Home > Economics - General > Can this census be saved?

Can this census be saved?

August 10th, 2016

It appears that, having crashed last night with only about 10 per cent of households having submitted data, the Census website is now off the air indefinitely. It’s hard for me to see how this exercise can be salvaged. Almost certainly, lots of people who tried and failed to fill in their forms last night will be unwilling to do so again, especially in the absence of any coherent explanation for the failure. It’s looking increasingly as if the only option will be to give up and try again in five years time. Coincidentally or not, a ten-yearly census was exactly what the leadership of ABS was suggesting a couple of years ago.

This fiasco seems to have “reform” written all over it, from the new entrepreneurial leadership of ABS to the contracting out of vital functions to the benign/malign neglect displayed by the Abbott-Turnbull government. Peter Martin is very good on this, as is Chris Graham at New Matilda.

Categories: Economics - General Tags:
  1. Matt
    August 10th, 2016 at 15:25 | #1

    Interesting to consider whether a 10-yearly census would increase the likelihood of these sorts of problems, though. For sure the technology available today is nothing like what was available in 2006, and in 2026 it will be different again. It would also be pretty difficult to retain any corporate knowledge between censuses – not only among the public servants who organise the thing but among the thousands of collectors and area managers who are engaged casually each time (my mum is one).

    None of that is a good argument for continuing to have the census every 5 years, but it’s a cost that needs to be considered.

  2. zoot
    August 10th, 2016 at 15:40 | #2

    I’m intrigued that the defence seems to be “There were DDOS attacks”. Surely 11 million people trying to access the site in a short period of time would look very much like a DDOS attack?
    Judging from Chris Graham’s article it seems that IBM has shafted the Australian taxpayer yet again.

  3. Newtownian
    August 10th, 2016 at 15:43 | #3

    Thanks for the Peter Martin and NM links. Three good things did eventuate.

    Firstly we found out the Bureau had actually kept our names lined up with the data for 18 months from the last census even though the spin suggested anonymity. A great betrayal of trust which Labor is clearly as responsible for as the coalition.

    Secondly we discovered government computing skills and outsourcing tendering processes in organizations where they should be best, are both pretty well shot. So as old Christopher Pyne reassured us, we dont need to be paranoid and wear a tinfoil hat, albeit because of government incompetence rather than any lack of malevolence and duplicity on their part.

    Thirdly perhaps the selfie obsessed and privacy naive younger generations will now understand the concerns of us older farts about giving all your personal data to the forces of darkness, and maybe go and read 1984 or leave Facebook.

  4. David Allen
    August 10th, 2016 at 16:05 | #4

    Also, see Naomi Klein “The Shock Doctrine”. F*ck it up then privatise it.

  5. Beethoven
    August 10th, 2016 at 16:07 | #5

    This fiasco seems to have “reform” written all over it, from the new entrepreneurial leadership of ABS to the contracting out of vital functions

    Once the ABS made the decision to put the census online, of course they contracted it out. What were they supposed to do, write their own software, in-house?

    As for the decision to contract it out to IBM (per the linked Graham article), the aphorism “Nobody ever got fired for choosing IBM” exists for a reason.

    All major IT projects are have initial stuff ups, always. It’s an iron law of nature. This is true of payroll systems, ticketing systems, databases of all kinds. Even Obama-care had major IT snafus, at least to begin with. The next online census in 5 years time will probably be smooth as silk.

  6. Collin Street
    August 10th, 2016 at 16:11 | #6

    In canada they basically abolished the census in extremely dubious circumstances, as I recall.

    Honestly? Census provides hard data. People-whose-personality-inclines-them-to-become-conservative-politicians don’t like hard data, because it has a nasty nasty tendency to be all biased and prove their ideas wrong.

    [as I’ve said before: I’m convinced that an impartial medical assessment would reveal that most of federal cabinet are not in fact legally qualified to vote.]

  7. GrueBleen
    August 10th, 2016 at 16:40 | #7

    Well I have to say that Malcolm Turnbull is totally consistent: starting with Grech and utegate, he hasn’t put a single foot right.

    And when we’ve finished agonizing over Turnbull’s many successes, can we look at Abbott’s success rate – particularly with Larry Marshall and CSIRO – which Turnbull’s lot will have to live with for a while to come.

  8. paul walter
  9. Willy Bach
    August 10th, 2016 at 16:59 | #9

    John, thanks. The trouble with neoliberal economics is that it relies on ideology, never evidence. So, they misread what is known as public good, preferring to believe that this kind of work can be done for a profit. Everything is a business. There have been many failures of this thinking, many ruined lives and public institutions.

    Once again, Turnbull’s bragging bunch of incompetents show us that they are not long for their positions of power. This government is fragile and may not see the year 2017. I can only say good riddance to this born-to-rule cohort of slaving neoliberal wreckers. Make it soon. The failed census is just a start.

  10. bjb
    August 10th, 2016 at 17:06 | #10

    @David Allen

    This was the first thing I thought – the Right will claim that privatising the ABS will solve this “problem”, notwithstanding the fact that’s it was essentially outsourced already to IBM.

  11. Ken_L
    August 10th, 2016 at 18:02 | #11

    The sad thing about the debacle is that it’s probably put online voting in elections back 20 years.

  12. MartinK
    August 10th, 2016 at 18:47 | #12

    @Ken_L
    ‘The sad thing about the debacle is that it’s probably put online voting in elections back 20 years.’
    As it should. People will believe the same incompetence would apply and they will be right.

  13. Peter T
    August 10th, 2016 at 18:50 | #13

    @Beethoven

    Pre Howard, most government IT was a mix of in-house and small-medium contractors for basic work. The in-house ranged from really very good (Social Security, Vets Affairs), through ordinary to poor – the usual range. Much was innovative, most was reliable, pretty much all were very large – routinely an order of magnitude bigger than most private systems. Pretty much all had very extensive interfaces with industry or clients, so was very reliant on the rest of the business for knowledge of partner reactions and business knowledge. Howard put almost all to large private tenders, won by IBM, Telstra, Accenture and a few others. The small and medium business lost out. Most staff transferred, then went on, so business knowledge was lost. The head contractors in turn contract, and staff turnover is high. Business design does not substitute for in-depth knowledge of the operation. IIRC, the Audit Office reckoned the cost to the Commonwealth of the new arrangements at 8 bn a year. But this cannot be unwound easily.

    Large IT stuffs up regularly, although the private debacles get much less press. This stuff-up is not unusual. But the arrangements make it hard to learn from it. Either ABS duplicates IBM, at extra cost, or goes on with IBM (or another). The next time round, the IT staff will have changed, their knowledge of the business be just as poor, and the contract terms no different.

  14. Collin Street
    August 10th, 2016 at 19:29 | #14

    The sad thing about the debacle is that it’s probably put online voting in elections back 20 years.

    On-line voting doesn’t work. I mean, a-priori: anonymity means breaking the connection between ballots and vote registrations, and you can’t do that and keep an audit trail. Anonymous or tamper-evident: one or the other.

  15. derrida derider
    August 10th, 2016 at 19:36 | #15

    “a ten-yearly census was exactly what the leadership of ABS was suggesting a couple of years ago”.

    Err, no – that was a Dept of Finance suggestion. The ABS publicly floated ways that the adverse impact of it could be minimised (on the assumption it was a fait accompli), but in private they were quite appalled. But Treasury’s vigorous opposition is what finally killed it.

    You can’t absolve those immediately responsible for this fiasco. I know, like and greatly respect David Kalisch but he has to take the fall – especially as these days his Ministers never will. But the underlying cause really is the death of the ABS by a thousand cuts – their ability to do any capacity investment has been degrading for a decade in the face of chronic budget squeezes.

    Of course, the beancounters will no doubt immediately say “they’re proven poor value – cut ’em again”. It’s an old story with government services – keep squeezing them over a long period until they can no longer deliver, then use that as an excuse to get rid of them.

  16. Beethoven
    August 10th, 2016 at 19:49 | #16

    @Peter T

    I don’t think what happened in the good old days is all that relevant. Back then, computers in Vet Affairs (which might have been just one mainframe in the basement) were not connected to any other computers. These days, every computer in the world is connected to every other computer. The expertise needed to make something on the scale of a census reliable, safe and robust is specialised and very few people know how to do it.

    I also don’t get why people are so worried about their privacy, on this occasion, when the government already holds much more sensitive information on its computers. The tax office holds information about their income, obviously, but also can find out about every dollar that has gone in or out of their bank accounts and super funds. The Medicare computers contain details of every doctor’s visit, and every procedure and operation everyone has ever had in public hospitals. But nobody worried about that information getting into the wrong hands. What’s so special about the census?

  17. Beethoven
    August 10th, 2016 at 20:13 | #17

    @derrida derider

    There is nothing remotely in Kalisch’s background (he’s spent his whole career in the public service) to suggest that he is “entrepreneurial”. He is not Larry Marshall.

  18. GrueBleen
    August 10th, 2016 at 20:37 | #18

    @Beethoven

    It wasn’t quite that primitive, Ludwig. Back in 1974 when I joined C’wealth Dept. Health as a trainee ADPer, the Pharmaceutical Benefits folks already had an Australia wide network of data entry minicomputers that transmitted PBS data to Canberra over PMG leased lines. Sure, the last stage of the process was to walk tapes over from the Canberra concentrator to be processed on the IBM mainframes, but that didn’t remain forever.

    But the thing about the Census (and you forgot Centre-Link which holds a lot of info on us pensioned oldies) is that as well as names, there is a complete address. Plus birthdates. Now who would like to suggest that a minor hack of the Telstra operated White Pages to put address and name data (divided up by age) together, wouldn’t also produce a lot of “confidential” data.

    So, who wants to suggest that we also don’t fill in our address or the ages of the residents and census night guests ?

    The difference was, back then, that networks were handled by privately operated leased lines – and to penetrate one of those you had to know which line to intercept in which PMG (Later Telecom) exchange.

  19. derrida derider
    August 10th, 2016 at 20:39 | #19

    Beethoven, you don’t know him. Believe me, he’s not a Sir Humphrey. Insofar as all this is the result of anything he did it’s almost certainly because he tried to be TOO entrepreneurial. Though as I implied most of the factors that led to this fiasco predated him.

    But he’s still gotta go – you take the money, you take the responsibility.

  20. Donald Oats
    August 10th, 2016 at 20:39 | #20

    I don’t think anyone should be taking the fall for this, not until such time as the mechanics of what went wrong, and why, are investigated. While I have some default antipathy towards LNP ministers, I really don’t think the current one can be credibly blamed for this shambles, and he shouldn’t take the fall—at least not on the information currently available.

    Still, it has been amusing to read some of the comments popping up around the traps. I won’t re-post any.

  21. Beethoven
    August 10th, 2016 at 21:29 | #21

    @derrida derider

    I am not defending Kalisch. If he has to fall in his sword, so be it. But the IBM contract was awarded before he was appointed as Statistician. The whole architecture of the census was a done deal while he was still running AIHW. He could hardly have put the whole thing on hold while thinking about whether it was a good idea when he started early last year. The census runs to a strict timetable.

  22. Douglas Hynd
    August 10th, 2016 at 21:35 | #22

    @Ken_L Nothing sad about that – current system works well why change?

  23. derrida derider
    August 10th, 2016 at 21:37 | #23

    Personalities aside, I do agree with Beethoven that this privacy malarky is tinfoil hat bullshit – Christopher Pyne is right for once:

    – The ABS go to enormous lengths to make sure it is not possible for users of their data to ever identify anyone – to the degree that it seriously inhibits usefulness of much of the more expensive stuff they produce. In over a century they have never had a leak about individuals.

    – Australia and NZ are the only developed countries that DON’T retain their census data indefinitely and its a non-issue elsewhere.

    – The ATO, Centrelink and various state and (especially) government bodies already know and retain your name and address and lots of other stuff about you that is far more sensitive than any Census question. And because these are mostly live accounts exposed to the web rather than an archived dataset on an isolated computer they are infinitely more liable to hacking than Census linking keys.

  24. paul walter
    August 10th, 2016 at 22:10 | #24

    Derrida Derider, you are worth your weight in gold. It is not so much about the census but an exercise in neoliberal privatisation/outsourcing/ secrecy ideological junketry and public conditioning…zombie economics and the surveillance state.

  25. jrkrideau
    August 10th, 2016 at 22:23 | #25

    @Collin Street
    The Cons (conservatives) did not completely abolish the census. We have a two tiered system with a short form and a long form.

    What they did was abolish the long form which is the one with the real meat. It supplies the key information that business and government planners can use to make sensible planning decisions and that allow for researchers to tease out what is happening.

    The Cons decided to replace the long form with a voluntary survey despite objections from any and everyone from banks to municipal planners to researchers. Response rates dropped from 90+% to 77% making valid inferences in many areas impossible

    The head of Statistics Canada resigned (in silent protest since as a loyal civil servant he did not attack the government)

    The decision to cancel appeared to be totally ideologically based though the Cons might have been cunning enough to realize that without the long form data it would be more difficult to attack some of the more egregious of their programs. Certainly the Cons never allowed facts to contradict ideology.

    I have an even longer rant on the earlier thread Should the census be compulsory?

  26. jrkrideau
    August 10th, 2016 at 22:30 | #26

    Re IBM designing the Census IT system

    Here in Canada IBM designed the new Federal Govenment pay system. Chaos has followed. Some people have not been paid for perhaps 18 months, some people have been wildly overpaid, too much tax has been deduced, no tax has been deducted

    The Government is scrambling madly to try to at least apply some patches. Of course it was a Conservative Gov’t who let the contract but the Liberals who are having to clean up the mess.

  27. Collin Street
    August 10th, 2016 at 23:15 | #27

    The Cons (conservatives) did not completely abolish the census. We have a two tiered system with a short form and a long form.

    Thank you for your clarifications.

  28. Donald Oats
    August 11th, 2016 at 01:46 | #28

    I wonder if this was part of the “disruptive innovation” that the current mob love pushing?

  29. Ernestine Gross
    August 11th, 2016 at 08:35 | #29

    @Beethoven

    I don’t agree with your critique of Peter T’s comparison of past and current operations. On the contrary, your argument rests on the assumption that ‘new’ is better and this is clearly not the case.

    Furthermore, Peter T’s post contains an example of how concentration of wealth occurs due to privatisation and how privatisation (IBM contract) reduces the ownership of technological knowhow within Australia. There are also negative BoP implications due to profit repatriation.

    As for privacy, is Beethoven your real name?

  30. Ernestine Gross
    August 11th, 2016 at 08:40 | #30

    “The Medicare computers contain details of every doctor’s visit, and every procedure and operation everyone has ever had in public hospitals.”

    Not true, strictly speaking. Doctor’s visits paid for privately and not claimed are not recorded in Medicare data systems.

  31. Ikonoclast
    August 11th, 2016 at 08:53 | #31

    Wow, this is great! <- Sarcasm.

    The Turnbull govt can't even run a census which is scarcely the most complicated thing a government needs to run. They have to be held responsible for it as it was they who gutted the ABS and set it on its new crash and burn path.

    I am baffled by some related issues. IBM has a long history of massive, egregious IT implementation failures. Why in blithering hellcat tarnations would anyone sign a contract with IBM for anything? Actually we might know. Let me ask a hypothetical. Would it be possible that there are secret promises of lucrative corporate gigs for key players, just a bit down the line when things quiet down and short public memory fizzles out?

    It's been a repeated feature of the neoliberal era that private enterprise's contract lawyers have clearly been far cleverer than the Australian PS contract lawyers, especially in the arena of IT. That is a failure which bears looking at. Governments should drive much harder bargains at the contract table to the point of placing sovereign rights conditions in contracts which conditions government reserves the right to activate at any time. Possibly, the best candidate strategy for implementing and activating such conditions would be to extend the concept of odious debt. In international law, odious debt, also known as illegitimate debt, is a legal theory that holds that the national debt incurred by a government for purposes which do not serve the best interests of the nation, should not be enforceable. In the case of egregious failure imperiling or damaging the nation, the government or their proper operations, the corporation should pay extensive damages to the state.

    Overall, this current fiasco is just another example of late stage neoliberal capitalism doing what it does best. It deliberately sabotages the democratic state via projects which the state pays for. Then it forces the state (meaning the people) to pay again for a privatised and vastly inferior product. It's exactly as Naomi Klein said. The plan is to sabotage it and then privatise it. And to make the people pay for every fiasco along this path.

    The institution to blame for this is not that of democratic state governance as such. It is the set of institutions and corporations which constitute the operations of neoliberal corporate capital and give them outrageous advantages over the rights of the people and their democratic government. The blame must be sheeted to the correct place.

    However, I have little to no hope for the time being. These forces will continue wrecking our society and world for a long time to come. They are still in near complete ascendancy and the bulk of the people simply do not yet understand what is happening and where the real problem lies.

  32. Beethoven
    August 11th, 2016 at 09:18 | #32

    @Ikonoclast

    Why in blithering hellcat tarnations would anyone sign a contract with IBM for anything?

    Because nobody ever got fired for choosing IBM. If you’re a risk averse public servant – and public servants are nothing if not risk averse – and you choose IBM, and the project goes tits up, you can always fall back on: “It’s IBM. They’ve been around for decades and are the number one company in the world for these kinds of projects”. Whereas, if you choose an unknown company and it goes wrong, then your career is in the toilet.

    Plus, in this case, IBM did the job for only $9 million, an unbelievably small sum for a job of this magnitude and importance. They were probably the cheapest by a long way.

  33. Beethoven
    August 11th, 2016 at 09:19 | #33

    @Beethoven

    #Formattingfail

  34. tony lynch
    August 11th, 2016 at 09:50 | #34

    @Beethoven
    I like Beethoven’ take on all this.. i) It went as brilliantly as could be expected; ii) It will be smooth as silk next time; iii) TINA. Boy with a future here.

  35. Peter T
    August 11th, 2016 at 09:58 | #35

    Beethoven

    First – Government pioneered IT interfaces (Social Security had to interface with the ATO, Finance and banks; all departments with Finance; Customs with airlines, freight, brokers, shipping, banks, ATO, Immigration, TGA…, Vets Affairs with Social and ATO and so on). Been a long time since it was a mainframe in a basement.

    Second – there are about 5 or 6 large IT firms, all much the same. Cost differences are marginal. One screws up, you get another – but staff transfer. The problem with outsourcing the whole of IT (rather than, say, maintenance, back-ups and such) is that you lose the understanding of the business. And you have no control over costs or timing. This is why large organisations keep core functions in house. Howard insisted all departments outsource the whole, and yielded for cases where the pushback was very strong (eg CSIRO and similar tech agencies know more about computers than IBM ever will, and their people fiddle with them all the time; some parts of Defence ditto, plus security). In many cases, we have ended with the worst of both worlds – a rebuilt in-house capability struggling to control contractors who have their hands on the actual equipment and control over access and changes.

  36. Ikonoclast
    August 11th, 2016 at 09:59 | #36

    @Beethoven

    If one has a low confidence in the automation process and a low confidence in the electronic security of the automation then one should retain manual processes which are historically proven to work successfully. This is the case when “one” is government and cost concerns are not (and should not be) the only metric for the project. Quality and security of the data and successful prosecution of the collection process are more important than cost.

    In any case, extra government spending on manual processes lifts employment. There is always a benefit to this when in a prolonged employment recession as we are now. Australia’s true rate of unemployment is much closer to 10% than it is to the rank lie peddled of 6%. See Morgan polls and Bill Mitchell on this topic.

    In J.Q.’s previous blog post on this census, I mentioned that the best data security can be obtained by combining physical security and electronic security. I wrote:

    An overall theory of data security must include the theory of “hybrid” security setups which employ physically secure processes and electronically secure processes in concert. For some applications it is not always wise to switch entirely to electronic data at all steps and thus restrict the system to the limitations of electronic security only. The census is a good example. It would be possible to maintain paper form collection, then use data entry into a mainframe not connected to the internet. Research of the census data could then proceed via government user requests into some sort of statistical analysis system which would run batch jobs on the unconnected mainframe. There is no genuine need for real-time queries of, or an internet connection to, census data, even for government analysts.

    The extra costs of such a system would not be important. Indeed, extra employment would be generated by the manual process components. If extra automation creates data vulnerability and more unemployment, then the savings and benefits overall are largely or even completely illusory. Costs and risks will be shifted to other parts of the economy and onto ordinary citizens. The mania for total automation and cost-cutting is the real problem not the compulsory nature of the census and the collecting of names.

    In summary, a major problem is in the blind assumption that more automation is always better and that cost-cutting goals should trump all other goals. How many times have we seen short-sighted cost-cutting result in more costs in the long run? Innumerable times would be my answer. This fiasco is another example.

  37. Ernestine Gross
    August 11th, 2016 at 10:18 | #37

    “Plus, in this case, IBM did the job for only $9 million, an unbelievably small sum for a job of this magnitude and importance”

    $10million according to smh. A 10% error is too big!

  38. Beethoven
    August 11th, 2016 at 10:19 | #38

    @Peter T

    I don’t disagree with any of this, as far as it goes. But the situation here was millions of people logging in to the ABS via the web to upload information. When has the public service ever had the expertise in that kind of IT?

    @tony lynch

    Pretty much. Millions of people and organisations send their tax returns to the ATO’s computers and it all seems to go OK. The same will be true for the next Census. The trick is to have a big enough computer to handle the traffic, which means not doing it on the cheap.

  39. Ernestine Gross
    August 11th, 2016 at 10:25 | #39

    @Peter T

    “The problem with outsourcing the whole of IT (rather than, say, maintenance, back-ups and such) is that you lose the understanding of the business. And you have no control over costs or timing.”

    Exactly. This is an example of the shortcomings of the ‘privatise and regulate’ policy framework. How are government departments supposed to regulate something they don’t know? That is, effective regulation presupposes the same or better knowledge of the technology that underlies an ‘industry’ which is to be regulated.

    (Education, health, banking, utilities provide similar examples.)

  40. Jim
    August 11th, 2016 at 10:59 | #40

    I’m not sure the problem is necessarily “the new entrepreneurial leadership of ABS”. but that was possibly a contributing factor.

    IMHO the public service (Commonwealth and the states) has lost much of its professional expertise and knowledge as most of the technical work is now contracted out (IT, science, economics etc.). The senior ranks are full of people who are able to meet job criteria like “strategic communicator”, but portfolio-specific skills or content knowledge is never a priority. The public service is now full project management types that are poorly equipped to deal with even remotely technical issues.

  41. Peter T
    August 11th, 2016 at 11:12 | #41

    For those interested, the lesson on outsourcing is over 200 years old. The Royal Navy contracted much shipbuilding, maintenance and supply but, after a couple of early lessons, always kept an in-house capacity both as a fall-back and a source of the necessary technical and pricing knowledge. The one area where private suppliers were able to make large profits was armour plate, where large initial costs plus economies of scale kept the RN from building its own capacity.

    Really, the reliance of much economics on very simple notions of competition is a standing intellectual reproach.

  42. Ivor
    August 11th, 2016 at 11:17 | #42

    I think it was a fiasco for two reasons. 1) As hinted by Derrider, cuts to ABS and the need to show a efficiency dividend – if this still exists.

    2) Failure to develop new system by modules and and development and test systems. They should have developed experience by for example saying in 2016 only ACT addresses have the option of online census. Next time expand the scope.

    There is no reason for anyone to resign except get rid of capitalist motives in society – the root cause.

  43. August 11th, 2016 at 11:30 | #43

    OOps I used a bad word in my comment so here it is again.

    I’m backing up Peter T here – I worked for NSW Health for a little while over the period of the Y2K bug and back then most sensitive IT stuff was done in-house (I was responsible for managing my little part, which worked really well I might add). This included various state-wide networks that held confidential patient data. When things went tits-up they usually did so for reasons unrelated to IT expertise or management (for example, rolling out a shared database of patient information across the whole of my health district was technically well within the abilities of the in-house IT, but various hospitals and clinics blocked it for privacy reasons). Our experience with outside contractors was mixed but they were usually poor – the sole exception being some specialists we brought in for Y2K. I remember back in the 1990s NSW Health already had the HOIST system which was a centralized database of health information that you could query from anywhere in the state to run analysis on.

    What we built wasn’t rocket science and it wasn’t pretty but it got stuff done and it got it done well. If the federal govt had maintained that capacity over the past 20 years, this census madness shouldn’t have happened.

  44. David Allen
    August 11th, 2016 at 11:59 | #44

    So far there have been unsubstantiated claims of hack, ddos, attack etc. from the ABS, the minister, &Turnbull. I’ve seen no evidence yet & those experts that have looked for it have found none. This blame shifting to anonymous parties just makes their incompetence look worse. “Our” ABC has run the attack line solidly despite the lack of evidence which will rebound on them as well.

    Millions of Australians accessing the census website after dinner, hitting F5 to refresh constantly, on Tuesday night is exactly what a ddos would look like.

  45. Beethoven
    August 11th, 2016 at 12:23 | #45

    @David Allen

    What is amazing is that the ABS seems to have assumed in its capacity planning that the load would be spread evenly across Tuesday when it was obvious that 95% of households in the eastern states would all be doing the census between 7.30 pm and 10 pm on Tuesday night.

    Turnbull’s comments today bode badly for David Kalisch. Just as the head of the AEC resigned after the lost WA Senate votes fiasco, even though he didn’t personally lose the voting papers, Kalisch might have to go too. As DD said upthread, this comes with the territory when you get paid the big bucks.

  46. Ken_L
    August 11th, 2016 at 12:49 | #46

    The fact that the site is STILL down has turned this affair into a complete farce. How long before they have to concede that no action will be taken to fine people who’ve spent days trying in good faith but unsuccessfully to complete their returns?

    I would hate to be one of the collectors who has the unhappy task next week of visiting households to ask why they haven’t submitted their census form yet.

  47. Beethoven
    August 11th, 2016 at 12:57 | #47

    @Ken_L

    The problem is once they tell people it’s back up, they risk everyone rushing onto it all at once, which will put it back down.

  48. may
    August 11th, 2016 at 13:26 | #48

    found the form at my front door a couple of days before the 9th.

    forgot to fill it in on the 9th

    filled it in next morning.

    stuck it in the letterbox.

  49. Brenton
    August 11th, 2016 at 14:49 | #49

    @Beethoven Its a while since I actively worked in network capacity planning so they may very well have tools to address this now. The problem that is often not taken into account is the number of retries that will occur from failed logins. So if we assume that 100 users will try and login within a given time but 50 percent of those attempts fail and the users try again, say 5 times (not unreasonable) then within that given time you have the 10 attempts plus 5 x 5 attempts or 35 login attempts all up. It is a cascade affect that rapidly increases the traffic form a predictable flow to a massive increase when things do not go smoothly.

    Of course I would think the boffins designing the ABS system would well understand this problem and it would have been taken into account. But, the information now dribbling out and crab walking away from a foreign DOS attack has dented this confidence.

  50. Brenton
    August 11th, 2016 at 14:52 | #50

    Spot the deliberate mistake! Out by a factor of 10, should be 350 login attempts. Maybe that is why I don’t work in this area anymore.

  51. GrueBleen
    August 11th, 2016 at 16:52 | #51

    @Ikonoclast

    It isn’t just IBM that has a “…a long history of massive, egregious IT implementation failures.”, Ikono, it’s just about everybody.

    You see, the reason we don’t learn from the mistakes of history is that mostly we weren’t around when the mistakes were made (and understood – Root Cause Analysis, anybody ?).

    But for the very few who were, I have just one word: Mandata 1973 (oh alright, one word plus one number). By the way, how’s your experience with Gantt Charts, PERT Charts and Critical Path Analysis ? Anybody ?

  52. Peter Chapman
    August 11th, 2016 at 18:05 | #52

    I am one of the few who seemingly were able to complete the forms online, submit and receive an emailed receipt, before all the problems occurred. What struck me was the continuing banality and superficiality of the questions, with nothing innovative compared with previous Censes. Questions about weekly income with no corresponding questions about assets and wealth, for example, can give no meaningful data about the distribution and scale of wealth. The data generated by this exercise are really very limited from the perspective of public policy. ABS gets better data for housing and social policy purposes, for example, through surveys, which can be completed more quickly and more often, and which deliver usable data in a shorter time frame. Better perhaps to spend the Census millions on even better surveys, with a less-frequent attempt at a full count for bench-marking purposes.

  53. Ikonoclast
    August 11th, 2016 at 19:47 | #53

    GrueBleen,

    We learned to reliably build skyscrapers, dams, bridges, ships, passenger aircraft etc. very well once they become mature technologies. There are relatively few catastrophic failures of these projects in developed nations. Yet we have failed to bring to maturity the techniques of implementing large scale computer systems. They fail time after time. As you say “it’s just about everybody”. I wonder why this is?

    In many engineering projects, but not often IT projects, people die if the designers and builders get it wrong. Maybe that factor is what concentrates minds and legislatures. The only possibility for achieving a similar concentration of project management minds in IT projects would seem to be very steeply rising penalties for failing to achieve deliverables on time and to specification. In other words, miss a key milestone for performance as has happened in the census project and the developer would lose a large percentage of the contract. I would suggest since only 10% data was collected by the deadline, IBM should only get 10% of the contracted monies. This would have had to have been in the contract of course. The chance to recoup a large percentage of the contract monies would then swing on excellent repair and support of the system for the next 5 years and a fully compliant performance the next time round. If private enterprise won’t sign such contracts then build up national in-house capability and build things in-house.

    As I have touched on at least a couple of times, there seem to be systematic failures in government contracting and serious weaknesses in public-private contract law. Some might find this article interesting.

    Contract Theory and the Failures of Public-Private Contracting – Wendy Netter Epstein

    http://scholarship.kentlaw.iit.edu/cgi/viewcontent.cgi?article=1205&context=fac_schol

  54. GrueBleen
    August 12th, 2016 at 02:57 | #54

    @Ikonoclast

    Your #53 of 11 Aug

    Fortunately, although it is mostly everybody, it isn’t every time … failures I mean. I spent 34 years in the ADP/EDP/IT/ICT industry (whichever acronym you prefer) and I never took part in a genuine disaster. Nor even a sizable failure. But that was definitely the luck of the game, because quite a few have happened in the time of my professional involvement.

    And the funny thing is that IBM – which I worked for 13 years – had the best methodological armory in the profession, and had implemented some sizable successes, even in Australia. But it seems that those salad days are long gone now, and one would have to say that IBM hasn’t succeeded in it’s transition from a mostly hardware – and hardware related software – company into a full spectrum IT services company, even after it acquired the PwC IT Consulting arm back in 2002 (for a measly US$3.5 billion according to the web).

    Conceptually, I’ve always liked the categorisation of “manufacturing” by Peter Drucker, viz
    [Cottage industry – which Drucker doesn’t actually mention, but I include for completeness]
    Unique product – eg large buildings, large vessels etc
    Single product mass production – usually from ‘raw’ materials into a single finished product, often used as inputs to =>
    Variable product mass production – ie TVs today, computer monitors tomorrow, same assembly line, a few different components
    System products – eg a refinery or a general hospital where the system is the product.

    That all makes sense and each kind of manufacturing has its own rules and success criteria, except that large scale IT partakes of a weird combination of cottage industry, unique product and system product. No two large buildings are identical, nor are most large scale vessels, but both of them utilise some very ‘standard’ components (not entirely dissimilar to ‘multiple product mass production’ in many respects). Large IT systems do not – unless they are large scale implementations of commercial software packages – eg SAP.

    In short, lots of opportunities all along the way for IT projects to ‘go wrong’, and many of them take full advantage of those opportunities.

    But I agree about vwery serious weaknesses in contract management and I’ll follow up your link a little later. In my experience, it’s almost a case of ‘common law’ versus ‘statute law’. Contracts partake of the nature of statute law – they attempt to set out the details of what is to be done, by when and for how much etc. But like all statute law, a contract cannot cover everything in micro detail – there is always an ongoing process whereby the vendor/supplier tries to limit commitment to precisely what the contract says and the customer tries to invoke the ‘reasonable expectations of a rational man’ to overcome imprecisions and omissions in the contract.

    If that whole process isn’t handled by rational adults of reasonable goodwill, disaster often follows. As we all have noticed.

  55. Ikonoclast
    August 12th, 2016 at 04:55 | #55

    GrueBleen,

    Interesting points. Just to be a snark, a product like an automobile is technically a system too; although I agree it sits somewhere in the range from single mass production product to variable mass production product.

    I could not understand a failure like the Qld Health pay system. I mean it’s not like it would be the first pay system ever built in the world. Indeed, it’s not like it would be the first state or national pay system ever built in the world. Such a projecy should be known territory with tried and tested modules and systems. It’s theoretically not even a difficult or unknown problem even given multiple pay rates, multiple awards and complex rosters. None of these things are unknown. These all should be well known parameters with an understanding of how to incorporate them. I would have thought SAP could handle it with relatively little tailoring of existing system modules.

    Speaking of SAP;

    “SAP wins first big Centrelink systems overhaul deal”

    http://www.itnews.com.au/news/sap-wins-first-big-centrelink-systems-overhaul-deal-432168

    Centrelink has run ISIS (Income Security Integrated System) for many years now. It is an in-house built system based on Model 204. Model 204 has worked very well. It was built when humans still knew how to build stuff. (OK, slight sarcasm there.) ISIS itself worked (and works) well considering that it was forced to be insanely complicated by insanely intricate welfare legislation from a series of idiotic governments of both colours and of course the sheer size of the database(s). Just about everyone in Australia is on it in one way or another; every child, every parent, every partner, every welfare recipient… that is to say basically everybody. IIRC, Centrelink was/is just about the last customer in the world still using Model 204 and it was being supported just for them.

    I have more confidence in SAP than in anything IBM does now. I have less confidence in SAP than in a legacy database built when human actually used to know how to build stuff. (There’s that sarcasm again.) Basically, I think we can expect mammoth screw-ups on an unprecedented scale from this project. Something has been lost. The competency to do large projects is declining worldwide. I think neoliberal managerialism and anti-science ideology is systematically destroying competency in the professions and trades. It’s a form of civilizational decline. It’s unfashionable to point out civilizational decline in the modern context but historically the rise and fall of civilizations is standard behaviour. Ours is declining, falling apart, IMO.

  56. Ikonoclast
    August 12th, 2016 at 05:11 | #56

    Footnote to above:

    “Model 2014 technology is supplied by a niche US provider, whose only remaining customers are Australia’s DHS and the CIA”.

  57. GrueBleen
    August 12th, 2016 at 09:24 | #57

    @Ikonoclast

    Your #55 of 12 Aug

    Yeah, lots of things these days are a bit more mixed up than in the simpler days of Peter Drucker’s time. Though I think cars are substantially ‘variable product mass production’ – especially back in the long ago days when GMH advertised “millions of different Holdens – choose your own variety” – but in these days of computerisation and GPS etc they are taking on more of the attributes of a ‘system’. The “millions of Holdens” thing was quite a while ago now (at least 3 decades I think) and it was a failure at the time – most people couldn’t be bothered “creating” their own “unique” Holden, so IIRC a company that just presented a well designed and well fitted out single ‘standard’ model started to climb the sales total count. And so began the era of Toyota, which indeed concentrated on a “single product mass production” and implemented the effective quality control to go with it.

    But pardon my flights of nostalgia, however, you’ve sent me on another one with your mention of Model 204. I used to be a DBA/DCA for some time, concentrating on IBM’s IMS, the Fujitsu’s AIM and later IMS ‘Fast Path’ (for banking online systems) the IBM’s DB2 and finally, to a somewhat lesser extent, Oracle. But Model 204 was the choice of the last holdout of ‘computer database purists’, and it was already moribund then. I was, by the way, heavily involved in Dept. of Employment (and Other Things) systems from the initial JOB Bank online system in Melbourne to the full Job Bank system in Canberra – that was done in Fujitsu’s AIM and I was a Fujitsu SE (Systems Engineer) at the time. The system was developed in COBOL, had-written by Departmental employees plus some ‘external’ help (it was the last time I ever wrote any significant amount of IBM style Assembler code 🙂 ) It was later replaced, quite effectively I think, by IBM when Dept. Employment (and Other Things) replaced their Fujitsu mainframes with IBM mainframes and a DB2 system.

    Could SAP be used for a payroll system ? But of course, and as a later IBM ‘consulting’ SE i was heavily involved in the implementation of Telstra’s HR-Pay system (back when Telstra still had around 70,000 employees Australia wide and they all, by labour agreement, had to receive a printed payslip every fortnight). So it goes.

    So, very large scale ‘all of enterprise’ packages like SAP have certainly changed the computer systems landscape, and yes, large scale inhouse development is a long faded art. How much that has to do with “neoliberal managerialism and anti-science ideology” I’m not sure – though I’d agree there is some connection. But I think it has a lot more to do with the thoughts that Fred Brooks (originally Software Development Manager and later the total boss of the IBM S360 project which catapulted IBM to the fore) expressed in his famous essay “No Silver Bullets” which was incorporated into the 2nd edition of his at least equally famous book “The Mythical Man Month” (which was almost my standin for a bible for some years).

    Brooks’ point was that large computer system development was simply irreducibly complex (how long did it take SAP to get up to scratch – 20 or more years ?). And when something is “irreducibly complex”, there are no ‘silver bullets’ to slay the complexity werewolf. But we keep pretending and hoping that there are (and Model204 and later DB2 and Visual Basic and … are expressions of our lasting hopes and pretenses).

    Thank you for providing the opportunity for my nostalgic rave, Ikono. Nobody much is interested in such things nowadays 🙁

  58. Ikonoclast
    August 12th, 2016 at 10:39 | #58

    @GrueBleen

    “Neoliberal managerialism and anti-science ideology” as I call it, has been deliberately deployed since the 1970s to degrade the effectiveness of democratic government outside those functions expressly required to guarantee capitalist relations and corporate. oligopolistic profit.

    From the Omega Project (Adam Smith Institute) onward, neoliberal theorists developed a systematic program for retrenching the welfare state and social democracy to ensure the claimed full rule of so-called “market forces”. It turns out, of course, that it is not market forces (an incomplete economic piloting system anyway) which are unleashed but corporate oligopolistic dictatorship of the TINA (there is no alternative) variety.

    Anti-science ideology is applied selectively. Production science, where it enables corporate profits and ever greater concentration of wealth and power in fewer and fewer hands along with military and police power to ensure the same ends, is encouraged and subsidised. Impact science (environmental science, climate science) is the science which is attacked and which has its institutions, agencies, funding sources and public credibility destroyed.

  59. Ikonoclast
    August 12th, 2016 at 10:45 | #59

    Footnote to above:

    “Retrenchment is generally an exercise in blame avoidance rather than credit claiming, primarily because the costs of retrenchment are concentrated (and often immediate), while the benefits are not.” – Paul Pierson.

    Pierson’s observation explains another reason why privatisation is a favoured ploy of politicians. (I mean in addition to lining the pockets of their supporters and themselves.) Privatisation facilitates political blame avoidance. Privatise power or mass public transport and then costs and failures can then be attributed to the “market”. The politicians are no longer directly to blame for anything. That’s the way uh huh, uh huh, they like it!

  60. derrida derider
    August 12th, 2016 at 14:30 | #60

    Yep, GrueBleen, one of my favourite aphorisms is Einstein’s “things should always be made as simple as possible – and no simpler”.

    Air safety investigators always proceed on the principle that disaster is rarely caused by a single big mistake – usually it’s a whole series of interlocking smaller ones. Such is certainly the case here, but I think the main mistakes by the ABS were:

    – failure to anticipate how a change with really minor privacy or security implications (giving yourself four years instead of eighteen months to check that you’ve turned names and addresses into anonymised SLKs) would be distorted by politicians on the make keen to exploit paranoia. Bluntly they needed to spend a lot more on PR and also hire a better advertising agency.

    – accepting an obvious lowball bid from IBM (did the tender panel really believe they could get a robust solution so cheap?)

    – the decision on the night to pull the whole site down when the odds of a successful DDOS attack increased as the geoblocking server went down (even Turnbull called this “an abundance of caution” – I call it a disastrous failure to assess comparative risk).

    Whatever, though, I think the answer to the OP’s original question is “no – it can’t be saved”. I wouldn’t like to be the hapless methodologists charged with converting the raw census data into something that is fit for statistical use.

    This is a disaster that extends well beyond the census, because the census data is used to design and benchmark just about every survey (public or private) in Australia. I don’t like the chances of getting accurate data on unemployment in a few years time, for example. Even political opinion polls will become less accurate (those of them that are fair dinkum, anyway). And expect a huge number of court challenges to electoral boundaries. Note all these are exactly the arguments people successfully made against ten year censuses.

  61. Beethoven
    August 12th, 2016 at 14:42 | #61

    @derrida derider

    It’s even worse than that. Everything the ABS does will now be treated with scepticism at best, contempt at worst. It took the ABS many decades to build its reputation as an unimpeachable statistical agency, and one night to wreck that reputation.

  62. GrueBleen
    August 12th, 2016 at 16:52 | #62

    @derrida derider

    Einstein also said things (supposedly) about World Wars that I found entertaining, too, DD.

    But “as simple as possible” isn’t an unique point solution, it’s a stage in a long process.

    For instance: “pay the employees their salary” is about as simple as the efinition of a payroll system can get. But (at least once upon a time) it probably took several million lines of COBOL (plus JCL, utility control statements etc) to take that statement to an actually operational level of “simplicity”.

    That said, I think you have basically covered the ground and there is nothing quite so pathetic as rapid response action forced on the unready (cf the Man Monis siege).

    I dunno, but apart from unrealistic expectations of what the present day IBM could deliver, and especially so cheaply, it just doesn’t sound as though anybody designed a scenario based test plan that worked through a Component Failure Impact Analysis. Such as we used to do maybe 20 – 30 years ago, when there was still a significant inhouse expertise and staffing level.

  63. GrueBleen
    August 12th, 2016 at 17:00 | #63

    @Ikonoclast

    Your points are taken, Ikono, but I’m not sure I’d credit it all to the wit and wisdom of the “neoliberals” (or econorats as I prefer to call them).

    I think the process has been aided – as is always the case – by lots of “useful idiots” who just take up the current fashion as though it was the delivered word of almighty truth. But that was ever thus, yes ?

    It’s been a long descent from “the light on the hill” though, hasn’t it.

  64. Greg McKenzie
    August 14th, 2016 at 08:49 | #64

    Perhaps this is also the hidden agenda for Medicare. If you strip it of personnel and
    Make it harder to get any service, then you drive people into private health insurance.
    A tax penalty for not having private health insurance certainly cannot hurt this plan.

  65. Ikonoclast
    August 14th, 2016 at 09:31 | #65

    Greg McKenzie,

    Certainly, this is a deliberate part of neoliberal strategy. When the CES (Commonwealth Employment Service) was abolished in favour of privatised job search providers, Employment National was also set up. It was the government provider which (sort of) replaced CES. There are studies and media reports pertinent to the time which argue that Employment National was deliberately set up to fail. And indeed, it soon did fail leaving the field wide open for private providers. The private job search industry created new millionaires like Kevin Rudd’s wife and made companies like Sarina Russo more lucrative.

    It is a standard neoliberal technique to starve public services of operational funds. Then they are criticised for being poor services and the push starts to privatise them because private enterprise “does everything better”.

    Here is Albanese criticising this closure of Employment National in 2000.

    http://anthonyalbanese.com.au/pm-closure-of-employment-national

    What is instructive is that when Labor gained power later (Kevin 07), they did not change the system back to a government system. Wikipedia tells us; “The sale of the Australian arm of Ingeus (Therese Rein’s company) took place in May 2007 to ensure there was no perceived conflict of interest as her husband, Kevin Rudd was the Leader of the Opposition (and later the Prime Minister of Australia). The Australian businesses sold in October and December 2007. Ingeus re-entered the Australian market with the acquisition of Assure Programs in October 2011.”

    One can see in retrospect that Rudd and Rein had a long term interest in not repealing or reversing the privatisation of employment services. Thus the conflict of interest, while technically and legalistically resolved for the duration of Rudd’s first Labour leadership, was not substantively, permanently or ethically resolved. This raises the general issue of how the “Labor Traitors”, as I call the entire Labor Party modern professional political class, have the same interests as all people of the neoliberal class. The Labor Party’s modern professional political class are neoliberals through and through.

    Neoliberalism is bi-partisan policy whereby the LNP implement the new neoliberal policies. The ALP pretend to oppose these policies in opposition. When the ALP gets into power, they go quiet on these policies for the most part and allow the neoliberal policy innovations to remain in place. They might throw out a few cosmetic sops to con or appease Labor voters. Thus neoliberal policy gets ever ratcheted to the right. The LNP moves the policy to the right each time and the ALP is the ratchet stop which holds the policy at that level until the LNP can get into power again.

    Given that big business in Australia donates to both parties, there is very likely a strategic policy understanding and intent in neoliberal circles for the entire dual main party system to function in this manner. There is a strategic manipulation and subornation of our entire mainstream politics by big business interests. Personally, I think it is the height of naivety to think that modern capitalist politics is not characterised by this level of strategic intent, insight, planning and Machiavellian spirit.

  66. GrueBleen
    August 14th, 2016 at 15:46 | #66

    @Ikonoclast

    As I once said to someone in here, Ikono, “Its been a long way down from the light on the hill”. Oh yes, that was you !

    But yes, look at the Labor leadership since then: Arthur Caldwell, Gough Whitlam, Bill Hayden (as least he brought Medicare/Medibank in), Bob Hawke, Paul Keating, Simon Crean, Mark Latham, Kim Beazley, Kevin Rudd, Julia Gillard, Bill Shorten. As hopeless a lot as you could find anywhere, any time.

    However, surely I don’t have to remind you that the LNP was no better ? Robert (Pig Iron Bob) Menzies, Harold Holt, John Gorton, Billy “Big Ears” McMahaon, Billy Snedden, Malcolm Fraser, Andrew Peacock, John Howard, Andrew Peacock, John Hewson, Alexander Downer, John Howaqrd, Brendan Nelson, Malcolm Turnbull, Tony Abbott, Malcolm Turnbull. Even worse, maybe, than the Labs !

    In short, I don’t trust any of them, with the possible exception of John Hewson of having the “smarts” to enact the “conspiracy” you think has been going on.

    So, are we really ruled by the Bilderbergers after all, or, as one of my now deceased friends used to say” “Have you ruled out stupidity ?”

    I haven’t. I just see an ongoing stream of intelligence-restricted (in both senses of ‘intelligence’) careerists who keep doing what we keep on voting them in to do.

  67. GrueBleen
    August 14th, 2016 at 16:12 | #67

    @Ikonoclast

    Ooh, just a wee bit of serendipity after I posted the above: do you read the Club Troppo blog ? If not, you should because it’s often worth a read, especially for Nick Gruen – who is a reformed PubServe and one time “privatization” believer. Now he’s just a touch more skeptical.

    Anyway, you may enjoy his latest post, just Google:
    Club Troppo Choice, competition, markets and human services: Some thoughts

    (I try to avoid URLs unless absolutely necessary as it often takes John several days to “moderate” my post when I include links in them).

  68. Ikonclast
    August 14th, 2016 at 17:11 | #68

    @GrueBleen

    Of course the LNP are even worse than the ALP. I take that as being so obvious that I don’t even bother mentioning it. 🙂

  69. J-D
    August 14th, 2016 at 17:14 | #69

    Good to see Arthur Caldwell in the same list as Billy McMahaon and John Howaqrd.

  70. GrueBleen
    August 15th, 2016 at 00:30 | #70

    @Ikonclast

    Yair, but I didn’t realize, until I typed that list in, just how much the Libs have “blown in the wind” over the years.

    But do look up that Club Troppo article, I think you’ll enjoy it.

  71. GrueBleen
    August 15th, 2016 at 00:32 | #71

    @J-D

    So good to have you on the job so diligently, J-D.

    You don’t live in NZ I take it.

Comments are closed.