Can this census be saved?

It appears that, having crashed last night with only about 10 per cent of households having submitted data, the Census website is now off the air indefinitely. It’s hard for me to see how this exercise can be salvaged. Almost certainly, lots of people who tried and failed to fill in their forms last night will be unwilling to do so again, especially in the absence of any coherent explanation for the failure. It’s looking increasingly as if the only option will be to give up and try again in five years time. Coincidentally or not, a ten-yearly census was exactly what the leadership of ABS was suggesting a couple of years ago.

This fiasco seems to have “reform” written all over it, from the new entrepreneurial leadership of ABS to the contracting out of vital functions to the benign/malign neglect displayed by the Abbott-Turnbull government. Peter Martin is very good on this, as is Chris Graham at New Matilda.

71 thoughts on “Can this census be saved?

  1. Interesting to consider whether a 10-yearly census would increase the likelihood of these sorts of problems, though. For sure the technology available today is nothing like what was available in 2006, and in 2026 it will be different again. It would also be pretty difficult to retain any corporate knowledge between censuses – not only among the public servants who organise the thing but among the thousands of collectors and area managers who are engaged casually each time (my mum is one).

    None of that is a good argument for continuing to have the census every 5 years, but it’s a cost that needs to be considered.

  2. I’m intrigued that the defence seems to be “There were DDOS attacks”. Surely 11 million people trying to access the site in a short period of time would look very much like a DDOS attack?
    Judging from Chris Graham’s article it seems that IBM has shafted the Australian taxpayer yet again.

  3. Thanks for the Peter Martin and NM links. Three good things did eventuate.

    Firstly we found out the Bureau had actually kept our names lined up with the data for 18 months from the last census even though the spin suggested anonymity. A great betrayal of trust which Labor is clearly as responsible for as the coalition.

    Secondly we discovered government computing skills and outsourcing tendering processes in organizations where they should be best, are both pretty well shot. So as old Christopher Pyne reassured us, we dont need to be paranoid and wear a tinfoil hat, albeit because of government incompetence rather than any lack of malevolence and duplicity on their part.

    Thirdly perhaps the selfie obsessed and privacy naive younger generations will now understand the concerns of us older farts about giving all your personal data to the forces of darkness, and maybe go and read 1984 or leave Facebook.

  4. This fiasco seems to have “reform” written all over it, from the new entrepreneurial leadership of ABS to the contracting out of vital functions

    Once the ABS made the decision to put the census online, of course they contracted it out. What were they supposed to do, write their own software, in-house?

    As for the decision to contract it out to IBM (per the linked Graham article), the aphorism “Nobody ever got fired for choosing IBM” exists for a reason.

    All major IT projects are have initial stuff ups, always. It’s an iron law of nature. This is true of payroll systems, ticketing systems, databases of all kinds. Even Obama-care had major IT snafus, at least to begin with. The next online census in 5 years time will probably be smooth as silk.

  5. In canada they basically abolished the census in extremely dubious circumstances, as I recall.

    Honestly? Census provides hard data. People-whose-personality-inclines-them-to-become-conservative-politicians don’t like hard data, because it has a nasty nasty tendency to be all biased and prove their ideas wrong.

    [as I’ve said before: I’m convinced that an impartial medical assessment would reveal that most of federal cabinet are not in fact legally qualified to vote.]

  6. Well I have to say that Malcolm Turnbull is totally consistent: starting with Grech and utegate, he hasn’t put a single foot right.

    And when we’ve finished agonizing over Turnbull’s many successes, can we look at Abbott’s success rate – particularly with Larry Marshall and CSIRO – which Turnbull’s lot will have to live with for a while to come.

  7. John, thanks. The trouble with neoliberal economics is that it relies on ideology, never evidence. So, they misread what is known as public good, preferring to believe that this kind of work can be done for a profit. Everything is a business. There have been many failures of this thinking, many ruined lives and public institutions.

    Once again, Turnbull’s bragging bunch of incompetents show us that they are not long for their positions of power. This government is fragile and may not see the year 2017. I can only say good riddance to this born-to-rule cohort of slaving neoliberal wreckers. Make it soon. The failed census is just a start.

  8. @David Allen

    This was the first thing I thought – the Right will claim that privatising the ABS will solve this “problem”, notwithstanding the fact that’s it was essentially outsourced already to IBM.

  9. The sad thing about the debacle is that it’s probably put online voting in elections back 20 years.

  10. @Ken_L
    ‘The sad thing about the debacle is that it’s probably put online voting in elections back 20 years.’
    As it should. People will believe the same incompetence would apply and they will be right.

  11. @Beethoven

    Pre Howard, most government IT was a mix of in-house and small-medium contractors for basic work. The in-house ranged from really very good (Social Security, Vets Affairs), through ordinary to poor – the usual range. Much was innovative, most was reliable, pretty much all were very large – routinely an order of magnitude bigger than most private systems. Pretty much all had very extensive interfaces with industry or clients, so was very reliant on the rest of the business for knowledge of partner reactions and business knowledge. Howard put almost all to large private tenders, won by IBM, Telstra, Accenture and a few others. The small and medium business lost out. Most staff transferred, then went on, so business knowledge was lost. The head contractors in turn contract, and staff turnover is high. Business design does not substitute for in-depth knowledge of the operation. IIRC, the Audit Office reckoned the cost to the Commonwealth of the new arrangements at 8 bn a year. But this cannot be unwound easily.

    Large IT stuffs up regularly, although the private debacles get much less press. This stuff-up is not unusual. But the arrangements make it hard to learn from it. Either ABS duplicates IBM, at extra cost, or goes on with IBM (or another). The next time round, the IT staff will have changed, their knowledge of the business be just as poor, and the contract terms no different.

  12. The sad thing about the debacle is that it’s probably put online voting in elections back 20 years.

    On-line voting doesn’t work. I mean, a-priori: anonymity means breaking the connection between ballots and vote registrations, and you can’t do that and keep an audit trail. Anonymous or tamper-evident: one or the other.

  13. “a ten-yearly census was exactly what the leadership of ABS was suggesting a couple of years ago”.

    Err, no – that was a Dept of Finance suggestion. The ABS publicly floated ways that the adverse impact of it could be minimised (on the assumption it was a fait accompli), but in private they were quite appalled. But Treasury’s vigorous opposition is what finally killed it.

    You can’t absolve those immediately responsible for this fiasco. I know, like and greatly respect David Kalisch but he has to take the fall – especially as these days his Ministers never will. But the underlying cause really is the death of the ABS by a thousand cuts – their ability to do any capacity investment has been degrading for a decade in the face of chronic budget squeezes.

    Of course, the beancounters will no doubt immediately say “they’re proven poor value – cut ’em again”. It’s an old story with government services – keep squeezing them over a long period until they can no longer deliver, then use that as an excuse to get rid of them.

  14. @Peter T

    I don’t think what happened in the good old days is all that relevant. Back then, computers in Vet Affairs (which might have been just one mainframe in the basement) were not connected to any other computers. These days, every computer in the world is connected to every other computer. The expertise needed to make something on the scale of a census reliable, safe and robust is specialised and very few people know how to do it.

    I also don’t get why people are so worried about their privacy, on this occasion, when the government already holds much more sensitive information on its computers. The tax office holds information about their income, obviously, but also can find out about every dollar that has gone in or out of their bank accounts and super funds. The Medicare computers contain details of every doctor’s visit, and every procedure and operation everyone has ever had in public hospitals. But nobody worried about that information getting into the wrong hands. What’s so special about the census?

  15. @derrida derider

    There is nothing remotely in Kalisch’s background (he’s spent his whole career in the public service) to suggest that he is “entrepreneurial”. He is not Larry Marshall.

  16. @Beethoven

    It wasn’t quite that primitive, Ludwig. Back in 1974 when I joined C’wealth Dept. Health as a trainee ADPer, the Pharmaceutical Benefits folks already had an Australia wide network of data entry minicomputers that transmitted PBS data to Canberra over PMG leased lines. Sure, the last stage of the process was to walk tapes over from the Canberra concentrator to be processed on the IBM mainframes, but that didn’t remain forever.

    But the thing about the Census (and you forgot Centre-Link which holds a lot of info on us pensioned oldies) is that as well as names, there is a complete address. Plus birthdates. Now who would like to suggest that a minor hack of the Telstra operated White Pages to put address and name data (divided up by age) together, wouldn’t also produce a lot of “confidential” data.

    So, who wants to suggest that we also don’t fill in our address or the ages of the residents and census night guests ?

    The difference was, back then, that networks were handled by privately operated leased lines – and to penetrate one of those you had to know which line to intercept in which PMG (Later Telecom) exchange.

  17. Beethoven, you don’t know him. Believe me, he’s not a Sir Humphrey. Insofar as all this is the result of anything he did it’s almost certainly because he tried to be TOO entrepreneurial. Though as I implied most of the factors that led to this fiasco predated him.

    But he’s still gotta go – you take the money, you take the responsibility.

  18. I don’t think anyone should be taking the fall for this, not until such time as the mechanics of what went wrong, and why, are investigated. While I have some default antipathy towards LNP ministers, I really don’t think the current one can be credibly blamed for this shambles, and he shouldn’t take the fall—at least not on the information currently available.

    Still, it has been amusing to read some of the comments popping up around the traps. I won’t re-post any.

  19. @derrida derider

    I am not defending Kalisch. If he has to fall in his sword, so be it. But the IBM contract was awarded before he was appointed as Statistician. The whole architecture of the census was a done deal while he was still running AIHW. He could hardly have put the whole thing on hold while thinking about whether it was a good idea when he started early last year. The census runs to a strict timetable.

  20. Personalities aside, I do agree with Beethoven that this privacy malarky is tinfoil hat bullshit – Christopher Pyne is right for once:

    – The ABS go to enormous lengths to make sure it is not possible for users of their data to ever identify anyone – to the degree that it seriously inhibits usefulness of much of the more expensive stuff they produce. In over a century they have never had a leak about individuals.

    – Australia and NZ are the only developed countries that DON’T retain their census data indefinitely and its a non-issue elsewhere.

    – The ATO, Centrelink and various state and (especially) government bodies already know and retain your name and address and lots of other stuff about you that is far more sensitive than any Census question. And because these are mostly live accounts exposed to the web rather than an archived dataset on an isolated computer they are infinitely more liable to hacking than Census linking keys.

  21. Derrida Derider, you are worth your weight in gold. It is not so much about the census but an exercise in neoliberal privatisation/outsourcing/ secrecy ideological junketry and public conditioning…zombie economics and the surveillance state.

  22. @Collin Street
    The Cons (conservatives) did not completely abolish the census. We have a two tiered system with a short form and a long form.

    What they did was abolish the long form which is the one with the real meat. It supplies the key information that business and government planners can use to make sensible planning decisions and that allow for researchers to tease out what is happening.

    The Cons decided to replace the long form with a voluntary survey despite objections from any and everyone from banks to municipal planners to researchers. Response rates dropped from 90+% to 77% making valid inferences in many areas impossible

    The head of Statistics Canada resigned (in silent protest since as a loyal civil servant he did not attack the government)

    The decision to cancel appeared to be totally ideologically based though the Cons might have been cunning enough to realize that without the long form data it would be more difficult to attack some of the more egregious of their programs. Certainly the Cons never allowed facts to contradict ideology.

    I have an even longer rant on the earlier thread Should the census be compulsory?

  23. Re IBM designing the Census IT system

    Here in Canada IBM designed the new Federal Govenment pay system. Chaos has followed. Some people have not been paid for perhaps 18 months, some people have been wildly overpaid, too much tax has been deduced, no tax has been deducted

    The Government is scrambling madly to try to at least apply some patches. Of course it was a Conservative Gov’t who let the contract but the Liberals who are having to clean up the mess.

  24. The Cons (conservatives) did not completely abolish the census. We have a two tiered system with a short form and a long form.

    Thank you for your clarifications.

  25. I wonder if this was part of the “disruptive innovation” that the current mob love pushing?

  26. @Beethoven

    I don’t agree with your critique of Peter T’s comparison of past and current operations. On the contrary, your argument rests on the assumption that ‘new’ is better and this is clearly not the case.

    Furthermore, Peter T’s post contains an example of how concentration of wealth occurs due to privatisation and how privatisation (IBM contract) reduces the ownership of technological knowhow within Australia. There are also negative BoP implications due to profit repatriation.

    As for privacy, is Beethoven your real name?

  27. “The Medicare computers contain details of every doctor’s visit, and every procedure and operation everyone has ever had in public hospitals.”

    Not true, strictly speaking. Doctor’s visits paid for privately and not claimed are not recorded in Medicare data systems.

  28. Wow, this is great! <- Sarcasm.

    The Turnbull govt can't even run a census which is scarcely the most complicated thing a government needs to run. They have to be held responsible for it as it was they who gutted the ABS and set it on its new crash and burn path.

    I am baffled by some related issues. IBM has a long history of massive, egregious IT implementation failures. Why in blithering hellcat tarnations would anyone sign a contract with IBM for anything? Actually we might know. Let me ask a hypothetical. Would it be possible that there are secret promises of lucrative corporate gigs for key players, just a bit down the line when things quiet down and short public memory fizzles out?

    It's been a repeated feature of the neoliberal era that private enterprise's contract lawyers have clearly been far cleverer than the Australian PS contract lawyers, especially in the arena of IT. That is a failure which bears looking at. Governments should drive much harder bargains at the contract table to the point of placing sovereign rights conditions in contracts which conditions government reserves the right to activate at any time. Possibly, the best candidate strategy for implementing and activating such conditions would be to extend the concept of odious debt. In international law, odious debt, also known as illegitimate debt, is a legal theory that holds that the national debt incurred by a government for purposes which do not serve the best interests of the nation, should not be enforceable. In the case of egregious failure imperiling or damaging the nation, the government or their proper operations, the corporation should pay extensive damages to the state.

    Overall, this current fiasco is just another example of late stage neoliberal capitalism doing what it does best. It deliberately sabotages the democratic state via projects which the state pays for. Then it forces the state (meaning the people) to pay again for a privatised and vastly inferior product. It's exactly as Naomi Klein said. The plan is to sabotage it and then privatise it. And to make the people pay for every fiasco along this path.

    The institution to blame for this is not that of democratic state governance as such. It is the set of institutions and corporations which constitute the operations of neoliberal corporate capital and give them outrageous advantages over the rights of the people and their democratic government. The blame must be sheeted to the correct place.

    However, I have little to no hope for the time being. These forces will continue wrecking our society and world for a long time to come. They are still in near complete ascendancy and the bulk of the people simply do not yet understand what is happening and where the real problem lies.

  29. @Ikonoclast

    Why in blithering hellcat tarnations would anyone sign a contract with IBM for anything?

    Because nobody ever got fired for choosing IBM. If you’re a risk averse public servant – and public servants are nothing if not risk averse – and you choose IBM, and the project goes tits up, you can always fall back on: “It’s IBM. They’ve been around for decades and are the number one company in the world for these kinds of projects”. Whereas, if you choose an unknown company and it goes wrong, then your career is in the toilet.

    Plus, in this case, IBM did the job for only $9 million, an unbelievably small sum for a job of this magnitude and importance. They were probably the cheapest by a long way.

  30. @Beethoven
    I like Beethoven’ take on all this.. i) It went as brilliantly as could be expected; ii) It will be smooth as silk next time; iii) TINA. Boy with a future here.

  31. Beethoven

    First – Government pioneered IT interfaces (Social Security had to interface with the ATO, Finance and banks; all departments with Finance; Customs with airlines, freight, brokers, shipping, banks, ATO, Immigration, TGA…, Vets Affairs with Social and ATO and so on). Been a long time since it was a mainframe in a basement.

    Second – there are about 5 or 6 large IT firms, all much the same. Cost differences are marginal. One screws up, you get another – but staff transfer. The problem with outsourcing the whole of IT (rather than, say, maintenance, back-ups and such) is that you lose the understanding of the business. And you have no control over costs or timing. This is why large organisations keep core functions in house. Howard insisted all departments outsource the whole, and yielded for cases where the pushback was very strong (eg CSIRO and similar tech agencies know more about computers than IBM ever will, and their people fiddle with them all the time; some parts of Defence ditto, plus security). In many cases, we have ended with the worst of both worlds – a rebuilt in-house capability struggling to control contractors who have their hands on the actual equipment and control over access and changes.

  32. @Beethoven

    If one has a low confidence in the automation process and a low confidence in the electronic security of the automation then one should retain manual processes which are historically proven to work successfully. This is the case when “one” is government and cost concerns are not (and should not be) the only metric for the project. Quality and security of the data and successful prosecution of the collection process are more important than cost.

    In any case, extra government spending on manual processes lifts employment. There is always a benefit to this when in a prolonged employment recession as we are now. Australia’s true rate of unemployment is much closer to 10% than it is to the rank lie peddled of 6%. See Morgan polls and Bill Mitchell on this topic.

    In J.Q.’s previous blog post on this census, I mentioned that the best data security can be obtained by combining physical security and electronic security. I wrote:

    An overall theory of data security must include the theory of “hybrid” security setups which employ physically secure processes and electronically secure processes in concert. For some applications it is not always wise to switch entirely to electronic data at all steps and thus restrict the system to the limitations of electronic security only. The census is a good example. It would be possible to maintain paper form collection, then use data entry into a mainframe not connected to the internet. Research of the census data could then proceed via government user requests into some sort of statistical analysis system which would run batch jobs on the unconnected mainframe. There is no genuine need for real-time queries of, or an internet connection to, census data, even for government analysts.

    The extra costs of such a system would not be important. Indeed, extra employment would be generated by the manual process components. If extra automation creates data vulnerability and more unemployment, then the savings and benefits overall are largely or even completely illusory. Costs and risks will be shifted to other parts of the economy and onto ordinary citizens. The mania for total automation and cost-cutting is the real problem not the compulsory nature of the census and the collecting of names.

    In summary, a major problem is in the blind assumption that more automation is always better and that cost-cutting goals should trump all other goals. How many times have we seen short-sighted cost-cutting result in more costs in the long run? Innumerable times would be my answer. This fiasco is another example.

  33. “Plus, in this case, IBM did the job for only $9 million, an unbelievably small sum for a job of this magnitude and importance”

    $10million according to smh. A 10% error is too big!

  34. @Peter T

    I don’t disagree with any of this, as far as it goes. But the situation here was millions of people logging in to the ABS via the web to upload information. When has the public service ever had the expertise in that kind of IT?

    @tony lynch

    Pretty much. Millions of people and organisations send their tax returns to the ATO’s computers and it all seems to go OK. The same will be true for the next Census. The trick is to have a big enough computer to handle the traffic, which means not doing it on the cheap.

  35. @Peter T

    “The problem with outsourcing the whole of IT (rather than, say, maintenance, back-ups and such) is that you lose the understanding of the business. And you have no control over costs or timing.”

    Exactly. This is an example of the shortcomings of the ‘privatise and regulate’ policy framework. How are government departments supposed to regulate something they don’t know? That is, effective regulation presupposes the same or better knowledge of the technology that underlies an ‘industry’ which is to be regulated.

    (Education, health, banking, utilities provide similar examples.)

  36. I’m not sure the problem is necessarily “the new entrepreneurial leadership of ABS”. but that was possibly a contributing factor.

    IMHO the public service (Commonwealth and the states) has lost much of its professional expertise and knowledge as most of the technical work is now contracted out (IT, science, economics etc.). The senior ranks are full of people who are able to meet job criteria like “strategic communicator”, but portfolio-specific skills or content knowledge is never a priority. The public service is now full project management types that are poorly equipped to deal with even remotely technical issues.

  37. For those interested, the lesson on outsourcing is over 200 years old. The Royal Navy contracted much shipbuilding, maintenance and supply but, after a couple of early lessons, always kept an in-house capacity both as a fall-back and a source of the necessary technical and pricing knowledge. The one area where private suppliers were able to make large profits was armour plate, where large initial costs plus economies of scale kept the RN from building its own capacity.

    Really, the reliance of much economics on very simple notions of competition is a standing intellectual reproach.

  38. I think it was a fiasco for two reasons. 1) As hinted by Derrider, cuts to ABS and the need to show a efficiency dividend – if this still exists.

    2) Failure to develop new system by modules and and development and test systems. They should have developed experience by for example saying in 2016 only ACT addresses have the option of online census. Next time expand the scope.

    There is no reason for anyone to resign except get rid of capitalist motives in society – the root cause.

  39. OOps I used a bad word in my comment so here it is again.

    I’m backing up Peter T here – I worked for NSW Health for a little while over the period of the Y2K bug and back then most sensitive IT stuff was done in-house (I was responsible for managing my little part, which worked really well I might add). This included various state-wide networks that held confidential patient data. When things went tits-up they usually did so for reasons unrelated to IT expertise or management (for example, rolling out a shared database of patient information across the whole of my health district was technically well within the abilities of the in-house IT, but various hospitals and clinics blocked it for privacy reasons). Our experience with outside contractors was mixed but they were usually poor – the sole exception being some specialists we brought in for Y2K. I remember back in the 1990s NSW Health already had the HOIST system which was a centralized database of health information that you could query from anywhere in the state to run analysis on.

    What we built wasn’t rocket science and it wasn’t pretty but it got stuff done and it got it done well. If the federal govt had maintained that capacity over the past 20 years, this census madness shouldn’t have happened.

  40. So far there have been unsubstantiated claims of hack, ddos, attack etc. from the ABS, the minister, &Turnbull. I’ve seen no evidence yet & those experts that have looked for it have found none. This blame shifting to anonymous parties just makes their incompetence look worse. “Our” ABC has run the attack line solidly despite the lack of evidence which will rebound on them as well.

    Millions of Australians accessing the census website after dinner, hitting F5 to refresh constantly, on Tuesday night is exactly what a ddos would look like.

  41. @David Allen

    What is amazing is that the ABS seems to have assumed in its capacity planning that the load would be spread evenly across Tuesday when it was obvious that 95% of households in the eastern states would all be doing the census between 7.30 pm and 10 pm on Tuesday night.

    Turnbull’s comments today bode badly for David Kalisch. Just as the head of the AEC resigned after the lost WA Senate votes fiasco, even though he didn’t personally lose the voting papers, Kalisch might have to go too. As DD said upthread, this comes with the territory when you get paid the big bucks.

  42. The fact that the site is STILL down has turned this affair into a complete farce. How long before they have to concede that no action will be taken to fine people who’ve spent days trying in good faith but unsuccessfully to complete their returns?

    I would hate to be one of the collectors who has the unhappy task next week of visiting households to ask why they haven’t submitted their census form yet.

  43. @Ken_L

    The problem is once they tell people it’s back up, they risk everyone rushing onto it all at once, which will put it back down.

  44. found the form at my front door a couple of days before the 9th.

    forgot to fill it in on the 9th

    filled it in next morning.

    stuck it in the letterbox.

  45. @Beethoven Its a while since I actively worked in network capacity planning so they may very well have tools to address this now. The problem that is often not taken into account is the number of retries that will occur from failed logins. So if we assume that 100 users will try and login within a given time but 50 percent of those attempts fail and the users try again, say 5 times (not unreasonable) then within that given time you have the 10 attempts plus 5 x 5 attempts or 35 login attempts all up. It is a cascade affect that rapidly increases the traffic form a predictable flow to a massive increase when things do not go smoothly.

    Of course I would think the boffins designing the ABS system would well understand this problem and it would have been taken into account. But, the information now dribbling out and crab walking away from a foreign DOS attack has dented this confidence.

  46. Spot the deliberate mistake! Out by a factor of 10, should be 350 login attempts. Maybe that is why I don’t work in this area anymore.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s