Can this census be saved?

It appears that, having crashed last night with only about 10 per cent of households having submitted data, the Census website is now off the air indefinitely. It’s hard for me to see how this exercise can be salvaged. Almost certainly, lots of people who tried and failed to fill in their forms last night will be unwilling to do so again, especially in the absence of any coherent explanation for the failure. It’s looking increasingly as if the only option will be to give up and try again in five years time. Coincidentally or not, a ten-yearly census was exactly what the leadership of ABS was suggesting a couple of years ago.

This fiasco seems to have “reform” written all over it, from the new entrepreneurial leadership of ABS to the contracting out of vital functions to the benign/malign neglect displayed by the Abbott-Turnbull government. Peter Martin is very good on this, as is Chris Graham at New Matilda.

71 thoughts on “Can this census be saved?

  1. Re IBM designing the Census IT system

    Here in Canada IBM designed the new Federal Govenment pay system. Chaos has followed. Some people have not been paid for perhaps 18 months, some people have been wildly overpaid, too much tax has been deduced, no tax has been deducted

    The Government is scrambling madly to try to at least apply some patches. Of course it was a Conservative Gov’t who let the contract but the Liberals who are having to clean up the mess.

  2. The Cons (conservatives) did not completely abolish the census. We have a two tiered system with a short form and a long form.

    Thank you for your clarifications.

  3. I wonder if this was part of the “disruptive innovation” that the current mob love pushing?

  4. @Beethoven

    I don’t agree with your critique of Peter T’s comparison of past and current operations. On the contrary, your argument rests on the assumption that ‘new’ is better and this is clearly not the case.

    Furthermore, Peter T’s post contains an example of how concentration of wealth occurs due to privatisation and how privatisation (IBM contract) reduces the ownership of technological knowhow within Australia. There are also negative BoP implications due to profit repatriation.

    As for privacy, is Beethoven your real name?

  5. “The Medicare computers contain details of every doctor’s visit, and every procedure and operation everyone has ever had in public hospitals.”

    Not true, strictly speaking. Doctor’s visits paid for privately and not claimed are not recorded in Medicare data systems.

  6. Wow, this is great! <- Sarcasm.

    The Turnbull govt can't even run a census which is scarcely the most complicated thing a government needs to run. They have to be held responsible for it as it was they who gutted the ABS and set it on its new crash and burn path.

    I am baffled by some related issues. IBM has a long history of massive, egregious IT implementation failures. Why in blithering hellcat tarnations would anyone sign a contract with IBM for anything? Actually we might know. Let me ask a hypothetical. Would it be possible that there are secret promises of lucrative corporate gigs for key players, just a bit down the line when things quiet down and short public memory fizzles out?

    It's been a repeated feature of the neoliberal era that private enterprise's contract lawyers have clearly been far cleverer than the Australian PS contract lawyers, especially in the arena of IT. That is a failure which bears looking at. Governments should drive much harder bargains at the contract table to the point of placing sovereign rights conditions in contracts which conditions government reserves the right to activate at any time. Possibly, the best candidate strategy for implementing and activating such conditions would be to extend the concept of odious debt. In international law, odious debt, also known as illegitimate debt, is a legal theory that holds that the national debt incurred by a government for purposes which do not serve the best interests of the nation, should not be enforceable. In the case of egregious failure imperiling or damaging the nation, the government or their proper operations, the corporation should pay extensive damages to the state.

    Overall, this current fiasco is just another example of late stage neoliberal capitalism doing what it does best. It deliberately sabotages the democratic state via projects which the state pays for. Then it forces the state (meaning the people) to pay again for a privatised and vastly inferior product. It's exactly as Naomi Klein said. The plan is to sabotage it and then privatise it. And to make the people pay for every fiasco along this path.

    The institution to blame for this is not that of democratic state governance as such. It is the set of institutions and corporations which constitute the operations of neoliberal corporate capital and give them outrageous advantages over the rights of the people and their democratic government. The blame must be sheeted to the correct place.

    However, I have little to no hope for the time being. These forces will continue wrecking our society and world for a long time to come. They are still in near complete ascendancy and the bulk of the people simply do not yet understand what is happening and where the real problem lies.

  7. @Ikonoclast

    Why in blithering hellcat tarnations would anyone sign a contract with IBM for anything?

    Because nobody ever got fired for choosing IBM. If you’re a risk averse public servant – and public servants are nothing if not risk averse – and you choose IBM, and the project goes tits up, you can always fall back on: “It’s IBM. They’ve been around for decades and are the number one company in the world for these kinds of projects”. Whereas, if you choose an unknown company and it goes wrong, then your career is in the toilet.

    Plus, in this case, IBM did the job for only $9 million, an unbelievably small sum for a job of this magnitude and importance. They were probably the cheapest by a long way.

  8. @Beethoven
    I like Beethoven’ take on all this.. i) It went as brilliantly as could be expected; ii) It will be smooth as silk next time; iii) TINA. Boy with a future here.

  9. Beethoven

    First – Government pioneered IT interfaces (Social Security had to interface with the ATO, Finance and banks; all departments with Finance; Customs with airlines, freight, brokers, shipping, banks, ATO, Immigration, TGA…, Vets Affairs with Social and ATO and so on). Been a long time since it was a mainframe in a basement.

    Second – there are about 5 or 6 large IT firms, all much the same. Cost differences are marginal. One screws up, you get another – but staff transfer. The problem with outsourcing the whole of IT (rather than, say, maintenance, back-ups and such) is that you lose the understanding of the business. And you have no control over costs or timing. This is why large organisations keep core functions in house. Howard insisted all departments outsource the whole, and yielded for cases where the pushback was very strong (eg CSIRO and similar tech agencies know more about computers than IBM ever will, and their people fiddle with them all the time; some parts of Defence ditto, plus security). In many cases, we have ended with the worst of both worlds – a rebuilt in-house capability struggling to control contractors who have their hands on the actual equipment and control over access and changes.

  10. @Beethoven

    If one has a low confidence in the automation process and a low confidence in the electronic security of the automation then one should retain manual processes which are historically proven to work successfully. This is the case when “one” is government and cost concerns are not (and should not be) the only metric for the project. Quality and security of the data and successful prosecution of the collection process are more important than cost.

    In any case, extra government spending on manual processes lifts employment. There is always a benefit to this when in a prolonged employment recession as we are now. Australia’s true rate of unemployment is much closer to 10% than it is to the rank lie peddled of 6%. See Morgan polls and Bill Mitchell on this topic.

    In J.Q.’s previous blog post on this census, I mentioned that the best data security can be obtained by combining physical security and electronic security. I wrote:

    An overall theory of data security must include the theory of “hybrid” security setups which employ physically secure processes and electronically secure processes in concert. For some applications it is not always wise to switch entirely to electronic data at all steps and thus restrict the system to the limitations of electronic security only. The census is a good example. It would be possible to maintain paper form collection, then use data entry into a mainframe not connected to the internet. Research of the census data could then proceed via government user requests into some sort of statistical analysis system which would run batch jobs on the unconnected mainframe. There is no genuine need for real-time queries of, or an internet connection to, census data, even for government analysts.

    The extra costs of such a system would not be important. Indeed, extra employment would be generated by the manual process components. If extra automation creates data vulnerability and more unemployment, then the savings and benefits overall are largely or even completely illusory. Costs and risks will be shifted to other parts of the economy and onto ordinary citizens. The mania for total automation and cost-cutting is the real problem not the compulsory nature of the census and the collecting of names.

    In summary, a major problem is in the blind assumption that more automation is always better and that cost-cutting goals should trump all other goals. How many times have we seen short-sighted cost-cutting result in more costs in the long run? Innumerable times would be my answer. This fiasco is another example.

  11. “Plus, in this case, IBM did the job for only $9 million, an unbelievably small sum for a job of this magnitude and importance”

    $10million according to smh. A 10% error is too big!

  12. @Peter T

    I don’t disagree with any of this, as far as it goes. But the situation here was millions of people logging in to the ABS via the web to upload information. When has the public service ever had the expertise in that kind of IT?

    @tony lynch

    Pretty much. Millions of people and organisations send their tax returns to the ATO’s computers and it all seems to go OK. The same will be true for the next Census. The trick is to have a big enough computer to handle the traffic, which means not doing it on the cheap.

  13. @Peter T

    “The problem with outsourcing the whole of IT (rather than, say, maintenance, back-ups and such) is that you lose the understanding of the business. And you have no control over costs or timing.”

    Exactly. This is an example of the shortcomings of the ‘privatise and regulate’ policy framework. How are government departments supposed to regulate something they don’t know? That is, effective regulation presupposes the same or better knowledge of the technology that underlies an ‘industry’ which is to be regulated.

    (Education, health, banking, utilities provide similar examples.)

  14. I’m not sure the problem is necessarily “the new entrepreneurial leadership of ABS”. but that was possibly a contributing factor.

    IMHO the public service (Commonwealth and the states) has lost much of its professional expertise and knowledge as most of the technical work is now contracted out (IT, science, economics etc.). The senior ranks are full of people who are able to meet job criteria like “strategic communicator”, but portfolio-specific skills or content knowledge is never a priority. The public service is now full project management types that are poorly equipped to deal with even remotely technical issues.

  15. For those interested, the lesson on outsourcing is over 200 years old. The Royal Navy contracted much shipbuilding, maintenance and supply but, after a couple of early lessons, always kept an in-house capacity both as a fall-back and a source of the necessary technical and pricing knowledge. The one area where private suppliers were able to make large profits was armour plate, where large initial costs plus economies of scale kept the RN from building its own capacity.

    Really, the reliance of much economics on very simple notions of competition is a standing intellectual reproach.

  16. I think it was a fiasco for two reasons. 1) As hinted by Derrider, cuts to ABS and the need to show a efficiency dividend – if this still exists.

    2) Failure to develop new system by modules and and development and test systems. They should have developed experience by for example saying in 2016 only ACT addresses have the option of online census. Next time expand the scope.

    There is no reason for anyone to resign except get rid of capitalist motives in society – the root cause.

  17. OOps I used a bad word in my comment so here it is again.

    I’m backing up Peter T here – I worked for NSW Health for a little while over the period of the Y2K bug and back then most sensitive IT stuff was done in-house (I was responsible for managing my little part, which worked really well I might add). This included various state-wide networks that held confidential patient data. When things went tits-up they usually did so for reasons unrelated to IT expertise or management (for example, rolling out a shared database of patient information across the whole of my health district was technically well within the abilities of the in-house IT, but various hospitals and clinics blocked it for privacy reasons). Our experience with outside contractors was mixed but they were usually poor – the sole exception being some specialists we brought in for Y2K. I remember back in the 1990s NSW Health already had the HOIST system which was a centralized database of health information that you could query from anywhere in the state to run analysis on.

    What we built wasn’t rocket science and it wasn’t pretty but it got stuff done and it got it done well. If the federal govt had maintained that capacity over the past 20 years, this census madness shouldn’t have happened.

  18. So far there have been unsubstantiated claims of hack, ddos, attack etc. from the ABS, the minister, &Turnbull. I’ve seen no evidence yet & those experts that have looked for it have found none. This blame shifting to anonymous parties just makes their incompetence look worse. “Our” ABC has run the attack line solidly despite the lack of evidence which will rebound on them as well.

    Millions of Australians accessing the census website after dinner, hitting F5 to refresh constantly, on Tuesday night is exactly what a ddos would look like.

  19. @David Allen

    What is amazing is that the ABS seems to have assumed in its capacity planning that the load would be spread evenly across Tuesday when it was obvious that 95% of households in the eastern states would all be doing the census between 7.30 pm and 10 pm on Tuesday night.

    Turnbull’s comments today bode badly for David Kalisch. Just as the head of the AEC resigned after the lost WA Senate votes fiasco, even though he didn’t personally lose the voting papers, Kalisch might have to go too. As DD said upthread, this comes with the territory when you get paid the big bucks.

  20. The fact that the site is STILL down has turned this affair into a complete farce. How long before they have to concede that no action will be taken to fine people who’ve spent days trying in good faith but unsuccessfully to complete their returns?

    I would hate to be one of the collectors who has the unhappy task next week of visiting households to ask why they haven’t submitted their census form yet.

  21. @Ken_L

    The problem is once they tell people it’s back up, they risk everyone rushing onto it all at once, which will put it back down.

  22. found the form at my front door a couple of days before the 9th.

    forgot to fill it in on the 9th

    filled it in next morning.

    stuck it in the letterbox.

  23. @Beethoven Its a while since I actively worked in network capacity planning so they may very well have tools to address this now. The problem that is often not taken into account is the number of retries that will occur from failed logins. So if we assume that 100 users will try and login within a given time but 50 percent of those attempts fail and the users try again, say 5 times (not unreasonable) then within that given time you have the 10 attempts plus 5 x 5 attempts or 35 login attempts all up. It is a cascade affect that rapidly increases the traffic form a predictable flow to a massive increase when things do not go smoothly.

    Of course I would think the boffins designing the ABS system would well understand this problem and it would have been taken into account. But, the information now dribbling out and crab walking away from a foreign DOS attack has dented this confidence.

  24. Spot the deliberate mistake! Out by a factor of 10, should be 350 login attempts. Maybe that is why I don’t work in this area anymore.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s