Can this census be saved?

It appears that, having crashed last night with only about 10 per cent of households having submitted data, the Census website is now off the air indefinitely. It’s hard for me to see how this exercise can be salvaged. Almost certainly, lots of people who tried and failed to fill in their forms last night will be unwilling to do so again, especially in the absence of any coherent explanation for the failure. It’s looking increasingly as if the only option will be to give up and try again in five years time. Coincidentally or not, a ten-yearly census was exactly what the leadership of ABS was suggesting a couple of years ago.

This fiasco seems to have “reform” written all over it, from the new entrepreneurial leadership of ABS to the contracting out of vital functions to the benign/malign neglect displayed by the Abbott-Turnbull government. Peter Martin is very good on this, as is Chris Graham at New Matilda.

71 thoughts on “Can this census be saved?

  1. @Ikonoclast

    It isn’t just IBM that has a “…a long history of massive, egregious IT implementation failures.”, Ikono, it’s just about everybody.

    You see, the reason we don’t learn from the mistakes of history is that mostly we weren’t around when the mistakes were made (and understood – Root Cause Analysis, anybody ?).

    But for the very few who were, I have just one word: Mandata 1973 (oh alright, one word plus one number). By the way, how’s your experience with Gantt Charts, PERT Charts and Critical Path Analysis ? Anybody ?

  2. I am one of the few who seemingly were able to complete the forms online, submit and receive an emailed receipt, before all the problems occurred. What struck me was the continuing banality and superficiality of the questions, with nothing innovative compared with previous Censes. Questions about weekly income with no corresponding questions about assets and wealth, for example, can give no meaningful data about the distribution and scale of wealth. The data generated by this exercise are really very limited from the perspective of public policy. ABS gets better data for housing and social policy purposes, for example, through surveys, which can be completed more quickly and more often, and which deliver usable data in a shorter time frame. Better perhaps to spend the Census millions on even better surveys, with a less-frequent attempt at a full count for bench-marking purposes.

  3. GrueBleen,

    We learned to reliably build skyscrapers, dams, bridges, ships, passenger aircraft etc. very well once they become mature technologies. There are relatively few catastrophic failures of these projects in developed nations. Yet we have failed to bring to maturity the techniques of implementing large scale computer systems. They fail time after time. As you say “it’s just about everybody”. I wonder why this is?

    In many engineering projects, but not often IT projects, people die if the designers and builders get it wrong. Maybe that factor is what concentrates minds and legislatures. The only possibility for achieving a similar concentration of project management minds in IT projects would seem to be very steeply rising penalties for failing to achieve deliverables on time and to specification. In other words, miss a key milestone for performance as has happened in the census project and the developer would lose a large percentage of the contract. I would suggest since only 10% data was collected by the deadline, IBM should only get 10% of the contracted monies. This would have had to have been in the contract of course. The chance to recoup a large percentage of the contract monies would then swing on excellent repair and support of the system for the next 5 years and a fully compliant performance the next time round. If private enterprise won’t sign such contracts then build up national in-house capability and build things in-house.

    As I have touched on at least a couple of times, there seem to be systematic failures in government contracting and serious weaknesses in public-private contract law. Some might find this article interesting.

    Contract Theory and the Failures of Public-Private Contracting – Wendy Netter Epstein

  4. @Ikonoclast

    Your #53 of 11 Aug

    Fortunately, although it is mostly everybody, it isn’t every time … failures I mean. I spent 34 years in the ADP/EDP/IT/ICT industry (whichever acronym you prefer) and I never took part in a genuine disaster. Nor even a sizable failure. But that was definitely the luck of the game, because quite a few have happened in the time of my professional involvement.

    And the funny thing is that IBM – which I worked for 13 years – had the best methodological armory in the profession, and had implemented some sizable successes, even in Australia. But it seems that those salad days are long gone now, and one would have to say that IBM hasn’t succeeded in it’s transition from a mostly hardware – and hardware related software – company into a full spectrum IT services company, even after it acquired the PwC IT Consulting arm back in 2002 (for a measly US$3.5 billion according to the web).

    Conceptually, I’ve always liked the categorisation of “manufacturing” by Peter Drucker, viz
    [Cottage industry – which Drucker doesn’t actually mention, but I include for completeness]
    Unique product – eg large buildings, large vessels etc
    Single product mass production – usually from ‘raw’ materials into a single finished product, often used as inputs to =>
    Variable product mass production – ie TVs today, computer monitors tomorrow, same assembly line, a few different components
    System products – eg a refinery or a general hospital where the system is the product.

    That all makes sense and each kind of manufacturing has its own rules and success criteria, except that large scale IT partakes of a weird combination of cottage industry, unique product and system product. No two large buildings are identical, nor are most large scale vessels, but both of them utilise some very ‘standard’ components (not entirely dissimilar to ‘multiple product mass production’ in many respects). Large IT systems do not – unless they are large scale implementations of commercial software packages – eg SAP.

    In short, lots of opportunities all along the way for IT projects to ‘go wrong’, and many of them take full advantage of those opportunities.

    But I agree about vwery serious weaknesses in contract management and I’ll follow up your link a little later. In my experience, it’s almost a case of ‘common law’ versus ‘statute law’. Contracts partake of the nature of statute law – they attempt to set out the details of what is to be done, by when and for how much etc. But like all statute law, a contract cannot cover everything in micro detail – there is always an ongoing process whereby the vendor/supplier tries to limit commitment to precisely what the contract says and the customer tries to invoke the ‘reasonable expectations of a rational man’ to overcome imprecisions and omissions in the contract.

    If that whole process isn’t handled by rational adults of reasonable goodwill, disaster often follows. As we all have noticed.

  5. GrueBleen,

    Interesting points. Just to be a snark, a product like an automobile is technically a system too; although I agree it sits somewhere in the range from single mass production product to variable mass production product.

    I could not understand a failure like the Qld Health pay system. I mean it’s not like it would be the first pay system ever built in the world. Indeed, it’s not like it would be the first state or national pay system ever built in the world. Such a projecy should be known territory with tried and tested modules and systems. It’s theoretically not even a difficult or unknown problem even given multiple pay rates, multiple awards and complex rosters. None of these things are unknown. These all should be well known parameters with an understanding of how to incorporate them. I would have thought SAP could handle it with relatively little tailoring of existing system modules.

    Speaking of SAP;

    “SAP wins first big Centrelink systems overhaul deal”

    Centrelink has run ISIS (Income Security Integrated System) for many years now. It is an in-house built system based on Model 204. Model 204 has worked very well. It was built when humans still knew how to build stuff. (OK, slight sarcasm there.) ISIS itself worked (and works) well considering that it was forced to be insanely complicated by insanely intricate welfare legislation from a series of idiotic governments of both colours and of course the sheer size of the database(s). Just about everyone in Australia is on it in one way or another; every child, every parent, every partner, every welfare recipient… that is to say basically everybody. IIRC, Centrelink was/is just about the last customer in the world still using Model 204 and it was being supported just for them.

    I have more confidence in SAP than in anything IBM does now. I have less confidence in SAP than in a legacy database built when human actually used to know how to build stuff. (There’s that sarcasm again.) Basically, I think we can expect mammoth screw-ups on an unprecedented scale from this project. Something has been lost. The competency to do large projects is declining worldwide. I think neoliberal managerialism and anti-science ideology is systematically destroying competency in the professions and trades. It’s a form of civilizational decline. It’s unfashionable to point out civilizational decline in the modern context but historically the rise and fall of civilizations is standard behaviour. Ours is declining, falling apart, IMO.

  6. Footnote to above:

    “Model 2014 technology is supplied by a niche US provider, whose only remaining customers are Australia’s DHS and the CIA”.

  7. @Ikonoclast

    Your #55 of 12 Aug

    Yeah, lots of things these days are a bit more mixed up than in the simpler days of Peter Drucker’s time. Though I think cars are substantially ‘variable product mass production’ – especially back in the long ago days when GMH advertised “millions of different Holdens – choose your own variety” – but in these days of computerisation and GPS etc they are taking on more of the attributes of a ‘system’. The “millions of Holdens” thing was quite a while ago now (at least 3 decades I think) and it was a failure at the time – most people couldn’t be bothered “creating” their own “unique” Holden, so IIRC a company that just presented a well designed and well fitted out single ‘standard’ model started to climb the sales total count. And so began the era of Toyota, which indeed concentrated on a “single product mass production” and implemented the effective quality control to go with it.

    But pardon my flights of nostalgia, however, you’ve sent me on another one with your mention of Model 204. I used to be a DBA/DCA for some time, concentrating on IBM’s IMS, the Fujitsu’s AIM and later IMS ‘Fast Path’ (for banking online systems) the IBM’s DB2 and finally, to a somewhat lesser extent, Oracle. But Model 204 was the choice of the last holdout of ‘computer database purists’, and it was already moribund then. I was, by the way, heavily involved in Dept. of Employment (and Other Things) systems from the initial JOB Bank online system in Melbourne to the full Job Bank system in Canberra – that was done in Fujitsu’s AIM and I was a Fujitsu SE (Systems Engineer) at the time. The system was developed in COBOL, had-written by Departmental employees plus some ‘external’ help (it was the last time I ever wrote any significant amount of IBM style Assembler code 🙂 ) It was later replaced, quite effectively I think, by IBM when Dept. Employment (and Other Things) replaced their Fujitsu mainframes with IBM mainframes and a DB2 system.

    Could SAP be used for a payroll system ? But of course, and as a later IBM ‘consulting’ SE i was heavily involved in the implementation of Telstra’s HR-Pay system (back when Telstra still had around 70,000 employees Australia wide and they all, by labour agreement, had to receive a printed payslip every fortnight). So it goes.

    So, very large scale ‘all of enterprise’ packages like SAP have certainly changed the computer systems landscape, and yes, large scale inhouse development is a long faded art. How much that has to do with “neoliberal managerialism and anti-science ideology” I’m not sure – though I’d agree there is some connection. But I think it has a lot more to do with the thoughts that Fred Brooks (originally Software Development Manager and later the total boss of the IBM S360 project which catapulted IBM to the fore) expressed in his famous essay “No Silver Bullets” which was incorporated into the 2nd edition of his at least equally famous book “The Mythical Man Month” (which was almost my standin for a bible for some years).

    Brooks’ point was that large computer system development was simply irreducibly complex (how long did it take SAP to get up to scratch – 20 or more years ?). And when something is “irreducibly complex”, there are no ‘silver bullets’ to slay the complexity werewolf. But we keep pretending and hoping that there are (and Model204 and later DB2 and Visual Basic and … are expressions of our lasting hopes and pretenses).

    Thank you for providing the opportunity for my nostalgic rave, Ikono. Nobody much is interested in such things nowadays 😦

  8. @GrueBleen

    “Neoliberal managerialism and anti-science ideology” as I call it, has been deliberately deployed since the 1970s to degrade the effectiveness of democratic government outside those functions expressly required to guarantee capitalist relations and corporate. oligopolistic profit.

    From the Omega Project (Adam Smith Institute) onward, neoliberal theorists developed a systematic program for retrenching the welfare state and social democracy to ensure the claimed full rule of so-called “market forces”. It turns out, of course, that it is not market forces (an incomplete economic piloting system anyway) which are unleashed but corporate oligopolistic dictatorship of the TINA (there is no alternative) variety.

    Anti-science ideology is applied selectively. Production science, where it enables corporate profits and ever greater concentration of wealth and power in fewer and fewer hands along with military and police power to ensure the same ends, is encouraged and subsidised. Impact science (environmental science, climate science) is the science which is attacked and which has its institutions, agencies, funding sources and public credibility destroyed.

  9. Footnote to above:

    “Retrenchment is generally an exercise in blame avoidance rather than credit claiming, primarily because the costs of retrenchment are concentrated (and often immediate), while the benefits are not.” – Paul Pierson.

    Pierson’s observation explains another reason why privatisation is a favoured ploy of politicians. (I mean in addition to lining the pockets of their supporters and themselves.) Privatisation facilitates political blame avoidance. Privatise power or mass public transport and then costs and failures can then be attributed to the “market”. The politicians are no longer directly to blame for anything. That’s the way uh huh, uh huh, they like it!

  10. Yep, GrueBleen, one of my favourite aphorisms is Einstein’s “things should always be made as simple as possible – and no simpler”.

    Air safety investigators always proceed on the principle that disaster is rarely caused by a single big mistake – usually it’s a whole series of interlocking smaller ones. Such is certainly the case here, but I think the main mistakes by the ABS were:

    – failure to anticipate how a change with really minor privacy or security implications (giving yourself four years instead of eighteen months to check that you’ve turned names and addresses into anonymised SLKs) would be distorted by politicians on the make keen to exploit paranoia. Bluntly they needed to spend a lot more on PR and also hire a better advertising agency.

    – accepting an obvious lowball bid from IBM (did the tender panel really believe they could get a robust solution so cheap?)

    – the decision on the night to pull the whole site down when the odds of a successful DDOS attack increased as the geoblocking server went down (even Turnbull called this “an abundance of caution” – I call it a disastrous failure to assess comparative risk).

    Whatever, though, I think the answer to the OP’s original question is “no – it can’t be saved”. I wouldn’t like to be the hapless methodologists charged with converting the raw census data into something that is fit for statistical use.

    This is a disaster that extends well beyond the census, because the census data is used to design and benchmark just about every survey (public or private) in Australia. I don’t like the chances of getting accurate data on unemployment in a few years time, for example. Even political opinion polls will become less accurate (those of them that are fair dinkum, anyway). And expect a huge number of court challenges to electoral boundaries. Note all these are exactly the arguments people successfully made against ten year censuses.

  11. @derrida derider

    It’s even worse than that. Everything the ABS does will now be treated with scepticism at best, contempt at worst. It took the ABS many decades to build its reputation as an unimpeachable statistical agency, and one night to wreck that reputation.

  12. @derrida derider

    Einstein also said things (supposedly) about World Wars that I found entertaining, too, DD.

    But “as simple as possible” isn’t an unique point solution, it’s a stage in a long process.

    For instance: “pay the employees their salary” is about as simple as the efinition of a payroll system can get. But (at least once upon a time) it probably took several million lines of COBOL (plus JCL, utility control statements etc) to take that statement to an actually operational level of “simplicity”.

    That said, I think you have basically covered the ground and there is nothing quite so pathetic as rapid response action forced on the unready (cf the Man Monis siege).

    I dunno, but apart from unrealistic expectations of what the present day IBM could deliver, and especially so cheaply, it just doesn’t sound as though anybody designed a scenario based test plan that worked through a Component Failure Impact Analysis. Such as we used to do maybe 20 – 30 years ago, when there was still a significant inhouse expertise and staffing level.

  13. @Ikonoclast

    Your points are taken, Ikono, but I’m not sure I’d credit it all to the wit and wisdom of the “neoliberals” (or econorats as I prefer to call them).

    I think the process has been aided – as is always the case – by lots of “useful idiots” who just take up the current fashion as though it was the delivered word of almighty truth. But that was ever thus, yes ?

    It’s been a long descent from “the light on the hill” though, hasn’t it.

  14. Perhaps this is also the hidden agenda for Medicare. If you strip it of personnel and
    Make it harder to get any service, then you drive people into private health insurance.
    A tax penalty for not having private health insurance certainly cannot hurt this plan.

  15. Greg McKenzie,

    Certainly, this is a deliberate part of neoliberal strategy. When the CES (Commonwealth Employment Service) was abolished in favour of privatised job search providers, Employment National was also set up. It was the government provider which (sort of) replaced CES. There are studies and media reports pertinent to the time which argue that Employment National was deliberately set up to fail. And indeed, it soon did fail leaving the field wide open for private providers. The private job search industry created new millionaires like Kevin Rudd’s wife and made companies like Sarina Russo more lucrative.

    It is a standard neoliberal technique to starve public services of operational funds. Then they are criticised for being poor services and the push starts to privatise them because private enterprise “does everything better”.

    Here is Albanese criticising this closure of Employment National in 2000.

    What is instructive is that when Labor gained power later (Kevin 07), they did not change the system back to a government system. Wikipedia tells us; “The sale of the Australian arm of Ingeus (Therese Rein’s company) took place in May 2007 to ensure there was no perceived conflict of interest as her husband, Kevin Rudd was the Leader of the Opposition (and later the Prime Minister of Australia). The Australian businesses sold in October and December 2007. Ingeus re-entered the Australian market with the acquisition of Assure Programs in October 2011.”

    One can see in retrospect that Rudd and Rein had a long term interest in not repealing or reversing the privatisation of employment services. Thus the conflict of interest, while technically and legalistically resolved for the duration of Rudd’s first Labour leadership, was not substantively, permanently or ethically resolved. This raises the general issue of how the “Labor Traitors”, as I call the entire Labor Party modern professional political class, have the same interests as all people of the neoliberal class. The Labor Party’s modern professional political class are neoliberals through and through.

    Neoliberalism is bi-partisan policy whereby the LNP implement the new neoliberal policies. The ALP pretend to oppose these policies in opposition. When the ALP gets into power, they go quiet on these policies for the most part and allow the neoliberal policy innovations to remain in place. They might throw out a few cosmetic sops to con or appease Labor voters. Thus neoliberal policy gets ever ratcheted to the right. The LNP moves the policy to the right each time and the ALP is the ratchet stop which holds the policy at that level until the LNP can get into power again.

    Given that big business in Australia donates to both parties, there is very likely a strategic policy understanding and intent in neoliberal circles for the entire dual main party system to function in this manner. There is a strategic manipulation and subornation of our entire mainstream politics by big business interests. Personally, I think it is the height of naivety to think that modern capitalist politics is not characterised by this level of strategic intent, insight, planning and Machiavellian spirit.

  16. @Ikonoclast

    As I once said to someone in here, Ikono, “Its been a long way down from the light on the hill”. Oh yes, that was you !

    But yes, look at the Labor leadership since then: Arthur Caldwell, Gough Whitlam, Bill Hayden (as least he brought Medicare/Medibank in), Bob Hawke, Paul Keating, Simon Crean, Mark Latham, Kim Beazley, Kevin Rudd, Julia Gillard, Bill Shorten. As hopeless a lot as you could find anywhere, any time.

    However, surely I don’t have to remind you that the LNP was no better ? Robert (Pig Iron Bob) Menzies, Harold Holt, John Gorton, Billy “Big Ears” McMahaon, Billy Snedden, Malcolm Fraser, Andrew Peacock, John Howard, Andrew Peacock, John Hewson, Alexander Downer, John Howaqrd, Brendan Nelson, Malcolm Turnbull, Tony Abbott, Malcolm Turnbull. Even worse, maybe, than the Labs !

    In short, I don’t trust any of them, with the possible exception of John Hewson of having the “smarts” to enact the “conspiracy” you think has been going on.

    So, are we really ruled by the Bilderbergers after all, or, as one of my now deceased friends used to say” “Have you ruled out stupidity ?”

    I haven’t. I just see an ongoing stream of intelligence-restricted (in both senses of ‘intelligence’) careerists who keep doing what we keep on voting them in to do.

  17. @Ikonoclast

    Ooh, just a wee bit of serendipity after I posted the above: do you read the Club Troppo blog ? If not, you should because it’s often worth a read, especially for Nick Gruen – who is a reformed PubServe and one time “privatization” believer. Now he’s just a touch more skeptical.

    Anyway, you may enjoy his latest post, just Google:
    Club Troppo Choice, competition, markets and human services: Some thoughts

    (I try to avoid URLs unless absolutely necessary as it often takes John several days to “moderate” my post when I include links in them).

  18. @Ikonclast

    Yair, but I didn’t realize, until I typed that list in, just how much the Libs have “blown in the wind” over the years.

    But do look up that Club Troppo article, I think you’ll enjoy it.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s