Archive for January, 2003

Strocchi meets Soon

January 22nd, 2003 Comments off

I don’t suppose this blog will ever be responsible for one of those Internet marriages we’re always reading about. But it can claim at least some credit for introducing Hayek fans, Jason Soon and Jack Strocchi, although their first encounter was scarcely promising. That was all sorted out, and now Jack is adding to the formidable intellectual firepower at Catallaxy.

As I see it, I’ve not so much lost a commentator as gained a fellow-blogger.

Categories: General Tags:

Great Minds Think Alike Dept.

January 22nd, 2003 Comments off

Jason Soon at Catallaxy Files writes

A new paper with a completely different spin on things can be found on the excellent AEI-Brookings Joint Centre for Reg. Studies website. The paper, by Mark Nadel, argues that there is little economic justification for laws against unauthorised copying because the higher revenues this generates for popular creations, are in ‘winner take all’ entertainment markets, generally used for promotional efforts and that such efforts crowds out many borderline creations.

In one of my first forays into this debate, in the AFR in 1998, I took an almost identical line, arguing

Turning to the intellectual property arguments, the key difficulty here is the failure to recognise the network externalities associated with fashion-driven markets like that for poular music. The popularity of, say, the Spice Girls, is not due to the fact that they are more talented than the next-best group or that their songs are better written, or even that their marketing is cleverer. Rather, the Spice Girls are popular primarily because they are popular. People, especially teenagers, want to listen to the same music their friends are listening to. Allowing record companies to extract the maximum possible rent through discriminatory pricing will not lead to the greater support for new and innovative music. Rather, it will encourage the dissipation of yet more resources in attempts to capture control of the next big hit.

I will read Nadel’s piece more carefully and perhaps post some comments.

Categories: General Tags:

The flat tax system is already here

January 22nd, 2003 Comments off

One of the debates that’s been raging in the US, and that I haven’t had time to mention here has arisen from the claim by the Wall Street Journal that the poor and working class are ‘lucky duckies’, who don’t pay their share of the tax burden (some ambiguity has arisen over whether the WSJ actually wants the poor to pay more taxes, or merely to exclude them from tax cuts until the WSJ constituency has had its tax burden driven as close to zero as possible). Among others, CalPundit has done a good job of demolishing this nonsense, which relies on an exclusive focus on Federal income taxes, excluding not only indirect taxes and state and local taxes, but even payroll taxes.

The only point I want to add is that all of this has long been known to anyone with even the slightest knowledge of tax policy. Looking at my not-so-best-seller Taxing Times, from 1998, for example, I find on p21

In Australia, and most other OECD countries, the tax system as a whole is roughly proportional. The progressivity of income taxes is offset by the regressivity of most other taxes, and by the fact that high-income earners have more extensive opportunities for tax avoidance than low-income earners.
The redistributive effect of taxation and expenditure in Australia arises from the fact that, while taxation payments are roughly proportional to income, the benefits of public expenditure are about the same for all members of the community.

. I am not, of course, claiming any originality for this observation – it’s so well-known that I didn’t even bother to source it.

All of this underlines the point that the WSJ and those who have supported them on this issue are either incompetent or dishonest. No-one qualified to write on tax policy could be unaware of the facts in this question.

Categories: General Tags:

Poetic Justice Dept

January 22nd, 2003 Comments off

I’ve argued already that recriminations about the Canberra bushfires are premature. However, the recriminations have started and the leading fingerpointer has been Commonwealth Regional Services Minister Wilson Tuckey. It’s with some interest, therefore, that I read the headline on the front page of today’s AFR, “Fires: Tuckey ignored his own advice”, above a story that establishes pretty clearly that Tuckey has failed in his responsibilities as badly as, or worse than, any other participant in the policy process, and bears a fair share of the blame for the disaster at the weekend. He should resign immediately, but of course he won’t. Failing that, Howard should sack him, but of course he won’t.

Categories: General Tags:


January 22nd, 2003 Comments off

My last AFR piece on unemployment has been reproduced at australian policy online. And today’s SMH has a piece, at least partly in response from John Edwards. The argument has also attracted some criticism from Stephen Kirchner . Stay tuned for a summary and response.

Categories: General Tags:

Good news in the Lott saga

January 22nd, 2003 Comments off

With the continuous uproar about academic dishonesty, it’s nice to report some good news, even if it concerns someone with whose views I disagree. Pro-gun Chicago academic John Lott has been widely suspected of fabricating a survey he claimed to have undertaken in 1997, concerning defensive uses of guns. He has now produced someone who claims to have been a respondent to the survey, and whose story appears credible. You can get the full story here, from Tim Lambert, an Australian critic of Lott’s work.CalPundit has more comments. In my view, unless there are new developments, the ‘case’ of fabrication against Lott must be dropped.

On the other hand, Lott’s own account shows the 1997 survey to have been slipshod in every respect, from survey design to documentation to statistical analysis. All Lott’s records of the survey were apparently wiped in a crash on his personal computer shortly after the survey had been undertaken, at a time when standard practice would call, not only for backups, but for the preservation of original interview records. And the sample size was clearly far too small to justify the conclusions Lott claimed to draw.

This survey is not of course, Lott’s most important work. The study by Lott and Mustard, purporting to show that laws allowing ‘concealed carry’ of firearms lead to a reduction in crime is much more important, and the data set for this study is publicly available. But as I pointed out in my post on data mining, with modern computer methods it’s easy to discover spurious correlations, either accidentally or deliberately. Only if you take painstaking care to record and justify your hypothesis testing procedure can the associated significance tests be taken seriously.

Given Lott’s own account of his methods, it is clear that he does not take the kind of care needed to avoid the pitfalls of data mining. Thus, his result must be regarded as suggesting a hypothesis for further testing, rather than providing substantive evidence in support of that hypothesis. The obvious further test is ‘out of sample’ testing using the same model with similar data, not used in the original estimation.

According to Tim Lambert , the work of Lott and Mustard fails this test.

In the case of Lott’s model we are in the fortunate position of being able to test its predictive power. Lott’s original data set ended in 1992. Between 1992 and 1996, 14 more jurisdictions (13 states and Philadelphia) adopted carry laws. We can test the predictive power of Lott’s model by seeing if it finds less crime in those jurisdictions. Ayres and Donahue?[2] have done this test. They found that, using Lott’s model, in those jurisdictions carry laws were associated with more crime in all crime categories . Lott’s model fails the predictive test.

Ayres and Donahue go on to examine all the states adopting carry laws using data up to 1997 and found that carry laws were associated with crime increases in more states than they were associated with decreases. They rather pointedly observe that

Those who were swayed by the statistical evidence previously offered by Lott and Mustard to believe the more guns, less crime hypothesis should now be more strongly inclined to accept the even stronger statistical evidence suggesting the crime- inducing effect of shall issue laws.

I haven’t checked these references myself, but unless Lambert has something badly wrong in his summary, I think it’s safe to disregard both the results Lott claims for his lost survey in 1997 and the statistical results of the Lott and Mustard study.

Categories: General Tags:

Thoughts for Thursday

January 22nd, 2003 Comments off

Because of the fires, I didn’t open up the message board as usual on Monday. So rather than put up my own “Thursday thoughts”, I’m asking for your comments (civilised discussion and no coarse language) on any topic. I’d particularly like further ideas on blame, chance and Fate, which has been the subject of some valuable discussion in earlier comments threads this week.

Categories: General Tags:

Back to normal?

January 21st, 2003 Comments off

As usually seems to be the case, it’s the first day of surprise when the fires do their worst. The latest news seems to indicate that, barring an unexpected change in the weather, the fires threatening Canberra have been kept behind their containment lines, and cool weather should also improve the situation further south. Like everyone else, I’ve focused first on the loss of life and the destruction of so many homes, but the destruction in the Alpine National Parks is also a huge blow.

Of course, fire is a natural phenomenon in Australia, as is the drought that sets the stage for it, and it’s very hard to distinguish random fluctuations from an underlying trend. Still, I think the evidence supports the view that the severity of the current drought reflects the impact of long-term global warming as well as shorter-term cycles such as El Nino. If so, we can expect more of the same in the future. I’ll post on this point in more detail in future.

Another issue that has been raised is the reliance of our culture on blame when things go wrong. This may be compared with other cultures which focus on a notion of Fate. I’d like to explore the question of chance, fate and blame in more detail in the future.

As all of this indicates, in the absence of further disasters, I’m planning a return to normal blogging on a range of issues, probably starting tomorrow.

Categories: General Tags:


January 21st, 2003 Comments off

As you might expect, it hasn’t taken long for the recriminations to start – not enough firebreaks, too many trees, disorganisation at the scene etc. No doubt some of this will prove justified, but there’s plenty of time to sort this out later. Given the kind of test provided by the worst drought in 100 years, and extreme weather conditions, it’s inevitable that mistakes will have been made.

Among the print media, PP McGuinness is predictably nasty and ill-informed. But as in many other things, the blog world outdoes the established media in nastiness. The prize goes to this piece from “Strawman” reproduced in full by Australian Libertarian Society* (summary: tree-huggers deserved all they got). I don’t know if this is meant as a joke and I don’t particularly care. Tim Dunlop has more.

A lot of the discussion has given a quite misleading impression of what the western edge of Canberra is actually like – you get the impression of a city built in a national park. Actually most suburbs are bordered by open grassland used for grazing cattle and horses. Judging by the media reports and the experience of the past, the big fuel source has been pine plantations that were set up (I think) during the Depression and have been part of the Canberra landscape ever since, though presumably no longer. Pines are more even flammable than eucalypts which, after all, evolved to be fire-tolerant.

* The ALS is based largely in Canberra – make of that what you will

Update In the comments thread, Jason Soon points out that the ALS site is organised as an unmoderated collective blog, and that “Strawman” is a member of the collective who can choose to repost his own thoughts freely. I had incorrectly assumed that some sort of conscious decision to reproduce this piece had been made by someone other than the author. As Jason says, his Catallaxy Files site is organised in the same way and he often ends up contradicting his own guest bloggers.

It was therefore, premature of me to suggest that the viewpoint of one ALS member, posted on their site, is representative of the group as a whole. I’ll be interested to see whether anyone chooses to put a different viewpoint.

Further update John Humphreys of ALS has posted a personal account of his bushfire experience in the Forum section of ALS (links are still not very good, but it’s not hard to find). This isn’t a response to “Strawman” and doesn’t seek to draw any political conclusions, but it certainly suggests that the post I referred to reflects on the character of its pseudonymous author rather than on ALS as a whole.

Categories: General Tags:

A farewell

January 21st, 2003 Comments off

Bloggers come and bloggers go, and at the moment departures seem to be outnumbering arrivals. I’m particularly disappointed to see the end of bertramonline a Danish blog I only recently discovered. As with many others, Henning Bertram decided that the time required to do a decent job of blogging is too much.

Categories: General Tags:


January 20th, 2003 Comments off

If one good thing could come out of these terrible fires, it’s that people might stop using “Canberra” as a synonym for the Federal government instead of what it is, an Australian city and the home of ordinary Australians who will, I am sure, overcome this disaster.

Categories: General Tags:


January 20th, 2003 Comments off

I probably won’t be posting much for the next couple of days, while I digest the impact of the dreadful fires in Canberra. As far as I can tell at present, my family and friends there haven’t been badly affected, though I haven’t heard news yet of some ANU colleagues who live in areas that have been hit hard. As usual, I can’t find much to say in the face of disasters like this, except to extend my sympathy to all those affected.

Categories: General Tags:

What I'm reading this week

January 19th, 2003 Comments off

Cold Mountain by Charles Frazier. The protagonist is a deserter from the Confederate armies, trying to make his way back home to the woman he hopes to marry. It’s bleak, but also inspiring.

I’m also continuing to reread the many works of Anthony Trollope. He’s my kind of writer, a craftsman rather than an artist. He claimed to turn out 2500 words before breakfast, and to regularly write at a rate of 250 words per quarter-hour. I suspect this is gilding the lily a bit . Even if he didn’t spend hours agonising over le mot juste, this pace would imply that he knew exactly what he was going to write when he started, allowing no time even for rearranging sentences. Nevertheless, with 47 3-volume novels to his credit, he must have averaged around 1000 words of final output every working day. I read somewhere that George Orwell wrote nearly 200 000 words in a single year. He would have been using a typewriter, as opposed to Trollope’s pen and ink. Of course, the romantics despise this kind of thing, but my view is that the more you write, the more good stuff you write.

Categories: General Tags:

Song for Saturday

January 18th, 2003 Comments off
Blue Thunder (A Modern Bluey Brink)

There once was a trendy who scorned worldly wealth
And devoted his life to a search for good health,
He spent all his money at his local health store
Took potions in gallons and pills by the score

But sad to relate every time that it rained
His house it was flooded from roots in the drains
To cure it the landlord gave him a pill
To flush down his dunny, them tree roots to kill

Next morning he followed his usual routine
By swallowing tablets in number thirteen
But somehow or other one extra crept in
It was big, it was blue, it was bitter as sin

This pill was so big that it stuck in his throat
It made him to splutter and it made him to choke
But he swallowed it down like he knew that he should
If it tasted that vile, well it had to be good

Now soon he was stricken with aches and with pains
As the pill started working to clear out his drains
He was stuck on the dunny for a night and a day
Weighed fifteen pounds less when he staggered away

Well it made him feel weak but it made him feel pure
So he told all his friends of his miracle cure
They asked him the secret but he wouldn’t tell
After buying for years he had something to sell

And now he is running his own health food shop
Selling soya bean sausage and Gaylord’s green glop
And his own extra-specialty, cure for all ills
They’re bio-dynamical Blue Thunder Pills

Categories: General Tags:

One for the record books

January 18th, 2003 Comments off

I just thought I’d record this statement (made in connection with the discovery of empty chemical warheads in Iraq) by Andrew Sullivan for future reference

There can be no further excuses. Saddam had one absolutely last chance and he lied. If we do not go to war now, then Bush, in turn, will have been shown to have lied in his countless statements declaring zero tolerance for future violations.

Given Sullivan’s generally hagiographic style in relation to Bush, even a future conditional use of the word ‘lie’ is striking.

While I’m on this point, today’s Age reproduces a piece from the LA Times giving some factual background, including the point:

Dozens of rockets are usually fired at once, to try to ensure that enemy troops are enveloped in a thick concentration of chemicals.

This seems relevant in assessing the status of the 12 warheads found so far.

Categories: General Tags:

Real jobs in rural and remote Aboriginal communities?

January 17th, 2003 Comments off

Following my posts the Windschuttle controversy, I promised to put forward some ideas on the current policy problems facing Aboriginal Australians, and particularly the problem of economic development. It’s always problematic for white ‘experts’ to tell black communities what to do and I want to make it clear that I’m not trying to do this. Although I have given economic advice to Aboriginal organisations on a range of issues, I don’t regard myself as an expert on the problems facing Aboriginal communities. My perspective on the issue comes more from a consideration of the general economic problems of rural Australia and particularly the general decline in population and employment.

Discussion of economic policy for rural and remote Aboriginal communities often poses a dichotomy between passive reliance on government welfare and economic independence. The assumption is that reliance on government is inherently demoralising.

I have come to the conclusion that, at least in the terms in which economic independence is commonly understood, this kind of thinking is mistaken. Against a background of generally declining demand for labour in rural and remote Australia, it is unlikely that the majority of Aboriginal communities will ever be economically independent in the sense that they produce enough in terms of market goods and services, to ‘pay their way’ in a free-market economy, even with an inflow of capital income from land and mineral rights. Aboriginal communities have all the economic problems facing declining country towns as well as the additional difficulties arising from a century or more of dispossession and discrimination. If most country towns can’t find a way to hold enough jobs for a generally shrinking population, it seems impossible that Aboriginal communities, many with growing populations, can ever ‘pay their way’.

Having said that, I will observe that for most of the 20th century, large sections of the Australian workforce did not ‘pay their way’ either, relying instead on tariff protection or subsidy schemes. This did not seem to do much harm to the self-respect of Australian workers. It’s my view that if ‘practical reconciliation’ is taken to include the objective of generating sustainable employment in rural and remote Aboriginal communities, we have to accept that employment will be sustained only with permanent government support, just as was true of large sections of manufacturing industry. The obvious approach is permanent wage subsidies, though in some cases output subsidies to enterprises based in Aboriginal communities might also work.

The closest approach so far has been the Community Employment Development Program under which communities can use social security benefits to pay for useful projects. This scheme (which has elements of ‘work for the dole’) has been possible only because it has been treated as welfare rather than as a labour market program. The idea of a permanent wage subsidy, explicitly confined to rural and remote Aboriginal communities breaks so many policy taboos that it is unlikely to make it on to the policy agenda for a long time. But I can’t see any alternative approach that will generate substantial numbers of permanent jobs.

Categories: General Tags:


January 17th, 2003 Comments off

Can any scientifically literate readers comment on thisABC Report, which says

research reported in the science journal Nature challenges basic beliefs of evolution – that wings evolved only once in insects and that if a trait is lost is cannot be regained.

It also opens a new direction for research because it shows that once a complex figure has evolved it can be maintained over long evolutionary period even if it is not apparent on the outside.

Over a 50 million year period, even though the stick insects did not have wings the genes for creating them appeared to have been maintained.

“The remarkable thing was that they had the ability to generate wings when they needed them,” Mr Whiting explained.

I was under the impression that the standard mechanism in the loss of things like wings was the selection of genes that ‘turned off’ crucial steps in development. If so, it would obviously be easier to reverse this process than to evolve wings for the first time. Am I wrong, or is the article over-dramatizing things?

UpdateThe BG strikes yet again. Brad DeLong has an almost identical reaction with a reference to “Darwin’s Radio ” – must read this.

Categories: General Tags:

The BlogGeist strikes again

January 17th, 2003 Comments off

I was going to comment on the disastrous decision of the US Supreme Court in Ashcroft vs Eldred, upholding new laws which amount to Copyright Perpetuity. I thought I’d point out that Australia still has a more reasonable position where copyright last 50 years after the author’s death, speculate on the implications for the proposed Free Trade Agreement and wrap up with a link to Kim Weatherall and a pithy comment. When I looked at her site, I found

Another thing we can look to, I think, in the current round of Free Trade Negotiations between Australia and the United States, is pressure from the States, freed from the spectre of constitutional challenge, to Australia to extend its copyright terms to match those of the United States. So perhaps what we REALLY need is some pressure back here in Australia.

So my pithy comment is “Me too!”

Categories: General Tags:

Limited liability and the nanny state

January 17th, 2003 Comments off

Ozplogger James Morrow has a piece in Wednesday’s Oz, saying

If a gambler sitting at the Crown Casino in Melbourne knew that the Government would cover his losses, no matter what, he’d have no incentive not to draw to an inside straight or hit on 17. Likewise, a board of directors, if they believe their business is politically “too big to fail” and could qualify for a government bailout, will surely be more inclined to take the sort of big risks that would in the long run cost a gambler his house and car and livelihood.

And though a casino is able to force a loser to pay his debts, in the corporate cowboy world of Adler and Williams, losses go unpaid while the gamblers walk around with huge personal fortunes.

I agree with the sentiment and with Morrow’s concerns about ‘moral hazard’, but I think he has aimed at the wrong target.

It’s not government bailouts that let Adler, Williams, Rich and others walk away from failed companies with their personal fortunes intact. Under the institution of limited liability, this is the rule, not the exception. Assuming the company is not preserved as a going concern, government bailouts of the kind we saw with HIH don’t give any additional help to the managers and shareholders, only the creditors (in this case, policyholders, and in other cases employees).

James writes for Reason and there are quite a few bloggers of a libertarian bent. I’d be interested to read their views on limited liability and, for that matter, personal bankruptcy.

Update As I hoped, this post has generated a lively comments thread. Come and have your say – I’m enjoying the bloggers’ privilege of asking a rhetorical question without revealing my own answer for a while.

PS I forgot to thank Gareth Parker, whose link I followed.

Categories: General Tags:

Creeping renationalisation

January 17th, 2003 Comments off

I’ve been arguing for some years that (re)nationalisation should be put back on the policy agenda. The starting point is in reversing obviously failed privatisations, such as that of British Rail. Following the replacement of the private Railtrack company (which owned the network) by the quasi-public Network Rail The Guardian reports the partial renationalisation of track maintenance. More precisely, Network Rail has sacked the private contractors who were doing this job on one of the most accident-prone stretches of track and taken it back in-house.

Network Rail’s announcement was the latest in a series of moves to increase central control over the railways, prompting suggestions of “creeping renationalisation”. In November the strategic rail authority announced that train operators would be moved on to shorter, stricter franchises.

The privatisation of maintenance was at the centre of Ken Loach’s great film The Navigators, which I reviewed for the Canberra Times (my only venture into film reviewing so far). In the review, which is available at Australian Policy Online, I observed

The tendency to shave safety margins an ever-present feature of the privatisation process. It reflects a shift of power from engineers to accountants, and more generally from a production objective to a profit objective.

APO has lots more good stuff and a very nice-looking site.

Categories: General Tags:

I Told You So Dept.

January 16th, 2003 Comments off

Ever since Bush’s turn to the UN, I’ve been debating the significance of UN resolution 1441 on Iraq with other bloggers, notably including Steven Den Beste, who initially thought Bush was knuckling under, then changed his mind. When the first draft was announced, that it was, in effect, a one-and-a-half resolution I observed that it effectively required a second UNSC vote

All that’s left in the reported draft resolution, and in the statements of the UK and US governments, is that, if inspectors report obstruction to the UNSC, the US and UK will not necessarily accept a veto on military action cast by, say, France.

Although, the US Administration is still resisting this interpretation, Tony Blair has now accepted it almost word for word

Mr. Blair said, “Of course we all want a second U.N. resolution. I believe we will get one.

But he added, “Where there is an unreasonable veto put down, we will not rule out action.

On this issue, Blair’s interpretation is more important than Bush’s. The US Administration has already made up its mind that it wants a war, whereas the British government has not.

Of course, all of this may turn out to be academic (in the pejorative sense) if the latest reported discovery of shells for chemical weapons turns out to provide the much-sought-after ‘smoking gun’. On the other hand, the precise interpretation of 1441 will be very important if the inspectors turn up incriminating, but not conclusive, evidence against Saddam. It’s still too early to tell on this one.

Categories: General Tags:

Thoughts on Thursday

January 16th, 2003 Comments off

My Op-Ed piece in today’s Financial Review (subscription required) is on unemployment. A short grab:

At no time since the election of the current government has unemployment been an issue of real concern. Second-order trivia like the GST and waterfront reform have had far more attention. And, sadly, the Australian public has become inured to chronic mass unemployment. In the absence of a severe economic downturn, the government will pay no real political price for its worst policy failure.

Also, today the Committee for the Economic Development of Australia released a review of Australian experience with privatisation. You can download the <a Introduction here (PDF). Most of the contributors are broadly sympathetic to privatisation, though less so in the case of regulated monopolies. I argue, as I have done for some time, that we should be looking at renationalisation in many cases.

Most interesting to me was a study by Jonathan Kelly and Joanna Sikora showing that public opinion against privatisation has hardened steadily over time. In 1986, views on the privatisation of Telstra were about evenly divided. By 2002, 70 per cent were opposed and only 16 per cent in favour. Similar views apply even to firms like the Commonwealth Bank and Qantas that have been privatised for years, and opposition is even stronger in the case of Australia Post, the only business in the study still in full public ownership.

It’s difficult to attribute this to emotional attachments to Aussie icons, in view of the fact that people were quite willing to contemplate privatisation in the 1980s. Advocates of privatisation hoped that experience would lead people to accept it. In fact, the reverse has been the case, here as in Britain and New Zealand. People have experienced privatisation and they don’t like it.

In view of the rhetoric about elites we’ve heard so much about lately, it’s worth pointing out that the divide between elite and popular opinion is far sharper here than in the case of asylum-seekers.

Categories: General Tags:

Data mining

January 15th, 2003 Comments off

‘Data mining’ is an interesting term. It’s used very positively in some academic circles, such as departments of marketing, and very negatively in others, most notably departments of economics. The term refers to the use of clever automatedsearch techniques to discover putatively significant relationships in large data sets.

The paradigm example, though a very old-fashioned one is ‘stepwise regression’. You take a variable of interest then set up a multivariate regression. The computer then tries out all the other variables in the data set one at a time. If the variable comes up significant, it stays in, otherwise it’s dropped. In the end you have what is, arguably, the best possible regression.

Economists were early and enthusiastic users of stepwise regression, but they rapidly became disillusioned. To see the problem, consider the simpler case of testing correlations. Suppose, in a given dataset you find that consumption of restaurant meals is positively correlated with education. This correlation might have arisen by chance or it might reflect a real causal relationship of some kind (not necessarily a direct or obvious one). The standard statistical test involves determining how likely it is that you would have seen the observed correlation if there was in fact no relationship. If this probability is lower than, say, 5 per cent, you say that the relationship is statistically significant.

Now suppose you have a data set with 10 variables. That makes 45 (=10*9/2) distinct pairs you can test. Just by chance you’d expect two or three correlations that appear statistically significant correlations. So if your only goal is to find a significant relationship that you can turn into a publication, this strategy works wonders.

But perhaps you have views about the ‘right’ sign of the correlation, perhaps based on some economic theory or political viewpoint. On average, half of all random correlations will have the ‘wrong’ sign, but you can at expect to find at least one ‘right-signed’ and statistically significant correlation in a set of 10 variables. So, if data mining is extensive enough, the usual statistical checks on spurious results become worthless.

In principle, there is a simple solution to this problem, reflecting Popper’s distinction between the context of discovery and the context of justification. There’s nothing wrong with using data mining as a method of discovery, to suggest testable hypotheses. Once you have a testable hypothesis, you can discard the data set you started with and test the hypothesis on new data untainted by the process of ‘pretesting’ that you applied to the original data set.

Unfortunately, at least for economists, it’s not that simple. Data is scarce and expensive. Moreover, no-one gets their specification right first time, as the simple testing model would require. Inevitably, therefore, there has to be some exploration (mining) of the data before hypotheses are tested. As a result, statistical tests of significance never mean precisely what they are supposed to.

In practice, there’s not much that can be done except to rely on the honesty of investigators in reporting the procedures they went through before settling on the model they estimate. If the results are interesting enough, someone will find another data set to check or will wait for new data to allow ‘out of sample’ testing. Some models survive this stringent testing, but many do not.

I don’t know how the marketing guys solve this problem. Perhaps their budgets are so large that they can discard used data sets like disposable syringes, never infecting their analysis with the virus of pretesting. Or perhaps they don’t know or don’t care.

Update Kevin Drum at CalPundit gives the perspective of a marketing guy, with lots of interesting points (for example, loyalty programs are there to collect data for mining). He doesn’t accept my main point and raises the dreaded “B” word – Bayesian.

For those familiar with debates among statisticians, this is the point at which things typically become both heated and incomprehensible (just like a lot of blogs, really). A real challenge, which I may tackle at some point, is to explain the Bayesian concept of statistical reasoning in ordinary language. For the moment, though, I’ll just agree that a debate like the one over data mining ultimately makes sense only if it’s cast in Bayesian terms, that is, with a discussion of the beliefs we hold before we begin the statistical analysis.

Categories: General Tags:

The heart of the matter

January 15th, 2003 Comments off

In two excellent articles, here and here,Thomas Friedman gets to the heart of the problem facing US Middle Eastern policy. To summarise, both Israelis and Palestinians are stuck with leaders who have repeatedly failed them, and led them into a cycle of attacks and reprisals. While the cycle goes on, neither side is prepared to gratify the other by dumping their failed leader and electing one who will make peace. By backing Sharon, the US ensures that it is hated by all those who support the Palestinians, which means virtually everybody in the region outside Israel. This in turn means that the idea of a democratic revolution in Iraq is doomed from Day One. Any democratically-elected government would be more fundamentally anti-American than Saddam (an American ally who merely miscalculated what he could get away with in Kuwait).

I’ll add my own analysis to this. What this means is that the Administration got its policy in the wrong order. It should first have leant hard on both sides until they agreed on peace terms essentially on the lines of the Clinton plan (though the paternity would obviously have to be denied). Once people in the rest of the region saw Israeli settlements actually being dismantled, Bush would have had a free pass as regards Iraq, and democratic reconstruction would actually have been possible.

It might still be possible to resurrect this strategy, by dragging out inspections and keeping the pressure on Saddam as long as possible. If the Israeli elections produce some swing to the left, and the US starts applying pressure, it might be possible for the Palestinian leadership to curb terror attacks long enough to reach an agreement. Then we might see a Palestinian state with democratic elections – Arafat might win the first of these, but not the second. And as I’ve said, success on this front would give Bush plenty of political capital to use in justifying an attack on Saddam.,

That’s a lot of ‘mights’ and ‘ifs’ but I think this offers a better chance than occupying Baghdad and then working out what to do next.

Categories: General Tags:

Cool consumers

January 15th, 2003 Comments off

In today’s Age, Thomas Frank writes:

One of the most tenacious myths of the “culture wars” that have been going on in America for more than 30 years is that youth counterculture possesses some sort of innate transgressive power – that the eternal battle between hippie and hard-hat, disco-dweller and churchgoer or individualist and conformist is every bit as important as the struggle between classes used to be.

In a review of Frank’s last book, One Market Under God, I wrote

The most interesting part of Frank’s story relates to the way in which the entrepreneurs of the Internet boom appropriated the rhetoric of the Vietnam-era left, to the extent that venture capitalists refer to themselves as VC and the employees of Internet firms call themselves ‘dotcommunists’. Simultaneously claiming victory in the cold war, they denounced both governments and Old Economy corporations as little better than Soviet commissars. In this story, the entrepreneurs stand for the liberation, not of the workers, but of ‘new money’.

A cameo appearance is made by the postmodernists, whose alleged drive to destroy Western civilisation formed the basis of numerous denunciations of ‘political correctness’ in the 1990s. The postmodernist beliefs that all truth is relative, and that ‘there is nothing outside the text’ find their natural 1990s expression in work for the burgeoning advertising industry. The tools of critical theory and radical anthropology are pressed into service in the creation of a mythical reality for, say, a brand of toothpaste. The ultimate outcome of this radical theorising is the view that, not only nations, but individuals are, ultimately, brands, and that most are in dire need or rebranding.

I did another version of this review here. The para I quoted above is from Frank’s latest The Conquest of Cool, which sounds great.

Categories: General Tags:

Peace by timetable

January 14th, 2003 Comments off

Steven Den Beste sets out his scenario for a unilateral US invasion of Iraq

The way that war will begin is that President Bush will pick up his telephone, call General Franks, and say, “OK, Tommie, go for it.” And then American (and British) jets will begin major bombing, and shortly thereafter American (and British) troops will move into Iraq from multiple directions. The war will be brief, and we’ll win.

The fatal flaw in this scenario is in the parentheses. President Bush can’t pick up his phone and order British troops to invade Iraq. Any British participation will require, at a minimum, a Cabinet decision, which in turn will certainly produce demands for a public debate, which will result in a demand for a UNSC resolution. Blair knows this, as does the US Department of State, which explains the recent backing away from Jan 27 as a deadline.

And at this point, the whole thing starts to unravel. In email, Den Beste argues that British participation is an optional extra (which would justify the parentheses). Even in strategic terms, this is unclear – some of the things the British are providing, such as in-flight refuelling, are apparently indispensable in the short run. And in tactical and political terms it’s silly. The British and Americans are co-operating closely on the premise of a joint invasion. A call from Bush saying that American forces only should invade would produce chaos. More importantly it would trash the alliance between the US and Britain which, from the British side, is entirely premised on the idea of a ‘special relationship’ which gives the British government leverage over American policy in return for fairly reliable, and tangible, support.

Admittedly, there’s an element of ‘peace by timetable’ about this argument, but I don’t have a problem with this – war and peace are not symmetric in this respect.

The other interesting thing that became clear in my email discussion with Steven Den Beste is that the rift between the US departments of State and Defense has re-opened. State, which is, as I’ve noted responding to the concerns of the British is trying to slow things down, while Defense is trying to stick to the timetable. Den Beste, who seems to be pretty well attuned to the thinking of the hawks within the US Administration, writes

What has become increasingly clear over the last year is that the State Department as a whole is in serious disrepute with the White House, in part because it’s coming to be seen not so much as America’s foreign policy agency towards the world, but rather as the world’s foreign policy agency toward the US.

Many now suspect that the career bureaucrats in State identify more strongly with the nations they’re assigned to study than with their own. In some cases it appears that they’ve actually been bought.

I think it’s useful to observe that, however much State and Colin Powell are distrusted by some, they won the debate in 2002 about the need to take the UN route and focus on weapons of mass destruction. I suspect they will win again in arguing for delay.

Categories: General Tags:

Roundup Ready vs Roundup Resistant

January 14th, 2003 Comments off

This report that, following the expansion in the use of GM-based ‘Roundup Ready’ crops, the widely used crop herbicide is losing weed resistance doesn’t have any obvious implications for the GM debate other than to remind us that biological systems are complex and hard to manage.

It’s a useful rule, though that, if a policy piece discussing the use of herbicides and pesticides doesn’t discuss or refer to the implications of resistance, it should be treated with caution. I’ll have more to say on this later.

Update As Jim Birch points out, the NYT headline should have read “Weeds are gaining resistance to widely used herbicides”. Embarrassing! This will teach me to rely on the “Blog This!’ default instead of writing in my own link!

Categories: General Tags:

Beaten to the punch

January 14th, 2003 Comments off

The blogosphere prides itself on beating print pundits to the punch. But in today’s Age, Gerard Henderson has a piece bagging the Howard government’s proposed appointment of John Carroll to vet the National Museum. A few blog-eons (months) ago we were treated to a full-length fisking of this guy from Tim Blair for his bizarre book about September 11. I thought about blogging on this a week or so ago, but couldn’t really be bothered. Irony alert on: I’m surprised, though, that Tim missed a chance to put the boot into the government for such a crazy appointment. Irony alert off

While we’re on the subject of the National Museum, I’m surprised by the hostility it’s aroused. Having produced a fair bit of it, I think my antennae are pretty well-tuned to detect material that will infuriate the average Quadrant reader and I found very little of it at the Museum. In its general feel, it reminded me of nothing so much as the opening ceremony of the Sydney Olympics – celebrating ordinary Australians of all kinds. While not ignoring conflict, I don’t think the Museum plays it up, either. The fact that it’s always packed with apparently ordinary Australians supports my view. Of course, most of the hostile reviews had been written before opening day and the Quadrant crew, like the Bourbons, forgets nothing and learns nothing.

Categories: General Tags:

Last rites

January 14th, 2003 Comments off

With the resignation of Steve Case as chairman of AOL Time Warner (soon to be just plain Time Warner again, it seems) we can pronounce the “New Economy” officially dead. When the biggest merger in history was announced almost exactly three years ago (it seems like a century), I wrote

If the accounting numbers are taken at face value, AOL Time Warner will have a price-earnings ratio of about 350 to 1, modest by Internet standards. When options are taken into account, the ratio is more like 1000 to 1. It is difficult to see how an economy in which investment decisions are based on numbers like these can avoid some sort of financial catastrophe.

So is AOL Time Warner a super-profitable monopolist in the making or a jerry-built piece of financial engineering ? It can scarcely be both. But in the miraculous world of the New Economy, anything is possible.

Of course, as the blogworld and other phenomena show, the death of the New Economy has had almost no impact on the development of the Internet. For that matter, neither did its rise. In economic terms, the Internet has provided a source of modest but significant productivity gains in the market sector. However, the trillion dollars or so dissipated in the bubble has probably negated the impact of several years worth of Internet productivity growth.

Coming back to blogs, the big impact of the Internet has been in non-market services, of which email has been the paradigm example, but for which blogs are an even better illustration. Apart from the commodity service of ISPs providing the connection, no-one has ever made any significant money out of these things and, probably, no-one ever will. That doesn’t mean that they don’t count towards economic welfare, but the New Economy was never about economics.

Categories: General Tags:

Lott's more trouble on the scientific front

January 14th, 2003 Comments off

I’ve been receiving a lot of interesting material on both Lomborg and global warming, but this time I’m going to stick to my plan of putting aside, for a while, the academic scandals that have been dominating the blog lately. In the meantime, those who want more can visit Ken Parish for what looks like a mirror-image of the Bellesiles story, involving pro-gun economist John Lott.

I’ll just offer the following thought. In the context of political debate, if a recently-reported scientific result seems too good to be true, it probably is.

Categories: General Tags: