Problems with probabilities

Peter Hartcher is an insightful commentator on political issues, but we are all prone to fallacious reasoning about probability, and this article about Australian views of the US election illustrates quite a few of them. I don’t mean to pick on Hartcher, whose errors here are trivial compared to the practice of deriving strong conclusions from trivial fluctuations in poll numbers, but this is, as they say, a learning opportunity. Hartcher notes that most Australians, like most people everywhere outside the US, would prefer Obama and goes on to say

But Australians’ answers to another poll question on the US election were troubling. Asked which candidate they expect to win, 65 per cent name Obama and only 9 per cent Romney in the poll conducted by UMR Research.

This is not a question about preferences but expectations. And it is far removed from the realities in the US. The contest for the presidency is finely balanced.

The average result of eight leading polls of US voting intentions shows 46.9 per cent of Americans support Obama and 45.5 per cent Romney, according to That’s a difference of just 1.4 percentage points, which is within the margin of polling error. For statistical purposes, it’s a dead heat.

”Australians could be in for an unpleasant surprise on November 6,” the UMR Research pollster Stephen Mills observes.

Read more:

There are lots of problems here.

The first is that the statistical margin of error in opinion polls, commonly stated as 3 per cent, is the 95 per cent confidence interval for a single poll with a sample size of about 1000. With 8 polls, the confidence interval is more like 1 per cent. (Going the other way is the fact that the real problem with polling data isn’t sample size, but sample biases and non-sampling errors, such as the difference between an answer to a poll question now and a voting decision in November.

The second is that, if you want to use statistics as a guide to prediction, you need a Bayesian approach, rather than the classical hypothesis testing approach associated with terms like “statistically significant”. On the Bayesian approach (and starting without strong prior beliefs), a lead of 1.4 per cent gives pretty good odds in favor of Obama, even if it were derived from a single poll. Both simulation approaches like that of Nate Silver’s 538 blog and actual betting markets have Obama at 2/1 on.

But the really big problem is this. Suppose Obama is only a slight favorite, say 52-48, and you are asked who you expect to win. Presumably if you have no special information, you will answer either “Obama” or “too close to call”. Now suppose 100 people are asked the same question. Intuition might suggest that they should divide 52-48. But if you think more carefully, everyone (except those with inside info or strongly held beliefs) is in the same position as you. That is, the only sensible answers are “Obama” or “too close to call”. Since 91 per cent of respondents gave one of those two answers, there’s no reason at all to regard Australians as deluded.

To give my own views, I think the current odds somewhat understate Obama’s chances. The last few weeks have been bad for the Repubs in ways that have yet to percolate through to public opinion. For example, the choice of Ryan as VP candidate was expected to please the centrist media as well as the base, but Ryan’s fraudulent claims to be a “deficit hawk” were shot down by Krugman and others before they could take wing. The handful of centrists who endorsed Ryan on this basis, such as William Saletan at Slate, are now licking their wounds, while the Akin flap in Missouri has highlighted Ryan’s extreme views on social issues. That’s not in the polling data, but it will affect the way the Repubs are covered beteween now and November. So, I’d put Obama more like 3 to 1 than 2 to 1. Still, Hartcher is absolutely right that this is too close for comfort for the overwhelming majority of people in the world who would certainly choose Obama, if not with the enthusiasm with which he was greeted in 2008.

53 thoughts on “Problems with probabilities

  1. “If you want to use statistics as a guide to prediction, you need a Bayesian approach.” Not true. Frequentist can produce forecasts with prediction errors. But I take your point that a hypothesis test of significance is not all that useful here.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s