Problems with probabilities

Peter Hartcher is an insightful commentator on political issues, but we are all prone to fallacious reasoning about probability, and this article about Australian views of the US election illustrates quite a few of them. I don’t mean to pick on Hartcher, whose errors here are trivial compared to the practice of deriving strong conclusions from trivial fluctuations in poll numbers, but this is, as they say, a learning opportunity. Hartcher notes that most Australians, like most people everywhere outside the US, would prefer Obama and goes on to say

But Australians’ answers to another poll question on the US election were troubling. Asked which candidate they expect to win, 65 per cent name Obama and only 9 per cent Romney in the poll conducted by UMR Research.

This is not a question about preferences but expectations. And it is far removed from the realities in the US. The contest for the presidency is finely balanced.

The average result of eight leading polls of US voting intentions shows 46.9 per cent of Americans support Obama and 45.5 per cent Romney, according to realclearpolitics.com. That’s a difference of just 1.4 percentage points, which is within the margin of polling error. For statistical purposes, it’s a dead heat.

”Australians could be in for an unpleasant surprise on November 6,” the UMR Research pollster Stephen Mills observes.

There are lots of problems here.

The first is that the statistical margin of error in opinion polls, commonly stated as 3 per cent, is the 95 per cent confidence interval for a single poll with a sample size of about 1000. With 8 polls, the confidence interval is more like 1 per cent. (Going the other way is the fact that the real problem with polling data isn’t sample size, but sample biases and non-sampling errors, such as the difference between an answer to a poll question now and a voting decision in November.

The second is that, if you want to use statistics as a guide to prediction, you need a Bayesian approach, rather than the classical hypothesis testing approach associated with terms like “statistically significant”. On the Bayesian approach (and starting without strong prior beliefs), a lead of 1.4 per cent gives pretty good odds in favor of Obama, even if it were derived from a single poll. Both simulation approaches like that of Nate Silver’s 538 blog and actual betting markets have Obama at 2/1 on.

But the really big problem is this. Suppose Obama is only a slight favorite, say 52-48, and you are asked who you expect to win. Presumably if you have no special information, you will answer either “Obama” or “too close to call”. Now suppose 100 people are asked the same question. Intuition might suggest that they should divide 52-48. But if you think more carefully, everyone (except those with inside info or strongly held beliefs) is in the same position as you. That is, the only sensible answers are “Obama” or “too close to call”. Since 91 per cent of respondents gave one of those two answers, there’s no reason at all to regard Australians as deluded.

To give my own views, I think the current odds somewhat understate Obama’s chances. The last few weeks have been bad for the Repubs in ways that have yet to percolate through to public opinion. For example, the choice of Ryan as VP candidate was expected to please the centrist media as well as the base, but Ryan’s fraudulent claims to be a “deficit hawk” were shot down by Krugman and others before they could take wing. The handful of centrists who endorsed Ryan on this basis, such as William Saletan at Slate, are now licking their wounds, while the Akin flap in Missouri has highlighted Ryan’s extreme views on social issues. That’s not in the polling data, but it will affect the way the Repubs are covered beteween now and November. So, I’d put Obama more like 3 to 1 than 2 to 1. Still, Hartcher is absolutely right that this is too close for comfort for the overwhelming majority of people in the world who would certainly choose Obama, if not with the enthusiasm with which he was greeted in 2008.

53 thoughts on “Problems with probabilities”

1. Way back in MAR 2010 I predicted Obama would win the 2012 election, based on a “triangulating strategy of tacking to the Centre in 2008-12 and Centre-Left for 2012-16”:

My gut feeling is that the current Tea Party movement is a flash in the pan, generating more smoke than fire and will burn out in a year or so. I therefore tentatively predict that Obama will suffer a bit of a short-term set back in the 2010 mid-term Congressional election. …
Over the medium term the absence of a falling-in-sky from this moderate health care reform will reassure white independent voters so he will probably recover and achieve a convincing victory in the 2012 election, better than Bush’s victory in the 2004 election.

Over the longer term he will shift the polity further to the Centre-Left, to readjust the welfare state to adjust to fiscal policy and ethno political reality. In effect over the tennies he will do what Clinton wanted to do over the nineties, but more easily due to the weakening opposition.

So far events have pretty much exactly followed this prediction, indicating that my underlying model is tracking reality. So I am pretty confident Obama will win the election.

I have elsewhere suggested that his major policy priority for the second term will be some kind of climate change legislation/regulation. But this prediction is not looking so hot now.

He will have to have some great cause to mobilise his support. One hopes it won’t be deficit reduction, which was Clinton’s greatest achievement. Thats not exactly going to set his supporters pulse rate racing.

BTW in the interests of scientific accountability I wonder what other pundits and commentators predicted the outcome of this election would be. Have any of them put any models up or is it horse race commentary all the way down?

2. Hartcher should steer clear of derivative trading.

3. Matt W says:

Not to mention that U.S. Presidential elections are determined by the electoral college, not popular vote. Electoral college votes are awarded by state to the winner of the popular vote in that state. They are nominally divided up by census, but it is certainly possible to win the election whilst losing the popular vote (c.f. George W. Bush). Most U.S. states are locked up for one candidate or the other (e.g. California is going to vote for President Obama, Texas for Romney), with only a few states actually in play (Florida, Virginia, Ohio, etc.) Therefore polls in those states are the ones to pay attention to; national polls do not tell the correct story. The progressive site Daily Kos publishes aggregate polling results for the so-called battleground states every week, and those polls should make Obama supporters breathe more easily (http://www.dailykos.com/story/2012/08/23/1123491/-Battleground-snapshot-The-race-in-the-Electoral-College-isn-t-tightening) The President seems fairly likely to win both Ohio and Pennsylvania at this point, and without one of those two states, Governor Romney has to rack up several wins in states that are currently leaning Democrat (e.g. Wisconsin, Michigan, Colorado.) And it’s game over if the President wins Florida or Virginia, both of which are also leaning his way at present.

4. Samuel Conner says:

It looks like a “representative agent” fallacy — think of the American public as a single voter who is 52% likely to vote for O. Just like the American economy is a single representative agent economic actor.

5. Freelander says:

With the binary options (answer Obama or Romney), one or other when asked who you expect to win, with those being asked having identical priors and the same new information every rational computational competent Bayesian would name the same person as the expected winner (presumably, in this case Obama). As they don’t seem to have asked for relative odds, just which more likely than the other, anticipating the magnitude of November :’surprise’ doesn’t seem too easy, or easy, simply based on the data. Statistics is shockingly difficult even with some measure of expertise, so some charity might be called for.

With all the great software churning out statistical studies has for far to long been too easy, with doing it right and interpreting right, not so, but increasingly lives depend on getting both modelling and interpretation right. Not in this case (I’d think).

6. rog says:

People that I know say that the US Republicans are extreme and mad and not to be compared with our own conservatives, the LNP coalition. I would urge caution in adopting that line, our version(s) are equally capable of deranged nuttiness.

7. Tom says:

Watching the US politics always makes me wonder why do any Americans (other than the elites) actually support the Republican at the moment. Romney’s economic paper written by 4 economists including John Taylor and Greg Mankiw is no different to a white paper, not only so, the independant economists, the Romney economists team cited have argued their work had been misrepresented. While Ryan want US to abandon paper currency and go back to gold coins.

Nothing in the US politics resemble anything sane, more like a playground for the ideology zealots.

8. wilful says:

I generally think Hartcher is more worth reading than most in the Oz. But that was a rubbish article – why is it “troubling” that a bunch of people with no influence on the outcome and who frankly don’t really care that much might not be aligned with US polling?

More broadly, the idea that the truly crazy ideas of the US far-right are now politically acceptable and mainstream is just barking mad.

9. Uncle Milton says:

Hohn’s reasoning assumes that people should take opinion polls at face value and as reliable sources of information which they use to predict who will win. But many people confuse who they want to win with who they think they will win, or they don’t believe the opinion polls because they it couldn’t possibly be the case that their non-preferred candidate/party is mor likely to win.Some people don’t even believe the result after the event. (“How could X have won? I don’t know anybody who voted for X”)

Back to Obama/Romney, I know the predictive abilities of markets are deeply unfashionable at present, but still, Centrebet has Obama at \$1.40 and Romney at \$2.90, implying winning probabilities of 77.4% and 32.6%. John, if you think that Obama really should be 3 to 1 on (that is, decimal odds of \$1.33), Centrebet is paying over the odds, and you should put some of your hard-earned on Obama.

10. Robert Merkel says:

I think you’re being too kind to Hartcher here.

This isn’t some subtle statistical quirk; this is a very basic piece of probabilistic reasoning in a problem domain in which Hartcher is supposed to be an expert.

11. Doug says:

Nat Silver at the blog site Fivethirtyeight has a sophisticated analysis of all the polling data – he currently has Obama at about 67.8% chance of winning http://fivethirtyeight.blogs.nytimes.com/

12. Tony says:

Maybe once Romney is elected, they’ll pass that nude negro law.

13. Freelander says:

One thing that has annoyed in realtion to us reportage recently, and that is, and includes Pulitzer winners, reporting on the plain packaging legislation in Oz as though it was not simply introducing that but was also introducing “gruesome” pictures (which were actually a 2004 measure). This mistake Ive seen in as decent a paper as the. NYtimes. But then competence has increasingly become old fashioned over the last couple of decades.

Hartcher’s error, quite forgivable.

14. Freelander says:

Should have included the qualification reportage in the US.

15. derrida derider says:

On the reasoning about probability, John’s post is of course correct. Of course he’s also correct to note that journalists, universally bereft of any interest in statistical reasoning, regularly commit much worse sins than Hartcher’s. I do like Jack’s comment @2 though.

On the guess about Obama’s actual odds I reckon the bookies have it roughly right, and people here are being a tad wishful.

Romney would hardly be the first President elected by ignoring the laws of arithmetic and telling the punters they can have it all (said punters being as innumerate as the journalists, and much more prone to hearing only what they want to hear). The US economy is very sick still. Plus with voluntary voting the enthusiasm of your supporters really matters and no-one’s really enthusiastic about Obama these days; “best of a bad bunch” is not a line calculated to make you want to wait in a long queue at the polling booth.

16. derrida derider says:

“John, if you think that … Centrebet is paying over the odds, you should put some of your hard-earned on Obama.” – Uncle Milton @9

But this assumes that money is the sole metric of utility. Me, I’d put my money on Romney as a laying-off strategy (assumng risk aversion). If the bugger does get up then my unhappiness at the result will be partly compensated by my cash winnings. It’s the same reason that, barring serious mispricing (in my view) by the bookies***, I’ll be betting on Abbott to be our next PM.

Of course given this strategy the amount I lay off will be proportionate to the utility effects of the result. So its a small bet on the US election and a bigger one on the Australian one.

***No, since you ask, I’ve never been much impressed by the EMH.

17. NickR says:

Another flap is the Isaac and the RNC. People appear to be distracted by the cyclone instead of paying attention to the convention, which will probably limit the boost that typically comes with such events. It could also remind people of Katrina, Bush, climate change etc and if there is damage the cleanup may make aggressive attitudes to govt look a little heartless.

Nevertheless I wouldn’t take anything for granted come November.

18. Ikonoclast says:

It is also possible to steal elections in the US in ways which are not possible in most real democracies. George W. and his party machine very clearly stole both of his election wins.

The U.S. is not a democracy. The “we are a democracy” claims made by the US are made only for foreign consumption. American intellectuals of the liberal left and the conservative right all concur that America is not a democracy. The liberal left deplore it. The Right boast about it; “American is not a democracy. It is a Republic.”

Barely half of Americans vote. The President is not elected by popular vote but by the electoral colleges system. The Senate is unrepresentative (like the Australian Senate) with 2 Senators per state, ignoring population size. The House of Reps features first past the post voting which badly distorts the popular will. Oligarchic and corporate monies buy the Presidency and the majority of Congress members (in one way or anoter). The two major parties are both very right wing on any broad global-historical standard of politics.

Why persist with this delusion that America is democracy? It was not even designed to be such by the Founding Fathers. They expressly designed major limitations to the popular will in the constitution and expressly designed special rights to enable the wealthy and priveleged to prevail.

19. NickR says:

@Ikonoclast
Nate Silver tends to think that voter id laws etc are not a big issue – perhaps enough to sway a very close election but not in most circumstances. And the electoral college appears to be favoring Obama this round.

Agree with you on the Senate though.

20. Uncle Milton says:

@derrida derider
Yes, but given that at 1.40 Obama is over the odds of 1.33 (and by implication, Romney is under) you can back Obama at 1.40, then when the odds move to the objective truth, back Romney. This way you can lock in a profit that you can console yourself with if Romney wins or can be a bonus if Obama wins.

21. Katz says:

In US presidential elections, aggregated polls mean little. If one candidate wins both Florida and Ohio, he will win the presidency. The polls in the above-mentioned states are the important electoral data in 2012.

From the point of view of electoral tactics, it is clear that Romney/Ryan aim to motivate the GOP base by eschewing determined courting of centrists in the hope that luke-warm Obama supporters will be too unmotivated to get out to vote.

22. John Quiggin says:

I’m still unable to login, while the site move is completed.

My reason for not betting my beliefs isn’t risk aversion, as in DD, but transactions costs. I did some betting on Intrade as a test of concept, but it turns out to be a bit of a pain to collect the winnings. I keep meaning to close the account, but haven’t got around to it yet.

23. John Quiggin says:

Hmm, looks as if I’m logged in now.

24. Uncle Milton says:

Open a Betfair account. Very liquid. And you can back Obama at \$1.61, which is way overs.

25. Mel says:

Uncle Milton: “John, if you think that Obama really should be 3 to 1 on (that is, decimal odds of \$1.33), Centrebet is paying over the odds, and you should put some of your hard-earned on Obama.”

What are the odds that Donald Trump will finally produce compelling evidence that Obama is a Kenyan?

26. Freelander says:

Haven’t heard about that law. Sounds intriguing.

27. Jim Rose says:

Romney heads to the Republican Convention with his “safe” and “lean” Electoral College votes unchanged at 108 and 74, respectively

Obama’s count shifted to 177 “safe” and 70 “lean.”

there are seven states (109 EC votes) that are considered a toss up

28. Freelander says:

Who really cares who wins anyway? Won’t make a great deal of difference. Either they retain their current “murderer in chief” or swap him for a god helps those who help themselves by choosing the right antecedents moron, sorry mormon (same different, as that story is even more incredible than the JC saga). Why is it now requisite that as well plus 34, born inside the empire, religious hypocrisy and moral depravity are also necessary to have a chance of your donors buying you the presidency? No wonder standards have dropped. Amongst the meager few who can tick all those boxes a quality candidate might be a difficult find. The submariner was a decent enough chap, but a long time since the office has had the calibre of a Nixon, an LBJ or an Eisenhower.

29. Katz says:

Obama is distinguishable from Romney on the bases of gender politics and dog whistling to the Fundos.

A clothes peg would mostly occlude the stench of Obama but not the stench of Romney.

30. Neil says:

Not defending Obama, but this kind of comment seems to me to be another zombie. No matter how many times it is refuted, it goes on being made. We saw it with the election of Howard in Australia (“there’s no real difference between coalition and labor”) and with the election of Cameron in the UK. Each time the remark is trotted out by people on the left who are (justifiably) disappointed with the incumbent. Each time we have seen that the conservatives in power are a much nastier lot than they pretend, and it is the poor who bear the brunt.

31. Freelander says:

Yes. You’re right.

When what you are putting up with currently (Obama, for example) is undeniably truly awful, it’s only wishful thinking to think worse is precluded.

But how to punish the current incumbents without embracing worse? That’s the conundrum!

32. Katz says:

It takes a special kind of incomprehension to interpret my comment above as meaning that Obama is not preferable to Romney.

33. Neil says:

@Katz
Indeed, it would take a special kind of incomprehensions to interpret your comment that way. I did not so interpret your comment (though I agree I could have made that clearer). I interpreted your comment as saying that there is little real difference (dog whistling is not a substantive difference; gender politics is, but you imply that’s the only differences). It is the small difference claim that I dispute.

34. Katz says:

Neil, I’d be happy to discuss the magnitude of other points of difference.

35. Freelander says:

And next topic of discussion.

“Do big fleas really have small fleas on their backs to bite them?”

36. Jim Rose says:

Berry and Bickers in “Forecasting the 2012 Presidential Election With State-Level Economic Indicators” predict the popular vote for each of 50 states and the District of Columbia.

They emphasize changes in each state in real per capita income, the national unemployment rate, the state-level unemployment rate, the vote received by that party in the previous election, a variable for whether the incumbent is a Democrat or a Republican, and for how many terms in a row the presidency would be held by a particular party.

they predict Romney will win the 2012 election with more than 52% of the popular vote. Obama will lose almost all of the swing states, including North Carolina, Virginia, New Hampshire, Colorado, Wisconsin, Minnesota, Pennsylvania, Ohio, and Florida!!

all in all, to close to call, which is a great achievement for Obama by being still in the race despite the great recession, which is the fault of his policies.

37. paul walter says:

Hartcher?
Wrote a crocka slagging Assange the the other day also and am losing time for him quite rapidly .
As to their election, basically what several others have said. At least Obama doesn’t valorise the worst of his stuff, much the same as Gillard here. The surprise is that people like Willard in the US and Abbott here are even “alive” politically, given the insanity and likely consequences of their policies; the austerity example of the lunatic Cameron government in Little Britain ought to warn even morons what’s in store.

38. Freelander says:

Always angered by the almost universal cowardice displayed in relation to Assange especially by those who profess themselves on the human rights, compassion end of the spectrum. Most display their bravery through silence, others by attempting to figuratively ‘pack rape’ him.

39. Patrickb says:

@Katz
On foreign policy you’re probably right. On US domestic policy you’re completely wrong. If you don’t buy that than you’ll need more than a terse dismissal to justify your opinion. Or not as the case may be.

40. Freelander says:
41. Freelander says:

Oops!

Increasing find myself have to agree with Socrates’ and Plato’s views on democracy. This, all to successful pandering to the lowest common denominator amongst the great unwashed by the demogogue of the day makes one hope that there might be a better way.

Abbott, and his f#t friend, Hockey look lodgeward bound using that simple but successful strategy of the big lie(s) told and repeated often. Rather than going off message and ever being interviewed, they drone on turning each or their photo Opps into another monologue, yet another party political.

42. paul walter says:

Patrickb, are you saying the republican domestic policies are anything but vicious, medieval and irrational?
If so, you are disinherited forthwith.
Truth is, Obama and his democrats, for all their flaws, at least represent survival.
Another four years of Obama offers a thread of hope. That thread lies in the possibility that the corrupt, reactionary octogenarians on the dysfunctional US Supreme Court may finally fall off their perches, offering the hope of a return to sanity in the interpreting of the law less encumbered by corruption and subjective fantasist Randian ideology.
Apart from that, the deterioration into Talibanisation is likely to be slower while the president remains a rational secularist rather than an idiot sockpuppet for the Tele-mullahs and the obscurantist tendencies they represent.

43. Nick says:

“On US domestic policy you’re completely wrong. If you don’t buy that than you’ll need more than a terse dismissal to justify your opinion.”

I’m also staggered by this comment, Patrickb and not just for its blatant pot/kettle-tude. What tea party free world have people been living in since 2009?

Talibanisation is the right word, Paul.

We just finished the first season of the Newsroom last night…ok, it can be more than a bit cheesy (kind of like Star Trek/The West Wing crossed with The Office), but it’s worth a look. Here are a few screenshots and a clip:

http://nick2012.minus.com/

That Bauer quote is sickening. From Wiki: In 2008, Bauer supported legislation for a new state license plate containing the words “I Believe” and a “cross superimposed on a stained glass window.”

44. NickR says:

@Jim Rose
I couldn’t find the Berry Bickers paper online but from the parts of the discussion I read it seems that they combine single equation methods for each state. This is a sensible and widely used method for building models of the Electoral College (I have a Probit EC model that uses this exact approach) however it ignores contemporaneous correlation between error terms across states. This is a big problem a long way out from the election, as polls and economic data will fluctuate together. Ignoring this phenomenon means ignoring a great deal of very relevant uncertainty.

Further it seems (although I may be wrong of course) that they are reported a deterministic estimate rather than a stochastic one, which would also be a big problem.
Let me illustrate with an example. Suppose I take a dice and draw an extra dot on the 5 to make it a 6, so that the range of options is 1, 2, 3, 4, 6, 6. If I throw this dice 10 times, the most likely individual sequence is 10 sixes. However this would also be an incredibly unlikely event – if I did the experiment I would almost certainly get some jumble of all the other numbers, and may not even get a particularly high proportion of sixes. So if I am to make a prediction of scores and order doesn’t matter, the correct thing to do is use the probability distribution instead of the most likely single draw.

From what I can tell BB have reported the “most likely” specific result in terms of electoral maps (as in expecting 10 sixes), when instead they should have reported the probability distribution, then figured out what is the probability of a particular candidate getting 270 or more votes. Right now they are anticipating a large number of very close races, and almost all of them being won by the Republicans by a tiny margin. Instead it is much more likely that some will go one way and some another, which of course would result in a much better result for Obama.

These issues may explain why the BB model is so out of step with 538, Intrade and bookmakers. All the other information I have seen indicates he is a moderate favorite. Of course it is possible that BB are right and everyone else is wrong, but it seems unlikely to me.

BTW

“all in all, to close to call, which is a great achievement for Obama by being still in the race despite the great recession, which is the fault of his policies.”

Not sure how sincere you are here Jim – I hope you are kidding, and suspect you may not be worth arguing with if not.

45. Jim Rose says:

thanks NickR, most presidential predictions end up riding on the economy, which is bad, which is why Obama is neck and neck with Mitt.

Kehoe and Prescott concludes that bad policies are responsible for causing depressions. while different sorts of shocks can lead to ordinary business cycle downturns, overreactions by the government can prolong and deepen the downturn, turning it into a depression.

Obama is strong on raising taxes, regulation and business costs.

46. Freelander says:

And the Republicans, just as they have since 1980, are aceing it in stupidity.

How Clint could get up there and make their day?

They now must be feeling lucky, punks.

The increasingly bizarre bunch of odd-balls now trotted out every four years difficult not to believe they’re not living in some comic book fiction.

47. Freelander says:

Quite right. In 2000 the only reason the supremes got close to a chance of electing Bush was because of the rorting in Florida, eliminating Democrat voters by accidentally purging them from rolls and by putting defective voting machines in areas where the voting was heavily Democratic so many more of their votes would be spoilt. Despite this “preplanning” if they hadn’t combined their “good work” with a bogus initial count the supremes wouldn’t have been provided the chance to choose W. In 2004 with electronic voting in many states some make a credible case that W lost again

48. Freelander says:

That is, lost in terms of minority of voters voting for him in too many states except that pattern not reflected in the pattern recorded by the significantly suspect and non verifiable electronic voting machines.

Yes progress is wonderful, and productivity is for ever increasing. What took so much planning and effort in 2000 could be achieved with the press of a button by 2004.

49. Freelander says:

Que? Moderation?

50. Freelander says:

Moderation, of inquiry about moderation. Teething troubles presumably or anti spam device…

51. Chris Lloyd says:

“If you want to use statistics as a guide to prediction, you need a Bayesian approach.” Not true. Frequentist can produce forecasts with prediction errors. But I take your point that a hypothesis test of significance is not all that useful here.