Signal and noise
Not surprisingly, given the mini-crisis surrounding Howard’s leadership, the Newspoll released yesterday, with the two party preferred position for Labor shifting from 59-41 to 55-45 was big news. But was there really any news, or just the usual random fluctuation. The ABC News on Monday night made it sound huge, describing it as the government cutting Labor’s lead by 8 per cent. On the other hand, if you started with the view that the true position was 57-43, neither this poll nor the last one would lead you to change this view. Two percentage points is well inside the usual allowance for sampling error in polls with a typical sample size of a bit over 1000. So, you might say, it’s all just a beatup.
Comparing the two polls gives a somewhat different answer, which my son Daniel kindly computed for me. Given two polls with 1000 respondents, and the stated results, you can reject, at the 5 per cent confidence level but not at the 1 per cent level, the null hypothesis that they are from the same population. That’s because, given no underlying change in the population, the chance of two variations in opposite directions is smaller than the chance of each variation considered separately.
But that doesn’t end the problems. If you run polls every fortnight, you’re bound, sooner or later to get two samples in a row that deviate in opposite directions, producing a big apparent swing.
FWIW, looking at the Newspoll results since July, I see no reason to think there’s been any real movement in either direction. Aggregating all the polls, and reducing the sampling error accordingly, I’d put the Newspoll 2PP vote somewhere in the range 56-57. The other polls seem to be much the same.
The real problem, though, is non-sampling error. If the question being asked doesn’t match up to the way people are actually going to vote, all of these statistical considerations count for nothing.