Misleading reporting of surveys

A lot of people have pointed to a survey reported in the Telegraph and accepted the misleading interpretation of the results given by the Tele’s Malcolm Farr. First, the results. Of those surveyed, 47 per cent thought Howard had lied on children overboard, 31 per cent did not believe he had lied, and 22 per cent were uncommitted. On a second question, 60 per cent said it would not influence the way they voted, 31 per cent said it would and 9 per cent were uncommitted. The heading is “If PM lied, few care” and the opening para is “A large majority of voters have said the issue won’t influence their polling booth choice – even if they think the Prime Minister didn’t tell the truth.” This is wrong in almost every way a survey report can be wrong.

The first problem here is the standard one of opposite majorities. A near-majority of voters think the PM lied, and a majority say it won’t influence their vote, but that doesn’t mean “most people think the PM lied, but won’t let it influence their vote”. Suppose, as a first approximation, that all those who say it will influence their vote think the PM lied[1]. Then the proportion left in the “lied, but don’t care” category is only 16 per cent. I expect the true number is a little bit higher, but, almost certainly, less than half of those who said the PM lied then said it wouldn’t affect their vote. This is the opposite of what Farr asserted.

The second problem is that, if the questions are taken literally, I and others like me would answer “Yes” in both cases. I think Howard lied, but I wasn’t going to vote for him anyway and it doesn’t influence my ranking of the other choices on offer. I expect that is true of most people who think Howard lied, but presumably a lot of people took the view that this strengthened a resolution they had already reached and therefore answered Yes.

The third problem is, perhaps the most significant. Suppose that we interpret “affect my vote” as “change from Liberal to Labor (directly or on preferences)” and suppose also that most people had forgotten about the issue until the latest revelations. Then even 5 per cent answering “Yes” would be sufficient to guarantee Howard’s defeat. Things are a bit more complicated than that, but this point illustrates the problem with treating a question like “will this influence your vote?” as one for which a majority “Yes” or “No” is what matters.

Meanwhile, Don Arthur reports on another dubious survey undertaken by Peter Saunders of the CIS>

fn1. Presumably some of those who say he was telling the truth also said it would influence their vote, meaning that they were more likely to support Howard.

3 thoughts on “Misleading reporting of surveys

  1. Labor by 11, with 3 to independents (none to Greens in the House). Greens up to 6 senate seats. Democrats one only. One Nation gone altogether. Malcolm Turnbull fails to get elected. Peter Garret home and hosed.

    Of course this all assumes a perfectly spherical voter to start with….

  2. In the case of Saunders and the CIS, I tend to agree that he knew what was dodgy about the survey methodology and results, but didn’t let this get in the way of reporting results which seemed to confirm his prejudices with a spin which strengthened this effect.

    In the case of Farr (and most journalists who write on political, social and cultural phenomena), I think’s it’s most likely that he simply doesn’t have the skills in quantitative data analysis to either understand the meaning of the survey results, or to communicate them effectively to his readers.

Comments are closed.