One of the justifications I make for the time I spend blogging is that it gives me a chance to try out arguments I use in my work. With that in mind, I’d very much appreciate comments on this short summary of the role of ideas and interests in explaining policy outcomes.
Economists seeking to explain policy outcomes use three main theories, sometimes explicitly and sometimes implicitly. These may be referred to as the public interest theory, the private interest theory and the ideological theory.
The public interest theory is rarely stated explicitly, but is implicit in much of the normative analysis of policy options. The central hypothesis of the public interest theory is that governments adopt policies that will maximise social welfare, subject to random error cause, for example, by ignorance about the issues. The utilitarian case for democracy is based on the argument that a government responsible to a democratic electorate will have an incentive to weight the interests of all voters equally, and will therefore promote the public good.
The private interest theory is commonly presented in conscious contrast with the public interest theory. The central hypothesis is that political outcomes are determined by interactions between interest groups, and that the relative weight of interest groups will determined by factors such as the effectiveness of their organisation, rather than by their significance in relation to some well-specified social welfare function. Marxism (at least in its simple ‘vulgar’ form) and public choice theory share this central hypothesis, along with some versions of liberal pluralism.
The ideological theory is most commonly associated with Keynes’ dictum that soon or late, it is ideas, not vested interests, which are dangerous for good or evil…” (General Theory, page 570)” On this view, it is changes in beliefs about the merits of policies such as privatisation, as opposed to changes in the actual costs and benefits or in the relative weight of competing interest groups, that do most to explain changes in policy outcomes.
Unless ideas are regarded as evolving independently of the real word, the ideological theory tends to reduce, in the long run, to some mixture of public interest and private interest theory. If ideas about the desirability of policy are adjusted in response to evidence (cf Keynes – when the facts change, I change my mind. What do you do?) then the public interest theory will be valid in the long run. If changes in ideas are determined by the rise and fall of dominant interest groups (as in many Marxist models, and in Schumpeter) private interest theories will be valid in the long run, even though people may believe themselves to be acting in the public interest.
John Quiggin 
A couple of comments.
1. The difference between the first and the second ‘theory’ of what influences policy is often not a difference in theory but a difference in ways of speaking. One can address oneself to what SHOULD be the case and one can address oneself to predicting what it will turn out to be. Because a lot of economic analysis sets out what should be the case, does not mean it is written by people naïve enough to think that pointing out what should be the case and getting it done is a simple process. Likewise, those who point to the role of interests in setting policies don’t REALLY believe that’s all there is to it – otherwise why would they bother participating in the debate at all? As things will turn out the way they will whatever they say (because of the determination of interests). Indeed if they’re audience were strictly of this school they’d all assume that what the author had written was simply a reflection of their own interests – so why read on?
2. The idea of these being two ‘theories’ of action is a bit much for me. I know this is how they’re often talked about in economics, but as a description of reality doesn’t it seem crude? Obviously both ‘theories’ are caricatures and capture different aspects of reality at any time. As Harry Johnson once wrote “The ‘adversary procedure’ of testing one hypothesis against another is a useful scientific procedure up to a point; but, when both hypotheses perform well and seem to be fairly evenly matched, it is not necessarily the best scientific procedure to send the challenger back to training camp with good advice on how to prepare for the next month.” After going on some wild goose chases for ‘universal laws of history’ in the 1950s and 60s historians are now much better than economists at working out where to go looking for laws, and where one cannot do better than more discursive analysis. Sometimes interests are important – though ideas and utility will influence the way they are played out. And sometimes they are less important. When the Menzies Govt was crafting policy for the car industry, it pandered to special interests and didn’t care that the policies it was adopting were messing up the industry’s structure. When Churchill was trying to win WWII, he was mainly trying to win WWII, though no doubt he did some favours along the way. We could go on trying to develop a ‘theory’ of when one mode will dominate, but it will change with changing pressures, ideas etc. We are in history.
Hope this makes some sense.
John,
I probably come out in the same place as Nick Gruen, but by another route.
There are ideas that really matter in public policy (or history) for which ‘evidence’ is irrelevant. Liberty, equity, home… etc. These are the ideas that win elections, bring out donations and send people off to war or other risky adventures. They are powerfully normative.
And there are ideas that are frequently part of the elite discourse on public policy that are fashionable or useful as gestures rather than truly motivating. The danger of smoking, the rightness of feminism and ‘sustainability’ are such ideas.
These ‘intellectual’ ideas, we know from experience, are frequently immune to evidence in the short term: contrary evidence may be invisible or derided. But they define the political battleground where private interests wrangle over short-term distributive decisions: they create clients, help to define alliances.
I’ll try to post something a bit longer and less cryptic on my site.
Best wishes
Peter, maybe I’m misreading you, but it appears you are suggesting that there is not much evidence regarding the dangers of smoking. Would you care to clarify?
John,
Of course the evidence for the dangers of smoking is overwhelming: or at least it SHOULD be.
But many people in their early teens are still taking it up. Why? The best explanation may be the most obvious: the (intellectually compelling) idea that smoking is risky is trumped by an idea like ‘liberty’ that has greater motivating power.
I suggest in post that I hope will appear on my site in an hour or so that ideas that have a very strong impact on voters (and on history… to be a little grander) are not intellectually compelling ideas but ideas like ‘liberty’ that have a more atavistic appeal.
The ideas that matter in the elite-dominated debates that form public policy in the short-run (most of them debates about the distribution of resources) are, at least superficially, ‘intellectual’ ideas where — as you say — evidence should matter.
But my skeptical view is that evidence doesn’t matter in most of these cases. Many of these ideas are not terribly sensitive to evidence at all.
I’m left supporting the notion that private interest plays a much bigger role in government than do ideas.
But you’ve posed a hard question…
I’d just add my “hear, hear” to Nick’s – I think he’s hit the nail on the head. But there is also definitely the influence of conventional wisdoms (a characteristically lovely coinage by Galbraith) that don’t seem much affected in the short run by either empiric evidence or narrow vested interests.
Not that conventional wisdoms aren’t sometimes actually wise – my point is just that their wisdom or otherwise doesn’t seem to have much to do either way with their becoming a convention.
John; I largely agree with Nick Gruen, but note that there is a body of theory out there that is distinctive and can be regarded as coherent and consistent – albeit a lot of people find it unconvincing – that is, rational actor political theory. IMO what most characterises it is its extreme individualism (Buchanan has many times commented on its atomistic and individual nature). The interesting thing is that when it comes to policy prescriptions public choice proponents turn out to be as ideological as anyone else – and it’s fascinating to observe the gulf. Some of this material however is useful in explaining particular policy outcomes – for example, the observation, provable, that the ordering of choices can make an enormous difference to the final policy decision (seen clearly in the recent republic referendum – if it had been a choice of a) do you want a republic and b) if yes, what sort – rather than as happened the other way around – then the result would have been the opposite).
The political science literature has a lot of material on interests, often termed “stakeholder theory” (although it is dignifying it too much to call it theory) that has addressed the question of policy outcomes. The other strand of political science that bears on this question is the literature on agenda setting – that is, defining the range of possible policy outcomes in advance.
An area where there is overlap between the politics and economics thinking is path dependency. If you want to explain policy outcomes, it might be worth exploring the fact that past policy choices constrain them – and can mean only a few outcomes (out of what seems potentially a large a number) are actually possible. That said, of course you then go back to saying “ok, we accept the choices are constrained, but what is that explains the choice”
Nicholas Gruen seems misinterpret John’s ‘public interest’ model. As I understand him, John is saying that some people believe, as a purely positive matter, that, if certain fundatmental conditions for democratic governance are in place, then one way or another, at the end of the day, something approximating the public good will prevail. And I think it’s reasonable, as an initial charactersiation, to contrast this with the belief that the actions of government, however democratically constituted, will inivetably prove detrtimental to either particular classes (Marxism) or the public good (public choice theory).
This is not to say that one might not hold intermediate positions, nor that the more cynically inclined cannot continue advocating policies to promote the public good.
My own objection is more to the third classification, but this is more of a terminological quibble. The term ideology has a strong association with Marxism itself, and the notion of false consciousness. Whether or not one fundamentally adheres to the marxian version of the private interset model, one has to concede that ideology plays an essential role in it as a means to domination by one interest group, especially in capitalism.
Keynes on the other hand is contrasting ideas to interests, and asserting the potential of ideas to exert an independent and arbitrary influence on events. I think we need a word or phrase for this other than ideological.
John,
I don’t think ideological theory always reduces to either the public interest or private interest theory.
Ideology is not just about factual beliefs. It’s also about values. According to some ideological positions maximizing ‘social welfare’ is not always the morally right thing to do. Welfare economics only appeals to people of a particular ideological persuasion.
When a political leader governs according to values other than social welfare maximization I’m not convinced that you can treat this as the pursuit of a private interest.
Public and private interest
My own experience tells me that the hardest-fought decisions made at a ‘political level’ are distributive decisions in which ideas of a sort play a role. But they are ideas that are … not empty but not rigorous or particularly sensitive to evidence.