I have a piece in The Conversation arguing against the common practice of publishing projections, based on holding constant parameters that are unlikely to remain so in practice. I suggest modellers need to bite the bullet, make predictions and stand by them.
A slight clarification, arising from discussion. To the extent that we are concerned with policy, it’s fine to make conditional predictions about the consequences of alternative policy packages.
8 thoughts on “Predictions and projections”
I recall a bet I made with you J.Q. in 2010 approx. It was something like me saying that in 2020 the world GDP (which condition you amended to World Income and I agreed) would be less in 2020 than 2010 after allowing for inflation. Clearly, I have lost that bet or will lose it about the end of this year. It was for $A100 indexed for inflation. Sometime early in the new year or at the start of the academic year 2020 I am going to have to pay up… unless a large meteor hits earth between now and then. We will have to settle on the precise amount owed, due date and payment method. You have my email of course so please email me as and when appropriate.
Of course, I was foolish to bet on something measured in nominal units, namely dollars. Nevertheless, there is no doubt that real production is also greater in 2020 so the bet would lost anyway. None of this means that the basic thesis (the current economic system is unsustainable and will collapse) is necessarily wrong. I still contend that this will occur eventually. Nonetheless bets are bets and agreed conditions are agreed conditions. And predictions are always risky, especially about the future!
Most unusual. A John Quiggin article with which I:
a) have professional experience in the subject under discussion; and
b) disagree with.
The point about forecasts is that they are almost always wrong, and (crucially) forecasts based on formal models are often almost as likely to be as wrong as anybody else’s. That’s especially so if they’re about the more distant future.
Knightian uncertainty rules (aka shit always happens). I once had dinner with an American actuary who had just been commisioned to do a review of the original 1938 US Social Security long range forecasts – he commented sadly “they’d have been dead on if not for World War 2”!
Projections, though, can be correct – not in the sense that they predict what will happen, but that they can predict the DIFFERENCE a particular parameter or parameters make. And for policy purposes that is what you want to know, not what will actually happen.You want to know what difference alternative courses of action you can control will make. The main reason models do better at predicting differences rather than levels is, of course, that fixed errors wash out when you difference a time series.
The persistent wrongness of the IEA model is odd. Why doesn’t it learn from experience? This can be done either algorithmically or by commonsense manual adjustment. It is strange to stick with parameters that are provably wrong. And have they heard of learning rates?
” Projections, though, can be correct – not in the sense that they predict what will happen, but that they can predict the DIFFERENCE a particular parameter or parameters make. ”
We are in furious agreement “Projections are useful in the development of models. All models are based on past experience and have to assume that in some respects the future will be like the past. By examining the projections generated under particular assumptions about which variables and parameters will remain constant, it is possible to understand how models works and make modelling choices.’
A technical complication. The IPCC for instance uses the projections from multiple models to make its own more reliable predictions. Generally speaking, a simplified version of this is what sensible people do all the time. If you know your model is going to be Delphied in this way, you can spare yourself the professional risk of making the prediction yourself.
Clearly, the IEA are ideologues not empiricists. If they were empiricists they might learn something. The IPCC demonstrates that consensus is conservative. Although empiricists they are tethered to a consensus model which owes something both to its own most conservative scientists and to the conservatism of the IPCC’s political paymasters. The consensus model of the IPCC has severely downplayed the likelihood and dangers of rapid climate change. Every dangerous trend predicted is occurring more rapidly than predicted. Many dangerous developments were not predicted at all even by the IPCC. Rather than climate change theory being the conspiracy, the conspiracy is climate change denial and even climate change minimization. I’m using “minimization” here to mean minimizing the rapidity with which climate change is happening and the imminence of the coming catastrophe.
Iko: you are probably right about the IEA, and certainly about the conservative bias in the IPCC method. But where does this come from? Is there any evidence that it comes from the way they average model results? More likely the bias is in the model cluster, in which both more conservative and more adventurous scientists are weighted equally. All the models necessarily ignore black swans and unidentified feedbacks. Finally, the publication delay means everything is two years out of date – noticeable for instance in the assumptions about renewable costs.
It looks as if the effect of political representation is on the tone of the language used to present results, clearly a bias to watering down. The recent 1.5 degree report was SFIK something of a departure in the robustness of its language.
“Clearly, the IEA are ideologues not empiricists.”
Maybe that is not so clear cut… I’ve lately been reading a bit of their reportage over time. The IEA have responded to the changing circumstances as presented by wider changes in knowledge and understanding with changes in their approach and forecasting. They seem open minded and seem ready to point out where they have adjusted with changed circumstances and why. Unlike ideologues the IEA has changed as the facts have changed. They can’t get ahead of the facts as they tread the difficult path of engaging numerous varied decision makers they need to best inform, advise, and influence. Surely that’s hard going? That’s the system. The system needed changing. Too late?