Hoist on their own petard
Having been involved in the debate over schools policy for quite a few years, I’m enjoying a bit of schadenfreude following the publication of a couple of regression analyses showing that students at charter schools (publicly funded US schools operating independently from the main public school system) score worse on standard tests than students at ordinary public schools. I don’t have a particularly strong view on the desirability or otherwise of charter schools, but I have long been critical of one of the most prominent rationales for charter schools and other programs of school reform.
This is the claim that “regression analyses show that students in small classes do no better than those in large classes”. If you believe this claim, you should believe the same claim with “charter schools” replacing “small classes” since both are supported by the same kind of evidence.
The class size claim goes back to the pioneering attempts of James Coleman to estimate “educational production functions”, which came up with the conclusion that almost everything is insignificant. A lot of other studies had similarly negative findings, and the natural interpretation, that this was because there was no effect, was promoted vigorously by a number or writers, notably including Ric Hanushek, who collected a bunch of studies and summarised the results as showing no effect.
An alternative interpretation is that the dependent variable in these regression analyses is so noisy as to make the results highly suspect. We want to know how much the kids learnt in a given period in school, but what we can typically measure is the difference between two test scores, before and after the relevant schooling. A test score is a pretty noisy measure of how much someone knows; the difference between two test scores is so noisy as to be close to useless.
Of course, problems of this kind arise in lots of contexts, and there’s a big literature on meta-analysis as a way of extracting meaningful results from disparate studies. Hanushek’s “counting studies” approach is a very crude kind of meta-analysis. A more sophisticated version, undertaken by Hedges, Laine and Greenwald shows that there is a significant relationship.
There’s also revealed preference to consider. If small class sizes are educationally valueless and are adopted as a response to, say, teacher union pressure, we’d expect to see a very different allocation of resources in private schools. In fact, though, wealthy private schools typically go for small classes – the share of budgets going to teaching staff doesn’t vary much between public and private schools.
Anyway, I’ve been hammering these points for years, without much of an impact. But now, I expect, a lot of people are suddenly going to discover that comparisons of school performance are tricky, that regression analysis is not infallible, and that there’s more to educational outcomes than test scores. I hope they will be consistent enough to revise their prior beliefs about class sizes.
fn1. As you might expect, there’s another study that gets the opposite result.
fn2. As always, I use the term ‘reform’ to mean ‘structural change’, without any connotation of approval or disapproval.
fn3. Sometimes, there’s only one test score, which makes everything even worse.
fn4. Hedges is one of the big names in meta-analysis