Hoist on their own petard

Having been involved in the debate over schools policy for quite a few years, I’m enjoying a bit of schadenfreude following the publication of a couple of regression analyses showing that students at charter schools (publicly funded US schools operating independently from the main public school system) score worse on standard tests than students at ordinary public schools[1]. I don’t have a particularly strong view on the desirability or otherwise of charter schools, but I have long been critical of one of the most prominent rationales for charter schools and other programs of school reform[2].

This is the claim that “regression analyses show that students in small classes do no better than those in large classes”. If you believe this claim, you should believe the same claim with “charter schools” replacing “small classes” since both are supported by the same kind of evidence.

The class size claim goes back to the pioneering attempts of James Coleman to estimate “educational production functions”, which came up with the conclusion that almost everything is insignificant. A lot of other studies had similarly negative findings, and the natural interpretation, that this was because there was no effect, was promoted vigorously by a number or writers, notably including Ric Hanushek, who collected a bunch of studies and summarised the results as showing no effect.

An alternative interpretation is that the dependent variable in these regression analyses is so noisy as to make the results highly suspect. We want to know how much the kids learnt in a given period in school, but what we can typically measure is the difference between two test scores, before and after the relevant schooling[3]. A test score is a pretty noisy measure of how much someone knows; the difference between two test scores is so noisy as to be close to useless.

Of course, problems of this kind arise in lots of contexts, and there’s a big literature on meta-analysis as a way of extracting meaningful results from disparate studies. Hanushek’s “counting studies” approach is a very crude kind of meta-analysis. A more sophisticated version, undertaken by Hedges, Laine and Greenwald[4] shows that there is a significant relationship.

There’s also revealed preference to consider. If small class sizes are educationally valueless and are adopted as a response to, say, teacher union pressure, we’d expect to see a very different allocation of resources in private schools. In fact, though, wealthy private schools typically go for small classes – the share of budgets going to teaching staff doesn’t vary much between public and private schools.

Anyway, I’ve been hammering these points for years, without much of an impact. But now, I expect, a lot of people are suddenly going to discover that comparisons of school performance are tricky, that regression analysis is not infallible, and that there’s more to educational outcomes than test scores. I hope they will be consistent enough to revise their prior beliefs about class sizes.

fn1. As you might expect, there’s another study that gets the opposite result.

fn2. As always, I use the term ‘reform’ to mean ‘structural change’, without any connotation of approval or disapproval.

fn3. Sometimes, there’s only one test score, which makes everything even worse.

fn4. Hedges is one of the big names in meta-analysis

10 thoughts on “Hoist on their own petard

  1. Maybe we should simply harpoon the word reform and use “structural change”.

    I’m going to reform the waterfront! – good, when do we start?

    I’m going to structurally change the waterfront!- um, why?

  2. Charter schools and vouchers are basically a “solution” to the “problem” of teachers’ unions. Better educational outcomes are beside the point.

    At its core, teaching is a form of performance art, which is why it has been so hard to institutionalize success. But even a great teacher will have a hard time stuck in classroom full of students who have bad attitudes about education. And even good students with good teachers won’t be ready for the modern world if they are taught an antiquated curriculum. I think a lot of incremental adjustments will produce better results than the various clean sheet approaches Americans seem compelled to seek.

  3. John, Caroline Hoxby was one of my supervisors, so you should take what I’m about to say with a grain of salt. But she has two useful studies on the points you discuss.

    * A recent charter school paper that uses the nearest public school as the comparison (not every public school in the state, as the AFT and DOE studies do), and finds small positive effects of charters:

    * A class size paper with a plausibly exogenous identification method that finds not an imprecisely estimated zero effect (which as you correctly point out wouldn’t tell us much), but a precisely estimated zero effect:

  4. “If small class sizes are educationally valueless and are adopted as a response to, say, teacher union pressure, we’d expect to see a very different allocation of resources in private schools”

    John – Private schools are very skilled at responding to what the market wants. Parents who stump up big wads of cash to send their kids to private schools, demand small class sizes.

    Private schools who want to keep their enrolments up give the parents what they want, and that means small class sizes. (Private schools must keep their enrolments up to stay financially viable. Because their costs are largely fixed costs, even small drops in enrolments make huge differences to their bottom line.)

    I’m not saying that small class sizes aren’t good, but you can infer very little about their educational value from what you observe private schools to be doing.

    Andrew – if class size has a precisely zero effect, does that mean you could have class sizes of 50, 60 or 70 without harming students ediucational outcomes? Presumably not. “Zero effect” must mean, at best, zero effect within a certain range.

  5. Off the top of my head, Andrew, one possible problem with Hoxby’s analysis is that schools may reallocate resources internally to compensate for class size variations, for example, by giving newer/weaker teachers the small classes.

    Milton, you need to say why parents demand small class sizes.

  6. “you need to say why parents demand small class sizes.”

    Because they believe them to be educationally beneficial.

    Which might or might not be true, but then a lot of parents also believe that six cuts of the cane are educationally beneficial, and that is another reason they send their kids to private schools. (These parents are then disappointed to discover that not even private schools do that any more, but that is another story.)

    Parents are important buyers of educational services, and they drive the market outcomes, but not all of them are well-informed buyers.

  7. Quoth Caroline Hoxby “it is not obvious that schools have stringent achievement maximization objectives imposed on them” from classsize_oct2000.pdf. This so misunderstands what a school is that you would conclude that this person has nothing to say about schools to people from planet Earth.In fact I would say it borders on the bizarre. A classic example of new speak that would have George Orwell spinning. Up there with “collateral damage”

  8. Milton, you’re right — the study was in Connecticut, and the relevant range was 20-30. Another Israeli study (based on Malmomides rule) found significant class size effects for classes in the 30-40 range.

    John, the instrument was a Connecticut rule that when classes in a given school went over X students, they got a new teacher. My guess is that this affected all classes in the school that year – most likely, schools would tend to have similar size classes.

    Bill, I think the key word is stringent. Here’s one example for you: http://www.washingtonpost.com/ac2/wp-dyn?pagename=article&contentId=A42155-2003Oct3&notFound=true

  9. Andrew, I don’t think you’re correct here. Hoxby compares cohorts within a school. Her argument is that, because of integer problems, some cohorts will consistently experience smaller classes than others. To the extent that the school as a whole can capture and redistribute benefits, as is implied by your comment, this won’t work.

Comments are closed.