Lies, damned lies, and statistics

Not sure why I read this stuff, as I have no connection whatever to education in the US, but a couple of posts recently caught my eye, one on the Freakanomics blog, and the other on Pissed Off (Teacher).

Stephen Dubner of Freakanomics, dips into the mailbag and quotes this correspondent:

Hi guys. I know you guys have railed against confusing correlation with causality, and also discussed various statistical problems, but you do not seem to have addressed one that affects many real world decisions. I have noticed what appears to be an increase in decisions based on single variant analysis and averaging, often ignoring consequences of those decisions and/or the context of the analysis. It seems you ought to address these, as they are right up your alley. Some simple common examples…

He gives 3: Post Office boxes, schools and testing, and global warming. Here’s an excerpt from the schools and testing example:

Another example happening in the SF Bay Area involves schools. The District looks at average standardized test scores of each school and closes the ones with the lowest average. They do not consider whether these ones which are lower have the poorest kids with least parent involvement and most transience and the like. But, by closing the school and moving them to a much larger school, these low performing kids continue to be low performers, they just do not pull down the average as much. Worse, these low performing kids often do worse (at least according to a recent study at Stanford) and/or drop out. Dropping out helps the averages, so you could argue it is a good policy (Texas allegedly uses that to improve their scores), but the consequences are likely dire for society (more crime or welfare or both). The problem is that the poor performing school may in fact have those particular kids doing much better than they would in another school (due to extra attention, etc), but decision makers are taught that “statistics do not lie”. This is the opposite of some of your situations (where statistics show “common sense” to be wrong).

Pissed Off steals a parody of NCLB:

My dentist is great! He sends me reminders so I don’t forget checkups He uses the latest techniques based on research. He never hurts me, and I’ve got all my teeth.
When I ran into him the other day, I was eager to see if he’d heard about the new state program. I knew he’d think it was great.
“Did you hear about the new state program to measure effectiveness of dentists with their young patients?” I said.
“No,” he said. He didn’t seem too thrilled. “How will they do that?” “It’s quite simple,” I said. “They will just count the number of cavities each patient has at age 10, 14, and 18 and average that to determine a dentist’s rating. Dentists will be rated as excellent, good, average, below average, and unsatisfactory. That way parents will know which are the best dentists. The plan will also encourage the less effective dentists to get better,” I said. “Poor dentists who don’t improve could lose their licenses to practice.”
“That’s terrible,” he said.
“What? That’s not a good attitude,” I said. “Don’t you think we should try to improve children’s dental health in this state?”
“Sure I do,” he said, “but that’s not a fair way to determine who is practicing good dentistry.”
“Why not?” I said. “It makes perfect sense to me.”
“Well, it’s so obvious,” he said. “Don’t you see that dentists don’t all work with the same clientele, and that much depends on things we can’t control? For example, I work in a rural area with a high percentage of patients from deprived homes, while some of my colleagues work in upper middle-class neighborhoods. Many of the parents I work with don’t bring their children to see me until there is some kind of problem, and I don’t get to do much preventive work. Also, many of the parents I serve let their kids eat way too much candy from an early age, unlike more educated parents who understand the relationship between sugar and decay. To top it all off, so many of my clients have well waterwhich is untreated and has no fluoride in it. Do you have any idea how much difference early use of fluoride can make?”
“It sounds like you’re making excuses,” I said. “I can’t believe that you, my dentist, would be so defensive. After all, you do a great job, and you needn’t fear a little accountability.”

There’s more. Go read the whole thing.