I've been reading (listening to podcasts?) about health-care reform and education reform lately, and both issues have one thing in common: We measure the inputs but not the outputs.
What does it mean to measure inputs but not outputs? Basically we measure what goes into the system, typically in terms of dollars, but we don't do a very good job measuring what comes out of the system.
In health care, this means that we measure health treatments (tests, medicines, doctor visits, specialists and so on) but we don't measure health outcomes (does the patient get better?). For schooling, we measure spending on education, but we don't have fine-grained data on student outcomes. It's true that NCLB has forced all states to use standardized testing, but only at a very course-grained level. We could do much, much better.
Why do we measure inputs? There are a number of reasons, but the most obvious, and probably most important reason, is that measuring inputs is fundamentally easier than measuring outputs.
Why is this the case? Well, consider the task of measuring health outcomes for a heart surgeon. A crude approximation is to measure the percentage of successful surgeries performed. Except maybe the best surgeons perform the most difficult surgeries, so they're going to fail more because they're better. So we need to take into account the difficulty of the procedure being done, which makes measuring the outcomes more difficult. Furthermore, some successful surgeries are more successful than other successes (and some failed procedures are worse than other failures), so we need to adjust for that as well. At the very least, we need standards for evaluating degrees of difficulty of the surgery to be performed, and we need to follow-up for years after the surgery to see how well the surgeon's work holds up.
Now consider measuring health outcomes for a general practitioner responsible for caring for people for many years. How do we measure a successful outcome? Average lifespan of the patient? Quality of life of the patient? What does a successful outcome even look like?
The point isn't that we can't measure health outcomes. We can track degree of difficulty for surgeries, we can do follow-up with the patient. We have the technology to do that. We can compare a GP's patients against the general population, while controling for factors like the region of the country, the age of the patients, their income, their heritable disease risks, and so on. We have the technology to store and analyze all this data, we just don't do it aggressively enough.
Once we can track health outcomes, we can evaluate doctors based on health outcomes. And once we can do that, we can stop paying doctors based on treatments administered, which can be very expensive, and start paying doctors for better health outcomes. So long as doctors are paid for treatments, there is incentive to order every test that a patient's insurance will pay for. I don't see how this can not lead to over-consumption of health care for people with insurance, since they're not paying for the extra tests, and litigation-averse doctors will happily order every test covered by insurance.
We have a very similar problem in education, namely that we measure how much money goes into the system, but we have crude measurements of what comes out of the system. And it's hard to evaluate teachers for the same reasons that it's hard to evaluate doctors: some teachers have wealthy students with highly-involved parents, and some teachers don't. Sometimes the best teachers educate the worst students, where 50% of the students passing a basic proficiency exam is a big success.
As in medicine, these challenges don't mean that it's impossible to evaluate student learning in a more fine-grained way. It just means that it's going to take some more work to collect better data.
One intriguing idea is Value-Added Testing (VAT), where you measure a class at the beginning of the year to establish a baseline of what they know, then again at the end of the year to see what they've learned---in other words, what value has the teacher added to the students' knowledge? This controls for some of the problem of diverse student bodies.
Another exciting trend is performance-based pay, or merit pay, for the best teachers. Right now, faculty salaries are determined by seniority and not much else. So if you work for 10 years as an industrial chemist, then decide to teach high school chemistry, you make the same money as a 22-year old college graduate. In fact you probably make less because you don't have the teaching certificate that the 22-year old has, and you have to go back to school to earn that certificate. And while I'm not very familiar with the literature, I'm pretty sure that there's very little correlation between teaching certifications and performance in the classroom.
So you can't switch into teaching and make good money, and if you happen to be an excellent 22-year-old teacher, the only way to get a raise is to wait until you've been there long enough. So long as you don't get fired, it doesn't matter how well or how poorly you teach, you get the same raises everyone else does. It doesn't matter if the teacher next door is terrible and you're great; you get paid the same. Very few competitive industries pay everyone the same; why do we expect it to work in education? I don't think there are enough selfless individuals who will work long hours for little respect and no chance at a raise.
Are there problems with merit-based pay? Certainly. Basing pay entirely on standardized testing or value-added testing puts an awful lot of pressure on tests, and tests never tell the whole story in education. Plus it creates a huge incentive to teach to the test, to the exclusion of everything else. Merit-pay will have to include other metrics, like classroom observation and follow-up studies on how well the students do down the road. But it's hard to imagine a worse system given the amount of money we spend on education in the US.
The point of all of this: Statistics matter. In 1986, we thought that batting average and home runs were the most important statistics; most fans didn't understand the value of on-base percentage. Hell, most general managers didn't understand the critical importance of OBP. Baseball stat geeks have revolutionized our understanding of the game of baseball by looking at the data in great detail.
How the heck can we understand baseball so well, but we don't know nearly enough about health care or education?
No comments:
Post a Comment