On a popular economics site, fivethirtyeight.com, authors use “hard numbers” to tell compelling stories, and this time the compelling story told is about value-added models and all of the wonders, thanks to the “hard numbers” derived via model output, they are working to reform the way “we” evaluate and hold teachers accountable for their effects.
In an article titled “The Science Of Grading Teachers Gets High Marks,” this site’s “quantitative editor” (?!?) – Andrew Flowers – writes about how “the science” behind using “hard numbers” to evaluate teachers’ effects is, fortunately for America and thanks to the efforts of (many/most) econometricians, gaining much-needed momentum.
Not to really anyone’s surprise, the featured economics study of this post is…wait for it…the Chetty et al. study at focus of much controversy and many prior posts on this blog (see for example here, here, here, and here). This is the study cited in President Obama’ 2012 State of the Union address when he said that, “We know a good teacher can increase the lifetime income of a classroom by over $250,000,” and this study was more recently the focus of attention when the judge in Vergara v. California cited Chetty et al.’s study as providing evidence that “a single year in a classroom with a grossly ineffective teacher costs students $1.4 million in lifetime earnings per classroom.”
These are the “hard numbers” that have since been duly critiqued by scholars from California to New York since (see, for example, here, here, here, and here), but that’s not mentioned in this post. What is mentioned, however, is the notable work of economist Jesse Rothstein, whose work I have also cited in prior posts (see, for example, here, here, here, and here,), as he has also countered Chetty et al.’s claims, not to mention added critical research to the topic on VAM-based bias.
What is also mentioned, not to really anyone’s surprise again, though, is that Thomas Kane – a colleague of Chetty’s at Harvard who has also been the source of prior VAMboozled! posts (see, for example, here, here, and here), who also replicated Chetty’s results as notably cited/used during the Vergara v. California case last summer, endorses Chetty’s work throughout this same article. Article author “reached out” to Kane “to get more perspective,” although I, for one, question how random this implied casual reach really was… Recall a recent post about our “(Unfortunate) List of VAMboozlers?” Two of our five total honorees include Chetty and Kane – the same two “hard number” economists prominently featured in this piece.
Nonetheless, this article’s “quantitative editor” (?!?) Flowers sides with them (i.e., Chetty and Kane), and ultimately declares victory for VAMs, writing that VAMs ultimately and “accurately isolate a teacher’s impact on students”…”[t]he implication[s] being, school administrators can legitimately use value-added scores to hire, fire and otherwise evaluate teacher performance.”
This “cutting-edge science,” as per a quote taken from Chetty’s co-author Friedman (Brown University), captures it all: “It’s almost like we’re doing real, hard science…Well, almost. But by the standards of empirical social science — with all its limitations in experimental design, imperfect data, and the hard-to-capture behavior of individuals — it’s still impressive….[F]or what has been called the “credibility revolution” in empirical economics, it’s a win.”
LOL, so who is funding the “editor” again?
If anyone is interested, you can read the working paper by Jesse Rothstein here:
http://www.princeton.edu/~ceps/workingpapers/170rothstein.pdf
Professor Rothstein’s main concern is that principals will make classroom assignments based on student characteristics that are not included in the data set.
And there is solid evidence, that this happens…and it happens in many ways that cannot be statistically controlled. While variables that might be correlated with some of these variable decisions can be controlled (although how well is certainly of debate), there is no way to control for, well, reality. See this detailed in: Paufler, N. A. & Amrein-Beardsley, A. (2014). The random assignment of students into elementary classrooms: Implications for value-added analyses and interpretations. American Educational Research Journal, 51(2), 328-362. doi: 10.3102/0002831213508299
Dr. Amein-Beardsley,
It is certainly true that every method that could be used to evaluate teachers might be subject to bias, so to reject this method there must be a further claim that this method is worse than others. Do you think that is the case? What is the evidence about biases in other methods of evaluation?
“It is certainly true that every method that could be used to evaluate teachers might be subject to bias, so to reject this method there must be a further claim that this method is worse than others.”
Utter nonsense, TE. All one has to do is to show the many inconsistencies, falsehoods and errors of the VAM methodology to show that it is COMPLETELY INVALID to show the absurdity of VAM. And then anyone who demonstrates that also has to claim that it is “worse than others” is well, ludicrous and risible. To use a COMPLETELY INVALID process for anything is the height of insanity and idiology (purposely misspelled) and has no place in any rationo-logical discussion of anything.
Duene,
Do all the posters on Dr. Ravitch’s blog understand that you believe that EVERY exam they have ever written and graded is completely invalid?
TE has a some skill at asking questions that struggle to rise to the level of being a non sequitur. As Click and Clack would put it, they are “time kill central”.
Confundus,
Duane has been posting about Noel Wilson’s criticism of exams at least biweekly for a couple of years on Dr. Ravitch’s blog.
Duane argues that Noel Wilson has shown that ALL exams, from PARC, SAT, and Woodcock Johnson III to teacher written exams in classes are ALL invalid.
Dr. Ravitch has been strangely silent about the validity of Noel Wilson’s work and I hope Duane can persuade her to publicly endorse Noel Wilson’s work if she finds it has merit or publicly dismiss his work if she finds it, like I do, word salad.
Fivethirtyeight should be ashamed for advancing such dishonesty. They’re using the same strategy as the tobacco industry & the petroleum industry employed to discredit known scientific findings of the effects of smoking and fossil fuels, respectively. Read ‘Merchants of Doubt’ 2010 by Naomie Oreskes. These dishonest economists are no better than the climate deniers paid to cast doubt on the science of climate changes. The disinformation campaign began in the 1980’s and it’s taken 30 long years to break through the climate change deniers public relations noise. She called these men “Merchants of Doubt”
http://www.nytimes.com/2015/06/16/science/naomi-oreskes-a-lightning-rod-in-a-changing-climate.html?_r=0
“Her core discovery, made with a co-author, Erik M. Conway, was twofold. They reported that dubious tactics had been used over decades to cast doubt on scientific findings relating to subjects like acid rain, the ozone shield, tobacco smoke and climate change. And most surprisingly, in each case, the tactics were employed by the same group of people.
The central players were serious scientists who had major career triumphs during the Cold War, but in subsequent years apparently came to equate environmentalism with socialism, and government regulation with tyranny
The goal- cast doubt on the science.”
Well said. I’m going to look into this further.
When the Chetty study came out I was immediately suspicious of the numbers since they were big, but spread out over a lifetime and over an entire classroom of kids? So, using simple division I came up with a difference in earnings of about $45 per year per kid. My memory may not be the best on this, but I’m pretty sure I’m solidly in the ball park. Obviously this increase in earnings does not hold true for hourly employees who work for minimum wage under a predefined pay rate increase. It also does not explain the mechanism by which these kids with one better teacher are awarded the chump change increase in salary nor what effect a downturn in the economy might have on them. Long story short, the Chetty study remains the most sparkly of fairy dust theories concerning teacher effects on the economy at large.
Jon,
At it’s heart, Chetty et al. shows that good teaching in K-12 has a lifelong impact on students. I am at a loss to figure out why teachers argue that this is not the case.
teachingeconomist, not only should one consider the alternative for rating teachers and/or schools, but let’s consider these decisions from the students perspective. Assume that there is an 85% chance that a teacher is bottom 5% and clearly ineffective. That means that there is a 15% chance that the teacher is just an average, run-of-the-mill effective teacher.
The question then becomes whose interests outweigh the other. We understand that teachers think it is unacceptable that any action be taken when there is a 15% chance the teacher is actually effective. However, considering there is an 85% the disadvantaged students are being saddled with an ineffective teacher, how can administrators and the public not demand action?! That action may not be to fire the teacher right away. It may be to refer the teacher for more professional development and removal from a non-core class. But when there is a > 50% chance (or I would argue even a smaller % chance) that a teacher is ineffective, how can one morally leave that questionable teacher in the classroom?
The column was so off base that I closed the page and had to re-open to re-read and see if I had been reading the “Onion”. The only accurate section was found in the comments section, where the posters were obviously better informed that the economists who were engaged in “almost-like-science-activities”, but not. Your Vamboozled website and you were frequently referenced in the rebuttals. Thank you for your work!
And what do these wizards say about the estimated 69% of teachers who have job assignments for which there are not data for calculating VAM? Talk about inferential leaps through thin air. Missing data? No problem. You and others have methodically exposed the VAM sham. Economists dabbling with handy data have reputations to defend and they will do whatever it takes in PR to do so.
I suspect Chetty et al. would say you can not use growth in test scores when there are no tests linked to the teachers.
My favorite line from Dr. Amrein-Beardsley’s 2008 paper is this:
“An expert teacher may be labeled above average, whereas an equally expert teacher with significantly fewer students might not be acknowledged at all, being misclassified as average because of having fewer student records. Inversely, an ineffective teacher who teaches a large class might be penalized for being below average, whereas an equally ineffective teacher who teaches a smaller class may go undetected.”
So she is concerned that since not all ineffective teachers will be identified, we should just give up on VAM. WOW!
It sounds like the complaints from my kids… “well, he hit me too so why aren’t you punishing him”. Dr. Amrein-Beardsley, meanwhile, couldn’t care less about the disadvantaged kids stuck in either of the ineffective teachers’ classrooms. She is just concerned about equity among ineffective teachers. This is why the opt-out, anti-VAM movement is a complete joke and merely a ruse to take future income from the pockets of disadvantaged students and put it in the paychecks of ineffective teachers. It’s really that simple.
How many days have you spent teaching in a high-needs classroom? I’ve spent plenty, sadly, and you accuse me of not caring less? Students like those who I have taught, very effectively I might add, are the exact reasons why I continue to take on folks like you who literally have no clue. Nice post, by the way, about you on Diane Ravitch’s blog: http://dianeravitch.net/2015/07/28/ny-teachers-offers-advice-to-vam-loving-reader/ If it is some level of fame you seek, try to do so with some integrity. If not, consider yourself blocked.
I would like to point you to another critic whose work has gotten much attention and to which Chetty has basically failed to provide any kind of convincing answer (though to his credit, he tried).
Moshe Adler, Review of Measuring the Impacts of Teachers, National Education Policy Center, http://nepc.colorado.edu/thinktank/review-measuring-impact-of-teachers
Among other severe issues with the Chetty paper, Adler points out that Chetty et al. report a small (albeit visually wildly exaggerated) but statistically significant effect of teacher VAM on student earnings at age 28 but they found no statistically significant effect at all on earnings at age 30. This finding was reported by the authors in a Working paper but not in the AER publication.
Further, the effect of VAM on earnings Chetty does report – even where it is statistically significant – is small. It is so small that Chetty had to resort to stretching the y-axes in all his graphs, something you learn in any introductory statistics course is a big No-No (Adler also points out other tricks in the way the earnings effect is reported to make it look bigger). All the graphs that were shown to the judge and public in the Vergara case have stretched y-axes. The earnings axis is stretched 15-fold to visually boost the claimed effect (also compare the graphs at http://nepc.colorado.edu/blog/revisiting-chetty-rockoff-friedman-molehill and http://www.washingtonpost.com/blogs/wonkblog/wp/2014/06/10/a-california-judge-just-ruled-that-teacher-tenure-is-bad-for-students/ – the reported effect mysteriously got bigger, based on the exact same data set). Furthermore, there are no error bars. If the error bars had been shown, they would almost certainly swamp the purported effect on earnings. This is more Fox News (http://www.mediaite.com/tv/fox-news-airs-seriously-misleading-obamacare-graphic/) than serious social science.