My First Book “Review”

Posted on June 26, 2014 by Audrey Amrein-Beardsley

I just came across the first external review of my new book, Rethinking Value-Added Models in Education on Amazon, and I thought I would share this review with you all in that I am happy to announce that it captures exactly what I meant to do with this book, particularly for those of you out there who are practitioners working within (and under) the realities the come along with the adoption of VAMs for increased accountability.

This reviewer gave the book five out of five stars (Thanks!!), and wrote: “This book provides a logical research based response to assumptions underlying VAM. Every educator should read this book as VAM may control our professional destiny. This book provides the research to challenge our candidates and policy makers as so many states are moving forward in adopting VAM. Statistical methodology is so simply explained that I almost feel like an expert in VAM.”

Mission accomplished, I hope! I hope as well that, as this reviewer wrote, this book will continue to “provide the research” you all might need to continue “to [wisely] challenge your candidates and policy makers” in these regards.

Onward!

For more information about the book, see it on Amazon or see a prior post about it here.

Also, if you have read this book so far, please do review it on or Amazon, or elsewhere for that matter. I’d prefer not to recruit folks (e.g., my graduate students) to write reviews of my book, as I truly want people to provide their honest, external reviews and feedback…nothing against my wonderful graduate students, of course 😉

Moshe Adler on Chetty et al.

Posted on June 24, 2014 by Audrey Amrein-Beardsley

Recall the study at focus of now many posts on this blog? Written by Raj Chetty and two of his colleagues at Harvard, cited first in President Obama’ 2012 State of the Union address when Obama said, “We know a good teacher can increase the lifetime income of a classroom by over $250,000,” and more recently the focus of much attention when the judge in Vergara v. California cited Chetty’s testimony and the aforementioned team’s research as evidence that “a single year in a classroom with a grossly ineffective teacher costs students $1.4 million in lifetime earnings per classroom?”

Well, one of this study’s biggest critics, and most important critics as an expert in economics himself, has written a response to all of this. More specifically, Moshe Adler – economist affiliated with Columbia University and Empire State College, State University of New York and author of the 2010 book, Economics for the Rest of Us: Debunking the Science That Makes Life Dismal – first (in April 2014) wrote a critique of Chetty et al.’s studies for the National Education Policy Center (NEPC) during which he also compared this study from its first release in 2011 (cited by Obama) to its current 2014 version (cited by the judge in Vergara v. California) that can be found here, after which Chetty et al. wrote a response (in May 2014) that can be found at the bottom of the page here, after which Adler released a final response (in June 2014) that can also be found at the very bottom of the page here.

For those of you interested in reading all three pieces, all three are really quite critical to understanding Chetty et al.’s methods, as well as their methods’ shortcomings; their assumptions, as well as their feasibility in context; their results, as unfortunately overstated as they are; and the like. For those of you who want just the highlights, though (as also referenced in the press release just released here), here are the nine key points taken from now the two studies. These are the nine concerns made and established by Moshe in the first round, to which Chetty et al. responded, but seemingly still stand as solid criticisms.

As per Chetty et al.’s study #1:

Value-added scores in this report and in general are unstable from year to year and from test to test, but the report ignores this instability.
The report inflates the effect of teacher value-added by assuming that a child’s test scores can be improved endlessly [over time].
The procedure that the report develops for calculating teacher value-added varies greatly between subjects within school levels (math or English in elementary/high school) and between schools within subjects (elementary or middle school math/English), indicating that the calculated teacher value-added may be random.
The commonly used method for determining how well a model predicts results is through correlations and illustrated through scatterplot graphs. The report does not [however] present either the calculations or graphs and instead invents its own novel graph to show that the measurements of value-added produce good predictions of future teacher performance. But this is misleading. Notwithstanding the graph, it is possible that the quality of predictions in the report was poor.

As per Chetty et al.’s study #2:

An earlier version of the report found that an increase in teacher value-added has no effect on income at age 30, but this result is not mentioned in this revised version. Instead, the authors state that they did not have a sufficiently large sample to investigate the relationship between teacher value-added and income at any age after 28, but this claim is untrue. They had 220,000 observations (p. 15), which is a more than sufficiently large sample for their analysis.
The method used to calculate the 1.34% increase [that a 1 standard deviation increase in teacher VA raises wages at age 28 by 1.34% (p. 37)] is misleading, since observations with no reported income were included in the analysis, while high earners were excluded. If done properly, it is possible that the effect of teacher value-added is to decrease, not increase, income at age 28 (or 30).
The increase in annual income at age 28 due to having a higher quality teacher “improved” dramatically from the first version of the report ($182 per year, report of December, 2011) to the next ($286 per year, report of September, 2013). Because the data sets are not identical, a slight discrepancy between estimates is to be expected. But since the discrepancy is so large, it suggests that the correlation between teacher value-added and income later in life is random.
In order to achieve its estimate of a $39,000 income gain per student, the report makes the assumption that the 1.34% increase in income at age 28 will be repeated year after year. Because no increase in income was detected at age 30, [however] and because 29.6% of the observations consisted of non-filers, this assumption is unjustified.
The effect of teacher value-added on test scores fades out rapidly. The report deals with this problem by citing two studies that it claims buttress the validity of its own results. This claim is [also] both wrong and misleading.

Gene Glass v. Chetty and Kane

Posted on June 23, 2014 by Audrey Amrein-Beardsley

Following the Vergara v. California decision now two weeks ago (see three recent VAMboozled posts about this court decision here, here, and here), one of my mentors and still friends/colleagues – Gene Glass, Regents’ Professor Emeritus at ASU – wrote the following on his twitter and Facebook account about Raj Chetty’s and Tom Kane’s testimonies, testimonies that played key roles in the judge’s (unfortunate) decision ruling teacher tenure as unconstitutional.

Testimony in Vergara by Harvard profs. Does anybody–other than Judge Treu–really believe these guys?! Amazing! — These dudes must be in love with their production functions. How anyone but a know-nothing judge could buy this stuff is amazing to me. A bad teacher costs a class of 28 kids $1.4 Million in life-time earnings. Really? What do a couple of lousy economists cost society?

Glass then followed up with a comment: “Aristotle in Nichomachean Ethics: “We must not expect more precision than the subject-matter admits.” (Chp. 3)…after which a Facebook friend replied…”Aristotle was 2.13 times more effective than Plato at teaching Aristotle’s ethics.” Funny!

Anyhow, somebody else at The Becoming Radical seemed to come across this post and wrote another blog post around it titled, “The Very Disappointing Teacher Impact Numbers from Chetty.” To read more about what this author has titled the now “very famous” but still “mostly hypothetical” Chetty et al. study and what their numbers ACTUALLY mean, even if true and accurate in the real world, click here.

Unidentified Bloggers Defending Themselves Anonymously

Posted on June 19, 2014 by Audrey Amrein-Beardsley

“People should stand behind their own words…that’s doubly-true of people in public life.”

For those of you who missed it, about two weeks ago Diane Ravitch posted a piece about (and including) a set of emails exchanged between Diane, Raj Chetty, and me (although this post did not include Chetty’s emails as he did not grant Diane permission to share) about Chetty et al.’s now infamous study, the study at the heart of the recent Vergara v. California win (see Diane’s post here). In the comments section of this blog post, a unidentified respondent by the name of “WT,” as in “What The…,” went after me and my review/critique of Chetty et al.’s study, also referenced in this same post. See WT’s comments, again here, about 40 comments down: “WT on June 2, 2014 at 4:00 p.m.”

Anyhow, it became clear after a few back-and-forths that “WT” was closer to this study than (s)he seemingly wanted to expose. Following these suspicions, I wrote that I was at that point “even more curious as to why the real “WT” [wasn’t] standing up? Who [was] hiding behind (a perhaps fictitious) set of initials, or perhaps an acronym? Whoever “WT” [was] seem[ed] to care pretty deeply about this study, not to mention know a lot about it to cite directly from [very] small sections of the 56 pages (something I for sure would not be able to do nor, quite frankly, would I take the time to do given I’ve already conducted two reviews of this study since 2011). Might “WT” [have been] somebody a bit too close to this study, hence the knee-jerk, irrational reactions, that (still) lack[ed] precision and care? Everyone else (besides Harold [another commentator on this blog post]) ha[d] fully identified themselves in this string. So what [was to say] WT?”

Well, “WT” said nothing, besides a bunch of nothingness surrounding his anonymous withdrawal from the conversation string. Perhaps this was Chetty as “WT?” Perhaps not, but I’d bet some serious cash whoever “WT” was was pretty darn close to the Chetty et al. study and didn’t want to show it.

Sooo…now getting to the best part of all of this and how this has, in an interesting and similar turn of events, evidenced itself elsewhere. This video just came out on our local news station in Phoenix about a very similar situation in the state of Arizona and the state’s Superintendent of Public Instruction – John Huppenthal – engaging in some equally shady blogging tactics. Do give it a watch if you want yet another check on reality.

View Video Here

Vergara v. California: Concerns Beyond Teachers

Posted on June 18, 2014 by Audrey Amrein-Beardsley

I received this “public letter” via email and felt compelled to share it with you all. It is written by Ed Johnson, “Advocate for Quality in Public Education” and devotee of W. Edwards Deming. It is about the quite honestly shameful testimony of Harvard Economics Professor Raj Chetty (the source of prior posts here and here), his testimony’s impact on the Vergara v. California decision last week (see two recent posts about this court decision here and here), and the deleterious implications of the ways that Chetty and those whom he advises (including Obama’s key leaders) think about educational measurement — implications that unfortunately span well beyond the teachers at the center of what is now so many similar VAM-based measurement systems.

June 17, 2014

Thanks mainly to Raj Chetty’s testimony, corporate school reformers won, or bought, the Vergara v. California case. Now, in consequence, there should be grave concern. California Superior Court’s ruling and Chetty’s cheerful and gleeful testimony offer insight into the kind of intelligence-without-wisdom that can only lead to hastening the demise of democratic ideals along with the rise of plutocratic ideals and attendant morally and ethically corrupt and corrupting ways of social life legitimized in public law. It is worth spending nearly two hours to watch and listen to Chetty’s testimony, so one may disallow oneself ever being able to claim, as some in Germany once claimed, “I didn’t know.” Be sure to listen to Chetty explain President Obama’s and the White House’s involvement.

Chetty based his testimony on having posed and answered for himself the question: “Are teachers’ impacts on students’ test scores (“value-added”) a good measure of their quality?” He testified that his research findings brought him to answer the question in the affirmative. As if nothing much beyond test scores matter, Chetty concluded that teachers’ impact on students’ test scores are a good measure of the quality of students’ teachers. Now just think about that for a minute and what it means and the potential precedent it holds for the future.

Chetty applied economic reasoning to “big data” sets of students’ test scores and relatable other kinds of data to, in effect, contrive little more than a method for distributing teachers over the mythical bell curve. Bottom-line result? “Déjà vu all over again” – that is, teachers ranked. The five percent or so of teachers forced into the head of the bell curve are deemed, arguably, highly valuable teachers. Similarly, the five percent or so of teachers forced into the tail of the bell curve are deemed, arguably, non-valuable teachers. Then Chetty intimated that the teachers ranked by his method establish the justification to fire the non-valuable teachers among them. And the quicker that is done, the better for the students. But, of course, “the quicker, the better” can only be done if teachers are disallowed tenure – that is, if teachers are allowed no rights to due process.

Now, the rub is, why would anyone today believe Chetty’s magical “value-added” genie of a measure will still be in its bottle in the future? Why would anyone believe corporatists will never apply Vergara v. California to employees? Why would anyone believe our government will never back Vergara v. California to be applied beyond public school teachers to include ever more citizens? Why would anyone believe “teachers” will never be generalized to “people” at the behest by plutocrats’ desires to get rid of non-valuable people the quicker, the better? Why would anyone believe eugenics will never be embraced, again?

Unlike Chetty’s, the works, teachings, and philosophy of W. Edwards Deming (and similar others) are morally, ethically and, yes, economically grounded in the aim to continually improve the quality of life for everyone. For example, Deming once asked, as I recall it, this powerful, and no doubt to some, unsettling question: “Even if you could rank people with accuracy, would you do it?”

In comparison to Chetty’s economic-only reasoning, Deming offers simple, profound wisdom that even elementary school children can be taught hence learn:

“Let X be the contribution of some individual, (YX) the effect of the system on his performance. Then suppose that we have some number for his apparent performance, such as eight mistakes during the year, or sales of $8,000,000.

“Then, X + (YX) = 8.

“We need X. Unfortunately, there are two unknowns and only one equation. Johnny in the sixth grade knows that no one can solve this equation for X. Yet people that use the merit system think that they are solving it for X. They ignore the other term (YX), which is predominant.

“There is another factor to take into account, the Pygmalion effect. Rated high at the start, anyone stays high. Rated low at the start, he stays low.

“Ranking creates competition between people, salesmen, teams, divisions. It demoralizes employees.

“Ranking comes from failure to understand variation from common causes.”

–W. Edwards Deming. The New Economics for Industry, Government, Education (pp. 25-26). Kindle Edition.

Raj Chetty’s value-added measure (VAM) contrivance simply invites staying stuck in the status quo of regressive and mechanistic ways of thinking that try to reduce life’s complexity to simplistic algorithms and scripts. Instead, the country could be progressing, provided it had leaders that held a stronger adherence to democratic ideals than to plutocratic ideals, and that held a more just, moral, and ethical social systemic world view (see Ackoff, 11/1993).

Frankly, where are the words with which to express deeply felt sorrow and revulsion with Raj Chetty’s VAM, California Superior Court’s ruling, and Obama’s involvement? How ironic that the country’s first self-proclaimed “African American” President serves a chief facilitator of attacks on public education.

Ed Johnson
Advocate for Quality in Public Education

“The foundation of every state is the education of its youth.”
Diogenes of Sinope (c. 412 – c. 323 BCE)

ACT’s Dan Wright on a Simple VAM Experiment

Posted on June 17, 2014 by Audrey Amrein-Beardsley

A VAMboozled! folllower, Dan Wright, who is also a statistician at ACT (the nonprofit company famously known for developing the college-entrance ACT test), wrote me an email a few weeks ago about his informed and statistical take on VAMs. I invited him to write a post for you all here. His response, with my appreciation, follows:

“I am part cognitive scientist, part statistician, and I work for ACT, Inc. About a year ago I was asked whether value-added models (VAMs) provide good estimates of teacher effectiveness. I found lots of papers that examined if different statistical models gave similar answers to each other and lots showing certain groups tended to have lower scores, but all of these papers seemed to bypass the question I was interested in: do VAMs accurately estimate teacher effectiveness? I began looking at this and Audrey invited me to write a guest blog post describing what I found.

The difficulty is that in the real world we don’t really know how effective a teacher, that is in “objective” terms, so that we do not have much with which to compare our VAM estimates. Hence, a popular alternative/way for statisticians to decide if an estimation method is good or bad is to simulate data using particular values, and then see if the method produces estimates that are similar to the values expected.

The trick is how to create the simulation data to do this. It is necessary to have some model for how the data arise. Hence, I will use a simple model to demonstrate the problem. It is usually best to start with simple models, make sure the statistical procedures work, and then progress to more complex models. On that note, I have an under review paper that goes into more details with other models with more simulation studies. Email me if you want a copy.

Anyhow, the model I used to create the data for this simulation starts with three variables. The first encapsulates everything about the students that is unique to them, including ability, effort, grit, and even the home environment (called AB in the code below). The second encapsulates all the neighborhood and environmental factors that can influence everything from financial spending in schools to which teachers are hired (called NE). This value is unique to each teacher (so it is simpler than real data where teachers are nested in schools). The third is teacher effectiveness (called TE). I created it from the neighborhood variable plus some random variation. These three variables would be unmeasured in a real data set (of course, some elements of them may be measured, but not all elements of them), but in a computer simulation their values are known.

In addition, there are two sets of test scores. The first are scores from before the student has encountered the teacher (called PRE). They are created by adding the student ability variable, the neighborhood variable, and some random variation. The second set of test scores are from after the student has encountered the teacher (called POST). They are created by adding the first set of test scores, the student ability variable, one-fifth of the teacher effectiveness variable (less than the other effects since the impact of a single teacher is usually estimated at about 10% or less of student achievement), and some random variation.

Again, however, this model is simpler than real educational data. Accordingly, there are no complications like missing values, teachers nested within schools, etc. Also, I will use a very simple VAM, just using the first set of scores to predict the second set of scores, but allowing for random variation by teachers and by students. Given the importance placed on the results of VAMs in many different countries and in many industries (not just in education), this method should work…Right?

Theories about causation actually suggest that there will be a problem. I won’t go into the details about this (again, email me for my paper if you want more information), but using the first set of scores in the VAM allows information to flow between the teacher effectiveness variable and the second set of test scores through the other unmeasured variables. This messes up trying to measure the effect of teachers on the final scores.

But let’s see if a simulation supports that this is problematic as well. The simulation used the freeware R and the code is below so that you can repeat it, if you want, or change the values. Download R onto your computer (follow the instructions on http://cran.us.r-project.org/), open it, copy the code, and paste it into R. The # sign means R ignores the remainder of the line so I have used that to make comments. If you think another set of numbers are more realistic, put those in. Or construct the data in some other way. One nice thing about simulations is you can keep trying all sorts of things.

For the values used, the correlation is -.13. A negative correlation means the teachers who are more effective tend to have lower estimated teacher effectiveness scores. That’s not good. That’s bad! Don’t get hung up with this specific number, though, as it moves around depending on how the data are constructed. The correlation can go up (e.g., change any one, but not two, of the first four values to -1) and down (e.g., increase the size of AB2POST to 2).

Given this, there is a conclusion, as well as a recommendation. The conclusion is that value-added estimates can be very inaccurate for what seem to be highly commonsensical and plausible models for how the data could arise, and where they are bad is predicted from theories of causation. The recommendation is that those promoting and those critical of VAMs should write down plausible models for how they think the data in which they are interested arose and see if the statistical procedures used perform well.

Dan Wright

*The personal opinions expressed do not necessarily represent the views and opinions of ACT, Inc.

Breaking News: Vergara Going “On Tour”

Posted on June 15, 2014 by Audrey Amrein-Beardsley

Following up on my most recent post regarding the “Vergara Case Lost in Los Angeles,” it seems “they” aren’t stopping in Los Angeles.

A VAMboozled! follower posted this recent op-ed on this topic on Aljazeera, in which the authors write about how “Students Matter, financier of the case (Vergara vs. State of California) with the funding of tech magnate David Welch, is preparing to bring similar suits to New York, Connecticut, Maryland, Oregon, New Mexico, Idaho and Kansas. This state-by-state strategy follows a historical precedent by similarly motivated groups, such as turn-of-the-20th-century women suffragists and contemporary gay marriage activists, seeking to challenge regulations at the state level to build a groundswell for national upheaval.” To read more, click here.

But also in breaking news, it seems that the Houston Independent School District (HISD) – the district in which the American and Houston Federation of Teachers are suing the district for their wrongful use of, and over-emphasis on VAMs (see a prior post about this here) – is to be defended by the same law firm who just won the Vergara trial.

While at the same time the HISD school board “quietly voted to dial back the weight student test scores have on teacher evaluations,” the school board hired Gibson, Dunn and Crutcher LLP to defend the district in the way the district is using its scores derived via its VAM (i.e., the Education Value-Added Assessment System [EVAAS]). “[A] grant from the [Houston-based] Laura and John Arnold Foundation will help fund the litigation.”

As for the national and local unions’ response? As per a joint press release: “Hiring a powerful law firm and accepting a grant from one of the country’s most conservative and well-endowed foundations shows the Houston school system will spend as much money as it can to defend what we believe are unjust and unfair actions, rather than act in the best interests of the district’s students and teachers.” To read more, click here.

Vergara Case Lost in Los Angeles

Posted on June 12, 2014 by Audrey Amrein-Beardsley

A few months ago, I wrote a few posts about the “Vergara v. California” case happening in Los Angeles, California, as well as posts about the testimonies of Thomas Kane (Economics Professor from Harvard University), David Berliner (Regents Professor Emeritus from Arizona State), Linda Darling-Hammond (Professor from Stanford), and Jesse Rothstein (Economics Associate Professor from the University of California – Berkeley).

The case involved nine public school students (backed by some serious corporate reformer funds) who were collected by the “Students Matter” to challenge five California state statutes that supported the state’s “ironclad [teacher] tenure system.” The prosecution’s argument was that students’ rights to a good education were being violated by teachers’ job protections…protections that were making it too difficult to fire “grossly ineffective” teachers. The protections at issue included the set of statutes that granted permanent employment status to teachers after 18 months on the job, that required a lengthy procedure to dismiss teachers, and that set up a seniority system in which the teachers most recently hired were the first to be fired when layoffs occurred, or in recent years needed to occur given economic crises and swings.

The suggested replacement to the “old” way of doing this, of course, was to use value-added scores to make “better” decisions, as based on “the data,” about whom to fire and whom to keep around as per whether they were positively impacting student achievement, and growth in student achievement. This is precisely why this case mattered so much for this audience here, particularly as we continue to think about VAMs and how they are being positioned, many times falsely, for that which they theoretically can do versus what they really can do in practice.

Anyhow, it seems, as of two days ago, the defense lost the case. As per a recent “Washington Post” post, and actually the best post I’ve read thus far on the verdict as written by Kevin Welner (attorney and Professor of Education policy at University of Colorado – Boulder), “what the court found is that the state failed to meet its burden to show a compelling interest to justify policies that the court found were causing the employment of several thousand “grossly ineffective” teachers scattered throughout California.”

The case will certainly be appealed to a higher court, however, so while a loss there is still hope, as per Welner.

Welner continued writing that there is likely/hopefully a silver lining to all of this. He writes, “the decision gives real teeth to the state’s Constitution, and that could be a very good thing. It’s those teeth that I find fascinating, since an approach like that used by the Vergara judge could put California courts in a very different role — as a guarantor of educational equality — [different] than we have thus far seen in the United States.” What continuing this case could bring to light, is the extent to which low income children and children of color in some of the highest needs schools are denied equal educational opportunities, not just as per their “grossly effective” teachers (which is an overstatement in and of itself, but that which was constructed and positioned with the case here, but more importantly with all other things causing the gross inequities that also matter, if not matter more. These include the educational policies surrounding issues with “transportation, school choice, buildings, funding formulas, access to computer technology, enriched curriculum, testing and accountability policies, and segregated and stratified schools.”

Welner concludes, “Would this be a bad thing? In my view, not necessarily. Courts play an extremely important role: protecting political minorities from the tyranny of the political majority. When they relinquish that role, stepping aside and granting discretion to the executive and legislative branches, the easily foreseeable consequence is that laws and rules will disadvantage that minority. The Vergara plaintiffs and Judge Treu, whether intentionally or unintentionally, are pointing us to a different model—what is sometimes denounced as “judicial activism.” But active engagement of courts to demand that the educational opportunities of minorities are protected could be a crucial step forward toward meaningfully closing opportunity gaps and thus achievement gaps. These days, we need all the silver linings we can find.”

Again, to read this post, please click here. It is definitely worth the balanced read and point of view.

Consequences of Non-Compliance in the State of Washington

Posted on June 10, 2014 by Audrey Amrein-Beardsley

In case you missed it, from a post on Diane Ravitch’s blog, about U.S. Secretary of Education Arne Duncan withdrawing the state of Washington’s No Child Left Behind (NCLB) Waiver for noncompliance re: VAMs:

The Education Department is pulling Washington state’s No Child Left Behind waiver because the state has not met the department’s timeline for tying teacher evaluations to student performance metrics.

Washington is the first state to lose its waiver. The loss will give local districts less flexibility in using federal funds. For instance, they may now be required to spend millions on private tutoring services for at-risk students. The waiver revocation could also result in nearly every school across the state being labeled as failing under NCLB.

Washington had pledged in its waiver application to make student growth a significant factor in teacher and principal evaluations by the 2014-15 school year. But the state Legislature refused to pass a bill mandating that student performance on statewide assessments be included in teacher evaluations. The department placed the state on “high-risk” status in August. Arizona, Kansas and Oregon are also at risk of losing their waivers.

VAMboozled!

VAMboozled! A blog by Audrey Amrein-Beardsley

Monthly Archives: June 2014

Top Three Summer Reads Recommended for Educators