Repeating Policy Behaviors Over and Over Again Despite the Evidence

According to Albert Einstein, the definition of insanity is to repeat the same behaviors over and over again in the hope that different results will materialize “the next time,” perhaps after this is fiddled with or that is fine-tuned. In the case of VAMs, it seems, educational policymaker are stuck in such a pattern of insanity, propagating and regenerating policies based on the use of students’ test scores to evaluate teachers, despite the fact that this has been a policy initiative of now almost 25 years past. Yes, we have research from now almost 25 years ago re: why this is a bad idea!

Gene V Glass published a study about just this in 1990. Click here to check it out. While this study was conducted pre-VAM (as we currently know it, particularly in terms of policy not form), the research-based reason why this is wrong remains the same. Regardless of the machine or statistical technique to be used, “the machine that is meant to do the world’s work must be observed in the actual world where [the VAM-based research] designs meet reality.”

The issues then, continue to be the issues today: “[t]he issue of ‘pupil performance in teacher evaluation’ needs to be discussed in a way that has not been separated from the many other pressures that shape a personnel system of assessment and rewards. The issue cannot be judged apart from how it exists in real places. The validity of such a process of evaluating teachers, its economy, its power to engender belief and win acceptance, depend on how it fits with many other realities of the teaching profession. What now must be addressed is how this notion gets applied in a context as complicated as education. How is the idea transformed as it moves from the statistician’s mind to the real world? How does it fare when the tradeoffs and balances are struck? Is the concept of evaluating teachers by student progress [to be] trusted?”

Glass then explores these questions using a case-based analysis of a number of district sites, concluding what we would still conclude and predict today, even with current and “more contemporary” approaches to the same policy “initiatives.” Glass concludes, that using student achievement data to evaluate teachers…

  1. …will nearly always be undertaken at the level of a school (either all or none of the teachers in a school are rewarded equally) rather than at the level of individual teachers since (a) no authoritative tests exist in most areas of the secondary school curriculum, nor for most special roles played by elementary teachers; and (b) teachers reject the notion that they should compete with their colleagues for raises, privileges and perquisites;
  2. …will always be combined with other criteria (such as absenteeism or extra work) which [continue to] prove to be the real discriminators between who is ultimately rewarded and who is not;
  3. …will always be too susceptible to intentional distortion and manipulation to engender any confidence in the data; moreover teachers and others who believe that no type of test nor any manner of statistical analysis can equate the difficulty of the teacher’s task in the wide variety of circumstances in which they work will further undermine the system;
  4. …will elevate tests themselves to the level of curriculum goals, obscuring the distinction between learning and performing on tests;
  5. …will often make room for symbolic administrative actions simply undertaken to reassure the lay public that student learning is valued and assiduously sought after…

Sound (insanely) familiar?!?


My Book on VAMs Released Today

My new book, titled “Rethinking Value-Added Models in Education: Critical Perspectives on Tests and Assessment-Based Accountability,” was officially released by my publisher – Routledge – today.

For those interested, you can order this book on Amazon, here, Barnes & Noble, here, or the Routledge site, hereThe book’s foreword is written by Diane Ravitch, the cover is below, and 100% of my personal royalties are headed to the ACODO Orphanage in Siem Reap, Cambodia.

For more information about the book, see a prior post about it here.

I am also speaking about the book this Saturday, at Arizona State University’s Tempe Campus from 10-11:30, for those of you in AZ who would like to attend. For more information about this event, and to RSVP, please click here.



Forcing the Fit Using Alternative “Student Growth” Measures

As discussed on this blog prior, when we are talking about teacher effectiveness as defined by the output derived via VAMs, we are talking about the VAMs that still, to date, only impact 30%-40% of all America’s public school teachers. These are the teachers who typically teach mathematics and/or reading/language arts in grades 3-8.

The teachers who are not VAM-eligible are those who typically teach in the primary grades (i.e., grades K-2), teachers in high-schools who teach more specialized subject areas that are often not tested using large-scale tests (e.g., geometry, calculus), and the teachers who teach out of the subject areas typically tested (e.g., social studies, science [although there is a current push to increase testing in science], physical education, art, music, special education, etc.). Sometimes entire campuses of teachers are not VAM-eligible.

So, what are districts to do when they are to follow the letter of the law, and the accountability policies being financially incentivized by the feds, and then the states (e.g., via Race to the Top and the NCLB waivers)? A new report released by the Institute of Education Sciences (IES), the research arm of the US Department of Education, and produced by Mathematica Inc. (via a contract with the IES) explains what states are up to in order to comply. You can find the summary and full report titled “Alternative student growth measures for teacher evaluation: Profiles of early-adopting districtshere.

What investigators found is that these “early adopters” are using end-of course exams, commercially available tests (e.g., the Galileo assessment system), and Student Learning Objectives (SLOs), which are teacher-developed and administrator-approved to hold teachers accountable for their students’ growth. Although an SLO is about as subjective as it gets in the company of the seemingly objective, more rigorous, and vastly superior VAMs. In addition, the districts sampled are also adopting the same VAM methodologies to keep all analytical approaches (except for the SLOs) the same, almost regardless of the measures used. If the measures exist, or are to be adopted, might as well “take advantage of them” to evaluate value-added because the assessments can be used (and exploited) to measure the value-added of more and more teachers. What?

This is the classic case of what we call “junk science.” We cannot just take whatever tests, regardless of to what standards they are aligned, or not, and run the data through the same value-added calculator in the name of accountability consistency.

Research already tells us that when using different tests, even on the same students of the same teachers at the same time, but using the same VAMs, gives us very, very different results (see, for example, the Papay article here).

Do the feds not see that forcing states to force the fit is completely wrong-headed and simply wrong? They are the ones who funded this study, but apparently see nothing wrong with the absurdity of the study’s results. Rather, they suggest, results should be used to “provide key pieces of information about the [sampled] districts’ experiences” so that results “can be used by other states and districts to decide whether and how to implement alternative assessment-based value-added models or SLOs.”

Force the fit, they say, regardless of the research or really any inkling of commonsense. Perhaps this will help to further line the pockets of more corporate reformers eager to offer, now, not only their VAM services but also even more tests, end-of-course, and SLO systems.

Way to lead the nation!

Breaking News: Houston Teachers Suing over their District’s EVAAS Use

It’s time!! While lawsuits are emerging across the nation (namely three in Tennessee and one in Florida), one significant and potentially huge lawsuit was just filed in federal court this past Wednesday in Houston on behalf of seven teachers working in the Houston Independent School District.

The one place I have conducted quite extensive research on VAMs, and the Education Value-Added Assessment System (EVAAS) and its use in this district in particular, is there in Houston. Accordingly, I’m honored to report that I’m helping folks move forward with their case.

Houston, the 7th largest urban district in the country, is widely recognized for its (inappropriate) using of the EVAAS for more consequential decision-making purposes (e.g., teacher merit pay and in the case of this article, teacher termination) more than anywhere else in the nation. Read a previous article about this in general, and also four teachers who were fired in Houston due in large part to their EVAAS scores here. See also a 12-minute YouTube video about these same teachers here.

According to a recent post in The Washington Post, the teachers/plaintiffs are arguing that  EVAAS output are inaccurate, the EVAAS is unfair, that teachers are being evaluated via the EVAAS using tests that do not match the curriculum they are to teach, that the EVAAS system fails to control for student-level factors that impact how well teachers perform but that are outside of teachers’ control (e.g., parental effects), that the EVAAS is incomprehensible and hence very difficult if not impossible to actually use to improve upon their instruction, and, accordingly, that teachers’ due process rights are being violated because teachers do not have adequate opportunities to change as a results of their EVAAS results.

“The suit further alleges that teachers’ rights to equal protection under the Constitution are being abridged because teachers with below-average scores find themselves receiving harsher scores on a separate measure of instructional practice.” That is, teachers are claiming that their administrators are changing, and therefore distorting other measures of their effectiveness (e.g., observational scores) because administrators are being told to trust the EVAAS data and to force such alignments between the EVAAS and observational output to (1) manufacture greater alignment between the two and (2) artificially inflate what are currently the very low correlations being observed between the two.

As well, and unique to but also very interesting in the case of the EVAAS, recalibration in the model is also causing teachers’ “effectiveness” scores to adjust retroactively when additional data are added to the longitudinal data system. This, according to EVAAS technicians, helps to make the system more rigorous, but this, in reality, causes teachers’ prior scores to change retroactively from years prior. So yes, a teacher who might be deemed ineffective one year and then penalized, might have his/her prior “ineffective” score retroactively changed to “effective” for the same year prior when more recent data are added and re-calibrated to refine the EVAAS output. Sounds complicated, I mean comical, but it’s unfortunately not.

This one is definitely one to watch, and being closest to this one I will certainly keep everyone updated as things progress as I can. This, in my mind, is the lawsuit that “could affect [VAM-based] evaluation systems well beyond Texas.”

Yes!! Let’s sure hope so…assuming all goes well, as it should, as this lawsuit is to be based on the research.

To watch an informative short (2:30) video produced by ABC news in Houston, click here. To download the complete lawsuit click here.


My Forthcoming Speaking Engagement in AZ about VAMs

If you reside in Arizona or plan to travel here soon, I invite you to join me as I discuss my new book, Rethinking Value-Added Models in Education: Critical Perspectives on Tests and Assessment-Based Accountability (available this Sunday, May 4th). To read more about this book, please click here re: a prior post with more of the book’s details.


This event is free and open to the public, and many representing many constituencies across the state have been invited. It will be held one week from this Saturday, on Saturday, May 10th, 2014 from 10:00am-11:30am in the Education Lecture Hall at Arizona State University (ASU’s main campus).

For more details and to RSVP, please use this link:

Coffee, tea, and light refreshments will be provided.

Hope to see you all there. If you cannot make it, I will also post the materials-post presentation, for all of you who follow this blog who do not reside in Arizona and feel not so inclined to fly in 😉