According to Albert Einstein, the definition of insanity is to repeat the same behaviors over and over again in the hope that different results will materialize “the next time,” perhaps after this is fiddled with or that is fine-tuned. In the case of VAMs, it seems, educational policymaker are stuck in such a pattern of insanity, propagating and regenerating policies based on the use of students’ test scores to evaluate teachers, despite the fact that this has been a policy initiative of now almost 25 years past. Yes, we have research from now almost 25 years ago re: why this is a bad idea!
Gene V Glass published a study about just this in 1990. Click here to check it out. While this study was conducted pre-VAM (as we currently know it, particularly in terms of policy not form), the research-based reason why this is wrong remains the same. Regardless of the machine or statistical technique to be used, “the machine that is meant to do the world’s work must be observed in the actual world where [the VAM-based research] designs meet reality.”
The issues then, continue to be the issues today: “[t]he issue of ‘pupil performance in teacher evaluation’ needs to be discussed in a way that has not been separated from the many other pressures that shape a personnel system of assessment and rewards. The issue cannot be judged apart from how it exists in real places. The validity of such a process of evaluating teachers, its economy, its power to engender belief and win acceptance, depend on how it fits with many other realities of the teaching profession. What now must be addressed is how this notion gets applied in a context as complicated as education. How is the idea transformed as it moves from the statistician’s mind to the real world? How does it fare when the tradeoffs and balances are struck? Is the concept of evaluating teachers by student progress [to be] trusted?”
Glass then explores these questions using a case-based analysis of a number of district sites, concluding what we would still conclude and predict today, even with current and “more contemporary” approaches to the same policy “initiatives.” Glass concludes, that using student achievement data to evaluate teachers…
- …will nearly always be undertaken at the level of a school (either all or none of the teachers in a school are rewarded equally) rather than at the level of individual teachers since (a) no authoritative tests exist in most areas of the secondary school curriculum, nor for most special roles played by elementary teachers; and (b) teachers reject the notion that they should compete with their colleagues for raises, privileges and perquisites;
- …will always be combined with other criteria (such as absenteeism or extra work) which [continue to] prove to be the real discriminators between who is ultimately rewarded and who is not;
- …will always be too susceptible to intentional distortion and manipulation to engender any confidence in the data; moreover teachers and others who believe that no type of test nor any manner of statistical analysis can equate the difficulty of the teacher’s task in the wide variety of circumstances in which they work will further undermine the system;
- …will elevate tests themselves to the level of curriculum goals, obscuring the distinction between learning and performing on tests;
- …will often make room for symbolic administrative actions simply undertaken to reassure the lay public that student learning is valued and assiduously sought after…
Sound (insanely) familiar?!?
