The Elephant in the Room – Fairness

ShareTweet about this on TwitterShare on Facebook1Email this to someoneShare on Google+0Share on LinkedIn0Share on Reddit0

While VAMs have many issues pertaining, fundamentally, to their levels of reliability, validity, and bias, they are wholeheartedly unfair. This is one thing that is so very important but so rarely discussed when those external to VAM-based metrics and metrics use are debating, mainly the benefits of VAMs.

Issues of “fairness” arise when a test, or more likely its summative (i.e., summary and sometimes consequential) and formative (i.e., informative) uses, impact some more than others in unfair yet often important ways. In terms of VAMs, the main issue here is that VAM-based estimates can be produced for only approximately 30-40% of all teachers across America’s public schools. The other 60-70%, which sometimes includes entire campuses of teachers (e.g., early elementary and high school teachers), cannot altogether be evaluated or “held accountable” using teacher- or individual-level VAM data.

Put differently, what VAM-based data provide, in general, “are incredibly imprecise and inconsistent measures of supposed teacher effectiveness for only a tiny handful [30-40%] of teachers in a given school” (see reference here). But this is often entirely overlooked, not only in the debates surrounding VAM use (and abuse) but also in the discussions surrounding how many taxpayer-derived funds are still being used to support such a (purportedly) reformatory overhaul of America’s public education system. The fact of the matter is that VAMs only directly impact the large minority.

While some states and districts are rushing into adopting “multiple measures” to alleviate at least some of these issues with fairness, what state and district leaders don’t entirely understand is that this, too, is grossly misguided. Should any of these states and districts also tie serious consequences to such output (e.g., merit pay, performance plans, teacher termination, denial of tenure), or rather tie serious consequences to measures of growth derived via any varieties of the “multiple assessment” that can be pulled from increasingly prevalent multiple assessment “menus,” states and districts are also setting themselves for lawsuits…no joke! Starting with the basic psychometrics, and moving onto the (entire) lack of research in support of using more “off-the-shelf” tests to help alleviate issues with fairness, would be the (easy) approach to take in a court of law as, really, doing any of this is entirely wrong.

School-level value-added is also being used to accommodate the issue of “fairness,” just less frequently now than before given the aforementioned “multiple assessment” trends. Regardless, many states and districts also continue to attribute a school-level aggregate score to teachers who do not teach primarily reading/language arts and mathematics, primarily in grades 3-8. That’s right, a majority of teachers receive a value-added score that is based on students whom they do not teach. This also calls for legal recourse, also in that this has been a contested issue within all of the lawsuits in which I’ve thus far been engaged.

ShareTweet about this on TwitterShare on Facebook1Email this to someoneShare on Google+0Share on LinkedIn0Share on Reddit0

3 thoughts on “The Elephant in the Room – Fairness

  1. Student learning Objectives (SLOs,) so-called distributed scores, and nationally normed end-of course tests were pushed as alternatives to VAM by the “Reform Support Network” created for Race to the Top initiatives.

    The American Institutes of Research still pushes Student learning Objectives– SLOs. SLOs have been marketed since 1999 by William J. Slotnick founder of the Community Training and Assistance Center (CTAC).

    All of these measures are based on a business concept of growth as an increase in a metric, not human growth, not even growth in learning—unless you believe scores on standardized tests are some sort of gold standard for learning—every kind, everywhere, for every kid. The nonsense continues.
    All three of these measures should be put in the trash bin. Thanks for exposing the fraud in all of these measures.

    Here is an up dated bibliography of “research” on SLOs. Nothing here warrants SLO use for every subject and grade. Nothing warrants SLO use for any purpose other than stack rating teachers and using that to justify pay for performance. http://www.air.org/resource/what-we-know-about-slos-annotated-bibliography-research-and-evaluations-student-learning

  2. All of these measures are based on a business concept of growth as an increase in a metric, not human growth, not even growth in learning—unless you believe scores on standardized tests are some sort of gold standard for learning—every kind, everywhere, for every kid.

  3. On fairness, the following (https://www.researchgate.net/publication/313837822_Ancova_and_Gain_Score_Models_Relevance_for_Baseball_and_Education) shows that ancova based methods (like VAMs) are biased against teachers and schools that serve primarily groups of students from historically low performing groups IF the covariates (usual prior scores) do not influence the school/teacher to whom the student is assigned. The paper is revised and resubmitted at Am Stat.

Leave a Reply

Your email address will not be published. Required fields are marked *