Research Study: Missing Data and VAM-Based Bias

A new Assistant Professor here at ASU, from outside the College of Education but in the College of Mathematical and Natural Sciences also specializes in value-added modeling (and statistics). Her name is Jennifer Broatch, she is a rising star in this area of research, and she just sent me an article I missed, just read, and certainly found worth sharing with you all.

The peer-reviewed article, published in Statistics and Public Policy this past November, is fully cited and linked below so that you all can read it in full. But in terms of its CliffsNotes version, researchers evidenced the following two key findings:

First, researchers found that, “VAMs that include shorter test score histories perform fairly well compared to those with longer score histories.” The current thinking is that we need at least two if not three years of data to yield reliable estimates, or estimates that are consistent over time (which they should be). These authors argue that with three years of data the amount of data that go missing are not worth shooting for that target. Rather, again they argue, this is an issue of trade-offs. This is certainly something to consider, as long as we continue to understand that all of this is about “tinkering towards a utopia” (Tyack & Cuban, 1997) that I’m not at all certain exists in terms of VAMs and VAM-based accuracy.

Second, researchers found that, “the decision about whether to control for student covariates [or background/demographic variables] and schooling environments, and how to control for this information, influences [emphasis added] which types of schools and teachers are identified as top and bottom performers. Models that are less aggressive in controlling for student characteristics and schooling environments systematically identify schools and teachers that serve more advantaged students as providing the most value-added, and correspondingly, schools and teachers that serve more disadvantaged students as providing the least.”

This certainly adds evidence to the research on VAM-based bias. While there are many researchers who still claim that controlling for student background variables is unnecessary when using VAMs, and if anything bad practice because controlling for such demographics causes perverse effects (e.g., if teachers focus relatively less on such students who are given such statistical accommodations or boosts), this study adds more evidence that “to not control” for such demographics does indeed yield biased estimates. The authors do not disclose, however, how much bias is still “left over” after the controls are used; hence, this is still a very serious point of contention. Whether the controls, even if used, function appropriately is still something to be taken in earnest, particularly when consequential decisions are to be tied to VAM-based output (see also “The Random Assignment of Students into Elementary Classrooms: Implications for Value-Added Analyses and Interpretations”).

Citation: Ehlert, M., Koedel, C., Parsons, E., & Podgursky, M. (2013, November). The sensitivity of value-added estimates to specification adjustments: Evidence from school- and teacher-level models in Missouri. Statistics and Public Policy, 1(1), 19-27. doi: 10.1080/2330443X.2013.856152

1 thought on “Research Study: Missing Data and VAM-Based Bias

  1. The Am Stat Assoc has made this journal accessible to everyone. It is on http://amstat.tandfonline.com/toc/uspp20/current#.UxtLyIXwpyR. In this issue there is another VAM paper. It is by Dan Goldhaber and colleagues and it also compares different models for estimating VAMs. The findings are (in an over-simplified half-sentence) the estimates are correlated, but that the classifications are different for many teachers.

    Another freely accessible paper (until Jan 2015) is Harvey Goldstein on school rankings using VAMs. It is http://amstat.tandfonline.com/toc/uspp20/current#.UxtLyIXwpyR. In the UK more emphasis is placed on using VAMs for school rankings. One of the problems, even addressed by a House of Commons, is the uncertainty in the ratings. Prof. Goldstein’s concerns in using these models are particularly noteworthy as he is one of the most important people (I might argue the most important) for developing these types of models within education. He has done a lot of work on how to convey the uncertainty and on technical aspects of the estimation. Other of his papers (only some of which are on VAMs, but the phrases league tables and school effectiveness are more common in Europe so these also mean the paper is on VAMs).

Leave a Reply

Your email address will not be published. Required fields are marked *