In a 2013 study titled “Re-testing PISA Students One Year Later: On School Value Added Estimation Using OECD-PISA” (Organisation for Economic Co‑operation and Development-Programme for International Student Assessment), researchers Bratti and Chechi explored a unique PISA international test score data set in Italy.
‘[I]n two regions of North Italy (Valle d’Aosta and the autonomous province of Trento) the PISA 2009 test was re-administered to the same students one year later.” Hence, authors had the unique opportunity to analyze what happens with school-level value-added when the same students were retested for two adjacent years, using a very strong standardized achievement test (i.e., the PISA).
Researchers found that “cross-sectional measures of school value added based on PISA…tend to be very volatile over time whenever there is a high year-to-year attrition in the student population.” In addition, some of this volatility can be mitigated when longitudinal measures of school value added take into account students’ prior test scores; however, higher consistency (less volatility) tends to be more evident in schools in which there is little attrition/transition. Inversely, lower consistency (higher volatility) tends to be more evident in schools in which there is much attrition/transition.
Researchers observed correlations “as high as 0.92 in Trento and is close to zero in Valle d’Aosta” when the VAM was not used to control for past test scores. When a more sophisticated VAM was used (accounting for students’ prior performance, and school fixed effects), however, researchers found that the ” coefficient [was] much higher for Valle d’Aosta than for Trento.” So, the correlations flip-flopped based on model specifications, the more advanced specs yielding “the better” or “more accurate” value-added output.
Researchers attribute this to panel attrition in that “in Trento only 8% of the students who were originally tested in 2009 dropped out or changed school in 2010, [but] the percentage [rose] to about 21% in Valle d’Aosta” at the same time.
Likewise, “[i]n educational settings characterized by high student attrition, this will lead to very volatile measures of VA.” Inversely, “in settings characterized by low student attrition (drop-out or school changes), longitudinal and cross-sectional measures of school VA turn out to be very correlated.”