No surprise, really, but Thomas Kane, an economics professor from Harvard University who also directed the $45 million worth of Measures of Effective Teaching (MET) studies for the Bill & Melinda Gates Foundation, is publicly writing in support of VAMs, again. His newest article was recently published on the website of the Brookings Institution, titled “Do Value-Added Estimates Identify Causal Effects of Teachers and Schools?” Not surprisingly as a VAM advocate, in this piece he continues to advance a series of false claims about the wonderful potentials of VAMs, in this case as also yielding causal estimates whereas teachers can be seen as directly causing the growth measured by VAMs (see prior posts about Kane’s very public perspectives here and here).
The best part of this article is where Kane (potentially) seriously considers whether “the short list of control variables captured in educational data systems—prior achievement, student demographics, English language learner [ELL] status, eligibility for federally subsidized meals or programs for gifted and special education students—include the relevant factors by which students are sorted to teachers and schools” and, hence, work to control for bias as these are some of the factors that do (as per the research evidence) distort value-added scores.
The potential surrounding the exploration of this argument, however, quickly turns south as thereafter Kane pontificates using pure logic that “it is possible that school data systems contain the very data that teachers or principals are using to assign students to teachers.” In other words, he painfully attempts to assert his non-research-based argument that as long as principals and teachers use the aforementioned variables to sort students into classrooms, then controlling for said variables should indeed control for the biasing effects caused by the non-random assortment of students into classrooms (and teachers into classrooms, although he does not address that component either).
In addition, he asserts that, “[o]f course, there are many other unmeasured factors [or variables] influencing student achievement—such as student motivation or parental engagement [that cannot or cannot easily be observed]. But as long as those factors are also invisible [emphasis added] to those making teacher and program assignment decisions, our [i.e., VAM statisticians’] inability to control for them” more or less makes not controlling for these other variables inconsequential. In other words, in this article Kane asserts as long as the “other things” principals and teachers use to non-randomly place students into classrooms are “invisible” to the principals and teachers making student placement decisions, these “other things” should not have to be statistically controlled, or factored out. We should otherwise be good to go given the aforementioned variables already observable and available.
As evidenced in a study I wrote with one of my current doctoral students that was recently published in the esteemed, peer-reviewed American Educational Research Journal on this very topic (see the full study here), we set out to better determine how and whether the controls used by value-added researchers to eliminate bias might be sufficient given what indeed occurs in practice when students are placed into classrooms.
We found that both teachers and parents play a prodigious role in the student placement process, in almost nine out of ten schools (i.e., 90% of the time). Teachers and parents (although parents are also not mentioned in Kane’s article) provide both appreciated and sometimes unwelcome insights, regarding what teachers and parents perceive to be the best learning environments for their students or children, respectively. Their added insights typically revolve around, in the following order, students’ in-school behaviors, attitudes, and disciplinary records; students’ learning styles and students’ learning styles as matched with teachers’ teaching styles; students’ personalities and students personalities as matched with teachers’ personalities; students’ interactions with their peers and prior teachers; general teacher types (e.g. teachers who manage their classrooms in perceptibly better ways); and whether students had siblings in potential teachers’ classrooms prior.
These “other things” are not typically if ever controlled for given current VAMs, nor will they likely ever be. In addition, these factors serve as legitimate reasons for class changes during the school year, although whether this, too, is or could be captured in VAMs is highly tentative at best. Otherwise, namely prior academic achievement, special education needs, giftedness, and gender also influence placement decisions. These are variables for which most current VAMs account or control, presumably effectively.
Kane, like other VAM statisticians, tend to (and in many ways have to if they are to continue with their VAM work, despite “the issues”) (over)simplify the serious complexities that come about when random assignment of students to classrooms (and teachers to classrooms) is neither feasible, nor realistic, or outright opposed (as was also clearly evidenced in the above article by 98% of educators, see again here).
The random assignment of students to classrooms (and teachers to classrooms) very rarely happens. Rather, the use of many observable and unobservable variables are used to make such classroom placement decisions, and these variables go well beyond whether students are eligible for free-and-reduced lunches or are English-language learners.
If only the real world surrounding our schools, and in particular the measurement and evaluation of our schools and teachers within them, was so simple and straightforward as Kane and others continue to assume and argue, although much of the time without evidence other than his own or that of his colleagues at Harvard (i.e., 8/17; 47% of the articles cited in Kane’s piece). See also a recent post about this here. In this case, much published research evidence exists to clearly counter this logic and the many related claims herein (see also the other research not cited in this piece but cited in the study highlighted above and linked to again here).
As a retired principal, I can attest to the fact that classroom placements are not at all random, and that the “controlled for” variables are not much of a part of it. The primary factors in classroom placements are many other things, including primarily student behavioral issues, interactions among students (so that difficult groups of students are separated, for example, and positive relationships fostered), student-teacher matches, and budget limitations that, for example, require placement of groups of students with similar needs (for example, ELL students) together in order to provide them services. Other factors include such things as placing all the clarinet or trumpet students together to make district staff schedules work, placing students with ADD/ADHD in classrooms that have P.E. in the morning, and the like. And, of course, most important, placing challenging students with the teachers who can best handle them, and giving new or less experienced teachers easier classes (and/or easier parents).