A Florida Media Arts Teacher on Her “VAM” Score

A Media Arts Teacher from the state of Florida wrote a piece for The Washington Post – The Answer Sheet by Valerie Strauss about her VAM score recently publicly released, even though she is a media arts teacher and does not teach the subject areas and many of the students tested and whose test scores are being used to hold her accountable.

Bizarre, right? Not really, as this too is a reality facing many teachers who teach out-of-subject areas, or more specifically subject areas that “don’t count,” and who teach students sometimes a lot yet sometimes never. They are being assigned “school-level” VAM scores, and these estimates regardless of their actual contributions are being used to make consequential decisions (e.g., in this case, about her merit pay).

She writes about “What it feels like to be evaluated on test scores of students I don’t have,” noting, more specifically, about what others “need to know about [her] VAM score.” For one, she writes, “As a media specialist, [her] VAM is determined by the reading scores of all the students in [her] school, whether or not [she] teach[es] them. [Her] support of the other academic areas is not reflected in this number.” Secondly, she writes, “Like most teachers, [she has] no idea what [her] score means. [She] know[s] that [her] VAM is related to school-wide reading scores but [she] do[es]n’t understand how it’s calculated or exactly what data is [sic] used. This number does not give [her] feedback about what [she] did for [her] students to support their academic achievement last year or how to improve [her] instruction going forward.” She also writes about issues with her school being evaluated differently from the state system given they are involved in a Gates Foundation grant, and she writes about her concerns about the lack of consistency in teacher-level scores over time, as based on her knowledge of the research. See the full article linked again here to read more.

Otherwise, she concludes with what a very real question, also being faced by many. She writes, “[W]hy do I even care about my VAM score? Because it counts. My VAM score is a factor in determining if I am eligible for a merit pay bonus, whether I have a job in the next few years, and how many times I’ll be evaluated this year.” Of course, she cares as she and many others are being forced to care about their professional livelihoods out from under a system that is out of her control, out of thousands of teachers’ control, and in so many ways just simply out of control.

See also what she has to offer in terms of what she frames as a much better evaluation system, that would really take into account her effectiveness and the numbers that are certainly much more indicative of her as a teacher. These are the numbers, if we continue to fixate on the quantification of effectiveness, that in all actuality should count.



On a More Positive Note…

Contrary to what is going on in South Carolina and its House Bill 4419 (thanks in large part to Rhee and her “supportive” efforts, as posted also today here), in the state of Washington its State Senate just voted “No” against evaluating teachers in the state using student test scores.

Interesting that this happened in Bill Gates’s homestead state, given his current and widespread educational “reform” initiatives in support of the opposite, but I digress.

As per at least one of the articles capturing this story, Washington State Senate Bill 5246 failed by a 28-19 vote, even though Washington (like many other states) has a federal No Child Left Behind (NCLB) waiver requiring the state to mandate the use of such tests for such evaluation systems. The state could now lose an undisclosed (and likely unknown) amount of federal funding, by not complying with the aforementioned condition of the NCLB waiver.

One of the State Senators said that she voted the bill down because “using state tests to measure student growth has not been proven to be an effective way to judge teachers.”

It’s about time!!! Silver linings, perhaps, but something certainly to celebrate!

But not for another State Senator who voted for it, particularly noting that “Losing the waiver would mean nearly every school in the state would have to send a letter home to parents saying they are failing to meet the requirements of the federal education law.”

Is this not something else to celebrate? Depends on your stance as a leader, I guess.

Follow-Up On Kane’s Testimony

As a follow-up to our most recent post, for those of you interested in reading more about Kane’s testimony in the Vergara v. California case, please see the following post written by John Thompson in another blog titled “Does the Gates Foundation’s Evidence Argue For or Against Vergara?

As the title implies, Thompson connects the dots between the Gates Foundation and the others who are also part of the “Billionaires’ Boys Club” who are supporting (although not directly financing) this case. Thompson also provides a much more thorough and detailed analysis of Kane’s testimony. Do give it a read.

As an aside, I also just happened across a press release announcing then that one of the case’s current star witnesses, who was actually the first called to the stand by the plaintiffs – Los Angeles Superintendent John Deasy – also starting in 2009 served as the Gates Foundation’s Deputy Director of Education. While it does not seem that Deasy is directly affiliated with the Gates Foundation anymore, perhaps this connection had something to do with the choice location of this lawsuit in Los Angeles Unified. In all fairness, however, Deasy is “providing fodder” for both sides, including testimony for the defense about the district’s current labor laws and how firing “ineffective” teachers really “comes down to the choices and competence of management, not the constitutionality of current regulations.”

Chicken or the Egg?

“Which came first, the chicken or the egg?” is an age-old question, but is more importantly a dilemma about identifying the real cases of cause and consequence.

Recall from a recent post that currently in the state of California nine public school students are challenging California’s teacher tenure system, arguing that their right to a good education is being violated by job protections that protect ineffective teachers, but do not protect the students from being instructed by said teachers. Recall, as well, that a wealthy technology magnate [David Welch] is financing the whole case, as also affiliated and backed by Students Matter. The ongoing suit is called “Vergara v. California.”

Welch and Students Matter have thus far brought to testify an all-star cast, most recently including Thomas Kane, an economics professor from Harvard University. Kane also directed the $45 million worth of Measures of Effective Teaching (MET) studies for the Bill & Melinda Gates Foundation, not surprisingly as a VAM advocate, advancing a series of highly false claims about the wonderful potentials of VAMs. Potentials that, once again, did not pass any sort of peer review, but that still made it to the US Congress. To read about the many methodological and other problems with the MET studies click here.

If I was to make a list of VAMboozlers, Kane would be near the top of the list, especially as he is increasingly using his Harvard affiliation to advance his own (profitable) credibility in this area. To read an insightful post about just this, read VAMboozled! reader Laura Chapman’s comment at the bottom of a recent post here, in which she wrote, “Harvard is only one of a dozen high profile institutions that has become the source of propaganda about K-12 education and teacher performance as measured by scores on standardized tests.”

Anyhow, and as per a recent article in the Los Angeles Times, Kane testified that “Black and Latino students are more likely to get ineffective teachers in Los Angeles schools than white and Asian students,” and that “the worst teachers–in the bottom 5%–taught 3.2% of white students and 5.4% of Latino students. If ineffective teachers were evenly distributed, you’d expect that 5% of each group of students would have these low-rated instructors.” He concluded that “The teaching-quality imbalance especially hurts the neediest students because ‘rather than assign them more effective teachers to help close the gap with white students they’re assigned less effective teachers, which results in the gap being slightly wider in the following year.”

Kane’s research was, of course, used to support the claim that bad teachers are causing the disparities that he cited, regardless of the fact the inverse could be also, equally, or even more true–that the value-added measures used to measure teacher effectiveness in these schools are biased by the very nature of the students in these schools that are contributing their low test scores to such estimates. As increasingly being demonstrated in the literature, these models are biased by the types of students in the classrooms and schools that contribute to the measures themselves.

So which one came first? The chicken or the egg? The question here, really, and that I wish defendants would have posed, was whether the students in these schools caused such teachers to appear less effective when in fact they might have been as equally effective as “similar” teachers teaching more advantaged kids across town. What we do know from the research literature is that, indeed, there are higher turnover rates in such schools, and oftentimes such schools become “dumping grounds” for teachers who cannot be terminated due to such tenure laws – this is certainly a problem. But to claim that teachers in such schools are causing poor achievement is certainly cause for concern, not to mention a professional and research-based ethics concern as well.

Kane’s “other notable finding was that the worst teachers in Los Angeles are doing more harm to students than the worst ones in other school systems that he compared. The other districts were New York City, Charlotte-Mecklenberg, Dallas, Denver, Memphis and Hillsborough County in Florida.” Don’t ask me how he figured that one out, across states that use different tests, different systems, and have in their schools entirely different and unique populations. Amazing what some economists can accomplish with advanced mathematical models…and just a few (heroic) assumptions.

The Gates Foundation and its “Strong Arm” Tactics

Following up on VAMboozled!’s most recent post, about the Bill & Melinda Gates Foundation’s $45 million worth of bogus Measures of Effective Teaching (MET) studies that were recently honored with a 2013 Bunkum (i.e., meaningless, irrelevant, junk) Award by the National Education Policy Center (NEPC), it seems that the Bill & Melinda Gates foundation are, once-again, “strong-arming states [and in this case a large city district] into adoption of policies tying teacher evaluation to measures of students’ growth.”

According to Nonprofit Quarterly, the Gates Foundation is now threatening to pull an unprecedented $40 million grant from Pittsburgh’s Public Schools “because the foundation is upset with the lack of an agreement between the school district and the teachers’ union over a core element of the grant” — the use of test scores to measure teachers’ value-added and to “reward exceptional teachers and retrain those who don’t make the grade.”

More specifically, the district and its teachers are not coming to an agreement about how they should be evaluated, rightfully because teachers understand better than most (even some VAM researchers) that these models are grossly imperfect, largely biased by the types of students non-randomly assigned to their classrooms and schools, highly unstable (i.e., grossly fluctuating from one year to the next when they should remain more or less consistent over time, if reliable), invalid (i.e., they do not have face validity in that they often contradict other valid measures of teacher effectiveness), and the like.

It seems, also, that Randi Weingarten, having recently taken a position against VAMs (as posted in VAMboozled! here and here), has also “added value,” at least in terms of the extent to which teachers in Pittsburgh are (rightfully) exercising some more authority and power over the ways in which they are to be (rightfully) evaluated. Unfortunately, however, money talks, and $40 million of it is a lot to give up for a publicly funded district like this one in Pittsburgh.

The 2013 Bunkum Awards & the Gates Foundation’s $45 MET Studies

Tis the award season, and during this time every year, the National Education Policy Center (NEPC) recognizing the “lowlights” in educational research over the previous year, in their annual Bunkum Awards. To view the entertaining video presentation of the awards, hosted by my mentor David Berliner (Arizona State University), please click here.

Lowlights, specifically defined, include research studies in which researchers present, and often oversell thanks to many media outlets, “weak data, shoddy analyses, and overblown recommendations.”  Like the Razzies are to the Oscars in the Academy of Film, are the Bunkums to the best educational research studies in the Academy of Education. And like the Razzies, “As long as the bunk [like junk] keeps flowing, the awards will keep coming.”

As per David Berliner, in his introduction in the video, “the taxpayers who finance public education deserve smart [educational] policies based on sound [research-based] evidence.” This is precisely why these awards are both necessary, and morally imperative.

One among this year’s deserving honorees is of particular pertinence here. This is the, drum roll: ‘We’re Pretty Sure We Could Have Done More with $45 Million’ Award — Awarded to the Bill & Melinda Gates Foundation for Two Culminating Reports they released this year from their Measures of Effective (MET) Project. To see David’s presentation on this award, specifically, scroll to minute 3:15 (to 4:30) in the aforementioned video.

Those at NEPC write about these studies: “We think it important to recognize whenever so little is produced at such great cost. The MET researchers gathered a huge data base reporting on thousands of teachers in six cities. Part of the study’s purpose was to address teacher evaluation methods using randomly assigned students. Unfortunately, the students did not remain randomly assigned and some teachers and students did not even participate. This had deleterious effects on the study–limitations that somehow got overlooked in the infinite retelling and exaggeration of the findings.

When the MET researchers studied the separate and combined effects of teacher observations, value-added test scores, and student surveys, they found correlations so weak that no common attribute or characteristic of teacher-quality could be found. Even with 45 million dollars and a crackerjack team of researchers, they could not define an “effective teacher.” In fact, none of the three types of performance measures captured much of the variation in teachers’ impacts on conceptually demanding tests. But that didn’t stop the Gates folks, in a reprise from their 2011 Bunkum-winning ways, from announcing that they’d found a way to measure effective teaching, nor did it deter the federal government from strong-arming states into adoption of policies tying teacher evaluation to measures of students’ growth.”

To read the full critique of both of these studies, written by Jesse Rothstein (University of California – Berkeley) and William Mathis (University of Colorado – Boulder), please click here.