Everything You Want to Know about VAM Research

A few weeks ago Diane Ravitch requested that I construct a list of the Top 10 and Top 25 research articles about VAMs, largely to inform others needing research to become more informed about (and defend themselves against) VAMs.

In turn, I constructed two lists in response: the Top 13 and Top 25. A few days ago, she posted these lists in a blog post titled “Everything You Want to Know about VAM Research,” so I thought I should send these out here, as well, for those who might have missed them.

You can find the Top 13 articles here, and the Top 25 here.

I also have two other running lists that currently include one list that includes the 17 articles published in the top journals sponsored by the American Educational Research Association (AERA), and another list that includes, thus far, 69 research articles that I’ve simply termed as all “good” articles about VAMs.

Hope this helps!!

Another Lawsuit in Rochester, New York

Following my most recent post, it seems, again, and as per another post in Diane Ravitch’s blog, that the Rochester, New York Teachers Association is also now “suing the state over its teacher evaluation system, alleging that it does not take into account the impact of poverty on classroom performance.”

As per the original post, “The Rochester Teachers Association today filed a lawsuit alleging that the Regents and State Education Department failed to adequately account for the effects of severe poverty and, as a result, unfairly penalized Rochester teachers on their APPR (Annual Professional Performance Review) evaluations.

The suit, filed in state Supreme Court in Albany by New York State United Teachers on behalf of the RTA and more than 100 Rochester teachers, argues the State Education Department did not adequately account for student poverty in setting student growth scores on state tests in grades 4-8 math and English language arts. In addition, SED imposed rules for Student Learning Objectives and implemented evaluations in a way that made it more difficult for teachers of economically disadvantaged students to achieve a score of “effective” or better. As a result, the lawsuit alleges the Regents and SED violated teachers’ rights to fair evaluations and equal protection under the law.

SED computes a growth score based on student performance on state standardized tests, which is then used in teacher evaluations.

Nearly 90 percent of Rochester students live in poverty. The lawsuit says SED’s failure to appropriately compensate for student poverty when calculating student growth scores resulted in about one-third of Rochester’s teachers receiving overall ratings of “developing” or “ineffective” in 2012-13, even though 98 percent were rated “highly effective” or “effective” by their principals on the 60 points tied to their instructional classroom practices. Statewide, just 5 percent of teachers received “developing” or “ineffective” ratings.

“The State Education Department’s failure to properly factor in the devastating impact of Rochester’s poverty in setting growth scores and providing guidance for developing SLOs resulted in city teachers being unfairly rated in their evaluations,” Iannuzzi said. “Rochester teachers work with some of the most disadvantaged students in the state. They should not face stigmatizing labels based on discredited tests and the state’s inability to adequately account for the impact of extreme poverty when measuring growth.”

RTA President Adam Urbanski said an analysis of Rochester teachers’ evaluations for 2012-13 demonstrated clearly the effects of poverty and student attendance, for example, were not properly factored in for teachers’ evaluations. As a result, “dedicated and effective teachers received unfair ratings based on student outcomes that were beyond their control. The way the State Education Department implemented the state testing portion of APPR adds up to nothing more than junk science.”

Urbanski stressed Rochester teachers embrace accountability, support objective and constructive evaluations, and accept that testing has a place in education. “Tests should be used to inform instruction and can be effective tools to help improve teaching and learning,” he said. “But SED’s obsession with standardized testing and data collection has perverted the goals of testing while, ironically, failing to accurately measure the one impact that matters most: the effects of poverty on student achievement.”

Another Lawsuit in Tennessee

As per Diane Ravitch’s blog, “The Tennessee Education Association filed a second lawsuit against the use if value-added assessment (called TVAAS in Tennessee), this time including extremist Governor Haslam and ex-TFA state commissioner Huffman in their suit.

As per a more detailed post about this lawsuit, “The state’s largest association for teachers filed a second lawsuit on behalf of a Knox County teacher, calling the use of the Tennessee Value-Added Assessment System (TVAAS), which uses students’ growth on state assessments to evaluate teachers, unconstitutional.

Farragut Middle School eighth grade science teacher Mark Taylor believes he was unfairly denied a bonus after his value-added estimate was based on the standardized test scores of 22 of his 142 students. “Mr. Taylor teaches four upper-level physical science courses and one regular eighth grade science class,” said Richard Colbert, TEA general counsel, in a press release. “The students in the upper-level course take a locally developed end-of-course test in place of the state’s TCAP assessment. As a result, those high-performing students were not included in Mr. Taylor’s TVAAS estimate.”

Taylor received ‘exceeding expectations’ classroom observation scores, but a low value-added estimate reduced his final evaluation score below the requirement to receive the bonus.

The lawsuit includes six counts against the governor, commissioner and local school board.

TEA’s general counsel argues the state has violated Taylor’s 14th Amendment right to equal protection from “irrational state-imposed classifications” by using a small fraction of his students to determine his overall effectiveness.

TEA filed its first TVAAS lawsuit last month on behalf of Knox County teacher Lisa Trout, who was denied the district’s bonus. The lawsuit also cites the arbitrariness of TVAAS estimates that use test results of only a small segment of a teacher’s students to estimate her overall effectiveness.

TEA says it expects additional lawsuits to be filed so long as the state continues to tie more high-stakes decisions to TVAAS estimates.”

UC Berkeley’s Jesse Rothstein at Vergara v. California

To read a decent and fair summary of the Vergara v. California trial, please see this piece posted by The Pew Charitable Trust. Also covered on VAMboozled! was a summary, but also the summarized testimonies of Linda Darling-Hammond and David Berliner.

It seems Jesse Rothstein, an Associate Professor of Economics at the University of California Berkeley, also testified for the defense. His research has been critical in terms of the evidence about bias in VAM estimates, and whether the sophisticated controls are indeed sophisticated enough to control for the nonrandom assignment of students to classrooms.

As per Rothstein’s testimony (again summarized by Pew) he cautioned “against placing too much weight on the “value-added models” now being used by many states to try to determine the impact of a teacher on student learning. The models try to tease out how much an individual teacher has contributed to a student’s academic progress.

Rothstein’s research has found that such models can yield very different findings for the same teacher from one year to the next, in part because 25 students or so are not a large enough sample size to create a reliable estimate of a teacher’s teaching ability. Value-added models also assume that students are assigned to teachers randomly, which very often is not the case.

“I think we want to think hard about how to use them,” Rothstein said of value-added models. “What I’m seeing is there are a lot of places that got way out ahead of the evidence and passed policies saying we should fire teachers based on value-added. It’s not as easy as just saying ‘just do this.’”

Rothstein also testified about possible unintended consequences of changes to teacher tenure and dismissal rules, saying it could be harder for school districts to attract and retain good teachers and encourage teaching to the test, for example.”

One of the First (of X) Lawsuits to Come?

From another blog post titled, “TEA files lawsuit on behalf of Knox County teacher:

“Tennessee Education Association has filed a lawsuit on the behalf of a Knox County teacher who believes she was unfairly denied a bonus as a result of her value-added test scores. The statewide organization expects this TVAAS (Tennessee Value-Added Assessment System) lawsuit to be the first of many as more districts tie high-staked decisions to  students’ achievement and growth scores on standardized tests.

Knox County Lisa Trout was denied a bonus after her value-added score was calculated. “After being told she would receive the system-wide TVAAS estimate because of her position in an alternative school, a guidance counselor incorrectly claimed 10 of Ms. Trout’s students for her TVAAS score without her knowledge,” said TEA general counsel Richard Colbert in a press release.  ”As a result, Ms. Trout ultimately received a lower TVAAS estimate than she should have and was denied the APEX bonus she had earned.”

TEA’s lawsuit also contests the arbitrariness TVAAS estimates that use test results of only a small segment of a teacher’s students to estimate her overall effectiveness.

“Ms. Trout’s situation illustrates the fundamental problem with using statistical estimates for high-stakes decisions that affect teacher pay,” Colbert said. “Her case raises great concerns over the constitutionality of such practices.”