Monthly Archives: November 2014

Surveys + Observations for Measuring Value-Added

Posted on by
2

Following up on a recent post about the promise of Using Student Surveys to Evaluate Teachers using a more holistic definition of a teacher’s valued added, I just read a chapter written by Ronald Ferguson — the creator of the Tripod student survey instrument and Tripod’s lead researcher — and written along with Charlotte Danielson — the creator of the Framework for Teaching and founder of The Danielson Group (see a prior post about this instrument here). Both instruments are “research-based,” both are used nationally and internationally, both are (increasingly being) used as key indicators to evaluate teachers across the U.S., and both were used throughout the Bill & Melinda Gates Foundation’s ($43 million worth of) Measures of Effective Teaching (MET) studies.

The chapter titled, “How Framework for Teaching and Tripod 7Cs Evidence Distinguish Key Components of Effective Teaching,” was recently published in a book all about the MET studies, titled “Designing Teacher Evaluation Systems: New Guidance from the Measures of Effective Teaching Project” written by Thomas Kane, Kerri Kerr, and Robert Pianta. The chapter is about whether and how data derived via the Tripod student survey instrument (i.e., as built on 7Cs: challenging students, control of the classroom, teacher caring, teachers confer with students, teachers captivate their students, teachers clarify difficult concepts, teachers consolidate students’ concerns) align with the data derived via Danielson’s Framework for Teaching, to collectively capture teacher effectiveness.

Another purpose for this chapter is to examine how both indicators also align with teacher level-value-added. Ferguson (and Danielson) find that:

  • Their two measures (i.e., the Tripod and the Framework for Teaching) are more reliable (and likely more valid) than value-added measures. The over-time, teacher-level classroom correlations, cited in this chapter, are r = 0.38 for value-added (which is comparable with the correlations noted in plentiful studies elsewhere), r = 0.42 for the Danielson Framework, and r = 0.61 for the Tripod student survey component. These “clear correlations,” while not strong particularly in terms of value-added, do indicate there is some common signal that the indicators are capturing, some stronger than the others (as should be obvious given the above numbers).
  • Contrary to what some (softies) might think, classroom management, not caring (i.e., the extent to which teachers care about their students and what their students learn and achieve), is the strongest predictor of a teachers’ value-added. However, the correlation (i.e., the strongest of the bunch) is still quite “weak” at an approximate r = 0.26, even though it is statistically significant. Caring, rather, is the strongest predictor of whether students are happy in their classrooms with their teachers.
  • In terms of “predicting” teacher-level value-added, and of the aforementioned 7Cs, the things that also matter “most” next to classroom management (although none of the coefficients are as strong as we might expect [i.e., r < 0.26]) include: the extent to which teachers challenge their students and have control over their classrooms.
  • Value-added in general is more highly correlated with teachers at the extremes in terms of their student survey and observational composite indicators.

In the end, while the authors of this chapter do not disclose the actual correlations between their two measures and value-added, specifically (although from the appendix one can infer that the correlation between value-added and Tripod output is around r = 0.45 as based on an unadjusted r-squared), and I should mention this is a HUGE shortcoming of this chapter (one that would not have passed peer review should this chapter have been submitted to a journal for publication), the authors do mention that “the conceptual overlap between the frameworks is substantial and that empirical patterns in the data show similarities.” Unfortunately again, however, they do not quantify the strength of said “similarities.” This only leaves us to assume that since they were not reported the actual strength of the similarities empirically observed between was likely low (as is also evidenced in many other studies, although not as often with student survey indicators as opposed to observational indicators.)

The final conclusion the authors of this chapter make is that educators “cross-walk” the two frameworks (i.e., the Tripod and the Danielson Framework) and use both frameworks when reflecting on teaching. I must say I’m concerned about these recommendations, as well, mainly given this recommendation will cost states and districts more $$$, and the returns or “added value” (using the grandest definition of this term) of doing so and engaging in such an approach does not have the necessary evidence I would say one might use to adequately justify such recommendations.

Interested in Earning a PhD in Ed Policy at ASU?

Posted on by
Reply

Any of you out there interested in earning a PhD in Education Policy at ASU? Especially if you are interested in conducting research on teacher evaluation, accountability, or value-added models (VAMs)?

DOCTOR OF PHILOSOPHY IN EDUCATIONAL POLICY AND EVALUATION

epe

The Doctor of Philosophy in Educational Policy and Evaluation is a full-time program
that prepares you to conduct rigorous, high-impact and interdisciplinary research on
education policy.

In this program, you will:
– Train in the theory, methods and practice of education policy research and evaluation
– Work closely with nationally and internationally recognized faculty members, including award-winning scholars and researchers
– Engage in systematic analyses of education policies in a variety of real-world settings
– Produce policy briefs, reports and scholarly articles
– Deepen your knowledge of education policy based on an area of specialization

The coursework for this full-time program is offered at ASU’s Tempe campus.

Available funding packages include a competitive stipend, tuition remission and health insurance.

CONTACT US

Anita Youmara
anita.youmara@asu.edu
(480) 727-6928

OR CONTACT ME DIRECTLY

Audrey Amrein-Beardsley
audrey.beardsley@asu.edu
(602) 561-4731

Badass Teachers (BATs) Respond to TIME Magazine

Posted on by
Reply

Following up on a recent post about “TIME Magazine Need[ing] a TIME Out,” the leaders of the Bad Ass Teachers Association (BAT) sent the following letter to TIME magazine in response. They also sent the letter to Diane Ravitch (see here), and I’ve also pasted their letter below. It’s certainly worth a read of what they wrote:

As delegates of an organization that represents the collective voices of 53,000 teachers, we take issue with the cover [see below] selected for the November 3 edition of Time. We believe that the image is journalistically irresponsible because it unfairly paints teachers and teacher tenure in a negative light.

teacher-cover

The gavel as a symbol of corporate education, smashing the apple – the universal symbol of education – reinforces a text applauding yet another requested deathblow to teacher tenure. Instead of clarity, this continues the misconception that tenure ensures a job for life. It does not. It ensures “just cause” rationale before teachers can be fired. [Related, see a recent Washington Post article about how an all round “excellent” teacher is suing the state of New York given her lack of due process here].

In addition, the cover perpetuates the pernicious myth of the “bad” teacher and tenure as the prime enablers of larger failures in American education. This is a false narrative. These failures are due to structural inequalities and chronic underfunding in our educational systems, not due to teachers and teacher tenure.

The cover feeds this narrative with the misleading statement, “It is nearly impossible to fire bad teachers.” A few months ago talk show host Whoopi Goldberg made similar statements suffering under the same basic misunderstanding of teacher tenure as something akin to what college professors enjoy rather than a simple guarantee of procedural due process which is its function in K-12 education.

Nevertheless, opponents of teacher tenure have consistently invoked the “bad teacher” argument as pretext to attack not only teachers but also teacher unions, arguing that they place the needs of students second to the protection of underperforming teachers.
In fact, teacher tenure has served as an important protection to allow teachers to advocate for students— especially with regard to maintaining manageable class sizes, safe instructional spaces, the needs of students who are English Language Learners and Students with Disabilities.

Given the massive increase in student enrollments, one of the greatest shortfalls is in the number of teachers themselves. A simple accounting of all the teaching positions lost in the great recessions reveals that the nation would need 377,000 more teachers in the classroom just to keep pace not to mention combat the shameful shortage of teachers of color.

In its haste to disparage teachers, the cover inadvertently tells a larger truth. The instrument used to destroy teacher tenure is wielded against the entire profession. It seeks to obliterate due process for all teachers rather than to ensure its proper use.

More significantly, the cover uncritically situates the tech millionaires as saviors without revealing their own self-interest in the tenure fight, the creation of a nation of corporate-run franchise schools taught by untrained teachers and measured by high stakes test[s] developed and administered by those same millionaires.

In an age where transparency in politics and journalism is sorely needed, we regret Time’s decision to proceed with a cover so clearly at odds with the truth.

The Badass Teachers Association
(Created by BAT Administrators and edited by Marla Kilfoyle, Melissa Tomlinson, Steven Singer, and Dr. Yohuru Williams)

A “Must Watch” Video on VAMs from Albuquerque, New Mexico

Posted on by
Reply

As written into a recent article in the Albuquerque Journal, an Albuquerque, New Mexico Public School Board member publicly, but in many ways appropriately, unleashed her frustration over the use of standardized tests and VAMs in Albuquerque’s public schools.

This is a “must watch” video that can be watched in just over two minutes here.

At one point she said that she is “so goddamn grateful she (her eldest of four children, a high school senior) is leaving the public schools” system at the end of the school year, although she still has three children left in the system.” Hence, her outrage, but also her call to her fellow board members to delay, or rather suspend indefinitely the use of the state of New Mexico’s new Common Core tests and to suspend the use of such test scores in teacher evaluations using VAMs, also suspect as per the board. In the state of New Mexico, test scores are to make up 50% of all test-eligible teacher’s evaluation scores. She also cited the American Statistical Association (ASA) recently released Position Statement on VAMs (discussed here and directly accessible here).

A fellow board member criticized Kort for her language choices that he defined as “poor.” He and a few other members also criticized her for not wanting to continue to work with the state that is pushing VAMs as working and negotiating with the state is in their best interests. Thereafter, “[t]he board passed a resolution calling on itself to draft a resolution addressing its concerns with the PARCC exam and its role in teacher evaluations and school grades. The board is looking to get feedback from parents and teachers before drafting its resolution.”

VAMboozled! One Year Later

Posted on by
2

This VAMboozled blog officially went live just over one year ago two weeks ago. You can read our first post, here, about the “Top 10” pieces of VAMmunition, or rather, ammunition, practitioners need to know to better protect themselves against the unfair implementation and use of VAMs. This one is definitely worth another read if you haven’t read it before as it captures, what I still believe are the same  research-based reasons, today, that VAMs don’t work as intended and as per the research. To read this list, click again here.

Otherwise, I thought I’d share with you all our “progress” report as per our Google analytics. We are now up to ≈ 8,200 subscribers with a loose approximate of about 100,000* hits or blog reads per month.

Hence, thank you all for your support, interest, sharing, action, and the like!! Onward!! And here’s to a productive year in terms of informing more about the research and research-based issues surrounding teacher evaluation, teacher accountability, and value-added models (VAMs) in America’s public schools.

*This number is calculated by ((subscribers x average number of posts per month) + external hits per month)), although external hits may also include subscribers as analytics cannot differentiate between subscribers and hits. These two indicators are not mutually exclusive.

Breaking News: Tennessee’s Huffman and Tulane’s Ayers Resign

Posted on by
1

Two public faces in the world of VAM announced this week that they are resigning from their leadership positions, so it is being claimed, given their high visibility (and perhaps failures) using VAMs.

First, Kevin Huffman, a big national champion of teacher accountability and perpetuator of unsubstantiated nonsense about how teachers need to “work harder” and be held more accountable when they don’t do so, just resigned from his position as the Tennessee Education Commissioner. As per an article in The Tennessean: “Huffman — a former Teach For America executive and ex-husband of controversial former Washington, D.C., Chancellor Michelle Rhee — made Tennessee a flashpoint nationally for debates over teacher evaluations, teacher licensing, Common Core standards and charter schools…In recent months, Huffman lost a handful of political fights. The state overturned a teacher license policy that tied test scores to advancing professionally and delayed a move to Common Core-aligned state testing…More than 50 superintendents had already publicly questioned his leadership, several teachers unions expressed “no confidence” even after [other] accolades; and most recently a group of 15 Republicans last summer called for his resignation.” See previous VAMboozled posts about Huffman’s more-than-controversial past in the state of Tennessee here, here, and here, but more importantly click here for the full article (from which I pulled excerpts above) explaining Huffman’s past and resignation.

Second, John Ayers, the executive director of Tulane University’s Cowen Institute for Education Initiatives, announced that he is to resign at the end of November in an article just posted in the npr_ED blog. One month ago, in what was being called a “high-profile embarrassment,” Ayers on behalf of the institute apologized for releasing a high-profile report based on faulty VAM research and then pulled the institute’s report from its once prevalent placement on its website. The report was celebrated widely in that it “proved” that students in post-Hurricane Katrina’s (largely charter) schools, despite the disadvantages they faced prior to the school-based reforms triggered by Katrina, were “beating the odds and “posting better test scores and graduation rates than predicted by their populations,” thanks to said reforms. Demographics were no longer predicting students’ educational destinies, and the institute had the VAM-based evidence to prove it. Let’s just say the leaders of the charter movement were all over this one, and also allegedly involved. It seems, now, this has at least in part led to the resignation of the institute’s leader, although “Ayers told the New Orleans Times-Picayune that the flawed report and his sudden departure are unrelated.” See a previous VAMboozled post about Ayer and this debacle here, but more importantly click here for the full nprED article in which the article author also writes quite eloquently about how “value-added modeling, it turns out, is really, really hard.”

Both incidents should “serve as a caution both to education researchers, and to those of us [e.g., policymakers, think tank organizations, members of the public] who are making decisions based on [VAM] findings.”

More from Moshe on Chetty, Kane, and their Vergara Testimonies

Posted on by
Reply

Moshe Adler is one of Chetty et al.’s biggest and most important critics, especially as an expert in economics and econometrics himself. Likewise, he has been debunking Chetty et al.’s works from the time Chetty et al.’s studies were first released in 2011 (when the studies were cited by by President Obama) through their most current 2014 versions (when the studies were cited by the judge in Vergara v. California). Examples of some examples of Adler’s criticisms of Chetty et al.’s works can be found here and here.

Adler is also a critic of the work of Thomas Kane, another person of interest (see prior posts about Kane here and here), as Kane is also involved with the Bill & Melinda Gates Foundation, he is a proponent of VAMs as aligned with the Gates Foundation’s interests, and he (like Chetty) served as a key witness for the prosecution in the same Vergara v. California trial. In many ways, Chetty and Kane together delivered the final blows that cost the defense its lost in the Vergara v. California case (as explained here).

That being said, Adler recently presented his work at a conference of the National Organization of Lawyers for Education Associations that was held in Washington DC in October of 2014. More specifically, he presented his review of Chetty’s and Kane’s Vergara v. California testimonies. He also agreed to share his PowerPoint presentation with a brief summary, below. Here’s what he wrote, although you can also click here for his full PowerPoint here.

In the Vergara trial in Los Angeles, the economist Raj Chetty testified that teachers with high teacher value-added scores have a life-long positive impact on the incomes of their students, and the economist Tom Kane testified that minority students in Los-Angeles lose more than a month of schooling each year because they are assigned low value-added teachers.

Chetty’s testimony is based on two articles that he co-authored with economists John Friedman and Jonah Rockoff. Tom Kane’s testimony is based on his work for the Bill and Melinda Gates Foundation.

However, what is highlighted in this PowerPoint presentation is that Chetty’s research, on which he based his testimony, constitutes bad science, and that Kane chose to present his findings in units that present trivial effects as significant [emphasis added].

When Chetty et al.’s articles were accepted for publication by the American Economic Review (AER), Chetty emailed me to inform me of this news (and to suggest that these articles should earn him the Nobel Prize). I immediately wrote to the editors of the AER to alert them of the problems with the articles. Curiously, the editors did not send my concerns to a third party reviewer or even to the referees who originally reviewed the papers for the review. Instead, they sent them to Chetty et al., who published a reply to which I have responded. For more information about all of this, see this aforementioned debate here, and see all of this email correspondence here.

But as for this PowerPoint, the main point is that Chetty et al.’s claim that an increase in teacher value-added leads to higher income throughout adulthood is contradicted by their own findings. Again, you can read more about this here.

Rothstein, Chetty et al., and (Now) Kane on Bias

Posted on by
Reply

Here’s an update to a recent post about research conducted by Berkeley Associate Professor of Economics – Jesse Rothstein.

In Rothstein’s recently released study, he provides evidence that puts the aforementioned Chetty et al. results under a more appropriate light. Rothstein’s charge, again, is that Chetty et al. (perhaps unintentionally) masked evidence of bias in their now infamous VAM-based study, which in turn biased Chetty et al.’s (perpetual) claims that teachers caused effects in student achievement growth over time. These effects, rather, might have been more likely caused by bias given the types of students non-randomly assigned to teachers’ classrooms versus “true teacher effects.”

In addition, while in his study Rothstein replicated Chetty et al.’s overall results using a similar data set, so did Thomas Kane – a colleague of Chetty’s at Harvard who has also been the source of prior VAMboozled! posts here, here, and here. During the Vergara v. California case last summer, the prosecuting team actually used Kane’s (and colleagues’) replication-study results to validate Chetty et al.’s initial results.

However, Rothstein did not replicate Chetty et al.’s findings when it came to bias (the best evidence of this is offered in Rothstein’s study’s Appendix B). Inversely, Kane’s (and colleagues’) study did not, then, have any of the prior year score analyses needed to analyze and assess bias, so the extent to which Chetty et al.’s results were due to bias was then more or less moot.

But after Rothstein released his recent study effectively critiquing Chetty et al. on this point, Kane (and colleagues) released the results Kane presented at the Vergara trial (see here). However, Kane (and colleagues) seemingly released an updated version of “Kane’s” initial results to seemingly counter Rothstein, in support of Chetty. In other words, Kane seems to have released his study (perhaps) more in support of his colleague Chetty than in the name of conducting good, independent research.

Oh the tangled web Chetty and Kane (purportedly) continue to weave.

See also Chetty et al.’s direct response to Rothstein here.

Recent Gallup Poll on Teacher Perceptions

Posted on by
Reply

In an “Update” published by the Gallup public opinion polling company titled “Teachers Favor Common Core Standards, Not the Testing,” some interesting data can be found regarding current teachers’ perceptions about some currently hot reform topics in education. These Gallup data include, as pertinent here, self-report data on how teachers feel about using students’ test scores to hold them accountable.

Here are what I see as the most interesting findings, again as pertinent here. For more information, however, do click here.

  • 76% of America’s public school teachers “reacted positively” to the primary goal of the Common Core State Standards (i.e., to have all states use the same set of academic standards for reading, writing and math in grades K-12).
  • 9% of America’s public school teachers were in favor of linking students’ test scores to teacher evaluations.
  • 89% of America’s public school teachers viewed linking students’ test scores to teacher evaluations negatively, and 2% reported having “no opinion” on the matter.
  • 78% of America’s public school teachers agreed that the tests to be used in general and also to hold them accountable for their effectiveness (e.g., using VAMs) take too much time away from teaching.

As one teacher who helped Gallup to interpret these findings put it: “The [Common Core] standards were positive until standardized testing was involved.”

As for linking students’ test scores to teacher performance, another teacher elaborated on this by stating, “Student populations are not spread evenly among teachers. Some students will have learning disabilities and some will have familial and environmental factors that do not allow them to progress at the same rate as others. If I have to be so concerned about a test score, I cannot address the needs of individual students.” And a Texas high school teacher concludes, “We must have different yardsticks for different student circumstances so teachers can focus on individual student growth.”

The teachers who participated in this survey research study included a random sample of 854 public K-12 school teachers, aged 18 and older, with Internet access, living in all 50 U.S. states and the District of Columbia.

Education-Related Election Updates

Posted on by
Reply

Great news! The state of Missouri did not pass its “VAM with a Vengeance” constitutional amendment proposing to tie teacher evaluation to student test scores and require teachers to be dismissed, retained, promoted, demoted, and paid based primarily on the test scores of their students. It was also to require teachers to enter into contracts of three years or less, eliminating seniority and tenure. Well done, voters of Missouri!!

Uncertain news! My state of Arizona is still counting ballots to determine who will be our new State Superintendent of Public Instruction. For the first time in history, we have somebody who actually knows a lot about education running – Dr. David Garcia – an associate professor colleague of mine at Arizona State University who also studies educational policy. He would also be the first Latino elected to a state office in over 30 years. He is running as a Democrat, though, and in a very red state, a win for him and others before him have often turned impossible. His opponent is Diane Douglas, a former school board member who focused everything about her campaign in protest of the Common Core Standards and the federal government’s involvement in state-level educational policies. She ran a low-key campaign, she dodged the media spotlight, and she avoided all but one debate with David during which David literally ate her lunch. BUT lucky for her, she ran on the Republican ticket. While there are still 300,000 votes to be counted, she has a narrow lead which is making many on both sides of the party line very upset with those in our state who perpetually vote along the party line. Even with David drawing some very significant endorsements from Republicans and conservative business groups throughout this red state, the state’s conservative voters continue to dominate. Many agree on this note and “see her success as part of the decisive Republican sweep and not a triumph of ideology.” So very sad for our state is this one here. The next move might just be amend the state constitution and make this a state appointed position in that David is by far the best and most informed candidate our state has likely every had in the running (anonymous phone conversation, this morning ;).

Unfortunate news! As per a recent post by Diane Ravitch, the other election news as related to education was bad. There were many victories for similar people “who hold the public sector in contempt and believe in Social Darwinism. It was a bad night for those who hope for a larger vision of the common good, some vision grander than each one on his own. American history and politics are cyclical. It may require the excesses of this time to bring a turn of the wheel. It’s always darkest just before dawn.”