Data Secrecy Violating Data Democracy in DC Public Schools

The District of Columbia Public Schools (DCPS) is soon to vote on yet another dramatic new educational policy that, as described in an email/letter to all members of the American Federation of Teachers (AFT) by AFT President Randi Weingarten, “would make it impossible for educators, parents and the general public to judge whether some of DCPS’ core instructional strategies and policies are really helping District children succeed.”

As per Weingarten: “Over a year ago, the Washington [DC] Teachers’ Union filed a Freedom of Information Act (FOIA) request to see the data from the school district’s IMPACT [teacher] evaluation system—a system that’s used for big choices, like the firing of 563 teachers in just the past four years, curriculum decisions, school closures and more [see prior posts about this as related to the IMPACT program here]. The FOIA request was filed because DCPS refused to provide the data….[data that are]…essential to understanding and addressing the DCPS policies and practices that impact” teachers and education in general.

Not only are such data crucial to build understandings, as noted, but they are also crucial in support of a functioning democracy, to allow others within a population concerned with a public institution test the mandates and policies they collectively support, in theory or concept (perhaps) but also via public taxes.

Regardless, soon after the DC union filed the FOIA, DCPS (retaliated, perhaps, and) began looking to override FOIA laws through “a radical new secrecy provision to hide the information that’s being used to make big decisions” like those associated with the aforementioned IMPACT teacher evaluation system.

Sound familiar? See prior posts about other extreme governmental moves in the name of secrecy, or rather educational policies at all costs, namely in New Mexico here and here.

You can send a letter to those in D.C. to vote NO on their “Educator Evaluation Data Protection” provisions by clicking here.

As per another post on this topic, in GFBrandenburg’s Blog — that is “Just a blog by a guy who’s a retired math teacher” — Brandenburg did leak some of the data now deemed “secret.” Namely, he “was leaked,” by an undisclosed source, “the 2009-10 IMPACT sub-scores from the Value-Added Monstrosity (VAM) nonsense and the Teaching and Learning Framework (TLF), with the names removed. [He] plotted the two [sets of] scores and showed that the correlation was very, very low, in fact about 0.13 [r-squared=0.33], or nearly random, as you [can] see here:”

vam-vs-tlf-dc-2009-10

In the world of correlation, this is atrocious, IF high-stakes (e.g., teacher termination, tenure, merit pay) are to be attached to such output. No wonder DCPS does not want people checking in to see if that which they are selling is true to what is being sold.

In Brandenburg’s words: “Value-Added scores for any given teacher jumped around like crazy from year to year. For all practical purposes, there is no reliability or consistency to VAM whatsoever. Not even for elementary teachers who teach both English and math to the same group of children and are ‘awarded’ a VAM score in both subjects. Nor for teachers who taught, say, both 7th and 8th grade students in, say, math, and were ‘awarded’ VAM scores for both grade levels: it’s as if someone was to throw darts at a large chart, blindfolded, and wherever the dart lands, that’s your score.”

US Secretary of Education Duncan “Loves Him Some VAM Sauce”

US Secretary of Education “Arne [Duncan] loves him some VAM sauce, and it is a love that simply refuses to die,” writes Peter Greene in a recent Huffington Post post. Duncan’s (simple-mind) loves it because, indeed, the plan is (too) overly simplistic. All that the plan requires are two simple ingredients: “1) A standardized test that reliably and validly measures how much students know 2) A super-sciency math algorithm that will reliably and validly strip out all influences except that of the teacher.”

Sticking with the cooking metaphor, however, Green writes “VAM is no spring chicken, and perhaps when it was fresh and young some affection for it could be justified. After all, lots of folks, including non-reformy folks, like the idea of recognizing and rewarding teachers for being excellent. But how would we identify these pillars of excellence? That was the puzzler for ages until VAM jumped up to say, “We can do it! With Science!!” We’ll give some tests and then use super-sciency math to filter out every influence that’s Not a Teacher and we’ll know exactly how much learnin’ that teacher poured into that kid.”

“Unfortunately, we don’t have either,” and we likely never will. Why this is the case is also highlighted in this post, with Greene explicitly citing three main sources for support: the recent oppositional statement released by the National Association of Secondary School Principals, the oppositional statement released this past summer by the American Statistical Association, and our mostly-oppositional blog Vamboozled! (by Audrey Amrein-Beardsley). Hopefully getting the research into the hands of educational practitioners, school board members, the general public, and the like is indeed “adding value” in the purest sense of this phrase’s meaning. I sure hope so!

Anyhow, in this post Greene also illustrates and references a nice visual (with a side of sarcasm) explaining the complexity behind VAMs in pretty clear terms. I also paste this illustration here, which Greene references as originally coming from a blog post from Daniel Katz, Ph.D. but I have seen similar versions elsewhere and prior (e.g., a New York Times article here).

2014-12-16-VAM1.jpg2014-12-16-VAM2.jpg

Greene ultimately asks why Duncan is still staying so fixated on a policy, disproportionally loaded and ever-increasingly rejected and unsupported?

Greene’s answer: ‘[I]f Duncan were to admit that his beloved VAM is a useless tool…then all his other favorite [reform-based] programs would collapse” around him…Why do we give the Big Test? To measure teacher effectiveness. How do we rank and evaluate our schools? By looking at teacher effectiveness. How do we find the teachers that we are going to move around so that every classroom has a great teacher? With teacher effectiveness ratings. How do we institute merit pay and a career ladder? By looking at teacher effectiveness. How do we evaluate every single program instituted in any school? By checking to see how it affects teacher effectiveness. How do we prove that centralized planning (such as Common Core) is working? By looking at teacher effectiveness. How do we prove that corporate involvement at every stage is a Good Thing? By looking at teacher effectiveness. And by “teacher effectiveness,” we always mean VAM (because we [i.e., far-removed educational reformers] don’t know any other way, at all).”

If Duncan’s “magic VAM sauce, is a sham and a delusion and a big bowl of nothing,” his career would literally fold in.

To read more from Greene, do click here to read his post in full.

“VAM with a Vengeance” on the Ballot in Missouri

On this election day, the state of Missouri is worth mentioning as it has a very relevant constitutional amendment on its ballot for the day. As featured on Diane Ravitch’s blog a few days ago here, this (see below) is “the worst constitutional amendment to appear on any state ballot in 2014.”

missouriballotissue

“It ties teacher evaluation to student test scores. It bans collective bargaining about teacher evaluation. It requires teachers to be dismissed, retained, promoted, demoted, and paid based primarily on the test scores of their students. It requires teachers to enter into contracts of three years or less, thus eliminating seniority and tenure.

This is VAM with a vengeance.

This ballot resolution is the work of the far-right Show-Me Institute, funded by the multi-millionaire Rex Sinquefeld. He is a major contributor to politics in Missouri and to ALEC.

The Center for Media and Democracy writes about him:

‘Sinquefield is doing to Missouri what the Koch Brothers are doing to the entire country. For the Koch Brothers and Sinquefield, a lot of the action these days is not at the national but at the state level.’

‘By examining what Sinquefield is up to in Missouri, you get a sobering glimpse of how the wealthiest conservatives are conducting a low-profile campaign to destroy civil society.’

‘Sinquefield told The Wall Street Journal in 2012 that his two main interests are “rolling back taxes” and “rescuing education from teachers’ unions.’

‘His anti-tax, anti-labor, and anti-public education views are common fare on the right. But what sets Sinquefield apart is the systematic way he has used his millions to try to push his private agenda down the throats of the citizens of Missouri.”

Five Nashville Teachers Face Termination

Three months ago, Tennessee Schools Director Jesse Register announced he was to fire 63 Tennessean teachers, of 195 total who for two consecutive years scored lowest (i.e., a 1 on a scale of 1 to 5) in terms of their overall “value-added” (as based on 35% EVAAS, 15% related “student achievement,” and 50% observational data). These 63 were the ones who three months ago were still “currently employed,” given the other 132 apparently left voluntarily or retired (which is a nice-sized cost-savings, so let’s be sure not to forget about the economic motivators behind all of this as well). To see a better breakdown of these numbers, click here.

This was to be the first time in Tennessee that its controversial, and “new and improved” teacher evaluation system would be used to take deliberate action against whom they deemed their “lowest-performing” teachers, as “objectively” identified in the classroom; although, officials at that time did not expect to have a “final number” to be terminated until fall.

Well, fall is here, and it seems this final number is officially five: three middle school teachers, one elementary school teacher, and one high school teacher, all teaching in metro Nashville.

The majority of these teachers come from Neely’s Bend: “one of 14 Nashville schools on the state’s priority list for operating at the bottom 5 percent in performance statewide.” Some of these teachers were evaluated even though their principal who evaluated them is  “no longer there.” Another is a computer instructor being terminated as based on this school’s overall “school-level value-added.” This is problematic in and of itself given teacher-level and in this case school-level bias seem to go hand in hand with the use of these models, and grossly interfere with accusations that these teachers “caused” low performance (see a recent post about this here).

It’s not to say these teachers were not were indeed the lowest performing; maybe they were. But I for one would love to talk to these teachers and take a look at their actual data, EVAAS and observational data included. Based on prior experiences working with such individuals, there may be more to this than what it seems. Hence, if anybody knows these folks, do let them know I’d like to better understand their stories.

Otherwise, all of this effort to ultimately attempt to terminate five of a total 5,685 certified teachers in the district (0.09%) seems awfully inefficient, and costly, and quite frankly absurd given this is a “new and improved” system meant to be much better than a prior system that likely yielded a similar termination rate, not including, however, those who left voluntarily prior.

Perhaps an ulterior motive is, indeed, the cost-savings realized given the mere “new and improved” threat.

Can Today’s Tests Yield Instructionally Useful Data?

The answer is no, or at best not yet.

Some heavy hitters in the academy just released an article that might be of interest to you all. In the article the authors discuss whether “today’s standardized achievement tests [actually] yield instructionally useful data.”

The authors include W. James Popham, Professor Emeritus from the University of California, Los Angeles; David Berliner, Regents’ Professor Emeritus at Arizona State University; Neal Kingston, Professor at the University of Kansas; Susan Fuhrman, current President of Teachers College, Columbia University; Steven Ladd, Superintendent of Elk Grove Unified School District in California; Jeffrey Charbonneau, National Board Certified Teacher in Washington and the 2013 US National Teacher of the Year; and Madhabi Chatterji, Associate Professor at Teachers College, Columbia University.

These authors explored some of the challenges and promises in terms of using and designing standardized achievement tests and other educational tests that are “instructionally useful.” This was the focus of a recent post about whether Pearson’s tests are “instructionally sensitive” and what University of Texas – Austin’s Associate Professor Walter Stroup versus Pearson’s Senior Vice President had to say on this topic.

In this study, authors deliberate more specifically the consequences of using inappropriately designed tests for decision-making purposes, particularly when tests are insensitive to instruction. Here, the authors underscore serious issues related to validity, ethics, and consequences, all of which they use and appropriately elevate to speak out, particularly against the use of current, large-scale standardized achievement tests for evaluating teachers and schools.

The authors also make recommendations for local policy contexts, offering recommendations to support (1) the design of more instructionally sensitive large-scale tests as well as (2) the design of other smaller scale tests that can also be more instructionally sensitive, and just better. These include but are not limited to classroom tests as typically created, controlled, and managed by teachers, as well as district tests as sometimes created, controlled, and managed by district administrators.

Such tests might help to create more but also better comprehensive educational evaluation systems, the authors ultimately argue. Although this, of course, would require more professional development to help teachers (and others, including district personnel) develop more instructionally sensitive, and accordingly useful tests. As they also note, this would also require that “validation studies…be undertaken to ensure validity in interpretations of results within the larger accountability policy context where schools and teachers are evaluated.”

This is especially important if tests are to be used for low and high-stakes decision-making purposes. Yet this is something that is way too often forgotten when it comes to test use, and in particular test abuse. All should really take heed here.

Reference: Popham, W. J., Berliner, D. C., Kingston, N. M., Fuhrman, S. H., Ladd, S. M., Charbonneau, J., & Chatterji, M. (2014). Can today’s standardized achievement tests yield instructionally useful data? Quality Assurance in Education, 22(4), 303-318 doi:10.1108/QAE-07-2014-0033. Retrieved from http://www.tc.columbia.edu/aeri/publications/QAE1.pdf

“No Exceptions and No Excuses”

In an excellent, descriptive article written in The New Yorker two weeks ago, the author describes how educators in a high-needs, struggling school in Atlanta (during the now infamous cheating scandal that surrounded Atlanta’s public schools in the late 2000’s) made a series of “shocking choice[s]” to artificially inflate and boost their students’ test scores to meet state/federal mandates (i.e., No Child Left Behind [NCLB]). Choices they justified so that they could pacify others,’ including said state/federal policymakers,’ “relentless focus” on data and data-driven accountability, mainly to keep their school in tact and what they believed to be safe in their community.

While this all occurred at the same time the “new and improved” value-added measures (VAMs) as we now know them were being introduced to the country, that is to (theoretically) support even better and more accurate data measurements and data-driven accountability policies, some pro-VAM folks still make explicit their beliefs that using VAMs will inherently (i.e., without research evidence) decrease incidences of cheating, artificial score inflation, system manipulation/distortion, and the like (as so poignantly described within this article).

It is important to note, and continuously recall should you read this article in full, that really nothing has changed since this or the other cheating scandals that arose post NCLB. Likewise, incidences such as these are still arguably widespread and still most common in schools like the school described herein. If anything, we are now even more “enthusiastic about judging teaching by what [now] appear[s] to be an [even more] objective metric” than the metric used before (i.e., Adequate Yearly Progress vs. VAMs in widespread use now); hence, there is absolutely no reason to believe (and again no research evidence to support) that such incidences (even with increased security measures) have whatsoever declined.

Accordingly, a bit of logic and a lot of Campbell’s Law (i.e., “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor)  suggest that the unintended consequences that accompany what is now teacher-based accountability as based on VAMs are if anything more, not less, widespread as VAMs permit the attachment of even more serious consequences to students’ test scores.

Just because VAM use is new, and research has yet to bear “objective” evidence that similar gaming behaviors continue to occur (behaviors, again, like those described herein and elsewhere within and outside of Georgia before and since), this does not mean such behaviors are not still live and well throughout America’s, especially high-needs schools. This all still with thanks to American policymakers’ continuous fixation on and obsession with increased accountability measures and their “no exceptions and no excuses” approach to test-based school reform.

Repeating Policy Behaviors Over and Over Again Despite the Evidence

According to Albert Einstein, the definition of insanity is to repeat the same behaviors over and over again in the hope that different results will materialize “the next time,” perhaps after this is fiddled with or that is fine-tuned. In the case of VAMs, it seems, educational policymaker are stuck in such a pattern of insanity, propagating and regenerating policies based on the use of students’ test scores to evaluate teachers, despite the fact that this has been a policy initiative of now almost 25 years past. Yes, we have research from now almost 25 years ago re: why this is a bad idea!

Gene V Glass published a study about just this in 1990. Click here to check it out. While this study was conducted pre-VAM (as we currently know it, particularly in terms of policy not form), the research-based reason why this is wrong remains the same. Regardless of the machine or statistical technique to be used, “the machine that is meant to do the world’s work must be observed in the actual world where [the VAM-based research] designs meet reality.”

The issues then, continue to be the issues today: “[t]he issue of ‘pupil performance in teacher evaluation’ needs to be discussed in a way that has not been separated from the many other pressures that shape a personnel system of assessment and rewards. The issue cannot be judged apart from how it exists in real places. The validity of such a process of evaluating teachers, its economy, its power to engender belief and win acceptance, depend on how it fits with many other realities of the teaching profession. What now must be addressed is how this notion gets applied in a context as complicated as education. How is the idea transformed as it moves from the statistician’s mind to the real world? How does it fare when the tradeoffs and balances are struck? Is the concept of evaluating teachers by student progress [to be] trusted?”

Glass then explores these questions using a case-based analysis of a number of district sites, concluding what we would still conclude and predict today, even with current and “more contemporary” approaches to the same policy “initiatives.” Glass concludes, that using student achievement data to evaluate teachers…

  1. …will nearly always be undertaken at the level of a school (either all or none of the teachers in a school are rewarded equally) rather than at the level of individual teachers since (a) no authoritative tests exist in most areas of the secondary school curriculum, nor for most special roles played by elementary teachers; and (b) teachers reject the notion that they should compete with their colleagues for raises, privileges and perquisites;
  2. …will always be combined with other criteria (such as absenteeism or extra work) which [continue to] prove to be the real discriminators between who is ultimately rewarded and who is not;
  3. …will always be too susceptible to intentional distortion and manipulation to engender any confidence in the data; moreover teachers and others who believe that no type of test nor any manner of statistical analysis can equate the difficulty of the teacher’s task in the wide variety of circumstances in which they work will further undermine the system;
  4. …will elevate tests themselves to the level of curriculum goals, obscuring the distinction between learning and performing on tests;
  5. …will often make room for symbolic administrative actions simply undertaken to reassure the lay public that student learning is valued and assiduously sought after…

Sound (insanely) familiar?!?

 

States Most Likely to Receive Race to the Top Funds

Since 2009, the US Department of Education via its Race to the Top initiative has given literally billions in federal, taxpayer funds to incentivize states to adopt its various educational policies, as based on many non-research-based or research-informed reforms. As pertinent here, the main “reform” being VAMs, with funding going to states that have them, are willing to adopt them, and are willing to use them for low- and preferably high-stakes decisions about teachers, schools, and districts.

Diane Ravitch recently posted a piece about the Education Law Center finding that there was an interesting pattern to the distribution of Race to the Top grants. The Education Law Center, in an Education Justice article, found that the states and districts with the least fair and equitable state school finance systems were the states that won a large share of RTTT grants.

Interesting, indeed, but not surprising. There is an underlying reason for this, as based on standard correlations anybody can run or calculate using state-level demographics and some basic descriptive statistics.

In this case, correlational analyses reveal that state-level policies that rely at least in part on VAMs are indeed more common in states that allocate less money than the national average for schooling as compared to the nation. More specifically, they are more likely found in states in which yearly per pupil expenditures are lower than the national average (as demonstrated in the aforementioned post). They are more likely found in states in which students perform worse, or have lower relative percentages of “proficient” students as per the US’s (good) national test (i.e., the National Assessment of Educational Progress (NAEP). They are more likely found in states that have more centralized governments, rather than those with more powerful counties and districts as per local control. They are more likely to be found in more highly populated states and states with relatively larger populations of poor and racial and language minority students. And they are more likely to be found in red states in which residents predominantly vote for the Republican Party.

All of these underlying correlations indeed explain why such policies are more popular, and accordingly adopted in certain states versus others. As well, these underlying correlations help to explain the correlation of interest as presented by the Education Law Center in its aforementioned Education Justice article.  Indeed, these states disproportionally received Race to the Top funds as their political and other state-level demographics would have predicted them to, as these are the states most likely to climb on board the VAMwagon (noting that some states had already done so prior to Race to the Top and hence won first-round Race to the Top funds [e.g., Tennessee]).

Please note, however, that with all imperfect correlations found in correlational research, there are outliers. In this case, this would include blue states that adopt VAMs for consequential purposes (e.g., Colorado) or red states who continue to move relatively slower in terms of their VAM-based policies and initiatives (e.g., Texas, Arizona, and Mississippi). (Co)related again, this would also include states with relatively fewer and relatively more poor and minority students and English Language Learners (ELLs), respectively.

For more about these correlations and state level research, please see: Collins, C., & Amrein-Beardsley, A. (2014). Putting growth and value-added models on the map: A national overview. Teachers College Record, 16(1). Retrieved from: http://www.tcrecord.org/Content.asp?ContentId=17291

Teachers’ Licenses No Longer Jeopardized by THE Tennessee Value-Added Assessment System (TVAAS)

On Friday, April 12th, the Tennessee Board of Education met and rescinded much of the policy that ties teachers’ licenses to their value-added, as determined by the growth in their students’ performance on tests over time as calculated by the all-too-familiar Tennessee Education Value-Added Assessment System (TVAAS), and its all-too-familiar mother-ship the Education Value-Added Assessment System (EVAAS). This means that “teachers’ licenses can no longer be jeopardized by the changes in their students’ test.”

What a victory for teachers in Tennessee! Especially given the TVAAS has been in place in the state since 1993, and Tennessee is the state with the longest running history using value-added (thanks to education “value-added” creator William Sanders).

Although, again, it might be too soon to celebrate, as rumors have it that some of the other consequences tied to TVAAS output are still to be pushed forward. Rather, this might be viewed or interpreted as a symbolic gesture or compromise, to open up room for further compromising in the future.

Accordingly, David Sevier, the Tennessee State Board’s Deputy Executive Director also noted that the Board of Education can “now begin to find a measure that is comparable to TVAAS that all parties can agree on.” While I’m not quite sure how to interpret what this means, though, in terms of whether they think it’s just the TVAAS that’s the problem in terms of evaluating their teachers (also for licensure purposes) and that another VAM can satisfy their needs and ideals, or not. Hopefully, the latter is true.

To read more about this from the original article, click here.

A Tennessee Teacher, On the TVAAS and Other Issues of Concern

Check out this 5-minute video to hear from a teacher in Tennessee – the state recognized for bringing to the country value-added models and VAM-based teacher accountability – as she explains who things are going in her state of Tennessee.

Diane Ravitch, in her call to all of us to share out this and other videos/stories such as these, writes that we should help this video, now with over 100,000 views, reach every parent and teacher across the country. “We can be the change,” and social media can help us counter the nonsense expressed so well herein.