My Forthcoming Speaking Engagement in AZ about VAMs

If you reside in Arizona or plan to travel here soon, I invite you to join me as I discuss my new book, Rethinking Value-Added Models in Education: Critical Perspectives on Tests and Assessment-Based Accountability (available this Sunday, May 4th). To read more about this book, please click here re: a prior post with more of the book’s details.

AZPresentation

This event is free and open to the public, and many representing many constituencies across the state have been invited. It will be held one week from this Saturday, on Saturday, May 10th, 2014 from 10:00am-11:30am in the Education Lecture Hall at Arizona State University (ASU’s main campus).

For more details and to RSVP, please use this link: http://whoozin.com/Events/EventU.aspx?e=WY6-VEW-X3QV

Coffee, tea, and light refreshments will be provided.

Hope to see you all there. If you cannot make it, I will also post the materials-post presentation, for all of you who follow this blog who do not reside in Arizona and feel not so inclined to fly in 😉

The Intersection of Standards and their Assessments: From an AZ Teacher

In January, I wrote a post about “An AZ Teacher’s Perspective on Her “Value-Added.” Valerie Strauss covered the same story in her The Answer Sheet blog for The Washington Post, validating for me that readers appreciate stories from the field that explain in better terms than I can what is actually happening as these VAM-based teacher accountability and evaluation systems are being “lived out” in practice.

Well, the same AZ teacher wrote to me another story that I encourage you all to read, about the intersection and alignment of standards and their assessments, or more specifically the lack thereof.

She writes:

A fundamental principal in education is the precise alignment of the teaching of learning objectives (standards) with the assessment of learning objectives (tests). Research has demonstrated that when an educator plans lessons that begin with an analysis of what students need to learn, coupled with how a student will demonstrate the learning, achievement tends to happen. This is a “best practice” in education.

Enter: standards’ reform. My school district saw the writing on the wall: Common Core implementation was going to be massive. Beyond a shift in the philosophical underpinnings of standards (college and career readiness vs. every state for themselves), Common Core implementation meant, in some cases, a shift in instructional approaches (inquiry vs. modeling). And frequently, Common Core implementation meant changes in what got taught and in which grade levels.

Like any well-meaning, responsible school district, my district realized these changes were going to take time.  And so they began Common Core implementation earlier than others in the 2012-2013 school year. And based on what we know about best practices, when the standards change, the assessments should change as well. But they didn’t—yet. For the past two years, I (and many, many others) have been teaching standards that are NOT fully aligned to the state assessment system. Instead, we’ve been frantically (and some may say schizophrenically!) trying to teach two sets of standards—the old (aligned with Arizona’s current state assessment) and the new Common Core.

Enter: value-added measures. Value added measures are statistical tools aimed at capturing a teacher’s impact on student achievement through student performance on standardized tests. A few years ago, Arizona passed a law that mandates up to 50% of a teacher’s evaluation be comprised of student test scores. And again, my school district did what any responsible, law-abiding district would do: implement a teacher evaluation system that complies with state law: 50% of a teacher’s evaluation is composed of student test scores and 50% is composed of classroom observations.

The intersection of these two policies is a problem for teachers (and students!). If the state assessment is not designed to precisely test the standards that are being taught, it cannot be legitimately claimed that value-added measures (or any other measure using student test scores!) are capturing a teacher’s impact on student achievement? One problem for teachers is that their employment status may hinge on the outcome. One problem for students is that what they are learning may not be what they are ultimately held accountable for on the state assessment.

In this case, compliance with law has superseded the use of best practices. Let us hope this doesn’t happen in another field–say, healthcare?

 

Rhee Coming to AZ?!?

Speaking of Rhee, mentioned in a prior post today, it seems that she is to be in my state of Arizona, as part of her most recent “tour.”

Arizona, continuously basking in the glow of negative attention, coming largely from the uber conservative policies supported by our current Republican Governor Jan Brewer, is to potentially have another gubernatorial leader – Scott Smith.

I was sent, in secret, an invitation to a “Reception in Honor” of gubernatorial candidate Smith…and his special guest of the evening…[insert drum roll]…Michelle Rhee.

The party is to celebrate their collective stance towards “Efficient, Effective & Accountable Leadership for Arizona.” Sound familiar?

Anyhow, a ticket to this event comes at a cost of a minimum campaign contribution set at $250 per person to a maximum campaign contribution of $4,000 per person. The event is to be held tonight, Friday, February 21, from 5:00-6:00 (yes, for one hour) at The Montelucia Resort in a wealthy suburb within the metropolitan Phoenix area called Paradise Valley.

It seems wealth is a politician’s common denominator of power, now doesn’t it?

On that note, I heard and then verified recently, that current AZ Governor Brewer’s highest level of education is that of an x-ray technician with her highest degree earned being a radiological technologist certificate. That too says a lot about the power of money and the power money can bring for (too often, in the case of Arizona) some very, let’s say, misguided policies. Thanks to Rhee, she can help my state in its misguidedness.

As related to Arizona’s teacher accountability system in particular, so far, the state department of education has at least tried to maintain some sanity in terms of its teacher accountability and related VAM-based policies, leaving much of this to be determined by and in the hands of districts and schools who still very much honor and appreciate their local control. I like to think that I have had at least a little to do with this current stance, while not ideal but reasonable given the current sociopolitical circumstances of my state.

But it looks like if this candidate wins, we might be in way worse shape than we are now, despite our best intentions…and, again, with special thanks to policy clots like Rhee.

An AZ Teacher’s Perspective on Her “Value-Added”

This came to me from a teacher in my home state – Arizona. Read not only what is becoming a too familiar story, but also her perspective about whether she is the only one who is “adding value” (and I use that term very loosely here) to her students’ learning and achievement.

She writes:

Initially, the focus of this note was going to be my 6-year long experience with a seemingly ever-changing educational system.  I was going to list, with some detail, all the changes that I have seen in my brief time as a K-6 educator, the end-user of educational policy and budget cuts.  Changes like (in no significant order):

  • Math standards (2008?)
  • Common Core implementation and associated instructional shifts (2010?)
  • State accountability system (2012?)
  • State requirements related to ELD classrooms (2009?)
  • Teacher evaluation system (to include a new formula of classroom observation instrument and value-added measures) (2012-2014)
  • State laws governing teacher evaluation/performance, labeling and contracts (2010?)

have happened in a span of, not much more than, three years. And all these changes have happened against a backdrop of budget cuts severe enough to, in my school district, render librarians, counselors, and data coordinators extinct.  In this note, I was going to ask, rhetorically: “What other field or industry has seen this much change this quickly and why?” or “How can any field or industry absorb this much change effectively?”

But then I had a flash of focus just yesterday during a meeting with my school administrators, and I knew immediately the simple message I wanted to relay about the interaction of high-stakes policies and the real world of a school.

At my school, we have entered what is known as “crunch time”—the three-month long period leading up to state testing.  The purpose of the meeting was to roll out a plan, commonly used by my school district, to significantly increase test scores in math via a strategy of leveled grouping. The plan dictates that my homeroom students will be assigned to groups based on benchmark testing data and will then be sent out of my homeroom to other teachers for math instruction for the next three months. In effect, I will be teaching someone else’s students, and another teacher will be teaching my students.

But, wearisomely, sometime after this school year, a formula will be applied to my homeroom students’ state test scores in order to determine close to 50% of my performance. And then another formula (to include classroom observations) will be applied to convert this performance into a label (ineffective, developing, effective, highly effective) that is then reported to the state.  And so my question now is (not rhetorically!), “Whose performance is really being measured by this formula—mine or the teachers who taught my students math for three months of the school year?” At best, professional reputations are at stake–at worse, employment is.

David Berliner’s “Thought Experiment”

My main mentor, David Berliner (Regents Professor at Arizona State University) wrote a “Thought Experiment” that Diane Ravitch posted on her blog yesterday. I have pasted the full contents here for those of you who may have missed it. Do take a read, and play along and see if you can predict which state will yield higher test performance in the end.

—–

Let’s do a thought experiment. I will slowly parcel out data about two different states. Eventually, when you are nearly 100% certain of your choice, I want you to choose between them by identifying the state in which an average child is likely to be achieving better in school. But you have to be nearly 100% certain that you can make that choice.

To check the accuracy of your choice I will use the National Assessment of Educational Progress (NAEP) as the measure of school achievement. It is considered by experts to be the best indicator we have to determine how children in our nation are doing in reading and mathematics, and both states take this test.

Let’s start. In State A the percent of three and four year old children attending a state associated prekindergarten is 8.8% while in State B the percent is 1.7%. With these data think about where students might be doing better in 4th and 8th grade, the grades NAEP evaluates student progress in all our states. I imagine that most people will hold onto this information about preschool for a while and not yet want to choose one state over the other. A cautious person might rightly say it is too soon to make such a prediction based on a difference of this size, on a variable that has modest, though real effects on later school success.

So let me add more information to consider. In State A the percent of children living in poverty is 14% while in State B the percent is 24%. Got a prediction yet? See a trend? How about this related statistic: In State A the percent of households with food insecurity is 11.4% while in State B the percent is 14.9%. I also can inform you also that in State A the percent of people without health insurance is 3.8% while in State B the percent is 17.7%. Are you getting the picture? Are you ready to pick one state over another in terms of the likelihood that one state has its average student scoring higher on the NAEP achievement tests than the other?

​If you still say that this is not enough data to make yourself almost 100% sure of your pick, let me add more to help you. In State A the per capita personal income is $54,687 while in state B the per capita personal income is $35,979. Since per capita personal income in the country is now at about $42,693, we see that state A is considerably above the national average and State B is considerably below the national average. Still not ready to choose a state where kids might be doing better in school?

Alright, if you are still cautious in expressing your opinions, here is some more to think about. In State A the per capita spending on education is $2,764 while in State B the per capita spending on education is $2,095, about 25% less. Enough? Ready to choose now?
Maybe you should also examine some statistics related to the expenditure data, namely, that the pupil/teacher ratio (not the class sizes) in State A is 14.5 to one, while in State B it is 19.8 to one.

As you might now suspect, class size differences also occur in the two states. At the elementary and the secondary level, respectively, the class sizes for State A average 18.7 and 20.6. For State B those class sizes at elementary and secondary are 23.5 and 25.6, respectively. State B, therefore, averages at least 20% higher in the number of students per classroom. Ready now to pick the higher achieving state with near 100% certainty? If not, maybe a little more data will make you as sure as I am of my prediction.

​In State A the percent of those who are 25 years of age or older with bachelors degrees is 38.7% while in State B that percent is 26.4%. Furthermore, the two states have just about the same size population. But State A has 370 public libraries and State B has 89.
Let me try to tip the data scales for what I imagine are only a few people who are reluctant to make a prediction. The percent of teachers with Master degrees is 62% in State A and 41.6% in State B. And, the average public school teacher salary in the time period 2010-2012 was $72,000 in State A and $46,358 in State B. Moreover, during the time period from the academic year 1999-2000 to the academic year 2011-2012 the percent change in average teacher salaries in the public schools was +15% in State A. Over that same time period, in State B public school teacher salaries dropped -1.8%.

I will assume by now we almost all have reached the opinion that children in state A are far more likely to perform better on the NAEP tests than will children in State B. Everything we know about the ways we structure the societies we live in, and how those structures affect school achievement, suggests that State A will have higher achieving students. In addition, I will further assume that if you don’t think that State A is more likely to have higher performing students than State B you are a really difficult and very peculiar person. You should seek help!

So, for the majority of us, it should come as no surprise that in the 2013 data set on the 4th grade NAEP mathematics test State A was the highest performing state in the nation (tied with two others). And it had 16 percent of its children scoring at the Advanced level—the highest level of mathematics achievement. State B’s score was behind 32 other states, and it had only 7% of its students scoring at the Advanced level. The two states were even further apart on the 8th grade mathematics test, with State A the highest scoring state in the nation, by far, and with State B lagging behind 35 other states.

Similarly, it now should come as no surprise that State A was number 1 in the nation in the 4th grade reading test, although tied with 2 others. State A also had 14% of its students scoring at the advanced level, the highest rate in the nation. Students in State B scored behind 44 other states and only 5% of its students scored at the Advanced level. The 8th grade reading data was the same: State A walloped State B!

States A and B really exist. State B is my home state of Arizona, which obviously cares not to have its children achieve as well as do those in state A. It’s poor achievement is by design. Proof of that is not hard to find. We just learned that 6000 phone calls reporting child abuse to the state were uninvestigated. Ignored and buried! Such callous disregard for the safety of our children can only occur in an environment that fosters, and then condones a lack of concern for the children of the Arizona, perhaps because they are often poor and often minorities. Arizona, given the data we have, apparently does not choose to take care of its children. The agency with the express directive of insuring the welfare of children may need 350 more investigators of child abuse. But the governor and the majority of our legislature is currently against increased funding for that agency.

State A, where kids do a lot better, is Massachusetts. It is generally a progressive state in politics. To me, Massachusetts, with all its warts, resembles Northern European countries like Sweden, Finland, and Denmark more than it does states like Alabama, Mississippi or Arizona. According to UNESCO data and epidemiological studies it is the progressive societies like those in Northern Europe and Massachusetts that care much better for their children. On average, in comparisons with other wealthy nations, the U. S. turns out not to take good care of its children. With few exceptions, our politicians appear less likely to kiss our babies and more likely to hang out with individuals and corporations that won’t pay the taxes needed to care for our children, thereby insuring that our schools will not function well.

But enough political commentary: Here is the most important part of this thought experiment for those who care about education. Everyone of you who predicted that Massachusetts would out perform Arizona did so without knowing anything about the unions’ roles in the two states, the curriculum used by the schools, the quality of the instruction, the quality of the leadership of the schools, and so forth. You made your prediction about achievement without recourse to any of the variables the anti-public school forces love to shout about –incompetent teachers, a dumbed down curriculum, coddling of students, not enough discipline, not enough homework, and so forth. From a few variables about life in two different states you were able to predict differences in student achievement test scores quite accurately.

I believe it is time for the President, the Secretary of Education, and many in the press to get off the backs of educators and focus their anger on those who will not support societies in which families and children can flourish. Massachusetts still has many problems to face and overcome—but they are nowhere as severe as those in my home state and a dozen other states that will not support programs for neighborhoods, families, and children to thrive.

This little thought experiment also suggests also that a caution for Massachusetts is in order. It seems to me that despite all their bragging about their fine performance on international tests and NAEP tests, it’s not likely that Massachusetts’ teachers, or their curriculum, or their assessments are the basis of their outstanding achievements in reading and mathematics. It is much more likely that Massachusetts is a high performing state because it has chosen to take better care of its citizens than do those of us living in other states. The roots of high achievement on standardized tests is less likely to be found in the classrooms of Massachusetts and more likely to be discovered in its neighborhoods and families, a refection of the prevailing economic health of the community served by the schools of that state.

Arizona’s Teacher Evaluation System, Not Strict Enough?

Last week, Arizona State Superintendent of Public Instruction, John Huppenthal, received the news that Arizona’s No Child Left Behind (NCLB) waiver extension request had been provisionally granted with a “high-risk” label (i.e., in danger of being revoked). Superintendent Huppenthal was given 60 days to make two revisions: (1) adjust the graduation rate to account for 20% of a school’s A-F letter grade instead of the proposed 15% and, as most pertinent here, (2) finalize the guidelines for the teacher and principal evaluations to comply with Elementary and Secondary Education Act (ESEA) Flexibility (i.e., the NCLB waiver guidelines).

Within 60 days, Superintendent Huppenthal and the Arizona Department of Education (ADE) must: (1) finalize its teacher and principal evaluation guidelines; (2) give sufficient weighting to student growth so as to differentiate between teachers/principals who have contributed to more/less growth in student learning and achievement; (3) ensure that shared attribution of growth does not mask high or low performing teachers as measured by growth; and (4) guarantee that all of this is done in time for schools to be prepared to implement for the 2014-2015 school year.

These demands, particularly #2 and #3 above, reflect some of the serious and unavoidable flaws with the new teacher evaluations that are based on student growth (e.g., and all other VAMs).

As per #2, the most blatant problem is with the limited number of teachers (typically around 30%, although reported as only 17% in the recent post about DC’s teacher evaluation system) who are eligible for classroom-level student growth data (i.e., value-added). Thus, one of the key expectations—to ensure sufficient weight to student growth scores so as to differentiate between teachers’/principals’ impact on student learning and achievement—is impossible for probably around seven out of every ten of Arizona’s and other states’ teachers. While most states, including Arizona, have chosen to remedy this problem by attributing a school-level (or grade-level) value-added score to classroom-level ineligible teachers (sometimes counting as much as 50% of the teacher’s overall evaluation), this solution does not (and likely never will) suffice as per #2 written above. It seems the feds do not quite understand that what they are mandating in practice leaves well over half of teachers’ evaluations based on both students and/or content that these teachers didn’t teach.

As per #3, Arizona (and all waiver-earning states) is also to demonstrate how the state will ensure that shared attribution of growth does not mask high or low performing teachers as measured by growth. Yet, again, when these systems are implemented in practice, 70+% of teachers are assigned a school-level student growth score, meaning that all teachers in any given school who fall into this group will all receive the same score. In what way is it feasible to “ensure” that no high or low performing teacher is “masked” by such a method of attributing student growth to teachers in this way? Yet this is another example of the type of illogical circumstances by which schools must abide in order to meet the arbitrary (and often impossible) demands of ESEA Flexibility (and Race to the Top).

If Arizona fails to comply with the USDOE requests within 60 days, they will lose their ESEA waiver and face the consequences of NCLB. In a statement to Education Week, however, AZ Superintendent Huppenthal stood by his position on providing school districts with as much flexibility as possible within the constraints of the waiver stipulations. He said he will not protest the “high risk” label and will instead attempt to “get around this and still keep local control for those school districts.” The revised application is due at the end of January.

Post contributed by Jessica Holloway-Libell