Please follow and like us:

A VAMboozled! follower posted this comment via Facebook the other day: “I was wondering if you had seen this video by The Value-Added Research Center [VARC], called the “Oak Tree Analogy” [it is the second video down]? My children’s school district has it on their web-site. What are your thoughts about VARC, and the video?”

I have my own thoughts about VARC, and I will share these next, but better than that I have somebody else’s much wiser thoughts about this video, as this video has in many ways gone “viral.”

Professor Edward Haertel, School of Education at Stanford University, wrote Linda Darling-Hammond (Stanford), Jesse Rothstein (Berkeley), and me an email a few years ago about just this video. While I could not find the email he eloquently drafted then, I persuaded (aka, begged) him to recreate what he wrote then, here, for all of you.

You might want to watch the video, first, to follow along, or least, to more critically view the contents of the video. You decide, but Professor Haertel writes:

“The Value-Added Research Center’s ‘Oak Tree’ analogy is helpful in conveying the theory [emphasis added] behind value-added models. To compare the two gardeners, we adjust away various influences that are out of the gardeners’ control, and then, as with value added, we just assume that whatever is left over must have been due to the gardener.  But, we can draw some important lessons from this analogy in addition to those highlighted in the presentation.

In the illustration, the overall effect of rainfall was an 8-inch difference in annual growth (+3 inches for one gardener’s location; -5 for the other). Effects of soil and temperature, in one direction or the other, were 5 inches and 13 inches. But the estimated effect of the gardeners themselves was only a 4-inch difference. 

As with teaching, the value-added model must sort out a small “signal” from a much larger amount of “noise” in estimating the effects of interest. It follows that the answer obtained may depend critically on just what influences are adjusted for. Why adjust for soil condition? Couldn’t a skillful gardener aerate the soil or amend it with fertilizer? If we adjust only for rainfall and temperature then Gardener B wins. If we add in the soil adjustment, then Gardener A wins. Teasing apart precisely those factors for which teachers justifiably should be held accountable versus those beyond their control may be well-nigh impossible, and if some adjustments are left out, the results will change. 

Another message comes from the focus on oak tree height as the outcome variable.  The savvy gardener might improve the height measure by removing lower limbs to force growth in just one direction, just as the savvy teacher might improve standardized test scores by focusing instruction narrowly on tested content. If there are stakes attached to these gardener comparisons, the oak trees may suffer.

The oak tree height analogy also highlights another point. Think about the problem of measuring the exact height of a tree—not a little sketch on a PowerPoint slide, but a real tree. How confidently could you say how tall it was to the nearest inch?  Where, exactly, would you put your tape measure? Would you measure to the topmost branch, the topmost twig, or the topmost leaf? On a sunny day, or at a time when the leaves and branches were heavy with rain?

The oak tree analogy does not discuss measurement error. But one of the most profound limitations of value-added models, when used for individual decision making, is their degree of error, referred to technically as low reliability. Simply put, if we compare the same two gardeners again next year, it’s anyone’s guess which of the two will come out ahead.”

Thanks are very much in order, Professor Haertel, for having “added value” to the conversations surrounding these issues, and, helping us collectively understand the not-so-simple theory advanced via this video.