Virginia SGP, also known as Brian Davison — a parent of two public school students in the affluent Loudoun, Virginia area (hereafter referred to as Virginia SGP) — has been an avid (and sometimes abrasive) commentator about value-added models (VAMs), defined generically, on this blog (see, for example, here, here, and here), on Diane Ravitch’s blog (see, for example, here, here, and here), and elsewhere (e.g., Virginia SGP’s Facebook page here). He is an advocate and promoter of the use of VAMs (which are in this particular case Student Growth Percentiles (SGPs); see differences between VAMs and SGPs here and here) to evaluate teachers, and he is an advocate and promoter of the release of teachers’ SGP scores to parents and the general public for their consumption and use.
Related, and as described in a Washington Post article published in March of 2016, Virginia SGP “…Pushed [Virginia] into Debate of Teacher Privacy vs. Transparency for Parents” as per teachers’ SPG data. This occurred via a lawsuit Virginia SGP filed against the state, attempting to force the release of teachers’ SGP data for all teachers across the state. More specifically, and akin to what happened in 2010 when the Los Angeles Times published the names and VAM-based ratings of thousands of teachers teaching in the Los Angeles Unified School District (LAUSD), Virginia SGP “pressed for the data’s release because he thinks parents have a right to know how their children’s teachers are performing, information about public employees that exists but has so far been hidden. He also wants to expose what he says is Virginia’s broken promise to begin using the data to evaluate how effective the state’s teachers are.” He thinks that “teacher data should be out there,” especially if taxpayers are paying for it.
In January, a Richmond, Virginia judge ruled in Virginia SGP’s favor, despite the state’s claims that Virginia school districts, despite the state’s investments, had reportedly not been using the SGP data, “calling them flawed and unreliable measures of a teacher’s effectiveness.” And even though this ruling was challenged by state officials and the Virginia Education Association thereafter, Virginia SGP posted via his Facebook page the millions of student records the state released in compliance with the court, with teacher names and other information redacted.
This past Tuesday, however, and despite the challenges to the court’s initial ruling, came another win for Virginia SGP, as well as another loss for the state of Virginia. See the article “Judge Sides with Loudoun Parent Seeking Teachers’ Names, Student Test Scores,” published yesterday in a local Loudon, Virginia news outlet.
The author of this article, Danielle Nadler, explains more specifically that, “A Richmond Circuit Court judge has ruled that [the] VDOE [Virginia Department of Education] must release Loudoun County Public Schools’ Student Growth Percentile [SGP] scores by school and by teacher…[including] teacher identifying information.” The judge noted that “that VDOE and the Loudoun school system failed to ‘meet the burden of proof to establish an exemption’ under Virginia’s Freedom of Information Act [FOIA].” The court also ordered VDOE to pay Davison $35,000 to cover his attorney fees and other costs. This final order was dated April 12, 2016.
“Davison said he plans to publish the information on his ‘Virginia SGP’ Facebook page. Students will not be identified, but some of the teachers will. ‘I may mask the names of the worst performers when posting rankings/lists but other members of the public can analyze the data themselves to discover who those teachers are,” Virginia SGP said.
I’ve exchanged messages with Virginia SGP prior to this ruling and since, and since I’ve explicitly invited him to also comment via this blog. While with this objective and subsequent ruling I disagree, although I do believe in transparency, it is nonetheless newsworthy in the realm of VAMs and for followers/readers of this blog. Comment now and/or do stay tuned for more.
The case makes it legal to publish invalid rankings of teachers and schools based on invalid measures of “effectiveness.” The prospects for a class-action lawsuit against the Loudoun County Public Schools and Virginia SGP appear to be nil.
How about stack rating parents by the scores of their students, since there is valid research that parents have more influence on in-school performance than teachers…that Coleman report plus overwealming evidence else where.
Taking your argument to its logical conclusion would mean removing children from poor parents or eugenics.
There are misapplications of VAMs but as far as I see they add real, meaningful insight into a student’s growth and the performance of our schools. Unfortunately, the public either seems to be unable to understand them and teacher’s unions excel as promoting a false narrative of “teaching to the test.”
It’s a shame no one care enough about Maricopa schools to wage a similar battle.
“There are misapplications of VAMs but as far as I see they add real, meaningful insight into a student’s growth and the performance of our schools.”
Please tell me how VAMs and SGPs, based on the invalid results of the processes of educational standards and standardized testing can provide “meaningful insight”. Educational malpractices such as VAM and SGP being rife with conceptual errors and falsehoods and filled with psychometric fudging (that fudging is a discussion for another day) have been proven by Noel Wilson to render any results “vain and illusory” or as this rural Show Me State resident says “horse manure”. To understand read the following and then go and read the study itself: “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700
Brief outline of Wilson’s “Educational Standards and the Problem of Error” and some comments of mine.
1. A description of a quality can only be partially quantified. Quantity is almost always a very small aspect of quality. It is illogical to judge/assess a whole category only by a part of the whole. The assessment is, by definition, lacking in the sense that “assessments are always of multidimensional qualities. To quantify them as unidimensional quantities (numbers or grades) is to perpetuate a fundamental logical error” (per Wilson). The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing the descriptive information about said interactions is inadequate, insufficient and inferior to the point of invalidity and unacceptability.
2. A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. And this error is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher).
3. Wilson identifies four “frames of reference” each with distinct assumptions (epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.
4. Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.”
In other words all the logical errors involved in the process render any conclusions invalid.
5. The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. And a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.
6. Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms crap in-crap out.
7. And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”
In other words it attempts to measure “’something’ and we can specify some of the ‘errors’ in that ‘something’ but still don’t know [precisely] what the ‘something’ is.” The whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?
My answer is NO!!!!!
One final note with Wilson channeling Foucault and his concept of subjectivization:
“So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self-evident consequences. And so the circle is complete.”
In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.
I only recently learned about value added metrics from a friend. From what I’ve read, it appears the “critics” of these measurements approach this topic with religious fervor and rabidly attack even the most basic and honest questions.
I’ve read some interesting studies and papers supporting the value of VAMs. Since VAMs compare a student to his/her own progress, it has the best socio-economic consideration you could possible attribute. I’ve looked and haven’t found anything to suggest they are “invalid results.” Opponents have criticized the study and pointed out that VAMs should not be used as a sole basis for teacher evaluations but I haven’t found a single school district in the entire country that tries to use VAMs as the sole basis of teacher evaluations.
The paper you cite is a 255 page rambling complaint by 70+ year old retired teacher who felt gripes about unfair categorization in his experience. None of which has anything to do with VAMs.
Your summary of Mr. Wilson’s paper
is a wordy argument that tests are invalid. Respectfully, you can make that argument better in your own words.
To the point, tests are a necessary component in assessments. Doctors use tests to determine if a treatment is working. Teachers use them to determine if students have learned assigned material. It’s cute to say you advocate for the “test-taker” not the “test-maker” but that learning is not a shared goal.
I’m a vicariously embarrassed by the information presented as “research” here. I argue that ASU is a world class institution. The obnoxious ravings with NO empirical research severely undermines my assertions. A critical and free thinker would have noticed the absence of evidence. Sadly, ASU professors and “experts” entirely miss the point.
I couldn’t agree more…
Laura Chapman, stay tuned for my post on this case. The rankings are completely valid. Yes, aptitude (as determined by the parents’ genes) plays the biggest role on the achievement score obtained by the students. But you know full well that SGPs measure growth and discount past performance. Approximately 90% of a student’s SES factors can be filtered out using prior score history so no we are not just measuring out-of-school factors in these rankings.
We have very disparate results in Loudoun County based on the quality of the teachers and principals. Many young teachers are not getting their due. Many senior teachers are “retired in place” despite $100K+ salaries (up to $130K when you factor in pension benefits). That promotion system is unfair, not the SGP rankings.
Why don’t we continue this discussion once my post is published here.
As you well know–or should know–not including student background characteristics can substantially alter the estimates and labels of teachers as low- or high-performing. So, even if we assume that the SGPs are accurate indicators of teacher effectiveness (which they are not), the fact that the SGPs do NOT include student background characteristics or other factors that are outside the control of educators. Now, I know you will argue that the inclusion of prior scores invalidates the need to control for other factors. While this is true at the individual student level, research demonstrates that the grouping of students at the classroom or school level results in biased estimates of teacher and school/principal effectiveness.
Further, I suspect SGPs dont work very well for kids scoring perfect on the test. How does one assess growth for students at or near perfect? This is an important question for teachers in wealthy districts which I believe includes Loudon county.
Also, all researchers agree that VAMs/SGPs should NOT be used in isolation. Yet, you are choosing to use them in this way. Ultimately, you will harm the teaching profession and, thus, harm students, by releasing inaccurate scores. Do you really think well-qualified people want to go into a profession in which an invalid indicator of their performance is released publicly?
Related to this point, I challenge you to publicly release your own work performance scores since you are more than willing to make parts of teacher evaluations public.You will likely say that your job is not funded by taxpayers. Yet, if your employer receives any subsidies or tax breaks, I would argue that we–the taxpayers–have a right to see YOUR performance reviews.
Thank you for your post Ed. There’s much that Virginia SGP does not understand about the research; he accordingly rides his arguments on only the research which support his a priori conclusions, all the while rejecting “truth” as evidenced via the majority of the research out there. See more coming as I’ve given him a chance to explain himself, fully aware that what we have to say on the topic will not matter much. It’s still important news and an important (albeit obtuse) perspective I think we also need to understand.
“The rankings are completely valid.”
How can that statement be true when the rankings are based on a process that is proven (and you agreed with me that there is no rebutting Wilson’s work) to be COMPLETELY INVALID? How does starting with invalidity end in quality?
Any craftsman knows that to build a quality product one must start with quality materials. Invalid standardized test scores are not a “quality material” with which to construct a quality teacher assessment product.
Ed, I agree that SGPs/VAMs are only one component that should be used in evaluating teachers. Districts in Virginia took $Ms from the feds in return for using SGPs in teacher evaluations and sharing them with the districts. Had they not received an NCLB waiver, every school would have been declared \”failing\”. California didn\’t ask for a waiver. You can\’t take a waiver and then defraud the feds. Prince William\’s teachers\’ SGPs won\’t be disclosed because they are virtually the only large district that followed the law. Seems like a lot of your anger should be directed towards the admins who knowingly committed fraud.
Duane, as you know Diane \”huge free speech advocate\” Ravitch banned all of my post after I linked to the closing arguments in this SGP case at your request. She didn\’t even wait for the (foregone conclusion) verdict. But you might be more interested in my free speech case against the Republican district attorney – Jim Plowman. He refused to prosecute LCPS attorneys for perjury (abuse of discretion but not a crime) and then blocked me when I posted on his official Facebook page. That, now, is unconstitutional. His big shot attorney has already admitted they violated the law but is having a hard time convincing the DA his political career is effectively over (not good when the DA is ruled to have violated constitutional rights). Interesting reading for others who are getting censored online on official pages.
I see the point of this as a very big for SGP regarding sunshine laws. Thank you for providing the transparency we are entitled to. Also, it is a black eye on the lower courts rulings against him.