This is another significant analysis of the use of VAM scores that are being used to make tenure and retention decisions about teachers. If you haven’t read any of Dr. O’Neil’s articles, here is a great one to start with, especially given the Vergara v California tentative decision in Los Angeles.
I think I’m supposed to come away impressed, but that’s not what happens. Let me explain.
Their data set for students scores start in 1989, well before the current value-added teaching climate began. That means teachers weren’t teaching to the test like they are now. Therefore saying that the current VAM works because an retrograded VAM worked in 1989 and the 1990?s is like saying I must like blueberry pie now because I used to like pumpkin pie. It’s comparing apples to oranges, or blueberries to pumpkins.
Figure 1. Words used to describe the statistical model used to evaluate a classroom teacher in Florida!
Earlier this week, Florida released the VAM scores of its teachers. Disturbing to say the least. The Florida Times-Union released links to teacher VAM data for the past two years. There are 116,723 teachers listed in the data base. I am not posting the link here, but it is out there.
A VAM score is a number that is derived using a covariate adjustment equation (Figure 1). The idea is to evaluate teachers using student test scores. In the Florida VAM big data release, VAM scores are reported for teachers who taught math and reading, and for those that didn’t teach math or reading. They reported next to each teacher’s name, a score that indicates the learning gains students made above or below what they were expected to learn (based on earlier performance, with OTHER teachers).
Here is equation used to figure teachers’ “value added effect.”
If you are interested here is the meaning of the equation.
Gobbledygook in Florida
Gobbledygook is nonsense, gibberish, balderdash, rubbish, or if you prefer garbage.
When you step back and think about your experience as a teacher you have been insulted by the antics of the Florida officials who financially supported and then hired guns to design and carry out the Florida VAM model, which in my estimation is Gobbledygook.
The reports that we read in the media do not go into any detail about how teacher VAM scores are determined. If the public really knew what school districts were doing to their children’s teachers, they would be furious.
Parents know that their child’s knowledge of science or history is not a number that emerges from a test given once a year. Parents also know that their children’s teachers should not be judged by a single numbers, especially when what goes into making the number is not only spurious, but lacks the credibility of the education research community.
It is true that you can go to an 80 page document (Florida Value-Added Model Technical Report), and read through the details of the VAM model that is used on teachers. Upon reading the paper, I became further outraged at the ends powerful officials will go to drum up a system that does not contribute one iota to the improvement of the work of professional teachers, or to the improvement of learning among students. What are they think?
Even the Bill and Melinda Gates Foundation is opposed to the release of the Florida VAM scores. But, its a bit late. They contributed millions of dollars to finance the development of VAM. Others have reported that the scores of 70% of the teachers were based on the scores of students they didn’t teach (See Valerie Strauss’s Blog).
The use of VAM to evaluate teacher’s is flawed, unreliable and invalid. To post scores based on “junk” science, is preposterous.
Chris Guerrieri Speaks about VAM
Finally, I want to include what Chris Guerrieri, a teacher and blogger in Florida has to say about VAM scores being released in his state. In his recent blog post, he quoted two Florida superintendents. Joseph Joyner, superintendent of a high scoring district said this:
I cannot express enough, my disappointment in the decision to publish VAM data, in any form. The push to create simplicity (test scores) out of an inherently complex process (teaching) is rooted, in my opinion, in the desire of media and policy makers to create lists with the ultimate goal of allowing for judgment In the end, we continue to treat teachers like sheep, being herded into a gate to have a number pinned to their ear. I question this treatment of professionals as we owe the success of our state and nation to great teachers, and the lack of respect and loss of dignity is appalling. (Guerrieri, Chris. Chris Guerrieri’s Education Matters. The Real Reason the FLDOE Fought the Release of VAM Data., 28 Feb. 2014. Web. 28 Feb. 2014.)
Guerrieri has written extensively not only about VAM, but other issues on Florida education. He suggests, that even though the Florida Department of Education (FLDOE) fought having the scores released, the reason was to protect themselves from the bad policy that they promoted. The release of the scores would embarrass the FLDOE. Here is how he put it:
Finally how could any reasonable education body think it was good policy to score teachers on subjects and or students they did not teach? It is unexplainable and indefensible but Florida’s depart of education did it anyway. I believe that, not because they were concerned about the public only getting half the picture and not because they were concerned about destroying teachers morale is why they fought to keep VAM scores from the public.
When the scores were released at first I was outraged. I believe the FLDOE who is an operative of the privatization agenda wants people only to get half the picture and them to take things out of context and for some sadly they will be that case. But after a few days of reflection I am glad they released the scores because now it will show those paying attention how incompetent the FLDOE, the commissioner and the State Board of Education are. (Guerrieri, Chris. Chris Guerrieri’s Education Matters. The Real Reason the FLDOE Fought the Release of VAM Data., 28 Feb. 2014. Web. 28 Feb. 2014.)
What is your opinion on the use and publishing of teacher VAM scores in Florida?
People are asking for better schools, with no clear idea how to improve eduction, or even how to define improvement of education (except to increase test performance on high-stakes tests).
Most people are in favor of improving education. But when asked how would they improve education, the suggestions are insufficient, and in some cases, even negative (See W. Edwards Deming. The New Economics for Industry, Government, Education (p. 8). Kindle Edition.)
Instead of reporting the details of how people and organizations want to improve education, such as corporate chiefs, philanthropists, and the U.S. Department of Education, I want to report on the work of Mr. Ed Johnson, an advocate for quality education, and has for more than a decade devoted himself to writing and talking about improving education in the Atlanta Public Schools.
Ed Johnson consults as Quality Information Solutions, Inc., with a commitment to human social and cultural systems to receive quality information from information systems for the continual improvement of life, work, and play. His commitment extends to advocating the transformation of K-12 public education systems to humanistic paradigms from prevailing mechanistic paradigms. Ed also is former president of Atlanta Area Deming Study Group.
Ed Johnson is a systems thinker.
In this regard, he believes schools can not be improved by trying to improve the parts separately. It is a sure path to failure. For example, some advocates of educational reform believe that student achievement can be improved by weeding out the bad teachers. Millions of dollars have been invested in using student high-stakes test scores to check teacher performance using a technique called Value Added Measure (VAM). Teachers whose VAM scores are low can be identified, and according to these experts, teachers with low scores must be bad teachers. Getting rid of “defects” in any system will not improve the system or the part that was identified. Instead, a better investment would be to ask how can we improve the quality of teaching, and what can be done to improve the teaching of all educators.
The above example highlights the current approach to reform. Identify a part of the system, and fix it. Bad teachers, get rid of them. Low achievement scores? Write “rigorous” standards, raise the bar, and give high-stakes tests. It’s that simple. We’ve had rigorous and not so rigorous standards in place for more than a decade, and as you will see ahead, changing standards doesn’t have any effect on student performance.
Systems thinking means that all parts of a school system are interdependent and must be taken as a whole. The Atlanta Public Schools (APS) is a system of interconnected and interdependent parts, and to improve the quality of the APS, it is critical to look at the APS as a whole. For example, closing schools (removing so-called underperforming schools), does not have an effect of improving the APS, or indeed saving money (as some would tell you). Fundamental questions about APS need to be asked, but in the context of the APS being a system, not a collection of schools, students, teachers, administrators, parents, curriculum, textbooks, technology.
Ed Johnson has contributed to my understanding of quality education, and it is my great honor to share his work on this blog, and in particular to look at teaching in urban schools, and in particular the Atlanta Public Schools.
Each story is about a system. In TUDA (Trial Urban District Assessment), there are particular TUD’s each as a system. So, Reading is a system. Mathematics is a system. 4th grades is a system. 8th grades is a system.
Johnson’s research is a longitudinal study of performance in reading and mathematics from 2002 – 2013. Using scores on reading and mathematics obtained from the National Center for Educational Statistics, he investigated the nature of a number of systems derived from the data. Some of these particular systems include:
Reading as a System
Mathematics as a System
4th Grades as a System
8th Grades as a System
Since these systems are part of the APS system, we know that each of these systems is interdependent with other systems, not just the ones identified here, but including parents (as a system), teachers (as a system), and so forth.
There were 21 urban school districts in the study. However, Ed has managed to make our work easier by highlighting with color coding just two systems, Atlanta Public Schools (Red) and District of Columbia (Purple).
Ed starts his study by giving us the “bottom line.” How did these systems (reading, math, 4th grade, 8th grade) do? Figure 1, is a summary of systemic TUD student performance in reading and math at the 4th and 8th grade from 2002 – 2011, and predictions for 2013 (all the predictions were accurate forecasts of student performance in 2013).
Only 4th grade reading showed some improvement over the period 2002 – 2011, and the improvement was slight and noticed only in Austin, Charlotte, and Hillsborough. In all other systems, no improvement was observed, meaning that the common causes that influence the system of math, or reading, or 4th or 8th grade inhibited improvement.
Student Improvement in Mathematics and Reading
In the TUDA study, a sample of students in each urban district was tested in reading and mathematics at the 4th and 8th grade level. To help us understand how to interpret data collected over the past dozen or so years, Mr. Johnson has produced a series of graphs (control charts) showing the natural variation of scores to be expected in each system (reading, math, 4th grade, 8th grade).
Figure 2 shows a control chart for reading, 4th grade. Figure 3 shows a control chart for mathematics, grade 4. Upper control limits and lower control limits were calculated for 2002, and then projected forward. Changes in scores from one test period to the next are shown in the Figure 2. If there is systemic change in reading at the 4th grade level, then scores would fall “outside” the upper or lower control limits. You’ll notice that all the variation, except for four points (Charlotte, 2009 and 2011, Austin, 2011, and Hillsborough, 211), was within the variation expected. In systems thinking, we mean that the variation for the most part was random, but there is evidence that some special causes were at work in the three districts mentioned here.
Mathematics is another story. As Mr. Johnson puts it in his study, “all districts have been on the same boat continuously since 2003 in mathematics at the 4th grade level. What means is that the variation shown in the graph is random, and not due to any special cause.
There is very little student improvement in reading or mathematics at the 4th grade level as shown in Figures 2 and 3.
As long as we continue to ignore the common causes of variation that exist in the system then we can expect very little to no improvement.
But as Mr. Johnson has said in other letters and reports, if fundamental questions about the purpose of schooling are not addressed and if we can not agree on these purposes, very little will change in the system. In the two systems explored here, reading at the 4th grade and math at the fourth grade, we need to ask: What is purpose of teaching reading in the elementary school? Why do we teach reading in the elementary school? What is goal of teaching mathematics in the elementary school? Why do we teach mathematics?
As Mr. Johnson has shown, why are these districts on the same boat for the teaching of mathematics? How can we used systems theory to look at mathematics teaching as a system and answer questions about how to improve mathematics learning? How can help students develop a love affair with mathematics?
Ed Johnson has examined a lot of data from the standpoint of systems thinking based in part of his work with Edward Deming, and other scholars in the field of systems thinking.
In the days ahead, I’ll revisit Mr. Johnson’s study, and report on his analysis of the “performance gap” variation that he has depicted as a series of images as shown in Figures 2 and 3. I’ll also explore systems thinking and school in more detail.
We’re spending a lot of money to find great teachers in Georgia. Admittedly, Georgia isn’t the only state where this is happening. Thanks to the Race to the Top Fund, the race is on to implement a system to score teachers based on pre-post test scores.
The $4 billion Race the Top Fund, which was distributed among 11 states and the District of Columbia, specified the reform goals that applicants had to include in their proposals. The reform goals were grouped into six categories, plus a seventh category which called for making a priority of STEM (Science, Technology, Engineering and Math). The categories and the relative value (total of 500 points) placed on them by the reviewers were these:
Great Teachers and Leaders (worth 138 points)
State Success Factors (125)
Standards and Assessment (70)
General Selection Criteria for Charters (55)
Turning Around Lowest-Achieving Schools (50)
Data Systems to Support Instruction (47)
How to Spend $400 Million
The relative scoring that was used to rate RT3 proposals prioritized the goals for educational reform. Georgia received nearly $400 million as shown in Figure 1. The most funding was allocated to the category of Great Teachers and Leaders (other than the amount allocated to the 26 LEA’s that are participating in the grant).
Nearly 30% of the RT3 budget has been allocated to the development of Great Teachers and Leaders. What is a great teacher? What is a great leader? Does the Georgia Department of Education (GDOE) have a plan that will encourage teachers to be creative and innovative, two of the qualities that many would claim are indicators of a great teacher.
Later in the post, we’ll find out that the Race to the Top administrators have something else in mind.
But, in the meantime, what is a great teacher?
In order to answer this question, I am going to write about the work that Dr. Peter Smagorinksy, Distinguished Research Professor of English at the University of Georgia (UGA) has done to highlight examples of Georgia teachers who understand what it takes to work with youth in our public schools. Dr. Smagorinksy has written several essays that are part of the Georgia Great Teacher Series. His essays engage us in the lives and professional work of a number of Georgia teachers that will help us understand what great teaching is, and who he considers to be a great teacher.
His essays are important in light of the present drive in Georgia to test teachers using simple metrics such as student test scores. In fact, in a recent report published by the inept organization, the National Council of Teacher Quality (NCTQ), it says that student test scores should be used in all matters of education including teacher salaries, tenure, and teaching licenses. It’s an organization that believes that the only aspect of schooling that has worth is academic achievement as measured on bubble tests.
Unfortunately, this belief is held by the U.S. Department of Education, and is articulated through the $4 billion Race to the Top Program. There is more to learning than how well our kids do on achievement tests.
Teachers sure are taking a beating these days. Not all teachers, however. If youre in a private school or charter school, you must be pretty danged good. No, its just the teachers with the hardest jobs who get the abuse heaped on them, day after day, by the great and small, named and anonymous: those in regular old public schools.
Teachers are subjected to increasingly urgent calls for accountability, no doubt because those high salaries and other cushy benefits need to be justified in the most rigorous, reliable, and valid of ways. No, make that way, not ways: The only measure of successful teaching these days is their students test scores. Everything that teachers do for kids can easily be boiled down to those scores, regardless of whether those kids have fridge full of healthy food or a clean change of clothes at home; or for that matter, a home at all.
Or, maybe teachers can be appreciated for other things that they do. In this series of columns, Id like to feature teachers I know of who do extraordinary work, often with kids whose life circumstances do not predict college attendance or other arenas where test scores matter to them enough to do their best. What I hope to accomplish is to provide profiles of outstanding teachers without referring to their ability to train students to fill in bubbles on machine-scored answer sheets.
I was always pleased to find him in my classes because of the ripple effect he had on other students. Its pretty hard to be in a class with someone of boundless inquisitiveness and vigor, and not get caught up in the momentum yourself. He set a high standard for engagement and participation that inevitably gave the classes vitality and purposecertainly for me as a teacher, and I believe also for the other students in the class, many of whom, like David, were coming to campus after a demanding and often exhausting day of teaching their own classes in Georgia schools.
As a teacher, David has “taken a special interest throughout his teaching and education in the quality of learning experienced by students from low socioeconomic status groups. In Athens, which is one of the poorest counties in the country, he was a leader in his passion for social-justice, and lead the way in founding the Red Clay Writing Project, which brought together teachers from North Georgia who became leaders in their respective schools, especially as it connected to the National Writing Project.
But, as Dr. Smargorinsky points out, David Ragsdale most remarkable achievement was his founding of two magazines at Clark Central High School, the Odyssey (news) and the Iliad (literary). Dr. Smargorinsky says:
David saw the need for students to take pride in and have outlets for their writing, and so revived the Iliad and launched the Odyssey. Taking this initiative in a school in which student writing was so little appreciated reveals much about Davids spirited optimism and faith in his students in a setting in which many have simply given up on students prospects for achievement. Serving as advisor to one or the other of these publications would be an onerous amount of work; founding and advising both while pursuing graduate studies and being a key player in the RCWP is simply remarkable.
Each of these magazines has, under Davids dynamic leadership, risen to national prominence in very short order.
There are hundreds of teachers who have the same kind of enthusiasm for their students, and for the work they do in their respective schools. When we look deeply into the professional work of teachers, it’s a disservice not only to our teachers, but more definitely to our student if we believe that achievement tests scores are what education is about.
Here are some additional biographies of Great Teachers by Dr. Smagorinsky.
In Georgia’s Race to the Top program, these teachers could teach the administrators of the RT3 a lot about Great Teachers and Leaders. But would they really listen.
I don’t think they would.
$60 Million on Great Teachers?
If you take a look at how the RT3 is spending nearly $60 million on Great Teachers, you might scratch your head, and wonder, what are they thinking? Figure 2 shows how the funds are used in the Great Teachers and Leaders work plan.
Notice that nearly $16 million will be spent developing a system to evaluate teachers using student achievement test scores. The state has begun the development of a Value Added Measure (VAM) that will be used to evaluate and rate teachers. In fact, our esteemed Georgia Legislature has passed a law requiring that at least half of a teachers evaluation must be based on student test scores.
What about Art Teachers?
But don’t stop with VAM. Look more closely at the budget. I’ve marked up the Great Teachers RT3 budget (Figure 3). I’ve highlighted three aspects of the budget. At the top, note that there is funding for STEM training and for increasing the “pipeline” of science and math teachers. This is one of the cornerstones of the Federal RT3, and Georgia has included it in their work plan. Now keep in mind that I am a science educator, and have been for a very long time. But I must ask, why are there no funds for art, or history, or music, or literature?
Focusing attention only on math, science and English/language arts is a mistake. There are many questions that need to asked about our increasing emphasis on these subjects, and at the peril of the arts, health, history, music and physical education.
A second part that I’ve highlighted is the categories listed for performance pay for principals and teachers. There is nearly $10 million for performance pay. Performance pay is a bonus that will be determined by how well students perform on achievement tests. Yet in studies done on the effect of performance pay schemes on student achievement, the results do not favor bonuses. In fact, we can ask serious questions about the cheating scandals that have cropped out in many school districts around the country, all of which put a premium on administrators getting the most out of its students—that is, their score on achievement tests. Check out this report from Vanderbilt University.
And then, take a look at the bottom of the graph, and note that nearly half of the Great Teacher budget is connected to using VAM as the measure of teacher effectiveness.
A large group of Georgia professors prepared a letter which was sent to key politicians including the governor, the GA state school superintendent, and the superintendents of 29 school systems. The letter challenges the teacher and leader evaluation system, identifies the unintended negative consequences, and recommends the state opt out of this invalidated and unreliable system. These research professors provided data to support the following conclusion/recommendations for using VAM scores.
Value Added Models are not proven;
GA is not prepared to implement this evaluation model;
This model is not the most useful way to spend education funds;
Students will be adversely affected by this Value Added Model.
Student achievement tests were not designed to measure teacher or administrator performance. They were designed to measure student learning, but for years researchers have warned not to try and make connections between student test scores and teacher performance. The Value Added Measure (VAM) is just as unreliable as the student test scores used to generate a VAM number. According to the Georgia Professors’ Letter, No evidence exists that evaluation systems that incorporate student test scores produce gains in student achievement. In order to determine if there is such a relationship, researchers recommend long-term, small-scale pilot testing of such systems. Furthermore, student test scores have not been found to be a strong predictor of the quality of teaching as measured by other instruments or approaches.
Georgia is headed down a path that will end up creating more problems that it solves. This plan will not improve student achievement, nor will it lead to the improvement of the quality of teaching. What it will do is demoralize teachers who have worked for years to improve what happens in the classroom.
Yet the myth persists that the teacher is primarily responsible for student scores and that great teachers can overcome the influence of family, poverty, disability status, language proficiency, and students own levels of interest and ability. Certainly, there are many people whose lives were changed by one teacher, but their stories typically describe teachers who were unusually inspiring, not the teacher who raised my test scores to the top. Teachers do have the power to change lives. But after more than a decade of No Child Left Behind, researchers are still searching for a nonselective school or a district where every student, regardless of his or her starting point, has achieved proficiency on state tests because that school or that district has only effective teachers. (Ravitch, Diane (2013-09-17). Reign of Error: The Hoax of the Privatization Movement and the Danger to Americas Public Schools (Kindle Locations 2144-2145). Knopf Doubleday Publishing Group. Kindle Edition).
What do you think? Are we heading down the wrong path by spending nearly $60 million on Great Teachers as described in Georgia’s Race to the Top?
“High-stakes plan could change teachers pay,” is the title of an article in the Atlanta Journal-Constitution paper. The article is a good review of the history behind why the state is going to move to “merit” pay, and how the Race to Top grant that Georgia won in 2010 provided the funds and a mandate to pay teachers on the basis of performance. But earlier this year, the Superintendent of Education, Dr. John Barge, a Republican (who has announced he will run for governor in 2014 against Republican Nathan Deal), said that the merit plan will not be implemented in the 26 districts that are part of the Georgia’s RTT.
However, Marietta City and Fulton County Schools, who are part of the RTT, are charging ahead with a merit based plan, and have hired the same Boston company to help them spend their money.
Using merit pay plans, especially if they are tied to student achievement scores, are based on incorrect conclusions.
Student achievement scores on high-stakes examinations are the preferred strategy of paying teachers for performance, or service rendered to its employer. The idea here is that if students do well on achievement tests, then the teacher should be rewarded–more pay. On the other hand, if the kids don’t do so well on the tests, then what? Punish the teacher? Less pay?
In Marietta, Georgia, the Marietta Board of Education has announced that it will freeze teacher salaries, and “float” the idea among faculty and staff of the school district that it might be better if they were paid based on performance and not on experience and their education level.
Marietta and Fulton County teachers are in jeopardy because the plan that the school boards and superintendents want to put in place to decide teacher compensation will be based on unscientific and unreliable methods. I’ve reported horror stories describing how outstanding and experienced teachers received poor evaluations here, here, and here. The plan will use student achievement, and other methods as shown in Figure 1.
Student achievement tests were not designed to measure teacher or administrator performance. They were designed to measure student learning, but for years researchers have warned not to try and make connections between student test scores and teacher performance. The measure that is actually used is the Value Added Measure (VAM), and it is just as unreliable as the student test scores used to generate a VAM number.
Many of you may be familiar with VAM, or Value Added Modeling. This is a teacher evaluation system that uses student test scores to evaluate teachers’ performance. The data used for Louisiana’s system is fraught with errors. The premise behind the system is flawed. And as it turned out, Louisiana’s system was corrupted on the inside for political reasons.
Diane Ravitch reported on her blog that researchers at the Atlanta Journal have uncovered a “surprising number of errors, though not surprising to those familiar with the testing industry.”
teachers who taught more remedial classes tended to have lower value-added scores than teachers who taught mainly higher-level classes. “That phenomenon was not due to the best teachers’ disproportionately teaching the more-rigorous classes, as is often asserted. Instead, the paper shows, even those teachers who taught courses at more than one level of rigor did better when their performance teaching the upper-level classes was compared against that from the lower-level classes.”
Georgia is headed down a path that will end up creating more problems that it solves. This plan will not improve student achievement, nor will it lead to the improvement of the quality of teaching. What it will do is demoralize teachers who have worked for years to improve what happens in the classroom.
Yet the myth persists that the teacher is primarily responsible for student scores and that great teachers can overcome the influence of family, poverty, disability status, language proficiency, and students’ own levels of interest and ability. Certainly, there are many people whose lives were changed by one teacher, but their stories typically describe teachers who were unusually inspiring, not “the teacher who raised my test scores to the top.” Teachers do have the power to change lives. But after more than a decade of No Child Left Behind, researchers are still searching for a nonselective school or a district where every student, regardless of his or her starting point, has achieved proficiency on state tests because that school or that district has only effective teachers. (Ravitch, Diane (2013-09-17). Reign of Error: The Hoax of the Privatization Movement and the Danger to America’s Public Schools (Kindle Locations 2144-2145). Knopf Doubleday Publishing Group. Kindle Edition).
Marietta and Fulton County teachers, this is a heads up.