The Atlanta Cheating Scandal: Suspicions Raised About the AJC Investigative Methods

A Fulton County grand jury indicted former Atlanta Public Schools Superintendent Beverly Hall and 34 others — top aides, principals, teachers and a secretary — for racketeering as well as theft by taking for the bonuses they received for good test scores or making false statements or writings, charges that provided the basis for the Racketeer Influenced and Corrupt Organization count.

The indictment came about 20 months after Governor Deal released the results of an investigation carried out by two attorney’s appointed by former Govern Sonny Perdue and an investigative team of about 50 GBI agents who fanned out into schools and classrooms to interrogate educators who were suspected of involvement in the “Atlanta Cheating Scandal.”

As stated in the Governor’s Investigative Report a “culture of fear” took over the Atlanta School System, and led to a conspiracy of silence which enabled the bubble sheet erasure scandal to happen.

In light of the grand jury indictments I am re-posting a blog article published a year ago on March 26, 2012.  My post was a reaction to an article written by the Atlanta Journal Constitution cheating investigative team which has published 30 articles on the cheating scandal not only in Atlanta, but throughout the country over the past five years.

Although there was cheating happening in some Atlanta schools, the data analysis conducted by the AJC has been questioned by testing experts, and they suggest that the AJC should have been very clear that other reasons could have accounted for wide swings in test results other than cheating.

Here’s the article published last year

Suspicions About the Atlanta Journal’s Investigation into Cheating Across the Nation 

| March 26, 2012 | Filed under: Assessment

On a Sunday in March 2012, the Atlanta Journal-Constitution (AJC), a Cox newspaper published the results of its investigation into “cheating” in American schools.  The article was entitled Suspicious Scores Across the Nation, and you can read it by following the link. The article was subtitled “Cheating Our Children.”

I was immediately suspicious of the report that the AJC published.  They have put into place an aggressive team of watchdog reporters, database specialists, and investigative reporters.  In whose interest motivates this team and this newspaper?  After reading the report, I reaffirm my suspicions.  Let me explain.

The data (collection of facts, observations, and measurements) that is available to this team is a database specialists’ fantasy.  Fifty-states provided standardized testing data from 69,000 schools, 14, 743 districts, 13 million students, representing 1.6 million records.  Most states provided the data immediately (public records law), but a few Departments of Education hedged a bit, but eventually sent on their data.

Comparing Average Scores, Not Individual Students

According to the researchers,

For each state, grade, cohort and year, we created a linear regression model, weighted by the number of students in a class, and compared the average score for a class with the score predicted by the model based on the previous year’s average score. We then calculated a p-value — an estimated probability that such a difference would occur by chance — using standardized residuals and the “T” probability distribution, which adjusts the probability upward for classes with fewer students. (Links Mine)

What the researchers looked for was scores rising or falling with probabilities less than 0.05.  These were flagged.  Maybe the bubble sheets were erased and correct answers added?  This would mean that some one cheated.  Wouldn’t it?  Or could there be other explanations?

Is cheating what caused the some scores to change at a probability level that you wouldn’t expect.  According to the Journal-Constitution,

A statistical analysis cannot prove cheating. It can only identify improbable events that can be caused by cheating and should be investigated.

If it smells like cheating, it must be cheating

But if you read the AJC’s initial article, and the one they published today (Cheating our children: AJC’s testing investigation spurs action), the only suggestion that the Journal makes is that the T-scores might indicate cheating.  And indeed, in the latter article, Georgia U.S. Senator Johnny Isakson said that this report is troubling, and if the districts don’t do something about it, then the Governors should.  If they don’t, then “I don’t think Congress should look the other way.” I am sorry Mr. Isakson, but the Congress has looked the other way since the NCLB Act was signed into law ten years ago.

Here is what the Atlanta Journal staff had to say about their research model:

A statistical analysis cannot prove cheating. It can only identify improbable events that can be caused by cheating and should be investigated.

Ideally we would look at how individual student test scores change from year to year, but federal privacy regulations precluded access to that data. The approximate cohorts we used were the only available substitute. It is unlikely that two groups of students in a cohort are perfectly identical. Urban districts in particular have high student mobility.

In the model that the data analysts used, average scores from one year to the next were not necessarily based on the same population of students.  They didn’t have student data.  The only had average data for a school by subject. So it is possible that they are making predictions based on unreliable data.

High student mobility might give us a clue about other possibilities to explain score changes given the demographics of the schools and districts that were highlighted.  For example if you look at the districts that were highlighted in the article (Amarillo, Atlanta, Baltimore, Chicago, Dallas, Detroit, East St. Louis, Fresno, Gary, IN, Houston, Los Angeles, & Mobile County, AL), these districts reported that at least 64% of the students were eligible for free or reduced-price meals.  The poverty concentration of these schools averaged 80%.  Mobility is high in these districts.

Critique of Methodology used by the Atlanta Journal Data Analysts 

Gary Miron, professor of education at Western Michigan University was one of four academics or test specialists that advised USA Today and its affiliated Gannett newspapers on a multi-state analysis of irregularities in assessment data.  In his article published on The Answer Sheet, Dr. Miron writes about some of his concerns about the research method used by the Atlanta-Journal Constitution.

My review, however, yielded serious concerns about the data used, the methods of analysis employed, and the conclusions drawn. I shared these concerns with journalists at the Dayton Daily News, which is one of the Cox affiliates involved in this story.

To be clear, the Cox analysis may accurately detect large variations in assessment results from year to year. But my analysis of the data suggests that these irregularities are less likely due to real cheating than due to mobility in student population (recall the lack of student-level data). Although the Cox news articles on this study offer a disclaimer that their analysis does not actually prove cheating, this disclaimer should be expanded considerably.

The evidence is that the AJC may be practicing sensationalism by throwing the data up against the wall, and hoping that some of it will stick.  The map they published is impressive, but it raises more questions than it answers, and the writers of the article were quick draw the “cheat” card. “Cheating” was mentioned 57 times in the first article of 3000 words, and 31 times in the second article, which was 978 words long. Mobility or any mention of changing student populations was mentioned once in both articles.  One might raise the issue that the publication of this article is self-serving.  Are they really interested in uncovering serious issues facing our nation’s schools, or are they more interested in selling newspapers, and receiving awards for investigative reporting.  I really don’t know.  But the thought crossed my mind.

Dr. Miron conveys his concerns about the method used by the AJC.  Here are his concerns:

  • As noted, the analysis is based on school-level data and not individual student-level data. Accordingly, it was not possible to ensure that the same students were in the group in both years.
  • The analysis of irregular jumps in test scores should have been coupled with irregularities in erasure data where this data was available.
  • The analysis by Cox generates predicted values for schools, but this does not incorporate demographic characteristics of the student population.
  • The limited details available on the study methods made it impossible to replicate and verify what the journalists were doing. Further, the rationale was unclear for some of the steps they took.
Stop Weighing the Cow, and Start Teaching Students to Think, Problem Solve, and Do Inquiry
Since NCLB was enacted in 2002, all students and schools have been subjected to high-stakes testing.  Its been relentless, and it has had it’s effect on teaching and learning.  Teaching has been reduced, especially in high poverty schools, to teaching to the test.  Students in many of these schools have a full diet of worksheets, drill and practice, and fill in the blanks.  Inquiry, and problem solving are nowhere to be seen.  It must be seen, however.
In her new book, Multiplication is for White People: Raising Expectations for Other People’s Children, Dr. Lisa Delpit made this important statement early in her book:

I am angry at the machinations of those who, with so little knowledge of learning, of teachers, or of children, are twisting the life out of schools.

She adds that the current use of standardized tests, which promotes competition between schools, and used to evaluate teachers and principals to decide their salaries bring out the worst in adults.  It should be no surprise to the Atlanta Journal staff, but they fail to pursue the data other than to blame schools for the current state of our schools. This is not about cheating.

According to Dr. Delpit, “the problem is that the cultural framework of our country has, almost since its start, dictated that “black” is bad and less than and in all arenas “white” is good and superior.”

To point to urban schools as the harbinger of cheating and surprising test results leaves us with many questions.

  • Why don’t we talk about how we got ourselves into this mess in the first place?
  • What has been the effect of the NCLB policy that has turned schooling into test factories, only interested in finding out what low-level of knowledge students might have learned in school?

We simply can not continue to test with out really reforming education.  And simply writing common standards (goals) for all kids regardless of where they live, and with little teacher flexibility or comments from teachers simply will only reinforce the authoritarian nature of schooling, but do little to improve the lives of teachers and students.  Testing students doesn’t make them smarter, any more than weighing the cows makes them heavier.

I agree with Dr. Delpit when she calls for us to create excellence in urban classrooms, and she suggests that we must do the following:

  • Recognize the importance of a teacher and good teaching, especially for the “school dependent” children of low-income communities.
  • Recognize the brilliance of poor, urban children and teach them more content, not less.
  • Whatever method or instructional program is used, demand critical thinking while at the same time assuring that all children gain access to “basic skills”—the conventions and strategies that are essential to success in American society.
  • Provide children with the emotional ego strength to challenge racist societal views of their own competence and worthiness and that of their families and communities.
  • Recognize and build on children’s strengths.
  • Use familiar metaphors and experiences from the children’s world to connect what students already know to school-taught knowledge.
  • Create a sense of family and caring in the classroom.
  • Monitor and assess students’ needs and then address them with a wealth of diverse strategies.
  • Honor and respect the children’s home cultures.
  • Foster a sense of children’s connection to community, to something greater than themselves. (Delpit, Lisa (2012-03-20). “Multiplication Is for White People”: Raising Expectations for Other People’s Children . Perseus Books Group. Kindle Edition.)

What do you think about the Atlanta Journal’s inference that cheating might be the reason for the unexpected changes in test scores?  Do you think the questions raised in this post have implications for the Atlanta educators indicted by the Fulton County grand jury?

K-12 Education Viewed Through the Lens of Conservative Values & World-View

Note: this is the second in a series of posts on the effect of conservative & progressive values on K-12 education.

Progressive values should set the ideals of teaching, and learning in American society.  Unfortunately, the “cloud of authoritarianism looms over education, making it very difficult to design instruction around progressive values.  We’ll refer to those who support this latter view as the “authoritarians.”

In order to understand the authoritarianism that has claimed schooling in America, we can glean insights into the effectiveness of this approach by using the values that conservatives use to frame their issues, and make their policy decisions. As George Lakoff says,  “radical conservatives have taken over the reins of government and have been controlling the terms of political debate for many years” (Lakoff, George (2006-10-03). Thinking Points: Communicating Our American Values and Vision (Kindle Locations 131-133). Macmillan. Kindle Edition).

Conservative World View

“World view refers to the culturally-dependent, generally subconscious, fundamental organization of the mind,” according to William W. Cobern, who has done extensive research on world-view and how it impinges science teaching.  Ones world view predisposes one to feel, think and act in predictable way, according to Cobern.  World-view inclines one to a particular way of thinking.

In my argument about trying to explain the current state of schooling in the U.S., we will look at two different political and social world-views, the conservative world-view, and the progressive world-view.  Both world-views have played significant roles in American history, including public education.  In this post we’ll examine the conservative world-view and how it has affected the way in which public schools determine curriculum, hold schools accountable for achievement of its students, and the effectiveness of schools and teachers.

According to research by George Lakoff and the Rockbridge Institute, the moral world-view of either conservatives or progressives can be understood by using the conceptual metaphor of Nation as Family.  Using this idea, ones political beliefs tend to be structured by how we think of family, and our early experiences in our own family which contribute to our beliefs.  Thinking of a nation as a family is a familiar notion, as in phrases such as Mother Russia, Fatherland, sending sons and daughters off to war, the founding fathers, Big Brother (see Joe Brewer, Rockbridge Institute, discussion here).  In Brewer’s thinking, the conceptual metaphor of nation as family organizes our brains in this way: homeland is home, citizens are siblings, the government (or head) is parent, and so forth.  The diagram below shows the organization of schooling according to a conservative world-view.

Continue reading “K-12 Education Viewed Through the Lens of Conservative Values & World-View”

More than 90% of Students Learned in Spite of the CRCT Erasure Scandal?

Are you surprised?

You probably know that Atlanta Schools are in the middle of a test cheating scandal in which student bubble answer sheets were changed by erasing wrong answers to right answers.  Did the students learn, in spite of some teachers’ and administrators’ behavior.  They did because the the teaching practices that were initiated, especially in reading and English/language arts, seem to hold as evidenced in CRCT test results the year AFTER the scandal.

Read ahead for more on this. You’ll find the results interesting…..

Parks Middle School. In the Atlanta bubble test erasure investigation, Parks Middle School was center-stage in the investigation.  According to the report, “cheating” occurred from 2005 – 2009. According to the report, the principal conspired with other administrators and some teachers to systematically changed answers on student bubble tests during these, and made an effort to keep this from the test coordinator.

But, during this period of time Parks was held up as a model of how to turn around an urban school. In fact a lengthy report in the form of a published paper (here) of Parks’ efforts and successes was included in the Governor’s Investigative report. Parks was involved in many innovative curriculum efforts designed to help students achieve success.

I examined the data at CRCT website (Georgia Department of Education) for a three year period, 2008 -2010. I wanted to find out how the scores changed (if at all) in 2010 in each subject area. As you can see in the areas of Reading and English/Language Arts Parks more than 90% of Park’s 8th graders met or exceeded the state target, even after the year when “cheating” was discovered.  In the areas of math, science and social studies, we do see an appreciable decline in CRCT results in 2010.

At Parks Middle School, the increase in reading scores rose dramatically from 2004 from 35% to 74%, and then to 98.5 in 2009.  According to the Governor’s investigative team, the scores in 2010 (the year in which we can be certain there was no cheating), students in the 8th grade at Parks still scored above 90%.  The same is true for English/Language Arts.

Why Parks’ Students Scores Increased Dramatically.  In a paper describing the Parks’ story of success, the dramatic gains in student test scores was attributed to effective leadership, data-driven planning and instruction, high expectations, strategic partners (corporate sponsors), increased student discipline, and more professional development.  There is evidence that all of these did indeed occur, although some might argue with the “effective leadership” attribute.

These dramatic increases in student performance were lauded, locally and nationally, and Parks received many awards, and enormous financial support during this period.  Superintendent Hall praised the work of the principal, Christoper Waller, and both were recognized for creating conditions that made learning successful for poor children.  Specialists in reading, special education and other areas were hired to provide staff development and instruction for students. Waller launched Project GRAD at Parks Middle School, a reform model that included professional development for teachers, on going support, coaching and re-training.  Twenty-five Atlanta elementary, middle and high schools currently participate in Project GRAD.   Project GRAD is a national program, and is in place in more than ten cities around the country.

Georgia State Department Involvement. The Georgia Department of Education was involved with Parks Middle through the NCLB “Needs Improvement” schools support.  The state assigned Dr. Cheryl Hunley to serve with Parks and six other area schools.  Working closely with the principal, she provided professional development, and worked very closely with the entire staff at Parks.

In addition to these two major sources of professional development, Parks was also part of the SRT 2 (School Reform Team 2), an initiative begun by Dr. Hall which was led by an executive director who oversaw several schools.  Training, support, and assistance was localized with in the district through SRTeams.

There is no doubt Parks was involved in innovative school improvement.  And given, the data that is shown in the Figure 1, we can conclude that these efforts must have contributed to some of the gains shown in student CRCT test results, especially in Reading and English/Language Arts.

Test Results. The results in Math, which did decrease in 2010, are disappointing.  The scores in science and social studies show the greatest losses.  But I remember several years ago that Dr. Hall was quoted as saying that there is no way that students will do well on the NAEP Science Test with out Reading and Math.  She indirectly was saying that schools should emphasize reading and math to the exclusion of science, and perhaps social studies.

The data reported by the Investigative Team of the Governor’s Office, and the CRCT data for these three years does not answer all of the questions.  Teachers may have cheated in changing student scores, but students did learn and improve, and they need to be informed that all of their gain was not due to teacher’s changing their papers.

Parks Middle School Reading English Language Arts Math Science Social Studies
2008 93.5 94.4 81.5 49.2 79
2009 98.5 96.9 85.4 58.5 66.9
2010 94 89.4 70.2 35 28
Average 95.3 93.5 79.0 47.5 57.9
Figure 1. Percent of Students Who Met or Exceeded the CRCT State Mandated Standard by Subject, 2008 – 2010 at Parks Middle School.  Note: 2009 was the year the Governor’s Office investigated excessive erasures in the APS.  In 2010, there were few, if any, erasures on bubble tests. 

Why did this happen?  What happened in Atlanta is still an open question. The interim Superintendent is moving swiftly to replace administrators and staff mentioned in the Governor’s Report, including the administrators at Parks Middle, and STR2. It is important to note that not only was the Superintendant of Atlanta aware of the dramatic increase in test scores at Parks, so was the Georgia Department of Education.

In whose interest? But it was in their interest to perhaps look the other way, and not ask the question that was asked by an Atlanta reporter. As I have said in this blog, the scrutiny of this scandal should stop at the Superintendent’s office in Atlanta, but should also include the Twin Towers, home of the Georgia Department of Education. Did they contribute to the culture of fear that the Governor has alluded to in the investigative report?

Finally, the data shows that student did improve their Reading and English/Language Arts scores over the years. Not to admit so is a diservice to the teachers at Parks, and more importantly to the students and their parents.