In Spite of the “System,” Urban Teachers Have a Record of Success

In spite of the “System” the evidence is that urban teachers have a record of success, not one that is spiraling down.  The present state of reform of American education is based on the idea that American students are doing poorly, and this will lead to disastrous economic consequences, and the loss of American’s place in the global economic competition.

But, education (for our students) should not be a competition.  There is no need to have winners and losers as outcomes of the school experience.  Education is about learning, and in an environment that has as its core belief that learning is the fundamental goal of schooling.  Students are living in the present, and their school experience should be based on their lives now, and should not be based on furthering the economic prosperity of society.  Schooling should not be based on job training, career readiness or college entry.  It should be based on fostering the creative and innovative aspects of youth, and create school as a learning environment designed to help students learn to collaborate, work with others to solve problems, and engage in content from the arts and the sciences that has personal meaning.

We’ve been told that urban education in America needs to be saved by pouring advise and money from the élite and influential corporations and philanthropic groups.  The problem is that these groups are focused on only one set of outcomes that all come down to increasing student academic performance measured by high-stakes examinations.

I want to show here that urban teachers have held their own for the past decade and half in spite of the problems they face in their schools day-to-day.  They not only have held their own, but the evidence shows that academic performance of their students (in mathematics at the 8th grade) in the example below has slowly but surly increased as shown in Figure 1.  As you can see, in Atlanta, students at the 25th, 50th and 75th percentile increased performance on NAEP tests given from 2003 through 2013.

Figure 1. Atlanta NAEP 8th Grade Mathematics Scores for Selected Percentiles 2003 - 2013.
Figure 1. Atlanta NAEP 8th Grade Mathematics Scores for Selected Percentiles 2003 – 2013.

Reading and Math in Urban Schools

Take a look at the next four figures (Figures 2 – 5).  They were compiled by Mr. Ed Johnson in his study of the NAEP Trial Urban District Assessment (TUDA).  Johnson, who is a student W. Edwards Deming, examined the TUDA results through a Deming Lens.  A Deming Lens means that to understand the behavior of a system, one must look at the system.  Breaking down a system into its parts (goals, policies, finances, curriculum, teachers, administrators, parents, directors) loses one’s ability to understand the system.

Each of the graphs below shows the behavior of these four systems over ten years.  You will notice that there is variation in the achievement scores of students in reading (grades 4 and 8) and mathematics (grades 4 and 8) from one testing period to the next.  But the variation is within upper and lower limits that would be expected in each system.

Causes of Variation in Scores

According to W. Edwards Deming 94% of the variation is due to the nature of the system, not the people who work in or make the system work.  Only 6% are attributable to special causes.  (W. Edwards Deming. The New Economics for Industry, Government, Education (p. 33). Kindle Edition).

As you look over the graphs you will see ONLY FOUR instances where the variation in scores lies outside the Upper Control Limit (UCL), and then only in 4th grade reading  Examples include: Charlotte, 2009, Austin, 2011, Charlotte, 2011, and Hillsborough, 2011.  Except for these four instances, all the variation is due to the nature of the system.

Figure 2. Location of Cities in the Trial Urban District Assessment; Gold triangle-higher than large city; Circle--not different; Blue triangle-lower than large city. Source: Nations Report Card
Figure 2. Location of Cities in the Trial Urban District Assessment; Gold triangle-higher than large city; Circle–not different; Blue triangle-lower than large city. Source: Nations Report Card.  Click on Map for more details.

The graphs below plot reading and math scores for 21 school urban school districts.  Mr. Johnson highlighted Atlanta (in red) and DC Public Schools (purple).  As you can note in the following graphs, achievement scores in reading and math for Atlanta and DC Public Schools fell within the Upper and Lower Control limits.  There is no radical change in scores, either up or down.  It appears that the teachers in these urban schools are doing the job they were hired to do and that is help their students learn how to read, and do mathematics.  And they’ve done this in spite of all the issues that surround schools in urban communities.

In systems thinking, as Mr. Johnson would tell us, there are two types of causes of variation in any system.  The most important cause of variation in any system is what we call “common causes” of variation that is really a function of the system itself.  Examples of common cause variation will fall within control limits on a graph (as shown below in Figures 2 – 5). Examples of common causes that influence variation (scores on tests, for example, or graduation rates) include 

  • High percentage of children from low SES groups.
  • Where the school is located. It’s zip code.
  • Age of the school building.
  • Size of the school system.
  • Underpinning policies, practices, procedures of the school which determines it’s culture.
  • Inadequate resources.

According to Deming, nearly all outputs of schooling are the result of common cause variation, and these would include drop out rates, achievement test scores, violence, bullying, gang activity, low self esteem, attitudes, under performance and literacy skills.

Defying Gravity

When we examine a school system from a systems thinking view, these outputs are causes by the day to day effects of common causes of variation.  As Deming and other systems thinkers, such as Ed Johnson would say, trying to seek achievement scores beyond what see in the graphs (Figures 2 – 5) is to “defy gravity.”  Reformers have charged ahead as if they can “defy gravity” and have put the blame of not improving test scores in the wrong place.

Managers (administrators) and workers (teachers) are not “common cause” variables.  However, since schools are based on a linear factory model, “reformers” ignore common causes, and instead claim that teachers and administrators can overcome the challenges posed by common causes.  When reformers insist on market reforms, and they don’t work, they blame the teachers and principles.  And to make matters worse, they use student test scores (which are the result of common causes) to evaluate teachers on the basis of false assumptions about schooling.

We think that the present system of reading and mathematics is fairly stable.  The output in reading and math (as measured by a test score) vary little, and one can make predictions about future reading and math output.

reading 4th

Screen Shot 2014-01-11 at 8.27.50 PM

Screen Shot 2014-01-11 at 8.28.18 PM

Screen Shot 2014-01-11 at 8.28.31 PM

We will explore systems thinking in future posts.  But for now, what do think about the analysis of the NAEP TUDA data as compiled by Ed Johnson?

A Systems Thinker Reviews The Atlanta Public Schools’ Performance in Reading & Math

Latest Story

People are asking for better schools, with no clear idea how to improve eduction, or even how to define improvement of education (except to increase test performance on high-stakes tests).

Most people are in favor of improving education.  But when asked how would they improve education, the suggestions are insufficient, and in some cases, even negative (See W. Edwards Deming. The New Economics for Industry, Government, Education (p. 8). Kindle Edition.)

Instead of reporting the details of how people and organizations want to improve education, such as corporate chiefs, philanthropists, and the U.S. Department of Education, I want to report on the work of Mr. Ed Johnson, an advocate for quality education, and has for more than a decade devoted himself to writing and talking about improving education in the Atlanta Public Schools.

Systems Thinking

Ed Johnson consults as Quality Information Solutions, Inc., with a commitment to human social and cultural systems to receive quality information from information systems for the continual improvement of life, work, and play. His commitment extends to advocating the transformation of K-12 public education systems to humanistic paradigms from prevailing mechanistic paradigms. Ed also is former president of Atlanta Area Deming Study Group.

Ed Johnson is a systems thinker.

In this regard, he believes schools can not be improved by trying to improve the parts separately.  It is a sure path to failure.  For example, some advocates of educational reform believe that student achievement can be improved by weeding out the bad teachers.  Millions of dollars have been invested in using student high-stakes test scores to check teacher performance using a technique called Value Added Measure (VAM).  Teachers whose VAM scores are low can be identified, and according to these experts, teachers with low scores must be bad teachers.  Getting rid of “defects” in any system will not improve the system or the part that was identified.  Instead, a better investment would be to ask how can we improve the quality of teaching, and what can be done to improve the teaching of all educators.

The above example highlights the current approach to reform.  Identify a part of the system, and fix it. Bad teachers, get rid of them.  Low achievement scores?  Write “rigorous” standards, raise the bar, and give high-stakes tests.  It’s that simple.  We’ve had rigorous and not so rigorous standards in place for more than a decade, and as you will see ahead, changing standards doesn’t have any effect on student performance.

Systems thinking means that all parts of a school system are interdependent and must be taken as a whole.  The Atlanta Public Schools (APS) is a system of interconnected and interdependent parts, and to improve the quality of the APS, it is critical to look at the APS as a whole.  For example, closing schools (removing so-called underperforming schools), does not have an effect of improving the APS, or indeed saving money (as some would tell you).  Fundamental questions about APS need to be asked, but in the context of the APS being a system, not a collection of  schools, students, teachers, administrators, parents, curriculum, textbooks, technology.

Ed Johnson has contributed to my understanding of quality education, and it is my great honor to share his work on this blog, and in particular to look at teaching in urban schools, and in particular the Atlanta Public Schools.

Trial Urban District Assessment (TUDA)

The National Assessment of Educational Progress (NAEP) created the Trial Urban District Assessment (TUDA) in 2002 to assess student achievement in the nation’s large urban districts.  Reading results were first reported in 2002 for six districts, and math results were reported in 2003 for 10 districts.

The NAEP provides data from 2002 through 2012 on math and reading and are comparable to NAEP national and state results because the same assessments are used.

Screen Shot 2013-12-28 at 7.00.56 PMUsing data from all of the Trial Urban Districts Assessments (2002 – 2013), available online at The Nations Report Card, Ed Johnson analyzed and created a presentation that is a series of systemic stories told by the data collected in the urban district studies.

Each story is about a system.  In TUDA (Trial Urban District Assessment), there are particular TUD’s each as a system.  So, Reading is a system.  Mathematics is a system.  4th grades is a system.  8th grades is a system.

Johnson’s research is a longitudinal study of performance in reading and mathematics from 2002 – 2013.  Using scores on reading and mathematics obtained from the National Center for Educational Statistics, he investigated the nature of a number of systems derived from the data.  Some of these particular systems include:

  • Reading as a System
  • Mathematics as a System
  • 4th Grades as a System
  • 8th Grades as a System

Since these systems are part of the APS system, we know that each of these systems is interdependent with other systems, not just the ones identified here, but including parents (as a system), teachers (as a system), and so forth.

There were 21 urban school districts in the study.  However, Ed has managed to make our work easier by highlighting with color coding just two systems, Atlanta Public Schools (Red) and District of Columbia (Purple).

Overall

Figure 1. Trial Urban District, Bottom Line.  Source: Ed Johnson edwjonhson@aol.com
Figure 1. Trial Urban District, Bottom Line. Source: Ed Johnson edwjonhson@aol.com

Ed starts his study by giving us the “bottom line.”  How did these systems (reading, math, 4th grade, 8th grade) do?  Figure 1, is a summary of systemic TUD student performance in reading and math at the 4th and 8th grade from 2002 – 2011, and predictions for 2013 (all the predictions were accurate forecasts of student performance in 2013).

Only 4th grade reading showed some improvement over the period 2002 – 2011, and the improvement was slight and noticed only in Austin, Charlotte, and Hillsborough.  In all other systems, no improvement was observed, meaning that the common causes that influence the system of math, or reading, or 4th or 8th grade inhibited improvement.

Student Improvement in Mathematics and Reading

In the TUDA study, a sample of students in each urban district was tested in reading and mathematics at the 4th and 8th grade level.  To help us understand how to interpret data collected over the past dozen or so years, Mr. Johnson has produced a series of graphs (control charts) showing the natural variation of scores to be expected in each system (reading, math, 4th grade, 8th grade).

Figure 2 shows a control chart for  reading, 4th grade.  Figure 3 shows a control chart for mathematics, grade 4.  Upper control limits and lower control limits were calculated for 2002, and then projected forward.  Changes in scores from one test period to the next are shown in the Figure 2.  If there is systemic change in reading at the 4th grade level, then scores would fall “outside” the upper or lower control limits.  You’ll notice that all the variation, except for four points (Charlotte, 2009 and 2011, Austin, 2011, and Hillsborough, 211), was within the variation expected.  In systems thinking, we mean that the variation for the most part was random, but there is evidence that some special causes were at work in the three districts mentioned here.

Figure 2.  NAEP TUDA, Reading, 4th Grade, All students prepared by edwjohnson@aol.com
Figure 2. NAEP TUDA, Reading, 4th Grade, All students prepared by edwjohnson@aol.com

Mathematics is another story.  As Mr. Johnson puts it in his study, “all districts have been on the same boat continuously since 2003 in mathematics at the 4th grade level.  What means is that the variation shown in the graph is random, and not due to any special cause.

Figure 3. NAEP TUDA, Mathematics, 4th Grade, All Students prepared by edwjohnson@aol.com
Figure 3. NAEP TUDA, Mathematics, 4th Grade, All Students prepared by edwjohnson@aol.com

There is very little student improvement in reading or mathematics at the 4th grade level as shown in Figures 2 and 3.

As long as we continue to ignore the common causes of variation that exist in the system then we can expect very little to no improvement.

But as Mr. Johnson has said in other letters and reports, if fundamental questions about the purpose of schooling are not addressed and if we can not agree on these purposes, very little will change in the system.  In the two systems explored here, reading at the 4th grade and math at the fourth grade, we need to ask: What is purpose of teaching reading in the elementary school?  Why do we teach reading in the elementary school?  What is goal of teaching mathematics in the elementary school?  Why do we teach mathematics?

As Mr. Johnson has shown, why are these districts on the same boat for the teaching of mathematics?  How can we used systems theory to look at mathematics teaching as a system and answer questions about how to improve mathematics learning?  How can help students develop a love affair with mathematics?

Ed Johnson has examined a lot of data from the standpoint of systems thinking based in part of his work with Edward Deming, and other scholars in the field of systems thinking.

I highly recommend that you check his study which you can get access to as a PDF file here: NAEP TUDA 2002-2011 Views through a Deming Lens.

In the days ahead, I’ll revisit Mr. Johnson’s study, and report on his analysis of the “performance gap” variation that he has depicted as a series of images as shown in Figures 2 and 3.  I’ll also explore systems thinking and school in more detail.

Testing in Georgia: Students Miss the Mark or Did the State Officials

In today’s Atlanta Journal there was an article that reported that Georgia Department of Education officials were shocked by the state math and social studies tests. You see on the end-of-year CRCT only 20 to 30 percent of the students passed the social studies test, and about 40 percent of the Georgia’s could be held back because they failed the math test (the truth is, they must fail both the math and reading test to be retained).

I know that this is a radical shift from the writing I’ve been doing about the earthquake in China, but this is the season when students have finished taking their high-stakes tests, and the time when departments of education release the results to parents, schools, and the general public. So here are some thoughts on these tests.

This is also the time that the state department “interprets” or tries to explain the results. Now you must understand that the State Superintendent of Schools in Georgia is infamous for suggesting the word “evolution” be banned from the State of Georgia Science Standards. She suggested this several years ago claiming that evolution was simply a “buzz word.” It did not happen mostly because of the hyper-mail and telephone calls she received from around the country.

When test scores go up, officials usually claim that that the standards are being interpreted and taught properly, and the students are meeting expectations. When the score go down, as they did this year in math and social studies, amazing reasons surface: maybe the standards aren’t clear; some of the test questions are just too complicated and difficult. If you listen to students, some of them say, “we never even covered that material in our curriculum.” I’ve always believed that the reasons state level officials give for scores (whether they go up, stay the same, or go down), have little to do with actual day-to-day teaching.

High Jumping
Administrators at the state level typically advise us that these high-stakes tests measure to what degree students have met or not met the standards, and in fact they liken this process to the Olympic sport of high jumping, e.g. raising the bar. Cox and other administrators suggest “if we ‘raise the bar’ or ‘standards’ on the students we might expect a bit of dip (on test scores), before the students soar over the bar. Apparently there was a dip in test scores in social studies and math this year in Georgia. To me this is an annual game played out in cities and states around the country, indeed around the world. See Svein Sjoberg’s critique of the international science testing program, PISA.

But we continue to fool ourselves with these high-stakes tests. They do not provide schools, teachers or students with information that will lead to improved learning. They simply put students in a position being betrayed by schooling when in fact schools should be in the service of students and parents. Over the past 20 years, the standards movement and associated high-stakes testing has led to teachers saying things like, “I used to be a good teacher; now all I do is pass out books, and get students ready for the test.”

What do you think about high-stakes testing? Are students missing the mark on these tests, as in the case of the Georgia results in mathematics and social studies, or have we missed the mark in not applying what we know about motivation, pedagogy, and evaluation?

Alfie Kohn, an outspoken and published critic of high-stakes testing, says this:

A plague has been sweeping through American schools, wiping out the most innovative instruction and beating down some of the best teachers and administrators. Ironically, that plague has been unleashed in the name of improving schools. Invoking such terms as “tougher standards,” “accountability,” and “raising the bar,” people with little understanding of how children learn have imposed a heavy-handed, top-down, test-driven version of school reform that is lowering the quality of education in this country.