Moving Beyond the Drama and Hyperbole that Dominate International Test Score Results

Moving Beyond the Drama and Hyperbole that Dominate International Test Score Results. The Guardian newspaper published a series of articles the 2013 PISA international test results.  In this post I want to focus on the article written by Finnish educator and scholar Pasi Sahlberg entitled, PISA 2012 scores show the failure of ‘market based’ education reform (Sahlberg, Pasi, 2013). Sahlberg is Director General of International Centre at the Finnish Ministry of Education and Culture in Helsinki.  He has written extensively about educational improvement and enhancing equity as the focus of improving education. Dr. Sahlberg points out that creating league tables that showcase or shame countries based on their student’s performance on standardized tests is simply not an proper use of international test results, in this case PISA. As I’ve reported many times on this blog, international test results fall prey to newspaper headlines that predict the collapse of economies, or the inability of its students to compete in the ‘global market.’  The ‘sky is falling’ mantra was alive and well last week.  Imagine reading the headlines in Helsinki after its students fell from second place to 12th in just three years.  Sahlberg reports that in Sweden, the test result for its students was considered a national disaster.  In the United States, the Secretary of Education (Arne Duncan) said that for the U.S. the results are “straightforward and stark: It is a picture of educational stagnation.”

Global Educational Reform

But Dr. Sahlberg suggests that the PISA results are proof that the Global Educational Reform Movement (GERM).  According to Sahlberg, GERM is a virus that has infected many nations in their march to “reform” education.  In his view, GERM is characterized by standardization (Common Core), core subjects (math, reading, science), teaching to the test, corporate management style, and test-based accountability.  When Duncan commented  (Guardian News, 2013) on the 2013 PISA results, he said it was clear that this “must serve as a wake-up call against educational complacency and low expectations.”  And to correct American education’s shortcomings, “we must invest in early learning, redesign high schools, raise standards and support great teachers.” Good examples of GERM schools can be found in the US, England, New Zealand, Australia, Sweden and Chile.  Here is how they fared in the PISA tests (Table 1).

PISA Results for Nations that have adopted the Global Educational Reform Movement (GERM)
Table 1. PISA Results for Nations that have adopted the Global Educational Reform Movement (GERM)
These nations have adopted a model of education based on competition, standardization, and test-based accountability.  In Sahlberg’s view,

GERM has acted like a virus that “infects” education systems as it travels around the world.

Non-Global Education Reform

But Sahlberg, if he were ever asked by Duncan how to improve American schools, would not suggest the “reforms” that Duncan has funded for the past five years.  Instead Dr. Sahlberg would suggest that the standard-corporate styled reforms (GERM) are based on premises that are rejected by educators and policy makers in nations that seem to be successful.  Disclaimer: There are many organizations, groups, individual educators and policy makers in nations that are infected by GERM who oppose these market-based reforms, and suggest that equity must be at the center of educational reform). According to Sahlberg, a school system is “successful” if it performs above the OECD average in mathematics, reading literacy and science, and if students’ socio-economic status has a weaker-than-average impact on students’ learning outcomes. The most successful education systems in the OECD are Korea, Japan, Finland, Canada and Estonia.

Table 2. PISA test scores for nations that are above the OECD average, and students socio-economic status has weaker-than-average impact on students' learning outcomes (Text: Sahlberg, 2013)
Table 2. PISA test scores for nations that are above the OECD average, and students socio-economic status has weaker-than-average impact on students’ learning outcomes (Text: Sahlberg, 2013)

Beyond GERM

1. Schools should have autonomy over its curricula and how students are assessed.  Teachers should work collaboratively to design and develop curriculum, and make decisions about the nature of instruction in their own classrooms.  This is contrary to the reforms that have dominated American education for decades, especially starting with the publication, Nation at Risk, followed by the No Child Left Behind Act during the Bush Administration, and The Race to the Top during the Obama administration.  Sahlberg says:

PISA shows how success is often associated with balanced professional autonomy with a collaborative culture in schools. Evidence also shows how high performing education systems engage teachers to set their own teaching and learning targets, to craft productive learning environments, and to design multiple forms student assessments to best support student learning and school improvement.

2. Schools need to focus on equity by giving priority to early childhood (one point for Duncan), comprehensive health and special education in schools, a balanced curriculum that sees the arts, music and sports as equals to math, reading and science. 3. School choice does not improve academic performance in a nation’s schools.  In fact, the overemphasis on school choice and competition between schools leads to greater segregation of schools. 4.  Successful schools are public schools and are controlled locally, not by a state or federal government. If we want to improve education in the US, we need to move away from the competitive, corporate-based model that is based on standardization and test accountability.  As Dr. Nel Noddings says in her new book, Education and Democracy in the 21st Century,

Education in the 21st century must put away some 20th-century thinking. All over the world today, many educators and policymakers believe that cooperation must displace competition as a primary form of relating. Competition is not to be abandoned— some competition is healthy and necessary— but it should no longer be the defining characteristic of relationships in an era of growing globalization. If we agree with this judgment, then we must consider how to prepare students for a cooperative world, not solely for one of competition.  (Noddings, Nel (2013-01-25).

American public schools are not failing.  The premise that they are failing is based on one factor–test scores.  We need to move beyond this concept of schooling and embrace collaboration, dialogue, interdependence, and creativity (Noddings, 2013).


Sahlberg, Pasi. “The PISA 2012 Scores Show the Failure of ‘market Based’ Education Reform.” Guardian News and Media, 08 Dec. 2013. Web. 12 Dec. 2013. “A Picture of Educational Stagnation’: Study Finds US Teen Students Lagging.” Guardian News and Media, 03 Dec. 2013. Web. 12 Dec. 2013. Noddings, Nel (2013-01-25). Education and Democracy in the 21st Century (Kindle Locations 103-107). Teachers College Press. Kindle Edition.

Does High Stakes Testing Produce Students Who are “Study Machines?”

Does High Stakes Testing Produce Students Who are “Study Machines?”

According to Emma Vanbergen, Shangahi-based study abroad director for BE Education, a company that places Chinese students in British schools and universities, it’s not surprising that Shanghai students are rated at the top of PISA World League Standings.

Why does she think so?

She says that the city schools are the most competitive in a country (China) where getting high scores on exams is the goal of education.

She indicates that the only measure that schools use to enable students to gain entry to the next level of education is a test score.  Nothing else.

Where is Shanghai?  Find the red dot.  It represents about 1.7 of the Chinese population.
Where is Shanghai? Find the red dot. It represents about 1.7 of the Chinese population.

And the higher rank of a school, the more difficult it is for students to meet the entrance requirement (test score).  She explains, that starting at a very young age, students go to school “in an endless cycle of learning, preparing for, and taking exams.

Because of her role in placing students for study abroad experiences in British schools and universities, she has some insight into the nature of Shanghai students.  Not surprisingly she says that the best students, that is those that carry around the top scores on achievement tests, are “not in fact super clever, great thinkers, or future academics.”

She says that

They are simply extremely hard-working study machines who memorize and churn out answers for tests in minutes.  They spend all their time on study, revision, homework, “pre-study” (a term I’d never encountered  until arriving in China), learning test techniques, and taking practice papers.”  (Edmma Vanbergen, Comments & Features, The Daily Telegraph, December 5, 2013)

According to Ms. Vanbergen, parents and teachers conspire inadvertently by bearing any cost to make sure their children get what is needed to score high marks on tests, and teachers are forced into the system because of the incentive of the “pay-for-student performance” for bonuses and promotions.

In China, the curriculum focuses on mathematics, science, Chinese and English.  Other subjects such as history, the arts, physical education, and the social sciences are not emphasized.  Chinese education is geared to the PISA exams, which test only mathematics, science and reading (English).

Ms. Vanbergen also tells us that the students who stay in school at age 15 in Shanghai schools are the best test takers.  Perhaps this sets them up for super performance on the triennial PISA tests.

She also reminds us that Shanghai is not representative of the system of Chinese education.  In her view, she says:

The super competitive, overpopulated, high-pressure nature of the schools, coupled with the significant financial backing of test-fixated parents, means students are conditioned from a young age to out-perform the competition in tests.  In that one respect–outperforming the majority of the world’s students in tests, in this case PISA–the system is clearly a great success.  (Emma Vanbergen, Comments & Features, The Daily Telegraph, December 5, 2013)

According to American newspapers and articles on blogs such as Education Week, US education stalls as other nations make gains.  In the eyes of the American press, and officials at the US Department of Education (ED), there is an even greater need for the standards-based test oriented curriculum of the Common Core.

If we are to accept the analysis of Ms. Vanbergen, then the solution to high scores on tests such as PISA is to turn students into “Study Machines.”

Is this the aim of an education in a democracy?  What do you think?


PISA Headlines from the UK: World League Standings!

As in America, the U.K. and European newspapers are having a field day reporting the results of the 2012 PISA results.   Here are few that I’ve read.

One of the sub-headlines reads: UK teenagers slip down world league in maths, science and reading.

PISA has a life of its own, and its existence of course is based on the drive of a few people at the OECD, in particular Andreas Schleicher, special advisor and chief developer of the test.  The Guardian refers to him at The OECD’s Pisa delivery man.

PBS aired an interview with Dr. Schleicher on December 3rd focusing on how American education measures up to schools around the globe.  The interview is instructive in that it is clear that Schleicher tells the interviewer that, Well, yes, I think the U.S. is an average performer.  Although the interviewer asked a few probing questions, Schleicher managed to stay to the OECD line that these tests are a valid measure of education world-wide, and that we’d better look at those nations whose students are at the top of World League Tables in maths, reading and science.

Is Asian Education the Answer?

If we look closely at the those countries (mainly in Asia) we might be surprised to find out that many educators within these systems are not happy with their education system.  Yong Zhao, who has written extensively about American and Chinese education wrote that it is a mistake to think that the day-to-day system of education has improved in these countries.  As he points out, students from these countries have always done well on standardized exams.

Dr. Zhao, in recent post on the National Education Policy Center blog, addressed this issue.  His comments are extremely important to help us sort out the meaning of these international tests, in particular, PISA.  He says:

The 2009 PISA results have already begun to shift the world’s attention away from Finland to Shanghai. I fear that the 2012 PISA will complete that shift and make Shanghai and other East Asian education systems THE model of education because the magic potion that East Asian success in international tests is very poisonous.

The recipe for the East Asian success is actually not that magical. It includes all the elements that have been identified as the symptoms of the GERM (Global Education Reform Movement) by the great Finnish education scholar Pasi Sahlberg: Competition, Standardization, Frequent Testing, and Privatization. In East Asian high PISA performing systems, these ingredients are more effectively combined and carried out to an extreme to result in entire societies devoted to ensure that their youngsters become excellent test takers.

While the East Asian systems may enjoy being at the top of international tests, they are not happy at all with the outcomes of their education. They have recognized the damages of their education for a long time and have taken actions to reform their systems. Recently, the Chinese government again issued orders to lesson student academic burden by reducing standardized tests and written homework in primary schools. The Singaporeans have been working reforming its curriculum and examination systems. The Koreans are working on implementing a “free semester” for the secondary students. Eastern Asian parents are willing and working hard to spend their life’s savings finding spots outside these “best” education systems. Thus international schools, schools that follow the less successful Western education model, have been in high demand and continue to grow in East Asia.

The Organization for Economic Co-Operation and Development (OECD), a 50 year-old organization monitors events in member nations and publishes policy recommendations based on their survey research.

OECD developed the Programme for International Student Assessment (PISA) which it gives to students from cooperating nations every three years.  The 2012 PISA was given to 510,000 students between the ages of 15 years 3 monte and 16 years 2 months.  However, each student who sat for the exam took only a small part of the PISA test.  Student scores are statistically joined to arrive at a nation’s average score in maths, reading and science.

The test is so long that no student could bear to sit for the hours that it would take.  This fact is often not mentioned in the news reports, and it one of the many issues that researchers raise when discussing PISA.

If you want to sit for a few minutes to answer a few of the PISA questions, follow this link.

The questions are paper and pencil, and each student sits for about 2 hours.  Each student completes a questionnaire (about their backgrounds, school and learning experiences) and then a mixture of open-ended and multiple-choice questions.  According to the PISA website, about 390 minutes of test items were covered, but again not by each student.

The PISA results need to be evaluated carefully, and it would be a mistake to listen to the initial comments by leaders of the testing movement such as Andreas Schleicher, or Arne Duncan.  We need time for researchers to look at the data, and write reports based on careful deliberation.



PISA Day: Your education today is your economy tomorrow

“Your education today is your economy tomorrow,” is a quote from Andreas Schleicher from the Organisation for Economic Co-operation and Development (OECD), who has become one of the world’s most influential figures in education. I took this statement off the PISA Day website.  In the view of Mr. Schleicher, a nation’s economy hinges on education (performance).  Although there is data to refute this, organizations such OECD, make the claim that your student’s performance on school tests will have a major impact on the nation’s economy. A nation’s economy and it’s ability to compete is much more complicated than laying the blame on its students and teachers. And one’s ability to “compete in the 21st century” is also more complicated than one’s report card.

Results as reported by PISA helps shape the public image of science education (or mathematics education), and it is unfortunate that educators allow this to happen.  Dr. Svein Sjøberg of the University of Oslo in a publication entitled Pisa and Real Life Challenges: Mission Impossible, questions the use of tests such as PISA.  He informs us that:

The PISA project sets the educational agenda internationally as well as within the participating countries. PISA results and advice are often considered as objective and value- free scientific truths, while they are, in fact embedded in the overall political and economic aims and priorities of the OECD. Through media coverage PISA results create the public perception of the quality of a country’s overall school system. The lack of critical voices from academics as well as from media gives authority to the images that are presented.

PISA measures only three areas of the curriculum (math, science, reading), according to Dr. Sjøberg , and the implication is that these are the most important areas, and areas such as history, geography, social science, ethics, foreign language, practical skills, arts and aesthetics are not as important to the goals of PISA.  The other international test, TIMSS, according to his analysis (and I would agree) is based on a science curriculum that many science educators want to replace, yet uses test items that could have been written 50 years ago.   In general the public is convinced that these international tests are valid ways of measuring learning, and that the results can be used to draw significant conclusions about the effectiveness of teaching and learning.

If you live in the world of psychometrics and modeling, the results that are gathered by these international testing bodies is a dream come true. Sjøberg puts it this way:

PISA is dominated and driven by psychometric concerns, and much less by educational. The data that emerge from these studies provides a fantastic pool of social and educational data, collected under strictly controlled conditions – a playground for psychometricians and their models. In fact, the rather complicated statistical design of the studies decreases the intelligibility of the studies. It is, even for experts, rather difficult to understand the statistical and sampling procedures, the rationale and the models that underlie the emergence of even test scores. In practice, one has to take the results at face value and on trust, given that some of our best statisticians are involved. But the advanced statistics certainly reduce the transparency of the study and hinder publicly informed debate.

PISA Day 2013

It’s PISA Day, and now everyone knows who is at the top and the bottom of the “leader Board.”

Screen Shot 2013-12-03 at 6.34.06 PM
PISA League Table showing the top dozen or so nations on the 2013 PISA international assessment of 15 year-olds. League Tables like this one create PISA envy. But the data also is manipulated to create fear which is good for those who benefit from constant reform of education, especially in the U.S.

On English TV, league tables are used to report the PISA results, with little else mentioned. The UK is falling behind, again, says one TV reporter. The whole enterprise is summed as a global race (to the top), and nations such as the UK and the US scored lower in the League Tables.

In England the argument becomes a political one, in that the present government puts the blame on the regime that was in power a decade ago.

Interestingly, the reports highlighted factors that seem to be characteristic of nations that fared well on the tests: teacher autonomy and decision-making power; emphasis on localized curriculum development; and emphasis on problem solving and not high-stakes testing.

These characteristics are absent in the UK and US education system. Each country has moved to an authoritarian standards-based and high-stakes testing. Teachers are removed from the decision-making process, and more and more, it’s the opinions of policy makers such as politicians, corporations, and private foundations run by a few very wealthy people.

The PISA assessments have gathered enormous amounts of data thanks to the cooperation of more than 510,000 15 year olds from nearly 70 countries.

Whose Happy in School?

One interesting aspect of the PISA assessment program was students’ perception of school. They were asked if they were happy at school? The results, by and large, showed that in nations where students did not perform at the highest level academically, the students reported a higher “happiness” score than the nations whose students were at the top of the League Standings. Students who were happiest at school included Indonesia, Albania, Peru, Thailand, and Columbia. The least happy students were in Korea, Czech Republic, Slovak Republic, Estonia, and Finland. Note: the U.S. was in the bottom third of the happiness scale, while the UK was half way up the scale.

Snapshots of Data

More to come, in the meantime:

Follow this link to see snapshots of the data collected by PISA.

Schools Stuck at Back of the Class According To Forthcoming PISA International Test Results

According to today’s lead newspaper story, the results to be released by the Organization for Economic Co-operation and Development (OECD), show the  country has simply “stagnated” since its lackluster performance four years ago.  The OECD results will be announced on Tuesday.

According to OECD 2009 test data, average score for the country’s fifteen year olds fell in maths, reading and science.  Scores are expected to fall further when the 2013 results are available on Tuesday.

According to one senior education official, the latest PISA results are a “wake up call” for school policy.  Because China, Shanghai, and Korea did so much better, politicians suggest that education policy officials go to China to find out how they do it.

Screen Shot 2013-10-09 at 6.44.47 PMAnd of course, the solution to poor performance on these international tests is to root out “bad teachers.”  In fact, one inspector of schools and newspaper columnist said that the problem with our schools is that there are thousands of “bad teachers.”   He suggested that the way to root them out is to raise the standards, and then pay teachers on the basis of their student’s test scores.

Now your probably thinking that this is a newspaper story in an American newspaper, but you would be wrong.

I am in Britain

I am in England for three weeks, and the lead story in today’s Sunday Times (London) is UK schools stuck at back of the class.

The last time the PISA results were released (2010) you would have thought the sky was falling if you listened to our Secretary of Education, Arne Duncan.  PISA is an international assessment that is administered to 15 year-old students in participating countries. The PISA assessment has been administered in 200020032006 and 2009.

The 2009 test results were released in December 2010.  In 2009, 65 countries participated in the test. In general average scores are used to make comparisons among countries.  The U.S. average was 502 (OECD average was 501).   According to the 2009 report, among the other 64 countries and education systems, 18 had higher average scores, 33 had lower average scores, and 13 had average scores that were not measurably different from the U.S. average score.

Sputnik Moments

If we rank order the countries according to average test score, the U.S. is in 19th place, and using the sports analogy, we are not at the top, and that that’s what causes politicians, corporate leaders, and state departments to make dire assessments of  the quality of American education.  The leaders of the U.S. government actually said that these test results (coming in 19th) was a “sputnik momement.”

Now we have another potential “Sputnik moment,” and it’s not happening only in the U.S.  Newspaper headlines throughout most Western nations will put their schools at the back of the class.

Sputnik moment or not, this is the predicted reaction of “leaders” when ever international (or national) test results are released.  In fact, the headlines of many nations’ national newspapers often are headlined with claims that the “sky is falling” and that the educational system is a failure.  Politicians, corporate heads, and others rush to make judgements, and lead their nations down paths that are harmful to the educational systems they claim is failing.

One problem here is the over reliance on test scores to make judgements about systems of education that in some cases are huge (the U.S. has 15,000 different school districts), very small (Singapore is City-State, perhaps comparable to one U.S. district), distinctly different about how many students live in poverty, differences in the way schools are funded, teachers prepared, and curriculum developed and implemented.

What is your predication about the response of newspapers and education officials when the PISA results are released on Tuesday?