Results Are In: NCTQ Report on Teacher Prep Rated with Four Cautions

In the last four posts, I have written articles that call out the National Council on Teacher Quality on the basis of their built-in bias against teacher preparation institutions, and their confrontational style of what they call research.

In the last post I used research by Michael Carolan, who investigated when does science become junk.  In one study, drawing from the Lexis-Nexis database, ten years (1995–2005) of newspaper articles containing the term ‘junk science’ in the headline were systematically analysed and coded according to how the term “junk science” was used.  From this analysis and coding,  11 definitions of junk science were identified.  Michael Carolan’s research is significant here because it offers a method to find how we can distinguish between junk science and science reports, especially if we think the report is published by non-refereed sources.  It’s vital to be able to answer this question, especially in the present age where the media gives equal time to every idea regardless of the idea’s validity.

To rate the NCTQ study, I used the 11 definitions as criteria to decide whether the study fits the definitions of junk science, or whether it might be considered a valid scientific investigation.  In future posts, I’ll report evaluations of other studies, especially two studies on science standards by the Fordham Institute.

Using the criteria listed in Figure 1, I examined the NCTQ Report on Teacher Prep.  For the criteria, the goal was find out if that particular definition could be applied to the NCTQ report.  If it did, then the criteria was marked YES.  To support the decision, examples from the NCTQ report are included in the last column of Figure 1.

Junk Science Definitions Junk Science? Score Comments
Bad policy based upon Yes 1 Assumes American education is declining; Believes the marketplace is the engine for change
Experts with agendas Yes 1 Assume that one of causes of America’s decline is teacher preparation institutions. NCTQ wants teacher education to be a training ground not an environment of learning
False data No 0 Data reported are ratings, which are dubious.  We could rate this “yes”
No data or unsubstantiated claims Yes 1 Data unclear; claim teacher education is “chaotic” with no data;
Failure to cite references Yes 1 No review of the literature; cherry picked three references;
Using non-certified experts Yes 1 There is no evidence that experts in the field of teacher education participated
Poor methodology Yes 1 Data was limited to college bulletins and syllabi.  Very few syllabi received to make decisions.
Too much uncertainty to arrive at conclusions drawn Yes 1 Universities did not cooperate thus creating great uncertainty in any information gathered by NCTQ
Revealing only that data which supports findings Yes 1 The data is in the form of ratings
Non-peer reviewed claims Yes 1 Not subjected to review by experts in the field of teacher education
Totals 10 yes; 1 no 10 The NCTQ report on Teacher Prep scored 10 on a scale of 0 -11.  The report is clearly junk science
Figure 1.  An Analysis of the NCTQ Report on Teacher Prep using established definitions of junk science. The NCTQ scored 10 out of 11 indicating that it a highly rated junk science study.  Definitions from Carolan, M.S., When Does Science Become Junk?

Screen Shot 2013-07-01 at 8.01.25 PMCaution

The NCTQ study scored 10 out of 11 on the Junk Science scale, and as result earned four “CAUTIONS, as indicated in Figure 2.  To earn 3 or 4 cautions is clear evidence based on the JS Score Method, that the study under consideration is pure junk.  The results should be considered  fraudulent, invalid, unreliable, and pure ideology.

Figure 2.  Junk Science Rating System Using JS (Junk Science) Score.  The higher the score, the more caution should be exercised using the results and conclusions.  Hit the delete button or drag the file to the waste basket if the study has a rating of more than 2 Cautions.

I invite you rate the NCTQ report on teacher prep using the JS Score Method, and then compare it to a study done by Linda Darling-Hammond, Powerful Teacher Education, Lessons from Exemplary Programs.  The NCTQ study compared to the Darling-Hammond  study is a classic example of junk science compared to science.

Do you think NCTQ study is junk science, or is it a credible study of teacher prep?


About Jack Hassard

Jack Hassard is a writer, a former high school teacher, and Professor Emeritus of Science Education, Georgia State University