What’s Wrong With Standardized Testing?

Todd Farley is the author of Making the Grades: My Misadventures in the Standardized Testing Industry, published this week by PoliPoint Press. http://p3books.com/ Todd’s book is a remarkable memoir of his 15-year involvement with the tests, first as a scorer of 4^th grade essays, later as a manager. Full disclosure: I’m Todd’s literary agent.

Todd’s book is an eloquent and withering attack on the methodology and practice of the tests. It is the first book by an insider. It also is one of the funniest books ever written., I mean side-splittingly funny, fall out of the bed funny. As in Catch 22  (an apt comparison), Todd shows that humor and inspired ridicule are the best weapons against pretense. 

I give a lot of talks to writers. In this publishing climate it is pretty hard to offer words of inspiration for the unpublished author. I always tell Todd’s story to them. Todd contacted me over the transom through a query letter. I was curious about the subject, so I asked him to send more.  We struggled to get the book published over objections that the author lacked “platform” or that the book  had a limited audience. (33 rejections, to be precise).  We found a publisher who believed in him. This week Todd has written the Op-ed essay in The New York Times and has been read by over  1,000,000 people worldwide. Limited audience, indeed!

 Andy: Todd, you are a smart guy. How did you end up in the boiler room of a test scoring company?

Todd:   I ended up in the business because I was a slacker in my 20’s, living the good life in Iowa City as a reader and drinker, and I stumbled upon a job scoring student responses to standardized tests.  I don’t know that I was in any way qualified to do that job, and I certainly had no interest in it, but the pay was decent so I took the job.  Then fifteen years of me working in testing just to pay the bills crept by and now here we are….

Andy: And when did you realize things were terribly wrong with the theory and practice of testing? Did you have a “Eureka” moment?

Todd:   I thought it was a screwed up process pretty much the first day I started. Kids answer dozens of questions in some far-distant state and then their tests get chopped up into all these pieces—multiple-choice answers go this way, fill in the blank questions go that way, essays questions somewhere else—and are then read and scored by a temporary employee counting the minutes till quitting time?  It took no time at all to see what a bad idea that was. And all those doubts I had about testing were confirmed over the next 15 years. 

 Andy: Your book is full of stories that are as hilarious as they are disturbing. Can you relate a story that epitomizes the failings of standardized testing? 

Todd: I have more anecdotes about the foolishness of the business than there is space on the Internet.  I think about the very first essay-scoring project I worked on, when all us temporary employees were told we would have to qualify to be able to keep the job scoring high school writing—we’d have to do a good enough job during a training session to prove we could assess essays in a “standardized” way.  Then, when nearly half the hundred people applying for the job failed the “qualifying tests” and were fired, you know what happened?  The for-profit testing company in charge of the project simply lowered the qualifying score to make us (yes, “us’) all eligible to work.  The point is the scoring company had these high standards about employee qualifications, at least until they realized firing us all would leave them short on personnel.  Once they realized that, all us “flunkies” were immediately un-fired and given jobs reading and scoring all those tests.

 Andy: Standardized testing has been the mantra of efforts to reform the education system. It has been embraced by politicians  of both parties, most recently by President Obama. From your own experience, what is the flaw in the reasoning? What is the essential fallacy of the concept? 

 Todd: For me the fallacy is the utter inefficacy of a system that trusts for-profit companies and for-profit people (like me) to make decisions about American education, something that is supposed to be separate from the bottom line.  Because there are so many tests to be scored each year, every imaginable shortcut has to be taken to meet deadlines, with the end result being that scores are returned to students that I don’t necessarily believe are the right scores.  As far as I’m concerned, they are just scores, random numbers that vary as much due to the vagaries of the testing business as to the quality of student work. 

Andy: And while we are at it, how is the application of the theory in the actual grading of tests at odds with the espoused objectives?

Todd:  Andy, that seems like a very complex question for someone of my limited intelligence, but I can see why people believe there needs to be accountability in schools.  On the other hand, the fifteen years I just spent in the K-12 testing business convinced me that for-profit testing companies can’t provide that accountability.  Imagine, the theory of standardized testing is in effect that the best way to see what’s going on in American education is to ask the opinion of a bunch of massive corporations situated hundred, if not thousands, of miles away from the classrooms.  That is simply illogical, as my book explains in great—albeit hilarious—detail.

Andy: You say in your book that when the statistical results of the grading of the groups are at variance with the predicted results, then the results are fudged. That is a pretty damning critique. Can you elaborate?

 Todd:  Look, in my time in testing, I was witness to (and party to) a near constant manipulation of statistics.  The testing companies (and I) changed reliability statistics, validity statistics, qualification stats, etc etc.  Our job in the testing industry was to score tests, and within a limited time frame, and while certainly attempts were done to make the scoring legitimate, the sheer volume of tests was always so overwhelming each year that the only way to get through the process was to cheat, cheat, cheat. The companies took shortcuts to get things done and we temporary employees took shortcuts to get things done.  Many of us employees were working to keep our jobs, get raises and promotions, get plum assignments in college towns around the country, and hence we fudged the numbers all the time to make things look good.  Am I proud of that fact?  No.  On the other hand, the problems with the testing industry are so many that I didn’t think a little cheating on top of a lot of insanity was that bad a thing. 

 Andy: Todd, this blog is primarily about writing and publishing – not about education policy. I am particularly proud of this book, because you wrote it in relative obscurity. Now you are writing the op-ed essay in the New York Times. How did you begin the process of getting this book published? Can you give some inspirational words to the struggling writer?

 Todd:  I’m not sure if I have much advice for the struggling writer, because my situation was a little unusual.  I wasn’t one of ten-thousand (or fifty-thousand) people writing novels last year—I was pretty much the only one who had written an expose of the standardized testing industry.  That means I didn’t have to write the best of thousands of books, I just had to find someone to publish my one, unique book.   I realized there was a huge hue and cry in this country against standardized testing, and I knew I had lots of information that all the people against testing would love to hear, and I figured I’d get someone to publish it. Frankly, I always thought this book would sell, because an insider’s account about an important industry like testing just seemed to make sense.  Although I did have my doubts during the long process of writing the book and getting rejections, I had little doubt it would eventually sell—I was really just worried one of my colleagues in testing would write a book like this before I got to it.

 Andy: When I got your query letter, I had never heard of you. I assume you sent it out to other agents. How many? 10, 100, 1000? Was anyone else curious about the project? 

Todd: I sent query letters both to agents and publishers, maybe fifty or sixty total over the course of 8-10 months, and I did get enough interest to convince me I was on the right path—so while there was rejection, I also got considerable interest, enough to keep me at it.  I got lots of initial rejections, yes, but I also probably got 20 or so bites from agents and publishers.  Most of them thought I had written something good, but all said it might be hard to place.  I’d written a book about standardized testing, but a cheeky memoir and not an academic treatise, so no one knew where it would go.  Academic publishers (like Harvard University Press, NYU Press) were intrigued, but when they read my book they said it was too… personal, funny, not serious… for them to publish.  The same was true of agents.  Some would ask to see it and then, after telling me how funny it was, would say they couldn’t imagine where to publish it or who would buy it.  Those poor, ignorant fools… I could have made them all rich, as I did with you, Andy Ross!

 Andy: Ah, yes. Excuse me a moment while I ask my butler to call for the car.  Anyway, Todd. Thanks for being interviewed, and thanks for writing the book.  I think you have done your part in making the world a little better.

Advertisement

Tags: andy ross, making the grades, new york times, op-ed, standardized tests, todd farley