I got 51 problems but a #2 pencil ain’t one.

Booz Allen Hamilton has a funny name but they’re all about serious business. The “global strategy and technology consulting firm” was contracted by the College Board in the fallout of the SAT scoring mistakes from last October. They audited the testing process and made a report, which was hidden away. At least until Senator LaValle used subpoena powers and demanded the report be made public–they were required to do so by New York’s “truth in testing” state laws but the College Board stated that they were “exempt from state law.” I love this choice quote: “Once again The College Board’s approach to problems is to shroud them in secrecy,” Senator LaValle said in a July 13 news release. “Their attitude gives rise to greater questions as to their true purpose.” What do you mean, Senator LaValle? Is there still a problem?

“The board’s lack of accountability in this matter goes to the heart of the arrogance that exists when an organization basically holds a monopoly in a certain market,” LaValle said in a statement following the release of the report. “This report still does not answer why such a large number of testing errors occurred on the October 2005 SAT. In addition, the board has not provided an explanation of why it took several months to inform over 4,000 students of inaccurate test scores.”

Additionally, The National Center for Fair and Open Testing responded to the report by releasing a list of questions that the study did not answer. Among them: What was the root cause of the problem? When did The College Board receive the first request for hand scoring of the October SAT? How many weeks did it take for the board to respond to that request? When did The College Board first discover a broad problem? Why was there a delay between that time and when test takers and others were informed of the problem?

[thanks, Daily Stanford]

Now, I went into that report (it’s a powerpoint presentation) and picked out some of the most fun parts of it. Actually, it’s more a case of me fearmongering by promoting potentially frightening sections of a report which just goes to show how flimsy the SAT is as part of a situation which just demonstrates how much that not-for-profit status is bullshit.

The report was released July 26 and has since been buried from the public as best the College Board can manage. It’s available on their website here; I’m not making this stuff up. Booz Allen Hamilton placed each of the 51 identified error-risk-locations into one of these categories:

There were some really cute flowcharts and diagrams, too–even little pie charts of “overall rating” for the likelihood a particular flaw in the testing chain could fail. There were several charts–one titled “Prioritization of Risk” (on slide 54 for those reading along) shows the killzone. These 4 were both of “higher impact” (i.e. “affects large part or whole answer sheet”) and “high likelihood,” remember the color scheme:

7. Scanner reading variation due to erasure

3. Scanner variation on marginal marks

4. Scanner reading variation due to alternative writing implement

26. Answer sheets left in test books

Don’t worry, they provide partial solutions on slide 56. For instance, pesky #7 can be solved by…

1. Continue double scan
2. Supply pencils/erasers at test centers

Wait, that’s it? That’s how they solve it? What about the blank spaces under “medium-term improvements” and “major future changes” or “follow-on analysis”? Why is there even the “follow-on analysis” table if there is nothing in it on that entire slide? Okay, whatever. How do we solve “Scanner reading variation due to alternative writing implement” ? We… supply pencils and erasers at test centers. Oh. And that finishes it? Oh, we can “Investigate software upgrades for consistent readings (color scanning, wavelength, etc.)” also? That’s great. And then eventually have an online test? That sounds stupid but I’ll take your word for it. Hmm, those problems sound easier to fix than some of the other ones, though. Error 51, Errors in deleting items from database–that sounds nasty. You would have to ” Ensure DB backup and rollback procedures in place” to solve that one.

Oh well, I’ll trust that stuff. I’ll just skip to the appendix and see if there is anything else of interest…Oh, look, here are some of those things that were of lower likelihood! These can’t be very important. Hang on, they could have “higher” impact? Okay, what happens if there is “Scanner variation due to smeared pencil marks.” That can’t be that bad. Smeared pencil marks due to manual handling–I can see that happening, I guess. And what’s the result? Inaccurate scores? Say it ain’t so! Well, at least there are now protections for that. Wait, there are “none” now or in the foreseeable future? Dang. Too bad there are several other ways for that to happen, too–not least among them a PEM database corruption which results in data storage failure and then… inaccurate scores. And which again has “none” ways against happening right now. Glad to see you have it all under control, College Board, but I’m still seeing a little too much red. Not you, though–all you see is green, green, cash. Good thing you still manage to keep the best interests of the students your balance books in mind.

Finish with more from the Daily Stanford article…

In the long run, the report makes the recommendation of offering the SAT online as a way to minimize scoring problems due to mechanical malfunctions, though the prospect of an online SAT raises many security questions. Certainly, the amount of weight given to SAT scores in college admissions necessitates that the process be carefully scrutinized for error.

Although The College Board insists that the mistakes that led to last fall’s incorrect exams will be fixed, and the problem was an isolated incident, the organization’s reputation as a supposedly non-profit, fair institution has been damaged by the mistake and its initial refusal to cooperate with the Senate Higher Education Committee. Moreover, this gives rise to the question of whether the SAT should continue to be used in college admissions.

Wooooo.

Oh, for dessert: here are all 51 problem areas: