The College Board cancelled the January 28, 2007 SAT scores of 900 Koreans because some students previously had access to the questions. The reason test items circulated in advance is that the exam was identical to the SAT administered in December 2005.

 

Though the December 2005 SAT was not made public under "Truth in Testing" provisions, which apply to questions and answers from only four of the seven SAT administrations each year, students post items from every exam on the Internet. In addition, some coaching schools have run sophisticated operations to collect entire exams, either by sending in teams of test-takers to memorize the exam or by obtaining entire forms.

 

The College Board offered no reason to believe that the prior-exposure problem was confined to one Asian nation.

Other thoughts: it must suck to be one of those test-question memorizers. I tend to remember the ones I think I missed, which haunt me until I get my scores... at which point they continue to haunt me.

Wait, it gets better. The College Board explanation for the repeated test?

The College Board, which owns the SAT, and its contractor, the Educational Testing Service, justified the test recycling practice by claiming that it costs "probably $350,000" to create each new exam. But 326,000 students took the January SAT, paying a base registration fee of $41.50. That means test-makers took in more than $13 million at this administration. Given that huge revenue stream and the fat surpluses historically enjoyed by the College Board and ETS), the companies had no credible financial excuse for cutting corners.

The lesson from all this? Taking those practice tests could be even better than you first expected!

Oh, by the way, that official online College Board SAT prep course is still 100% free, check out my post on it from last year.

There are two reasons I didn't take the ACT, aside from its lack of trendiness here in the Northeast.

• Reason number one is that I had good SAT scores in hand by the time I had the opportunity to ever sit for the ACT.
• Reason number two is that although looking at practice books made the ACT look like a better and more fun test than the SAT, it was almost impossible to take from my then-HQ in Exeter, NH. There were one or two changes to take the ACT throughout the entire year--and both meant getting up around 4:30 and driving to Maine. I don't know how bad the situation is back home around Boston, but the unavailability of testing centers in the Northeast scuttled my desire to take the ACT.

Now you know!

I got 51 problems but a #2 pencil ain't one.

Booz Allen Hamilton has a funny name but they're all about serious business. The "global strategy and technology consulting firm" was contracted by the College Board in the fallout of the SAT scoring mistakes from last October. They audited the testing process and made a report, which was hidden away. At least until Senator LaValle used subpoena powers and demanded the report be made public--they were required to do so by New York's "truth in testing" state laws but the College Board stated that they were "exempt from state law." I love this choice quote: “Once again The College Board’s approach to problems is to shroud them in secrecy,” Senator LaValle said in a July 13 news release. “Their attitude gives rise to greater questions as to their true purpose.” What do you mean, Senator LaValle? Is there still a problem?

“The board’s lack of accountability in this matter goes to the heart of the arrogance that exists when an organization basically holds a monopoly in a certain market,” LaValle said in a statement following the release of the report. “This report still does not answer why such a large number of testing errors occurred on the October 2005 SAT. In addition, the board has not provided an explanation of why it took several months to inform over 4,000 students of inaccurate test scores.”

Additionally, The National Center for Fair and Open Testing responded to the report by releasing a list of questions that the study did not answer. Among them: What was the root cause of the problem? When did The College Board receive the first request for hand scoring of the October SAT? How many weeks did it take for the board to respond to that request? When did The College Board first discover a broad problem? Why was there a delay between that time and when test takers and others were informed of the problem?

[thanks, Daily Stanford]

Now, I went into that report (it's a powerpoint presentation) and picked out some of the most fun parts of it. Actually, it's more a case of me fearmongering by promoting potentially frightening sections of a report which just goes to show how flimsy the SAT is as part of a situation which just demonstrates how much that not-for-profit status is bullshit.

The report was released July 26 and has since been buried from the public as best the College Board can manage. It's available on their website here; I'm not making this stuff up. Booz Allen Hamilton placed each of the 51 identified error-risk-locations into one of these categories:

There were some really cute flowcharts and diagrams, too--even little pie charts of "overall rating" for the likelihood a particular flaw in the testing chain could fail. There were several charts--one titled "Prioritization of Risk" (on slide 54 for those reading along) shows the killzone. These 4 were both of "higher impact" (i.e. "affects large part or whole answer sheet") and "high likelihood," remember the color scheme:

7. Scanner reading variation due to erasure

3. Scanner variation on marginal marks

4. Scanner reading variation due to alternative writing implement

26. Answer sheets left in test books

Don't worry, they provide partial solutions on slide 56. For instance, pesky #7 can be solved by...

1. Continue double scan
2. Supply pencils/erasers at test centers

Wait, that's it? That's how they solve it? What about the blank spaces under "medium-term improvements" and "major future changes" or "follow-on analysis"? Why is there even the "follow-on analysis" table if there is nothing in it on that entire slide? Okay, whatever. How do we solve "Scanner reading variation due to alternative writing implement" ? We... supply pencils and erasers at test centers. Oh. And that finishes it? Oh, we can "Investigate software upgrades for consistent readings (color scanning, wavelength, etc.)" also? That's great. And then eventually have an online test? That sounds stupid but I'll take your word for it. Hmm, those problems sound easier to fix than some of the other ones, though. Error 51, Errors in deleting items from database--that sounds nasty. You would have to " Ensure DB backup and rollback procedures in place" to solve that one.

Oh well, I'll trust that stuff. I'll just skip to the appendix and see if there is anything else of interest...Oh, look, here are some of those things that were of lower likelihood! These can't be very important. Hang on, they could have "higher" impact? Okay, what happens if there is "Scanner variation due to smeared pencil marks." That can't be that bad. Smeared pencil marks due to manual handling--I can see that happening, I guess. And what's the result? Inaccurate scores? Say it ain't so! Well, at least there are now protections for that. Wait, there are "none" now or in the foreseeable future? Dang. Too bad there are several other ways for that to happen, too--not least among them a PEM database corruption which results in data storage failure and then... inaccurate scores. And which again has "none" ways against happening right now. Glad to see you have it all under control, College Board, but I'm still seeing a little too much red. Not you, though--all you see is green, green, cash. Good thing you still manage to keep the best interests of the students your balance books in mind.

Finish with more from the Daily Stanford article...

In the long run, the report makes the recommendation of offering the SAT online as a way to minimize scoring problems due to mechanical malfunctions, though the prospect of an online SAT raises many security questions. Certainly, the amount of weight given to SAT scores in college admissions necessitates that the process be carefully scrutinized for error.

Although The College Board insists that the mistakes that led to last fall’s incorrect exams will be fixed, and the problem was an isolated incident, the organization’s reputation as a supposedly non-profit, fair institution has been damaged by the mistake and its initial refusal to cooperate with the Senate Higher Education Committee. Moreover, this gives rise to the question of whether the SAT should continue to be used in college admissions.

Wooooo.

Oh, for dessert: here are all 51 problem areas:

I wrote yesterday about an upcoming paper from MSU researchers about prediction of future academic success from current measured background characteristics. Realizing that I could, I requested a prepublication copy of the paper. Fewer than five hours later, at 7:41 this morning, what arrives in my inbox but a response from head researcher and chairperson of the MSU psychology department Dr. Neal Schmitt? So I quickly leapt at the 14,000 word, 56 page paper and accompanying 120 pages of side material. I mostly just skimmed those supplements, but they were interesting too. I see why people want to take psych 101 so much.

Now, I'm not going to pretend that I caught all the references (there were something like 50 papers cited, and I stopped looking them all up after a few pages. Kristof-Brown et al. 2005 (CONSEQUENCES OF INDIVIDUALS' FIT AT WORK: A META-ANALYSIS OF PERSON–JOB, PERSON–ORGANIZATION, PERSON–GROUP, AND PERSON–SUPERVISOR FIT) was very popular. And I'll be honest again--even after reading the study, the result tables (e.g., Results of Multivariate Latent Growth Modeling of Fit, Satisfaction, and Student Outcomes, table 4 p.49) didn't make 100% sense to me. Which is good, because that fact is essentially an argument for graduate study in psychology, and I don't pretend to be a savant for psychology. I just try to sound out the words and derive meaning from context as best is possible.

It's not exactly the most accessible of beach reading; the hypotheses are phrased like this: [H1] Perceptions of person-organization fit will lead to satisfaction, which in turn will lead to behavioral outcomes such as intent to turnover and performance. Totally straightforward for the next 48 pages, just like that. (Just kidding, there are actually 7 hypotheses with subsections for some of them.)
Sample was of 2,771 undergraduates, though there was not insignificant attitrition over the course of the study. Related to that, here's a fun snippet:

As can be seen, the final sample has a smaller proportion of African American students and a correspondingly greater proportion of Caucasians and Asian Americans. The average SAT/ACT scores and high school GPAs of the final sample were higher than those of the initial respondent group.

Blacks were also intentionally over-represented. : \

In any case, here is what the study sounds like:

In similar fashion, we believe that those who are highly extraverted will perceive increased fit and satisfaction with college life over time. At most universities, including those attended by the students in our study, most college freshmen spend their first year living in close proximity in dormitories. In addition, in most dorms, students will have one or more roommates, will likely share bathrooms and other facilities, eat some or all of their meals in college cafeterias and take part in social activities organized by their resident assistant. Those students who are extraverted are more likely to make friends in these situations and perceive that they fit in this atmosphere, whereas less extraverted students will not enjoy the social closeness and lack of privacy they tend to encounter in dormitory life. This close living proximity inevitably requires tolerance of others’ behavior and idiosyncrasies as well as willingness to share and compromise, so we believe that Agreeableness will also be positively associated with perceptions of fit and satisfaction with students’ social situation. Hence it seems reasonable to pose the following hypothesis:
H7: Extraversion and Agreeableness should be related positively to changes in perceptions of social fit and social satisfaction.

Now, the bolded section (emphasis mine) does illustrate the fact that not all of the hypotheses here are exactly mind-blowing. Still, very interesting stuff, as everything has nice clean data behind it. Some of the hypotheses were more spectacular than the seventh one, above.

In any case, I do take issue with some of the logic and reasoning here. It doesn't seem airtight in many places. For instance:

Skipping classes and being late to classes may be proximal evidence of physical and psychological withdrawal from the university community, which in turn may be correlated with eventual transfer to another college or with dropping out of college entirely.

This despite recent studies which have shown a huge portion of college populations to skip class fairly regularly; it's just a fact of college life, when you can make your own schedule. Is it becuase they're listening to the lectures on their iPods? Maybe, but it might just be that they're extracting themselves from class to do other things around the college community. I'm not saying that many people cut class for community service, but it isn't always just for sleeping in.

Some of it was just a little bit... sketchy.

To measure CWB, we adapted items from Bennett and Robinson’s (2000) organizational deviance scale. Items reflect targets of deviance that are both interpersonal (e.g., “made fun of another student at your school”) and organizational (e.g., “stuck gum under your desk”) in nature.

My favorite part was the citing of one question on the "vocational interest" section of their survey which had "Refinish Furniture" as an option. It did not go unselected as a favorite pasttime. After the abstract and hypotheses and methodology there was another... 30 pages or so, of math and statistics, which wasn't very exciting. After that, some charts. Again, if you want to know more, I can definitely send along those papers or you can contact Dr. Schmitt et al. yourself. It's interesting, but not far enough along--and I hope it never is. The title to one of the accompanying papers alone tells us just how frightened we should be of this sort of standardization-heavy approach to admissions: The Use of Background and Ability Profiles to Predict College Student Outcomes. That doesn't sound so friendly to me... I'll just leave you with the abstract, which is fairly fuzzy:

College admissions tests predict college performance well, particularly first-year grade point average (Kuncel, Hezlett, & Ones, 2001, 2004). Noncognitive measures such as those measuring interests, background experiences, and motivational characteristics may add incremental validity to traditional cognitive college admissions tests and high school grade point average, in that they assess a broader range of dimensions reflecting the potential of college students – such as those measuring leadership, interpersonal skills, and ethics.

Oh, and one more thing, from one more paper (Developing a Biodata Measure and Situational Judgment Inventory as Predictors of College Student Performance,

Colleges and universities seek out the best students possible for their institutions, where ‘best’ can be defined in many ways. Traditionally, selection systems for college admissions typically use standardized tests of verbal and mathematical skills, and possibly records of achievement in specific subject matter areas. Such systems have worked well for decades ...

Start laughing and crying whenever you like.

Plus!

Bonus featurette from an AIM chat with Tom Gram about the study:

13:32:07 Tom Gram: u got that paper already?
13:32:56 me: check your email
13:33:47 me: schmitt hit me up at 741 this morning, less than 5 hours after I sent it
13:34:18 Tom Gram: thats hardcore
13:34:37 me: yeah he's chair of the department too
13:34:40 me: guess he has time to respond to emails
13:35:00 me: I thought about signing my email "future psych major" but decided against it
13:35:52 Tom Gram: haha, your email is quite friendly and enthusiastic about his work
13:36:06 me: that's how I got such a quick reply, of course
13:36:29 me: "dude u sux y u gotta be hatin on HS studnets" does not a prompt reply bring
13:41:21 Tom Gram: haha i like this so far, hes basically like....there was evidence that students' liking toward the learning enviornment affects performance
13:41:45 me: satisfaction correlating with a positive change in environmental fit perception, yes

see also this one from Mike Lee:
Mike Lee: I thought the whole personality thing was supposed to be judged by the essays/extracurrics
me: oh god no
me: that's not STANDARDIZED

(Article truncated here)

update Aug 2: the Research being done at MSU by MSU researchers, MSU has belatedly put up a nice, extra-informative article on the subject HERE.

update Aug 3: I emailed Dr. Schmitt, chair of the MSU psych department, and got the prepublication papers, which are very interesting. I'm reading through them now (~175 pages, so sort of skimming) and will report on them soon. Yeah, I know, excellent use of my time.

Professors Find Ways to Predict Student Success

By KATHLEEN POLESNAK

http://www.statenews.com/article.phtml?pk=37042
The State News

Grade-point averages and SAT scores can only do so much to predict student success at college. That's why two MSU professors researched other ways to anticipate how student life beyond the books plays into college performance.After looking at students' interests, background experiences and motivational characteristics from 10 universities during a four-year period, psychology professors Neal Schmitt, Frederick Oswald and a team of undergraduate and graduate students found predictors for student potential, including their likeliness to cheat, drop out of school and attend classes regularly.

The College Board approached Schmitt and Oswald about conducting the study, which it plans to use to enhance college admissions.

"I think it's a step forward in terms of learning about college performance more broadly and how to admit students that will create a more well-rounded student body," Oswald said.

Oswald and Schmitt started by examining college mission statements to find which student traits universities desire. They narrowed them down to 12 categories, including artistic and cultural appreciation, social responsibility and citizenship, and career orientation. [...]

"It is daunting in a sense that no test is perfect in predicting students' success," Oswald said. "We did the best as we could to make the test as reliable and predictable of student outcomes as possible." "You lose some of that individual richness, but what you gain is the consistency, and it's interpreted in a similar way on the admissions side of things," Oswald said.

The next step in utilizing the information is extending the research to high school seniors, said Viji Sathy, an associate research scientist for the College Board.

Good:

• "If you expand your selection beyond that of ability, you can look at these elements and expand the array of (students at) the college."
• "I think it's a step forward in terms of learning about college performance more broadly and how to admit students that will create a more well-rounded student body," Oswald said.

Bad:

• "We're just trying to find a way to measure people,"
• "You lose some of that individual richness"

Worst:

• The College Board approached Schmitt and Oswald about conducting the study, which it plans to use to enhance college admissions.
• The next step in utilizing the information is extending the research to high school seniors, said Viji Sathy, an associate research scientist for the College Board.

Scary...

Read part 2 of this exciting story here: Feed it Gossip hear your future GPA pt. 2

BIO: 770 LIT: 760 and M2C: 760.

My impressions: Biology was very easy, English Literature was very arbitrary and obtuse, and Math was very easy with one or two obnoxious places. The Literature was just hugely, hugely obnoxious. I took the old SAT II Writing when I was applying for Exonian copy editor Lower (read: Sophomore) year at Exeter, and it was really not bad, but the English Literature exam was gruesome because on so many questions (all of them) you had to fit your head into the ETS “retard” box. Interpretation was not open. Just as the 20 minute essays are flawed because of the fact that they are twenty minutes long, so too I feel the literature test is inherently broken because literature can be interpreted different ways, so long as multiple choice isn’t the only format. Whatever.

Shameful and disappointing for me, of course. I do ‘well’ and all I’m left with is an emptiness knowing that I didn’t reach my full potential. Quite depressing. I’ve trackbacked something like 20 other blogs found via Technorati on the subject here, of SAT2 tests, because most of them share what I can only regard as a tragic perspective: they are accepting and in fact pleased with low scores. There are sort of exceptions here and there, “anything below 750 would be unfortunate” type moanings, like those complaints you’ve just read, but for the most part people don’t seem to mind that they’re getting these 610s or whatever the hell low scores they have. I’m not saying that they’re stupid, just that those scores are lower than they could be. Which is true. Average is pretty… average, so even being moderately above average is moderately lame.

Anyways back to my discussion of relativism in standardized testing. The point is that I compare myself not to the national average this or that, or anything like that. I set my baseline somewhere around the 99.5th percentile. I’m not saying that I’m in the top .5 percent, I’m just saying that’s around where the standard lies. For example, my best friend in the entire world, my neighbor Greta Friar, always does much better than me on these tests. We both took the Eng Lit; she got somewhere something like an 800. Maybe she got one wrong, but I think she still got an 800. That was my recollection from the discussion which went “what did you get?” [minutes of filer] “something something 800″ “oh nice” [more on]. She’s very modest about it so it isn’t soul crushing or anything but combine that with the 2370 SAT I and it can be a little bit harrowing. So the point is that when I look at my peers in the higher percentiles even a good performance of mine is transformed into a half-assed or half-hearted one and so it’s pretty bleak in that regard. But it’s all handled on the fly, subconsciously, most of the time. Only rarely do I need to actively think about diluting my achievements in the oceans of others accomplishments; most of the time the job is quite simple.

College admissions officers in Massachusetts and elsewhere yesterday scrambled to deal with the applications of thousands of students whose SAT scores were too low because of a technical glitch, one of the biggest mistakes ever made on the high-stakes exam.

Ooops! Someone messed up. Ostensibly, a computer, but really there’s a human behind it in the end. This was covered by the NYT yesterday, but now the Boston Globe has some interesting notes about how it affects Massachusetts colleges.

Officials at The College Board, which administers the test, said technical glitches led to errors in roughly 4,000 students’ October 2005 tests, resulting in some students not getting credit for some of their correct answers. …

The College Board said that 83 percent of the incorrect scores were too low by 40 points or less. Five percent involved 100 points or more, and 16 students’ scores were more than 200 points off on the 2,400-point exam.

Sucks to be one of those ~4,000 people, I suppose.