The GRE is useful; range restriction is a thing

The above figure is from Beyond the Threshold Hypothesis: Even Among the Gifted and Top Math/Science Graduate Students, Cognitive Abilities, Vocational Interests, and Lifestyle Preferences Matter for Career Choice, Performance, and Persistence. It shows that even at very high levels of attainment on standardized tests there are differences in life outcome based on variation. The old joke is that results on intelligence tests don’t matter beyond a certain point…that point being whatever your own position is! But these results show that mathematics SAT outcomes at age 13 can still predict a lot of things across a wide range.

From personal experience people outside of psychology are pretty unaware of the power of cognitive aptitude testing. This includes many biologists. I was reminded of the above figure as I read portions of Richard Haier’s The Neuroscience of Intelligence. If you are a biologist curious about the topic, this is a highly recommended book.

The main reason I am posting this is because a friend in academia suggested it might be useful. There has recently been a backlash against the GRE exam, with support from the highest echelons of the science media. Additionally, many researchers in public forums are expressing objections to the GRE very vocally. Naturally this has resulted in counterarguments…but respondents have to be very careful how the couch their disagreement, because they fear being accused of being racist, sex, or classist. Such accusations might trigger social media mobs, which no one wants to be the target of (and if past experience is any guide, friends and colleagues will stand aside while the witch is virtually burned, hoping to avoid notice).

Because of the request above I finally decided to look at the two papers which are eliciting the current wave of GRE-skepticism, The Limitations of the GRE in Predicting Success in Biomedical Graduate School and Predictors of Student Productivity in Biomedical Graduate School Applications. To my eye they suffer from the same problem as all earlier criticisms: range restriction.

The issue is that if a university is using the GRE and other metrics well as filters for those admitted then there shouldn’t be that much variation to be left to be explained by those measures (the outcome being publications or some other important metric which actually leads to the production of science, as opposed to test scores and grades). The two papers above look at those admitted to biomedical programs at UNC and Vanderbilt, while another study looked at UCSF. These are all universities with standards high enough that there are either explicit or implicit cut-off scores so that many students are removed from the applicant pool immediately (the mean scores are well above the 50th percentile, you can see them in the paper yourself).

When I was in graduate school I was on a fellowship committee for several years, and I had access to GRE scores and grades. But I didn’t really pay much attention to them because there wasn’t that much range. And to be honest if the student was beyond their first year I didn’t look at all as time went on. In contrast, I did look really closely at the recommendations from their advisors. From talking to others on the committee this seemed typical. Once students were admitted they were judged based on how they were doing in graduate school. And how they were doing in graduate school had to do with research, not their graduate school GPA or what they scored on the GRE to get in.

As an empirical matter I do think that it is likely many universities will follow the University of Michigan in dropping the GRE as a requirement. There will be some resistance within academia, but there is a lot of reluctance to vocally defend the GRE in public, especially from younger faculty who fear the social and professional repercussions (every time a discussion pops up about the GRE I get a lot of Twitter DMs from people who believe in the utility of the GRE but don’t want to be seen defending it in public because they fear becoming the target of accusations of an -ism). My prediction is that after the GRE is gone people will simply rely on other proxies.

If the GRE is not required, but can be taken, then students who do well on the GRE will put that on their application. Sometimes strong students encounter tragedies in their undergraduate years which strongly impact their grade point averages, and very strong GREs can help show admissions committees that they can do the coursework despite their undergraduate record (I’m not positing a hypothetical, but recounting real individuals I’ve known of and seen). It seems cruel to deny these students the chance to submit their test scores. This means that those professors who believe the GRE is valid will show preference to students who take the test and have strong scores (and to be sure, many more care about the GRE when it means someone concretely joining their lab, as opposed to the abstraction of who gets admitted to the department).

More broadly, professors who are taking students will look more at proxies for GRE score, such as undergraduate institution, or the prestige of the people writing recommendation letters. That is, pedigree will matter a lot more. In some places, such as Britain, standardized testing emerged in part as a way to identify strong students from underprivileged backgrounds. These are not the type of students who would ever be able to present a prestigious letter of recommendation. This is a sort of student which still exists (often they are from non-academic backgrounds, being the first to graduate from college in their family; what they lack in polish they compensate for in aptitude, but that takes the right environment to express).

The recourse to other variables besides the GRE score will likely have mixed results at best. Consider the successful campaign to ban asking for job applicants’ criminal records. It turns out that just increased discrimination against all young black men, because employers could not longer differentiate. In general I think removing the GRE would probably hurt graduates of less prestigious state universities the most (and of course students from East Asia, who tend to have a comparative advantage on standardized tests). I’m pretty sure we’ll see, as the experiment will be run.

Addendum: There are professors at relatively prestigious research universities who had mediocre or sub-par GRE scores. We all know them. To some extent I think many of these individuals almost take pride in the fact that they accomplished so much in science despite negative feedback due to their unimpressive test scores. But remember that we’re talking about trends and averages, not deterministic predictions. Nothing in science is guaranteed, and even if you start at Harvard with undergraduate publications (not first author, but still) in Nature you may not make it that far (I’m thinking of a friend of mine, alas, who picked the wrong lab/project and couldn’t recover).

6 thoughts on “The GRE is useful; range restriction is a thing

  1. The alternative to objective measures is subjective measures. They include the old boy network and the 3 martini lunch. Because the primary stress on elite parents is the worry about their ability to pass their status on to their dull or lazy children, they will favor subjective measures so that they can promote their own offspring. works like a charm.

  2. * Graduate admissions currently overweights test scores and underweights undergrad GPA relative to their respective predictive values. http://washparkprophet.blogspot.com/2016/10/what-predicts-law-school-grades.html

    * While Physics GRE test scores are predictive of graduate school success for most physics graduate students this is far less true of astronomers, for reasons that probably have less to do with range restriction than the pertinence of the material on the test to the subfield. https://arxiv.org/abs/1512.03709

    * A couple of law schools are starting to accept the GRE because the GRE and the LSAT predict the same things. https://www.nytimes.com/2017/08/10/business/dealbook/law-school-gre.html

    * There are at least two dimensions of career success that aren’t captured by GRE/IQ type measures. One corresponds more or less to the Big Five Personality Trait conscientiousness which is probably one of the important reasons that grades are more predictive of graduate school success than test scores. The other corresponds more or less to the Big Five Personality Trait extraversion or more generally to social skills. http://washparkprophet.blogspot.com/2011/04/beyond-heritable-iq-in-socioeconomic.html

    * Not quite on topic but interesting. The ACT is more accurate if you drop two of its four sections. http://washparkprophet.blogspot.com/2017/01/act-math-and-english-much-more.html

  3. “Graduate admissions currently overweights test scores and underweights undergrad GPA relative to their respective predictive values.”

    The link seems to suggest that graduate admissions currently underweight test scores but underweight undergrad GPA even more. Low correlation from the selected population is the expected result with proper use.

    Under an assumption that people generally go to the “best” school which admits them, there should be almost no correlation between standardized test scores and performance:
    People with lower standardized test scores who are still admitted should have compensating factors that cause them to do just as well as the people with higher standardized test scores, and vice versa people with higher standardized test scores should do worse on the other key factors that matter.

    People who score better on all axes should get into even better schools and generally choose to go them, in a (multidimensional) range restriction effect that doesn’t apply when you just have a selection cutoff. This doesn’t apply at the very top schools, but there you have a range restriction in the raw scores on the standardized test (“everybody in a top math grad program gets a 990 on the Math Subject Test”) plus a range restriction on grades due to grade inflation (“everybody at a top program deserves an A.”)

    Note that especially at undergrad, the applicants who are overqualified and admitted at “better” schools but matriculate anyway are disproportionately likely to have other ties to the school, like being a legacy. This is why schools can point to legacies being, on average, just as qualified or overqualified compared to ordinary applicants even as they do get an advantage in admissions– the overqualified people who choose to go to their parents’ alma mater balance out the people who wouldn’t get in without the advantage. This doesn’t mean that being a legacy isn’t an advantage (and maybe shouldn’t be), just that we should expect the stats of legacies to be even better if it weren’t.

  4. JT, yeah, obv a weighted thing re: admissions. if an applicant had a lot of undergrad research & some pub(s) then GRE & GPA mattered a lot less pretty clearly. also if the grad dept. was strong in a particular sub-field of student interest then you’d get stronger applicants judging by various metrics.

    it’s not rocket science.

  5. I took the GRE about 10 years ago before they changed the scoring scale and had the computer adapt the questions on the fly to how you were doing.

    I scored a perfect 800 on the math, which put me in the 94th percentile. Seemed a pretty dull instrument at the time. Especially since my math skills topped out at basic calculus (though I don’t think that any questions even required that.)

    The test was a formality for me, as I was applying to library school. Having taken the GRE was a requirement for admission; score was immaterial.

Comments are closed.