Patterns in international GRE scores

Why writing up my earlier post I stumbled onto to some interesting GRE data for applicants for various countries. I transcribed the results for all nations with sample sizes greater than 500. What you see above is a plot which shows mean quantitative and verbal scores on the GRE by nations.

The correlation in this set of countries between subtests of the GRE are as so:

Quant & verbal = 0.33

Verbal & writing = 0.84

Quant & writing = 0.21

Basically, the writing score and verbal score seem to reflect the lack of English fluency in many nations.

Many of these results are not too surprising if you’ve ever seen graduate school applications in the sciences (I have). Applicants from the United States tend to have lower quantitative and higher verbal scores. This is what you see here. It’s rather unfair since the test is administered in English, and that’s the native language of the United States. No surprise the United Kingdom and Canada score high on verbal reasoning. Ireland, Australia, and New Zealand didn’t have enough test takers to make the cut, but they all do as well as the United Kingdom. Singapore has an elite group which uses English as the medium of instruction in school.

I didn’t include standard deviation information, even though it’s in there. India has a pretty high standard deviation on quantitative reasoning, at 9.1. In contrast, China only has a standard deviation of 5.2 for quantitative reasoning. More than twice as many Indians as Chinese take the GRE.

Finally, I want to observe Saudi Arabia, as opposed to Iran. Both countries have about 5,000 people taking the GRE every year. About 2.5 times as many people live in Iran as opposed to Saudi Arabia. But the results for Saudi Arabia are dismal, while Iranian students perform rather well on the quantitative portion of the GRE.* This is not surprising to me, having seen applications from Saudi and Iranian students.

Saudi Arabia wants to move beyond being purely a resource-driven economy. These sorts of results show why many people are skeptical: in the generations since the oil-boom began the Saudi state has not cultivated and matured the human capital of its population. To get a better sense, here are the scores with N’s of MENA nations and a few others:

Country N Quantitative
Saudi 4462 141.6
Libya 113 146.2
Iraq 148 146.6
Oman 98 146.9
UAE 238 147.2
Qatar 85 147.3
Kuwait 386 147.8
Algeria 86 149.5
Yemen 68 149.9
Bahrain 55 150.9
Ethiopia 353 151.3
Jordan 472 152.1
Egypt 1044 153.2
Morocco 191 153.7
Tunisia 128 154.1
Georgia 71 154.2
Lebanon 691 154.7
Armenia 84 154.9
Azerbaijan 125 155.1
Eritrea 223 155.2
Israel 344 156.8
Iran 5319 157.3
Turkey 2370 158.9

 The “natural break” is between the Saudis and everyone else. In recent years Saudis indigenized their non-essential workforce. I’m broadly skeptical of the consequences of this.

The data for the plot at the top is below the fold.

Read More

GRE utility for graduate school and conditioning on the dependent variable

One of the things that seems to be popular in biological sciences right now is the push to get rid of the GRE as part of the criteria for entrance. Two of the major rationales are that it’s expensive, so discriminates against lower socioeconomic status candidates, and, that it makes it harder to recruit underrepresented minorities since on average they score lower on the GRE (many departments have either explicit or implicit GRE cut-offs).

I’m not going to litigate these issues. To be honest I believe it is a fait accompli that many departments will stop using the GRE. This will probably increase diversity in some ways. But I also suspect it will result in a greater bias toward more “polished” candidates since very high GRE scores sometimes indicate to admissions committees that applicants who are otherwise spotty or irregular may have promise.

But, I do want to enter into the record a major problem with the argument that GRE does not correlate with academic success at the graduate level (supported by research). Yes, part of the issue may simply be range restriction. But there is another issue which many biological scientists may not be familiar with.

First, right now this paper from early this year is getting a lot of attention, The Limitations of the GRE in Predicting Success in Biomedical Graduate School.

It was, of course, a political scientist who objected immediately:

This blog post is of interest for those curious, That one weird third variable problem nobody ever mentions: Conditioning on a collider. Basically, it is well known that at many universities graduate admittees exhibit a weak negative association between GRE scores and grade point averages. This was commented on as far back as the 1970s in ScienceGraduate Admission Variables and Future Success:

The standard variables considered in selecting students for graduate school do not correlate well with later measures of the success or attainments of the selected students (1, 2). The low correlations have led at least one investigator (3) to propose abandoning one of these standard variables, the Graduate Record Examination (GRE). The purpose of the present report is to demonstrate that variables that are the basis for admitting students to graduate school must have low correlations with future measures of the success of these students.

What’s going on?

As noted in the paper there are some universities which are first-choices for graduate school in a field to such an extent that they will admit candidates who have very high GPAs and very high GREs. In this case, neither of the criteria will predict success because there is very little variation to generate a correlation. But, at many universities, there is a negative correlation between admittee GRE score and undergraduate GPA. That is because very few applicants will be admitted with both low GRE and GPA scores, but some will be admitted with high GRE scores and low(er) GPAs and others with higher GPAs and low(er) GREs (usually there is still a GPA and GRE floor).

Consider the relation:

    \[ R^2 = \frac{r_1^2 + r_2^2 - 2r_1r_2r}{1 - r^2} \]

Where \R^2 is the proportion of the variance of the variable you want to predict, and r_1^2 and r_2^2 are the correlations between GRE and GPA and that the variable of interest, and r is the correlation between GRE and GPA.

Basically, when you have negative correlations you’re going to get into a situation where r_1^2 and r_2^2 are not going to be able to explain a lot of the variance in what you want to predict.

This may seem like a nerdy issue. And it is well known to social scientists. But since the people I see talking about the GRE are academics in the biological sciences I thought I would at least highlight this nerdy issue.

As I said above, I do think GRE is going to be dropped as a requirement at many universities for graduate programs. This is going to be a natural experiment, so we’ll be able to test many hypotheses. The paper above ends like so:

…Without a study in which a sample of the applicants-rather than of the selected students is evaluated, it is impossible to tell [the validity of the criteria -RK]. Yet such a study is completely infeasible. Even if rejected applicants are monitored throughout the rest of their working careers, it is impossible to evaluate how they would have done had they been admitted, because the rejection itself constitutes an important “treatment” difference between them and the selected students. The alternative is to admit a sample of the applicant population without using the standard admission variables to select them-preferably, to select at random.

Selection may not be random, but I believe we may be able to test some hypotheses in the next generation by testing a set of students later on after admittance on the GRE and see what the future correlation is.