Your cognitive ability cut-off is doing less work than you think

At a campus drive in Pune last November, 1,400 final year students sat a 90 minute online ability test. The company made 420 offers to the top 30 percent, and the HR head defended the cut with one number: .51.

That is the correlation between cognitive ability and job performance, he said. The best predictor we have.

He was quoting a real paper correctly. The number has since been revised twice, downward, and the way it was revised matters more than the new figure.

The .51 came from a correction that was too generous

Schmidt and Hunter's 1998 meta-analysis is the paper behind almost every vendor deck in this category. It placed general mental ability at .51 against job performance, level with structured interviews and just behind work samples at .54.

In 2022, Paul Sackett, Charlene Zhang, Christopher Berry and Filip Lievens reworked the arithmetic underneath those figures. Their conclusion was not that the tests fail. It was that the corrections had been applied too generously, and that the field had overestimated the validity of nearly everything it sold. Cognitive ability came down from .51 to .31. Every other procedure dropped by .10 to .20 points. Structured interviews came out on top at .42.

In 2023 the same group went further and used only studies published this century. Across 153 samples and 40,740 people, the mean observed validity was .16, and .22 after correction for criterion unreliability and range restriction.

A correlation of .22 explains under 5 percent of the variance in how those hires performed. The test is not useless. It is doing less work than the meeting believed.

Why the correction was wrong

A validity study is usually run on people who already work for you, which means they already passed your selection. Their scores are bunched. To estimate how the test would behave across a full applicant pool, meta-analysts had to guess how much that range had been squeezed, and the usual guess was built from hiring rates. A hiring rate assumes the test was the only thing that decided who got in. It never is. Interviews, referrals and a manager's preference all take a slice.

The correction removed restriction that was not there. The test looked sharper than it was in every summary that quoted it.

The same squeeze is happening in your applicant pool

Range restriction is not an artefact in old papers. It decides how much your test can do on the day.

A test ranks people by how far apart their scores are. If everyone in the room already cleared the same board exam, or came from the six colleges you recruit from every year, the spread of ability in front of you is narrower than the spread the test was built to measure. The software still prints a rank, so the output looks precise. The differences it is sorting on are smaller than the error it carries.

A test with a standard deviation of 12 points and a reliability of .85 has a standard error of measurement of 4.6 points. Two candidates 4 points apart are, statistically, the same person. In a 1,400 person drive that is hundreds of adjacent pairs, and your cut lands inside one.

What the test predicts, and what it does not

Keep the part that survives. Cognitive ability predicts how fast someone acquires job knowledge, and it predicts task performance. In the revised table job knowledge tests sit at .40, and much of what they capture is ability already applied. If the ramp matters in your role, ability is a legitimate input.

It is quiet on most of the rest. The 2023 paper notes that the older studies behind .51 measured task performance almost exclusively, and that citizenship and counterproductive behaviour correlate far less with ability. If your definition of a good hire includes how someone handles a frustrated customer or a difficult colleague, the test has little to say.

It also does not predict better in complex jobs, at least not in the newer data. The 2023 analysis tested whether jobs that work more with information show higher validity, using an O*NET scale. The correlation was .00. That undercuts the common rule of testing only for senior roles because it supposedly matters more there.

The criterion is weaker than the test. Zhou and colleagues, summarised in the same paper, put the reliability of a single supervisor rating at .46 for managerial jobs and .61 for non-managerial jobs. A rating that is more than half error cannot correlate strongly with anything.

Step one: measure the spread before you set the cut

Most teams keep a pass list. Keep the raw score.

Before the next drive, produce three numbers from last cycle: applicants, mean score, standard deviation. Compare that standard deviation with the one in the test's norm sample, which your vendor has even when the report does not show it. If your pool's spread is 70 percent of the norm group's, your 70th percentile cut is not cutting where you think it is.

Then do the same for the offer list. If you made 120 offers from 1,400 applicants, your selection ratio is 8.6 percent, and the mean standard score of the people you hired tells you how much the test actually separated them. A hired-group mean z of 1.2 and one of 2.0 describe two very different hiring processes. If you are about to rewrite the job description, the same discipline applies to deciding what the job actually requires before you choose an instrument.

Step two: compute what the cut-off buys

In a roughly normal applicant pool, the expected performance of the people you hire, in standard deviations of the performance distribution, is the validity multiplied by the average standard score of the selected group:

expected gain = validity × mean z of those selected

For a top 30 percent cut the mean z is 1.16. For a top 5 percent cut it is 2.06. Use .22 as the validity unless you have your own number.

The shape is the argument. The first cut, from the whole pool to the top 30 percent, buys .26 of a standard deviation. Tightening from 30 percent to 5 percent, which rejects five times as many people, buys another .19. That is the trade being made in the room where the cut is set.

Step three: set a floor, then band the scores

Stop using the test to rank. Use it to admit.

A floor is a score below which someone is unlikely to cope with the learning demands of the job. It is a requirement, and you defend it from the work itself: the reading load in month one, the arithmetic in the role, the pace of the queue. A floor set at the 70th percentile is not a requirement. It is a ranking wearing a requirement's clothes.

Above the floor, band the scores by one standard error of measurement so that everyone inside a band is treated as tied. Then break the tie with something job specific: a 45 minute work sample, a job knowledge test, a structured interview scored against written anchors. Those are the procedures that came out at .33 to .42 in the revised table, ahead of ability. Running the test, the work sample and the rubric on one candidate record is what an assessment platform is for: see how the assessment modules fit together.

Banding is also the honest answer to adverse impact. Under the Uniform Guidelines on Employee Selection Procedures, a selection rate for any group below four-fifths of the highest group's rate is generally treated as evidence of adverse impact. That is a flag that sends you to examine whether the cut is job related. It is not a verdict that the test is biased. A floor you can defend plus a job specific tie-breaker is a stronger answer than a percentile you cannot.

Step four: one administration per cycle

Repetition raises scores on its own. Across 75 samples, taking the same or an equivalent test again improved scores by almost a quarter of a standard deviation, and by the third administration the pooled effect reached d = .51. That is the same size as the validity of the test being used.

Allow unlimited retakes and your ranking increasingly measures who practised. Pick a policy and put it in the candidate instructions: one attempt per hiring cycle, or a retake after twelve months, applied to everyone without exception. The policy matters less than its uniformity.

A worked example

Figures below are illustrative, built to show the arithmetic rather than to describe a real client.

1,400 applicants. Scores with a mean of 50 and a standard deviation of 12. 120 offers to make.

The current rule ranks everyone and takes the top 120, which puts the cut near 65. About 144 people clear it. A candidate at 65.2 and one at 60.6 differ by 4.6 points, exactly one standard error of measurement at a reliability of .85, so the test cannot support the claim that the first will perform better than the second. At the margin, the cut is being decided by noise.

The banded rule. A floor at 56, defensible as the reading and arithmetic load of the role, passes 432 people. Band the rest in steps of 4.6 points. Everyone above 69.8, 69 people, is in. Everyone between 60.6 and 69.8, 195 people, goes to a 45 minute work sample scored by two raters on a four item rubric, and 51 of them are hired on that score.

The test still does real work. It removes two thirds of the pool at the floor and settles 69 offers beyond argument. It no longer decides the 51 offers inside measurement error. That decision moves to a task that looks like the job.

The limits of this advice

Three caveats worth carrying into the meeting.

.22 is an average across jobs, sectors and criteria. It is not your validity. The only number that describes your hiring is the one measured where you hire, and most teams never measure it.

Measuring it locally is harder than it sounds. To detect a correlation of .22 with 80 percent power you need roughly 165 people who have both a test score and a performance rating. For .31 you need about 85. If your last validation study had 40 people in it, it could not have found the effect either way, and reporting "no relationship" from it would have been wrong in both directions.

The criterion is the weakest link. If performance ratings are political, or they are one manager's memory in December, no predictor will look good against them. That is a reason to fix the ratings, not to buy a better test. The same logic drives why so much training evidence fails to convince a CFO: the measurement, not the intervention, is usually the problem.

Do this before the next drive

Pull last cycle's raw scores, not the pass list. Compute the standard deviation. Compute validity × mean z for the cut you used. If the answer is under a quarter of a standard deviation, your test is a floor and a coarse sorter, not a ranking, and the offers inside the band should be decided by a work sample.

That is an afternoon of arithmetic on data you already own, and it will change who gets hired more than switching to a different test provider will.