Retesting candidates: what a second attempt actually measures

A candidate scores 58 against a cut-off of 60. Six weeks later he sits the test again and scores 63. The file now says 63, and the 58 is a footnote nobody reads.

Ask what changed between the two sittings and the honest answer is usually not reasoning ability. He knew the instructions, he knew how the timer behaved, he knew which sections eat time, and if the company reused the form he may have remembered a handful of items. On a 15-point scale, those things are worth about four points. He moved four.

That number is not a guess. It is one of the better established findings in selection research, and it has a policy consequence most hiring teams have never written down.

The average gain is about four points, and it is not learning

The best single estimate comes from a meta-analysis by John Hausknecht and colleagues in the Journal of Applied Psychology. They pooled 50 studies, 107 samples and 134,436 participants and found an adjusted overall effect size of 0.26 for retesting a cognitive ability test. They also note that somewhere between 25% and 50% of applicants in organisational and educational settings are retested, so this is not an edge case.

On the familiar 15-point scale, 0.26 of a standard deviation is about 3.9 points. The gain was larger when practice came with test coaching, and larger again when the identical form was used.

A second meta-analysis, by Jana Scharfen and colleagues in Intelligence, extended this to repeated administrations. Across 174 samples, 122 studies and 153,185 people, the pooled gain at the second sitting was 0.29 of a standard deviation and at the third 0.33, with no further gains after the third. The size of the gain depended on the ability being tested, whether the form was equivalent, how long the interval was, and the candidate's age.

Why the score rises matters more than how much it rises. The usual account names three causes. The underlying ability genuinely improved, which is implausible over six weeks. Distorting factors such as anxiety, unfamiliarity and incomplete understanding of the rules were reduced. And item-specific memory kicked in. Almost all of a single retest gain comes from the second and third causes, both of which are properties of the test rather than the person.

An identical form is a memory test of your own item bank

The moderator finding is the one to act on. When the retest uses the same items, some of the gain is recall: this question looked like that one, the last block had two answers in a row that were the same letter. When the retest uses an equivalent form, the memory route is closed and only the familiarity route remains. So a policy that says "retake the same test" is partly measuring how well the candidate remembers your bank, which is precisely what your test security controls are supposed to prevent.

That distinction matters for high-volume hiring in India. A 200-item bank reused across four campuses in one season has been seen by hundreds of candidates, so the retest score is contaminated by shared memory, not just personal practice.

Same test, ten attempts: what 4,726 candidates showed

The field study worth knowing is older and less quoted. Hausknecht, Trevor and Farr followed 4,726 candidates for law enforcement positions who retook identical selection tests. Scores rose significantly between the first and second sitting and again between the second and third.

Then the interesting part. Among 1,515 candidates who were eventually hired, the number of tests a person needed to gain entry was positively associated with later training performance and negatively associated with turnover probability. Candidates who had to come back more often did slightly better in training and left less often.

Read that carefully, because it cuts both ways. The encouraging reading is that persistence carries information the first score does not, though the authors explain it through persistence and continuance commitment, which is a hypothesis about a construct rather than proof that attempt count measures grit.

The uncomfortable reading is that if you keep the highest score and discard the attempt count, you are selecting on persistence anyway, without saying so and without validating it. A retest needs a free day, travel, and in many Indian pipelines a fee, so attempt count also tracks money and leave. Treat persistence as a criterion only if you will defend it as one and monitor it for adverse impact.

Decide what the retest is for before you write the rule

Most retest policies are one sentence someone added after a candidate complained. Three different problems get that sentence, and only one is a genuine retest.

The third row is the common one, and in a shortlist of 30 it quietly adds a variable called willingness to retake, weighted by however many points practice adds, and leaves it out of the validation file.

The interval is the lever you control, and it fights the calendar

Retest gains shrink as the interval between sittings grows, one of the moderators Scharfen and colleagues identified. Long intervals let item memory and anxiety reduction fade.

So you have one dial and one constraint: push the retest out, but a hiring decision cannot wait a quarter. Pick the interval deliberately and write down why. A four-week minimum for a boundary retest, inside the same cycle, is defensible. Leaving it unstated is not, because two candidates with the same score then get different treatment depending on when they applied.

Judge one person's delta against the standard error, not the average gain

Here is the arithmetic almost nobody runs.

The standard error of measurement tells you how far a single score can sit from the person's true score. For a scale with a standard deviation of 15 and a reliability of 0.90, the standard error is 15 times the square root of 0.10, which is 4.7 points. A 95% band around any individual score is roughly plus or minus 9.3 points.

Compare that with the average gain of 3.9 points. The average retest gain is smaller than the error band around one person's score. So when a candidate moves from 58 to 63, the data say the second score is not distinguishable from the first. The five-point move is inside the noise.

The group average is real. Aggregate it across a hundred retests and the practice effect emerges from the error because the errors cancel. One candidate is not a hundred candidates.

A decision rule that survives this:

1. Record both scores, the attempt number, the form identifier and the days between sittings.
2. Compute delta and compare it with 1.96 times the standard error for that instrument.
3. If the delta is inside that band, write the honest sentence in the file: the second score is not distinguishable from the first. Decide on the work sample and the interview.
4. If the delta clears the band, ask what caused it. On an equivalent form the candidate may have settled. On an identical form, suspect exposure first.

A worked example, with illustrative numbers

Take a pool of 480 candidates for an operations role. Cut-off at 60 on a 15-point scale, reliability 0.90, so the standard error is 4.7 points. Ninety-six candidates retest, a rate of 20%, and the mean gain on the second sitting is 4.1 points.

Under a keep-the-highest rule, 11 of them cross 60. Their original scores run from 52 to 59. Every one sits inside the 95% band of about plus or minus 9.3 points, so each was plausibly a 60 the first time and plausibly a 55 the second. The procedure did not identify eleven stronger candidates. It re-rolled the dice eleven times and recorded the favourable roll.

The process looks like it is working, because 2.3% of the pool changed decision, enough for a recruiter to feel the option earns its keep. It cost 96 extra sittings. A visible benefit, an invisible cost, and no way to separate them in a spreadsheet that stores only the final score.

Keep the attempt count, never fold it into the score

Two fields, not one. Score of sitting one. Score of sitting two. Plus attempt number, form identifier and interval in days. That needs a system that keeps each sitting as its own record rather than overwriting the last one, which is how BolderBrain's assessment reports store attempts.

What not to do is average them. The mean of 58 and 63 is 60.5, and no sitting ever produced that number. It mixes a contaminated score with a clean one. Averaging only helps when both measurements come from the same conditions, and these did not.

If you use attempt count, give it a weight, record it for every candidate including those who never retested, and review its impact by group each cycle. A variable that enters through administrative convenience is how a selection system acquires a criterion nobody chose.

Three numbers to check every hiring cycle

You do not need a psychometrician to watch these, and each one points at a different failure.

Retest rate. The share of candidates who sit twice. A jump from 8% to 30% in one season usually means items are circulating, not that candidates got keener.

Mean delta by form type. Compute the gain separately for identical and equivalent forms. If the equivalent-form delta matches the identical-form delta, either your forms are not equivalent or the items were shared. The research predicts a larger gain on identical forms, so a departure is information.

Share of retests that changed the decision. If it is zero, you are paying for sittings that change nothing. If it is large, your cut-off sits inside the noise band, and the fix is a band-based decision rather than more retests. The cut-off question is a separate problem with a separate answer.

The honest limit

None of this tells you whether a candidate who gained four points is better at the job. It tells you whether the score moved beyond what the instrument can resolve, which is a narrower and more useful question.

It also does not fix an exposed item bank. If candidates are sharing questions on a group chat, an equivalent-form policy reduces the damage and does not repair it.

And the average hides the individual. The pooled 0.26 is a mean across 134,436 people. Some gained nothing and some gained fifteen points. Without item-level data you cannot tell which kind of candidate is in front of you, so the error band stays your only honest tool at a sample of one.

What to do on Monday

Open the document that says what happens when a candidate retests. If the sentence reads "the higher score will be retained" and nothing follows, replace it with five lines: two sittings maximum, equivalent form on the second, a stated minimum interval, both scores kept with attempt number and form identifier, and a rule for when the difference falls inside 1.96 times the standard error.

Five lines does more for the defensibility of your selection than a better item bank would, because it is the part a candidate or an auditor can check.