Your test got easier. That is not the same as better candidates
Raw scores from two hiring cycles do not share a scale. Freeze a reference date, put anchors on every paper, and test the anchors for drift before you link.
Rajendra Kumar · 22 November 2026 · 9 min read

Representational image.
The 2026 campus batch scored 66 percent on your cognitive test. The 2023 batch scored 60. The deck says the hiring bar has moved up, and nobody in the room objects.
Then someone prints the item analysis. Eight of the forty questions have appeared on the paper every year since 2023. Those eight went from 62 percent correct to 71 percent correct. The candidates did not get smarter in the gaps you thought you were measuring. The paper got easier exactly where it had been reused.
That gap is not noise. It is the difference between a score and a scale.
A raw score is a claim about a date, not about a person
Item difficulty in classical test theory is the proportion answering correctly, the p-value, and it is computed from whoever happened to sit the test. Change the sample and the number changes, even when the item and the construct have not moved. Teaching material on traditional item statistics puts it plainly: the statistics are dependent on your sample of test takers.
Two things follow, and both bite HR teams.
First, 66 percent in 2026 and 60 percent in 2023 are not the same unit. Raw scores compare only when both cycles sat a paper of equal difficulty with candidates of equal composition. In most Indian hiring funnels neither condition holds for long. Intake channels change, college mix changes, the screening filter changes, and the paper is partly reused because writing new items is slow.
Second, a test without a scale cannot support the claims stacked on top of it: the bar is up, training worked, this cohort is weaker. Those are statements about change, and change needs a fixed unit. A percentile has its own version of this problem, which is why a percentile can mislead even when the maths is right.
Three forces move item difficulty. Only one of them is about candidates learning.
Exposure. Reused items leak. Coaching centres collect them, candidates compare notes, and the same twenty anchors repeated twice a year get steadily easier. Ofqual's guidance on maintaining standards names overuse of items as a direct cause of drift: items become easier and their parameters move.
Cohort composition. A batch drawn from a new set of colleges, or hired after you changed the screening cut-off, is a different sample. The p-value moves because the group moved.
Construct change. An item about a technology or a regulation means something different five years on. The Rasch measurement literature has a clean example: an HIV question that was obscure immunology in 1986 had become current affairs by 1992. A correct answer on the two dates reflects different knowledge, so the same key measures two different things.
None of the three is fixed by buying a better test. All three are fixed by refusing to let the scale float.
Name a reference administration and freeze it
The scale needs one anchor point in time. Choose a single administration, ideally one with a decent sample and a clean item analysis already done, and declare its item difficulties to be the definition of the scale.
Record four things in the item bank: the date, the form, the sample size, and the composition of the calibration sample (college tier, experience band, gender split, whatever your funnel actually varies on). From then on, every score you report is expressed on that scale: "reported on the August 2025 scale".
The date does not move to suit a convenient year. If you eventually have to move it, that is a scale change, and every report afterwards needs a note saying so, plus the link between the old and new reference administrations.
This is the part teams skip because it feels bureaucratic. It is the only part that makes the rest arithmetic instead of argument.
Put a common block on every paper, twenty percent or more
An anchor block is a set of items that appears on every form, unchanged, and is used to translate each new cycle onto the reference scale.
Practical rules:
- Size. The standard rule of thumb reported in Ofqual's review of equating practice is that more than one fifth of the items should overlap between two forms for reliable equating. On a forty-item paper that is eight to ten anchors.
- Spread. Pick anchors across the difficulty range, not the eight easiest questions. If every anchor sits where 70 percent of candidates score, you learn nothing about the top or the cut-off band.
- Position. Embed them through the paper. Items placed at the end get rushed and skipped more in large rooms.
- Handling. Where you can, keep anchors out of individual candidate decisions: they exist to carry the scale, not to rank people. Never show them in candidate feedback, and retire an anchor after a set number of exposures before it becomes a known quantity.
Skipping the anchor block is what makes year-on-year score comparisons fiction. Adding a parallel form instead of reused items is decent practice, but adaptive and fixed forms fail in different ways, and both need common items to be linked back to a reference scale.
Test the anchors before you link them
An anchor is only a ruler if it has not changed length. So the drift check comes before the equating, never after.
Take each anchor item and compare its current difficulty with its frozen value, in logits. If you have no IRT software, logits from the p-value are close enough for a decision: logit equals the natural log of p divided by one minus p. If you do have item parameter estimates, use the displacement between the anchored value and the value the current data would have produced.
A displacement threshold is a judgement call with published precedent:
- Wright and Douglas found random displacement below 0.5 logits has little effect on the instrument.
- Draba recommended 0.5 on the reasoning that item difficulties usually span about minus 2.5 to plus 2.5 logits, so 0.5 is ten percent of the range.
- The American Board of Family Medicine treats 0.6 logits or more as excessive and typically flags 10 to 25 percent of its anchors. The American Board of Pediatrics flags 5 to 15 percent on its largest exams, where n exceeds 500, and 10 to 30 percent on low-volume subspecialty exams.
Pick one rule, write it down, and keep it. A 3-sigma p-value rule and an absolute logit threshold answer different questions, and switching rules between cycles is how a team talks itself into a result.
Decide by count and symmetry, not by convenience
The pattern of flags matters more than any single flag. Three cases, three actions.
| What the flags look like | What it means | What to do |
|---|---|---|
| Few flags, scattered evenly above and below zero | Ordinary calibration noise, roughly what ABP reports on large exams | Unanchor those items, re-estimate the link on the rest, publish the comparison |
| Many flags, all in the same direction | The paper got easier, or the cohort got weaker. You cannot tell which from the data | Do not link. Report the cycle as an unreferenced administration and say the series broke |
| One or two flags with an obvious story (a leaked paper, a rule change, an academy coaching a specific drill) | A specific item died | Retire the item, keep the link on the remaining anchors, note the retirement |
The mistake to avoid is treating every flag as an item to fix and re-anchoring until nothing is flagged. The same article that gives the 0.6 threshold makes the counter-argument: too much flexibility in letting items float makes the substantive meaning of the construct fuzzy. If you unanchor a third of the block every year, you are not equating, you are relabelling.
Report in the anchored metric, with the error attached
Once the link is estimated, every number in the results deck changes unit and gets an uncertainty.
Do not report "mean score 26.4". Report "mean score 22.8 on the August 2025 scale, link estimated from eight anchor items". If the corrected difference has a confidence interval, print it. A difference whose interval crosses zero is not a finding, and the honest line is "no measurable change this cycle".
That line costs you a slide. Not printing it costs you a decision made on a number that drifted.
A worked example
Numbers below are illustrative, sized to a mid-volume Indian hiring funnel.
Reference administration: August 2025, 210 candidates, forty-item paper, mean 24.0 items (60.0 percent). Eight anchors, mean p of 0.62, which is 0.49 in logits.
Current administration: June 2026, 248 candidates, same paper plus four new items, mean 26.4 (66.0 percent). The eight anchors average p of 0.71, which is 0.90 in logits.
Anchor displacement is 0.41 logits. Under the 0.5 rule, no single anchor is flagged for excessive drift. Read only the flags and the paper looks fine.
Now convert the shift into score points. The anchor block moved 0.09 in proportion terms, which on a forty-item paper is 3.6 items of pure paper difficulty. Correct the observed mean by that amount: 26.4 minus 3.6 gives 22.8, or 57.0 percent, against 60.0 percent in the reference cycle.
The headline changes from a six-point gain to a three-point loss, and neither number is trustworthy on its own. The combined standard error of the anchor difference works out to about 0.044 in proportion terms, or 1.8 items. Double it for a rough 95 percent interval and the correction carries roughly plus or minus 3.5 items of uncertainty. The honest reading: the six-point gain was mostly the paper, and the data cannot distinguish "no change" from a four-item drop.
That is a less satisfying sentence than the one in the original deck. It is also the only one that survives a challenge from the business head.
Where this stops working
The block-mean correction assumes uniform drift. It applies one shift to the whole score distribution, which is what a simple anchor correction does. If the easy items leaked and the hard ones did not, the correction is wrong at the tails, and the tails are where your cut-off lives. Item-level methods exist for exactly this reason. For a hiring test, the practical fix is to check the flagged items' positions before trusting a corrected mean near a cut-off.
Cohort size limits what you can certify. To resolve a one-item difference at two standard errors with items near p of 0.6, you need on the order of 1,500 candidates per cycle. A funnel of 250 can detect a shift of about three to four items, no finer. Small cycles can avoid a false positive. They cannot certify a real two-point gain, and no amount of clever arithmetic changes that.
The drift threshold and the reporting threshold are different questions. In the worked example, 0.41 logits of anchor movement passes every published drift rule, and it simultaneously manufactured six points of apparent improvement. "Our anchors passed" is not the same sentence as "our comparison is valid". Keep both checks.
Equating does not rescue a leaked item. It records that the item is dead and stops it from corrupting the scale. The item still has to be retired, and the cohort that saw it still has a score built partly on exposure rather than ability.
Do this before the next cycle
Open the item bank this week and write the reference date into it, with the sample composition next to it. Put eight anchors on the next paper, spread across the difficulty range, embedded mid-paper. Compute the anchor p-value before anyone sees the cycle mean. Then add one line to the results deck: "Scores are reported on the August 2025 scale, linked through eight anchor items, standard error of the link X."
If that line cannot be written, the comparison should not go in the deck. Which is a better outcome than the alternative, where a drifted paper quietly decides who gets hired and who gets told they were not good enough. Run the measurement through a system that keeps the history in one place, like the reporting built into BolderBrain's assessment platform, and retesting the same candidates with the same items stops looking like a free comparison.
The habit is small and unglamorous: every score belongs to a date, and every date needs a ruler you have not moved.
Next step
Run the same measurement on your own team
Bring a real team and a real competency. We set up the assessment, run one cycle, and walk you through the report so you can judge the output yourself.
Written by
Rajendra Kumar
Head of Enterprise Business, Conceptial
Runs enterprise work at Conceptial, the training and consulting firm behind BolderBrain. Two decades of building assessment and development programmes for Indian enterprises.