A percentile is only as good as the norm group behind it

A shortlist reached me last month with a column headed "percentile". Every candidate in the top band had scored above the 80th percentile on a reasoning test, the HR head said, so the band was the shortlist. Forty openings, ninety names, one number per name.

Two questions took the column apart. Which year were the norms collected? Nobody knew. Who was in the norm group? Nobody knew that either, and the vendor manual, when we found it, answered both in a footnote: a United States sample, tested in 2015.

A percentile is a statement about a group, not a person

This is the part that gets skipped in every score-report walkthrough. A percentile says: this person scored at or above this share of one specific comparison sample. The 82nd percentile means 82 percent of that sample scored at or below this candidate. It does not mean 82 percent correct, and it does not mean 82 percent of your applicants.

So the number can move without the candidate answering a single extra question. Three things move it: the date of the norming study, the population that was sampled, and the size of the sample in the cell you are reading. Each one has a check, and each check costs about ten minutes.

Norms go stale on a schedule you can estimate

Tests do not stay calibrated to the people taking them. Scores on the same test rise over time as populations change, which means an old norm table flatters a current candidate. The standard meta-analysis of this effect pooled 285 studies and 14,031 participants and put the average at 2.31 standard score points per decade, with a 95 percent confidence interval of 1.99 to 2.64. For modern Wechsler and Stanford-Binet tests the figure was 2.93 points per decade (Trahan et al., 2014).

The same paper states the practical consequence plainly: a test overestimates an individual's score by roughly 0.3 points per year between the year the norms were collected and the year the test was administered.

Put that into a hiring decision. A norm table collected eleven years ago carries a drift of about 2.5 points. On a standard score scale with a standard deviation of 15, one standard deviation is worth about 34 percentile points near the middle of the distribution, so 2.5 points lands at roughly six percentile points. A candidate who is truly at the 74th percentile reads as the 80th. Multiply that by ninety shortlisted names and the band above your cut line is partly made of drift.

The Standards for Educational and Psychological Testing are direct about whose job this is. Standard 5.11 says that while a test remains in print, the publisher is responsible for renorming it with enough frequency to keep score interpretations accurate (AERA, APA, NCME, 2014). If your vendor has not renormed in a decade, that is a fact about the product you are buying, not a detail.

One caution on the manual: the publication year is not the norming year. Manuals routinely report a norming window that closed years before print. Ask for the window.

The comparison group has to look like your candidates

Date is the easy failure. Population is the expensive one, and India gives us a clean natural experiment.

A 2026 study tested 1,752 children in Delhi, Hyderabad and Patna on Raven's Coloured Progressive Matrices, then scored them against the existing CPM norms for India. Under those norms, 78.5 percent of the sample landed in the bottom tenth of the distribution, with an estimated IQ of 80 or lower (Vogelzang, Wigdorowitz and Tsimpli, 2026).

Read that number twice. A distribution in which four children out of five sit in the bottom decile is not a description of the children. It is a description of the norm sample. The authors went and looked: the existing norm group was about one fifth the size of their sample, had an underspecified linguistic background, and drew mainly from higher socioeconomic families in private schools, while roughly two thirds of Indian pupils attend government schools. The norms were valid arithmetic applied to the wrong population.

Standard 5.8 puts the obligation in one sentence: norms should refer to clearly described populations, and those populations should include the groups with whom test users will ordinarily wish to compare their own examinees. When the description does not match your pool, the percentile is not a weaker number. It is the wrong number.

For a hiring pool, the variables that move scores most are years of English-medium instruction, school board, age band and region. If the norm table is keyed on "graduate, 25 to 34, United States", a 23 year old diploma holder from a Hindi-medium school is being ranked against people who share almost none of that.

How wide is the number you are quoting

Even a well-matched, recent norm table gives you an estimate, not a measurement. The percentile is computed from a finite sample, so it carries sampling error the same way a mean does.

Publisher confidence intervals usually do not cover this. They reflect the unreliability of the test itself and ignore the uncertainty created by sampling people into the norming study, which is the point of a 2019 paper that proposes a method for including it (Voncken, Albers and Timmerman, 2019). Standard 5.9 is the related requirement: technical documentation should indicate the precision of the norms themselves.

You can get a usable approximation with the binomial standard error for a proportion, where n is the number of people in the norm cell you are reading:

SE(percentile) = 100 × sqrt(p × (1 - p) / n)

At a percentile near the middle and a 95 percent band, that gives:

Those are arithmetic, not findings, and they assume the cell is a simple random sample, which norm cells rarely are. Treat them as a floor on the uncertainty.

The rule that follows is simple. Two candidates whose percentile bands overlap have not been shown to differ. A shortlist built by ranking the top forty from ninety on a table with plus or minus seven point bands has sorted a lot of noise into an order. The differences you can defend start around one full band width.

The checks, in the order the work happens

This is the sequence I now run before any assessment score reaches a hiring manager.

1. Date the norms. Get the norming window from the manual, not the print date. Record the number of years since it closed.
2. Find the cell size. Norm tables are keyed by age, education or both. The n for your candidate's cell, not the total norm group, sets the band.
3. Read the population definition. Compare it to your applicant pool on education, language of instruction and age. If two of the three do not match, stop quoting the percentile.
4. Compute the band. Apply the formula above to the cell size. Write the band next to the number, not in an appendix.
5. Choose the comparison and say which one. Published norm, local norm from your own pool, or a raw score with a validated cut. Each one is defensible if the report names it.

Steps one to four take an afternoon for a test you already use. Step five is a policy decision and needs a name attached to it.

A worked example, with illustrative numbers

Take a 60 item supervisory reasoning test. Your applicant pool over the last year is 1,240 people. The vendor manual reports a norm group from 2015, with 90 people in the relevant age band, a mean raw score of 33.2 and a standard deviation of 8.4. Your own pool, scored on the same items, averages 37.9 with a standard deviation of 7.6.

A candidate scores 41.

On the vendor table that is a z of 0.93, or the 82nd percentile. Against your own applicants it is a z of 0.41, or the 66th. Same paper, same pen, a 16 point gap created entirely by the comparison group.

The band at n equals 90 is about plus or minus 10 points, so the vendor percentile could be anywhere from 72 to 92. The norm date adds roughly six percentile points of flattery in the other direction. Honest summary of that candidate: somewhere in the sixties or seventies, on a norm group that does not resemble him.

The shortlist rule that survives all of this is not a percentile at all. Set the first cut on raw score, say 39 and above, because that is the number the panel can discuss and the number you can validate against performance later, and report percentile beside it with the norm named and the band shown. How to validate that cut against actual job performance is a separate problem, and the honest way to set a floor is covered in what a cognitive cut-off can and cannot do.

A local norm you can build from your own pipeline

Once you have 200 or more completed cases in a cell, your own applicant pool is a better comparison than a foreign table, and Standards 5.8 and 12.5 both allow it. The 12.5 comment is worth reading in full: local norms are useful alongside published norms, especially when the local population differs markedly from the published one, and in some cases may be used alone.

What it takes in practice: keep raw scores, not just percentiles, because percentiles cannot be re-binned later; report "percentile within our 2026 applicant pool, n equals 1,240"; refresh annually; and stop reporting the percentile in cells under 200, where the band is wider than any difference you are trying to detect.

The cost is bookkeeping. The gain is that the comparison group is made of people who actually applied for the job, in your language, from your colleges, at your salary band.

The honest limit

Drift correction from a meta-analysis is an average across mostly North American and European samples, and it assumes a linear trend that recent studies have questioned. Do not apply 2.31 points per decade as a precise correction to an Indian applicant pool. Use the norm date as a red flag that tells you to stop trusting the tail of the distribution, and use local data when you need precision.

Local norms have their own distortion. They rank people inside the population you already recruit, so if your sourcing is narrow, your norm is narrow, and a candidate who would perform well but comes from outside your usual colleges will look weak. A local norm is a better ruler, not a neutral one.

And none of this rescues a test that does not predict anything. Norms govern how a score is positioned. They say nothing about whether the score relates to the work. If you have not checked that, start with why a selection ratio is not evidence of adverse impact and work backwards to the score itself.

What to do on Monday

Pull the manual for the assessment your organisation uses most. Find three facts: the norming window, the n in the cell you actually read, and the population the norms describe. Write them at the top of the score report template, above the percentile column, so nobody can quote a percentile without seeing where it came from.

Then take last quarter's shortlist and recompute those candidates against your own applicant pool. Count how many names move across the cut. That number is the size of the problem, and it is the only argument you need to change the cut, the test or the vendor. The score report your managers actually see is built in the assessment platform, and a report that shows the norm beside the number prevents most of this before it starts.