Adverse impact in cognitive testing: the ratio is not the finding

A selection test can pass the eighty percent rule and still be the reason your hiring process fails an audit. It can fail the eighty percent rule and be nothing at all, just twelve people landing on the wrong side of a small sample. The rule cannot tell you which situation you are in, and a report that shows one ratio has not told you either.

Most assessment reports show one ratio. That is the first thing to fix.

The rule, in one paragraph

The Uniform Guidelines on Employee Selection Procedures, the 1978 US regulation that most adverse impact practice is built on, say this: a selection rate for any race, sex or ethnic group that is less than four-fifths of the rate for the group with the highest rate will generally be regarded by federal enforcement agencies as evidence of adverse impact. A rate above four-fifths generally will not (29 CFR 1607.4(D)).

The same paragraph then takes most of that back. Smaller differences may still be adverse impact where they are significant in statistical and practical terms. Greater differences may not be, where they rest on small numbers and are not statistically significant. The regulation itself calls the ratio a first inference, not a conclusion, and the EEOC repeated it in its 2023 guidance on algorithmic hiring tools, calling the four-fifths rule "merely a rule of thumb" that may be "inappropriate under certain circumstances".

You are probably not subject to this regulation. If you hire in India, no Indian statute gives you a four-fifths threshold. The rule still reaches you, because your vendor's validity report is written to it and a client audit will ask for it in that form. Treat it as the shared vocabulary, not as your legal standard.

The bottom line is not a defence

Here is the trap that produces the single-ratio report.

The regulation is generous about where you look. Section 1607.4(C) says that if the total selection process shows no adverse impact, agencies will in usual circumstances not expect you to evaluate the components. Read that and a clean bottom line looks like permission to stop measuring stages.

Because the regulation governs enforcement discretion, and a court governs an individual's claim. Those are different questions.

In Connecticut v. Teal, 457 U.S. 440 (1982), a written supervisory examination went to 48 black and 259 white candidates. Fifty-four percent of the black candidates passed, about 68 percent of the white pass rate, and the plaintiffs failed it. By the time the case reached the Supreme Court the employer could point to a bottom line facing the other way: 22.9 percent of black candidates promoted against 13.5 percent of white candidates. Connecticut called that a complete defence.

The Court said no. The bottom line does not stop a candidate establishing a case and does not give the employer a defence. Title VII guarantees the individual "the opportunity to compete equally", and a pass-fail barrier has to be shown to measure skills related to the job, whatever the final tally says (full opinion).

The practical translation is short. A clean bottom line is not evidence that your stages are clean. It is evidence that one stage's damage was offset by another stage's generosity, and offsets are not something you planned.

Step one: draw the funnel before you touch the data

Write down every point where a person or a system can say no. Not every point you think of as an assessment. Every rejection.

For a typical operator hire that is: application screen, online aptitude test, recorded interview, group exercise, final panel. Five stages, and most teams report one number for the whole thing.

Two rules. Include the stages run by the vendor: the EEOC's 2023 guidance is blunt that an employer can be responsible for a tool even when a vendor designed and administered it, so a "job fit" score needs its own row. And keep the order, because if stage two and three swap places next quarter, last year's pooled numbers no longer describe this year's process.

Step two: build the 2x2 at every stage

For each stage and each group you are monitoring, you need four numbers: how many entered, how many advanced, the rate, and the ratio against the highest group.

Three things there are why a single bottom line hides everything. The written test sits at exactly 0.80, and the regulation flags rates "less than four-fifths", so exactly four-fifths does not trip it. Half a person would have. The group exercise flags at 0.71 on 52 people. The final panel has 12 people in group B, so it cannot carry a verdict.

Step three: the second number

Add one test to the ratio. A two-proportion z test, which anyone with a spreadsheet can run, answers the question the ratio cannot: is this gap larger than chance would produce from pools this size.

Take the two rates, work out the pooled rate, compute the standard error of the difference, divide. The conventional threshold is 2, roughly the 95 percent mark. Above 2, the gap is real and needs an explanation. Below 2, you do not yet have a finding.

Watch how much the ratio moves with two people. In the group exercise above, group B advanced 12 of 52. Move that to 14 and the ratio goes from 0.71 to 0.83, and the flag disappears. Nothing about your process changed. Two candidates changed. That fragility is not an argument for ignoring the rule, only for never reporting it alone.

The four readings, and what each one means

Every stage lands in one of four boxes. Most reports only ever compute the first column.

The top right box is where most panic lives. A flagged stage on 40 people is a question, not a finding, and rewriting a test over it destroys comparability.

The bottom left box is the one nobody checks, and the more expensive mistake. A stage can sit at 0.82, look compliant, and still show a gap a real test calls significant, which the regulation anticipated when it said smaller differences may constitute adverse impact where they are significant in statistical and practical terms.

A worked example, with illustrative numbers

The figures below are invented to show the arithmetic. The structure is the point.

A plant hires 300 operator trainees. Group A supplies 200 applicants, group B 100. Three stages, one cycle.

• Written test: A 130 of 200 (65.0%), B 52 of 100 (52.0%). Ratio 0.80, z = 2.17.
• Group exercise: A 42 of 130 (32.3%), B 12 of 52 (23.1%). Ratio 0.71, z = 1.23.
• Final panel: A 30 of 42 (71.4%), B 8 of 12 (66.7%). Ratio 0.93, z = 0.32.
• Bottom line: A 30 of 200 (15.0%), B 8 of 100 (8.0%). Ratio 0.53, z = 1.72.

The written test passes the rule at exactly 0.80 and fails the significance test at 2.17. A real gap sitting one decimal place inside the line, and a ratio-only report would have called it compliant.

The group exercise flags on the ratio and fails on significance. One cycle is not enough, so pool the stage across three cycles with the same instrument:

Three cycles of a flag that never cleared significance become one pooled result that does, which is what happens when a real effect is measured with a sample too small to see it. The final panel is noise. The bottom line flags at 0.53 and is not significant on its own: it is the product of one real stage and one that probably is not.

What goes in the file

The documentation clause is the part teams skip, and it decides how the conversation goes. Section 1607.4(A) asks each user to keep records disclosing the impact of its selection procedures by group, then adds that where a user has not kept that data, an agency may infer adverse impact from the failure if the group is under-represented against the relevant labour market.

A missing file is not neutral. It is evidence.

Five things belong in it:

1. The stage list, in order, with dates and who administered each stage.
2. The four counts per stage per group: entered, advanced, rate, ratio.
3. The method: which rule, which significance test, which threshold, and why that threshold.
4. The decision and the name of the person who made it. "No action, pooled pending cycle 3" is a decision. Write it down.
5. The validation evidence for the instrument, and the less discriminatory alternative you considered.

Item five is what survives scrutiny. A ratio gets you to a question. The answer is whether the test predicts performance in this job and whether something less exclusionary would do the same work. For a general cognitive ability screen that evidence is weaker than vendors imply, an argument with its own method.

The honest limits

The rule is US-derived and India has no equivalent statutory test, so what you buy here is defensibility and client compliance, not statutory safe harbour. Say that plainly in your file rather than letting an auditor imply more.

A stage with fewer than about 30 people in the smaller group cannot carry a verdict. Pool it, or report it as a question.

A clean ratio does not validate a test, and a dirty ratio does not invalidate one. Impact and validity are two questions asked in sequence, and teams that conflate them scrap a good instrument or defend a bad one.

Pooling fails too if you go too far back and average away a change you made last quarter. Pool the window in which the process stayed the same.

The rule is gameable in one way. Merge the stages into a single score and the ratio gets cleaner while the barrier gets less visible, because the stage doing the damage stops being a stage. The Teal opinion flagged the same move. It is the shortcut that looks worst in a file.

Before your next hiring window

Open the last adverse impact report you signed and count the stages it reports. If the answer is one, you have a bottom line, not an analysis, and you do not know whether the number that cleared you was a clean process or an offset.

Then take the stage that rejects the most people, rebuild its 2x2 over two cycles, add the z, and put both numbers in front of whoever owns the test. If they agree you are done in an hour. If they disagree, you have found the argument worth having before a candidate finds it for you.

BolderBrain keeps cognitive assessment data at item and candidate level, which is what makes a stage-level 2x2 possible without a data request to the vendor. The reporting model is on the platform features page, and design choices that change how much a ratio can tell you are covered in the comparison of adaptive and fixed tests.