Test security is an item supply problem, not a proctoring problem

The hiring manager in Pune had 340 candidates and three days. By the second session, a WhatsApp group had forty-one screenshots of the numerical reasoning paper. Nothing in the platform was breached. No camera was covered, no extension was installed, no candidate logged in twice. One person had photographed the paper in the first session, and the paper had been the same for everyone.

The afternoon cohort scored above the 90th percentile at twice the expected rate. The interviewers noticed, said nothing formally, and quietly discounted every score from that week. That is the real cost: the numbers stop being usable, and nobody can tell you which numbers.

Most organisations respond by buying something. Remote proctoring, lockdown browser, ID verification, an AI flag engine. All of those operate at the session, which is the last and weakest place to intervene. The failure happened in the item pool, weeks earlier, and the fix belongs there.

A fixed form has a shelf life of one session

A test paper that every candidate sees is a secret shared with everyone who has taken it. The first ten people to sit it may be honest. The eleventh can photograph eleven items and the value of those items is gone permanently, along with any ability to know which of the next three hundred candidates had them.

Estimates of how many people do this vary widely: a review of the research puts the range at 7% to 50% of people cheating on web-based tests, and a 2006 vendor study found about 10% admitting it. Employers are more optimistic than the research. In a survey of organisations using computerised testing, roughly half estimated fraud on unproctored tests at below 10%, and only about one in ten expected more than 10% of applicants to cheat. The same survey found 40% of these organisations using unproctored delivery for all their testing, including high-stakes hiring, against 20% still using supervision.

The gap shows up in the scores. A three-level meta-analysis of 108 effect sizes from 49 studies, covering 100,434 people, found unproctored ability scores averaging 0.20 standard deviations higher than proctored ones, 95% CI [0.10, 0.31]. The same analysis found rank ordering across contexts correlated only .58, which means the person who ranks third unproctored is not reliably the person who ranks third under supervision. And in a review of the countermeasure literature, added controls did not close that gap. People cheated when the opportunity existed, whatever the stakes and whatever was installed.

That is not an argument for giving up. It is an argument that the controls being tested were the wrong ones.

Start with the pool: cap how often any item is used

In an adaptive test, item selection does not spread the load. It concentrates it. The most informative item at a given ability level is chosen every time a candidate lands near that level, so a handful of items do the work while the rest of the bank is decoration.

A 680-item bank from a Dutch adaptive test makes this concrete. With items selected at the candidate's ability estimate and no exposure control, the most-used item appeared in 68% of administrations, nineteen items in more than 40%, and only 226 of the 680 were ever used. With selection at the pass mark, twenty-two items appeared in 100% of tests and 90% of the bank went unused. Twenty-two items is not a pool. It is a leaflet with a hundred and twenty pages of padding.

The fix is old and cheap. Define a maximum exposure rate, the share of administrations in which an item may appear, and constrain selection so it cannot exceed that. Capping at 0.35 kept every item under the ceiling, raised items used from 226 to 245, and cost 0.3 of an item in length. Combining the cap with a method that pulls rarely used items forward brought maximum exposure to 0.28-0.32 and added one item on average.

One item. That is the price of not handing the same twenty-two questions to an entire hiring season.

Two practical rules follow. Compute the exposure rate per item from your own administration log, because vendors rarely report it and it drifts as your candidate population clusters. Then retire items at a threshold rather than at your convenience; 0.25 is a reasonable starting point for a pool in continuous use.

A third rule vendors will not mention: an item used for one client in a city is compromised for every other client in that city. Rotation across tenants matters more than most security features, because the leak that damages you usually happened on someone else's test.

Write items that are useless to look up

If cheating worked uniformly, security would be a single number. It does not. In a randomised experiment, 178 participants were told to cheat on a 68-item online ability test, with a week's notice and cash prizes for high scores. Their average score was 0.40 standard deviations above the 248-person control group, but the gain sat almost entirely in particular subtests.

Vocabulary, number series and calculations are all things a calculator, a dictionary or a search engine can help with. Figure sequences, verbal analogies and syllogisms require holding a rule in mind and applying it to material that does not exist anywhere except on the screen. Looking them up costs more time than solving them.

That is the design principle. Weight the score towards items where the answer is derivable in thirty to forty-five seconds from the stimulus alone, and where the stimulus is novel. Per-item time limits do more work than most people expect: two-thirds of the cheaters said cheating had been difficult, and 42% gained nothing measurable.

One more number: only 7% of the group told to cheat expected they would cheat in a real selection process. People do not plan to cheat; they take the opportunity when it appears.

Detect afterwards from response data, not from honesty

If you cannot prevent it, you can price it. Self-report measures are close to worthless here. In a study of 315 people who took a knowledge test unproctored and then again in a proctored lab, honesty and overclaiming scales did not predict who cheated. What did predict it was test data, specifically performance on extremely difficult items, and paradata: response times and tab switching.

Response times are the most underused signal in Indian hiring stacks. A candidate who has seen a subset of items answers those faster than their own baseline on comparable unseen items. The statistic that compares speed on compromised items with speed on clean ones has a known null distribution, so you can fix your false-positive rate before you look at a single name. That matters more than its power, because accusing an honest candidate is not recoverable.

In 2002 the Educational Testing Service found candidates in several countries using websites that showed live GRE items. Average GRE Verbal scores rose by 100 points out of 800 in one country and 50 in another. That is not a better cohort. That is a pool that had stopped measuring anything.

Tab switching is the other cheap signal, and it needs no camera. A script that records when the test page loses focus detected abandonment with very high sensitivity and specificity in a controlled study, and a popup warning reduced the behaviour. A five-line listener beats another model of facial geometry.

Three flags belong in the score report: response time on the hardest items against the candidate's own median, count of page abandonments, and similarity of the response vector to others in the same session window. None is proof. Together they give a reviewer something to open.

What proctoring is actually for

Remote proctoring does one thing nothing else does: it binds a face to an identity at the moment of the test. Proxy test taking, where a stronger colleague sits the test in the candidate's place, is the one failure mode that item rotation and response-time analysis cannot touch. If your funnel is large and the test is the main filter, identity binding is worth paying for.

Beyond that, be careful what you believe. A before-and-after study of 384 pharmacy students moving to remote proctoring found scores significantly lower for first-years and significantly higher for second-years, with no anxiety or GPA subgroup explaining it. Proctoring changes the conditions of the test. It does not reliably produce a cleaner score.

Flag engines deserve the same scepticism. A systematic review of online proctoring notes that because a flag is a lead for human decision-making, not a verdict, flagging has to be over-cautious, which necessarily produces a large share of false positives. A flag is a reason to open a recording, not a finding.

A worked example, with illustrative numbers

A company runs 1,200 candidates over three months on a 300-item pool, forty items per test. The numbers are illustrative, chosen to show the arithmetic.

Before: no exposure cap. The twelve most-used items appear in 55% to 70% of sessions. One branch manager photographs a full test in month one, and every candidate at that branch for the rest of the quarter has an advantage nobody can quantify. Two hot items are verbal, where look-up is easy.

After: a 0.25 cap, which raises average test length from 40 items to about 41.6. The hottest item drops to 24% exposure and the number of items actually used rises from roughly 140 to 188. Six verbal items are rewritten as figure sequences of the same difficulty, and the score report carries a tab-switch flag and a response-time ratio. In the first month, 9 of 380 candidates are flagged, 4 confirmed on review, 5 cleared.

Added cost: 1.6 items per candidate, six rewrites, about a day of engineering. Removed cost: an unusable quarter of hiring data.

The honest limit

None of this proves a test is clean. Every measure raises the cost of cheating and none eliminates it. A determined candidate with a second device and a cooperative friend beats a tab-switch flag, and a pool leaked from a vendor's demo environment defeats an exposure cap inside your tenant.

Detection also carries a moral debt vendors rarely put in the deck. A false positive in hiring is not a cancelled exam; it is a person who does not get a job and never learns why. Set thresholds for that asymmetry, require review of every flag, and record the outcome.

And the honest bill is item writing. A 300-item pool is months of work for a trained writer, and every retirement consumes some of it.

One action this week

Pull the administration log for your last 500 sessions and compute the exposure rate of every item served. Sort descending. Anything above 25% goes on a retirement list this quarter, and the top five go today. That one table will tell you more about your test's real security than any proctoring demo. If you are still choosing a platform, ask the vendor for it before you ask about their camera. How BolderBrain handles item banking and reporting is documented, and the piece on adaptive versus fixed forms explains why pool depth decides whether adaptivity helps or hurts, which is the same reason a cut-off score does less work than it looks.