Adaptive versus fixed difficulty tests: what actually changes

A plant HR head in Pune asked me to cut her supervisor aptitude test from 75 minutes to 40 without losing the ability to rank candidates. The vendor's answer was immediate: make it adaptive. Same precision, half the time.

That claim is roughly true, and it is also incomplete in a way that costs money later. Adaptive delivery does not make a test better. It moves where the items sit, and everything the candidate experiences follows from that move.

The difference is where the items land

Every test item has a difficulty level and a narrow band of ability where it separates people well. An item that almost everyone answers correctly tells you little about a strong candidate. An item that almost everyone fails tells you little about a weak one. Maximum information sits where the probability of a correct answer is around one half.

A fixed form cannot know who is sitting it. So it spreads its items across the whole ability range the test is meant to cover. If the form is 30 items and the range runs from -2.5 to +2.5 on the logit scale, then roughly 18 of those 30 items sit more than one logit away from any given candidate. Those items are still scored. They just carry very little information about that person.

An adaptive test does the opposite. It starts near the middle, looks at the response, updates the estimate, and picks the next item closest to the new estimate. The National Council of State Boards of Nursing describes the loop plainly in its own explainer: the ability estimate changes after every question, and the uncertainty around it shrinks. That is the entire mechanism. Nothing smarter is happening. The item pool is just being aimed.

The consequence is arithmetic, not marketing. Because each item lands near the candidate's level, each one carries close to maximum information, and you need fewer of them to reach a given standard error. That is why the National Council can run the NCLEX as a variable length exam with a minimum of 85 items and a maximum of 150: candidates whose ability sits far from the pass line get a decision early, and candidates near the line keep answering until the estimate is tight enough.

What the candidate actually notices

The psychometrics are invisible to the person at the screen. Four things are not.

Length varies, and candidates compare. On the NCLEX, finishing at 85 items can mean a pass or a fail, and the council says so explicitly because candidates read length as a verdict. If your programme has no pass line and you stop on a precision rule, one candidate answers 22 items and another answers 48. Someone will notice and someone will complain.

Review usually disappears. A full adaptive test cannot let you go back, because the item you would return to was chosen using the answer you now want to change. The MedEdPublish review of adaptive testing puts this as a design cost: adaptive delivery by difficulty requires unidirectional navigation. Section-adaptive designs buy the review back at the price of coarser targeting. The GRE is the clean example: two sections per measure, 12 questions in the first at average difficulty, 15 in the second chosen by how the first went, and you can move back and forth freely inside a section.

Difficulty feels personal. A fixed form gives everyone the same experience of the paper. An adaptive test hands a strong candidate a run of hard items and a weaker one a run of easier items. Both are being measured properly, and both may walk out convinced they did badly. This is the single most common complaint in pilot feedback, and no amount of psychometric accuracy fixes it.

The candidate cannot estimate their own score. With a fixed form, a candidate who answered 24 of 30 knows roughly where they stand. With adaptive delivery, the raw count means nothing without the difficulty of the items served. Expect more queries about results, and expect them to be harder to answer in one line.

The arithmetic before the platform decision

Three numbers decide whether adaptive is available to you at all, before anyone talks about algorithms.

The precision you need. In item response theory, the standard error of the ability estimate is one over the square root of the total information collected. Reliability of .90 corresponds to a standard error of about .316, and .80 corresponds to about .447. Decide the target first. A development feedback report can live with .45. A selection decision that gates a job should not.

The bank you have. Adaptive only works if there are calibrated items at every level you might need to measure. The published simulation study of a national Chinese medical exam started with 300 items and kept 121 after psychometric screening, and that retained set is what the adaptive engine ran on. The screening is not optional: badly fitting items distort the estimate exactly where the algorithm is aiming.

The calibration sample. Item parameters have to be estimated from real responses before the adaptive engine can use them. The MedEdPublish review notes that item response theory models need more than 200 candidates to be robust, which is why the successful large scale examples are national exams with thousands of candidates. The Sahin and Weiss simulation work is more forgiving for small programmes: a 200 item bank calibrated on as few as 150 examinees can be usable, provided the bank has enough information at the ability levels you actually target. That is a real option for an in-company test with a few hundred sitters a year, and it is not an option for a 60 item bank.

If you cannot meet the bank and calibration conditions, adaptive is not a cheaper version of your test. It is a different test that you cannot score properly.

A worked example: 21 items or 30 to 41

These numbers are illustrative, computed here rather than taken from a published study. I built a 90 item bank on a simple Rasch model with difficulties spread from -2.5 to +2.5, then compared a 30 item fixed form spanning that range against an adaptive form drawing from the same bank, targeting the same standard error of .45.

• Fixed form at the middle of the range: 30 items to reach .45.
• Fixed form at either edge of the range: 41 items to reach .45.
• Adaptive form at every level: 21 items.

Read the second line again. The fixed form needs a third more items for a candidate at the edge of the range than for one at the middle, and it is the edge candidates who are hardest to classify. Precision is not just lower off centre, it is lowest exactly where you most want it.

Now hold length constant instead. Give everyone 20 items:

• Fixed form: standard error .551 in the middle, .64 at the edges.
• Adaptive form: .453 at every level.

A fixed 20 item form is about 45 percent less precise than an adaptive form of the same length. If your current test is 20 items and your decisions are borderline, that gap is the reason your borderline cases keep going to a second interview.

When a fixed form is the honest choice

Adaptive is not the default good. It is a purchase with three payments: a calibrated bank, an item screening cycle, and the loss of review.

Take the fixed form when the cohort is small. Below roughly 200 sitters a year you are not maintaining a bank, you are maintaining a form, and every item you retire hurts. Take it when the bank cannot cover the tails, because an adaptive test that runs out of hard items gives its strongest candidates a run of medium ones and quietly compresses the top of your distribution. Take it when the decision is appealable. If a candidate can ask why they were rejected, a process where they answered different items from the person beside them is harder to defend, however sound the statistics are. The MedEdPublish review is candid that adaptive delivery trades transparency for precision, and that trade is not always worth taking.

Take the section-adaptive middle when you need both. It keeps review, keeps a predictable structure that candidates can prepare for, and captures most of the precision gain at a fraction of the bank requirement.

The limits nobody puts in the proposal

Adaptive delivery does not fix a bad item. It aims the item better. A bank full of ambiguous questions, double barrelled stems and culturally loaded examples produces a confident estimate of the wrong thing, and it does so faster than a fixed form would.

It also lengthens tests once you add content constraints. Every rule about blueprint coverage, no repeated passage, and no more than two items on one topic removes the most informative item from consideration, and the algorithm compensates by asking more questions. The efficiency gain survives, but it shrinks.

And the fairness objection does not go away because the psychometrics are correct. Candidates who receive different items than their peers will suspect the process, particularly in a country where the same paper for everyone has been the norm. The mitigation is communication and a documented blueprint, not a better algorithm.

What to do on Monday

Pull last cycle's item level data and answer one question: what share of the items each candidate answered sat more than one logit from their final score? If that number is above 40 percent, you are paying for items that tell you almost nothing, and adaptive or section-adaptive delivery is worth costing. If it is below 20 percent, your form is already tight and the gain will not justify the bank.

Then count your calibrated items by difficulty band. If any band you need for a decision has fewer than three items, stop. No delivery design will save that. Write items first, and revisit the platform question next cycle.

For how a score gets used once it exists, the cut-off arithmetic matters more than the delivery design, and the case against a single cut score covers that. If you are also deciding how many assessment platforms to keep alive, the tool count question is the same decision in a different costume. And if you want to see how the item bank and reporting side works in practice, the assessment features and the platform comparison are the places to look.