Your pre and post test measures the test, not the training

A manufacturing L&D manager sends a Level 2 report to her plant head. Twenty questions on the new pump commissioning procedure, given before the workshop and again four weeks after. Mean score 58 percent, then 71 percent. A 13 point gain, printed in bold.

The plant head looks at it for a while and asks the question nobody in the room had prepared for. If the forty eight engineers had spent that day on the line instead, and we had tested them twice anyway, how much of that 13 points would we still have?

Nobody knows, because the same twenty questions were used both times, and no one was tested twice without training in between. The number is real. The attribution is not.

The gain is real, and part of it is item memory

Two mechanisms inflate a same-items pre and post design, and both are documented well enough to size.

The first is retest effect. Sitting the same test twice raises the score because the items are now familiar, not because the ability changed. Scharfen, Peters and Holling pooled 122 studies, 174 samples and 153,185 people, 786 test outcomes in all, and found a standardised mean change of 0.34 standard deviations from the first administration to the second. When the second administration used a parallel form rather than the identical items, the gain was significantly smaller, by 0.15 standard deviations (p < 0.001). They also found no further score gains after the third administration, which tells you most of the effect is front-loaded familiarity.

On a test with a spread of 12 points, 0.34 standard deviations is about four points of pure sitting-there. That figure comes from cognitive ability tests, where the material is abstract and hard to memorise. A twenty item quiz about your own SOPs is easier to remember, so treat four points as a floor, not a ceiling.

The second mechanism is less intuitive. The pre-test is itself an intervention. Attempting questions you cannot yet answer improves later retention of the answers, even when every initial guess is wrong. In a 2025 experiment in the Journal of Cognition, the group that attempted a pretest before studying recalled 60.86 percent on the final test, against 48.24 percent for a group that only read the same material. That is a 12.6 point advantage produced by the pretest alone, with no training at all.

Put the two together and the honest reading of a same-items gain is uncomfortable: a meaningful share of it was manufactured by your own instrument. Which is why the design decision, not the statistics, is what decides whether you have evidence.

Two forms, one blueprint

The fix is a parallel form. Not a reworded form, a parallel one.

The difference matters. Rewording turns "which of these is NOT a commissioning step" into "which is not a commissioning step" and calls it form B. That is the same item with different punctuation, and it retains every bit of item memory. A parallel form tests the same objective with different content, at matched difficulty.

The work happens in this order.

Step one: write the item blueprint before writing a single item. One row per item: the objective it tests, the operation it requires (recall, apply, diagnose), and the difficulty band you intend. Twenty items means twenty rows. This document is what makes two forms comparable, and it is also the thing that stops your subject matter expert writing eleven questions about the part of the procedure he personally finds interesting.

Step two: build both forms in the same sitting. Forty items, split A and B by alternating down the blueprint so each form carries the same mix of operations and difficulty. If you write form B a month later, after seeing how form A performed, you will unconsciously make it easier. That is not sabotage, it is how humans write tests.

Step three: counterbalance the order. Half the cohort sits A then B. The other half sits B then A. This costs one extra line on the attendance sheet and it removes form difficulty from the comparison entirely, because a slightly harder form B is penalised when it serves as the pre-test and credited when it serves as the post-test. Without counterbalancing, a five point difference in form difficulty becomes a five point fake training effect.

Step four: fix the pass bar before you see any scores. Decide what counts as a meaningful change and write it down. Two items on a twenty item test. One full band on a rubric. Do it in advance, because after the data arrives every team finds a defensible reason why three points is actually a lot.

Also check the ceiling before you commit. If the pre-test mean is already above 80 percent, the instrument cannot show a gain that means anything, and no amount of analysis will rescue it. That is a design failure discovered at the right time, which is before the workshop runs.

Step five: score the person, not only the average. A group mean hides the distribution, and the distribution is the finding. More on this below.

Score the person, not just the mean

The reliable change index answers a narrow, useful question: is this one person's change larger than measurement error would produce on its own?

The arithmetic is short. Take the standard error of measurement, SEM = SD × √(1 − r), where r is the test-retest reliability of your instrument. Because both scores carry error, the standard error of the difference is Sdiff = √2 × SEM. The index is the change divided by Sdiff, and anything above 1.96 in absolute value is reliable change at the 5 percent level. The method comes from Jacobson and Truax, and the refinements that matter in practice are to use a test-retest reliability measured over a comparable interval, and to subtract your known practice effect from the observed change before you divide.

Two warnings on r. Use test-retest reliability, not internal consistency. Cronbach's alpha is routinely higher than a four week retest correlation on the same instrument, and using it lowers the bar, which is the wrong direction for a claim you have to defend. If you have no retest estimate at all, substitute a conservative value such as 0.70, say in the report that you did, and accept the higher bar.

The point most pre and post reports miss: a group mean can move reliably while almost no individual in it did. Averaging shrinks error by the square root of the sample size. An individual score does not get that help. So report both. The mean is the programme's result. The count of people who cleared the bar is the manager's result.

A worked example, with illustrative numbers

Numbers below are illustrative, constructed to show the arithmetic rather than to report a real programme.

Forty eight field engineers, a twenty item test on a new commissioning procedure, forms A and B counterbalanced across two batches, four weeks between sittings, test-retest reliability of 0.80 and a pre-test standard deviation of 12 points.

The headline is the one from the opening: mean 58, then 71, a 13 point gain.

Now the individual bar. SEM = 12 × √(1 − 0.80) = 5.37. Sdiff = √2 × 5.37 = 7.59. Reliable change at the 5 percent level = 1.96 × 7.59 = 14.9 points. A single engineer needs to gain about fifteen points before you can say the change is more than noise on this instrument.

If individual gains scattered around 13 with a spread of roughly 8 points, about four in ten would clear 14.9. That is 19 engineers who moved reliably, 9 who moved by less than 5 points, and 4 who went backwards. That distribution is the report a plant head can act on, because it names the people who need a second session and the four whose scores fell.

The mean, meanwhile, is still trustworthy at group level. The standard error of the mean gain is 7.59 / √48 = 1.1 points, so a 13 point gain sits about twelve standard errors from zero. Averaging does for the group what it cannot do for the individual. Both statements are true at once, and a report that gives only one of them is misleading.

Then subtract your own practice effect. If a no-training comparison group of twelve engineers, tested twice on the same schedule, moved 4 points, the programme's honest contribution is 9 points, not 13. That single subtraction is what turns a score into a claim.

The honest limits

Parallel forms cost real work. Forty item specifications and a day of a subject matter expert is not free, and for a half-day compliance briefing it may be more rigour than the decision deserves. If you cannot build a second form, use the same items and report the gain with the caveat attached. Do not report it as impact.

Alternate forms do not eliminate retest effect, they shrink it. The published estimate is roughly 0.22 standard deviations of procedural gain even with a fresh form. The only way to measure your own residual is a small untrained group tested on the same schedule. Twelve people is enough to get a usable number.

The reliable change index also has a ceiling of ambition. It tells you a change is bigger than noise. It says nothing about whether the change was worth the money, and it says nothing about whether anyone does the procedure differently on the shopfloor on Monday. That last question needs an observation schedule, not a test, which is what measuring behaviour change without a research team is for.

And the whole method assumes the test is worth taking. A knowledge test with poor items will produce a precise measurement of the wrong thing. If you have never run an item analysis, you are one step behind this article, not ahead of it.

What to do this week

Take the test you already used, and write twenty new items from a blueprint. Not reworded, new. Then find twelve people who will not be trained this quarter, test them twice, three weeks apart, with no training in between.

Whatever that second group moves by is your practice effect. Subtract it from every gain you report from now on, and put the number in the slide deck next to the gain rather than in a footnote. A gain of 13 with a practice effect of 4 is a programme that worked. A gain of 13 with a practice effect of 11 is a test that got familiar, and it is better to find that out from your own data than from your plant head. If you need a place to keep the results so the next programme inherits them, the assessment workspace is built for that, and if you are still assembling the measurement order itself, start with why most training impact stays unprovable.