A training control group that fits inside one plant
You cannot hold half a plant back from training. Stagger the go-live by line or site instead, then read the waves against each other as a designed comparison.
Rajendra Kumar · 21 November 2026 · 9 min read

Representational image.
The programme ran in two waves because one trainer could not cover eight lines at once. Lines 2 and 3 went in March, the other six in June. By September rejection was down everywhere and the L&D manager had a slide saying the programme cut rejection by 0.8 points.
The plant head asked the obvious question. What would that number have been if we had done nothing? Nobody knew. The comparison had been sitting inside the rollout plan the whole time, unused.
Every programme delivered in waves already contains a comparison group: the cohort that has not gone yet. What it lacks is an order not chosen by the outcome, a comparison held still, and an analysis that reads the waves against each other instead of reporting each one alone.
The second wave is a control only if you keep it one
A control group does not need a research team and a refusal. It needs some units trained later than others, which is true of almost every real programme. The problem is what happens to the later wave between the two dates.
Three things inflate a before-and-after reading, and all three are worse when the order was set by convenience.
Time. Rejection rates, attrition, reopen rates and error counts move on their own. Process fixes land, mix changes, the worst performers leave, the busy season ends. The March wave gets five extra months of whatever else was happening.
Attention. A unit just trained gets visited. The trainer comes back, the HR business partner drops in, the supervisor raises it at the shift meeting. Some of the gain is the programme, some is somebody now watching.
Selection. Units are usually ordered by need, and the neediest go first. Lines reading 6 percent will fall further than lines at 3 percent with no training at all, because extreme readings are partly noise. Ordering by need and claiming the drop measures regression to the mean.
The fourth problem is the analysis itself. Report each wave's own before and after and you have two uncontrolled claims, not one comparison. Nothing in the deck shows the counterfactual.
Fix the unit, the outcome and the window before the calendar is signed
Do this in the order the work happens. Each decision constrains the next.
Choose the unit of assignment. Training goes to people, but the outcome is usually counted on a group: a line, a shift, a branch, a depot. The unit is whatever the metric is counted on. If the metric is monthly rejection per line, the unit is the line, and everyone on it goes in the same wave. Splitting a line across waves is the most common way this design fails.
Pick one primary outcome and define it before anything runs. Not three. One number you will defend, from a system that already collects it: quality record, rework log, dispatch error log, incident register. If the metric only starts existing because someone will now count it, you have no baseline. Write the definition and its exclusions down, because "rejection rate" means four things to four departments.
Set two windows of equal length. A baseline window and a follow-up window, neither sitting on a known seasonal peak, and a read date fixed in advance. Unequal windows are the quiet error: two months against five will show improvement in almost anything that is drifting.
Pair the units, then toss a coin
Pairing is what turns eight units into four usable comparisons. Two rules.
Pair on trend, not on level. Lines reading 6.0 and 3.0 can be a good pair if their last three periods moved in parallel, because the difference-in-differences removes the level difference by construction. Two lines both at 4.5 heading in opposite directions are a bad pair, whatever today's chart looks like. Write the three prior readings for each unit into the plan first.
Randomise the order inside each pair. One coin flip per pair decides which member goes in wave 1. That is the whole randomisation, and it is what separates a designed comparison from a story about two groups of lines. It also protects the L&D manager from the accusation that the easy units went first, which is the first thing a sceptical plant head will say.
Record the draw in one line: date, pair, result, witness. A design that cannot be shown to have been decided before the results is not much better than the anecdote it replaced.
Then hold the comparison still for the length of the window. This is the part teams skip.
| Threat | What it looks like | What you do about it |
|---|---|---|
| Contamination | One trainer covers both waves, or a wave 1 supervisor is transferred onto a wave 2 line | Different trainer per wave where possible, log every transfer, hold cascade training until after the read date |
| Attention | Walk-rounds and coaching concentrate on the units just trained | Same cadence to both waves with the training content removed |
| Concurrent change | New SOP, machine overhaul, incentive revision, shift pattern change | One-page quiet-window memo signed by the plant head; if the change is unavoidable, move the read date, never adjust the number afterwards |
| Partial delivery | Half a line trained because of shift cover | Move the whole unit to the next wave |
In most plants a training result is unprovable not because the design was wrong but because two other things changed in the same quarter and nobody wrote that down.
Tell the comparison units what is happening
A wait-list design is the version of this that survives an ethics conversation. Everyone is trained, some later, and the later group is given a date and a reason for the draw. The American Evaluation Association's practitioner guidance describes exactly this structure: measure both groups at the end of wave 1, train the wait-list group afterwards, compare across the same window. Nobody is denied anything and the comparison is real.
Do not let the wait-list group become a rumour. If wave 2 hears that wave 1 got something and they did not, the measured change includes the rumour.
The arithmetic, and the number to write down
The estimator is one line. Take the change in the trained units, subtract the change in the comparison units.
Difference-in-differences = (trained after minus trained before) minus (comparison after minus comparison before)
Here is a worked example. Every number below is illustrative, built to show the mechanics rather than to report a real programme.
Eight lines at one plant, four pairs, two waves. Rejection per thousand units, monthly average over equal windows.
| Pair | Wave 1 line | Before | After | Wave 2 line | Before | After |
|---|---|---|---|---|---|---|
| A | Line 1 | 4.8 | 3.9 | Line 2 | 4.6 | 4.3 |
| B | Line 3 | 6.2 | 5.1 | Line 4 | 6.0 | 5.5 |
| C | Line 5 | 3.4 | 2.8 | Line 6 | 3.5 | 3.3 |
| D | Line 7 | 5.1 | 4.4 | Line 8 | 5.0 | 4.8 |
The naive claim is the average fall in the trained lines: 0.83 points. The comparison lines fell 0.30 points over the same period without the programme. The programme's share is the difference, 0.53 points.
Now the interval, because a point estimate will not survive a finance review. Compute the difference-in-differences per pair: 0.60, 0.60, 0.40, 0.50. Those four pair-level values are the observations. Their standard deviation is 0.10, so the standard error of the mean is 0.10 divided by the square root of 4, or 0.05. Four pairs give three degrees of freedom, so the 95 percent multiplier is 2.78, not 1.96. The interval runs from 0.39 to 0.66 points.
That is the sentence to write: rejection fell 0.53 points more on the trained lines than on the comparison lines, 95 percent CI 0.39 to 0.66. And the sentence that makes it credible: roughly a third of the naive improvement was already happening on the lines we had not trained.
One technical point that changes the answer more than any other. The observations in that calculation are the pairs, not the individual rejections or the people. Compute the standard error across thousands of production units and you get an interval a few hundredths wide, claiming a precision the design does not have. Four pairs is four observations. That is the honest sample size.
Do the power sum before the programme, not after. With four pairs and this much noise, anything smaller than about 0.2 points will not separate from zero. Put that number in the design note.
If both waves sit in one system with the cycle, the unit and the date on the same record, the analysis is an afternoon in a spreadsheet. BolderBrain's assessment and reporting layer keeps the cycle and the score together per person, which is what this arithmetic needs. If the wave label lives in a notebook, the comparison is gone.
What this design cannot tell you
The comparison rests on an assumption you cannot test. Difference-in-differences requires that the two groups would have moved in parallel without the programme. Columbia's methods reference is blunt: it is the most critical requirement, the hardest to satisfy, and there is no statistical test for it. Three parallel pre-period readings make it plausible, not proven. If the comparison lines were on a different trajectory, the estimate is biased and no arithmetic rescues it.
Staggered rollouts carry a specific version of that risk. Hemming and colleagues, writing on stepped wedge designs in the BMJ, note that because more units are exposed towards the end of the study, the effect can be confounded with any underlying temporal trend, and a result that looks like an effect may be the trend itself. Both the sample size and the analysis have to allow for clustering and for time. In a plant, that means a pair-level analysis and a pre-period trend check, not a pooled average across everyone trained.
Wait-list comparisons also tend to overstate. A reanalysis in Frontiers in Psychiatry found one intervention showing an effect roughly six times larger against waiting-list controls than against the other control conditions. Waiting is not a neutral state: expectations, deterioration and the sense of being on hold all move the outcome. Treat your wave comparison as an upper bound, and say so when you present it.
With four to eight units you also cannot detect small effects, and in a plant where every unit must go live on the same date for a compliance or audit reason, there is no comparison at all. Do not manufacture one out of a different site with a different product mix. In that case the honest output is a prediction registered before the programme and read afterwards, which is the method that survives when no control group exists.
The instrument still has to work. A comparison run on a weak pre and post instrument measures noise twice over, which is why the choice of forms, timing and rater matters as much as the wave design. And if the metric moved for a reason you have not ruled out, the four rival explanations still have to be checked before you attribute anything.
What to do with the next rollout plan
Open the plan for your next programme and add one column: go-live date per unit.
If every row has the same date, you have no comparison group. Say so in the business case rather than discovering it in the review, and register a prediction instead.
If the column has two or more dates, you have an experiment that is currently unanalysed. Pair units on their last three readings, toss a coin for the order inside each pair, write the draw down with a witness, sign the quiet-window memo, and put the read date in the calendar before the first session runs. That is the entire design. It costs a coin, a column and one page, and it is the difference between a number you can defend and a number you have to hope nobody questions.
Next step
Run the same measurement on your own team
Bring a real team and a real competency. We set up the assessment, run one cycle, and walk you through the report so you can judge the output yourself.
Written by
Rajendra Kumar
Head of Enterprise Business, Conceptial
Runs enterprise work at Conceptial, the training and consulting firm behind BolderBrain. Two decades of building assessment and development programmes for Indian enterprises.