When the metric moves and training did not cause it

The monthly review is at the plant head's office, and the HR manager has one slide. Attrition among first-line supervisors fell from 18% to 14% in the six months after the supervisor development programme. She has the attendance sheets, the feedback scores, the certificate count. The plant head looks at the number and asks the only question that matters: compared to what?

The number is real. The attribution is not yet earned, and everyone in the room knows it, including her. What follows is the order of checks that turns a real number into a claim you can defend, or tells you honestly that you cannot.

A metric that moves has more than one author

Business metrics are open systems. Between the two readings, hiring slowed, a competitor opened a plant two districts away, the appraisal cycle ended, one site changed how it codes separations, and the plant head started a daily standup. All of that is inside the 4-point gap.

A before-and-after comparison quietly assigns all of it to your programme. The World Bank's impact evaluation guidance is blunt about why that fails: you cannot draw causal conclusions from before-and-after changes alone, because other factors influence the outcome over time, and you cannot simply compare trained and untrained groups either, because they differ in ways you did not measure.

So the work is not to prove the programme worked. The work is to eliminate the rival explanations one at a time, in an order that gets cheaper as you go.

Check the definition before you check the effect

Start with the least glamorous rival: the metric itself may have changed. Definitions drift without anyone deciding to change them. Attrition is computed against average headcount in one report and closing headcount in another. A category gets reclassified. An acquired site starts reporting in month eight and lands in the year-end number. A field office that used to send a spreadsheet feeds the HRIS directly, and its previously invisible exits appear.

It is cheap to rule out. Pull the definition document as it stood at both dates, plus the reporting population. Recompute the earlier period under the current definition, or the reverse. If the two series cannot be put on one definition, you do not have a trend, you have two unrelated numbers. Rebuilding it is half a day with the HRIS team, and that half day decides whether the rest of the analysis is worth doing.

A quick test: ask who submits the data and whether that list changed. A metric whose reporting population grew by 15% will move for reasons that have nothing to do with behaviour.

Take the calendar out before anything else

Seasonality is the rival most often mistaken for an effect, because it is large and it repeats. The Office for National Statistics describes seasonal adjustment as removing systematic calendar-related variation so that consecutive periods can be compared at all. Their example is retail sales rising every December. Nobody calls that a successful Christmas campaign.

Indian workplaces have their own calendar. Supervisors resign after increments land in April and after appraisal outcomes in May and June. Festive demand pulls temporary hiring up in September and October and releases it in November. February and March carry budget flushes and transfer decisions. Attendance in July and August depends on the monsoon in ways a training plan cannot fix.

Compare like with like: same quarter last year, or the same three months across three years. If the programme ran March to August, the comparison is March to August of the previous year, not January to June. With no history, take the calendar out through the comparison group instead: the untrained branches had the same April increment.

A 2-point fall in a March-to-June window is not evidence of anything until you have seen what March-to-June usually does.

Decompose the number before you explain it

The third rival is composition. An aggregate can move in one direction while every part of it moves in the other, and this is not a statistical curiosity. It is common enough to have a name and a literature.

Armstrong and Wattenberg give a clean real example: United States median weekly wages rose 0.9% between 2000 and 2013, while median wages fell in every education segment. Workers with no degree lost 7.9%, those with some college 7.6%, high school only 4.7%, and bachelor's degree holders 1.2%. The overall figure rose because employment shifted toward the higher-educated, higher-earning group, which grew 33% while the no-degree group shrank 21%.

Your version: average training hours per employee rise because a large plant with low hours per head closed. Average handling time falls because the mix shifted to simpler cases. Engagement rises because the response rate dropped among the most disengaged teams.

The check is mechanical. Split the metric by the three or four dimensions that changed at the same time: site, grade, tenure band, shift, product line. Then see whether the overall movement survives inside the cells. If the aggregate moved one way and every cell moved the other, the story is composition, and training has no claim on it.

Manager attention is a real effect, and it is not your training

The fourth rival is the one practitioners argue about most. When a programme starts, someone starts paying attention. The plant head visits. The supervisor gets a weekly call. A dashboard appears. Behaviour changes because it is being watched.

The evidence is messier than the folklore. McCambridge and colleagues reviewed nineteen studies designed to measure research participation effects. Most reported some effect, but they were so heterogeneous that little can be securely known about its size, conditions or mechanism. That is the honest state of the Hawthorne effect: a possibility, not a number you can apply.

So do not estimate it, equalise it. If trained supervisors got a monthly visit from the HR business partner, the comparison group gets the same visit with the training content removed. Same cadence, same measurement, no programme. Whatever improvement shows up in both groups is attention, measured rather than argued about.

Then use a comparison group, and check its trend

Only now do you spend effort on a control group. The method is difference-in-differences: the before-and-after change in the trained group minus the before-and-after change in the untrained group, with the remainder treated as the effect. The World Bank guidance states the assumption plainly. It holds if the two groups would have moved in parallel without the programme, and that cannot be proved, only assessed: check the trends ran parallel for several periods before the programme started, and run a placebo test on a group the programme never touched.

One more trap sits inside this step. If you chose the trained group because it was the worst, part of the improvement is guaranteed before anyone is trained. Barnett and colleagues show the size of the problem in a real trial. In the placebo arm of a betacarotene study, serum levels appeared to rise by 0.24 on the log scale after a baseline cut-off; their estimate of the regression-to-the-mean effect was 0.29. The improvement was an artefact of selecting low baseline values.

Training the ten worst branches is exactly this design. The worst branches improve on the next measurement with or without you, because extreme readings are partly noise. Do not stop targeting the worst. Put the near-worst in the comparison and expect a rise in both groups.

A worked example, with invented numbers

Twelve branches in one region, same HRIS, same definition, same reporting population throughout. The four branches with the highest supervisor attrition were trained in April. The other eight were not, and their attrition was lower to begin with.

The headline the programme owner wanted was 3.8 points. The defensible number is 1.6, and even that is soft. With four branches, the uncertainty around each rate is wide enough that 1.6 points is not distinguishable from zero. Add the selection problem: the trained branches started 3.5 points higher, so part of their larger fall is regression to the mean, not training.

The honest slide says: attrition fell in both groups, consistent with the April-to-June calendar; the trained group fell further, but the gap is within noise at this size; we will re-read it at twelve months against the same comparison. Less exciting, and it survives a challenge from the plant head, the CFO and whoever inherits the file.

What skipping this costs

The cost is not one wrong slide. It is the credibility of the next ten. An L&D function that claims a 4-point attrition win on one before-and-after comparison gets audited once, loses, and then finds its accurate claims discounted too. That is the subject of why a single training ROI number is usually fiction.

There is a second cost: false negatives. A programme that worked gets buried because it was measured against a group already improving faster. Bad attribution cuts both ways.

The honest limit

Difference-in-differences cannot prove equal trends. With four or eight units, no test will rescue you, and a confidence interval that crosses zero means you report no detectable effect, not that you look harder for a subgroup that moved. In some cases there is no credible comparison group at all: a company-wide policy change has no untrained twin.

When that happens, change the claim rather than the analysis. Say what you changed, what moved, what else moved at the same time, and what you cannot separate. Decision-makers can act on a plausible mechanism plus a leading indicator. What they cannot forgive is a business metric presented as proof when the only evidence is a line going down after a training room emptied.

It is the same discipline that makes a pre and post test credible, applied one level up at the business metric. And it is why order matters more than technique: baseline first, then programme, then claim.

What to do before the next review

Take the one metric you intend to claim this quarter and write a four-line attribution note. One: the definition at both dates, and whether the reporting population changed. Two: the same period last year, and what the calendar usually does. Three: the metric split by site and grade, to see whether composition is doing the work. Four: the comparison group and its pre-programme trend.

If any line is blank, the claim is a hypothesis, not a result, and it gets a date to be re-checked. If all four hold, you have a number that will survive the plant head's question, which is the only test that matters in that room.

Put the note in the appendix of your reporting pack and keep it there. The report is the evidence; the slide is the summary. Our assessment and reporting workflow keeps the definition, the group and the baseline travelling with the number, because the alternative is reconstructing all three from memory six months later, in front of the person who signs the budget.