Three training needs analysis instruments that cost nothing

The last training needs analysis I watched get approved arrived as a 38 page document. It cost a little over four lakh, took eleven weeks, and surveyed 640 people on their self-rated ability across 22 competencies. Its top three findings were shopfloor documentation, supervisor feedback and dispatch accuracy.

I had read all three the previous quarter, in a file nobody had opened. The plant's rejection register: 211 rejected batches, each with a reason code and a line leader's name against it. The analysis spent money to produce what the plant already knew and had not read.

The problem is not a shortage of instruments. It is a shortage of the evidence sitting in the next room, read by someone with the patience to convert a count into a rate.

Why the survey is the default, and why it is the wrong default

There is data on what people actually use. A European Centre for Disease Prevention and Control review counted the methods inside 13 real training needs assessments, 32 uses in total. Questionnaires were 12 of those, 37 per cent. Interviews were 7, desk-based reviews of existing documents 6, focus groups 4. Nine of the 12 questionnaire studies paired it with another method, which tells you the people running them did not trust it alone.

The questionnaire dominates for a reason that has nothing to do with evidence. It is the only instrument that produces a spreadsheet by itself.

Then there is the measurement problem. A needs survey asks people to rate their own competence, and self-rating is a weak proxy for measured competence. Davis and colleagues' systematic review of physicians found a poor relationship between self-ratings and ratings from external observers of the same performance. The accompanying JAMA editorial put the sharper point on it: the inaccuracy was worst among the least competent, who overestimated their ability.

The people who most need the training are the least likely to name it, and the most likely to mark themselves proficient. The error runs in one direction, hardest where the money should go. That does not make the survey useless. It makes it a second instrument, not a first one.

Instrument one: the records you already own

Every operating unit produces evidence of its own failures, and almost none of it is read by the people who decide what training to buy. This is the desk-based review, used in 19 per cent of the method instances ECDC counted, and it is cheap because it is already paid for. If half the reason codes read "other", the register is a diary. Fix the field first.

Four rules turn a register into evidence.

Convert every count into a rate. Not 211 rejections, but 211 out of 24,300 batches, 0.87 per cent. A bare count cannot be compared to anything, including last quarter.

Slice by the unit that can act. Line, shift, product family, tenure band. A plant-wide rate hides the fact that one shift is four times the other, which is the only actionable thing in the file.

Take a quarter at minimum, three if volume is low. One bad month is noise, and the fastest way to lose a plant head is a plan built on a bad fortnight.

Write down what the record cannot tell you. That list is the interview agenda. A register tells you where; it never tells you why.

Budget for this: about six hours of an analyst's time, most of it spent arguing with the export.

Instrument two: eight critical incident interviews

The critical incident technique is old and unglamorous. Flanagan developed it in the US Army Air Forces aviation psychology programme, asking people to describe exactly what they had done when a task went well and when it went badly. It collects factual episodes, then sorts them into themes.

What it does not do is ask what training people want. That question produces "communication skills" from roughly every manager in every plant, because it is safe, unfalsifiable and sounds like leadership.

Four probes carry the interview. They come from the standard protocol summarised in the vocational education literature on the technique:

• "Describe what led to the situation."
• "Exactly what was done that was especially effective or ineffective?"
• "What was the outcome or result of this action?"
• "Why was this action effective?"

The discipline is in refusing to move on until you have an episode. If the answer contains an adjective instead of an action, ask again: "Tell me about the last time that happened. What did the person actually do?"

Eight interviews is enough. Two line leaders, four strong performers, and two people who joined in the last six months. New joiners are the most underused source in the building, because they still remember precisely what nobody told them. Thirty minutes each, same four probes. If the same episode shape appears in six of eight interviews, you have a mechanism rather than a complaint.

Write each incident as one line: situation, action, outcome. Sort the lines into themes and stop when the last two interviews produce no new one, which for a single job role happens somewhere between six and ten interviews. Six hours of manager time, no cash.

Instrument three: one observed work sample

A work sample is a short piece of the real job, done by the real person, scored against a few observable criteria by someone who can do it.

Schmidt and Hunter's meta-analysis of 85 years of selection research put work sample tests at a validity of .54 for predicting job performance, above structured interviews at .51 and job knowledge tests at .48. The same paper adds two caveats: work samples are much more costly than alternatives, and they only work with people who already know the job.

Both caveats invert when the purpose is diagnosis rather than selection. The people in front of you already know the job, and you are not building an instrument that will be defended in a tribunal. You are checking whether a gap that exists on paper also exists in a pair of hands. That takes twenty minutes and a checklist, not a test centre.

Build it in an afternoon. Take the task that appears most often in the rejection register. Write three to five criteria as pass or fail observations, never a 1 to 5 scale, because a scale invites opinion and pass or fail invites an argument about the standard, which is the conversation you want. Run it on 8 to 12 people across two shifts, supervisor scoring while you record.

Two rules keep it honest. Tell people what it is before they start: a check on the process, not on them. Keep the scores out of the personnel file, because the day it becomes a test the next sample is worthless.

The one-page template

Everything above collapses into six columns. If it does not fit on a page, it is not finished.

One rule makes the template do its job. Any row backed by a single instrument gets marked single source and cannot be funded above a small fixed ceiling until a second instrument confirms it. That constraint kills most of the wish list, because a wish list row is always single source and usually a manager's memory. The distinction the template depends on is set out in why a list of requested courses is not an analysis.

A worked example, with illustrative numbers

A 220-person auto component plant, dispatch errors, one quarter. The records show 340 consignments with a wrong part number out of 24,300 dispatched, 1.4 per cent. Rework runs about 180 rupees per event, and eleven customer debit notes averaged 4,000 rupees, so call it 105,000 rupees for the quarter. By shift, B sits at 2.6 per cent and A at 0.7.

The interviews produce the mechanism. Six of eight describe the same episode: a part number changes mid-shift, the change reaches the line leader as a message, and the picker works from the pick list printed at the start of the shift. The planning system is updated. The paper is not.

The work sample confirms it: ten pickers, both shifts, given a mid-shift change in the format the plant really uses. Nine of ten pick from the outdated list when the change arrives verbally. Eight of ten pick correctly when it arrives as a reprinted list.

The gap is not part number knowledge. It is a missing step in the changeover, plus no rule about where a change is allowed to arrive. The decision: one process change owned by the shift supervisor, a 45 minute briefing across three shifts, a re-measure in 30 days. Content cost nothing. The programme the analysis would have bought instead was a two day attention to detail course for 40 people, roughly 1.6 lakh, training them to compensate for a broken process.

Where this method fails

The critical incident technique does not suit every job. It suits work with many correct ways to behave. For a job with one fixed procedure, read the procedure and watch it being done.

Eight interviews and twelve work samples describe a process, not a population. Do not put a percentage on them. Work samples also carry observer bias, so have two people score the first three and compare.

None of these instruments finds a need that has no history. A new machine, a new regulation, a product line that does not exist yet: those need a task analysis of the future job, which takes an engineer's time and money.

And the honest limit of the whole approach: it finds gaps that leave a trace. A supervisor who never gives feedback leaves none in any register. That gap needs a different instrument, which is why who owns the analysis and who pays for it matters more than which questionnaire you pick. Where you need to measure a capability rather than infer it, a proper assessment platform is the one part of this method that money genuinely improves.

Start on Monday

Pick one job role, not a function. Pull last quarter's register for it and build the six column sheet before you speak to anyone. Book eight half hour interviews for next week using the four probes as written, then the work sample for week three.

If you cannot fill the evidence column for a row, that row is not a need yet. It is a request. Put it on a watch list until a record or an episode brings it back.

The register is already in the building. Somebody wrote it at the end of a bad shift. Reading it is the cheapest hour in the whole cycle.