Running Kirkpatrick level 3 without a research team

A telecom services company I worked with ran a six-day programme for forty branch managers. The evaluation plan had all four levels, neatly tabulated, and one line under level 3: supervisor observation at 90 days. When the 90 days arrived, twenty-two of the forty had changed reporting managers, nine had been transferred or were on long leave, and the observation form, which nobody had tested before printing it, ran to thirty-four items. Four forms came back. Nobody was being lazy. The plan had simply never been built to survive contact with a branch.

The usual diagnosis is that managers do not value evaluation. That is comfortable and mostly wrong. Level 3 fails because it is treated as paperwork to be collected after the programme rather than as a measurement instrument to be designed before it. Instruments have three parts: something countable, someone whose job already puts them next to the thing being counted, and a schedule that starts before the behaviour is supposed to change. Get those three right and an L&D team of two can run level 3 on a programme of forty people. Get them wrong and a research team with a budget cannot rescue it.

Level 3 is an observation schedule, not a survey

Level 3 answers one question: did the behaviour happen at work, at the frequency the programme promised, while the participant had support to do it. Kirkpatrick Partners describe the level as measuring the degree to which the target audience performs the critical behaviours in its environment and is supported and accountable for that performance. They also make a point that most internal evaluation plans ignore: measurement of behaviour cannot begin at 90 days after learning, because by then you have lost the chance to correct the programme while it is still running.

That single sentence changes the shape of the work. Level 3 is not a follow-up survey. It is a set of scheduled observations that begin before the programme and run alongside it, on a form short enough that a busy supervisor fills it without resenting it.

There is a second reason to measure the environment and not just the person. A meta-analytic dissertation at the University of Central Florida found that supervisor support explained the largest share of variance among the work-environment supports studied, and that those supports correlated with transfer in the range of 0.15 to 0.38. Blume, Ford, Baldwin and Huang, in their 2010 meta-analysis, found post-training knowledge and trainee reactions were only modest predictors of whether people applied what they learned, while transfer climate predicted better.

Read both findings as an instruction for the form. If you collect only whether the manager did the behaviour, you learn whether the person complied. If you also collect what stopped them, you learn whether the programme was set up to work. The second column is the one that gets budget.

Write the behaviour as a countable event

Most level 3 plans fail at the first sentence, before any observation happens. The behaviour is written as an adjective: coaches well, communicates clearly, thinks strategically. Nobody can count an adjective, so two observers watching the same week reach different verdicts, and the data becomes opinion with a timestamp.

Use one test. Could a colleague who has never met this person count the behaviour from a document or a meeting they can already see, without asking anyone for an interpretation? If not, rewrite it. And take the behaviour from the gap you already established, not from the course outline: a need and a want look identical in a training request, and only one of them produces an observable behaviour.

Three things follow. The sentence has to name a frequency window, because "did it happen" produces a yes that means once in six weeks. It has to name the artifact where the evidence appears, because that decides who observes. And some competencies will not convert, so say so in the plan rather than pretending. A framework of nine competencies usually yields four to six countable behaviours for one programme. Three is better than six.

Pick the observer who is already in the room

The biggest cost in level 3 is not the form. It is the observer's time, and the scheduling friction of creating a new meeting. So choose the source that already exists.

The default that works: one work artifact plus the line manager inside the 1:1 that is already in the calendar. Never create a new meeting for evaluation. A monthly review that exists for other reasons will still exist in month four. A dedicated observation slot will not.

If the artifact carries the evidence on its own, skip the manager rating. A ticketing system that records whether the issue field was filled is a cleaner instrument than a supervisor's memory of last Tuesday.

Book the observations before the programme starts

The schedule is where most Indian programmes lose the data: the batch runs at the end of the quarter and the evaluation lands after the quarter closes.

The baseline is the step teams skip, and it is the only one that makes the rest interpretable. Six of twelve managers logging the escalation field at week six means nothing on its own. It means a great deal if the baseline was two, and very little if the baseline was eight.

Four data points per person is the floor I would trust at this scale. That is judgment, not a research finding, and it comes from how often one week turns out to be atypical because of a stocktake, an audit or a festival.

The two-minute form

The form is the whole system in one page. Six fields, phone-friendly, no rating scale.

1. Date, programme code and observation source.
2. Participant code, not a name, if the form reaches HR.
3. Behaviour one: count in the window, as 0, 1, 2 or 3 or more.
4. Behaviour two: count in the same window.
5. Behaviour three: count in the same window.
6. One specific example, or the words "none seen". Plus a blocker field, ten words maximum.

Two rules matter more than they look. Avoid the one-to-five scale: a scale invites a verdict about the person, while a count is a fact about the week. Cap the free text at a line, because a long comment box is a deferred form.

The example field is not decoration. It is your calibration data, and it is what stops an unsure supervisor from rounding up.

Calibrate both observers before you trust a number

Give two supervisors the same written case, or a five-minute role play, and ask both to complete the form. Then compare. They should land in the same frequency band, or one band apart. If they land three bands apart, the behaviour sentence is broken, not the managers. Rewrite the sentence and test again.

This costs twenty minutes, once. Without it, week six data reflects two definitions of the same competency, and the gap between 40 and 70 per cent compliance may be nothing more than which supervisor owned the form.

Read the result as two columns, not one score

Do not average the counts into a behaviour score out of five. Report two things per cohort: the proportion of participants at the threshold, baseline against week six, and a ranked list of blockers with counts.

Here is an illustrative example for a twelve-person branch manager programme on escalation handling. All numbers are invented to show the shape of the output.

Top blockers across 33 forms: nineteen reports of "the ticket field is still optional in the CRM", eleven of "the team leader reassigns the call before it ends", six of "no time during the shift", four of "the customer refuses to be put on hold".

That page supports three decisions. The behaviour moved and then slipped, so the coaching is not holding. The largest blocker is a configuration change, which is cheap and permanent. The training was not the weak link, so re-running it would have wasted the budget. Compare that with the usual report, which is a satisfaction score and a photograph.

When the behaviour data has to feed a business case, pair it with the lagging metric from the same cohort, and be explicit about what the pairing can and cannot prove. The method for that sits in how to prove a training programme changed anything.

Where this stops working

Level 3 proves behaviour, not results. A manager who logs the escalation field has done the behaviour; whether customers were retained is a level 4 question with a different design and a longer timeline. Do not let a clean level 3 report stand in for an impact claim.

The observer is also the coach, so expect inflation. Counting artifacts is the correction, along with keeping the count out of the observer's own appraisal. If a team leader learns the form affects their rating, the counts become a gift.

Small numbers do not support statistics. Twelve people is a count, not a sample. Report the counts and resist the significance test, which on a group this size mostly measures how the numbers were rounded.

Some behaviours cannot be observed at all: senior strategy work, confidential HR conversations, anything that happens once a quarter. Use an artifact the role produces, or write in the plan that level 3 is not available. That sentence is more honest than a form filled in by three people who guessed.

And the hard one. Level 3 collects evidence about people, so it will be used to judge them unless you decide otherwise. If the blockers named on the form never reach anyone who can fund a fix, the exercise becomes a list of names. The instrument is either a support tool or a surveillance tool. It cannot be both, and the difference is entirely in what happens to the blocker column.

This week

Take one live programme. Write a single behaviour sentence and test it on a colleague who was not in the design meeting: can they count it from a document they already have access to? If the answer is no, rewrite it until it is yes. Then find the baseline window, which for most Indian programmes is simply the two weeks before the next batch starts, and fill the form yourself twice before asking a manager to. The first two forms will tell you more about your evaluation design than any amount of planning, and they cost you twenty minutes. When you are ready to run the whole cycle on one system rather than four spreadsheets, the assessment and reporting side is where BolderBrain fits.