Designing a TNA for a distributed field workforce

The Google Form went into the field sales WhatsApp group on a Tuesday. 1,412 officers were in the group. By Friday 612 had answered, and the report said 71 per cent of the field wanted negotiation skills training.

The 612 were not a sample of the field. They were the sample of the field that sits within signal, has a phone that opens Forms without dying, and had a spare eight minutes between calls because their beat is compact. The 800 who did not answer were on route in the districts with patchy data, which is where the discounting problems actually live. The training was designed for the people who were easy to hear.

Nothing in that pipeline was dishonest. The failure came from treating a distributed workforce as if it were a room.

Three things break when the workforce is not in one place

A TNA designed for a plant works because the population is fixed, present and observable. Take that same design to a field force and three assumptions collapse.

Attendance is not the population. In a plant, the roster and the room are the same list. In the field, the roster is 1,400 people and the reachable list is whoever the supervisor can get on the phone this week. If you sample from the reachable list, you have sampled supervisor conscientiousness, not capability.

The instrument travels through a different environment. A 22-item Likert survey is fine at a desk. In a truck cab, on a phone, with one bar of signal, it is 30 per cent completion and a lot of straight-lining.

The rating comes from someone with an incentive. Most field TNAs lean on supervisor ratings because they are cheap. A supervisor who rates his team low is rating his own supervision. That does not make the ratings useless. It makes them something you have to calibrate before you compare them across territories.

Sample sites, not people, and then correct for it

The first decision is not sample size. It is the unit. For a field workforce the natural unit is the site, beat, depot, branch or route, because that is how the work and the supervision are organised.

This is the same problem public health surveys solved decades ago. The WHO Expanded Programme on Immunization standardised a 30-cluster design with 7 children per cluster in 1978, and it became the default for coverage measurement because walking every household was impossible. Later work compared 30 by 30, 33 by 6 and 67 by 3 designs and found the small-cluster designs are more efficient for outcomes that are strongly clustered within a site, and less efficient where there is real variation inside the cluster.

That trade-off is the whole design. Clustered sampling loses precision, and the loss is quantified by the design effect:

Deff = 1 + (n - 1) × ρ

where n is the average number of respondents per cluster and ρ is the intra-cluster correlation, the degree to which people in the same site resemble each other. Worked in public health teaching material: four practices enrolling 25 patients each looks like 100 people, but at ρ = 0.017 the effective sample is 71. The other 29 interviews bought almost nothing.

Field workforces have much higher ρ than that, because people in one territory share a manager, a customer mix and a set of local problems. Assume 0.10 to 0.20 until you have your own estimate.

Same field cost, meaningfully different precision. The fix is more sites and fewer people per site, right up to the point where travel between sites costs more than the extra interview. That crossover is usually the real constraint, and it is a cost question, not a statistics question.

Two practical rules follow. Draw the sample from the duty roster, not attendance and not the WhatsApp group. And give every sampled person a replacement rule decided in advance, so a supervisor cannot substitute the officer who is easiest to reach.

Non-response is a bias question, not a rate question

The instinct is to chase a response rate. The American Association for Public Opinion Research notes that rates have declined across every mode, that comparisons against census benchmarks have questioned the link between response rate and accuracy, and that the field now examines surveys for non-response bias regardless of how high the rate is.

Applied to a field TNA, that means two concrete things.

First, report who answered by site, tenure band and reporting manager, not just a total. If two of your forty sites contribute nothing, say so in the findings, because those two sites are probably the ones with the worst connectivity or the worst manager, and both are informative.

Second, run a short non-response check. Take 20 non-respondents, call them, and ask the three questions that matter most. If their answers differ from responders by more than a few points, the headline finding is not safe.

The instrument has to survive a truck cab

Design for the worst device and the worst signal in the population, not the best.

The last row matters more than it looks. An officer who believes a low self-rating will reach his manager has an incentive to mark himself proficient. Davis and colleagues reviewed 20 comparisons of physician self-rating against external observation: 13 showed little, no or an inverse relationship, and the worst accuracy sat with the least skilled and the most confident. If your instrument invites a self-rating and signals that it is visible, expect that tail to be the loudest part of your data.

Language: translate the question, not the sentence

A TNA translated by one bilingual person in head office produces questions that are grammatically correct and semantically wrong. The classic example is asking about "escalation" in a context where the field word is "raising it to the SO". Respondents answer the question they think you asked.

Two steps fix most of it, and both are cheap.

Write the question in the field's vocabulary, gathered from four or five interviews with people in the role. Then back-translate: two people who did not write the original translate the local version back into English independently. Where the two back-translations disagree, the item is ambiguous and gets rewritten. Do not skip this because the wording looks obvious to you.

Free text needs the same care. Keep one open question in the local language and have it coded by someone fluent. Machine-translating 400 responses into English and coding them loses the operational detail you were paying for.

Calibrate supervisors before you compare territories

If supervisor ratings are part of the design, the ratings are only comparable if the supervisors are using the scale the same way. Most are not. Some rate everyone 3. Some never give 4.

Calibration is a half-day, and it is mechanical:

1. Write three short vignettes describing a field officer's behaviour on the target competency. Real, specific, taken from interviews.
2. Write the anchors so each level has an observable marker: what the person does, not how good they are.
3. Have every supervisor rate all three vignettes independently, before any discussion.
4. Compute exact agreement and within-one-point agreement against the reference rating.
5. If within-one-point agreement is below 70 per cent, the anchors are too vague. Rewrite them and repeat.

Then check drift. Have supervisors re-rate one vignette three weeks later without warning. A supervisor whose own two ratings of the same vignette disagree is adding noise to every territory average they contribute.

Do not let the calibration round become a session on being nicer. The goal is agreement on what the evidence means, not consensus on the person.

A worked example, with illustrative numbers

A distribution business, 1,200 field officers across 96 territories, 12 to 14 officers each. The question: can officers handle a credit-blocked retailer conversation without escalating?

Design: 80 territories sampled at random from the roster, 4 officers drawn at random inside each, 320 interviews. Average cluster size 4. From a pilot on 3 territories, ρ for the supervisor-rated item came out at 0.11, so Deff = 1 + 3 × 0.11 = 1.33, and the effective sample is 241.

The headline estimate came back at 46 per cent of officers handling the conversation unaided, with a 95 per cent interval of roughly plus or minus 6 points given the clustering. Unclustered arithmetic would have claimed plus or minus 5.5. The honest number is wider, and the difference is not academic when the decision is whether to spend on a national programme or on the 20 worst territories.

The site-level output is where the value was. Twelve territories came in under 25 per cent, and nine of those twelve shared a reporting manager. That pattern is not a training need in the usual sense. It is either a manager who has not been taught the conversation himself, or a credit policy that makes escalation rational. The CDC's needs assessment guidance makes the same point at the analysis stage: a performance gap can sit in policy, technology or standard operating procedure, and training is only one candidate fix.

That finding cost one extra column in the analysis and changed the intervention.

Where this method stops

It gives you site-level prevalence and a defensible ranking of territories. It does not give you an individual diagnosis. Four people per site cannot support a per-person learning plan, and you should not let a manager read it that way.

Supervisor ratings still contain the supervisor. Calibration reduces the between-rater noise; it does not remove the incentive to rate a team in a way that reflects well on its management. Where the decision is expensive, pair the rating with an observed work sample on a small subsample, even 30 calls, and check whether the two agree.

And the design assumes your roster is accurate. If 15 per cent of the roster has left and nobody has updated it, your sample is drawn from a list that includes ghosts and misses new joiners. Fix the roster first. It is unglamorous and it is the actual prerequisite.

Take four sites this week

Pick four territories at random from the duty roster, with a random number generator and a witness. Draw four officers in each. Ask them one question about the task that costs you the most money when it goes wrong, in their own words, and ask their supervisor to rate the same four on one anchored item.

You will learn three things in a week: whether the roster is real, whether the question survives translation, and how far apart supervisors sit on the same scale. Those three answers decide whether a national TNA is worth funding, and none of them require a survey platform. If the roster turns out to be the weak link, that is the finding. Read the three instruments that cost nothing before you buy a tool, and settle who holds the budget for the analysis before you promise anyone a programme. If the gap turns out to be environmental rather than a skill gap, the four-question filter will save you the spend entirely. When you are ready to run the cycle across every territory, the assessment workflow handles the sampling and the report in one place.