Your training sample may be missing the people who need it

The plant HR team draws a neat sample from its active-employee spreadsheet. Every selected name has a department, role and supervisor. The survey goes out, responses arrive, and the report says maintenance needs refresher training.

Then a supervisor points out that several contract technicians were never on the spreadsheet. They work the same shifts, handle the same equipment and are involved in the same incidents. The sample was tidy. The population was wrong.

That is not a sampling problem. It is a frame problem. Before asking who to assess, establish who belongs in the population and whether the list can find them. Otherwise, a larger sample only gives you a more precise answer about the wrong group.

The United Nations Statistics Division’s survey handbook distinguishes the target population from the frame used to reach it. Undercoverage leaves eligible units out; overcoverage includes ineligible or duplicate units. In a TNA, either can distort what looks like a skills gap. The handbook is written for surveys, not corporate HR, but the distinction transfers directly.

First define the population, not the spreadsheet

Write one sentence that says who the assessment is meant to represent. Include the work, site, employment relationship and reference date. “All people who operate or maintain line 3 equipment at Plant A during the November assessment” is better than “Plant A employees.”

This definition makes awkward cases visible. Does it include agency technicians? Employees on temporary assignment? New starters who have not completed onboarding? People on leave? A contractor who operates the equipment daily but reports to a vendor? Do not resolve these cases by convenience. Decide based on the question the TNA must answer.

Next list the records that might describe that population. In a typical site review, these can include the HRIS roster, payroll extract, access or gate register, contractor deployment list, and shift allocation. These are not interchangeable sources of truth. Payroll can establish who was paid in a period but may not identify every person doing the target task. A gate register shows recorded access, not competence or employment status. HRIS often captures employees well and non-employees poorly.

Treat each source as evidence about a different edge of the population. The point is not to crown one system the winner. It is to find the people who appear in one source but not another and resolve why.

Build a three-way reconciliation

Freeze extracts to a common reference window. Use a period that matches the intended population, not whichever export is easiest to obtain. Record extraction dates, inclusion rules and the owner of each file. Work from a copy with only the fields needed for matching. Avoid circulating personal identifiers more widely than necessary.

Create a common matching key. An employee ID is usually useful for employees, but contractors may have vendor IDs, badge IDs or names with inconsistent spellings. Keep the original identifiers in a restricted crosswalk. In the working file, use a stable audit ID. Do not match people on name alone when a second identifier is available.

Then build a row per person or unresolved candidate, with flags for presence in each source:

The table is a triage tool, not a verdict. “Present in all three” does not prove the person performs the target task. “Missing from one” does not prove an error. The business owner needs to decide the exception using evidence and record the reason.

Resolve exceptions before drawing names

Use an exception register with one line per unresolved case. Include the audit ID, sources where it appears, likely issue, evidence checked, decision, decision owner, date and action. The decision should be one of: include, exclude, correct the frame, or hold for clarification. Give each case an owner and a due date. Otherwise a register becomes a list of interesting mysteries.

Separate the main error types. An omission is an in-scope person not represented. A duplicate is one person represented more than once. An outdated record is someone whose status or location changed before the reference date. A classification error puts a person in the wrong role, site or shift stratum. A legitimate difference is not an error, but it still needs a documented rule so the next analyst does not reopen it.

Do not quietly substitute one person for another to make a sample easy to reach. If the selected person is unavailable, record non-response and follow the prespecified replacement or follow-up rule. Substitution can hide a frame defect or systematically remove night-shift, remote or contractor workers. If you use a non-probability sample, say so plainly and limit the claims accordingly.

For assessment purposes, validate task eligibility as well as employment status. An access record may show entry to a plant, not that the person operates a machine. Confirm the task against a shift assignment, supervisor confirmation or work record. The supervisor is useful for resolving a specific case, not for rewriting the population from memory.

Use rates as flags, not magic thresholds

Calculate at least three things: unresolved exception count, confirmed in-scope omissions, and confirmed in-scope duplicates. Divide by the best documented estimated population, and show the numerator and denominator. Where one source is known to miss a category, calculate the rate separately for that category rather than diluting it in the employee total.

There is no universal percentage at which an HR roster becomes unusable. A low overall error rate can hide a serious omission in a small but operationally important group. A site might have few unmatched people overall and still miss most contractor operators. Conversely, several differences may all be explainable and have no effect on the intended task population.

Set the stop rule before reviewing results. A practical rule is: pause sampling when any exception could change who is eligible, alter a planned stratum, or leave an important work group without a known selection probability. Resume only when those cases are resolved or the assessment explicitly narrows its claims. This is more defensible than inventing a universal 5% cutoff.

If a threshold is required for governance, establish it from the cost of missing a person, the size and importance of the affected group, and the planned analysis. Keep the threshold separate from the number of unresolved cases. A frame with one unresolved exception can still be unsafe if that case represents the only night-shift specialist.

Worked example: a plant maintenance TNA

All figures below are illustrative. A plant plans to assess people who perform preventive maintenance on a production line during a defined month. The HR roster has 84 names with the relevant job family. Payroll has 82 matching employee IDs for the same period. Gate records show 91 distinct badges associated with the line’s maintenance area. These counts are not expected to match automatically: gate access includes people who may not do the target work, and some employees may enter through another gate.

The analyst first matches exact employee IDs. Eighty names appear in both HR and payroll. Two HR-only records are explained by a transfer effective after the payroll cut-off. Two payroll-only records are active employees assigned to another site, so they are excluded from the target population. The gate list contains nine additional badges. A deployment list identifies six as vendor technicians assigned to preventive maintenance, two as logistics staff who enter the area, and one as a duplicate badge record.

The six technicians are in the defined work population even though the employee roster omits them. The logistics staff and duplicate badge do not belong in the assessment frame. The analyst confirms that the two transfer cases are also outside the reference month and records the rationale. The working population is then updated with the six technicians, each identified through the vendor deployment record and confirmed by the site owner.

The key finding is not simply “six more people.” The employee-only frame would have excluded an entire employment category that performs the target task. If the assessment had sampled only from HRIS, no weighting adjustment could recover their missing responses. The frame needs to be corrected before selection.

The analyst now draws the sample from the reconciled list, retains a stratum for vendor technicians if their work differs, and reports coverage by employment category. The population count, exclusions and matching decisions go in the methods note. The precise sample size comes later. A sample cannot repair a list that never contained the people it was meant to represent.

Record what you cannot reconcile

Sometimes the sources cannot be linked reliably. A shared badge, common name, missing vendor roster or late payroll update may leave identity unresolved. Do not force a match to make the spreadsheet clean. Keep an unresolved category, state its likely size or bounds if you can support them, and explain what that uncertainty means for the TNA.

A frame audit also has a boundary. It can establish whether the list plausibly covers the defined population. It cannot prove that the chosen task matters, that a test measures competence, or that training is the right intervention. Those are separate decisions. The TNA-to-training-plan method starts after the gap has credible evidence. The assessment features page describes how structured assessments can support that next stage, not fix a bad frame.

Do this before the next assessment invite

Before selecting participants, write the population sentence and ask HR operations, payroll and the site owner for extracts covering the same dates. Reconcile them, log every unmatched case, and agree the stop rule before sampling begins.

If nobody can explain who is missing from the list, do not send the assessment yet. Spend the day fixing the frame. It is cheaper than spending the next month explaining why the people with the most relevant work never appeared in the results.