One result, one level: competency mapping without double counting

A plant head once showed me a talent review sheet where a maintenance manager sat at level 3 on "Customer focus" and level 4 on "Communication". I asked which items produced those two numbers. He pulled up the assessment workbook. The 360 item "listens to the shift before proposing a fix" appeared under both competencies. So did a supervisor form item about handling escalations. Five rows in total were counted twice, and because those five rows scored high, the manager's Customer focus average moved from 3.1 to 3.4, which crossed the company's 3.3 cut for the top band.

Nobody had cheated. The map was built after the data arrived, by an analyst who filed each item where it seemed to fit, and a well-written item fits in two places. That is the whole failure mode. Competency mapping is not a design document. It is arithmetic, and the arithmetic has to be fixed before the responses land.

Why the map drifts after the data arrives

Assessment data arrives as items. Competencies are a summary layer you impose on top. The summary is only as good as the rule that connects one to the other, and if that rule is written afterwards, it gets written to be plausible rather than exclusive.

Three things push it off course.

The first is that item wording is deliberately broad. "Listens before responding" is good item design because it is observable in many situations. It is also a sentence that a reasonable person can file under Customer focus, Communication, or Coaching. Breadth in the item becomes ambiguity in the map.

The second is that instruments overlap by construction. A manager rating form, a 360, and a structured interview all watch the same behaviour from different angles. That is useful when you keep the sources separate. It becomes double counting when you pool them into one average and forget that three sources watching one behaviour are not three behaviours.

There is a harder problem underneath. The dimension scores you are mapping may not be as separate as the framework implies. In the assessment centre literature, ratings of the same dimension across different exercises have repeatedly failed to behave like a single construct, and ratings of different dimensions within one exercise have failed to stay apart. Russell and Domm, testing an alternative explanation, found that task-based ratings carried the criterion signal: correlations of .22 to .28 with superiors' appraisals and .32 to .35 with store profit, while the traditional trait-based dimension ratings did not. If the underlying dimensions are that entangled, a map that multiplies them creates precision that the data does not contain.

Step one: build the map before the first response lands

Sit with the item bank, not the framework document. Every item gets exactly one row, one competency. That is the only rule that survives contact with a real analyst.

The map is a three-column table:

Two constraints make this table do work.

First, an item that will not sit in one competency is a bad item, not a hard call. If "listens before responding" could belong to three competencies, rewrite it until it belongs to one, or delete it. The rewrite usually means adding the object of the behaviour: "listens to the customer's complaint before proposing a fix" files cleanly. The cost of deleting is small. The cost of ambiguity is a number you cannot defend in a review.

Second, write each competency definition with a boundary sentence: what it is, and what it is not. "Commercial judgement: decides what to do with a margin problem. Not: decides how to communicate the decision." Boundaries do the filing work that adjectives cannot.

Build this table while the instruments are being finalised. If you cannot fill a cell, the instrument is not ready. If you are still choosing instruments, the overlap test in this piece on tool consolidation tells you which two are measuring the same thing before you map either of them.

Step two: run the row budget

The row budget catches double counting in ten minutes and works on a map someone else built. Count the rows per competency and compare the total to the number of distinct items across all instruments. If the row total is higher than the item count, items have been filed more than once. That is the whole test.

Do it per competency too, because averages hide it. If Customer focus has 14 rows and you know the 360 contributes 8 items and the manager form 4, the extra 2 rows are duplicates.

Set a floor at the same time. A competency with fewer than three distinct items behind it does not produce a level. It produces a note. Three items from two instruments is the minimum I would defend in a talent review, and the two-instrument condition matters more than the count: three items from one rater are one opinion asked three times.

Step three: average the raters before you score the competency

This step fixes the 360 weight problem, and it is one line of arithmetic most workbooks skip.

For each 360 item, average across raters first. Now each item is one number, and the rater group is one source with one perspective. Then the competency score is the mean of item scores, and each item counts once regardless of how many people answered it.

The alternative, pooling all rater rows, does something specific and wrong: it treats agreement between raters as additional evidence about the person, when it is mostly evidence about the raters. Interrater reliability is the number that tells you how much of the variance in those rows is the person and how much is the rater. Salgado and Moscoso pooled 224 independent samples and put the observed interrater reliability of supervisory ratings of overall performance at 0.61 when collected for research and 0.45 when collected for administrative purposes. Their figures are for overall performance ratings, not competency ratings, so treat them as an order of magnitude rather than a precise transfer. But the direction is clear: a large share of the differences between two raters is rater, not ratee. Pooling those rows multiplies noise.

The same source used twice is also not twice the evidence. A manager's rating and the 360 both feeding a competency are correlated, because both are somebody's impression of the same person at the same time. Averaging them reduces a little error. Summing them multiplies it.

Step four: convert scores to levels with a rule fixed in advance

Write the band rule down before you see the data. Two workable versions:

The count rule is stricter and slower. It also survives the question "why is he a 4?" because the answer is two named items, not a decimal. For talent reviews where someone will argue, use the count rule.

The band rule needs a caveat printed next to it. With a four-level scale and a typical spread of about 0.7 levels, an interrater reliability of 0.45 gives a standard error of measurement of roughly half a level. That means a 90 percent band around any single level is about plus or minus 0.85 levels. A person at 3.1 and a person at 3.6 are, on that evidence, not distinguishable. Report the band, not the decimal, and never rank people on differences smaller than the band. If you are running this in a tool rather than a workbook, the scoring model should be able to show the band by default; that is one of the things to check in a competency and assessment platform before you commit a cycle to it.

Step five: weight evidence, not instruments

Once each competency has its items, weights decide how much each source counts inside that competency. Three rules.

Weights sum to 1 within a competency, not across the whole framework. A competency scored from two sources uses 0.7 and 0.3, or 0.5 and 0.5. Weights of 1 and 1 quietly make the second source twice as important as the first.

Weight by what the source can see, not by what it cost. An ability test sees maximum performance on a task. A manager sees typical performance over months. For "analytical reasoning", the test deserves the heavier weight. For "coaching a junior through a mistake", the manager does, and the test should not be in the map at all.

Do not spend a week optimising the weights. Dawes showed in 1979 that equal weights on standardised predictors produced a composite correlating .48 with later faculty ratings, higher than the .38 obtained by cross validating optimally derived weights on the same data. The lesson transfers: weights chosen to look sophisticated usually buy noise, and simple weights that everyone understands survive a challenge in a review meeting. If two sources are genuinely comparable, use 0.5 and 0.5 and move on.

The quarterly drift audit

The map decays. Instruments get revised, items get reworded, a new analyst files by feel, and within two cycles you are back to the sheet I opened with. A one-hour quarterly audit catches it.

Take ten people at random from the last cycle. For each, pick two competencies and trace the reported level back to named items. Count three things:

• Orphan levels. A level with no item behind it, or fewer than three. Any orphan is a defect.
• Double-filed items. Any item ID appearing under two competencies. Target is zero.
• Source monoculture. A competency where every item comes from one instrument. Flag it, because the level is one perspective.

Then re-run the row budget on the whole framework. If the audit keeps finding the same defects, the framework itself is the problem, and the retranslation test for competency anchors is the cheaper fix than another year of patching. If the row total exceeds the item count, the drift is systemic and the map needs rebuilding.

Keep the audit output. One page per quarter, with the ten names, the traced items and the defects found, is the document that answers "how do you know these numbers are right?" It is also the only way to see whether the fix is holding.

A worked example, with illustrative numbers

A 40-person auto components manufacturer, six competencies, three instruments: a 30-item ability test, a 24-item 360 with five raters, and a 12-item manager form. That is 66 distinct items.

The first pass pooled every row. The 360 contributed 120 rows, and nine items had been filed under two competencies, so the workbook held 171 rows for 66 items. Every competency average was pulled toward the 360.

After the fix: raters averaged per item, nine items assigned to a single competency, three items deleted for ambiguity. The workbook held 63 rows for 63 items.

One manager's Customer focus score moved from 3.4 to 3.1, dropping him out of the top band and out of the succession slate he had been added to a month earlier. His Commercial judgement score did not move at all, because none of the duplicated items touched it. That is the useful part of the exercise: it does not move everyone down. It moves the people whose scores were resting on repeated evidence.

The numbers are illustrative. The direction is not.

Where this stops working

A clean map does not make an instrument valid. It makes the arithmetic honest. If the items are badly written, you now have a tidy, traceable, wrong answer, which is worse than a messy one because it looks defensible.

The method also assumes the competencies are distinct enough to file items into. If "Customer focus" and "Client orientation" are separate competencies, the map will show they are one cell within an hour, and the fix is to merge them.

It also does not decide what you do with the gap once a level is defensible. That is a separate step, covered in turning a competency gap into a funded training plan.

And small teams break the 360 part. With three raters, averaging across raters is still right, but the result is one person's view with a thin veil. Below five raters, do not report a level from the 360 at all. Report the comments.

Do this on Monday

Take the last talent review sheet. Pick one person. For each competency level on that sheet, name the items behind it. If you cannot name three, the level is a guess with a decimal point. That trace takes twenty minutes, which is the same twenty minutes the audit runs on ten people each quarter. Do it once before you defend the sheet in a room where someone will ask.