Keeping a competency framework alive after year one

The framework was signed off in March. Eleven competencies, four levels each, anchors written by a working group that met six times. Within twelve months, 38% of supervisors sat at level 3 or above on every competency. By the second March, 61% did.

Nothing visible had gone wrong. The training was delivered, the calibration session happened, the intranet page still loaded. What changed is that the instrument moved, and nobody was measuring the movement.

A framework has a shelf life, and it is shorter than the review cycle almost every company gives it. Most organisations review it when someone complains, which is never, or when a new HR head arrives, for the wrong reason.

A framework is an instrument, and instruments drift

The reason drift is so hard to see is that a framework produces one output: a level, assigned by a manager, once a year, about a person the manager also appraises. There is no standard error printed next to the number, no flag when two raters disagree, no note on the report saying which version of the anchors was used.

So the only visible signal is the distribution, and a distribution that tidies itself up looks like progress. More people at level 3 reads as a stronger team. It might be a softer scale.

Certification bodies solved this problem decades ago, because they had to. The NCCA standards, which accredit professional certification programmes in the United States, treat the underlying job analysis as perishable and require a review mechanism rather than a single study: a job analysis every five years as a general guideline, every one to three years where the profession changes fast, and as long as seven or eight years where the content is stable. The standard also asks for an ongoing mechanism to review and confirm that the content specifications still apply, separate from the scheduled study.

When was your last job analysis, and where is the review interval written down? For most L&D teams that interval does not exist anywhere, which is the finding.

Three things drift, and only one of them is the work

Most framework reviews look at exactly one of these and then declare the document fit for another year.

Content drift. The work changed. A new ERP, a merged role, a regulation, a shift from counter sales to app sales. The anchors now describe a job fewer people do. This is the drift everyone expects, because managers complain about it out loud.

Rater drift. The people applying the scale changed. In measurement terms this is not noise, it is a parameter: the many-facet Rasch literature models rater severity explicitly and has documented that severity shifts across occasions, which is why time-specific severity models exist at all. Your plant head who got tougher after a bad quarter, and your HR business partner who grades generously because she has to deliver the feedback, are both moving the scale under the same anchors.

Composition drift. The population changed. Restructuring, new hiring channels, an acquisition. Level 3 in a team of fifteen-year veterans is a different standard of performance from level 3 in a team hired last year, even when the anchors are word for word identical. The framework did not change. The meaning of a level did.

These three produce the same symptom: a distribution that moves. That is why a review that only reads the document cannot find the problem.

Freeze the numbers before you touch a word

The single most common mistake in a framework review is editing the anchors first. Once the wording changes, every historic score becomes non-comparable, and you have destroyed the only baseline you will ever have.

So export first, before the workshop, before the survey, before anyone opens the document:

• Assigned levels for the last two cycles, by person, competency, rater and date.
• The rater identity against each rating. If your system anonymises this, you cannot run the rater check, and that is a design decision worth revisiting.
• One outcome measure per competency that exists independently of the framework: quality escapes, rework hours, first-90-day attrition, ticket reopen rate, sales conversion, whatever the business already counts.

Store that export with the date in the filename, outside the shared drive the framework lives in.

Three drift checks you can run on data you already have

None of these needs a psychometrician. All three run in a spreadsheet, and they take an afternoon.

The fourth check is the one nobody runs and the cheapest. It needs no statistics. Ask the HR team to list any decision where a specific competency level was the reason. If a competency produced nothing, you do not improve its anchors. You delete it.

Version the anchors, not the spreadsheet

A framework edited by six people over three years is not one framework. It is six frameworks wearing the same title, and any trend line drawn across them is fiction.

The discipline that fixes this is small and boring:

• One named owner. A person, not a committee. The committee can advise. Someone has to be accountable for the version that is live.
• Stable IDs. A competency keeps its ID for life. If "Customer Focus" becomes "Customer Ownership", the ID does not change and the change log records the retitle.
• A change log with six fields. Date, competency ID, what changed, why, who approved, and which historic scores are now non-comparable. That last field is the one that stops a well-meaning edit from quietly invalidating a promotion decision.
• Retire, never delete. Retired competencies move to an archive section with a retirement date. Deleting one breaks every historic score you might need in a dispute, an audit, or a grievance.
• Freeze the level count inside a cycle. Adding a fifth level mid-year rescales everything that follows. If four levels were not enough, that is a finding for the next re-baseline, not an edit for this month.
• One published version per year, dated. The version string appears on any report that cites a level, the way a date appears on a lab result.

It is the minimum needed to answer one honest question: when two managers rated the same person a level apart, was that opinion, or were they reading different documents?

Set the review interval from how fast the work moves

Two clocks, not one.

The light review runs once a year and takes a day. It is the four drift checks above, a read of the change log, and a decision on each competency: keep, retire, or re-anchor. No workshops, no external consultant, no new document.

The full rebuild runs on an interval set by the rate of change in the work, using the same logic the certification bodies use: three years for functions where the technology or regulation moves (digital, compliance, field service with new equipment), five years for stable operational roles, longer for roles whose content genuinely has not changed. Do not pick one number for the whole company. Pick per job family and write it down.

Then add event triggers that override the calendar. A new system that changes the work, a merger, a regulatory change, a new job family, or a strategy that reorders what the company rewards. Each one starts a rebuild, because after any of those the old anchors describe a job that no longer exists.

A worked example, with illustrative numbers

A three-plant manufacturer, 42 supervisors, 11 competencies, four levels. Numbers below are illustrative, not a client case.

Year one: 34% of supervisor ratings sat at level 3 or above, averaged across competencies. Manager severity ran from 2.1 to 3.4 on the four-point scale, with a company mean of 2.6. Adjacent-level separation held on 8 of the 11 competencies.

Year two: the share at level 3 or above was 47%. Severity now ran from 2.0 to 3.7. Separation failed on three competencies, including "Quality Orientation", where people at level 4 had 0.04 more quality escapes per person than people at level 3. The top level was measuring seniority, not quality.

The review took one day and produced four decisions:

1. "Quality Orientation" was re-anchored to three levels, with the outcome attached to each level from the plant's existing rework data.
2. Two managers more than 0.5 levels from the company mean were put through a calibration session on nine named cases where performance was not in dispute.
3. "Documentation Discipline" was retired. It had driven zero decisions in eighteen months and every manager had rated everyone at level 3.
4. The full rebuild was scheduled for year three, not year five, because the plant was installing a new MES.

Everything else stayed untouched. A good annual review changes two or three things out of eleven, and can prove why.

Where this advice stops

The rater check needs enough raters. With fewer than about twenty managers producing ratings, severity differences are indistinguishable from real differences in their teams, and the check chases noise. At fifteen reports, stop running statistics and calibrate on real cases instead.

The diagnostics tell you that the distribution moved. They do not tell you which of the three drifts caused it. For that you need the outcome column: if the outcome moved with the level, the work changed and the anchors need updating. If the level moved and the outcome did not, a rater or the population moved, and rewriting the anchors will not help.

Version control does not rescue a badly built framework. It only makes the flaws visible sooner. If the original job analysis was a working group inventing eleven competencies from memory, the annual review will keep finding problems because there are problems. The build method is covered in how to write a framework managers will actually use.

And a strategy change is not drift. If the company now rewards something the framework never measured, no amount of versioning fixes it. Rebuild, and keep the old version archived so you can explain the discontinuity later.

What to do this week

Pick one competency. Pull last year's assigned levels from whatever system holds them, and count how many people sit at level 3 or above.

If that number is over 50 and it rose from the year before, you have your first drift reading and you can get it before lunch. Then open the document and check whether the anchors for that competency are attached to a version and a date. If they are not, that is the second finding: you cannot currently tell which framework produced last year's scores.

Two readings in an afternoon justify the day-long review. If you want the levels, the evidence sources and the change log in one place rather than three spreadsheets, that is what the competency and assessment module is for, and the mapping rules that stop one result counting twice are in one result, one level.