A 360 question bank that survives ten different raters
Ten raters read the same item and answer ten different questions. The two-rater retranslation test finds those items before the cycle, not after the report.
Rajendra Kumar · 19 November 2026 · 8 min read

Representational image.
I once watched a 360 debrief stop dead on a single line. The manager was rated 2.1 out of 5 on "keeps people informed". Six of his eight raters had given him a 2. Two had given him a 5. He asked the only sensible question: what did the two people who gave me a 5 see that the six did not?
Nobody in the room could answer. The item had been in the bank since 2019, copied from a vendor template, and it had never been tested for whether two raters read the same sentence the same way. In that room we had six people answering "does he push information down to me before I need it" and two answering "does he protect confidential things I tell him". Both are defensible readings of "keeps people informed". Only one of them is what the framework meant.
The variance was not in his behaviour. It was in the question.
Where the noise actually enters
Survey methodologists model an answer as four steps: comprehension, retrieval, judgment, response. The rater has to understand what is being asked, recall specific events, decide how those events map onto a scale, and pick a number. Most 360 quality work concentrates on the last step, arguing about whether the scale should have five points or seven. The damage is usually earlier, at comprehension and judgment.
There is direct evidence for this in instrument development. A 2026 study applying that four-stage framework to a 95-item talent scale found most problems concentrated at comprehension (16 items) and judgment (11 items), leading to 20 items being revised. An earlier cognitive interviewing study on a comfort questionnaire found 11 of its items needed rewording and six were missing entirely, and concluded that without the pretest those defects would have gone undetected.
A 360 bank has the same exposure, with one extra multiplier. In a self-report survey the respondent and the person who wrote the item are at least in the same role. In a 360 the item is answered by a direct report, a peer and a boss, who observe different slices of the job. Nowack and Mashihi, in their review of 360 practice for the APA, list four reasons raters on the same person disagree: they focus on different aspects of the job, they get different opportunities to observe, they weight the same observed behaviour differently, and the wording of the question itself influences the appraisal.
That last one is the only one you control before the cycle opens.
The evidence that the scale is not your problem
Two numbers from the same review are worth carrying into any argument about item design. Conway and Huffcutt's meta-analysis put the correlation between two supervisors rating the same person at .50, between two peers at .37, and between two subordinates at .30. Separate raters in the same group are only loosely agreeing.
And Dai and colleagues, on an archive of 737 responses, found self-other agreement was lower when a competency was stated abstractly than when it was concrete.
So the ceiling is set by observation, and the floor is set by wording. You cannot fix the first number. You can stop making the second one worse. A bank of abstract items pushes the within-group correlation below .30, and the report then presents the average of that disagreement as a finding about the manager.
The test: two raters, one item, no item text
The retranslation test is simple enough to run in a week and hard enough that most teams skip it. It borrows from cross-language instrument work, where the standard diagnostic is to have a second translator render a translated item back into the source language without seeing the original. The 360 version does not need a second language.
For each item, separately brief two raters from the group that will actually answer it. Show them the item, then ask two questions and write down the answers verbatim.
- What would this person have to do, in the last 30 days, for you to answer 5?
- What would they have to do for you to answer 2?
That is the whole instrument. You are collecting the rater's private definition of both ends of the scale. Compare the two written answers.
| What the two cards show | Reading | Action |
|---|---|---|
| Same behaviour, same window, same direction | Item works | Keep |
| Same behaviour, but one card has no time window | Retrieval problem | Add a window, retest |
| Two different behaviours, both plausible | Comprehension failure | Rewrite or cut |
| Same behaviour, but one card describes an outcome and the other an action | Judgment problem | Split into one action item |
| Both cards vague, both plausible | Item is a personality question | Cut. Do not reword |
The pass condition is not agreement on a number. It is that both cards describe the same observable event. Raters can differ on how often it happened and still pass. If they describe different events, the item is measuring the rater.
The wording rules that survive the test
Six rules do most of the work. They come out of the same survey-methodology literature, and each one maps to the failure mode it prevents.
One behaviour per item. A double-barrelled item ("communicates clearly and sets expectations") forces the rater to average two different observations. Split it. The test will fail these immediately because one card describes a meeting and the other describes a written brief.
A visible verb. If the item can be true while nothing observable happened, it is an inference item. "Explains the reasoning behind a decision" is visible. "Is transparent" is not, and no amount of scale tuning fixes it. This is the same principle behind rewriting abstract 360 items into incident questions, applied to a whole bank rather than six items.
A window and a count. "In the last 30 days" forces retrieval of instances. "Generally" invites a summary feeling, which is the rater's impression of you, not your behaviour.
Anchors that carry the frequency, not the item. Put the time reference in the item and the frequency in the scale (never, once or twice, monthly, weekly, daily). If the item says "often" and the scale says "often", the rater has to resolve two frequencies at once, and they will resolve them differently.
No negations. "Does not interrupt" produces response errors because raters have to hold the behaviour, negate it, then map it. Ask for the behaviour: "lets a colleague finish a sentence before responding".
No adjectives lifted from the competency framework. Framework language like "strategic" or "collaborative" is written for calibration meetings, not for observers. The retranslation test fails these items reliably, which is why the framework and the question bank should never be the same document.
A worked example, with illustrative numbers
Take a 24-item bank used for first-line managers across two sites. You run the retranslation test with four raters per site, so eight raters, each of whom sees eight items. That is 64 item-cards, roughly a day of interviews if you batch them.
The numbers below are illustrative, built to show the shape of the output, not measured from a real programme.
| Result | Items | What happens |
|---|---|---|
| Both cards describe the same behaviour | 15 | Keep as written |
| Same behaviour, missing or inconsistent window | 4 | Add "in the last 30 days", retest with two fresh raters |
| Two different behaviours, both defensible | 3 | Rewrite to one behaviour, retest |
| Item survives no rewrite | 2 | Cut from the bank |
The important column is the last one. Two items out of 24 do not survive. In practice those are almost always the ones that describe a disposition rather than an act, and the temptation is to keep them because the framework has a competency that needs coverage. Keeping a failed item is worse than an uncovered competency: the competency can be marked as needing a different evidence source, while the item silently produces a number that looks like data.
At the end you have 22 items, each with two documented cards showing that two independent raters read it the same way. That record is worth more than the bank itself. When a manager disputes a score in the debrief, you can put the card on the table.
What this costs and what it does not buy
A retranslation test on 24 items costs about a day of eight raters' time plus a day of yours to compare cards. Rerun it when you add items, when you change a scale, and when you translate the bank. That is the honest scope: it is a pretest, not a validation study.
Three limits are worth stating plainly.
It does not fix observation. Two raters who genuinely see different parts of the job will still disagree, and they should. If a rater has worked with the manager for six weeks, no wording change makes their rating comparable to someone who has watched for two years. That is a rater selection problem, and it belongs in your rater exclusion rules, not in the question bank.
It does not fix translation. If the bank runs in two languages, the retranslation test tells you the English items work. It says nothing about whether the Hindi or Tamil version asks the same question. That needs the two-translator protocol and an item-level stop rule, the same discipline described in why field TNAs fail at the translation step.
And it does not tell you the item is worth asking. An item can survive retranslation and still be irrelevant to the job. That is a content validity question, answered by asking managers which behaviours separate good from poor performance, not by asking raters how they read a sentence.
What to run before the next cycle
Pick the five items your reports have argued about most in the last two cycles. Those are the ones producing the disagreement you have been blaming on the manager. Run the two-card test on each, with two raters, this week. You will get your answer in an afternoon, and it will usually be that at least one of the five is not measuring what the framework thought it was measuring.
If you would rather not maintain the bank by hand, BolderBrain's 360 module keeps item versions and rater group assignment in one place, so a rewritten item stops silently inheriting historical averages from the version before it. That part matters: an item that changed wording and kept its score history has a trend line built from two different questions.
Next step
Run the same measurement on your own team
Bring a real team and a real competency. We set up the assessment, run one cycle, and walk you through the report so you can judge the output yourself.
Written by
Rajendra Kumar
Head of Enterprise Business, Conceptial
Runs enterprise work at Conceptial, the training and consulting firm behind BolderBrain. Two decades of building assessment and development programmes for Indian enterprises.