How to run a 360 cycle without it becoming a popularity contest

Fourteen managers go through a 360 at a plant in Pune. Each one submits a list of twelve raters. The reports come back in November, and eleven of the fourteen score above 4.0 out of 5 on collaboration. The two managers the plant head privately worries about score 4.3 and 4.1. Nobody in the room says anything, because the numbers look fine.

The plant head's first question is the usual one: are the questions biased? They are not the problem. The bias entered three weeks earlier, in an email where each manager named twelve people.

A 360 score is a mean of a small sample, and the person being rated usually draws the sample. That is the whole mechanism. It does not require anyone to lie. It only requires the arithmetic of a small, self-selected sample to do what small, self-selected samples do.

Why liking lands hardest on the people you pick

Two published numbers explain most of it.

The first is agreement. In a meta-analysis of 360 ratings, the average correlation between two supervisors rating the same person was .50. Between two peers it was .37. Between two direct reports it was .30 (Conway and Huffcutt, 1997, summarised in Nowack and Mashihi, 2012). A correlation of .30 means the two ratings share about 9 percent of their variance. Two colleagues who both work with you every day describe you almost independently.

So a group of two or three raters is not a measurement. It is a coin toss with a decimal point.

The second is affect. Antonioni and Park (2001) tested whether how much a rater likes the person moves the rating, across downward, upward and peer feedback. The effect was significantly stronger in upward and peer feedback than in ratings from above. Liking moves the people who report to you and the people who sit beside you more than it moves your manager.

Put the two findings together and you get the shape of the problem. The groups where liking distorts the score most are the groups a manager is most likely to nominate personally. And those same groups are usually the smallest in the report, so the distortion has nothing to average against.

Size the pool before you design anything

You cannot fix a mean of three. You can only add raters until the mean means something.

The standard correction is the Spearman-Brown formula, which tells you the reliability of an average of n raters given the correlation between any two of them:

R = n × r / (1 + (n − 1) × r)

Run it with the correlations above and the answer is blunt:

Five direct reports give you .68. Six give you .72. That is the difference between a number you can defend and a number you cannot.

The published benchmark is more conservative still. Greguras and Robie (1995), cited in the same review, put the requirement at four supervisors, eight peers and nine direct reports to reach a reliability of .70 or better, and separate work suggests that two or fewer respondents in a group is not enough for reliable measurement at all.

You do not have to hit those numbers in every organisation. You do have to write down a floor and hold it. A workable policy: no rater group with fewer than three respondents appears as a score in the report. Under three, the comments go in and the number does not. That single rule kills most of the popularity problem, because it removes the groups that were small enough to be captured.

Nomination rules that survive a friendly list

The fix is not to take the list away from the manager being rated. It is to stop the list being the only input.

Fixed slots, not a free list. The subject proposes peers, capped at half the peer slots. The manager fills the rest from people who worked on the same projects in the last six months. Direct reports are invited by census, not by selection: if the manager has eight or fewer, all of them are invited.

A recency rule. At least half of every group must have worked with the subject in the last six months. This is the cheapest quality control in the whole cycle. A rater who last saw you in 2023 is rating a memory.

Record who nominated whom. One column in the export. If nine of ten raters came from the subject, the report says so on the cover page. Visibility changes nomination behaviour faster than any policy.

One rater the subject did not choose, in every group. The manager adds one peer and one direct report the subject did not name. Not to catch anyone out. To stop the report being a mirror.

Calibrate the raters in twenty minutes

Most 360 cycles spend weeks on item wording and nothing on the people answering them. Rater training is one of the few interventions with consistent support for improving the quality of ratings (Hoffman and Baldwin, 2012, and Woehr, 2008, both cited in the same review).

You do not need a workshop. You need twenty minutes at the start of the rating window:

1. Give every rater three short written incidents from a different team, describing behaviour at the low, middle and high end of one competency.
2. Ask them to place each incident on the rating scale before they see the real form.
3. Show the reference placement and the two most common disagreements.
4. Then open the form.

The effect is that raters stop using their own private scale. Half the spread in a typical 360 report is not disagreement about the person. It is disagreement about what a 4 means.

Report by source group, not one blended score

The blended score is where a decent dataset turns back into a popularity contest. When four groups with different vantage points get averaged into one number, the largest group dominates, and the disagreements that carried the actual information disappear.

Rater source differences are not noise to be averaged away. They reflect real differences in what each group is positioned to see (Lance and colleagues, 2008, as summarised in the same review).

So the report page carries four rows per competency, each with its n:

• Manager, with n
• Peers, with n
• Direct reports, with n
• Self

Never a number without its n beside it. When a reader sees 4.3 from three peers, they read it differently from 4.3 from eight. That is the point. It also has to be a constraint on whatever holds the data, because a spreadsheet will average the groups back into one number the moment somebody sorts it.

The delta between self and others is where the development conversation starts, and it needs items a rater can actually answer from memory. The wording that makes a 360 item answerable covers that part of the design. If your cycle also has to decide who sees which comment and at what threshold, that is a separate decision and it should be made before the window opens, not after the reports are printed.

A worked example, with illustrative numbers

Numbers here are illustrative. The arithmetic is not.

A plant general manager has three supervisors, seven peers and five direct reports. His original nomination list gives him two supervisors, three peers and three direct reports, all of them people he has worked closely with for years.

In the curated cycle, two of the three groups sit below .70, and the peer group is a coin toss with a decimal point. In the fixed cycle, every group clears the line.

The cost of the change is five extra forms. At four minutes each, that is twenty minutes of somebody's afternoon across the whole plant. There is no version of this where the money is the constraint.

What the report then shows: the peer score falls from 4.3 to 3.6, and the fall is concentrated in one behaviour, not spread across the competency. The comments explain why. In the curated cycle, the three peers were the three people he had shipped two successful projects with. The wider peer group had seen him in the cross-functional review where he does not do the same thing at all.

The honest reading is not that the first number was fake. It is that the first number described a narrower population than the report implied.

The honest limit

Selecting your own raters is not automatically corrupting. A study comparing selected and non-selected raters found the selected ones were as accurate, or more accurate, than the ones chosen for them (Nieman-Gonder and colleagues, 2006, cited in the review above). Choosing your raters may even raise acceptance of the feedback. The problem is not that people pick; it is that they pick too few. Fix the number and you can keep most of the choice.

Small teams cannot reach the benchmark. A manager with two direct reports will never get a reliable upward score. Do not fake it with a weighted average. Report that group as narrative only, and say in the report why.

A well-built instrument can still do damage in the debrief. In the largest meta-analysis of feedback interventions, 607 effect sizes across 23,663 observations, the average effect was d = .41, and performance actually declined in one third of the studies analysed (Kluger and DeNisi, 1996, as summarised in the review). Better arithmetic does not fix a badly handled conversation. It only means the conversation is about the right thing.

Administrative use breaks the model. If the 360 feeds a promotion decision, raters know it, and they rate accordingly. No nomination rule survives that. At that point you are running an appraisal with extra steps, and you should say so in the policy rather than pretend otherwise.

And the limit that matters most: this fixes the arithmetic of popularity, not the politics. A manager who is genuinely difficult to work with will still score low. That is not the instrument failing. That is the instrument working, and it is the reason to do the work of sizing the pool properly.

Three changes before your next window opens

Write the minimum into the policy this week: three respondents per group before a score appears, manager-filled slots alongside the subject's list, direct reports invited by census. Then add the twenty-minute calibration and put n beside every number in the report.

None of that is a new platform. It is four lines in a document that already exists, and it is the difference between a report that describes fourteen managers and a report that describes fourteen managers' friends.