Rater fatigue is why your 360 completion rate collapses

The cycle closed at 47 per cent. Sixty three invitations, twenty nine submissions, and four participants whose peer group came back with two raters each. The HR team's post mortem said the usual thing: people are busy, engagement is low, we will send more reminders next time.

None of that was the problem. The form asked every rater seventy four questions. Several of the raters had been nominated by four different colleagues, so the same person was being asked to answer three hundred questions about other people in a three week window. They did not stop caring. They ran out of attention, and the tail of the list is where attention goes first.

The load you measured was per participant, not per rater

Most 360 reviews are scoped in the wrong unit. The plan says eight raters per participant, sixty items, twenty four participants. That reads as reasonable. Nobody multiplies it out.

The number that predicts your completion rate is ratings per rater:

ratings per rater = (participants × raters per participant × items) ÷ size of the rater pool

In the example above: 24 × 8 × 74 ÷ 30 = 474 ratings per person. A rating takes about five seconds when it is easy and thirty when the rater has to think. Call it fifteen seconds. That is close to two hours of unpaid form filling, asked of thirty people who still have their day jobs.

The same arithmetic explains why a 360 at a 600 person plant can close at 85 per cent while a 360 at a 40 person startup closes at 40 per cent. The startup's raters are the same eight people for everybody. The pool is small, so the load per person is large. Nothing about their culture is worse. The denominator is smaller.

Why adding items buys almost nothing

A 360 does not measure a person. It measures how a set of people see a person. The second part carries most of the error, and no amount of extra questions fixes it.

The numbers here are old but they have not been overturned. In Conway and Huffcutt's meta-analysis of 360 ratings, the average correlation between two supervisors rating the same person was 0.50, between two peers 0.37, and between two subordinates 0.30. Two peers looking at the same colleague agree at about the level of two people guessing at the same partially visible object.

Run those correlations through the Spearman-Brown formula, which tells you the reliability of an average of several raters:

Read the peer column. Two peers give you .54. Six give you .78. Going from six to twelve raters adds six points. That curve is the whole argument: reliability is bought with raters, and it gets cheap in the middle of the table.

Now the item side. Adding questions raises the internal consistency of one rater's score, which sounds like the same thing and is not. A rater who is systematically generous, or who only ever sees your best work, stays systematically generous across ninety items. You have measured their perspective more precisely. You have not added a second perspective, which is what the correlation of .37 says you need.

What item count actually costs

Item count does have a measurable price, just a smaller one than people assume. The SOM Institute at the University of Gothenburg ran a clean experiment in 2020 on a random sample of Gothenburg residents: a 9 page questionnaire with 149 questions against a 12 page version with 205 questions. Response fell from 48.5 per cent to 45.8 per cent, a drop of 2.7 percentage points. The longer form lost about one respondent in twenty.

That is a real effect and it is smaller than the folk wisdom. Which is the useful finding. Length alone does not collapse a survey. Length multiplied by repetition does. The Gothenburg sample answered one questionnaire once. Your raters answer six questionnaires, each of them long, in the same fortnight, and the sixth one lands after they have already formed an opinion about your process.

There is a second cost that does not appear in response rates. The Canadian medical education literature on multisource feedback is blunt about it: items a rater cannot observe push them into guessing, and ratees who receive feedback on things they know their raters never saw stop trusting the report. A shopfloor supervisor asked to rate a colleague on "strategic thinking" is not measuring strategy. They are pattern matching from a meeting they half remember. You paid a question for noise.

Do the arithmetic before the window opens

Three numbers, ten minutes, a spreadsheet. Do this before you send anything.

1. Ratings per rater. The formula above. Under 150 and you will close near your invitation rate. Over 300 and you are choosing attrition.
2. Items the median rater can actually observe. Go through the instrument and mark each item: seen it happen, or heard about it. Anything in the second pile is a guessing item. In most first draft instruments this is 20 to 35 per cent of the bank.
3. Raters available per group. If your platform reports by group, a group with two responders is not a small sample, it is an anecdote. Recent work cited in the practitioner literature is clear that two or fewer respondents per group is inadequate for reliable measurement, and the practical recommendation from the medical MSF reviews is 5 to 11 assessors from one observer category, 10 to 20 from another, and far more where the observers are patients.

If the first number is high and the second is low, you have your diagnosis without looking at a single comment.

The method, in the order the work happens

Step 1: fix the pool before you fix the form. Count the people who are eligible to rate anyone in the cohort. That number, not the participant count, sets your ceiling. If the pool is 30 and the cohort is 24, you cannot ask for eight raters each without asking each person to rate six colleagues. Widen the pool first: add a second layer of peers, include one skip-level, include internal customers. The pool is the cheapest variable you have.

Step 2: cut the bank to observed behaviour. Target 15 to 25 items, and make each one a thing a person could have watched happen: "responds to a customer complaint within the same shift", not "is customer centric". Item reduction is not dumbing down. It is the difference between a rater answering from memory and a rater answering from inference.

Step 3: spread the load with round-robin nomination. Instead of every participant nominating eight people from the same clique, assign raters so that each person rates a maximum of three colleagues. In a 24 person cohort with a 30 person pool, three ratings each is the cap. This one rule usually does more for completion than any reminder sequence.

Step 4: set the anonymity floor before invitations go out. State the minimum in the invitation: three responses per group before the group is reported. Then merge groups that cannot reach it, rather than reporting a single peer score built from two people. If you want the mechanics of that trade-off, the case for thresholds and attributed groups works through it.

Step 5: two reminders, placed by day not by guilt. The meta-analytic evidence on reminders is unusually strong. Across 38 studies and 246,040 invitations, a single reminder raised response from 49.5 to 65.6 per cent, sixteen percentage points, with almost no cost in retention. What it does not support is a daily drip. Send one reminder on day four, one on day nine, and stop. The third and fourth reminders recover very little and cost you goodwill you will need next cycle.

Step 6: report by group, with n printed next to every mean. A peer score of 3.6 built on two raters and a manager score of 3.6 built on one are not the same claim. Printing n is the cheapest honesty available, and it also quietly teaches the organisation why the next cycle needs a wider pool.

A worked example, with illustrative numbers

A 24 person cohort, a 30 person rater pool, two designs. The completion figures are illustrative, built to sit in the range these designs typically produce, not measured.

Design B asks for less than half the total work, 5,184 item-ratings against 11,520, and produces a report you can defend. Design A collected more raw answers, 4,838 against 3,525, and still landed at a reliability of .59 because those answers came from 3.4 people. That is the trap: the longer form looks like it gathered more evidence. It gathered more answers from the same narrow set of observers.

The honest limit

This advice holds when the constraint is rater attention. It holds less well in three situations.

Small teams. A team of six cannot give you eight raters without asking people to rate their own manager, which corrupts the data more than a low n does. In that case, keep the group size honest, report n of four, and use the numbers for conversation rather than comparison. The reliability table does not rescue you.

Highly technical behaviours. Cutting the bank to observed behaviour works when the behaviour is visible. For a specialist role where the work happens inside someone's head or a private screen, most items fail the observation test, and shortening the form removes the only content you had. There, use fewer raters and longer interviews, and do not pretend the questionnaire is the assessment.

The independence assumption. Spearman-Brown assumes raters err independently. In a real team they do not: peers who sit together share the same view of the same project. Correlated error means the real gain from the ninth rater is smaller than the table shows. Treat the table as an upper bound, which is still enough to make the case for raters over items.

Three changes before your next window opens

Open last cycle's configuration and compute ratings per rater. Write the number down. If it is over 300, cut the item bank to twenty observable behaviours and cap each rater at three colleagues before you invite anyone. Then move the reminder to day four and delete the rest of the sequence. Three changes, one afternoon, and you will know by the close whether the diagnosis was rater fatigue or something else. If completion stays flat with a load of 150, you have a real engagement problem and a clean baseline to prove it. The reporting side, per-group scores with n shown and a threshold enforced, is the part the 360 module in the platform was built around, and if you are rewriting the questions themselves, the six items that surface a blind spot is the place to start.