Who should not be on a 360 rater list

Fourteen managers at a Nashik plant went through a 360 in September. One report came back with a 1.8 on collaboration from direct reports, against a plant average near 4.0. HR ran the usual post-mortem: item wording, translation, scale anchors, the debrief script. All clean.

The explanation was in a file nobody had opened. One of the four direct reports had a grievance against that manager, pending for four months, with the plant HR head as the responding officer. The rating was not dishonest. It was an answer to a different question than the one the form asked, and the report averaged it in as if it were the same thing.

Every 360 programme I have seen writes inclusion criteria. Worked together at least six months. At least two interactions a month. Same grade for peers. Almost none writes the other half: who must be kept off the list, on what evidence, and who is allowed to decide.

A rater list is a sampling frame

The report treats every returned form as one observation of the same behaviour, drawn from the same population. That assumption is what makes a mean meaningful and what makes an exclusion necessary. Some forms are not observations of that behaviour at all. They fall into three groups.

Salience. A rater whose contact with the subject in the window is dominated by one event, a dispute, an escalation, a failed audit, is not sampling six months of behaviour. They are sampling the event. The score carries the event's weight, and the comments explain it in detail, which makes it feel more credible than it is.

Bargaining. A rating is cheap to give and hard to trace. Where the rater's own increment, confirmation or grievance outcome sits with the subject, or could sit with them next quarter, the form becomes a move in a negotiation. Anonymity is supposed to remove this. It does not remove it inside a group of three, where the manager can work out who said what from the comments alone.

An empty observation window. A manager who took over the team seven weeks ago cannot rate six months of behaviour. They can rate an impression, and impressions are formed fast. This is not a harshness problem. Van Scotter and colleagues had 104 experienced US Air Force officers rate four colleagues delivering short briefings, and the ratings from officers who knew the person being rated were more accurate than the ratings from officers who did not (Journal of Management Development, 2007). Acquaintance improved accuracy. A new manager is not biased against you. They are simply less accurate, and the report gives their number the same weight as someone who has watched you for two years.

The published standard for pooling ratings assumes raters had an opportunity to observe the performance being rated (Nowack and Mashihi, 2012, reviewing Murphy, 2008). That condition is not a formality. It is the thing that licenses the arithmetic.

Four exclusions worth writing down

The test matters more than the principle, because a principle with no test becomes a judgement call, and a judgement call in a 360 becomes a political act.

Read the fourth row carefully. It is the one most programmes miss, and it is arithmetic rather than ethics. If a manager has three direct reports and one is excluded, two remain. Two respondents in a group is not enough for reliable measurement at all, per the review above, which cites 3D Group's 2009 finding that two or fewer responses per group are inadequate. So in small teams the exclusion and the anonymity threshold collide, and you have to decide which one gives way before the window opens, not after the report prints.

Two more are common enough to name. A rater in a close personal relationship with the subject, including family, is out. So is a rater who has already seen the subject's self-assessment or draft comments.

Who decides, and on what evidence

The subject does not decide, and neither does the subject's manager alone. Both have an interest in the shape of the list, and the exclusion list is the easiest part of a 360 to capture. Give the decision to one named person, usually the HR business partner running the cycle, working from records: the case register, the requisition list, the calendar export, the reporting line.

The rule is that a rater is excluded by an entry in a system, not by a feeling described in an email. If the entry does not exist, the rater stays on the list and the concern goes into the debrief notes instead. This one constraint removes most of the gaming, because inventing a grievance to remove a critic is a much bigger act than complaining about one.

Tell the subject which reason code was applied, not who was removed. That is enough for them to know the process was applied, and not much else.

Excluding without replacing is how a report gets thinner

Here is where good intentions break the arithmetic. Every exclusion reduces n, and n is the only thing standing between a group mean and a coin toss. A programme that removes four raters from a cycle of forty has improved validity and quietly degraded reliability, and the report will not say so.

So exclusion and replacement are one step, not two. For every rater removed, the same group gets a like-for-like replacement from the census: another peer at the same grade with the same exposure test applied, another direct report from the full team list. Replacement is chosen by the manager and the HR partner from the census; the subject sees only the final group size.

The floor does not move. If the group still cannot reach its minimum after replacement, it reports as narrative only, with a line saying why. Three respondents for a score is the floor most Indian organisations can hold.

A worked example, with illustrative numbers

Numbers here are illustrative. The structure is not.

A plant runs 14 managers through a cycle. The nomination sheets produce 46 peer slots and 38 direct report slots. The exclusion audit finds nine problems: two open grievances, one promotion rivalry between two peers, four managers in post under 90 days, and two direct reports in a group of three whose manager signs their appraisal.

Without replacement, one manager's direct report group falls from three to two and disappears from the report, and he loses his entire upward score. The plant head reads that as "no data" rather than as "this team is too small to measure upward", which is a more useful fact.

With replacement, seven of the nine slots are refilled from the census. The two that are not are the grievance cases in teams of three, where there is nobody left to substitute. Those two report as narrative, with the reason stated on the page.

The visible change in the scores is small and worth stating plainly: the manager with the 1.8 moves to 3.6 on direct reports, with four respondents. That is still a low score. The difference is that the number now describes how four people experience working for him rather than how one person's pending case does.

What to do with the excluded rater's data

Do not delete it. An excluded rating is often the most important signal in the cycle, and throwing it away is the second mistake programmes make after averaging it in.

Route it to a confidential channel instead of the score. The HR partner holds a one-page note: who was excluded, the reason code, and the substance of what the rater said, unattributed. The debrief lead reads it before the conversation and uses it to decide which questions to ask. The manager being rated never sees a number and never sees a name.

This is the difference between removing a data point and ignoring a problem. A grievance is a fact about the manager's team that the 360 was never designed to carry. It needs a human process, and usually it already has one.

If your cycle reports comments to the subject, the anonymity decision has to be made before the window opens, and the exclusion list is part of that decision rather than a separate policy.

The honest limit

Exclusions are gameable in both directions. A manager who wants a critic removed can describe a disagreement as a conflict. That is why the test has to be a record in a system and the decision has to sit with someone who does not report to the subject. No rule survives a determined senior person, but a record-based rule at least leaves a trail.

Excluding the complainant can silence a real complaint. This is the serious cost, and it should be said out loud in the policy. Removing an aggrieved direct report from the upward group protects the score and removes that person's only structured channel to say something about their manager. The confidential note is not a substitute for handling the case. If the case is open, the case is the process that matters.

A clean exclusion audit is not a clean cycle. It fixes one class of error. It does not fix a group of three that stays at three, a rater who has never seen the manager run a meeting, or a debrief that turns a 3.6 into an argument about the number. Sizing the pool properly is still the larger lever, and rater calibration still matters more than either.

Small plants will hit the floor constantly. With three or four direct reports, upward scores stay fragile however carefully you exclude. Report them as narrative and say so in the summary.

Three lines to add this week

Take the nomination sheet you already use and add two columns: reason code, and replacement name. Then write one line into the policy naming who decides, and one line setting the floor at three respondents per group after replacement. The evidence lives in systems you already run, so the audit is an hour of work per cycle, not a project.

The test of whether it worked is simple. At the end of the next cycle, every group that appears as a score should have at least three respondents, none of whom is in an open case with the subject, competing for the same role, or less than 90 days into knowing them. If that holds, the numbers in the report are worth arguing about. If it does not, you are arguing about a report that was decided by a list three weeks before anyone answered a question.