Multi-Rater Performance Evaluation and Calibration: Managing Multiple Opinions

Image credit: Unsplash


This paper studies multi-rater performance evaluation and calibration. Specifically, we examine how supervisors use their discretion to weight multi-rater assessments and how calibration committees make decisions to adjust these employee performance evaluations. Using data from an e-commerce company, we document that supervisors use their discretion to weight multi-rater assessments consistent with the aim to improve the informativeness of employee performance evaluations by emphasizing (deemphasizing) ratings that are relatively more (less) informative. However, we find that these weighting efforts can be constrained by a high information load placed on the supervisor and supervisor bias. Furthermore, we document that calibration committees analyze the performance information provided and are less likely to adjust supervisors’ weighting decisions when the supervisor relied more strongly on more informative multi-raters’ assessments and when the supervisor provided more substantiated argumentation for their decision. We also find that the extent to which the calibration committee relies on different information sources depends on job type, and more focus is placed on those employee cases that are more suspect of supervisor bias.

Daniel Schaupp
Daniel Schaupp
Assistant Professor of Accounting

My research interests include transparency, opportunisitc behavior in organizations and performance evaluation and feedback systems.