Multi-Rater Performance Evaluation and Calibration: Managing Multiple Opinions

Abstract

This study examines a performance evaluation process that incorporates multiple raters and calibration committees—practices increasingly adopted by organizations but underexplored in academic research. Using proprietary data, we investigate how supervisors aggregate multi-rater assessments into initial evaluations and how calibration committees adjust these aggregation decisions. We find that when aggregating multi-rater assessments, supervisors place more weight on more informative assessments, but this holds true only when their cognitive load is not too high. Regarding calibration, we find that committees are less likely to adjust supervisors’ aggregation decisions when those decisions are consistent with more informative multi-rater assessments and when supervisors provide well-substantiated justifications. Moreover, calibration committees place greater emphasis on the inconsistency between supervisors’ decisions and the assessments from more informative multi-raters as well as supervisors’ justifications in employee cases where supervisors face a high compared to low information load. Our results, therefore, suggest that calibration committees strategically focus on information that may not have been fully incorporated into the initial performance ratings, highlighting the value of combining multi-rater systems with calibration.

Daniel Schaupp

Assistant Professor of Strategy and Managerial Accounting

My research interests include performance evaluation, feedback, incentives and the impact of new digital technologies on management control.