Multi-Rater Performance Evaluation and Calibration: Managing Multiple Opinions

Abstract

This paper studies two contemporary performance evaluation systems, multi-rater performance evaluation and calibration. Specifically, we use a rich dataset of performance evaluations at an e-commerce company to examine how supervisors use their discretion to weight multi-rater assessments and how calibration committees make decisions to adjust these weighting decisions. We document that supervisors use their discretion to weight multi-rater assessments consistent with the aim of improving the informativeness of employee performance evaluations by emphasizing (deemphasizing) ratings that are relatively more (less) informative. However, we find that these weighting efforts can be constrained by a high information load placed on the supervisor. Furthermore, we document that calibration committees analyze the performance information provided and are less likely to adjust supervisors’ weighting decisions when they are more in line with emphasizing more informative multi-raters assessments as well as when the supervisor provided more substantiated argumentation for their decision. In contrast to the constraining role of a high information load on supervisors’ weighting efforts, we find that calibration committees consider the available information more strongly when the information load is high.

Daniel Schaupp

Assistant Professor of Management Accounting

My research interests include contemporary performance management systems, transparency and the impact of digitalization on management accounting.