Multi-Rater Performance Evaluation and Calibration: Managing Multiple Opinions

Image credit: Unsplash

Abstract

Many organizations rely on performance evaluation processes where employees are assessed by multiple raters and final decision rights about employees’ ratings are allocated to calibration committees. Using proprietary data from an organization that employs multi-rater evaluation and calibration, we investigate how supervisors aggregate multi-rater assessments and how committees calibrate the resulting evaluations. We find that when aggregating multi-rater assessments, supervisors place more weight on more informative assessments – but this holds true only when their cognitive load is not too high. For calibration, we find that committees are less likely to adjust supervisors’ aggregation decisions (i) if they are consistent with more informative multi-rater assessments and (ii) when supervisors’ justifications of their decisions are well substantiated. Notably, we find that cognitive load affects calibration committees differently than supervisors. That is, we find that calibration committees appear to focus their attention specifically on information that might not have been (completely) incorporated.

Daniel Schaupp
Daniel Schaupp
Assistant Professor of Management Accounting

My research interests include contemporary performance management systems, transparency and the impact of digitalization on management accounting.

Related