The Measurement of Agreement in Medical Research: A Permutation Approach

Abstract

Medical researchers have long been strong advocates of permutation statistical methods as permutation statistical methods require neither random sampling nor any distributional assumptions. In addition, permutation statistical methods can often provide exact probability values. A generalized, chance-corrected measure of inter-observer agreement is introduced and illustrated. Example applications include categorical, rank, and interval observations with both two and more than two observers. Exact and Monte Carlo resampling permutation procedures are utilized to obtain both exact and approximate probability values.

Keywords: agreement, chance-corrected, Cohen’s kappa, exact tests, permutation methods, resampling