An examination of the inter-rater reliability of the diagnostic and statistical manual of mental disorders, third edition (DSM-III)

Abstract

An examination of the inter-rater reliability of DSM-III is reported. Two groups of judges diagnosed a series of 18 clinical case studies utilizing the DSM-III nomenclature. Diagnoses and ratings were made on Axes I, II, IV, and V for each case. One group of judges was composed of 10 clinically experienced psychologists. A second group of diagnostic raters was composed of 10 undergraduate students. Reliability of Axis I and Axis II was assessed by means of the statistic, Kappa, a chance-corrected measure of inter-rater agreement. Reliability estimates are reported for mental disorders at the category, class, and major class levels. The inter-rater reliabilities of Axes IV and V were evaluated utilizing the intraclass correlation coefficient. Results obtained from each of the two groups of diagnosticians were compared. Reliability estimates obtained from the group of psychologists on each of the four axes were found to be higher than the reliabilities obtained from the student group. The findings obtained in the present study utilizing psychologists were also compared with the reliability estimates reported for DSM-III by previous researchers. Overall, it was found that diagnoses of mental disorders made by psychologists showed greater reliability than diagnoses made by the untrained students. However, the reliability estimates obtained in the present study from the psychologist group are lower than the reliabilities reported for earlier studies of DSM-III. The implications of these results are discussed. The finding of higher reliability of diagnoses for the psychologist group relative to the student group is interpreted to indicate that higher diagnostic reliability is associated with greater professional experience. With regard to the differences in findings for the present and previous studies of DSM-III, three possible explanations are discussed. First, these discrepancies may result from differences in professional training between the samples of diagnostic judges used in the present and previous studies. A second possible explanation is that methodological differences between the present and previous studies may account for the discrepant findings. A third reason may be that a combination of professional and methodological differences contributed to the discrepant findings. Recommendations are made for further research to resolve the methodological questions raised by the results, and to more definitively assess the reliability of DSM-III.