Consistency of observations from echograms made centrally in the Collaborative Ocular Melanoma Study. COMS report no. 13

PURPOSE. To describe the methods used by the Collaborative Ocular Melanoma Study (CAMS) Echography Center for grading tumor echograms and to assess reliability of the grading system. METHODS. Tumor echograms were graded at the COMS Echography Center using a specific protocol. To assess consistency, a five-percent random sample of all echographic gradings received as of June 30, 1996 was selected by the COMS Coordinating Center for re-evaluation at the COMS Echography Center. The results were compared with original data. Agreement between the two sets of gradings was evaluated by calculating overall percent agreement, and by using the kappa statistic for categorical features and intra-class correlation coefficients (ICC) for continuous measures. RESULTS. Overall agreement between original gradings and regradings was high. Agreement, based on the kappa statistic, between the original grading and the regrading was classified as 'moderate,' 'substantial,' or 'almost perfect' for nearly every variable graded. Kappas ranged from 0.47 for extrascleral extension to 0.82 for tumor shape. Intra-class correlation coefficients were 0.99 or higher for tumor height measurements and for the clock hour designation of the tumor apex. The only variable graded by the COMS Echography Center that did not have good agreement was 'confidence in tumor measurement.' CONCLUSIONS. The level of agreement (after adjusting for chance agreement) ranged from 'moderate' to 'almost perfect.' Grading for 'confidence of tumor measurement' differed between the original grading and the regrading but there was little difference in the tumor measurements. The COMS Echography Center has demonstrated that its grading protocol is consistent over time.