Comparison of Intra- and Inter-Observer Consistency (Intra-Expert Reliability, Inter-Expert Reliability) in Assessing the Extent of COVID-19 Pneumonia Lesions on Chest Computed Tomography

Background. The SARS-CoV-2 pandemic has set new challenges for the radiological community: early diagnosis of interstitial pneumonia, estimation of its severity, and dynamics. Computed tomography has become the method of choice for assessing lung tissue in COVID-19 patients, which is due to the low sensitivity of radiography in detecting a decrease in airiness by the “ground glass opacity” type. The criteria for assessing visual signs of lung tissue damage often have a degree of subjectivity, and the conclusion based on them affects the patient’s management tactics. Aims — to determine the inter- and intra-expert consistency in the assessment of the percentage of lesions and the CT stage of COVID-19 pneumonia among experts-radiologists with different experience, to analyze the level of precision depending on the prevalence of the lesion and other factors. Materials and methods. The research analyzed CT scans of 221 patients with confirmed SARS-CoV-2 by PCR. Patients with additional lung pathology and some patients with lesions up to 50% were excluded to create a uniform degree sample of 100 patients. Four expert radiologists determined the percentage of lung damage and the CT stages. The results of the expert assessment are analyzed using the methods of classical descriptive statistics and the analysis of intra-and inter-expert consistency. Results. The correlation of the level of lung damage, when evaluating intra-expert convergence (after 6 months), as a percentage between the first and second reading was R = 0.86 (p < 0.05) for expert 1 (high level of training), R = 0.84 (p < 0.05) for expert 2 (high level of training). Within the expert agreement, Kappa (for K-degree) was 0.54 for expert 1 and 0.46 for expert 2, which corresponds to a moderate level of consistency. When assessing inter-expert convergence, the connectivity between the level of lung damage as a percentage between experts 1 and 2 (high level of training) was R = 0.87 (p < 0.05), between experts 3 and 4 (low level of training) —R = 0.78 (p < 0.05). The measure of inter-expert agreement Kappa was 0.51 for experts 1 and 2 and 0.56 for experts 3 and 4. The average assessment of experts in the sample varied up to 4.5%, and when analyzing the differences in expert opinions, the difference varied evenly, both in the degree of increase and decrease in the volume of the lesion. Conclusions. In spite of the high level of correlation in the assessment of the percentage of lesion, the convergence of the Cap on the CT degree was moderate, not significantly differing from the degree of expert training. More often, differences in the level of damage are observed in “non-classical” patterns: “reverse halo”, curvilinear lines, etc. The difference in opinions doesn’t represent a systematic error. Hereby the expert assessment of the volume of lung damage “empirically” has a moderate, insufficient level of reliability.