Comparison of Intra- and Inter-Observer Consistency (Intra-Expert Reliability, Inter-Expert Reliability) in Assessing the Extent of COVID-19 Pneumonia Lesions on Chest Computed Tomography

Cover Page

Cite item

Full Text

Open Access Open Access
Restricted Access Access granted
Restricted Access Subscription or Fee Access


Background. The SARS-CoV-2 pandemic has set new challenges for the radiological community: early diagnosis of interstitial pneumonia, estimation of its severity, and dynamics. Computed tomography has become the method of choice for assessing lung tissue in COVID-19 patients, which is due to the low sensitivity of radiography in detecting a decrease in airiness by the “ground glass opacity” type. The criteria for assessing visual signs of lung tissue damage often have a degree of subjectivity, and the conclusion based on them affects the patient’s management tactics. Aims — to determine the inter- and intra-expert consistency in the assessment of the percentage of lesions and the CT stage of COVID-19 pneumonia among experts-radiologists with different experience, to analyze the level of precision depending on the prevalence of the lesion and other factors. Materials and methods. The research analyzed CT scans of 221 patients with confirmed SARS-CoV-2 by PCR. Patients with additional lung pathology and some patients with lesions up to 50% were excluded to create a uniform degree sample of 100 patients. Four expert radiologists determined the percentage of lung damage and the CT stages. The results of the expert assessment are analyzed using the methods of classical descriptive statistics and the analysis of intra-and inter-expert consistency. Results. The correlation of the level of lung damage, when evaluating intra-expert convergence (after 6 months), as a percentage between the first and second reading was R = 0.86 (p < 0.05) for expert 1 (high level of training), R = 0.84 (p < 0.05) for expert 2 (high level of training). Within the expert agreement, Kappa (for K-degree) was 0.54 for expert 1 and 0.46 for expert 2, which corresponds to a moderate level of consistency. When assessing inter-expert convergence, the connectivity between the level of lung damage as a percentage between experts 1 and 2 (high level of training) was R = 0.87 (p < 0.05), between experts 3 and 4 (low level of training) —R = 0.78 (p < 0.05). The measure of inter-expert agreement Kappa was 0.51 for experts 1 and 2 and 0.56 for experts 3 and 4. The average assessment of experts in the sample varied up to 4.5%, and when analyzing the differences in expert opinions, the difference varied evenly, both in the degree of increase and decrease in the volume of the lesion. Conclusions. In spite of the high level of correlation in the assessment of the percentage of lesion, the convergence of the Cap on the CT degree was moderate, not significantly differing from the degree of expert training. More often, differences in the level of damage are observed in “non-classical” patterns: “reverse halo”, curvilinear lines, etc. The difference in opinions doesn’t represent a systematic error. Hereby the expert assessment of the volume of lung damage “empirically” has a moderate, insufficient level of reliability.

Full Text

Restricted Access

About the authors

Sergey S. Pervushkin

Samara State Medical University

ORCID iD: 0000-0002-7574-283X
SPIN-code: 3089-5082


Russian Federation, 89, Chapaevskaya str., 443099, Samara

Pavel M. Zelter

Samara State Medical University

ORCID iD: 0000-0003-1346-5942
SPIN-code: 3678-3932

MD, PhD, Associate Professor

Russian Federation, 89, Chapaevskaya str., 443099, Samara

Evgeniya K. Kramm

Samara State Medical University

ORCID iD: 0000-0003-3029-8787
SPIN-code: 4826-5241


Russian Federation, 89, Chapaevskaya str., 443099, Samara

Elizaveta A. Sartakova

Samara State Medical University

Author for correspondence.
ORCID iD: 0000-0002-2439-197X
SPIN-code: 6825-5077


Russian Federation, 89, Chapaevskaya str., 443099, Samara


  1. Kim H. Outbreak of novel coronavirus (COVID-19): What is the role of radiologists? Eur Radiol. 2020;30(6):3266–3267. doi:
  2. Fang Y, Zhang H, Xie J, et al. Sensitivity of Chest CT for COVID-19: Comparison to RT-PCR. Radiology. 2020;296(2):E115–E117. doi:
  3. Salehi S, Abedi A, Balakrishnan S, et al. Coronavirus Disease 2019 (COVID-19): A Systematic Review of Imaging Findings in 919 Patients. AJR Am J Roentgenol. 2020;215(1):87–93. doi:
  4. Prokop M, van Everdingen W, van Rees Vellinga T, et al. CO-RADS: A Categorical CT Assessment Scheme for Patients Suspected of Having COVID-19-Definition and Evaluation. Radiology. 2020;296(2):E97–E104. doi:
  5. Salehi S, Abedi A, Balakrishnan S, et al. Coronavirus disease 2019 (COVID-19) imaging reporting and data system (COVID-RADS) and common lexicon: a proposal based on the imaging data of 37 studies. Eur Radiol. 2020;30(9):4930–4942. doi:
  6. Авдеев С.Н., Адамян Л.В., Алексеева Е.И.,и др. Профилактика, диагностика и лечение новой коронавирусной инфекции (COVID-19): Временные методические рекомендации/ Министерство здравоохранения РФ; утверждено зам. министра здравоохранения РФ Е.Г. Камкин. — M., 2020. — 236 с. [AvdeevNS, AdamjanLV, AlekseevaEI, i dr. Profilaktika, diagnostika I lechenie novoj koronavirusnoj infekcii (COVID-19): Vremennye metodicheskie rekomendacii / Ministerstvo zdravoohranenija RF; utverzhdeno zam. Ministra zdravoohranenija RF E.G. Kamkin. Moscow; 2020. 236 s. (In Russ.)]
  7. Hadied MO, Patel PY, Cormier P, et al. Interobserver and Intraobserver Variability in the CT Assessment of COVID-19 Based on RSNA Consensus Classification Categories. Acad Radiol. 2020;27(11):1499–1506. doi:
  8. De Jaegere TMH, Krdzalic J, Fasen BACM, et al. Radiological Society of North America Chest CT Classification System for Reporting COVID-19 Pneumonia: Interobserver Variability and Correlation with Reverse-Transcription Polymerase Chain Reaction. Radiol Cardiothorac Imaging. 2020;2(3):e200213. doi:
  9. Sushentsev N, Bura V, Kotnik M, et al. A head-to-head comparison of the intra- and interobserver agreement of COVID-RADS and CO-RADS grading systems in a population with high estimated prevalence of COVID-19. BJR Open. 2020;2(1):20200053. doi:
  10. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977:159–174.
  11. Harmon SA, Sanford TH, Xu S, et al. Artificial intelligence for the detection of COVID-19 pneumonia on chest CT using multinational datasets. Nat Commun. 2020;11(1):4080. doi:
  12. Adly AS, Adly AS, Adly MS. Approaches Based on Artificial Intelligence and the Internet of Intelligent Things to Prevent the Spread of COVID-19: Scoping Review. J Med Internet Res. 2020;22(8):e19104. doi:
  13. Zhao W, Zhong Z, Xie X, et al. Relation between Chest CT Findings and Clinical Conditions of Coronavirus Disease (COVID-19) Pneumonia: A Multicenter Study. AJR Am J Roentgenol. 2020;214(5):1072–1077. doi:
  14. Kumar V, Abbas AK, Fausto N, etal. Robbins and Cotran Pathologic Basis of Disease (Robbins Pathology). Saunders. ISBN:B005WV2Q86.
  15. Зайратьянц О.В., Cамсонова М.В., Михалева Л.М., и др. Патологическая анатомия COVID-19: Атлас / под общ. ред. О.В. Зайратьянца. — М.: НИИОЗММ, 2020. — 140 с. [Zajrat’yanc OV, Camsonova MV, Mihaleva LM, i dr. Patologicheskaya anatomiya COVID-19: Atlas / pod obshch. red. O.V. Zajrat’yanca. Moscow: NIIOZMM; 2020. 140 s. (In Russ.)]
  16. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276–282.
  17. Порядок госпитализации в медицинские организации пациентов с установленным диагнозом новой коронавирусной инфекции COVID-19 в зависимости от степени тяжести заболевания: Приложение № 12 к приказу Минздрава России от 19.03.2020 № 198н. Available from:
  18. Эволюция пандемии COVID-19 / под ред. Н.А. Белякова, С.Ф. Багненко. — СПб.: Балтийский медицинский образовательный центр, 2021. — 411 с. [Evolyuciya pandemii COVID-19 / pod red. N.A. Belyakova, S.F. Bagnenko. Saint Petersburg: Baltijskij medicinskij obrazovatel’nyj centr; 2021. 411 s. (In Russ.)]
  19. Морозов С.П., Чернина В.Ю., Блохин И.А., и др. Прогнозирование исходов при лабораторно верифицированном COVID-19 по данным компьютерной томографии органов грудной клетки: ретроспективный анализ 38 051 пациента // Digital Diagnostics. — 2020. — Т. 1. — № 1. — C. 27–36. [Morozov SP, Chernina VYu, Blokhin IA, Gombolevskiy VA. Chest computed tomography for outcome prediction in laboratory-confirmed COVID-19: Aretrospective analysis of 38,051 cases. Digital Diagnostics. 2020;1(1):27−36. (In Russ.)] doi:

Supplementary files

Supplementary Files
1. Fig. 1. Research scheme

Download (296KB)
2. Fig. 2. Diagram of the values of the percentage of defeat of experts 1 (A) and 2 (B)

Download (165KB)
3. Fig. 3. Frequency histograms inside the expert difference of experts 1 (A) and 2 (B), %

Download (111KB)
4. Fig. 4. Diagram of the values of the percentage of defeat between experts: A — 1 and 2 (high level of training); B — 3 and 4 (low level of training)

Download (167KB)
5. Fig. 5. Frequency histograms of the interexpert difference in percentages: A — high level of training; B — low level of training

Download (110KB)
6. Fig. 6. Diagram of the values of the average percentage of defeat between experts of different levels of training

Download (99KB)
7. Fig. 7. Computed tomography of the chest organs of patient T.

Download (303KB)
8. Fig. 8. Computed tomography of the chest organs of patient B.

Download (278KB)

Copyright (c) 2022 "Paediatrician" Publishers LLC

This website uses cookies

You consent to our cookies if you continue to use our website.

About Cookies