Evaluation of diagnostic accuracy of the automatic system for the analysis of digital lung X-ray for detection of spherical masses

Cover Page

Cite item

Abstract

Rationale: Most data on the effectiveness of systems for the analysis of digital X-ray images have been provided by their developers and require a  high-quality validation in databases prepared independently of the developer.

Aim: To analyze the information content of automatic identification of spherical lung masses with digital X-ray imaging using one of the widely available diagnostic algorithms on publicly unaccessible reference datasets.

Materials and methods: The study was based on the recognition and analysis of digital X-ray images from two publicly inaccessible reference datasets that have the state registration (Russian Federation) with one of the publicly available diagnostic algorithms (FutureMed Analyzer). The study was performed using two models of X-ray screening as examples: Model  1 consisted of 100  X-ray images of the lungs with a  normal: abnormal ratio of 94%: 6%; Model  2 consisted of 5150 chest X-ray images with a normal: abnormal ratio of 97%: 3%.

Results: According to the results of the analysis of the X-ray images with the diagnostic system, 98%  of the images were correctly interpreted with Model 1 and 95% of the images, with Model 2. 83% of the cases from Model  1 and 69% from Model  2% were interpreted as images with lung abnormalities. The percentage of correct answers for differentiation of the chest X-ray images into two categories (normal vs. abnormal) for Model 1 and Model  2 was 95% and 98%, respectively. The sensitivity for detection of abnormal masses ranged from 69% to 83%. The specificity was 99% for the Model 1 chest X-ray images and 96% for the Model  2 chest X-ray images. The underdiagnosis rate was quite low ranging for Model 1 – 17%, and for Model 2  – 31%. The area under the curve for Model 1 was 0.91 and for Model 2 0.85.

Conclusion: The diagnostic efficiency of the automatic image analysis based on the convolutional neuronal networks approaches that of the radiologists. This system of automatic identification of abnormalities was unable to solve the most complex problems of detecting low density spherical masses (like "ground glass" area on computed tomography) and that of shadow summation for abnormalities located in such difficult to interpret zones as lung apices, clavicles, ribs, etc. To select a  suitable system, medical institutions need to conduct preliminary testing in their own models equivalent to the studies performed in a  given institution (parameters for radiography, nature and frequency of abnormalities).

About the authors

P. V. Gavrilov

Saint Petersburg Research Institute for Phthisiopulmonology

Author for correspondence.
Email: spbniifrentgen@mail.ru
ORCID iD: 0000-0003-3251-4084

Pavel V. Gavrilov – MD, PhD, Leading Research Fellow, Head of Radiology Area

2–4 Ligovskiy prospekt, Saint Petersburg, 191036

Russian Federation

U. A. Smolnikova

Saint Petersburg Research Institute for Phthisiopulmonology

Email: ulamonika@mail.ru
ORCID iD: 0000-0001-9568-3577

Uliana A. Smolnikova – Postgraduate Student, Department of Radiology 

2–4 Ligovskiy prospekt, Saint Petersburg, 191036

Russian Federation

References

  1. Тюрин ИЕ. Лучевая диагностика в Российской Федерации в 2016 г. Вестник рентгенологии и радиологии. 2017;98(4):219–226. doi: 10.20862/0042-4676-2017-98-4-219-226.
  2. Трофимова ТН, Козлова ОВ. Лучевая диагностика 2018 в цифрах и фактах. Лучевая диагностика и терапия. 2019;(3):100–102. doi: 10.22328/2079-5343-2019-10-3-100-102.
  3. Yerushalmy J, Harkness JT, Cope JH, Kennedy BR. The role of dual reading in mass radiography. American Review of Tuberculosis. 1950;61:443–464.
  4. Nakamura K, Ohmi A, Kurihara T, Suzuki S, Tadera M. [Studies on the diagnostic value of 70 mm radiophotograms by mirror camera and the reading ability of physicians]. Kekkaku. 1970;45(4):121–128. Japanese.
  5. Гаврилов ПВ, Ушков АД, Смольникова УА. Выявление округлых образований в легких при цифровой рентгенографии: роль опыта работы врача-рентгенолога. Медицинский альянс. 2019;(2):51–56.
  6. Lakhani P, Sundaram B. Deep Learning at Chest Radiography: Automated Classification of Pulmonary Tuberculosis by Using Convolutional Neural Networks. Radiology. 2017;284(2):574–582. doi: 10.1148/radiol.2017162326.
  7. Jaeger S, Karargyris A, Candemir S, Folio L, Siegelman J, Callaghan F, Zhiyun Xue, Palaniappan K, Singh RK, Antani S, Thoma G, Yi-Xiang Wang, Pu-Xuan Lu, McDonald CJ. Automatic tuberculosis screening using chest radiographs. IEEE Trans Med Imaging. 2014;33(2):233–245. doi: 10.1109/TMI.2013.2284099.
  8. Морозов СП, Владзимирский АВ, Ледихова НВ, Соколина ИА, Кульберг НС, Гомболевский ВА. Оценка диагностической точности системы скрининга туберкулеза легких на основе искусственного интеллекта. Туберкулез и болезни легких. 2018;96(8):42–49. doi: 10.21292/2075-1230-2018-96-8-42-49.
  9. Падалко МА, Наумов АМ, Назариков СИ, Лушников АА. Применение технологий искусственного интеллекта для диагностики туберкулеза и онкологических заболеваний. Туберкулез и болезни легких. 2019;97(11):62. doi: 10.21292/2075-1230-2019-97-11-62-62.
  10. Морозов СП, Владзимирский АВ, Кляшторный ВГ, Андрейченко АЕ, Кульберг НС, Гомболевский ВА, Сергунова КА. Клинические испытания программного обеспечения на основе интеллектуальных технологий (лучевая диагностика): методические рекомендации. Серия «Лучшие практики лучевой и инструментальной диагностики». Вып. 57. М.; 2019. 53 с.
  11. Васильев АЮ, Малый АЮ, Серова НС. Анализ данных лучевых методов исследования на основе принципов доказательной медицины: учебное пособие. М.: ГЭОТАР-Медиа; 2008. 32 с.
  12. Fawcett T. An introduction to ROC analysis. Pattern recognition letters. 2006;27(8):861–874. doi: 10.1016/j.patrec.2005.10.010.
  13. Macskassy SA, Provost F, Rosset S. ROC confidence bands: An empirical evaluation. In: ICML '05: Proceedings of the 22nd international conference on Machine learning. 2005. p. 537–544. doi: 10.1145/1102351.1102419.
  14. Nam JG, Park S, Hwang EJ, Lee JH, Jin KN, Lim KY, Vu TH, Sohn JH, Hwang S, Goo JM, Park CM. Development and Validation of Deep Learning-based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest Radiographs. Radiology. 2019;290(1): 218–228. doi: 10.1148/radiol.2018180237.
  15. Sim Y, Chung MJ, Kotter E, Yune S, Kim M, Do S, Han K, Kim H, Yang S, Lee DJ, Choi BW. Deep Convolutional Neural Network-based Software Improves Radiologist Detection of Malignant Lung Nodules on Chest Radiographs. Radiology. 2020;294(1):199–209. doi: 10.1148/radiol.2019182465.

Supplementary files

There are no supplementary files to display.


Copyright (c) 2021 Gavrilov P.V., Smolnikova U.A.

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

This website uses cookies

You consent to our cookies if you continue to use our website.

About Cookies