Evaluation of diagnostic accuracy of the automatic system for the analysis of digital lung X-ray for detection of spherical masses

Rationale: Most data on the effectiveness of systems for the analysis of digital X-ray images have been provided by their developers and require a  high-quality validation in databases prepared independently of the developer.

Aim: To analyze the information content of automatic identification of spherical lung masses with digital X-ray imaging using one of the widely available diagnostic algorithms on publicly unaccessible reference datasets.

Materials and methods: The study was based on the recognition and analysis of digital X-ray images from two publicly inaccessible reference datasets that have the state registration (Russian Federation) with one of the publicly available diagnostic algorithms (FutureMed Analyzer). The study was performed using two models of X-ray screening as examples: Model  1 consisted of 100  X-ray images of the lungs with a  normal: abnormal ratio of 94%: 6%; Model  2 consisted of 5150 chest X-ray images with a normal: abnormal ratio of 97%: 3%.

Results: According to the results of the analysis of the X-ray images with the diagnostic system, 98%  of the images were correctly interpreted with Model 1 and 95% of the images, with Model 2. 83% of the cases from Model  1 and 69% from Model  2% were interpreted as images with lung abnormalities. The percentage of correct answers for differentiation of the chest X-ray images into two categories (normal vs. abnormal) for Model 1 and Model  2 was 95% and 98%, respectively. The sensitivity for detection of abnormal masses ranged from 69% to 83%. The specificity was 99% for the Model 1 chest X-ray images and 96% for the Model  2 chest X-ray images. The underdiagnosis rate was quite low ranging for Model 1 – 17%, and for Model 2  – 31%. The area under the curve for Model 1 was 0.91 and for Model 2 0.85.

Conclusion: The diagnostic efficiency of the automatic image analysis based on the convolutional neuronal networks approaches that of the radiologists. This system of automatic identification of abnormalities was unable to solve the most complex problems of detecting low density spherical masses (like "ground glass" area on computed tomography) and that of shadow summation for abnormalities located in such difficult to interpret zones as lung apices, clavicles, ribs, etc. To select a  suitable system, medical institutions need to conduct preliminary testing in their own models equivalent to the studies performed in a  given institution (parameters for radiography, nature and frequency of abnormalities).