AI-Driven COVID-19 Tools to Interpret, Quantify Lung Images
Qualitative interpretation is a good thing when it comes to reading lung images in the fight against coronavirus 2019 disease (COVID-19), but quantitative analysis makes radiology reporting much more comprehensive. To that end, several research groups have begun looking to artificial intelligence (AI) as a tool for reading and analyzing X-rays and computed tomography (CT) scans, and helping to diagnose and monitor COVID-19.
The interest in lung imaging arose when clinicians and radiologists in Wuhan, Hubei Province of China, began noting shared abnormalities in lung CT images among patients with COVID-19 . As the contagion spread, radiologists at health care facilities around the world saw the same thing in their chest CT scans, as well as X-rays, and started using images to help determine whether patients were infected and the extent of lung involvement, and to track patient progress during the course of their treatments. As that practice spread, however, some health organizations noted concerns about whether image interpretation could give a clear diagnosis, partially because lung images from COVID-19 and other respiratory infections are very difficult to discern through a qualitative analysis .
This is where AI can contribute, according to Dirk Smeets, Ph.D. (Figure 1), chief technology officer at icometrix of Leuven, Belgium, which is developing AI for scrutinizing CT scans. While radiologists can see indicative patterns in lung images, a determination of the amount of lung that is affected requires quantification at a level that is impossible by just looking at a CT scan, he said. Yet, that type of quantitative and objective detail is very useful for COVID-19, he said, pointing to research showing that negative outcome prediction improved when clinicians had precise measurements of the amount of lung tissue that was affected .
AI can also play a role in spotting and quantifying even the slightest nuances in abnormalities that appear in images, said Preetham Sreenivas (Figure 2), AI scientist and one of the founding members of Qure.ai, Mumbai, India. The company is focusing its efforts on X-ray analysis for COVID-19, and is finding that AI-driven approaches can be used to differentiate COVID-19 and other respiratory illnesses.
The idea with all of this work is not to replace, but to augment the radiologist or frontline clinician, so that COVID-19 can be detected and monitored more quickly and objectively, remarked Maryellen L. Giger (Figure 3), Ph.D., A.N. Pritzker Professor of Radiology at the University of Chicago (UChicago). Her group is leading a joint COVID-19 effort with the research center Argonne National Laboratory to develop AI for analyzing both CT and X-ray images. This capacity, she said, will be particularly important if transmission of the COVID-19 virus spikes again in the coming fall and winter months, or if the coronavirus turns out to be a year-round pathogen.
AI for both
“When COVID happened, we were already analyzing lung images—both chest X-ray and CT—so it was logical for us to apply our knowledge and expertise to the COVID-19 problem, and that’s what we did,” Giger said.
In previous work, her group developed computer-vision and deep-learning approaches to find and point out features in lung images that are associated with any of a range of respiratory-related diseases, such as emphysema and pneumonia. As part of that research, the group developed a computational method called cascade-based, deep-transfer learning that trains the AI to analyze X-rays and CT scans, and pick up pertinent patterns connected to different diseases.
For the new UChicago-Argonne project, the researchers plan to retrain the AI on features of COVID-19. They have already begun collecting a database of COVID-19 images, and have performed additional preliminary work to train the AI on images from other diseases so it can single out distinctions associated with COVID-19 while also ruling out other respiratory diseases, Giger said.
The UChicago/Argonne project is currently funded by a 12-month grant from the c3.ai Digital Transformation Institute , and by the time the grant is up next spring, Giger hopes it will have churned out a clinical system that has at least been tested on an independent dataset.
She is also seeking to extend that funding, and to apply for additional grants to keep the work moving forward “because this isn’t going to go away in a year,” she said. Ultimately, she envisions the project as having two primary benefits: detecting COVID-19 in any patient coming in for standard lung imaging, which will be useful in preventing unknowing transmission, and monitoring the progress of COVID-19 in already-diagnosed patients as a way to manage treatments for the optimal effects.
Deploying X-ray AI
Qure.ai has been developing deep-learning technology and AI for medical imaging since the company’s inception in 2016, and one of its products is qXR. Trained on a dataset of 2.5 million chest X-rays, the AI in qXR automatically perceives 20 lung abnormalities, and uses those findings to identify probable cases of tuberculosis  and other respiratory illnesses, Sreenivas described. Once the COVID-19 crisis began, the company immediately thought about expanding qXR to also screen for findings that are strongly suggestive of that disease.
Although a large dataset of COVID-19 X-rays was not yet available to train qXR when Qure.ai began this work, it did have access to reports from the front lines about X-ray abnormalities that strongly indicated a diagnosis of COVID-19 in patients, Sreenivas said. These included the localization of lesions lower in the lungs, as well as hazy areas known as ground-glass opacities (GGOs) and consolidations (the amount of the lung that is filled with fluid instead of air) that were peripheral and bilateral in nature. At the same time, radiologists were also noting other abnormalities, including the presence of certain cavities, nodules, and pleural effusions (water outside the lungs), that excluded a COVID-19 diagnosis.
Because the earlier development of qXR included the ability to find and quantify all of those abnormalities, Qure.ai scientists were able to quickly build a downstream application that could generate a concise and instantaneous report about COVID-19 risk, Sreenivas explained. The risk report is displayed right on the X-ray (Figure 4), he said, “so radiologists and clinicians who are using this don’t have to deviate from their regular workflow.” Tests showed that it had sensitivity and specificity (the ability to identify correctly those with the disease and those without) of 91% and 78%, respectively, for COVID-19 , he said.
As of June 1, qXR had been adopted by more than 60 clinical sites, including San Raffaele Hospital in Milan, Italy, which was an early epicenter for COVID-19. In hospitals and other health settings, Sreenivas described qXR as providing value by speeding X-ray interpretation, a benefit that is especially helpful for triage purposes in very busy hospitals, and also for monitoring patients who are undergoing treatment. For the latter, he remarked, “The typical way to assess whether a patient is getting better is to take a chest scan and see if the disease is improving. qXR provides an objective mechanism, so even if the (patient’s lung involvement) is only decreasing by 5 percent, that will show up. That is a significant and important use” (Figure 5).
In addition, Qure.ai has developed and commercialized qScout , an app-based platform it had been designing to help track tuberculosis, but has now been expanded to include COVID-19. “This is something that was already in the works, but we accelerated the development once COVID hit,” Sreenivas said. qScout includes an electronic medical record (EMR) and an AI-based tracking component that maintains secure patient information about initial and continuing individual health status, so clinicians can provide personalized care; and tracks more general epidemiological and geographical data, which can help health organizations do contact tracing and begin appropriate mitigation efforts, he said, adding that the ministry of Oman is already deploying qScout as part of its nationwide COVID-19 contact-monitoring work. He anticipates the company will do a larger launch if the disease doesn’t subside over the summer.
AI for CT scans
CT scans are the focus of icometrix, which has developed AI algorithms, called icolung (Figure 6), to identify COVID-19 and monitor patient progression. When it embarked on icolung, the company’s core technology was brain-image processing, so it was already well-versed in network models, deep-learning, and AI technologies. It did not, however, have experience with lung scans, so it got in touch with those who did. “We basically reached out to all the hospitals that we were already in contact with for our daily business and asked if they would help us by providing lung images,” said Dirk Smeets, chief technology officer at icometrix. The company also sought and received help from nearly three dozen radiologists from around the world who provided their expert evaluations, or delineations, of image features. This assistance, along with an AI COVID-19-detecting framework developed by the University of Wuhan  and collaborative efforts with Belgian academic centers in Leuven, Antwerp, and Brussels, helped icometrix initialize its deep-learning network, and train its icolung algorithm, he said.
Within a matter of weeks, icometrix had enough data to validate its first algorithm, and applied for and received the European Conformity (CE) marking in April. “That was a big milestone because it allowed us to roll it out into clinical practice,” Smeets said. “We now offer it to medical centers pro bono and typically ask if we can use the (resulting) data to improve the algorithms, because the accuracy of deep-learning algorithms depends on whether they receive enough variability of data.” So far, he said, 70 centers have provided data to icometrix, which has used it to continue improving icolung.
Since the CE marking, icometrix has also received temporary permission of the U.S. Food and Drug Administration (FDA) to use icolung clinically for quantification of lung pathology on chest CT scans in admitted COVID-19 patients, and the company is seeking an FDA extension for the technology.
In the meantime, icolung’s focus has shifted from assessing the severity of lung involvement in symptomatic patients, which is what hospitals needed at the height of the pandemic, and toward detection of infection in asymptomatic individuals, which will require a different network approach, Smeets said. It is also developing icolung’s ability to distinguish COVID-19 from other lung diseases. For both of these objectives, icometrix will again be involving radiologists not only to ensure that icolung is providing accurate interpretations, but also to build radiologists’ confidence that the AI is indeed keying on the correct features, he said. “It can be very difficult to really understand why deep-learning networks are making decisions, so this (collaboration) is important so radiologists will trust the algorithm.”
Developing AI to quantify chest images makes good sense, agreed all three researchers. “Humans are good at detecting certain patterns, but they have more difficulty assessing specific amounts unless they are relative,” Smeets said. “As an example, it’s not easy to tell how tall a person is or what his weight is if he is standing alone, but if you can compare him with another person, you can say if he’s larger or smaller. The same is true with ground glass opacity: it’s very difficult to say how much there is in a lung image, yet these quantitative measurements are of key importance.”
This quantification is especially useful in settings with limited diagnostic resources, such as rural or economically disadvantaged parts of the world, said Sreenivas. There, AI offers clear and actionable information about the amount of lung involvement, and in the case of qXR, provides an instant risk assessment that appears directly on the X-ray. This would be particularly useful for monitoring progression of the infection, evaluating how well the patient is responding to treatment, and determining if treatment changes might be necessary. Smeets concurred. “AI solutions would help to bring the average radiologist more to the level of the expert radiologist.”
Looking more broadly, Giger believes these types of AI-based approaches for COVID-19 could be expanded to pick up all sorts of respiratory illnesses, even new coronaviruses that may arise in the future. In other words, AI could help turn a standard CT scan or X-ray into a versatile tool to assist with speedy diagnosis, which would be particularly useful capability for spotting easily spread infections. “If you don’t catch an infection, and the person goes out and causes more people to get it, that’s not good, so why not look for COVID-19 or other diseases when doing any chest radiograph or low-dose CT for lung screening?” She added, “It is a logical step.”
- L. Xiaoming et al., “CT imaging changes of corona virus disease 2019 (COVID-19): A multi-center study in Southwest China,” J. Transl. Med., vol. 18, no. 1, p. 154, Apr. 6, 2020.
- American College of Radiology, “ACR recommendations for the use of chest radiography and computed tomography (CT) for suspected COVID-19 infection,” Mar. 11, 2020. Accessed: May 31, 2020. [Online]. Available: https://www.acr.org/Advocacy-and-Economics/ACR-Position-Statements/Recommendations-for-Chest-Radiography-and-CT-for-Suspected-COVID19-Infection
- D. Colombi et al., “Well-aerated lung on admitting chest CT to predict adverse outcome in COVID-19 pneumonia,” Radiology, Apr. 17, 2020. Accessed: May 27, 2020. [Online]. Available: https://pubs.rsna.org/doi/pdf/10.1148/radiol.2020201433
- R. Mitchum, University of Chicago, “UChicago joins new academic/industry consortium to accelerate AI innovation,” press release, Mar. 26, 2020. Accessed: May 29, 2020. [Online]. Available: https://news.uchicago.edu/story/uchicago-joins-new-academicindustry-consortium-accelerate-ai-innovation
- E. Engle, A. Gabrielian, A. Long, D. E. Hurt, and A. Rosenthal, “Performance of Qure.ai automatic classifiers against a large annotated database of patients with diverse forms of tuberculosis,” PLoS One, vol. 15, no. 1, e0224445, Jan. 24, 2020.
- T. Raj, B. Reddy, M. TLD, P. Sreenivas, and P. Rao, “Re-purposing qXR for COVID-19,” Qure.ai Blog, Apr. 10, 2020. Accessed: Jun. 11, 2020. [Online]. Available: http://blog.qure.ai/notes/chest-xray-AI-qxr-for-covid-19
- Qure.ai, “qScout,” video. Accessed: Jun. 5, 2020. [Online]. Available: http://qure.ai/covid.html
- L. Lin et al., “Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT,” Radiology, Mar. 19, 2020. Accessed: Jun. 5, 2020. [Online]. Available: https://pubs.rsna.org/doi/10.1148/radiol.2020200905