AI is about to face a major test: Can it differentiate Covid-19 from flu?

It’s long past hackathon time.

With Covid-19 cases surging in parts of the U.S. at the start of flu season, developers of artificial intelligence tools are about to face their biggest test of the pandemic: Can they help doctors differentiate between the two respiratory illnesses, and accurately predict which patients will become severely ill?

Numerous AI models are promising to do exactly that by sifting data on symptoms and analyzing chest X-rays and CT scans. For now, the increased availability of coronavirus testing means AI is unlikely to be relied upon for frontline detection and diagnosis. But it will become increasingly important for figuring out how aggressively to treat patients and which ones are likely to need intensive care beds, ventilators, and other equipment that could become scarce if there’s a Covid-flu “twindemic.”

“That’s on the forefront of everyone’s mind right now,” said Anna Yaffee, an emergency medicine physician at Emory University who helped build an online symptom checker to assess Covid-19 patients. “Although both viruses are similarly managed, they are not the same entity, and patients will need different things.”

In studies, AI models have demonstrated increasing ability to home in on patterns that define Covid-19 and identify patients likely to require supplemental oxygen, stepped-up care in the hospital, and continued monitoring after they are sent home

But those systems were not developed amid flu season, and health informatics experts said it remains to be seen how AI will perform in a more complex clinical environment where large numbers of patients with different illnesses are arriving with similar symptoms. Another question is whether these systems can be effectively scaled up and integrated into the process of caring for patients.

“Imaging-based products are going to have a harder time spreading and being implemented without either integration into the electronic health record or some company licensing the technology out to universities directly,” said Karandeep Singh, a physician at the University of Michigan who researches the use of AI and mobile technology in health care.

He said imaging-analysis AI systems have shown the most promise in identifying the impact of Covid-19 on the lungs, and helping doctors understand the extent of disease progression and how to treat patients. Researchers in Beijing  just published a paper in the journal Nature describing an AI model that could differentiate between Covid-19, influenza, and non-viral pneumonia with a .98 AUC, indicating a high-degree of accuracy in a measure commonly used by machine learning researchers.

Singh said the study was conducted rigorously, but the tool would likely be useful only if Covid-19 combined with a bad flu season to significantly reduce the supply of PCR tests that are most frequently used to diagnose Covid-19. He said the most valuable tools are those that can help predict the course of a patient’s disease and help hospitals allocate resources.

The Food and Drug Administration recently granted an emergency use authorization to a product that uses AI to spot early warning signs of severe illness. Developed by the San Francisco Bay Area company Dascena, the product crunches data in electronic health records to flag high-risk patients likely to require intubation.

That is far from the only product developed to predict deterioration of Covid-19 patients. A tool distributed by the electronic health record vendor Epic is also widely used for that purpose, and many hospitals are building their own products to predict mortality risk and separate patients with mild symptoms from those likely to need careful management even after they leave the hospital.

Meanwhile, developers of automated symptom checkers are also ramping up their efforts to help hospitals navigate the flu season. Boston-based Buoy Health has so far screened about 1 million people for symptoms of Covid-19, helping to reduce demand on hospital emergency departments and direct patients to self-isolate or visit a local testing center.

The challenge now is to comb the data for patterns that could identify clusters of symptoms or certain clinical details that could differentiate Covid-19 from flu and other illnesses. That could help both patients and caregivers make more accurate decisions and preserve medical  resources.

Darin Baumgartel, director of data science and analytics at Buoy, said machine learning is ideally suited for that task. But so far, he said, the company has not been able to get enough data on the outcomes of patients who have used its screening tool. That information would allow it to train a machine learning algorithm based on a ground truth, so that it could automatically correlate certain patterns of symptoms with patients’ test results and the severity of symptoms they experienced.

“A lot of the future potential lies in the ability to label all those individuals as Covid-positive or not,” he said. “Rounding out that data set is the next step in that journey.”