Researchers at Microsoft have developed an AI tool that diagnoses patients sequentially, mimicking the approach of human doctors. According to a study published in Science, this tool was more than twice as accurate as general physicians in solving complex medical cases. The study involved testing the AI tool on over 300 unusual and confusing medical dilemmas. The AI tool's performance was also evaluated in scenarios drawn from a major database of emergency room admissions.
The AI tool uses an "orchestrator" that coordinates the steps to reach a diagnosis by employing a simulated panel of clinical experts. This approach differs from existing medical AI models, which typically examine all patient symptoms, lab results, and imaging studies at once to generate a diagnosis. The orchestrator enables the AI tool to identify and prioritize questions to ask about a patient's history and physical exam, and then decide which tests to order.
The AI tool's sequential approach involves updating its working hypotheses and plans as it gathers more information. It first decides what to ask about the patient's history and what is significant in the physical exam. Based on what it learns, it determines which tests to order, considers the results, and orders further testing as needed. This approach allows the AI tool to mirror the diagnostic process of human doctors.
In the study, the AI tool outperformed human physicians in diagnosing complex medical cases. General physicians using textbooks and the internet were compared to the AI tool in terms of accuracy. The AI tool was also evaluated in scenarios drawn from a major database of emergency room admissions, and it performed well in these tests. According to Dr. Dominic King, corporate vice president of the health unit at Microsoft AI, existing AI systems are often evaluated using multiple-choice questions, which do not reflect real-world scenarios.
Dr. King noted that AI systems have become very good at achieving high pass rates on multiple-choice questions, but this does not translate to real-world performance. He emphasized that no patient presents with a neat summary of their symptoms, and AI systems need to be able to handle complex and nuanced cases. The researchers also found that the diagnostic processes designed by the AI tool were cost-effective.
The researchers, including Harsha Nori, head of AI at Microsoft AI Health, evaluated not only the accuracy of the AI tool but also how judiciously and carefully it went through a diagnostic work-up. The study found that the AI tool was able to design cost-effective diagnostic work-ups. The researchers plan to collaborate with Beth Israel Deaconess Medical Center in Boston to test the tool's ability to triage patients in emergency department waiting rooms.
The next step for the researchers is to test the AI tool in a real-world setting. The collaboration with Beth Israel Deaconess Medical Center will evaluate the tool's performance in triaging patients in emergency department waiting rooms. This study will provide further insights into the tool's effectiveness and help to refine its performance.
Key points
- The AI tool was more than twice as accurate as general physicians in solving complex medical cases.
- The AI tool uses a sequential approach to diagnosis, mirroring the approach of human doctors.
- The researchers plan to test the AI tool in a real-world setting, starting with a collaboration with Beth Israel Deaconess Medical Center in Boston.
