ChatGPT is as good as A&E doctors at diagnosing patients, a study has suggested, with researchers saying that it could be used to slash waiting times.
Research in the Netherlands compared the accuracy of diagnoses made by the language model, which is powered by artificial intelligence (AI), with those made by medics.
The study found that ChatGPT was at least as accurate as A&E doctors in making the right diagnosis.
Researchers in the Netherlands entered physicians’ notes on signs, symptoms and physical examinations from 30 patients who visited an emergency department last year, as well as anonymous doctors’ notes, into ChatGPT versions 3.5 and 4.0.
They also provided the chatbot with results of lab tests, such as blood and urine analysis. For each case, they compared the shortlist of probable diagnoses generated by the chatbot with the shortlist made by emergency medicine doctors and to the patient’s correct diagnosis.
The AI analysis was compared with two clinicians who made a diagnosis based on the same information, both with and without laboratory data.
Doctors had the correct diagnosis within their top five probable diagnoses in 87 per cent of cases, compared with 97 per cent for ChatGPT 3.5 and 87 per cent for ChatGPT 4.0.
The team said that while ChatGPT was “able to suggest medical diagnoses much like a human doctor would”, more work is needed before it is applied in the real world.
They said that their findings suggest that the technology could one day support doctors working in emergency medicine, potentially leading to shorter waiting times for patients.
The study was led by Dr Hidde ten Berg and Dr Steef Kurstjens from Jeroen Bosch Hospital ’s-Hertogenbosch in the Netherlands.
Dr ten Berg said: “Like a lot of people, we have been trying out ChatGPT and we were intrigued to see how well it worked for examining some complex diagnostic cases. So we set up a study to assess how well the chatbot worked compared to doctors with a collection of emergency medicine cases from daily practice.”
Dr ten Berg said that there was a lot of overlap between probable diagnoses made by ChatGPT and the doctors, with a 60 per cent overlap between differential diagnoses.
He said: “Simply put, this indicates that ChatGPT was able to suggest medical diagnoses much like a human doctor would.
“For example, we included a case of a patient presenting with joint pain that was alleviated with painkillers, but redness, joint pain and swelling always recurred. In the previous days, the patient had a fever and sore throat. A few times there was a discolouration of the fingertips.
“Based on the physical exam and additional tests, the doctors thought the most likely diagnosis was probably rheumatic fever, but ChatGPT was correct with its most likely diagnosis of vasculitis.”
Dr ten Berg stressed that ChatGPT “is not a medical device” and there are “concerns over privacy” when using the programme with medical data.
“However, there is potential here for saving time and reducing waiting times in the emergency department,” he added.
“The benefit of using artificial intelligence could be in supporting doctors with less experience, or it could help in spotting rare diseases.”
The findings of the study, published in the medical journal Annals of Emergency Medicine, will be presented at the European Emergency Medicine Congress (EUSEM) 2023 in Barcelona.
Prof Youri Yordanov, from the emergency department of St Antoine Hospital in Paris, is chairman of the EUSEM 2023 abstract committee.
He said: “We are a long way from using ChatGPT in the clinic, but it’s vital that we explore new technology and consider how it could be used to help doctors and their patients.
“People who need to go to the emergency department want to be seen as quickly as possible and to have their problem correctly diagnosed and treated. I look forward to more research in this area and hope that it might ultimately support the work of busy health professionals.”
The Telegraph’s technology editor, James Titcomb, has answered the most pressing of readers’ queries about AI and what it holds for the future, here