Artificial intelligence (AI)-driven tools used in emergency triage may be susceptible to the same clinical stigmas associated with human assessments, researchers at UT Southwestern Medical Center reported in a study published in npj Digital Medicine.
AI Triage Tools Susceptible to Human Biases A study by UT Southwestern Medical Center reveals that AI tools used in emergency triage can inherit human biases, potentially leading to harmful and incorrect patient prioritization in emergency departments. The Impact of Stigmatizing Language Dr. Philip Jarrett highlights that the specific wording used to describe patients influences AI models. Stigmatizing language can misdirect AI, preventing it from prioritizing patients who need care most, underscoring the importance of careful clinical information presentation for safe AI-assisted triage. Study Methodology and Initial Findings Researchers analyzed emergency department patient records from Stanford Medicine and Beth Israel Deaconess Medical Center. They used three open-source AI large language models (LLMs) to predict patient deterioration. Initially, without sensitive or stigmatizing data, models successfully identified high-risk patients. The study then progressively added various patient details, like mental health, substance use history, age, and ED visit patterns, using both neutral and stigmatizing language to observe changes in AI prioritization decisions. Specific Instances of AI Bias The study found that details linked to clinical stigma significantly influenced AI prioritization. Information unrelated to a patient's immediate medical condition, such as being a 'frequent ED user' or having a history of psychiatric emergencies, caused AI models to deprioritize patients who later deteriorated. This mirrored human biases and resulted in incorrect downgrading for nearly 10% of patients in certain scenarios. Nuances in AI Response to Patient Data The study observed varied AI responses to different types of information. While some substance use descriptions yielded unexpected outcomes, highlighting the intricate nature of AI's interpretation of clinical language, age-related influences were largely attributed to higher medical risks in older patients rather than inherent age bias. Recommendations and Future Research Dr. Jarrett recommends that each AI tool be rigorously tested against real patient outcomes due to inherent model variability. Future research will focus on developing AI models that predict emergency department deterioration using both structured data and clinical text. The findings emphasize that how patient information is documented significantly influences AI's ability to make fair and accurate triage decisions as hospitals increasingly adopt AI. Scientific Understanding and Collaborative Effort Dr. Andrew Jamieson emphasizes the ongoing scientific endeavor to understand AI system behavior, including how their responses are shaped and vary in clinical practice. He underscores that clinical needs drive their research to enhance tools for real-world problems. The study also acknowledges collaborative contributions from other key researchers in Emergency Medicine and Bioinformatics. Funding and Support The study received institutional funding from the UT Southwestern Medical Center Office of the President and additional grant funding from the National Foundation of Emergency Medicine for follow-up research. About UT Southwestern Medical Center UT Southwestern Medical Center is a premier academic medical institution integrating biomedical research, clinical care, and education. Its distinguished faculty, including Nobel laureates, is dedicated to translating scientific discoveries into clinical treatments. The center's physicians provide comprehensive care across over 80 specialties, managing a large volume of hospitalized patients, emergency room cases, and outpatient visits annually.
AI Triage Tools Susceptible to Human Biases
A study by UT Southwestern Medical Center reveals that AI tools used in emergency triage can inherit human biases, potentially leading to harmful and incorrect patient prioritization in emergency departments.
The Impact of Stigmatizing Language
Dr. Philip Jarrett highlights that the specific wording used to describe patients influences AI models. Stigmatizing language can misdirect AI, preventing it from prioritizing patients who need care most, underscoring the importance of careful clinical information presentation for safe AI-assisted triage.
Study Methodology and Initial Findings
Researchers analyzed emergency department patient records from Stanford Medicine and Beth Israel Deaconess Medical Center. They used three open-source AI large language models (LLMs) to predict patient deterioration. Initially, without sensitive or stigmatizing data, models successfully identified high-risk patients. The study then progressively added various patient details, like mental health, substance use history, age, and ED visit patterns, using both neutral and stigmatizing language to observe changes in AI prioritization decisions.
Specific Instances of AI Bias
The study found that details linked to clinical stigma significantly influenced AI prioritization. Information unrelated to a patient's immediate medical condition, such as being a 'frequent ED user' or having a history of psychiatric emergencies, caused AI models to deprioritize patients who later deteriorated. This mirrored human biases and resulted in incorrect downgrading for nearly 10% of patients in certain scenarios.
Nuances in AI Response to Patient Data
The study observed varied AI responses to different types of information. While some substance use descriptions yielded unexpected outcomes, highlighting the intricate nature of AI's interpretation of clinical language, age-related influences were largely attributed to higher medical risks in older patients rather than inherent age bias.
Recommendations and Future Research
Dr. Jarrett recommends that each AI tool be rigorously tested against real patient outcomes due to inherent model variability. Future research will focus on developing AI models that predict emergency department deterioration using both structured data and clinical text. The findings emphasize that how patient information is documented significantly influences AI's ability to make fair and accurate triage decisions as hospitals increasingly adopt AI.
Scientific Understanding and Collaborative Effort
Dr. Andrew Jamieson emphasizes the ongoing scientific endeavor to understand AI system behavior, including how their responses are shaped and vary in clinical practice. He underscores that clinical needs drive their research to enhance tools for real-world problems. The study also acknowledges collaborative contributions from other key researchers in Emergency Medicine and Bioinformatics.
Funding and Support
The study received institutional funding from the UT Southwestern Medical Center Office of the President and additional grant funding from the National Foundation of Emergency Medicine for follow-up research.
About UT Southwestern Medical Center
UT Southwestern Medical Center is a premier academic medical institution integrating biomedical research, clinical care, and education. Its distinguished faculty, including Nobel laureates, is dedicated to translating scientific discoveries into clinical treatments. The center's physicians provide comprehensive care across over 80 specialties, managing a large volume of hospitalized patients, emergency room cases, and outpatient visits annually.
