搜索结果：AMIA ... Annual Symposium proceedings. AMIA Symposium

Recent advances in vision-language models (VLMs) have enabled automatic radiology report generation, yet current evaluation methods remain limited to general-purpose NLP metrics or coarse classification-based clinical scores. In this study, we propose a clinically informed evaluation framework for VLM-generated radiology reports that goes beyond traditional performance measures. We define a taxonomy of 12 radiology-specific error types, each annotated with clinical risk levels (low, medium, high) in collaboration with physicians. Using this framework, we conduct a comprehensive error analysis of three representative VLMs, i.e., DeepSeek VL2, CXR-LLaVA, and CheXagent, on 685 gold-standard, expert-annotated MIMIC-CXR cases. We further introduce a risk-aware evaluation metric, the Clinical Risk-weighted Error Score for Text-generation (CREST), to quantify safety impact. Our findings reveal critical model vulnerabilities, common error patterns, and condition-specific risk profiles, offering actionable insights for model development and deployment. This work establishes a safety-centric foundation for evaluating and improving medical report generation models. The source code of our evaluation framework, including CREST computation and error taxonomy analysis, is available at https://github.com/guanharry/VLM-CREST.

Reference errors, such as citation and quotation errors, are common in scientific papers. Such errors can result in the propagation of inaccurate information, but are difficult and time-consuming to detect, posing a significant threat to the integrity of scientific literature. To support automatic detection of reference errors, we evaluated the ability of large language models in OpenAI's GPT family to detect quotation errors. Specifically, we prepared an expert-annotated, general-domain dataset of statement-reference pairs from journal articles, one-third of which is in biomedicine. Large language models were evaluated in different settings with varying amounts of reference information provided by retrieval augmentation. Results showed that large language models are able to detect erroneous citations with limited context and without fine-tuning. This study contributes to the growing literature that seeks to utilize artificial intelligence to assist in the writing, reviewing, and publishing of scientific papers as well as grounding of language model responses.

Natural products are essential in drug discovery, chemical biology, and medicinal chemistry. Despite their widespread use, NP data remains fragmented across various databases, limiting their utility for whole person health research, which requires comprehensive, interoperable resources. This study explores and compares three major NP databases: COCONUT, NP-MRD, and GSRS, assessing their scope, structural representation, metadata completeness, and accessibility. COCONUT provides extensive chemical diversity, NP-MRD emphasizes spectral and physical property data, and GSRS focuses on regulatory classification. Despite their strengths, overlap between databases is moderate to small, and significant gaps remain in integrating medical and pharmaceutical information. Improved interoperability and harmonization are needed to support advanced computational models for whole person health. Our findings highlight critical gaps and opportunities to enhance NP database integration, laying the groundwork for developing comprehensive resources that better support data-driven investigations of natural products.

Of the millions of Americans with chronic health conditions (CHCs), a growing number are coming to identify as disabled due to their CHCs. Their definition of disability differs substantially from how health informatics has traditionally thought about disability and CHCs. Rather than seeing disability as worsening CHCs that ought to be prevented, disability community definitions see disability as a form of social difference, akin to race and gender. To understand the impact of this perspective on disability on people with CHCs, we interviewed 15 participants who identify as disabled due to their CHCs. We found that it was often difficult to develop a disability identity, but doing so had significant benefits: greater self-acceptance, accessibility, and community. We conclude by identifying opportunities for health informatics to enable more people with CHCs to develop and benefit from a disability identity.

Subject: This study aims to evaluate the effectiveness of Complementary and Integrative Health (CIH) therapies in reducing the incidence and severity of Postpartum Depression (PPD) using real-world data and target trial emulation. Methods: Using electronic health records (EHR) from a large healthcare system, we emulated target trials for CIH approaches including acupuncture, chiropractic, aromatherapy, and omega-3 fatty acids. CIH usage was identified and extracted from clinical notes using natural language processing (NLP) techniques. Logistic regression-based propensity score matching was employed to address confounding factors. The primary outcome was the incidence of PPD within 12 months postpartum, defined by diagnostic codes or antidepressant initiation. Secondary outcomes included changes in PHQ-9 scores and subgroup analyses by treatment type. Results: For the primary outcome, none of the treatments significantly reduced PPD risk intervals (CIs). However, omega-3 fatty acids and chiropractic care significantly reduced PHQ-9 scores in the treatment groups (omega-3 fatty acids: p<0.001, chiropractic care: p = 0.021), with no comparable improvements in controls. Aromatherapy showed mixed results, with reduced severe depression in the treatment group but increased severity in controls. Acupuncture had no significant effect (p > 0.05). These findings suggest that omega-3 fatty acids and chiropractic care may alleviate PPD symptoms, while the effects of aromatherapy, acupuncture and chiropractic remain inconclusive and warrant further investigation. Conclusion: This study provides approach to evaluating CIH interventions in real-world settings. These findings underscore the importance of integrating non-traditional treatment options into clinical practice to improve outcomes for individuals affected by PPD.

The rapid development of Large Language Models (LLMs) has opened up new possibilities for their role in supporting research. This study assesses whether LLMs can generate "thoughtful" research plans in the domain of Medical Informatics and whether LLM-generated critiques can improve such plans. Using an LLM pipeline, we prompt four LLMs to generate primary research plans. Subsequently, these plans are mutually critiqued and then the LLMs are prompted to refine their outputs based on these critiques. These original and improved responses are then reviewed by human evaluators for errors, hallucinations, etc. We employ ROUGE scores, cosine similarity, and length differences to quantify similarities across responses. Our findings reveal variations in outputs among four LLMs, the impact of critiques, and differences between primary and secondary outputs. All LLMs produce cogent outputs and critiques, integrating feedback when generating improved outputs. Human evaluators can distinguish between primary and secondary responses in most cases.

Electronic cigarette (vaping) usage in the U.S. has steadily increased, raising significant public health concerns. Extensive research demonstrates various negative health outcomes associated with vaping. However, many potential harms remain understudied, especially those directly reported by users. Social media platforms such as Reddit offer rich, real-time sources of unfiltered personal accounts, presenting a unique opportunity to explore health outcomes beyond traditional clinical research. In this study, we systematically investigated potential negative health outcomes (NHOs) by analyzing millions of posts and comments from 15 active vaping-related subreddits in 2019. Employing robust data-driven methodologies, including advanced natural language processing (NLP) techniques such as sentiment analysis, UMLS tagging, and topic modeling, we identified distinct patterns of vaping-related health concerns. Our findings highlight the value of user-generated content for early detection of emerging risks, guiding clinicians, policymakers, and public health initiatives aimed at mitigating vaping-related harms, particularly among younger populations.

As teenagers increasingly turn to social media for health-related information, understanding the values of teen-targeted content has become important. Although videos on healthy lifestyles and self-improvement are gaining popularity on social media platforms like YouTube, little is known about how these videos benefit and engage with teenage viewers. To address this, we conducted a thematic analysis of 44 YouTube videos and 66,901 comments. We found that these videos provide various advice on teenagers' common challenges, use engaging narratives for authenticity, and foster teen-centered communities through comments. However, a few videos also gave misleading advice to adolescents that can be potentially harmful. Based on our findings, we discuss design implications for creating relatable and intriguing social media content for adolescents. Additionally, we suggest ways for social media platforms to promote healthier and safer experiences for teenagers.

De-identified health data are frequently used in research. As AI advances heighten the risk of re-identification, it is important to respond to concerns about transparency, data privacy, and patient preferences. However, few practical and user-friendly solutions exist. We developed iAGREE, a patient-centered electronic consent management portal that allows patients to set granular preferences for sharing electronic health records and biospecimens with researchers. To refine the iAGREE portal, we conducted a mixed-methods usability evaluation with 40 participants from three U.S. health systems. Our results show that the portal received highly positive usability feedback. Moreover, participants identified areas for improvement, suggested actionable enhancements, and proposed additional features to better support informed granular consent while reducing patient burden. Insights from this study may inform further improvements to iAGREE and provide practical guidance for designing patient-centered consent management tools.

Artificial intelligence and machine learning are transforming healthcare by improving clinical risk predictions and diagnostic precision. However, their performance can be compromised by data drifts due to changes in patient populations and evolving clinical practices. This study investigated performance drift in models predicting Acute Kidney Injury (AKI) using electronic health records from 249,749 inpatient encounters over ten years, analyzing performance across both the overall population and nine subgroups with unique health profiles. To mitigate the performance drift, we implemented two model updating strategies: an Overall Population Update (OPU) and a Specific Subgroup Update (SSU). Our results demonstrated significant reductions in drift, with OPU increasing the average area-under-the-precision-recall-curve (AUPRC) by 0.14 in the overall population and 0.11 across subgroups, and SSU improving the average AUPRC by 0.10 among subgroups. These findings highlight the importance of continuous model surveillance and adaptive updates to maintain reliable predictive performance in dynamic clinical environments.

Workforce challenges are a significant issue facing many healthcare organizations.1 One variable contributing to this market dynamic is provider burnout, which remains high and is largely driven by administrative demands that continue to increase.2 Healthcare organizations are rapidly adopting enabling digital capabilities, such as generative artificial intelligence (AI) technologies, that have the potential to decrease administrative burden. One such tool is ambient documentation, which aims to make clinical documentation workflows smarter and more efficient. In September 2024, ambient documentation became the first broad clinical use of generative AI at Geisinger, when the technology was deployed to 100 ambulatory providers. This paper outlines Geisinger's evaluation and implementation approach to ambient documentation and the impact this technology has made on administrative burden, provider burnout, and patient experience.

Clinical decision-making is inherently complex and fast-paced, particularly in emergency departments (EDs) where rapid and high-stakes decisions are made. Clinical Decision Rules (CDRs) are standardized evidence-based tools that combine signs, symptoms, and clinical variables into decision trees to make consistent and accurate diagnoses. CDR usage is often hindered by the clinician's cognitive load, limiting their ability to quickly recall and apply the appropriate rules. We introduce CDR-Agent, a novel LLM-based system designed to enhance ED decision-making by autonomously identifying and applying the most appropriate CDRs based on unstructured clinical notes. To validate CDR-Agent, we curated two novel ED datasets: synthetic and CDR-Bench, although CDR-Agent is applicable to non ED clinics. CDR-Agent achieves a 56.3% (synthetic) and 8.7% (CDR-Bench) accuracy gain relative to the standalone LLM baseline in CDR selection, with overall prediction accuracy improvements of 134.0% (synthetic) and 20.4% (CDR-Bench). Moreover, CDR-Agent significantly reduces computational overhead.

American Indian and Alaska Native (AI/AN) communities not only face significant health disparities but are often underrepresented in health research dissemination. Existing communication tools may fail to effectively reach these communities in culturally relevant and accessible ways, limiting their ability to benefit from critical health research. We co-designed and evaluated a prototype for health research results dissemination for AI/AN communities. We created and evaluated the prototype drawing from previous co-design workshops with AI/AN people. 38 participants completed an evaluation providing feedback for further iteration and highlighting key features such as search functionality, ease of use, visual and interactive elements, and content accessibility. Participants emphasized the importance of community connection, educational resources, and personalized experiences. We feature an alternative design approach we call Indigenous Community-Centered Design to create more accessible and engaging health research communication tools for AI/AN communities, fostering stronger connections and more accurate research representation.

Cardiovascular event adjudication is essential in clinical trials but relies on manual chart review that is slow, variable, and expensive. We present a two-stage framework that automates adjudication of cardiovascular deaths using large language models (LLMs). First, a few-shot LLM extracts structured evidence (event, span, negation, date) from unstructured clinical documents. Second, a Tree-of-Thoughts adjudicator aligns its reasoning with clinical endpoint committee (CEC) guidelines to classify deaths as cardiovascular or non-cardiovascular and produce an auditable rationale. On Lilly clinical-trial data, extraction achieved precision 0.96, recall 0.71 (F1 0.82), and adjudication attained 0.68 accuracy (GPT-4 ToT), outperforming a summarizer-plus-adjudicator baseline. We introduce CLEART, a rubric-based automated score that quantifies rationale quality across clarity, consistency, detail, guideline adherence, relevance, and timeline accuracy (overall 0.67), highlighting temporal reasoning and relevance as key areas for improvement. This approach can reduce adjudication time and variability while increasing transparency.

Documenting clients, screenings and vaccinations administered is of particular importance during mass vaccination, since information regarding uptake is critical for monitoring adverse effects and vaccine efficacy. This is especially essential when a newly-developed vaccine is being dispensed or when multiple doses of vaccine are needed per person. Despite these needs, there is no uniform or integrated system for effective vaccine data collection. This work focuses on modernizing public health infrastructure through informatics. In this paper, we describe and analyze five types of electronic technologies for registration and screening in vaccination clinics. We contrast their functionalities, usability and operations performance based on time-motion studies and service data collected during actual influenza vaccination campaigns. We evaluate their dispensing performance under an optimal dispensing clinic design. Our analysis shows that each of these electronic technologies can improve overall throughput by 16% to 45%. Based on our findings, we design a prototypical registration and screening system with integrated information flow that can be used for dispensing, monitoring and assessing mass vaccination. The system connects to the local Immunization Information System and electronic medical record systems. The design is flexible and adaptable for different types of medical countermeasures, and is suitable for regional public health departments. Our approach bridges research and public health informatics in a practical way, demonstrating how data can guide both system design and public health response planning.

Relation extraction (RE) is a standard information extraction task playing a major role in downstream applications such as knowledge discovery and question answering. Although decoder-only large language models are excelling in generative tasks, smaller encoder models are still the go to architecture for RE. In this paper, we revisit fine-tuning such smaller models using a novel dual-encoder architecture with a joint contrastive and cross-entropy loss. Unlike previous methods that employ a fixed linear layer for predicate representations, our approach uses a second encoder to compute instance-specific predicate representations by infusing them with real entity spans from corresponding input instances. We conducted experiments on two biomedical RE datasets and two general domain datasets. Our approach achieved F1 score improvements ranging from 1% to 2% over state-of-the-art methods with a simple but elegant formulation. Ablation studies justify the importance of various components built into the proposed architecture.

Stigmatizing language results in healthcare inequities, yet there is no universally accepted or standardized lexicon defining which words, terms, or phrases constitute stigmatizing language in healthcare. We conducted a systematic search of the literature to identify existing stigmatizing language lexicons and then analyzed them comparatively to examine: 1) similarities and discrepancies between these lexicons, and 2) the distribution of positive, negative, or neutral terms based on an established sentiment dataset. Our search identified four lexicons. The analysis results revealed moderate semantic similarity among them, and that most stigmatizing terms are related to judgmental expressions by clinicians to describe perceived negative behaviors. Sentiment analysis showed a predominant proportion of negatively classified terms, though variations exist across lexicons. Our findings underscore the need for a standardized lexicon and highlight challenges in defining stigmatizing language in clinical texts.

Health literacy significantly impacts patient outcomes, yet many struggle to understand complex medication information. Medication non-adherence often results from poor comprehension of drug instructions and contributes to preventable hospitalizations and poor treatment outcomes. With the increasing use of digital health interventions, AI-powered chatbots present an opportunity to improve patient access to understandable and personalized medication information. This study evaluates the usability, accessibility, and effectiveness of EMPATHICA, an AI-powered chatbot designed to provide patient-centric medication information. The study assesses whether chatbot-generated responses improve patient comprehension and engagement. The research observed the interaction between participants and the web-based application, where participants asked the chatbot to measure the chatbot's usability and accuracy using various qualitative and quantitative measures and expert physician evaluation of the responses. AI-driven chatbots have the potential to bridge health literacy gaps by providing clear and accessible medication information. By evaluating EMPATHICA, this study contributes to the growing field of AI applications supporting patient-informed medication use.

This study is part of the OsteoPorotic fracTure preventION System (OPTIONS) project which aims to develop an evidence-based mobile application for older adults transitioning from skilled nursing facilities (SNFs) back to the community after lower limb fractures. The app promotes exercise, nutrition, and bone health medications to prevent future fractures. Using a Design science framework, app requirements were identified by synthesizing scientific knowledge, clinical expertise, and end-user needs. An initial mockup was developed based on these specifications and iteratively refined through design sessions incorporating end-user feedback. The final OPTIONS app features four core functions: (1) task-based self-management support, (2) personalized exercises and education, (3) motivational messaging, and (4) progress tracking allowing users to monitor their progress through visualizations. By addressing usability challenges for older adults, the app provides a personalized, engaging experience for continuous health management.