共找到 7 条结果
Background Large-scale secondary use of clinical databases requires automated tools for retrospective extraction of structured content from free-text radiology reports. Purpose To share data and insights on the application of privacy-preserving open-weights large language models (LLMs) for reporting content extraction with comparison to standard rule-based systems and the closed-weights LLMs from OpenAI. Materials and Methods In this retrospective exploratory study conducted between May 2024 and September 2024, zero-shot prompting of 17 open-weights LLMs was preformed. These LLMs with model weights released under open licenses were compared with rule-based annotation and with OpenAI's GPT-4o, GPT-4o-mini, GPT-4-turbo, and GPT-3.5-turbo on a manually annotated public English chest radiography dataset (Indiana University, 3927 patients and reports). An annotated nonpublic German chest radiography dataset (18 500 reports, 16 844 patients [10 340 male; mean age, 62.6 years ± 21.5 {SD}]) was used to compare local fine-tuning of all open-weights LLMs via low-rank adaptation and 4-bit quantization to bidirectional encoder representations from transformers (BERT) with different subsets of reports (from 10 to 14 580). Nonoverlapping 95% CIs of macro-averaged F1 scores were defined as relevant differences. Results For the English reports, the highest zero-shot macro-averaged F1 score was observed for GPT-4o (92.4% [95% CI: 87.9, 95.9]); GPT-4o outperformed the rule-based CheXpert [Stanford University] (73.1% [95% CI: 65.1, 79.7]) but was comparable in performance to several open-weights LLMs (top three: Mistral-Large [Mistral AI], 92.6% [95% CI: 88.2, 96.0]; Llama-3.1-70b [Meta AI], 92.2% [95% CI: 87.1, 95.8]; and Llama-3.1-405b [Meta AI]: 90.3% [95% CI: 84.6, 94.5]). For the German reports, Mistral-Large (91.6% [95% CI: 90.5, 92.7]) had the highest zero-shot macro-averaged F1 score among the six other open-weights LLMs and outperformed the rule-based annotation (74.8% [95% CI: 73.3, 76.1]). Using 1000 reports for fine-tuning, all LLMs (top three: Mistral-Large, 94.3% [95% CI: 93.5, 95.2]; OpenBioLLM-70b [Saama]: 93.9% [95% CI: 92.9, 94.8]; and Mixtral-8×22b [Mistral AI]: 93.8% [95% CI: 92.8, 94.7]) achieved significantly higher macro-averaged F1 score than did BERT (86.7% [95% CI: 85.0, 88.3]); however, the differences were not relevant when 2000 or more reports were used for fine-tuning. Conclusion LLMs have the potential to outperform rule-based systems for zero-shot "out-of-the-box" structuring of report databases, with privacy-ensuring open-weights LLMs being competitive with closed-weights GPT-4o. Additionally, the open-weights LLM outperformed BERT when moderate numbers of reports were used for fine-tuning. Published under a CC BY 4.0 license. Supplemental material is available for this article. See also the editorial by Gee and Yao in this issue.
To evaluate the feasibility of a locally deployable large language model (LLM) system for automated MRI protocol selection addressing data privacy, annotation burden, and scalability limitations. This retrospective study included 598 German-language MRI order entries from three neuroradiology domains (brain, head/neck, spine) between June 2018 and January 2023. A radiologist labeled entries for 27 protocol classes based on institutional standard operating procedures (SOP). An SOP-grounded AI system using MedGemma 27B was developed to predict the MRI protocol from the order entry. The system was optimized using Stochastic Introspective Mini-Batch Ascent (SIMBA), a self-reflective prompt optimization algorithm, and compared with a hierarchical system that first classified the body region and then the MRI protocol. Data efficiency was evaluated using training subsets of 10-119 examples across 3 optimization runs per subset size. The flat zero-shot model achieved 73.07% accuracy in the three-domain setting on the held-out dataset (n = 479). In the hierarchical model, prompt optimization yielded 73.90% ± 3.24% with 30 labeled examples and a maximum of 74.46% ± 3.82% but did not outperform the flat approach. Prompt optimization benefited the hierarchical system, whereas the flat model already performed strongly without labeled examples for optimization. Performance on brain and head/neck cases remained broadly stable after expansion from 22 to 27 protocol classes, i.e., including spine. A locally deployable SOP-grounded open-weight LLM can support MRI protocol selection while preserving data privacy and needing minimal labeled data. In this dataset, hierarchical routing and prompt optimization did not improve overall performance over the flat baseline, although they altered optimization behavior and error profile. These findings support prospective evaluation in human-in-the-loop clinical workflows.
暂无摘要（点击查看详情）
Large language models (LLMs) increasingly inform mental health decisions by patients and clinicians. Inference-time activation steering can shift model behavior on a target dimension without altering weights or prompts and without disclosure to users, allowing treatment recommendations to be silently changed for commercial or ideological reasons. To determine whether directional activation steering can shift an open-weights LLM's depression treatment recommendations. This non-human subjects study applied directional activation steering to an open-weights LLM (DeepSeek V4 Flash) responding to 12 depression-advice scenarios (4 favoring medication, 4 favoring avoidance, 4 neutral), generated at 30 amplitudes from -1.5 to +1.5 in 0.1 increments plus an unsteered baseline. A single steering direction contrasting antidepressant medication with self-directed approaches (diet, exercise, meditation, dietary supplements), constructed from 16 paired training prompts and applied at the attention output of every transformer block; weights and system prompt were held constant. The extent to which medication and four self-care categories were addressed, scored 0 to 3 by a human-validated LLM rater (Claude Opus 4.7), the medication-versus-self-care balance, and clinician referral, estimated per unit of amplitude using mixed-effects models with a scenario random intercept. Across 372 generations, steering produced a graded, dose-dependent shift in the medication-versus-self-care balance, which declined by 0.32 per unit of amplitude (β = -0.32; 95% CI, -0.39 to -0.25; P < .001); medication extent fell and self-care extent rose. The shift was largest for scenarios with no stated treatment preference (β = -0.44; 95% CI, -0.54 to -0.34; P < .001). A clinician referral appeared in 322 of 372 responses (87%) and did not vary with steering amplitude (P = .63). In this open-weights LLM providing depression treatment information, inference-time activation steering shifted treatment recommendations without altering weights, prompt structure, or safety outputs, with the largest effect among users expressing no treatment preference. These findings suggest a need for LLM disclosure standards and independent auditing as such models inform clinical decisions.
Extracting information from radiology reports can provide critical data to empower many radiology workflows. For spinal compression fractures, these data can facilitate evidence-based care for at-risk populations. Manual extraction from free-text reports is laborious, and error-prone. Large language models (LLMs) have shown promise; however, fine-tuning strategies to optimize performance in specific tasks can be resource intensive. A variety of prompting strategies have achieved similar results with fewer demands. Our study pioneers the use of Meta's Llama 3.1, together with prompt-based strategies, for automated extraction of compression fractures from free-text radiology reports, outputting structured data without model training. We tested performance on a time-based sample of CT exams covering the spine from 2/20/2024 to 2/22/2024 acquired across our healthcare enterprise (637 anonymized reports, age 18-102, 47% Female). Ground truth annotations were manually generated and compared against the performance of three models (Llama 3.1 70B, Llama 3.1 8B, and Vicuna 13B) with nine different prompting configurations for a total of 27 model/prompt experiments. The highest F1 score (0.91) was achieved by the 70B Llama 3.1 model when provided with a radiologist-written background, with similar results when the background was written by a separate LLM (0.86). The addition of few-shot examples to these prompts had variable impact on F1 measurements (0.89, 0.84 respectively). Comparable ROC-AUC and PR-AUC performance was observed. Our work demonstrated that an open-weights LLM excelled at extracting compression fractures findings from free-text radiology reports using prompt-based techniques without requiring extensive manually labeled examples for model training.
While Americans are using herbal dietary supplements (natural products) more than ever, the consumption of natural products with prescription drugs can lead to harmful interactions. Pharmacovigilance of natural products depends on careful expert review and interpretation of a wide variety of evidence. In prior work, we demonstrated the value of knowledge graph (NP-KG) for assisting with natural product safety investigations. However, scaling the NP-KG from 33 natural products to the thousands on the market requires computer-assisted data extraction, particularly from visual elements (figures or tables) of pharmacology literature. We evaluated the accuracy and resilience of 3 open-weights and 7 closed-weights multimodal models by performing visual information extraction from select tables and images. The best performing models could accurately extract 90% of tabular data and 45% of data reported figures with a modified relative error rate of 0.05. Image resolution and information density were primary hindrances to better extraction performance.
Patient access to radiology reports has heightened the need for patient-friendly communication. Automated generation of patient-centered summaries using large language models (LLMs) is a promising solution. However, their use on real-life reports is limited by privacy concerns. Here, we evaluate the safety and effectiveness of on-premise, privacy-preserving LLMs for generating lay summaries of real French brain MRI reports for emergency presentations of headache. In this retrospective study, we sampled 105 brain MRI reports (January–December 2022) for radiologist evaluation and a subset of 30 reports for non-physician evaluation. Three open-weights models (Llama 3.3 70B, Athene V2, Mistral Small) generated French lay summaries via a single standardized prompt. Radiologists’ mean ratings across models were high for exactness (4.10, 95% CI: 4.04–4.16), exhaustiveness (4.34, 95% CI: 4.29–4.39), didacticness (3.83, 95% CI: 3.79–3.88), and readiness for clinical use (3.84, 95% CI: 3.79–3.89). Non-physicians reported higher perceived understanding with summaries, from 2.85 (95% CI: 2.67–3.04) to 4.27 (95% CI: 4.15–4.38, p < 0.001). The correct identification rate for reports increased from 75.2% to 83.6% (p < 0.001). The ability to identify causal findings also improved, from 80.6% to 84.8% (p < 0.001) overall. Overall error rate in LLM-generated lay summaries was 19.7% (62/315), warranting expert oversight.