共找到 20 条结果
Background: Reliable structured-output generation is a prerequisite for using large language models (LLMs) in automated clinical documentation workflows, but many evaluations focus on clinical quality before testing whether outputs are parseable, schema-compliant, and stable. Methods: We evaluated three locally deployed open-weight LLMs in the 7- to 8-billion-parameter range (Llama3-Med42-8B, Meta-Llama-3-8B-Instruct, and Mistral-7B-Instruct-v0.3) for structured admission-note editing. Seventy de-identified English-language admission notes (35 internal medicine and 35 surgical) were processed by each model in three independent runs under two output-control conditions: a free-text JSON prompt and a schema-enforced structured-output condition. A total of 1260 local inferences were performed in LM Studio on consumer-grade hardware. Automated proxy metrics assessed JSON/schema validity, run-to-run stability, instruction compliance, verbosity, numeric-token preservation, and uncertainty-marker change without clinician adjudication of clinical correctness. Results: Under the free-text JSON prompt, the tested Mistral-7B-Instruct-v0.3/embedded-prompt configuration had the weakest structural reliability (74.3-78.6% first-pass validity per run; 18.6-21.4% persistent parse/schema failures after retry), with at least one final failure for 17 of 70 notes. In a message-format sensitivity analysis using Meta-Llama-3-8B-Instruct, embedding system instructions in the user message increased first-attempt invalid outputs compared with separate system/user roles (55/700, 7.9% vs. 12/700, 1.7%). Under schema enforcement, all models produced 70 of 70 first-pass valid, schema-compliant outputs in every run. Documentation behavior nevertheless differed by model, including differences in verbosity and numeric-token preservation. Conclusions: Schema enforcement removed parsing failures in this sample but did not eliminate model-specific editing behavior. Proxy-based screening can identify structurally unstable model-prompt or model-format configurations before clinician review.
Generative language models can drift mid-generation from reliable, helpful continuations to more undesirable, misleading, or unsafe ones. Such shifts can be hard to notice because the early part of a response can be fluent and correct, so downstream users (or automations) may not trigger checks until after harm occurs. Here, we isolate a minimal, first-principles mechanism for such "good-to-bad" output shifts. The mechanism originates within a single effective attention head, where the dot products of embedded vectors drive a competition between output basins; mapping to a multispin thermal system yields a closed-form expression for the shift. Crucially, multilayer processing does not wash out this single-head mechanism-it amplifies it. Tracking hidden states through all layers of open-weight models, we find that the competing dot-product gaps grow by up to 617 × from the first layer to the penultimate layer, and that successive layers fuse competing basins into a shared geometric subspace-constructing precisely the configuration the single-head formula assumes. A cross-architecture study on six independently trained models confirms that the resulting closed-form expression, with zero free parameters, correctly predicts the output-shift regime in 94 of nonambiguous cases. Our physical picture is of course extremely simplified-just as a paper plane cannot capture all the details of modern aviation, or a simple model of an atom cannot capture all the details of a complex material-but it helps fill a current need for generative AI explainability, and it provides a simple yet concrete platform for discussing how training, fine-tuning and prompts might shift generative AI output.
Egocentric videos are inherently long-form, as they provide a continuous, first-person perspective of daily life, capturing complex social interactions and routines that naturally span days or weeks. Understanding and reasoning over egocentric videos that span hours or even days poses significant challenges due to their length, multimodal nature, and complex temporal dependencies over long time horizons. To this end, we introduce Ego-R1, a novel framework for reasoning over ultra-long (i.e., days and weeks) egocentric videos. Ego-R1 leverages a structured Chain-of-Tool-Thought (CoTT) process, orchestrated by an Ego-R1 Agent trained via reinforcement learning (RL). Inspired by human problem-solving strategies, CoTT decomposes complex reasoning into modular steps, empowering the agent to act as a high-level controller that dynamically invokes specialized tools-such as hierarchical memory retrievers and multimodal perceptors-to iteratively and collaboratively answer sub-questions. This approach enables effective temporal abstraction, long-horizon dependency tracking, and step-by-step multimodal reasoning. The framework is built upon a flexible toolkit designed for efficient temporal retrieval and granular visual analysis: Hierarchical RAG (H-RAG), a text-based module that performs efficient top-down temporal localization by aggregating video logs from day-level summaries down to 10-minute intervals; Video-LLM, a short-horizon perception module that analyzes local temporal windows to interpret dynamic interactions; and VLM, a fine-grained vision-language model used to extract high-resolution details, such as text or object attributes, from specific frames. We design a two-stage training paradigm involving supervised fine-tuning (SFT) of a pretrained language model using CoTT data, to enable dynamic tool proposal for long-range reasoning; followed by RL, to enhance the performance of plan smartly with tools. To facilitate training, we construct Ego-R1 Data, which consists of Ego-CoTT-25 K for SFT and Ego-QA-4.4 K for RL. Furthermore, we evaluate Ego-R1 on a newly curated week-long video QA benchmark, Ego-R1 Bench, which contains hybrid-source, human-verified QA pairs. Extensive experiments show that our 3B-parameter Ego-R1 Agent achieves the strongest performance among open-weight and tool-agent baselines, while offering interpretable tool-grounded reasoning trajectories. On Ego-R1 Bench, Ego-R1 achieves 46.0% accuracy, substantially outperforming Gemini-1.5-Pro (38.3%) and LLaVA-Video (29.0%); we further report Gemini-3.1-Pro as a stronger closed-source reference at 53.7%. Moreover, the framework exhibits strong generalization to standard exocentric video benchmarks; by leveraging the long-video nature of egocentric data to train the orchestrator's planning capabilities rather than overfitting the perceptors to a specific view, our modular design remains robust across domains. Ego-R1 Agent achieves 64.9% accuracy on Video-MME (long), surpassing leading open-weight models. These results validate that dynamic, tool-augmented reasoning effectively bridges the gap between limited context windows and the demands of understanding both week-long first-person experiences and general long-form video content.
To systematically identify and characterize original rheumatology research using large language models (LLMs), quantify reliance on closed/proprietary versus open systems, and assess reporting practices relevant to reproducibility (model versions, prompts, inference settings, and availability of code/data), in order to identify recurring reporting gaps and inform pragmatic considerations for improving transparency. We searched PubMed/MEDLINE for English-language peer-reviewed original research published from 1 November 2022 to 23 January 2026. Two reviewers independently screened title/abstract and full texts, with a third resolving disagreements. Data were extracted using an LLM-assisted workflow and then independently verified against source articles by the authors. Extracted items included study characteristics, model families and openness, access mode, versioning/timing, and transparency indicators (prompts, code, data). Of 185 screened records, 63 studies were included. Most were research (n = 26/63; 41.27%) or education-focused (n = 20/63; 31.75%). Studies predominantly used closed/proprietary LLMs (n = 50/63; 79.37%), with limited exclusive use of open-weight models (n = 4/63; 6.35%) and some hybrid use (n = 9/63; 14.29%). OpenAI models were most common (n = 55/63; 87.30%). Reporting was heterogeneous: interaction language was often not reported (n = 42/63; 66.67%); access mode was reported in (n = 19/63; 30.16%); output generation date in (n = 33/63; 52.38%). Prompts were shared in (n = 48/63; 76.19%), but code was publicly available in (n = 4/63; 6.35%), and data in (n = 13/63; 20.63%). LLM-based rheumatology research largely depends on closed models with inconsistent reporting of reproducibility-critical details and minimal code sharing. Clearer reporting practices and, in the longer term, consensus-based standards will be needed to strengthen the methodological robustness of this field.
To evaluate whether a locally hosted open-weight large language model (LLM) can extract documented psychosocial factors from pediatric psychiatric intake notes and apply validated extraction to a large emergency psychiatry cohort. We identified emergency department presentations at Cincinnati Children's Hospital Medical Center from January 1, 2016, through December 31, 2024, among patients <18 years with psychiatric billing diagnoses. Using full-text intake notes, gpt-oss:120b classified peer conflict, sleep disruption, and school-related academic, attendance, and disciplinary issues as detected, negated, or indeterminate. Four human raters independently reviewed 50 notes. We compared Fleiss' κ among humans alone versus humans plus the LLM, assessed repeated-query stability across 50 independent calls per note, and applied the workflow to all eligible notes. Among 37,315 eligible admissions, 22,284 had eligible intake notes; 22,270 produced parseable JSON. In detected-vs-not-detected coding, human-plus-LLM reliability did not differ significantly from human-only reliability across measures (human κ, 0.71-0.94; human-plus-LLM κ, 0.70-0.93). Stability was associated with human agreement: mean LLM-human agreement increased from 42.6% for classifications with <80% stability to 82.7% for 100% stability (Pearson r=0.36). Full-cohort extraction showed frequent and overlapping documented factors: sleep disruption was most frequently detected (57.7%), followed by peer conflict (47.2%), academic issues (43.4%), disciplinary issues (43.3%), and attendance issues (16.9%). Agreement varied by construct and was strongest when repeated model outputs were stable. Locally hosted open-weight LLMs can support scalable structured extraction of documented psychosocial factors from pediatric psychiatric intake notes after local validation.
To compare open-weight and proprietary large language models (LLMs), a rule-based extractor (RBE) and radiologists for labelling pulmonary embolism CT reports, and to test whether a hybrid RBE-LLM workflow improves labelling performance. This single-centre retrospective study included structured CT reports from October 2021 to March 2025. Three labelling pipelines were evaluated: an RBE; a model-agnostic LLM extractor (18 open-weight, four GPT-4 variants); and a hybrid pipeline routing only RBE failures to an LLM. Ground truth was defined at the report-text level by deterministic schema matching for initially RBE-valid fields and blinded adjudication of RBE-invalid fields by two attending radiologists. Eight radiologists provided a human baseline. Outcomes included F1 scores, accuracy, LLM-based salvage of RBE failures, and labelling time. In total, 2,923 reports from 2,923 patients (mean age 66 ± 17 years; 1,465 women) were included. Falcon3-10b and GPT-4.1-mini achieved similar item-level performance (F1 0.98 [95% CI, 0.97-0.98] for both; p = 0.70) and both exceeded the RBE (F1 0.81 [95% CI, 0.80-0.82]; p < 0.001). Salvage of RBE failures was comparable between open-weight and proprietary models (88.1% vs 91.9%; p = 0.12). The hybrid RBE-LLM workflow achieved 99.8% accuracy and F1 0.99 (0.98-0.99), exceeding both the RBE and pooled radiologists (F1 0.92 [95% CI, 0.90-0.93]; all p < 0.001). Schema-constrained open-weight and proprietary LLMs exceeded rule-based extraction and, at the upper end of performance, matched a pooled radiologist label-transfer baseline. A rules-first, targeted LLM workflow enabled near-perfect extraction from finalised structured pulmonary embolism CT reports. A rules-first LLM workflow can automate high-fidelity extraction of structured CT findings from finalised radiology reports, enabling scalable, auditable, and more consistent cohort curation for clinical research, registries, and quality improvement. A hybrid rules-first workflow combining a rule-based extractor (RBE) with targeted large language model (LLM) salvage achieved the highest overall performance for labelling of pulmonary embolism CT reports (F1, 0.99; accuracy, 99.8%). The top standalone open-weight and proprietary LLMs (Falcon3-10b and GPT-4.1-mini) both exceeded the RBE and, at the upper end of performance, matched a pooled radiologist label-transfer baseline. The hybrid workflow reduced cohort-curation time from 32.2 h for radiologists to 1.0 h while reducing LLM calls by 85.6%, because the LLM was only triggered for rule-failed fields.
Researchers who classify text with large language models often possess codebooks written for human annotators. We ask how these codebooks can serve as prompts. Using three codebooks from experimental economics (one on promise classification, two on strategic thinking), we vary the level of information in the prompt, then vary its formatting, framing and wording at the lowest and highest information levels, across two proprietary and two open-weight models. Used as prompts, the codebooks reach 82-88% agreement with human annotators across the three tasks. On the recognition-heavy task (promise classification), model choice accounts for most of the variation in accuracy; on the learning-heavy tasks (strategic thinking classification), the level of detail in the classification instructions carries comparable weight. These information components partly substitute for one another, whereas model reasoning does not reliably compensate for missing content and yields little or no improvement once the content is present. Larger models make better use of additional information and are more robust to formatting, framing, and wording of the prompt, while smaller models can be hurt by extra information and are more sensitive to how the information is presented. Our results advocate for a shift in focus from prompt engineering techniques (formatting, framing, reasoning, etc.) to the content of the prompt: preparing instructions as one would for human annotators, with detailed context, category definitions, and examples.
Accurate identification of clinical symptoms and signs (S&S) is essential for the early detection of high-burden cardiorespiratory conditions, including lung cancer, chronic obstructive pulmonary disease, and heart failure. Although symptom data play a central role in diagnostic reasoning and predictive modeling, most S&S information remains embedded in unstructured electronic health record notes, limiting their use in automated phenotyping, surveillance, and clinical decision support. Traditional natural language processing systems struggle with domain variability and contextual nuance in clinical text. Recent advances in large language models (LLMs) offer a promising alternative, yet challenges remain in hallucinations, overinference, and safe deployment. This study evaluated whether locally deployed open-source models could reliably extract cardiorespiratory S&S and map them to ICD-10-CM (International Classification of Diseases, Tenth Revision, Clinical Modification) codes using optimized prompting strategies. This study aims to assess the accuracy of open-source LLMs in extracting explicitly stated cardiorespiratory S&S from clinical notes and mapping them to ICD-10-CM codes (R00-R09) and to compare performance across 4 prompt-engineering strategies, including a multimodule LLM framework. A total of 593 clinical notes from the MTSamples database were manually reviewed, with 93 notes used for prompt development and comparison using Llama 3.3-70B, and 500 notes used as testing data for the final best prompt setting using both Llama 3.3-70B and gpt-oss-120B. Four prompting conditions were evaluated: (1) instruction-only, (2) ICD-10-CM definition-based prompts, (3) assumption-free prompts, and (4) a multimodule LLM framework with postprocessing. Performance was measured using precision, recall, and F1-score for both S&S extraction and ICD-10-CM code generation. Across all prompt strategies, model performance improved as more structure and constraints were added. Instruction-only prompting demonstrated high recall but poor precision. Incorporating ICD-10-CM definitions improved coding accuracy, and assumption-free prompting further balanced precision and recall. The multimodule approach with postprocessing achieved the highest performance during prompt development. On the independent test corpus, entity-level microaveraged evaluation showed that gpt-oss-120B outperformed Llama 3.3-70B in both tasks. For S&S extraction, Llama 3.3-70B achieved a precision of 0.63, a recall of 0.86, and an F1-score of 0.73, whereas gpt-oss-120B achieved a precision of 0.89, a recall of 0.87, and an F1-score of 0.88. For ICD-10-CM code mapping, Llama 3.3-70B achieved a precision of 0.59, a recall of 0.83, and an F1-score of 0.69, whereas gpt-oss-120B achieved a precision of 0.90, a recall of 0.84, and an F1-score of 0.87. Locally deployed LLMs, when paired with optimized prompting and multimodule orchestration, can accurately extract cardiorespiratory S&S and generate ICD-10-CM codes from unstructured clinical notes. This approach increases the level of data safety by enabling on-premises processing without external data transmission and demonstrates strong potential for scalable, domain-adaptive symptom extraction pipelines in biomedical informatics. Future work should expand datasets and evaluate generalizability across clinical domains.
We study the security resilience of large language models to prompt-chain attacks that exploit context retention and sequencing. Existing evaluations emphasize single-turn or fixed-domain settings and thus under-measure vulnerabilities to sequence-aware adversaries. We introduce SecReEvalBench, a scenario-based benchmark with six attack sequences (one-off, successive, successive-reverse, alternative, sequential ascending, sequential descending) and four metrics tailored to multi-turn security: Prompt Attack Resilience Score, Prompt Attack Refusal-Logic Score, Chain Attack Resilience Score, and Chain Attack Refusal Timing Score. The accompanying dataset spans seven security domains, sixteen attack techniques, and four maliciousness levels with dual-LLM adjudication for intent labels. Our evaluation protocol combines an unsafe-content detector (Llama-Guard 3) with a refusal classifier to disentangle eventual unsafe cooperation from refusal timing, yielding bounded, auditable, and sequence-sensitive measurements. Experiments on five representative open-weight models (Llama 3.1, Gemma 2, Mistral v0.3, DeepSeek-R1, Qwen 3) and two proprietary models (gpt-3.5-turbo and gpt-4o-mini) show that sequence-aware metrics surface vulnerabilities that single-turn tests miss, including delayed or absent refusals under escalation and leakage through visible intermediate reasoning despite surface-level refusals. SecReEvalBench standardizes multi-turn safety assessment and provides a reproducible basis for developing domain- and sequence-aware defenses. Our dataset is made publicly available at https://kaggle.com/datasets/5a7ee22cf9dab6c93b55a73f630f6c9b42e936351b0ae98fbae6ddaca7fe248d.
Genomic foundation models can dramatically accelerate biological research by learning general-purpose representations of genomic data that transfer across tasks, enabling researchers to predict variant effects, regulatory elements, and molecular function, among others. To safeguard against potential biosecurity threats and malicious misuse of open-weight models, a common strategy involves excluding human-infecting viral genomes from the model's training corpora. This strategy, however, can be easily circumvented by fine-tuning models on abundantly available viral data. Weight-locking with spectral deformation has been proposed as a potential method to prevent fine-tuning of neural networks, but has not been systematically evaluated in biological AI models. We applied spectral deformation locking to the Evo-1-8k-base genomic foundation model and evaluated a panel of attack configurations spanning naive fine-tuning, low-rank adaptation (LoRA), a simple inserted-layer bypass baseline, and a white-box singular value decomposition (SVD)-chain factorisation at chain lengths k ∈ {2, 3, 5}. Recovered virological capability was quantified on three Human Virome Understanding Evaluation (HVUE) tasks. The lock defended against the naive attacker by either standard pipeline. Naive full fine-tuning under the strong lock drove downstream virological capability significantly below the pretrained baseline on pathogenicity and host tropism, converting the attack into a capability loss rather than a gain, while naive low-rank adaptation neither moved held-out perplexity (PPL) nor recovered downstream capability above pretrained. Thus, we conclude that by neither route does the naive attacker reach the gain achieved by fine-tuning an unlocked model. Consistent with previous results in non-biological models, an informed attacker who implements the SVD-chain construction does recover capability on pathogenicity prediction, at the cost of increased computational requirements for the fine-tuning process. https://github.com/Georgakopoulos-Soares-lab/glm-locking .
Large language models (LLMs) are rapidly incorporated into medical education and examination preparation; yet, most benchmarking evidence is derived from English-language material. Whether frontier commercial models and Brazilian Portuguese domain-specialized systems perform equivalently on high-stakes Brazilian medical examinations remains unclear. This study aims to quantify and compare the performance of 9 frontier commercial LLMs and 1 Brazilian Portuguese domain-specialized system (Charcot, Voa Health) on 2026 Brazilian National Medical Education Examination (Exame Nacional de Avaliação da Formação Médica [ENAMED] 2026) and to describe the patterns of systematic between-model error as complementary quality signal. All 100 items of ENAMED 2026 (99 valid after annulment) were administered to 10 frontier-panel models across 5 independent runs under identical Portuguese prompts (temperature=0; top-p=.95). Commercial models were accessed through a unified OpenRouter client layer (DeepSeek provider-pinned). The primary outcome was mean accuracy against the preliminary key; the secondary outcomes were convergence error (CE), normalized mean response time (NMRT), and intermodel agreement. Accuracy was analyzed with Shapiro-Wilk, Levene, Kruskal-Wallis (ε2), Dunn-Holm post hoc, and a binomial generalized linear mixed model with question and run random intercepts. NMRT excluded Charcot (different stack) and Grok 4 (latency outlier). A total of 7 open-weight and small language models were assessed as a small language model (SLM) substudy. Frontier-panel accuracy ranged from 73.74% (365/495) for GPT-4o-mini to 96.97% (480/495) for Charcot. Accuracy was nonnormal (Shapiro-Wilk, W=0.82; P<.001) with homogeneous variance (Levene P=.26). Kruskal-Wallis showed large between-model differences (H9=47.65; P<.001; ε2=0.97). Dunn-Holm flagged 8 of 45 pairs: Charcot was separable from GPT-4o-mini, DeepSeek v3.2-exp, and Grok 4, but not from the top frontier cluster. The generalized linear mixed model preserved the ranking (all comparators odds ratio<1 vs Charcot; upper CI<1 except GPT-5). A total of 9 items met the default CE criterion; item 77 showed 10-of-10 convergence, later confirmed by Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira rectification, and sensitivity analysis preserved the CE set (5-20 items). Intermodel agreement was high (Fleiss κ=0.852; Krippendorff α=.852). Among 8 retained commercial models, NMRT correlated positively with accuracy (Spearman ρ=0.74; P=.04); these results are interpreted as descriptive of an orchestration-level latency-accuracy law. SLM accuracy ranged from 47.47% (Gemma 3 4B) to 82.22% (GPT-OSS 120B), with lower agreement (Fleiss κ=0.508) and 17 CE items. The best SLM-panel model lagged every frontier-panel model except GPT-4o-mini, with a 12-15 percentage-point gap against the top cluster. On ENAMED 2026, a Brazilian Portuguese domain-specialized system ranked first, indistinguishable from frontier commercial cluster and above subfrontier and open-weight systems. Charcot's architecture is not publicly disclosed; these findings should be interpreted as comparative black-box evidence of performance and not as mechanistic evidence of specialization. CE was stable, and it prospectively flagged 1 rectified item, supporting its use as a quality assurance screen.
Large Language Models (LLMs) extracts structured data from unstructured Ga-68 Prostate Specific Membrane Antigen Positron Emission Tomography/Computed Tomography (PSMA PET/CT) reports by interpreting context and semantics, enabling reliable conversion of free-text into structured, queryable data. We evaluated whether locally deployed Llama 3.2:3b and Gemma 2:2b models could accurately perform zero-shot extraction of key diagnostic features. We retrospectively selected 50 de-identified Ga-68 PSMA PET/CT reports of histologically confirmed prostate cancer (01 January 2020-30 June 2024), ensuring rigorous expert annotation and inter-reader agreement. Reports were batch-processed using standardized zero-shot and few-shot prompts applied to locally deployed Llama 3.2:3b and Gemma 2:2b models, with model outputs benchmarked against an adjudicated dual-expert ground truth reference derived from independent annotations and consensus resolution of discrepancies using standard performance metrics. Both Llama 3.2:3b and Gemma 2:2b achieved rapid, reliable extraction of key diagnostic features from Ga-68 PSMA PET/CT reports, with excellent inter-reader agreement (average κ = 0.882). Llama 3.2:3b offered superior sensitivity (83.7%) and negative predictive value (NPV) (97.3%), while Gemma 2:2b demonstrated greater overall accuracy (86.2%) and specificity (88.5%). Model performance varied with question prevalence and complexity, highlighting distinct strengths for clinical integration. Open-weight Llama 3.2:3b and Gemma 2:2b LLMs enable rapid extraction of key PET/CT findings (16-20x faster than manual review), with Llama excelling in sensitivity and Gemma in specificity. However, modest performance on rare but clinically important findings and limited positive predictive value indicate these models are best suited for initial screening or human-assisted workflows rather than autonomous extraction. The online version contains supplementary material available at 10.1007/s13139-026-00991-z.
Clinical narrative text contains crucial patient information, yet reliable extraction remains challenging due to linguistic variability, documentation habits, and differences across care settings. Large language models (LLMs) have shown strong accuracy on clinical information extraction (IE), but their reproducibility (stability under repeated runs) and robustness (stability under small, natural prompt variations) are less consistently quantified, despite being central to clinical deployment. In this study, we evaluate three open-weight LLMs representing distinct modeling choices: a dense general-purpose model (Llama 3.3), a mixture-of-experts (MoE) general-purpose model (Llama 4), and a domain-tuned medical model (MedGemma). We focus on binary clinical IE aligned with four mobility classes from the International Classification of Functioning, Disability and Health (ICF) framework. Using a controlled experimental design, we quantify (1) intra-prompt reproducibility across repeated sampling and (2) inter-prompt robustness across paraphrased prompts. We jointly report predictive performance (F1-score) and stability (Fleiss' Kappa). And we test factor effects using three-way ANOVA with post-hoc comparisons. Results show that increasing temperature generally degrades agreement, but the magnitude depends on model and task; furthermore, prompt paraphrasing can substantially reduce stability, with particularly large drops for the MoE model. Finally, we evaluate a practical mitigation, self-consistency via majority voting, which improves κ substantially and often improves or preserves F1-score, at the cost of additional inference. Together, these findings provide a reproducible framework and concrete recommendations for evaluating and improving LLM reliability in clinical IE.
Biomedical named entity recognition (NER) is a high-utility natural language processing task, and large language models (LLMs) show promise in few-shot settings. In this article, we address performance challenges for few-shot biomedical NER by investigating innovative prompting strategies involving retrieval-augmented generation. Using five biomedical NER datasets, we implemented and evaluated a systematically-structured multi-component static prompt and a dynamic prompt engineering technique, where the prompt is dynamically updated via retrieval with most relevant in-context examples based on the input texts. Static prompting with structured components increased average F1-scores by 12% for GPT-4, and 11% for GPT-3.5 and LLaMA 3-70B, relative to basic static prompting. Dynamic prompting further boosted performance and was evaluated on GPT-4, LLaMA 3-70B, and the recently released open-weight GPT-OSS-120B model, with TF-IDF based retrieval yielding the best results, improving average F1-scores by 8.8% and 6.3% in 5-shot and 10-shot settings, respectively. An ablation study on retrieval pool size demonstrated that strong performance can be achieved with relatively small number of annotated samples, reinforcing the annotation efficiency and scalability of our framework in real-world settings.
Hospital discharge reports (HDRs) support continuity of care; yet, their specialized terminology may hinder patient understanding and postdischarge self-management, particularly among individuals with limited health literacy (HL). AIM-HEALTH (Artificial Intelligence-Mediated Discharge Document for Accessible Healthcare) aims to develop and evaluate a clinician-validated, artificial intelligence (AI)-powered supplementary discharge document (SDD) to support comprehension of HDR content, tailored to patients' HL levels. This prospective, observational, noninterventional study will enroll 200 adults from the nephrology and cardiology units at Azienda Socio-Sanitaria Territoriale Spedali Civili hospital. Following written informed consent, participants' HL will be assessed and combined with HDR structured data and an audio-recorded discharge interview to generate two outputs using locally deployed agentic AIs powered by an open-weight large language model: (1) an HL-tailored SDD for patient use and (2) a clinical informational performance report (CIPR) highlighting omissions and inconsistencies between the HDR and the discharge interview to support clinical safety. Clinicians will validate the SDD using a QUEST (quality, understanding, expression style, safety, and trust)-informed tool assessing accuracy, completeness, clarity, utility, and safety domains. All outputs will undergo clinical assessment; only suitable outputs will be retained, while corrections and unsuitable outputs will be used to iteratively refine the system. Patients will assess SDD perceived accessibility, comprehensibility, usefulness, and engagement. Data processing follows on-premise, Data Protection Impact Assessment-defined safeguards (data minimization, pseudonymization, and security or incident management). Expected results include (1) technical feasibility and workflow indicators (completion and SDD withholding rates); (2) clinician-rated accuracy, appropriateness, completeness, and safety of SDDs; (3) perceived utility of the clinical informational performance report for identifying omissions and inconsistencies between HDRs and discharge interviews; (4) patient-reported accessibility, comprehensibility, usefulness, and engagement at ~1-month follow-up; and (5) associations between SDD readability index scores and perceived comprehensibility. This study is approved by the local institutional review board (NP 6761-62, February 24, 2026). Data collection commenced on May 29, 2026. AIM-HEALTH is intended to generate preliminary feasibility, safety, and acceptability evidence on the integration of an AI-generated, HL-adapted SDD into routine workflows, particularly for patients with chronic conditions. The findings are intended to inform the design of future comparative studies addressing the potential role of such tools in postdischarge communication, under on-premise data protection and mandatory clinician validation. PRR1-10.2196/95782.
Accurate risk stratification in oncology is essential for guiding treatment decisions, yet current algorithms rely on a narrow set of structured variables, and hence potentially ignore the rich signal in narrative pathology reports. These reports contain nuanced morphological descriptions and expert clinical judgmentThis narrative information remains largely unused in clinical decision-making as it gets lost in "prose" text-based reports. We hypothesized that large language models (LLMs) could extract prognostic information from complete free-text pathology reports and convert it into a binary survival biomarker. We used the open-weight LLaMA 3.3 70B model to generate risk scores directly from publicly available pathology reports across three gastrointestinal cancer types. The model was prompted to synthesize the complete narrative reports into a binary prognostic score. We evaluated associations between the LLM-generated scores and survival outcomes, including overall survival, progression-free survival, and disease-specific survival. In colorectal cancer, LLM-generated risk scores demonstrated significant prognostic value for overall survival (Hazard ratio (HR) = 2.77, 95% confidence interval (CI) = 1.92-3.97, p < 0.001), progression-free survival (HR = 2.93, 95% CI = 2.11-4.08, p < 0.001), and disease-specific survival (HR = 5.85, 95% CI = 3.66-9.36, p < 0.001). Multivariate analysis confirmed the LLM-generated risk score as an independent prognostic factor for progression-free survival. LLMs can turn narrative pathology reports into a single, independent survival biomarker. This approach leverages routinely available free-text documentation without requiring additional tissue analysis or pathologist workload, providing a deployable method to enhance risk stratification for treatment decision-making.
Background The growth of biomedical literature increasingly exceeds the capacity of manual evidence synthesis. Large language models (LLMs) may support abstract screening and structured extraction, but many current workflows depend on proprietary cloud APIs, creating challenges for governance, reproducibility, and scalable deployment. Methods I developed a fully local, open-weight, schema-constrained pipeline (gpt-oss-20b, deployed via Ollama on Apple M1 Max) for title/abstract-based scoping workflows. The pipeline combined deterministic metadata filtering, LLM-assisted screening, and structured abstract extraction. Performance was benchmarked against three published systematic reviews (ketamine/neuroimaging; clozapine/suicidality; clozapine patient/caregiver perspectives) using precision, recall, and F1 against reference inclusion sets. I also report audit-adjusted estimates (i.e., performance metrics recalculated after manual full-text adjudication of discrepant records) alongside standard reference-set performance. Results In the ketamine/neuroimaging benchmark, the pipeline retained all 41 studies included in the original review; after audit adjustment, recall was 100.0% (46/46), accuracy 99.4% (156/157), precision 97.9% (46/47), and F1 98.9%. For clozapine/suicidality, recall was 79.3% (46/58), and F1 was 76.0%, with missed studies largely attributable to missing or non-informative abstracts. For clozapine patient/caregiver perspectives, recall was 88.9% (56/63), and F1 was 83.6%, with similar abstract-level constraints. Abstract-level extraction recovered audited metadata fields without detected errors and generated evidence maps that were thematically concordant with the main narrative structure of the reference reviews. Conclusions As a proof-of-concept, a fully local LLM pipeline can support scalable and auditable abstract-based scoping and high-level evidence mapping. Because performance was benchmarked against three reviews with partly audit-adjusted reference sets, the findings require confirmation in larger, independently adjudicated evaluations. Random human audit remains advisable, and expert full-text synthesis remains necessary when abstracts are non-informative or when mechanistic precision is required.
Unstructured oncology consultation notes contain rich clinical information that may support survival prediction. Open-weight large language models (LLMs) can utilize these notes with zero-shot inference or fine-tuning, but their relative value for this setting remains unclear. The objective of this study is to evaluate open-weight LLMs for predicting 60-month survival from initial oncology consultation notes, comparing (i) zero-shot performance, (ii) performance after fine-tuning, and (iii) smaller natural language processing models trained on the same dataset in prior work. We used Meta's Llama models to predict patients' 60-month survival using oncology consultation notes from a dataset of 59 800 patients. We tested both zero-shot and fine-tuning approaches. Metrics included balanced accuracy (BA) and weighted F1. Zero-shot performance was limited. Llama-2-13B performed best among the zero-shot configurations (average performance across prompts: BA 0.596, weighted F1 0.644; performance on Prompt 4: BA 0.766, weighted F1 0.802). Fine-tuning improved performance across models: Llama-2-13B achieved BA 0.842, weighted F1 0.846, area under the receiver operating characteristic curve (AUC) 0.905; Llama-2-7B achieved BA 0.840, weighted F1 0.843, AUC 0.911; Llama-3.1-8B achieved BA 0.829, weighted F1 0.829, AUC 0.881. Performance was numerically similar to smaller models trained on the same task and data. For predicting 60-month survival from initial oncology consultation documents, fine-tuning open-weight LLMs meaningfully improves performance compared with zero-shot use, but does not consistently outperform smaller language models. This may suggest that both fine-tuned LLMs and smaller models merit continued investigation, with the most appropriate approach likely to depend on the outcome of interest, clinical context, and practical considerations such as hardware, privacy, and deployment feasibility.
To evaluate the feasibility of a locally deployable large language model (LLM) system for automated MRI protocol selection addressing data privacy, annotation burden, and scalability limitations. This retrospective study included 598 German-language MRI order entries from three neuroradiology domains (brain, head/neck, spine) between June 2018 and January 2023. A radiologist labeled entries for 27 protocol classes based on institutional standard operating procedures (SOP). An SOP-grounded AI system using MedGemma 27B was developed to predict the MRI protocol from the order entry. The system was optimized using Stochastic Introspective Mini-Batch Ascent (SIMBA), a self-reflective prompt optimization algorithm, and compared with a hierarchical system that first classified the body region and then the MRI protocol. Data efficiency was evaluated using training subsets of 10-119 examples across 3 optimization runs per subset size. The flat zero-shot model achieved 73.07% accuracy in the three-domain setting on the held-out dataset (n = 479). In the hierarchical model, prompt optimization yielded 73.90% ± 3.24% with 30 labeled examples and a maximum of 74.46% ± 3.82% but did not outperform the flat approach. Prompt optimization benefited the hierarchical system, whereas the flat model already performed strongly without labeled examples for optimization. Performance on brain and head/neck cases remained broadly stable after expansion from 22 to 27 protocol classes, i.e., including spine. A locally deployable SOP-grounded open-weight LLM can support MRI protocol selection while preserving data privacy and needing minimal labeled data. In this dataset, hierarchical routing and prompt optimization did not improve overall performance over the flat baseline, although they altered optimization behavior and error profile. These findings support prospective evaluation in human-in-the-loop clinical workflows.
Single-cell multimodal omics offer unprecedented resolution of cellular networks, yet translating continuous computational attributions into structured, testable biological mechanisms remains a persistent bottleneck. To address this limitation, we introduce an analytical pipeline employing decision trees to discretize continuous neural network attributions into explicit regulatory thresholds. These boundaries then structurally constrain large language models, enabling them to integrate established literature with empirical data to synthesize context-specific hypotheses. Applying this continuous-to-discrete framework across sparse datasets yielded novel biological mechanisms. Specifically, the framework articulated a cytoskeletal gating hierarchy governing EGF-stimulated pathways, identified transcriptomic drivers of input resistance in cortical interneurons, and delineated translational logic predicting Ki-67 abundance within spatial transcriptomics. Retrospective benchmarking validated the capacity of the framework to autonomously reconstruct published regulatory logic. Supported by a locally deployable open-weight language model and a code-free interface, this approach establishes an auditable methodology to extract robust experimental hypotheses from high-dimensional single-cell data.