共找到 20 条结果
As AI models evolve, their application in specialized fields like colorectal cancer requires rigorous validation. This pilot study aimed to comparatively assess the knowledge retention, safety, and reasoning limitations of six advanced AI chatbots using a constrained zero-shot multiple-choice question format. 137 text-based MCQs covering 12 core colorectal cancer modules were adapted from the 2023 Chinese guidelines and administered to Gemini 3 Pro Preview, GPT-5.1, Kimi K2 Thinking, DeepSeek V3.2, Qwen3-Max, and Claude Opus 4.5 under zero-shot conditions with a prompt prohibiting reasoning steps. Both quantitative statistical analysis and qualitative error analysis were performed. Overall accuracy was low: Kimi K2 Thinking 27.74%, SD = 0.45, Claude Opus 4.5 26.28%, SD = 0.44, Gemini 3 Pro 16.06%, SD = 0.37, DeepSeek V3.2 15.33%, SD = 0.36, GPT-5.1 14.60%, SD = 0.35, and Qwen3-Max 13.87%, SD = 0.34. Significant module-wise disparities emerged, with Kimi K2 scoring 37.04% in Endoscopic Imaging versus Qwen3-Max at 7.41%. Qualitative analysis revealed four failure patterns: semantic association bias, hierarchical logic failure, fact retrieval error, and hallucinations. No correlation existed between item difficulty and accuracy. Under constrained prompts, next-generation AI chatbots demonstrate unsatisfactory colorectal cancer performance, often relying on keyword matching rather than physiological simulation. This leads to dangerous clinical errors, highlighting the critical need for chain-of-thought prompting, expert oversight, and domain-specific fine-tuning before unsupervised use.
There are significant differences in the diagnosis and treatment of chronic non-bacterial osteitis (CNO), and there is an urgent need for health education efforts to enhance awareness of this condition. Deepseek V3, Doubao, and Kimi1.5 are highly popular language models in China that can provide knowledge related to diseases. This article aims to investigate the accuracy and reproducibility of the responses provided by these three artificial intelligence (AI) language models in answering questions about CNO. According to the latest expert consensus, 16 questions related to CNO were collected. The three AI language models were separately asked these questions at three different times. The answers were independently evaluated by two orthopedic experts. Among the responses of the three AI models to 16 CNO-related questions across three rounds of testing, only Doubao received "Completely incorrect" ratings (accounting for 6.25%) in the third round of scoring by Reviewer 2. During the answering process, Doubao had the shortest response time and provided the most words in its answers. In the first and third rounds of scoring by the first expert, Kimi scored the highest (3.938 ± 0.342, 3.875 ± 0.873), while in the second round, Doubao scored the highest (3.875 ± 0.5). In the second round of scoring by the second expert, Doubao received the highest score (3.812 ± 0.403). In the first and third rounds, Kimi1.5 received the highest score (3.812 ± 0.602, 3.812 ± 0.704). Deepseek V3, Doubao, and Kimi1.5 are capable of answering most questions related to CNO with good accuracy and reproducibility, showing no significant differences.
Pediatric cataract occurs during the critical period of visual development, and early intervention is essential to avoid irreversible visual impairment. The health literacy and self-management ability of children and their parents directly affect treatment adherence and prognosis. With the rapid development of artificial intelligence, this study aims to evaluate the accuracy, completeness, and repeatability of domestic open-source large language model (LLM) in answering common clinical questions from pediatric cataract patients, and to explore their application potential as an online health information resource tool for pediatric cataract patients. The research team collected real patient questions from mainstream online medical platforms since 2016, and categorized them into 5 major domains: Risk factors, disease diagnosis, symptoms and staging, screening and examinations, treatment and prognosis. After expert review, 40 high-attention questions were finalized and manual reference answers were provided by experts. Four domestic open-source LLM (Kimi chat, Doubao, ERNIE Bot 3.5, DeepSeek) were selected. Each question was asked repeatedly 4 times, including 2 times with a "patient-physician" role prompt. Three cataract specialists with the title of associate chief physician or above scored the answers blindly using a 4-level accuracy scale, 3-level completeness scale, and 3-level reproducibility scale. The evaluation followed a two-stage assessment scheme: Stage 1 preliminarily tested the 4 LLM using 6 questions of recognized lower difficulty; Stage 2 performed a full evaluation of all 40 questions on the highest-scoring LLM from Stage 1. In the first stage of evaluation, regardless of whether role prompts were included, among the 4 LLM, Kimi chat performed the best, followed by Doubao and ERNIE Bot 3.5, and finally DeepSeek. In Stage 1, regardless of role prompting, Kimi chat performed best, followed by Doubao and ERNIE Bot 3.5, with DeepSeek ranking last. The proportion of answers from Kimi chat scoring accuracy=4, completeness=3, and reproducibility=3 was higher than Doubao, ERNIE Bot 3.5, and DeepSeek. In Stage 2, Kimi chat completed all 40 questions. Its median answer length was 531 (277, 1 059) words, significantly higher than the manual reference 369 (162, 707) words (Z=-4.096, P<0.001). However, answer length showed no significant correlation with accuracy or completeness (both P>0.05). Across 240 model responses, the proportions were: accuracy ≥ 3: 83.8%, completeness=3: 77.9%, and repeatability≥70%: 66.7%. 62.1% (149/240) of evaluators selected Kimi chat answers as their top preference. Reasons for not selecting included off-topic responses, controversial suggestions, and redundant information. Domestic open-source LLM, especially Kimi chat, demonstrated relatively good performance in pediatric cataract health education scenarios, providing medical information with good accuracy, completeness, and reproducibility for parents. LLM have great potential in the healthcare field, but information security, hallucination, and bias remain key challenges, and they still cannot replace clinical physicians. In the future, LLM are expected to collaborate with physicians to deliver more efficient and personalized medical services and promote the development of healthcare. 目的: 儿童白内障发生于视觉发育关键期，早期干预对避免不可逆视力损害至关重要。患儿及家长的健康素养及自我管理能力直接影响治疗依从性与预后。目前人工智能快速发展，本研究旨在评估国内开源大语言模型(large language model，LLM)回答儿童白内障患者常见诊疗问题的准确性、完整性及可重复性，探讨其作为儿童白内障患者在线健康信息资源工具的应用潜力。方法: 研究团队从主流互联网医疗平台收集2016年以来患者真实提问，将其归纳为危险因素、疾病诊断、症状与分期、筛查与检查、治疗与预后5大类别。经专家审核最终确定40个高关注度问题并给出人工回答。选取4个国内开源LLM(Kimi chat、豆包、文心一言3.5、DeepSeek)，每题重复提问4次，其中2次加入“患者-医师”角色提示。由3位副主任及以上职称白内障专科医师采用4级准确性、3级完整性及3级重复性量表盲法对所有回答评分。研究采用2阶段评估方案，第1阶段选择公认难度较低的6个题目对4个LLM进行初步测评;第2阶段则对第1阶段得分最高的LLM进行题库中40个题目的完整评估。结果: 在第1阶段评估中，无论是否加入角色提示，在4个LLM中，Kimi chat表现最佳，其次为豆包和文心一言3.5，最后为DeepSeek。Kimi chat准确性评分为4及完整性评分、重复性评分为3的比例均优于豆包、文心一言3.5、DeepSeek。第2阶段评估中Kimi chat完成全部40题评估，其回答平均字数为531(277，1 059)字，显著高于人工的369(162，707)字(Z=-4.096，P<0.001)，但字数与准确性、完整性均无显著相关性(均P>0.05)。在总体240次回答中，准确性≥3分的比例为83.8%，完整性=3分的比例为77.9%，重复性≥70%的比例为66.7%。62.1%(149/240)的评估者首选Kimi chat答案，未选原因包括答非所问、争议性建议及冗余信息。结论: 测评国内开源LLM特别是Kimi chat在儿童白内障健康教育场景下表现较佳，可为家长提供准确性、完整性、可重复性良好的医学信息。LLM在医疗健康领域的应用具有巨大潜力，但存在信息安全、“幻觉”现象和偏见等问题，目前仍无法取代临床医师。未来LLM有望通过与医师协同工作，为患者提供更高效、个性化的医疗服务，推动医疗健康领域的发展。.
This study aims to evaluate the efficacy of large language models (LLMs) in health management for urological and andrological conditions by comparing two Chinese LLMs: DeepSeek and Kimi. The responses of DeepSeek and Kimi to 30 questions on six urological and andrological diseases were quantitatively assessed using quality assessment tools, expert assessments, and readability metrics. In quality assessments, DeepSeek outperformed Kimi in only two instances: responses to "What are the surgical options for kidney stones?" (PEMAT-AI: P = 0.012; DISCERN-AI: P = 0.024) and "What are the treatment modalities for bladder cancer?" (PEMAT-AI and DISCERN-AI: P < 0.001). Expert assessments revealed that DeepSeek demonstrated superior accuracy and safety compared to Kimi only in addressing "Dietary considerations for kidney stone patients" (accuracy: P < 0.001; safety: P = 0.02). No statistically significant differences were found in quality assessments or expert assessments between the models for disease-specific categories. However, Kimi outperformed DeepSeek in readability evaluations. Both DeepSeek and Kimi exhibit substantial efficacy and clinical applicability in managing urological and andrological diseases. To enhance their practical use, Kimi requires improvements in accuracy and safety, while DeepSeek should focus on improving readability.
With the increasingly widespread application of artificial intelligence technology, generative artificial intelligence has become an important tool for people to obtain health information due to its convenience and flexibility in health education or health promotion. However, the readability and accuracy of such AI-generated materials still need to be evaluated. To comprehensively evaluate and compare the quality and readability of health education texts about diabetes generated by different generative artificial intelligence (AI) models. We followed a fixed list of ten questions without modifications, systematically presenting the same inquiries to seven generative AI models and exporting their results into defined forms in the text generation process. Five experts were invited to evaluate the texts based on five criteria. The readability index, a readability formula, was used to evaluate the text's readability. Kendall's coefficient of concordance was employed to assess inter-rater reliability. The linear mixed model was used to compare the differences in five dimensions and readability among the health education texts generated by different AI models. Kimi-K1.5 and Doubao attained the highest overall scores in scientific accuracy, whereas iFlytek Spark-V3.5 received lower scores compared to other models. In terms of practical value and logical clarity, Kimi-K1.5 received the highest scores, while iFlytek Spark-V3.5 scored the lowest. In the dimension of reference basis, Kimi-K1.5 and ERNIE Bot-3.5 received relatively high scores, while iFlytek Spark-V3.5 and Doubao scored lower. In the assessment of text readability, higher R-value scores indicate poorer readability. The health education text generated by Doubao had the highest R-value, while iFlytek Spark-V3.5 had the lowest R-value. Kimi-K1.5 performed better across multiple assessment parameters in the overall evaluation of diabetes-related health education texts created by different generative AI models. Notably, among all the models tested, iFlytek Spark-V3.5 showed the best readability.
To assess the quality, readability, and actionability of hypertension patient education materials generated by six selected patient-facing large language model (LLM) platforms, and to characterize cross-platform heterogeneity to guide the optimization of AI-generated patient education materials. Six publicly accessible or commonly accessible platforms were evaluated using standardized prompts to generate patient education materials. Understandability and actionability were assessed using the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P). Information quality was evaluated using the expanded Ensuring Quality Information for Patients scale (EQIP-36), and overall quality was rated using the Global Quality Score (GQS). Readability was compared using seven metrics, including the Flesch Reading Ease Score (FRES) and the Flesch-Kincaid Grade Level (FKGL). Overall educational quality and understandability were generally favorable, but substantial cross-platform heterogeneity was observed. Readability remained challenging, with low FRES values and grade-level indices generally exceeding commonly recommended thresholds for patient education materials. Qwen3-Max-Thinking-Preview achieved the highest PEMAT-P total score (77.00), followed by ChatGPT 5.2-Thinking (72.22). For EQIP-36, Qwen3-Max-Thinking-Preview scored highest (49.28), followed by DeepSeek-R1 (45.48). DeepSeek-R1 generated the most readable materials among the evaluated platforms, with a median FRES of approximately 42.14 and FKGL of approximately 10.10, whereas Qwen3-Max-Thinking-Preview showed lower readability, with a median FRES of approximately 16.82 and FKGL of approximately 14.38. Kimi K2 showed high PEMAT-P understandability (76.90) but low actionability (10.00). Post hoc analyses showed that Qwen3-Max-Thinking-Preview significantly outperformed ERNIE Bot 4.5 Turbo, Doubao, and Kimi K2 on PEMAT-P total score, and outperformed Kimi K2 and ERNIE Bot 4.5 Turbo on EQIP-36. DeepSeek-R1 also outperformed Kimi K2 and ERNIE Bot 4.5 Turbo on EQIP-36. Across content domains, actionability was significantly higher for Daily Care and Prevention than for Basic Understanding of the Disease and Complications, Psychological and Social Aspects. Generative AI shows promise for hypertension patient education, particularly in improving understandability. However, actionability remains a major limitation of current outputs, highlighting the need for platform-aware optimization strategies that explicitly strengthen step-by-step and action-oriented guidance.
Congenital cataract (CC) is a time-critical cause of preventable childhood visual impairment. After diagnosis, parents frequently experience uncertainty and increasingly seek guidance online. The safety, readability, and counseling quality of large language models (LLMs) responses for CC remain insufficiently benchmarked, particularly for explanations involving lens development, etiology, and genetic risk. We performed a cross-sectional comparative evaluation of five publicly accessible Chinese conversational LLMs (ChatGPT-5.2, Gemini 3 Pro, DeepSeek-V3.1, Doubao, and Kimi K2). Thirty standardized parent-facing CC questions were developed by senior ophthalmologists and mapped to five domains, with specific incorporation of scenarios requiring translation of lens developmental pathology and genetic counseling knowledge. Two researchers independently performed standardized zero-shot querying and response recording under identical conditions. Output efficiency and textual structure were extracted. Two blinded ophthalmologists rated each response on a 5-point Likert scale across Accuracy, Logic, Coherence, Safety, and Content Accessibility; inter-rater agreement was assessed using quadratic weighted Cohen's kappa. Group differences were tested using ANOVA or Kruskal-Wallis H tests with Bonferroni-corrected pairwise comparisons. Significant between-model differences were observed in output efficiency and text characteristics (all P < 0.001). ChatGPT-5.2 was fastest (17.94 ± 5.11), whereas DeepSeek-V3.1 and Kimi K2 were slowest (41.46 ± 3.22 and 40.02 ± 4.67). DeepSeek-V3.1 generated the longest responses (1,456.93 ± 224.99 words) and Kimi K2 the shortest (640.83 ± 252.95). ChatGPT-5.2 showed the strongest tendency toward structured/tabular output [2.00 (1.00, 2.00)] followed by Gemini 3 Pro [1.00 (1.00, 1.25)], while the other models rarely produced tables. Quadratic weighted Cohen's kappa indicated good inter-rater reliability (0.686-0.767). Content quality differed significantly across models (Accuracy H = 41.15, Logic H = 32.95, Content accessibility H = 41.33; all P < 0.001). ChatGPT-5.2 and Gemini 3 Pro achieved higher overall profiles and did not differ significantly from each other, whereas Kimi K2 scored lower on multiple dimensions. LLM performance in translating lens developmental pathology and genetics for CC parent counseling is model-dependent. Longer outputs did not necessarily translate into higher quality; structured presentation was more closely associated with better safety and accessibility. These findings provide quantitative benchmarks for safer, parent-centered deployment of LLMs in pediatric ophthalmology education and support more reliable translation of complex disease-related knowledge into actionable parent guidance.
Axial spondyloarthritis (axSpA) is a chronic autoinflammatory disease with heterogeneous clinical features, presenting considerable complexity for sustained patient self-management. Although the use of large language models (LLMs) in health care is rapidly expanding, there has been no rigorous assessment of their capacity to provide axSpA-specific health guidance. This study aimed to develop a patient-centered needs assessment tool and conduct a systematic evaluation of the quality of LLM-generated health advice for patients with axSpA. A 2-round Delphi consensus process guided the design of the questionnaire, which was subsequently administered to 84 patients with axSpA and 26 rheumatologists. Patient-identified key concerns were formulated and input into 5 LLM platforms (GPT-4.0, DeepSeek R1, Hunyuan T1, Kimi k1.5, and Wenxin X1), with all prompts and model outputs in Chinese. Responses were evaluated using 2 techniques: an accuracy assessment based on guideline concordance, with independent double blinding by 2 raters (interrater reliability analyzed via Cohen κ), and the AlphaReadabilityChinese analytic tool to assess readability. Analysis of the validated questionnaire revealed age-related differences. Patients younger than 40 years prioritized symptom management and medication side effects more than those older than 40 years. Distinct priorities between clinicians and patients were identified for diagnostic mimics and drug mechanisms. LLM accuracy was highest in the diagnosis and examination category (mean score 20.4, SD 0.9) but lower in treatment and medication domains (mean score 19.3, SD 1.7). GPT-4.0 and Kimi k1.5 demonstrated superior overall readability; safety remained generally high (disclaimer rates: GPT-4.0 and DeepSeek-R1 100%; Kimi k1.5 88%). Needs assessment across age groups and observed divergences between clinicians and patients underline the necessity for customized patient education. LLMs performed robustly on most evaluation metrics, and GPT-4.0 achieved 94% overall agreement with clinical guidelines. These tools hold promise as scalable adjuncts for ongoing axSpA support, provided complex clinical decision-making remains under human oversight. Nevertheless, the prevalence of artificial intelligence hallucinations remains a critical barrier. Only through comprehensive mitigation of such risks can LLM-based medical support be safely accelerated.
This study aimed to evaluate the comprehensive performance of five large language models (LLMs), namely ChatGPT, DeepSeek, Doubao, Kimi, and Qwen, in addressing medication consultation inquiries for people living with HIV (PLWH) in a Chinese-language context, thereby providing evidence for their clinical application and further model optimization. A total of 55 real-world medication consultation questions covering mainstream antiretroviral drugs for PLWH were screened and classified from Beijing Youan Hospital, Capital Medical University, a specialized infectious disease hospital in China. Five LLMs were queried within a fixed period, and expert evaluations were conducted across five dimensions: accuracy, relevance, completeness, clarity, and reliability. The comprehensive scores ranked from highest to lowest were DeepSeek (4.47), Qwen (4.33), Kimi (4.24), Doubao (4.13), and ChatGPT (3.41), with highly significant differences were observed among all models (H=182.14, p < 0.001). Regarding dimensional scores, the ranking was clarity (4.53) > reliability (4.33) > completeness (4.13) > accuracy (3.95) > relevance (3.62). ChatGPT exhibited statistically significant differences compared with all other models (p < 0.001); no significant differences were found between DeepSeek and Qwen or between Doubao and Kimi (p > 0.05). Significant differences were observed in the capacity of the five LLMs to address medication consultations for PLWH within the Chinese-language context. DeepSeek and Qwen achieved optimal overall performance, Doubao excelled in clarity, whereas ChatGPT yielded the poorest results. All models demonstrated significant limitations when handling complex pharmaceutical inquiries and cannot fully replace professional clinical pharmacists. Further optimization focusing on high-quality medical domain dataset training and algorithm refinement is therefore warranted.
This study evaluated the performance of four popular large-scale language models (ChatGPT o3-mini, Gemini 2.0 pro experimental, Deep Seek Thinking R1, and Kimi Thinking K1.5) in addressing frequently asked patient questions about cataracts and cataract surgery in Chinese. DeepSeek Thinking R1 performed comparably to Gemini 2.0 pro experimental in accuracy, while outperforming both ChatGPT o3-mini and Kimi Thinking K1.5. In terms of completeness and consistency, DeepSeek Thinking R1 showed superior performance over the other three LLMs. Regarding legibility and safety, DeepSeek Thinking R1, Gemini 2.0 pro experimental, and ChatGPT o3-mini exhibited comparable results, all performing better than Kimi Thinking K1.5. Deep Seek Thinking R1 demonstrated the strongest overall performance among the four LLMs in this comparative evaluation. The modern LLMs are promising tools for public education in ophthalmology while human oversight is still required.
The rapid evolution of large language models (LLMs) in the medical field, particularly in automating medical tasks and supporting diagnosis and treatment, has shown promising potential. However, their accuracy, comprehensiveness, and safety in managing complex cardiovascular diseases have not been systematically assessed. This study aims to evaluate and compare the diagnostic, therapeutic, and safety performance of two large language models-ChatGPT-4o and Kimi-in managing complex cardiovascular diseases, and to explore their potential for future clinical application. A total of 200 complex cardiovascular cases published in JACC: Case Reports between January 2020 and August 2024 were included. All cases were standardized and de-identified before being input into ChatGPT-4o and Kimi using identical prompts. Each model independently generated diagnostic, treatment, and long-term management plans. Three cardiovascular specialists independently evaluated the outputs in a blinded manner, scoring accuracy and comprehensiveness using Likert scales. Safety was assessed using a risk matrix analysis. Additionally, 50 cases were randomly selected for triangulation to compare model-generated recommendations with clinical guidelines. Statistical analysis was performed using the Wilcoxon signed-rank test with Benjamini-Hochberg correction for multiple comparisons. In preliminary diagnostic accuracy, the two models performed similarly (P = 0.663, r = 0.044), but ChatGPT-4o showed superior comprehensiveness (P < 0.001, r = 0.484). For treatment recommendations, ChatGPT-4o outperformed Kimi in both accuracy (P = 0.004, r = 0.321) and comprehensiveness (P < 0.001, r = 0.644). In long-term management, ChatGPT-4o demonstrated significant advantages in accuracy (P < 0.001, r = 0.717) and comprehensiveness (P < 0.001, r = 0.690). Safety assessment showed a lower proportion of high-risk outputs with ChatGPT-4o (1.5%) compared to Kimi (4.5%). LLMs, particularly ChatGPT-4o, exhibit significant promise in the diagnosis and treatment of complex cardiovascular diseases, showing superior accuracy, comprehensiveness, and safety compared to Kimi. Despite their high accuracy and safety, LLMs still require clinician oversight, especially in the formulation of personalized treatment plans and complex decision-making scenarios, to ensure their reliable integration into clinical practice.
Diabetic retinopathy is a leading cause of preventable vision loss, and patients increasingly seek disease related information through online consultations. Large language models may support patient education, but their reliability and usability vary across systems, particularly in disease specific settings. Thirty common patient questions about diabetic retinopathy were developed from guidelines and organized into five domains: disease overview, screening and diagnosis, treatment and follow up, lifestyle and prevention, and prognosis and complication management. From November 10 to 15, 2025, two researchers independently submitted all questions to five models (ChatGPT-5, DeepSeek-V3.1, Doubao, Wenxinyiyan 4.5 Turbo, and Kimi) on public platforms under identical conditions without system prompts. Chat histories were reset before each question. Response time, response length, structural metrics, and table outputs were extracted. Two retinal specialists rated each answer on a 1 to 5 Likert scale across accuracy, logical consistency, coherence, safety, and content accessibility. Inter rater agreement was assessed with the intraclass correlation coefficient. Group differences were analyzed using analysis of variance or the Kruskal-Wallis H test with Bonferroni corrected pairwise comparisons. Significant between model differences were observed in output efficiency and textual characteristics (all P < 0.001). ChatGPT-5 responded fastest (15.92 ± 4.48 s), whereas Wenxinyiyan 4.5 Turbo and DeepSeek-V3.1 were slowest (41.89 ± 5.09 s and 38.20 ± 2.96 s). DeepSeek-V3.1 generated the longest answers (1396.37 ± 189.23 words), while Kimi produced the shortest (579.40 ± 182.96 words). Only ChatGPT-5 consistently generated structured tables (median 2.00, IQR 1.00-2.00). Content quality differed significantly across all five dimensions (H = 15.34-37.19, all P ≤ 0.004). ChatGPT-5 achieved the highest median scores for accuracy (5.00, IQR 4.00-5.00) and logical consistency (4.50, IQR 4.00-5.00), whereas Kimi showed the lowest accuracy (3.50, IQR 3.00-4.00). The intraclass correlation coefficient indicated good inter rater reliability (0.87). Performance of large language models in diabetic retinopathy patient consultations is model dependent. ChatGPT-5 demonstrated the best overall usability, combining faster responses, clearer structure, and higher factual accuracy. Other Chinese optimized models provided comparable professional information coverage but require improved accessibility and stability for safe patient facing use.
Digital tools are reshaping public health education and training, yet evidence on whether large language models (LLMs) can generate specialist ophthalmic teaching materials remains limited. High myopia (HM), a vision-threatening condition with long-term management needs and public health relevance, provides a suitable setting for evaluating this capability. This study compared five LLMs in generating HM-related multiple-choice questions (MCQs) for ophthalmic education. Five LLMs (ChatGPT-5.4, Gemini 3, DeepSeek, Kimi K2.5, and Doubao) completed 60 predefined HM MCQ generation tasks each, yielding 300 MCQs. A standardized blueprint covered four domains: basic knowledge, clinical cases, diagnosis and treatment decision-making, and screening/follow-up management. Objective evaluation included structural completeness, format compliance, keyed-answer accuracy, output features, and response time. Two ophthalmology experts rated six domains using 5-point Likert scales, and Spearman analyses examined associations among text features, response time, and expert ratings. All models achieved 100.0% initial structural acceptability, structural completeness, and format compliance. Keyed-answer accuracy was highest for ChatGPT-5.4 and Gemini 3 (both 100.0%), followed by DeepSeek (98.3%) and Kimi K2.5 and Doubao (both 95.0%). Significant between-model differences were observed across all output features and response time (all p < 0.001). ChatGPT-5.4 generated the shortest stems, Gemini 3 the shortest explanations and fastest responses, and Kimi K2.5 and Doubao the longest explanations and total outputs. Inter-rater agreement was good (ICC range, 0.835-0.885). Significant differences were found in clarity, distractor quality, and mean subjective score (all p < 0.001), but not in content rigor, educational usefulness, cognitive-level alignment, or overall usability. DeepSeek achieved the highest median mean score, while direct usability was highest for ChatGPT-5.4 (91.7%) and Gemini 3 (90.0%). Content rigor was strongly associated with overall usability (ρ = 0.85, p < 0.05), whereas distractor quality was negatively associated with explanation length (ρ = -0.43, p < 0.05) and total output length (ρ = -0.37, p < 0.05). LLMs can reliably generate structurally valid HM-related MCQs under standardized Chinese prompting conditions. Their value may lie in supporting digital ophthalmic education and public health training, although expert oversight remains necessary because meaningful differences persist in factual accuracy, distractor quality, and direct usability.
Chronic obstructive pulmonary disease (COPD) requires personalised, guideline-based management according to the Global Initiative for Chronic Obstructive Lung Disease (GOLD) Report. As large language models (LLMs) are increasingly used for medical guidance, their adherence to updated clinical recommendations requires systematic evaluation. This study assessed and compared ChatGPT-5, DeepSeek-V3.2, Gemini 3, Grok, Manus, and Kimi K1.5 in delivering 2025 GOLD-consistent responses, focusing on accuracy, consistency, and clinical reasoning. A standardised dataset of 90 questions derived from the 2025 GOLD Report was categorised into YES/NO, multiple-choice, and open-ended formats. During a 7-day longitudinal study in December 2025, all questions were administered daily across the six platforms, generating 3,780 responses. Performance was evaluated against the 2025 GOLD Report, and open-ended responses were assessed for quality and clinical comprehensiveness. Performance was assessed across two separate primary outcomes. For binary accuracy (YES/NO and MCQ questions), Kimi K1.5 achieved the highest rate (97.1%; 408/420; 95% CI, 95.1-98.4%), and Gemini 3 the lowest (89.5%; 376/420; 95% CI, 86.2-92.1%). For open-ended clinical reasoning quality, Gemini 3 achieved the highest weighted score (96.5%; 95% CI, 93.3-98.4%) and DeepSeek-V3.2 the lowest (91.8%; 95% CI, 87.4-94.9%), reflecting a performance inversion between the two outcome domains. All platforms exhibited statistical stability across the seven-day study period (p > 0.05 for all models). Large language models showed high but format-dependent adherence to the 2025 GOLD COPD guidelines, with clear divergence between binary accuracy and open-ended reasoning. While they may serve as supplementary tools under expert supervision, generalization to other settings requires further validation. Residual performance gaps persisted across both domains, with binary error rates ranging from 2.9% (Kimi K1.5) to 10.5% (Gemini 3), and open-ended weighted-score deficits ranging from 3.5% (Gemini 3) to 8.2% (DeepSeek-V3.2).
Venous thromboembolism (VTE) is a major cause of maternal morbidity and mortality, and nursing plays a central role in prevention, patient education, and follow-up. Large language models (LLMs) have attracted increasing attention in healthcare; however, their comparative performance in maternal VTE nursing contexts remains insufficiently explored. Five representative LLMs-DeepSeek, GPT-4.1, Claude 3.7, Huatuo, and Kimi-were evaluated across six clinical domains (etiology, diagnosis, treatment, prognostic assessment, home care, prevention) and five performance dimensions (accuracy, comprehensibility, logical coherence, reliability, safety). An expert-informed Delphi framework comprising 41 items guided the evaluation. Three nursing experts independently rated each model's responses, and inter-rater reliability was assessed using Fleiss's Kappa. GPT-4.1, Claude 3.7, and DeepSeek demonstrated superior overall performance, particularly in patient education, individualized care planning, and preventive guidance. Huatuo and Kimi showed limitations in treatment and prognostic reasoning. Inter-rater reliability was excellent (Kappa = 0.892). The findings highlight relative strengths and limitations of different LLMs across nursing-relevant domains in maternal VTE care. While certain models performed better in educational and supportive contexts, the current study does not assess clinical adequacy or readiness for real-world nursing deployment. Future research incorporating patient perspectives and real-world validation is needed to inform the safe and appropriate integration of LLMs into nursing practice.
To compare the differences in content quality, readability, and actionability of patient education texts for self-management of chronic heart failure (CHF) generated by five mainstream large language models (LLMs) in China, and to provide a basis for platform selection and assessment framework construction for clinical use. A standardized set of 20 questions was developed based on literature review, guidelines, and consensus from cardiovascular experts, covering disease awareness, diagnosis and classification, treatment and rehabilitation, daily management and prevention, and psychosocial dimensions. Using a uniform prompt, responses were generated by DeepSeek-R1, Doubao, ERNIEBot 4.5 Turbo, Qwen3-Max-Thinking-Preview, and Kimi K2. The PEMAT-P scale was used to assess understandability and actionability, 36-item expanded EQIP (EQIP-36 score) scale was used to evaluate information completeness and standardization, and Global Quality Score (GQS) was used to assess overall quality. Additionally, seven readability formulas, including Flesch Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL), were computed for comparison. Overall quality was high [GQS median 5.00 (4.00-5.00)] with significant between-platform differences (χ2 = 14.47, P = 0.006). Doubao and Kimi K2 achieved the highest GQS [both 5.00 (5.00-5.00)]. DeepSeek-R1 showed the greatest information completeness [EQIP-36 39.20 (36.17-44.23); χ2 = 25.07, P < 0.001] but the lowest readability [FRES 19.32 (17.94-36.89) and FKGL 14.28 (13.02-15.85); both P < 0.001]. ERNIEBot 4.5 Turbo and Qwen3-Max-Thinking-Preview were most readable (FRES ≈ 59; FKGL ≈ 8; both P < 0.001) but had lower EQIP-36 scores. Actionability was limited overall [PEMAT-P actionability 20.00% (0.00-40.00); χ2 = 26.40, P < 0.001] and varied by topic, with daily management and prevention outperforming disease knowledge and diagnosis/classification (χ2 = 20.86, P < 0.001). LLMs show potential for use in patient education for CHF, but there is a structural trade-off between information detail and readability, as well as gaps in actionability and verifiability. It is recommended to combine enhanced search and structured template generation strategies, and establish a governance feedback loop involving prompt engineering, clinical expert review, and continuous monitoring to improve readability alignment, completeness of action instructions, and patient safety.
This study investigates the integration of different large language models (LLMs) into the Medical Cell Biology Laboratory Course (MCBLC) to enhance bioethics training for undergraduate medical students in China. It further compares the effectiveness of these LLMs in improving teaching outcomes and student learning performances. Key challenges encountered during implementation were identified, and potential strategies to address them were also explored. First-year undergraduate medical students from three medical majors were assigned to five groups. The study involved three phases: instructor-led course introduction, LLM-assisted experimental practice addressing procedural, conceptual, and psychological challenges, and post-training evaluation via questionnaires and blind-graded laboratory reports. Four domestic robust LLMs (DeepSeek, Doubao, KIMI, ChatGLM) were compared to assess their impact on bioethics integration, instructional effectiveness, and student learning outcomes, while documenting students' perceptions and concerns regarding LLM use. The study demonstrated that all four LLMs supported first-year undergraduate medical students in consolidating foundational knowledge, enhancing bioethics proficiency during laboratory practice, and developing critical competencies for future physicians. Questionnaires from 86 students across three majors indicated generally high satisfaction. For Medical Imaging Technology students, DeepSeek (mean 4.3, SD 0.7) and KIMI (mean 4.3, SD 0.8) were rated significantly higher than Doubao (mean 3.9, SD 0.7) and ChatGLM (mean 3.3, SD 0.6). KIMI was also preferred among Health Surveillance and Quarantine (mean 4.4, SD 0.5) and Medical Prevention (mean 4.5, SD 0.5) students. Nevertheless, students expressed concerns regarding potential academic inaccuracies, bias, and possible impact on independent thinking. This study suggested that recent LLMs, particularly KIMI and DeepSeek, may support integrating bioethics into undergraduate medical laboratory courses in a university in China. By assisting students in accessing information, reflecting on ethical issues, and navigating practical challenges, these tools can facilitate learning and foster ethical awareness, competent future physicians. These findings, as an initial exploration and context-specific, indicate that LLMs may support bioethics learning in undergraduate medical laboratory courses and help foster ethically aware, competent future physicians.
Perinatal medication consultation is a core clinical pharmacy service that involves a complex benefit-risk assessment for both maternal and fetal safety. Large language models (LLMs) have emerged as potential tools to improve access to medication information, yet their performance and safety in real-world, pharmacist-led perinatal consultation settings, particularly in non-English contexts, remain insufficiently evaluated. To evaluate and compare the performance of multiple advanced large language models in addressing real-world Chinese perinatal medication consultation queries and to assess their potential role as supervised adjunctive tools within clinical pharmacy services. This cross-sectional study evaluated seven LLMs using real-world clinical data from pharmacist-led medication consultations at the Pharmacy Clinic of the Beijing Obstetrics and Gynecology Hospital, Capital Medical University. A standardized test set of 64 perinatal medication consultation questions was developed from 15,280 electronic consultation records collected between April 2014 and April 2024. The evaluated models included international (GPT-5.1, Grok 3, Gemini 3.0) and domestic (DeepSeek, Wenxin Yiyan, Kimi K2, Tongyi Qianwen) models. Senior clinical pharmacologists independently assessed responses across four dimensions-relevance, accuracy, usefulness, and empathy-using a 10-point Likert scale. Results are reported primarily as median (IQR), with mean ± SD additionally provided as a secondary descriptor to facilitate comparison with prior literature. Among the 448 model-generated responses, inter-rater consistency was excellent (ICC = 0.91, 95% CI 0.88-0.94). Significant differences in overall performance were observed among the models (Kruskal-Wallis H = 187.4, p < 0.001; ε2 = 0.41, large effect). GPT-5.1 achieved the highest median total score [9.3 (IQR: 8.8-9.6); mean ± SD: 9.1 ± 0.8], outperforming all other models (all Bonferroni-corrected p < 0.01; all r > 0.50, large effect sizes), followed by Kimi K2 [8.5 (IQR: 7.9-9.1); mean ± SD: 8.4 ± 1.2] and DeepSeek [8.3 (IQR: 7.6-8.9); mean ± SD: 8.2 ± 1.1]. Tongyi Qianwen demonstrated the lowest overall performance [6.7 (IQR: 5.9-7.4); mean ± SD: 6.8 ± 1.3]. Accuracy was the primary determinant of performance differences. Performance gaps were more pronounced in complex clinical scenarios involving comorbidities or benefit-risk trade-offs, whereas domestic models demonstrated relative advantages in consultations involving traditional Chinese medicine. LLMs have demonstrated variable performance in response to perinatal medication consultation queries. While high-performing models show potential to support pharmacist-led perinatal medication consultations by improving access to information, their current performance supports use only as supervised, adjunctive decision-support tools rather than independent sources of medication counseling, with human oversight essential prior to broader integration.
Hypersensitivity pneumonitis (HP) is a complex, immüne mediated interstitial lung disease in which accurate diagnosis and long term management require integration of clinical, radiologic, and exposure-related information. Patients increasingly use artificial intelligence (AI) based chatbots to obtain disease related information; however, the quality, readability, and patient usability of such content remain unclear. This study aimed to evaluate the quality, reliability, readability, and patient-centered usability of AI chatbot generated information on HP. Using Google Trends, we identified four of the most frequently searched patient-oriented questions regarding HP: (1) What is HP and what causes it? (2) What are the clinical features of HP? (3) How is HP treated? (4) How is HP diagnosed? These questions were submitted verbatim to eight AI chatbots (ChatGPT-5.1, Claude 3, Microsoft Copilot, DeepSeek V3, Gemini Pro, Grok 4, Kimi K2, Perplexity AI). A total of 32 responses were independently evaluated in a blinded fashion by four pulmonology professors specializing in interstitial lung diseases. Content quality and reliability were assessed using DISCERN; understandability and actionability with PEMAT-P; global written readability with the Written Readability Rating (WRR); and structural readability with the Flesch-Kincaid Grade Level (FKGL). All chatbot outputs required advanced literacy, with FKGL scores ranging from 20.17 to 29.07 and a mean of approximately 24-25, indicating college or postgraduate reading level. No chatbot produced content within the recommended patient-appropriate range (FKGL ≤ 8). WRR scores declined with increasing clinical complexity, from 67.85 for definitional content (Q1) to 51.227 for diagnostic explanations (Q4). DISCERN scores varied substantially across models (35.001-57.103), with most chatbots falling into the "fair-good" range, reflecting partially reliable but incomplete information. [..] Conclusion: AI chatbots can generate clinically rich explanations of HP but currently produce content that is too complex and insufficiently actionable for most patients. [..].
Standard echocardiography reports use complex terminology, limiting patient comprehension and exacerbating preconsultation anxiety. Large language models (LLMs) can transform technical data into patient-friendly narratives by incorporating longitudinal comparisons with prior examinations. This study aims to develop an LLM-based patient-friendly echocardiography reporting system and evaluate its professional safety, patient comprehension, and impact on short-term anxiety. This study consisted of 2 stages. In the retrospective development stage, 60 patients were included. Clinical diagnosis, hospitalization records, and serial echocardiographic data were integrated as model inputs using DeepSeek-V3.2. Generated reports followed a standardized 4-module structure. Report quality was independently evaluated by 2 clinicians and an external LLM (Kimi 2.5, Moonshot AI) across 4 domains: data accuracy, information completeness, appropriateness of interpretation, and reasonableness of recommendations. In the prospective clinical evaluation stage, 100 patients undergoing echocardiography and 85 family members were enrolled. Participants received both conventional and LLM-generated patient-friendly reports. A 5-point Likert scale assessed helpfulness in understanding results, effectiveness in addressing concerns, helpfulness in improving disease-related knowledge, and anxiety relief. Anxiety was measured using the STAI-6 (6-item short-form State-Trait Anxiety Inventory) at 3 time points: after echocardiography, after conventional report release, and after reading the patient-friendly report. All 60 reports in the retrospective stage were successfully generated. Professional evaluation showed high overall quality scores from both clinicians and the external LLM, with no significant difference between evaluators (mean total scores 18.15, SD 1.36 vs 18.28, SD 1.26; P=.55). One hallucination event was identified. In the prospective stage, all 100 patients received patient-friendly reports. Both patients and family members rated the reports highly, with no significant between-group difference in total scores (17.61, SD 1.60 vs 17.62, SD 1.03; P=.95). Subgroup analyses showed greater perceived benefit among older patients and outpatients (both P<.001); these subgroup findings should be considered exploratory given the lack of adjustment for multiple comparisons. Patients with chronic heart failure, reduced left ventricular ejection fraction (≤40%), and left ventricular enlargement (>55 mm) reported higher scores for addressing concerns (all P<.001). Anxiety scores increased significantly after conventional report release and decreased significantly after reading the patient-friendly report (both P<.001). Older patients (>60 y) and outpatients showed significantly higher anxiety change rates than their counterparts (both P<.001). The reduction in anxiety was positively correlated with subjective anxiety relief ratings (r=0.531; P<.001). The LLM-based patient-friendly echocardiography reporting system with longitudinal comparison demonstrated good feasibility and promising preliminary clinical usefulness. While maintaining high professional quality, it was associated with improved patient understanding of echocardiographic findings and was associated with reduced short-term anxiety, particularly among older adults and outpatients. Causality cannot be inferred from this nonrandomized sequential design, and longer-term outcomes remain to be evaluated.