Pediatric cataract occurs during the critical period of visual development, and early intervention is essential to avoid irreversible visual impairment. The health literacy and self-management ability of children and their parents directly affect treatment adherence and prognosis. With the rapid development of artificial intelligence, this study aims to evaluate the accuracy, completeness, and repeatability of domestic open-source large language model (LLM) in answering common clinical questions from pediatric cataract patients, and to explore their application potential as an online health information resource tool for pediatric cataract patients. The research team collected real patient questions from mainstream online medical platforms since 2016, and categorized them into 5 major domains: Risk factors, disease diagnosis, symptoms and staging, screening and examinations, treatment and prognosis. After expert review, 40 high-attention questions were finalized and manual reference answers were provided by experts. Four domestic open-source LLM (Kimi chat, Doubao, ERNIE Bot 3.5, DeepSeek) were selected. Each question was asked repeatedly 4 times, including 2 times with a "patient-physician" role prompt. Three cataract specialists with the title of associate chief physician or above scored the answers blindly using a 4-level accuracy scale, 3-level completeness scale, and 3-level reproducibility scale. The evaluation followed a two-stage assessment scheme: Stage 1 preliminarily tested the 4 LLM using 6 questions of recognized lower difficulty; Stage 2 performed a full evaluation of all 40 questions on the highest-scoring LLM from Stage 1. In the first stage of evaluation, regardless of whether role prompts were included, among the 4 LLM, Kimi chat performed the best, followed by Doubao and ERNIE Bot 3.5, and finally DeepSeek. In Stage 1, regardless of role prompting, Kimi chat performed best, followed by Doubao and ERNIE Bot 3.5, with DeepSeek ranking last. The proportion of answers from Kimi chat scoring accuracy=4, completeness=3, and reproducibility=3 was higher than Doubao, ERNIE Bot 3.5, and DeepSeek. In Stage 2, Kimi chat completed all 40 questions. Its median answer length was 531 (277, 1 059) words, significantly higher than the manual reference 369 (162, 707) words (Z=-4.096, P<0.001). However, answer length showed no significant correlation with accuracy or completeness (both P>0.05). Across 240 model responses, the proportions were: accuracy ≥ 3: 83.8%, completeness=3: 77.9%, and repeatability≥70%: 66.7%. 62.1% (149/240) of evaluators selected Kimi chat answers as their top preference. Reasons for not selecting included off-topic responses, controversial suggestions, and redundant information. Domestic open-source LLM, especially Kimi chat, demonstrated relatively good performance in pediatric cataract health education scenarios, providing medical information with good accuracy, completeness, and reproducibility for parents. LLM have great potential in the healthcare field, but information security, hallucination, and bias remain key challenges, and they still cannot replace clinical physicians. In the future, LLM are expected to collaborate with physicians to deliver more efficient and personalized medical services and promote the development of healthcare. 目的: 儿童白内障发生于视觉发育关键期，早期干预对避免不可逆视力损害至关重要。患儿及家长的健康素养及自我管理能力直接影响治疗依从性与预后。目前人工智能快速发展，本研究旨在评估国内开源大语言模型(large language model，LLM)回答儿童白内障患者常见诊疗问题的准确性、完整性及可重复性，探讨其作为儿童白内障患者在线健康信息资源工具的应用潜力。方法: 研究团队从主流互联网医疗平台收集2016年以来患者真实提问，将其归纳为危险因素、疾病诊断、症状与分期、筛查与检查、治疗与预后5大类别。经专家审核最终确定40个高关注度问题并给出人工回答。选取4个国内开源LLM(Kimi chat、豆包、文心一言3.5、DeepSeek)，每题重复提问4次，其中2次加入“患者-医师”角色提示。由3位副主任及以上职称白内障专科医师采用4级准确性、3级完整性及3级重复性量表盲法对所有回答评分。研究采用2阶段评估方案，第1阶段选择公认难度较低的6个题目对4个LLM进行初步测评;第2阶段则对第1阶段得分最高的LLM进行题库中40个题目的完整评估。结果: 在第1阶段评估中，无论是否加入角色提示，在4个LLM中，Kimi chat表现最佳，其次为豆包和文心一言3.5，最后为DeepSeek。Kimi chat准确性评分为4及完整性评分、重复性评分为3的比例均优于豆包、文心一言3.5、DeepSeek。第2阶段评估中Kimi chat完成全部40题评估，其回答平均字数为531(277，1 059)字，显著高于人工的369(162，707)字(Z=-4.096，P<0.001)，但字数与准确性、完整性均无显著相关性(均P>0.05)。在总体240次回答中，准确性≥3分的比例为83.8%，完整性=3分的比例为77.9%，重复性≥70%的比例为66.7%。62.1%(149/240)的评估者首选Kimi chat答案，未选原因包括答非所问、争议性建议及冗余信息。结论: 测评国内开源LLM特别是Kimi chat在儿童白内障健康教育场景下表现较佳，可为家长提供准确性、完整性、可重复性良好的医学信息。LLM在医疗健康领域的应用具有巨大潜力，但存在信息安全、“幻觉”现象和偏见等问题，目前仍无法取代临床医师。未来LLM有望通过与医师协同工作，为患者提供更高效、个性化的医疗服务，推动医疗健康领域的发展。.
使用 AI 将内容摘要翻译为中文，便于快速阅读
使用 AI 分析这篇文章的核心发现、关键要点和深度见解
由 DeepSeek AI 提供分析 · 首次使用需配置 API Key
arXiv · 2025-06-06
arXiv · 2026-01-22
arXiv · 2025-04-05
arXiv · 2026-04-07