共找到 20 条结果
Coding-agent reinforcement learning treats execution infrastructure as a background implementation detail, despite relying on large numbers of interactive software rollouts. This is a missed opportunity: measuring infrastructure overhead can reveal practical efficiency gains for RL post-training, where small per-rollout savings compound at scale. We present a comparative study of four execution substrates: single containers, hosted sandboxes, Kubernetes-orchestrated containers, and cloud virtual machines. We find up to $110\times$ variation in cold-start latency and a $1.8\times$ spread in projected worker-hours for one million 150-step trajectories. Our results suggest that future coding-agent RL systems should optimize execution substrates as part of the training system itself, not merely as deployment plumbing.
Scientific machine learning papers typically make computational claims, e.g., that the relative mean square error is less than 5% or that the 95% predictive credible interval covers the test data. A coding agent can be prompted to replicate those claims from paper materials alone, but the prompt does not by itself reliably preserve progress or check whether generated evidence supports the paper's claims. We introduce Paper-replication, a workflow that makes each selected paper claim a target with recorded evidence, and implement it as a coding-agent skill. The workflow makes the agent record those targets, reconstruct the paper's method, run computational experiments, link generated outputs to provenance and comparisons with the paper's claims, record where matched evidence appears in the replication report, and pass validation checks before completion. We evaluate Paper-replication on twelve independent runs across four scientific machine learning papers. All twelve workspaces pass the completion gate, and all 158 recorded targets are matched with report coverage. Even in this completed workspace state, repeated runs differ in how papers are divided into targets, in numerical fide
Harnesses are now central to coding-agent performance, mediating how models interact with tools and execution environments. Yet harness engineering remains a manual craft, because automating it faces a heterogeneous action space across editable components, voluminous trajectories that bury actionable signal, and edits whose effect is hard to attribute. We introduce Agentic Harness Engineering (AHE), a closed loop that addresses these challenges through three matched observability pillars: (1) component observability gives every editable harness component a file-level representation so the action space is explicit and revertible; (2) experience observability distills millions of raw trajectory tokens into a layered, drill-down evidence corpus that an evolving agent can actually consume; and (3) decision observability pairs every edit with a self-declared prediction, later verified against the next round's task-level outcomes. Together, these pillars turn every edit into a falsifiable contract, so harness evolution proceeds autonomously without collapsing into trial-and-error. Empirically, ten AHE iterations lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-de
Benchmark scores tell you what an agent got right; they do not tell you how it got there. In this work, we introduce methods for comparing agents procedurally in different contexts, where the model, tasks, and approaches vary. We compare ten agents and find that they are identifiable by their behavioral habits, which we define as fingerprints: a probe over these procedural signatures attributes an unseen trajectory to the correct agent at 85.7% accuracy, controlling for leakage across tasks. We develop procedural representations for agent problem-solving procedures with an emergent vocabulary induction technique that is meant to be maximally compressive to avoid surface-level variation while being expressive enough to unveil the quirks of the models' patterns. We apply our framework to the software engineering evaluation dataset SWE-Bench to study the structural distinctness of agent trajectories and find that behavior is most similar between models from similar release periods and those that are distilled from one another (e.g., a distilled student model and its teacher have a Jensen-Shannon divergence of 0.25, about half the distance between other model pairs). As more models sat
AI coding agents increasingly accept assigned software tasks, modify repositories under bounded authority, and return work packages for review. Prior work proposed the software delegation contract, covering the task, authority, returned work package, and acceptance context, as the unit of analysis for delegated coding work, but did not measure its effects. This paper reports a controlled pilot study of explicit delegation contracts for coding agents. We built a dependency-free TypeScript API task environment with seeded defects and documentation gaps, authored ten tasks across five families, and ran 64 agent executions across two model tiers under three conditions: a realistic issue-style prompt, an explicit delegation contract, and a contract with a required evidence bundle. Each run was scored with hidden acceptance tests, mutation checks, and scope analysis, then reviewed by three independent condition-blinded model-based reviewers using a fixed rubric, for 192 reviews. Explicit contracts did not improve objective task outcomes: all 64 runs passed hidden acceptance checks, with zero scope violations. They did improve reviewability. Evidence sufficiency improved in 22 of 30 paire
Configuring an advanced scientific simulator, translating a modeling goal into a valid, runnable input deck, is a persistent bottleneck that costs domain scientists hours to days. Input decks are executable interfaces: simulator-specific vocabulary, cross-file references, schema constraints, and validation rules must align before a simulation can run. We show that this bottleneck can be substantially reduced with a lightweight adapter around an off-the-shelf coding agent, rather than a bespoke simulator agent. Coding agents already navigate files, edit code, run commands, and repair outputs; what they lack is the simulator's executable contract, and rebuilding the agent loop risks discarding harness-calibrated tool-use and self-correction behavior. We introduce SIGA, a coding-agent adapter that supplies this contract through retrieval, procedural memory, agent-callable validation, and validation-gated termination while leaving the model and loop frozen. Because this contract is small and external, SIGA also supports adapter self-evolution: prior trajectories can rewrite the adapter contents without modifying the underlying agent. On GEOS, a multiphysics subsurface simulator, SIGA's
Implementing target trial emulation (TTE) studies as standardized, reproducible analytic workflows is technically demanding. We developed Text-guided Health-study Estimation and Specification Engine Using Strategus (THESEUS), which uses large language models (LLMs) to translate free-text study descriptions into structured analytic specifications and Strategus R scripts within the Observational Health Data Sciences and Informatics (OHDSI) ecosystem. THESEUS executes 2 steps: an LLM maps study descriptions to a JavaScript Object Notation (JSON) schema, and validated specifications are converted into Strategus R scripts through rule-based logic. For standardization evaluation, we compared specifications generated by 8 LLMs using 15 OHDSI-based TTE studies and 15 non-OHDSI studies under primary-analysis and full-analyses settings. Under the primary-analysis setting, overall standardization accuracy ranged from 0.93 to 0.97 across models in OHDSI studies and from 0.82 to 0.95 in non-OHDSI studies. Gemini-3.1-Pro achieved the highest overall accuracy in OHDSI studies, while Gemini-3.1-Pro and Gpt-5.5 jointly achieved the highest overall accuracy in non-OHDSI studies. Under the full-analyses setting, field-level sensitivity ranged from 0.83 to 0.97 in OHDSI studies, with 0.07-0.80 false positives (FPs) per study, and from 0.77 to 0.89 in non-OHDSI studies, with 0.53-1.20 FPs per study. Gpt-5.5 performed best at the field level. THESEUS was implemented as a web application and coding-agent tools. Pairing a standardized data model with a structured analysis framework enables reliable LLM-assisted interpretation of study descriptions and deterministic workflow construction in observational research. THESEUS supports translation of natural language study descriptions into executable, shareable code in standardized observational research settings.
Coding agents powered by large language models (LLMs) have gained traction for automating code generation through iterative problem-solving with minimal human involvement. Despite the emergence of various frameworks, e.g., LangChain, AutoML, and AIDE, ML scientists still struggle to effectively review and adjust the agents' coding process. The current approach of manually inspecting individual outputs is inefficient, making it difficult to track code evolution, compare coding iterations, and identify improvement opportunities. To address this challenge, we introduce a visual analytics system designed to enhance the examination of coding agent behaviors. Focusing on the AIDE framework, our system supports comparative analysis across three levels: (1) Code-Level Analysis, which reveals how the agent debugs and refines its code over iterations; (2) Process-Level Analysis, which contrasts different solution-seeking processes explored by the agent; and (3) LLM-Level Analysis, which highlights variations in coding behavior across different LLMs. By integrating these perspectives, our system enables ML scientists to gain a structured understanding of agent behaviors, facilitating more effective debugging and prompt engineering. Through case studies using coding agents to tackle popular Kaggle competitions, we demonstrate how our system provides valuable insights into the iterative coding process.
Effective management of Water Distribution Networks (WDNs) is essential to ensure efficient and reliable water supply in cities. However, many management tasks require complex system modelling and optimization approaches, which heavily rely on specialized domain expertise and human resources. Recent advancements in Large Language Models (LLMs) offer promising opportunities to automate complex hydraulic decision-making tasks. This study presents an LLM-based agent framework to automate WDN management tasks. Two tasks are considered to evaluate the feasibility and limitations of LLM agents: hydraulic model calibration and pump operation optimization. The key component of the proposed framework is an Orchestrating Agent that interprets tasks and system states, generates update strategies or executable code, and interacts with three specialized agents to carry out implementation: a Knowledge Agent performing reasoning based on hydraulic principles, a Modelling Agent that interfaces with hydraulic simulation tool EPANET, and a Coding Agent that executes code and returns output feedback. To assess the capabilities of these agents, the framework was systematically tested on two benchmark WDNs - Net2 and Anytown. The results indicate that the reasoning capability demonstrated through interaction with the Knowledge Agent effectively replicates expert-level hydraulic thinking, though it lacks numerical precision. In contrast, the Modelling Agent, which integrates external simulation tools, enhances reliability, although interpreting and enforcing numerical constraints expressed in natural language remain challenging, particularly in looped networks such as Anytown where the agent often converged to suboptimal solutions. Furthermore, the Coding Agent, where code for optimization algorithms is iteratively generated and executed, delivers the most consistent and accurate performance across both networks, underscoring its practical potential. These findings highlight the potential of LLM-based agents for automated, accurate hydraulic optimization, and represent a significant step toward LLM-driven multi-agent frameworks for hydraulic decision-making. This work establishes a foundation for future advancements in specialized, domain-focused LLM applications in complex hydraulic management scenarios.
The rapid adoption of multi-agent frameworks for automated code generation has significantly enhanced software development efficiency, yet simultaneously introduced critical security challenges that remain largely unexplored. While extensive research has investigated jailbreaking vulnerabilities in single-agent large language models, existing studies have overlooked the unique security risks arising from collaborative dynamics in multi-agent systems, where distributed decision-making and social interactions may amplify rather than mitigate adversarial threats. To address this gap, we propose the first comprehensive security assessment framework for multi-agent code generation, introducing Implicit Multi-agent Attack (IMA), a novel jailbreaking strategy that exploits social engineering and collaborative reinforcement within agent networks. Our evaluation encompasses four prominent frameworks (MetaGPT, CrewAI, AutoGen, and ChatDev) using the established RMCBench benchmark (Resistance to Malicious Code Benchmark) with 282 malicious code generation tasks across text-to-code, function-level, and block-level completion scenarios. Compared to traditional explicit attacks and single-agent baselines, IMA demonstrates substantially higher effectiveness, achieving an average attack success rate of 89.01% and revealing collaborative harm amplification factors up to 114.9%. The results expose fundamental vulnerabilities in current multi-agent architectures, with defense mechanisms showing alarmingly low detection rates below 30%. Crucially, by isolating the final coding agent via a direct jailbreak baseline (SADJ), we demonstrate that multi-agent collaboration itself amplifies attack success by an average of 11.4% (CADA), confirming that the vulnerability lies in the architecture rather than the underlying model alone.
Large language models (LLMs) show promise for clinical decision support but often struggle with case-specific reasoning. We present Ophtimus-V2-Tx, an 8-billion-parameter ophthalmology-specialized LLM fine-tuned on more than 10,000 case reports. Evaluation is conducted on a pre-collected dataset. Alongside text metrics (ROUGE-L, BLEU, METEOR) and a semantic similarity score, we use CliBench to map outputs to standardized codes (ICD-10-CM, ATC, ICD-10-PCS) and compute hierarchical F1 (L1-L4 and Full), with code mapping used strictly as an evaluation tool. Ophtimus-V2-Tx is competitive with a state-of-the-art general model and stronger in several settings. It improves text metrics (ROUGE-L 0.40 vs. 0.18; BLEU 0.26 vs. 0.05; METEOR 0.45 vs. 0.29) with comparable semantic similarity. On CliBench, it attains a higher full-code score for secondary diagnosis and ties or leads at selected granular levels for primary diagnosis, while medication and procedure results are close with overlapping confidence intervals. Relative to other ophthalmology-tuned baselines, it shows consistently higher text-generation scores. These findings indicate that a compact, domain-adapted model can approach-or in targeted settings, exceed-large general LLMs on clinically grounded outputs while remaining feasible for on-premise use. We also describe an auditable evaluation pipeline (frozen coding agent, identical prompts, hierarchical metrics) to support reproducibility and future benchmarking.
Humans have believed in gods and spirits since the earliest days of the Holocene, and many people still believe in them today. Although the existence of religious belief has been a human constant, the nature and prevalence of religion has changed dramatically throughout human history. Here we describe the emerging science of religious change. We first outline a multilevel framework for studying religious change drawn from theories of socioecological psychology and cultural evolution. We illustrate this framework with four case studies featuring two ancient religious changes (the rise of punitive religions and doctrinal rituals) and two modern religious changes (the rise of atheism and nontraditional religions). We then review useful methods for examining religious change, including ethnographic coding, agent-based modeling, and time-series analysis. Next, we explore future directions, highlighting the need for predictive forecasts, nonlinear models, and non-Western samples. We also outline ten key questions that need to be answered for a fuller understanding of religious change. (PsycInfo Database Record (c) 2021 APA, all rights reserved).
This paper focuses on identifying structural features responsible for resolution of heavy isotope coded peptides during reversed-phase chromatography. This was achieved by using labeled coding agents that varied in structure, number of deuterium atoms, placement of deuterium in the coding agent, and the functional group targeted by the reagent. Six coding agents were examined. Deuterated versions of the coding agents studied included succinic anhydride-2H4, acetic acid 2,5-dioxopyrrolidin-1-yl ester-2H3, propionic acid 2,5-dioxopyrrolidin-1-yl ester-2H5, pentanoic acid 2,5-dioxopyrrolidin-1-yl ester-2H9, [3-(2,5-dioxopyrrolidin-1-yloxycarbonyl)-propyl]-trimethylammonium chloride-2H9, and the commercial ICAT-2H8 reagent. It was found that these labeling agents vary widely in both their absolute and relative contribution to the chromatographic isotope effect. Relative effects were evaluated by normalizing resolution for the number of deuterium atoms in the derivatized peptide. The single, most dominant effect was the placement of deuterium atoms relative to hydrophilic functional groups in the coding agent. It was concluded that the probability of a deuterium atom interacting with the stationary phase of a reversed-phase chromatography (RPC) column and impacting resolution is greatly diminished by placing it adjacent to a hydrophilic group, as explained by solvophobic theory. But peptide size and coding agent size were also seen to correlate inversely with the magnitude of the isotope effect. This effect was explained as being due to the relative size of the coding agent versus that of the coding agent-peptide conjugate.
This paper reports studies comparing the relative degree of sialylation among human serum glycoproteins carrying complex biantennary N-linked, hybrid, and high-mannose oligosaccharides. Comparisons were made by coupling lectin affinity selection with stable isotope coding of peptides from tryptic digests of serum. After proteolysis, samples were split and differentially acetylated with stable isotope coding agents according to either origin or the separation method by which they would be fractionated. A lectin column prepared from Sambucus nigra agglutinin (SNA) was used to select and compare the concentration of sialic acid containing glycopeptides. The relative standard deviation in quantification using this method was 4%. Using this method the concentration of sialic acid containing glycoproteins from a normal individual were compared to those in a pooled serum sample from a large number of normal individuals. It was found that sialylation varied less than 2-fold in all but four or five glycoproteins. Further studies were done on the degree of sialylation within glycoproteins. Samples labeled with the light isoform of the coding agent were applied to a set of serial lectin columns consisting of a concanavalin A (Con A) column coupled to an SNA column for selecting sialic acid appended to glycopeptides with complex biantennary N-linked, hybrid, and high-mannose glycans. In contrast, samples labeled with the heavy isoform of the coding agent were applied to a Con A lectin column alone to select glycopeptides containing complex biantennary N-linked, hybrid, and high-mannose glycans, without regard to sialylation. Glycopeptides thus selected were mixed, deglycosylated by PNGase F, and fractionated by reversed-phase chromatography (RPC). The RPC fractions were then analyzed by ESI-MS. The relative standard deviation of the method was 4%. All glycopeptides identified contained sialic acid except one. Peptides in which the relative abundance of isotopic isoforms was equal were considered to indicate that the protein parent was fully sialylated at that specific glycosylation site.
Coding agents are rapidly becoming a major application of agentic LLMs, but serving them efficiently remains challenging. Progress on this challenge requires understanding real workload patterns, yet the data needed for such analysis is largely absent. Existing public traces and benchmarks do not capture real, day-to-day coding-agent usage across multiple agents and model families for serving-system analysis. To help fill this gap, we collect and release a trace of roughly 4,300 coding-agent sessions, containing about 350,000 LLM steps and 430,000 tool calls from our own day-to-day use of Claude Code and Codex. Our analysis shows that coding-agent workloads feature long autonomous loops, long contexts with short outputs, diverse and heavily-tailed tool calls, and high but imperfect prefix cache hit rates. These findings point to concrete opportunities for optimizing serving, including lower-overhead tool calling, append-length-aware prefill, semantic-aware tool-latency prediction, and improved KV-cache management around human-paced gaps. We release the dataset, trace collection pipeline, and analysis code at https://github.com/uw-syfi/TraceLab.git the project website is https://tra
Interactive LLM agents are becoming part of daily work, but they do not reliably become easier to work with over time: a correction remembered in one session may still be violated in the next. We study this gap between preference access and preference compliance. In tasks derived from anonymized real-user friction cases, Mem0 memory still leaves 57.5% of applicable preference checks violated. We introduce Test-time Rule Acquisition and Compiled Enforcement (TRACE), a drop-in skill-layer pipeline for coding-agent runtimes that mines user corrections, rewrites them as atomic rules, and compiles them into runtime checks that must pass before an agent completes future tasks. Unlike runtime checks written ahead of time by developers, TRACE skills come from the user's own chat corrections. We evaluate TRACE with simulated user-in-the-loop experiments on ClawArena coding-agent tasks and MemoryArena-derived memory-intensive tasks. On ClawArena, TRACE reduces held-out preference violation from 100.0% to 37.6% on in-distribution tasks and from 100.0% to 2.0% on out-of-distribution tasks. On MemoryArena-derived tasks, TRACE reduces in-distribution violation from 100.0% to 60.5% while matching
Coding-agent benchmarks evaluate whether a single uninterrupted agent can resolve a repository issue. Real software work is messier: tasks are interrupted, reassigned, reviewed, and resumed from partial states left by another agent or engineer. We study this missing dimension through \emph{handoff debt}: the rediscovery cost imposed when a predecessor's work is opaque or incomplete. Our takeover protocol interrupts a coding agent at deterministic handoff points, freezes the repository, and evaluates successor agents under four handoff views: repository state only, raw trace, summary notes, and structured notes. Across 75 source tasks, the protocol generates 181 handoff-point tasks and 724 takeover runs per successor model. Across three successor models, context-bearing handoffs reduce median agent events by 20--59\% and cumulative prompt tokens by 42--63\% relative to repository-only takeover. Solved-rate effects are smaller and model-dependent, but efficiency gains are consistent. These findings suggest that coding-agent evaluation should report not only whether a task is solved, but also how costly that work is for another agent to resume.
Coding agents are increasingly used to automate software engineering tasks. To guide their behavior, these agents commonly rely on configuration files, typically named AGENTS. md or CLAUDE. md, which provide instructions about architecture, workflows, coding conventions, and testing practices. Despite their growing importance, little is known about common problems affecting the definition and maintenance of these files. In this paper, we present the first catalog of smells for coding-agent configuration files. To identify such smells, we first conducted a grey literature review and a repository mining analysis. As a result, we identified six configuration smells and proposed automated heuristics to detect them. To evaluate the prevalence of the proposed smells, we analyzed 100 popular open-source repositories containing either an AGENTS. md or a CLAUDE. md file. Our results show that configuration smells are widespread. Lint Leakage was the most common smell, affecting 62% of the files, followed by Context Bloat (42%) and Skill Leakage (35%). We further show that several smells frequently co-occur, particularly Context Bloat, Skill Leakage, and Conflicting Instructions.
We evaluate an initial coding-agent system for ARC-AGI-3 in which the agent maintains an executable Python world model, verifies it against previous observations, refactors it toward simpler abstractions as a practical proxy for an MDL-like simplicity bias, and plans through the model before acting. The system is intentionally direct: it uses a scripted controller, predefined world-model interfaces, verifier programs, and a plan executor, but no hand-coded game-specific logic. The agent-facing prompts, workspace, and controller contain no game-specific code, game-specific prompts, hand-coded heuristics, hidden solutions, or other game-specific information; the same agent and prompts are used across games. Because the coding agent has broad system access, we audit unintended information channels, describe earlier vulnerable harnesses, and explain how the current harness closes observed leakage channels while reducing benchmark-specific information exposure. We report results on the 25 public ARC-AGI-3 games. Each playthrough starts from a fresh agent instance and clean workspace, with no access to files or conversation state from earlier playthroughs. With GPT-5.5 high reasoning eff
Large Language Model (LLM) coding agents have achieved strong results on software engineering tasks, yet repository exploration remains a major bottleneck: locating relevant code consumes substantial token budget and pollutes the agent's context with irrelevant snippets. In most agents, the same model explores the repository and solves the task, leaving exploratory reads and searches in the solver's history. We present FastContext, a dedicated exploration subagent that separates repository exploration from solving. Invoked on demand, FastContext issues parallel tool calls and returns concise file paths and line ranges as focused context. FastContext is powered by specialized exploration models spanning 4B--30B parameters. We bootstrap them from strong reference-model trajectories and refine them with task-grounded rewards for broad first-turn search, multi-turn evidence gathering, and precise citation generation. Across SWE-bench Multilingual, SWE-bench Pro, and SWE-QA, integrating FastContext into Mini-SWE-Agent improves end-to-end resolution rates up to 5.5% while reducing coding-agent token consumption up to 60%, with marginal overhead. These results show that repository explora