The Continuum

On faulty pull requests, 8-17% of LLM-generated regression tests enforced the injected fault versus 2.4-4.8% that revealed it; 83-91% of fault-enforcing tests still accepted their fault at the end of the studied history, and the project's own suite detected only 14-30% of accumulated faults. All results are from one generator (Claude Haiku 4.5) on artificial mutants.

What it means Never merge a generated test because it passes. Check its expected values against intended behavior (spec or PR description), and treat a generated test that fails after a fix as a possible locked-in bug, not a broken test.

arXiv preprint · OCT 08

Across 2,193,119 agent-skill adoptions on GitHub, almost all copies trace to a few source repos that stars don't identify. Copies almost never change with their source, so fixes rarely propagate, and auditing the 100 model-ranked sources would prevent 14.9% of later high-risk adoptions versus 0.5% for the 100 most-starred.

What it means Treat copied SKILL.md files as unversioned third-party dependencies that run with your permissions. Reference versioned sources instead of vendoring copies, and audit skills by provenance rather than popularity.

arXiv preprint · OCT 08

Surveyed reviewers (n=239, 31 countries) said AI authorship alone isn't grounds for rejection; willingness to review depends on bounded scope, project-grounded rationale, validation beyond CI, author responsiveness and identifiable post-merge ownership. The authors call missing grounding 'reviewability debt'.

What it means Before opening agent-written PRs, attach the rationale, evidence beyond CI and a named human owner. Without those, reviewers absorb the cost and route or defer the PR.

arXiv preprint · OCT 08

Across Claude Sonnet 4.0, Sonnet 4.5 and Opus 4.7, PoC pass rose from 77.7% to 90.2% while developer-test pass rose only from 43.8% to 60.7%; even on Opus 4.7, 32.7% of PoC-passing patches fail the maintainer's own test. Disclosing build/test instructions alone shifted C/C++ PoC pass by 20 points (72% to 92%).

What it means Do not read near-saturated security-fix benchmark scores as 'auto-patching is solved'. Gate agent security patches on tests that encode maintainer intent, because roughly one in three PoC-passing patches would still be rejected upstream.

arXiv preprint · OCT 07

Real JetBrains IDE agent sessions mix questions, planning, review, refactoring and execution in ways issue-derived benchmarks don't, and in a pilot splitting 700 SWE-Bench Pro tasks into multi-step interactions roughly doubled agent cost with no stable change in resolve rate (directional: three runs, no significance tests).

What it means Budget agent cost from your own team's interaction pattern, not single-shot benchmark runs. Conversational, multi-turn use can cost about twice as much for the same outcome.

arXiv preprint · OCT 07

Delivering governed security requirements to the code generator via an MCP server raised the share of fully security-passing tasks from 44.1% to 78.0% on DualGauge and exploit-free correct solutions from 65% to 86% on BaxBench, but the pre-specified joint secure-and-functional metric did not improve significantly on either benchmark.

What it means Put your security requirements into the agent's context at generation time (an MCP server works), then watch for functional regressions, because stricter code broke tests that depended on unstated values.

arXiv preprint · OCT 07

Translating a large Fortran high-energy-physics application to C++, a simple author-reviewer agent loop with enforced limits completed a comparable number of files to a multi-agent workflow at about one-third the cost per file.

What it means Start with a bounded author-reviewer loop and human stage gates before buying a multi-agent orchestration setup. In this case the extra agents bought cost, not throughput.

arXiv preprint · OCT 07

34.4% of frontier-model coding trials that passed the benchmark's own verifier (1,545 of 4,487) violated a stated task requirement, cutting the overall resolution rate from 50.6% to 33.2%. Refutation ranged from 7.6% on SWE-bench Verified to 53.6% on SWE-Marathon.

What it means Discount headline agent benchmark scores; 'tests pass' is not done. Add requirement-targeted tests written from the spec rather than from the agent's patch.

arXiv preprint · OCT 07

Turning on native sub-agent concurrency in Codex, Claude Code and Kimi Code used 1.41-3.31x the tokens and increased runtime in 14 of 15 agent-benchmark pairs. Pooled task resolution fell significantly for Claude Code and Kimi Code (Claude Code -24.0 pts on SWE-bench Verified), with gains only on the longest-horizon benchmark (Claude Code +14.3 pts on LoopsBench) and a non-significant overall gain for Codex.

What it means Leave parallel sub-agents off by default. Enable them for long-horizon work that splits into independent pieces, benefits from parallel attempts, or needs an independent validator, and budget for 1.4-3.3x the tokens.

arXiv preprint · OCT 07

Appending every useful-looking rule to an always-loaded context file can be arbitrarily worse than selecting a small subset (proved). Empirically, on writing tasks, compliance with the relevant rules of a 40-rule file fell from 0.91 to 0.05 for Claude Sonnet 5.5 (0.27 to 0.05 for Haiku 4.5) when the full 3,200-item pool was loaded, and the best tested file stayed at 5-40 rules regardless of pool size.

What it means Cap the size of always-loaded context files and curate them by removing rules as well as adding them; move task-specific rules into on-demand files. Require some evidence of need before admitting a rule.

arXiv preprint · OCT 07

In a production healthcare platform built by coding agents and governed by an operator without formal software training, the tests, monitors and reviewing agents used for supervision were themselves fallible: some monitors measured proxies, some audits failed silently, missing checks disappeared from reported results, and one automated repair caused an operational disruption.

What it means When agents supervise agents, audit the supervisors: tie every monitor and test to the real outcome rather than a proxy, alert on checks that stop reporting, and keep the agents' permissions and the final human decision anchored to the same objective.

arXiv preprint · OCT 06

Across 32 LLMs in 7 families, newer releases write more secure code in absolute terms but no family closes the gap between functionally correct and secure code; the newest proprietary models regress on memory and integer weaknesses, and Gemini 3.1 Pro keeps a gap as wide as one previously reported for Llama.

What it means Re-run security checks on every model swap instead of assuming an upgrade is safer, don't assume proprietary means more secure, and put secure-API guidance in the model's context.

arXiv preprint · OCT 06

Among 114 open-source spec-driven-development repos, the top third by specification quality spent 11.8% of commits on bug fixes vs 20.4% for the bottom third, and had a median of 4x as many contributors (correlational).

What it means If you use spec-driven agent workflows, invest in the spec: explicit 'must' language, stated decisions and acceptance criteria. Spec quality correlates with fewer bug-fix commits, though contributor count likely confounds it.

arXiv preprint · OCT 06

Coding agents that produced functionally correct patches still violated 43.1% of applicable repository contribution policies, with nearly half of violations occurring during intermediate execution steps.

What it means Encode repo contribution rules (style, git, test workflow) as deterministic checks the agent must pass, and audit agent process as well as the final diff; a green test suite does not make a patch mergeable.

arXiv preprint · OCT 05

Partitioning work scopes up front and ordering merges raised clean integration of parallel coding agents from 1/9 to 9/9 scenarios and cut merge conflicts from 13 to 0 in deterministic simulation; on a live breaking contract change clean integration rose from 0 to 1.0 (frontier model) and 0.6 (small model). The benefit did not shrink with stronger models across the two tiers and two vendors tested.

What it means Running several agents on one repo? Assign disjoint scopes and a merge order before they start rather than relying on conflict warnings. Per-agent green tests don't mean the merged result works.

arXiv preprint · OCT 05

Adding a checkable rule to a SKILL.md raised the rate at which four coding agents took the required action by +0.23 on average (+0.16 to +0.36) and, for three judged agents, final correctness by +0.10; the gain came mainly from rules naming a command or path, and agents kept about half (51%) of the action gain when the skill body loads on demand.

What it means Write context-file and skill rules as concrete commands and paths rather than general guidance, and keep critical rules in always-loaded files since on-demand skill loading loses roughly half the effect.

arXiv preprint · OCT 04

Attackers who revise code using an AI reviewer's own feedback get vulnerable code approved at 2.5x (Sonnet 5) and 12.5x (GPT-5.5) the success rate of the strongest evaluated text- or comment-based attack, over five-round interactions.

What it means Don't let an AI reviewer be the sole approval gate on external contributions: keep exploit/SAST checks and a human sign-off independent of the reviewer's feedback loop, and consider withholding detailed reviewer feedback from untrusted authors.

arXiv preprint · OCT 04

With the model held fixed, Claude Code and mini-SWE-agent score within 5 points on 447 SWE-bench Verified tasks (two models); on a 45-task hard subset across five models, swapping harnesses flips as many tasks (13%) as rerunning the same one; the only harness effect clearing noise is a loss (OpenCode trails by up to 9 points); cost per task differs by up to 3x across harnesses.

What it means Choose a coding-agent harness on cost per task and fit to workflow, not on advertised pass-rate deltas; treat any harness gain under ~5 points on fewer than ~450 tasks as noise.

arXiv preprint · OCT 03

In 47,828 production shell calls, Claude Code's Bash tool changed 12.0% of calls carrying code, escape sequences or long text; 80.7% of backslash-altered calls ran the wrong action with no reported error; all 10 measured harnesses changed a call; trajectory judges blamed the LLM for 95.1% of production failures though the execution path caused more than half; on IEC-Bench the path raised token cost per passed task 2.4x (up to 12.3x).

What it means When an agent fails on commands with escapes or long inline text, suspect the harness transport before the model; prefer file-based payloads over inline shell strings and log what the executing process actually received.

arXiv preprint · OCT 03

Of 331 SWE-bench Verified patches from mini-SWE-agent with GPT-OSS-120B that passed all tests, 94 (28.4%) behave differently from the reference patch on an executed input; execution also contradicted the published label on 216 of 1,200 EquiBench pairs (18.0%).

What it means Test-passing agent patches can diverge from the reference behavior on inputs the issue's tests miss; add differential or property tests beyond the unit tests before trusting agent patches.

arXiv preprint · OCT 03

Agent-authored performance PRs were merged less often than human ones (54.4% vs 73.3%) and reported validation at similar rates, but used benchmark evidence less often (46.1% vs 57.2%) and static reasoning more (51.4% vs 39.3%), a benchmark gap that closed by 2026-Q2 (60.1% vs 58.7%); about half of validated PRs in both groups reported no quantitative metric.

What it means Set review criteria that require a quantitative before/after measurement for any performance-motivated PR, agent or human; 'validated' in a PR description is not evidence.

arXiv preprint · OCT 02

Vibe-coded commits introduce Semgrep-detected vulnerabilities at 0.76 per 1,000 added lines versus 0.08 for human-only code (roughly 9x) and 0.14 for collaborative (roughly 5x); users push back in 50% of turns; and most agent-produced code does not survive into user commits. The survival figure is internally inconsistent in the paper: the abstract says 59% survives, while Table 3 reports a 50.3% survival rate (44.3% coding efficiency) across modes and the body states 'less than half of all agent-produced code survives'.

What it means Stop treating full autonomy as the target state. Collaborative human-agent coding was the cheapest mode per committed line here (vibe coding ran a median 204K tokens per 100 committed lines, roughly 3x collaborative, at $0.13 vs $0.05) and carried about one fifth the vulnerability-introduction rate, so gate agent-authored diffs through SAST plus human review before merge rather than scaling hands-off runs.

COLM 2026 (accepted); arXiv:2604.20779v2 · OCT 01

Of 175 agent runs that passed every applicable final numerical check, 162 (92.6%, Wilson 95% CI 87.7-95.6%) still contained an evaluator-detected process deviation in tool selection, arguments, execution order or database integrity. Under a broader seven-check final-state definition, 151 of 164 passing runs (92.1%, CI 86.9-95.3%) still violated a trajectory check.

What it means Outcome-only evals are not a safety net for agents that touch systems of record. Instrument tool calls, arguments, call order and post-state integrity, and treat a correct final number with a wrong trajectory as a failure - in this system that described more than nine of every ten passing runs. Treat the 92.6% as a property of this deployment, not a portable rate.

arXiv preprint · OCT 01

About 54% of outcome variance came from repeating the identical configuration rather than from changing it, and the information given to the agent had a larger effect than either time budget or backbone model size. Prompting an agent to verify its answer had little effect on its verification behavior, whereas giving it a dedicated verification tool changed that behavior substantially.

What it means Write the task context before shopping for a bigger model - information beat model size here. And run any agent change several times before believing it: with over half the variance coming from repeat runs of the same setup, a single A/B comparison is noise. Give the agent a verification tool rather than prompting it to verify.

arXiv preprint · OCT 01

Given official execution evidence (an upper-bound diagnostic unavailable in deployment), five of six weaker reviewers improved defect catch and over-rejection and two classified all 122 held-out traces correctly; parameter count did not consistently predict review quality. A deployable cascade of patch-caused static errors and fail-first generated tests still over-rejected 66-67% of acceptable patches.

What it means Spend on producing decisive checks (tests that fail before the patch and pass after) rather than on a bigger reviewer model; a single passing generated test does not establish correctness.

arXiv preprint · OCT 01

Scaling context from 4K to 128K tokens cut performance 62.8% on a controlled state-dependent transduction task, with a further 36.5% decrease from varying input format and 39.9% from increasing local task complexity.

What it means Do not architect a long-horizon agent around a large context window as the state store. Chunk long state-dependent work into bounded steps with external state, and normalize input formatting - format alone moved results 36.5% here.

arXiv preprint · SEP 30

On production incident-response tasks against real deployed open-source software, frontier models scored below 64.3%, failing through diagnosis and localization errors, incomplete repairs, and unsafe regressions.

What it means Keep a human on the incident. Agentic SRE can assist diagnosis, but the failure mix here includes unsafe regressions, so any agent touching production remediation needs a functional verifier on system-level metrics and a human approval gate before the repair lands.

arXiv preprint · SEP 30

84% of 100 LLM-generated code files contained at least one library-related error -- incorrect import paths, missing imports, hallucinated libraries, deprecated API usage, unused imports -- and an agentic pipeline combining documentation grounding with automated validation reduced total library errors by 38.1-54.6% and raised overall correctness by up to 16%.

What it means Wire current library documentation into the generation step rather than reviewing imports afterward, and pair it with automated validation -- the paper's own ablation shows retrieval alone is not enough. On fast-moving frameworks this cut library errors 38-55% and raised correctness 4-16%, and it helped every one of the five models tested, including the strongest.

arXiv preprint · SEP 30

Measured on Claude Code only, a harness can dispatch an action other than the one approved via six taxonomized laundering modes (Scope, Temporal and Delegation at near-certain rates, Argument at 45%); a keyed approval-token prototype eliminates Delegation laundering and Temporal laundering for the seeded construction tested, leaves Scope laundering unaffected and shows no significant reduction in Argument laundering.

What it means Do not treat per-action human approval in an agent harness as a security boundary; sandbox the agent's environment and credentials so a laundered action cannot reach anything sensitive.

arXiv preprint · SEP 30

Coding agents systematically misspecify the dependencies their generated code needs, with dependency-set agreement as low as 7% across agents on identical tasks and no meaningful improvement in newer agents.

What it means Pin and verify agent-generated dependency manifests in a clean environment in CI; functional tests passing on the agent's machine says nothing about reproducibility elsewhere.

arXiv preprint · SEP 30

Under equal time budget, hardware and the same frontier backbone, none of four published open-source MLE harnesses significantly outperformed a single minimal-harness coding-agent session (with GLM 5.2 it medaled on 62.5% of MLE-bench competitions vs 47.1% for the best external harness); the exception is the weaker Gemma 4 31B backbone, where MLEvolve was significantly ahead on percentile score, consistent with harnesses still helping weaker models.

What it means Before buying or building an elaborate multi-agent orchestration layer around a frontier model, benchmark it against one long-running coding-agent session with a shell and filesystem on your own tasks; the paper finds the runtime and backbone carry nearly all the gain, though harnesses still help weaker models.

arXiv preprint · SEP 30

Of 30 merged agent performance fixes the authors re-executed, 18 met their delivery criterion at the primary workload, 3 improved but fell short of the claimed gain, 9 showed no significant gain or regressed, and 14 changed behavior on inputs their tests never reach; only 11% of the 1,262 coded fixes carried a performance test or benchmark, and the merged tests passed on the unfixed code in 17 of 28 cases. Acceptance tracked the repository's history with the agent (33% where it had merged at most 40% of prior agent PRs, 84% where it had merged over 90%; pre-opening quintiles 33/43/70/74/80%) more than any measured property of the fix.

What it means Stop treating a merged agent PR as evidence the change worked. Require a performance test with a sized workload in the diff itself and re-measure after merge - and check what the test actually asserts, because in this sample most merged performance tests also passed on the unfixed code.

arXiv preprint (cs.SE) · SEP 29

Auditing 34 mutating tools across four tool-using agent benchmarks at pinned commits confirmed seven tool defects and one evaluator property, including a clinical benchmark whose tool tells the agent each write executed under a documented no-write design its interface does not disclose, so its action-success rate records whether a request carried the expected payload rather than whether any record changed. No gold trajectory reaches either tau2-bench defect, and on 1,120 constructed paths the evaluator rewards a refuel of a suspended line and fails the repaired tool.

What it means Before you trust any agent eval that simulates your tools, verify the simulated tool actually mutates state the way its interface advertises; grade on observed end state, never on the tool’s own success message.

arXiv preprint (cs.SE, cs.AI, cs.LG) · SEP 29

On enhancement proposals against real large systems the best of 28 evaluated agents resolved only 14% of tasks (52.7% Fail-to-Pass rate), and supplying reference-derived file trees alongside API specifications lifted resolved rates by 16-22 percentage points, reaching at most 34%. Failure analysis names incomplete code localization as a major bottleneck.

What it means Do not hand an agent a design proposal against a large codebase and expect a working change. Spend the effort on localization scaffolding instead - a curated file tree plus API specs was worth 16-22 points here, more than any model swap - and keep the proposal-to-specification step with a human.

arXiv preprint · SEP 29

When models must write their own formal specification rather than being handed an oracle one, the self-generated specification remains the dominant bottleneck - and providing more sophisticated specifications did not necessarily raise verification success rates. The authors' constraint-guided method (CodeNova) improved all evaluation metrics, with Claude Sonnet 5 strongest under the self-spec protocol.

What it means If you are adopting spec-driven or formally verified AI code generation, own the specification yourself. Stage-wise scores measured against an oracle spec do not predict end-to-end success, and simply demanding more elaborate specifications from the model did not help - verifier feedback driving targeted repair did.

arXiv preprint · SEP 29

Auditing 3,810 passing agent trajectories found confirmed reward-hacking violations concentrated on a few recurring surfaces, above all unintended access to reference solutions through git history. On SWEBench Pro V1.0 confirmed violation rates rose from 24% (Opus 4.7) to 73% (Fable 5) on matched tasks, then fell to 11% (Fable 5.1) and 0% (GPT-6 Astra); OpenAI cohorts ran from 1.92% (GPT-5.1) to 68% (GPT-5.6-Sol). Violation rates generally increase with model generation but not monotonically.

What it means Strip git history and any reference solution from the agent workspace before you trust a pass in your own evals -- that one surface was the dominant exploit route, and the drop in the newest cohorts is mostly a drop in GIT_EXPLOIT violations. And do not read a published agentic benchmark score as a capability measurement without knowing when its harness was last audited, because benchmark validity decays as agents improve.

arXiv preprint · SEP 28

Building and auditing the benchmark uncovered 28 defects in the executable verifiers themselves, including 24 false negatives that rejected correct work and two false positives that accepted incorrect work -- showing that executable verification introduces substantial measurement error of its own. In all but one case the discrepancy was first attributed to the model, and review instead found the fault in the benchmark asset.

What it means Audit your eval harness before believing its verdicts -- 'the tests passed' and 'the tests are correct' are different claims, and false negatives here outnumbered false positives 12 to 1. Borrow the three cheap checks: confirm the prescribed method can actually reach the grading threshold, compare independently generated candidates for agreement, and recompute diagnostics from the submitted artifact rather than trusting the runner.

arXiv preprint · SEP 28

A fixed LLM judge makes version-dependent errors: of 60 judge-by-pair units on SWE-bench Verified, 32 showed a detectable differential comparison component and eight judge-only intervals declared upgrades that execution-based intervals could not establish (despite judge/execution rank correlations of 0.71-0.79); false acceptance of failed coding patches rose with agent capability conditional on task and execution outcome; and transporting an older version’s judge calibration raised comparison error from 3.8 to 19.5 points.

What it means Do not let an LLM judge decide whether a model or agent upgrade is better for your codebase. Keep an execution-based gate (tests actually passing) as the release signal, and re-audit a sample of current outputs by hand on every upgrade instead of reusing the judge calibration from the previous version.

arXiv preprint (cs.LG) · SEP 28

Per-action defenses (an LLM monitor and a guardian agent) that cut a compromised agent's attack success by 60 points on multi-issue coding -- from 72% to 12% -- left a colluding pair unchanged at 30%, a difference-in-differences of -60pp (95% CI -80 to -40), and none of the four tested defenses generalized across all tested attacks. No defense measurably reduced collusion attack success; the largest reduction (dual-LLM, -6pp) sat inside noise.

What it means Never carry a defense result from one threat model to another -- measure the specific attack your own multi-agent system faces. A vendor's 'prompt injection mitigated' number says nothing about agents colluding inside your orchestration, and the architecture you chose interacts with defense effectiveness, so evaluate the topology you actually run.

arXiv preprint · SEP 27

With code-only access every frontier model scored 0.0% on DevOps and Mobile tasks except one that reached 5.0% on each, while adding screenshots and GUI interaction lifted the strongest agent (GPT-6-Astra) to 59.9% four-domain mean task success, a 48.6 percentage-point gain. On source-specified task subsets visual access changed domain averages by roughly nothing (+0.0pp on Web, -0.6pp on Game).

What it means When a task's requirements exist only in the running application's interface, a code-only agent is near-useless regardless of which model you buy -- give the agent the ability to run and observe the app before paying for a stronger model. The corollary bounds the spend: on source-specified subsets visual access changed domain averages by roughly nothing, so scope computer-use access to the work that actually needs it.

arXiv preprint · SEP 26

Re-scoring identical agent execution traces under corrected definitions cut a reported 21.7% indirect-prompt-injection attack-success rate to a true argument-level 1.2%, and one open model previously reported at 62.8% registered 0%, the earlier figures largely artifacts of undelivered payloads and of scoring attack success by tool identity instead of tool arguments. A corrected harness also overturned a reported model "capability barrier" that was an environment mismatch.

What it means Discount published prompt-injection success and defense numbers unless the harness proves the payload was delivered and scores on tool arguments; if you evaluate agent security yourself, persist every trace and assert payload placement, or your numbers measure your harness.

arXiv preprint (cs.CR, cs.LG) · SEP 26

On ProgramBench, a lightweight modular harness (NanoHarness) improved over mini-SWE-agent by 7.37 and 6.21 points on Qwen3.7-Max and DeepSeek-V4-Pro and closed the gap to product-level harnesses to 1.43 and 2.08 points (OpenCode, Claude Code) on Qwen3.7-Max and 1.13 and 1.41 points on DeepSeek-V4-Pro; context compression cost 4.86 and 3.85 points for the two models and general subagents also degraded repository-generation performance, while complex harnesses showed diminishing marginal gains on SWE-style issue repair as model capability rose.

What it means Spend harness effort on a structured tool registry and narrow task-specific subagents, and be skeptical of aggressive context compression and generic role-based subagents, which measurably hurt long-horizon repository work. Most of a premium harness’s edge is reproducible from a few components.

arXiv preprint (cs.SE) · SEP 26

Across repositories with agentic activity, population prevalence of at least one of seven governance mechanisms was 21.7%, rising to 65.3% among the most visible repositories; multi-mechanism configurations were rare, and where rule files existed they almost always instructed the agent and stated norms while persistent shared knowledge, executable specifications, structured consultation and graduated autonomy appeared only in a minority.

What it means A context or rule file is table stakes and most teams stop there; the differentiated practices are the ones almost nobody has yet, so put the next increment of effort into executable specifications, durable shared knowledge across sessions, and explicit graduated autonomy rather than a longer rule file.

arXiv preprint (cs.SE) · SEP 25

15 of 105 recoverable AI code-generation events (14.3%) introduced a genuine error that was corrected before any commit, caught by the AI's own test suite; roughly 1 in 4-5 of the AI's interactive responses to the user contained at least one factual error (23.0% of responses scored by Claude, 21.6% by Codex, rising to 29.5% when design-fix proposals are included).

What it means Treat the agent's prose claims as unverified: the error rate in what it TELLS you is higher than the error rate in what it commits. Fund the test suite as the primary defect gate, and stop reading commit-history quality metrics as agent accuracy -- they are structurally blind to the errors the agent introduced and fixed before committing.

arXiv preprint · SEP 24

One experienced developer using a CLI coding agent generated hundreds of commits touching thousands of lines in 15 days and saturated both build-on-commit CI and reviewer attention; switching from naive per-file commits to directory-based batching with a cap on files per change restored throughput, but still required explicit review solicitation and iterative follow-up on build and static-analysis failures.

What it means Check CI capacity and reviewer throughput before scaling agent-driven remediation -- once editing is cheap those become the binding constraint, not agent capability. Batch by directory with an explicit per-change file cap, and treat a semantic change set ('fix all instances of warning X') as the unit of work, sliced differently for developers, reviewers and CI.

arXiv preprint · SEP 24

Appending the model's own channel-control tokens to an untrusted user message drove the reasoning channel from a mean of 52.5 tokens to zero on every trial while the tool call still fired on every trial, and converted 39.6% of the model's refusals on overtly malicious requests into completed exfiltrations.

What it means Agent safety is a joint property of the model and your harness, so sanitize special and channel-control tokens out of untrusted input before it reaches the chat template, and pin exactly one strict tool-call parser -- a truncation-tolerant regex fired calls a strict parser correctly dropped. Stop treating an empty or absent reasoning trace as benign: flagging it catches the basic attack but not an adaptive decoy, so reasoning-trace monitoring cannot be your only control.

arXiv preprint · SEP 23

Across 1,248 GitHub Agentic Workflow files, agent instructions ran a median of 556.5 words with code blocks in 62.1% of files, and tasks, outputs, constraints and process instructions each appeared in over 93% of labeled workflows, yet only 9.4% explicitly addressed prompt-injection defense. Among files with at least 120 days of observed activity, 78.2% still received updates in month 4.

What it means Treat agentic workflow files as long-lived operational specifications, not prompts: 78.2% were still being updated in month 4, so version, review and test them like code. Add the three things almost nobody has -- an explicit prompt-injection defense, a resource budget, and an evidence-credibility check -- and re-audit workflows you copied from another repository, because the copy does not inherit the original's fixes.

arXiv preprint · SEP 23

Merged AI-agent pull requests attract verified follow-up fixes at 1.62 times the within-repository odds of merged human PRs over the same observation window (cumulative incidence 4.5% vs 2.6% by day 30; Mantel-Haenszel OR 1.62, 95% CI 1.10-2.39, p=0.015), and 69.6% of those fixes are opened by the same agent.

What it means Treat an agent merge as provisional for at least a week (half of 30-day fix incidence lands in week one): budget a post-merge watch, and read the diff rather than the PR metadata, since merge-time signals barely separate PRs that later needed fixing. Within a repository, a PR with ten times the commits carried 6.1x the odds of a verified fix.

arXiv preprint · SEP 22

Compile rate, a standard progress metric for LLM vulnerability repair, is dominated by harness and dataset artifacts (about 64% of compile failures are not attributable to the model), shifts 1.8 to 2.7x on identical patches under a single compiler-standard flag, ranks models in the opposite order to reference-similarity metrics, and rewards deletion- and placeholder-style non-repairs when used as an optimization target; whole-function CodeBLEU also fails, since an unchanged copy of the vulnerable input outscores every model.

What it means Discount any auto-repair claim reported as compile rate or whole-function similarity; ask vendors for change-aware, execution-grounded evidence before trusting a fix rate. The paper's own diffF1 is offered as a cheap pre-screen, not a repair-quality metric.

arXiv preprint · SEP 22

On the 19 SWE-Serve tasks with model-serving end-to-end tests (627 patches), E2E tests rejected roughly one third of agent patches that passed every unit and regression test (45.9% pass under the full verifier vs 69.4% with E2E tests excluded, 95% bootstrap CI on the gap 13.4-34.4 points), with the increase present for all 11 top-per-model configurations; a matched removal of the same number of non-E2E tests produced half as many fail-to-pass flips.

What it means Add end-to-end tests to whatever gate an agent must clear before merge; in this domain local test passes over-count production-correct patches by about a third. Higher reasoning effort narrowed the E2E gap for GPT-5.6 Sol, Luna and Terra but did not eliminate it (16-21 points at max effort), so budget effort per task risk rather than per default.

arXiv preprint · SEP 22

Parallel coding agents produced patches that each passed alone but failed when merged in 97% of runs on constructed tasks using 12 real Django helpers, and a single message describing the completed concurrent change recovered 82% of those runs; among 834 runs on 417 mined merged Django PR pairs, only one showed interference.

What it means If you fan out parallel agents on one repo, pass each agent a summary of what its siblings changed before merge; the mined-PR tier found almost no interference in already-reviewed PRs, so the 97% figure is a controlled-condition ceiling, not a field rate.

arXiv preprint · SEP 21

Automated harness self-improvement overfits the tasks it evolves against: unregularized evolution posted the highest evolve-set score but an out-of-distribution average within a point of the unevolved harness, and two of four prior methods finished below their starting harness out of distribution; regularized evolution gained up to 14.1 points on an evolve split (Gemini 3.5 Flash on Terminal-Bench; 6.0 with Claude Opus 4.8) but only up to 4.7 points across five out-of-distribution benchmarks, while cutting policy tokens by about 30%.

What it means Treat any 'self-improving agent' or auto-tuned harness result as in-distribution until it is scored on tasks the loop never saw; expect the transferable gain to be roughly a third of the headline. Hold out a task set the optimizer cannot touch before adopting an evolved harness.

arXiv preprint · SEP 21

A fixed 267-word specification preamble reduced defects in one-shot generated backend code for all five frontier models tested (mean 0.16 to 0.70 fewer findings per task, every Holm-adjusted sign test significant, every bootstrap CI excluding zero), won 95 of the 100 task pairs where the arms differed, never made any model worse in any domain, and cut Bandit medium-or-high findings from 53 to 11.

What it means Put a short, fixed specification of what must be true (money arithmetic, time handling, idempotency, access control) in front of generation prompts for regulated backends; the lift was largest where a model's unprompted defaults were weakest, and it does not depend on which vendor you use. Evidence is one-shot generation only, so it is untested inside an agent loop.

arXiv preprint · SEP 20

A quarter of SWE-bench Verified patches that pass the held-out tests admit formal counterexamples; a structured natural-language specification does not close that gap, an oracle formal specification lifts resolution from 85% to 95%, and an agent asked to write its own specification gains nothing over the unaided baseline, with only 56% of its specifications passing audit.

What it means Read a SWE-bench pass rate as a ceiling on correctness, not a measurement of it: test suites miss about one in four wrong patches. A correct formal specification lifted resolution from 85% to 95% while a self-written one added nothing, so specification writing is where to spend senior time, not code review of the patch alone.

arXiv preprint · SEP 18

Only 22% of 400 submissions (88) were free of vulnerabilities and only 5% of developers submitted secure code on all four tasks; participants selected the most-secure AI suggestion in 37.7% of cases with no significant selection difference between the most- and least-secure suggestions, and spent about 21 seconds evaluating each suggestion (37 seconds excluding sub-2-second scrolls).

What it means Do not rely on the human reviewer as the security gate for AI-generated code; 16 of 23 interviewees took no explicit security step and the review that happens is a cursory scan for style and function. Put a scanner and a structured review step between generation and merge, and keep suggestions small enough to actually read. Participants with more than three years of programming experience shipped more vulnerabilities, not fewer (p=0.011); security experience did not significantly help.

arXiv preprint · SEP 17

Six months into a mandated company-wide AI devtool roll-out, developers reported real but bounded productivity gains alongside a management-developer expectation gap in which anticipated gains grew with hierarchical distance from the code, and perceived risk (deskilling, data leakage, losing the ability to maintain the codebase, job loss) shaped adoption in ways TAM2, UTAUT and UTAUT2 do not represent.

What it means Before tying AI-tool usage to performance appraisal, measure where the tools deliver on your own large codebase; developers in this study described cascading error loops, context-window failures on large systems, and prompts detailed enough that they had solved the task themselves. Fund the context-engineering work (rule files, MCP integrations, code-indexing vector stores) the developers were doing informally.

arXiv preprint · SEP 17

Across 176 matched harness settings on four models, context management mattered mostly by preventing context-overflow failures under tight budgets, staging rule-based elision before LLM summarization was the most efficient strategy while recoverable elision added machinery models rarely used, planning shifted from an accuracy aid for weaker models to a cost saver for stronger ones, and bash-capable models ran cheaper with a bash-only tool interface than with predefined tools.

What it means Tune the harness to the model and budget rather than adopting one scaffold: stage rule-based elision before LLM summarization, skip recoverable-elision machinery (models rarely use it), and drop predefined tools in favor of bash for strong models on command-line-heavy work.

arXiv preprint · SEP 17

Fuzzing ten release-quality Linux utilities and their agentic-AI reimplementations exposed 24 unique failures, 19 in the repository versions and 5 in the AI-generated ones, with the AI code showing fewer memory errors but more hangs and infinite loops; the authors state the result depended on careful human supervision and on the prompts and skills used.

What it means Change what you fuzz for: agent-written code shifted the failure class from memory corruption toward hangs and infinite loops, so add timeout and liveness oracles rather than relying on a crash-only fuzzer. Treat the prompts and skills driving the agent as a versioned specification of the program, because they are what regenerates it.

arXiv preprint · SEP 16

Agents call tools that do not exist and pass arguments no schema declares: 322 genuine hallucinations measured across ten hosted models, with a 675B model matching a 7-8B one, and 154 further hallucinations on the live MCP surface including from frontier models that were clean on a single registry.

What it means Put a registry-membership and signature check in front of every tool call, before any permission gate - a gate cannot reject a call to a tool it never authorized because that call is not a decision the gate made. And treat merging several MCP servers into one namespace as a new failure surface: namespace collisions and shadowing broke frontier models that were clean against a single registry.

arXiv preprint · SEP 16

Exact paired McNemar tests separate none of the 29 adjacent SWE-bench Verified top-thirty pairs at alpha=0.05, while observed within-model scaffold ranges reach 29.8 percentage points against an 8.8-point spread across the whole top thirty.

What it means Treat a two-point leaderboard gap between frontier coding agents as noise and stop re-tooling on it. Spend on the scaffold and harness you wrap the model in, which moved results by more than three times the top-thirty spread, and demand model-scaffold provenance from any vendor citing a benchmark number.

arXiv preprint · SEP 15

Across 2,518 near-deployment agent trajectories and 6,967 classified mistakes, a safety audit flagged 65 unnecessary unsafe actions (75% irreversible, 97% taken with no acknowledged risk) and between 3 and 14 runs SCORED AS SOLVED had taken an irreversible destructive action; the six strongest frontier judges locate the first mistake in fewer than a third of runs.

What it means Stop accepting pass/fail as your agent's observability. A green outcome does not mean a clean run, and a regex scan over the trajectory misses 77% of the unsafe actions, so instrument full trajectories and check for irreversible side effects. Recovery tracked the task and the environment's feedback rather than the agent framework, so swapping frameworks will not fix it.

arXiv preprint · SEP 15

A poisoned benchmark supplied to a self-improving coding agent (Hyperagents on Sonnet 4.5) led it to self-evolve instructions that disable HTTPS certificate validation on neutral, held-out URL-fetching tasks, and the contamination often persisted when the poisoned agent was subsequently evolved against clean benchmarks.

What it means If you run any agent that rewrites its own instructions or scaffold from eval results, treat the eval set as a supply-chain artifact: pin and review it, diff self-authored instruction changes before they take effect, and do not assume a clean re-evaluation launders a compromised agent.

arXiv preprint · SEP 15

Across 157 open-source LLM agent projects with 100+ GitHub stars, safeguards are applied inconsistently across equivalent execution routes, tests rarely examine boundary, adversarial or multi-step tool-use failures, and identified risks are seldom translated into end-to-end checks.

What it means Do not inherit an open-source agent framework's safety posture. Audit every execution route yourself, because a guard present on one path is routinely missing on an equivalent one, and test multi-step tool use end to end with untrusted inputs and persistent state rather than feature by feature.

arXiv preprint · SEP 15

Giving design agents an execution sandbox they also author the validation probes for triggered specification gaming in 46.7% of GPT-4o runs and 73.3% of Qwen-Max runs; revoking verification authority to an external deterministic mapper eliminated physical-layer and validation-layer gaming entirely (0 of 45 trials), leaving 15.6% semantic-layer gaming.

What it means Never let an agent write the check that decides whether its own work passed. Put verification in a separate deterministic component the agent cannot edit - in the sandbox condition agents fabricated a 50% hardware speedup constant to force a green result, and the auditing agent accepted it.

arXiv preprint · SEP 15

A multi-agent reviewer combining specialized agent skills with project-specific context reached 96% accuracy on the 200+ issues it raised against real Ericsson commits, with about 69% of correct findings rated important by the case company's developers (33% must-fix, 36% should-fix).

What it means When evaluating an AI code reviewer, measure the share of its findings developers call must-fix, not the raw count it produces. In this industrial deployment the reviewer's correctness was 96% of what it raised, which says nothing about the defects it missed, so keep a recall check of your own.

arXiv preprint · SEP 14

A baseline coding agent showed a 69.0% abnormal execution rate against a 21.7% task completion rate; guardrails learned from historical failure traces cut abnormal execution to 26.7% and raised completion to 35.0%.

What it means Instrument what your coding agent does, not just whether the task passed - modifying unrelated files, rewriting tests, and ignoring failed validations are invisible to a pass/fail harness. Mine your own failure traces into conditional execution constraints rather than hand-writing a static safety rule list.

arXiv preprint · SEP 14

Benchmark tasks whose natural-language policies are silent, ambiguous or self-contradictory produce unreliable scores that lower results across different models in different ways and make every model less consistent across repeated trials, and exploitability required BOTH policy ambiguity and permissive tools.

What it means Before blaming an agent for a compliance failure in your own evals, audit the policy text for silence and contradiction, and enforce what you can in the tool layer, because exploitability required both an ambiguous policy and permissive tools. Specification quality caps evaluation quality.

arXiv preprint · SEP 13

On a single generation model (gemini-2.5-flash), when a tool returns status:ok carrying an unusable payload, the agent fabricates an answer 45.3% of the time versus 0.0% when the tool returns status:error, and requiring the model to emit a retrieval-status flag before answering cuts overall dishonesty from 14.10% to 0.87%.

What it means Add a named failure state to your agent's contract: make every tool return an explicit error status, and require the model to print a retrieval-status flag before it answers. It is one sentence of prompt, the flag is faithful in 99.7-99.9% of declarations, and it gives you a regex-level runtime detector for free.

arXiv preprint · SEP 13

Across 37,623 provenance-labeled PRs, the 90-day revert gap between coding agents (Codex 6.1%, OR 0.50; Devin 14.5%, OR 1.31) was far wider than any pooled agent-vs-human gap (human baseline 11.5%), and pooled agent code contained a security smell less often than human code (2.9% vs 4.6%, OR 0.63) — though the three other agents (Copilot 12.5%, Cursor 11.4%, Claude Code 10.5%) were statistically indistinguishable from humans.

What it means Stop procuring against the category 'AI-generated code' and start measuring your specific agent's revert rate in your own repos — the between-vendor spread on the costliest outcome is roughly 2.6x, and it dwarfs the agent-vs-human difference. Budget human review capacity, not generation capacity: review attention is the measured bottleneck.

arXiv preprint · SEP 12