Small BenchmarkOn faulty pull requests, 8-17% of LLM-generated regression tests enforced the injected fault versus 2.4-4.8% that revealed it; 83-91% of fault-enforcing tests still accepted their fault at the end of the studied history, and the project's own suite detected only 14-30% of accumulated faults. All results are from one generator (Claude Haiku 4.5) on artificial mutants.
On the Risks of using LLM-Generated Tests for Regression TestingWhat it means Never merge a generated test because it passes. Check its expected values against intended behavior (spec or PR description), and treat a generated test that fails after a fix as a possible locked-in bug, not a broken test.
arXiv preprint · OCT 08
Large BenchmarkAcross 2,193,119 agent-skill adoptions on GitHub, almost all copies trace to a few source repos that stars don't identify. Copies almost never change with their source, so fixes rarely propagate, and auditing the 100 model-ranked sources would prevent 14.9% of later high-risk adoptions versus 0.5% for the 100 most-starred.
Skill Constellations: Tracing the Supply Chain of Agent Skills on GitHubWhat it means Treat copied SKILL.md files as unversioned third-party dependencies that run with your permissions. Reference versioned sources instead of vendoring copies, and audit skills by provenance rather than popularity.
arXiv preprint · OCT 08
Case StudySurveyed reviewers (n=239, 31 countries) said AI authorship alone isn't grounds for rejection; willingness to review depends on bounded scope, project-grounded rationale, validation beyond CI, author responsiveness and identifiable post-merge ownership. The authors call missing grounding 'reviewability debt'.
Who Pays the Review Cost? Triage, Fairness, and Accountability in AI-authored Pull RequestsWhat it means Before opening agent-written PRs, attach the rationale, evidence beyond CI and a named human owner. Without those, reviewers absorb the cost and route or defer the PR.
arXiv preprint · OCT 08
Small BenchmarkAcross Claude Sonnet 4.0, Sonnet 4.5 and Opus 4.7, PoC pass rose from 77.7% to 90.2% while developer-test pass rose only from 43.8% to 60.7%; even on Opus 4.7, 32.7% of PoC-passing patches fail the maintainer's own test. Disclosing build/test instructions alone shifted C/C++ PoC pass by 20 points (72% to 92%).
On the Reliability of LLM-Based Vulnerability Patching BenchmarksWhat it means Do not read near-saturated security-fix benchmark scores as 'auto-patching is solved'. Gate agent security patches on tests that encode maintainer intent, because roughly one in three PoC-passing patches would still be rejected upstream.
arXiv preprint · OCT 07
Large BenchmarkReal JetBrains IDE agent sessions mix questions, planning, review, refactoring and execution in ways issue-derived benchmarks don't, and in a pilot splitting 700 SWE-Bench Pro tasks into multi-step interactions roughly doubled agent cost with no stable change in resolve rate (directional: three runs, no significance tests).
Coding-Agent Benchmarks Should Match Their Users' Task FlowsWhat it means Budget agent cost from your own team's interaction pattern, not single-shot benchmark runs. Conversational, multi-turn use can cost about twice as much for the same outcome.
arXiv preprint · OCT 07
Small BenchmarkDelivering governed security requirements to the code generator via an MCP server raised the share of fully security-passing tasks from 44.1% to 78.0% on DualGauge and exploit-free correct solutions from 65% to 86% on BaxBench, but the pre-specified joint secure-and-functional metric did not improve significantly on either benchmark.
Applying Security by Design at the Point of Execution: How Governed Security Requirements Affect the Security of AI-Generated CodeWhat it means Put your security requirements into the agent's context at generation time (an MCP server works), then watch for functional regressions, because stricter code broke tests that depended on unstated values.
arXiv preprint · OCT 07
Case StudyTranslating a large Fortran high-energy-physics application to C++, a simple author-reviewer agent loop with enforced limits completed a comparable number of files to a multi-agent workflow at about one-third the cost per file.
Designing Collaborative AI-Driven Workflows for Scientific Software EngineeringWhat it means Start with a bounded author-reviewer loop and human stage gates before buying a multi-agent orchestration setup. In this case the extra agents bought cost, not throughput.
arXiv preprint · OCT 07
Large Benchmark34.4% of frontier-model coding trials that passed the benchmark's own verifier (1,545 of 4,487) violated a stated task requirement, cutting the overall resolution rate from 50.6% to 33.2%. Refutation ranged from 7.6% on SWE-bench Verified to 53.6% on SWE-Marathon.
TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator EvolutionWhat it means Discount headline agent benchmark scores; 'tests pass' is not done. Add requirement-targeted tests written from the spec rather than from the agent's patch.
arXiv preprint · OCT 07
Controlled StudyTurning on native sub-agent concurrency in Codex, Claude Code and Kimi Code used 1.41-3.31x the tokens and increased runtime in 14 of 15 agent-benchmark pairs. Pooled task resolution fell significantly for Claude Code and Kimi Code (Claude Code -24.0 pts on SWE-bench Verified), with gains only on the longest-horizon benchmark (Claude Code +14.3 pts on LoopsBench) and a non-significant overall gain for Codex.
When Sub-Agents Work in Parallel: The Promises and Pitfalls of Dynamic Concurrency in Long-Horizon Coding TasksWhat it means Leave parallel sub-agents off by default. Enable them for long-horizon work that splits into independent pieces, benefits from parallel attempts, or needs an independent validator, and budget for 1.4-3.3x the tokens.
arXiv preprint · OCT 07
Small BenchmarkAppending every useful-looking rule to an always-loaded context file can be arbitrarily worse than selecting a small subset (proved). Empirically, on writing tasks, compliance with the relevant rules of a 40-rule file fell from 0.91 to 0.05 for Claude Sonnet 5.5 (0.27 to 0.05 for Haiku 4.5) when the full 3,200-item pool was loaded, and the best tested file stayed at 5-40 rules regardless of pool size.
Curating Always-Loaded Context for LLM Agents: A Capacitated Assortment Model with Censored FeedbackWhat it means Cap the size of always-loaded context files and curate them by removing rules as well as adding them; move task-specific rules into on-demand files. Require some evidence of need before admitting a rule.
arXiv preprint · OCT 07
Case StudyIn a production healthcare platform built by coding agents and governed by an operator without formal software training, the tests, monitors and reviewing agents used for supervision were themselves fallible: some monitors measured proxies, some audits failed silently, missing checks disappeared from reported results, and one automated repair caused an operational disruption.
A Case Study in Assuring AI-Written SoftwareWhat it means When agents supervise agents, audit the supervisors: tie every monitor and test to the real outcome rather than a proxy, alert on checks that stop reporting, and keep the agents' permissions and the final human decision anchored to the same objective.
arXiv preprint · OCT 06
Large BenchmarkAcross 32 LLMs in 7 families, newer releases write more secure code in absolute terms but no family closes the gap between functionally correct and secure code; the newest proprietary models regress on memory and integer weaknesses, and Gemini 3.1 Pro keeps a gap as wide as one previously reported for Llama.
Newer and Bigger, but Safer? A Longitudinal Study of the Functionality-Security Gap in LLM-Generated CodeWhat it means Re-run security checks on every model swap instead of assuming an upgrade is safer, don't assume proprietary means more secure, and put secure-API guidance in the model's context.
arXiv preprint · OCT 06
Case StudyAmong 114 open-source spec-driven-development repos, the top third by specification quality spent 11.8% of commits on bug fixes vs 20.4% for the bottom third, and had a median of 4x as many contributors (correlational).
The EPIC Framework for Spec-Driven DevelopmentWhat it means If you use spec-driven agent workflows, invest in the spec: explicit 'must' language, stated decisions and acceptance criteria. Spec quality correlates with fewer bug-fix commits, though contributor count likely confounds it.
arXiv preprint · OCT 06
Large BenchmarkCoding agents that produced functionally correct patches still violated 43.1% of applicable repository contribution policies, with nearly half of violations occurring during intermediate execution steps.
Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding AgentsWhat it means Encode repo contribution rules (style, git, test workflow) as deterministic checks the agent must pass, and audit agent process as well as the final diff; a green test suite does not make a patch mergeable.
arXiv preprint · OCT 05
Small BenchmarkPartitioning work scopes up front and ordering merges raised clean integration of parallel coding agents from 1/9 to 9/9 scenarios and cut merge conflicts from 13 to 0 in deterministic simulation; on a live breaking contract change clean integration rose from 0 to 1.0 (frontier model) and 0.6 (small model). The benefit did not shrink with stronger models across the two tiers and two vendors tested.
Verifying Coordination in Parallel Coding Agents: NP-Bench and a Scheduling PlannerWhat it means Running several agents on one repo? Assign disjoint scopes and a merge order before they start rather than relying on conflict warnings. Per-agent green tests don't mean the merged result works.
arXiv preprint · OCT 05
Controlled StudyAdding a checkable rule to a SKILL.md raised the rate at which four coding agents took the required action by +0.23 on average (+0.16 to +0.36) and, for three judged agents, final correctness by +0.10; the gain came mainly from rules naming a command or path, and agents kept about half (51%) of the action gain when the skill body loads on demand.
Agent Skill Evolution: How Revisions Affect Coding AgentsWhat it means Write context-file and skill rules as concrete commands and paths rather than general guidance, and keep critical rules in always-loaded files since on-demand skill loading loses roughly half the effect.
arXiv preprint · OCT 04
Large BenchmarkAttackers who revise code using an AI reviewer's own feedback get vulnerable code approved at 2.5x (Sonnet 5) and 12.5x (GPT-5.5) the success rate of the strongest evaluated text- or comment-based attack, over five-round interactions.
Adaptive Code Revision Attacks on AI Pull Request ReviewersWhat it means Don't let an AI reviewer be the sole approval gate on external contributions: keep exploit/SAST checks and a human sign-off independent of the reviewer's feedback loop, and consider withholding detailed reviewer feedback from untrusted authors.
arXiv preprint · OCT 04