The Continuum

Benchmarks sharing the same label (e.g. 'bug fix') differ statistically on at least two of three task-demand axes (Spread–Novelty–Centrality), agent success concentrates uniformly in low-demand tasks for every model family and scale, and the behavioral signature of success is model-family-specific (Claude succeeds by matching gold-solution scope, Qwen by exceeding it).

What it means When comparing agent scores across benchmarks, check what the benchmark actually demands rather than trusting its category label — and expect deployed agents to succeed mostly on low-spread, low-novelty, low-centrality tasks, reserving the hard cross-cutting changes for humans.

arXiv preprint · SEP 01

On 203 real-world dependency-upgrade tasks with hidden code-level breakage (DEPBENCH), the best coding-agent configuration solved only 51.2% (104/203), with substantial variation across harnesses, models, and ecosystems.

What it means Do not hand agents unattended dependency upgrades — the maintenance work teams most want to delegate is precisely where agents fail half the time; keep upgrade PRs behind full test suites and human review, and expect ecosystem-specific gaps.

arXiv preprint · AUG 31

How a user invokes a coding agent changes its vulnerability to poisoned repositories by up to 4.5x in attack success rate across task types (Run-Tests 45.5% ASR vs Fix-Bug 8.6%), with test-execution tasks forming a silent attack surface (high attack success, low agent alerting).

What it means Treat 'run the tests' in an untrusted third-party repo as the riskiest agent instruction, not the safest — sandbox agents when bootstrapping from external repos, and know that prompt phrasing and supplied rules shift the attack surface rather than eliminating it.

arXiv preprint · AUG 31

A single factor explains 74.5% of common variance across twelve frontier benchmarks and substantially tracks model release date (R²=0.505), so most of the gap between models released months apart is calendar; under the pre-registered dimensionality rule the four economic benchmarks form no distinct capability factor — though a leave-one-benchmark-out test shows a multi-factor representation still predicts held-out economic scores better than a single general index, so they carry incremental predictive information without constituting a distinct latent capability.

What it means Date-adjust before buying on leaderboard gaps: a small score difference between contemporaneous models is mostly noise around a time trend, not a capability difference — pick by trying models on your own workload, not by leaderboard deltas.

arXiv preprint · AUG 29

Contamination types are best organized by which mitigation each defeats (direct, derivative, temporal, distributional, acquired) — a private held-out test set closes only the first — and in an audit of 41 evaluation documents, elicitation budgets were reported in just 13% of documents and none addressed all five types.

What it means When a benchmark score informs a buy decision, ask which contamination mitigations the eval actually applied — a held-out private test set defeats only direct leakage, and current reporting practice almost never discloses enough to tell capability from leakage.

arXiv preprint · AUG 29

Rewriting SWE-bench tasks to match how real users phrase requests cuts resolution rates by 6.4 percentage points on average and can change model rankings; 88% of real prompts carry only a bare problem statement (alone or with limited context) versus 7% of benchmark problems, and stating Desired Behavior and Motivation measurably lifts performance while Environment Information and Reproduction Steps add tokens without benefit.

What it means State the desired behavior and the motivation in every agent request — those two fields measurably lift resolution while reproduction steps and environment info just add tokens; and discount SWE-bench scores as an upper bound, since real phrasing costs ~6 points and can reorder models.

arXiv preprint · AUG 28

A manager-worker scaffold over a shared filesystem helps some models dramatically (+8 to +30 points on the 100 hardest recent LiveCodeBench problems, up to +42 in single-pass configs) and is null or negative for others (-1 to -9), roughly triples token cost, but buys accuracy more cheaply than upgrading to a larger model in the cases studied.

What it means Test multi-agent scaffolds per-model before adopting: the gain is conditional, largest for smaller models and reasoning-off configs, and near-zero for large reasoning models — and compare the ~3x token bill against simply buying the bigger model, which this study found is often the worse deal.

arXiv preprint · AUG 27

LLM-generated backend services that pass functional tests still show statistically significant memory-growth trends under 48-hour sustained execution in most application-language combinations, indicating software-aging defects that correctness testing does not catch — and the aging trends also appear in some human-written implementations.

What it means Soak-test AI-generated services before deploying them as long-running processes — passing the test suite says nothing about memory behavior at hour 40; add memory-trend monitoring to acceptance criteria for generated backends.

arXiv preprint · AUG 26

Teams absorbing high-volume AI code generation are shifting from line-by-line review toward layered supervision: machine-readable architectural conventions as preventive guardrails, linting/testing/CI repurposed as supervision infrastructure, and human review refocused on architectural reasoning and operational explainability.

What it means If AI generation volume is straining code review, don't add reviewers — externalize architectural intent into machine-checkable rules and let CI carry first-pass supervision, reserving humans for architecture and maintainability judgment.

arXiv preprint (accepted, ESEM 2026 SEIP track) · AUG 26

On end-to-end tasks drawn from market-validated AI-startup workflows, the strongest evaluated agent completes only ~30% of StartupBench despite substantial partial progress, with complex instruction-following and domain expertise as the main failure sources.

What it means Scope agent deployments to the ~30% of real end-to-end workflows they can actually finish; keep humans on complex multi-step instructions and domain-specific steps, and validate against market-representative tasks rather than researcher-chosen benchmarks.

arXiv preprint · AUG 18

Vibe-coding tools (Lovable, v0, Replit) produce structurally distinct and uneven code quality from a single prompt: Lovable concentrates lower-severity issues but has a much higher code-smell density per KLOC, while v0 and Replit produce more aggressive severity profiles.

What it means Treat vibe-coded output as a first draft carrying tool-specific structural debt: run static analysis and budget remediation before shipping, and pick the tool on its quality trade-off, not just perceived speed.

arXiv preprint · AUG 17

A scan-fix-rescan pipeline cuts static-analyzer findings in AI-generated code by 29-69% across four Claude models, but remediation itself introduces new vulnerabilities in 15-22% of cases, and the best code-generation model (Opus 4.8) was not the best pipeline performer.

What it means Gate AI-generated code behind an automated scan-fix-RESCAN loop rather than a single fix pass, since remediation adds new vulnerabilities ~1 in 5 times; and choose the security-remediation model on measured residual findings, not on general coding leaderboard rank.

arXiv preprint · AUG 17

In a longitudinal study of professional developers, the dominant efficiency bottlenecks were organizational dependencies and waiting for external validation (structurally stable over time), while a generative-AI usage barrier emerged mid-study and became the most frequently coded interview theme.

What it means Do not expect AI tooling to move developer efficiency while organizational dependencies and validation waits dominate; measure efficiency continuously (not one cross-sectional survey) and treat evolving AI-tool friction as a first-class, addressable barrier.

arXiv preprint · AUG 17

Across a system-level synthesis, many apparent coding-agent 'model failures' actually originate in the harness, retrieval, state management, or verification layers, and layer-level improvements often fail to propagate to end-to-end outcomes.

What it means Evaluate and buy the coding-agent SYSTEM, not the model: attribute failures to the specific layer (harness, retrieval, state, verification) before swapping models, and instrument each layer so a fix is proven to reach end-to-end outcomes.

arXiv preprint · AUG 14

On SWE-bench Verified, the higher-recall retriever setting (gold file present in 0.878 vs 0.806 of packs) LOWERS issue resolution; disabling per-file deduplication to favor within-file depth raises single-shot resolve rate +7.6pp for gpt-5.6-sol (39.2%->46.8%, n=500, p=0.0003), replicated on open weights (+3.6pp, n=499).

What it means Stop tuning code-assistant retrieval to recall@k; A/B the context-packing policy against actual task resolution, and at a tight token budget do not hard-deduplicate by file (favor within-file depth over file breadth).

arXiv preprint · AUG 14

The structure of a developer workspace (directory depth, modularity, injection position, context framing) measurably changes indirect-prompt-injection success against agentic coding assistants, with highly modular codebases showing significantly lower attack success rates.

What it means Treat any third-party code an agent ingests as an injection surface: constrain filesystem scope, prefer modular workspace layouts, add security-cue framing, and test agents in an uncontaminated environment before trusting IPI-resistance claims.

arXiv preprint · AUG 14

In two high-velocity AI-infrastructure repos, PR throughput rose 21x (vLLM) and 17.9x (SGLang) during the agentic-coding era, but bot-authored PRs accounted for less than 0.2% of that growth, indicating the velocity increase was overwhelmingly human-driven while PR size stayed stable.

What it means Read the AI-coding productivity story as augmentation, not autonomy: expect throughput and reviewer-participation gains from AI-assisted humans, not from bots authoring the work — staff for more human review, not fewer humans.

arXiv preprint · AUG 14

For command-issuing coding agents, a matched headline score can hide large offsetting effects: GPT-5.6-sol's matched gap of -3.6 points masks -64.3 points of execution-path damage compensated by +60.7 points, and the deployment configuration reorders model rankings.

What it means Never rank command-issuing agents on a single matched score; benchmark under your actual execution path (shell serialization/escaping) and report the operating point and final-state validator, because the ranking flips with deployment config.

arXiv preprint · AUG 13

At repository scale, the strongest frontier coding agent fully solves only 27 of 43 joint implementation-and-proof instances and closes no specifications on the hardest repositories, showing current agents fall short of verified repo-scale synthesis.

What it means Do not rely on coding agents for machine-checked correctness guarantees on multi-module codebases yet; use verified generation only where a human specifies and audits, and expect the hardest modules to fail entirely.

arXiv preprint · AUG 13