B4 Research September 2, 2026

Benchmarks sharing the same label (e.g. 'bug fix') differ statistically on at least two of three task-demand axes (Spread–Novelty–Centrality), agent success concentrates uniformly in low-demand tasks for every model family and scale, and the behavioral signature of success is model-family-specific (Claude succeeds by matching gold-solution scope, Qwen by exceeding it).

What it means When comparing agent scores across benchmarks, check what the benchmark actually demands rather than trusting its category label — and expect deployed agents to succeed mostly on low-spread, low-novelty, low-centrality tasks, reserving the hard cross-cutting changes for humans.

arXiv preprint · SEP 01

On 203 real-world dependency-upgrade tasks with hidden code-level breakage (DEPBENCH), the best coding-agent configuration solved only 51.2% (104/203), with substantial variation across harnesses, models, and ecosystems.

What it means Do not hand agents unattended dependency upgrades — the maintenance work teams most want to delegate is precisely where agents fail half the time; keep upgrade PRs behind full test suites and human review, and expect ecosystem-specific gaps.

arXiv preprint · AUG 31

How a user invokes a coding agent changes its vulnerability to poisoned repositories by up to 4.5x in attack success rate across task types (Run-Tests 45.5% ASR vs Fix-Bug 8.6%), with test-execution tasks forming a silent attack surface (high attack success, low agent alerting).

What it means Treat 'run the tests' in an untrusted third-party repo as the riskiest agent instruction, not the safest — sandbox agents when bootstrapping from external repos, and know that prompt phrasing and supplied rules shift the attack surface rather than eliminating it.

arXiv preprint · AUG 31

A single factor explains 74.5% of common variance across twelve frontier benchmarks and substantially tracks model release date (R²=0.505), so most of the gap between models released months apart is calendar; under the pre-registered dimensionality rule the four economic benchmarks form no distinct capability factor — though a leave-one-benchmark-out test shows a multi-factor representation still predicts held-out economic scores better than a single general index, so they carry incremental predictive information without constituting a distinct latent capability.

What it means Date-adjust before buying on leaderboard gaps: a small score difference between contemporaneous models is mostly noise around a time trend, not a capability difference — pick by trying models on your own workload, not by leaderboard deltas.

arXiv preprint · AUG 29

Contamination types are best organized by which mitigation each defeats (direct, derivative, temporal, distributional, acquired) — a private held-out test set closes only the first — and in an audit of 41 evaluation documents, elicitation budgets were reported in just 13% of documents and none addressed all five types.

What it means When a benchmark score informs a buy decision, ask which contamination mitigations the eval actually applied — a held-out private test set defeats only direct leakage, and current reporting practice almost never discloses enough to tell capability from leakage.

arXiv preprint · AUG 29

Rewriting SWE-bench tasks to match how real users phrase requests cuts resolution rates by 6.4 percentage points on average and can change model rankings; 88% of real prompts carry only a bare problem statement (alone or with limited context) versus 7% of benchmark problems, and stating Desired Behavior and Motivation measurably lifts performance while Environment Information and Reproduction Steps add tokens without benefit.

What it means State the desired behavior and the motivation in every agent request — those two fields measurably lift resolution while reproduction steps and environment info just add tokens; and discount SWE-bench scores as an upper bound, since real phrasing costs ~6 points and can reorder models.

arXiv preprint · AUG 28

A manager-worker scaffold over a shared filesystem helps some models dramatically (+8 to +30 points on the 100 hardest recent LiveCodeBench problems, up to +42 in single-pass configs) and is null or negative for others (-1 to -9), roughly triples token cost, but buys accuracy more cheaply than upgrading to a larger model in the cases studied.

What it means Test multi-agent scaffolds per-model before adopting: the gain is conditional, largest for smaller models and reasoning-off configs, and near-zero for large reasoning models — and compare the ~3x token bill against simply buying the bigger model, which this study found is often the worse deal.

arXiv preprint · AUG 27

LLM-generated backend services that pass functional tests still show statistically significant memory-growth trends under 48-hour sustained execution in most application-language combinations, indicating software-aging defects that correctness testing does not catch — and the aging trends also appear in some human-written implementations.

What it means Soak-test AI-generated services before deploying them as long-running processes — passing the test suite says nothing about memory behavior at hour 40; add memory-trend monitoring to acceptance criteria for generated backends.

arXiv preprint · AUG 26

Teams absorbing high-volume AI code generation are shifting from line-by-line review toward layered supervision: machine-readable architectural conventions as preventive guardrails, linting/testing/CI repurposed as supervision infrastructure, and human review refocused on architectural reasoning and operational explainability.

What it means If AI generation volume is straining code review, don't add reviewers — externalize architectural intent into machine-checkable rules and let CI carry first-pass supervision, reserving humans for architecture and maintainability judgment.

arXiv preprint (accepted, ESEM 2026 SEIP track) · AUG 26