Software categories don't sit still.

From neutral prompts, three frontier models (Fable 5, Opus 4.8, Opus 5) shipped code with unprompted SOC 2 conformance of 47-88% and real vulnerabilities (a reachable Werkzeug debugger RCE, an unauthenticated download, an endpoint returning every stored name and email); adding one sentence naming the SOC 2 standard lifted every case to 86-100% (worth 23-50 points) and removed every insecure construction.

What it means Name the compliance standard explicitly in the prompt and gate AI-generated infra/auth/PII code behind a security review; unprompted output ships RCE-class defects, controls outside the model's default conception (MFA, cookie flags, account lifecycle) survive even a named standard, and a newer model won't close the gap.

arXiv preprint · AUG 07

Across 3.52M production code changes at a billions-of-users enterprise (Apr 2025-Apr 2026), AI-generated C++ carried a distinct quality profile - higher interface/coupling burden, more copy and allocation overhead, explicit loops over optimized standard APIs - translating to increased review effort and a 5-8% rise in compute consumption; taxonomy-informed static-analysis feedback cut targeted warnings 11.1%.

What it means Budget for the hidden tax on AI-authored code - extra review effort and ~5-8% more compute at runtime - and route it through taxonomy-informed static-analysis feedback; velocity gains are real but not free.

arXiv preprint · AUG 06

Recurrent context compression in long-horizon agents weakens the influence of recent interactions and measurably increases blocked actions, repeated exploration, and run-to-run instability; a verifier-guided compaction framework (TRACE) recovers task performance and multi-run reliability.

What it means Treat context compaction as a reliability risk rather than a free cost optimization — measure multi-run stability before enabling it on long-horizon agents, since naive compression trades tokens for blocked actions and nondeterminism.

arXiv preprint · AUG 06

A survey of 119 practitioners found LLM use has become a habitual component of professional software engineering, with reported patterns of functional dependence and overreliance — prioritizing LLMs over documentation or peer consultation — alongside less-common addiction-related difficulty moderating use.

What it means Build trust-calibration and mandatory verification into team practice — the productivity that drives adoption also drives overreliance (skipping docs and peers), so make 'verify the output' and 'consult a human/source' explicit workflow steps rather than assumed judgment.

arXiv preprint · AUG 06

Coding agents spend most of their token budget finding the file to patch rather than patching it — a 30B OpenHands agent averages 23 rounds and 631K tokens per resolved SWE-Bench issue — and a dedicated RL-trained retrieval step preserved resolve rate (27.0% vs 25.8%) while cutting 15% of rounds and 19% of tokens, but only above a retrieval-precision threshold below which retrieval degraded the agent.

What it means Attack agent cost at repository retrieval, not just the model — a precise file-finding step cuts ~19% of tokens, but a weak retriever (BM25-grade) makes things worse, so measure retrieval precision before bolting one on.

arXiv preprint · AUG 06

Engineers report a productivity-experience paradox: 84% report productivity improvement at both time points, yet the share of matched participants reporting worsened developer experience in at least one dimension nearly doubled from 14% to 27% over six months, with flow state and cognitive load eroding while feedback loops improved.

What it means Track developer experience alongside throughput when rolling out AI tooling — the work is shifting to supervisory correction of AI output, and that erosion shows up months after the productivity win.

arXiv preprint · MAY 22

In legacy-code modernization, semantic-trap snippets silently changed behavior in 39.7% of attempts (versus 7.0% on benign controls), and the producing model silently endorsed 31.7% of its own drift cases — sometimes while correctly articulating the exact Python 2/3 semantic distinction that broke its output.

What it means Never use the model that performed a migration as its own behavioral reviewer — verify semantic preservation with executable oracles (characterization tests), especially around numeric semantics.

arXiv preprint · MAY 20

10.7% of passing SWE-agent trajectories are 'Lucky Passes' — regression cycles, blind retries, or missing verification that happen to end green — with per-model Lucky rates ranging 0.5% to 23.2%, enough to move models up to five leaderboard rank positions when scored by process quality.

What it means When comparing agents for production use, inspect trajectories, not just pass rates — a model that passes by chaotic trial-and-error will be more expensive and less predictable on your real work than its score suggests.

arXiv preprint · MAY 13

Enterprise-grade coding agents executed hidden malicious commands smuggled inside benign-looking skill files in 95.5-96.1% of runs (Gemini CLI) and 71.6-74.0% (Qwen Code), nearly invariant to the generating model, with explicit safety recognition in only 1.99% of runs.

What it means Treat third-party agent skill/plugin files as untrusted executable code - sandbox and human-review them before adoption and assume the agent's own safety recognition is near-zero.

arXiv preprint · AUG 05

When the issue report is removed and agents must discover as well as fix bugs, most state-of-the-art coding agents struggle - showing limited ability to locate and resolve recorded bugs, handle multiple-bug scenarios, and find valid latent bugs.

What it means Do not extrapolate SWE-bench-style resolve rates to unscoped bug-hunting - agents lean heavily on a human-written issue report to perform; proactive discovery still needs a person to frame the problem.

arXiv preprint · AUG 05

Repository-preserved Agent Plan files are rare but informative: screening 36,710 GitHub repositories surfaced only 85 Markdown plan files from 10 repositories, and where present they guided agent execution most often through implementation steps, concrete files and locations, and testing/validation information.

What it means Preserve agent plan files in-repo with the elements that actually steer execution — numbered implementation steps, concrete file paths, and test/validation criteria — they double as durable human-agent task-intent documentation, but adoption is still near-zero so this is convention to establish, not a settled norm.

arXiv preprint · AUG 05

Chat panels, terminal agents, generated diffs, and streaming status output in AI developer tools create real visual accessibility barriers for blind, low-vision, and color-vision-deficient developers, clustering into screen-reader/AT barriers, contrast and differentiation problems, and readability/scaling limits — with prominence varying by tool ecosystem.

What it means Include screen-reader, contrast, and scaling checks when adopting or building AI coding-tool interfaces — the agent/diff/streaming surfaces introduce accessibility barriers that general IDE accessibility work does not cover.

arXiv preprint · AUG 05

Prompt wording changes where agent effort is spent without changing success: 'multiple approaches' phrasing inflates reasoning 2.4-7.4x with no success gain, 'maximum certainty' phrasing drives redundant-verification runs costing 18x the clean-run median, and harness design shifts cost per successful task 5-30x.

What it means Strip effort-inflating phrases ('explore multiple approaches', 'be absolutely certain') from agent prompts and use bounded-efficiency wording — prompt style is a direct cost lever with no measured correctness payoff.

arXiv preprint · AUG 02

13.6% of SWE-bench Verified instances have misaligned PR-issue pairings — the problem statement does not actually match what the graded patch fixes — across five misalignment patterns.

What it means Discount small SWE-bench Verified score differences between models — a double-digit fraction of the instances measure something other than issue resolution, so single-digit leaderboard gaps are within construction noise.

arXiv preprint · JUL 30