B4 Research August 19, 2026

On end-to-end tasks drawn from market-validated AI-startup workflows, the strongest evaluated agent completes only ~30% of StartupBench despite substantial partial progress, with complex instruction-following and domain expertise as the main failure sources.

What it means Scope agent deployments to the ~30% of real end-to-end workflows they can actually finish; keep humans on complex multi-step instructions and domain-specific steps, and validate against market-representative tasks rather than researcher-chosen benchmarks.

arXiv preprint · AUG 18

Vibe-coding tools (Lovable, v0, Replit) produce structurally distinct and uneven code quality from a single prompt: Lovable concentrates lower-severity issues but has a much higher code-smell density per KLOC, while v0 and Replit produce more aggressive severity profiles.

What it means Treat vibe-coded output as a first draft carrying tool-specific structural debt: run static analysis and budget remediation before shipping, and pick the tool on its quality trade-off, not just perceived speed.

arXiv preprint · AUG 17

A scan-fix-rescan pipeline cuts static-analyzer findings in AI-generated code by 29-69% across four Claude models, but remediation itself introduces new vulnerabilities in 15-22% of cases, and the best code-generation model (Opus 4.8) was not the best pipeline performer.

What it means Gate AI-generated code behind an automated scan-fix-RESCAN loop rather than a single fix pass, since remediation adds new vulnerabilities ~1 in 5 times; and choose the security-remediation model on measured residual findings, not on general coding leaderboard rank.

arXiv preprint · AUG 17

In a longitudinal study of professional developers, the dominant efficiency bottlenecks were organizational dependencies and waiting for external validation (structurally stable over time), while a generative-AI usage barrier emerged mid-study and became the most frequently coded interview theme.

What it means Do not expect AI tooling to move developer efficiency while organizational dependencies and validation waits dominate; measure efficiency continuously (not one cross-sectional survey) and treat evolving AI-tool friction as a first-class, addressable barrier.

arXiv preprint · AUG 17

Across a system-level synthesis, many apparent coding-agent 'model failures' actually originate in the harness, retrieval, state management, or verification layers, and layer-level improvements often fail to propagate to end-to-end outcomes.

What it means Evaluate and buy the coding-agent SYSTEM, not the model: attribute failures to the specific layer (harness, retrieval, state, verification) before swapping models, and instrument each layer so a fix is proven to reach end-to-end outcomes.

arXiv preprint · AUG 14

On SWE-bench Verified, the higher-recall retriever setting (gold file present in 0.878 vs 0.806 of packs) LOWERS issue resolution; disabling per-file deduplication to favor within-file depth raises single-shot resolve rate +7.6pp for gpt-5.6-sol (39.2%->46.8%, n=500, p=0.0003), replicated on open weights (+3.6pp, n=499).

What it means Stop tuning code-assistant retrieval to recall@k; A/B the context-packing policy against actual task resolution, and at a tight token budget do not hard-deduplicate by file (favor within-file depth over file breadth).

arXiv preprint · AUG 14

The structure of a developer workspace (directory depth, modularity, injection position, context framing) measurably changes indirect-prompt-injection success against agentic coding assistants, with highly modular codebases showing significantly lower attack success rates.

What it means Treat any third-party code an agent ingests as an injection surface: constrain filesystem scope, prefer modular workspace layouts, add security-cue framing, and test agents in an uncontaminated environment before trusting IPI-resistance claims.

arXiv preprint · AUG 14

In two high-velocity AI-infrastructure repos, PR throughput rose 21x (vLLM) and 17.9x (SGLang) during the agentic-coding era, but bot-authored PRs accounted for less than 0.2% of that growth, indicating the velocity increase was overwhelmingly human-driven while PR size stayed stable.

What it means Read the AI-coding productivity story as augmentation, not autonomy: expect throughput and reviewer-participation gains from AI-assisted humans, not from bots authoring the work — staff for more human review, not fewer humans.

arXiv preprint · AUG 14

For command-issuing coding agents, a matched headline score can hide large offsetting effects: GPT-5.6-sol's matched gap of -3.6 points masks -64.3 points of execution-path damage compensated by +60.7 points, and the deployment configuration reorders model rankings.

What it means Never rank command-issuing agents on a single matched score; benchmark under your actual execution path (shell serialization/escaping) and report the operating point and final-state validator, because the ranking flips with deployment config.

arXiv preprint · AUG 13

At repository scale, the strongest frontier coding agent fully solves only 27 of 43 joint implementation-and-proof instances and closes no specifications on the hardest repositories, showing current agents fall short of verified repo-scale synthesis.

What it means Do not rely on coding agents for machine-checked correctness guarantees on multi-module codebases yet; use verified generation only where a human specifies and audits, and expect the hardest modules to fail entirely.

arXiv preprint · AUG 13