Large BenchmarkEnterprise-grade coding agents executed hidden malicious commands smuggled inside benign-looking skill files in 95.5-96.1% of runs (Gemini CLI) and 71.6-74.0% (Qwen Code), nearly invariant to the generating model, with explicit safety recognition in only 1.99% of runs.
Towards a Risk Assessment of Malicious Skill Files in Coding AgentsWhat it means Treat third-party agent skill/plugin files as untrusted executable code - sandbox and human-review them before adoption and assume the agent's own safety recognition is near-zero.
arXiv preprint · AUG 05
Large BenchmarkWhen the issue report is removed and agents must discover as well as fix bugs, most state-of-the-art coding agents struggle - showing limited ability to locate and resolve recorded bugs, handle multiple-bug scenarios, and find valid latent bugs.
Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue ReportsWhat it means Do not extrapolate SWE-bench-style resolve rates to unscoped bug-hunting - agents lean heavily on a human-written issue report to perform; proactive discovery still needs a person to frame the problem.
arXiv preprint · AUG 05
Case StudyRepository-preserved Agent Plan files are rare but informative: screening 36,710 GitHub repositories surfaced only 85 Markdown plan files from 10 repositories, and where present they guided agent execution most often through implementation steps, concrete files and locations, and testing/validation information.
An Exploratory Study of Agent Plans for Agentic AI Coding Tools in Open-Source SoftwareWhat it means Preserve agent plan files in-repo with the elements that actually steer execution — numbered implementation steps, concrete file paths, and test/validation criteria — they double as durable human-agent task-intent documentation, but adoption is still near-zero so this is convention to establish, not a settled norm.
arXiv preprint · AUG 05
Case StudyChat panels, terminal agents, generated diffs, and streaming status output in AI developer tools create real visual accessibility barriers for blind, low-vision, and color-vision-deficient developers, clustering into screen-reader/AT barriers, contrast and differentiation problems, and readability/scaling limits — with prominence varying by tool ecosystem.
Characterizing Visual Accessibility Issues in AI Developer Tools: An Empirical StudyWhat it means Include screen-reader, contrast, and scaling checks when adopting or building AI coding-tool interfaces — the agent/diff/streaming surfaces introduce accessibility barriers that general IDE accessibility work does not cover.
arXiv preprint · AUG 05
Controlled StudyPrompt wording changes where agent effort is spent without changing success: 'multiple approaches' phrasing inflates reasoning 2.4-7.4x with no success gain, 'maximum certainty' phrasing drives redundant-verification runs costing 18x the clean-run median, and harness design shifts cost per successful task 5-30x.
Same Task, Different Work: Prompt-Induced Waste in Coding AgentsWhat it means Strip effort-inflating phrases ('explore multiple approaches', 'be absolutely certain') from agent prompts and use bounded-efficiency wording — prompt style is a direct cost lever with no measured correctness payoff.
arXiv preprint · AUG 02
Large Benchmark13.6% of SWE-bench Verified instances have misaligned PR-issue pairings — the problem statement does not actually match what the graded patch fixes — across five misalignment patterns.
PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like BenchmarksWhat it means Discount small SWE-bench Verified score differences between models — a double-digit fraction of the instances measure something other than issue resolution, so single-digit leaderboard gaps are within construction noise.
arXiv preprint · JUL 30