arXiv:2611.08666v1 Announce Type: new Abstract: Interactive debuggers are indispensable to human software development, yet coding agents overwhelmingly rely on edit-execute-observe loops, using common bash tools, test scripts, and print statements to debug via repeated shell commands. Stepping through a debugger provides live access to runtime state, which may enable different problem-solving capabilities. However, prior work typically introduces the debugger as an additional tool or through a new subagent, making its individual contribution difficult to isolate from the associated scaffold changes. To address this, we compare a debugger-only agent against a bash-only agent on all 500 SWE-bench Verified instances, holding the rest of the scaffold fixed. We evaluate both architectures on two models, GPT-OSS-120B and Nemotron-3-Ultra-550B, with three independent runs per architecture $\times$ model configuration. Crucially, to distinguish interface specialization from expanded coverage under repeated sampling, we introduce pass$_X$@$k$, a cross-agent pass@$k$ that provides an unbiased estimator of the probability that at least one of $k$ agents is successful. Although the bash and debugger agents solve partially disjoint subsets of the benchmark in any single pair of runs, our pass$_X$@$2$ estimator reveals that this is largely due to run-to-run variance rather than tool complementarity. Across both models, we find that two bash rollouts outperform the combination of one bash and one debugger rollout. Our findings highlight the importance of accounting for stochastic variance in evaluating agent and tool diversity, and we hope that pass$_X$@$k$ will serve as a useful framework for the community.
Pdb or Not to Be: Comparing Debugging Interfaces With Pass$_{X}$@$k$
arXiv:2611.08666v1 Announce Type: new Abstract: Interactive debuggers are indispensable to human software development, yet coding agents overwhelmingly rely on edit-execute-observe loops, using common bash tools, test scripts, and print statements to debug via repeated shell…
AI Summary & Key Takeaways
- arXiv:2611.08666v1 Announce Type: new Abstract: Interactive debuggers are indispensable to human software development, yet coding agents overwhelmingly rely on edit-execute-observe loops, using common bash tools, test scripts, and print statements to debug via repeated shell…
Pdb or Not to Be: Comparing Debugging Interfaces With Pass$_{X}$@$k$
Related stories
Ripple-65B Preview: Model for Agentic Coding
🌟 Ripple-65B Preview: Model for Agentic Coding. Recursive Intelligence has released an open preview of a 65‑billion‑parameter agentic programming model focused on response speed. The company was founded by Anna Goldie and Azalia Mirhoseini, former Google researchers…
FLEX Platform Teaches Robots to Understand Hidden Human Intentions
MIT Unveils FLEX Platform: Robots Learn to Understand Hidden Human Intentions and Generalize Commands Based on Language Model.
Top AI News of the Week
📌 Top AI News of the Week 🔊 ElevenLabs Sound Effects v3 — generates sounds from descriptions: from footsteps and explosions to ambient backgrounds. 🎨 OpenAI Images 2.6 — creates GIF animations, generates images faster, and edits existing images more precisely. 🎬 Gemini Omni 1.2 Flash —…
Toward LLM Self-Play Under Data Scarcity: Effectiveness, Generalizability, and Trustworthiness
arXiv:2611.08908v1 Announce Type: new Abstract: The rapid development of large language models (LLMs) has been driven largely by scaling model size and human-curated training data, yet this paradigm is increasingly constrained by data scarcity and diminishing returns from human…