arXiv:2611.08684v1 Announce Type: new Abstract: Clients and proxies in front of an LLM engine retry calls that time out. After a transient overload, these retries can keep the engine from recovering even when calls arrive below its capacity. Since the engine may keep computing for clients that have left, propagating cancellation looks like the natural remedy. In this paper, we test it on vLLM with overload experiments on three open models, and with scheduler traces of one that follow each client attempt to the execution it creates in the engine. We find that with retries on, cancellation alone did not restore service in any run, and only 12-38% of the calls that arrived after the overload were delivered. The traces suggest why: retries that vLLM ran typically started prefill less than half a second before they timed out. Executions that vLLM later aborted took 79-85% of the prompt tokens it computed, and vLLM's prompt-token counter missed a third to half of all computed prompt tokens. A retry budget restored service by shedding retries but left many of the overload's calls undelivered. Request coalescing delivered every call from the overload onward, some late, unless cancellation had already aborted the running execution. Overload controls should thus be judged by the executions that retries add and the calls that clients receive.
Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine
arXiv:2611.08684v1 Announce Type: new Abstract: Clients and proxies in front of an LLM engine retry calls that time out. After a transient overload, these retries can keep the engine from recovering even when calls arrive below its capacity. Since the engine may keep computing…
AI Summary & Key Takeaways
- arXiv:2611.08684v1 Announce Type: new Abstract: Clients and proxies in front of an LLM engine retry calls that time out
- After a transient overload, these retries can keep the engine from recovering even when calls arrive below its capacity
- Since the engine may keep computing…
Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine
Related stories
Ripple-65B Preview: Model for Agentic Coding
🌟 Ripple-65B Preview: Model for Agentic Coding. Recursive Intelligence has released an open preview of a 65‑billion‑parameter agentic programming model focused on response speed. The company was founded by Anna Goldie and Azalia Mirhoseini, former Google researchers…
FLEX Platform Teaches Robots to Understand Hidden Human Intentions
MIT Unveils FLEX Platform: Robots Learn to Understand Hidden Human Intentions and Generalize Commands Based on Language Model.
Top AI News of the Week
📌 Top AI News of the Week 🔊 ElevenLabs Sound Effects v3 — generates sounds from descriptions: from footsteps and explosions to ambient backgrounds. 🎨 OpenAI Images 2.6 — creates GIF animations, generates images faster, and edits existing images more precisely. 🎬 Gemini Omni 1.2 Flash —…
Toward LLM Self-Play Under Data Scarcity: Effectiveness, Generalizability, and Trustworthiness
arXiv:2611.08908v1 Announce Type: new Abstract: The rapid development of large language models (LLMs) has been driven largely by scaling model size and human-curated training data, yet this paradigm is increasingly constrained by data scarcity and diminishing returns from human…