arXiv:2611.08684v1 Announce Type: new Abstract: Clients and proxies in front of an LLM engine retry calls that time out. After a transient overload, these retries can keep the engine from recovering even when calls arrive below its capacity. Since the engine may keep computing for clients that have left, propagating cancellation looks like the natural remedy. In this paper, we test it on vLLM with overload experiments on three open models, and with scheduler traces of one that follow each client attempt to the execution it creates in the engine. We find that with retries on, cancellation alone did not restore service in any run, and only 12-38% of the calls that arrived after the overload were delivered. The traces suggest why: retries that vLLM ran typically started prefill less than half a second before they timed out. Executions that vLLM later aborted took 79-85% of the prompt tokens it computed, and vLLM's prompt-token counter missed a third to half of all computed prompt tokens. A retry budget restored service by shedding retries but left many of the overload's calls undelivered. Request coalescing delivered every call from the overload onward, some late, unless cancellation had already aborted the running execution. Overload controls should thus be judged by the executions that retries add and the calls that clients receive.
Research2 分钟阅读
取消还不够:重试在过载的LLM引擎中留下了什么
arXiv:2611.08684v1 通告类型:新 摘要:LLM引擎前的客户端和代理会重试超时的调用。在暂时性过载后,这些重试会阻止引擎恢复,即使调用到达率低于其容量。因为引擎可能继续计算…
arXiv — cs.AI作者: Xindan Zhang, Fei Wang2026年11月9日 (2小时前)
提供 4 种语言版本AI 摘要:要点
- arXiv:2611.08684v1 通告类型:新 摘要:LLM引擎前的客户端和代理会重试超时的调用。在暂时性过载后,这些重试会阻止引擎恢复,即使调用到达率低于其容量。因为引擎可能继续计算…
Research
KZ·AI A435Инсайт дня
取消还不够:重试在过载的LLM引擎中留下了什么
全文为英语
内容来自原始来源: arXiv — cs.AI
查看原文查看原文
标签:#arXiv
相关文章
Ripple-65B Preview:面向智能体编程的模型
🌟 Ripple-65B Preview:面向智能体编程的模型。Recursive Intelligence 发布了一个拥有 650 亿参数的智能体编程模型的开放预览版,专注于响应速度。该公司由前 Google 研究员 Anna Goldie 和 Azalia Mirhoseini 创立……
Telegram — AI Machine Learning Big Data
SMNTCN (Semanticon) — глобальные новости ИИ
本周AI热门新闻榜
📌 本周AI热门新闻榜 🔊 ElevenLabs Sound Effects v3 — 根据描述生成声音:从脚步声、爆炸声到环境背景音。 🎨 OpenAI Images 2.6 — 创建GIF动画,图像生成更快,编辑现有图像更精准。 🎬 Gemini Omni 1.2 Flash —…
Telegram — GPT Time | Нейросети | ИИ
面向数据稀缺条件下的LLM自我博弈:有效性、泛化能力与可信度
arXiv:2611.08908v1 公告类型:新 摘要:大语言模型(LLM)的快速发展主要得益于模型规模扩大和人工策划的训练数据,但这一范式正日益受到数据稀缺和人工收益递减的制约…
arXiv — cs.AI