Interchangeable KV Importance Signals under One Compiled FlexAttention Budget with FlexBudget

Описание

When you run a model over a long context, the KV cache gets huge, so you have to throw some of it away. There are competing rules for choosing what to drop: keep the most recent tokens, keep the ones that get the most attention, and so on. The problem is that every method ships…