# The Price of Token Boundaries: Compression Certificates and Prediction

> 全文为英语

**Published:** 2026年9月30日
**Author:** Yuhao Du, Shunian Chen
**Source:** arXiv — cs.AI
**Language:** en-US
**HTML:** https://aiqazaqstan.com/zh/articles/the-price-of-token-boundaries-compression-certificates-and-prediction

arXiv:2609.35869v1 Announce Type: new Abstract: Pre-tokenisation restricts which text fragments can become prediction units, but its compression cost is obscured when tokenisers are compared only under the same boundaries. We measure this cost by bounding the minimum token count…

> 内容来自原始来源： **arXiv — cs.AI**
> [在 arXiv — cs.AI 阅读全文](https://arxiv.org/abs/2609.35869)

**标签：** #arXiv

---

*AI Qazaqstan — non-commercial informational portal about AI in Kazakhstan. Machine translations and summaries are AI-generated.*