Quantiles, Not Points: Streaming Quantile Matching for Ordinal-Aware LLM-as-a-Judge Calibration
arXiv:2611.00447v1 Announce Type: new Abstract: LLM-as-a-judge scores are often distributionally shifted from human ratings on any given evaluation task, and post-hoc calibration (mapping LLM to human scores in a small labeled anchor set) has become a popular method to correct…
AI Summary & Key Takeaways
- arXiv:2611.00447v1 Announce Type: new Abstract: LLM-as-a-judge scores are often distributionally shifted from human ratings on any given evaluation task, and post-hoc calibration (mapping LLM to human scores in a small labeled anchor set) has become a popular method to correct…
Quantiles, Not Points: Streaming Quantile Matching for Ordinal-Aware LLM-as-a-Judge Calibration
Related stories
RUCLEAR измерила налог на токенизатор в разных языках
📌CLEAR измерила налог на токенизатор в разных языках Объединение Coalition for Language Equity in AI Research (CLEAR), опубликовало первое из своих исследований языкового равенства. Поставщики моделей берут плату за токен независимо от языка, а токенизаторы обучены в основном…
ChatGPT will soon be able to create animated GIFs.
ChatGPT will soon be able to create animated GIFs. OpenAI has already added support for generating GIF animations to Codex. The function is behind the new image_generation_animation flag. Users will be able to set the number of frames and the duration of each of them, as well as...
StateBench: Testing how language models track state changes over long documents
A Blog post by Ai2 on Hugging Face
Anthropic is still grappling with the economics of compute
At a moment when Anthropic's compute margins should be getting better, they’re getting worse.