AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory
arXiv:2611.05556v1 Announce Type: new Abstract: LLM-as-a-judge is a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias. Judge pools mix judges of very different reliability, while the human-labeled…
AI Summary & Key Takeaways
- arXiv:2611.05556v1 Announce Type: new Abstract: LLM-as-a-judge is a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias
- Judge pools mix judges of very different reliability, while the human-labeled…
AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory
Related stories
Anthropic has launched a website for its upcoming IPO
✔️ Anthropic has launched a website for the upcoming IPO. The company has opened a laconic portal ipo.anthropic.com. So far, the main page displays only minimalistic animation and an inscription about the imminent appearance of details. Anthropic plans to raise private capital through brokerage partners...
Claude even bans paid accounts. How not to lose everything?
Claude even bans paid accounts. How not to lose everything? ⠀ Imagine: you pay $200 for Claude Max, manage projects, store correspondence, and one day you see Account Suspended. Access was lost, but the work remained inside. ⠀ In October, complaints about blocking became more frequent, including after...
ChatGPT can now create GIF animations and remember your pets
🎞 ChatGPT can now create GIF animations and remember your pets OpenAI has released Images 2.6, an updated model for generating images. Now in ChatGPT you can create not only pictures, but also animated GIFs. What else is new? • Image generation has become…
Who May Overrule the Agent? Evidence-Gated Authority in LLM Review Institutions
arXiv:2611.05390v1 Announce Type: new Abstract: Multi-agent language-model systems often add a reviewer to improve an upstream agent's decision. Yet a reviewer can both repair errors and replace decisions that were already correct. We study this as a problem of institutional…