Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurement
LLM judges are increasingly used to assess whether AI outputs meet workplace requirements, but agreement on response rankings does not establish agreement on acceptance rates or occupational aggregates. We introduce O*NET-BENCH, an audit suite derived from an existing survey of…
- 01.LLM judges are increasingly used to assess whether AI outputs meet workplace requirements, but agreement on response rankings does not establish agreement on acceptance rates or occupational aggregates
- 02.We introduce O*NET-BENCH, an audit suite derived from an existing survey of…
This material is provided as an extended analytical summary.
Read full article at sourceRelated Intelligence
All stories →OpenAI announces three big science discoveries and a new way to work with academics
Less than a month after its mass release of hundreds of manuscripts that it said contained solutions to long-standing mathematics questions, OpenAI is back with three more "major" discoveries in math, computer science, and physics. What's different this time is the choice to…
Anthropic files for IPO
The placement volume and price per share have not yet been announced
Anthropic uses gas to fuel AI growth
Anthropic reports on its centers' dependence on natural gas and the associated risks.