# Когда F1 недостаточно: Аудит кодбуков для аннотации с помощью LLM

> Полный текст доступен на английском

**Published:** 9 ноября 2026 г.
**Author:** Yuehong Cassandra Tai, Qunhua Li
**Source:** arXiv — cs.AI
**Language:** en-US
**HTML:** https://aiqazaqstan.com/ru/articles/when-f1-is-not-enough-auditing-codebooks-for-llm-assisted-annotation

arXiv:2611.08846v1 Тип объявления: new Аннотация: Большие языковые модели (LLM) всё чаще используются для аннотации текстов в социальных науках, но стандартные метрики производительности не объясняют, почему происходят ошибки. Низкий F1 может отражать ограничения системы аннотации, дефекты в...

arXiv:2611.08846v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for text annotation in social science, but standard performance metrics do not explain why errors occur. Low F1 may reflect limitations of the annotation system, defects in the written codebook, or inconsistency in reference labels. We propose a pre-deployment codebook audit for LLM-assisted annotation. The audit uses class-specific metrics and confusion matrices to identify problematic classes and boundaries, structured document-level review to attribute error sources, and targeted codebook revision followed by held-out evaluation. We apply the audit to two published multi-class codebooks from distinct domains. In both applications, the audit identifies under-specified or conflicting coding rules, and targeted revisions reduce the diagnosed errors on held-out data. Analyses with two LLM annotators show that some problematic boundaries and codebook revisions transfer across models, whereas other revision effects are annotator-specific. The results show that LLM annotation errors can help identify weaknesses in the written measurement instrument, not only in the model. Codebook auditing complements prompting, retrieval, and fine-tuning by clarifying whether persistent errors call for model improvement, codebook revision, or renewed scrutiny of reference labels.

---

*Материал агрегирован из первоисточника: [arXiv — cs.AI](https://arxiv.org/abs/2611.08846). Открыть оригинал.*

**Теги:** #arXiv

---

*AI Qazaqstan — non-commercial informational portal about AI in Kazakhstan. Machine translations and summaries are AI-generated.*