# 当F1不足时：审核LLM辅助标注的代码本

> 全文为英语

**Published:** 2026年11月9日
**Author:** Yuehong Cassandra Tai, Qunhua Li
**Source:** arXiv — cs.AI
**Language:** en-US
**HTML:** https://aiqazaqstan.com/zh/articles/when-f1-is-not-enough-auditing-codebooks-for-llm-assisted-annotation

arXiv:2611.08846v1 发布类型：新 摘要：大语言模型（LLM）在社会科学文本标注中的应用日益增加，但标准性能指标无法解释错误发生的原因。低F1可能反映了标注系统的局限性、代码本中的缺陷...

arXiv:2611.08846v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for text annotation in social science, but standard performance metrics do not explain why errors occur. Low F1 may reflect limitations of the annotation system, defects in the written codebook, or inconsistency in reference labels. We propose a pre-deployment codebook audit for LLM-assisted annotation. The audit uses class-specific metrics and confusion matrices to identify problematic classes and boundaries, structured document-level review to attribute error sources, and targeted codebook revision followed by held-out evaluation. We apply the audit to two published multi-class codebooks from distinct domains. In both applications, the audit identifies under-specified or conflicting coding rules, and targeted revisions reduce the diagnosed errors on held-out data. Analyses with two LLM annotators show that some problematic boundaries and codebook revisions transfer across models, whereas other revision effects are annotator-specific. The results show that LLM annotation errors can help identify weaknesses in the written measurement instrument, not only in the model. Codebook auditing complements prompting, retrieval, and fine-tuning by clarifying whether persistent errors call for model improvement, codebook revision, or renewed scrutiny of reference labels.

---

*内容来自原始来源： [arXiv — cs.AI](https://arxiv.org/abs/2611.08846). 查看原文.*

**标签：** #arXiv

---

*AI Qazaqstan — non-commercial informational portal about AI in Kazakhstan. Machine translations and summaries are AI-generated.*