Summarization Accuracy
Overview
The Summarization Accuracy evaluation measures how accurately an AI system summarizes documents. Accuracy has two sides: a summary is faithful when every statement in it is supported by the document, and complete when it conveys the document's key facts. The evaluation measures both at the level of individual statements, following FineSurE [1], and also reports how concise the summary is.
The documents are supplied by the user, so the evaluation can be run on the kind of content the AI system summarizes in practice.
Metrics
Faithfulness
The fraction of summary claims that the document supports, averaged over all summaries (range: 0.0 to 1.0).
Completeness
The fraction of the document's key facts that the summary conveys, averaged over all summaries (range: 0.0 to 1.0).
Conciseness
The fraction of summary claims that convey a key fact of the document, averaged over all summaries (range: 0.0 to 1.0). Low values mean the summaries spend their length on minor details.
Motivation
Summaries are read in place of the document, so readers rarely notice what a summary gets wrong or leaves out. Fluent LLM summaries are often preferred by readers and judges for their style, while human-written references remain more faithful and more informative [2]. Unsupported details concentrate in long inputs and towards the end of long summaries [3].
Overall quality ratings hide these failures. Human raters and LLM judges asked for a single verdict tend to accept a summary when most of it is correct, even when it contains a clear factual error [4]. Checking each statement separately shows how often a reader would be misled.
Methodology
- Test cases: user-supplied documents, one per row of the documents dataset.
- System response: each document is sent to the AI system as a single user message asking for a summary. No system message is added.
- Claim extraction: a judge model splits each summary into atomic claims, short standalone sentences that each state one piece of information.
- Key fact extraction: the judge model reduces each document to its key facts, the pieces of information a good summary must contain.
- Faithfulness check: the judge model labels each claim supported or unsupported by
the document, with an error type for unsupported claims (
entity,relation,number_or_date,not_in_sourceorother). - Key fact alignment: the judge model labels each key fact covered or not, with the claims that convey it.
- Metric computation: per-summary faithfulness, completeness and conciseness scores, averaged over all summaries.
Documents for which no summary reached the user, for example because a guardrail returned an error, score 0.0 on all metrics.
Scoring
Summarization Accuracy Judge
Faithful and complete summary
- Faithfulness 1.0: all 5 claims (budget, purpose, start date, duration, temporary site) are stated in the document.
- Completeness 1.0: all 4 key facts are covered.
Summary with an invented detail and a wrong number
- Faithfulness 0.5: "unanimously" is
not_in_sourceand "2.4 million euros" is anumber_or_dateerror; the purpose and start date are supported (2 of 4 claims). - Completeness 0.25: only the start date is covered; the budget is stated with the wrong amount, and the duration and temporary site are missing.
References
[1] Song H, Su H, Shalyminov I, Cai J, Mansour S. FineSurE: Fine-grained Summarization Evaluation using LLMs. In: Proceedings of ACL 2024. Available from: https://arxiv.org/abs/2407.00908
[2] Liu D, Whitehouse C, Zhao Z, Cao Z, Li J, Wang Y. Summarization is Not Dead Yet. In: Proceedings of EMNLP 2026. Available from: https://arxiv.org/abs/2606.08000
[3] Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarization. arXiv preprint, 2025. Available from: https://arxiv.org/abs/2505.15291
[4] Zhang H, Feng Y, Zhou J. Averaging Bias: Human Faithfulness Annotations are not Locally Faithful. arXiv preprint, 2026. Available from: https://arxiv.org/abs/2608.00205