Answer Consistency

Overview

The Answer Consistency evaluation measures whether an AI system gives the same answer when it is given the exact same prompt several times. The prompts are provided by the user, so the evaluation can be tailored to any domain, product or knowledge base. No reference answers are needed: the evaluation measures agreement between the answers, not whether they are correct.

Metrics

Consistency

The average, over all prompts, of the share of answers that agree with the most common answer to that prompt (range: 1/N to 1.0, where N is the number of answers per prompt).

Motivation

Most AI systems generate text with some randomness, and many users send the same prompts. If the answer changes from one request to the next, two users sending the same prompt can be told different things, and a single user who sends it again can receive a different answer than the first time. Neither user can tell which answer to trust.

Inconsistency is also a warning sign that is independent of correctness. An AI system that is inconsistent on a prompt is, at best, correct only some of the time on that prompt, and its answers do not reflect stable knowledge or a stable policy. Because the evaluation needs no reference answers, it can be run on real user prompts for which no correct answer has been written down.

Methodology

1. Test Cases: The user provides a dataset of prompts. Only the prompt column is needed; its name is configurable.
2. Model Response: Each prompt is sent to the AI system N times (configurable, default 5) as separate requests with no shared conversation history.
3. Consistency Scoring: A judge model receives the prompt and all N answers. It groups together answers that give the same answer to the prompt and returns the share of answers in the largest group.
4. Metric Computation: The mean score across all prompts yields the consistency.

Scoring

Answer Consistency

Examples