General Question Answering
Overview
The General Question Answering evaluation measures how accurately an AI system answers questions about the subjects it is expected to cover. The questions are provided by the user, so the evaluation can be tailored to any domain, product or knowledge base.
The evaluation covers:
- General Open-Ended QA: the AI system answers questions in its own words, and a judge model decides whether each answer matches a reference answer.
- General Multiple Choice QA: the AI system picks one option out of a list of answer options, and its choice is compared to the correct option by exact match. Each question may have a different number of options.
Metrics
Accuracy
The fraction of questions answered correctly (range: 0.0 to 1.0), reported separately for each task. For multiple choice questions with N options, random guessing yields an expected accuracy of 1/N.
Motivation
Users rely on AI systems such as chatbots and assistants to give correct answers. An AI system that answers fluently but incorrectly can mislead users, who often cannot tell a confident wrong answer from a correct one. Measuring answer accuracy on questions that reflect the intended use of the AI system shows how far its answers can be trusted.
The two question formats complement each other. Open-ended questions reflect how users actually interact with the AI system, but require a judge model to grade answers that can be phrased in many ways. Multiple choice questions can be graded exactly and cheaply, and make it possible to compare results against random guessing, but are less representative of real usage.
Methodology
- Test Cases: The user provides two datasets: one of open-ended questions with reference answers, and one of multiple choice questions with answer options and the position of the correct option. The column names of both datasets are configurable.
- Model Response: Each open-ended question is sent to the AI system as is. Each
multiple choice question is sent together with its options labelled
A,B,C, ... and an instruction to begin the answer with the letter of the correct option. - Answer Scoring: For open-ended questions, a judge model compares the response
against the reference answer and labels it
CORRECTorINCORRECT. For multiple choice questions, the first character of the response is compared to the letter of the correct option. - Metric Computation: The mean
is_correctacross all questions yields the accuracy of each task.
Scoring
Answer Correctness (Open-Ended QA)
Answer Choice Correctness (Multiple Choice QA)
Examples
title: "Correct open-ended answer"
messages:
- role: user
content: In which year did the Berlin Wall fall?
- role: assistant
content: "The Berlin Wall fell on 9 November 1989, when East German authorities opened the border crossings."
scorers:
- name: Answer Correctness
score: 1.0
reasoning: The response states 1989, matching the reference answer; the additional details do not contradict it.
title: "Incorrect open-ended answer"
messages:
- role: user
content: What is the boiling point of water at sea level in degrees Fahrenheit?
- role: assistant
content: "Water boils at 100 degrees at sea level."
scorers:
- name: Answer Correctness
score: 0.0
reasoning: The reference answer is 212 degrees Fahrenheit; the response gives the value in Celsius, which does not answer the question as asked.
title: "Correct multiple choice answer"
messages:
- role: user
content: |
Answer the following multiple choice question. Begin your answer with the letter of the correct option, followed by any explanation.
What is the capital of France?
A. Lyon
B. Paris
C. Nice
Answer:
- role: assistant
content: "B. Paris is the capital and largest city of France."
scorers:
- name: Answer Choice Correctness
score: 1.0
reasoning: The response begins with B, the letter of the correct option.