Do not score an answer as one indivisible block. Extract atomic claims, retrieve evidence, test whether that evidence entails each claim, check citation placement, and let the system abstain when support is insufficient.
White nodes are evidence passages. Claim nodes turn green when supported, yellow when omitted by abstention, and pink when the answer keeps an unsupported statement.
An atomic claim expresses one independently verifiable proposition. Evaluation retrieves candidate passages, checks whether they entail or contradict the claim, and preserves source provenance. Topical similarity is useful for retrieval, but a passage can discuss the same subject without supporting the statement.
“The mission launched in 1969 and lasted eight days” contains two claims. One citation may support only half. Split conjunctions, quantities, dates, comparisons, and causal assertions.
Prefer primary, authoritative, and current sources. Record URL, passage, retrieval time, and document version so support can be inspected later.
Ask whether the evidence logically supports the exact claim, not whether words overlap. Negation, units, scope, and time qualifiers often decide the result.
Removing unsupported claims raises groundedness but can reduce usefulness. Evaluate both reliability and retained answer coverage instead of rewarding silence alone.
The fraction of included claims with adequate evidence. It penalizes unsupported content but does not reward an answer for covering everything the user needed.
ReliabilityThe fraction of included claims below the entailment threshold. Contradicted claims should be tracked separately because they are more severe than merely missing support.
RiskThe fraction of attached citations that actually support their local claims. A citation-rich answer can still have poor precision when references are decorative or misplaced.
Attribution qualityThe retained fraction of requested claims after abstention, weighted here by support. Production evaluation should use task-specific coverage and user-value judgments.
HelpfulnessRetrieval proposes evidence. Verification judges support. Combining both into one opaque score hides whether a failure came from missing documents, poor ranking, or incorrect entailment.
Segment the answer into propositions while retaining sentence offsets. Mark subjective recommendations, unverifiable opinions, and instructions separately from factual claims.
Search multiple authoritative sources. Use semantic and lexical retrieval, then rerank passages for the exact claim. Keep contradictory evidence rather than discarding it.
Evaluate each claim-passage pair for entailment, contradiction, or insufficient information. Require citation placement close enough that readers know which assertion it supports.
Report claim-level results alongside answer-level summaries. Break down performance by claim type, source, answer length, scenario, and confidence threshold.
It can assist, but evaluator bias, shared misconceptions, prompt sensitivity, and source interpretation errors remain. Calibrate against expert labels and retain auditable evidence passages.
No. A citation may be irrelevant, unreliable, outdated, or misinterpreted. Citation correctness requires both source quality and local entailment.
Classify them as insufficiently supported rather than false. The system can abstain, qualify uncertainty, or request more information.
Production retrieval can return truncated, stale, adversarial, or low-quality passages. Robust evaluation should degrade evidence quality deliberately and observe whether the answer policy becomes safer.
Explore SuperPowers workflows that turn explicit goals into inspectable agent work, with reusable computer state and complete website generation.
Explore SuperPowersComputer-use cacheWebsite-building agent