--- title: RAG Evaluation — context, faithfulness, correctness and coverage description: Evaluate a RAG pipeline by separating context relevance, answer faithfulness, correctness against a reference and coverage, with visible weights and uncertainty intervals. keywords: RAG evaluation, RAGAS, RAG faithfulness, context relevance, answer correctness, RAG hallucination, LLM evaluation hide: - toc - navigation - footer ---
Tools · RAG · 08

Locate the failure: retrieval, faithfulness, correctness or coverage.

Label a synthetic set of contexts and claims to see each evaluation dimension change independently. The goal is not a single magic number; it is to distinguish poor evidence retrieval from unsupported generation, incorrect answers and missing reference facts.

Contextis the retrieved evidence relevant?
Faithfulnessis each answer claim supported?
Answeris it correct and sufficiently complete?

Reference and coverage

Coverage measures completeness, not faithfulness.

Composite-score weights

While at least one weight is positive, weights are normalized automatically. If all four are zero, the composite score is disabled. It is a configurable decision aid, not a standard metric or a reproduction of RAGAS/ARES.

Context relevance——
Faithfulness——
Correctness——
Reference coverage——
Relevant contexts—
Supported claims—
Correct claims—
Reference facts covered—
Weighted score—
Diagnosis—
What to inspect next

—

—

Retrieved evidence

Mark whether each passage is relevant to the question.

6 synthetic passages
Generated answer

Keep evidence support separate from correctness against a reference.

5 synthetic claims
Method

Do not collapse “good answer” into a single cause.

Context relevance is a deterministic proportion of retrieved passages manually labelled relevant. It isolates the quality of the material that reaches the generator.

context_relevance = relevant_contexts / retrieved_contexts

Faithfulness measures the proportion of answer claims supported by retrieved context. A claim may be true in the world and still be unfaithful to the supplied context if that evidence does not support it.

faithfulness = supported_claims / answer_claims

Correctness uses a separate judgment against a reference or external criterion. This prevents “supported by context” and “correct” from becoming synonyms.

answer_correctness = correct_claims / answer_claims

Coverage measures the fraction of reference facts represented in the answer. A short answer can be perfectly faithful and correct while still being incomplete.

reference_coverage = covered_reference_facts / reference_facts

Uncertainty. Each proportion shows a 95% Wilson interval over the labelled units. An 80% score from five claims is not as stable as 80% from thousands. Treat it as a statistical interval only when units can reasonably be treated as an approximately independent, representative sample; in this synthetic scenario it mainly visualizes small-sample instability. It does not calibrate annotator or judge errors.

Diagnosis. The notice uses explicit teaching thresholds: context relevance < 50%, and faithfulness, correctness or coverage < 70%. If several dimensions fail, it reports multiple failures instead of choosing one arbitrary cause. These are not recommended production acceptance gates.

Composite score. If you must rank configurations, combine the four dimensions with explicit weights. The playground normalizes those weights and never presents the resulting score as a standard. If your product requires near-perfect faithfulness, encode that in the weights or, preferably, as a hard gate.

This is not an LLM judge. RAGAS and ARES automate parts of RAG evaluation with models/judges. Here labels are visible and manipulable so you can study which dimension changes. Production evaluation still requires a representative dataset, annotation criteria, evaluator calibration and human validation.

Sources: Es et al., RAGAS (EACL 2024) and Saad-Falcon et al., ARES (2023/2024). Both separate dimensions such as context relevance and faithfulness; ARES also emphasizes calibration with human annotations.

01RAG Retrieval LabMeasure ranking quality with Precision@k, Recall@k, MRR and nDCG before evaluating generation.Tool 02Context budgetCheck how much retrieved context actually fits beside the rest of the prompt.Tool 03Model evaluationDesign the metric from the decision you need to make, not the other way around.Concept