Locate the failure: retrieval, faithfulness, correctness or coverage.
Label a synthetic set of contexts and claims to see each evaluation dimension change independently. The goal is not a single magic number; it is to distinguish poor evidence retrieval from unsupported generation, incorrect answers and missing reference facts.
—
—
Mark whether each passage is relevant to the question.
Keep evidence support separate from correctness against a reference.
Do not collapse “good answer” into a single cause.
Context relevance is a deterministic proportion of retrieved passages manually labelled relevant. It isolates the quality of the material that reaches the generator.
Faithfulness measures the proportion of answer claims supported by retrieved context. A claim may be true in the world and still be unfaithful to the supplied context if that evidence does not support it.
Correctness uses a separate judgment against a reference or external criterion. This prevents “supported by context” and “correct” from becoming synonyms.
Coverage measures the fraction of reference facts represented in the answer. A short answer can be perfectly faithful and correct while still being incomplete.
Uncertainty. Each proportion shows a 95% Wilson interval over the labelled units. An 80% score from five claims is not as stable as 80% from thousands. Treat it as a statistical interval only when units can reasonably be treated as an approximately independent, representative sample; in this synthetic scenario it mainly visualizes small-sample instability. It does not calibrate annotator or judge errors.
Diagnosis. The notice uses explicit teaching thresholds: context relevance < 50%, and faithfulness, correctness or coverage < 70%. If several dimensions fail, it reports multiple failures instead of choosing one arbitrary cause. These are not recommended production acceptance gates.
Composite score. If you must rank configurations, combine the four dimensions with explicit weights. The playground normalizes those weights and never presents the resulting score as a standard. If your product requires near-perfect faithfulness, encode that in the weights or, preferably, as a hard gate.
This is not an LLM judge. RAGAS and ARES automate parts of RAG evaluation with models/judges. Here labels are visible and manipulable so you can study which dimension changes. Production evaluation still requires a representative dataset, annotation criteria, evaluator calibration and human validation.
Sources: Es et al., RAGAS (EACL 2024) and Saad-Falcon et al., ARES (2023/2024). Both separate dimensions such as context relevance and faithfulness; ARES also emphasizes calibration with human annotations.