RAG System Evaluation SOP: Measure Retrieval and Generation Quality with Ragas/TruLens/DeepEval

The most dangerous judgment after building a RAG system is "the answer looks right." Ship on that gut call and the failures always come from the cases you never tested: retrieval missed a key clause, generation leaked the model's own knowledge into the answer, or the response drifted off-topic while sounding perfectly coherent. RAG differs from a plain LLM app-it's a pipeline (retrieval -> generation), and you must measure each stage separately, or you won't even know whether to tweak chunk size or rewrite the prompt. This SOP walks through quantifying RAG quality with three mainstream frameworks-Ragas, TruLens, and DeepEval: break quality into three dimensions, prepare an eval set, then give copy-paste code layer by layer, with pitfalls and FAQ at the end. Unlike our "Build an enterprise knowledge-base RAG with Dify" SOP, which covers building, this one covers evaluating-evaluation is the non-negotiable step between build and launch.

1. Three Quality Dimensions: Turn "Good/Bad" Into Measurable Axes

RAG quality isn't one score; it's three dimensions, each mapping to a different pipeline stage:

Remember: retrieval quality sets the ceiling, generation faithfulness sets the floor, answer relevance sets the user experience. All three must score high for a RAG to be solid. Look at only one and you misjudge-e.g., faithfulness is perfect but context recall is 0 (relevant docs not retrieved): the model is "confidently making things up from partial context." The floor is fine, but the answer is off-target.

Metric-name correspondence across the three frameworks (semantics map 1:1, names differ):

2. Prepare the Eval Set: Without ground_truth You Can Only Evaluate Half

This is the most-skipped step and the biggest source of pitfalls. The eval set caps your evaluation quality, not the framework. A minimal eval sample looks like this:

eval_samples = [
    {
        "question": "How many days of annual leave does the company offer?",  # user query
        "ground_truth": "10 days at 1 year tenure, 15 days at 10 years.",   # human-labeled gold answer
        "contexts": ["...retrieved chunk 1...", "...chunk 2..."],          # actually retrieved context
        "answer": "According to the rules, annual leave is 10 days.",       # the RAG system's actual answer
    },
    # aim for 50-100, covering factual / comparison / multi-hop / abstention questions
]

The four fields determine which metrics you can compute:

• With question + answer + contexts: you can compute faithfulness, answer_relevancy, context_precision.
• Add ground_truth: only then can you compute context_recall and answer_correctness.

This means: without ground_truth you can never measure "did retrieval miss something" and "is the answer correct"-exactly where RAG fails most. Many teams skimp and feed only question/answer/contexts, get high faithfulness and answer_relevancy, declare victory, then at launch users surface cases where retrieval missed a key doc. Prefer a small, clean eval set (50 hand-labeled) over a large, noisy one (1000 auto-generated, unlabeled).

3. Retrieval Quality: Reading context precision / recall

Retrieval is the RAG foundation. It splits two ways: how much of what should be retrieved got retrieved (recall), and whether what came back is ranked well (precision).

context_precision (Ragas) / ContextualPrecisionMetric (DeepEval): whether relevant chunks rank near the top. Ideally the relevant chunk sits at top-1/top-2; if it lands at position 8, precision is low. It needs ground_truth to judge which chunks are "relevant."

context_recall: whether all information in the gold answer was actually retrieved. Splits ground_truth into individual claims, checks each whether it can be found in contexts. Low recall means retrieval is missing things.

Here's Ragas computing all core metrics in one call:

# pip install ragas langchain-openai
from ragas import evaluate
from ragas.metrics import (
    context_precision, context_recall,
    faithfulness, answer_relevancy,
)
from datasets import Dataset

dataset = Dataset.from_list(eval_samples)   # the set from the previous section

result = evaluate(
    dataset,
    metrics=[context_precision, context_recall, faithfulness, answer_relevancy],
)
print(result)               # average score per metric
print(result.to_pandas())   # per-sample scores, to find bad cases

Don't read only the averages. A context_precision of 0.85 looks fine, but open to_pandas() and pull the sub-0.5 samples-those are the real retrieval cases to fix, usually a proper noun the embedding missed, requiring an embedding swap or a keyword-retrieval boost.

4. Generation Faithfulness: Measuring faithfulness

Faithfulness measures "is the answer hallucinated." Ragas' faithfulness works by first splitting the answer into verifiable claims, then checking each claim against the retrieved context. All supported = 1; one unsupported dings the score. It needs no ground_truth-which is its most useful trait: even without a gold answer you can detect "is the answer fabricated on top of the context."

TruLens calls this same dimension groundedness, one of its "RAG triad":

# pip install trulens trulens-provider-openai
from trulens.core import Tru
from trulens.core import Feedback
from trulens.providers.openai import OpenAI

provider = OpenAI()                                     # evaluator LLM
groundedness = Feedback(provider.groundedness_measure)  # is answer supported by context
context_rel = Feedback(provider.qs_relevance)           # is context relevant to query
answer_rel = Feedback(provider.qs_relevance)            # is answer on-topic (swap input side)
# Use .on(...).on_output() to bind feedback to the app's inputs/outputs/retrieved context.
# Wrap your RAG app with Tru recording; running the eval set auto-computes the triad.
tru = Tru()
tru.run(app=your_rag_app)

If you'd rather not pull in a framework yet, the minimum viable option is a manual LLM-as-judge prompt for faithfulness, mirroring Ragas' internal logic:

You are a strict RAG evaluator. Below are the retrieved contexts (CONTEXT) and the
system-generated answer (ANSWER). Break ANSWER into verifiable factual claims, then
judge each whether it is supported by CONTEXT.

Rules:
1. Judge only against CONTEXT; do not use your own knowledge.
2. Label each claim supported / not_supported / unverifiable.
3. Any not_supported claim means the model is fabricating; faithfulness is low.
4. Final faithfulness = supported claims / total claims (0-1).

CONTEXT: {retrieved_contexts}
ANSWER: {answer}

Low faithfulness is usually not retrieval's fault-it's a generation-stage problem: either the system prompt didn't constrain "answer only from context" and the model leaked its own knowledge, or too much irrelevant context got stuffed in and the model ran off the noise. The fix is to tighten the generation prompt, not to tune retrieval.

5. Answer Relevance: answer relevancy / correctness

answer_relevancy: whether the answer is on-topic. It reverse-generates several "possible questions" from the answer, then measures their similarity to the original query. An off-target answer (answering B when asked A) scores low. No ground_truth needed.

answer_correctness: compares the answer to ground_truth claim by claim; needs the labeled gold answer. This is the strictest metric, catching "fluent but the numbers are wrong" cases.

DeepEval usage (same idea as Ragas, different field names):

# pip install deepeval
from deepeval import evaluate
from deepeval.metrics import (
    FaithfulnessMetric, AnswerRelevancyMetric,
    ContextualPrecisionMetric, ContextualRecallMetric,
)
from deepeval.test_case import LLMTestCase

case = LLMTestCase(
    input="How many days of annual leave does the company offer?",  # = question
    actual_output="According to the rules, annual leave is 10 days.",  # = answer
    retrieval_context=["...chunk 1...", "...chunk 2..."],  # = contexts
    expected_output="10 days at 1 year tenure, 15 days at 10 years.",   # = ground_truth (optional)
)
metrics = [
    FaithfulnessMetric(),
    AnswerRelevancyMetric(),
    ContextualPrecisionMetric(),
    ContextualRecallMetric(),
]
evaluate([case], metrics)   # generates a report; bad cases flag failure reasons

Note DeepEval's field mapping: input=question, actual_output=answer, retrieval_context=contexts, expected_output=gold answer. Different names from Ragas, same semantics-when porting, swap fields per this table.

6. Pitfalls

Pitfall 1: Wanting context_recall without labeling ground_truth. context_recall and answer_correctness both depend on a gold answer. Without labels you can only evaluate faithfulness, answer_relevancy, and partial context_precision. Prefer a small, real eval set-start with 50 hand-labeled samples.

Pitfall 2: Using the same LLM as both generator and evaluator. Self-rewarding bias-the model grades its own answers systematically high. Use a stronger or different-vendor model for the evaluator (e.g., generate with GPT-4o-mini, evaluate with GPT-4o or Claude).

Pitfall 3: Drawing conclusions from too few samples. An average of 0.9 over 10 samples is statistically meaningless; one or two bad cases skew the mean. Use at least 50, covering factual, comparison, multi-hop, and abstention question types.

Pitfall 4: Reading only the average, ignoring bad cases. A 0.85 average can hide three 0-score high-risk cases. Always export per-sample results, sort ascending, and scrutinize the worst 10%-those are what will bite you in production.

Pitfall 5: context_recall is high but faithfulness is low, so you tune retrieval. Wrong direction. Retrieval already surfaced the right docs (high recall); generation didn't use them or drifted (low faithfulness). Tighten the generation prompt, don't touch chunks.

Pitfall 6: Using an English judge model on Chinese answers. The evaluator's claim-splitting and judgment on Chinese answers is systematically off-faithfulness runs low, answer_relevancy gets noisy. For Chinese scenarios pick an evaluator strong in Chinese or one specifically tuned for it; don't reach for the default English config to save effort.

FAQ

Q1: How do I choose among Ragas, TruLens, and DeepEval? Different strengths: Ragas has the most complete metrics and thickest docs, ideal for offline batch evaluation and regression testing; TruLens excels at the "RAG triad" and real-time dashboards, ideal for continuous monitoring of a live app; DeepEval's API is pytest-like, lets you assert_test directly and run in CI, ideal for engineering integration into pipelines. In one line: offline eval -> Ragas; continuous monitoring -> TruLens; CI integration -> DeepEval.

Q2: Do I have to label ground_truth? What can I evaluate without labels? You can evaluate without labels, but only half. Without ground_truth you can still compute faithfulness, answer_relevancy, and context_precision, measuring "is there hallucination, is it off-topic, are relevant chunks ranked high." But you cannot measure context_recall (missed retrieval) or answer_correctness (is it correct)-exactly the RAG failure hotspots. Strongly recommend labeling at least 50.

Q3: Which evaluator LLM? Can I use a local model? All three frameworks let you swap the evaluator LLM: Ragas wraps any LLM via LangChain; TruLens has a provider abstraction; DeepEval likewise. Local models work, but the evaluator must be strong enough-using a 7B model as judge for a 70B model's answers won't judge accurately. When cost-sensitive, do a coarse pass with a local or cheap model, then re-check uncertain bad cases with a strong model.

Q4: How much API cost per evaluation? How to save? Ragas over 100 samples x 4 metrics with GPT-4o runs in the single-digit-dollar range, mostly spent on faithfulness' claim-splitting and per-claim verification. Three ways to save: run a small subset (20 samples) to tune params before full runs; use a cheaper eval model (GPT-4o-mini) to pre-filter bad cases; re-check only low-score samples with a strong model. Ragas / DeepEval both support caching results to avoid recompute.

Q5: How do I continuously monitor a live RAG, not just evaluate once before launch? A pre-launch eval proves only "it worked that moment"-docs and query patterns keep changing. For continuous monitoring use TruLens: wrap the live app with recording, auto-compute the triad per real Q&A into a dashboard, and alert when scores drop. Ragas / DeepEval are better as scheduled regression jobs-rerun the eval set after doc updates and diff scores for regressions.

Reference Sources

• Ragas Docs - Available Metrics
• Ragas GitHub Repository
• TruLens Docs - RAG Triad
• TruLens GitHub Repository
• DeepEval Docs - Metrics Introduction
• DeepEval GitHub Repository