How to evaluate an LLM app: a practical guide to evals

Every LLM app reaches the same moment. You change a prompt to fix one bad answer, and three other answers quietly get worse. Or a new model version comes out, everyone says it's better, and you have no idea whether it's better for your use case.

The answer is evals: a repeatable way to measure how well your app performs on the tasks that matter to you. Evals have gone from a nice-to-have to the thing that decides whether a team can ship with confidence. Without them, every change is a guess.

This guide shows how to build an evaluation setup from scratch, without a big platform or a research team.

What an eval actually is

An eval is three things:

1. A set of inputs: realistic questions or tasks your app will face.
2. A way to score outputs: checks that say whether each answer is good.
3. A comparison: the scores for this version versus the last one.

Run the loop on every meaningful change.

That's it. The hard part is not the code; it's choosing good cases and good scoring.

Step 1: build a golden dataset

Your golden set is a collection of inputs with known good outcomes.

Where cases come from

• Real usage. The best cases are real user questions (anonymised), especially the ones that went wrong.
• Domain experts. Ask the people who know the answers to write the questions they'd expect.
• Edge cases. Ambiguous questions, questions with no answer in your data, multi-part questions, other languages, adversarial inputs.
• Synthetic generation, carefully. A model can draft variations of real questions, but review them; synthetic sets drift toward easy cases.

How big?

Start with 30–50 cases that cover your main use cases and known failures. That is enough to catch obvious regressions. Grow toward a few hundred as the product matures, adding every production failure you find as a new case.

What to store per case

{
  "id": "refund-policy-07",
  "input": "Can I get a refund after 45 days if the item was damaged?",
  "expected": {
    "must_include": ["damaged items", "contact support"],
    "must_not_include": ["no refunds after 30 days"],
    "reference_answer": "Damaged items can be refunded after the 30-day window; contact support with photos.",
    "source_doc": "refund-policy.md"
  },
  "tags": ["refunds", "edge-case"]
}

Tags let you break scores down by area, so you see "refunds dropped, shipping improved" instead of one blurry average.

Step 2: choose how to score

Use the cheapest scorer that works for each property, and combine them.

Code-based checks (fast, cheap, deterministic)

Use code wherever there's an objective answer:

• Format: is the output valid JSON? Does it match the schema?
• Must include / must not include: required facts present, forbidden claims absent.
• Exact or numeric match: extracted dates, totals, categories.
• Tool use: did the agent call the right tool with valid arguments?
• Length and latency: within limits?

These run in milliseconds and should cover as much as possible.

LLM-as-judge (flexible, needs care)

For qualities that code can't check (helpfulness, correctness against a reference, tone, faithfulness to sources), use a model as a grader with a clear rubric:

You are grading an answer from a customer support assistant.
 
Question: {input}
Reference answer: {reference_answer}
Assistant answer: {output}
 
Score each criterion as PASS or FAIL:
1. Correct: agrees with the reference on all key facts.
2. Grounded: makes no claims that aren't in the reference.
3. Complete: answers every part of the question.
 
Return JSON: {"correct": "...", "grounded": "...", "complete": "...", "reason": "..."}

LLM judges have known biases, so design around them:

• Use binary or small scales (PASS/FAIL, 1–3) instead of 1–10 scores, which are noisy.
• Grade one criterion at a time rather than an overall "quality" score.
• Provide a reference answer whenever possible: judging against a reference is far more reliable than judging from scratch.
• Watch for position and length bias. Judges tend to prefer the first option in pairwise comparisons and longer answers, so randomise order and penalise padding.
• Validate the judge. Have a person grade 30–50 outputs and check how often the judge agrees. If agreement is low, fix the rubric before trusting the numbers.

Human review (gold standard, expensive)

People are the ground truth for subjective quality. Use them to calibrate the judge, to spot-check new versions, and to review failures. You don't need humans to grade every run, but you need them in the loop regularly.

Step 3: evaluate retrieval separately (for RAG)

If your app retrieves documents, a wrong answer has two possible causes: retrieval found the wrong context, or generation misused the right context. Measure them separately.

If recall is low, fix search (chunking, filters, hybrid search, reranking). If recall is high but answers are wrong, fix the prompt, the context order or the model.

Step 4: make it a regression test

Evals pay off when they run automatically:

• Run the suite on every prompt, model, retrieval or tool change, just like unit tests.
• Compare against a baseline: the last version you shipped.
• Report by tag, so you see which areas moved.
• Set a bar: for example, no tag may drop more than a few points, and critical cases must all pass.
• Keep run-to-run noise in mind: LLM outputs vary, so re-run borderline cases or run each case a few times before concluding something got worse.

A simple results table in your pull request is often enough:

                 baseline   candidate   change
refunds          18/20      19/20       +1
shipping         14/15      11/15       -3  ⚠
account          10/10      10/10        0
format (JSON)    45/45      45/45        0

That ⚠ is the eval doing its job: the prompt tweak that fixed refunds broke shipping, and you found out before your users did.

Step 5: evaluate in production too

Offline evals can't cover everything users will ask. Add lightweight online signals:

• Explicit feedback: thumbs up/down, "this didn't help".
• Implicit signals: users rephrasing the same question, escalating to a human, abandoning the chat.
• Sampled judging: run your LLM judge on a small random sample of real traffic.
• Failure harvesting: every confirmed bad answer becomes a new golden case.

Keeping evals cheap

• Run code checks on everything and LLM judges only where needed.
• Use a smaller model as the judge when your agreement checks show it's good enough.
• Keep a fast smoke set (20 critical cases) for every commit and the full suite before release.
• Cache outputs for unchanged cases when only the scorer changes.

Common mistakes

• Only testing happy paths. Your golden set must include the hard and weird cases.
• One overall score. Averages hide regressions; break down by tag.
• Trusting an unvalidated judge. Always measure judge-human agreement.
• Evals that never grow. Every production failure should become a test case.
• Running evals once. Their value is in running them on every change.

Evaluating AI agents

Agents need more than "was the final answer right?", because they can reach a correct answer through a wasteful or risky path, or fail despite reasonable steps. Evaluate three levels:

1. Task success: did the agent achieve the goal? Verify in code wherever possible (the record exists, the tests pass, the email draft contains the right order number).
2. Trajectory: were the steps sensible? Did it call the right tools with valid arguments, avoid unnecessary calls and stay within budget? Some frameworks score the whole sequence of intermediate steps with an LLM judge.
3. Tool-level accuracy: for each tool call, were the arguments correct? Measuring this separately shows whether failures come from planning or from tool use.

Also track cost and steps per task: an agent that succeeds with three times the tokens may not be an improvement.

Popular evaluation tools

You can run evals with a spreadsheet and a script, but tools save time as your suite grows:

• promptfoo: open-source, config-driven testing of prompts and models that runs locally or in CI, and also supports red-teaming.
• DeepEval: an open-source Python framework with pytest-style tests and ready-made metrics.
• Ragas: metrics focused on RAG (context precision, context recall, faithfulness, answer relevancy), plus agent and tool-use metrics.
• Hosted platforms such as LangSmith and Braintrust add dataset management, experiment tracking, dashboards and production tracing.

Pick based on where you want evals to run (CI vs dashboard), your language, and whether you need production monitoring. The method in this guide matters more than the tool.

A minimal eval script

If you want to start today without any framework:

import cases from "./golden-set.json";
 
let passed = 0;
for (const c of cases) {
  const output = await answer(c.input); // your app
  const checks = [
    ...c.expected.must_include.map((s: string) => output.toLowerCase().includes(s.toLowerCase())),
    ...c.expected.must_not_include.map((s: string) => !output.toLowerCase().includes(s.toLowerCase())),
  ];
  const ok = checks.every(Boolean);
  if (ok) passed++;
  else console.log(`FAIL ${c.id}\n  ${output.slice(0, 200)}`);
}
console.log(`${passed}/${cases.length} passed`);

It's crude, but it turns "I think it's better" into a number you can compare between versions, and you can add an LLM judge later.

Frequently asked questions

What are evals in AI?

Evals (evaluations) are repeatable tests that measure how well an AI system performs on a set of realistic inputs. They combine a dataset of cases, a scoring method such as code checks, an LLM judge or human review, and a comparison between versions to catch regressions.

How do you evaluate an LLM application?

Build a golden dataset of real and edge-case inputs with expected outcomes, score outputs with code checks for objective properties and an LLM judge or humans for subjective ones, evaluate retrieval separately if you use RAG, and run the suite on every change against a baseline.

What is LLM-as-a-judge?

LLM-as-a-judge means using a language model to grade another model's outputs against a rubric or reference answer. It is flexible and cheap compared with human review, but it has biases (toward longer answers or the first option shown), so validate it against human grades before trusting it.

How many test cases do I need for LLM evals?

Start with 30–50 cases covering your main use cases and known failures; that is enough to catch obvious regressions. Grow to a few hundred over time by adding every production failure as a new case.

How do you evaluate a RAG system?

Measure retrieval and generation separately. For retrieval, check whether the right documents appear in the top results (recall and precision at K). For generation, check faithfulness to the retrieved context and correctness against a reference answer.

What's the difference between offline and online evaluation?

Offline evaluation runs a fixed test set before you ship, to compare versions. Online evaluation measures real production behaviour through user feedback, behavioural signals and sampled grading of live traffic. You need both: offline to prevent regressions, online to find what your test set missed.

How do you evaluate an AI agent?

Measure task success (verified in code where possible), the quality of the trajectory (right tools, valid arguments, no wasted steps), tool-level accuracy, and cost and steps per task. Run a set of realistic scenarios on every change and compare against the previous version.

What are the best LLM evaluation tools?

Popular options include promptfoo for config-driven testing in CI, DeepEval for pytest-style tests in Python, Ragas for RAG and agent metrics, and hosted platforms such as LangSmith and Braintrust for datasets, experiments and production tracing. A simple script is enough to start.