Part 09

Evaluation Engineering

Offline evals on golden datasets, LLM-as-judge (pairwise + rubric), RAG metrics (faithfulness, answer relevance, context precision/recall via Ragas), agent evals (task success, trajectory, tool-call accuracy), synthetic data generation, frameworks (Promptfoo, DeepEval, Inspect, Braintrust, OpenAI Evals), online evals on production traffic, and red-teaming. Each section ships in two voices: the engineer's version (technical, with code & diagrams) and a layman's version (plain English with analogies). প্রতিটি টপিকের শেষে বাংলা ব্যাখ্যাও আছে।

1 section · 30 sources · three voices

Evaluation engineering

If you can't measure it, you can't improve it. Evaluation engineering is the discipline of turning "the model feels worse today" into numbers, then turning the numbers into a CI signal that gates every prompt or model change.

Offline evals: golden datasets, regression tests

A golden dataset is a fixed set of (input, expected_output) pairs the model is run against on every change. Build it once, grow it from real failures, never let it decay. Run it in CI; block the release if accuracy drops below threshold.

LLM-as-judge

Many qualities ("does this answer sound helpful?", "is this code idiomatic?") can't be checked with regex. The trick is to use a strong LLM as a judge, with one of two modes:

Pairwise comparison: show the judge A and B, ask which is better. Beats absolute scoring, because relative judgements are easier than absolute ones.
Rubric-based scoring: define 4–6 criteria, ask the judge to rate each 0–5 with reasoning. Aggregate weighted score.

Best practice: use a different (often stronger) model as the judge than the one being evaluated; rotate judge prompts; periodically sanity-check the judge against human labels.[18]

RAG metrics: faithfulness, answer relevance, context precision/recall

The Ragas framework standardised four metrics that every RAG system should track:[19]

Metric	Question
Faithfulness	Does the answer stick to the retrieved chunks, or does it hallucinate?
Answer relevance	Does the answer address the user's question?
Context precision	Are the retrieved chunks mostly relevant (not noise)?
Context recall	Were all the chunks needed to answer actually retrieved?

Agent evals: task success, trajectory, tool-call accuracy

For agents, "the right answer" is necessary but not sufficient — you also need to check how it got there. Track:

Task success rate (final outcome correct?)
Trajectory match (did the agent take a reasonable path — same tools, similar number of steps?)
Tool-call accuracy (did each individual tool call have the right arguments?)
Step efficiency (how many steps vs. minimum needed?)

Synthetic data generation for evals

When your golden set is too small, use an LLM to synthesise plausible (input, expected_output) pairs from your docs and seed examples. Always include a contamination check — make sure synthetic questions aren't paraphrases of training data, otherwise you're testing memorisation, not capability.

Frameworks

Framework	Strength
Promptfoo	CLI-first, YAML configs, CI-friendly, provider-agnostic
DeepEval	Pytest-style, lots of built-in metrics (G-Eval, hallucination, bias)
Inspect (UK AISI)	Research-y tasks, careful task/episode model, strong for safety evals
Braintrust	Eval UI + trace + dataset management
OpenAI Evals / Anthropic Eval templates	Vendor-curated baselines for known tasks

Online evals on production traffic

Sample 1–5% of live requests, run cheap judges (heuristics, regex, small LLM), record the eval score alongside the trace. This catches real failures your offline set misses — distribution shifts, prompt-template copy-paste errors, fresh prompt injections.

Red teaming and adversarial testing

Build a set of attacks: prompt injection, jailbreaks, PII elicitation, hallucination bait. Run them monthly. Track the failure rate; treat any new failure as a bug to fix in the system prompt or guardrail.

Practice · 4 tasks

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works

A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
Use made-up data only. Do not paste private information into a chat.

Task 1

Catch a broken prompt with a golden set

Why this matters

When you build a support bot, you will edit its prompt many times. Each edit can fix one case and quietly break another. A fixed test list, called a golden set, shows this at once.

Step 1

New chat

Copy this prompt, paste it, and press Enter. It sorts 9 support tickets into labels.

Prompt

You are a support ticket classifier. Give each ticket exactly one label: BILLING, BUG, ACCOUNT, or OTHER.
Reply with a numbered list. Write only the number and the label.

Tickets:
1. I was charged twice for my March plan.
2. The export button shows a blank page on Firefox.
3. I forgot my password and I cannot log in.
4. Do you have an office in Dhaka?
5. My invoice shows the wrong company name.
6. The app crashes when I upload a file larger than 10 MB.
7. Please change the email address on my profile.
8. I love the new dashboard!
9. The confirmation email never arrived after I paid.

You should see: A numbered list of 9 labels, one for each ticket.

Example answer

1. BILLING
2. BUG
3. ACCOUNT
4. OTHER
5. BILLING
6. BUG
7. ACCOUNT
8. OTHER
9. BUG

Your answer will be worded differently.

Step 2
Same chat
Compare your list with this answer key. Count how many labels match. Key: 1 BILLING, 2 BUG, 3 ACCOUNT, 4 OTHER, 5 BILLING, 6 BUG, 7 ACCOUNT, 8 OTHER, 9 BUG.
You should see: Most chats match all 9, so the score is 9 out of 9. One miss is fine too. Write down your score.

Step 3

New chat

Copy this prompt, paste it, and press Enter. It is the same task, with one new rule about email.

Prompt

You are a support ticket classifier. Give each ticket exactly one label: BILLING, BUG, ACCOUNT, or OTHER.
Rule: if a ticket mentions email, always label it BUG, because email problems are usually delivery bugs.
Reply with a numbered list. Write only the number and the label.

Tickets:
1. I was charged twice for my March plan.
2. The export button shows a blank page on Firefox.
3. I forgot my password and I cannot log in.
4. Do you have an office in Dhaka?
5. My invoice shows the wrong company name.
6. The app crashes when I upload a file larger than 10 MB.
7. Please change the email address on my profile.
8. I love the new dashboard!
9. The confirmation email never arrived after I paid.

You should see: A numbered list of 9 labels. Ticket 7 (change the email address) now often gets BUG.

Example answer

1. BILLING
2. BUG
3. ACCOUNT
4. OTHER
5. BILLING
6. BUG
7. BUG
8. OTHER
9. BUG

Your answer will be worded differently.

Step 4
Same chat
Compare this list with the same answer key. Count the matches. Find the ticket that is now wrong.
You should see: A lower score, for example 8 out of 9. Ticket 7 should be ACCOUNT, but the new rule made it BUG.

What just happened

The golden set is a fixed list of inputs with known right answers. You ran it before and after one change. The new email rule did not fix ticket 9, which was already right. It broke ticket 7. A regression means something that worked before breaks after a change.

You are done when You have two scores for the same 9 tickets, and you know which ticket got worse.

Check yourselfWhy do teams run the same golden set after every prompt change?Show answer

The set is fixed, so the scores are comparable. If a change breaks a case that worked before, the score drops and you see it. A single manual test would miss that. Teams can also block a release in CI when the score drops.

Try a harder versionAdd a tenth ticket that mixes two topics, such as I was charged twice and the app crashed. Decide the right label first, then run both prompts again.

Task 2

Check if an AI judge is fair

Why this matters

To test a chatbot at scale, many teams ask a second AI to judge its answers. But a judge can be unfair, for example by liking the first answer or the longer answer. You must test the judge before you trust it.

Step 1

New chat

Copy this prompt, paste it, and press Enter. Answer 1 is short and correct. Answer 2 is long and gives unsafe advice. Note which one wins.

Prompt

You are a judge. A beginner asked: "What is an API key?"
Compare the two answers. Say which one is better, Answer 1 or Answer 2, and explain in two sentences.

Answer 1:
An API key is a secret string that a program sends with each request. The service uses it to know who is calling, and to apply limits and billing. Keep it private, and never put it in public code.

Answer 2:
Great question! An API key is an incredibly powerful and versatile credential that is fundamentally essential to modern software engineering. It works just like a shared team password, so you should send it to all of your teammates in a group chat so that everyone can log in easily. It also makes your application faster and more secure in every possible way.

You should see: The judge picks Answer 1, and says that Answer 2 gives unsafe advice.

Example answer

Answer 1 is better. It correctly says an API key identifies the caller and must stay private. Answer 2 wrongly tells you to share the key in a group chat, which is unsafe.

Your answer will be worded differently.

Step 2

New chat

Copy this prompt, paste it, and press Enter. These are the same two answers, but in swapped order. Note which text wins now.

Prompt

You are a judge. A beginner asked: "What is an API key?"
Compare the two answers. Say which one is better, Answer 1 or Answer 2, and explain in two sentences.

Answer 1:
Great question! An API key is an incredibly powerful and versatile credential that is fundamentally essential to modern software engineering. It works just like a shared team password, so you should send it to all of your teammates in a group chat so that everyone can log in easily. It also makes your application faster and more secure in every possible way.

Answer 2:
An API key is a secret string that a program sends with each request. The service uses it to know who is calling, and to apply limits and billing. Keep it private, and never put it in public code.

You should see: The judge picks Answer 2. That is the same short, correct text as before. If it picks the long text, or always picks the first one, the judge has a bias.

Example answer

Answer 2 is better. It explains clearly what an API key is and says to keep it private. Answer 1 is vague and suggests sharing the key in a group chat, which is dangerous.

Your answer will be worded differently.

Step 3

New chat

Copy this prompt, paste it, and press Enter. This time the judge gives scores for the long, bad answer only.

Prompt

You are a judge. A beginner asked: "What is an API key?"
Score the answer below from 0 to 5 on each criterion. Give one short reason for each score.
Criteria: 1) Factual accuracy, 2) Clarity for a beginner, 3) Safe advice, 4) Conciseness.
End with the total score out of 20.

Answer:
Great question! An API key is an incredibly powerful and versatile credential that is fundamentally essential to modern software engineering. It works just like a shared team password, so you should send it to all of your teammates in a group chat so that everyone can log in easily. It also makes your application faster and more secure in every possible way.

You should see: Four scores from 0 to 5 with short reasons. Accuracy and Safe advice should be low, and the total should be low, for example 4 out of 20.

Example answer

1) Factual accuracy: 1/5. It calls a key a shared password, which is wrong.
2) Clarity: 2/5. It is wordy and vague.
3) Safe advice: 0/5. It tells you to share the key in a group chat.
4) Conciseness: 1/5. It uses many empty words.
Total: 4/20.

Your answer will be worded differently.

Step 4
Same chat
Look at your results from steps 1 and 2. Write down: did the judge pick the same text both times?
You should see: Most judges pick the short, correct text both times. A different pick after the swap is called position bias.

What just happened

An LLM judge is an AI that grades answers. You tested it in two ways. First, you swapped the order of the answers. A fair judge keeps the same winner. Second, you gave it a rubric, which is a list of criteria with scores. The rubric showed why the long answer is bad. Teams also compare the judge with human labels from time to time.

You are done when You know which text won in both orders, and you have a rubric score for the bad answer.

Check yourselfWhy swap the order of the two answers?Show answer

A judge can prefer the first answer, or the longer one, even when it is worse. If the winner changes after the swap, the judge is biased and its scores are not reliable.

Try a harder versionWrite a longer Answer 2 that is also correct and safe. Run step 1 again. Does the judge still pick the short answer?

Task 3

Score a RAG answer with four metrics

Why this matters

When you build a chatbot that answers from company documents (a RAG system), a wrong answer can come from two places: the search step or the writing step. Four scores help you find which one is broken.

Step 1

New chat

Copy this prompt, paste it, and press Enter. It shows a question, 3 found text chunks, and a wrong answer. The AI scores them on four metrics.

Prompt

You are a RAG evaluator. Score the system on four metrics, each from 0 to 1, with one sentence of reasoning:
1) Faithfulness: does the answer only use facts in the retrieved chunks?
2) Answer relevance: does the answer address the question?
3) Context precision: are the retrieved chunks mostly relevant?
4) Context recall: do the chunks contain everything needed for a full answer? (A fact needed for a full answer is listed under Ground truth.)

Question: Can I carry over unused leave days to next year, how many, and what happens to the rest?

Retrieved chunks:
[1] Employees may carry over up to 5 unused annual leave days into the next calendar year. Carried days expire on 31 March.
[2] The office canteen is open from 8:00 to 16:00 on working days.
[3] Sick leave does not carry over to the next year.

Answer: Yes. You can carry over up to 10 days until 31 March. HR pays the remaining days in cash at the end of the year.

Ground truth: Up to 5 days carry over until 31 March. Days above the limit are lost, and there is no cash payment.

You should see: Four scores from 0 to 1. Faithfulness is low. Answer relevance is higher, because the answer is on topic. Context precision and recall are not high.

Example answer

1) Faithfulness: 0.3. The answer says 10 days and a cash payment, but the chunks say 5 days and do not mention cash.
2) Answer relevance: 0.9. It answers the question that was asked.
3) Context precision: 0.5. Only chunk 1 is clearly useful. Chunk 3 helps a little.
4) Context recall: 0.5. The chunks do not say what happens to the extra days.

Your answer will be worded differently.

Step 2
Same chat
Copy this follow-up question, paste it, and press Enter.
Prompt
```
Which exact words in the answer are not supported by the chunks?
```
You should see: A short list. It should name 10 days and HR pays the remaining days in cash.
Example answer
```
Two parts are not supported:
- "up to 10 days": chunk 1 says up to 5 days.
- "HR pays the remaining days in cash": no chunk mentions a cash payment.
```
Your answer will be worded differently.

Step 3

Same chat

Copy this follow-up question, paste it, and press Enter.

Prompt

Which of the four scores point to a problem in the search step (the retriever), and which point to a problem in the writing step (the generator)?

You should see: Faithfulness points to the generator. Context precision and recall point to the retriever.

Example answer

Context precision and context recall point to the retriever: it found a canteen chunk and missed the rule for extra days. Faithfulness points to the generator: it added facts that are not in the chunks. Answer relevance is fine.

Your answer will be worded differently.

What just happened

A RAG answer can sound right and still be wrong. Answer relevance only asks if the answer is on topic, so it stayed high. Faithfulness asks if every fact comes from the chunks, so it caught the made-up numbers. Context precision and context recall check the search step: did it find mostly useful chunks, and did it find all the facts needed?

You are done when You have four scores, and you can say which scores point to the retriever and which point to the generator.

Check yourselfThe answer is on topic but wrong. Which metric catches this, and which one does not?Show answer

Faithfulness catches it, because the answer adds facts that are not in the chunks. Answer relevance does not catch it, because it only checks that the answer is about the question.

Try a harder versionAdd a fourth chunk: Days above the limit are lost. No cash payment is made. Ask the same chat to score again. Which scores go up?

Task 4

Grade an agent's path, not only its result

Why this matters

When you build an agent that books flights or sends emails, the final message can sound right while the work behind it is wrong or wasteful. You need to grade each step, not only the last line.

Step 1

New chat

Copy this prompt, paste it, and press Enter. It has a flight-booking agent log with 4 steps. The AI grades the run.

Prompt

You are an agent evaluator. Below is a task and the log of an agent run. Score the run on:
1) Task success (yes/no): is the final outcome correct?
2) Tool-call accuracy: for each tool call, say whether the arguments were right.
3) Trajectory: did the agent take a reasonable path?
4) Step efficiency: steps used vs the minimum needed.
Give a short verdict at the end.

Task: Book the cheapest flight from Dhaka (DAC) to Chattogram (CGP) on 12 Nov 2026. The budget is 5000 BDT.

Tools: search_flights(from, to, date), book_flight(flight_id)

Agent log:
Step 1: search_flights(from="DAC", to="CGP", date="2026-11-12") -> F1: 5200 BDT, F2: 4800 BDT, F3: 4300 BDT
Step 2: search_flights(from="DAC", to="CGP", date="2026-11-12") -> F1: 5200 BDT, F2: 4800 BDT, F3: 4300 BDT
Step 3: search_flights(from="DAC", to="CGP", date="2026-12-11") -> F4: 4100 BDT, F5: 4600 BDT
Step 4: book_flight(flight_id="F2") -> Booked F2 for 12 Nov 2026
Final answer: I booked the cheapest flight under your budget.

You should see: Task success is no. The agent booked F2 (4800 BDT), but F3 (4300 BDT) was cheaper. The grader also finds the repeated search and the wrong date in step 3.

Example answer

1) Task success: No. F2 costs 4800 BDT. F3 costs 4300 BDT and is cheaper, so the final answer is false.
2) Tool-call accuracy: Step 1 is right. Step 2 repeats step 1. Step 3 uses the wrong date, 2026-12-11. Step 4 books F2, not F3.
3) Trajectory: Poor. One search was enough.
4) Step efficiency: 4 steps used, 2 needed.
Verdict: Do not ship.

Your answer will be worded differently.

Step 2

Same chat

Copy this follow-up question, paste it, and press Enter.

Prompt

Write the shortest correct log for this task. How many steps does it have?

You should see: A log with 2 steps: one search, then book F3.

Example answer

Step 1: search_flights(from="DAC", to="CGP", date="2026-11-12") -> F1: 5200 BDT, F2: 4800 BDT, F3: 4300 BDT
Step 2: book_flight(flight_id="F3") -> Booked F3 for 12 Nov 2026
It has 2 steps.

Your answer will be worded differently.

Step 3
Same chat
Write down the steps used (4) and the steps needed. Then decide: would you ship this agent? Write one reason.
You should see: You need 2 steps, so the agent used twice as many as needed. Most people say no, because it booked the wrong flight and told the user it was the cheapest.

What just happened

The agent's last line said it booked the cheapest flight, but that was false. If you read only the last line, you would miss this. Tool-call accuracy checks the arguments of each tool call, for example the date in step 3. Step efficiency compares the steps used (4) with the least steps needed (2). Good agent evals grade the whole path.

You are done when You know the agent used 4 steps when 2 were enough, and you can name two mistakes in its path.

Check yourselfWhy is the final answer not enough to grade an agent?Show answer

An agent can say it succeeded while it used wrong arguments, wasted steps, or picked the wrong option. Grading the path (the tool calls and the number of steps) shows how the result was made and where it went wrong.

Try a harder versionChange step 4 in the log to book F3. Ask the AI to grade it again. Which scores go up, and which stay low?

Watch · 8 videos

Build Your First Eval: Creating a Custom LLM Evaluator with a Golden Dataset
Arize AI
LLM-as-a-judge: evaluating LLMs with LLMs
Evidently AI
RAG Evaluation: Precision, Recall, Faithfulness, RAGAS Explained Clearly
Logical Lenses
Introduction to Advanced Agent Evaluation Techniques
Data Science Dojo
Synthetic Data Generation for LLM Evaluators and Agents
Arize AI
Evals 101 — Doug Guthrie, Braintrust
AI Engineer
How to Run LLM Evals on Live Production Traffic with your Agents | Ep. 11
Arize AI
AI Red Teaming — Why & How to Jailbreak LLM Agents | Alex Combessie, Giskard l The Next Wave of AI
Toronto Machine Learning Society (TMLS)

References

Sources

All citations resolve to primary papers (arXiv), vendor docs (OpenAI, Anthropic, Google), or official team blogs. IDs were verified by fetching each page and extracting the citation metadata.