AI Evals: Test AI Before You Trust It
An eval is a test for an AI system: a set of questions with known good answers, and a way to score the AI's answers. You will build one for Ask Corri, the help assistant of a made-up home insurer: check a test set of 120 questions, run a scoring script, check an AI grader against people, compare two versions and find where the newer one got worse, and set up a release check that stops a worse version from going live. You run real scripts; no coding experience needed to start. All practice data is made up. Educational, not legal advice.
"We have a test set that reflects our real questions, a score we trust, and a release check that blocks a version that got worse"
6 Interactive Sessions
Short, interactive sessions: watch it work, try it, then build it yourself. You can go deeper at any time in the written chapters.
- 1
Why “it looks good” is not a test
AI answers read well even when they are wrong. A quick look misses the mistakes. An eval checks every answer against known good answers, the same way every time.
- 2
Building a test set
A test is only as good as its questions. They should match what customers really ask, include hard cases and questions the AI must refuse, and each needs a known good answer.
- 3
Scoring answers with a script
A scoring script checks every answer the same way, every time. It uses three kinds of check: exact facts, rules the answer must follow, and a graded score for open questions.
- 4
Using AI to grade AI, carefully
An AI grader can score open answers quickly, but it can be wrong in its own ways. Check it against answers people have graded before you trust its scores, and keep checking.
- 5
Comparing two versions
A better overall score can hide a topic that got worse. Compare versions topic by topic and group by group, find the cause of any drop, and explain it.
- 6
A release check that blocks a worse version
Agree the pass marks before you see the results, check every topic as well as the total, and run the check before every change. A version that fails waits until it is fixed.
Skills you'll practise
Practice data included
- 120 test questions with reference answers, tagged by topic, difficulty and customer group
- Answers from two versions of Ask Corri (A and B)
- 20 answers graded by people
What you walk away with
Shareable portfolio
A public URL showing your module timeline, patterns mastered, and completion status.
Everything you build
Download all your work as a ZIP file. It is yours to keep.
Module walkthrough
Each module documented with deliverables and the production pattern you implemented.
Ready to start AI Evals: Test AI Before You Trust It?
First course free. $20 per course after that.