← All courses
$20

AI Specs & Evals for PMs

The product manager's lens on AI: write a feature spec an engineer can build from, then define and read the eval that says whether it's good enough to ship. Runs a real, deterministic eval over a support-triage assistant — and teaches the skill that matters most: a single accuracy number will lie to you. Learn to find the false pass the headline metric hides.

"87.5% accuracy looked shippable — the eval caught the tax question it hallucinated. That's the failure that ships."

6 Interactive Sessions

Short, interactive sessions: watch it work, try it, then build it yourself. You can go deeper at any time in the written chapters.

  1. 1

    Decide what the assistant must not do

    Before anything is built, write down the job an AI feature is for and, just as carefully, what it must refuse or hand to a person. Those non-goals are what every later step tests.

  2. 2

    Write a spec an engineer can build and a test can check

    Turn the frame into a spec with two concrete parts: the exact fields the assistant returns, and the rules for when it must not answer. Each rule you write is also a test you can run.

  3. 3

    Decide what good means before you see a score

    Turn "is it good?" into a few measurements with pass marks agreed in advance. Most can be averaged. The one that guards against harm has to be a floor that nothing else can make up for.

  4. 4

    Write the cases that can catch a failure

    An eval only knows about the tickets you wrote down. Build a set of cases that covers every refusal rule, tag each case so the eval knows what a pass means, and make sure each case is scored by a gate.

  5. 5

    Read the report and find the case the headline hides

    Run the eval, read every score against its gate, and find the one ticket a reassuring headline number waves through.

  6. 6

    Write the go/no-go memo

    Turn the eval report into a decision a person owns: the call, the evidence, the condition that would change it, and what to watch once the assistant is live.

Skills you'll practise

AI Feature SpecsOutput Contracts & GuardrailsEval RubricsEval Sets (adversarial/must-refuse)Reading an Eval ReportEval-Gated Go/No-Go

Practice data included

  • AI feature spec
  • Eval rubric (dimensions + gates)
  • Eval case set (happy/adversarial/must-refuse)
  • Pre-recorded assistant outputs
  • Go/no-go memo

How this certificate is scored

Your certificate shows how many of your written answers met the rubric below, checked by an AI grader. Anyone who checks your certificate sees it.

  • You answer 6 short questions, one for each module, in your own words (at least 40 words each).
  • An AI grader checks each answer against what its module asks for: it must answer the question, use the course's practice case specifically, and be correct.
  • To pass, at least 5 of the 6 answers must meet that bar. The certificate says the answers were graded by AI.

The rubric

  1. Module 1, Frame the Problem: A framed problem: users, JTBD, non-goals, where AI fails.
  2. Module 2, Write the AI Spec: An AI spec: output contract + guardrails/refusal rules.
  3. Module 3, Define Good: A rubric of measurable dimensions + ship thresholds.
  4. Module 4, Build the Eval Set: Happy + adversarial + must-refuse eval cases.
  5. Module 5, Read an Eval Report: A read that finds the false pass and the failing gate.
  6. Module 6, Go / No-Go Memo: An eval-gated decision + post-launch watchlist.

What you walk away with

Shareable portfolio

A public URL showing your module timeline, patterns mastered, and completion status.

Everything you build

Download all your work as a ZIP file. It is yours to keep.

Module walkthrough

Each module documented with deliverables and the production pattern you implemented.

Ready to start AI Specs & Evals for PMs?

First course free. $20 per course after that.