Building AI Data Pipelines

8 Interactive Sessions

Short, interactive sessions: watch it work, try it, then build it yourself. You can go deeper at any time in the written chapters.

1. 1

   Land the data exactly as it arrives

   See why a pipeline that runs without errors can still publish a wrong number, and why the first step is to copy the source data in exactly as it arrives.

2. 2

   Give every column its proper type

   Turn the all-text raw copy into a typed staging table you can calculate with, and see why a step that rebuilds its table every run is safe to run again.

3. 3

   Catch the bad row and set it aside

   Test staging with questions that should find nothing, and set aside the rows that fail: kept for review, kept out of the numbers, never deleted.

4. 4

   Publish the table the business reads

   Build the daily summary table from clean staging only, and see in a single number how the quality gate protects what people read.

5. 5

   Spot data that has quietly shifted

   Detect a pattern in the data that has shifted while every count stayed normal, by comparing today with a saved normal week, and keep the alarm believable.

6. 6

   Notice when the data stops arriving

   Catch the quiet failure where the pipeline runs on time but the source stopped sending data, using a freshness time limit, and keep a record of each run.

7. 7

   Let AI draft the routine work, then review every line

   Use an AI model to draft routine pipeline work (new columns, tests, plain explanations) while your own review stays the gate on every line that ships.

8. 8

   Decide which alerts wake someone up

   Turn the pipeline's checks into an alert policy people trust: wake someone only for the failures that hurt if missed, send a warning for the rest, and feed what you learn back into the checks.

How this certificate is scored

Your certificate shows how many of your written answers met the rubric below, checked by an AI grader. Anyone who checks your certificate sees it.

• You answer 8 short questions, one for each module, in your own words (at least 40 words each).
• An AI grader checks each answer against what its module asks for: it must answer the question, use the course's practice case specifically, and be correct.
• To pass, at least 7 of the 8 answers must meet that bar. The certificate says the answers were graded by AI.

The rubric

1. Module 1, Land Raw: A raw schema holding a faithful, untyped copy of the source.
2. Module 2, Type & Derive: A typed staging table with a derived resolution_minutes.
3. Module 3, Quality Gates: A quality gate that quarantines a bad row, publishing clean staging.
4. Module 4, Build the Marts: marts.daily_ticket_metrics built from clean staging.
5. Module 5, Detecting Drift: A PSI drift report vs a saved baseline + new-category detection.
6. Module 6, Freshness & Lineage: A freshness gate against an SLA + an auditable run manifest.
7. Module 7, AI at the Keyboard: AI-drafted transform, tests, and drift explanation — human-reviewed.
8. Module 8, Recall-First Monitoring: A monitor that pages only for bad-data/stale, WARNs for the rest.

What you walk away with