Six Sigma in the Age of AI

How Sure Is That AI Score? Error Bars for LLM Evals and the Central Limit Theorem

Episode 2 of Six Sigma in the Age of AI teaches the central limit theorem and confidence intervals through the place they break: small AI evals.

Published Length 16:34

Playing the video loads it from YouTube, and YouTube's privacy policy applies.

Chapters

About this episode

Your AI agent passes 19 of 20 tasks. The usual error bar puts its true pass rate between 85 and 105 percent. Let it pass all 20, and the same recipe reports 100 percent, plus or minus zero. This episode shows why, and what to report instead.

Episode 2 of Six Sigma in the Age of AI teaches the central limit theorem and confidence intervals through the place they break: small AI evals. It is for AI engineers who run evals and report scores, and for quality professionals, who will recognize the fix.

Covered:

  • How AI evals work, and the two sources of randomness in a score: which questions were chosen, and the model's own run-to-run variation
  • The central limit theorem, and what a 95% confidence interval actually promises
  • Three ways the standard CLT error bar fails on small evals: the wall at 0 and 100%, coverage that can fall far below the promised 95%, and questions that come in families
  • The Green Belt rule that already flags it: at least five passes and five fails
  • What to use instead: the Wilson interval, a Bayesian interval from a flat prior, and a cluster-aware Bayesian model
  • Sizing an eval for the decision it supports, running each task more than once, and comparing agents on the same tasks

Every figure in the video is a hypothetical example, computed exactly.

I'm Tomasz Kaczorowski, a PhD chemist and ASQ Certified Six Sigma Green Belt. I run Smart-Suited Tech, where we build AI automation for companies that live in documents and care about quality.

Transcript

{{ series }}

{{ title }}

{{ summary }}

Published Length {{ duration }}

{{ playerIframe }}

{{ privacyNote }}

This episode's copy (content.js beside this page) could not be loaded.

Chapters

About this episode

{{ para.text }}

{{ para.lead }}

  • {{ item.text }}

Transcript

{{ transcriptLabel }}

All lessons