Playing the video loads it from YouTube, and YouTube's privacy policy applies.
Chapters
About this episode
Your AI agent passes 19 of 20 tasks. The usual error bar puts its true pass rate between 85 and 105 percent. Let it pass all 20, and the same recipe reports 100 percent, plus or minus zero. This episode shows why, and what to report instead.
Episode 2 of Six Sigma in the Age of AI teaches the central limit theorem and confidence intervals through the place they break: small AI evals. It is for AI engineers who run evals and report scores, and for quality professionals, who will recognize the fix.
Covered:
- How AI evals work, and the two sources of randomness in a score: which questions were chosen, and the model's own run-to-run variation
- The central limit theorem, and what a 95% confidence interval actually promises
- Three ways the standard CLT error bar fails on small evals: the wall at 0 and 100%, coverage that can fall far below the promised 95%, and questions that come in families
- The Green Belt rule that already flags it: at least five passes and five fails
- What to use instead: the Wilson interval, a Bayesian interval from a flat prior, and a cluster-aware Bayesian model
- Sizing an eval for the decision it supports, running each task more than once, and comparing agents on the same tasks
Every figure in the video is a hypothetical example, computed exactly.
I'm Tomasz Kaczorowski, a PhD chemist and ASQ Certified Six Sigma Green Belt. I run Smart-Suited Tech, where we build AI automation for companies that live in documents and care about quality.
Transcript
Welcome back. I'm Tom, a PhD chemist and a certified Six Sigma Green Belt. On this channel, I teach the classic tools of quality and process improvement, with an AI example next to each one. If you're new here, the first episode covers the fundamentals, and it's linked below.
In that episode, we said that before trusting an accuracy figure, you should ask: per what unit? Today we ask the next question. Out of how many? A score from 20 tests and a score from 20,000 tests can look identical, and mean very different things.
Here's the scenario. A team builds an AI agent that uses software tools to finish tasks. They test it on 20 tasks, and it passes 19. That's 95%. Then they add the standard error bar, the usual kind, and it says that the true rate lies between about 85 and 105%.
105%. An agent can't pass more tasks than exist. And if it had passed all 20, the same method would report 100%, plus or minus zero. Perfect certainty, from 20 tests. Something in the method is broken, and today we'll find out exactly what.
First, a little background. When people measure an AI model or an agent, they run an eval. Think of it as an exam. There's a set of questions or tasks, a way to grade each answer, and a final score, which is usually just a share of answers that passed.
The grader can be an answer key, or a test the output has to pass, like code that must run. More and more often it's another AI model acting as the judge, which raises the measurement questions on its own. We'll come back to this in a later episode. Today, we focus on the score.
The big public benchmarks have thousands of questions. But the evals that matter most to a business are often much smaller. A specialist benchmark for one field. A set of agent tasks, that take minutes to run. Or a company's own acceptance test, with a few dozen of real documents.
And every one of those scores is an estimate. A different set of questions would give a different score. So would the same questions, run again, because the model itself has some randomness. What we really want to know is how far the true number could be from the one we saw.
Here's a way to picture it. Think of each question as a thumbtack, where point up means pass. Drop it, and it lands point up or falls over. An easy question is a wide, heavy head on a short pin, landing nine times in ten. A hard one is nearly all pin and it falls over far more often.
Running the eval means dropping every tack once and counting how many land point up. Drop the same tacks again, and you get a slightly different count. Swap in a different box of tacks, meaning a different set of questions, and the count moves again. Those are the two kinds of randomness an error bar is supposed to capture.
If you remember the first episode, this is common cause variation. Nothing went wrong between the two runs. The process simply varies, and the error bar's job is to say by how much.
Now imagine an eval with plenty of questions, and run it thousands of times, plotting every score you get. Each drop is just pass or fail. But the scores, the averages of many drops, pile up into a smooth, symmetric hill. That's the bell curve, the normal distribution.
This is the central limit theorem. Average enough independent results, and the average follows a bell curve, almost whatever the individual results look like. That's why the normal distribution shows up all over Six Sigma. An X-bar control chart plots the average of a few parts at a time, and the average follows the bell even when single parts don't.
The bell also gets narrower as the sample grows. Its width shrinks with the square root of the number of questions. So four times the questions gives you half the width. A hundred times the questions gives you a tenth.
A confidence interval puts a number on that spread. The usual recipe measures the bell's width as a standard error, then adds and subtracts about two of them from the observed score. The result is a 95% confidence interval.
Here's what the 95% means, and it's widely misunderstood. It doesn't mean there is a 95% chance that the true value is in this particular interval. It means the recipe, used over and over, catches the true value 95 times in 100. It's a promise about the method.
And like any promise, it comes with fine print. The central limit theorem is a statement about large samples. It doesn't say how large is large enough. And small AI evals are often nowhere near it.
The first problem is a wall. A pass rate can't go below zero or above 100%. The bell curve doesn't know that. It's symmetric, and it stretches forever in both directions.
When a score sits far from the wall, that hardly matters. But good AI agents score close to 100%, right up against it. With only 20 tasks, the real spread of possible scores is lumpy and lopsided. The symmetric bell just pushes through it.
That's where our 105% came from. And here's where the plus or minus zero came from. The recipe measures uncertainty from how much the results disagree with each other. 20 passes in a row never disagree, so it measures none.
But an agent that scores 20 of 20 isn't more certain than one that scores 19 of 20. The formula only thinks so because it was handed a sample with no variation in it. A perfect score on a small test comes out looking like perfect certainty. And that's exactly backwards.
Quality people meet the same wall from the other side, because a good process has almost no defects. A Green Belt pulls 20 parts and finds one defective. The same recipe says 5%, plus or minus about 10. A negative defect rate, from the same broken method.
The second problem is quieter, and more serious. Suppose an agent's true pass rate really is 95%, and you test it on 20 tasks. Repeat that test many times, and the standard 95% interval catches the true value only about 64% of the time. Nearly all the misses are perfect scores, where the method reported no uncertainty.
For an agent whose true rate is 98%, the interval catches it only about a third of the time. The label still says 95%. The error bar looks reassuring, and is simply too narrow. With a few hundred tasks, the promise mostly holds again.
Picture it as a landscape. Across one direction runs the agent's true pass rate, and across the other, the number of tasks. The height is how often the standard error bar really catches the truth. It should be a flat plateau at 95%. Near 100%, with few tasks, it falls into canyons.
This is the warning of a 2025 position paper by Sam Bowyer, Laurence Aitchison and Desi Ivanova, presented at ICML, one of the main machine learning conferences. Its title is the advice. Don't use the central limit theorem for language model evals with fewer than a few hundred data points.
If you've studied for a Green Belt, this should sound familiar. When you approximate a pass or fail rate with the normal curve, the textbook rule is that you need at least five passes and at least five fails. Some textbooks ask for 10.
Our agent passed 19 tasks and failed one. One fail, against a minimum of five. The quality toolbox has carried this warning for decades. Machine learning is now rediscovering it.
The third problem is independence. The central limit theorem assumes each question adds new, independent evidence. In real evals, questions often come in families. 20 tasks built from one template, or 40 documents in one supplier's format.
Picture 200 test documents that came only from 8 supplier templates. If the agent handles a template well, it tends to get all that template's documents right. If it misreads the template, it can get them all wrong together. They are not 200 separate witnesses.
Depending on how tightly they cluster, those 200 documents can carry no more evidence than two or three dozen independent ones. The standard recipe still counts 200, so its error bar comes out much too narrow. It's the first episode's lesson again. Decide what your unit really is.
So what should you use instead? For a simple pass rate, use a method that respects the wall. The Wilson interval is one. It works backward, asking which true pass rates would make the score that you saw unsurprising. It's available in most statistics tools.
For our agent with 19 passes out of 20, Wilson says that the true rate is somewhere between 76 and 99%. For 20 out of 20, roughly 84 to 100. Wider, and honest. That's what 20 tests can actually tell you.
The other approach is Bayesian. Start by treating every pass rate from 0 to 100 as possible. Every pass or fail reshapes that belief a little. Because the curve lives only between 0 and 100, it never spills over the wall. And it does carry the plain reading, a 95% chance the truth is inside, given that starting assumption.
For our agent, with a flat starting belief, the Bayesian answer is about 76 to 99%. For 20 of 20, about 84 to 100. Almost exactly where Wilson landed. Two roads, one honest answer, and both of them know exactly where the wall is.
When questions come in families, use a method that knows about the families, such as a Bayesian model with a level for each cluster. At the very least, count your families, and report that number next to the score. Eight templates is a warning, whatever the document count says.
Remember the Green Belt with 20 parts and one defective? They don't argue about the error bar. They open the sampling plan, where the sample size was set before anyone inspected a part, for the decision the lot has to pass.
20 parts was never going to tell you whether that lot runs at 1% defective or 10. The plan knows that, and it says how many parts to pull for the call that you need to make. An AI eval needs the same thing before it runs.
Size the test for the decision it has to support. Say you must show an agent is above 90%, and it really runs at 95. 20 tasks can't show it at all, and you'd want about 250 to have a good chance.
Also run each task more than once, what DOE calls replicates, to see how much of the score is the model's own randomness. And when you compare two agents, give them the same tasks, and look at the differences task by task. That's a hypothesis test, and it's the topic for the next episode.
Let's recap. An eval score is an estimate, and an error bar tells you how far to trust it. The standard error bar rests on the central limit theorem, which needs plenty of data. Small evals don't have it, and scores near 100% need even more. Questions that come in families make the recipe count the wrong thing.
When that happens, the standard error bar comes out too narrow, and says you know more than you do. Use an interval that respects the wall, count your families, and size the test before you run it. None of this is new to quality professionals.
So the next time someone shows you an AI score, ask three questions. Out of how many? How independent were they? And how was the error bar calculated? Three questions, and you'll know how much the number is worth.
At Smart-Suited Tech, we build AI automation for companies that live in documents and care about quality, and we run the discipline we teach here. If it was useful, like the video and subscribe, so you won't miss the next episode on hypothesis testing. And tell me in the comments which topic you'd like after that. Thanks for watching.