Sample Size and Detectable Difference in Quality Measurement
Sample size and detectable difference govern whether an observed gap in a quality score reflects a real difference in capability or the ordinary variation of counting. The question arises constantly in workforce and vendor management: one site scores 88% and another 91%, and a decision follows about training, work placement or a supplier relationship. Whether that three-point gap means anything depends almost entirely on how many interactions were scored, and the answer is frequently that it means nothing at all.
Two results drive most of the practical consequences. Precision is set by the absolute number of observations, not by the share of interactions they represent — so full monitoring of a small unit still yields a small sample. And a comparison between two units is limited by whichever one you know least about — so measuring a large site more thoroughly does not make it comparable to a small one.
Throughout, an observation means one interaction that was actually scored — by a returned survey, a quality review, or automated monitoring. It is not the number of contacts handled. A site handling 20,000 contacts with a 3% survey return has 20,000 contacts and 600 observations, and it is the 600 that determines what can be concluded.
This page sets out the arithmetic, with runnable code for each step. For the separate problem of whether two units are doing comparable work in the first place, see Quality Monitoring, the case-mix discussion below, and the deployable method at Wiki:Packs/Service Quality Comparability.
First: which question is being asked
Two questions are routinely conflated, and almost every disagreement about measurement precision traces back to it.
| The question | What uncertainty applies | |
|---|---|---|
| Enumerative | "What was this unit's score last month?" | If every interaction was scored, none. The number is exact. |
| Analytic | "Is this unit's capability different from that one's — and will next month look like this?" | The period is one draw from an ongoing process. Uncertainty remains however completely it was measured. |
The distinction is Deming's.[1] An enumerative study aims to describe a frame that has already been observed; an analytic study aims at action on the process that will produce future output.
Nearly every management decision is analytic. Deciding where to place work next quarter, whether to expand a site, whether a supplier is underperforming — these are claims about a process, not about a month that has closed. So the analytic frame governs, and the arithmetic below applies even where measurement coverage is complete.
This resolves a common and genuine disagreement. Someone arguing that full monitoring removes sampling error is right about the enumerative question. Someone arguing that a small unit's score remains unreliable is right about the analytic one. Both can hold their position indefinitely because they are answering different questions.
One unit: how precise is a single score?
First, two different percentages
Two numbers on this page are both written as percentages and mean completely different things. Keeping them apart is most of the difficulty.
| What it is | Example | Does it change? | |
|---|---|---|---|
| The score | The thing being measured — the share of interactions rated positively | 90% | Yes. It moves with performance, and with luck. |
| The confidence level | How cautious we choose to be when stating a range | 95% | No. It is a convention we pick once and leave alone. |
A 95% confidence level means: if this exercise were repeated many times, the range quoted would contain the true score in about 95 of every 100 of them. It says nothing about quality. It describes how much caution is built into the range — and 95% is simply the customary choice.
So a sentence like "the score is 90%, and the 95% range is 83% to 94%" is saying: the team was rated positively on 90% of what we looked at, and the underlying truth is somewhere between 83% and 94%. The 95% never appears in the answer. It only sets how wide the range must be to be safe.
Every range on this page uses 95%. Nothing below requires thinking about it again.
What a single score can actually mean
Take a unit whose true performance is exactly 90% — it does not improve and it does not decline. Its reported score will still move around, purely because a different set of interactions gets scored each period.
How far it moves depends entirely on how many interactions were scored:
| Interactions scored | Reported score could plausibly be anywhere from… | Give or take |
|---|---|---|
| 50 | 78.6% to 95.7% | ± 8.5 pp |
| 100 | 82.6% to 94.5% | ± 6.0 pp |
| 250 | 85.7% to 93.1% | ± 3.7 pp |
| 500 | 87.1% to 92.3% | ± 2.6 pp |
| 1,000 | 88.0% to 91.7% | ± 1.9 pp |
| 3,000 | 88.9% to 91.0% | ± 1.1 pp |
| 10,000 | 89.4% to 90.6% | ± 0.6 pp |
Read the second row as: a team that is genuinely at 90% will, on 100 scored interactions, report anything between 83% and 94% — and every one of those numbers is the same team doing the same job.
The practical reading: below roughly 250 scored interactions a unit's score is not a measurement, it is an impression.
The arithmetic
The range around a proportion is a binomial confidence interval. The familiar textbook form, , behaves badly at the high proportions typical of quality scores and at small sample sizes — exactly the conditions that matter here — so the Wilson score interval is used throughout this page instead.[2]
# Paste into a notebook. Requires numpy and scipy.
import numpy as np
from scipy import stats
def wilson(k, n, conf=0.95):
# Plausible range for the true rate, given k positives out of n scored.
z = stats.norm.ppf(1 - (1 - conf) / 2)
p = k / n
d = 1 + z**2 / n
centre = (p + z**2 / (2 * n)) / d
half = z * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / d
return centre - half, centre + half
# a unit whose TRUE rate is 90%, scored at different volumes
print(f"{'scored':>10} {'plausible range':>22} {'give or take':>14}")
for n in [50, 100, 250, 500, 1000, 3000, 10000]:
lo, hi = wilson(round(0.90 * n), n)
print(f"{n:>10} {f'{lo*100:.1f}% to {hi*100:.1f}%':>22} {f'±{(hi-lo)/2*100:.1f} pp':>14}")
Two units: can you tell them apart?
Comparing two units requires the standard error of the difference:
The smallest difference distinguishable from noise is approximately — the middle column below. The third column is stricter: the gap you would reliably catch rather than merely be able to see, which needs the larger multiplier . Both use the same 95% confidence level described above.
| Interactions scored, each unit | Smallest gap you could believe | Smallest gap you'd reliably catch |
|---|---|---|
| 50 | 11.8 pp | 16.8 pp |
| 100 | 8.3 pp | 11.9 pp |
| 250 | 5.3 pp | 7.5 pp |
| 500 | 3.7 pp | 5.3 pp |
| 1,000 | 2.6 pp | 3.8 pp |
| 3,000 | 1.5 pp | 2.2 pp |
| 10,000 | 0.8 pp | 1.2 pp |
def detectable(n1, n2, p=0.90, conf=0.95, power=None):
"""Smallest difference in proportions distinguishable from noise."""
za = stats.norm.ppf(1 - (1 - conf) / 2)
se = np.sqrt(p * (1 - p) / n1 + p * (1 - p) / n2)
if power is None:
return za * se
return (za + stats.norm.ppf(power)) * se
print(f"{'obs each':>10} {'detectable':>12} {'80% power':>12}")
for n in [50, 100, 250, 500, 1000, 3000, 10000]:
print(f"{n:>10} {f'{detectable(n,n)*100:.1f} pp':>12} "
f"{f'{detectable(n,n,power=0.80)*100:.1f} pp':>12}")
A four-percentage-point gap — the kind that triggers action — requires roughly 500 scored interactions per unit before it can be distinguished from noise, and closer to 900 before it would be reliably detected when real.
Comparing a big unit with a small one
Why it works this way
Your ability to see a difference between two things is limited by whichever one you know least about.
Imagine weighing two suitcases to settle a bet about which is heavier. One goes on a laboratory scale accurate to a gram. The other goes on a bathroom scale that reads to the nearest five kilos. You cannot settle a two-kilo bet — and buying a better laboratory scale does not help you at all, because the uncertainty is all coming from the other side.
Measurement of a service unit works the same way. A large site with tens of thousands of scored interactions has a score pinned down to a fraction of a point. A small team with a hundred has a score that could sit anywhere across a ten-point band. When the two are compared, almost all of the uncertainty in the answer comes from the small team — and improving the large site's measurement is polishing a number that was already precise.
The arithmetic
The uncertainty of a comparison combines the uncertainty of both sides:
The two terms add. As the large unit's observations grow, its term shrinks toward nothing — but the small unit's term does not move, because nothing was done to the small unit. Once the first term is negligible, the whole expression is just the second one, and adding more data to the large unit changes the answer by nothing.
That remaining value is the floor: the best resolution obtainable from this comparison, set entirely by the smaller side.
With the small team fixed at 100 scored interactions and a 90% baseline:
| Observations on the large unit | Smallest difference you could detect |
|---|---|
| 100 | 8.32 pp |
| 500 | 6.44 pp |
| 1,000 | 6.17 pp |
| 10,000 | 5.91 pp |
| 100,000 | 5.88 pp |
| a billion | 5.88 pp — the floor |
print(f"{'large unit':>14} {'detectable':>12}")
for n_large in [100, 500, 1_000, 10_000, 100_000, 10**9]:
print(f"{n_large:>14,} {f'{detectable(n_large, 100)*100:.2f} pp':>12}")
# the floor: what the small unit alone permits
floor = stats.norm.ppf(0.975) * np.sqrt(0.9 * 0.1 / 100)
print(f"
floor set by the small unit alone: {floor*100:.2f} pp")
What to do instead
Three options, and the third is a legitimate answer that is almost never given.
| Option | What it does | Resolution |
|---|---|---|
| Wait. Let the small team accumulate a quarter rather than a month | 120 observations become 360 | 5.4 pp → 3.1 pp |
| Pool. Combine it with two comparable small teams and assess them as a group | 120 become 1,080 | 5.4 pp → 1.8 pp |
| Decline. State that the comparison cannot be made at this volume | — | — |
Pooling costs nothing and is usually available: small teams doing similar work under similar conditions can be assessed together, and individual scores retained for diagnosis rather than for judgement. Waiting costs only patience. Declining costs a slide.
What does not work is buying more measurement for the large unit, which is the response the numbers most often provoke because it is the one that feels like doing something.
What full monitoring does and does not fix
Moving from sampled measurement to complete monitoring is a large gain and is frequently mischaracterised in both directions.
What it fixes. It removes sampling error from the enumerative question, and it multiplies observations. Going from three per cent capture to full coverage is roughly a thirty-fold increase, which moves a mid-sized unit from an unusable interval to a decisive one. It also removes response bias — survey-based measurement is answered by a self-selected minority, and complete monitoring does not have that problem.
What it does not fix. It does not create observations that do not exist. A unit handling a hundred interactions in a period yields a hundred observations whether it is sampled or monitored in full, and remains subject to the floor described in the previous section.
The consequence is uncomfortable and worth stating plainly: full monitoring substantially improves precision for mid-sized and large units, and barely improves it for the smallest ones — which are precisely the units whose scores swing most and which are therefore most often placed under review. Volatility in a small unit's score is not evidence of instability in its operation.
Thresholds, and why stable operations get flagged
Applying a fixed threshold to a measured score, repeatedly, across several units, generates failures that have no operational cause.
Consider eight sites with identical true capability of 91.5% against a 90% target, each scored on 300 interactions a month for a year. Nothing distinguishes them. Nothing changes.
rng = np.random.default_rng(7)
sites, months, n_per, p_true, target = 8, 12, 300, 0.915, 0.90
draws = rng.binomial(n_per, p_true, size=(sites, months)) / n_per
below = draws < target
print(f"site-months below target : {below.sum()} of {sites*months}")
print(f"sites flagged at least once: {below.any(axis=1).sum()} of {sites}")
print(f"worst single site-month : {draws.min()*100:.1f}%")
Fifteen of ninety-six site-months fall below target, and all eight sites are flagged at least once during the year — despite every one of them genuinely exceeding the standard.
Two mechanisms are at work.
Repeated comparison. Any threshold applied often enough, to enough units, will be crossed by chance. The expected number of false flags grows with the number of units and the number of periods, and no individual flag carries information on its own.
A target set at the process mean. Where the target equals current average performance, roughly half of all units miss it in any period, permanently and by construction. This is worth checking before a target is defended: a goal set at the mean is a goal designed to be missed half the time.
Regression to the mean
The companion effect, and the reason remediation appears to work when it does nothing.[3]
Units selected because they scored badly were selected partly on genuine performance and partly on bad luck. The luck does not persist. Their next measurement will tend to be better whether or not anything was done.
p_true, n_per = 0.915, 300
draws = rng.binomial(n_per, p_true, size=(2000, 2)) / n_per # identical units, two periods
worst = draws[:, 0] < np.quantile(draws[:, 0], 0.15) # bottom 15% in period 1
print(f"bottom 15%, period 1 : {draws[worst,0].mean()*100:.2f}%")
print(f"same units, period 2 : {draws[worst,1].mean()*100:.2f}%")
print(f"apparent improvement : {(draws[worst,1]-draws[worst,0]).mean()*100:+.2f} pp")
print("no intervention was applied")
The selected units improve by roughly three percentage points with no intervention whatsoever. Any remediation programme triggered by a low score will therefore appear to succeed, and a programme evaluated against its own trigger cannot distinguish its effect from this one. Evaluating remediation requires a comparison group selected the same way and left alone.
Precision and bias are different problems
Everything above concerns precision — how much a measurement moves for reasons unrelated to performance. It is fixed by observations.
Bias is separate: whether the two units are doing comparable work at all. A unit handling harder cases scores lower at identical capability, and no volume of observations corrects that. Case-mix adjustment does, or comparison restricted to matched complexity bands.
The two failure modes should not be confused:
- Insufficient precision produces confident conclusions from noise.
- Uncorrected bias produces precise conclusions about the wrong thing.
A large sample of the wrong comparison is confidently wrong. Both must be addressed, and more data only addresses one of them. See Quality Monitoring and Wiki:Packs/Service Quality Comparability.
What to report
Six items, none expensive, which together prevent almost every misreading described above.
- Observation count beside every score. Not the coverage rate — the count.
- The interval, not the point estimate alone.
- The detectable difference beside every comparison, so a reader can see whether the gap clears it.
- The case-mix adjustment applied, or an explicit statement that none was.
- A suppression rule for units below a minimum count, published rather than applied silently.
- How the target was derived — external benchmark, historical baseline, or judgement — and where it sits relative to the current process mean.
Maturity Model considerations
- Levels 1–2. Scores are compared as point estimates. Small units appear volatile and are managed as though they were unstable.
- Level 3. Coverage rates are reported and improved, generally in the belief that raising coverage resolves comparability. Intervals are still absent.
- Level 4. Observation counts and intervals accompany scores, comparisons carry a detectable difference, and suppression rules are published.
- Level 5. Targets are derived rather than asserted, remediation is evaluated against a comparison group rather than against its own trigger, and precision and case-mix bias are addressed as separate problems.
See Also
- Quality Monitoring
- Quality Management in Contact Centers
- Wiki:Packs/Service Quality Comparability — the deployable comparability method
- Sourcing Design Axes: Node and Client Ownership
- Doubly Stochastic Arrivals and Demand Variance Decomposition
- Quality Management
- Multi-Skill Pooling and the Double-Counting Trap
- Statistical Process Control
References
- ↑ Deming, W.E. (1975). On probability as a basis for action. The American Statistician 29(4), 146–152.
- ↑ Agresti, A., Coull, B.A. (1998). Approximate is better than "exact" for interval estimation of binomial proportions. The American Statistician 52(2), 119–126.
- ↑ Barnett, A.G., van der Pols, J.C., Dobson, A.J. (2005). Regression to the mean: what it is and how to deal with it. International Journal of Epidemiology 34(1), 215–220.
