Doubly Stochastic Arrivals and Demand Variance Decomposition

Demand variance decomposition separates the observed variability of contact arrivals into components that differ in what can be done about them. Under a doubly stochastic arrival model, part of the variance is irreducible randomness that no forecast can ever predict, part is structure a forecast can capture, and part is uncertainty in the arrival rate that remains after forecasting has done its best. The three parts call for different responses — pooling, forecast investment, and supply elasticity respectively — so measuring their relative sizes is a prerequisite for deciding where planning effort should go. The decomposition requires only interval-level actuals and the forecast that was published against them.

This page assumes familiarity with Poisson Process in Contact Centers, which covers the baseline arrival model and the conditions under which it holds.

The doubly stochastic model

A homogeneous Poisson process has a single source of randomness: the rate $\lambda$ is fixed, and arrivals scatter around it with the defining property $\mathrm{Var}(N)=E[N]$.

A doubly stochastic Poisson process — also called a Cox process or a mixed Poisson process — stacks two layers:

1. The rate is itself a random variable $\mathrm{\Lambda }$. A given interval-of-week does not have a demand level; it has a distribution of possible levels.
2. Conditional on $\mathrm{\Lambda }=\lambda$, arrivals in the interval are Poisson with that rate.

Applying the law of total variance to $N\mid \mathrm{\Lambda }\sim \text{Poisson}(\mathrm{\Lambda })$ gives the identity the whole method rests on:

$\mathrm{Var}(N)=\underset{\text{Poisson noise}}{\underbrace{E[\mathrm{\Lambda }]}}+\underset{\text{rate uncertainty}}{\underbrace{\mathrm{Var}(\mathrm{\Lambda })}}$

Observed variance therefore exceeds the mean whenever the rate is uncertain. Overdispersion is the signature of rate uncertainty, and there is no other source of it under this model.

The empirical case for the doubly stochastic form in contact centres is well established. Jongbloed and Koole found call arrival counts to be overdispersed relative to the Poisson distribution and proposed a Poisson mixture — imposing a probabilistic structure on the intensity — as the remedy. Brown and colleagues, in the canonical statistical study of a call centre, reached the same conclusion from operational data. Avramidis, Deslauriers and L'Ecuyer built daily arrival models around a random "busyness" factor common to the whole day.

Kim and Whitt add an important refinement. Applying Kolmogorov–Smirnov tests to banking call centre and hospital emergency arrivals, they find the data is consistent with a nonhomogeneous Poisson process — but only when analysed carefully, with the rate estimated per day rather than pooled across days. The practical reading is precise and useful: the process is Poisson given the rate; the overdispersion lives in the rate, across days. That is exactly the structure the decomposition below exploits.

Why the split governs investment

The two terms of the identity have different managerial consequences, and a further split inside the second term is where the decision actually sits.

Component	What it is	What acts on it
Poisson noise — $E[\mathrm{\Lambda }]$	Independent customers making independent decisions	Nothing forecasts it. It is buffered by staffing headroom. Because it scales as $\sqrt{\lambda }$, it shrinks in relative terms as pools grow — making it a pooling and fragmentation question, not a forecasting one
Explained rate variation	Rate movement attributable to day of week, seasonality, holidays, known events, campaigns	The only component forecast investment can convert
Residual rate variation	Rate movement left unattributed after the model has done its best	Not predictable by construction. Met only by response capability — reallocation, flexible supply, automated capacity

Three diagnoses follow, and they point in different directions:

• Explained variation dominant — a forecasting problem that is genuinely solvable. Invest in the demand side.
• Residual variation dominant — forecast investment faces a ceiling. The return is on the supply side.
• Poisson noise dominant — neither. The estate is fragmented into pools too small to average out, and consolidation or chaining is the lever.

Most operations have never separated these, which is why the forecasting-versus-flexibility argument is usually conducted on assertion.

Measuring the decomposition

What the data must contain

Tier	Fields	What it answers
A — minimum	Interval-level actual offered volume at the interval actually staffed to (15 or 30 min), by queue or skill, for 52 weeks or more; and the forecast that was published for those same intervals, at a stated lock horizon	The full three-way split, the residual rate CV, and the regime test
B — workload	Total handle seconds per interval (not average handle time), and the forecast handle time for the same intervals	Whether volatility is worse in workload than in volume, and whether volume and handle time move together
C — attribution	Forecast vintages — the same interval forecast at several lead times; contact type or complexity flag; event markers; queue open, capped and outage flags; abandonment and redial or contact-chain identifiers	Which horizon each variance component resolves at, and whether the variance sits in simple or complex work

Tier A is a low bar and is usually already available. Three numbers per cell — the mean of actuals, the variance of actuals, and the mean squared forecast error — produce the entire decomposition.

Step 1: test for overdispersion

Group interval counts into interval-of-week cells (for example, Tuesday 10:00–10:30 across all Tuesdays) and compute the index of dispersion $D=s^2/\overline{x}$. Under a pure Poisson process $D\approx 1$; the statistic $(n-1)D$ is distributed ${\chi }_{n-1}^2$, which gives a formal test.

From $D$, the rate variance and its coefficient of variation follow directly:

$\mathrm{Var}(\mathrm{\Lambda })=(D-1)\mu CV(\mathrm{\Lambda })=\sqrt{\frac{D-1}{\mu }}$

Detrend first. If trend and seasonality are left in, growth over the observation window is measured as arrival-rate randomness. Growth is predictable rate variation and belongs in the explained component, not the residual one. This is the most common error in applying the test.

Step 2: use the forecast to split the rate variance

The forecast is what separates explained from residual rate variation, and it does so through a second identity. For a forecast $F$ fixed at the time it was published:

$E[(N-F{)}^2]=E[\mathrm{\Lambda }]+E[(\mathrm{\Lambda }-F{)}^2]$

The mean squared forecast error contains the Poisson noise floor. Subtracting it isolates the rate error the forecast failed to capture:

$\underset{\text{residual rate variation}}{\underbrace{E[(\mathrm{\Lambda }-F{)}^2]}}=\text{MSE}-\mu$

Two consequences are worth stating plainly:

• A forecast whose MSE equals the mean is already perfect. All remaining error is Poisson noise, and no further forecasting investment can improve service outcomes. The ratio $\text{MSE}/\mu$ is therefore a natural efficiency measure, with 1 as the floor.
• Forecast error is not the same thing as unforecastable variation. Reporting error without subtracting the Poisson floor overstates the remaining opportunity, and does so most severely on low-volume queues. Steckley, Henderson and Mehrotra make the converse point from the modelling side: treating a forecast volume as if it were the true Poisson rate introduces significant error, because forecast errors in practice are large.

The three components then follow from three numbers:

Component	Formula
Poisson noise	$\mu$
Explained rate variation	$\mathrm{Var}(N)-\text{MSE}$
Residual rate variation	$\text{MSE}-\mu$
Total	$\mathrm{Var}(N)$

Two refinements matter for honest results. Remove forecast bias first — decompose MSE into squared bias plus error variance, since bias is a correctable process defect rather than evidence about variability. And compute the split at two levels: across all intervals, the explained component includes the intraday and day-of-week profile and will flatter the forecast; within interval-of-week cells, it measures only what the forecast adds beyond a naive seasonal baseline. The second is the honest basis for deciding on further investment.

Step 3: separate recoverable from irreducible residual

The residual component from Step 2 is what the current process failed to explain, not what is unexplainable in principle. Splitting it requires a model bake-off.

Fit a well-specified count model — a negative binomial or Poisson GLM, or a mixed-effects model with a random day effect, following the approaches compared by Ibrahim and L'Ecuyer — with day-of-week by time-of-day profile, seasonality, holiday and event indicators, trend, and available exogenous regressors. Evaluate it out of sample, on a held-out period, using the same lock horizon as the production forecast so the comparison is fair.

• Recoverable $={\text{MSE}}_{\text{current}}-{\text{MSE}}_{\text{model}}$ — a forecasting execution gap, closable with better method
• Irreducible $={\text{MSE}}_{\text{model}}-\mu$ — genuine rate uncertainty, addressable only by elasticity

Under a negative binomial fit with $\mathrm{Var}=\mu +{\mu }^2/\theta$, the dispersion parameter gives the residual rate CV directly as $1/\sqrt{\theta }$.

Step 4: the regime test

A variance figure means nothing until compared with the staffing buffer it forces. Two buffers scale differently:

• Poisson noise requires safety staffing of order $\beta \sqrt{\lambda }$
• Rate uncertainty requires a buffer of order $z\cdot CV\cdot \lambda$ — linear in volume

Their ratio is $(z/\beta )\cdot CV\cdot \sqrt{\lambda }$, so the crossover sits near $CV\approx \beta /(z\sqrt{\lambda })$. At 400 contacts in a staffing interval, that is a residual CV of roughly 5%.

This is the formal content of the result that when the arrival rate is itself random, the classical square-root safety staffing prescription must be revisited. The operational implication is sharper than the statistics suggest: where rate uncertainty dominates, the buffer grows in proportion to volume rather than to its square root, so scale stops rescuing the operation. Consolidating queues raises absolute buffer requirements even as it improves the Poisson term.

The test should use the residual CV after forecasting, not the total rate CV, because the explained portion is already handled by the forecast.

Step 5: map variance to horizon

Every variance component becomes knowable at some lead time. Every lever needs a lead time to act. Where variance resolves later than its lever requires, forecasting is structurally incapable and only elasticity remains.

If forecast vintages are available, this is directly measurable: compute MSE for the same intervals forecast at 13 weeks, 4 weeks, 1 week and 1 day, and plot how it falls. The shape of that curve is the answer.

Lever	Actuation lag
Hiring	3–6 months, gated by time to proficiency
Schedule publication	2–4 weeks
Shift bid, overtime, voluntary time off, contingent call-up	1–7 days
Intraday reallocation, offline work movement	15–60 minutes
Routing, overflow, automated capacity	Immediate

Variance that only resolves inside a lever's lag cannot be converted by that lever no matter how accurate the forecast becomes. Laying the resolution curve against this table converts the decomposition into an allocation of effort across both sides.

Worked example

A queue cell with mean 400 contacts per half-hour interval, observed variance 6,400, and a published forecast whose MSE against those intervals is 3,600.

Quantity	Calculation	Value
Index of dispersion	$6400/400$	16.0 — strongly overdispersed
Poisson noise	$\mu$	400  (6.3% of variance)
Explained rate variation	$6400-3600$	2,800  (43.7%)
Residual rate variation	$3600-400$	3,200  (50.0%)
Residual rate CV	$\sqrt{3200}/400$	14.1%
Forecast efficiency $\text{MSE}/\mu$	$3600/400$	9.0  (floor is 1.0)
Regime ratio	$(1.64/1.5)\times 0.141\times \sqrt{400}$	≈ 3.1 — rate uncertainty dominates

A model bake-off then returns an out-of-sample ${\text{MSE}}_{\text{model}}$ of 2,400. The residual splits again:

• Recoverable by better forecasting: $3600-2400=1200$ — 37.5% of the residual
• Irreducible: $2400-400=2000$ — 62.5% of the residual

The reading: this operation has both problems. There is a real forecasting execution gap worth closing, and beneath it a larger block of irreducible rate uncertainty that no forecasting programme will reach. Roughly a third of the addressable variance belongs to the demand side and two thirds to the supply side — a split that would have been invisible from forecast accuracy reporting alone, which would have shown only that the forecast was missing by an RMSE of 60.

Figures are illustrative, not measured.

Extending to handle time and workload

Staffing responds to workload, not to contacts. For $N$ contacts with handle times of mean $m$ and variance $s^2$, workload $W$ satisfies:

$CV(W{)}^2=\frac{CV(T{)}^2}{E[N]}+CV(N{)}^2$

The first term is divided by the contact count, so per-contact handle-time scatter washes out at scale and is largely irrelevant for large queues. What matters is variation in the interval mean handle time $M$ — the exact analogue of arrival-rate uncertainty. Treating $W\approx N\cdot M$:

$CV(W{)}^2\approx CV(N{)}^2+CV(M{)}^2+2\rho CV(N)CV(M)$

The covariance term is the one that is routinely omitted and that matters most in disruption-driven operations, where volume and handle time rise together and $\rho >0$. With $CV(N)=14\mathrm{\%}$, $CV(M)=10\mathrm{\%}$ and $\rho =0.5$, workload CV is 20.9% against 17.2% at $\rho =0$. Measuring $\rho$ is cheap and is frequently the difference between a plan that holds in disruption and one that does not.

Use total handle seconds per interval rather than reported average handle time, which is a ratio of averages and carries its own bias.

Data traps

• Redials are endogenous. Offered contacts include customers calling back because the operation was understaffed, which inflates measured variance and couples it to the operation's own service level. De-duplicate to customer-initiated contacts where the data allows.
• Censoring. Intervals in which the queue was closed, capped or overflowing record capacity, not demand. Left in, they bias the variance and distort the profile.
• Interval alignment. Variance measured daily and variance measured at 30 minutes are different quantities. Measure at the interval actually staffed to.
• Pooling across cells. Computing dispersion across intervals with genuinely different rates measures the profile, not the randomness — the error Kim and Whitt identify as the reason arrivals can appear non-Poisson when they are not.
• Forecast vintage drift. Comparing actuals against whichever forecast is currently in the system, rather than the one locked at the decision horizon, measures hindsight rather than forecast performance.

Interpreting the result

Pattern	Diagnosis	Where effort belongs
$D\approx 1$	Rate is stable; observed variability is irreducible noise	Pooling and fragmentation; forecast investment has no return here
High $D$, large recoverable share	Forecasting execution gap	Method, regressors, event handling, bias correction
High $D$, large irreducible share, residual CV above the regime threshold	Rate uncertainty dominates	Supply elasticity — reallocation, flexible staffing models, automated capacity
Resolution curve flat until inside the lever lag	Variance is knowable too late to convert	Elasticity only; further accuracy work cannot pay
High $\rho$ between volume and handle time	Workload volatility exceeds volume volatility	Plan on workload distributions; treat disruption as a joint event

Maturity Model considerations

• Levels 1–2. Variance is reported as forecast accuracy against a point estimate. The decomposition is not available because the Poisson floor is never separated, so all error appears addressable and forecast investment absorbs effort that cannot pay.
• Level 3. Interval data and forecast history exist, making Tier A feasible. The decomposition typically surfaces the first evidence that accuracy improvement has a ceiling.
• Level 4. Forecast vintages and scenario simulation make Step 5 measurable, and the resolution curve begins to drive lever design rather than describing it.
• Level 5. The decomposition is a standing input to capacity decisions, with the irreducible component sized and explicitly matched to elastic supply including automated capacity.

Running the test

The steps above are stated as method. In execution order, and with the effort each requires:

#	Step	Needs	Produces
1	Extract at the staffing interval, joining the forecast as locked at the decision horizon	SQL against contact and forecast tables	The analysis dataset
2	Build interval-of-week cells; exclude and count censored intervals	Grouping only	Cells with observation counts
3	Dispersion, computed within short blocks and pooled	Summary statistics	$D$, $CV(\mathrm{\Lambda })$
4	The three-way split, with bias decomposed out	Summary statistics	Poisson, explained and residual shares
5	Regime test on the residual CV	Arithmetic	Classical or uncertainty regime
6	Bake-off against a challenger model	Model fitting, out of sample	Recoverable vs irreducible residual
7	Horizon resolution curve	Repeat step 1 per lead time	Variance mapped to lever lead times
8	Workload extension	Summary statistics	$CV(M)$, $\rho$, workload CV

Only step 6 requires modelling. Every other step runs on means, variances and mean squared errors, which makes the core result reachable by a forecasting team in a day once the extract exists. The extract is usually the long pole, and the field most often missing is the forecast vintage — without it, actuals can only be compared against revised forecasts, which measures hindsight rather than forecast performance.

Use this with Claude

A ready-to-deploy instruction set and reference files for running this test are at Wiki:Packs/Demand Variance Decomposition (CP-WFM-006).

See Also

• Supply Elasticity in Workforce Planning
• Chaining and Flexibility Design
• Poisson Process in Contact Centers
• Probabilistic Forecasting
• Staffing to Percentile vs. Mean Forecast
• Forecast Accuracy Metrics
• Monte Carlo Simulation in WFM
• Discrete Event Simulation for Workforce Capacity Planning
• Intraday Reforecasting and Real-Time Forecast Updates
• Operational Volatility Index (OVIX)
• Multi-Skill Pooling and the Double-Counting Trap
• Erlang-A

For the broader methodological context, see the survey by Ibrahim and colleagues on modelling and forecasting call centre arrivals and the practice-oriented overview by Koole and Li.