Forecast Accuracy Governance (generic process)
Forecast Accuracy Governance (generic process) is the generic process card, GP-106, for the recurring process by which a workforce function scores every forecast it published at the horizon it was used for, attributes the error to the step that produced it, flags systematic bias, assigns and verifies corrective actions, and decides on that evidence which automation candidates in the forecasting processes are released or withdrawn. The measures themselves are documented on MAPE WAPE and Forecast Bias and Forecast Accuracy Metrics; the step-by-step scoring is Forecast Value Added in Workforce Management; the detection and correction of bias is Forecast Bias Detection and Correction. None of that is restated here. This card carries the operating process, who scores what and on which day, with the L0 profile, the variant classes, the L3 job aids and the automation candidates. It is part of Process Standardization and sits under the metrics and controls blocks of the forecasting section of the function catalog in Anatomy of a ROC Standard.
The process governs the other forecasting cards: the annual and monthly cycles, the weekly reforecast, the intraday reforecast and the supply forecast each record scores that this process reviews. The weekly forecast-versus-actual review is its operational first rung: that card scores and reviews one week with the floor and files the week's scores; this one scores the month, reviews the pattern across weeks with the standards committee and sets the measures the weekly review reports. Its founding idea is that a forecasting process is a sequence of steps each of which can add or destroy accuracy, and that only a step which is measured can be kept or removed on evidence.[1]
Process profile
The profile extends the L0 identity fields of Process Decomposition (L0–L3) with the process-level facts a standards committee needs before any step is drawn. Local values are held by the deploying function and marked SET LOCALLY.
| Field | Generic value |
|---|---|
| Card | GP-106 · local card id assigned by the standards committee at deployment
|
| Class | supporting, in the catalog's four-class vocabulary (strategic · real-time · both · supporting) |
| Trigger | the governance calendar: the monthly review opens when the month's actuals close and the weekly scores from the weekly review are on file, at a business-day offset (SET LOCALLY); the quarterly review coincides with the annual cycle's checkpoint; an out-of-cycle review when a bias flag exceeds a band (SET LOCALLY) |
| Cadence | weekly scores arrive from the weekly review; this process scores the month and reviews it monthly; release and withdrawal decisions are quarterly |
| Scope in | the measure set and the benchmark per horizon; the stairstep per forecasting process; the bias monitor; the adjustment and reforecast logs; the corrective action register; the release and withdrawal of forecasting automation candidates on their catch rate; the accuracy report |
| Scope out | the metric definitions; the forecasting methods; the weekly review's agenda with the floor; the mathematics of bias correction; adherence and schedule measures; individual performance management |
| Inputs | every vintage with its stored layers and decision date (Forecast Vintages); actuals at the register's definitions; the weekly scores; the adjustment logs (GP-103) and reforecast logs (GP-104); the supply vintage scores (GP-105); the bias threshold table (SET LOCALLY); the catch-rate ledger of released rules |
| Outputs | the monthly accuracy report, a stairstep per process at the horizon used, by client portfolio and segment; the bias flag list with root cause; the corrective action register with owner, due date and the measure that will show the fix; the release and withdrawal decisions; the measure-set change log; the quarterly summary |
| Upstream | every forecasting process; the vintage store; the weekly review |
| Downstream | the forecasting processes (corrective actions); GP-601 (changes to the standard); the standards committee (release recommendations); GP-604 (the published report) |
| Resources | the vintage store; a scoring routine; the bias monitor; the action register; the committee calendar. Platforms are named at L3, never here |
| Decision points | is a layer-to-layer difference real or noise? is a flagged bias a data, structural or organizational cause? which correction? is a layer that adds nothing removed? does a candidate's catch rate meet the release condition, and has a released rule fallen below it? does the measure set change? |
| Authority tier | scoring at tier 1 (forecaster); the decision per flagged item at tier 2, the planner who owns each client portfolio, in a monthly review the standardization lead chairs; release and withdrawal at tier 4 (function head) on the standards committee's recommendation; a change to the measure set goes through GP-601 |
| Risk and compliance flags | accuracy used as an individual's performance measure invites the sandbagging the bias page describes; scoring at a horizon the forecast was not used for; a layer merged into the published number; an accuracy clause in a partner contract on a definition other than the register's |
| Variant-prone steps | the benchmark for a client portfolio with no history; partner contracts with their own accuracy clause; client portfolios too small for percentage measures |
| Automation candidates | eight L2 sub-steps (five ranks), listed below |
L1 flow
One owner per step; owners are seats. Steps are scored, never seats: a human-edit layer is scored on the client portfolios and days it touched, and the absence of edits is reported as such.
- Confirm the scoring base: the month's actuals are closed, the weekly scores are on file, and every vintage used in the month is stored with its layers and its decision date (data steward).
- Score the month: every vintage used in the month, at the horizon it was used for, on the measure set per client portfolio and segment (forecaster).
- Build the stairstep per forecasting process, layer by layer against the layer before and against the naive benchmark (forecaster).
- Decision: is each layer's difference larger than noise under the paired test? If not, record "no measurable value" for that layer (forecaster).
- Run the bias monitor by segment against the thresholds (SET LOCALLY) (forecaster).
- Decision: is any segment in breach? If yes, open a root-cause item classed as data, structural or organizational before any correction is applied (forecaster).
- Review the adjustment and reforecast logs: direction and size of each adjustment against outcome; an adjustment repeated every period is flagged for the model (forecaster).
- Score the supply vintage separately and attribute its error (supply planner).
- Monthly review: chair the review at which the planner who owns each client portfolio produces a decision per flagged item, keep, correct, remove the layer or escalate (standardization lead).
- Assign corrective actions with an owner, a due date and the measure that will show the fix (planner).
- Decision, quarterly: does a candidate's published catch rate meet the release condition, and has any released rule fallen below it? Release or withdraw on the standards committee's recommendation, with a stated reason (function head).
- Publish the report in the answer-first format to the function head and the standing forums (standardization lead).
- Close the loop: verify that last month's corrective actions resolved the flag or moved it; log lessons; raise change requests to the standard through GP-601 (forecaster).
L2 step table
Generic sub-steps only; the deploying function adds the columns Process Decomposition (L0–L3) leaves open. Durations are typical business days [A], SET LOCALLY.
| Step | Sub-step | Owner (seat) | Inputs | Outputs | Duration | Automation candidate | Documentation join key |
|---|---|---|---|---|---|---|---|
| 1 | 1.1 Check actuals closed, weekly scores filed, every vintage stored with layers and decision date | data steward | the vintage store, the actuals feed, the weekly scores | the scoring base | 0.5 BD | Y | scoring base checklist |
| 2 | 2.1 Score each vintage on the measure set at the horizon used | forecaster | the base, the measure set | the score table | 1 BD | Y | scoring routine instruction |
| 3 | 3.1 Build the stairstep per process by client portfolio and segment | forecaster | the score table | the stairstep | 0.5 BD | Y | stairstep template |
| 4 | 4.1 Run the paired test per layer; mark unmeasurable gains | forecaster | the stairstep | the tested stairstep | 0.5 BD | Y | significance test instruction |
| 5 | 5.1 Compute the bias measures by segment against thresholds | forecaster | the score table, the threshold table | the flag list | 0.5 BD | Y | bias monitor layout |
| 6 | 6.1 Open a root-cause item per breach, classed data, structural or organizational | forecaster | the flag list, the logs | the root-cause items | 2 BD | N | root-cause protocol |
| 7 | 7.1 Compare each logged adjustment with its outcome; flag repeats for the model | forecaster | the adjustment and reforecast logs | the adjustment review | 1 BD | N | adjustment review template |
| 8 | 8.1 Score the supply vintage and attribute the error | supply planner | the supply vintage, the roster | the supply score | 1 BD | Y | vintage scoring template |
| 9 | 9.1 Circulate the review pack ahead of the meeting | standardization lead | steps 2 to 8 | the review pack | 1 BD | Y | monthly review pack |
| 9 | 9.2 Produce a decision per flagged item | planner | the review pack | the decision record | 0.5 BD | N | monthly review pack |
| 10 | 10.1 Enter each action with owner, due date and proving measure | planner | the decision record | the action register | 0.5 BD | N | corrective action register |
| 11 | 11.1 Compare each candidate's catch rate with the release condition; list rules below it | standardization lead | the catch-rate ledger | the recommendation list | 1 BD | N | release and withdrawal form |
| 11 | 11.2 Decide release or withdrawal with a stated reason | function head | the recommendation list | the signed decision | 1 BD | N | release and withdrawal form |
| 12 | 12.1 Assemble the report in the answer-first format | standardization lead | the review outputs | the draft report | 0.5 BD | Y | report template |
| 12 | 12.2 Review the narrative and publish | standardization lead | the draft report | the published report | 0.5 BD | N | report template |
| 13 | 13.1 Verify last month's actions against their proving measure | forecaster | the action register, the score table | closed or carried actions | 0.5 BD | N | corrective action register |
| 13 | 13.2 Raise change requests to the standard | forecaster | the lessons, the measure-set change log | change requests | as needed | N | GP-601 intake |
Variants
The known variant classes for this process, recorded per Standard and Variant in Process Documentation:
- Statutory. No statutory variant changes a step. Where an accuracy figure forms part of a regulated service report, the retention period of the vintages and scores is a parameter, SET LOCALLY.
- Contractual. A partner contract with its own accuracy clause, on its own measure and horizon, takes a variant row at steps 2 and 12: the partner view is scored on the contract's measure beside the register's, never instead of it. Owned by the placement lead, reviewed at renewal.
- Structural. A client portfolio too small for percentage measures substitutes a scale-free measure; that is a parameter of the measure set per client portfolio, SET LOCALLY. A new client portfolio with no history for a naive benchmark takes a variant row at step 3, scored against the forecast of record only, with a review at its first full year.
- Transitional. A unit from a predecessor organization whose vintages are not stored with layers is scored on the total only under a variant row with expiry at cut-over; its automation effect reads blocks against rank 3 for that unit.
L3 register
A localizing team writes these job aids against the generic L2; names only, the content is the function's. The measure set per horizon and client portfolio, the bias thresholds and the release condition are L3 values marked SET LOCALLY in the L2 row.
| Job aid | Anchored to | Form |
|---|---|---|
| The measure set per horizon and client portfolio, with the benchmark | step 2 | work instruction; choice on MAPE WAPE and Forecast Bias |
| The scoring base checklist | step 1 | work instruction |
| The scoring routine and the stairstep template | steps 2 and 3 | work instruction; method on Forecast Value Added in Workforce Management |
| The significance test instruction | step 4 | work instruction; method on Model Evaluation and Validation |
| The bias threshold table and monitor layout | step 5 | work instruction; detection on Forecast Bias Detection and Correction |
| The root-cause protocol: data, structural, organizational | step 6 | playbook |
| The adjustment review template | step 7 | work instruction |
| The monthly review pack and the decision record | step 9 | work instruction |
| The corrective action register | steps 10 and 13 | work instruction |
| The release and withdrawal form, with the catch-rate evidence | step 11 | playbook; condition on Human Gates and Number Grades |
| The report template | step 12 | work instruction; format on Answer-First Reporting |
| The out-of-cycle review path on a bias breach | the trigger | runbook |
Automation candidates
Flagged at L2 per the method on Process Decomposition (L0–L3); released per class of action on a published catch rate per Human Gates and Number Grades, never on a maturity label. The list is ordered by how easy a catch rate is to publish, and every row marked Y above appears in it.
- The scoring base check (1.1): deterministic; a missing vintage or an unclosed month is unambiguous, and first because it is caught the day it occurs.
- Vintage scoring (2.1, 8.1): computed, with the right answer defined by the measure set and the actuals; it ranks after the base check because a score is only as good as the base it was run on.
- The stairstep and the paired test (3.1, 4.1): computed, and the test that separates a real gain from noise is itself a rule;[2] it ranks after scoring because the test consumes the scores and its own errors need several months of layers to surface.
- The bias monitor (5.1): computed on a tracking signal and a bias ratio against thresholds;[3] it ranks after the stairstep because a false flag is only known to be false after the root-cause item it opened has been closed.
- Report assembly (9.1, 12.1): assembled from the review outputs with the narrative reviewed; last because an error here is caught by the reader rather than by a rate of its own.
Three steps are never candidates. The decision per flagged item (9.2) is the gate at which a planner must produce a keep, correct, remove or escalate, and a gate that asks only for a click is not a gate. The release and withdrawal decision (11.2) is the gate that governs every other gate in the forecasting pillar, and a rule cannot certify its own release. The root-cause classification (6.1) stays human because the organizational causes of bias, sandbagging and anchoring among them, are not in the data the monitor reads; the root-cause protocol on Forecast Bias Detection and Correction places the cause question before any correction for that reason. The choice of measure follows the scale-free arguments of the forecasting literature, which is why the measure set is a parameter per client portfolio rather than a single figure for the whole operation.[4]
Maturity Model Position
Four scales on this wiki use the word level; the launch page states which is which. On the WFM Labs Maturity Model™ a function that scores every published forecast on a defined measure at a defined cadence and keeps an action register is at Level 2; one that scores by layer, tests for significance, monitors bias by segment automatically and has released ranks 1 to 4 above on a published catch rate is at Level 3; one that uses the stairstep as the release list for forecasting automation, queue by queue, under the earned-learning and earned-auto-approval rules of Forecast Value Added in Workforce Management, is at Level 4.
See Also
- Process Standardization — the hub and the inventory this card belongs to
- Forecast Value Added in Workforce Management — the stairstep and the earned-automation rules
- MAPE WAPE and Forecast Bias — the measures and their failure modes
- Forecast Accuracy Metrics — the wider measure set
- Forecast Bias Detection and Correction — detection, correction and the organizational causes
- Forecast Vintages — the bookkeeping that makes scoring at the horizon used possible
- Weekly Forecast-vs-Actual Review (generic process) — the operational first rung
- Weekly Rolling Reforecast (generic process) — the process whose layers are scored
- Standards Governance and Change Control (generic process) — where a measure-set change goes
- Standard and Variant in Process Documentation — how the variants above are recorded
- Wiki:Packs/Process Decomposition — the pack that localizes this card (CP-OPS-001)
References
- ↑ Gilliland, M. (2010). The Business Forecasting Deal: Exposing Myths, Eliminating Bad Practices, Providing Practical Solutions. John Wiley & Sons. Chapter on forecast value added analysis.
- ↑ Diebold, F. X., & Mariano, R. S. (1995). "Comparing Predictive Accuracy". Journal of Business & Economic Statistics 13(3), 253–263.
- ↑ Trigg, D. W. (1964). "Monitoring a Forecasting System". Operational Research Quarterly 15(3), 271–274.
- ↑ Hyndman, R. J., & Koehler, A. B. (2006). "Another look at measures of forecast accuracy". International Journal of Forecasting 22(4), 679–688.
