Forecast Accuracy Governance (generic process)

From WFM Labs

Forecast Accuracy Governance (generic process) is the generic process card, GP-106, for the recurring process by which a workforce function scores every forecast it published at the horizon it was used for, attributes the error to the step that produced it, flags systematic bias, assigns and verifies corrective actions, and decides on that evidence which automation candidates in the forecasting processes are released or withdrawn. The measures themselves are documented on MAPE WAPE and Forecast Bias and Forecast Accuracy Metrics; the step-by-step scoring is Forecast Value Added in Workforce Management; the detection and correction of bias is Forecast Bias Detection and Correction. None of that is restated here. This card carries the operating process, who scores what and on which day, with the L0 profile, the variant classes, the L3 job aids and the automation candidates. It is part of Process Standardization and sits under the metrics and controls blocks of the forecasting section of the function catalog in Anatomy of a ROC Standard.

The process governs the other forecasting cards: the annual and monthly cycles, the weekly reforecast, the intraday reforecast and the supply forecast each record scores that this process reviews. The weekly forecast-versus-actual review is its operational first rung: that card scores and reviews one week with the floor and files the week's scores; this one scores the month, reviews the pattern across weeks with the standards committee and sets the measures the weekly review reports. Its founding idea is that a forecasting process is a sequence of steps each of which can add or destroy accuracy, and that only a step which is measured can be kept or removed on evidence.[1]

Process profile

The profile extends the L0 identity fields of Process Decomposition (L0–L3) with the process-level facts a standards committee needs before any step is drawn. Local values are held by the deploying function and marked SET LOCALLY.

Field Generic value
Card GP-106 · local card id assigned by the standards committee at deployment
Class supporting, in the catalog's four-class vocabulary (strategic · real-time · both · supporting)
Trigger the governance calendar: the monthly review opens when the month's actuals close and the weekly scores from the weekly review are on file, at a business-day offset (SET LOCALLY); the quarterly review coincides with the annual cycle's checkpoint; an out-of-cycle review when a bias flag exceeds a band (SET LOCALLY)
Cadence weekly scores arrive from the weekly review; this process scores the month and reviews it monthly; release and withdrawal decisions are quarterly
Scope in the measure set and the benchmark per horizon; the stairstep per forecasting process; the bias monitor; the adjustment and reforecast logs; the corrective action register; the release and withdrawal of forecasting automation candidates on their catch rate; the accuracy report
Scope out the metric definitions; the forecasting methods; the weekly review's agenda with the floor; the mathematics of bias correction; adherence and schedule measures; individual performance management
Inputs every vintage with its stored layers and decision date (Forecast Vintages); actuals at the register's definitions; the weekly scores; the adjustment logs (GP-103) and reforecast logs (GP-104); the supply vintage scores (GP-105); the bias threshold table (SET LOCALLY); the catch-rate ledger of released rules
Outputs the monthly accuracy report, a stairstep per process at the horizon used, by client portfolio and segment; the bias flag list with root cause; the corrective action register with owner, due date and the measure that will show the fix; the release and withdrawal decisions; the measure-set change log; the quarterly summary
Upstream every forecasting process; the vintage store; the weekly review
Downstream the forecasting processes (corrective actions); GP-601 (changes to the standard); the standards committee (release recommendations); GP-604 (the published report)
Resources the vintage store; a scoring routine; the bias monitor; the action register; the committee calendar. Platforms are named at L3, never here
Decision points is a layer-to-layer difference real or noise? is a flagged bias a data, structural or organizational cause? which correction? is a layer that adds nothing removed? does a candidate's catch rate meet the release condition, and has a released rule fallen below it? does the measure set change?
Authority tier scoring at tier 1 (forecaster); the decision per flagged item at tier 2, the planner who owns each client portfolio, in a monthly review the standardization lead chairs; release and withdrawal at tier 4 (function head) on the standards committee's recommendation; a change to the measure set goes through GP-601
Risk and compliance flags accuracy used as an individual's performance measure invites the sandbagging the bias page describes; scoring at a horizon the forecast was not used for; a layer merged into the published number; an accuracy clause in a partner contract on a definition other than the register's
Variant-prone steps the benchmark for a client portfolio with no history; partner contracts with their own accuracy clause; client portfolios too small for percentage measures
Automation candidates eight L2 sub-steps (five ranks), listed below

L1 flow

One owner per step; owners are seats. Steps are scored, never seats: a human-edit layer is scored on the client portfolios and days it touched, and the absence of edits is reported as such.

  1. Confirm the scoring base: the month's actuals are closed, the weekly scores are on file, and every vintage used in the month is stored with its layers and its decision date (data steward).
  2. Score the month: every vintage used in the month, at the horizon it was used for, on the measure set per client portfolio and segment (forecaster).
  3. Build the stairstep per forecasting process, layer by layer against the layer before and against the naive benchmark (forecaster).
  4. Decision: is each layer's difference larger than noise under the paired test? If not, record "no measurable value" for that layer (forecaster).
  5. Run the bias monitor by segment against the thresholds (SET LOCALLY) (forecaster).
  6. Decision: is any segment in breach? If yes, open a root-cause item classed as data, structural or organizational before any correction is applied (forecaster).
  7. Review the adjustment and reforecast logs: direction and size of each adjustment against outcome; an adjustment repeated every period is flagged for the model (forecaster).
  8. Score the supply vintage separately and attribute its error (supply planner).
  9. Monthly review: chair the review at which the planner who owns each client portfolio produces a decision per flagged item, keep, correct, remove the layer or escalate (standardization lead).
  10. Assign corrective actions with an owner, a due date and the measure that will show the fix (planner).
  11. Decision, quarterly: does a candidate's published catch rate meet the release condition, and has any released rule fallen below it? Release or withdraw on the standards committee's recommendation, with a stated reason (function head).
  12. Publish the report in the answer-first format to the function head and the standing forums (standardization lead).
  13. Close the loop: verify that last month's corrective actions resolved the flag or moved it; log lessons; raise change requests to the standard through GP-601 (forecaster).

L2 step table

Generic sub-steps only; the deploying function adds the columns Process Decomposition (L0–L3) leaves open. Durations are typical business days [A], SET LOCALLY.

Step Sub-step Owner (seat) Inputs Outputs Duration Automation candidate Documentation join key
1 1.1 Check actuals closed, weekly scores filed, every vintage stored with layers and decision date data steward the vintage store, the actuals feed, the weekly scores the scoring base 0.5 BD Y scoring base checklist
2 2.1 Score each vintage on the measure set at the horizon used forecaster the base, the measure set the score table 1 BD Y scoring routine instruction
3 3.1 Build the stairstep per process by client portfolio and segment forecaster the score table the stairstep 0.5 BD Y stairstep template
4 4.1 Run the paired test per layer; mark unmeasurable gains forecaster the stairstep the tested stairstep 0.5 BD Y significance test instruction
5 5.1 Compute the bias measures by segment against thresholds forecaster the score table, the threshold table the flag list 0.5 BD Y bias monitor layout
6 6.1 Open a root-cause item per breach, classed data, structural or organizational forecaster the flag list, the logs the root-cause items 2 BD N root-cause protocol
7 7.1 Compare each logged adjustment with its outcome; flag repeats for the model forecaster the adjustment and reforecast logs the adjustment review 1 BD N adjustment review template
8 8.1 Score the supply vintage and attribute the error supply planner the supply vintage, the roster the supply score 1 BD Y vintage scoring template
9 9.1 Circulate the review pack ahead of the meeting standardization lead steps 2 to 8 the review pack 1 BD Y monthly review pack
9 9.2 Produce a decision per flagged item planner the review pack the decision record 0.5 BD N monthly review pack
10 10.1 Enter each action with owner, due date and proving measure planner the decision record the action register 0.5 BD N corrective action register
11 11.1 Compare each candidate's catch rate with the release condition; list rules below it standardization lead the catch-rate ledger the recommendation list 1 BD N release and withdrawal form
11 11.2 Decide release or withdrawal with a stated reason function head the recommendation list the signed decision 1 BD N release and withdrawal form
12 12.1 Assemble the report in the answer-first format standardization lead the review outputs the draft report 0.5 BD Y report template
12 12.2 Review the narrative and publish standardization lead the draft report the published report 0.5 BD N report template
13 13.1 Verify last month's actions against their proving measure forecaster the action register, the score table closed or carried actions 0.5 BD N corrective action register
13 13.2 Raise change requests to the standard forecaster the lessons, the measure-set change log change requests as needed N GP-601 intake

Variants

The known variant classes for this process, recorded per Standard and Variant in Process Documentation:

  • Statutory. No statutory variant changes a step. Where an accuracy figure forms part of a regulated service report, the retention period of the vintages and scores is a parameter, SET LOCALLY.
  • Contractual. A partner contract with its own accuracy clause, on its own measure and horizon, takes a variant row at steps 2 and 12: the partner view is scored on the contract's measure beside the register's, never instead of it. Owned by the placement lead, reviewed at renewal.
  • Structural. A client portfolio too small for percentage measures substitutes a scale-free measure; that is a parameter of the measure set per client portfolio, SET LOCALLY. A new client portfolio with no history for a naive benchmark takes a variant row at step 3, scored against the forecast of record only, with a review at its first full year.
  • Transitional. A unit from a predecessor organization whose vintages are not stored with layers is scored on the total only under a variant row with expiry at cut-over; its automation effect reads blocks against rank 3 for that unit.

L3 register

A localizing team writes these job aids against the generic L2; names only, the content is the function's. The measure set per horizon and client portfolio, the bias thresholds and the release condition are L3 values marked SET LOCALLY in the L2 row.

Job aid Anchored to Form
The measure set per horizon and client portfolio, with the benchmark step 2 work instruction; choice on MAPE WAPE and Forecast Bias
The scoring base checklist step 1 work instruction
The scoring routine and the stairstep template steps 2 and 3 work instruction; method on Forecast Value Added in Workforce Management
The significance test instruction step 4 work instruction; method on Model Evaluation and Validation
The bias threshold table and monitor layout step 5 work instruction; detection on Forecast Bias Detection and Correction
The root-cause protocol: data, structural, organizational step 6 playbook
The adjustment review template step 7 work instruction
The monthly review pack and the decision record step 9 work instruction
The corrective action register steps 10 and 13 work instruction
The release and withdrawal form, with the catch-rate evidence step 11 playbook; condition on Human Gates and Number Grades
The report template step 12 work instruction; format on Answer-First Reporting
The out-of-cycle review path on a bias breach the trigger runbook

Automation candidates

Flagged at L2 per the method on Process Decomposition (L0–L3); released per class of action on a published catch rate per Human Gates and Number Grades, never on a maturity label. The list is ordered by how easy a catch rate is to publish, and every row marked Y above appears in it.

  1. The scoring base check (1.1): deterministic; a missing vintage or an unclosed month is unambiguous, and first because it is caught the day it occurs.
  2. Vintage scoring (2.1, 8.1): computed, with the right answer defined by the measure set and the actuals; it ranks after the base check because a score is only as good as the base it was run on.
  3. The stairstep and the paired test (3.1, 4.1): computed, and the test that separates a real gain from noise is itself a rule;[2] it ranks after scoring because the test consumes the scores and its own errors need several months of layers to surface.
  4. The bias monitor (5.1): computed on a tracking signal and a bias ratio against thresholds;[3] it ranks after the stairstep because a false flag is only known to be false after the root-cause item it opened has been closed.
  5. Report assembly (9.1, 12.1): assembled from the review outputs with the narrative reviewed; last because an error here is caught by the reader rather than by a rate of its own.

Three steps are never candidates. The decision per flagged item (9.2) is the gate at which a planner must produce a keep, correct, remove or escalate, and a gate that asks only for a click is not a gate. The release and withdrawal decision (11.2) is the gate that governs every other gate in the forecasting pillar, and a rule cannot certify its own release. The root-cause classification (6.1) stays human because the organizational causes of bias, sandbagging and anchoring among them, are not in the data the monitor reads; the root-cause protocol on Forecast Bias Detection and Correction places the cause question before any correction for that reason. The choice of measure follows the scale-free arguments of the forecasting literature, which is why the measure set is a parameter per client portfolio rather than a single figure for the whole operation.[4]

Maturity Model Position

Four scales on this wiki use the word level; the launch page states which is which. On the WFM Labs Maturity Model™ a function that scores every published forecast on a defined measure at a defined cadence and keeps an action register is at Level 2; one that scores by layer, tests for significance, monitors bias by segment automatically and has released ranks 1 to 4 above on a published catch rate is at Level 3; one that uses the stairstep as the release list for forecasting automation, queue by queue, under the earned-learning and earned-auto-approval rules of Forecast Value Added in Workforce Management, is at Level 4.

See Also

References

  1. ↑ Gilliland, M. (2010). The Business Forecasting Deal: Exposing Myths, Eliminating Bad Practices, Providing Practical Solutions. John Wiley & Sons. Chapter on forecast value added analysis.
  2. ↑ Diebold, F. X., & Mariano, R. S. (1995). "Comparing Predictive Accuracy". Journal of Business & Economic Statistics 13(3), 253–263.
  3. ↑ Trigg, D. W. (1964). "Monitoring a Forecasting System". Operational Research Quarterly 15(3), 271–274.
  4. ↑ Hyndman, R. J., & Koehler, A. B. (2006). "Another look at measures of forecast accuracy". International Journal of Forecasting 22(4), 679–688.