Platform Degradation Runbook and Buffer Staffing (generic process)

From WFM Labs

Platform Degradation Runbook and Buffer Staffing (generic process) is the generic process card, GP-504, for the event in which an automation or AI containment layer degrades or fails and the contacts it was resolving return to the human queue. It has two parts that share one card because one sizes the other: the runbook, from the monitoring entry condition through severity mapping, buffer activation, routing fallback, communication and verification to exit; and the standing buffer-staffing policy, which sizes the human capacity held in reserve from the containment at risk and the platform's recovery time. No existing page carries either, so the card owns its method; it draws on AI Containment Rate and Its Workforce Implications for the containment arithmetic, AI Agent Failure Modes and Recovery for the failure taxonomy and the circuit breaker, and Human AI Supervision and Escalation Frameworks for the supervising seat, and defers to each. A degradation rated severity 1 or 2 fires the continuity wire and calls Crisis Trigger and Re-Plan (generic process); business continuity remains the partner that owns activation. The card is part of Process Standardization and sits under the continuity-interface section of the function catalog in Anatomy of a ROC Standard.

The buffer exists because the recovery pattern assumes it. In the circuit-breaker pattern an open breaker stops calling the failing component and runs a fallback until a half-open test passes;[1] mapped onto a contact queue, the failing component is the containment layer and the fallback is the human queue, which in a contact operation is staffed hours in advance or not at all.

Process profile

The profile extends the L0 identity fields of Process Decomposition (L0–L3). Local values are held by the deploying function and marked SET LOCALLY.

Field Generic value
Card GP-504 · local card id assigned by the standards committee at deployment
Class both: the runbook is real-time; the buffer policy is strategic and re-sized each capacity cycle
Trigger the entry condition from monitoring: containment by interval below its band for a run of intervals, the circuit breaker open, or a platform health alert from the platform owner (values SET LOCALLY)
Cadence event-driven runbook; the buffer re-sized monthly from the failure record inside Capacity Planning Cycle and re-based annually
Scope in entry; discrimination from a mix shift; severity mapping; returning load; buffer activation; routing fallback; communication; verification; exit; review; buffer re-size
Scope out diagnosis and repair of the platform (the platform owner); the breaker's own thresholds; the supervision ratio; the incident process this runbook runs inside (Incident Management for Contact Centers)
Inputs containment by interval and its forecast; breaker state; the automated node's supply card and drift terms; the buffer policy; the roster; the severity matrix
Outputs the incident record (nine fields); the buffer activation and routing fallback orders; the communications; the verification and exit records; the buffer-size recommendation
Upstream the platform owner's monitoring; the automation overseer; the real-time desk
Downstream the human queues and their node owners; GP-501 at severity 1 or 2; Real-Time Lever Deployment (generic process); the capacity cycle; Post-Mortem and RCA for Workforce Operations
Resources the monitoring feed; the routing control at contact-type grain; the incident ledger; a buffer roster of cross-trained agents; the sizing model. Platforms are named at L3, never here
Decision points degradation or mix shift; which severity row; inside the buffer or beyond it; is containment back at baseline for the exit window
Authority tier entry and record at tier 1; buffer activation at tier 2 or 3 by the lever's own tier; levers beyond the buffer through GP-501; the half-open test is the platform owner's call with the ROC duty lead's concurrence; a changed client commitment to the executive sponsor by exception
Risk and compliance flags contacts trapped between the two nodes (an escalation loop); a human-handling clause in a client contract; overtime and rest rules on the buffer; an exit taken before containment is confirmed
Variant-prone steps the fallback step on a platform without contact-type routing; communication where a client contracts a human-handling service level; the buffer where a works council governs emergency schedule changes
Automation candidates eight L2 sub-steps (listed below)

L1 flow

  1. Detect the entry condition: containment below band for the run length, the breaker open, or a platform alert. Owner: the automation overseer.
  2. Is this platform degradation, or a contact-mix shift that lowers containment with the platform intact? Check the intent mix and the health telemetry. Owner: the automation overseer.
  3. Open the incident record and map the severity, using containment at risk as the impact parameter; at severity 1 or 2 the continuity wire fires and GP-501 opens. Owner: the real-time analyst.
  4. Compute the returning load by interval: offered volume, times containment lost, times the share of contained contacts that return to a human rather than abandon, times handle time under escalation enrichment. Owner: the real-time analyst.
  5. Is the returning load inside the standing buffer? If not, the re-plan's levers are called. Owner: the ROC duty lead.
  6. Activate the buffer: cross-trained agents to the returning queues, off-phone pulls cancelled, overtime offered, time-off reversed, in that order. Owner: the ROC duty lead.
  7. Apply the routing fallback: graceful degradation by contact type where the platform supports it, full fallback where the breaker is open; confirm the human queue is receiving. Owner: the real-time analyst, with the platform owner.
  8. Communicate per the severity row: node owners, operations, the continuity owner at severity 1 or 2, and the customer-messaging request. Owner: the real-time analyst.
  9. Verify: human-queue service and backlog inside band at the degraded target, no escalation loop between the nodes, buffer attendance. Owner: the real-time analyst.
  10. Is containment back at baseline for the exit window after the half-open test? The reverse check of GP-501 applies: step down on measured recovery, never on a projected one. Owner: the ROC duty lead, on the automation overseer's measurement.
  11. Exit: restore routing in stages, release the buffer, close the record with the nine fields. Owner: the real-time analyst.
  12. Review: the post-incident review of the platform owner and the post-event review of the function run on one timeline. Owner: the process owner.
  13. Re-size the buffer from the record: update the failure-duration distribution and the containment at risk; recommend the size to the capacity cycle. Owner: the capacity planner.

L2 step table

The buffer-sizing arithmetic in 13.1 is the card's own: capacity in reserve equals the returning load at a chosen percentile of the failure-duration distribution, net of the levers the function can raise inside the recovery time, at the degraded service target. The percentile and the target are SET LOCALLY.

Step Sub-step Owner (seat) Inputs Outputs Duration Automation candidate Documentation join key
1 1.1 Compare containment by interval to its band; read breaker state and alerts automation overseer monitoring feed entry fired, timestamp and zone per interval Y entry-condition sheet
2 2.1 Compute intent-mix and health-telemetry deviation from baseline automation overseer telemetry deviation figures minutes Y (compute) discrimination checklist
2 2.2 Decide: degradation, or mix shift routed to the reforecast automation overseer deviation figures decision minutes N discrimination checklist
3 3.1 Open the record; set severity on containment at risk real-time analyst matrix, load estimate record at a severity row minutes Y (propose) severity mapping table
3 3.2 Notify per the row; fire the wire at severity 1 or 2 real-time analyst record notifications sent minutes N severity mapping table
4 4.1 Compute returning load by interval and queue [C] real-time analyst volume, containment lost, return share [E], enriched handle time [E] load by interval minutes Y returning-load calculator
5 5.1 Compare load to the buffer; decide ROC duty lead load, buffer roster inside, or GP-501 called minutes N buffer policy
6 6.1 Raise levers in policy order ROC duty lead buffer roster, lever menu activation order SET LOCALLY Y (propose) buffer activation checklist
7 7.1 Change routing by contact type or in full real-time analyst, platform owner fallback map routing changed SET LOCALLY N (the breaker does it; the human verifies) fallback runbook
7 7.2 Confirm the human queue is receiving real-time analyst queue state receipt confirmed minutes N fallback runbook
8 8.1 Send communications per the row real-time analyst templates sent log minutes Y (draft) communication templates
9 9.1 Measure service, backlog, loop count, attendance real-time analyst actuals verification record per interval Y (measure) verification criteria
10 10.1 Concur in the platform owner's half-open test; read the overseer's containment measurement over the exit window; decide ROC duty lead test result, actuals exit decision window SET LOCALLY N exit checklist
11 11.1 Restore routing in stages; release buffer; close with nine fields real-time analyst order, record closed record SET LOCALLY N exit checklist
12 12.1 Joint review on one timeline process owner both records findings, problem record within SET LOCALLY days N review agenda
13 13.1 Update the failure-duration distribution and containment at risk; compute the size at the percentile capacity planner records, policy buffer size [C] monthly Y (compute) sizing model
13 13.2 Recommend the size to the capacity cycle; re-base annually capacity planner size, cost recommendation monthly N sizing model

Variants

Recorded per Standard and Variant in Process Documentation:

  • Statutory. Overtime caps and rest minimums on the buffer are parameters, SET LOCALLY. Where a works council co-determines temporary changes to hours, a notification step is inserted between steps 5 and 6, before the buffer is activated, owned by the works-council liaison and approved by the function head; where the agreement provides an emergency exception, the step becomes a notification immediately after activation and the variant row says so.
  • Contractual. A partner's surge clause is a parameter of the lever menu, no row. A client whose contract fixes a human-handling service level inserts a client-notification step at step 8 under a variant row, reviewed at renewal.
  • Structural. A node with no automated layer does not run the card; it is marked not inventoried on the shell, per Process Shells for a Workforce Standard. A platform that cannot route by contact type varies step 7 to all-or-nothing fallback with a review date.
  • Transitional. An automated node migrating platforms carries two breaker definitions under a variant row expiring at cut-over.

L3 register

Job aid Anchored to Form
The entry-condition sheet: bands, run length, breaker states step 1 work instruction
The discrimination checklist: degradation versus mix shift step 2 runbook
The severity mapping table for degradation events step 3 work instruction
The returning-load calculator with graded inputs step 4 work instruction
The buffer policy and the buffer roster steps 5 to 6 playbook
The fallback runbook per platform step 7 runbook
The communication templates per severity row step 8 work instruction
The verification criteria and the exit checklist steps 9 to 11 runbook
The joint review agenda step 12 playbook
The sizing model and its annual re-base step 13 work instruction

Automation candidates

Flagged at L2 per Process Decomposition (L0–L3); released per class of action on a published catch rate per Human Gates and Number Grades. The eight rows marked Y are ranked by how easy a catch rate is to publish for each.

  1. Entry detection (1.1): thresholds against a feed that is already measured by interval, and the gap between onset and detection is the one gap the function can shorten without touching the platform.
  2. The returning-load computation (4.1): arithmetic on graded inputs; after #1 because two of its inputs, the return share and the enriched handle time, are [E].
  3. The buffer re-size (13.1): the same arithmetic over the failure record; after #2 because it runs monthly, so a catch rate accrues slowly.
  4. The discrimination computation (2.1): deviation from baseline; after #3 because the baseline moves with the contact mix, which is the thing being tested.
  5. Verification measurement (9.1): four signals against bands; after #4 because the bands are set locally per node.
  6. The severity proposal (3.1): the matrix applied to the load; after #5 because the load is [E] and a person sets the row.
  7. The buffer activation proposal (6.1): inside the guardrails of Real-Time Lever Deployment (generic process); after #6 because it proposes changes to people's schedules, where a miss costs more than a wrong number.
  8. The communication drafts (8.1): last because the output is prose, whose errors are counted by review rather than by rule.

The exit decision (10.1) is never a candidate, nor is the routing restoration it authorizes: the platform under suspicion is the one whose signals would drive an automated exit, and supervisors who have watched a reliable system reduce their vigilance precisely when it degrades.[2] The discrimination at step 2 stays human because a mix shift and a degradation call for opposite responses, a reforecast and a fallback; writing the runbook before the page is what makes that call fast.[3] The buffer is a standing cost rather than an overtime line because the staffing curve is convex: the returning load lands on an operation already sized to its own load, where each added contact costs more than the last.[4]

Maturity Model Position

Four scales on this wiki use the word level; the launch page states which is which. On the WFM Labs Maturity Model™ a function with a written fallback and an overtime response is at Level 2; one with the runbook as written, a buffer policy sized from its own failure record and a reverse-checked exit is at Level 3; one whose entry detection, load computation and activation proposal are released on published catch rates is at Level 4, where AI Agent Failure Modes and Recovery places operational circuit breakers.

See Also

References

  1. ↑ Nygard, M. T. (2018). Release It! Design and Deploy Production-Ready Software (2nd ed.). Pragmatic Bookshelf, chapter 5, "Stability Patterns": circuit breaker and fallback.
  2. ↑ Parasuraman, R., & Riley, V. (1997). "Humans and Automation: Use, Misuse, Disuse, Abuse". Human Factors 39(2), 230–253.
  3. ↑ Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.) (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly, chapters 11 and 14 (being on-call; managing incidents). sre.google.
  4. ↑ Gans, N., Koole, G., & Mandelbaum, A. (2003). "Telephone Call Centers: Tutorial, Review, and Research Prospects". Manufacturing & Service Operations Management 5(2), 79–141.