The Interim Data Layer

From WFM Labs
Seven canonical weekly files against three planning estates, and which view each one feeds.

The interim data layer is a deliberately temporary mechanism for producing a single consistent planning view across estates that do not share definitions, systems or practices. It consists of a canonical minimum dataset — the smallest set of fields that produces every required view — delivered as plain files on a weekly cadence, and a renderer that reads those files.

It is not an architecture. Enterprise Data Platform and WFM Data Infrastructure and Integration Architecture describe the permanent build; WFM Data Governance and Quality describes the discipline that should govern it. This page describes what a planning function does in the eighteen to twenty-four months before any of that arrives, and how doing it by hand produces a better specification for the permanent version than designing the permanent version first.

The condition it addresses

A function formed by merger or acquisition typically inherits several planning estates, each with its own workforce management system, its own reporting stack and its own definitions, plus a layer of individually maintained spreadsheets carrying whatever no system holds. Consolidation is on a roadmap measured in years.

Meanwhile the function must report weekly. The response is usually one of two failures. Either a person assembles the view by hand each week, which works until that person is unavailable and produces figures nobody can reconstruct; or the function waits for the platform, and reports nothing consistent in the interim.

The interim layer is the third option, and it rests on an observation about where the effort actually is: producing a view is cheap once the inputs have a fixed shape. Almost all of the cost is in the shape.

The canonical minimum dataset

The design rule is subtractive. Start from the views that must be produced, list every field each one consumes, and remove everything not consumed by at least one view. What remains is typically far smaller than an estate's reporting surface — commonly five to eight files rather than the dozens a system emits.

A dataset supporting a weekly planning pack, a forward risk view and a portfolio summary reduces to something like:

File Grain Carries
Demand day × estate × market × channel Transactions and contacts
Performance day × estate × channel Offered, handled, in-target, service level, speed of answer, abandonment, handle time
Supply week × estate × source Headcount, productive hours, shrinkage, and the handle-time basis
Plan one row per supply wave Volume, training and nesting duration, live date
Assumptions one row per assumption Value, grade, sensitivity, owner, date
Reference one row per supply pool Size, client count, value tier, visibility
Events one row per event Date, scope, description

Two properties matter more than the field list itself.

Every file carries its own definitional qualifiers. A supply row carries whether its handle-time basis is elapsed or worked, because on a concurrent channel the two differ by the concurrency factor and a workload computed on the wrong one is wrong by that factor. A reference row carries whether its population is measured or estimated, so a view can render an estimate as an estimate.

Some files have no system source at all, and that is expected. Plans, assumptions and events are small, human-maintained records. They need a home and a habit rather than an integration, and the absence of a system for them is not a reason to omit them from the contract.

Definitional reconciliation comes first

Extraction does not fix a field that means two things, and a layer built over unreconciled definitions produces a consistent-looking view that is wrong in a way nobody can see. The dimensional-modeling literature treats this as the conformance problem: shared dimensions must carry identical meaning across every fact they join, or cross-source comparison is invalid by construction.[1] The reconciliations that recur:

  • Transaction definition — gross against net, and which lines of business are counted.
  • Rate denominators — two figures carrying the same name and differing by a factor, because one is computed over a broader population than the other.
  • Touched or agent-assisted volume — frequently unrecorded at the desktop, which makes any ratio built on it unreliable and not repairable after the fact.
  • Handle-time basis — elapsed against worked, as above.
  • Handled against answered-within-target — publishing the second in a column labeled as the first understates workload by the miss rate, which is largest exactly when the operation is performing worst.
  • Unit identity — no shared key joining client, team, queue and location. This is the reconciliation that gates any view organized by supply unit, and it is the one the permanent ontology exists to solve.

The interim answer to the last is a hand-maintained register with permanent identifiers covering the units that matter, rather than the whole estate. A register of a hundred or so rows is tractable; one of several thousand is a program.

Build sequence

  1. One location, one shape. The files exist, with the same columns, named by week. Nothing clever. Hours of effort.
  2. One extract per estate. Where an automated report exists it needs a column mapping; where it does not, the manual source is re-cut into the same shape and is labeled as manual wherever it appears. Days of effort, plus however long the broken feed takes.
  3. The human files. Plans, assumptions and events get an owner and a weekly habit. The reference register is the only one requiring analysis.
  4. Point the renderer at the files. The view scripts read the files instead of holding values. A day, once the files exist.

After the fourth step, producing the weekly view costs the time to check it rather than the time to build it — which is the entire point, because the checking is the part that requires judgment.

What it deliberately does not do

  • No warehouse, pipeline or integration project.
  • No reconciliation of the estates into one model. They stay separate, are reported side by side, and their differences are named on the page.
  • No automation of the assumption record, which is a human artifact and should remain one.
  • No coverage of populations that churn without visibility. Those are marked as estimated and rendered as estimated.

Each limit should be stated when the layer is presented. A temporary mechanism presented as a permanent one attracts requirements it was never built to carry, and then fails against them.

Why running it by hand improves the permanent build

The strongest argument for the interim layer is not that it fills a gap. It is that a requirement written after running a process by hand for two quarters is materially better than one written from imagining it. The observation is old: a first system built to learn from, and expected to be discarded, tends to produce a better second system than a first attempt at the permanent one.[2] The manual version reveals which fields are actually consumed, which definitional collisions actually bite, which sources are actually reliable, and which parts of the weekly cycle actually take the time. Specifications written before that knowledge exists tend to over-specify the visible parts and under-specify the reconciliations.

Each element of the interim layer therefore has a named successor: the files become a platform feed, the reference register becomes the ontology, the definitional crosswalk becomes a governance artifact, and the hand-run scripts become scheduled jobs. Presenting the layer with its successors named is what distinguishes it from shadow IT.

Failure modes

  • The layer becomes permanent by default. No successor is named, the files acquire consumers, and the temporary mechanism is still running three years later. This is the accumulation of glue and configuration debt that surrounds any system assembled from whatever was available.[3] Name the successor for each element at the outset.
  • Scope grows to match the systems rather than the views. Fields are added because a source emits them. The dataset is defined subtractively from the views, or it is not minimal.
  • Definitions are deferred to the second phase. The layer then produces a consistent view of incomparable things.
  • One person owns all the extracts. The mechanism that was meant to remove a single point of failure becomes one.
  • Manual sources are not labeled as manual. A hand-assembled figure rendered identically to an automated one will be trusted identically.

Maturity Model Position

At Level 1–2 each estate reports separately and cross-estate views are assembled by hand on request. At Level 3 a canonical minimum dataset exists, is produced weekly, and the views are generated from it. At Level 4 the definitional reconciliations are documented and enforced at extraction, and every field carries its qualifiers. At Level 5 the interim layer has been retired into the permanent platform, and the specification for that platform is traceable to what the interim layer revealed.

See Also

References

  1. ↑ Kimball, R., & Ross, M. (2013). The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling, 3rd ed. Indianapolis: Wiley. The treatment of conformed dimensions and declared grain is the formal counterpart of the reconciliations listed here.
  2. ↑ Brooks, F. P. (1995). The Mythical Man-Month: Essays on Software Engineering, anniversary ed. Reading: Addison-Wesley. The argument that the first build should be planned as disposable, and the later caution that the point is iterative delivery rather than literal discard, both apply here.
  3. ↑ Sculley, D., Holt, G., Golovin, D., et al. (2015). "Hidden Technical Debt in Machine Learning Systems". Advances in Neural Information Processing Systems 28, 2503–2511. The described patterns — glue code, pipeline jungles, configuration debt and undeclared consumers — describe the failure mode of an interim layer left in place.
Retrieved from "https://wiki.wfmlabs.org/w/index.php?title=The_Interim_Data_Layer&oldid=5105"