Demand-Forecasting Agent
Agentic ML pipeline for retail demand forecasting
View on GitHubA demand-forecasting system for retail sales data in the M5 canonical format — a sales table (series × time), a calendar, and weekly sell prices. The ML core is built and validated as plain, defensible code first; an agent layer (planner → executor → critic → report) wraps it last. Format-general and threshold-adaptive by design: it ingests any M5-format retail dataset and calibrates its decision thresholds to that dataset's own distribution.
Roadmap
Five phases — each gated by review before proceeding.
Three branch signals over M5; four sub-datasets selected at pattern extremes; calibrated thresholds.
Plain L2 (Ridge regression) baseline per dataset — the number every later model must beat. WMAPE on a rolling-origin holdout, frozen feature set, sparse-series mean fallback.
L2 vs Tweedie vs Poisson vs quantile-median on the frozen feature set, so only the objective varies. Tweedie wins every dataset and every branch cohort; Croston/SBA takes the sparse fallback. Emits a machine-readable model contract for the agent.
Wrap the working core: planner → executor → critic → report. The critic must actually gate acceptance.
Run the finished agent end-to-end on the held-out sub-dataset D.
Agent architecture
The agent (Phase 4) orchestrates a core that already stands on its own.
Confirms M5 schema: required columns present, dates continuous, no missing crucial fields. Flags problems, does not repair.
Runs EDA, interprets the three branch signals (intermittency, spikes, day-of-week η²), and does preprocessing.
Applies deterministic, documented branch rules and reads the Phase 3 model contract — selects which branch applies, does not invent or re-tune modeling decisions.
WMAPE vs threshold + sanity checks (non-negative forecasts, plausible magnitudes). A model that fails is flagged, not accepted. Phase 3 showed the bounds never fire on current winners, so Phase 4 must prove the gate with a deliberately-failing case.
Short, structured summary to the user with per-dataset results and branch coverage.
Branch logic
Deterministic, documented rules the executor applies — the agent selects which branch, it does not invent decisions.
Zero-inflated demand; Tweedie's compound Poisson-Gamma form fits, unlike squared error.
Phase 3 · Prior vindicated: on A's zero-heavy cohort, L2 collapses to 1.385 WMAPE_28 vs Tweedie's 0.472.
Expected L2 or Poisson to win lower-intermittency series. Explicit complement of B1.
Phase 3 · Refuted: Tweedie wins here too, so the B1/Standard split earns nothing — slated to collapse into one branch in Phase 4.
Not enough signal for a heavy model; knowing when not to use it is deliberate.
Phase 3 · SBA beats the Phase 2 mean floor (0.716 vs 0.741) — a real but modest ~3% gain, reported as such.
Requirement-driven — a range answers a different question than a point.
Phase 3 · q90 well calibrated (0.906 vs 0.90 target on A); crossing rate ≤ 0.33%. Low quantiles are unidentifiable on zero-inflated counts — stated, not hidden.
Selected sub-datasets
Each pick sits at a pattern extreme and stresses one branch. D is held out, frozen until Phase 5.
Phase 3 bake-off results
WMAPE_28 on per-series 28-day totals, 5-fold rolling origin, winner picked on selection folds only. A zero forecast scores exactly 1.000; lower is better.
Absolute error is minimised by the conditional median — which is 0 when zero-share > 0.5. A forecaster handed the true window mean still scores 1.078 (B) and 1.517 (C), worse than predicting nothing. The quantile-median entrant ‘wins’ C daily by predicting all zeros.
WMAPE_28 (28-day totals — what a replenishment decision consumes) became the pre-registered decision metric. Zero still scores exactly 1.000. Phase 2's ‘baseline loses to doing nothing’ on B/C turned out to be a metric artifact.
On A, a squared-error LightGBM is worse than the linear Ridge (0.378 vs 0.326); only switching to Tweedie recovers and beats it. The LGBM-L2 control was in the field precisely to isolate this.
Winner chosen on folds 0–3 only; fold 4 is selection-free. The gap tracks fold difficulty, not entrant — no detectable selection overfitting. 250-series samples reproduce full cells to within 0.2%.
Design decisions
Each is documented in the plans and enforced by a test.
The ML core is built and validated as plain, defensible code. The agent wrapper (Phase 4) orchestrates what already works — it is not load-bearing.
Branch cuts are percentiles of each dataset's own distribution, so they adapt to new data. Literature constants (Syntetos–Boylan) stay absolute and cited.
WMAPE is volume-weighted and robust on zero-heavy series. WRMSSE re-imports the squared-error sensitivity that Tweedie avoids — rejected by design.
Dataset D informs no distribution and no threshold. The four sub-datasets are disjoint by construction with symmetric cell exclusions.
Day-of-week explains only ~1% of median item variance (η² ≈ 0.012). The per-series gate was dropped; every series gets calendar features instead.
Branches survive only if the data backs them. The seasonal branch was retired after Phase 1; the B1/Standard split is next, after Tweedie won both cohorts in Phase 3.
Phase 3 emits a machine-readable model contract — one frozen config per branch, completeness over the router asserted by test. The cost of freezing a single config is reported: under 0.6% relative.
Seeds, deterministic LightGBM and fixed thread counts on every fit — then checked empirically: three independent bake-off runs, bit-identical results.
Validation and the critic flag problems rather than guessing or silently patching. A model that fails the critic is flagged, not accepted.
Tech stack
Thanks for reading. Want to see the rest?
Back to all work