MLOps at Scale: Operating 440,000 Forecasting Models

Forecasting demand per store, per product, per day produces an enormous fleet of models — in one program, 440,000 of them. At that scale, every manual step you would happily do for a single model becomes impossible: you cannot hand-tune, hand-deploy, or eyeball the accuracy of nearly half a million models. This article describes the platform architecture that trains, serves, monitors, and retrains that fleet automatically, with governance that holds up to audit.

The scale problem reframed

The instinct to "train 440,000 models" is the wrong abstraction. You build one templated pipeline and execute it across 440,000 series partitions. Everything — features, training, evaluation, registration, serving, monitoring — is defined once and parameterised by the partition key. This turns an unmanageable model zoo into a single governed pipeline run many times.

Features once, point-in-time correct

The most expensive bug in forecasting is label leakage — training on a feature value that wasn't yet known at the forecast time. A feature store with point-in-time joins makes the correct behaviour the default: training reads from the offline store with as-of semantics; serving reads identical feature definitions from the online store, eliminating train/serve skew. See our feature-store deep-dive for the modelling side.

Sharded training, not 440,000 jobs

You do not launch 440,000 containers. You group partitions into shards and train many lightweight models (gradient-boosted or statistical) per worker, parallelising shards across a cluster. Each model's params, metrics, and artifact are logged to the registry under its partition key, so the fleet stays individually addressable while training runs as a handful of bulk jobs.

A templated, parallelised training task (one shard of partitions)

import mlflow, lightgbm as lgb

def train_shard(partition_ids, run_date):
    fs = FeatureStore()
    for pid in partition_ids:                       # e.g. 2,000 series per shard
        df = fs.get_training_df(pid, end=run_date)  # point-in-time correct
        model = lgb.LGBMRegressor(objective="quantile", alpha=0.5,
                                  n_estimators=400).fit(df.X, df.y)
        smape = backtest(model, df, folds=4)        # rolling-origin CV
        if smape < baseline_smape(pid):             # gate vs seasonal-naive
            with mlflow.start_run(run_name=f"fc-{pid}"):
                mlflow.log_metric("smape", smape)
                mlflow.lightgbm.log_model(model, "model",
                    registered_model_name=f"forecast_{pid}")
                client.set_registered_model_alias(f"forecast_{pid}", "prod",
                    version=latest_version(f"forecast_{pid}"))

Serving: mostly batch, selectively online

Demand forecasts are consumed by planning systems that run on a cadence, so the dominant pattern is batch scoring into a forecast table partitioned by date and key — cheap, cacheable, and easy to reconcile. Only series that need intra-day updates get an online endpoint that pulls live features from the online store. The registry's prod alias decouples "which model serves" from the serving code, so promotions and rollbacks are metadata changes, not deployments.

Monitoring half a million models

You cannot watch 440,000 dashboards, so monitoring is exception-based. Each partition streams realised error (sMAPE/MASE) and input-feature drift; the platform surfaces only the partitions breaching thresholds, ranked by business impact (forecast value × error). Teams act on a short, prioritised list, not a wall of charts.

Drift- and error-triggered retraining (scoped, not fleet-wide)

def retrain_triggers(run_date):
    flagged = []
    for pid in all_partitions():
        err  = realised_error(pid, window="28d")     # vs delivered forecasts
        psi  = feature_drift(pid, window="28d")       # population stability index
        if err > err_threshold(pid) or psi > 0.2:
            flagged.append(pid)
    # only the flagged partitions are retrained this cycle
    for shard in batched(rank_by_impact(flagged), size=2000):
        submit(train_shard, shard, run_date)
    log_event("retrain_scope", n=len(flagged))        # auditable

Governance and lineage

For an audited program, every served number must be reproducible. The registry ties each forecast to a model version, that version to a training run, and the run to a feature snapshot and code commit. Given any historical forecast, you can answer "what model, what data, what code produced this?" — which is what reproducibility, rollback, and audit all depend on.

What this delivers

The fleet stays accurate without anyone manually retraining it; deployments are reproducible and reversible; and engineering effort goes to the small set of partitions that actually need attention. The same templated-pipeline-plus-registry pattern scales down to dozens of models and up to millions — the architecture, not the count, is what makes it tractable.