MLOps Pipelines: Train, Test and Ship Models Automatically

Deployment
MLOps
CI/CD
What MLOps really means, the three maturity levels, an animated pipeline run, experiment tracking and a CI/CD workflow with a quality gate.
Author

MLStation

Published

October 1, 2026

Training PayGuard once by hand is easy. Retraining it every month, testing it, comparing it with the live model and shipping it without mistakes is hard. MLOps applies DevOps habits (automation, testing, monitoring) to machine learning, with one extra twist: the data changes too, so models must be retrained and re-validated, not just re-deployed (Google Cloud).

TipWhat you will learn
  • The three MLOps maturity levels and what each one automates
  • The stages of a training pipeline, shown as an animation
  • How data validation stops bad data before it reaches a model
  • How experiment tracking and a quality gate decide what ships
NoteThe Deployment series

Start: 1. Overview: from notebook to production

Build: 2. Tools · 3. MLOps · 4. Docker and containers · 5. Cloud and platforms

Operate: 6. Scale · 7. Monitoring · 8. Maintenance · 9. Updates and versioning

Three levels of maturity

Google describes three levels: level 0, manual process; level 1, ML pipeline automation (continuous training); and level 2, CI/CD pipeline automation (the pipeline code itself is tested and deployed automatically) (Google Cloud).

Table 1: MLOps maturity levels
Level Who retrains How often Typical risk
0 Manual A data scientist runs a notebook A few times a year Stale model, steps forgotten
1 Pipeline A scheduled or triggered pipeline Weekly or monthly Pipeline code changes are untested
2 CI/CD Pipeline code is tested and released like software On every change Higher setup effort

Why does it matter? The simulation below follows PayGuard for a year while fraud patterns slowly change. The manual team retrains twice; the pipeline retrains every month.

Show the code 
import numpy as np, plotly.graph_objects as go
from ms_charts import COLORS, PLOTLY_CONFIG
rng = np.random.default_rng(3)
days = np.arange(365)
def quality(retrain_days):
    q, last = [], 0
    for d in days:
        if d in retrain_days: last = d
        q.append(0.93 - 0.00045 * (d - last) - 0.0000012 * (d - last) ** 2 + rng.normal(0, 0.004))
    return np.array(q)
manual = quality({0, 180})
auto = quality(set(range(0, 365, 30)))
fig = go.Figure()
fig.add_scatter(x=days, y=manual, name="Level 0: manual, twice a year", line=dict(color=COLORS["rose"], width=2.5))
fig.add_scatter(x=days, y=auto, name="Level 1: pipeline, monthly", line=dict(color=COLORS["teal"], width=2.5))
fig.update_layout(height=380, xaxis_title="Day", yaxis_title="AUC on live data", legend=dict(orientation="h", y=1.12))
fig.show(config=PLOTLY_CONFIG)
Figure 1: Simulated model quality over a year. Automated monthly retraining keeps the model close to its best.

Averaged over the year, the automated model scores 0.924 AUC against 0.876 for the manual one. In fraud detection, that gap is money.

Anatomy of a training pipeline

A pipeline is a chain of steps where each step must pass before the next one starts.

flowchart TD
  A[Ingest<br/>new payments] --> B{Validate<br/>data}
  B -- fail --> X[Stop and alert]
  B -- pass --> C[Build<br/>features] --> D[Train] --> E[Evaluate]
  E --> F{Quality<br/>gate}
  F -- worse than live --> Y[Keep current model]
  F -- better --> G[Register<br/>new version] --> H[Deploy<br/>canary]

Press play to run the pipeline. Then tick “inject a bad data batch” and run it again: validation catches the problem before any training happens.

Data validation in practice

Most pipeline failures start with data, not code. A few simple checks catch the common ones. Here is a real check on a simulated batch where something went wrong upstream.

import pandas as pd
rng = np.random.default_rng(5)
batch = pd.DataFrame({
    "amount": rng.lognormal(3.5, 1, 1000),
    "hour": rng.integers(0, 24, 1000).astype(float),
    "km_from_home": rng.exponential(15, 1000),
})
batch.loc[:119, "amount"] *= -1          # an upstream bug flips signs
batch.loc[500:540, "hour"] = np.nan      # a broken field

rules = {
    "amount > 0":        (batch.amount > 0).mean(),
    "hour not missing":  batch.hour.notna().mean(),
    "hour in 0..23":     batch.hour.dropna().between(0, 23).mean(),
    "km_from_home >= 0": (batch.km_from_home >= 0).mean(),
}
report = pd.DataFrame({"pass_rate": rules}).assign(
    ok=lambda d: d.pass_rate >= 0.99)
report.style.format({"pass_rate": "{:.1%}"})
  pass_rate ok
amount > 0 88.0% False
hour not missing 95.9% False
hour in 0..23 100.0% True
km_from_home >= 0 100.0% True

Two rules fail, so the pipeline stops. Libraries such as Great Expectations, Pandera and TFX Data Validation turn these checks into reusable, documented suites.

Experiment tracking: which run wins?

During tuning, a team may train dozens of versions. Tracking tools (MLflow, Weights & Biases) store each run’s settings and scores, so the best one is easy to find and reproduce. Below are 40 simulated tuning runs.

Show the code 
lr = 10 ** rng.uniform(-2.5, -0.3, 40)
depth = rng.choice([3, 5, 8, 12], 40)
auc_runs = 0.95 - 0.02 * (np.log10(lr) + 1.3) ** 2 - 0.0012 * (depth - 6) ** 2 + rng.normal(0, 0.003, 40)
best = int(np.argmax(auc_runs))
fig = go.Figure()
for d, c in zip([3, 5, 8, 12], [COLORS["slate"], COLORS["teal"], COLORS["indigo"], COLORS["violet"]]):
    m = depth == d
    fig.add_scatter(x=lr[m], y=auc_runs[m], mode="markers", name=f"max_depth {d}", marker=dict(size=10, color=c, opacity=0.85))
fig.add_scatter(x=[lr[best]], y=[auc_runs[best]], mode="markers", name="Best run",
                marker=dict(symbol="star", size=20, color=COLORS["amber"], line=dict(width=1, color="white")))
fig.update_layout(height=380, xaxis=dict(type="log", title="Learning rate"), yaxis_title="Validation AUC", legend=dict(orientation="h", y=1.14))
fig.show(config=PLOTLY_CONFIG)
Figure 2: Forty simulated tuning runs. Each dot is one training run; the star is the best by validation AUC.

The quality gate

A new model is promoted only if it is at least as good as the live model on the same held-out data, and not worse on any important slice (for example, small merchants or one country).

\text{promote} \iff \text{AUC}_{\text{new}} \ge \text{AUC}_{\text{live}} - \varepsilon \;\text{ and }\; \min_{s \in \text{slices}} \big(\text{AUC}^{(s)}_{\text{new}} - \text{AUC}^{(s)}_{\text{live}}\big) \ge -\delta \tag{1}

Here \varepsilon is a small tolerance (say 0.002) and \delta protects slices (say 0.01). Writing the gate as code removes arguments and gut feelings from the release decision.

CI/CD for ML

At level 2, a push to the repository triggers tests, a training run and the gate. This GitHub Actions workflow is a compact starting point.

.github/workflows/train.yml
name: train-and-gate
on:
  push: {branches: [main]}
  schedule: [{cron: "0 3 1 * *"}]   # also monthly, 03:00 on the 1st
jobs:
  pipeline:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.11"}
      - run: pip install -r requirements.txt
      - run: pytest tests/                  # unit tests for feature code
      - run: python pipeline/validate.py    # data checks, exits 1 on failure
      - run: python pipeline/train.py       # logs run to MLflow
      - run: python pipeline/gate.py        # compares with live model
      - run: python pipeline/register.py    # only reached if gate passes

Common pitfalls

Pitfall Fix
Training notebook and pipeline drift apart Move code into a package; notebooks import it
No fixed random seeds or data snapshots Log seed, data version and code commit with every run
Gate only checks the overall metric Add slice checks and a latency check
Pipeline fails silently Alert on failure and on “no run in 35 days”

Summary

  • MLOps automates training, testing and release, because data keeps changing.
  • Level 0 is manual, level 1 automates training, level 2 also tests and ships the pipeline itself.
  • Validate data first, track every experiment, and let a quality gate decide what ships.
  • Next: package everything so it runs anywhere with Docker and containers.
Back to top