gitGraph commit id: "v1.0 first model" commit id: "v1.1 monthly retrain" branch experiment commit id: "new feature: merchant age" commit id: "tuned depth" checkout main commit id: "v1.2 monthly retrain" merge experiment id: "v1.3 adds merchant age" commit id: "v2.0 new input schema"
Model Updates and Versioning: Ship New Models Without Fear
Deployment
Versioning
Rollouts
MLflow
What to version, how a model registry works, and how shadow, canary, blue-green and A/B rollouts protect users, with an animated canary release and auto-rollback.
PayGuard version 15 scores better offline than version 14. Should it replace v14 for every payment right now? Not yet. Offline tests use old data; real traffic can still surprise you. A safe update answers three questions: exactly what changed, how do we expose it gradually, and how do we undo it in one step?
TipWhat you will learn
- The six things you must version together
- Semantic versions and registry aliases such as “champion” and “challenger”
- Shadow, canary, blue-green and A/B releases, and when to use each
- An animated canary release that rolls itself back when things go wrong
NoteThe Deployment series
Start: 1. Overview: from notebook to production
Build: 2. Tools · 3. MLOps · 4. Docker and containers · 5. Cloud and platforms
Operate: 6. Scale · 7. Monitoring · 8. Maintenance · 9. Updates and versioning
Version everything that changes the prediction
A model’s output depends on more than the model file. If any of these change silently, you cannot reproduce or explain a prediction.
| Artifact | Example | Tool |
|---|---|---|
| Code | Feature functions, training script | Git commit |
| Data | Training snapshot of 2026-09-01 | DVC, lakeFS, dated tables |
| Features | Definition of tx_last_hour |
Feature store, code review |
| Model | payguard version 15 |
MLflow Model Registry |
| Config | Threshold 0.5, hyperparameters | Config file in Git |
| Environment | Python 3.11, scikit-learn 1.5 | Docker image tag |
A simple semantic scheme makes versions readable:
- MAJOR (2.0.0): the input or output contract changes, for example a new required field. Callers must update.
- MINOR (1.5.0): retrained on new data or with new internal features, same contract.
- PATCH (1.5.1): a bug fix that does not change model behaviour intentionally.
Over time, model versions branch and merge like code:
The model registry
A registry stores every model version with its metrics and lineage. Instead of hard-coding “version 15”, the service loads a model by alias. Promoting or rolling back means moving the alias, with no code change and no redeploy (MLflow docs).
promote.py
import mlflow
from mlflow import MlflowClient
client = MlflowClient()
# v15 passed the quality gate: make it the challenger
client.set_registered_model_alias("payguard", "challenger", 15)
# the service always loads whatever "champion" points to
model = mlflow.pyfunc.load_model("models:/payguard@champion")
# after a successful rollout, promote; rolling back is the same call with 14
client.set_registered_model_alias("payguard", "champion", 15)Four ways to roll out
| Strategy | User risk | Extra cost | Answers the question |
|---|---|---|---|
| Shadow | None | Double compute | Does it behave sanely on real inputs? |
| Canary | Small, limited to the slice | Low | Is it safe to roll out? |
| Blue-green | Brief, but instant rollback | Two environments | Can we switch cleanly? |
| A/B test | Half of users see the new model | Time | Is it better for the business? |
A common sequence for PayGuard is shadow for a few days, then canary, then full rollout, with an A/B test when the change affects business outcomes such as false declines.
An animated canary release
Press play. Traffic to v15 steps up through 1%, 5%, 25%, 50% and 100%. At every step the system compares v15’s error rate with v14’s. Tick “bad release” to see the automatic rollback.
The point of a canary is containment. A bug that would have hit every customer touched only a small slice, for a few minutes.
Is v15 actually better? An A/B test
Safety is not the same as improvement. To measure the business effect, PayGuard split traffic 50/50 for three weeks and counted false declines (good customers wrongly blocked). The simulation below tracks the difference and its 95% confidence interval as data accumulates.
Show the code
import numpy as np, plotly.graph_objects as go
from ms_charts import COLORS, PLOTLY_CONFIG
rng = np.random.default_rng(17)
days, per_day = 21, 4000
a = rng.binomial(per_day, 0.0210, days).cumsum() # v14 false declines
b = rng.binomial(per_day, 0.0185, days).cumsum() # v15 false declines
n = per_day * np.arange(1, days + 1)
pa, pb = a / n, b / n
diff = (pb - pa) * 100
se = np.sqrt(pa * (1 - pa) / n + pb * (1 - pb) / n) * 100
d = np.arange(1, days + 1)
fig = go.Figure()
fig.add_scatter(x=np.r_[d, d[::-1]], y=np.r_[diff + 1.96 * se, (diff - 1.96 * se)[::-1]], fill="toself",
fillcolor="rgba(20,184,166,0.18)", line=dict(width=0), name="95% interval")
fig.add_scatter(x=d, y=diff, line=dict(color=COLORS["teal"], width=3), name="v15 minus v14")
fig.add_hline(y=0, line_color=COLORS["slate"], line_dash="dash")
fig.update_layout(height=360, xaxis_title="Day of test", yaxis_title="Difference in false-decline rate (points)", legend=dict(orientation="h", y=1.12))
fig.show(config=PLOTLY_CONFIG)The difference in rates and its interval are
\hat\Delta = \hat p_{15} - \hat p_{14}, \qquad \hat\Delta \pm 1.96\sqrt{\frac{\hat p_{14}(1-\hat p_{14})}{n_{14}} + \frac{\hat p_{15}(1-\hat p_{15})}{n_{15}}} \tag{1}
After 21 days, v15 changes the false-decline rate by -0.25 points (95% interval -0.38 to -0.12). Decide the test length before starting, and do not stop early the first time the interval looks good.
Rollback must be boring
flowchart LR A[Metric breach<br/>or human call] --> B[Move 'champion'<br/>alias back to v14] B --> C[Service reloads<br/>within seconds] C --> D[Verify metrics<br/>recover] D --> E[Keep v15 for<br/>investigation]
Practise rollback regularly (the Maintenance calendar includes a quarterly drill). Keep the previous image and model available, and make sure database or schema changes are backward compatible so v14 can still run.
Summary
- Version code, data, features, model, config and environment together.
- Use semantic versions and registry aliases; promotion and rollback become one call.
- Roll out with shadow → canary → full, and use an A/B test to prove business value.
- Automate the rollback trigger and practise it until it is boring.
- Back to the start: the Deployment overview shows how all nine parts fit together.