ML Deployment, Explained: From Notebook to Production

Deployment
MLOps
The big picture of putting a machine learning model to work: deployment patterns, the request journey, the latency budget and the build-and-operate loop.
Author

MLStation

Published

October 1, 2026

A model in a notebook is a prototype. A model in production answers real requests, every second, for months, while the world around it changes. Deployment is everything between those two states. This series follows one example from start to finish: PayGuard, a fraud-detection model that scores every card payment before it is approved.

TipWhat you will learn
  • The five ways a model can be deployed, and how to choose
  • What happens, step by step, when a request hits a model in production
  • How a latency budget is spent, and why the model is rarely the slowest part
  • The build-and-operate loop that the rest of this series explores
NoteThe Deployment series

Start: 1. Overview: from notebook to production

Build: 2. Tools · 3. MLOps · 4. Docker and containers · 5. Cloud and platforms

Operate: 6. Scale · 7. Monitoring · 8. Maintenance · 9. Updates and versioning

The deployment lifecycle

Deployment is not a single step at the end. It is a loop: you build a model and package it, ship it to an environment, serve predictions, then operate it by watching, scaling and updating it. Monitoring feeds back into building the next version.

flowchart TB
  subgraph Build
    direction LR
    A[Data and training] --> B[Package model<br/>tools, Docker]
  end
  subgraph Ship
    direction LR
    C[CI/CD pipeline<br/>MLOps] --> D[Cloud or edge<br/>platform]
  end
  subgraph Operate
    direction LR
    E[Serve and scale] --> F[Monitor] --> G[Maintain and update]
  end
  B --> C
  D --> E
  G -. new data, new version .-> A

Five ways to deploy a model

Table 1: Deployment patterns
Pattern How it works Typical latency Example
Batch Score a whole table on a schedule Minutes to hours Nightly churn scores for the sales team
Online API A request comes in, a prediction goes out Milliseconds PayGuard scoring a card payment
Streaming Score events as they flow through a queue Sub-second Flagging suspicious log-ins from an event stream
Edge The model runs on the device Milliseconds, works offline Barcode reader in a phone app
Embedded in an app The model is a library inside another service Microseconds Spam filter inside an email server

Answer three questions and see which pattern fits.

The journey of one request

When a shopper taps “Pay”, PayGuard has a strict budget: the bank wants an answer in under 100 ms. Press play and follow one payment through the system. The timings come from a simulation of 10,000 requests.

Where do those milliseconds go? Here is the median time for each stage.

Show the code 
import plotly.graph_objects as go
from ms_charts import COLORS, PLOTLY_CONFIG
fig = go.Figure(go.Waterfall(
    x=[k.replace(" ", "<br>") for k in stages] + ["Total"], y=list(med[list(stages)]) + [0],
    measure=["relative"] * len(stages) + ["total"],
    text=[f"{v:.1f} ms" for v in med[list(stages)]] + [f"{med['total']:.0f} ms"], textposition="outside",
    increasing=dict(marker=dict(color=COLORS["indigo"])), totals=dict(marker=dict(color=COLORS["teal"])),
    connector=dict(line=dict(color="rgba(138,147,163,0.4)"))))
fig.add_hline(y=100, line_dash="dash", line_color=COLORS["rose"], annotation_text="100 ms budget", annotation_position="top left")
fig.update_layout(height=380, yaxis_title="Milliseconds", showlegend=False, yaxis=dict(range=[0, 112]), xaxis=dict(tickangle=0))
fig.show(config=PLOTLY_CONFIG)
Figure 1: Median latency per stage across 10,000 simulated payments. The model itself is a small slice; fetching features costs more.

The median is only half the story. Users feel the slow tail, so teams track the 95th and 99th percentile (p95, p99).

Show the code 
p50, p95, p99 = np.percentile(lat.total, [50, 95, 99])
fig = go.Figure(go.Histogram(x=lat.total, nbinsx=80, marker_color=COLORS["indigo"], opacity=0.8))
for v, name, c, pos in [(p50, "p50", COLORS["teal"], "top"), (p95, "p95", COLORS["amber"], "top left"), (p99, "p99", COLORS["rose"], "top right")]:
    fig.add_vline(x=v, line_color=c, line_width=2.5, annotation_text=f"{name} {v:.0f} ms", annotation_position=pos)
fig.update_layout(height=340, xaxis_title="Total latency (ms)", yaxis_title="Requests", bargap=0.02, showlegend=False)
fig.show(config=PLOTLY_CONFIG)
Figure 2: Distribution of total latency. The p99 is far above the median.

Half of payments finish in 40 ms, but 1 in 100 takes more than 74 ms. Most speed work in production attacks the features and the network, not the model.

Build vs operate

Phase Question it answers Article
Tools How do I turn a model file into a service? Tools
MLOps How do I train, test and ship automatically? MLOps
Containers How do I make it run the same everywhere? Docker
Cloud Where should it run, and what will it cost? Cloud
Scale How do I handle 10× traffic on Black Friday? Scale
Monitoring How do I know it is still working? Monitoring
Maintenance How do I keep it healthy for years? Maintenance
Updates How do I release a new version safely? Updates
WarningThe hidden cost

The model code is a small part of a production ML system. The surrounding data pipelines, configuration, serving and monitoring create most of the long-term maintenance cost (Sculley et al., 2015). Plan for the whole system from day one.

Summary

  • Deployment is a loop: build, ship, serve, operate, and back again.
  • Pick the pattern from the latency need, connectivity and data shape: batch, online API, streaming, edge or embedded.
  • Measure latency per stage and watch the p95 and p99, not just the average.
  • The rest of this series walks through each part of the loop with PayGuard.
Back to top