Scaling ML Inference: Surviving the Black Friday Spike

Deployment
Scale
Autoscaling
Why latency explodes near full capacity, how autoscaling reacts to a traffic spike (animated), and how batching, caching and right-sizing control cost.
Author

MLStation

Published

October 1, 2026

On a normal day PayGuard handles a few hundred payments per second. On Black Friday traffic jumps 8× in an hour. If the service cannot keep up, payments time out and the shop loses sales. If it keeps 8× capacity all year, the cloud bill explodes. Scaling is the art of having just enough capacity, just in time.

TipWhat you will learn
  • Little’s law and why latency explodes as servers approach 100% busy
  • Horizontal vs vertical scaling
  • How an autoscaler reacts to a spike, in an animated simulation you control
  • Batching, caching and model compression as cost levers
NoteThe Deployment series

Start: 1. Overview: from notebook to production

Build: 2. Tools · 3. MLOps · 4. Docker and containers · 5. Cloud and platforms

Operate: 6. Scale · 7. Monitoring · 8. Maintenance · 9. Updates and versioning

Two numbers rule everything

Throughput \lambda is requests per second. Latency W is how long each request spends in the system. Little’s law links them to the number of requests in flight, L:

L = \lambda \, W \tag{1}

If PayGuard receives 400 requests per second and each takes 0.05 s, then on average 400 \times 0.05 = 20 requests are in progress at once. You need enough workers to hold them.

Why “80% busy” is the danger zone

When servers are nearly full, new requests wait in a queue. Queueing theory shows that waiting time does not grow in a straight line: it explodes as utilization approaches 100%. The chart uses the standard M/M/c queue model for a group of 4 replicas, each taking 20 ms per request.

Show the code 
import numpy as np, plotly.graph_objects as go
from math import factorial
from ms_charts import COLORS, PLOTLY_CONFIG

def mmc_latency(rho, c=4, service_ms=20):
    a = rho * c                                    # offered load
    s = sum(a**k / factorial(k) for k in range(c))
    pc = a**c / factorial(c) / (1 - rho)
    erlang_c = pc / (s + pc)                       # probability of waiting
    wait = erlang_c * service_ms / (c * (1 - rho))
    return service_ms + wait

rho = np.linspace(0.05, 0.97, 200)
w = [mmc_latency(r) for r in rho]
fig = go.Figure(go.Scatter(x=rho * 100, y=w, line=dict(color=COLORS["indigo"], width=3.5)))
fig.add_vrect(x0=80, x1=97, fillcolor=COLORS["rose"], opacity=0.08, line_width=0,
              annotation_text="danger zone", annotation_position="top left")
fig.update_layout(height=360, xaxis_title="Utilization (%)", yaxis_title="Average latency (ms)", showlegend=False)
fig.show(config=PLOTLY_CONFIG)
Figure 1: Average latency vs utilization for 4 replicas (M/M/4 queue, 20 ms service time). Past about 80% the curve turns sharply upward.

At 50% busy, a request takes about 22 ms. At 95% busy it takes about 109 ms. That is why autoscalers usually target 60–70% utilization, not 100%: the spare room absorbs bursts.

Scale up or scale out

flowchart LR
  subgraph V[Vertical: scale up]
    direction TB
    V1[1 small server] --> V2[1 bigger server<br/>more CPU, more RAM]
  end
  subgraph H[Horizontal: scale out]
    direction TB
    LB[Load balancer] --> R1[Replica 1]
    LB --> R2[Replica 2]
    LB --> R3[Replica 3]
  end

Vertical (bigger machine) Horizontal (more machines)
Limit The biggest machine available Practically unlimited
Failure One machine, one point of failure One replica fails, others carry on
Best for Large models that need lots of memory or a GPU Stateless APIs like PayGuard

Autoscaling a Black Friday spike

An autoscaler watches utilization and adds or removes replicas. It cannot react instantly: it needs time to notice the spike and to start new containers. Press play and adjust the settings to see the trade-off between cost and overload.

Try these experiments:

  • Raise minimum replicas to 20: overload almost disappears, but the cost roughly doubles.
  • Lower the target utilization to 0.5: more headroom during the spike, more replica-hours.
  • Increase the scale-up delay to 15 minutes: the spike arrives before new replicas are ready.

The practical answer is a mix: a sensible minimum, a target near 0.65, fast-starting containers (small images help, see Docker), and scheduled pre-scaling before known events like Black Friday.

On Kubernetes, a Horizontal Pod Autoscaler expresses the same idea:

hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: {name: payguard}
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: payguard}
  minReplicas: 4
  maxReplicas: 60
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: {type: Utilization, averageUtilization: 65}

Batching: throughput vs latency

Grouping requests into batches raises throughput, because fixed overhead is shared (we measured this in Tools). But each request waits for its batch to fill. The simulation below models a server with 4 ms of fixed overhead per batch plus 0.15 ms per row.

Show the code 
from plotly.subplots import make_subplots
b = np.array([1, 2, 4, 8, 16, 32, 64, 128])
batch_ms = 4 + 0.15 * b
throughput = b / batch_ms * 1000
latency = batch_ms + 0.5 * b * 0.6          # compute time + average wait to fill the batch
fig = make_subplots(specs=[[{"secondary_y": True}]])
fig.add_scatter(x=b, y=throughput, name="Throughput (req/s)", line=dict(color=COLORS["teal"], width=3), secondary_y=False)
fig.add_scatter(x=b, y=latency, name="Latency (ms)", line=dict(color=COLORS["rose"], width=3, dash="dot"), secondary_y=True)
fig.update_xaxes(type="log", title="Batch size", tickvals=b)
fig.update_yaxes(title_text="Requests per second", secondary_y=False)
fig.update_yaxes(title_text="Latency (ms)", secondary_y=True)
fig.update_layout(height=360, legend=dict(orientation="h", y=1.12))
fig.show(config=PLOTLY_CONFIG)
Figure 2: Simulated batching trade-off. Throughput rises quickly with batch size; latency rises slowly at first, then steadily.

A batch of 16 to 32 gives most of the throughput gain while keeping latency well inside PayGuard’s 100 ms budget.

Other cost levers

Lever How it helps PayGuard example
Caching Reuse answers for repeated inputs Cache merchant risk features for 5 minutes
Smaller models Distillation, pruning, quantization cut compute per request An int8 model scores faster on CPU
Right-sizing Match CPU and memory to real usage Drop from 4 vCPU to 2 when CPU sits at 20%
Async and queues Smooth spikes for work that can wait Send low-risk refunds to a queue
Scale to zero Pay nothing when idle Night-time demo endpoints

Summary

  • Little’s law L = \lambda W tells you how much concurrency you need.
  • Latency explodes past roughly 80% utilization, so keep headroom.
  • Scale out stateless services; autoscalers need time, so set a minimum and pre-scale for known events.
  • Batching, caching and smaller models cut cost per prediction.
  • Next: know when something goes wrong with Monitoring.
Back to top