A model can achieve 99% accuracy and be useless. It can have excellent ROC-AUC and produce poor probabilities. It can dominate random cross-validation and fail next month. Evaluation is therefore a design problem, not merely a metric calculation.
If model parameters are learned from \mathcal D_{\text{train}}, performance must be estimated on data not used to fit those parameters.
ALL DATA
│
├── TRAIN ── learn parameters
│
├── VALIDATION ── choose/tune
│
└── TEST ── final unbiased-ish assessmentIf you repeatedly inspect test performance and change the model, the test set becomes part of model development.
import numpy as np, pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X,y=make_classification(
n_samples=6000,n_features=12,n_informative=6,
n_redundant=2,weights=[.86,.14],
class_sep=1.0,flip_y=.025,random_state=42
)
X_train,X_test,y_train,y_test=train_test_split(
X,y,test_size=.25,stratify=y,random_state=42
)For imbalanced classification, predicting the majority class may yield high accuracy.
from sklearn.dummy import DummyClassifier
dummy=DummyClassifier(strategy="prior")
dummy.fit(X_train,y_train)
print("Dummy accuracy:",dummy.score(X_test,y_test))Dummy accuracy: 0.852Your model should beat a meaningful baseline—not merely produce a large-looking number.
For binary classification:
| Predicted − | Predicted + | |
|---|---|---|
| Actual − | TN | FP |
| Actual + | FN | TP |
From these four counts, many recurring metrics emerge.
\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}.
Useful when classes and error costs are reasonably balanced.
Dangerous when the positive class is rare or errors have asymmetric cost.
\text{Precision}=\frac{TP}{TP+FP}
asks:
Of the cases we flagged, how many were truly positive?
\text{Recall}=\frac{TP}{TP+FN}
asks:
Of all true positives, how many did we catch?
Fraud investigation with limited analysts: precision matters.
Cancer screening where missing a case is costly: recall can matter strongly.
Neither metric should be selected without the decision context.
F_1= 2\frac{\text{Precision}\cdot\text{Recall}} {\text{Precision}+\text{Recall}}.
F1 is a harmonic mean and is useful when you want one number balancing precision and recall.
But one number hides the tradeoff.
A classifier may produce a probability-like score p.
Decision:
\hat y= \begin{cases} 1,&p\ge t\\ 0,&p<t. \end{cases}
The threshold t=0.5 is not a law of nature.
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score, f1_score
clf=LogisticRegression(max_iter=1000).fit(X_train,y_train)
prob=clf.predict_proba(X_test)[:,1]
rows=[]
for t in np.linspace(.05,.95,19):
pred=(prob>=t).astype(int)
rows.append({
"threshold":t,
"precision":precision_score(y_test,pred,zero_division=0),
"recall":recall_score(y_test,pred,zero_division=0),
"f1":f1_score(y_test,pred,zero_division=0)
})
threshold_df=pd.DataFrame(rows)import plotly.express as px
long=threshold_df.melt("threshold",var_name="metric",value_name="score")
fig=px.line(long,x="threshold",y="score",color="metric",
markers=True,title="Threshold changes the operating point")
fig.show()First learn a scoring/ranking model. Then choose an operating threshold based on costs, capacity, risk and product objectives.
ROC plots
TPR \quad \text{vs.}\quad FPR
over thresholds.
AUC measures ranking discrimination: how well positives tend to receive higher scores than negatives.
from sklearn.metrics import roc_curve,roc_auc_score
fpr,tpr,_=roc_curve(y_test,prob)
auc=roc_auc_score(y_test,prob)
fig,ax=plt.subplots(figsize=(5,5))
ax.plot(fpr,tpr,label=f"AUC={auc:.3f}")
ax.plot([0,1],[0,1],linestyle="--")
ax.set(xlabel="False Positive Rate",ylabel="True Positive Rate",
title="ROC curve")
ax.legend()
plt.show()
For rare positive classes, PR curves often expose the useful operating region more directly.
from sklearn.metrics import precision_recall_curve,average_precision_score
precision,recall,_=precision_recall_curve(y_test,prob)
ap=average_precision_score(y_test,prob)
fig,ax=plt.subplots(figsize=(5,5))
ax.plot(recall,precision,label=f"AP={ap:.3f}")
ax.set(xlabel="Recall",ylabel="Precision",title="Precision–Recall curve")
ax.legend()
plt.show()
For binary classification:
-\frac1n\sum_i [y_i\log p_i+(1-y_i)\log(1-p_i)].
Confident wrong predictions are punished strongly.
Use when probability quality matters, not merely hard labels.
A calibrated model satisfies approximately:
P(Y=1\mid \hat p\approx0.8)\approx0.8.
from sklearn.calibration import calibration_curve
obs,pred=calibration_curve(y_test,prob,n_bins=10)
fig,ax=plt.subplots(figsize=(5,5))
ax.plot(pred,obs,marker="o")
ax.plot([0,1],[0,1],linestyle="--")
ax.set(xlabel="Predicted probability",ylabel="Observed rate",
title="Calibration")
plt.show()
Ranking and calibration are separate properties.
Suppose residuals are
e_i=y_i-\hat y_i.
MAE=\frac1n\sum|e_i|.
Easy to interpret in target units and less dominated by large errors than MSE.
MSE=\frac1n\sum e_i^2.
Large errors receive quadratic penalty.
RMSE=\sqrt{MSE}.
Returns the error to target units.
R^2=1-\frac{\sum(y_i-\hat y_i)^2} {\sum(y_i-\bar y)^2}.
Measures improvement relative to predicting the mean under the usual definition.
rng=np.random.default_rng(8)
errors=rng.normal(0,10,500)
errors[:5]=[60,-75,90,-110,130]
mae=np.mean(np.abs(errors))
rmse=np.sqrt(np.mean(errors**2))
print("MAE :",round(mae,2))
print("RMSE:",round(rmse,2))MAE : 8.89
RMSE: 13.89RMSE reacts much more strongly to rare large misses.
That can be desirable if large misses are disproportionately costly.
A single regression metric cannot tell you how a model is wrong.
Plot residuals against:
Look for:
A single split can be lucky or unlucky.
K-fold CV:
Fold 1: VALID | train train train train
Fold 2: train | VALID | train train train
Fold 3: train train | VALID | train train
...from sklearn.model_selection import cross_validate
scores=cross_validate(
LogisticRegression(max_iter=1000),
X,y,
cv=5,
scoring=["roc_auc","average_precision","neg_log_loss"]
)
pd.DataFrame(scores).filter(like="test_").describe()| test_roc_auc | test_average_precision | test_neg_log_loss | |
|---|---|---|---|
| count | 5.000000 | 5.000000 | 5.000000 |
| mean | 0.880999 | 0.646018 | -0.271305 |
| std | 0.008790 | 0.029056 | 0.010784 |
| min | 0.872216 | 0.614416 | -0.282189 |
| 25% | 0.872343 | 0.630749 | -0.280157 |
| 50% | 0.881902 | 0.632345 | -0.274053 |
| 75% | 0.886150 | 0.668374 | -0.262065 |
| max | 0.892385 | 0.684208 | -0.258062 |
Report variation, not only the mean.
Random K-fold may be appropriate.
Use stratification.
Use grouped splitting to prevent entity leakage.
Use time-ordered validation.
A sophisticated metric on an unrealistic split produces a sophisticated-looking wrong answer.
from sklearn.model_selection import StratifiedKFold,cross_val_score
vals=[]
for seed in range(15):
cv=StratifiedKFold(n_splits=5,shuffle=True,random_state=seed)
s=cross_val_score(
LogisticRegression(max_iter=1000),
X,y,cv=cv,scoring="roc_auc"
)
vals += [{"seed":seed,"fold":i+1,"auc":v} for i,v in enumerate(s)]
cv_df=pd.DataFrame(vals)fig=px.scatter(cv_df,x="seed",y="auc",color="fold",
title="Performance varies across validation samples")
fig.show()Hyperparameters are selected using training/validation information.
Examples:
Tools include:
Never tune repeatedly on the final test set.
Plot training and validation performance against training-set size.
Interpretation:
train high, validation low
→ overfitting / variance problem
both low
→ underfitting / bias or weak features
validation still improving with n
→ more data may helpVary one hyperparameter and compare train vs validation scores.
Example: tree depth.
depth too small → underfit
depth moderate → useful region
depth huge → training perfect, validation deterioratesA metric is an estimate from a finite sample.
Bootstrap evaluation:
test rows
↓ sample with replacement
metric₁
repeat B times
↓
distribution of metric
↓
uncertainty intervalrng=np.random.default_rng(99)
boots=[]
idx=np.arange(len(y_test))
for _ in range(1000):
b=rng.choice(idx,size=len(idx),replace=True)
if len(np.unique(y_test[b]))==2:
boots.append(roc_auc_score(y_test[b],prob[b]))
print(np.quantile(boots,[.025,.5,.975]))[0.82901722 0.85677176 0.88489216]Overall performance can hide failures.
Evaluate by:
A model can have excellent aggregate AUC while performing poorly on a strategically important subgroup.
Suppose:
Then a threshold can be evaluated by expected value, not only F1.
V(t)=120TP(t)-15FP(t)-200FN(t).
The best statistical metric is not automatically the best business operating point.
Offline:
Online:
A model that wins offline still needs evidence that it improves the real system.
Evaluation does not end at launch.
Monitor:
BASELINE → SPLIT → METRICS → THRESHOLD → DIAGNOSTICS → CROSS-VALIDATE → UNCERTAINTY → SEGMENTS → BUSINESS VALUE → ONLINE TEST → MONITOR
A trustworthy evaluation answers more than “What is the score?”
Ask: