Show the code
import numpy as np
rng=np.random.default_rng(42)
X=rng.normal(size=(1000,8))
beta=np.array([2,-1,.5,0,0,1.2,-.8,.3])
y=X@beta+rng.normal(0,1,1000)
print(X.shape,y.shape)(1000, 8) (1000,)Machine Learning Tools: From Notebook Experiment to Reproducible System
ML
Tools
MLOps
Data Science
A teaching-oriented map of the modern Data Science toolkit: environments, data, modeling, pipelines, tuning, tracking, explainability, testing, orchestration, deployment and monitoring.
🧰 A Data Scientist’s toolkit is not a pile of libraries. It is a chain of responsibilities.
The question is not “How many ML tools do I know?” It is:
Can I reproduce the data, model, metric and decision that produced this result—and move it safely toward production?
1. The Modern ML Workflow
ENVIRONMENT → DATA → EDA → FEATURES → MODEL → TUNE → EVALUATE → TRACK → EXPLAIN → PACKAGE → DEPLOY → MONITOR
2. The Tool Map
🐍 Compute
Python · NumPy · SciPy
🧾 Tabular data
pandas · Polars · SQL
📊 Visualization
Matplotlib · Seaborn · Plotly
🤖 Classical ML
scikit-learn · XGBoost · LightGBM · CatBoost
🧠 Deep learning
PyTorch · TensorFlow/Keras
🔬 Tuning
Grid/Random search · Optuna
🧪 Tracking
MLflow · Weights & Biases-like platforms
🔎 Explainability
Permutation importance · SHAP
📦 Reproducibility
Git · environments · Docker
⚙️ Pipelines
sklearn Pipeline · workflow orchestrators
🚀 Serving
FastAPI · batch jobs · model-serving systems
📡 Monitoring
metrics · logs · drift · model/business performance
3. Environments: “Which Python Ran This?”
A reproducible project begins with an isolated environment.
python -m venv .venv
source .venv/bin/activate
python -m pip install numpy pandas scikit-learnRecord direct dependencies:
numpy
pandas
scikit-learn
plotlyThen verify:
which python
python --version
python -m pip --versionEnvironment bugs masquerade as ML bugs
A surprising number of “model problems” are actually incompatible package versions, wrong Python executables or inconsistent environments.
4. NumPy — The Numerical Substrate
Core ideas:
- arrays,
- vectorization,
- broadcasting,
- linear algebra,
- random generators.
5. pandas and Polars — Work with Tables
pandas remains central in Data Science workflows:
summary = (
df.groupby("segment")
.agg(
customers=("customer_id","nunique"),
revenue=("revenue","sum")
)
.reset_index()
)Columnar DataFrame tools such as Polars can be useful for performance-oriented workflows.
The important transferable skill is dataframe reasoning:
select → filter → derive → group → aggregate → join → reshape6. SQL — Move Computation Toward the Data
Real company data often begins in a warehouse, not a CSV.
SELECT
customer_id,
COUNT(*) AS orders_90d,
SUM(amount) AS spend_90d
FROM orders
WHERE order_date >= CURRENT_DATE - INTERVAL '90 days'
GROUP BY customer_id;A Data Scientist should understand both SQL and Python because the boundary between them is an engineering decision.
7. scikit-learn — The Classical ML Workbench
A consistent estimator interface:
model.fit(X_train,y_train)
pred=model.predict(X_test)Transformers:
transformer.fit(X_train)
X_new=transformer.transform(X_test)This consistency makes composition possible.
8. Pipelines — Make Preprocessing Part of the Model
Show complete sklearn pipeline
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
X,y=make_classification(
n_samples=2500,n_features=12,n_informative=7,
weights=[.8,.2],random_state=42
)
Xtr,Xte,ytr,yte=train_test_split(
X,y,test_size=.25,stratify=y,random_state=42
)
pipe=Pipeline([
("scale",StandardScaler()),
("model",LogisticRegression(max_iter=1000))
])
pipe.fit(Xtr,ytr)
p=pipe.predict_proba(Xte)[:,1]
print("AUC:",roc_auc_score(yte,p))AUC: 0.8083605283605284Pipeline principle
If a transformation must be repeated at inference time, it belongs in a reproducible transformation pipeline—not hidden in notebook cell 17.
9. Tree-Boosting Libraries
For tabular ML, important libraries include:
- XGBoost,
- LightGBM,
- CatBoost.
Example with XGBoost’s sklearn-style API:
from xgboost import XGBClassifier
model=XGBClassifier(
n_estimators=400,
max_depth=5,
learning_rate=.05,
subsample=.8,
colsample_bytree=.8
)
model.fit(X_train,y_train)The transferable concepts matter more than memorizing package arguments:
- boosting,
- learning rate,
- tree complexity,
- row/column subsampling,
- early stopping,
- validation.
10. Hyperparameter Search
Grid search
Try all specified combinations.
Random search
Sample combinations from distributions.
Smarter optimization
Tools such as Optuna automate trial generation and pruning/search strategies.
Conceptually:
suggest hyperparameters
↓
train model
↓
validation metric
↓
optimizer learns where to search next11. Simulate a Hyperparameter Surface
Show tuning-surface simulation
import pandas as pd
rng=np.random.default_rng(7)
rows=[]
for depth in range(2,11):
for lr in np.linspace(.01,.3,25):
score=(
.83
-.0035*(depth-5)**2
-.75*(lr-.08)**2
+rng.normal(0,.002)
)
rows.append({"depth":depth,"learning_rate":lr,"auc":score})
tune=pd.DataFrame(rows)Show the code
import plotly.express as px
fig=px.scatter_3d(
tune,x="depth",y="learning_rate",z="auc",
color="auc",
title="Hyperparameter tuning is search over a response surface"
)
fig.show()12. Experiment Tracking — Stop Naming Files final_v7_really_final.pkl
A model run has context:
code version
data version
parameters
metrics
artifacts
model
time
environmentExperiment tracking systems such as MLflow organize these into runs/experiments.
import mlflow
mlflow.set_experiment("churn")
with mlflow.start_run():
mlflow.log_param("max_depth",5)
mlflow.log_metric("validation_auc",0.842)Tracking makes questions answerable:
Which code, data and hyperparameters produced the model we selected?
13. Git — Track the Code, Not Just the Notebook
git status
git add src/train.py
git commit -m "Add leakage-safe preprocessing pipeline"Use Git for:
- code,
- configuration,
- documentation,
- tests.
Large raw datasets and model artifacts usually need purpose-built storage/versioning rather than ordinary Git history.
14. Data and Artifact Versioning
Reproducibility requires identifying:
code version + data snapshot + configuration + environmentTools may include:
- warehouse snapshot/table versions,
- object-storage paths,
- DVC-like data versioning,
- experiment artifacts,
- feature-store versions.
15. Explainability Tools
Permutation importance
Shuffle a feature and measure performance degradation.
SHAP
Shapley-value-based methods attribute a prediction relative to a reference/background formulation.
Explanation is not causation
SHAP values, coefficients and feature importance explain aspects of a predictive model. They do not prove that intervening on the feature causes the predicted outcome to change.
16. Testing ML Code
ML systems need ordinary software tests.
def test_feature_ratio_is_finite():
out = build_features(sample_df)
assert np.isfinite(out["spend_per_visit"]).all()Also test:
- schema,
- ranges,
- missingness,
- row counts,
- deterministic behavior where expected,
- train/serve feature consistency.
17. Data Validation
Before training:
expected columns?
types correct?
unexpected nulls?
categories changed?
range violations?
duplicate keys?
timestamp sane?Model quality cannot compensate for a broken input contract.
18. Docker — Package the Runtime
A container records the runtime environment more explicitly than “it works on my laptop.”
Conceptually:
code
+ Python
+ dependencies
+ OS-level runtime
──────────────
container imageThis helps training and serving environments agree.
19. Orchestration — Turn Steps into a Workflow
A production workflow might be:
extract_data
↓
validate_data
↓
build_features
↓
train
↓
evaluate
↓
register_if_qualifiedWorkflow tools coordinate dependencies, schedules, retries and observability.
Examples in the ecosystem include Airflow, Prefect and Dagster-style orchestrators.
20. Batch vs Online Inference
Batch
nightly data → model → scores tableGood when predictions can be precomputed.
Online
request → features → model → responseGood when predictions must respond to current context with low latency.
The serving architecture should follow the product requirement—not fashion.
21. APIs
A simple prediction service can expose an endpoint:
from fastapi import FastAPI
app=FastAPI()
@app.post("/predict")
def predict(features: dict):
# validate → transform → model → response
...But production serving also needs validation, logging, security, scaling and monitoring.
22. Monitoring
Monitor three layers.
System
- latency,
- errors,
- throughput,
- CPU/memory.
Data/model
- missingness,
- feature distributions,
- prediction distributions,
- drift,
- calibration/performance when labels arrive.
Business
- conversion,
- loss prevented,
- retention,
- operational workload,
- guardrails.
23. Interactive Drift Simulation
Show drift simulation
rng=np.random.default_rng(18)
train_feature=rng.normal(0,1,2500)
drift_rows=[]
for shift in [0,.25,.5,.75,1.0]:
current=rng.normal(shift,1,1200)
drift_rows += [{"shift":shift,"value":v} for v in current]
drift=pd.DataFrame(drift_rows)Show the code
fig=px.histogram(
drift,x="value",
animation_frame="shift",
nbins=45,
range_x=[-4,5],
title="A feature distribution moves after deployment"
)
fig.show()24. Notebooks vs Production Code
Notebook:
- exploration,
- visualization,
- narrative,
- prototyping.
Modules/scripts/packages:
- reusable logic,
- tests,
- automation,
- deployment.
A healthy transition:
notebook discovery
↓
stable function
↓
module
↓
test
↓
pipeline25. A Practical Project Layout
ml-project/
├── README.md
├── pyproject.toml
├── notebooks/
│ └── exploration.ipynb
├── src/
│ ├── features.py
│ ├── train.py
│ └── predict.py
├── tests/
├── configs/
├── artifacts/
└── reports/26. Tool Selection Principles
Do not ask:
What is the most advanced tool?
Ask:
- Does the team already support it?
- Does it solve a recurring pain?
- Can others reproduce it?
- Is operational complexity justified?
- Does it integrate with the data/platform?
- Can we observe and debug it?
Tool maturity means knowing when NOT to add a tool
A Python script, SQL query and scheduled job can be the correct architecture. Complexity is a cost.
27. Today’s Data Scientist: Three Layers
ANALYSIS
NumPy · pandas · SQL · visualization
↓
MODELING
scikit-learn · boosting · PyTorch
↓
SYSTEM
Git · tracking · pipelines · containers
orchestration · serving · monitoringNot every Data Scientist owns every production layer, but understanding the interfaces makes collaboration dramatically better.
28. The Reproducibility Test
Can a teammate answer:
- Which dataset trained this model?
- Which code revision?
- Which feature definitions?
- Which hyperparameters?
- Which environment?
- Which validation protocol?
- Which metric selected it?
- Where is the artifact?
- How is it served?
- How will we know it degraded?
If not, the ML system has undocumented state.
29. Tool Learning Order
Python/NumPy → pandas/SQL → visualization → scikit-learn → Git → pipelines → boosting → tuning → tracking → testing → containers → orchestration → deployment → monitoring
Summary
Tools should support a lifecycle:
understand → build → reproduce → compare → package → operate → observe.
The professional skill is not knowing every library. It is understanding which responsibility each tool solves, and keeping the system as simple as the problem permits.