CLI for Data Science & ML

Use shell pipelines, environments, processes, SSH and text tools to work effectively on real Data Science systems.

Data Science eventually leaves the notebook.

Models run on servers. Jobs write logs. Environments break. Files become too large to open manually.

That is where CLI fluency pays off.

TipKey ideas

grep find sort uniq wc cut awk sed ssh scp curl tar ps top kill PATH venv

Inspect data without opening it

head -5 data.csv
wc -l data.csv
du -h data.csv
grep "missing" pipeline.log

Find files:

find . -name "*.parquet"

Watch a running job:

tail -f training.log

Understand redirection

python train.py > train.log 2>&1

means:

stdout ─┐
        ├──▶ train.log
stderr ─┘

This tiny idea is enormously useful for long-running jobs.

Environments

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

Know which Python is executing:

which python
python --version
python -m pip --version

Resources

The Missing Semester:
https://missing.csail.mit.edu/

Ubuntu CLI tutorial:
https://ubuntu.com/desktop/docs/en/latest/tutorial/the-linux-command-line-for-beginners/

Bash manual:
https://www.gnu.org/software/bash/manual/

TipProfessional checkpoint

You do not need to become a Linux administrator. For DS/ML, aim to navigate confidently, inspect files/logs, manage environments, understand processes, use SSH, and compose small shell pipelines.