20 Famous Open Data Sources for Analytics, and How to Extract Data πŸ—„οΈ

Resources
Datasets
Open Data
APIs
The twenty best-known free data sources, from Kaggle and UCI to FRED, the World Bank, NASA and SEC EDGAR, with a filterable catalogue, extraction code for each access method and a simulated end-to-end pipeline.
Author

MLStation

Published

October 1, 2026

Good practice data is the fuel of every project. The twenty sources below are free, famous and reliable. Some give you a CSV with one click; others need an API call, a Python package or a SQL query. This guide shows what each source is good for and exactly how to pull data from it. πŸš€

Tip🎯 What you will get
  • πŸ—‚οΈ A filterable catalogue of 20 open data sources
  • 🧭 A decision flow for choosing the right extraction method
  • πŸ’» Copy-ready extraction code for downloads, APIs, Python packages, SQL and cloud buckets
  • πŸ”„ An animated extract β†’ clean β†’ store pipeline and a worked, simulated example
  • βš–οΈ Notes on licences, rate limits and good manners
NoteπŸ“š The Resources hub

🧭 Overview: where to find resources Β· πŸ›€οΈ Learning paths Β· πŸ“ Cheat sheets Β· πŸ“– Glossary Β· πŸ—„οΈ 20 open data sources Β· πŸ“„ Paper notes (coming soon)

πŸ—‚οΈ The catalogue

How do you get the data out? The chart counts the main access method for each source.

NoteπŸ“ Note

The FRED CSV link works without a key, for example fredgraph.csv?id=UNRATE (FRED); the full API needs a free key (FRED API docs).

🧭 Which extraction method?

flowchart TD
  A["❓ What does the source offer?"] --> B{"A download link?"}
  B -- yes --> C["πŸ“₯ pandas.read_csv / read_parquet<br/>straight from the URL"]
  B -- no --> D{"A Python package?"}
  D -- yes --> E["πŸ“¦ ucimlrepo, datasets,<br/>fetch_openml, earthaccess, kaggle"]
  D -- no --> F{"A REST API?"}
  F -- yes --> G["🌐 requests + JSON<br/>pagination, keys, rate limits"]
  F -- no --> H{"A SQL warehouse<br/>or cloud bucket?"}
  H -- SQL --> I["πŸ—ƒοΈ BigQuery client"]
  H -- bucket --> J["☁️ S3 with --no-sign-request"]

πŸ’» Extraction recipes

Each tab is a working pattern. Replace dataset codes and keys with your own. Keys belong in environment variables, never in notebooks you share.

import pandas as pd

# Our World in Data: every chart has a CSV
le = pd.read_csv(
    "https://ourworldindata.org/grapher/life-expectancy.csv")

# FRED: one series, no key needed
unrate = pd.read_csv(
    "https://fred.stlouisfed.org/graph/fredgraph.csv?id=UNRATE",
    parse_dates=["observation_date"])

# NYC TLC: copy a monthly Parquet link from the TLC page
trips = pd.read_parquet("<paste-yellow-tripdata-link>.parquet")
# UCI repository
from ucimlrepo import fetch_ucirepo          # pip install ucimlrepo
wine = fetch_ucirepo(id=109)
X, y = wine.data.features, wine.data.targets

# OpenML through scikit-learn
from sklearn.datasets import fetch_openml
titanic = fetch_openml("titanic", version=1, as_frame=True).frame

# Hugging Face
from datasets import load_dataset            # pip install datasets
imdb = load_dataset("imdb", split="train").to_pandas()

# Kaggle (token in ~/.kaggle/kaggle.json)
# kaggle datasets download -d blastchar/telco-customer-churn --unzip
import os, requests, pandas as pd

# World Bank: no key; JSON when format=json
url = ("https://api.worldbank.org/v2/country/US;IN;BR"
       "/indicator/NY.GDP.MKTP.CD")
page = requests.get(url, params={"format": "json",
                                 "per_page": 500}).json()
meta, rows = page                        # [paging info, records]
gdp = pd.json_normalize(rows)[["country.value", "date", "value"]]

# FRED API: free key in an environment variable
obs = requests.get(
    "https://api.stlouisfed.org/fred/series/observations",
    params={"series_id": "CPIAUCSL", "file_type": "json",
            "api_key": os.environ["FRED_API_KEY"]}).json()
cpi = pd.DataFrame(obs["observations"])

# SEC EDGAR: no key, but identify yourself
hdr = {"User-Agent": "Your Name your.email@example.com"}
apple = requests.get(
    "https://data.sec.gov/submissions/CIK0000320193.json",
    headers=hdr).json()
import requests, pandas as pd

# OpenStreetMap Overpass: cafΓ©s in central Austin
query = """
[out:json][timeout:25];
node["amenity"="cafe"](30.25,-97.76,30.29,-97.72);
out;
"""
r = requests.post("https://overpass-api.de/api/interpreter",
                  data={"data": query}).json()
cafes = pd.DataFrame([{"name": e["tags"].get("name"),
                       "lat": e["lat"], "lon": e["lon"]}
                      for e in r["elements"]])
# pip install google-cloud-bigquery ; gcloud auth application-default login
from google.cloud import bigquery
client = bigquery.Client(project="your-project-id")
sql = """
SELECT name, SUM(number) AS babies
FROM `bigquery-public-data.usa_names.usa_1910_current`
WHERE year >= 2000
GROUP BY name ORDER BY babies DESC LIMIT 10
"""
top_names = client.query(sql).to_dataframe()
# AWS Open Data and Common Crawl allow anonymous access
aws s3 ls s3://commoncrawl/crawl-data/ --no-sign-request

# copy one file to your machine
aws s3 cp s3://<bucket>/<path>/file.parquet . --no-sign-request

BigQuery gives every project the first 1 TB of queries per month free, and Google pays to store the public datasets (Google Cloud). NOAA’s API needs a token in the request header and allows 5 requests per second and 10,000 per day (NOAA).

πŸ”„ The extraction pipeline, animated

Real projects wrap extraction in a small pipeline so it can be re-run safely. Press play and follow a batch of records.

πŸ§ͺ Worked example: cleaning API data (simulated)

APIs rarely return a tidy table. Below we simulate a World Bank-style JSON response for five fictional countries, with typical problems: nested fields, values as strings, missing years and one unit error. Then we clean it in a few lines.

import numpy as np, pandas as pd, plotly.express as px
from ms_charts import COLORS, PLOTLY_CONFIG
rng = np.random.default_rng(4)

# --- simulate a nested API response (fictional countries) ---
countries = [("Aurelia", 120), ("Borealis", 80), ("Caldera", 45),
             ("Deltora", 30), ("Equinox", 15)]
records = []
for c, start in countries:
    level = start
    for year in range(2005, 2026):
        level *= 1 + rng.normal(0.035, 0.025)
        # values arrive as strings, about 6% are missing
        v = None if rng.random() < 0.06 else str(round(level * 1e9))
        records.append({"country": {"id": c[:2].upper(), "value": c},
                        "date": str(year), "value": v})
# one record has a unit error (1000x too big)
if records[30]["value"] is None:
    records[30]["value"] = "1e9"
records[30]["value"] = str(float(records[30]["value"]) * 1000)

# --- clean ---
df = pd.json_normalize(records)                  # flatten nested JSON
df = df.rename(columns={"country.value": "country"})
df["year"] = df["date"].astype(int)
df["gdp_bn"] = pd.to_numeric(df["value"]) / 1e9  # strings β†’ numbers
med = df.groupby("country").gdp_bn.transform("median")
df.loc[df.gdp_bn > 20 * med, "gdp_bn"] = np.nan  # drop the unit error
df["gdp_bn"] = df.groupby("country").gdp_bn.transform(
    lambda s: s.interpolate(limit_direction="both"))  # fill gaps
print(f"{df.value.isna().sum()} missing values filled, 1 unit error fixed")
2 missing values filled, 1 unit error fixed
Show the code 
fig = px.line(df, x="year", y="gdp_bn", color="country", markers=True,
              color_discrete_sequence=[COLORS["indigo"], COLORS["teal"], COLORS["amber"], COLORS["rose"], COLORS["violet"]])
fig.update_layout(height=380, yaxis_title="GDP (billions, simulated)", xaxis_title=None, legend_title=None)
fig.show(config=PLOTLY_CONFIG)
Figure 1: Cleaned, simulated GDP series for five fictional countries.

πŸ›‘οΈ Be a good data citizen

Rule Why How
πŸ“œ Read the licence Some data cannot be used commercially or needs attribution Check the source’s terms page; cite it in your charts
⏱️ Respect rate limits Too many calls get you blocked Sleep between calls, use bulk downloads when offered
πŸ’Ύ Cache what you fetch Re-downloading wastes time and quota Save raw responses to Parquet or JSON with a date
πŸ”‘ Hide your keys Leaked keys get abused Environment variables, .env files in .gitignore
🧾 Record provenance You must be able to reproduce results Log URL, parameters and download date
πŸ” Protect people Even open data can identify individuals Aggregate, drop identifiers, follow privacy rules

A small helper that caches and retries covers most needs:

fetch.py
import time, json, pathlib, hashlib, requests

def fetch_json(url, params=None, cache="raw", retries=3):
    h = hashlib.sha1(f"{url}{params}".encode()).hexdigest()[:16]
    key = pathlib.Path(cache) / f"{h}.json"
    if key.exists():
        return json.loads(key.read_text())          # cached copy
    for attempt in range(retries):
        r = requests.get(url, params=params, timeout=30)
        if r.status_code == 429:                    # rate limited
            time.sleep(2 ** attempt); continue
        r.raise_for_status()
        key.parent.mkdir(exist_ok=True)
        key.write_text(r.text)
        return r.json()
    raise RuntimeError("Gave up after retries")

🧩 Summary

  • Start with Kaggle, UCI, OpenML and Hugging Face for ML practice; use FRED, World Bank, Eurostat, Census and OWID for economics and society; NOAA, NASA and OSM for climate and maps; BigQuery, AWS and Common Crawl for big data.
  • Pick the extraction method from what the source offers: download β†’ package β†’ API β†’ SQL or bucket.
  • Flatten, convert, validate and cache what you fetch.
  • Follow licences, rate limits and privacy rules.
  • Put the data to work with a learning path πŸ›€οΈ.
Back to top