Spark & Big Data: Think Distributed

Understand why distributed data processing is different and learn Spark’s core execution mental model before memorizing APIs.

Spark matters when the problem is no longer simply:

Which pandas function should I call?

but:

How should this computation execute across machines and partitions?

TipKey ideas

driver executor cluster partition DataFrame transformation action lazy evaluation shuffle stage task cache

The distributed picture

             DRIVER
                │
          execution plan
                │
       ┌────────┼────────┐
       ▼        ▼        ▼
   Executor  Executor  Executor
   partitions partitions partitions

Spark divides data and computation into work that can be performed across executors.

Transformations are lazy

filtered = df.filter(df.amount > 100)
summary = filtered.groupBy("region").count()

Spark can build a plan before executing it. An action such as:

summary.show()

causes work to be carried out.

Start here

Apache Spark Quick Start:
https://spark.apache.org/docs/latest/quick-start.html

PySpark User Guide:
https://spark.apache.org/docs/latest/api/python/user_guide/index.html

Spark SQL / DataFrames:
https://spark.apache.org/docs/latest/sql-programming-guide.html

Databricks PySpark overview:
https://docs.databricks.com/en/pyspark/

The concept to fear—in a healthy way

local operation
     ↓
possible network SHUFFLE
     ↓
data moves between partitions

Joins, aggregations and repartitioning can move large amounts of data across a cluster. Distributed performance is often about reducing unnecessary movement.

NoteDo not start with RDDs

For most modern structured-data learning, begin with DataFrames and Spark SQL. Learn lower-level abstractions later when you have a concrete reason.

Next: PySpark for Practical Data Science