← All modules

PS0 — PySpark

Why this module exists

PySpark looks like pandas and behaves like a distributed query compiler. Almost every mistake people make comes from that mismatch.

The single most useful thing you can hold in your head is that your Python code mostly doesn't run on your data. It builds a plan. The plan runs on the JVM. And the moments where Python does touch the data — UDFs, collect(), toPandas() — are exactly the moments where performance falls off a cliff.

Once you see the boundary, the rules stop being folklore. You know why a UDF is slow, why collect() kills the driver, and why withColumn in a loop produces a plan that takes longer to compile than to execute.

Scope, and what's deliberately elsewhere

Spark engine internals are not in this module. Execution model, shuffle mechanics, join strategy selection, Catalyst and adaptive query execution, memory management — those are covered in depth by module E7 of the sibling db-training repo, and lakehouse architecture by A0. Repeating them here would be duplication.

This module is about the Python API specifically: the boundary, the DataFrame API, UDF cost and Arrow, and writing PySpark that is testable and survives running on Glue and EMR. Where engine behaviour matters, it's named and pointed at rather than re-explained.

The one idea to hold onto

   YOUR PYTHON PROCESS          │          THE JVM EXECUTORS
   ─────────────────────        │          ──────────────────
   df = spark.read...           │
   df = df.filter(...)          │   builds a plan — nothing runs
   df = df.withColumn(...)      │
                                │
   df.write.parquet(...)   ────▶│   NOW it runs, on the JVM,
                                │   nowhere near your Python
   ═══════════════════════════════════════════════════════════
   THE EXPENSIVE CROSSINGS:
   udf(...)          → every row serialised to Python and back
   collect()         → all data to the driver
   toPandas()        → all data to the driver, as pandas

Every performance question in PySpark is "how often, and how much, are we crossing that line?"

What you'll be able to do

1. Explain the Python/JVM boundary, and predict which operations cross it.
2. Read a PySpark script and say where the plan is built versus where it executes.
3. Choose between a built-in function, a pandas UDF, and a plain Python UDF — and defend it with a mechanism, not a rule.
4. Name the Arrow configuration keys and say what they change.
5. Spot the driver-killers before code review does.
6. Write PySpark that can be unit-tested without a cluster.
7. Answer the standard PySpark interview questions with the why.

Lessons

Then: Cheat sheet · Lab · Quiz · Interview questions

A note on versions

PySpark's defaults change between versions, and several of the Arrow settings in lesson 3 changed recently. Facts here are verified against the current PySpark documentation on spark.apache.org as of 2026-08-12, and lesson 3 flags where behaviour is version-dependent.

⚠️ This matters more on AWS than anywhere else, because Glue and EMR pin specific Spark versions that are usually behind the current release. Always check which Spark version your Glue version or EMR release actually ships before assuming a default. Lesson 5 covers how.

A worked example of why this course fetches rather than remembers: while researching lesson 3, a web search asserted that spark.sql.execution.arrow.pyspark.enabled "doesn't appear in the current documentation". Fetching the actual page showed it does, and documents a default of true. The search result was wrong. Cite the page you fetched.

Facts verified 2026-08-12 against the pages cited in each lesson.

Keeps playing into the following modules — 73 min from here to the end of the course.