BrevFeed
Sign in

1. What runs here — and how to read a lesson

📖 Reading · 5 min
💡 Most code boxes below are live — edit one and hit Run. Boxes without a Run button are reference-only (they can't run in your browser).

This is a practical PySpark course. You won't watch slides about clusters; you'll write DataFrame code and run it, lesson after lesson, until the API is muscle memory. Everything runs right here in your browser — no install, no cluster, no Databricks account.

The one thing to know up front

Real Spark runs on a cluster of machines coordinated by a Java process. You can't fit that in a browser tab. But 90% of the PySpark you'll ever write is the DataFrame API — select, filter, groupBy, joins, window functions — and that we can run locally against a real query engine. So the code in these lessons is genuine PySpark:

from pyspark.sql import functions as F

The same lines work unchanged on a real Spark cluster. Where a topic truly needs a cluster (shuffle internals, RDDs, explain plans), it's covered in the companion video course, not typed here.

Three kinds of code box

1. Playground — editable, with a ▶ Run button. Change it, run it, break it. This one builds a tiny DataFrame and prints it:

people = spark.createDataFrame(
    [("Ada", "London"), ("Alan", "Manchester"), ("Grace", "New York")],
    ["name", "city"],
)
people.show()

You already have a spark session and F (for pyspark.sql.functions) ready in every box — the course sets those up for you.

2. Setup data — a collapsed "Setup data" box you'll see on most pages. It builds the sample DataFrames and runs before every Run, so each run starts clean no matter what you did last time.

3. Your turn — a graded exercise. Write the answer, hit ✓ Check, and it compares your result to the expected one. Assign your answer to a variable named result. Complete a lesson's exercises and it's marked done.

Here's one. The setup below gives you people; keep only the people in London and return just their name column.

people = spark.createDataFrame(
    [("Ada", "London"), ("Alan", "Manchester"), ("Grace", "New York"), ("Edsger", "London")],
    ["name", "city"],
)

# Assign your answer to `result`.
result = people  # <- filter to London, then select only "name"
result.show()

result = people.filter(F.col("city") == "London").select("name")

That's the whole loop: read → run → your turn. Next up: creating DataFrames properly and the handful of methods you'll use constantly.

Next →