3. Transformations, actions, and lazy evaluation

This is the one mental model that makes PySpark click. Every DataFrame method is one of two kinds:

• Transformation — describes a change and returns a new DataFrame. select, filter, withColumn, groupBy, joins. These do not run anything. They just build up a recipe.
• Action — asks for a result, which forces the recipe to actually execute. show, count, collect, write.

Spark is lazy: it stacks up your transformations without touching data, then — the moment you call an action — plans the whole chain and runs it in one go. This is what lets it optimize across your entire pipeline instead of one step at a time.

See it happen

Building the transformation prints nothing — no data moves. Only the action at the end does work:

films = spark.createDataFrame(
    [(1, "Chinatown", 1974, 8.1), (2, "Alien", 1979, 8.5), (3, "Coco", 2017, 8.4)],
    ["id", "title", "year", "rating"],
)

hits = films.filter(F.col("rating") >= 8.4)   # transformation — nothing runs
print("built the recipe, no data touched yet")

hits.show()                                    # action — NOW it runs

The recipe is a query

Under the hood, your chain of transformations compiles to a query. You can see it — every DataFrame can show you the query it built with .sql():

films = spark.createDataFrame(
    [(1, "Chinatown", 1974, 8.1), (2, "Alien", 1979, 8.5), (3, "Coco", 2017, 8.4)],
    ["id", "title", "year", "rating"],
)

recipe = films.filter(F.col("rating") >= 8.4).select("title")
print(recipe.sql())   # the SQL your DataFrame chain becomes

That's the deep truth this whole course leans on: a DataFrame pipeline is a query. If you know SQL, you already know what PySpark is doing; if you don't, you'll pick up the shape of it here. (Module 7 makes the connection explicit — you'll even run SQL directly.)

Why laziness matters in practice

Because nothing runs until an action, a bug in a transformation often doesn't blow up where you wrote it — it surfaces later, at the show() or count(). When an error points at an action, the real mistake is usually somewhere up the chain that built the DataFrame.

Your turn

The setup gives you films. Build a DataFrame (assign to result) of the title and year of every film from the year 2000 or later.

films = spark.createDataFrame(
    [(1, "Chinatown", 1974, 8.1), (2, "Alien", 1979, 8.5), (3, "Coco", 2017, 8.4)],
    ["id", "title", "year", "rating"],
)

result = films  # <- keep year >= 2000, select title & year
result.show()

result = films.filter(F.col("year") >= 2000).select("title", "year")