2. SparkSession and your first DataFrame

Everything in PySpark starts from a SparkSession — the object named spark. It's your handle to the engine: you use it to create DataFrames, read files, and run SQL. In a real script you'd build one yourself:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("my-job").getOrCreate()

In this course that line is already done for you — spark is waiting in every code box.

Creating a DataFrame from rows

A DataFrame is a table: named, typed columns and rows. The quickest way to make one is createDataFrame from a list of tuples plus column names:

films = spark.createDataFrame(
    [(1, "Chinatown", 1974, 8.1), (2, "Alien", 1979, 8.5), (3, "Coco", 2017, 8.4)],
    ["id", "title", "year", "rating"],
)
films.show()

show() prints the first rows as a table. It's an action — it actually runs the query (more on that next lesson).

Looking at a DataFrame

Three methods you'll reach for constantly:

films = spark.createDataFrame(
    [(1, "Chinatown", 1974, 8.1), (2, "Alien", 1979, 8.5), (3, "Coco", 2017, 8.4)],
    ["id", "title", "year", "rating"],
)
films.printSchema()          # column names + inferred types
print("rows:", films.count())  # how many rows
films.show(2)                # just the first 2 rows

Notice year came out as a long (an integer) and rating as a double — Spark inferred the types from your data. That's convenient now; the next lesson shows when you'll want to spell types out yourself.

Careful with show vs collect. show() prints a preview and is safe. collect() pulls every row back into Python — fine for 3 rows, a way to crash a real job on 3 billion. Preview with show; only collect when you know the result is small.

Your turn

The setup gives you films. Return the title and rating columns for films rated 8.4 or higher, as result.

films = spark.createDataFrame(
    [(1, "Chinatown", 1974, 8.1), (2, "Alien", 1979, 8.5), (3, "Coco", 2017, 8.4)],
    ["id", "title", "year", "rating"],
)

result = films  # <- select title & rating, keep rating >= 8.4
result.show()

result = films.select("title", "rating").filter(F.col("rating") >= 8.4)