31. Structs — nested records

A struct is a record nested inside a column: several named fields grouped together. In events, each row has a user struct (id, country) and an optional payment struct:

events.select("event_id", "user", "payment").show(6, truncate=False)

Reaching into a struct — dot notation

Access a nested field with parent.field, either as a string in select or through F.col:

events.select(
    "event_id",
    F.col("user.id").alias("user_id"),
    F.col("user.country").alias("user_country"),
    F.col("payment.amount").alias("paid"),
).show()

Events without a payment show null for payment.amount — accessing a field of a null struct is null, not an error. That's the usual way to flatten a nested record into flat columns.

Building a struct

F.struct(...) bundles columns into a struct — useful for grouping related fields, or nesting before writing JSON:

orders.select(
    "order_id",
    F.struct("product", "category", "unit_price").alias("product_info"),
).select("order_id", "product_info", "product_info.product").show(5, truncate=False)

You can select the whole struct or dot into it — both work on the same column.

Flattening a struct into columns

Pull each field up to a top-level column by selecting it and aliasing:

events.select(
    "event_id",
    F.col("user.id").alias("id"),
    F.col("user.country").alias("country"),
).show()

struct.* (real Spark). On a cluster, select("user.*") expands every field of a struct at once — a one-liner flatten. The in-browser engine doesn't support the star form, so name the fields explicitly (as above). The notebook below runs user.* on real Spark.

# colab: 06-complex-nested-data/01-structs
# Real Spark: the star form expands every field of the struct in one go.
events.select("event_id", "user.*").show()
# +--------+----+-------+
# |event_id|  id|country|
# +--------+----+-------+
# |       1|u101|     IN|
# ...

# Handy inverse: pack loose columns back into a struct.
events.select("event_id", F.struct("user.id", "user.country").alias("user")).printSchema()

Your turn

Flatten the user struct in events: return event_id, the user's id as user_id, and their country as user_country. Assign to result.

result = events  # <- select event_id, user.id AS user_id, user.country AS user_country
result.show()

result = events.select(
    "event_id",
    F.col("user.id").alias("user_id"),
    F.col("user.country").alias("user_country"),
)