29. Arrays: `explode`, `size`, `array_contains`

Real data is rarely flat. Our events DataFrame is a clickstream where each event carries an array of items:

events.printSchema()
events.select("event_id", "items").show(5, truncate=False)

items is an array of structs — a list, where each element has sku and qty.

explode — one array element per row

explode turns each array element into its own row, copying the other columns. It's how you go from "an order with a list of items" to "one row per item":

events.select("event_id", F.explode("items").alias("item")).select(
    "event_id", "item.sku", "item.qty"
).show()

Note the count: explode drops rows whose array is empty or null (one event has no items, so it vanishes).

explode_outer and posexplode (real Spark). On a cluster, explode_outer keeps the empty-array rows (with nulls), and posexplode also gives you each element's index. The in-browser engine here only has plain explode — the notebook below runs those variants on real Spark.

# colab: 06-complex-nested-data/00-arrays
# explode drops the no-items event; explode_outer keeps it with a null item.
events.select("event_id", F.explode_outer("items").alias("item")).show()

# posexplode adds the element's position within its array.
events.select("event_id", F.posexplode("items").alias("pos", "item")).show()
# +--------+---+-----------+
# |event_id|pos|       item|
# +--------+---+-----------+
# |       2|  0|  {B2, 2}  |
# |       2|  1|  {C3, 1}  |

size and membership

size gives an array's length; combine higher-order functions with array_contains to test membership. Since items is an array of structs, first pull out the skus with transform (next lesson), then test:

events.select(
    "event_id",
    F.size("items").alias("n_items"),
    F.array_contains(F.transform("items", lambda x: x.sku), "A1").alias("has_A1"),
).show()

F.size returns -1 for a null array (not 0) — guard with coalesce if that matters.

Your turn

Explode the events items so there's one row per item, and return event_id, the item's sku, and its qty. Assign to result.

result = events  # <- select event_id, explode(items) AS item, then event_id, item.sku, item.qty
result.show()

result = events.select("event_id", F.explode("items").alias("item")).select(
    "event_id", "item.sku", "item.qty"
)