14. Aggregating into lists: `collect_list` and `collect_set`

Most aggregates reduce a group to a single number. Sometimes you instead want to gather the values themselves into an array — every product a customer bought, every country a category shipped to. That's collect_list and collect_set.

orders.dropDuplicates().groupBy("category").agg(
    F.collect_set("country").alias("countries")
).show(truncate=False)

• collect_list keeps every value, including duplicates, in group order-ish.
• collect_set keeps only the distinct values (like a set).

Use collect_set when you want "which distinct X", collect_list when counts or repetition matter:

orders.dropDuplicates().groupBy("customer_id").agg(
    F.collect_list("product").alias("products_bought"),
    F.count("*").alias("n_orders"),
).orderBy(F.col("n_orders").desc()).show(5, truncate=False)

The result column is an array — a first taste of the complex types that Module 6 is all about. You can F.size(...) it, F.explode(...) it back into rows, or index into it.

Careful at scale. collect_list pulls every value in a group into one array held in memory. On a group with millions of rows that's an out-of-memory risk. Great for "products per customer" (small groups); dangerous for "all events per day" (huge groups).

Your turn

For each country (deduplicated orders), collect the distinct categories ordered/shipped there into an array called categories. Return country and categories, as result.

result = orders  # <- dropDuplicates, groupBy country, collect_set(category) AS categories
result.show(truncate=False)

result = (
    orders.dropDuplicates()
    .groupBy("country")
    .agg(F.collect_set("category").alias("categories"))
)