32. Maps — key/value columns

A map column holds key→value pairs (like a Python dict) in a single cell. Unlike a struct, the keys aren't fixed in the schema — handy for sparse or dynamic attributes.

Building a map

create_map(k1, v1, k2, v2, …) builds one from alternating key/value expressions:

attrs = orders.select(
    "order_id",
    F.create_map(
        F.lit("category"), F.col("category"),
        F.lit("country"), F.col("country"),
    ).alias("attrs"),
)
attrs.show(5, truncate=False)

Reading a map

• element_at(map, key) — look up one key's value (the key must be a Column, so wrap literals in F.lit).
• map_keys(map) / map_values(map) — get all keys or values as arrays.

attrs = orders.select(
    "order_id",
    F.create_map(
        F.lit("category"), F.col("category"),
        F.lit("country"), F.col("country"),
    ).alias("attrs"),
)
attrs.select(
    "order_id",
    F.element_at("attrs", F.lit("category")).alias("category"),
    F.map_keys("attrs").alias("keys"),
).show(5, truncate=False)

element_at on a missing key returns null — the map equivalent of a dict .get().

When to use a map vs a struct

• Struct — a fixed set of known fields (user.id, user.country). Schema-enforced, dot-access.
• Map — an open-ended set of keys that varies row to row (feature flags, event properties, tag→value). Access by element_at.

If you always know the field names, prefer a struct — it's typed and self-documenting.

Your turn

Build a map column attrs on orders with two entries — "product" → product and "category" → category — then read the product back out with element_at as product. Return order_id, attrs, and product. Assign to result.

result = orders  # <- create_map("product"->product, "category"->category) AS attrs; element_at(attrs,"product") AS product
result.show(truncate=False)

result = orders.select(
    "order_id",
    F.create_map(
        F.lit("product"), F.col("product"),
        F.lit("category"), F.col("category"),
    ).alias("attrs"),
).withColumn("product", F.element_at("attrs", F.lit("product")))