18. Conditional logic: `when` / `otherwise`

F.when(condition, value) is PySpark's CASE WHEN — it builds a column whose value depends on a test. Chain .when(...) for more branches and finish with .otherwise(...):

orders.select(
    "product",
    "unit_price",
    F.when(F.col("unit_price") >= 100, "premium")
     .when(F.col("unit_price") >= 30, "mid")
     .otherwise("budget")
     .alias("price_band"),
).show(8)

The branches are tried top to bottom; the first match wins. Leave off .otherwise(...) and unmatched rows get null — sometimes what you want, often a bug. Be deliberate.

Bucketing and flags

when is how you label rows into categories or build boolean flags:

orders.withColumn(
    "is_bulk", F.when(F.col("quantity") >= 3, True).otherwise(False)
).select("order_id", "product", "quantity", "is_bulk").show(6)

Because conditions are Column expressions, you can combine them with & / | just like in filter:

orders.select(
    "product", "category", "unit_price",
    F.when(
        (F.col("category") == "electronics") & (F.col("unit_price") >= 100),
        "premium electronics",
    ).otherwise("other").alias("segment"),
).show(6)

when respects nulls. If the condition is null (e.g. it tests a null quantity), that row falls through to the next branch or otherwise — the same null logic as filter. Handle nulls before bucketing if it matters.

Your turn

Add a price_band column to orders: "high" when unit_price is 100 or more, "medium" when 30 or more, otherwise "low". Return product, unit_price, and price_band. Assign to result.

result = orders  # <- when >=100 "high"; when >=30 "medium"; otherwise "low" AS price_band
result.show()

result = orders.select(
    "product",
    "unit_price",
    F.when(F.col("unit_price") >= 100, "high")
     .when(F.col("unit_price") >= 30, "medium")
     .otherwise("low")
     .alias("price_band"),
)