8. `withColumn`, `withColumnRenamed`, `drop`

select rebuilds the column list from scratch. When you just want to add or change one column and keep the rest, use withColumn.

orders.withColumn("line_total", F.col("quantity") * F.col("unit_price")).show(3)

Pass an existing name and it replaces that column:

orders.withColumn("unit_price", F.col("unit_price") * 1.2).show(3)  # +20%, same column

Immutability — the no-op bug

DataFrames are immutable. withColumn returns a new DataFrame; the original is untouched. The classic beginner bug is forgetting to assign the result:

orders.withColumn("line_total", F.col("quantity") * F.col("unit_price"))
# ^ runs fine, changes nothing — the new DataFrame was thrown away
orders = orders.withColumn("line_total", F.col("quantity") * F.col("unit_price"))
# ^ this is what you meant

Rename and drop

orders.withColumnRenamed("unit_price", "price").drop("order_date", "country").show(3)

drop silently ignores names that don't exist — convenient, but a typo like drop("prodct") drops nothing and says nothing. Double-check your spelling.

Performance footnote. Each withColumn adds a step to the query plan; a loop calling it hundreds of times bloats plan analysis. For bulk column work, do it in one select. It doesn't matter at 41 rows — file it away for when you're at scale.

Your turn

Add a line_total column (quantity * unit_price) to orders, then return just order_id and line_total, as result.

result = orders  # <- add line_total, then select order_id and line_total
result.show()

result = (
    orders
    .withColumn("line_total", F.col("quantity") * F.col("unit_price"))
    .select("order_id", "line_total")
)