6. Referring to columns: `col`, `expr`, `lit`

There are four ways to point at a column in PySpark. Knowing which to use — and which quietly break — saves you a lot of confusion.

orders.select("product").show(2)          # 1. string name
orders.select(orders["product"]).show(2)  # 2. bracket on the DataFrame
orders.select(orders.product).show(2)     # 3. attribute
orders.select(F.col("product")).show(2)   # 4. col() — the course default

• String names work wherever a bare column is expected. Simple selection, grouping keys — fine.
• The moment you need an expression (quantity * unit_price, price > 100), you need a real Column object, and a plain string won't do.
• col("name") builds that Column object and works everywhere. Attribute style (orders.product) breaks on names with spaces or dots and turns ambiguous after joins (Module 4). Course rule: use F.col() for anything beyond a bare name.

expr — a SQL string as an expression

F.expr lets you write the expression as a SQL snippet. Same result, sometimes more readable:

orders.select(
    "product",
    F.expr("quantity * unit_price").alias("revenue"),
    F.expr("upper(category)").alias("cat"),
).show(3)

lit — a constant as a column

When you need a literal value where a Column is expected — a fixed tag, a constant in a calculation — wrap it in F.lit:

orders.select(
    "product",
    F.lit("EUR").alias("currency"),
    (F.col("unit_price") * F.lit(1.2)).alias("with_tax"),
).show(3)

Without lit, Spark would look for a column called "EUR" and fail. lit says "this is a value, not a column name."

Your turn

Using orders, return product and a revenue column (unit_price * quantity) — build the revenue with F.expr this time — as result.

result = orders  # <- select product and F.expr("unit_price * quantity") AS revenue
result.show()

result = orders.select("product", F.expr("unit_price * quantity").alias("revenue"))