9. `cast` — changing a column's type

Every column has a type (orders.printSchema() shows them). cast returns the column as a different type. You'll reach for it constantly — CSV columns arrive as strings, IDs need to become text for joining, numbers need widening.

orders.select(
    "order_id",
    F.col("order_id").cast("string").alias("id_str"),
    F.col("unit_price").cast("string").alias("price_str"),
).show(3)

cast accepts type names ("int", "double", "string", "date", "boolean") or type objects (IntegerType()).

The common case: text → number

Data read loosely often lands as strings. Cast it to compute with it:

prices = spark.createDataFrame([("9.99",), ("18.50",), ("32.40",)], ["price"])
prices.select(F.col("price").cast("double").alias("price_num")).show()

A cast can create nulls — count them

The reason to respect cast: on real Spark, a value that can't be converted becomes null, silently — "abc" cast to int is just null, no error. So after casting questionable data, count the nulls you created before trusting the result. (Module 3 gives you the null tools.)

Engine note. This course runs on an in-browser engine (DuckDB) that is stricter than a real Spark cluster about casts: an impossible cast here raises an error instead of nulling, and it rounds float→int where Spark truncates (9.99 → 10 here, 9 on Spark). The everyday casts above behave identically; the notebook below runs the exact Spark ANSI semantics if you want to see the difference.

# colab: 03-dataframe-fundamentals/01-select-filter-withcolumn
# Real Spark: a bad cast nulls instead of raising, and float->int truncates.
bad = spark.createDataFrame([("abc",), ("9.99",)], ["v"])
bad.select(F.col("v").cast("int").alias("as_int")).show()
# +------+
# |as_int|
# +------+
# |  NULL|      <- "abc" is unconvertible: null, no error
# |  NULL|      <- "9.99" isn't an int literal either
# +------+

spark.createDataFrame([(9.99,)], ["v"]).select(
    F.col("v").cast("int").alias("as_int")   # 9, not 10 — Spark truncates
).show()

Your turn

On orders, cast quantity to a double and order_id to a string, then return order_id and quantity, as result.

result = orders  # <- cast quantity to double, order_id to string; select both
result.show()

result = (
    orders
    .withColumn("quantity", F.col("quantity").cast("double"))
    .withColumn("order_id", F.col("order_id").cast("string"))
    .select("order_id", "quantity")
)