15. String functions

Real data is full of text that needs cleaning and reshaping. PySpark's string functions all live on F and return Column expressions, so they drop straight into select / withColumn.

orders.select(
    "product",
    F.upper("category").alias("cat_upper"),
    F.length("product").alias("name_len"),
).show(5)

Splitting and slicing

split turns a string into an array on a delimiter; substring takes a fixed slice (1-indexed):

orders.select(
    "product",
    F.split("product", " ").alias("words"),
    F.substring("product", 1, 4).alias("first4"),
).show(5, truncate=False)

show(truncate=False) stops Spark from cutting long values off at 20 characters — handy when you're inspecting text or arrays.

Cleaning: trim, concat_ws, regexp_replace

orders.select(
    F.concat_ws(" — ", "category", "product").alias("label"),
    F.trim("country").alias("country"),
    F.regexp_replace("product", "[0-9]", "#").alias("digits_masked"),
).show(5, truncate=False)

• concat_ws(sep, a, b, …) joins columns with a separator (and skips nulls) — cleaner than chaining +.
• trim (also ltrim/rtrim) strips whitespace — the fix for "why doesn't == 'US' match 'US '".
• regexp_replace(col, pattern, replacement) is your regex swiss army knife: mask digits, strip punctuation, normalize separators.

Your turn

Return each product's name in UPPERCASE (alias product_upper) plus its character length (alias name_len), from orders. Assign to result.

result = orders  # <- select F.upper(product) AS product_upper, F.length(product) AS name_len
result.show()

result = orders.select(
    F.upper("product").alias("product_upper"),
    F.length("product").alias("name_len"),
)