37. Reading and writing CSV

Real pipelines read files in and write files out. CSV is the messy, universal starting point. You've been using DataFrames read from CSV all course; now you'll do the reading and writing yourself.

Writing

df.write.<format>(path) writes a DataFrame out. Use mode to say what happens if the path exists:

orders.dropDuplicates().write.mode("overwrite").option("header", True).csv("my_orders")
print("written to my_orders/")

• mode("overwrite") replaces existing output; "append" adds to it; "error" (default) refuses; "ignore" skips.
• Spark writes a directory of part-files, not a single file — that's how it writes in parallel across a cluster. The path is a folder.

Reading

spark.read.<format>(path) reads it back. option("header", True) uses the first row as column names:

orders.dropDuplicates().write.mode("overwrite").option("header", True).csv("my_orders")

back = spark.read.option("header", True).csv("my_orders")
back.printSchema()
back.show(3)

Options you'll actually use: header, sep (delimiter — "\t", "|"), quote, escape, nullValue. Pass them via .option(k, v) or as keyword args: spark.read.csv(path, header=True, sep="|").

Spark's inferSchema. On a real cluster, CSV columns come in as all strings unless you pass inferSchema=True (which costs an extra scan). The in-browser engine here always sniffs types for you — so you see real types even without it. Either way, the next lessons show why you often want to pin types yourself.

Your turn

Round-trip orders: write deduplicated orders to a folder called orders_out as CSV with a header (overwrite mode), then read it back with header=True. Assign the read-back DataFrame to result.

result = orders  # <- write dropDuplicates() to "orders_out" (csv, header, overwrite); read it back
result.show()

orders.dropDuplicates().write.mode("overwrite").option("header", True).csv("orders_out")
result = spark.read.option("header", True).csv("orders_out")