39. Parquet — the format you should default to

CSV and JSON are for exchanging data with the outside world. For data that lives inside your pipeline, the right default is Parquet.

orders.dropDuplicates().write.mode("overwrite").parquet("orders_pq")

back = spark.read.parquet("orders_pq")
back.printSchema()
back.show(3)

Notice the schema came back exactly — order_id as bigint, order_date as date, unit_price as double — with no options and no inference. That's the whole point.

Why Parquet wins

• Columnar. Values are stored by column, not by row. A query that reads 2 of 20 columns only touches those 2 — huge I/O savings (column pruning).
• Self-describing. The schema and types are stored in the file. No header, no inferSchema, no surprises — dates stay dates.
• Compressed. Columnar data compresses far better than text; files are a fraction of the CSV size.
• Pushdown-friendly. Engines can skip whole chunks using built-in min/max stats, so filters read less data.

CSV is rows of text with no types; Parquet is typed, compressed columns. For anything you'll read back more than once, Parquet is the answer.

Rule of thumb

• Data crossing a boundary (received from / handed to another system, humans): CSV / JSON.
• Data staying in your pipeline (intermediate tables, outputs you'll re-read): Parquet (or a Parquet-based table format like Delta — video track).

Your turn

Round-trip orders through Parquet: write deduplicated orders to a folder orders_parquet (overwrite), then read it back. Assign the read-back DataFrame to result.

result = orders  # <- write dropDuplicates() to "orders_parquet" (parquet, overwrite); read it back
result.show()

orders.dropDuplicates().write.mode("overwrite").parquet("orders_parquet")
result = spark.read.parquet("orders_parquet")