38. Reading and writing JSON

JSON is how nested data usually travels. Spark reads JSON Lines by default — one JSON object per line — which is exactly our events file.

Reading nested JSON

spark.read.json(path) infers structs and arrays automatically — nested fields become struct columns you can dot into (Module 6):

events.write.mode("overwrite").json("events_out")

back = spark.read.json("events_out")
back.printSchema()
back.select("event_id", "user.id", "user.country").show(5)

The user struct and items array survive the round-trip — JSON keeps the shape that a flat CSV would have to flatten away.

Writing JSON

df.write.json(path) writes one JSON object per line, struct/array columns and all:

customers.write.mode("overwrite").json("customers_json")
spark.read.json("customers_json").show(3)

Ragged records and options

JSON is forgiving — records can be missing fields, and reading fills the gaps with null (some of our events have no payment; that column is just null for them). Useful options:

• multiLine=True — for files where one JSON object spans many lines (a pretty-printed array) instead of one-per-line.
• primitivesAsString, dropFieldIfAllNull — tune inference.

On a real cluster, malformed JSON lines are handled with mode="PERMISSIVE" (the default — bad records go to a _corrupt_record column) or "DROPMALFORMED". That error-handling machinery is a video-track / Colab topic; here every record is clean.

Your turn

Round-trip customers through JSON: write customers to a folder cust_json (overwrite), then read it back. Assign the read-back DataFrame to result.

result = customers  # <- write to "cust_json" (json, overwrite); read it back
result.show()

customers.write.mode("overwrite").json("cust_json")
result = spark.read.json("cust_json")