BrevFeed
Sign in

40. Schema on read — infer vs. declare

📖 Reading · 5 min
💡 Most code boxes below are live — edit one and hit Run. Boxes without a Run button are reference-only (they can't run in your browser).

When you read text (CSV/JSON), Spark has to decide each column's type. There are two ways, and the choice matters more than it looks.

Inference

Letting the reader guess is convenient. But inference has costs: it scans the data first (slow on big files), and a column that's numeric most of the time but text in one stray row gets guessed as a string — silently changing every downstream calculation.

orders.dropDuplicates().write.mode("overwrite").option("header", True).csv("s_in")
inferred = spark.read.option("header", True).csv("s_in")
inferred.printSchema()

Declaring types yourself

The production habit is to not guess — state the types you expect. The portable way that runs everywhere is read-then-cast: read the raw columns, then cast each to its intended type. Now the types are a contract, not a guess:

raw = spark.read.option("header", True).csv("s_in")
typed = (
    raw
    .withColumn("quantity", F.col("quantity").cast("int"))
    .withColumn("unit_price", F.col("unit_price").cast("double"))
    .withColumn("order_id", F.col("order_id").cast("long"))
)
typed.printSchema()

Spark's one-liner: spark.read.schema(...). On a real cluster you hand the reader a StructType up front — spark.read.schema(my_schema).csv(path) — and it skips inference entirely, applying your types as it reads. The in-browser engine here doesn't expose .schema() on the reader, so we cast after reading (same result). The notebook below runs the .schema() form on real Spark.

# colab: 05-data-sources-io/00-file-formats-and-options
from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DoubleType

ORDERS_SCHEMA = StructType([
    StructField("order_id", IntegerType()),
    StructField("customer_id", StringType()),
    StructField("order_date", StringType()),
    StructField("product", StringType()),
    StructField("category", StringType()),
    StructField("quantity", IntegerType()),
    StructField("unit_price", DoubleType()),
    StructField("country", StringType()),
])

# No inference pass at all: Spark applies these types as it reads.
orders = spark.read.schema(ORDERS_SCHEMA).csv(f"{DATA_DIR}/orders.csv", header=True)
orders.printSchema()

And this is why Parquet is nice

All of this — inference cost, wrong guesses, casting — is a text-format problem. Parquet stores the schema in the file, so there's nothing to infer or declare. Read messy text once, cast carefully, write Parquet, and every read after that is typed and fast.

Your turn

Read the CSV back and pin the types: cast quantity to int and unit_price to double, then return order_id, quantity, unit_price. Assign to result.

orders.dropDuplicates().write.mode("overwrite").option("header", True).csv("sor")

raw = spark.read.option("header", True).csv("sor")
result = raw  # <- cast quantity->int, unit_price->double; select order_id, quantity, unit_price
result.show()

raw = spark.read.option("header", True).csv("sor")
result = (
    raw
    .withColumn("quantity", F.col("quantity").cast("int"))
    .withColumn("unit_price", F.col("unit_price").cast("double"))
    .select("order_id", "quantity", "unit_price")
)
← Previous Next →