BrevFeed
Sign in

4. Schemas: inferred vs explicit

📖 Reading · 4 min
💡 Every code box below is live — edit it and hit Run.

A DataFrame's schema is its list of columns and their types. So far we've let Spark infer it from the data. That's fine for quick work, but in a real pipeline you'll often spell the schema out yourself — and this lesson shows why.

Inferred: convenient, sometimes wrong

When you hand createDataFrame bare rows, Spark guesses the types by looking at the values:

films = spark.createDataFrame(
    [(1, "Chinatown", 1974, 8.1), (2, "Alien", 1979, 8.5)],
    ["id", "title", "year", "rating"],
)
films.printSchema()

id/year become long, rating becomes double. Good guesses here. But inference has real costs on actual data: to guess types when reading a file, Spark has to scan the data first (slow on big files), and a column that's numbers most of the time but text in one row can get inferred as a string — silently changing your results.

Explicit: you declare the shape

Define a StructType and hand it in instead of column names. Now the types are exactly what you say, no guessing:

from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DoubleType

schema = StructType([
    StructField("id", IntegerType()),
    StructField("title", StringType()),
    StructField("year", IntegerType()),
    StructField("rating", DoubleType()),
])

films = spark.createDataFrame(
    [(1, "Chinatown", 1974, 8.1), (2, "Alien", 1979, 8.5)],
    schema,
)
films.printSchema()
films.show()

Each StructField is (name, type, nullable=True). The common types: StringType, IntegerType, LongType, DoubleType, BooleanType, DateType, TimestampType.

When to use which

  • Inferred — exploring, tiny data, throwaway scripts.
  • Explicit — anything that runs more than once or reads files. It's faster (Spark skips the inference scan), it's a contract (a source that changes shape fails loudly instead of corrupting output), and it documents your data. In production, explicit wins.

You'll meet schemas again in Module 8 (reading files) and Module 6 (from_json needs one).

Your turn

Build a DataFrame from the rows below with an explicit schema where year is an IntegerType and rating a DoubleType, then return just the title and year as result.

from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DoubleType

# 1) define `schema` with columns: id (int), title (string), year (int), rating (double)
# 2) build `films` from these rows using that schema
# 3) result = title & year
rows = [(1, "Chinatown", 1974, 8.1), (2, "Alien", 1979, 8.5)]

result = None

from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DoubleType
schema = StructType([
    StructField("id", IntegerType()),
    StructField("title", StringType()),
    StructField("year", IntegerType()),
    StructField("rating", DoubleType()),
])
films = spark.createDataFrame(
    [(1, "Chinatown", 1974, 8.1), (2, "Alien", 1979, 8.5)],
    schema,
)
result = films.select("title", "year")
← Previous Next →