10. `orderBy`, `distinct`, `dropDuplicates`, `limit`

The last of the everyday verbs: sorting, de-duplicating, and taking a slice.

Sorting

orderBy (or sort — same thing) sorts by one or more columns. Ascending by default; use .desc() for descending:

orders.orderBy(F.col("unit_price").desc()).select("product", "unit_price").show(5)

Sort by several columns — each with its own direction:

orders.orderBy("category", F.col("unit_price").desc()).select("category", "product", "unit_price").show(8)

Distinct rows and duplicate rows

distinct() removes fully-duplicate rows. Our orders has a deliberately duplicated row (order 1021 appears twice) — watch the count drop by one:

print("all rows:", orders.count())
print("distinct:", orders.distinct().count())

dropDuplicates(["cols"]) is more surgical — dedup by specific columns, keeping one row per combination:

orders.dropDuplicates(["category"]).select("category", "product").show()

Taking a slice

limit(n) keeps the first n rows — pair it with orderBy for a top-N:

orders.orderBy(F.col("unit_price").desc()).limit(3).show()

Determinism. limit without orderBy gives you some rows, not a defined set — and on a cluster the "first" rows can change run to run. If you want the top N, always sort first. And when sort keys tie, add a tiebreaker column so the order is reproducible.

Your turn

Return the 5 most expensive distinct orders: drop duplicate rows, sort by unit_price descending, take 5, and select product and unit_price. Assign to result.

result = orders  # <- dropDuplicates, orderBy unit_price desc, limit 5, select product & unit_price
result.show()

result = (
    orders.dropDuplicates()
    .orderBy(F.col("unit_price").desc())
    .select("product", "unit_price")
    .limit(5)
)