7. `filter` / `where` — keep the rows you want

filter and where are the same method with two names — use whichever reads better. Both keep the rows where a condition is true.

orders.filter(F.col("category") == "electronics").show()

Two rules that cause 90% of filter bugs

1. Use & | ~, never and or not. Python's keywords can't be overloaded for columns; using them raises the baffling "Cannot convert column into bool" error.

2. Parenthesize every clause. In Python & binds tighter than ==, so col("a") == 1 & col("b") == 2 parses wrongly. Wrap each comparison:

orders.where(
    (F.col("category") == "electronics") & (F.col("quantity") >= 2)
).show()

& = and, | = or, ~ = not. Combine freely — just keep the parentheses:

orders.where(
    (F.col("category") == "electronics") | (F.col("unit_price") > 100)
).show()

String conditions

You can also pass a SQL string. Handy for quick filters, though less composable than column expressions:

orders.filter("unit_price > 100").show()

Nulls and filter — a trap for later. A row where the condition is null (e.g. quantity >= 2 when quantity is missing) is dropped — and it's dropped by the negated condition too. Both filter(cond) and filter(~cond) can lose the same row. Module 3 handles nulls head on.

Your turn

From orders, keep only electronics orders with a quantity of 2 or more. Assign to result.

result = orders  # <- category == "electronics" AND quantity >= 2
result.show()

result = orders.where(
    (F.col("category") == "electronics") & (F.col("quantity") >= 2)
)