BrevFeed
Sign in
← All courses

PySpark: Practical DataFrames

Learn PySpark by writing it - the DataFrame API end to end: select and filter, aggregation, joins, window functions, complex/nested data, Spark SQL, and reading/writing files. Every code box runs right in your browser (real pyspark.sql on DuckDB), with a graded exercise in every lesson.

intermediate Free pysparksparkdataframesbig-datapython
📖 44 readings ⚡ 44 exercises · 175 min total
Start course →
10 modules · 44 lessons
0. Getting Started 4 lessons · 19 min
  1. 1 What runs here — and how to read a lesson 5 min
  2. 2 SparkSession and your first DataFrame 5 min
  3. 3 Transformations, actions, and lazy evaluation 5 min
  4. 4 Schemas: inferred vs explicit 4 min
1. DataFrame Essentials — the core verbs 6 lessons · 25 min
  1. 5 `select` — choose and compute columns 4 min
  2. 6 Referring to columns: `col`, `expr`, `lit` 4 min
  3. 7 `filter` / `where` — keep the rows you want 4 min
  4. 8 `withColumn`, `withColumnRenamed`, `drop` 4 min
  5. 9 `cast` — changing a column's type 4 min
  6. 10 `orderBy`, `distinct`, `dropDuplicates`, `limit` 5 min
2. Aggregation & Grouping 4 lessons · 13 min
  1. 11 `groupBy` and `agg` — summarizing data 4 min
  2. 12 Multiple aggregates, aliasing, and filtering groups 3 min
  3. 13 Conditional aggregation and `pivot` 3 min
  4. 14 Aggregating into lists: `collect_list` and `collect_set` 3 min
3. Columns in the Real World — strings, dates, nulls, conditionals 4 lessons · 18 min
  1. 15 String functions 4 min
  2. 16 Date and time functions 4 min
  3. 17 Handling nulls 6 min
  4. 18 Conditional logic: `when` / `otherwise` 4 min
4. Joins 5 lessons · 20 min
  1. 19 Inner join 4 min
  2. 20 Left, right, and full outer joins 4 min
  3. 21 Semi and anti joins — filtering with a join 4 min
  4. 22 Cross joins, self-joins, and ambiguous columns 4 min
  5. 23 Diagnosing wrong row counts 4 min
5. Window Functions 5 lessons · 19 min
  1. 24 The window spec — `partitionBy` and `orderBy` 4 min
  2. 25 Ranking: `row_number`, `rank`, `dense_rank`, `ntile` 3 min
  3. 26 Analytic functions: `lag`, `lead`, `first`, `last` 4 min
  4. 27 Frames: running totals and moving averages 4 min
  5. 28 Practical window patterns 4 min
6. Complex & Nested Data 5 lessons · 24 min
  1. 29 Arrays: `explode`, `size`, `array_contains` 5 min
  2. 30 Higher-order functions on arrays 6 min
  3. 31 Structs — nested records 5 min
  4. 32 Maps — key/value columns 4 min
  5. 33 JSON in a column: `from_json` and `to_json` 4 min
7. Spark SQL — the other half of the API 3 lessons · 9 min
  1. 34 Temp views and `spark.sql` 3 min
  2. 35 SQL vs the DataFrame API — same engine 3 min
  3. 36 When to use which — and proof they're the same 3 min
8. Reading & Writing Data 4 lessons · 15 min
  1. 37 Reading and writing CSV 4 min
  2. 38 Reading and writing JSON 3 min
  3. 39 Parquet — the format you should default to 3 min
  4. 40 Schema on read — infer vs. declare 5 min
9. Capstone Projects — writing real PySpark 4 lessons · 13 min
  1. 41 Capstone 1 — clean a messy orders feed 3 min
  2. 42 Capstone 2 — customer analytics 3 min
  3. 43 Capstone 3 — flatten a nested clickstream 3 min
  4. 44 Capstone 4 — rewrite a SQL report as a DataFrame pipeline 4 min

0 Comments