Polars 2.0 Released with Out-of-Core Support, SQL Enhancements, and Performance Gains

Key Features in Polars 2.0

Polars 2.0 introduces several new features and improvements. A major addition is the initial version of out-of-core (spill-to-disk) support, allowing Polars to handle datasets larger than available RAM. The release also includes significant core performance enhancements across the engine.

Other notable features are first-class SQL support, a new Map data type, and stricter type enforcement for faster feedback and AI iteration.

SQL as a First-Class Citizen

With Polars 2.0, SQL is now treated as a first-class citizen, reflecting a dramatic increase in Polars' SQL coverage. This integration aims to enable Polars for more workloads, including those traditionally handled by SQL databases.

To support this, the release includes numerous improvements to the optimizer and engine, such as join reordering, enhanced common-subplan-elimination, and dynamic predicates/bloom filters.

Benchmark Performance Against Competitors

Polars SQL was benchmarked against DuckDB (versions 1.5.6 and 2.0 alpha) and DataFusion (version 54.0.0) using TPC-H and TPC-DS1 datasets. Tests were conducted on c7a.4xlarge and c7a.metal AWS instances.

The benchmarks showed Polars outperforming both DataFusion and DuckDB in query execution times, indicating its competitive performance in SQL workloads. Polars and DuckDB completed all queries, while DataFusion timed out on some.