DuckDB 2.0 Alpha Introduces Asynchronous I/O for Faster S3 Parquet Reads

DuckDB 2.0 Alpha Release

The alpha version of DuckDB 2.0 has been released, with the full version expected in the fall. This update brings performance improvements, particularly for users building data tables and pipelines. The primary focus of the performance gains is on how data is handled, especially with external sources.

Asynchronous I/O for S3 Parquet Reads

A significant new feature in DuckDB 2.0 is asynchronous I/O. This change directly impacts queries that read large Parquet files from remote storage like S3. Previously, workers would sequentially download data, wait, then decode it, leading to idle CPU time during network waits. With async I/O, a dedicated pool of threads handles downloads, buffering data, while another pool of workers focuses solely on decoding, ensuring continuous CPU utilization.

Performance Improvements in Practice

Testing a query that reads a 2.2 GB Parquet file from S3 showed performance gains. The file, containing 228 million rows across 2268 row groups, had one column (230 MB) read and processed. The new asynchronous I/O mechanism keeps dozens of row groups in flight for download, ensuring that decoding workers always have data ready, thus making network and CPU operations concurrent.

Configuration and Impact

The asynchronous I/O feature is enabled by default, controlled by the `read_ahead_depth` setting. This setting determines how many row groups the download pool can fetch ahead of the workers. A default value of -1 enables automatic sizing based on thread count, while setting it to 0 reverts to the previous synchronous behavior. This enhancement means users can see speed improvements without altering their existing queries, provided their data access patterns benefit from concurrent I/O and processing.