← All stories
● Covered by 1 source · 1 reportLow impact1 neutral

DuckLake 1.1 Released, Stores Data Lake Metadata in SQL Databases

🔄 Updated 7d ago
New to BrevFeed? We gather this story from every outlet covering it into one summary — ranked by real-world impact, not just the latest headline — so you never miss what matters. What is BrevFeed? →

Key points

  • DuckLake 1.1 launched alongside DuckDB 2.0.0 LTS.
  • Stores table metadata in a standard SQL database.
  • Reduces file complexity in data lakes by consolidating metadata.
  • Addresses the 'small files problem' using inlining and flushing.

DuckLake 1.1 Release

DuckLake 1.1, an open-source table lakehouse format developed by DuckLabs, has been released. This version launched concurrently with DuckDB 2.0.0 LTS. The core innovation of DuckLake is its method of managing data lake metadata.

SQL-Based Metadata Management

Unlike traditional table formats that distribute manifest files across object storage, DuckLake stores table metadata within a standard SQL database. This design treats lakehouse metadata as a relational database problem, aiming to simplify data management. Pedro Holanda, lead DuckLake developer, notes that this approach leverages familiar SQL concepts for schema and data tracking.

Addressing Data Lake Complexity

The primary goal of DuckLake is to reduce the complexity often associated with data lakes, especially at AI scale. By centralizing metadata in a SQL database, it reduces the number of files that need to be managed and traversed during queries. This simplification allows users to query data more directly, as the catalog updates while the underlying Parquet files remain unchanged.

Solving the Small Files Problem

DuckLake 1.1 also addresses the 'small files problem' common in streaming data lakes and lakehouses. It uses techniques like inlining and flushing to consolidate numerous small files into larger ones, which can reduce the number of requests needed for data access and simplify maintenance tasks for data lake operators.

✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →

The daily brief

One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.

One email a day. Unsubscribe in one click, any time.

Today's brief

Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.

~4 min · 3 stories · Nov 21

▶ Play today's brief Listen on Spotify

New every morning, and the back catalogue is archived by date.

Primary sources

GitHub duckdb/duckdb

Reporting from

DuckLake 1.1 has been released, an open-source table lakehouse format that manages data lake metadata within a standard SQL database. This approach aims to simplify data lake operations by treating metadata as a relational database problem, reducing the complexity associated with scattered manifest files in object storage.