Benchmarking DuckDB vs. SQLite for Observability Data on Low-Cost Hardware

Performance Comparison on Identical Hardware

A benchmark was conducted to compare the performance of DuckDB and SQLite as embedded databases for observability data. The tests were run on a $16.49/month Hetzner CCX13 server using the Traceway binary, with each database implemented as a storage engine.

The results showed that DuckDB achieved write throughput 3x to 15x higher than SQLite. For dashboard queries, DuckDB was able to handle 100x more data points at equal or better latency compared to SQLite.

Data Ingestion and Storage Efficiency

DuckDB demonstrated its capability to ingest a billion metric points in under an hour, occupying 10.8 GB of disk space. This scale was not achievable by SQLite in previous benchmarks.

The benchmark also highlighted DuckDB's particular strength with log data, which was previously recommended to be kept off SQLite entirely.

Methodology and Context

The comparison aimed to evaluate how far an embedded database could go before needing a client-server OLAP solution like ClickHouse. The methodology involved implementing DuckDB as a valid storage engine for Traceway and running its existing benchmarking suite.

This direct comparison addresses a gap in published numbers for these two embedded database options when self-hosting on a single, inexpensive server.

Implications for Self-Hosting

The findings suggest that DuckDB enables self-hosting a full OpenTelemetry (OTel) stack capable of handling large data volumes on a relatively small server. This provides a viable option for users seeking to manage telemetry data without the overhead of a separate client-server OLAP database.