Spotify Introduces Random Access Parquet (RAP) for Low-Latency Queries on Data Lake

Introducing Random Access Parquet (RAP)

Spotify has launched Random Access Parquet (RAP), a storage architecture designed to facilitate low-latency point queries directly against data stored in its data lake. This innovation allows online services and AI applications to access individual records without the need to duplicate datasets into separate operational databases.

Addressing Data Lake Challenges

Modern data lakes serve as central repositories for analytical and AI workloads, but retrieving individual records from them has been inefficient. Distributed query engines are optimized for analytical scans, not key-based lookups. While cloud object stores offer low latency, query planning and metadata traversal add overhead for point queries. Spotify manages petabytes of online data in Bigtable and exabytes in its Google Cloud Storage-based data lake, making large-scale data replication costly.

How RAP Works

RAP solves this by introducing an external indexing layer that maps lookup keys, such as user IDs, directly to Parquet files and specific row locations. Instead of scanning numerous files, a query uses the index to resolve the key and then performs a targeted ranged read against object storage. An index builder generates append-only index fragments for new data written into Apache Iceberg tables, without altering immutable Parquet files. This approach allows the same datasets to support analytical processing, machine learning pipelines, notebooks, AI agents, and latency-sensitive online applications without maintaining duplicate storage systems.

Industry Context and Comparison

Spotify's RAP aligns with broader industry efforts to extend open data lake technologies beyond analytical processing. Google Cloud recently described an Apache Iceberg-based lakehouse architecture for AI applications with similar goals of reducing data duplication and enabling operational access. However, RAP distinguishes itself by introducing a dedicated external indexing layer specifically optimized for point lookups, while maintaining compatibility with existing Parquet files and Iceberg tables.