Exploring SQLite Integration into Nixpkgs-Multiverse for Efficient Data Handling

Current Data Handling in Nixpkgs-Multiverse

The core of Nixpkgs-multiverse relies on JSON files to index package versions and revisions. These files, such as versions.json (5.3 MiB) and history.json (7.5 MiB), contain data for over 300,000 package versions across thousands of packages and revisions. The Nix API loads these files using builtins.fromJSON.

Challenges with Large JSON Files

The primary issue with the current setup is that builtins.fromJSON is an eager parser. When any part of the JSON data is accessed, the entire file is parsed and materialized on the Nix heap. This means that looking up a single package version incurs the same parsing and memory allocation cost as loading the entire index, which is inefficient for large datasets.

The Need for Efficient Data Access

The project's goal is to minimize downloaded Nixpkgs, but replacing large Nixpkgs with large, inefficiently handled JSON files does not provide a clear benefit. The current approach necessitates careful data encoding to keep JSON files compact. For more complex query patterns and efficient data storage, a database solution is required.

Proposing SQLite as a Solution

SQLite is identified as a suitable technology to address these data handling inefficiencies. It offers efficient data encoding and declarative querying capabilities, which would allow for more targeted data access without parsing the entire dataset. Nixpkgs-multiverse already exports a SQLite database as a package for data exploration, indicating its potential for deeper integration.