BrevFeed
Sign in
← All blogs

Embedded LevelDB Cache in Go: Billions of Records on One Small EC2

BrevFeed Jul 17, 2026 26 min read 6 reads
GoLevelDBCachingDatabasesPerformance

Our data pipeline had a habit of asking the same question a few billion times. It ingests scholarly article metadata from several sources — OpenAlex, Crossref, sometimes Semantic Scholar — and nearly every stage of that ETL keeps needing the answer to one small question: what do we already know about this article? Given an identifier, fetch the record. That's the whole query. An exact-key point lookup returning a small blob of metadata — no joins, no ranges, no transactions.

The catch is the scale of the thing being looked into. The OpenAlex snapshot alone is around 150 GB compressed (billions of rows once you unpack it), and the other sources pile on top. Meanwhile the pipeline asks its little question constantly: matching incoming records against ones we've already seen, enriching a record from one source with fields from another, deduplicating across sources. A single processing run means hundreds of millions of lookups, in effectively random key order, because article identifiers have no ordering that matches how records arrive.

And the constraint that makes it interesting: this all had to run on a small EC2 box. Not a cluster, not a managed cache fleet. One modest instance, because this is internal plumbing, not a user-facing product, and internal plumbing has a budget.

This post is about how we solved it: a key-value cache built in Go on an embedded store called LevelDB, holding billions of records on that one small machine, serving lookups to the rest of the pipeline. I'll walk through why the obvious options lost, and then build the whole service step by step.

The options we burned through first

Option A — a table and an index in the database we already had. Postgres, one table, identifier as PRIMARY KEY. This is the correct default, and it's where we started. Postgres remained an excellent general-purpose database throughout this story; nothing below changes that. But for our workload — billions of random-key inserts and lookups, small values, a modest EC2 instance — the B-tree index eventually became the dominant bottleneck. The initial load slowed to a crawl once the index outgrew RAM, and every lookup carried a network round trip, query parsing, and usually a random disk read. The next section explains the mechanics; the short version is that this particular combination of random keys, huge row count, and small machine is the B-tree's worst case, and we were living in it.

Option B — Redis / ElastiCache. Wonderful latency, and the default answer for "we need a cache." But Redis holds everything in RAM, and our dataset is a couple hundred gigabytes. RAM at that size means a very large node or a cluster; on a managed tier, thousands of dollars a month. For an internal cache feeding ETL jobs, that pricing is absurd. Redis wasn't wrong technically; it was priced out of the problem.

Option C — an embedded store. The cache lives inside a process, backed by files on cheap disk, and a lookup is a local read, not a network round trip. The one real constraint: LevelDB locks its files to a single process, so no second program can open the same store. That sounds limiting until you notice it isn't. Put a thin REST API in front of that one process and it becomes a network cache service that any number of consumers (ETL stages, Python scripts, services on other boxes in the VPC) can hit concurrently. One process owns the files; everything else talks to that process. That's what we build below, and it's why Option C stops looking exotic and starts looking obvious: disk is cheap enough to hold everything, and local reads are fast enough to serve everything.

The whole architecture fits in one picture:

   ETL stages        scripts (any language)     jobs on other boxes
        │                     │                          │
        └──────────── HTTP GET/PUT /cache/{id} ──────────┘
                              │
              ┌───────────────▼───────────────┐
              │  Go cache service (goleveldb) │  one process owns the
              │  RSS: tens of MB              │  files; serves everyone
              ├───────────────────────────────┤
              │  OS page cache                │  the rest of the 8 GB —
              │                               │  hot keys served from RAM
              ├───────────────────────────────┤
              │  /data on a gp3 EBS volume    │  billions of rows on
              │  (LevelDB sorted files + WAL) │  cheap disk
              └───────────────────────────────┘
                     one small EC2 instance

Why a B-tree index struggled with this workload

This is worth understanding, because it's the difference between "Postgres was slow for us, shrug" and knowing whether — and when — the same thing would happen to your workload.

Postgres and MySQL both index data with a B-tree by default. A B-tree keeps keys sorted in a tree of fixed-size pages, and it's genuinely excellent for the workload it was designed for: mixed reads and writes, range scans, transactional guarantees, at row counts from thousands into the hundreds of millions.

The trouble starts when writes arrive in random key order at very large scale — which is exactly what loading article identifiers looks like, since they have no natural ordering. Every random-order insert can touch a B-tree page that isn't already in memory, forcing a random disk read before the write can even happen, and occasionally splits a page, rippling changes upward through the tree. At a few million rows this is noise. At a few billion rows, with an index that no longer fits in RAM, it's the dominant cost — and Postgres additionally carries per-row overhead (the tuple header alone is 23+ bytes) that can rival the size of a small metadata record, plus VACUUM has to keep revisiting the whole table as records get updated.

LevelDB uses a different structure, an LSM-tree (log-structured merge-tree), built for exactly this shape of problem. Writes always go to an in-memory table plus an append-only log, which is a sequential disk write regardless of how random the keys are. Once that in-memory table fills up, it's flushed to disk as a sorted, immutable file. A background process periodically merges (compacts) those files together. The result: writes are cheap and sequential no matter what order the keys arrive in, and reads check a small number of sorted files instead of walking a tree that's mostly not in memory. This isn't a niche trick — it's why systems that manage genuinely huge, randomly-keyed datasets (Bitcoin Core's block index, go-ethereum's state data) reach for LevelDB or its descendants instead of a relational database.

The two write paths, side by side:

B-tree, random-key insert                 LSM-tree, any insert
─────────────────────────                 ────────────────────
find the target page        ┐             append to write-ahead log   (sequential disk)
  often a random disk read  ┘             insert into memtable        (RAM)
update the page in place    → random I/O        │
sometimes split the page    → more I/O          │ memtable full
  ...ripples up the tree                        ▼
                                          flush one sorted file       (sequential disk)
                                          background compaction merges files

To be fair to both sides: the LSM-tree is a trade, not a free lunch. What it gives up is on the read path — a lookup may have to check the memtable and then several on-disk files across levels before finding the key or concluding it's absent (read amplification), and the background compaction that keeps the file count low competes with your reads for disk I/O while it runs. Overwritten and deleted values also occupy space until compaction reclaims them. For a workload that's mostly reads over stable data those costs are modest — and you'll see exactly what they look like in the tail latencies of the benchmark below — but they're real, and a B-tree that fits in RAM doesn't pay them.

Step 1 — a Go HTTP server that does nothing yet

Before touching LevelDB, get a web server running, because if this doesn't work, nothing after it will either. The HTTP layer is what turns a single-process embedded store into a shared service: LevelDB only lets one process open the files, but one Go process comfortably serves thousands of concurrent HTTP requests, so every consumer — pipeline stages, Python scripts, jobs on other machines — goes through this API instead of the files.

// main.go
package main

import (
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/ping", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("pong"))
    })

    log.Println("listening on :8080")
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Run it with go run main.go, hit curl localhost:8080/ping, confirm you get pong. Everything from here is additive to this same file.

Step 2 — open a LevelDB store

Pull in the library:

go mod init cache-service
go get github.com/syndtr/goleveldb/leveldb

goleveldb is a pure-Go port — no C dependencies to compile, which keeps the build simple on a plain EC2 box with no extra toolchain setup. It's worth knowing the family tree here, because everything in this post transfers: RocksDB is Facebook's heavily-tuned fork of LevelDB (C++, more knobs, and features like read-only secondary instances), and Pebble is the pure-Go LSM engine that powers CockroachDB. Same architecture, same API shape — if you outgrow goleveldb, the code changes are small and the mental model doesn't change at all. Open a database at startup:

package main

import (
    "log"
    "net/http"

    "github.com/syndtr/goleveldb/leveldb"
)

var db *leveldb.DB

func main() {
    var err error
    db, err = leveldb.OpenFile("/data/cache", nil)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    http.HandleFunc("/ping", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("pong"))
    })

    log.Println("listening on :8080")
    log.Fatal(http.ListenAndServe(":8080", nil))
}

/data/cache will live on a dedicated EBS volume, not the EC2 instance's root disk — more on that in Step 5.

If you're reading closely: defer db.Close() never actually runs, because log.Fatal calls os.Exit, which skips deferred functions. I left it that way on purpose, because the interesting question is about crash safety anyway. What happens when this process dies without a clean shutdown? The answer is in the LSM design. Every write goes to the append-only log before it's acknowledged, and that log is a write-ahead log in the full database sense — on the next OpenFile, LevelDB replays whatever hadn't made it into sorted files yet. A kill -9, an OOM, a kernel panic mid-write: the store comes back consistent. Clean shutdown is still polite (it saves the replay), but durability doesn't depend on it.

Step 3 — read and write keys

The API is one route: GET /cache/{id} returns the stored record, PUT /cache/{id} stores one. Keys are article identifiers; values are the metadata JSON exactly as we want it back — the cache doesn't parse or validate it, it just stores bytes.

func handleCache(w http.ResponseWriter, r *http.Request) {
    key := strings.TrimPrefix(r.URL.Path, "/cache/")
    if key == "" {
        http.Error(w, "missing key", http.StatusBadRequest)
        return
    }

    switch r.Method {
    case http.MethodGet:
        val, err := db.Get([]byte(key), nil)
        if err == leveldb.ErrNotFound {
            http.Error(w, "miss", http.StatusNotFound)
            return
        }
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }
        w.Write(val)

    case http.MethodPut:
        body, err := io.ReadAll(r.Body)
        if err != nil {
            http.Error(w, err.Error(), http.StatusBadRequest)
            return
        }
        if err := db.Put([]byte(key), body, nil); err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }
        w.WriteHeader(http.StatusNoContent)

    default:
        http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
    }
}

leveldb.ErrNotFound is a specific sentinel error — that's a cache miss, worth handling separately from a real error like disk failure, because to the pipeline a miss is an answer ("we haven't seen this article"), not a failure. Wire it up with http.HandleFunc("/cache/", handleCache), and the loop works end to end:

curl -X PUT localhost:8080/cache/W2741809807 \
  -d '{"title":"...","year":2018,"venue":"...","cited_by":412}'
curl localhost:8080/cache/W2741809807

Step 4 — one store, several sources

The same article shows up in OpenAlex, Crossref, and Semantic Scholar under different identifier schemes, and the pipeline needs to look it up by whichever one it has in hand. Rather than running one store per source, prefix the keys:

oa:W2741809807   → {…}
cr:184827172     → {…}
s2:0d5efbc0a5f1  → {…}

This costs three bytes per key and buys two things. First, one process, one volume, one service to operate, no matter how many sources join later. Second — because LevelDB keeps keys sorted — all of one source's keys sit physically together, so you can iterate exactly one namespace when you need to (rebuilding one source's slice of the cache, counting its records) without touching the others:

iter := db.NewIterator(util.BytesPrefix([]byte("cr:")), nil)
for iter.Next() {
    // only Crossref-namespaced records
}
iter.Release()

A hash-based store can't give you that; the sorted layout is one of the quiet perks of an LSM-tree.

Step 5 — an EC2 box sized for billions of rows, not thousands

Two things need headroom here: RAM for LevelDB's in-memory tables and OS page cache, and disk — LSM-trees temporarily use extra space during compaction (old and newly-merged files coexist briefly), so budget roughly 1.5–2x your steady-state data size.

# on the instance, after attaching the EBS volume
sudo mkfs -t ext4 /dev/xvdf
sudo mkdir -p /data
sudo mount /dev/xvdf /data
sudo chown $USER /data

Install Go, pull the code, build it:

wget https://go.dev/dl/go1.22.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.22.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin

git clone <your-repo> cache-service && cd cache-service
go build -o cache-service .

Step 6 — run it as a real service

A go run in a terminal dies the moment you disconnect. Give it a systemd unit so it survives reboots and reconnects:

# /etc/systemd/system/cache-service.service
[Unit]
Description=LevelDB cache service
After=network.target

[Service]
ExecStart=/home/ec2-user/cache-service/cache-service
Restart=on-failure
User=ec2-user

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now cache-service
curl localhost:8080/ping

Step 7 — loading billions of records from the snapshots

The initial load is its own problem. Don't do it by calling the HTTP PUT endpoint a few billion times — per-request overhead would stretch the load into weeks. Write a separate one-off loader that streams the compressed snapshot files and talks to the LevelDB library directly, using batch writes, which group thousands of puts into one disk sync instead of one sync per record:

// loader/main.go — one-off program, not part of the running service
batch := new(leveldb.Batch)
for rec := range records { // streamed straight from the snapshot files
    batch.Put([]byte("oa:"+rec.ID), rec.MetaJSON)
    if batch.Len() >= 5000 {
        if err := db.Write(batch, nil); err != nil {
            log.Fatal(err)
        }
        batch.Reset()
    }
}

Because LSM writes are sequential no matter how random the keys are, the load rate holds steady from the first million rows to the last billion — background compaction makes it oscillate, but it never falls off a cliff the way a B-tree load does when the index outgrows RAM. A full snapshot load is an overnight job on this hardware, not a week-long one. When a new snapshot drops, the same loader re-runs: puts overwrite in place, no DELETE-then-INSERT, no vacuum debt.

What it looks like in production

This is the part that still surprises people, so here's the honest picture of a couple-hundred-GB store running on an 8 GB machine:

What this doesn't give you

The design above works because the problem has a very specific shape, so it's worth being explicit about what we gave up:

Embedded LevelDB (this post) Postgres / MySQL (B-tree index) Redis / ElastiCache
Underlying structure LSM-tree — sequential writes, sorted immutable files, background compaction B-tree — in-place page updates, random I/O under random-key writes In-memory hash table
Behavior at billions of small, randomly-keyed rows Designed for this — write cost stays flat regardless of key order Degrades — random-order inserts cause page splits and random disk reads once the index outgrows RAM Fast, but the whole dataset must fit in RAM (or you pay for a very large node)
Lookup cost Local function call + disk/page-cache read Query parse/plan + index traversal + network round trip Network round trip (~sub-ms in-VPC)
Shared across consumers Yes, through its REST API — many clients, but one process owns the files Yes Yes
Transactions, joins, range queries No (prefix scans only) Yes — this is what you're paying the overhead for No
Ops burden A systemd unit Full RDBMS operations: vacuum, backups, replication, connection limits Patching/failover, or AWS's problem (ElastiCache) — but you pay for it
Best fit Huge row counts, point lookups, rebuild-from-source data, one box's worth of throughput You need transactions, joins, or range queries — not just lookups by key Shared, evictable, RAM-sized hot working sets

The real ceiling isn't consumer count — the REST API happily serves many clients — it's that all of them share one box's throughput and one process's lock on the files. When you hit that ceiling, you have escalation paths that keep the same architecture: replicate the whole store onto more boxes behind a load balancer (cheap, and safe precisely because it's derived data), or step up the engine — RocksDB adds read-only secondary instances that let extra processes read the same store, and Pebble gives you the same LSM design in pure Go with a more actively developed core. And one caveat that isn't about scale: if the data weren't rebuildable from snapshots, "just re-run the loader" would stop being a recovery plan, and this design would need real backups.

The numbers: a three-way benchmark

"Postgres was too slow" and "Redis was priced out" are the kind of claims that deserve receipts, so I rebuilt the comparison as a controlled benchmark. It doesn't use the real article data — that would be awkward to redistribute and impossible for you to reproduce — but a synthetic dataset with exactly the same shape: 2 billion rows of short random key → ~70-byte value, ≈150 GB raw, with keys generated by a deterministic bijection so every backend loads identical data in identical (pseudo-random) order, and anyone can regenerate it from the code.

One logistical confession: the tutorial above deploys to EC2, but the benchmark itself ran on Google Cloud — an e2-standard-2 VM, the same 2 vCPU / 8 GB spec as the t3.large, with a 400 GB balanced persistent disk at /data, plus Cloud SQL for the Postgres phase. Same CPU count, same RAM ceiling, comparable disk class; nothing in the comparison hinges on the vendor logo. The setup:

Total cloud bill for the whole exercise, all three phases: about six dollars.

Round 1 — the write path: LevelDB swallowed 2 billion rows; Postgres didn't finish

The LevelDB loader ingested all 2 billion rows in 9h 45m — roughly 57k rows/s average. The rate oscillated between 30k and 110k rows/s as background compaction waves came and went, but it never collapsed. That's the LSM promise from earlier, kept: write cost stayed flat no matter how deep into the load we got.

Postgres told a very different story. COPY started at a healthy 95k rows/s at the 10-million-row mark — faster than LevelDB, in fact. Then the primary-key index outgrew RAM, and the decay set in:

Rows loaded Postgres load rate
10M 95,000 rows/s
100M 34,600 rows/s
180M 5,200 rows/s
260M 2,243 rows/s

That's a 97.7% collapse before reaching 13% of the dataset. After 11 hours Postgres had 260M rows loaded, and extrapolating the curve put completion at over a week, so I stopped it there. This is the failure mode from the B-tree section playing out on schedule: every random-key insert forcing a random read of an index page that's no longer in memory. In the time Postgres managed 260 million rows, LevelDB ingested all 2 billion. To be precise about what that proves — not that Postgres is slow, but that random-key bulk inserts far past RAM, on a deliberately small machine, are exactly the regime B-trees handle worst. More RAM moves the cliff further out; it doesn't remove it. For a pipeline that reloads from fresh snapshots, this round alone is decisive. One backend makes that an overnight job on this hardware; the other makes it a week-long one.

Round 2 — reads, where Postgres gets its revenge (sort of)

Here's the part a less honest benchmark would bury: Postgres won the raw read-throughput number.

Against its 260M-row table, Postgres served ~5,700 lookups/s (p50 ≈ 10.4 ms, p99 ≈ 26 ms). The LevelDB service, against the full 2-billion-row store, sustained ~3,800 lookups/s (p50 ≈ 11.8 ms, p99 ≈ 79 ms). But look at what each was being asked to do: Postgres's 7.9 GB primary-key index approximately fits in an 8 GB machine's memory, so most lookups never touched disk. LevelDB was serving a 161 GB store from 8 GB of RAM — a dataset 20x memory — and the bottleneck was simply the disk's IOPS budget, not CPU (the Go process itself used 75 MB under load; the rest of RAM was OS page cache).

So I ran the fair rematch: a fresh LevelDB store with the same 260M rows Postgres had.

260M rows each LevelDB Postgres
Bulk load time 45 minutes ~11 hours (14.6x slower)
On-disk size 21 GB (80.8 B/row) 34 GB (131 B/row)
Lookups/s (median) ~6,000 ~5,700
p50 latency 8.1 ms 10.4 ms
p95 / p99 26.6 / 40.5 ms 18 / 26 ms
Worst request seen 134 ms ~1.4 s outliers, every run

At equal size, reads are roughly a wash — LevelDB takes the median and the worst-case, Postgres has the tighter p95/p99. That tail gap is the read amplification promised in the theory section, showing up on schedule: an unlucky LevelDB lookup checks several files across levels, and compaction I/O occasionally competes with reads, so LevelDB's p95/p99 stretches wider even while its median wins. The decisive gaps aren't in reads at all: they're the 14.6x write throughput and the 1.6x storage overhead.

That storage number is worth unpacking, because it's the tuple-header-and-MVCC-bookkeeping cost from earlier, now measured instead of asserted. Postgres's 131 bytes/row decomposes roughly as: ~77 bytes of actual key + value, a 23-byte tuple header plus a 4-byte item pointer on every heap row, page-level overhead and fill factor spread across rows — and then a second copy of every key in the primary-key B-tree, with its own per-entry and per-page overhead (the index alone was 7.9 GB, ~30 bytes/row). LevelDB stores the key and value once, contiguously inside a sorted block, plus a few bytes of framing: 80.8 bytes/row for the same ~77-byte records. For small records, Postgres's bookkeeping costs about as much as the data; LevelDB's costs a rounding error.

Round 3 — Redis is spectacular, for the 2.5% of the data that fits

Redis ran on literally the same VM as the LevelDB service (Round 1's hardware, reused), capped at 7 GB of memory with eviction off. It fit 50.3 million rows before hitting the ceiling — 2.5% of the dataset — at ~149 bytes/row of in-memory overhead, nearly twice LevelDB's on-disk footprint.

For the slice that fit, it was untouchable: ~58–64k lookups/s at p50 ≈ 1.05 ms — 10–15x LevelDB's throughput at a tenth of the latency, and that ~1 ms is mostly just the in-zone network round trip. This is why Redis is the default answer for hot working sets.

But scaling it to the full dataset means buying ~20x the RAM: a ~176 GB-RAM machine, or a managed-cache tier where 150 GB runs to thousands of dollars a month (the quoted managed rate put it north of $5,000/mo; check current pricing, but the order of magnitude is the point). The LevelDB box serving all 2 billion rows cost ~$0.12/hour — about $90/month. That's the pricing math that ruled Redis out of our pipeline.

Scoreboard

LevelDB (2B rows) Postgres (stopped at 260M) Redis (50M — all that fit)
Ingest 2B rows in 9h 45m, rate never collapsed 97.7% rate collapse by 13% of dataset; projected >1 week ~250k rows/s — until RAM ran out
Bytes per row 80.5 on disk 131 on disk ~149 in RAM
Lookups/s ~3,800 (dataset 20x RAM) · ~6,000 at 260M ~5,700 at 260M ~60,000
p50 / p99 11.8 / 79 ms (2B) · 8.1 / 40.5 ms (260M) 10.4 / 26 ms 1.05 / 1.8 ms
Monthly cost at full 2B-row scale ~$90 (VM + disk) couldn't load it in reasonable time at this size ~$5,000+ managed, or a 176 GB-RAM box

The same story as two charts:

lookups/sec, warmed steady state (64 clients)
Redis     (50M rows — all that fit in RAM)  ██████████████████████████████  ~60,000
LevelDB   (260M rows)                       ███                             ~6,000
Postgres  (260M rows)                       ██▉                             ~5,700
LevelDB   (2B rows — dataset 20x RAM)       █▉                              ~3,800

monthly cost to serve the FULL 2B-row dataset
Redis (managed, ~150+ GB of RAM)            ██████████████████████████████  ~$5,000+
LevelDB (small VM + 400 GB disk)            ▏                               ~$90

Redis's throughput lead is real, but it applies to 2.5% of the data. LevelDB's cost lead applies to all of it.

Caveats, so you can beat these numbers

A few things I deliberately left on the table, any of which would make LevelDB look better — flagged so the comparison stays conservative:

The takeaway

For the row counts most applications have, a database table with an index is the right cache, and building this would be over-engineering. The benchmark even showed Postgres reading faster than the 2-billion-row LevelDB store, and matching it at equal size — if your index fits in RAM and you mostly read, Postgres is genuinely fine.

What the numbers refuse to negotiate on is the write path and the footprint. The embedded-store move earns its keep at a specific intersection: billions of rows, small values, random key order, point lookups only, and a budget measured in one small server. Our ETL sat exactly on that intersection. Postgres didn't lose the benchmark there — it failed to finish loading it, with the ingest rate down 97.7% before 13% of the data was in. Redis was the right answer for the 2.5% that fit in RAM and the wrong bill for the rest. An LSM-tree on cheap disk, wrapped in 150 lines of Go, turned "billions of records" into a problem one modest box handles without noticing — for about ninety dollars a month.

The lesson I'd carry to other problems: none of these systems is "the fast one" or "the slow one." Each won the round its data structure was built for. Start from the workload — key order, value size, read/write mix, what has to fit in RAM, who consumes it — and let that pick the storage engine, not the other way around.

0 Comments