Developer builds a personal search engine for 500,000 domains in a weekend for $10

Project Conception and Goal

A developer, frustrated with the prevalence of corporate documentation and SEO-optimized content in mainstream search results, decided to build a personal search engine. The goal was to create an index specifically for portfolios, zines, art projects, and independent software, avoiding the 'SEO sludge' of the broader web.

Rapid Development and Cost

The project began on a Sunday and by Wednesday lunch, the developer had cataloged 560,183 homepages. The total cost for this endeavor was approximately $10, which included an overnight GPU rental. The process involved a few hours of active steering while the system ran autonomously.

Technical Approach

The search engine was designed for a single user and did not include features like IP scanning, Redis, full page HTML storage, or multi-tenancy. It utilized a crawler that only fetched homepages, and a small local language model to read each page, generating a name, a brief description, a category, and tags. The system stored metadata for each domain, with page text being discarded after processing by the model.

System Architecture

The setup involved four processes: three running on the developer's personal computer and one on a rented GPU. The GPU was used for language model processing and did not directly interact with the database. Fetched data was temporarily stored in the domain's database row until the model completed its analysis, after which the text was wiped.

Implications for Personal Search

This project highlights the feasibility of creating highly customized, small-scale search indexes. It demonstrates that individuals can build effective tools to curate their own web experience, focusing on specific types of content and filtering out unwanted noise, with minimal financial and time investment.