Homebench Tool Benchmarks Local LLMs for Speed, Memory, and Quality

Introduction to Homebench

Homebench is a new single-command terminal user interface (TUI) designed to benchmark local large language models (LLMs). It evaluates models based on speed (tokens/sec, time-to-first-token), memory footprint, and a curated quality suite. The tool presents results in a live comparison leaderboard directly in the terminal.

Functionality and Supported Runners

The tool automatically discovers LLMs installed in local runners such as Ollama, LM Studio, llama.cpp, vLLM, or any OpenAI-compatible server. It offers zero-configuration setup, requiring no API keys or cloud services. Users can install it via pip and run it with a simple command.

Addressing a Benchmarking Gap

Existing tools often focus on either speed or quality, but not both in a local-first, user-friendly manner. For example, llama-bench measures speed only, while lm-evaluation-harness focuses on quality but lacks a polished laptop user experience for common local model runners. Homebench aims to fill this gap by providing a unified solution for local LLM evaluation.

Usage and Customization

Homebench offers various command-line options for customization. Users can benchmark all discovered models, run a full quality suite, specify particular models, or limit the number of models tested. It also supports different providers, exporting results to Markdown or JSON, and using custom task packs for quality evaluation.