Coding Harnesses Struggle with Local LLM Performance on Laptops

The Challenge of Local LLMs

Developers attempting to switch from API calls to local large language models (LLMs) often encounter performance issues. While benchmarks like llama-bench might suggest high token generation rates, the actual development experience with local models on laptops is inconsistent and often slow. This discrepancy arises because most existing coding harnesses were not built with local model execution in mind.

Inefficiencies in Harness Design

Coding harnesses typically conspire against local execution in several ways. One major factor is the use of large system prompts and tool schemas. For example, a harness might have a system prompt and tool schema combination of 2,008 tokens for Qwen 3.8 27B, or up to 18,046 tokens for Opencode. On a laptop with a read speed of 90 tokens per second, this translates to prefill times of 22 to 226 seconds before the model even begins to generate output. This is in stark contrast to data center GPUs, which can prefill at 10,000+ tokens per second, making these delays negligible.

Impact on Context Window

Another issue is the inefficient use of the LLM's context window. After the system prompt is loaded, the remaining context window for actual work is significantly reduced. For instance, with a 32,000-token context budget, a harness using 18,046 tokens for its initial prompt leaves only 44% of the context available for development tasks. This limits the model's ability to process and generate relevant code effectively, further degrading the local development experience.