Wayfinder Router Enables Offline Prompt Routing for LLMs

Introduction to Wayfinder Router

Wayfinder Router allows users to determine whether to send a prompt to a local or cloud-based large language model (LLM) by analyzing its features in real-time. This system enhances efficiency by avoiding additional model calls, which typically introduce latency and extra costs.

How Wayfinder Works

The router assesses the prompt's structure, including its length and formatting, along with its wording attributes such as mathematical proofs and constraints. Based on these evaluations, it makes a routing decision in microseconds. Unlike typical routers that utilize a model for classification, Wayfinder performs this process offline, relying solely on its analysis to provide a recommendation.

Cost Optimization and Performance

By sending simpler prompts to less expensive local models and reserving more complex tasks for cloud-based options, Wayfinder significantly minimizes costs associated with LLM usage. It avoids high fees for routine tasks by accurately identifying when to use local resources.

Comparison with Existing Solutions

Wayfinder's approach is distinct from other routing systems such as OpenRouter or Bifrost, which focus on provider choice rather than prompt difficulty. This new router targets the cost-effectiveness of query processing by categorizing prompts into 'cheap' and 'expensive' tiers, allowing for enhanced financial management of model calls.

Evaluation and Effectiveness

Initial benchmarking indicates that Wayfinder is not focused on achieving the highest accuracy but aims to provide reliable and instantaneous routing decisions. Users have the option to opt-in for more nuanced lexical analyses, although preliminary tests show that structural assessments yield consistent results.