● Covered by 1 source · 1 reportMedium impact1 neutral
Microsoft.Extensions.AI introduces new primitives for AI model routing and failover
Key points
- New primitives handle routing and failover for AI models.
- Four experimental types: RoutingChatClient, SemanticRoutingChatClient, FailoverChatClient, OrderedFailoverChatClient.
- Routing can be based on message content using embedding similarity.
- Failover strategies include retrying with different clients upon failure.
New AI Routing and Failover Capabilities
Microsoft.Extensions.AI has introduced new experimental primitives designed to manage routing and failover for AI models and providers. These additions aim to improve cost efficiency, application uptime, and reduce latency in AI-powered applications.
The new primitives allow developers to implement strategies for directing requests to different AI models or providers based on specific criteria, and to handle situations where a provider becomes unavailable.
Key Experimental Types Introduced
Four new experimental types are available: RoutingChatClient, SemanticRoutingChatClient, FailoverChatClient, and OrderedFailoverChatClient. These types function as IChatClient implementations, providing a structured way to integrate routing and failover logic.
RoutingChatClient serves as the base class for selecting and forwarding requests to other clients. SemanticRoutingChatClient routes requests by message content, using embedding similarity against example utterances. FailoverChatClient adds a retry loop for reselecting clients upon failure, and OrderedFailoverChatClient is a concrete implementation that processes a list of clients sequentially.
Routing Implementation Details
The RoutingChatClient abstract class calls SelectClientAsync on each request to determine the target client. Developers can use RoutingChatClient.Create with a callback for simple routing or derive from RoutingChatClient to override SelectClientAsync for more complex, stateful policies.
Each call to GetResponseAsync or GetStreamingResponseAsync generates a RoutingContext, which includes request messages and a clone of ChatOptions. This design separates request-level options, which persist across retries, from route-level options, which are applied by client wrappers.
Semantic Routing Functionality
The SemanticRoutingChatClient enables routing based on the meaning of messages. This is achieved by comparing the embedding of the last user message against a set of example utterances provided for each client, allowing for intelligent content-based routing decisions.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Primary sources
GitHub aurelio-labs/semantic-router GitHub dotnet/extensions GitHub joshuajyue/routing-cli-sampleReporting from
Microsoft.Extensions.AI has released new experimental primitives for routing and failover in AI applications, addressing cost, uptime, and latency concerns. These primitives allow developers to route requests based on content or implement failover strategies for AI models and providers.