Tokio Task Scheduling Prioritizes Progress Over Strict Ordering in High-Throughput Workloads

Event-Driven Service Architecture

The described service operates on an event-driven architecture, reading events from a message queue. For each incoming event, the service spawns a Tokio task to handle its processing. This design is common for systems requiring high throughput and concurrent processing of numerous independent units of work.

Task Fan-Out and Fan-In Pattern

Each event can contain up to 1000 user tokens. For every user token, an outbound API call is made, and the responses are collected. This involves a fan-out pattern where multiple Tokio tasks are spawned concurrently for each token, followed by a fan-in to gather all responses before generating a final response event. The primary goal of this setup is to maximize throughput.

Tokio Scheduler Behavior

The expectation was that short-lived Tokio tasks, typically finishing within milliseconds, would complete in a roughly ordered fashion, with earlier spawned tasks finishing earlier. However, observations from logs during a burst of 1000 events (generating approximately 1 million tasks) indicated that the Tokio scheduler prioritizes making progress across all active tasks rather than ensuring strict completion order based on their initiation time. This means tasks from later events or later within an event might complete before tasks from earlier events or earlier within an event.

Implications for Throughput and Ordering

This scheduling characteristic implies that while individual outbound calls might finish out of order, the overall system continues to make progress on a large number of tasks simultaneously. For applications where strict ordering of task completion is not a primary requirement and overall throughput is paramount, this behavior is acceptable. However, it highlights that developers should not assume completion order based solely on task spawning order in Tokio's asynchronous runtime.