Expedia Develops AI-Assisted Service Telemetry Analyzer (STAR) for Incident Investigation

Introduction of Service Telemetry Analyzer (STAR)

Expedia Group has developed and implemented Service Telemetry Analyzer (STAR), an internal AI-assisted observability platform. This system is designed to assist engineers in investigating production incidents by analyzing service telemetry data and providing structured root cause assessments. The primary goal of STAR is to minimize the time engineers spend identifying the source of service degradation.

How STAR Functions

STAR integrates operational metrics with large language models (LLMs) through a series of predefined diagnostic workflows. Unlike autonomous AI agents, the platform follows a deterministic workflow: telemetry is collected, analyzed using domain-specific prompts, consolidated into intermediate findings, and then summarized into a final report. This report includes potential root causes and recommended next steps, with human validation and decision-making remaining central to the process.

Technical Architecture and Data Sources

The platform is built as a FastAPI application that connects with Datadog for retrieving service metrics and an internal generative AI gateway for managing LLM access. The workflow utilizes prompt chaining to perform multiple specialized analyses before producing a consolidated diagnosis. STAR focuses on standardized infrastructure telemetry from Kubernetes-based services and JVM applications, analyzing metrics such as request throughput, latency, error rates (HTTP, gRPC, GraphQL), CPU and memory utilization, container restart events, Kubernetes probe failures, Java heap utilization, and garbage collection activity. This approach provides a consistent view across services developed with different programming languages and frameworks.

Asynchronous Processing Model

Expedia evolved STAR's architecture by replacing FastAPI background tasks with an asynchronous model using Redis as both the message broker and result backend. The engineering team noted that most processing involves I/O bound operations, specifically telemetry retrieval and LLM requests. This asynchronous execution model enables STAR to process multiple analysis tasks concurrently, improving efficiency.