AI Model Versioning Insufficient for Production AI Reliability

The Problem with Model-Only Versioning

AI applications can fail in production even when the model itself remains unchanged and healthy. This occurs because other components, such as input processing, prompts, retrieval configurations, tool contracts, and serving settings, can independently alter the application's behavior. For instance, a change in retrieval that sends more context can increase generation time, leading to timeouts and service degradation, even if the model version is the same.

Defining the AI Release Boundary

To ensure reliable AI infrastructure, a release boundary must encompass all components that interact to form the application. This extends beyond just the model version. MLOps workflows should then focus on testing this complete release, observing its performance under real traffic, and providing a safe rollback mechanism. The practical starting point for this is not a larger platform, but a versioned release manifest, a meaningful evaluation gate, and a tested rollback path.

The Release Manifest Approach

Traditional ML systems face similar issues, such as training-serving skew, which Google's MLOps guidance addresses through data and model validation. Generative AI applications further expand the set of dependencies that need management. A release manifest, like the illustrative example provided, would include identifiers for the application revision, model revision, prompt revision, retrieval components (index, embedding, pipeline), runtime configuration, and evaluation suite. Each reference in the manifest should resolve to retained and inspectable configurations or artifacts.

Comprehensive Versioning for Reliability

The runtime revision within the manifest should cover all settings affecting execution, including token limits, batching, timeouts, and resource placement. If the application integrates with external tools, their schemas and adapters should also be versioned. The manifest should store references to secrets, not the secret values themselves. While a manifest does not guarantee bit-for-bit reproducibility due to external service changes and the non-deterministic nature of generation, it provides a crucial framework for managing AI application deployments.