Checkout.com Migrates to Managed Service for Apache Airflow to Improve Efficiency

Transition from Self-Managed to Managed Service

Checkout.com previously operated a self-managed Apache Airflow environment on a different cloud platform, which required considerable maintenance effort. The team faced challenges like instability during peak loads, complex dependency management, and long DAG sync times, which limited operational efficiency.

To address these issues, the Data Platform team decided to migrate to Google Cloud's Managed Service for Apache Airflow (Gen 3), which offers a fully managed solution designed to alleviate the burden of infrastructure management.

Benefits of Migration

Post-migration, Checkout.com observed immediate improvements in reliability and cost structure. The Managed Airflow service allows for built-in dynamic scaling, eliminating the need for the team to over-provision resources for peak times, thereby reducing costs.

Additionally, the migration simplified dependency management by handling package compatibility at the image level, enhancing developer velocity by reducing manual processes and optimizing resource allocation.

Operational Improvements and Outcomes

The migration has resulted in more efficient onboarding of new teams and a streamlined process for deploying data pipelines. Teams no longer need to manually create secrets and variables, significantly speeding up the development process. The reduced operational friction helps ensure stability even under high-load conditions.