Pine59 Migrates to Managed Airflow 3 on Google Cloud for Improved Data Pipeline Orchestration

Modernizing Data Orchestration

Pine59, a company specializing in location intelligence data, processes millions of complex data points daily to feed predictive models. To manage its large data pipelines, including a Daily Foot Traffic metric that computes data for up to 14 million locations in a single job, Pine59 relies entirely on Google Cloud, utilizing BigQuery for heavy lifting and Managed Service for Apache Airflow (formerly Cloud Composer) for orchestration.

Transition to Airflow 3

As Pine59's data volume and machine learning workloads increased, the company decided to modernize its monorepo, which contains hundreds of directed acyclic graphs (DAGs). They stress-tested production workloads against the newly available Managed Airflow (Gen 3) architecture running Airflow 3. This test demonstrated immediate and significant improvements in processing speed, task scheduling, and overall stability, leading to a full transition to the new environment.

Enhanced MLOps Capabilities

A core aspect of the migration involved optimizing the orchestration of Pine59's ML inference workloads. Previously, standard Kubernetes operators were used for these tasks. By moving to Managed Airflow (Gen 3), which offers an optimized and abstracted infrastructure layer, Pine59's engineering team refined its MLOps architecture. This included setting up a dedicated Google Kubernetes Engine (GKE) cluster specifically optimized for model inference and integrating it into the new Airflow environment.