LinkedIn has announced the replacement of Argo CD with an in-house developed GitOps engine for managing its Kubernetes deployments. This new system is responsible for deploying applications across more than 250 Kubernetes clusters, supporting over 4,000 services, and handling in excess of 100,000 deployments each week.
The implementation of the custom engine has led to significant improvements in deployment reliability and speed. Deployment infrastructure failures decreased from 925 to 63, even with a 3.7x increase in deployment volume. The median time required to detect and resolve a failed deployment also saw a substantial reduction, falling from 16.1 minutes to 2.45 minutes.
LinkedIn initially used Argo CD with one instance per cluster, a setup chosen for managing cluster add-ons. However, at LinkedIn's scale, with clusters running up to 5,000 nodes, several issues emerged. These included the inability to link sync and health status to specific deployments due to the lack of an `observedGeneration` field, and unintended revisions being deployed due to stale repo-server cache entries.
Architectural limitations also became apparent. Argo CD's single watch per group-version-resource across applications caused etcd churn from one application to slow down syncs for all applications. Additionally, Kubernetes' truncation of `managedFields` after 10 distinct field managers, combined with server-side-apply PATCH operations, led to issues where removed containers could persist on live Pods, particularly in AI workloads with multiple containers per Pod.
After evaluating alternatives like Flux, which shared similar controller paradigms, LinkedIn opted to build its own engine as a library. This design ensures that each deployment runs in its own pod without shared controller state. The new system uses PUT operations for full replacements instead of PATCH, and incorporates retries with exponential backoff for `resourceVersion` conflicts. Drift detection is performed using a 3-way diff between the live object, the previously deployed manifest, and the newly rendered manifest.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's briefNew every morning, and the back catalogue is archived by date.
LinkedIn has replaced Argo CD with its own custom GitOps engine to manage deployments across over 250 Kubernetes clusters, supporting 4,000+ services and 100,000+ weekly deployments. This change reduced deployment infrastructure failures by 93% and decreased the median time to resolve failed deployments from 16.1 to 2.45 minutes. The move addresses scalability issues encountered with Argo CD, such as slow syncs and issues with server-side apply PATCH operations in large-scale Kubernetes environments.