Amazon EKS shares lessons from scaling Kubernetes controllers

Kubernetes Controllers and Their Functionality

Kubernetes controllers are essential for translating a declarative model into actionable operations. They observe the current state of the system and reconcile it with the intended state, ensuring that the system continues to operate correctly as conditions change.

Challenges of Scaling Controllers

As clusters grow larger, maintaining the correctness of controllers becomes increasingly difficult. Issues such as cache lag and object churn are exacerbated when scaling, demanding a comprehensive view to enforce network traffic rules effectively.

Amazon EKS Controllers

In Amazon EKS, two specific controllers address the challenge of traffic enforcement: the Network Policy Controller and the VPC Resource Controller. The former manages communication protocols within the cluster, while the latter assigns AWS security groups to selected pods for controlling external access.

Impact of Security Group Changes

Previously, pods inherited the security group of their node, necessitating broad access for resources like RDS databases. Now, individual pods can have their own security groups, allowing fine-grained access control—enhancing security and alignment with operational intent as pods are provisioned and decommissioned.

Takeaway for Developers

The learnings from operating these controllers at scale highlight crucial considerations for developers working with large Kubernetes deployments. Ensuring the accuracy of the state and the efficiency of the reconciliation process is vital to maintain operational intent across varied cluster sizes.