In January, Google Kubernetes Engine (GKE) customers using Private Service Connect (PSC) for private control-plane access experienced intermittent timeouts when pods attempted to reach the Kubernetes API server. This issue blocked approximately 800 developers from deploying to production environments.
Packet captures revealed the control plane responded to new TCP connection requests with stale ACK packets from previously closed connections. Google engineers from GKE, PSC, networking, hostnet, and kernel teams collaborated with the Linux upstream community to identify the root causes. The investigation found three distinct, interacting bugs across different networking layers.
The problem was traced to three specific issues: packet reordering occurring within Private Service Connect, Envoy's half-close behavior preventing TCP connections from properly closing, and a bug in the Linux kernel that incorrectly discarded certain reordered FIN packets. This combination left sockets indefinitely stuck in the FIN_WAIT_2 state, causing I/O timeouts.
Akash Thakur, a senior technical solutions engineer at Google Cloud, noted the complexity of the case, as symptoms appeared consistent across PSC, the kernel's TCP state machine, and Envoy's connection management. Each layer presented plausible but incomplete explanations, requiring a deep dive into cross-layer interactions.
The issue was resolved through a series of coordinated fixes over several weeks. Google initially rolled back a Private Service Connect change to restore connectivity. Subsequently, proxy protocol handling was corrected, and regression tests were added. The final resolution involved a patch to the Linux kernel itself to address the FIN packet handling.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's briefNew every morning, and the back catalogue is archived by date.
Google identified three interacting bugs across Google Kubernetes Engine (GKE), Private Service Connect (PSC), and the Linux kernel that caused intermittent Kubernetes API connectivity failures for approximately 800 developers. The issue stemmed from packet reordering in PSC, Envoy's half-close behavior, and a Linux kernel bug that discarded reordered FIN packets, leading to sockets stuck in FIN_WAIT_2 state.