← All stories
● Covered by 1 source · 1 reportMedium impact1 neutral

Google Traces GKE API Timeouts to Interacting Bugs in PSC, Envoy, and Linux Kernel

🔄 Updated 20h ago
New to BrevFeed? We gather this story from every outlet covering it into one summary — ranked by real-world impact, not just the latest headline — so you never miss what matters. What is BrevFeed? →

Key points

  • Intermittent GKE API timeouts affected 800 developers.
  • Three interacting bugs were found in PSC, Envoy, and Linux kernel.
  • PSC packet reordering, Envoy half-close, and kernel FIN packet discard caused the issue.
  • A Linux kernel patch was the final fix for stuck FIN_WAIT_2 sockets.

Intermittent GKE API Failures

In January, Google Kubernetes Engine (GKE) customers using Private Service Connect (PSC) for private control-plane access experienced intermittent timeouts when pods attempted to reach the Kubernetes API server. This issue blocked approximately 800 developers from deploying to production environments.

Complex Root Cause Analysis

Packet captures revealed the control plane responded to new TCP connection requests with stale ACK packets from previously closed connections. Google engineers from GKE, PSC, networking, hostnet, and kernel teams collaborated with the Linux upstream community to identify the root causes. The investigation found three distinct, interacting bugs across different networking layers.

Three Interacting Bugs Identified

The problem was traced to three specific issues: packet reordering occurring within Private Service Connect, Envoy's half-close behavior preventing TCP connections from properly closing, and a bug in the Linux kernel that incorrectly discarded certain reordered FIN packets. This combination left sockets indefinitely stuck in the FIN_WAIT_2 state, causing I/O timeouts.

Multi-Layered Troubleshooting

Akash Thakur, a senior technical solutions engineer at Google Cloud, noted the complexity of the case, as symptoms appeared consistent across PSC, the kernel's TCP state machine, and Envoy's connection management. Each layer presented plausible but incomplete explanations, requiring a deep dive into cross-layer interactions.

Coordinated Fixes Implemented

The issue was resolved through a series of coordinated fixes over several weeks. Google initially rolled back a Private Service Connect change to restore connectivity. Subsequently, proxy protocol handling was corrected, and regression tests were added. The final resolution involved a patch to the Linux kernel itself to address the FIN packet handling.

✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →

The daily brief

One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.

One email a day. Unsubscribe in one click, any time.

Today's brief

Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.

~4 min · 3 stories · Nov 21

▶ Play today's brief Listen on Spotify

New every morning, and the back catalogue is archived by date.

Reporting from

Google identified three interacting bugs across Google Kubernetes Engine (GKE), Private Service Connect (PSC), and the Linux kernel that caused intermittent Kubernetes API connectivity failures for approximately 800 developers. The issue stemmed from packet reordering in PSC, Envoy's half-close behavior, and a Linux kernel bug that discarded reordered FIN packets, leading to sockets stuck in FIN_WAIT_2 state.