Home / Blog / How Google Traced GKE API Timeouts to a Linux Kernel TCP Bug
Tech News

How Google Traced GKE API Timeouts to a Linux Kernel TCP Bug

Google resolved intermittent GKE API timeouts blocking 800 developers by rolling back a routing change and backporting a Linux kernel patch to version 1.33.5.

By Dillip Chowdary • Nov 21, 2026 • Source: InfoQ

How Google Traced GKE API Timeouts to a Linux Kernel TCP Bug
Image: InfoQ

At a glance

Affected users
Around 800 developers
Detection month
January 2026
Initial workaround
Private Service Connect rollback
Upstream kernel fix
Linux 6.9.7
Backported branches
Linux 6.6 and Linux 6.1 LTS
GKE availability
Version 1.33.5 and later

Google engineers detailed a multi-layer networking investigation after three interacting flaws across Google Kubernetes Engine, Private Service Connect, and the Linux kernel disrupted control-plane access for approximately 800 developers. The incident, first detailed in InfoQ's report, left deployment pipelines blocked when pods experienced intermittent input-output timeouts while attempting to reach the Kubernetes API server. Diagnostic packet captures showed the control plane answering incoming connection attempts with stale acknowledgment packets from closed sessions instead of completing new handshakes.

This breakdown covers the architectural sequence behind the outage, the three overlapping defects across the proxy and kernel stack, and the deployment of upstream fixes across long-term support kernels. It is intended for platform engineers, site reliability teams, and infrastructure operators managing private Kubernetes control planes. The failure demonstrates how valid individual protocol implementations across cloud routing layers, virtual proxies, and the operating system network stack can combine into hard-to-diagnose production timeouts.

01Intermittent control plane connection timeouts

The failure began in January inside an environment running multiple clusters on Google Kubernetes Engine configured with Private Service Connect for private control-plane connectivity. Worker pods attempting to contact the Kubernetes API server suffered intermittent input-output timeouts that prevented developers from shipping software. Because each layer reported an incomplete connection attempt, standard triage initially pointed to several plausible causes without revealing the true failure mechanism.

Akash Thakur, senior technical solutions engineer at Google Cloud, observed that typical networking incidents isolate to a single tier such as a miscalculated maximum transmission unit, an exhausted connection pool, or a firewall misconfiguration. In this case, the symptom of an uncompleted connection remained identical whether engineers examined virtual network interfaces, proxy connection handling, or kernel connection tracking. Around 800 internal software developers remained unable to deploy to production while engineering teams worked to isolate the breakdown across several layers.

Engineers across Google Kubernetes Engine, Private Service Connect, cloud networking, host networking, and operating system kernel groups convened to capture traffic across the communication path. Deep packet inspections at the control plane confirmed that incoming TCP connection handshakes failed to complete because the control plane replied using out-of-date acknowledgment packets from older, terminated sessions. Correlating rollout events with simultaneous network path traces allowed investigators to uncover three distinct bugs executing in combination.

02Three interacting networking layer defects

The root cause stemmed from three independent technical behaviors spanning the managed routing layer, the proxy protocol layer, and the Linux kernel network stack. First, Private Service Connect introduced packet reordering during packet transit to the private control plane.

Second, Envoy proxy connection management implemented a half-close routine that prevented targeted TCP connections from shutting down in an orderly manner. Third, an operating system bug caused the Linux kernel to silently drop specific reordered finish packets during session termination.

Because the kernel discarded those reordered finish packets, sockets became stuck indefinitely in the FIN_WAIT_2 state. That retained state prevented the operating system from clearing closed connection records. When pods later initiated new connections using recycled source port pairs, the control plane interpreted incoming SYN packets as belonging to the lingering FIN_WAIT_2 session and returned stale acknowledgment packets.

Akash Thakur noted that every individual layer had a plausible explanation for the stalled handshake, yet every isolated theory remained incomplete without evaluating the combined stack. Rohan Karne, managing director at Google Cloud, described the response as the unglamorous reality of mission-critical platform engineering, requiring teams to systematically eliminate obvious suspects before locating the kernel flaw in packet traces.

03Sequential mitigation across cloud services

Advertisement

Tech Pulse Daily

Get tomorrow's pulse first

Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.

Google deployed three sequential remediations over several weeks to stabilize cluster traffic and permanently address each defective behavior. The engineering group first rolled back a pending Private Service Connect networking update, immediately eliminating transit packet reordering and restoring developer access to deployment pipelines.

Next, engineers modified proxy protocol handling within Envoy to correct half-close connection processing and introduced automated regression tests to verify proper TCP socket termination. Finally, former Google software engineer Eric Dumazet authored a dedicated patch for the upstream Linux kernel to ensure reordered finish packets are handled correctly by the operating system state machine rather than discarded.

Guangba Yu, a postdoctoral researcher at the Chinese University of Hong Kong, highlighted that the transferable takeaway for chaos engineering and automated operations involves simultaneously gathering network path observations, protocol states, and rollout schedules to separate operational symptom recovery from underlying state-machine corrections. Yu also noted that the published case study remains anonymized, omits broad population failure rates, and does not account for all Kubernetes API timeouts.

04Operating system backports and releases

Sidero field chief technology officer Justin Garrison pointed out that while identifying an issue requiring a kernel patch is compelling, Google Kubernetes Engine clusters rely on the older 6.1 long-term support kernel branch, meaning the underlying fix had to be backported from upstream releases. The kernel defect was formally merged into upstream Linux release 6.9.7 before maintainers backported the change into the Linux 6.6 and Linux 6.1 long-term support branches.

Google packaged the backported operating system fix directly into its managed Kubernetes distribution to ensure control-plane endpoints process reordered finish packets cleanly without requiring configuration adjustments. The patch rolled out to production clusters running Google Kubernetes Engine version 1.33.5 and later, eliminating the condition that left connection sockets abandoned in the FIN_WAIT_2 state.

Note: The patch reached Google Kubernetes Engine customers starting in version 1.33.5 and requires running that version or later to receive the backported Linux kernel fix.

05Multi-tier diagnostics for distributed infrastructure

The incident demonstrates that standard network monitoring focused on single boundaries can fail when cloud encapsulation, reverse proxies, and operating system kernels interact. Analyzing packet captures simultaneously across host interfaces and virtual interconnects provided the only reliable path to tracing why fresh connection requests received old acknowledgments.

Correlating software deployment logs with live protocol state transitions allowed engineers to connect an upstream routing rollout with downstream socket state retention. By addressing the proxy half-close routine, rollback of routing changes, and operating system state handling in parallel, infrastructure operators resolved the cascading failure across all affected layers.

Platform teams maintaining private control-plane connections can apply these diagnostic practices when debugging intermittent connection timeouts that elude single-layer network telemetry. Documenting interactions between proxy connection pooling and kernel socket lifetimes remains critical as enterprise clusters scale across managed cloud boundaries.

Frequently asked questions

What caused the intermittent GKE API connection timeouts?

Three interacting defects caused the issue: packet reordering in Private Service Connect, improper TCP half-close handling in Envoy, and a Linux kernel bug that dropped reordered FIN packets, stranding sockets in the FIN_WAIT_2 state.

How did Google resolve the problem for affected developers?

Google rolled back a Private Service Connect change to restore connectivity, fixed Envoy proxy protocol handling with added regression tests, and backported an upstream Linux kernel patch authored by Eric Dumazet.

Which versions of Google Kubernetes Engine contain the permanent kernel fix?

The fix was merged into Linux 6.9.7, backported to the Linux 6.6 and 6.1 LTS branches, and delivered to GKE customers in version 1.33.5 and later.

Sources

Dillip Chowdary

Author

Dillip Chowdary

Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.

Related on Tech Bytes

Advertisement

5-min tech signal

Weekday briefing for engineers who skip the noise.

No spam · Unsubscribe anytime

Advertisement

✈️ CareerPilot

Your AI job-search copilot

Match your resume against live Ashby, Greenhouse & Lever openings — fit scores, job-specific resume optimization and email alerts.

Find matching jobs →

Free Tools

Browse all tools →