Home / Blog / When a sync window closes: How Argo CD self-heal skipped…
Tech News

When a sync window closes: How Argo CD self-heal skipped the migration hook

Argo CD automated self-heal operations bypassed database migration hooks across two regional rollouts, leaving services misaligned for up to 44 minutes.

By Dillip Chowdary • Nov 21, 2026 • Source: CNCF Blog

When a sync window closes: How Argo CD self-heal skipped the migration hook
Image: CNCF Blog

At a glance

Product
Argo CD
Publication date
November 20, 2026
Source organization
Cloud Native Computing Foundation
Author role
CNCF Ambassador
Test container
busybox:1.36
Observed impact
36 to 44 minutes of schema mismatch

Cloud Native Computing Foundation Ambassador Afzal Ansari detailed a critical race condition in Argo CD where closed sync windows cause self-heal operations to silently bypass database migration hooks, as documented in CNCF Blog's report. The defect surfaced when five regional deployments resumed synchronization after an off-hours deny window, leaving two regions operating on mismatched database schemas for up to forty-four minutes despite reporting fully healthy synchronization states.

This analysis examines how Argo CD internal refresh mechanisms evaluate revisions when execution windows reopen and outlines why standard resource hooks fail under automated drift correction. Platform engineers and Kubernetes operators managing automated delivery pipelines with scheduled change freezes will find concrete architectural remediation strategies to guarantee database migrations run reliably across every deployment cycle.

01Regional rollout failures and silent drift

An enterprise deployment experienced immediate schema desynchronization after releasing new application software during a project deny window. When the scheduled window expired, five regional instances initiated automated rollout routines for the identical Git commit. Three regions executed the necessary migration Job, but two skipped the task entirely, forcing updated code to query an incompatible database schema for 36 to 44 minutes.

The operational failure remained entirely invisible within the administrative dashboard. Both failing regional clusters displayed Synced and Healthy indicators on the new release commit, while operation logs reported that all tasks had executed successfully. Because standard monitoring showed spotless deployment states, operators had no immediate signal that underlying schema migrations had been omitted.

Diagnostic inspection of controller logs revealed the root operational discrepancy. The two anomalous operations recorded thirteen specific resources alongside a self-heal attempt counter of one. Instead of executing a fresh release deployment with configured hooks, Argo CD had treated the event as routine configuration drift, triggering an isolated reconciliation that bypassed lifecycle hooks.

02Controlled reproduction in a four-minute window

Investigation required building an isolated test environment consisting of an automated Application with prune and self-heal enabled. A ConfigMap and Deployment carried release identifiers to force an OutOfSync status, while a busybox 1.36 container handled schema updates under a standard Sync hook annotation. The project applied a four-minute deny window starting at the active minute to track exact expiration timing.

Initial tests failed to reproduce the bug because periodic background reconciliation coincided with the window expiration, executing an ordinary full deployment. Adjusting the test script to trigger a manual refresh fifteen seconds prior to window closure delayed the subsequent reconciliation timer. Upon window expiration, the application synced release v11 in one second while the migration Job remained stalled on release v10.

Subsequent testing demonstrated consistent vulnerability, reproducing the skipped migration five times across six independent executions. The operational failure required no bizarre configurations or edge-case cluster instability. Any deployment pipeline relying on automated sync and self-heal across scheduled change windows risks bypassing critical database initialization whenever event timing aligns unfavorably.

03Dual refresh loops and revision mistakes

Advertisement

Tech Pulse Daily

Get tomorrow's pulse first

Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.

Detailed controller debug logs revealed two distinct synchronization drivers operating within Argo CD. A level-2 refresh triggers every two to three minutes from an internal reconciliation timer, evaluating the Application directly against the remote Git repository. In contrast, a level-1 refresh responds immediately to managed resource modifications and relies exclusively on previously cached controller state.

Sync window expirations occur without broadcast notifications, allowing whichever refresh type arrives first to govern deployment behavior. While the window remains active, timer-driven refreshes update the status sync revision field without executing actual changes. When the window closes, a level-1 refresh inspects this recorded revision against itself, mistakenly assumes the commit has already synced, and initiates an automated partial drift correction.

Because partial sync routines only reconcile flagged drift across modified resources, the controller skips all lifecycle hooks and omits history entries. Alternative adjustments like switching to PreSync hooks or activating RunHooksOnPartialSync fail to resolve the problem. The controller ignores partial sync hook options during automated sync operations, leaving the underlying timing race intact across every window closure.

04Migration jobs ordered by sync waves

Reliable execution requires treating database migrations as core desired state rather than transient lifecycle side effects. Operators should eliminate hook annotations, assign the migration Job an argocd.argoproj.io/sync-wave annotation of negative one, and append dynamic release identifiers such as the container tag to the Job name. This structure ensures every synchronization pass, whether partial or full, identifies the unapplied Job as missing.

In simulated test runs, wave-based migrations executed reliably through the broken self-heal pathway, finishing one second before dependent Deployments and ConfigMaps updated. Argo CD halts progression until earlier waves report healthy status, guaranteeing complete schema readiness before serving application traffic. Meanwhile, retaining automated pruning ensures the cluster automatically cleans up obsolete migration Jobs from preceding software releases.

Note: Operators must avoid setting ttlSecondsAfterFinished on migration Jobs. When Kubernetes deletes a completed Job, automated self-heal identifies an unexpected deletion and triggers duplicate migration execution. Teams should maintain strictly idempotent migration logic and validate failure handling in non-production environments prior to cluster rollout.

05Automated synchronization policies and drift tradeoffs

For environments where immediate Helm chart modifications are impractical, disabling automated self-heal provides a temporary operational workaround. When self-heal is disabled, the controller logs an auto-sync skip message and halts deployment until a timer-driven reconciliation initiates approximately two minutes later. This delayed reconciliation correctly detects revision divergence, executes a full synchronization sequence, and runs the configured migration hook.

Choosing between configuration strategies involves balancing automation speed against manual drift protection. While turning off self-heal prevents schema mismatches, manual cluster alterations remain marked OutOfSync until engineers push subsequent commits. Converting migrations into declarative sync waves eliminates the timing race permanently by embedding schema progression directly into Git-managed cluster state.

Comparing migration management options highlights the tradeoffs between operational latency and drift resilience across production Kubernetes clusters. Teams must evaluate whether temporary synchronization delays or complete declarative wave definitions best protect backend databases from unapplied schema versions. The comparison outlines how each deployment configuration behaves when sync windows expire.

StrategyHook ExecutionSync TypeSchema Safety
Self-Heal Active (Hook)Skipped on Level-1Partial SyncSchema Mismatch (36–44m)
Self-Heal Disabled (Hook)Runs on Level-2Full Sync (2m wait)Schema Safe
Sync Waves (Desired State)Runs as Wave -1Full or PartialSchema Safe

Frequently asked questions

Why did Argo CD skip the database migration hook after the sync window closed?

A level-1 refresh triggered by object drift mistakenly treated the latest Git revision as already synced. Argo CD then initiated an automated partial self-heal sync, which skips all lifecycle hooks and omits history entries.

How can platform engineers ensure database migrations run reliably in Argo CD?

Teams should convert migration Jobs from Sync hooks into standard managed resources using sync waves set to negative one with unique release names. This ensures any synchronization pass detects the missing Job and applies schema updates before dependent applications roll out.

Does disabling self-heal resolve the skipped migration issue?

Yes, setting self-heal to false causes the controller to skip the faulty partial sync and wait for the reconciliation timer. Two minutes later, a timer-driven level-2 refresh executes a full sync that runs the migration hook properly.

Can setting RunHooksOnPartialSync prevent this Argo CD race condition?

No, Argo CD ignores the RunHooksOnPartialSync setting for automated sync operations. Because self-heal is automated by definition, partial syncs will continue to bypass lifecycle hooks.

Sources

Dillip Chowdary

Author

Dillip Chowdary

Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.

Related on Tech Bytes

Advertisement

5-min tech signal

Weekday briefing for engineers who skip the noise.

No spam · Unsubscribe anytime

Advertisement

✈️ CareerPilot

Your AI job-search copilot

Match your resume against live Ashby, Greenhouse & Lever openings — fit scores, job-specific resume optimization and email alerts.

Find matching jobs →

Free Tools

Browse all tools →