How an ArgoCD sync loop saturated our cluster CPU and crashed our monitoring

## The Incident **Duration:** 35 minutes of degraded monitoring. **Impact:** Cluster alerts disabled; SRE dashboards lost visibility into system metrics. **Root Cause:** Circular reference inside custom admission controller webhook resource templates, triggering infinite sync reconciliation loops. --- ### Timeline **09:40** - Engineer deploys a custom webhook configuration updates. **09:42** - ArgoCD starts resource reconciliation. The CPU utilization of the ArgoCD Application Controller pod spikes to 100%. **09:45** - System alerts stop firing. Dashboard metrics freeze. **09:50** - SRE checks cluster nodes and discovers that the node hosting the Prometheus pod has entered a NotReady status due to CPU starvation. **09:55** - SRE identifies the infinite reconciliation storm and terminates the ArgoCD deployment controllers. **10:10** - The node returns to normal. ArgoCD is restored with reconciler resource quotas set. **10:15** - Monitoring services are fully restored. --- ### What Went Wrong **1. Infinite Sync Loop** A Helm template was configured to auto-generate resource parameters during every reconciliation loop. ArgoCD detected these changes and attempted to redeploy, generating a new set of parameters and initiating a new sync loop infinitely. **2. Missing Resource Quotas** The ArgoCD controllers did not have CPU limits configured, allowing the application controller pod to consume all available resources on the shared node. --- ### What We Changed **Fix 1: Configure Resource Quotas** We added CPU limits to our ArgoCD controller manifests to prevent resource starvation on host nodes: ```yaml resources: requests: cpu: "500m" memory: "512Mi" limits: cpu: "2000m" memory: "2Gi" ``` **Fix 2: Set Safe Sync Parameters** We disabled automatic pruning and auto-sync triggers on resource templates that generate values dynamically during execution runs.