How a single pod memory exhaustion crashed our payment system

## The Incident **Duration:** 47 minutes of degraded service, 12 minutes of complete outage. **Impact:** Checkout gateway unresponsive; payment processing failed for ~2,400 active sessions. **Root Cause:** Memory leak in coupon cache dict + restrictive Pod memory limits + lack of Horizontal Pod Autoscaling (HPA). --- ### Timeline **14:32** - Automated deployment of v2.4.1 completes. The build contains the new coupon lookup cache. Pod replicas show Running. **14:41** - First warning fires: `payment-service` p99 query latency exceeds 2.5 seconds. On-call engineer initiates log checks. **14:43** - Critical alert: API gateway error rate climbs past 5%. Gateway logs show socket timeouts. **14:44** - Pod status check reveals container terminations: ```bash kubectl get pods -n payments # payment-service-5c8f8-xj2p 0/1 OOMKilled 1 3m # payment-service-5c8f8-ab4c 1/1 Running 0 8m # payment-service-5c8f8-qw9z 0/1 OOMKilled 2 3m ``` **14:46** - With two pods killed, all incoming traffic gets routed to the single remaining pod. The extreme load spikes its memory usage, triggering a third OOMKill within 90 seconds. **14:47** - Complete service outage. AWS Application Load Balancer (ALB) health checks fail, returning 503 Service Unavailable. **14:49** - On-call engineer decides to roll back to v2.4.0 immediately. ```bash kubectl rollout undo deployment/payment-service -n payments kubectl rollout status deployment/payment-service -n payments ``` **14:54** - Rollback completes. All pods return to healthy status. Payments resume. --- ### What Went Wrong **1. Memory Leak in Cache Dict** The coupon feature cached query validations in a module-level dictionary that was never pruned or expired: ```python # The Bug: unbound dictionary grows infinitely in RAM _validation_cache = {} def validate_coupon(code: str, user_id: int) -> bool: key = f"{code}:{user_id}" if key not in _validation_cache: # DB record cached in memory _validation_cache[key] = db.query(Coupon).filter_by(code=code).first() return _validation_cache[key].is_valid ``` **2. Restrictive Memory Limits** The pod memory limit was hardcoded to `256Mi` based on legacy development profiles. Active production usage was already averaging `190Mi`, leaving no headroom for growth. **3. Missing Autoscaling** No HPA was configured. When one container was terminated, the cluster did not scale out to handle the distributed traffic. --- ### What We Changed **Fix 1: Remove the Memory Leak** The inline dictionary was removed, replacing validation lookups with a fast direct query. A Redis-backed cache with a strict TTL was added in a subsequent release. **Fix 2: Set Safe Memory Limits** We updated the manifest specifications to provide adequate memory buffer sizes: ```yaml resources: requests: memory: "256Mi" limits: memory: "512Mi" # Safe headroom for unexpected heap usage ``` **Fix 3: Implement HPA** We added a Horizontal Pod Autoscaler targeting memory limits: ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: payment-hpa namespace: payments spec: minReplicas: 3 maxReplicas: 10 metrics: - type: Resource resource: name: memory target: type: Utilization averageUtilization: 75 ```