A pipeline sync bug that cleared our frontend S3 bucket

## The Incident **Duration:** 8 minutes of complete outage. **Impact:** Public landing page and checkout portal displayed blank screens or XML AccessDenied logs for all visitors. **Root Cause:** Missing staging/production namespace validation in CI script + active `--delete` flag. --- ### Timeline **11:04** - Developer merges a patch to the staging branch, triggering the staging deployment pipeline. **11:05** - Staging deployment job starts. Due to a copy-paste error, the environment variable `BUCKET_NAME` was hardcoded to the production bucket identifier (`devopsmanual-prod-assets`). **11:06** - The deploy runner executes: ```bash aws s3 sync ./dist s3://$BUCKET_NAME --delete ``` Because the build directory `./dist` only contained staging assets, the `--delete` flag instructed AWS to delete all existing files in the production bucket that did not match the build payload. Within 15 seconds, 95% of production static files were purged. **11:07** - Automated monitors alert on 404 file errors at the CDN layer. Engineers assemble on Zoom. **11:09** - Lead SRE identifies the bucket wipe and leverages S3 bucket versioning to initiate an automated rollback script. **11:13** - Rollback script completes, restoring deleted assets. The site is healthy. --- ### What Went Wrong **1. Lack of Environment Gates** The deploy script did not verify the target bucket suffix before executing destructive actions, allowing a staging job to target production S3 assets. **2. Destructive Delete Flag** Using `aws s3 sync --delete` in continuous integration builds is highly risky without bucket locks or confirmation checks. --- ### What We Changed **Fix 1: CI Environment Separation** We decoupled deployment permissions using IAM OIDC role authentication. The staging runner is now physically blocked from writing to the production bucket. **Fix 2: Stricter Build Checks** We added environment checks inside deployment scripts: ```bash if [[ "$ENV" != "production" && "$BUCKET_NAME" == *"-prod-"* ]]; then echo "CRITICAL: Staging job attempted to write to production bucket!" exit 1 fi ``` **Fix 3: MFA Delete & Object Versioning** Object versioning was set to active on the S3 bucket with strict MFA delete requirements for object purges.