Rotation Runbook

Overview

A credential rotation looks trivial — issue a new secret, revoke the old one — and that is exactly why it goes wrong. The recurring failure modes are not in the rotation itself but around it: a consumer nobody remembered, a verification probe that returns green without authenticating, a validation regression hiding behind the auth failure, and a secret written back to a path no other document agrees on. This skill formalizes the discipline distilled from repeated real rotation exercises.

When To Use

• Rotating any credential: API key, access token, OAuth client secret, webhook signing secret, TLS key, database password.
• Responding to a leaked or exposed credential (accidental commit, plaintext file, audit finding).
• Verifying that a past rotation actually converged (secret store ↔ producer truth-check).

The Runbook

1. Inventory every consumer first. Enumerate every place the credential is read: CI/CD secret stores, env files, secret-manager paths, service configs, cron jobs, sibling projects. Perform a truth-check between the secret store and the producer — treat "consumer copy ≠ producer truth" as the first hypothesis for any HTTP 401 from a previously working integration.
2. Choose the rotation window deliberately. Prefer a grace window (dual validity) when the provider supports it; immediate revoke (grace = 0) is required for an actively leaked credential and otherwise acceptable only with atomic consumer updates.
3. Verify with auth-scoped endpoints only. Public or catalog endpoints return success without authentication — a false green. Required both directions: old credential → explicit rejection; new credential → success. Allow for provider propagation delay before declaring failure.
4. Replay the full producer payload, not a minimal probe. An auth failure can mask a validation regression accumulated while the integration was broken. Only the full end-to-end replay proves the integration works.
5. Store the new secret at the canonical path defined by the project's secret-store schema document — never at a path invented at plan time.
6. Record the rotation — date, credential name, reason, rejection/success evidence, consumers updated — in the project's credential document.
7. Leak response extras: rotate first, scrub second; keep a local mirror as the pre-scrub backup (a backup tag on the same remote is destroyed by force-push); verify the scrub from a fresh clone.

Failure-Mode Checklist

• Consumer copy ≠ producer truth → 401 from a working integration → step 1 truth-check.
• False-green verification → revoked key still "works" → step 3 auth-scoped endpoint.
• Hidden validation regression → 400 right after fixing the 401 → step 4 full replay.
• Secret-path convention drift → three documents name three paths → step 5 canonical schema.
• Backup tag lost to force-push → pre-scrub state unrecoverable → step 7 local mirror.
• Premature verification failure → new key rejected seconds after issue → step 3 propagation wait.