The infra-automation Skill — Server Operations Without Guesswork

Running the same command across four servers, checking whether all HTTP services respond, or cataloguing every active listener before a migration — these are repetitive tasks where an ad-hoc approach accumulates small errors that compound. The infra-automation skill gives agents a fixed set of patterns for these operations so the approach is consistent and auditable.

The skill covers the Arcana server fleet: four machines (WWW, PROD, DB, and Trading), each with a public IP and a Tailscale mesh IP. Before any batch automation, host keys must be added to ~/.ssh/known_hosts through an explicit bootstrap step. That bootstrap event is documented; every subsequent automated SSH call uses BatchMode=yes and a five-second connect timeout so unknown hosts fail fast rather than hanging.

Batch execution and health checks

For a command that runs across all servers, the pattern is a single loop with explicit output headers per host and a line count cap on the output. This avoids the failure mode where one server produces a wall of log output that buries the results from the others. Health checks for HTTP services follow the same structure: a loop over service port and name pairs, with each check reporting either an HTTP status code or the string UNREACHABLE.

The safety rules are explicit. Destructive commands — deletes, drops — run one server at a time with confirmation, never in a loop. Every change is verified on one server before the loop runs across all of them. Operations are logged to a file for an audit trail.

Pre-migration inventory

Before moving a server, the skill specifies what to enumerate: web server virtual hosts, active TLS certificates, DNS records pointing to the server IP, all active listeners on the machine, cron jobs and systemd timers, and outgoing connections like webhooks and backups. A missed service is the most common cause of a migration going wrong, so the list is the starting point rather than an afterthought.

For config changes on a live server, the pattern is: backup, edit, diff, validate syntax, reload (not restart), then verify with curl. A reload keeps existing connections alive; a restart does not.

Compose deploy race and tracked artefacts

One specific pattern in the skill addresses a race condition in docker compose up -d --build: when a service uses restart: unless-stopped, a previous container may still hold its name when a new container tries to claim it. The fix is an explicit docker compose down --remove-orphans before the build. The teardown is idempotent on a cold start because of || true, and named volumes survive because down without -v does not touch them.

Any script or config installed on a production server and referenced by an acceptance criterion must be tracked in the repository before the criterion ships. An untracked artefact has no diff history and drifts invisibly from its intended behaviour.

Read what Datarim is for broader context, or see the related post on the network-exposure-baseline skill for how port binds are classified before any networking change.