Infrastructure Automation

Overview

Infrastructure Automation provides reusable patterns for SSH-based operations across your server fleet. It includes a server inventory with public and Tailscale IPs, batch execution templates, Tailscale mesh connectivity tests, and HTTP health check scripts for PROD services.

Server Inventory

A placeholder table of server roles (for example web, prod, db) with public IPs, Tailscale IPs, and SSH connection strings. Fill it from your own private inventory and verify it before use; real addresses are never hardcoded in the shipped skill.

SSH Batch Execution

Run a command on all (or selected) servers with proper flags: -o BatchMode=yes prevents interactive prompts from hanging automation, -o ConnectTimeout=5 prevents hanging on unreachable hosts. Output is limited to prevent flooding. Host keys are added to ~/.ssh/known_hosts once, through an explicit bootstrap step, before any batch automation; mesh reachability is checked per node first, and unreachable nodes fall back to the public IP or are deferred with a log entry.

Ping Matrix (Tailscale)

Tests N-by-N connectivity across all devices in the Tailscale mesh. Each source server pings every other server's Tailscale IP, producing a matrix of latencies. Useful for diagnosing mesh connectivity problems or verifying new node additions.

Health Checks

Checks all PROD services by port: curl to each service's /health endpoint from the PROD server itself, reporting HTTP status code or UNREACHABLE. Also includes templates for Docker service status, Tailscale status across all servers, disk usage checks, and nginx config validation.

Safety Rules

• Never run destructive commands via batch — one server at a time with explicit confirmation
• Always use BatchMode=yes to prevent password prompts
• Verify command on one server first before running batch
• Keep SSH sessions short — run and exit, no persistent sessions
• Log all operations for audit trail when making changes
• Probe service labels (launchd/systemd) live before writing them into a runbook — never from memory or placeholders