Infrastructure Automation

Overview

Infrastructure Automation provides reusable patterns for SSH-based operations across the Arcana server fleet. It includes a server inventory with public and Tailscale IPs, batch execution templates, Tailscale mesh connectivity tests, and HTTP health check scripts for PROD services.

Server Inventory

Maintains the current list of Arcana servers (WWW, PROD, DB, Trading) with their public IPs, Tailscale IPs, and SSH connection strings. Always verified against the latest memory references before use.

SSH Batch Execution

Run a command on all (or selected) servers with proper flags: -o BatchMode=yes prevents interactive prompts from hanging automation, -o ConnectTimeout=5 prevents hanging on unreachable hosts. Output is limited to prevent flooding.

Ping Matrix (Tailscale)

Tests N-by-N connectivity across all devices in the Tailscale mesh. Each source server pings every other server's Tailscale IP, producing a matrix of latencies. Useful for diagnosing mesh connectivity problems or verifying new node additions.

Health Checks

Checks all PROD services by port: curl to each service's /health endpoint from the PROD server itself, reporting HTTP status code or UNREACHABLE. Also includes templates for Docker service status, Tailscale status across all servers, disk usage checks, and nginx config validation.

Safety Rules

• Never run destructive commands via batch — one server at a time with explicit confirmation
• Always use BatchMode=yes to prevent password prompts
• Verify command on one server first before running batch
• Keep SSH sessions short — run and exit, no persistent sessions
• Log all operations for audit trail when making changes