Cron-Agent Patterns

Overview

A cron-orchestrated agent that makes external API calls (LLM CLI, HTTP client, subprocess) can hang, drift past its window, or overlap the next cron tick. This skill defends against that with nested timeouts — each tier strictly smaller than the one enclosing it — plus symmetric deadline guards and explicit headroom for the next fallback tier. Load it when implementing or reviewing any agent that runs on a schedule (cron, systemd timer, sleep-loop) and makes calls that can block indefinitely.

The Layered Timeout Ladder

Order the timeouts so each inner budget is strictly smaller than the budget enclosing it. Violating the strict-nesting invariant means an inner tier can never fire before the outer one preempts it — the inner guard becomes dead code.

• (a) Per-call timeout — bound every individual external call (HTTP read timeout, subprocess timeout, LLM CLI --timeout). Must fit inside (b) including retries and fallbacks.
• (b) Per-cycle deadline budget — compute a monotonic deadline once at cycle start; check it before every external call.
• (c) SIGALRM safety net — a library-level timeout cannot interrupt a call blocked in a C-level syscall; arm a hard alarm to force the process back into a handler.
• (d) Shell outermost guard — wrap the whole invocation in timeout --kill-after=<grace> <hard-limit> <cmd>, the last line of defense when everything in-process has wedged.

The pattern is stack-agnostic — the tier names map onto any runtime (Python signal/subprocess, Node timers/AbortController, Go context, shell timeout).

Anti-Patterns

Each of these caused a real production incident:

• Per-call timeout == cycle budget. If the per-call limit equals the whole cycle budget, the math forbids the call from ever fitting alongside retries or the next-tier fallback.
• max(N, deadline - now) floor. Flooring a per-call timeout at a constant re-inflates a doomed call after the deadline has already passed. Never floor the remaining budget — if it is zero or negative, skip the call.
• except Exception: pass swallowing the alarm. A bare exception handler around the call body silently eats the SIGALRM-raised exception, so the safety net never propagates. Re-raise the timeout exception before the generic handler.

Required Guards

• Symmetric deadline-passed guards in every fallback tier. A fallback chain (primary → secondary → tertiary provider) must check the deadline before each tier, not only before the first.
• Explicit headroom reservation for the next-tier fallback. Subtract a named reserve constant from the budget handed to each tier, so a fallback can still run after the primary times out. Without headroom the primary consumes the whole budget and the fallback is dead on arrival.

Checklist

• Each timeout tier is strictly smaller than the tier enclosing it (a<b<c<d)
• Cycle deadline computed once with a monotonic clock at cycle start
• Per-call timeout never floored back above the remaining budget
• Deadline checked before every external call and every fallback tier
• SIGALRM (or equivalent) armed for C-level blocked syscalls
• Timeout exception re-raised before any generic exception handler
• Explicit headroom reserved for the next fallback tier
• Shell timeout --kill-after wraps the whole invocation as outermost guard