A single failed SSH attempt or any unrelated exception in
ServerConnectionCheckJob marked the server unreachable and unusable, so
deployments failed with "Server is not functional" while the server was
online.
- Retry a failed SSH attempt once after 3 seconds within the same check.
- Mark the server unreachable only after UNREACHABLE_THRESHOLD (2)
consecutive failed checks; notifications use the same threshold.
- Keep the stored state when the job fails for a reason other than SSH or
Docker (database, cache, bugs).
- Raise the job timeout to 60 seconds and the overlap lock above it. The
old 15 second timeout killed checks before they wrote a result.
- Run the Docker and Compose version commands once with a 15 second
timeout. A failed Compose check no longer marks the server unusable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Horizon workers kept Server::findCached() results between jobs, so scheduled
database backups used stale reachability and subscription state. A backup that
was skipped with the stale state counted as done for that day. Flush the
Server identity map before each queue job.
Add debug logs for the full path of a scheduled run, so a multi-node cloud
instance shows where a run is lost:
- manager start delay, gap since the previous start, host, pid, memory, and
a warning when the start is later than the catch-up window
- skip logs with the server state that was used and the current database state
- occurrence enqueued, claimed, rejected, completed and failed, with host and
the delay since the due time; a warning when the previous run is still open
- count of occurrences still open after 15 minutes on each manager run
- database backup runs that end without a backup
- server connection state changes with the checking node and the SSH error
Also keep publishing pending occurrences when one of them fails.
Andras BacsaiandClaude Opus 5.5
Kevin Dench