Commit Graph
10 Commits
Author SHA1 Message Date
Andras Bacsai b081673c98 fix(server): skip Sentinel checks on unreachable servers
- CheckAndStartSentinelJob returns early when the server is not
  functional. It now reads the container state with `docker ps --filter`,
  so an SSH failure throws and is no longer treated as a missing
  container that needs a restart.
- ServerManagerJob only dispatches the Sentinel version check for
  functional servers.
- On cloud, ServerPatchCheckJob runs on the maintenance queue and
  ServerLimitCheckJob runs on the webhooks queue.
- The default number of cloud maintenance workers goes from 10 to 60.
- excludeCertainErrors drops its unused ignore list and is typed as
  `never`.
- Docs and the Horizon config comments are updated, and tests cover the
  new behavior.
2026-10-03 23:39:38 +02:00
Andras Bacsai a3ed963aed fix(server): stagger Sentinel version and patch checks per server
Servers no longer all connect at minute 0. The hourly Sentinel check uses
a per-server minute, and the weekly patch check uses a per-server time on
Sunday between 04:00 and 23:59.
2026-10-03 21:25:00 +02:00
Andras Bacsai 51e6949c36 fix(sentinel): stop hourly check job pile-up and self-heal Sentinel
- Start Sentinel with --restart unless-stopped so Docker restarts it
  after a crash or reboot.
- Make CheckAndStartSentinelJob unique per server so copies cannot
  collect in a slow queue.
- Read X-Sentinel-Version on push and update an outdated Sentinel
  without SSH; skip the hourly SSH version check for servers that
  report their version.
2026-10-02 09:41:04 +02:00
Andras Bacsai b5393e24ec fix(server): recheck unreachable servers before failing deployments
Deployments failed with "Server is not functional" when the server was
still flagged unreachable after a single failed scheduled check. Add
Server::isFunctionalAfterRecheck(), which runs a live SSH and Docker
connection check when the server is marked unreachable. It skips the
check for disabled servers and servers with placeholder IPs, then
re-evaluates isFunctional(). ApplicationDeploymentJob now uses it.

ServerManagerJob no longer skips the SSH connection check for servers
with a live Sentinel when the server is flagged unreachable or unusable.
The Sentinel heartbeat does not restore these flags, so without the
check they would never recover.

Add feature tests for the recheck behavior and unit tests covering the
Sentinel skip condition.
2026-10-01 09:30:21 +02:00
Andras BacsaiandClaude Opus 5.5 a79ee909c6 chore: remove unused components and markup-only tests
- Delete Blade components, component classes, Livewire components and
  views that nothing uses after the redesign.
- Delete tests that only compare Blade/CSS source text; keep the source
  checks for authorization gates, stable wire:key values and escaping.
- Update stale tests to the current UI and behaviour.
- Update DESIGN.md to the current navigation and layout.
- Add a lesson: test behaviour, not source markup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 00:18:16 +02:00
Andras Bacsai 22fbbd74ff fix(sentinel): restore hourly version checks for enabled servers
Dispatch Sentinel checks hourly for eligible servers, including healthy
installations, while continuing to exclude build servers.
2026-09-14 15:43:52 +02:00
Andras Bacsai 424dbd36ff feat(sentinel): make sentinel mandatory on regular servers
Remove the enable/disable toggle from the server UI, logs page, and
Sentinel API so is_sentinel_enabled is derived and read-only. Enable
existing regular servers via migration, start Sentinel after validate-
and-install, and drop the daily ServerManagerJob restart.
2026-09-09 06:30:32 +02:00
Andras Bacsai 7ae76ebc79 test(factories): add missing model factories for app test suite
Enable `HasFactory` on `Environment`, `Project`, `ScheduledTask`, and
`StandaloneDocker`, and add dedicated factories for related models to
stabilize feature/unit tests.

Also bump `visus/cuid2` to `^6.0` and refresh `composer.lock` with the
resulting dependency updates.
2026-03-03 09:50:05 +01:00
Andras Bacsai a0c177f6f2 feat(jobs): add queue delay resilience to scheduled job execution
Implement dedup key-based cron tracking to make scheduled jobs resilient to queue
delays. Even if a job is delayed by minutes, it will catch the missed cron window
by tracking previousRunDate in cache instead of relying on isDue() alone.

- Add dedupKey parameter to shouldRunNow() in ScheduledJobManager
  - When provided, uses getPreviousRunDate() + cache tracking for resilience
  - Falls back to isDue() for docker cleanups without dedup key
  - Prevents double-dispatch within same cron window

- Optimize ServerConnectionCheckJob dispatch
  - Skip SSH checks if Sentinel is healthy (enabled and live)
  - Reduces redundant checks when Sentinel heartbeat proves connectivity

- Remove hourly Sentinel update checks
  - Consolidate to daily CheckAndStartSentinelJob dispatch
  - Crash recovery handled by sentinelOutOfSync → ServerCheckJob flow

- Add logging for skipped database backups with context (backup_id, database_id, status)

- Refactor skip reason methods to accept server parameter, avoiding redundant queries

- Add comprehensive test suite for scheduling with various delay scenarios and timezones
2026-02-28 15:06:25 +01:00
Andras Bacsai 4002044877 Refactor: Move sentinel update checks to ServerManagerJob and add tests for hourly dispatch 2025-12-04 14:58:18 +01:00