Feat(datasets): one updater engine for every offline dataset

MaxMind and CAIDA each had their own updater with its own lock, state file,
publish step, boot download and timer. One engine (common/dataset-updater.js)
now owns all of that, and each dataset is a row in common/datasets.js that
holds only its own logic: find the newest remote version, fetch or distill
it, validate, reload.

- Locks: proper-lockfile, a heartbeat lock shared by every process, so a
  crashed holder's lock goes stale on its own; a booting server waits for a
  live holder and loads what it published.
- State: one file per dataset, written with write-file-atomic; the old
  .maxmind-update-state.json / .caida-update-state.json are read once, so an
  upgrade doesn't re-download.
- Schedule: croner at a fixed time (DATASET_UPDATE_CRON, default 04:30 local)
  instead of 24 h from process start, plus a catch-up run at boot for a check
  a downtime skipped. DATASET_AUTO_UPDATE (default on) replaces
  MAXMIND_AUTO_UPDATE / CAIDA_AUTO_UPDATE, which still apply when it is unset.
- Reload: the engine watches every dataset's files, so a publish by another
  process or a hand-placed .mmdb is picked up; MaxMind's own watcher goes.

MaxMind's City and ASN editions now publish together under one version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
jason5ng32andClaude Opus 5.5 committed 2026-10-05 02:21:58 +08:00
1 parent 152906b762
commit 287c50fe1a
17 files changed
+1122 -1093

No files matched your search

+4 -1
View File
@@ -42,6 +42,7 @@
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"concurrently": "^10.0.5",
"croner": "^10.0.1",
"dagre": "^0.8.5",
"dotenv": "^17.4.2",
"express": "^5.2.1",
@@ -55,6 +56,7 @@
"pino": "^10.3.1",
"pino-http": "^11.0.0",
"pino-pretty": "^13.1.3",
"proper-lockfile": "^4.1.2",
"reka-ui": "^2.10.4",
"tailwind-merge": "^3.7.0",
"tar": "^7.5.22",
@@ -69,7 +71,8 @@
"vue-router": "^5.3.1",
"vue-sonner": "^2.0.9",
"whoiser": "^1.18.0",
"world-atlas": "^2.0.2"
"world-atlas": "^2.0.2",
"write-file-atomic": "^8.0.0"
},
"devDependencies": {
"@sentry/vite-plugin": "^5.4.0",