publish took the combined signal (caller timeout + lock loss) and treated
any abort as a lost lock, so a scheduled or boot timeout firing between
renames skipped the rollback and cleanup while this process still held the
lock, leaving a mixed set and .next / .bak files. The combined signal still
cancels, but only the lock's own signal now stops rollback and cleanup.
Once every rename is done, only a lost lock keeps the version from being
recorded, so a timeout landing after a complete publish no longer forces a
re-download.
The lock-loss test now trips the engine's real onCompromised handler; a
new test covers the timeout path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CauPFca6bYL9qpuYEmsU9H
- The watcher started only after every boot step, so a dataset published
by another process (or placed by hand) while another step was running
became the watcher's baseline and was never loaded. It now starts before
the boot downloads.
- The CAIDA readers serve the newest *.txt under any name, but the boot
download checked only the published names, so an upgrade with a
hand-named snapshot re-downloaded it. A row can now supply hasSnapshot();
as2org / as-rel use their reader's own rule.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rows update one after another, each with up to 30 minutes, so four datasets
can legitimately take two hours while the monitor failed a run after one.
maxRuntime is now every scheduled row's timeout plus a 10-minute margin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rolling back after a lost lock could overwrite what the lock's new holder
had just published. Once the lock is lost, publish now touches nothing more
(no rollback, no cleanup). To keep a half-done set from passing for
current, the state names no version from the start of publishing until the
final write, so whoever runs next — after a lost lock, a crash or a kill —
re-fetches and publishes a whole set. A rename failure with the lock still
held is rolled back as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With connectivity now an error when no as-rel snapshot is loaded, the
network-down handler test has no working source on a data-less checkout
(CI), so the handler rightly answers 502. The test asserts that there and
keeps its 200 expectations where a snapshot exists.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lock-loss signal was checked only before publishing began, so a lock
compromised mid-copy or mid-rename let publication continue. It is now
checked before every copy and rename, an abort taking the same rollback,
and again before the state is written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- PeeringDB is 'disabled' only without a Cloudflare key; configured but not
loaded (still downloading, or the download failed) is an 'error', so the
ASN Profile isn't cached for a week without it.
- Connectivity without an as-rel snapshot is an 'error' too, not an empty
graph; a missing as2org still only blocks while it downloads, as RIPEstat
names the nodes.
- A CAIDA as2org / as-rel file that parses to nothing (corrupt) no longer
counts as loaded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A compromised lock (failed heartbeat; another process may take it over)
aborts the update holding it, download included, before it publishes,
instead of only being logged.
- A failed publish removes files it had just added when the dataset had none
before, as well as restoring replaced ones from backup: never half a set.
- The watcher reloads hand-placed or externally published files only once
they have held still for the settle interval and passed the row's
validator, so a file still being copied never replaces healthy data.
- Back to write-file-atomic 8, with engines aligned to its range
(^24.15.0 || >=26.0.0).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Scheduled and catch-up runs give each dataset 30 minutes, download
included, so a stalled upstream can't hold up the rest or, under croner's
overrun protection, every later run.
- Downloads go through fetchUpstream (project User-Agent, a timeout on the
wait for a response); the body stays bounded by the caller's signal.
- Catch-up asks whether a scheduled run fell between the last check and now,
instead of a 25-hour age that missed a check skipped just after one.
- A remote that names no version (no Last-Modified) is always fetched, not
taken for unchanged; maxmindIdentifier is null when an edition omits it.
- The watcher records a dataset's files as loaded only after the reload
succeeds, so a failed load is retried on their next change.
- write-file-atomic 7, which supports the project's Node >=24 (8 needs 24.15).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The backend downloaded every missing dataset before listening, so a first
boot (or a newly added dataset) refused every request for a minute or more.
Snapshots already on disk now load before the listener opens and missing
ones download behind it. Meanwhile only routes reading still-missing data
answer 503 (never edge-cached): /api/maxmind, /api/asn-connectivity and
/api/service-status. An ASN Profile section whose data is still loading
answers 'error' on its own, and the connectivity graph client doesn't cache
a 503, so both recover once the data lands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxMind and CAIDA each had their own updater with its own lock, state file,
publish step, boot download and timer. One engine (common/dataset-updater.js)
now owns all of that, and each dataset is a row in common/datasets.js that
holds only its own logic: find the newest remote version, fetch or distill
it, validate, reload.
- Locks: proper-lockfile, a heartbeat lock shared by every process, so a
crashed holder's lock goes stale on its own; a booting server waits for a
live holder and loads what it published.
- State: one file per dataset, written with write-file-atomic; the old
.maxmind-update-state.json / .caida-update-state.json are read once, so an
upgrade doesn't re-download.
- Schedule: croner at a fixed time (DATASET_UPDATE_CRON, default 04:30 local)
instead of 24 h from process start, plus a catch-up run at boot for a check
a downtime skipped. DATASET_AUTO_UPDATE (default on) replaces
MAXMIND_AUTO_UPDATE / CAIDA_AUTO_UPDATE, which still apply when it is unset.
- Reload: the engine watches every dataset's files, so a publish by another
process or a hand-placed .mmdb is picked up; MaxMind's own watcher goes.
MaxMind's City and ASN editions now publish together under one version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Shares over the part of the space walked before the budget ran out would
read as the ASN's whole IPv4 distribution. An exhausted budget now gives no
shares, like a lookup without an answer, and the answer stays uncached.
Real ASNs use at most ~30k of the 100k lookups.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
buildCountryShares now reports whether its walk completed: a lookup with
no answer (no database, or a failed lookup) and the lookup budget, now a
hard cap across blocks, both mark it incomplete, and the bgp-prefixes view
serves such an answer uncached. A prefix-less AS needs no lookups, so it
stays cacheable even without MaxMind. ASN_PROFILE_VERSION goes to 6 so
profiles cached with first-address country shares are not served on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The profile read `q` only on mount, so a new ASN link or history step with
the drawer open changed the URL but not the page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With MaxMind unavailable (a failed download) the bgp-prefixes view answered
`countries: []` as complete, and it was edge-cached for a week. The answer
is now marked partial, and the ASN Profile's prefixes section incomplete,
so neither is cached.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Country shares credited a whole announced prefix to the country of its
first address, so a /8 spanning many countries counted as one. Each prefix
is now walked one MaxMind network at a time, within a per-request lookup
budget.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The panel's IPv4/IPv6 prefix counts came from Radar's routes/stats, an
endpoint that fails often and, as one of the profile's summary segments,
kept complete ASN Profiles out of the edge cache. The ASN Profile already
shows prefix counts (from the pfx2as list) and relationships (the CAIDA
graph), so the panel drops both: no more routes/stats call, and the Radar
asn view no longer reads the local CAIDA snapshot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pino-pretty transport runs in a worker thread. In a loaded parallel
run, a spec that logs shortly before finishing could occasionally be kept
alive by that thread and never exit, stalling the whole suite until the
CI timeout. `pnpm test` now preloads tests/setup.js, which switches the
logger to its synchronous JSON output unless LOG_FORMAT is already set.
Runtime logging is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review findings on the ASN Profile work, mostly one class of bug: a failed
upstream read as "no data", which the 7-day edge cache would then pin.
- AS Rank: a 200 carrying GraphQL errors is a failure, not an unknown AS.
- Radar: an answer built from partly failed segments is still shown but
flagged `incomplete` and kept out of the cache, on the aggregate and on
the standalone asn view. A 404 is Radar's "no such AS" and counts as
empty, as does a segment that simply has no figures.
- Reputation: a malformed or non-200 answer is an error.
- Prefixes: a payload without `prefix_origins` is an error; [] stays valid.
- Radar is `empty` when only local CAIDA counts are present.
- requireValidASN enforces AS1–AS4294967295 and canonicalizes; the range
rule lives once in common/asn-input.js, shared with the frontend.
- The PeeringDB listing lookup goes through fetchUpstream.
- New functions follow the const-arrow convention; test stubs compare
parsed hostnames instead of URL substrings.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The tool is titled as a lookup, its card note is one short line, and the
intro names PeeringDB among its sources — across all six locales.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A third offline dataset joins the CAIDA updater: the daily PeeringDB dump,
distilled by a streaming pass into a compact per-ASN index (network type,
scope, peering policy, exchange and facility presence). Contact tables are
never parsed and the raw dump is deleted after distilling. It downloads
only when the Cloudflare key is configured, since the tool is hidden
without it.
The profile shows network type, scope and website in the header and a
"Peering & facilities" card whenever the AS lists an exchange or facility.
Aggregate deadlines now sit above the fetch timeouts they wrap, so a
source that degrades internally still delivers its partial answer.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Adds the v7.7.0 changelog line for the new tool, drops two exports
nothing imports, and aligns the reputation test fixtures with the fields
the private API actually returns.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A full profile of one autonomous system at /tools/asn?q=AS<n>, served by
a single /api/asn-profile request: identity and registration with the raw
Whois record, key facts (rank, announced IPv4, prefixes, users), traffic
shares and connection quality, where its IPv4 space is used, the announced
prefix list with RPKI status, and connectivity with the topology graph.
A reputation card appears beside the header when the backend returns it.
- The ASN Info panel now opens the profile instead of linking out.
- ASNConnectivity gains `expandable` / `bordered` so the profile can embed
it without the nested drawer and the second border.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
/api/asn-profile gathers every source for one AS in parallel and answers
once: Radar entity and traffic data, announced prefixes with country
shares, the CAIDA topology and neighbours, the RDAP registration record,
AS Rank, and (when the private API is configured) reputation. Each source
has its own deadline; a slow or failed one is simply absent, flagged in a
`status` map, and only complete answers are edge-cached.
- Upstream / peer / customer counts now come from the local CAIDA
snapshot with the same rule the topology graph applies, for both the
profile and the ASN Info panel; the graph builder itself is unchanged.
- The ASN data family (aggregate, Radar asn and bgp-prefixes views)
shares a 7-day edge cache.
- /api/asrank is folded into the aggregate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Registry gate `requiresConfig`: a tool is listed in the Advanced Tools
grid and the nav only when its backend config is present; the listing
filter is one pure function. ASN Profile requires the Cloudflare key.
- `useAsnInfo` owns the Radar ASN fetch and its session cache, shared by
the IP cards and the query dialog.
- ASN Info's traffic-share and connection-quality blocks become small
components over pure helpers, reusable by the profile page.
- ASN Profile lands as a mock-data page for now.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- cfradar `view=asn` keeps the RPKI route counts from routes/stats.
- New cfradar `view=bgp-prefixes`: the AS's announced prefixes (pfx2as)
plus IPv4 country shares resolved against the local MaxMind database.
- New /api/asrank: CAIDA AS Rank lookup (rank, customer cone, degree).
Misses answer 200 with null fields and stay out of the edge cache via
the new `cacheIf` option on `cacheable`.
- /api/asn-connectivity adds `neighbours`: provider / peer / customer
counts and name lists from the local AS-relationship snapshot.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Whois tool and /api/whois now take `AS<n>` (prefix in any case) next
to IPs and domains, answered from the RIR's RDAP autnum record. A bare
number stays invalid input, so it can never be mistaken for anything else.
The raw blocks no longer start with the template's own indentation.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Follows the v3 score output of MyIP-api (this frontend no longer reads
the v2 shape; deploy it right after the API).
- A toggle next to the quality score opens a panel: a short explanation
assembled from the score dimensions (network type, anonymity with its
last-observed date, reputation, the ASN's neighbourhood, the verdict),
and all boolean attributes as check / cross / question-mark icons.
Residential proxies seen only in the API's own sighting history read
as "suspected".
- The card's "Proxy" field becomes "Anonymizer": Tor exit, iCloud
Private Relay, residential proxy, proxy, VPN (each with a suspected
variant when only one source agrees) or not detected. Card fields are
anonymity / anonymityCode / anonymityProvider / anonymityProtocol; the
provider line hides when the API returns null. Texts live under
ipInfos.anonymity.*, with one code-to-key map for card and report.
- Diagnostic reports store the anonymity code (REPORT_VERSION 2; v1
links show the unsupported-version page, reworded to fit either
direction). The persona collector no longer sends the unused isProxy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ninety days is longer than most visitors need for a local record. Stored
preferences above the new cap clamp to 30 on read, and older day buckets
are pruned on the next load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collapse repeated daily IP records into summaries ordered by last seen.
Keep country and IP-type filters across both views and expand dates only
for IPs recorded on multiple days. Preserve the daily storage format.
Co-Authored-By: GPT-6 <noreply@openai.com>
jason5ng32andClaude Opus 5.5
Jason Ng
