Files
GeoLibre/scripts/fetch-basemaps-assets.mjs
Qiusheng Wuandpre-commit-ci[bot] 183642bc59 feat(pmtiles): styled offline basemaps + consolidate offline menus + CORS worker (#1259)
* feat(pmtiles): styled offline basemaps + consolidate Project offline menus

Turn an extracted PMTiles archive into a proper styled basemap (roads, water,
landcover, buildings, labels) instead of a flat single-colour overlay, and
unify the offline story under one Project menu.

- @geolibre/map: buildProtomapsBasemapStyle() generates a MapLibre style from
  @protomaps/basemaps for a pmtiles source in a chosen flavor (light/dark/
  white/grayscale/black). Applied as the basemap via an offline-basemap
  sentinel that map-controller's resolveMapStyle expands from a registry
  (mirrors the planetary-basemap pattern; falls back gracefully if the
  session-scoped archive is gone on reload).
- Bundle Protomaps glyphs (Latin/Cyrillic/Greek ranges) + all flavor sprites
  under public/basemaps-assets (1.3 MB) so it renders fully offline; refresh
  with scripts/fetch-basemaps-assets.mjs.
- BasemapExtractPanel: a "Basemap style" flavor picker. A Protomaps-schema
  vector archive is applied as the styled basemap; anything else (or "None")
  is added as the flat overlay layer.
- Project menu: replace "Download offline areas" + "Manage offline areas"
  (service-worker tile cache) with a single "Offline Basemap…" opening the
  extract panel. Removes OfflineRegionDialog/OfflineManagerDialog.
- workers/tiles: add a /pmtiles/<name>.pmtiles CORS range proxy for the
  Protomaps daily planet builds (scoped to build.protomaps.com, range-only),
  so the extractor can pull them from any origin.

Verified in a real browser: extracted Florence z0-15 from the live planet
build and applied it as a dark Protomaps basemap — roads, water, parks, POI
icons, and labels all render from the bundled offline assets.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix(pmtiles): extract panel polish — default proxy URL, resize, dark dropdown

- Default the Archive URL to the Protomaps planet build via the deployed CORS
  proxy (tiles.geolibre.app/pmtiles/<recent-date>.pmtiles), computed at open
  time so it stays fresh; widen the panel so the full URL is visible.
- Fix the dark-mode flavor dropdown contrast by using the themed @geolibre/ui
  Select instead of a bare <select>.
- Don't auto-seed the bbox on open (a globe/world view produced a near-global
  box whose overlay drew a stray diagonal line); guard the overlay for
  near-global boxes too. The user draws a box or clicks "Use map view".
- Make the panel resizable via a bottom-end grip; fix a position jump by
  pinning the parent-relative top-start corner (matching the drag).
- Show the estimated output size + tile count during download, highlighted
  when large.
- Default the panel to the top-start corner with a small gap; equal-width
  "Draw on map" / "Use map view" buttons; drop the now-redundant URL hint.

* feat(pmtiles): manage saved offline basemaps (list, rename, delete, re-apply)

Add a device-local catalogue of extracted basemaps and a "Saved basemaps"
section in the panel:

- lib/offline-basemaps.ts: a localStorage manifest (name, bbox, zoom range,
  flavor, size, saved path, date) with load/upsert/rename/delete + a change
  subscription. Each extract records an entry.
- The panel lists saved basemaps with size/zoom/flavor and per-row actions:
  use-as-basemap, rename (inline), delete. Re-apply regenerates the Protomaps
  style and reuses the live in-session archive, or (on desktop) reloads its
  bytes from the saved file; raster archives can't be styled.
- @geolibre/map: export hasPMTilesArchive and readLocalFileBytes for the
  live-vs-reload decision.

* feat(pmtiles): reload saved basemaps on web, per-entry flavor, clear-bbox

- "Use as basemap" now reloads a saved basemap whose in-memory archive is gone
  this session: from the remembered path on desktop, or via a file picker on
  web (browsers give no stable path to a downloaded file). Removes the dead-end
  "reopen on desktop" error — the affordance is the button itself.
- Each saved basemap has its own flavor dropdown; changing it re-styles the
  basemap (instantly if its archive is loaded, else on next apply) since the
  archive's tiles are flavor-independent. Adds setOfflineBasemapFlavor.
- Add a Clear (eraser) button beside Draw on map / Use map view that clears the
  drawn bounding box.

* fix(pmtiles): re-styling the active offline basemap now takes effect

registerOfflineBasemapStyle keyed the sentinel by id alone, so changing the
flavor of the currently active basemap re-registered the style under the same
URL. MapCanvas early-returns when basemapStyleUrl is unchanged, so map.setStyle
never fired and the map kept the old flavor. Give each apply a unique sentinel
(an incrementing suffix, dropping the id's prior entry) so the store value
changes and the basemap re-renders. Verified: switching a loaded basemap
Light -> Dark now restyles the map live.

* feat(pmtiles): latest.pmtiles proxy route, static default URL, delete confirm

- workers/tiles: /pmtiles/latest.pmtiles resolves to the newest daily Protomaps
  build by probing backward from today (no "latest" alias or index upstream)
  and caching the resolved date, so every range request in one extraction hits
  the same file. Deployed.
- Default the extract URL to the static latest.pmtiles proxy URL — always
  current with no client-side date to go stale.
- Deleting a saved basemap now asks for inline confirmation instead of removing
  on the first click.

* feat(basemap): style a PMTiles URL as a basemap; open local PMTiles; tidy menus

- BasemapPickerDialog + NewProjectDialog: a custom URL ending in .pmtiles (or
  a pmtiles:// URL) now reveals a Protomaps flavor selector and applies the
  archive as a styled offline basemap (new lib/pmtiles-basemap-url.ts).
- Fix: an active offline-basemap sentinel no longer seeds the custom URL field
  and trips the "invalid URL" error — treat it as its own choice
  (export isOfflineBasemapSentinel from @geolibre/map).
- BasemapExtractPanel: Saved basemaps section is always visible with an
  "Open file…" action to apply a local .pmtiles as a basemap without
  downloading; default archive URL seeds to latest.pmtiles unless a non-proxy
  URL was last used.
- Remove "Offline Basemap Extract" from the Add Data menu (the flow lives under
  Project → Offline Basemap); drop the now-dead addLayer.basemapExtract wiring.
- Rename the extract panel's "Archive URL" label to "Basemap URL".

* fix(basemap): persist the custom URL and PMTiles flavor in the picker

Applying a PMTiles URL swaps the basemap to an opaque offline sentinel, so
reopening the picker left the custom URL field blank. Remember the last applied
custom URL (and flavor) in localStorage and re-seed the field on open.

* fix(basemap): register the pmtiles protocol for a remote PMTiles basemap

A raw basemap style never runs the layer-sync path that registers the
pmtiles:// protocol and its archive, so a styled remote PMTiles basemap
applied no tiles — MapLibre had no handler for its source. Add
ensureRemotePMTilesArchive() (mirrors ensurePMTilesProtocol for a FetchSource-
backed remote URL) and call it when building the basemap style. Verified
against data.source.coop/opengeos/protomaps/planet/latest.pmtiles: range GETs
return 206 and the light basemap renders land, water, and place labels.

* Address Claude and CodeRabbit review feedback

- workers/tiles: bound the PMTiles range proxy — reject open-ended
  (`bytes=0-`) or oversized ranges (>32 MB) so it can't stream a 100+ GB
  planet build through the Worker; the presence-of-Range check alone did
  not cap the span.
- workers/tiles: fold the forwarded `Range` header into the sub-fetch
  cacheKey so a cached 206 for one byte range can't be served for a
  different range of the same file (default CF cache key is URL-only).
- protomaps-basemap: resolve bundled glyphs/sprites against
  `import.meta.env.BASE_URL` (caller passes assetsBaseUrl) so styled
  offline basemaps keep labels/icons when served under a sub-path.
- BasemapExtractPanel: free the previous styled basemap's in-memory
  PMTiles archive (and evict its style-registry entry) when a different
  one is applied or the active one is deleted — styled basemaps aren't
  GeoLibreLayers, so removeLayerFromMap never freed them (memory leak).
- BasemapExtractPanel: tighten the Protomaps-schema heuristic to require
  a distinctive `earth`/`water` layer (plus ≥2 schema layers), so a
  generic vector archive with only `roads`/`places` isn't styled with
  Protomaps expressions and rendered blank.
- BasemapExtractPanel: "Use as basemap" now also requires a non-null
  flavor, so overlay-saved entries can't be forced through the styled
  path into a broken basemap.
- BasemapExtractPanel: make the resize grip work in RTL layouts (invert
  the drag delta, measure available width to the left, pin the right
  edge via pos.x).
- packages/map: revert the incidental react/react-dom dependency
  widening (^19.0.0 → ^19.2.7) to match sibling workspaces.

* Address second-round review feedback

- workers/tiles: document that the /pmtiles range proxy is intentionally
  not origin-gated (Jupyter/embed builds on arbitrary origins need it),
  and that abuse is bounded by the per-request range cap plus Cloudflare
  platform/zone rate limiting — mirroring the OAM route's throttling note.
- tests: add unit coverage for the new pure-logic modules —
  offline-basemaps catalogue CRUD (upsert/rename/flavor/delete + malformed
  data) and the Protomaps offline-basemap style builder + sentinel
  registry (register/get/evict/isSentinel, sub-path assets base).

* Address third-round review feedback

- BasemapExtractPanel: hide the per-entry flavor <select> (and no-op
  handleSavedFlavorChange) for overlay-saved entries (flavor === null),
  so a flavor change can't force an incompatible archive through the
  styled-basemap path that applySaved's guard was added to block.
- BasemapExtractPanel: openPmtilesAsBasemap ("Open file…") now also
  requires isProtomapsCompatible, matching handleExtract — a non-Protomaps
  vector archive no longer gets force-styled into a blank basemap.
- pmtiles-basemap-url: buildRemotePmtilesBasemap now passes assetsBaseUrl
  = `${BASE_URL}basemaps-assets`, so remote-PMTiles basemaps applied via
  the New Project / Basemap Picker dialogs keep labels/icons under a
  sub-path deployment (GEOLIBRE_APP_BASE).
- NewProjectDialog: translate the custom-basemap flavor picker via t()
  (basemapExtract.style / basemapExtract.flavor.*), matching
  BasemapPickerDialog instead of hardcoded English.
- workers/tiles: drop `cf.cacheEverything` from the range proxy — caching
  206s under a URL-only key risks serving the wrong range's bytes, and
  `cf.cacheKey` only works on Enterprise plans, so don't rely on it;
  refuse a 200 (range-ignored) response so the whole file can't stream
  through; reject ranges with offsets past 2^53 (Number precision).

* Address fourth-round review feedback

- BasemapExtractPanel: deleting the currently-applied styled basemap now
  resets the basemap to DEFAULT_BASEMAP before freeing its archive, so the
  live MapLibre style isn't left pointing at a pmtiles:// source whose
  bytes were just released (which would 404 on pan/zoom).
- BasemapExtractPanel: "Open file…" now derives the saved-basemap name via
  layerNameFromPath, so a desktop absolute path yields "tuscany" rather
  than a mangled "home-alice-Downloads-tuscany".
- BasemapExtractPanel: move the active-styled-basemap tracker onto
  globalThis (matching the style registry / archive-key set this PR adds)
  so it survives a Vite HMR reload and doesn't leak the live archive.

* Address fifth-round review feedback

- workers/tiles: stop edge-caching the resolveLatestBuildDate existence
  probe. `cacheEverything` cached the 1-byte `bytes=0-0` 206 under the
  URL-only cache key for `<date>.pmtiles` — the same URL the range proxy
  fetches next — so a later real range read could be served the 1-byte
  body. The resolved date is already memoised in `latestCache`, so no
  edge cache is needed; this reintroduced the exact corruption the
  main-proxy fix removed.
- NewProjectDialog / BasemapPickerDialog: use the themed @geolibre/ui
  Select for the custom-basemap flavor picker instead of a bare native
  <select>, matching the fix already applied to the main extract-panel
  picker (dark-mode contrast + consistency).
- NewProjectDialog: translate the "Enter a valid style.json or .pmtiles
  URL." validation message via t() (new newProject.invalidCustomUrl key),
  matching BasemapPickerDialog's basemapPicker.invalidUrl.

* Address sixth-round review feedback

- BasemapExtractPanel: compare against the live store `basemapStyleUrl`
  (via a new isLiveOfflineBasemap helper + exported
  OFFLINE_BASEMAP_SENTINEL_PREFIX) instead of the panel-local tracker,
  which only sees applies made from this panel and goes stale once the
  basemap is switched via the picker / New Project dialog. Fixes two
  bugs: (1) deleting a saved entry could reset an unrelated, currently-
  active basemap to the default; (2) changing a non-displayed entry's
  flavor could silently swap it in as the live basemap. Now the default-
  reset on delete and the live re-style on flavor change only fire when
  the entry is actually the basemap on the map.

* Address seventh-round review feedback

- map-controller: resolveMapStyle now returns a structuredClone of the
  registered offline-basemap style instead of the shared registry object
  (matching the planetary path, which builds a fresh object). In
  split/compare view two Map instances resolve the same sentinel;
  MapLibre mutates the style it's handed, so sharing one object let the
  two panes corrupt each other's style state.
- pmtiles-basemap-url: use the percent-encoded URL as the remote-PMTiles
  registry id instead of a 32-bit rolling hash, removing the (unlikely
  but real) chance that two distinct URLs collide and one silently evicts
  the other's registry entry.

* fix(pmtiles): absolute basemap-assets URL + skip layer-control fetch of offline sentinel

Two runtime errors surfaced when a styled offline basemap was applied on
the deploy preview (relative "./" base):

- "Invalid sprite URL ... must be absolute": the sub-path fix built the
  glyphs/sprites base as `${import.meta.env.BASE_URL}basemaps-assets`,
  which is `./basemaps-assets` under the embed/demo build's relative base
  — and MapLibre rejects non-absolute sprite/glyph URLs. Resolve it to a
  fully-qualified URL via `new URL(base, document.baseURI).href` in both
  BasemapExtractPanel and pmtiles-basemap-url; this stays absolute for
  "/", a "/sub-path/", and a relative "./" base alike.

- "Failed to fetch geolibre://offline-basemap/…": the layer control
  fetches the basemap style URL to introspect its layers, but an offline
  basemap uses a non-fetchable `geolibre://` sentinel. MapController
  already substitutes BLANK_BASEMAP for planetary sentinels; extend that
  to offline-basemap sentinels so the control skips the fetch (and shows
  a single background entry) instead of throwing.

Verified in a real browser: applying a remote styled Protomaps basemap
now renders labels/water/landcover with 0 console errors (previously the
sprite + sentinel-fetch errors).

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-07-15 08:41:18 -04:00

60 lines
2.3 KiB
JavaScript
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
// Downloads the Protomaps basemaps glyphs + sprites this app bundles so an
// extracted PMTiles archive can render as a fully offline styled basemap
// (no CDN at runtime). Output: apps/geolibre-desktop/public/basemaps-assets/,
// which bakes into the web and desktop builds and is served at
// /basemaps-assets/. Re-run to refresh:
//
// node scripts/fetch-basemaps-assets.mjs
//
// Glyphs are bundled for ranges 0-1023 (Latin, Latin Extended, IPA, Greek,
// Cyrillic) for the three fontstacks @protomaps/basemaps uses. With the style
// generated at lang:"en" this covers labels for most of the world; scripts
// outside these ranges (CJK, Arabic, …) fall back to unrendered labels offline
// while all map geometry still renders.
import { mkdir, writeFile } from "node:fs/promises";
import { fileURLToPath } from "node:url";
const BASE = "https://protomaps.github.io/basemaps-assets";
const OUT = fileURLToPath(
new URL("../apps/geolibre-desktop/public/basemaps-assets/", import.meta.url),
);
const FONTSTACKS = ["Noto Sans Regular", "Noto Sans Medium", "Noto Sans Italic"];
const RANGES = ["0-255", "256-511", "512-767", "768-1023"];
const FLAVORS = ["light", "dark", "white", "grayscale", "black"];
const SPRITE_FILES = (flavor) => [
`${flavor}.json`,
`${flavor}.png`,
`${flavor}@2x.json`,
`${flavor}@2x.png`,
];
async function download(url, destPath) {
const res = await fetch(url);
if (!res.ok) throw new Error(`${res.status} for ${url}`);
const bytes = new Uint8Array(await res.arrayBuffer());
await mkdir(destPath.slice(0, destPath.lastIndexOf("/")), { recursive: true });
await writeFile(destPath, bytes);
return bytes.length;
}
let total = 0;
for (const fontstack of FONTSTACKS) {
for (const range of RANGES) {
const url = `${BASE}/fonts/${encodeURIComponent(fontstack)}/${range}.pbf`;
const dest = `${OUT}fonts/${fontstack}/${range}.pbf`;
total += await download(url, dest);
}
}
for (const flavor of FLAVORS) {
for (const file of SPRITE_FILES(flavor)) {
const url = `${BASE}/sprites/v4/${file}`;
const dest = `${OUT}sprites/v4/${file}`;
total += await download(url, dest);
}
}
console.log(
`Wrote ${OUT} (${(total / 1024 / 1024).toFixed(2)} MB: ` +
`${FONTSTACKS.length}×${RANGES.length} glyph ranges, ${FLAVORS.length} sprite flavors)`,
);