fix(python): render local rasters as XYZ tiles in Colab

This commit is contained in:
giswqs committed 2026-08-25 12:29:25 -04:00
1 parent 642cc9834b
commit 45599eff6a
11 files changed
+267 -21

No files matched your search

+8 -4
View File
@@ -27,12 +27,12 @@ conda install -c conda-forge geolibre
```
Optional extras provide GeoPandas/Shapely support for GeoDataFrames and local
vector files, plus xarray/rioxarray/rasterio support for in-memory rasters:
vector files, plus xarray/rioxarray/rasterio/rio-tiler support for in-memory rasters:
```bash
pip install "geolibre[all]" # both
pip install "geolibre[vector]" # GeoPandas/Shapely only
pip install "geolibre[raster]" # xarray/rioxarray/rasterio only
pip install "geolibre[raster]" # xarray/rioxarray/rasterio/rio-tiler only
```
`[all]` is the union of the two, so it now installs rasterio (and GDAL with it);
@@ -86,8 +86,10 @@ the app's own Earth Engine panel manages.
`add_raster` / `add_cog` also accept a **local** GeoTIFF path on the kernel host:
the file is served by the bundled localhost server so the app can read it. This
works directly in local Jupyter and VS Code. Google Colab routes the file through
its built-in kernel port proxy; JupyterHub can do the same when
works directly in local Jupyter and VS Code. In Google Colab, where the kernel
proxy does not preserve the byte-range semantics required by browser COG
rendering, GeoLibre renders local rasters as PNG XYZ tiles in the kernel instead.
JupyterHub can route the COG through the kernel port when
`jupyter-server-proxy` is available. A static-server-extension-only deployment
cannot expose kernel files, so pass a hosted URL there. The served URL is
session-scoped, so a project saved with a local raster will not restore it when
@@ -109,6 +111,8 @@ m.add_raster(
The temporary Cloud-Optimized GeoTIFF backing an xarray layer is removed when the widget is
closed, so xarray layers have the same session-only limitation as local files.
Locally the browser reads that COG directly; in Colab rio-tiler renders it into
ordinary PNG XYZ tiles to avoid Colab's incompatible byte-range proxy behavior.
Call `m.close()` when you are done to remove it promptly; otherwise it is removed
when the `Map` is garbage collected or when the kernel exits normally. A kernel
that is killed outright leaves the file behind in the system temp directory.
+3 -1
View File
@@ -203,7 +203,9 @@ anywhere untrusted, or use `Map.save_project`, which redacts by default.
- Optional extras: `pip install "geolibre[all]"` adds GeoPandas/Shapely support
for `add_geojson(geodataframe)` and for reading **local** vector files
(`add_vector`/`add_geoparquet`/`add_flatgeobuf`/`add_shp`/`add_kml`/`add_gpkg`),
plus xarray/rioxarray/rasterio support for in-memory rasters. `[vector]` and
plus xarray/rioxarray/rasterio/rio-tiler support for in-memory rasters.
Colab uses rio-tiler to serve these as PNG XYZ tiles because its proxy does
not preserve browser COG byte ranges. `[vector]` and
`[raster]` install just one half each -- `[all]` pulls in rasterio (and GDAL).
The kernel reads local vectors and inlines them as GeoJSON; remote URLs for
those formats stream through the in-browser vector control and need no extras.
+2 -2
View File
@@ -25,9 +25,9 @@ dependencies = ["anywidget>=0.9", "traitlets>=5", "jupyter-ui-poll>=0.2"]
# `all` is the union of the data extras. It now pulls in the raster stack
# (rasterio ships GDAL), so installs that only need GeoDataFrames and local
# vector files can ask for `vector` instead.
all = ["geopandas", "shapely", "xarray", "rioxarray", "rasterio"]
all = ["geopandas", "shapely", "xarray", "rioxarray", "rasterio", "rio-tiler"]
vector = ["geopandas", "shapely"]
raster = ["xarray", "rioxarray", "rasterio"]
raster = ["xarray", "rioxarray", "rasterio", "rio-tiler"]
# The MCP server (geolibre.mcp) is the only thing that imports the SDK, so it
# stays optional: the widget and the project builders work without it.
mcp = ["mcp>=2.0"]
+24 -5
View File
@@ -71,7 +71,7 @@ function proxiedLocalFileUrl(raw, base, port) {
parsed.hostname.endsWith(`-${port}-colab.googleusercontent.com`);
if (
((!loopback || parsed.port !== String(port)) && !colabProxy) ||
!parsed.pathname.startsWith("/_geolibre_local/")
!["/_geolibre_local/", "/_geolibre_tiles/"].some((prefix) => parsed.pathname.startsWith(prefix))
) {
return raw;
}
@@ -81,10 +81,15 @@ function proxiedLocalFileUrl(raw, base, port) {
// Mark these URLs so the embedded app can carry each requested byte range in
// the query string instead; the token-protected Python route translates it
// back into a normal 206 response. Other Jupyter proxies keep using headers.
if (rewritten.hostname.endsWith(`-${port}-colab.googleusercontent.com`)) {
if (
rewritten.pathname.startsWith("/_geolibre_local/") &&
rewritten.hostname.endsWith(`-${port}-colab.googleusercontent.com`)
) {
rewritten.searchParams.set("__geolibre_range_proxy", "1");
}
return rewritten.href;
// URL serialisation escapes XYZ placeholders, but MapLibre requires the
// literal braces in its template in order to substitute tile coordinates.
return rewritten.href.replace(/%7B([zxy])%7D/gi, "{$1}");
}
export function rewriteProxiedLocalFileUrls(project, base, port) {
@@ -96,12 +101,26 @@ export function rewriteProxiedLocalFileUrls(project, base, port) {
if (!layer || typeof layer !== "object") return layer;
const rawUrl = layer.source?.url;
const url = proxiedLocalFileUrl(rawUrl, base, port);
const rawTiles = layer.source?.tiles;
const tiles = Array.isArray(rawTiles)
? rawTiles.map((tile) => proxiedLocalFileUrl(tile, base, port))
: rawTiles;
const sourcePath = proxiedLocalFileUrl(layer.sourcePath, base, port);
if (url === rawUrl && sourcePath === layer.sourcePath) return layer;
const tilesChanged =
Array.isArray(rawTiles) && tiles.some((tile, index) => tile !== rawTiles[index]);
if (url === rawUrl && !tilesChanged && sourcePath === layer.sourcePath) return layer;
changed = true;
return {
...layer,
...(url !== rawUrl ? { source: { ...layer.source, url } } : {}),
...(url !== rawUrl || tilesChanged
? {
source: {
...layer.source,
...(url !== rawUrl ? { url } : {}),
...(tilesChanged ? { tiles } : {}),
},
}
: {}),
...(sourcePath !== layer.sourcePath ? { sourcePath } : {}),
};
});
+103
View File
@@ -34,8 +34,13 @@ _port: int | None = None
# only files the user explicitly registered are reachable. Trailing slash so the
# token starts immediately after it.
_LOCAL_FILE_PREFIX = "/_geolibre_local/"
_RASTER_TILE_PREFIX = "/_geolibre_tiles/"
# token -> absolute filesystem path for files registered via register_local_file.
_local_files: dict[str, Path] = {}
# token -> (path, bands, colormap, rescale) for Colab-safe XYZ rendering.
_raster_tiles: dict[
str, tuple[Path, tuple[int, ...] | None, str | None, tuple[tuple[float, float], ...] | None]
] = {}
# Chunk size for streaming a (possibly ranged) local file response.
_STREAM_CHUNK = 64 * 1024
@@ -134,6 +139,66 @@ class _QuietHandler(SimpleHTTPRequestHandler):
finally:
handle.close()
def _match_raster_tile(
self,
) -> (
tuple[
Path,
tuple[int, ...] | None,
str | None,
tuple[tuple[float, float], ...] | None,
int,
int,
int,
]
| None
):
"""Resolve a tokenized ``z/x/y.png`` request to its raster settings."""
path = unquote(urlsplit(self.path).path)
if not path.startswith(_RASTER_TILE_PREFIX):
return None
parts = path[len(_RASTER_TILE_PREFIX) :].split("/")
if len(parts) != 4 or not parts[3].endswith(".png"):
return None
token, z_raw, x_raw, y_raw = parts
try:
z, x, y = int(z_raw), int(x_raw), int(y_raw[:-4])
except ValueError:
return None
with _lock:
registered = _raster_tiles.get(token)
if registered is None:
return None
return (*registered, z, x, y)
def _serve_raster_tile(self, matched: tuple, *, head_only: bool) -> None:
"""Render one PNG tile from a registered kernel-side raster."""
file_path, bands, colormap, rescale, z, x, y = matched
try:
from rio_tiler.colormap import cmap
from rio_tiler.io import Reader
with Reader(file_path) as source:
image = source.tile(x, y, z, indexes=bands)
if rescale:
image.rescale(rescale)
color_map = cmap.get(colormap) if colormap and len(image.data) == 1 else None
payload = image.render(img_format="PNG", colormap=color_map)
except ImportError:
self.send_error(500, "rio-tiler is required for Colab raster tiles")
return
except Exception as exc: # pragma: no cover - rio-tiler error text varies
self.send_error(500, f"Could not render raster tile: {exc}")
return
self.send_response(200)
self.send_header("Content-Type", "image/png")
self.send_header("Content-Length", str(len(payload)))
self.send_header("Cache-Control", "no-store")
self.send_header("Access-Control-Allow-Origin", "*")
self.end_headers()
if not head_only:
self.wfile.write(payload)
@staticmethod
def _parse_single_range(range_header: str, size: int) -> tuple[int, int] | None:
"""Parse the first byte range of a ``Range: bytes=`` header.
@@ -166,6 +231,10 @@ class _QuietHandler(SimpleHTTPRequestHandler):
return start, min(end, size - 1)
def do_GET(self) -> None: # noqa: N802 - http.server naming
tile = self._match_raster_tile()
if tile is not None:
self._serve_raster_tile(tile, head_only=False)
return
matched = self._match_local_file()
if matched is not None:
self._serve_local_file(matched, head_only=False)
@@ -173,6 +242,10 @@ class _QuietHandler(SimpleHTTPRequestHandler):
super().do_GET()
def do_HEAD(self) -> None: # noqa: N802 - http.server naming
tile = self._match_raster_tile()
if tile is not None:
self._serve_raster_tile(tile, head_only=True)
return
matched = self._match_local_file()
if matched is not None:
self._serve_local_file(matched, head_only=True)
@@ -311,6 +384,33 @@ def register_local_file(path: str | os.PathLike[str]) -> str:
return f"{base}{_LOCAL_FILE_PREFIX.lstrip('/')}{token}/{quote(file_path.name)}"
def register_raster_tiles(
path: str | os.PathLike[str],
*,
bands: list[int] | None = None,
colormap: str | None = None,
rescale: list[list[float]] | None = None,
) -> str:
"""Register a raster and return a tokenized XYZ URL on the app server."""
file_path = Path(path).expanduser().resolve()
if not file_path.is_file():
raise ValueError(f"Local file not found: {path}")
normalized_bands = tuple(bands) if bands else None
normalized_rescale = tuple(tuple(pair) for pair in rescale) if rescale else None
settings = (file_path, normalized_bands, colormap, normalized_rescale)
with _lock:
if _base_url is None:
raise RuntimeError("The GeoLibre static server is not running; create a Map first.")
token = next(
(tok for tok, registered in _raster_tiles.items() if registered == settings), None
)
if token is None:
token = secrets.token_urlsafe(16)
_raster_tiles[token] = settings
base = _base_url
return f"{base}{_RASTER_TILE_PREFIX.lstrip('/')}{token}/{{z}}/{{x}}/{{y}}.png"
def unregister_local_file(path: str | os.PathLike[str]) -> None:
"""Drop the registration for ``path``, if it has one.
@@ -334,6 +434,9 @@ def unregister_local_file(path: str | os.PathLike[str]) -> None:
for token, registered in list(_local_files.items()):
if registered == file_path:
del _local_files[token]
for token, registered in list(_raster_tiles.items()):
if registered[0] == file_path:
del _raster_tiles[token]
def app_port() -> int | None:
+38 -3
View File
@@ -27,7 +27,13 @@ import traitlets
from . import authoring as _authoring
from . import project as _project
from ._server import app_port, register_local_file, serve_app, unregister_local_file
from ._server import (
app_port,
register_local_file,
register_raster_tiles,
serve_app,
unregister_local_file,
)
from .basemaps import resolve_basemap
from .polyline import polyline_to_geojson
@@ -1666,6 +1672,7 @@ class Map(anywidget.AnyWidget):
*,
tile_size: int = 256,
attribution: str | None = None,
bounds: list[float] | None = None,
**style: Any,
) -> str:
"""Add a raster XYZ tile layer.
@@ -1675,6 +1682,7 @@ class Map(anywidget.AnyWidget):
name: Layer display name.
tile_size: Tile size in pixels.
attribution: Optional attribution string.
bounds: Optional ``[west, south, east, north]`` request bounds.
**style: Style overrides.
Returns:
@@ -1686,6 +1694,7 @@ class Map(anywidget.AnyWidget):
url,
tile_size=tile_size,
attribution=attribution,
bounds=bounds,
**style,
)
)
@@ -1869,7 +1878,7 @@ class Map(anywidget.AnyWidget):
def add_cog(
self,
url: str,
url: str | os.PathLike[str],
name: str = "COG",
*,
bands: list[int] | None = None,
@@ -1896,6 +1905,31 @@ class Map(anywidget.AnyWidget):
Returns:
The id of the added layer.
"""
if self._running_on_colab() and not (
isinstance(url, str) and url.startswith(("http://", "https://"))
):
tile_url = register_raster_tiles(
url,
bands=bands,
colormap=colormap,
rescale=rescale,
)
try:
import rasterio
from rasterio.warp import transform_bounds
with rasterio.open(url) as dataset:
bounds = list(
transform_bounds(
dataset.crs,
"EPSG:4326",
*dataset.bounds,
densify_pts=21,
)
)
except Exception: # pragma: no cover - tile renderer reports invalid rasters
bounds = None
return self.add_tile_layer(tile_url, name, bounds=bounds, **style)
return self._add_layer(
_project.cog_layer(
name,
@@ -1965,7 +1999,8 @@ class Map(anywidget.AnyWidget):
raise TypeError("add_raster() missing required argument: 'source'")
raster_source = source
if not isinstance(source, (str, os.PathLike)):
is_xarray = not isinstance(source, (str, os.PathLike))
if is_xarray:
raster_source = self._materialize_xarray(source, array_args)
elif array_args:
warnings.warn(
+4
View File
@@ -454,6 +454,7 @@ def tile_layer(
*,
tile_size: int = 256,
attribution: str | None = None,
bounds: list[float] | None = None,
**style: Any,
) -> dict[str, Any]:
"""Build a raster XYZ tile layer (e.g. an ``{z}/{x}/{y}`` template).
@@ -463,6 +464,7 @@ def tile_layer(
url: The XYZ tile URL template.
tile_size: Tile size in pixels (typically 256).
attribution: Optional attribution string.
bounds: Optional ``[west, south, east, north]`` request bounds.
**style: Style overrides merged into the default layer style.
Returns:
@@ -477,6 +479,8 @@ def tile_layer(
}
if attribution:
source["attribution"] = attribution
if bounds:
source["bounds"] = bounds
layer["source"] = source
layer["metadata"] = {"sourceKind": "xyz-url"}
return layer
+32
View File
@@ -685,6 +685,38 @@ def test_add_raster_xarray_round_trip_with_real_rioxarray(monkeypatch, m):
assert not captured["path"].exists()
def test_add_raster_xarray_uses_xyz_tiles_on_colab(monkeypatch, m):
DataArray, _ = _fake_xarray_modules(monkeypatch)
monkeypatch.setattr(Map, "_running_on_colab", staticmethod(lambda: True))
captured = {}
monkeypatch.setattr(
gmod,
"register_raster_tiles",
lambda path, **options: (
captured.update(path=path, options=options)
or "http://127.0.0.1:1234/_geolibre_tiles/token/{z}/{x}/{y}.png"
),
)
m.add_raster(
DataArray(),
name="Temperature",
bands=[1],
colormap="turbo",
rescale=[[-5, 35]],
)
layer = _last_layer(m)
assert layer["type"] == "xyz"
assert layer["name"] == "Temperature"
assert "_geolibre_tiles/token/{z}/{x}/{y}.png" in layer["source"]["tiles"][0]
assert captured["options"] == {
"bands": [1],
"colormap": "turbo",
"rescale": [[-5, 35]],
}
def test_add_raster_accepts_deprecated_url_keyword(m):
"""The pre-rename `url=` keyword still works, with a DeprecationWarning."""
with pytest.warns(DeprecationWarning, match="add_raster\\(url=...\\) is deprecated"):
+28
View File
@@ -29,6 +29,7 @@ def served(tmp_path, monkeypatch):
monkeypatch.setattr(_server, "_base_url", None)
monkeypatch.setattr(_server, "_port", None)
monkeypatch.setattr(_server, "_local_files", {})
monkeypatch.setattr(_server, "_raster_tiles", {})
bundle = tmp_path / "app"
bundle.mkdir()
@@ -132,6 +133,33 @@ def test_register_same_file_reuses_token(served):
assert len(_server._local_files) == 1
def test_registered_raster_tile_returns_png(served, tmp_path):
rasterio = pytest.importorskip("rasterio")
pytest.importorskip("rio_tiler")
np = pytest.importorskip("numpy")
from rasterio.transform import from_bounds
raster = tmp_path / "world.tif"
with rasterio.open(
raster,
"w",
driver="GTiff",
width=8,
height=8,
count=1,
dtype="float32",
crs="EPSG:4326",
transform=from_bounds(-180, -85, 180, 85, 8, 8),
) as dst:
dst.write(np.arange(64, dtype="float32").reshape(1, 8, 8))
template = _server.register_raster_tiles(raster, bands=[1], colormap="turbo", rescale=[[0, 63]])
status, headers, body = _get(template.format(z=0, x=0, y=0))
assert status == 200
assert headers["Content-Type"] == "image/png"
assert body.startswith(b"\x89PNG\r\n\x1a\n")
def test_unregister_local_file_drops_the_token(served, tmp_path):
url, _payload = served
assert len(_server._local_files) == 1
+6 -6
View File
@@ -92,11 +92,10 @@ m.add_polyline(...)
### In-memory xarray rasters
`add_raster` also accepts an `xarray.DataArray` or `xarray.Dataset`, which needs
`pip install "geolibre[raster]"` (xarray + rioxarray + rasterio). The kernel
writes it to a temporary GeoTIFF — Cloud-Optimized by default — and serves that
the same way it serves a local file, so every caveat in the next section applies
to it as well. Keep the default: the app range-reads a COG directly, and a plain
GeoTIFF makes it warn and convert the whole file in the browser first.
`pip install "geolibre[raster]"` (xarray + rioxarray + rasterio + rio-tiler). The kernel
writes it to a temporary GeoTIFF — Cloud-Optimized by default. Locally the app
range-reads that COG directly. In Colab, whose proxy does not preserve COG byte
ranges, rio-tiler renders ordinary PNG XYZ tiles in the kernel instead.
```python
m.add_raster(data_array, name="Temperature", colormap="viridis")
@@ -136,7 +135,8 @@ durable.
`add_raster` / `add_cog` accept a **local** GeoTIFF path on the kernel host: the
bundled localhost server serves it so the app can read it. This works directly
in local Jupyter and VS Code, through Colab's built-in kernel port proxy, and on
in local Jupyter and VS Code. Colab uses kernel-rendered PNG XYZ tiles instead
of browser COG reads. JupyterHub works through the kernel port proxy when
JupyterHub when `jupyter-server-proxy` is available. A deployment that can only
serve the static app extension cannot expose kernel files, so use a hosted URL.
The served URL is session-scoped, so a saved project or exported HTML will not
+19
View File
@@ -77,6 +77,25 @@ describe("rewriteProxiedLocalFileUrls", () => {
assert.equal(rewritten.layers[0].sourcePath, rewritten.layers[0].source.url);
});
it("routes kernel-rendered XYZ tiles through the Colab proxy", () => {
const template = "http://127.0.0.1:41123/_geolibre_tiles/token/{z}/{x}/{y}.png";
const project = {
layers: [{ source: { tiles: [template], url: template } }],
};
const rewritten = rewriteProxiedLocalFileUrls(
project,
"https://session-41123-colab.googleusercontent.com/",
41123,
);
const expected =
"https://session-41123-colab.googleusercontent.com/" +
"_geolibre_tiles/token/{z}/{x}/{y}.png";
assert.equal(rewritten.layers[0].source.url, expected);
assert.deepEqual(rewritten.layers[0].source.tiles, [expected]);
});
it("does not rewrite unrelated, wrong-port, or already-remote URLs", () => {
const project = {
layers: [