Coverage for gws-app/gws/base/grabber/__init__.py: 100%
2 statements
« prev ^ index » next coverage.py v7.16.2, created at 2026-10-05 13:35 +0200
1"""Raster grabber.
3A grabber provides raster images for a layer: it fetches them from a source,
4aligns them to a tile grid, reprojects them into the target CRS, stores tiles
5and reads them back. It runs in-process; there is no separate tile server,
6no generated configuration and no HTTP between GWS and the tile machinery.
8Submodules
9----------
11- ``core`` - the base grabber (`core.Object`) and its construction options
12 (`core.Options`). It implements the public ``get_*`` API: tile and box
13 lookup in the stores, block locking, mosaicking from stored tiles, and the
14 shared helpers for encoding, decoding, warping and empty images.
15- ``box`` - the base grabber for sources that render arbitrary boxes. It
16 meta-tiles blocks, splits large requests into chunks and requests boxes in
17 the source CRS.
18- ``tile`` - the base grabber for sources addressed as tile pyramids. It
19 mosaics source tiles of the best matching matrix and warps them onto the
20 requested box.
22The interface is `gws.Grabber`; the tile stores are in ``gws.gis.cache.store``
23and the grid functions in ``gws.lib.grid``.
25Grids
26-----
28There is one fixed tile grid per CRS, shared by all grabbers in that CRS
29(``gws.lib.grid``). Projected CRS use the web mercator square as frame, one
30256px tile at level 0 and the resolution ladder ``156543.03 / 2^z``;
31geographic CRS use ``(-180, -90, 180, 90)``, two tiles at level 0 and
32``0.703125 / 2^z`` degrees. The same level means the same ground resolution
33in every projected CRS, and web mercator tile sources align 1:1 with the
34EPSG:3857 grid. The ladder has no bottom: any resolution is served by
35downsampling from the coarsest level that is not coarser than the target.
36Levels beyond ``MAX_LEVEL`` (30) are not served.
38Each grabber restricts the grid to the layer's extent: a tile range per level
39(``tile_range_for_level``), derived from the layer's WGS extent clipped to
40the area of use of the CRS. Tiles outside the range are transparent.
42Grabbers and layers
43-------------------
45An image layer holds one grabber per CRS the application supports
46(``layer.grabbers``, keyed by SRID), created in
47``base.layer.image.Object.post_configure_grabbers`` via the layer type's
48``create_grabber``. A layer whose extent does not intersect a CRS gets no
49grabber for it. Grabbers are plain objects, not nodes.
51Concrete grabbers live next to their provider (``plugin/.../grabber.py``) and
52extend one of two base classes:
54- `box.Object` for sources that render arbitrary boxes (WMS, QGIS server,
55 in-process MapServer for raster and MBTiles layers). Subclasses implement
56 ``fetch_box_as_bytes`` and ``fetch_box_as_image``, one source request each.
57- `tile.Object` for sources addressed as tile pyramids (tile services, WMTS).
58 Subclasses provide ``sourceTms`` and ``fetch_tile_as_bytes``.
60Fetching and composing
61----------------------
63The public API (`gws.Grabber`) is ``get_tile_as_bytes/image``,
64``get_tiles_as_bytes/image_dict`` and ``get_box_as_bytes/image``. ``get``
65may hit the store, ``compose`` assembles from parts and may issue several
66source requests, ``fetch`` is exactly one source request. Both return forms
67exist because encoding and decoding are not free; a caller asks for the
68form it needs and never pays a round trip.
70Box sources are meta-tiled: a block of ``requestTiles`` x ``requestTiles``
71tiles is rendered as one request with a ``requestBuffer`` (pixels) around it,
72then cut into tiles. This gives fewer source requests and consistent label
73placement across tile seams. Requests larger than the provider's
74``maxRequestPixels`` are split into chunks with the same buffer.
76Tile sources fetch the source tiles covering a target tile at the best
77matching source matrix (never upscaling), mosaic and warp them. Fetched
78source tiles are kept in the ephemeral store, so neighbouring target tiles
79reuse them. There is no meta-tiling for tile sources.
81Reprojection is done by the base: source images are fetched in the source
82CRS, clipped to the source CRS area of use, and warped into the target grid
83with GDAL. A box outside the source area is transparent. For box sources, a
84cross-CRS source request is capped at ``box.MAX_SOURCE_PIXEL_RATIO`` source
85pixels per target pixel per side.
87Boxes at a stored level are mosaicked from stored tiles; otherwise (uncached
88layers, levels beyond ``cache.maxLevel``, dynamic requests) they are composed
89directly at the exact resolution, which keeps print output free of pyramid
90resampling.
92Caching
93-------
95Every grabber has two tile stores with the same layout
96(``gws.gis.cache.store``): the persistent store under
97``MAP_CACHE_DIR/<cache name>_<srid>`` holds levels up to ``cache.maxLevel``
98when the layer has ``withCache`` and a positive ``cache.maxAge``; everything
99else goes to an ephemeral store under the ephemeral directory with a fixed
100lifetime of ``EPHEMERAL_MAX_AGE`` (60 s), so that a block is composed once
101per viewport even without a cache.
103The cache name is a content hash of the source binding (provider, source
104layers, style, format, extent, request shaping), or ``cache.name`` when
105configured; layers with identical bindings share stores. ``cache.crs``
106limits persistent caching to the listed CRS.
108Block composition is locked on block identity (``gws.u.server_lock``), so
109concurrent requests for tiles of one block trigger one source fetch. A lock
110busy for longer than ``BLOCK_LOCK_TIMEOUT`` (60 s) yields the transparent tile
111and a warning.
113Dynamic requests
114----------------
116``params`` (e.g. the visible leaf set of a composite QGIS layer, passed by the
117layer as ``lri.renderParams``) marks a request as dynamic. Dynamic requests
118never touch the persistent store; they use an ephemeral store keyed by a
119hash of the params and are meta-tiled and locked under the same key. Layers
120must pass distilled params only, or every request goes uncached.
122Configuration
123-------------
125On the layer:
127- ``withCache`` (default true) and ``cache``: ``maxAge`` (7d), ``maxLevel``
128 (18, a level of the global grid), ``requestTiles`` (4), ``requestBuffer``
129 (64), ``crs``, ``name``.
130- ``imageFormat``: the stored and returned format (default PNG8); JPEG and
131 other opaque formats render transparency as a white background.
132- ``display``: ``tile`` (grabber tiles), ``box`` (grabber boxes) or, for
133 ``tile`` and ``wmts`` layers, ``client`` (the browser fetches source tiles
134 directly; requires a source grid in the map CRS).
136Per provider: ``maxRequestPixels`` for WMS (4096); MapServer-backed sources
137use 9000.
139Cache management is ``gws cache status | seed | drop`` (``gws.gis.cache``),
140which works on the layers' grabbers and stores.
142Example
143-------
145A box grabber for a source that renders images on request::
147 class Object(gws.base.grabber.box.Object):
148 def __init__(self, opts: gws.base.grabber.Options, provider, sourceCrs: gws.Crs):
149 super().__init__(opts)
150 self.provider = provider
151 self.sourceCrs = sourceCrs
152 self.maxRequestPixels = provider.maxRequestPixels
154 def fetch_box_as_bytes(self, bounds, w, h, params=None):
155 return self.provider.get_map(bounds, w, h, self.mimeType)
157 def fetch_box_as_image(self, bounds, w, h, params=None):
158 return self.to_image(self.fetch_box_as_bytes(bounds, w, h, params))
160The layer creates it in ``create_grabber`` and reads images from it::
162 def create_grabber(self, opts):
163 return my_grabber.Object(opts, provider=self.provider, sourceCrs=self.sourceCrs)
165 grabber = layer.grabbers[crs.srid]
166 tile = grabber.get_tile_as_bytes((x, y, z))
167 img = grabber.get_box_as_image(extent, 800, 600)
169Notes and open issues
170---------------------
172- Source requests for the tiles covering one target tile are sequential;
173 cross-CRS tile sources pay several upstream round trips per cold tile.
174- Fetched source tiles stay in the ephemeral store for up to two hours,
175 independent of the layer's cache settings.
176- Meta-tiled composite QGIS layers need label placement that does not depend
177 on the request extent (polygon labels: centroid of the whole polygon) and a
178 ``requestBuffer`` at least as wide as the widest label, or labels differ
179 across block seams.
180- A source answering with an image of unexpected size raises
181 ``gws.ExternalServiceError`` (empty tile, log entry) rather than being
182 padded.
183- The background color for opaque formats is not configurable.
184- The own WMTS service composes a box per tile instead of reading stored
185 tiles directly.
186"""
188from .core import Options, Object
189from . import box, tile