Roadmap

Ordered by impact. Items move to the docs proper when they land; this page must not describe shipped features as future work.

1. Materialize-on-browse with a disk cache. Opt-in, off by default. Back the cache with a real zarr on local or shared storage so it persists, is shared across workers, and doubles as a partially computed output. Add a background filler that expands outward from requested chunks: the viewer becomes the job scheduler. (Ordering the work by what clients ask for, and dropping what they stop waiting for, has shipped; see caching. Inference ops would join its queues. A first piece has shipped in both engines: an explicit write of a dataset, or a box of it, into a store as the frontends serve it, from Python and the command line (chunkmirage materialize, formats), and in the browser into a folder the visitor picks or memory, its chunks computed in the background behind the viewer's (formats). The cache on disk itself, in either engine, its sharing across workers and the filler that grows it from what is browsed are still to do.)
2. Lazy-compute story for non-viewer clients. Demonstrate dask/tensorstore reading a served pipeline and running downstream analysis with no intermediate written. (The read itself is shown in getting started; the analysis on top of it is what remains.)
3. DAG pipelines, multi-source ops. Named stages with fan-out (one model, many post-processors) and a Combine op taking another pipeline as input. Unlocks masking, the difference of any two sources (two models' outputs, two conditions), registration overlays. (Two pieces have shipped: stack:// serves images on one grid as channels and an op such as contacts consumes them, and dataset:// reads another served dataset in-process through its stage cache, an edit upstream rebuilding what reads it, so passes of a computation chain without writing anything to disk between them; see formats and dataset sources. Named stages within one pipeline, fan-out and a Combine op remain.) With it, structured sources: a pipeline's source as a JSON object as well as a URL, such as {"register": {"moving": ..., "fixed": ..., "affine": [...]}}, typed by the same schema (chunkmirage schema) so nested sources and long parameters need no escaping and the browser engine, an MCP server or an agent build specs instead of strings. The URL schemes (register://, scene://, ...) stay as shorthand for the command line and links; register:// becomes a two-input node rather than a source.
4. More registration inputs, live landmarks, fusion. Readers for bigstream/EASI-FISH (affine.mat plus a deform zarr), BigWarp landmarks and ANTs/ITK fields into the same transform model; landmark pairs drawn in Neuroglancer, fitted to a thin-plate spline and served live; several scene images fused into one volume (for example through multiview-stitcher). Resampling through OME-Zarr 0.6 transformations has shipped as scene:// sources, and tiles stitched by interest points and fused as read as stitch:// sources (BigStitcher projects of OME-Zarr tiles; other loaders, multi-view registration and saving the result back to the project are still to do).
5. Adaptive caching. Measure stage compute time at runtime and cache automatically when it exceeds a threshold, removing the manual cache flag.
6. MCP surface. list_ops, set_pipeline, try (a spec on one block: its shape, dtype, value range and time, or the traceback), neuroglancer_link. Off by default, local only. (The REST API, /api/events stream, control page and python-neuroglancer viewer it would wrap already exist; see Interactivity. Ops from source need no tool of their own: the python op carries its code in the spec.)
7. Deployment hardening. Shared-cache multi-worker mode. (--https with an auto-generated self-signed certificate, and --token on /api/*, have shipped; see the CLI reference.)
8. Own-hosted Neuroglancer with a service worker. The zero-install browser demo with WebGPU ops and ONNX Runtime Web inference, sharing the JSON pipeline spec with the Python server. See FAQ for why it cannot target the hosted appspot viewer. A first piece exists: web/, TypeScript whose types are generated from chunkmirage's JSON Schema (chunkmirage schema), fits register://'s deformable registration on the viewer's GPU, reading OME-Zarr straight from its URLs (see the design notes), and serves the registered volume to Neuroglancer through a service worker, computed by web workers. The docs site hosts it with its own Neuroglancer build, and a gallery of demos whose pipelines run chunkmirage's own ops in the page through Pyodide (see Demos). A page of another project builds on the engine, which loads Python only for the views that need it (see the design notes). What remains, in order:
   • Cards built from the schema: a form for each card's parameters, so a visitor edits the pipeline as the control page edits a server's.
   • Ops on the GPU. The pointwise and filter ops as WebGPU compute shaders, each with a parity test against Python on seeded synthetic data; synthetic:// in TypeScript (a hash of world coordinates, exact at every scale) for the same reason. (Through Pyodide, synthetic://, label and the rest already run in the page, and the fractal is a computed pyramid 2^28 voxels across that costs nothing.) The same volume served as N5 and precomputed by the service worker, beside its zarr v2 and v3.
   • A layer of how well refine has fitted each block (its correlation before and after), in both engines. (Where any dataset's stage has computed, the full-resolution result or a coarse preview, is served by coverage:// in Python.)
   • User code on the GPU. Array functions exported to ONNX run in both engines (ONNX Runtime and ONNX Runtime Web). (The CPU tier has shipped: the python op runs in the page through Pyodide and natively in Python, where a server takes it through the REST API only with --allow-code; the demos' chat asks a model for it. In the browser, an ONNX Runtime Web stage runs a model on WebGPU as one stage of a view a page computes in stages, with the model's output kept while its postprocessing changes; see the design notes. Python's ONNX Runtime op is still to do.)
   • A CI job that runs the page headless and diffs its chunks against Python's, so the engines cannot drift. Whether the page's WebGPU shaders can also serve Python (wgpu-py), as one implementation of the solver for both, is still to be measured against PyTorch.
9. Stages on the GPU. Heavy work gains on a GPU (block matching, models, large kernels); cheap ops do not: a chunk is read and encoded on the CPU, so an op moved on its own would copy its block there and back for work that takes as long as the copies. So a stage would move its block to the device once, run its ops there and move it back once. Ops get a numpy or CuPy array and use that array's library (chunkmirage.gpu's array_module and ndimage; cupyx.scipy.ndimage takes scipy.ndimage's calls, which most ops make), saying which they support (as packages says what they import); the rest run on the CPU, with slots bounding the calls as now. The rest of the core's own computation (downsample, the overlap blend) is written the same way, and contrib's ops move over heaviest first (register:// already solves on a GPU). (The resampler of warp:// and scene:// has moved: serve --device, see formats.)

Untapped potential

• Every zarr reader is a client. dask, xarray, tensorstore, napari, Fiji and cellmap-analyze can open a served pipeline as a lazy array, making the server a general lazy compute node with a shared cache.
• Derived datasets as specs. A pipeline spec is a few hundred bytes; publishing a derived view of a public dataset costs no storage, and stateless chunks could run serverless.
• Invalidation when outside state changes. An op's cache_token and a dataset refresh give a stage new keys when the files or weights it depends on change, and viewers refetch (caching). Loops built on it, such as annotating, fine-tuning and serving the new model, belong to the applications that use chunkmirage (cellmap-flow's, for models), not to chunkmirage.
• Claude in the loop. With Neuroglancer's screenshot endpoint plus the REST API, an agent can render, look, adjust and repeat.