Changelog
0.10.2 (2026-11-13)
0.10.1's fix, on tensorstore 0.1.86. No placement change, no unit table change, and no default changes.
- A
gs://open with a stale Google login falls back to the public https URL on tensorstore 0.1.86 too. 0.10.1 knew the failed login by the source locations in tensorstore's error, which 0.1.86 leaves out, so with the newest tensorstore (what a fresh install gets) the open still raised. The token endpoint's own error code (invalid_grant, and the other OAuth 2.0 codes) is now matched as well. On 0.1.86 a token endpoint out of reach is raised as any unreachable host is, not passed over. - CI also runs the tests on the newest release of every dependency, weekly and on every pull request; 0.10.1 passed on the lock's tensorstore 0.1.78.
0.10.1 (2026-11-13)
A fix for gs:// with a stale Google login. No placement change, no unit table change,
and no default changes.
- A
gs://open falls back to the bucket's public https URL when Google's default credentials fail to refresh. An expired gcloud login (a Workspace reauth policy) is refused with HTTP 400invalid_grant, which tensorstore does not report as a refusal, so the open raised instead of reading a public bucket. Any error from getting the access token now counts as a refusal, and so does a token endpoint out of reach. - When every way in fails after a refusal, the error raised carries the first refusal as its cause, so a stale login stays in the traceback of a private bucket's 403.
0.10.0 (2026-11-03)
Axes in another order, on request. No placement change, no unit table change, and no
default changes: without axes_order, every array opens as it did in 0.9.
open,open_pyramidandmetadatatakeaxes_order:"stored"(the default, for good),"zyx"(the axes named z, y and x last, in that order, the others first in stored order) or a sequence naming each axis once. An OME-Zarr written x, y, z (TensorSwitch's) then reads as z, y, x, and a channel-last array channel first. The names move with the data, and shape, chunks, shards, units, voxel size and translation with them, in.ts,meta,grid, reads, writes,blocks, a pyramid's placements andlevel_meta, level choice by voxel size,base_voxel_size,reopenand a pickle.open_tensorstorestays in the store's order.- On such a handle,
chunk_key,parse_chunk_keyandlist_chunkstake and give indices in the presented order (a selected axis in its stored place); the keys are the store's.add_leveltakes its per-axis arguments in the pyramid's order and creates the array in the group's.
0.9.1 (2026-10-26)
A fix for exit. No placement change, no unit table change, and no default changes.
- The watchdog thread stops at exit, before the interpreter finalizes. On Python up to
3.13 a process could abort once all its work was done ("terminate called without an
active exception", exit 134) when finalization began while the thread was inside a
tensorstore call; an LSF job that did this looked failed. Now an
atexithandler, which runs once the program's other threads are joined, ends the thread and waits for it, and no new one starts after that.
0.9.0 (2026-10-23)
A job script can size each worker's threads. No placement change, no unit table change, and
no default changes: with WORLDARRAY_THREADS unset and no file_io_floor, every context
is what 0.8 made.
thread_budget()readsWORLDARRAY_THREADSahead ofOMP_NUM_THREADSand the job's slots, for a job script to give each dask worker its share of the job's cores whereOMP_NUM_THREADSstays 1 for numpy andLSB_DJOB_NUMPROCcounts the whole job (for examplemax(1, ncpus // processes)). A handle opened on the driver on the shared context is sized in each worker by that worker's value. Anything but a positive integer raisesValueError; blank is unset.shared_context(file_io_floor=)sets the least the file I/O pool takes under the thread budget (defaultFILE_IO_FLOOR, 8);1keeps file I/O at the budget. A pickle carries it, and withthreads=Northreads=0it is ignored.
0.8.0 (2026-10-23)
Faster writes from the contexts the package makes, and the create open timed out. No
placement change and no unit table change. Requires tensorstore 0.1.68 or later (was
0.1.60), the first with the file_io_locking setting.
shared_contextandcapped_contextwrite without fsync (file_io_sync: false) and without OS file locks (file_io_lockingmode"none");fsync=Trueandlocking="os"restore tensorstore's defaults. A whole-unit write stays atomic, a temporary file renamed into place; only concurrent read-modify-writes of one unit (partial=True) lose their guarantee, which was never given (Behaviour 13). fsync cost 6.5 ms a chunk on a local disk and nothing on NFS; the locks cost about 10 ms a chunk on NFS.- The default context's file I/O pool is
thread_budget()or 8, whichever is larger; decoding stays atthread_budget(). File I/O threads wait on storage rather than compute, and one of them wrote a block's chunk files one NFS round trip after another.capped_context(file_io=)sizes it apart fromconcurrency(default unchanged). - From one-thread workers through the default context, against 0.7: a 64^3 label chunk to
NFS 23 to 12.2 ms (zarr-python 2: 13.1), an 8-chunk label block 159 to 66.5 ms
(zarr-python 2: 102), an 8-chunk EM block 330 to 202 ms; to a local disk a label block
51.5 to 4.1 ms. Through
capped_context(1, network=False)the NFS label block went 166 to 91 ms;file_io=8gives it the rest. - The open that creates an array is timed out and retried under
retries(Behaviour 6). A retry that finds the array made by an attempt of its own that timed out adopts it after the usual comparison, even underexists="error". That open keeps OS locking, so create-if-absent stays atomic.
0.7.1 (2026-10-23)
Faster opens over http. No placement change, no unit table change, no API change.
- The documents a call needs from one directory are read at once instead of in turn. An
http open of a zarr v2 level made its eight requests one after another (five probe keys,
the level's
.zattrs, the parent'szarr.jsonand.zattrs); now they go in three rounds. At 10 ms an answer,wa.opentook 98 ms and takes 52 (cellmap-flow's measurement). A server that never answers still costs one key's attempts, not five.
0.7.0 (2026-10-23)
Opens time out. No placement change and no unit table change.
- Every metadata document read or written and every store opened is timed out and
retried under
Retry, as data reads are (Behaviour 6): by default 5 s times the attempt, ten attempts. A server that accepts a connection and never answers heldwa.openandwa.metadataforever before; it now raisesTimeoutErrornaming the URL and the attempts (cellmap-flow, 2026-10-23). The policy is the call'sretries, whichwa.metadata,wa.detectandwa.open_tensorstorenow take aswa.open,wa.open_pyramidandwa.createalready did. Listings and the open that creates an array are not timed out.
0.6.0 (2026-10-23)
One new argument, at chunkmirage's request. No placement change and no unit table change; nothing changes for a caller that does not pass it.
wa.shared_context(cache_bytes, threads=):None(the default) limits the decoding and file I/O pools tothread_budget()as in 0.5, a positive number limits them to that many, and0writes no limit, so tensorstore sizes them itself (every CPU on the node). Each(cache_bytes, threads)is one watched context for the process, so a server can share one cache pool across every store it opens at the thread count it chose, without building and watching its own context. A pickled handle on such a context comes back on the same one;threads=Nonestill lets the other process size by its own job.- Documented what a pool of one thread costs (spec Behaviour 7,
capped_context_spec): nothing on one-chunk blocks; on 8-chunk blocks, half the speed of 4 in a two-slot job but only 7 to 19% slower in a one-slot one (c-elegans on /nrs), so a worker per slot loses little at the default.
0.5.0 (2026-10-23)
Two bugs that hung or crashed cellmap-analyze 0.6 on /nrs (salamander's and c-elegans' runs, 2026-10-22), and tensorstore's threads sized to the job. No placement change and no unit table change.
- A read whose first attempt timed out raised
IndexErrorinstead of retrying: the warning took the first line of the error's message, and tensorstore'sTimeoutError()has none. The warning now names the error's type when the message is empty, and the error after the last attempt names the operation, the path and the attempts. - Every context the package makes or opens a store on is watched: once a second a daemon
thread gives its decoding and file I/O pools one no-op task each (an in-memory copy and
a local read of a missing key; no store is touched). tensorstore up to at least 0.1.85
can strand a queued task: a pool worker idle for 20 ms exits as the task arrives, and
the task never runs (fixed upstream in google/tensorstore ed51718085, in no release
yet). With
data_copyandfile_iocapped at 1, as cellmap-analyze's workers were, dozens of workers per run waited forever inwrite_indexwith no tensorstore thread alive, and reads timed out into theIndexErrorabove. One more task on the same pool wakes it, which salamander's and c-elegans' sessions proved on live stuck workers, so a stranded task now runs within about a second, at any concurrency, whoever waits on it: worldarray,vol.tsdirectly orawait. Reproduced on purpose on tensorstore 0.1.75, 0.1.78 and 0.1.85 (opt-intests/test_lost_wakeup.py): cellmap-analyze's block loop gave 29IndexErrors and 55 writes stuck past 20 s on 0.4.0, none on 0.5.0, slowest 0.98 s. Spec open question 14 says when this goes. write_index,writeandwrite_chunkare timed out and retried as reads are (Behaviour 6), by a newretries=argument or the handle'sretries. A write used to wait on its future with no timeout at all.- Without
context, the shared context's decoding and file I/O pools are limited to the newwa.thread_budget():OMP_NUM_THREADSwhen set (dask sets 1 in each worker), else the job's slots (LSB_DJOB_NUMPROC,SLURM_CPUS_PER_TASK,NSLOTS), else the CPUs the process may run on. A server that wants more threads than its job's slots asks for more slots or passes its own context. tensorstore alone sizes both to every CPU on the node, whateverbsub -nasked for, so a dask worker's tensorstore now takes the same one thread as its numpy (pinned to one CPU, tensorstore's default context started 43 to 45 pool threads, the capped one 4 to 5). Against cellmap-analyze 0.5.0's own block I/O on NFS (128^3 uint64 label blocks, each one chunk), its 0.6.1 on this release reads a block in 26 ms instead of 40 and writes it in 45 ms instead of 62 from one process at one thread per pool, and about 1.15 times faster from 8. Speed gains grow with blocks of several chunks and more threads.
0.4.0 (2026-10-22)
cellmap-analyze's three asks, so it can call worldarray directly with no wrapper, and two of cellmap-flow's. No placement change and no unit table change.
vol.read_onto(grid, begin=None, end=None, ...)reads voxels[begin, end)of aGrid's spatial axes from the array, exactly that shape (Behaviour 20). It is thereadof the grid's index box at the grid's voxel size anchored at its corner, so at the array's own voxel size nothing is resampled, whatever the sub-voxel offset, and otherwise the array is resampled onto the grid. The grid's units are converted.- A pickled handle, and
reopen, use the path resolved against the working directory at open, so a worker that changes directory before unpickling opens the same store; it used to raiseFileNotFoundErrorfor a relative path. write_index,writeandwrite_chunktake a scalar, which fills the box. It is converted to the dtype only when its value survives (0intouint8;300,-1,0.5and NaN raise), and the fill value over whole units leaves them absent.vol.source_translation: the stored level's translation inunits, besidesource_voxel_size, under resample and relabel too (cellmap-flow'sstored_placement).create(exists="adopt")also compares the keys ofattrswith the existing array's attributes and raises naming each that differs or is missing; keys not given are not compared. An adopt given differentattrsused to keep the old ones silently, as funlib'sprepare_dskept an old offset (cellmap-flow's_refuse_another_region).
0.3.1 (2026-10-22)
source_voxel_sizereports the stored level's voxel size underon_mismatch="relabel", as the spec says; it reported the relabelled size (cellmap-flow's report). Reads,voxel_size,translationandcornerare unchanged.boxof an array opened with a resample target is the extent of the target voxelsshapecounts, as the spec's[corner, corner + shape * voxel_size)says and asGrid.from_meta(meta, voxel_size=t).boxalready was; it was the stored data's extent, soread(box)could return fewer voxels thanshape(cellmap-flow's 7 voxels of 8 nm at 24 nm: box ended at 56 nm, now 72). No voxel moves.- Unit table change. The unit table knows kilometres (
kilometer,kilometre,km, 1e3 m), so they convert, abbreviate tokmand are written as OME'skilometer(chunkmirage's ask). This should have been a minor release (spec section 8): cellmap-flow exports the table and its CI failed until it moved its pin to>=0.3.1. - README: the placement rule gains the corner formula, a worked example, why the centre is the model, and how the read rule follows from it.
0.3.0 (2026-10-22)
Grids and blocks, built to spec draft 3 (Behaviours 17 to 19). Nothing on the read or write side changes.
blocks(shape, block_shape, context=0, box=None, clip=True): the array (or a voxel box, which may reach past the array) cut into write boxes that tile it with no overlap and no gap, each with a read box grown bycontext(one int, one per axis, or(before, after)pairs) and clipped to the array whenclip; C order,len,blocks[id]in O(1),index_of(begin); aBlockcarriesindex,id,begin,end,read_begin,read_end,at_startandat_endper axis andinner, the write box's slices inside the read box. Plain picklable records; nothing is scheduled.vol.blocks()defaults the block towrite_shape, so default blocks are aligned writes.vol.write_chunk(index, data)writes one whole unit.Grid(shape, voxel_size, translation, axes, units, chunks, shards): aWorldArraywithout a store, withbox,corner,index_box,chunks_covering,write_shape,blocksandmeta(dtype);Grid.from_meta(meta, voxel_size=)is an existing array's grid or the resample target grid over it.create(path, grid=g, dtype=)creates from it.level_factors(voxel_size, ratio=1.5, ...): the anisotropy-aware pyramid schedule mesh-n-bone and cellmap-analyze share, per level.units.abbreviation, andstore.join,store.parent,store.basenameas public URL-aware path helpers.
0.2.0 (2026-10-22)
Metadata and placed creation, built to spec draft 3 (Behaviours 10 to 16). Nothing on the read side changes.
ome_multiscales(axes, units, levels, version)builds the complete OME-NGFF 0.4 or 0.5 group document with one explicit placement per level;level_placementandlevel_shapederive a level from factors and an origin (Convention 1's rule inverted, so a level built from a ROI origin is placed from it;level_shapeabsorbs float-noise factors such as0.3 / 0.1). Units are written in OME's names ("nm"becomes"nanometer";units.ome_name), an unknown unit as given; the multiscale'stypeis written when given. Expected documents are pinned intests/fixtures/write.json, shared with chunkmirage's TypeScript builder, each case with its resolved per-axis units and types, fifteen cases.create(path, shape, dtype, voxel_size, translation, chunks=..., ...)writes the group document (made when absent, else its one dataset entry added or replaced, every other attribute kept) and then the array, as zarr v2 with OME 0.4 or zarr v3 with OME 0.5 following the group's format.chunksis required, codecs are explicit with a zstd level 3 default, shards are a zarr v3 option.existsis"error"or"adopt"(an identical array is opened for writing, a different one raises naming every difference, a creation lost to another process is adopted after a short backoff). Nothing is ever deleted, truncated or overwritten. Write units equal to the fill value are not stored unlessstore_fill=True, and a stored unit written so loses its key, as zarr-python'swrite_empty_chunks=Falsedoes.createalso writes a bare group document for every missing ancestor down from the.zarrroot and follows the nearest existing ancestor's zarr version, so a level made in a zarr v2 container is zarr v2 with OME 0.4 and the hierarchy browses (cellmap-analyze, cellmap-flow). Adopting an identical array writes no group document; after a write the document is read again and a lost entry is written again, then raises (registration-station's concurrent block jobs).ensure_group_entryis public: list an array the package did not create, such as a symlinkeds0(mesh-n-bone), or a level filled before it is listed.open(path, mode="r+")opens an existing array for writing; aWorldArraypickles by its arguments and its context's spec and reopens in the receiving process, a capped context staying capped (mesh-n-bone's workers).write.codecs_oftakes a path.write_index(begin, end, data)writes whole write units only (the shard when sharded, else the chunk; the clipped last unit allowed), so writers in different processes need no coordination;partial=Truedoes tensorstore's read-modify-write.write(box, data)maps a world box by the centre rule first. No cast, no padding, no resampling on write.open_pyramid(path, mode="r+").add_level(name, factors=, origin=)creates a level's entry and empty array, placed and sized from the level below with its dtype, chunks, shards, codecs and fill value copied; the downsampling itself stays with the caller.chunk_keyandparse_chunk_keyfor zarr v2, zarr v3 (both encodings, sharded arrays keyed by shard), N5 and precomputed, as free functions and onWorldArray;list_chunksuses the same parser.write_shape,ArrayMeta.shard_shape,chunks_covering(write=True).
0.1.1 (2026-10-22)
- funlib
voxel_size/offseton the parent group place an array that carries none, for zarr and N5 alike; funlib's keys on an N5 array are read in the array's C order, as zarr-python's N5 store writes them (both from mesh-n-bone's review of its PR #48). - A precomputed path that names a scale is that scale under
level=0or"first"too; other levels warn and are ignored, as on an array path. - The GCE metadata guard is applied only off Google Cloud (decided from the DMI product
name and the
GCE_METADATA_*variables), so metadata-server credentials keep working on a Google VM. Pyramid.placements(),voxel_sizes()andtranslations()read the group document when the pyramid was opened through one of its level paths too; they used to open every level from there, and a missing level made them raise.- Placement change. An N5 level with
downsamplingFactorsand no offset now sits with its corner at the origin at every level, as Neuroglancer draws N5: translationf / 2 * base. 0.1.0 followed BigDataViewer forpixelResolutionpyramids, centring the base's voxel 0 on the origin ((f - 1) / 2 * base, half a base voxel lower), while placingresolutionpyramids with the corner at the origin, so the two disagreed. Two shared fixture cases move and are renamed:n5_group_pixelresolution_and_downsampling_factors_corner_at_origin(translation 4, 4, 4) andn5_bigstitcher_resolution_with_downsampling_factors_corner_at_origin(0.5, 0.157, 0.157).
0.1.0 (2026-10-21)
- First version, built to
docs/spec.md: open zarr v2, zarr v3, N5 and Neuroglancer precomputed arrays and pyramids through tensorstore; one metadata precedence per format with the voxel-centre rule for translations; index-space and world-box reads with padding; level selection; nearest and linear resampling; retries; shared and capped tensorstore contexts. Built and trialled in one day against the readers it replaces in cellmap-flow, cellmap-analyze, mesh-n-bone and chunkmirage, and checked on their own data by the CineMap, liconn-err, registration-station and salamander repos; seedocs/spec.md, "Trial status". Hot-path reads cost 0.1 to 0.2 ms over a raw tensorstore read.