From 97adeecb8f6568ce39756c0419d65dac95a88a50 Mon Sep 17 00:00:00 2001 From: Max Jones <14077947+maxrjones@users.noreply.github.com> Date: Wed, 28 Oct 2026 19:16:58 +0000 Subject: [PATCH 1/4] docs(blog): add post on the Zarr Python 3.1.0 data type overhaul Adds Davis Bennett's design post on the extensible data type model introduced in 3.1.0, dated to the 3.1.0 release (2025-07-14). Converted for the mkdocs-material blog: front matter and excerpt marker, blank lines before lists so Python-Markdown renders them, 4-space list continuations, mkdocs footnote syntax, and relative/autoref links in place of the old Sphinx readthedocs URLs. Fixed the moved examples/custom_dtype link, a broken OpenTeams link, and typos. The unfinished "This would allow writers to define" sentence is dropped and the Arrow paragraph now introduces both proposed features. Code samples verified against current main; mkdocs build renders the post with no new warnings. Co-Authored-By: Claude Opus 5.5 --- docs/blog/posts/3.1.0-data-types.md | 341 ++++++++++++++++++++++++++++ 1 file changed, 341 insertions(+) create mode 100644 docs/blog/posts/3.1.0-data-types.md diff --git a/docs/blog/posts/3.1.0-data-types.md b/docs/blog/posts/3.1.0-data-types.md new file mode 100644 index 0000000000..a5e51d3f28 --- /dev/null +++ b/docs/blog/posts/3.1.0-data-types.md @@ -0,0 +1,341 @@ +--- +date: 2025-07-14 +authors: + - d-v-b +categories: + - Release +--- + +# New data types in Zarr Python 3.1.0 + +The [3.1.0 release](../../release-notes.md#310-2025-07-14) of Zarr Python overhauls our internal model of array data types. This is exciting for two reasons: + +First, we added Zarr V3 support for a lot of NumPy data types, including datetimes, timedeltas, Unicode strings, null-terminated bytes, raw bytes, and structured data types. This unblocks Zarr V2 users who couldn't migrate to Zarr V3 until these data types were added. + +Second, Zarr Python data types are now *extensible*. If you have a NumPy-compatible data type class that Zarr Python doesn't already support, like [`bfloat16`](https://github.com/jax-ml/ml_dtypes?tab=readme-ov-file#bfloat16), you can define a Zarr data type that wraps that data type! Here's a [self-contained, executable example](https://github.com/zarr-developers/zarr-python/blob/main/examples/custom_dtype/custom_dtype.py) that walks through the entire process. + + + +There's more to the data type extensibility story than just implementing a new data type in Zarr Python. A widely-used data type should have a *published standard* so different Zarr libraries can all consistently implement that data type. + +That's why the Zarr community has created the [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) project. `zarr-extensions` is a central repository for community-defined data types (and other extensions to the Zarr data model, like codecs and chunk grids). Check out the [specification document](https://github.com/zarr-developers/zarr-extensions/tree/main/data-types/bfloat16) for the `bfloat16` data type for an example. + +To dive right in to this new feature, head over to our new [data type documentation page](../../user-guide/data_types.md) and read more about how this new data type system works in Zarr Python. + +But perhaps you'd like to know more about the *context* for this new feature. The best way to convey that is to answer the following two questions: + +- What is Zarr's "data type model" anyway? +- How did we make data types extensible in Zarr Python? And why now? + +That's what the rest of this post is about. + +## Zarr's data type model + +This section will explain how data types were declared in Zarr V2, why we did it that way, why that turned out to be problematic, and how we improved things in Zarr V3. + +### NumPy x Zarr + +The Zarr format and the Zarr Python library are heavily influenced by NumPy's approach to arrays and data types. NumPy's influence is even written explicitly into the Zarr V2 specification: + +> Simple data types are encoded within the array metadata as a string, +> following the [NumPy array protocol type string (typestr) format](https://numpy.org/doc/stable/reference/arrays.interface.html#the-array-interface-protocol). +> +> -- [The Zarr V2 spec](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html#data-type-encoding) + +On the implementation side, Zarr Python uses NumPy internally, and we expose Zarr array data to users in the form of NumPy arrays or arrays that follow the NumPy API (like [cupy arrays](https://docs.cupy.dev/en/stable/reference/ndarray.html)). + +In some areas Zarr and NumPy fit together very well. They both describe arrays, after all. But the two projects have *very* different scopes: typical NumPy arrays live in memory of a single Python program. Zarr arrays are often persisted to external storage and accessed by many different programs, in many different programming languages, over many years. + +In my opinion, these fundamentally divergent design goals limit how much Zarr can borrow directly from NumPy. + +One of the big changes in the Zarr V3 spec was dropping NumPy compatibility in its data type definition. To inform some of the reasons for this choice, the next section will show just how tightly coupled Zarr V2 was to NumPy, and how this was ultimately problematic. + +### NumPy's influence on Zarr V2 + +Every Zarr V2 array is defined by a JSON metadata document. This document declares the information necessary to read from the array, including the array's shape, data type, chunks, etc. All of that metadata must be JSON serializable. + +For data types in particular, the Zarr V2 format defined a JSON serialization scheme that simply copied a particular[^1] string representation of NumPy data types. This is demonstrated in the following script: + +```python +# /// script +# requires-python = ">=3.11" +# dependencies = ["zarr>=3.1"] +# /// + +import numpy as np +import zarr +import pprint + + +dtype_str=', + "configuration": {...} + } + ``` + +This brings immediate advantages over the Zarr V2 data type model. + +#### Extensible data types + +Instead of copying NumPy, the Zarr V3 specification defines a list of core data types that all Zarr implementations ought to support. But we also think people should be able to create new data types, and we don't want to update the core Zarr V3 specification every time that happens. + +Instead, the Zarr community maintains the [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) project, which collects specification documents for community-defined data types. `zarr-extensions` is a lightweight coordination mechanism that ensures people designing their own data types can avoid name collisions with other pre-existing community-defined data types. + +We also think writing data type specifications in the open will drive awareness and adoption of new data types across multiple Zarr implementations. + +#### Using more of JSON + +By ditching the Zarr V2 string-only data type definition, we can use JSON objects to more explicitly model data types that have significant parametrization. + +For example, NumPy defines [two data types](https://numpy.org/doc/stable/reference/arrays.datetime.html) for temporal *moments* (`datetime64`) and *intervals* (`timedelta64`). Both of these data types are parametrized by a time unit and a scaling factor. Zarr V2 used the NumPy string representation of these data types, leading to [stringly typed](https://www.hanselman.com/blog/stringly-typed-vs-strongly-typed) data type declarations like `"b1"`, `" Date: Wed, 28 Oct 2026 20:26:08 +0000 Subject: [PATCH 2/4] docs(blog): mark 3.1.0 data type post code blocks as tested The doctests job failed in test_no_unvalidated_blocks because the three python fences in the new blog post had no exec/test attribute. Mark them test="true" (each in its own session, since they are standalone scripts) so they run under the docs harness; they already embed their expected output, so build-time exec rendering is not needed. Co-Authored-By: Claude Opus 5.5 --- docs/blog/posts/3.1.0-data-types.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/blog/posts/3.1.0-data-types.md b/docs/blog/posts/3.1.0-data-types.md index a5e51d3f28..733902eec4 100644 --- a/docs/blog/posts/3.1.0-data-types.md +++ b/docs/blog/posts/3.1.0-data-types.md @@ -56,7 +56,7 @@ Every Zarr V2 array is defined by a JSON metadata document. This document declar For data types in particular, the Zarr V2 format defined a JSON serialization scheme that simply copied a particular[^1] string representation of NumPy data types. This is demonstrated in the following script: -```python +```python test="true" session="blog-310-v2-dtype" # /// script # requires-python = ">=3.11" # dependencies = ["zarr>=3.1"] @@ -116,7 +116,7 @@ Prior to NumPy 2.0 (when Zarr V2 was defined) there was no data type for an arra The "Object" data type has a string identifier of `"|O"`, and "Object" data type arrays can contain *any* Python object, including but not limited to variable length strings. So two object arrays with very different contents will have the same data type identifier: -```python +```python test="true" session="blog-310-object-dtype" # /// script # requires-python = ">=3.11" # dependencies = ["numpy"] @@ -245,7 +245,7 @@ Each `ZDType` defines the following behaviors: We can demonstrate everything with a simple example: -```python +```python test="true" session="blog-310-dtype-api" # /// script # requires-python = ">=3.11" # dependencies = ["zarr==3.1.0"] From 5ac5e721ad8781936a9111a0dea9739f9f50ea8d Mon Sep 17 00:00:00 2001 From: Max Jones <14077947+maxrjones@users.noreply.github.com> Date: Wed, 28 Oct 2026 20:39:08 +0000 Subject: [PATCH 3/4] docs(blog): render 3.1.0 data type post examples at build time Switch the code blocks from test="true" with hand-written output to exec="true" source="above" result="ansi", matching the other blog posts, so the shown output is generated by markdown-exec rather than pasted in docstrings that nothing checks. Blocks are split per print call (sharing a session) to keep each output next to the call that produced it. Also drop the PEP 723 script headers (the zarr==3.1.0 pin would be false once output comes from the docs build), and use pprint.pformat with print because markdown-exec only captures print(), not pprint.pp. Co-Authored-By: Claude Opus 5.5 --- docs/blog/posts/3.1.0-data-types.md | 82 +++++++---------------------- 1 file changed, 19 insertions(+), 63 deletions(-) diff --git a/docs/blog/posts/3.1.0-data-types.md b/docs/blog/posts/3.1.0-data-types.md index 733902eec4..dffd0ac5ea 100644 --- a/docs/blog/posts/3.1.0-data-types.md +++ b/docs/blog/posts/3.1.0-data-types.md @@ -54,20 +54,14 @@ One of the big changes in the Zarr V3 spec was dropping NumPy compatibility in i Every Zarr V2 array is defined by a JSON metadata document. This document declares the information necessary to read from the array, including the array's shape, data type, chunks, etc. All of that metadata must be JSON serializable. -For data types in particular, the Zarr V2 format defined a JSON serialization scheme that simply copied a particular[^1] string representation of NumPy data types. This is demonstrated in the following script: - -```python test="true" session="blog-310-v2-dtype" -# /// script -# requires-python = ">=3.11" -# dependencies = ["zarr>=3.1"] -# /// +For data types in particular, the Zarr V2 format defined a JSON serialization scheme that simply copied a particular[^1] string representation of NumPy data types. This is demonstrated in the following example: +```python exec="true" session="blog-310-v2-dtype" source="above" result="ansi" import numpy as np import zarr import pprint - -dtype_str=' Date: Fri, 30 Oct 2026 15:54:21 +0100 Subject: [PATCH 4/4] docs(blog): correct facts and wording in the 3.1.0 data type post (#9) - Say that "|O" was a Zarr-Python 2 convention. The V2 spec defines no "O" type code. - Use one list of the data types that 3.1 added. Drop variable-length bytes, which 3.0 already supported for V3. - Describe native data types as NumPy data types. CuPy has no data type classes of its own. - Add a note that the post was written for 3.1.0 and that the examples run against the current release. - Mark the placeholder metadata block as text, so it is not highlighted as invalid JSON. - Spell the project name "Zarr-Python", as the other blog posts do. - Fix small wording issues. Assisted-by: ClaudeCode:claude-opus-5-5 --- docs/blog/posts/3.1.0-data-types.md | 72 +++++++++++++++-------------- 1 file changed, 37 insertions(+), 35 deletions(-) diff --git a/docs/blog/posts/3.1.0-data-types.md b/docs/blog/posts/3.1.0-data-types.md index dffd0ac5ea..a211ce956e 100644 --- a/docs/blog/posts/3.1.0-data-types.md +++ b/docs/blog/posts/3.1.0-data-types.md @@ -6,49 +6,52 @@ categories: - Release --- -# New data types in Zarr Python 3.1.0 +# New data types in Zarr-Python 3.1.0 -The [3.1.0 release](../../release-notes.md#310-2025-07-14) of Zarr Python overhauls our internal model of array data types. This is exciting for two reasons: +!!! note + This post was written for the 3.1.0 release in July 2025. The code examples run against the current release of Zarr-Python. -First, we added Zarr V3 support for a lot of NumPy data types, including datetimes, timedeltas, Unicode strings, null-terminated bytes, raw bytes, and structured data types. This unblocks Zarr V2 users who couldn't migrate to Zarr V3 until these data types were added. +The [3.1.0 release](../../release-notes.md#310-2025-07-14) of Zarr-Python overhauls our internal model of array data types. This is exciting for two reasons: -Second, Zarr Python data types are now *extensible*. If you have a NumPy-compatible data type class that Zarr Python doesn't already support, like [`bfloat16`](https://github.com/jax-ml/ml_dtypes?tab=readme-ov-file#bfloat16), you can define a Zarr data type that wraps that data type! Here's a [self-contained, executable example](https://github.com/zarr-developers/zarr-python/blob/main/examples/custom_dtype/custom_dtype.py) that walks through the entire process. +First, we added Zarr V3 support for a lot of NumPy data types, including datetimes, timedeltas, fixed-length Unicode strings, null-terminated bytes, raw bytes, and structured data types. This unblocks Zarr V2 users who couldn't migrate to Zarr V3 until these data types were added. + +Second, Zarr-Python data types are now *extensible*. If you have a NumPy-compatible data type class that Zarr-Python doesn't already support, like [`bfloat16`](https://github.com/jax-ml/ml_dtypes?tab=readme-ov-file#bfloat16), you can define a Zarr data type that wraps that data type! Here's a [self-contained, executable example](https://github.com/zarr-developers/zarr-python/blob/main/examples/custom_dtype/custom_dtype.py) that walks through the entire process. -There's more to the data type extensibility story than just implementing a new data type in Zarr Python. A widely-used data type should have a *published standard* so different Zarr libraries can all consistently implement that data type. +There's more to the data type extensibility story than just implementing a new data type in Zarr-Python. A widely-used data type should have a *published standard* so different Zarr libraries can all consistently implement that data type. That's why the Zarr community has created the [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) project. `zarr-extensions` is a central repository for community-defined data types (and other extensions to the Zarr data model, like codecs and chunk grids). Check out the [specification document](https://github.com/zarr-developers/zarr-extensions/tree/main/data-types/bfloat16) for the `bfloat16` data type for an example. -To dive right in to this new feature, head over to our new [data type documentation page](../../user-guide/data_types.md) and read more about how this new data type system works in Zarr Python. +To dive right into this new feature, head over to our new [data type documentation page](../../user-guide/data_types.md) and read more about how this new data type system works in Zarr-Python. But perhaps you'd like to know more about the *context* for this new feature. The best way to convey that is to answer the following two questions: - What is Zarr's "data type model" anyway? -- How did we make data types extensible in Zarr Python? And why now? +- How did we make data types extensible in Zarr-Python? And why now? That's what the rest of this post is about. ## Zarr's data type model -This section will explain how data types were declared in Zarr V2, why we did it that way, why that turned out to be problematic, and how we improved things in Zarr V3. +This section explains how data types used to be declared in Zarr V2, why we did it that way, why that turned out to be problematic, and how we improved things in Zarr V3. ### NumPy x Zarr -The Zarr format and the Zarr Python library are heavily influenced by NumPy's approach to arrays and data types. NumPy's influence is even written explicitly into the Zarr V2 specification: +The Zarr format and the Zarr-Python library are heavily influenced by NumPy's approach to arrays and data types. NumPy's influence is even written explicitly into the Zarr V2 specification: > Simple data types are encoded within the array metadata as a string, > following the [NumPy array protocol type string (typestr) format](https://numpy.org/doc/stable/reference/arrays.interface.html#the-array-interface-protocol). > > -- [The Zarr V2 spec](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html#data-type-encoding) -On the implementation side, Zarr Python uses NumPy internally, and we expose Zarr array data to users in the form of NumPy arrays or arrays that follow the NumPy API (like [cupy arrays](https://docs.cupy.dev/en/stable/reference/ndarray.html)). +On the implementation side, Zarr-Python uses NumPy internally, and we expose Zarr array data to users in the form of NumPy arrays or arrays that follow the NumPy API (like [cupy arrays](https://docs.cupy.dev/en/stable/reference/ndarray.html)). In some areas Zarr and NumPy fit together very well. They both describe arrays, after all. But the two projects have *very* different scopes: typical NumPy arrays live in memory of a single Python program. Zarr arrays are often persisted to external storage and accessed by many different programs, in many different programming languages, over many years. In my opinion, these fundamentally divergent design goals limit how much Zarr can borrow directly from NumPy. -One of the big changes in the Zarr V3 spec was dropping NumPy compatibility in its data type definition. To inform some of the reasons for this choice, the next section will show just how tightly coupled Zarr V2 was to NumPy, and how this was ultimately problematic. +One of the big changes in the Zarr V3 spec was dropping NumPy compatibility in its data type definition. To explain this choice, the next section will show just how tightly coupled Zarr V2 was to NumPy, and how this was ultimately problematic. ### NumPy's influence on Zarr V2 @@ -86,13 +89,13 @@ There are some advantages to copying NumPy data type strings: - Strings are JSON serializable. We can easily store these NumPy strings in Zarr metadata documents without defining a special JSON serialization procedure. - Using NumPy strings gives Zarr easy NumPy compatibility -- we can just call `np.dtype` on the string to get the right NumPy data type. -If Zarr only supported integers and floating point numbers, maybe the story could end here. But Zarr V2 supports other data types, namely variable-length data types, (because Zarr V2 users *needed* these data types), and with these data types we see the limits of copying strings from NumPy. +If Zarr only supported integers and floating point numbers, maybe the story could end here. But Zarr V2 supports other data types, namely variable-length data types (because Zarr V2 users *needed* these data types), and with these data types we see the limits of copying strings from NumPy. #### Variable-length data types The biggest problem with using NumPy data type strings in Zarr is that *distinct Zarr data types have the same NumPy string representation*. In other words, there are more Zarr data types than NumPy data types. -Prior to NumPy 2.0 (when Zarr V2 was defined) there was no data type for an array of strings with variable length, like `["a", "bb", "ccc"]`. NumPy supported these arrays with the ["Object" data type](https://numpy.org/doc/stable/reference/arrays.scalars.html#numpy.object_), which supports arrays of arbitrary Python objects (including strings) by storing a buffer of references. +When Zarr V2 was defined, years before NumPy 2.0, NumPy had no data type for an array of strings with variable length, like `["a", "bb", "ccc"]`. NumPy supported these arrays with the ["Object" data type](https://numpy.org/doc/stable/reference/arrays.scalars.html#numpy.object_), which supports arrays of arbitrary Python objects (including strings) by storing a buffer of references. The "Object" data type has a string identifier of `"|O"`, and "Object" data type arrays can contain *any* Python object, including but not limited to variable length strings. So two object arrays with very different contents will have the same data type identifier: @@ -106,18 +109,18 @@ print(vlen_str.dtype.str) print(vlen_array.dtype.str) ``` -The above example is relevant because Zarr Python 2 supported arrays of variable length strings *and* arrays of variable length arrays. According to the [Zarr V2 spec](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html#data-type-encoding), these two distinct data types are both declared as `"dtype": "|O"` in metadata documents (because they are both the "same" data type from NumPy's POV). +The above example is relevant because Zarr-Python 2 supported arrays of variable length strings *and* arrays of variable length arrays. Zarr-Python 2 declared both of these distinct data types as `"dtype": "|O"` in metadata documents (because they are both the "same" data type from NumPy's POV), even though `"O"` is not one of the type codes defined in the [Zarr V2 spec](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html#data-type-encoding). This is clearly a problem. From a Zarr point of view a variable-length array of strings is *very different* from a variable-length array of arrays. Zarr needs to store arrays on disk, and the storage procedure is very different for these two distinct data types. -Zarr Python solved this with a hack: an array of variable-length strings was *operationally* defined as an array with the following configuration in metadata: +Zarr-Python solved this with a hack: an array of variable-length strings was *operationally* defined as an array with the following configuration in metadata: - the data type is declared as `"|O"`. - the chunk encoding for the array included a special "variable-length UTF8" codec. This could be found either in the `filters` field or the `compressor` field. And an array of variable-length arrays was operationally defined similarly, although with a *different* special codec (one designed specifically for variable-length arrays). -Clearly it's problematic if the metadata field responsible for declaring the data type for the array (`dtype`) doesn't fully declare the data type for the array in some cases! Zarr Python hacked around this by storing some data-type-specific information in the filters / compressor for an array, but this is far from ideal (and this behavior was never fully specified). +Clearly it's problematic if the metadata field responsible for declaring the data type for the array (`dtype`) doesn't fully declare the data type for the array in some cases! Zarr-Python hacked around this by storing some data-type-specific information in the filters / compressor for an array, but this is far from ideal (and this behavior was never fully specified). ### Zarr Version 3 @@ -128,7 +131,7 @@ Instead, data types in Zarr V3 are declared in one of two ways: - as a string, like `"int64"`, which declares the signed 64-bit integer data type. Note that unlike NumPy, Zarr V3 data types are generally not defined with a byte order. - as a JSON object with the following form: - ```json + ```text { "name": , "configuration": {...} @@ -141,7 +144,7 @@ This brings immediate advantages over the Zarr V2 data type model. Instead of copying NumPy, the Zarr V3 specification defines a list of core data types that all Zarr implementations ought to support. But we also think people should be able to create new data types, and we don't want to update the core Zarr V3 specification every time that happens. -Instead, the Zarr community maintains the [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) project, which collects specification documents for community-defined data types. `zarr-extensions` is a lightweight coordination mechanism that ensures people designing their own data types can avoid name collisions with other pre-existing community-defined data types. +That's the job of the [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) project mentioned above. It is a lightweight coordination mechanism that ensures people designing their own data types can avoid name collisions with other pre-existing community-defined data types. We also think writing data type specifications in the open will drive awareness and adoption of new data types across multiple Zarr implementations. @@ -149,7 +152,7 @@ We also think writing data type specifications in the open will drive awareness By ditching the Zarr V2 string-only data type definition, we can use JSON objects to more explicitly model data types that have significant parametrization. -For example, NumPy defines [two data types](https://numpy.org/doc/stable/reference/arrays.datetime.html) for temporal *moments* (`datetime64`) and *intervals* (`timedelta64`). Both of these data types are parametrized by a time unit and a scaling factor. Zarr V2 used the NumPy string representation of these data types, leading to [stringly typed](https://www.hanselman.com/blog/stringly-typed-vs-strongly-typed) data type declarations like `"