diff --git a/docs/blog/posts/3.1.0-data-types.md b/docs/blog/posts/3.1.0-data-types.md new file mode 100644 index 0000000000..a211ce956e --- /dev/null +++ b/docs/blog/posts/3.1.0-data-types.md @@ -0,0 +1,299 @@ +--- +date: 2025-07-14 +authors: + - d-v-b +categories: + - Release +--- + +# New data types in Zarr-Python 3.1.0 + +!!! note + This post was written for the 3.1.0 release in July 2025. The code examples run against the current release of Zarr-Python. + +The [3.1.0 release](../../release-notes.md#310-2025-07-14) of Zarr-Python overhauls our internal model of array data types. This is exciting for two reasons: + +First, we added Zarr V3 support for a lot of NumPy data types, including datetimes, timedeltas, fixed-length Unicode strings, null-terminated bytes, raw bytes, and structured data types. This unblocks Zarr V2 users who couldn't migrate to Zarr V3 until these data types were added. + +Second, Zarr-Python data types are now *extensible*. If you have a NumPy-compatible data type class that Zarr-Python doesn't already support, like [`bfloat16`](https://github.com/jax-ml/ml_dtypes?tab=readme-ov-file#bfloat16), you can define a Zarr data type that wraps that data type! Here's a [self-contained, executable example](https://github.com/zarr-developers/zarr-python/blob/main/examples/custom_dtype/custom_dtype.py) that walks through the entire process. + + + +There's more to the data type extensibility story than just implementing a new data type in Zarr-Python. A widely-used data type should have a *published standard* so different Zarr libraries can all consistently implement that data type. + +That's why the Zarr community has created the [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) project. `zarr-extensions` is a central repository for community-defined data types (and other extensions to the Zarr data model, like codecs and chunk grids). Check out the [specification document](https://github.com/zarr-developers/zarr-extensions/tree/main/data-types/bfloat16) for the `bfloat16` data type for an example. + +To dive right into this new feature, head over to our new [data type documentation page](../../user-guide/data_types.md) and read more about how this new data type system works in Zarr-Python. + +But perhaps you'd like to know more about the *context* for this new feature. The best way to convey that is to answer the following two questions: + +- What is Zarr's "data type model" anyway? +- How did we make data types extensible in Zarr-Python? And why now? + +That's what the rest of this post is about. + +## Zarr's data type model + +This section explains how data types used to be declared in Zarr V2, why we did it that way, why that turned out to be problematic, and how we improved things in Zarr V3. + +### NumPy x Zarr + +The Zarr format and the Zarr-Python library are heavily influenced by NumPy's approach to arrays and data types. NumPy's influence is even written explicitly into the Zarr V2 specification: + +> Simple data types are encoded within the array metadata as a string, +> following the [NumPy array protocol type string (typestr) format](https://numpy.org/doc/stable/reference/arrays.interface.html#the-array-interface-protocol). +> +> -- [The Zarr V2 spec](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html#data-type-encoding) + +On the implementation side, Zarr-Python uses NumPy internally, and we expose Zarr array data to users in the form of NumPy arrays or arrays that follow the NumPy API (like [cupy arrays](https://docs.cupy.dev/en/stable/reference/ndarray.html)). + +In some areas Zarr and NumPy fit together very well. They both describe arrays, after all. But the two projects have *very* different scopes: typical NumPy arrays live in memory of a single Python program. Zarr arrays are often persisted to external storage and accessed by many different programs, in many different programming languages, over many years. + +In my opinion, these fundamentally divergent design goals limit how much Zarr can borrow directly from NumPy. + +One of the big changes in the Zarr V3 spec was dropping NumPy compatibility in its data type definition. To explain this choice, the next section will show just how tightly coupled Zarr V2 was to NumPy, and how this was ultimately problematic. + +### NumPy's influence on Zarr V2 + +Every Zarr V2 array is defined by a JSON metadata document. This document declares the information necessary to read from the array, including the array's shape, data type, chunks, etc. All of that metadata must be JSON serializable. + +For data types in particular, the Zarr V2 format defined a JSON serialization scheme that simply copied a particular[^1] string representation of NumPy data types. This is demonstrated in the following example: + +```python exec="true" session="blog-310-v2-dtype" source="above" result="ansi" +import numpy as np +import zarr +import pprint + +dtype_str = ', + "configuration": {...} + } + ``` + +This brings immediate advantages over the Zarr V2 data type model. + +#### Extensible data types + +Instead of copying NumPy, the Zarr V3 specification defines a list of core data types that all Zarr implementations ought to support. But we also think people should be able to create new data types, and we don't want to update the core Zarr V3 specification every time that happens. + +That's the job of the [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) project mentioned above. It is a lightweight coordination mechanism that ensures people designing their own data types can avoid name collisions with other pre-existing community-defined data types. + +We also think writing data type specifications in the open will drive awareness and adoption of new data types across multiple Zarr implementations. + +#### Using more of JSON + +By ditching the Zarr V2 string-only data type definition, we can use JSON objects to more explicitly model data types that have significant parametrization. + +For example, NumPy defines [two data types](https://numpy.org/doc/stable/reference/arrays.datetime.html) for temporal *moments* (`datetime64`) and *durations* (`timedelta64`). Both of these data types are parametrized by a time unit and a scaling factor. Zarr V2 used the NumPy string representation of these data types, leading to [stringly typed](https://www.hanselman.com/blog/stringly-typed-vs-strongly-typed) data type declarations like `"b1"`, `"