ZCollection#

This project is a Python library manipulating data split into a collection of groups stored in Zarr format.

This collection allows dividing a dataset into several partitions to facilitate acquisitions or updates made from new products. Possible data partitioning is: by date (hour, day, month, etc.) or by sequence.

A collection partitioned by date, with a monthly resolution, may look like on the disk:

collection/
├── year=2022
│    ├── month=01/
│    │    ├── time/
│    │    │    ├── 0.0
│    │    │    ├── .zarray
│    │    │    └── .zattrs
│    │    ├── var1/
│    │    │    ├── 0.0
│    │    │    ├── .zarray
│    │    │    └── .zattrs
│    │    ├── .zattrs
│    │    ├── .zgroup
│    │    └── .zmetadata
│    └── month=02/
│         ├── time/
│         │    ├── 0.0
│         │    ├── .zarray
│         │    └── .zattrs
│         ├── var1/
│         │    ├── 0.0
│         │    ├── .zarray
│         │    └── .zattrs
│         ├── .zattrs
│         ├── .zgroup
│         └── .zmetadata
└── .zcollection

Partition updates can be set to overwrite existing data with new ones or to update them using different strategies.

The Dask library handles the data to scale the treatments quickly.

It is possible to create views on a reference collection, to add and modify variables contained in a reference collection, accessible in reading only.

This library can store data on POSIX, S3, or any other file system supported by the Python library fsspec. Note, however, only POSIX and S3 file systems have been tested.

Indices and tables#