zarr-python

# Zarr Python

## Overview

Zarr is a Python library for storing large N-dimensional arrays with chunking and compression. Apply this skill for efficient parallel I/O, cloud-native workflows, and seamless integration with NumPy, Dask, and Xarray.

## Quick Start

### Installation

```bash
uv pip install zarr
```

Requires Python 3.11+. For cloud storage support, install additional packages:
```python
uv pip install s3fs  # For S3
uv pip install gcsfs  # For Google Cloud Storage
```

### Basic Array Creation

```python
import zarr
import numpy as np

# Create a 2D array with chunking and compression
z = zarr.create_array(
    store="data/my_array.zarr",
    shape=(10000, 10000),
    chunks=(1000, 1000),
    dtype="f4"
)

# Write data using NumPy-style indexing
z[:, :] = np.random.random((10000, 10000))

# Read data
data = z[0:100, 0:100]  # Returns NumPy array
```

## Core Operations

### Creating Arrays

Zarr provides multiple convenience functions for array creation:

```python
# Create empty array
z = zarr.zeros(shape=(10000, 10000), chunks=(1000, 1000), dtype='f4',
               store='data.zarr')

# Create filled arrays
z = zarr.ones((5000, 5000), chunks=(500, 500))
z = zarr.full((1000, 1000), fill_value=42, chunks=(100, 100))

# Create from existing data
data = np.arange(10000).reshape(100, 100)
z = zarr.array(data, chunks=(10, 10), store='data.zarr')

# Create like another array
z2 = zarr.zeros_like(z)  # Matches shape, chunks, dtype of z
```

### Opening Existing Arrays

```python
# Open array (read/write mode by default)
z = zarr.open_array('data.zarr', mode='r+')

# Read-only mode
z = zarr.open_array('data.zarr', mode='r')

# The open() function auto-detects arrays vs groups
z = zarr.open('data.zarr')  # Returns Array or Group
```

### Reading and Writing Data

Zarr arrays support NumPy-like indexing:

```python
# Write entire array
z[:] = 42

# Write slices
z[0, :] = np.arange(100)
z[10:20, 50:60] = np.random.random((10, 10))

# Read data (returns NumPy array)
data = z[0:100, 0:100]
row = z[5, :]

# Advanced indexing
z.vindex[[0, 5, 10], [2, 8, 15]]  # Coordinate indexing
z.oindex[0:10, [5, 10, 15]]       # Orthogonal indexing
z.blocks[0, 0]                     # Block/chunk indexing
```

### Resizing and Appending

```python
# Resize array
z.resize(15000, 15000)  # Expands or shrinks dimensions

# Append data along an axis
z.append(np.random.random((1000, 10000)), axis=0)  # Adds rows
```

## Chunking Strategies

Chunking is critical for performance. Choose chunk sizes and shapes based on access patterns.

### Chunk Size Guidelines

- **Minimum chunk size**: 1 MB recommended for optimal performance
- **Balance**: Larger chunks = fewer metadata operations; smaller chunks = better parallel access
- **Memory consideration**: Entire chunks must fit in memory during compression

```python
# Configure chunk size (aim for ~1MB per chunk)
# For float32 data: 1MB = 262,144 elements = 512×512 array
z = zarr.zeros(
    shape=(10000, 10000),
    chunks=(512, 512),  # ~1MB chunks
    dtype='f4'
)
```

### Aligning Chunks with Access Patterns

**Critical**: Chunk shape dramatically affects performance based on how data is accessed.

```python
# If accessing rows frequently (first dimension)
z = zarr.zeros((10000, 10000), chunks=(10, 10000))  # Chunk spans columns

# If accessing columns frequently (second dimension)
z = zarr.zeros((10000, 10000), chunks=(10000, 10))  # Chunk spans rows

# For mixed access patterns (balanced approach)
z = zarr.zeros((10000, 10000), chunks=(1000, 1000))  # Square chunks
```

**Performance example**: For a (200, 200, 200) array, reading along the first dimension:
- Using chunks (1, 200, 200): ~107ms
- Using chunks (200, 200, 1): ~1.65ms (65× faster!)

### Sharding for Large-Scale Storage

When arrays have millions of small chunks, use sharding to group chunks into larger storage objects:

```python
from zarr.codecs import ShardingCodec, BytesCodec
from zarr.codecs.blosc import BloscCodec

# Create array with sharding
z = zarr.create_array(
    store='data.zarr',
    shape=(100000, 100000),
    chunks=(100, 100),  # Small chunks for access
    shards=(1000, 1000),  # Groups 100 chunks per shard
    dtype='f4'
)
```

**Benefits**:
- Reduces file system overhead from millions of small files
- Improves cloud storage performance (fewer object requests)
- Prevents filesystem block size waste

**Important**: Entire shards must fit in memory before writing.

## Compression

Zarr applies compression per chunk to reduce storage while maintaining fast access.

### Configuring Compression

```python
from zarr.codecs.blosc import BloscCodec
from zarr.codecs import GzipCodec, ZstdCodec

# Default: Blosc with Zstandard
z = zarr.zeros((1000, 1000), chunks=(100, 100))  # Uses default compression

# Configure Blosc codec
z = zarr.create_array(
    store='data.zarr',
    shape=(1000, 1000),
    chunks=(100, 100),
    dtype='f4',
    codecs=[BloscCodec(cname='zstd', clevel=5, shuffle='shuffle')]
)

# Available Blosc compressors: 'blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib', 'zstd'

# Use Gzip compression
z = zarr.create_array(
    store='data.zarr',
    shape=(1000, 1000),
    chunks=(100, 100),
    dtype='f4',
    codecs=[GzipCodec(level=6)]
)

# Disable compression
z = zarr.create_array(
    store='data.zarr',
    shape=(1000, 1000),
    chunks=(100, 100),
    dtype='f4',
    codecs=[BytesCodec()]  # No compression
)
```

### Compression Performance Tips

- **Blosc** (default): Fast compression/decompression, good for interactive workloads
- **Zstandard**: Better compression ratios, slightly slower than LZ4
- **Gzip**: Maximum compression, slower performance
- **LZ4**: Fastest compression, lower ratios
- **Shuffle**: Enable shuffle filter for better compression on numeric data

```python
# Optimal for numeric scientific data
codecs=[BloscCodec(cname='zstd', clevel=5, shuffle='shuffle')]

# Optimal for speed
codecs=[BloscCodec(cname='lz4', clevel=1)]

# Optimal for compression ratio
codecs=[GzipCodec(level=9)]
```

## Storage Backends

Zarr supports multiple storage backends through a flexible storage interface.

### Local Filesystem (Default)

```python
from zarr.storage import LocalStore

# Explicit store creation
store = LocalStore('data/my_array.zarr')
z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))

# Or use string path (creates LocalStore automatically)
z = zarr.open_array('data/my_array.zarr', mode='w', shape=(1000, 1000),
                    chunks=(100, 100))
```

### In-Memory Storage

```python
from zarr.storage import MemoryStore

# Create in-memory store
store = MemoryStore()
z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))

# Data exists only in memory, not persisted
```

### ZIP File Storage

```python
from zarr.storage import ZipStore

# Write to ZIP file
store = ZipStore('data.zip', mode='w')
z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))
z[:] = np.random.random((1000, 1000))
store.close()  # IMPORTANT: Must close ZipStore

# Read from ZIP file
store = ZipStore('data.zip', mode='r')
z = zarr.open_array(store=store)
data = z[:]
store.close()
```

### Cloud Storage (S3, GCS)

```python
import s3fs
import zarr

# S3 storage
s3 = s3fs.S3FileSystem(anon=False)  # Use credentials
store = s3fs.S3Map(root='my-bucket/path/to/array.zarr', s3=s3)
z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))
z[:] = data

# Google Cloud Storage
import gcsfs
gcs = gcsfs.GCSFileSystem(project='my-project')
store = gcsfs.GCSMap(root='my-bucket/path/to/array.zarr', gcs=gcs)
z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))
```

**Cloud Storage Best Practices**:
- Use consolidated metadata to reduce latency: `zarr.consolidate_metadata(store)`
- Align chunk sizes with cloud object sizing (typically 5-100 MB optimal)
- Enable parallel writes using Dask for large-scale data
- Consider sharding to reduce number of objects

## Groups and Hierarchies

Groups organize multiple arrays hierarchically, similar to directories or HDF5 groups.

### Creating and Using Groups

```python
# Create root group
root = zarr.group(store='data/hierarchy.zarr')

# Create sub-groups
temperature = root.create_group('temperature')
precipitation = root.create_group('precipitation')

# Create arrays within groups
temp_array = temperature.create_array(
    name='t2m',
    shape=(365, 720, 1440),
    chunks=(1, 720, 1440),
    dtype='f4'
)

precip_array = precipitation.create_array(
    name='prcp',
    shape=(365, 720, 1440),
    chunks=(1, 720, 1440),
    dtype='f4'
)

# Access using paths
array = root['temperature/t2m']

# Visualize hierarchy
print(root.tree())
# Output:
# /
#  ├── temperature
#  │   └── t2m (365, 720, 1440) f4
#  └── precipitation
#      └── prcp (365, 720, 1440) f4
```

### H5py-Compatible API

Zarr provides an h5py-compatible interface for familiar HDF5 users:

```python
# Create group with h5py-style methods
root = zarr.group('data.zarr')
dataset = root.create_dataset('my_data', shape=(1000, 1000), chunks=(100, 100),
                              dtype='f4')

# Access like h5py
grp = root.require_group('subgroup')
arr = grp.require_dataset('array', shape=(500, 500), chunks=(50, 50), dtype='i4')
```

## Attributes and Metadata

Attach custom metadata to arrays and groups using attributes:

```python
# Add attributes to array
z = zarr.zeros((1000, 1000), chunks=(100, 100))
z.attrs['description'] = 'Temperature data in Kelvin'
z.attrs['units'] = 'K'
z.attrs['created'] = '2024-01-15'
z.attrs['processing_version'] = 2.1

# Attributes are stored as JSON
print(z.attrs['units'])  # Output: K

# Add attributes to groups
root = zarr.group('data.zarr')
root.attrs['project'] = 'Climate Analysis'
root.attrs['institution'] = 'Research Institute'

# Attributes persist with the array/group
z2 = zarr.open('data.zarr')
print(z2.attrs['description'])
```

**Important**: Attributes must be JSON-serializable (strings, numbers, lists, dicts, booleans, null).

## Integration with NumPy, Dask, and Xarray

### NumPy Integration

Zarr arrays implement the NumPy array interface:

```python
import numpy as np
import zarr

z = zarr.zeros((1000, 1000), chunks=(100, 100))

# Use NumPy functions directly
result = np.sum(z, axis=0)  # NumPy operates on Zarr array
mean = np.mean(z[:100, :100])

# Convert to NumPy array
numpy_array = z[:]  # Loads entire array into memory
```

### Dask Integration

Dask provides lazy, parallel computation on Zarr arrays:

```python
import dask.array as da
import zarr

# Create large Zarr array
z = zarr.open('data.zarr', mode='w', shape=(100000, 100000),
              chunks=(1000, 1000), dtype='f4')

# Load as Dask array (lazy, no data loaded)
dask_array = da.from_zarr('data.zarr')

# Perform computations (parallel, out-of-core)
result = dask_array.mean(axis=0).compute()  # Parallel computation

# Write Dask array to Zarr
large_array = da.random.random((100000, 100000), chunks=(1000, 1000))
da.to_zarr(large_array, 'output.zarr')
```

**Benefits**:
- Process datasets larger than memory
- Automatic parallel computation across chunks
- Efficient I/O with chunked storage

### Xarray Integration

Xarray provides labeled, multidimensional arrays with Zarr backend:

```python
import xarray as xr
import zarr

# Open Zarr store as Xarray Dataset (lazy loading)
ds = xr.open_zarr('data.zarr')

# Dataset includes coordinates and metadata
print(ds)

# Access variables
temperature = ds['temperature']

# Perform labeled operations
subset = ds.sel(time='2024-01', lat=slice(30, 60))

# Write Xarray Dataset to Zarr
ds.to_zarr('output.zarr')

# Create from scratch with coordinates
ds = xr.Dataset(
    {
        'temperature': (['time', 'lat', 'lon'], data),
        'precipitation': (['time', 'lat', 'lon'], data2)
    },
    coords={
        'time': pd.date_range('2024-01-01', periods=365),
        'lat': np.arange(-90, 91, 1),
        'lon': np.arange(-180, 180, 1)
    }
)
ds.to_zarr('climate_data.zarr')
```

**Benefits**:
- Named dimensions and coordinates
- Label-based indexing and selection
- Integration with pandas for time series
- NetCDF-like interface familiar to climate/geospatial scientists

## Parallel Computing and Synchronization

### Thread-Safe Operations

```python
from zarr import ThreadSynchronizer
import zarr

# For multi-threaded writes
synchronizer = ThreadSynchronizer()
z = zarr.open_array('data.zarr', mode='r+', shape=(10000, 10000),
                    chunks=(1000, 1000), synchronizer=synchronizer)

# Safe for concurrent writes from multiple threads
# (when writes don't span chunk boundaries)
```

### Process-Safe Operations

```python
from zarr import ProcessSynchronizer
import zarr

# For multi-process writes
synchronizer = ProcessSynchronizer('sync_data.sync')
z = zarr.open_array('data.zarr', mode='r+', shape=(10000, 10000),
                    chunks=(1000, 1000), synchronizer=synchronizer)

# Safe for concurrent writes from multiple processes
```

**Note**:
- Concurrent reads require no synchronization
- Synchronization only needed for writes that may span chunk boundaries
- Each process/thread writing to separate chunks needs no synchronization

## Consolidated Metadata

For hierarchical stores with many arrays, consolidate metadata into a single file to reduce I/O operations:

```python
import zarr

# After creating arrays/groups
root = zarr.group('data.zarr')
# ... create multiple arrays/groups ...

# Consolidate metadata
zarr.consolidate_metadata('data.zarr')

# Open with consolidated metadata (faster, especially on cloud storage)
root = zarr.open_consolidated('data.zarr')
```

**Benefits**:
- Reduces metadata read operations from N (one per array) to 1
- Critical for cloud storage (reduces latency)
- Speeds up `tree()` operations and group traversal

**Cautions**:
- Metadata can become stale if arrays update without re-consolidation
- Not suitable for frequently-updated datasets
- Multi-writer scenarios may have inconsistent reads

## Performance Optimization

### Checklist for Optimal Performance

1. **Chunk Size**: Aim for 1-10 MB per chunk
   ```python
   # For float32: 1MB = 262,144 elements
   chunks = (512, 512)  # 512×512×4 bytes = ~1MB
   ```

2. **Chunk Shape**: Align with access patterns
   ```python
   # Row-wise access → chunk spans columns: (small, large)
   # Column-wise access → chunk spans rows: (large, small)
   # Random access → balanced: (medium, medium)
   ```

3. **Compression**: Choose based on workload
   ```python
   # Interactive/fast: BloscCodec(cname='lz4')
   # Balanced: BloscCodec(cname='zstd', clevel=5)
   # Maximum compression: GzipCodec(level=9)
   ```

4. **Storage Backend**: Match to environment
   ```python
   # Local: LocalStore (default)
   # Cloud: S3Map/GCSMap with consolidated metadata
   # Temporary: MemoryStore
   ```

5. **Sharding**: Use for large-scale datasets
   ```python
   # When you have millions of small chunks
   shards=(10*chunk_size, 10*chunk_size)
   ```

6. **Parallel I/O**: Use Dask for large operations
   ```python
   import dask.array as da
   dask_array = da.from_zarr('data.zarr')
   result = dask_array.compute(scheduler='threads', num_workers=8)
   ```

### Profiling and Debugging

```python
# Print detailed array information
print(z.info)

# Output includes:
# - Type, shape, chunks, dtype
# - Compression codec and level
# - Storage size (compressed vs uncompressed)
# - Storage location

# Check storage size
print(f"Compressed size: {z.nbytes_stored / 1e6:.2f} MB")
print(f"Uncompressed size: {z.nbytes / 1e6:.2f} MB")
print(f"Compression ratio: {z.nbytes / z.nbytes_stored:.2f}x")
```

## Common Patterns and Best Practices

### Pattern: Time Series Data

```python
# Store time series with time as first dimension
# This allows efficient appending of new time steps
z = zarr.open('timeseries.zarr', mode='a',
              shape=(0, 720, 1440),  # Start with 0 time steps
              chunks=(1, 720, 1440),  # One time step per chunk
              dtype='f4')

# Append new time steps
new_data = np.random.random((1, 720, 1440))
z.append(new_data, axis=0)
```

### Pattern: Large Matrix Operations

```python
import dask.array as da

# Create large matrix in Zarr
z = zarr.open('matrix.zarr', mode='w',
              shape=(100000, 100000),
              chunks=(1000, 1000),
              dtype='f8')

# Use Dask for parallel computation
dask_z = da.from_zarr('matrix.zarr')
result = (dask_z @ dask_z.T).compute()  # Parallel matrix multiply
```

### Pattern: Cloud-Native Workflow

```python
import s3fs
import zarr

# Write to S3
s3 = s3fs.S3FileSystem()
store = s3fs.S3Map(root='s3://my-bucket/data.zarr', s3=s3)

# Create array with appropriate chunking for cloud
z = zarr.open_array(store=store, mode='w',
                    shape=(10000, 10000),
                    chunks=(500, 500),  # ~1MB chunks
                    dtype='f4')
z[:] = data

# Consolidate metadata for faster reads
zarr.consolidate_metadata(store)

# Read from S3 (anywhere, anytime)
store_read = s3fs.S3Map(root='s3://my-bucket/data.zarr', s3=s3)
z_read = zarr.open_consolidated(store_read)
subset = z_read[0:100, 0:100]
```

### Pattern: Format Conversion

```python
# HDF5 to Zarr
import h5py
import zarr

with h5py.File('data.h5', 'r') as h5:
    dataset = h5['dataset_name']
    z = zarr.array(dataset[:],
                   chunks=(1000, 1000),
                   store='data.zarr')

# NumPy to Zarr
import numpy as np
data = np.load('data.npy')
z = zarr.array(data, chunks='auto', store='data.zarr')

# Zarr to NetCDF (via Xarray)
import xarray as xr
ds = xr.open_zarr('data.zarr')
ds.to_netcdf('data.nc')
```

## Common Issues and Solutions

### Issue: Slow Performance

**Diagnosis**: Check chunk size and alignment
```python
print(z.chunks)  # Are chunks appropriate size?
print(z.info)    # Check compression ratio
```

**Solutions**:
- Increase chunk size to 1-10 MB
- Align chunks with access pattern
- Try different compression codecs
- Use Dask for parallel operations

### Issue: High Memory Usage

**Cause**: Loading entire array or large chunks into memory

**Solutions**:
```python
# Don't load entire array
# Bad: data = z[:]
# Good: Process in chunks
for i in range(0, z.shape[0], 1000):
    chunk = z[i:i+1000, :]
    process(chunk)

# Or use Dask for automatic chunking
import dask.array as da
dask_z = da.from_zarr('data.zarr')
result = dask_z.mean().compute()  # Processes in chunks
```

### Issue: Cloud Storage Latency

**Solutions**:
```python
# 1. Consolidate metadata
zarr.consolidate_metadata(store)
z = zarr.open_consolidated(store)

# 2. Use appropriate chunk sizes (5-100 MB for cloud)
chunks = (2000, 2000)  # Larger chunks for cloud

# 3. Enable sharding
shards = (10000, 10000)  # Groups many chunks
```

### Issue: Concurrent Write Conflicts

**Solution**: Use synchronizers or ensure non-overlapping writes
```python
from zarr import ProcessSynchronizer

sync = ProcessSynchronizer('sync.sync')
z = zarr.open_array('data.zarr', mode='r+', synchronizer=sync)

# Or design workflow so each process writes to separate chunks
```

## Additional Resources

For detailed API documentation, advanced usage, and the latest updates:

- **Official Documentation**: https://zarr.readthedocs.io/
- **Zarr Specifications**: https://zarr-specs.readthedocs.io/
- **GitHub Repository**: https://github.com/zarr-developers/zarr-python
- **Community Chat**: https://gitter.im/zarr-developers/community

**Related Libraries**:
- **Xarray**: https://docs.xarray.dev/ (labeled arrays)
- **Dask**: https://docs.dask.org/ (parallel computing)
- **NumCodecs**: https://numcodecs.readthedocs.io/ (compression codecs)

{
  "format": "aiskill/v1",
  "skill": {
    "name": "zarr-python",
    "slug": "zarr-python",
    "version": "1.0.0",
    "description": "Chunked N-D arrays for cloud storage. Compressed arrays, parallel I/O, S3/GCS integration, NumPy/Dask/Xarray compatible, for large-scale scientific computing pipelines.",
    "category": "Research",
    "tags": [
      "research-science",
      "infrastructure",
      "python"
    ],
    "platforms": [
      "chatgpt",
      "claude-code",
      "cursor"
    ],
    "tested_environment": "Agent Skills convention (SKILL.md) — works with Claude Code, Cursor, ChatGPT/Codex and compatible agents",
    "license": "MIT LICENSE"
  },
  "prompts": {
    "system": "# Zarr Python\r\n\r\n## Overview\r\n\r\nZarr is a Python library for storing large N-dimensional arrays with chunking and compression. Apply this skill for efficient parallel I/O, cloud-native workflows, and seamless integration with NumPy, Dask, and Xarray.\r\n\r\n## Quick Start\r\n\r\n### Installation\r\n\r\n```bash\r\nuv pip install zarr\r\n```\r\n\r\nRequires Python 3.11+. For cloud storage support, install additional packages:\r\n```python\r\nuv pip install s3fs  # For S3\r\nuv pip install gcsfs  # For Google Cloud Storage\r\n```\r\n\r\n### Basic Array Creation\r\n\r\n```python\r\nimport zarr\r\nimport numpy as np\r\n\r\n# Create a 2D array with chunking and compression\r\nz = zarr.create_array(\r\n    store=\"data/my_array.zarr\",\r\n    shape=(10000, 10000),\r\n    chunks=(1000, 1000),\r\n    dtype=\"f4\"\r\n)\r\n\r\n# Write data using NumPy-style indexing\r\nz[:, :] = np.random.random((10000, 10000))\r\n\r\n# Read data\r\ndata = z[0:100, 0:100]  # Returns NumPy array\r\n```\r\n\r\n## Core Operations\r\n\r\n### Creating Arrays\r\n\r\nZarr provides multiple convenience functions for array creation:\r\n\r\n```python\r\n# Create empty array\r\nz = zarr.zeros(shape=(10000, 10000), chunks=(1000, 1000), dtype='f4',\r\n               store='data.zarr')\r\n\r\n# Create filled arrays\r\nz = zarr.ones((5000, 5000), chunks=(500, 500))\r\nz = zarr.full((1000, 1000), fill_value=42, chunks=(100, 100))\r\n\r\n# Create from existing data\r\ndata = np.arange(10000).reshape(100, 100)\r\nz = zarr.array(data, chunks=(10, 10), store='data.zarr')\r\n\r\n# Create like another array\r\nz2 = zarr.zeros_like(z)  # Matches shape, chunks, dtype of z\r\n```\r\n\r\n### Opening Existing Arrays\r\n\r\n```python\r\n# Open array (read/write mode by default)\r\nz = zarr.open_array('data.zarr', mode='r+')\r\n\r\n# Read-only mode\r\nz = zarr.open_array('data.zarr', mode='r')\r\n\r\n# The open() function auto-detects arrays vs groups\r\nz = zarr.open('data.zarr')  # Returns Array or Group\r\n```\r\n\r\n### Reading and Writing Data\r\n\r\nZarr arrays support NumPy-like indexing:\r\n\r\n```python\r\n# Write entire array\r\nz[:] = 42\r\n\r\n# Write slices\r\nz[0, :] = np.arange(100)\r\nz[10:20, 50:60] = np.random.random((10, 10))\r\n\r\n# Read data (returns NumPy array)\r\ndata = z[0:100, 0:100]\r\nrow = z[5, :]\r\n\r\n# Advanced indexing\r\nz.vindex[[0, 5, 10], [2, 8, 15]]  # Coordinate indexing\r\nz.oindex[0:10, [5, 10, 15]]       # Orthogonal indexing\r\nz.blocks[0, 0]                     # Block/chunk indexing\r\n```\r\n\r\n### Resizing and Appending\r\n\r\n```python\r\n# Resize array\r\nz.resize(15000, 15000)  # Expands or shrinks dimensions\r\n\r\n# Append data along an axis\r\nz.append(np.random.random((1000, 10000)), axis=0)  # Adds rows\r\n```\r\n\r\n## Chunking Strategies\r\n\r\nChunking is critical for performance. Choose chunk sizes and shapes based on access patterns.\r\n\r\n### Chunk Size Guidelines\r\n\r\n- **Minimum chunk size**: 1 MB recommended for optimal performance\r\n- **Balance**: Larger chunks = fewer metadata operations; smaller chunks = better parallel access\r\n- **Memory consideration**: Entire chunks must fit in memory during compression\r\n\r\n```python\r\n# Configure chunk size (aim for ~1MB per chunk)\r\n# For float32 data: 1MB = 262,144 elements = 512×512 array\r\nz = zarr.zeros(\r\n    shape=(10000, 10000),\r\n    chunks=(512, 512),  # ~1MB chunks\r\n    dtype='f4'\r\n)\r\n```\r\n\r\n### Aligning Chunks with Access Patterns\r\n\r\n**Critical**: Chunk shape dramatically affects performance based on how data is accessed.\r\n\r\n```python\r\n# If accessing rows frequently (first dimension)\r\nz = zarr.zeros((10000, 10000), chunks=(10, 10000))  # Chunk spans columns\r\n\r\n# If accessing columns frequently (second dimension)\r\nz = zarr.zeros((10000, 10000), chunks=(10000, 10))  # Chunk spans rows\r\n\r\n# For mixed access patterns (balanced approach)\r\nz = zarr.zeros((10000, 10000), chunks=(1000, 1000))  # Square chunks\r\n```\r\n\r\n**Performance example**: For a (200, 200, 200) array, reading along the first dimension:\r\n- Using chunks (1, 200, 200): ~107ms\r\n- Using chunks (200, 200, 1): ~1.65ms (65× faster!)\r\n\r\n### Sharding for Large-Scale Storage\r\n\r\nWhen arrays have millions of small chunks, use sharding to group chunks into larger storage objects:\r\n\r\n```python\r\nfrom zarr.codecs import ShardingCodec, BytesCodec\r\nfrom zarr.codecs.blosc import BloscCodec\r\n\r\n# Create array with sharding\r\nz = zarr.create_array(\r\n    store='data.zarr',\r\n    shape=(100000, 100000),\r\n    chunks=(100, 100),  # Small chunks for access\r\n    shards=(1000, 1000),  # Groups 100 chunks per shard\r\n    dtype='f4'\r\n)\r\n```\r\n\r\n**Benefits**:\r\n- Reduces file system overhead from millions of small files\r\n- Improves cloud storage performance (fewer object requests)\r\n- Prevents filesystem block size waste\r\n\r\n**Important**: Entire shards must fit in memory before writing.\r\n\r\n## Compression\r\n\r\nZarr applies compression per chunk to reduce storage while maintaining fast access.\r\n\r\n### Configuring Compression\r\n\r\n```python\r\nfrom zarr.codecs.blosc import BloscCodec\r\nfrom zarr.codecs import GzipCodec, ZstdCodec\r\n\r\n# Default: Blosc with Zstandard\r\nz = zarr.zeros((1000, 1000), chunks=(100, 100))  # Uses default compression\r\n\r\n# Configure Blosc codec\r\nz = zarr.create_array(\r\n    store='data.zarr',\r\n    shape=(1000, 1000),\r\n    chunks=(100, 100),\r\n    dtype='f4',\r\n    codecs=[BloscCodec(cname='zstd', clevel=5, shuffle='shuffle')]\r\n)\r\n\r\n# Available Blosc compressors: 'blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib', 'zstd'\r\n\r\n# Use Gzip compression\r\nz = zarr.create_array(\r\n    store='data.zarr',\r\n    shape=(1000, 1000),\r\n    chunks=(100, 100),\r\n    dtype='f4',\r\n    codecs=[GzipCodec(level=6)]\r\n)\r\n\r\n# Disable compression\r\nz = zarr.create_array(\r\n    store='data.zarr',\r\n    shape=(1000, 1000),\r\n    chunks=(100, 100),\r\n    dtype='f4',\r\n    codecs=[BytesCodec()]  # No compression\r\n)\r\n```\r\n\r\n### Compression Performance Tips\r\n\r\n- **Blosc** (default): Fast compression/decompression, good for interactive workloads\r\n- **Zstandard**: Better compression ratios, slightly slower than LZ4\r\n- **Gzip**: Maximum compression, slower performance\r\n- **LZ4**: Fastest compression, lower ratios\r\n- **Shuffle**: Enable shuffle filter for better compression on numeric data\r\n\r\n```python\r\n# Optimal for numeric scientific data\r\ncodecs=[BloscCodec(cname='zstd', clevel=5, shuffle='shuffle')]\r\n\r\n# Optimal for speed\r\ncodecs=[BloscCodec(cname='lz4', clevel=1)]\r\n\r\n# Optimal for compression ratio\r\ncodecs=[GzipCodec(level=9)]\r\n```\r\n\r\n## Storage Backends\r\n\r\nZarr supports multiple storage backends through a flexible storage interface.\r\n\r\n### Local Filesystem (Default)\r\n\r\n```python\r\nfrom zarr.storage import LocalStore\r\n\r\n# Explicit store creation\r\nstore = LocalStore('data/my_array.zarr')\r\nz = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))\r\n\r\n# Or use string path (creates LocalStore automatically)\r\nz = zarr.open_array('data/my_array.zarr', mode='w', shape=(1000, 1000),\r\n                    chunks=(100, 100))\r\n```\r\n\r\n### In-Memory Storage\r\n\r\n```python\r\nfrom zarr.storage import MemoryStore\r\n\r\n# Create in-memory store\r\nstore = MemoryStore()\r\nz = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))\r\n\r\n# Data exists only in memory, not persisted\r\n```\r\n\r\n### ZIP File Storage\r\n\r\n```python\r\nfrom zarr.storage import ZipStore\r\n\r\n# Write to ZIP file\r\nstore = ZipStore('data.zip', mode='w')\r\nz = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))\r\nz[:] = np.random.random((1000, 1000))\r\nstore.close()  # IMPORTANT: Must close ZipStore\r\n\r\n# Read from ZIP file\r\nstore = ZipStore('data.zip', mode='r')\r\nz = zarr.open_array(store=store)\r\ndata = z[:]\r\nstore.close()\r\n```\r\n\r\n### Cloud Storage (S3, GCS)\r\n\r\n```python\r\nimport s3fs\r\nimport zarr\r\n\r\n# S3 storage\r\ns3 = s3fs.S3FileSystem(anon=False)  # Use credentials\r\nstore = s3fs.S3Map(root='my-bucket/path/to/array.zarr', s3=s3)\r\nz = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))\r\nz[:] = data\r\n\r\n# Google Cloud Storage\r\nimport gcsfs\r\ngcs = gcsfs.GCSFileSystem(project='my-project')\r\nstore = gcsfs.GCSMap(root='my-bucket/path/to/array.zarr', gcs=gcs)\r\nz = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))\r\n```\r\n\r\n**Cloud Storage Best Practices**:\r\n- Use consolidated metadata to reduce latency: `zarr.consolidate_metadata(store)`\r\n- Align chunk sizes with cloud object sizing (typically 5-100 MB optimal)\r\n- Enable parallel writes using Dask for large-scale data\r\n- Consider sharding to reduce number of objects\r\n\r\n## Groups and Hierarchies\r\n\r\nGroups organize multiple arrays hierarchically, similar to directories or HDF5 groups.\r\n\r\n### Creating and Using Groups\r\n\r\n```python\r\n# Create root group\r\nroot = zarr.group(store='data/hierarchy.zarr')\r\n\r\n# Create sub-groups\r\ntemperature = root.create_group('temperature')\r\nprecipitation = root.create_group('precipitation')\r\n\r\n# Create arrays within groups\r\ntemp_array = temperature.create_array(\r\n    name='t2m',\r\n    shape=(365, 720, 1440),\r\n    chunks=(1, 720, 1440),\r\n    dtype='f4'\r\n)\r\n\r\nprecip_array = precipitation.create_array(\r\n    name='prcp',\r\n    shape=(365, 720, 1440),\r\n    chunks=(1, 720, 1440),\r\n    dtype='f4'\r\n)\r\n\r\n# Access using paths\r\narray = root['temperature/t2m']\r\n\r\n# Visualize hierarchy\r\nprint(root.tree())\r\n# Output:\r\n# /\r\n#  ├── temperature\r\n#  │   └── t2m (365, 720, 1440) f4\r\n#  └── precipitation\r\n#      └── prcp (365, 720, 1440) f4\r\n```\r\n\r\n### H5py-Compatible API\r\n\r\nZarr provides an h5py-compatible interface for familiar HDF5 users:\r\n\r\n```python\r\n# Create group with h5py-style methods\r\nroot = zarr.group('data.zarr')\r\ndataset = root.create_dataset('my_data', shape=(1000, 1000), chunks=(100, 100),\r\n                              dtype='f4')\r\n\r\n# Access like h5py\r\ngrp = root.require_group('subgroup')\r\narr = grp.require_dataset('array', shape=(500, 500), chunks=(50, 50), dtype='i4')\r\n```\r\n\r\n## Attributes and Metadata\r\n\r\nAttach custom metadata to arrays and groups using attributes:\r\n\r\n```python\r\n# Add attributes to array\r\nz = zarr.zeros((1000, 1000), chunks=(100, 100))\r\nz.attrs['description'] = 'Temperature data in Kelvin'\r\nz.attrs['units'] = 'K'\r\nz.attrs['created'] = '2024-01-15'\r\nz.attrs['processing_version'] = 2.1\r\n\r\n# Attributes are stored as JSON\r\nprint(z.attrs['units'])  # Output: K\r\n\r\n# Add attributes to groups\r\nroot = zarr.group('data.zarr')\r\nroot.attrs['project'] = 'Climate Analysis'\r\nroot.attrs['institution'] = 'Research Institute'\r\n\r\n# Attributes persist with the array/group\r\nz2 = zarr.open('data.zarr')\r\nprint(z2.attrs['description'])\r\n```\r\n\r\n**Important**: Attributes must be JSON-serializable (strings, numbers, lists, dicts, booleans, null).\r\n\r\n## Integration with NumPy, Dask, and Xarray\r\n\r\n### NumPy Integration\r\n\r\nZarr arrays implement the NumPy array interface:\r\n\r\n```python\r\nimport numpy as np\r\nimport zarr\r\n\r\nz = zarr.zeros((1000, 1000), chunks=(100, 100))\r\n\r\n# Use NumPy functions directly\r\nresult = np.sum(z, axis=0)  # NumPy operates on Zarr array\r\nmean = np.mean(z[:100, :100])\r\n\r\n# Convert to NumPy array\r\nnumpy_array = z[:]  # Loads entire array into memory\r\n```\r\n\r\n### Dask Integration\r\n\r\nDask provides lazy, parallel computation on Zarr arrays:\r\n\r\n```python\r\nimport dask.array as da\r\nimport zarr\r\n\r\n# Create large Zarr array\r\nz = zarr.open('data.zarr', mode='w', shape=(100000, 100000),\r\n              chunks=(1000, 1000), dtype='f4')\r\n\r\n# Load as Dask array (lazy, no data loaded)\r\ndask_array = da.from_zarr('data.zarr')\r\n\r\n# Perform computations (parallel, out-of-core)\r\nresult = dask_array.mean(axis=0).compute()  # Parallel computation\r\n\r\n# Write Dask array to Zarr\r\nlarge_array = da.random.random((100000, 100000), chunks=(1000, 1000))\r\nda.to_zarr(large_array, 'output.zarr')\r\n```\r\n\r\n**Benefits**:\r\n- Process datasets larger than memory\r\n- Automatic parallel computation across chunks\r\n- Efficient I/O with chunked storage\r\n\r\n### Xarray Integration\r\n\r\nXarray provides labeled, multidimensional arrays with Zarr backend:\r\n\r\n```python\r\nimport xarray as xr\r\nimport zarr\r\n\r\n# Open Zarr store as Xarray Dataset (lazy loading)\r\nds = xr.open_zarr('data.zarr')\r\n\r\n# Dataset includes coordinates and metadata\r\nprint(ds)\r\n\r\n# Access variables\r\ntemperature = ds['temperature']\r\n\r\n# Perform labeled operations\r\nsubset = ds.sel(time='2024-01', lat=slice(30, 60))\r\n\r\n# Write Xarray Dataset to Zarr\r\nds.to_zarr('output.zarr')\r\n\r\n# Create from scratch with coordinates\r\nds = xr.Dataset(\r\n    {\r\n        'temperature': (['time', 'lat', 'lon'], data),\r\n        'precipitation': (['time', 'lat', 'lon'], data2)\r\n    },\r\n    coords={\r\n        'time': pd.date_range('2024-01-01', periods=365),\r\n        'lat': np.arange(-90, 91, 1),\r\n        'lon': np.arange(-180, 180, 1)\r\n    }\r\n)\r\nds.to_zarr('climate_data.zarr')\r\n```\r\n\r\n**Benefits**:\r\n- Named dimensions and coordinates\r\n- Label-based indexing and selection\r\n- Integration with pandas for time series\r\n- NetCDF-like interface familiar to climate/geospatial scientists\r\n\r\n## Parallel Computing and Synchronization\r\n\r\n### Thread-Safe Operations\r\n\r\n```python\r\nfrom zarr import ThreadSynchronizer\r\nimport zarr\r\n\r\n# For multi-threaded writes\r\nsynchronizer = ThreadSynchronizer()\r\nz = zarr.open_array('data.zarr', mode='r+', shape=(10000, 10000),\r\n                    chunks=(1000, 1000), synchronizer=synchronizer)\r\n\r\n# Safe for concurrent writes from multiple threads\r\n# (when writes don't span chunk boundaries)\r\n```\r\n\r\n### Process-Safe Operations\r\n\r\n```python\r\nfrom zarr import ProcessSynchronizer\r\nimport zarr\r\n\r\n# For multi-process writes\r\nsynchronizer = ProcessSynchronizer('sync_data.sync')\r\nz = zarr.open_array('data.zarr', mode='r+', shape=(10000, 10000),\r\n                    chunks=(1000, 1000), synchronizer=synchronizer)\r\n\r\n# Safe for concurrent writes from multiple processes\r\n```\r\n\r\n**Note**:\r\n- Concurrent reads require no synchronization\r\n- Synchronization only needed for writes that may span chunk boundaries\r\n- Each process/thread writing to separate chunks needs no synchronization\r\n\r\n## Consolidated Metadata\r\n\r\nFor hierarchical stores with many arrays, consolidate metadata into a single file to reduce I/O operations:\r\n\r\n```python\r\nimport zarr\r\n\r\n# After creating arrays/groups\r\nroot = zarr.group('data.zarr')\r\n# ... create multiple arrays/groups ...\r\n\r\n# Consolidate metadata\r\nzarr.consolidate_metadata('data.zarr')\r\n\r\n# Open with consolidated metadata (faster, especially on cloud storage)\r\nroot = zarr.open_consolidated('data.zarr')\r\n```\r\n\r\n**Benefits**:\r\n- Reduces metadata read operations from N (one per array) to 1\r\n- Critical for cloud storage (reduces latency)\r\n- Speeds up `tree()` operations and group traversal\r\n\r\n**Cautions**:\r\n- Metadata can become stale if arrays update without re-consolidation\r\n- Not suitable for frequently-updated datasets\r\n- Multi-writer scenarios may have inconsistent reads\r\n\r\n## Performance Optimization\r\n\r\n### Checklist for Optimal Performance\r\n\r\n1. **Chunk Size**: Aim for 1-10 MB per chunk\r\n   ```python\r\n   # For float32: 1MB = 262,144 elements\r\n   chunks = (512, 512)  # 512×512×4 bytes = ~1MB\r\n   ```\r\n\r\n2. **Chunk Shape**: Align with access patterns\r\n   ```python\r\n   # Row-wise access → chunk spans columns: (small, large)\r\n   # Column-wise access → chunk spans rows: (large, small)\r\n   # Random access → balanced: (medium, medium)\r\n   ```\r\n\r\n3. **Compression**: Choose based on workload\r\n   ```python\r\n   # Interactive/fast: BloscCodec(cname='lz4')\r\n   # Balanced: BloscCodec(cname='zstd', clevel=5)\r\n   # Maximum compression: GzipCodec(level=9)\r\n   ```\r\n\r\n4. **Storage Backend**: Match to environment\r\n   ```python\r\n   # Local: LocalStore (default)\r\n   # Cloud: S3Map/GCSMap with consolidated metadata\r\n   # Temporary: MemoryStore\r\n   ```\r\n\r\n5. **Sharding**: Use for large-scale datasets\r\n   ```python\r\n   # When you have millions of small chunks\r\n   shards=(10*chunk_size, 10*chunk_size)\r\n   ```\r\n\r\n6. **Parallel I/O**: Use Dask for large operations\r\n   ```python\r\n   import dask.array as da\r\n   dask_array = da.from_zarr('data.zarr')\r\n   result = dask_array.compute(scheduler='threads', num_workers=8)\r\n   ```\r\n\r\n### Profiling and Debugging\r\n\r\n```python\r\n# Print detailed array information\r\nprint(z.info)\r\n\r\n# Output includes:\r\n# - Type, shape, chunks, dtype\r\n# - Compression codec and level\r\n# - Storage size (compressed vs uncompressed)\r\n# - Storage location\r\n\r\n# Check storage size\r\nprint(f\"Compressed size: {z.nbytes_stored / 1e6:.2f} MB\")\r\nprint(f\"Uncompressed size: {z.nbytes / 1e6:.2f} MB\")\r\nprint(f\"Compression ratio: {z.nbytes / z.nbytes_stored:.2f}x\")\r\n```\r\n\r\n## Common Patterns and Best Practices\r\n\r\n### Pattern: Time Series Data\r\n\r\n```python\r\n# Store time series with time as first dimension\r\n# This allows efficient appending of new time steps\r\nz = zarr.open('timeseries.zarr', mode='a',\r\n              shape=(0, 720, 1440),  # Start with 0 time steps\r\n              chunks=(1, 720, 1440),  # One time step per chunk\r\n              dtype='f4')\r\n\r\n# Append new time steps\r\nnew_data = np.random.random((1, 720, 1440))\r\nz.append(new_data, axis=0)\r\n```\r\n\r\n### Pattern: Large Matrix Operations\r\n\r\n```python\r\nimport dask.array as da\r\n\r\n# Create large matrix in Zarr\r\nz = zarr.open('matrix.zarr', mode='w',\r\n              shape=(100000, 100000),\r\n              chunks=(1000, 1000),\r\n              dtype='f8')\r\n\r\n# Use Dask for parallel computation\r\ndask_z = da.from_zarr('matrix.zarr')\r\nresult = (dask_z @ dask_z.T).compute()  # Parallel matrix multiply\r\n```\r\n\r\n### Pattern: Cloud-Native Workflow\r\n\r\n```python\r\nimport s3fs\r\nimport zarr\r\n\r\n# Write to S3\r\ns3 = s3fs.S3FileSystem()\r\nstore = s3fs.S3Map(root='s3://my-bucket/data.zarr', s3=s3)\r\n\r\n# Create array with appropriate chunking for cloud\r\nz = zarr.open_array(store=store, mode='w',\r\n                    shape=(10000, 10000),\r\n                    chunks=(500, 500),  # ~1MB chunks\r\n                    dtype='f4')\r\nz[:] = data\r\n\r\n# Consolidate metadata for faster reads\r\nzarr.consolidate_metadata(store)\r\n\r\n# Read from S3 (anywhere, anytime)\r\nstore_read = s3fs.S3Map(root='s3://my-bucket/data.zarr', s3=s3)\r\nz_read = zarr.open_consolidated(store_read)\r\nsubset = z_read[0:100, 0:100]\r\n```\r\n\r\n### Pattern: Format Conversion\r\n\r\n```python\r\n# HDF5 to Zarr\r\nimport h5py\r\nimport zarr\r\n\r\nwith h5py.File('data.h5', 'r') as h5:\r\n    dataset = h5['dataset_name']\r\n    z = zarr.array(dataset[:],\r\n                   chunks=(1000, 1000),\r\n                   store='data.zarr')\r\n\r\n# NumPy to Zarr\r\nimport numpy as np\r\ndata = np.load('data.npy')\r\nz = zarr.array(data, chunks='auto', store='data.zarr')\r\n\r\n# Zarr to NetCDF (via Xarray)\r\nimport xarray as xr\r\nds = xr.open_zarr('data.zarr')\r\nds.to_netcdf('data.nc')\r\n```\r\n\r\n## Common Issues and Solutions\r\n\r\n### Issue: Slow Performance\r\n\r\n**Diagnosis**: Check chunk size and alignment\r\n```python\r\nprint(z.chunks)  # Are chunks appropriate size?\r\nprint(z.info)    # Check compression ratio\r\n```\r\n\r\n**Solutions**:\r\n- Increase chunk size to 1-10 MB\r\n- Align chunks with access pattern\r\n- Try different compression codecs\r\n- Use Dask for parallel operations\r\n\r\n### Issue: High Memory Usage\r\n\r\n**Cause**: Loading entire array or large chunks into memory\r\n\r\n**Solutions**:\r\n```python\r\n# Don't load entire array\r\n# Bad: data = z[:]\r\n# Good: Process in chunks\r\nfor i in range(0, z.shape[0], 1000):\r\n    chunk = z[i:i+1000, :]\r\n    process(chunk)\r\n\r\n# Or use Dask for automatic chunking\r\nimport dask.array as da\r\ndask_z = da.from_zarr('data.zarr')\r\nresult = dask_z.mean().compute()  # Processes in chunks\r\n```\r\n\r\n### Issue: Cloud Storage Latency\r\n\r\n**Solutions**:\r\n```python\r\n# 1. Consolidate metadata\r\nzarr.consolidate_metadata(store)\r\nz = zarr.open_consolidated(store)\r\n\r\n# 2. Use appropriate chunk sizes (5-100 MB for cloud)\r\nchunks = (2000, 2000)  # Larger chunks for cloud\r\n\r\n# 3. Enable sharding\r\nshards = (10000, 10000)  # Groups many chunks\r\n```\r\n\r\n### Issue: Concurrent Write Conflicts\r\n\r\n**Solution**: Use synchronizers or ensure non-overlapping writes\r\n```python\r\nfrom zarr import ProcessSynchronizer\r\n\r\nsync = ProcessSynchronizer('sync.sync')\r\nz = zarr.open_array('data.zarr', mode='r+', synchronizer=sync)\r\n\r\n# Or design workflow so each process writes to separate chunks\r\n```\r\n\r\n## Additional Resources\r\n\r\nFor detailed API documentation, advanced usage, and the latest updates:\r\n\r\n- **Official Documentation**: https://zarr.readthedocs.io/\r\n- **Zarr Specifications**: https://zarr-specs.readthedocs.io/\r\n- **GitHub Repository**: https://github.com/zarr-developers/zarr-python\r\n- **Community Chat**: https://gitter.im/zarr-developers/community\r\n\r\n**Related Libraries**:\r\n- **Xarray**: https://docs.xarray.dev/ (labeled arrays)\r\n- **Dask**: https://docs.dask.org/ (parallel computing)\r\n- **NumCodecs**: https://numcodecs.readthedocs.io/ (compression codecs)",
    "user_template": ""
  },
  "variables": [],
  "install": {
    "steps": [
      {
        "title": "Download & unzip",
        "detail": "Download the .zip package from SkillVault and unzip it. The zarr-python/ folder contains SKILL.md plus any companion scripts, references and assets."
      },
      {
        "title": "Copy into your agent skills directory",
        "detail": "Move the folder into your agent's skills directory — e.g. ~/.claude/skills/ (Claude Code), .codex/skills/, .cursor/skills/ or your project's .claude/skills/ folder — then restart or reload the session."
      },
      {
        "title": "Use it",
        "detail": "Describe the task in natural language; the agent loads the skill automatically when the request matches the skill description."
      }
    ]
  },
  "usage": {
    "steps": []
  },
  "metadata": {
    "author": "SkillVault",
    "source": "https://skillvault.top/skills/zarr-python",
    "exported_at": "2026-11-03T07:40:45.096Z"
  }
}

format: aiskill/v1
skill:
  name: zarr-python
  slug: zarr-python
  version: 1.0.0
  description: >-
    Chunked N-D arrays for cloud storage. Compressed arrays, parallel I/O, S3/GCS integration,
    NumPy/Dask/Xarray compatible, for large-scale scientific computing pipelines.
  category: Research
  tags:
    - research-science
    - infrastructure
    - python
  platforms:
    - chatgpt
    - claude-code
    - cursor
  tested_environment: >-
    Agent Skills convention (SKILL.md) — works with Claude Code, Cursor, ChatGPT/Codex and
    compatible agents
  license: MIT LICENSE
prompts:
  system: "# Zarr Python\r\n\r\n## Overview\r\n\r\nZarr is a Python library for storing large N-dimensional arrays with chunking and compression. Apply this skill for efficient parallel I/O, cloud-native workflows, and seamless integration with NumPy, Dask, and Xarray.\r\n\r\n## Quick Start\r\n\r\n### Installation\r\n\r\n```bash\r\nuv pip install zarr\r\n```\r\n\r\nRequires Python 3.11+. For cloud storage support, install additional packages:\r\n```python\r\nuv pip install s3fs  # For S3\r\nuv pip install gcsfs  # For Google Cloud Storage\r\n```\r\n\r\n### Basic Array Creation\r\n\r\n```python\r\nimport zarr\r\nimport numpy as np\r\n\r\n# Create a 2D array with chunking and compression\r\nz = zarr.create_array(\r\n    store=\"data/my_array.zarr\",\r\n    shape=(10000, 10000),\r\n    chunks=(1000, 1000),\r\n    dtype=\"f4\"\r\n)\r\n\r\n# Write data using NumPy-style indexing\r\nz[:, :] = np.random.random((10000, 10000))\r\n\r\n# Read data\r\ndata = z[0:100, 0:100]  # Returns NumPy array\r\n```\r\n\r\n## Core Operations\r\n\r\n### Creating Arrays\r\n\r\nZarr provides multiple convenience functions for array creation:\r\n\r\n```python\r\n# Create empty array\r\nz = zarr.zeros(shape=(10000, 10000), chunks=(1000, 1000), dtype='f4',\r\n               store='data.zarr')\r\n\r\n# Create filled arrays\r\nz = zarr.ones((5000, 5000), chunks=(500, 500))\r\nz = zarr.full((1000, 1000), fill_value=42, chunks=(100, 100))\r\n\r\n# Create from existing data\r\ndata = np.arange(10000).reshape(100, 100)\r\nz = zarr.array(data, chunks=(10, 10), store='data.zarr')\r\n\r\n# Create like another array\r\nz2 = zarr.zeros_like(z)  # Matches shape, chunks, dtype of z\r\n```\r\n\r\n### Opening Existing Arrays\r\n\r\n```python\r\n# Open array (read/write mode by default)\r\nz = zarr.open_array('data.zarr', mode='r+')\r\n\r\n# Read-only mode\r\nz = zarr.open_array('data.zarr', mode='r')\r\n\r\n# The open() function auto-detects arrays vs groups\r\nz = zarr.open('data.zarr')  # Returns Array or Group\r\n```\r\n\r\n### Reading and Writing Data\r\n\r\nZarr arrays support NumPy-like indexing:\r\n\r\n```python\r\n# Write entire array\r\nz[:] = 42\r\n\r\n# Write slices\r\nz[0, :] = np.arange(100)\r\nz[10:20, 50:60] = np.random.random((10, 10))\r\n\r\n# Read data (returns NumPy array)\r\ndata = z[0:100, 0:100]\r\nrow = z[5, :]\r\n\r\n# Advanced indexing\r\nz.vindex[[0, 5, 10], [2, 8, 15]]  # Coordinate indexing\r\nz.oindex[0:10, [5, 10, 15]]       # Orthogonal indexing\r\nz.blocks[0, 0]                     # Block/chunk indexing\r\n```\r\n\r\n### Resizing and Appending\r\n\r\n```python\r\n# Resize array\r\nz.resize(15000, 15000)  # Expands or shrinks dimensions\r\n\r\n# Append data along an axis\r\nz.append(np.random.random((1000, 10000)), axis=0)  # Adds rows\r\n```\r\n\r\n## Chunking Strategies\r\n\r\nChunking is critical for performance. Choose chunk sizes and shapes based on access patterns.\r\n\r\n### Chunk Size Guidelines\r\n\r\n- **Minimum chunk size**: 1 MB recommended for optimal performance\r\n- **Balance**: Larger chunks = fewer metadata operations; smaller chunks = better parallel access\r\n- **Memory consideration**: Entire chunks must fit in memory during compression\r\n\r\n```python\r\n# Configure chunk size (aim for ~1MB per chunk)\r\n# For float32 data: 1MB = 262,144 elements = 512×512 array\r\nz = zarr.zeros(\r\n    shape=(10000, 10000),\r\n    chunks=(512, 512),  # ~1MB chunks\r\n    dtype='f4'\r\n)\r\n```\r\n\r\n### Aligning Chunks with Access Patterns\r\n\r\n**Critical**: Chunk shape dramatically affects performance based on how data is accessed.\r\n\r\n```python\r\n# If accessing rows frequently (first dimension)\r\nz = zarr.zeros((10000, 10000), chunks=(10, 10000))  # Chunk spans columns\r\n\r\n# If accessing columns frequently (second dimension)\r\nz = zarr.zeros((10000, 10000), chunks=(10000, 10))  # Chunk spans rows\r\n\r\n# For mixed access patterns (balanced approach)\r\nz = zarr.zeros((10000, 10000), chunks=(1000, 1000))  # Square chunks\r\n```\r\n\r\n**Performance example**: For a (200, 200, 200) array, reading along the first dimension:\r\n- Using chunks (1, 200, 200): ~107ms\r\n- Using chunks (200, 200, 1): ~1.65ms (65× faster!)\r\n\r\n### Sharding for Large-Scale Storage\r\n\r\nWhen arrays have millions of small chunks, use sharding to group chunks into larger storage objects:\r\n\r\n```python\r\nfrom zarr.codecs import ShardingCodec, BytesCodec\r\nfrom zarr.codecs.blosc import BloscCodec\r\n\r\n# Create array with sharding\r\nz = zarr.create_array(\r\n    store='data.zarr',\r\n    shape=(100000, 100000),\r\n    chunks=(100, 100),  # Small chunks for access\r\n    shards=(1000, 1000),  # Groups 100 chunks per shard\r\n    dtype='f4'\r\n)\r\n```\r\n\r\n**Benefits**:\r\n- Reduces file system overhead from millions of small files\r\n- Improves cloud storage performance (fewer object requests)\r\n- Prevents filesystem block size waste\r\n\r\n**Important**: Entire shards must fit in memory before writing.\r\n\r\n## Compression\r\n\r\nZarr applies compression per chunk to reduce storage while maintaining fast access.\r\n\r\n### Configuring Compression\r\n\r\n```python\r\nfrom zarr.codecs.blosc import BloscCodec\r\nfrom zarr.codecs import GzipCodec, ZstdCodec\r\n\r\n# Default: Blosc with Zstandard\r\nz = zarr.zeros((1000, 1000), chunks=(100, 100))  # Uses default compression\r\n\r\n# Configure Blosc codec\r\nz = zarr.create_array(\r\n    store='data.zarr',\r\n    shape=(1000, 1000),\r\n    chunks=(100, 100),\r\n    dtype='f4',\r\n    codecs=[BloscCodec(cname='zstd', clevel=5, shuffle='shuffle')]\r\n)\r\n\r\n# Available Blosc compressors: 'blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib', 'zstd'\r\n\r\n# Use Gzip compression\r\nz = zarr.create_array(\r\n    store='data.zarr',\r\n    shape=(1000, 1000),\r\n    chunks=(100, 100),\r\n    dtype='f4',\r\n    codecs=[GzipCodec(level=6)]\r\n)\r\n\r\n# Disable compression\r\nz = zarr.create_array(\r\n    store='data.zarr',\r\n    shape=(1000, 1000),\r\n    chunks=(100, 100),\r\n    dtype='f4',\r\n    codecs=[BytesCodec()]  # No compression\r\n)\r\n```\r\n\r\n### Compression Performance Tips\r\n\r\n- **Blosc** (default): Fast compression/decompression, good for interactive workloads\r\n- **Zstandard**: Better compression ratios, slightly slower than LZ4\r\n- **Gzip**: Maximum compression, slower performance\r\n- **LZ4**: Fastest compression, lower ratios\r\n- **Shuffle**: Enable shuffle filter for better compression on numeric data\r\n\r\n```python\r\n# Optimal for numeric scientific data\r\ncodecs=[BloscCodec(cname='zstd', clevel=5, shuffle='shuffle')]\r\n\r\n# Optimal for speed\r\ncodecs=[BloscCodec(cname='lz4', clevel=1)]\r\n\r\n# Optimal for compression ratio\r\ncodecs=[GzipCodec(level=9)]\r\n```\r\n\r\n## Storage Backends\r\n\r\nZarr supports multiple storage backends through a flexible storage interface.\r\n\r\n### Local Filesystem (Default)\r\n\r\n```python\r\nfrom zarr.storage import LocalStore\r\n\r\n# Explicit store creation\r\nstore = LocalStore('data/my_array.zarr')\r\nz = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))\r\n\r\n# Or use string path (creates LocalStore automatically)\r\nz = zarr.open_array('data/my_array.zarr', mode='w', shape=(1000, 1000),\r\n                    chunks=(100, 100))\r\n```\r\n\r\n### In-Memory Storage\r\n\r\n```python\r\nfrom zarr.storage import MemoryStore\r\n\r\n# Create in-memory store\r\nstore = MemoryStore()\r\nz = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))\r\n\r\n# Data exists only in memory, not persisted\r\n```\r\n\r\n### ZIP File Storage\r\n\r\n```python\r\nfrom zarr.storage import ZipStore\r\n\r\n# Write to ZIP file\r\nstore = ZipStore('data.zip', mode='w')\r\nz = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))\r\nz[:] = np.random.random((1000, 1000))\r\nstore.close()  # IMPORTANT: Must close ZipStore\r\n\r\n# Read from ZIP file\r\nstore = ZipStore('data.zip', mode='r')\r\nz = zarr.open_array(store=store)\r\ndata = z[:]\r\nstore.close()\r\n```\r\n\r\n### Cloud Storage (S3, GCS)\r\n\r\n```python\r\nimport s3fs\r\nimport zarr\r\n\r\n# S3 storage\r\ns3 = s3fs.S3FileSystem(anon=False)  # Use credentials\r\nstore = s3fs.S3Map(root='my-bucket/path/to/array.zarr', s3=s3)\r\nz = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))\r\nz[:] = data\r\n\r\n# Google Cloud Storage\r\nimport gcsfs\r\ngcs = gcsfs.GCSFileSystem(project='my-project')\r\nstore = gcsfs.GCSMap(root='my-bucket/path/to/array.zarr', gcs=gcs)\r\nz = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))\r\n```\r\n\r\n**Cloud Storage Best Practices**:\r\n- Use consolidated metadata to reduce latency: `zarr.consolidate_metadata(store)`\r\n- Align chunk sizes with cloud object sizing (typically 5-100 MB optimal)\r\n- Enable parallel writes using Dask for large-scale data\r\n- Consider sharding to reduce number of objects\r\n\r\n## Groups and Hierarchies\r\n\r\nGroups organize multiple arrays hierarchically, similar to directories or HDF5 groups.\r\n\r\n### Creating and Using Groups\r\n\r\n```python\r\n# Create root group\r\nroot = zarr.group(store='data/hierarchy.zarr')\r\n\r\n# Create sub-groups\r\ntemperature = root.create_group('temperature')\r\nprecipitation = root.create_group('precipitation')\r\n\r\n# Create arrays within groups\r\ntemp_array = temperature.create_array(\r\n    name='t2m',\r\n    shape=(365, 720, 1440),\r\n    chunks=(1, 720, 1440),\r\n    dtype='f4'\r\n)\r\n\r\nprecip_array = precipitation.create_array(\r\n    name='prcp',\r\n    shape=(365, 720, 1440),\r\n    chunks=(1, 720, 1440),\r\n    dtype='f4'\r\n)\r\n\r\n# Access using paths\r\narray = root['temperature/t2m']\r\n\r\n# Visualize hierarchy\r\nprint(root.tree())\r\n# Output:\r\n# /\r\n#  ├── temperature\r\n#  │   └── t2m (365, 720, 1440) f4\r\n#  └── precipitation\r\n#      └── prcp (365, 720, 1440) f4\r\n```\r\n\r\n### H5py-Compatible API\r\n\r\nZarr provides an h5py-compatible interface for familiar HDF5 users:\r\n\r\n```python\r\n# Create group with h5py-style methods\r\nroot = zarr.group('data.zarr')\r\ndataset = root.create_dataset('my_data', shape=(1000, 1000), chunks=(100, 100),\r\n                              dtype='f4')\r\n\r\n# Access like h5py\r\ngrp = root.require_group('subgroup')\r\narr = grp.require_dataset('array', shape=(500, 500), chunks=(50, 50), dtype='i4')\r\n```\r\n\r\n## Attributes and Metadata\r\n\r\nAttach custom metadata to arrays and groups using attributes:\r\n\r\n```python\r\n# Add attributes to array\r\nz = zarr.zeros((1000, 1000), chunks=(100, 100))\r\nz.attrs['description'] = 'Temperature data in Kelvin'\r\nz.attrs['units'] = 'K'\r\nz.attrs['created'] = '2024-01-15'\r\nz.attrs['processing_version'] = 2.1\r\n\r\n# Attributes are stored as JSON\r\nprint(z.attrs['units'])  # Output: K\r\n\r\n# Add attributes to groups\r\nroot = zarr.group('data.zarr')\r\nroot.attrs['project'] = 'Climate Analysis'\r\nroot.attrs['institution'] = 'Research Institute'\r\n\r\n# Attributes persist with the array/group\r\nz2 = zarr.open('data.zarr')\r\nprint(z2.attrs['description'])\r\n```\r\n\r\n**Important**: Attributes must be JSON-serializable (strings, numbers, lists, dicts, booleans, null).\r\n\r\n## Integration with NumPy, Dask, and Xarray\r\n\r\n### NumPy Integration\r\n\r\nZarr arrays implement the NumPy array interface:\r\n\r\n```python\r\nimport numpy as np\r\nimport zarr\r\n\r\nz = zarr.zeros((1000, 1000), chunks=(100, 100))\r\n\r\n# Use NumPy functions directly\r\nresult = np.sum(z, axis=0)  # NumPy operates on Zarr array\r\nmean = np.mean(z[:100, :100])\r\n\r\n# Convert to NumPy array\r\nnumpy_array = z[:]  # Loads entire array into memory\r\n```\r\n\r\n### Dask Integration\r\n\r\nDask provides lazy, parallel computation on Zarr arrays:\r\n\r\n```python\r\nimport dask.array as da\r\nimport zarr\r\n\r\n# Create large Zarr array\r\nz = zarr.open('data.zarr', mode='w', shape=(100000, 100000),\r\n              chunks=(1000, 1000), dtype='f4')\r\n\r\n# Load as Dask array (lazy, no data loaded)\r\ndask_array = da.from_zarr('data.zarr')\r\n\r\n# Perform computations (parallel, out-of-core)\r\nresult = dask_array.mean(axis=0).compute()  # Parallel computation\r\n\r\n# Write Dask array to Zarr\r\nlarge_array = da.random.random((100000, 100000), chunks=(1000, 1000))\r\nda.to_zarr(large_array, 'output.zarr')\r\n```\r\n\r\n**Benefits**:\r\n- Process datasets larger than memory\r\n- Automatic parallel computation across chunks\r\n- Efficient I/O with chunked storage\r\n\r\n### Xarray Integration\r\n\r\nXarray provides labeled, multidimensional arrays with Zarr backend:\r\n\r\n```python\r\nimport xarray as xr\r\nimport zarr\r\n\r\n# Open Zarr store as Xarray Dataset (lazy loading)\r\nds = xr.open_zarr('data.zarr')\r\n\r\n# Dataset includes coordinates and metadata\r\nprint(ds)\r\n\r\n# Access variables\r\ntemperature = ds['temperature']\r\n\r\n# Perform labeled operations\r\nsubset = ds.sel(time='2024-01', lat=slice(30, 60))\r\n\r\n# Write Xarray Dataset to Zarr\r\nds.to_zarr('output.zarr')\r\n\r\n# Create from scratch with coordinates\r\nds = xr.Dataset(\r\n    {\r\n        'temperature': (['time', 'lat', 'lon'], data),\r\n        'precipitation': (['time', 'lat', 'lon'], data2)\r\n    },\r\n    coords={\r\n        'time': pd.date_range('2024-01-01', periods=365),\r\n        'lat': np.arange(-90, 91, 1),\r\n        'lon': np.arange(-180, 180, 1)\r\n    }\r\n)\r\nds.to_zarr('climate_data.zarr')\r\n```\r\n\r\n**Benefits**:\r\n- Named dimensions and coordinates\r\n- Label-based indexing and selection\r\n- Integration with pandas for time series\r\n- NetCDF-like interface familiar to climate/geospatial scientists\r\n\r\n## Parallel Computing and Synchronization\r\n\r\n### Thread-Safe Operations\r\n\r\n```python\r\nfrom zarr import ThreadSynchronizer\r\nimport zarr\r\n\r\n# For multi-threaded writes\r\nsynchronizer = ThreadSynchronizer()\r\nz = zarr.open_array('data.zarr', mode='r+', shape=(10000, 10000),\r\n                    chunks=(1000, 1000), synchronizer=synchronizer)\r\n\r\n# Safe for concurrent writes from multiple threads\r\n# (when writes don't span chunk boundaries)\r\n```\r\n\r\n### Process-Safe Operations\r\n\r\n```python\r\nfrom zarr import ProcessSynchronizer\r\nimport zarr\r\n\r\n# For multi-process writes\r\nsynchronizer = ProcessSynchronizer('sync_data.sync')\r\nz = zarr.open_array('data.zarr', mode='r+', shape=(10000, 10000),\r\n                    chunks=(1000, 1000), synchronizer=synchronizer)\r\n\r\n# Safe for concurrent writes from multiple processes\r\n```\r\n\r\n**Note**:\r\n- Concurrent reads require no synchronization\r\n- Synchronization only needed for writes that may span chunk boundaries\r\n- Each process/thread writing to separate chunks needs no synchronization\r\n\r\n## Consolidated Metadata\r\n\r\nFor hierarchical stores with many arrays, consolidate metadata into a single file to reduce I/O operations:\r\n\r\n```python\r\nimport zarr\r\n\r\n# After creating arrays/groups\r\nroot = zarr.group('data.zarr')\r\n# ... create multiple arrays/groups ...\r\n\r\n# Consolidate metadata\r\nzarr.consolidate_metadata('data.zarr')\r\n\r\n# Open with consolidated metadata (faster, especially on cloud storage)\r\nroot = zarr.open_consolidated('data.zarr')\r\n```\r\n\r\n**Benefits**:\r\n- Reduces metadata read operations from N (one per array) to 1\r\n- Critical for cloud storage (reduces latency)\r\n- Speeds up `tree()` operations and group traversal\r\n\r\n**Cautions**:\r\n- Metadata can become stale if arrays update without re-consolidation\r\n- Not suitable for frequently-updated datasets\r\n- Multi-writer scenarios may have inconsistent reads\r\n\r\n## Performance Optimization\r\n\r\n### Checklist for Optimal Performance\r\n\r\n1. **Chunk Size**: Aim for 1-10 MB per chunk\r\n   ```python\r\n   # For float32: 1MB = 262,144 elements\r\n   chunks = (512, 512)  # 512×512×4 bytes = ~1MB\r\n   ```\r\n\r\n2. **Chunk Shape**: Align with access patterns\r\n   ```python\r\n   # Row-wise access → chunk spans columns: (small, large)\r\n   # Column-wise access → chunk spans rows: (large, small)\r\n   # Random access → balanced: (medium, medium)\r\n   ```\r\n\r\n3. **Compression**: Choose based on workload\r\n   ```python\r\n   # Interactive/fast: BloscCodec(cname='lz4')\r\n   # Balanced: BloscCodec(cname='zstd', clevel=5)\r\n   # Maximum compression: GzipCodec(level=9)\r\n   ```\r\n\r\n4. **Storage Backend**: Match to environment\r\n   ```python\r\n   # Local: LocalStore (default)\r\n   # Cloud: S3Map/GCSMap with consolidated metadata\r\n   # Temporary: MemoryStore\r\n   ```\r\n\r\n5. **Sharding**: Use for large-scale datasets\r\n   ```python\r\n   # When you have millions of small chunks\r\n   shards=(10*chunk_size, 10*chunk_size)\r\n   ```\r\n\r\n6. **Parallel I/O**: Use Dask for large operations\r\n   ```python\r\n   import dask.array as da\r\n   dask_array = da.from_zarr('data.zarr')\r\n   result = dask_array.compute(scheduler='threads', num_workers=8)\r\n   ```\r\n\r\n### Profiling and Debugging\r\n\r\n```python\r\n# Print detailed array information\r\nprint(z.info)\r\n\r\n# Output includes:\r\n# - Type, shape, chunks, dtype\r\n# - Compression codec and level\r\n# - Storage size (compressed vs uncompressed)\r\n# - Storage location\r\n\r\n# Check storage size\r\nprint(f\"Compressed size: {z.nbytes_stored / 1e6:.2f} MB\")\r\nprint(f\"Uncompressed size: {z.nbytes / 1e6:.2f} MB\")\r\nprint(f\"Compression ratio: {z.nbytes / z.nbytes_stored:.2f}x\")\r\n```\r\n\r\n## Common Patterns and Best Practices\r\n\r\n### Pattern: Time Series Data\r\n\r\n```python\r\n# Store time series with time as first dimension\r\n# This allows efficient appending of new time steps\r\nz = zarr.open('timeseries.zarr', mode='a',\r\n              shape=(0, 720, 1440),  # Start with 0 time steps\r\n              chunks=(1, 720, 1440),  # One time step per chunk\r\n              dtype='f4')\r\n\r\n# Append new time steps\r\nnew_data = np.random.random((1, 720, 1440))\r\nz.append(new_data, axis=0)\r\n```\r\n\r\n### Pattern: Large Matrix Operations\r\n\r\n```python\r\nimport dask.array as da\r\n\r\n# Create large matrix in Zarr\r\nz = zarr.open('matrix.zarr', mode='w',\r\n              shape=(100000, 100000),\r\n              chunks=(1000, 1000),\r\n              dtype='f8')\r\n\r\n# Use Dask for parallel computation\r\ndask_z = da.from_zarr('matrix.zarr')\r\nresult = (dask_z @ dask_z.T).compute()  # Parallel matrix multiply\r\n```\r\n\r\n### Pattern: Cloud-Native Workflow\r\n\r\n```python\r\nimport s3fs\r\nimport zarr\r\n\r\n# Write to S3\r\ns3 = s3fs.S3FileSystem()\r\nstore = s3fs.S3Map(root='s3://my-bucket/data.zarr', s3=s3)\r\n\r\n# Create array with appropriate chunking for cloud\r\nz = zarr.open_array(store=store, mode='w',\r\n                    shape=(10000, 10000),\r\n                    chunks=(500, 500),  # ~1MB chunks\r\n                    dtype='f4')\r\nz[:] = data\r\n\r\n# Consolidate metadata for faster reads\r\nzarr.consolidate_metadata(store)\r\n\r\n# Read from S3 (anywhere, anytime)\r\nstore_read = s3fs.S3Map(root='s3://my-bucket/data.zarr', s3=s3)\r\nz_read = zarr.open_consolidated(store_read)\r\nsubset = z_read[0:100, 0:100]\r\n```\r\n\r\n### Pattern: Format Conversion\r\n\r\n```python\r\n# HDF5 to Zarr\r\nimport h5py\r\nimport zarr\r\n\r\nwith h5py.File('data.h5', 'r') as h5:\r\n    dataset = h5['dataset_name']\r\n    z = zarr.array(dataset[:],\r\n                   chunks=(1000, 1000),\r\n                   store='data.zarr')\r\n\r\n# NumPy to Zarr\r\nimport numpy as np\r\ndata = np.load('data.npy')\r\nz = zarr.array(data, chunks='auto', store='data.zarr')\r\n\r\n# Zarr to NetCDF (via Xarray)\r\nimport xarray as xr\r\nds = xr.open_zarr('data.zarr')\r\nds.to_netcdf('data.nc')\r\n```\r\n\r\n## Common Issues and Solutions\r\n\r\n### Issue: Slow Performance\r\n\r\n**Diagnosis**: Check chunk size and alignment\r\n```python\r\nprint(z.chunks)  # Are chunks appropriate size?\r\nprint(z.info)    # Check compression ratio\r\n```\r\n\r\n**Solutions**:\r\n- Increase chunk size to 1-10 MB\r\n- Align chunks with access pattern\r\n- Try different compression codecs\r\n- Use Dask for parallel operations\r\n\r\n### Issue: High Memory Usage\r\n\r\n**Cause**: Loading entire array or large chunks into memory\r\n\r\n**Solutions**:\r\n```python\r\n# Don't load entire array\r\n# Bad: data = z[:]\r\n# Good: Process in chunks\r\nfor i in range(0, z.shape[0], 1000):\r\n    chunk = z[i:i+1000, :]\r\n    process(chunk)\r\n\r\n# Or use Dask for automatic chunking\r\nimport dask.array as da\r\ndask_z = da.from_zarr('data.zarr')\r\nresult = dask_z.mean().compute()  # Processes in chunks\r\n```\r\n\r\n### Issue: Cloud Storage Latency\r\n\r\n**Solutions**:\r\n```python\r\n# 1. Consolidate metadata\r\nzarr.consolidate_metadata(store)\r\nz = zarr.open_consolidated(store)\r\n\r\n# 2. Use appropriate chunk sizes (5-100 MB for cloud)\r\nchunks = (2000, 2000)  # Larger chunks for cloud\r\n\r\n# 3. Enable sharding\r\nshards = (10000, 10000)  # Groups many chunks\r\n```\r\n\r\n### Issue: Concurrent Write Conflicts\r\n\r\n**Solution**: Use synchronizers or ensure non-overlapping writes\r\n```python\r\nfrom zarr import ProcessSynchronizer\r\n\r\nsync = ProcessSynchronizer('sync.sync')\r\nz = zarr.open_array('data.zarr', mode='r+', synchronizer=sync)\r\n\r\n# Or design workflow so each process writes to separate chunks\r\n```\r\n\r\n## Additional Resources\r\n\r\nFor detailed API documentation, advanced usage, and the latest updates:\r\n\r\n- **Official Documentation**: https://zarr.readthedocs.io/\r\n- **Zarr Specifications**: https://zarr-specs.readthedocs.io/\r\n- **GitHub Repository**: https://github.com/zarr-developers/zarr-python\r\n- **Community Chat**: https://gitter.im/zarr-developers/community\r\n\r\n**Related Libraries**:\r\n- **Xarray**: https://docs.xarray.dev/ (labeled arrays)\r\n- **Dask**: https://docs.dask.org/ (parallel computing)\r\n- **NumCodecs**: https://numcodecs.readthedocs.io/ (compression codecs)"
  user_template: ""
variables: []
install:
  steps:
    - title: Download & unzip
      detail: >-
        Download the .zip package from SkillVault and unzip it. The zarr-python/ folder contains
        SKILL.md plus any companion scripts, references and assets.
    - title: Copy into your agent skills directory
      detail: >-
        Move the folder into your agent's skills directory — e.g. ~/.claude/skills/ (Claude Code),
        .codex/skills/, .cursor/skills/ or your project's .claude/skills/ folder — then restart or
        reload the session.
    - title: Use it
      detail: >-
        Describe the task in natural language; the agent loads the skill automatically when the
        request matches the skill description.
usage:
  steps: []
metadata:
  author: SkillVault
  source: https://skillvault.top/skills/zarr-python
  exported_at: "2026-11-03T07:40:45.096Z"