Library Integration

---
name: library-integration
description: Guides adding support for new image/video augmentation libraries to the benchmark suite. Use when integrating a new library, adding library support, or when the user mentions adding a new augmentation library to test.
---

# Library Integration

Add support for new augmentation libraries to the benchmark suite.

## Integration Checklist

```
- [ ] Create transform implementation file
- [ ] Create requirements file
- [ ] Register matrix spec maps, requirements, and environment groups
- [ ] Add or update config examples if the library affects paper/common run configs
- [ ] Add or update matrix/config/job tests
- [ ] Test with sample data
- [ ] Generate baseline results
- [ ] Update documentation
```

## Step 1: Create Transform Implementation

Create `benchmark/transforms/{library}_impl.py`:

```python
"""Transform implementations for {library}."""

# Import library
import {library}

# Define library name
LIBRARY = "{library}"

def __call__(transform, image):
    """Apply transform to image.

    Adapt this to library's calling convention:
    - Some use transform(image)
    - Some use transform(image=image)
    - Some require specific formats
    """
    return transform(image)

# Define transforms to benchmark
TRANSFORMS = [
    {
        "name": "HorizontalFlip",
        "transform": {library}.HorizontalFlip(),
    },
    {
        "name": "VerticalFlip",
        "transform": {library}.VerticalFlip(),
    },
    # Add more transforms...
]
```

### Image Loading

Add loader to `benchmark/utils.py` if needed:

```python
def get_image_loader(library: str):
    """Get appropriate image loader for library."""
    if library == "new_library":
        def load_new_library(path):
            # Load in library's expected format
            return img
        return load_new_library
```

## Step 2: Create Requirements File

Create `requirements/{library}.txt`:

```txt
{library}>=1.0.0
numpy>=1.19.0
opencv-python>=4.5.0
```

Add to `requirements/requirements.txt` if base dependencies needed.

## Step 3: Register Matrix Support

### For image benchmarks

Add the library to the image spec and requirement maps in `benchmark/matrix.py`:

```python
IMAGE_SPECS["newlib"] = "benchmark/transforms/newlib_impl.py"
IMAGE_REQUIREMENTS["newlib"] = "requirements/newlib.txt"
```

### For video benchmarks

Add the library to `VIDEO_SPECS` and `VIDEO_REQUIREMENTS` in `benchmark/matrix.py`.
If it can share dependencies with existing libraries, add it to the relevant `ENV_GROUPS` entry instead of creating a separate venv.
If it needs a non-standard backend, add a `BenchmarkJob.backend` value in `benchmark/jobs.py` and route it through
`benchmark/orchestrator.py`; do not add a library-specific branch to `benchmark/cli.py`.
If it needs different media defaults or slow-skip thresholds, add them to `benchmark/policy.py`, not to individual
runners.
If it needs new user-facing run options, add them to `BenchmarkRunConfig` in `benchmark/config/models.py` first, then
update YAML loading/overrides in `benchmark/config/resolve.py`, dry-run expansion in `benchmark/config/plan.py`, and
examples under `configs/`.
Keep result filename changes in `benchmark/output_naming.py`, and detached GCP path changes in `benchmark/cloud/paths.py`.

Update tests when the matrix changes:

```bash
python -m pytest tests/test_matrix.py tests/test_config_models.py tests/test_config_plan.py tests/test_jobs_orchestrator.py
```

## Step 4: Test Integration

```bash
# Prefer a config-first smoke run, with CLI overrides for temporary values.
python -m benchmark.cli plan --config configs/examples/local_rgb_micro_cpu.yaml --num-items 10 --num-runs 1
python -m benchmark.cli run --config configs/examples/local_rgb_micro_cpu.yaml \
  --output test_output/newlib \
  --num-items 10 \
  --num-runs 1

# Verify JSON output
python -c "import json; print(json.load(open('test_output/newlib/image-rgb/micro/newlib_micro_results.json'))['metadata']['library_versions'])"
```

## Step 5: Generate Baseline Results

```bash
# Full benchmark run
python -m benchmark.cli run --config configs/examples/local_rgb_micro_cpu.yaml \
  --output output/newlib_rgb_micro \
  --num-items 2000 \
  --num-runs 5
```

## Step 6: Update Documentation

Create `docs/images/newlib_metadata.yaml` or `docs/videos/newlib_metadata.yaml`:

```yaml
library_name: NewLib
version: "1.0.0"
description: Brief description of the library
documentation: https://newlib.readthedocs.io
repository: https://github.com/org/newlib
```

Run comparison:
```bash
./tools/update_docs.sh
```

## Common Issues

### Import Errors
- Ensure library is in requirements file
- Check virtual environment activation
- Verify compatible Python version

### Transform Failures
- Check transform API matches library version
- Verify image format (RGB vs BGR, HWC vs CHW)
- Test transforms individually first

### Performance Issues
- Verify thread settings (`OMP_NUM_THREADS=1`)
- Check warmup convergence
- Monitor early stopping conditions

## Video Library Integration

For video libraries, create `{library}_video_impl.py` and adapt:

```python
import numpy as np


def __call__(transform, video):
    """Apply transform to one clip. Shape conventions:
    - (T, H, W, C) for Albumentations (NumPy) — use batch API when available
    - (T, C, H, W) for torch / Kornia tensors
    """
    # Albumentations: native multi-frame API (one param draw per clip)
    return np.ascontiguousarray(transform(images=video)["images"])
```

For Kornia/torchvision, see `kornia_video_impl.py` / `torchvision_video_impl.py`. Video loaders differ — check `benchmark/utils.py` (`get_video_loader`) and `benchmark/video_runner.py`.