Camera Trap Ai Eval

---
name: camera-trap-ai-eval
description: Evaluates AI-powered platforms for processing camera trap images, measuring their ability to detect animals and classify species against ground truth labels. Use when the user wants to benchmark on Colombian rainforest camera trap images, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2202.02283
  bibtex_key: velez2022choosing
  confidence: high
---

# camera-trap-ai-eval

> Choosing an Appropriate Platform and Workflow for Processing Camera Trap Data using Artificial Intelligence — Vélez et al. (2022) (arXiv:2202.02283, 2022)

## What this evaluates

Evaluates AI-powered platforms for processing camera trap images, measuring their ability to detect animals and classify species against ground truth labels.

## Datasets

- **Colombian rainforest camera trap images** — total 112247; splits: test (112247)

## Metrics

- `precision` — range: [0, 1]
  - Precision = TP / (TP + FP). Measures the proportion of predicted positive instances that are actually correct.
- `recall` — range: [0, 1]
  - Recall = TP / (TP + FN). Measures the proportion of actual positive instances that are correctly identified.
- `F1 score` **(primary)** — range: [0, 1]
  - F1 = 2 * (Precision * Recall) / (Precision + Recall). Harmonic mean of precision and recall, used as the headline composite metric.

## Input / output format

**Input**: Raw camera trap images.

**Output**: Predicted species labels or object categories (e.g., 'Animal', 'Blank') with associated confidence scores.

## Scoring recipe

```python
def evaluate(predictions, gold, ct):
    tp = fp = fn = 0
    for pred, true_label in zip(predictions, gold):
        if pred.confidence >= ct:
            if pred.label == true_label:
                tp += 1
            else:
                fp += 1
        else:
            fn += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return precision, recall, f1
```

## Common pitfalls

- Species classifiers often exhibit high precision but very low recall (<54%), meaning they miss many actual animals despite being accurate when they do predict.
- Performance is highly sensitive to the confidence threshold; lowering it increases recall but decreases precision, requiring careful trade-off selection.
- Platforms are best used in semi-automated workflows requiring expert review of low-confidence or 'Blank' predictions rather than fully automated deployment.

## Evidence (verbatim from paper)

> WI and MLWIC2 had high precision values for some species (at a CT = 0.65), suggesting that when computer vision predicted a species label, it was usually correct. However, all species had low recall values (less than 54%, at a CT = 0.65), indicating that these platforms missed many of the animals present in the images. MD, which identifies broader categories of objects, had a precision of 98% at a 93% recall (at a CT = 0.65) for the "Animal" class, and consequently also had a high F1 score.

## Citation

```bibtex
@misc{velez2022choosing,
  title={Choosing an Appropriate Platform and Workflow for Processing Camera Trap Data using Artificial Intelligence},
  author={Vélez et al. (2022)},
  year={2022},
  note={arXiv:2202.02283}
}
```

- arXiv: 2202.02283