Lamot Eval

---
name: lamot-eval
description: Evaluates open-vocabulary multi-object tracking by requiring models to follow multiple targets in video sequences guided by natural language descriptions. It probes the model's ability to jointly perform text-grounded detection and long-term identity association across diverse, real-world scenarios. Use when the user wants to benchmark on LaMOT, or asks about evaluating this task. Reports HOTA.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.08324
  bibtex_key: li2024lamot
  confidence: high
---

# lamot-eval

> LaMOT: Language-Guided Multi-Object Tracking — Li et al. (2024) (arXiv:2406.08324, 2024)

## What this evaluates

Evaluates open-vocabulary multi-object tracking by requiring models to follow multiple targets in video sequences guided by natural language descriptions. It probes the model's ability to jointly perform text-grounded detection and long-term identity association across diverse, real-world scenarios.

## Datasets

- **LaMOT** — total 1660; splits: full (1660); repo https://github.com/Nathan-Li123/LaMOT

## Metrics

- `HOTA` **(primary)** — range: [0, 100] percent
  - Higher Order Tracking Accuracy, a composite metric that balances detection accuracy (DetA) and association accuracy (AssA) across IoU thresholds.
- `MOTA` — range: [0, 100] percent
  - Multiple Object Tracking Accuracy, measuring tracking performance based on false negatives, false positives, and identity switches.
- `IDF1` — range: [0, 100] percent
  - ID F1 score, the harmonic mean of ID precision and ID recall, evaluating identity consistency over time.

## Input / output format

**Input**: Video frames with corresponding natural language descriptions specifying the target objects to track.

**Output**: Time-series bounding boxes with assigned track IDs, matched to the provided language queries.

## Scoring recipe

```python
def evaluate_mot(preds, gold):
    tp, fp, fn = match_boxes(preds, gold, iou_thresh=0.5)
    ids = count_identity_switches(preds, gold)
    mota = 1 - (fn + fp + ids) / max(len(gold), 1)
    hota = compute_hota(tp, fp, fn, ids)
    idf1 = harmonic_mean(id_precision(preds, gold), id_recall(preds, gold))
    return {'HOTA': hota, 'MOTA': mota, 'IDF1': idf1}
```

## Common pitfalls

- Feature-based ReID yields minimal gains because language descriptions make visually distinct targets appear semantically similar.
- Models fail on open-vocabulary queries if not explicitly designed for text-grounded detection, as they cannot recognize unseen categories.
- Scenario difficulty varies drastically; drone footage is significantly harder due to high target density and low resolution, skewing aggregate scores if not analyzed per scenario.

## Evidence (verbatim from paper)

> As in Tab.[3], LaMOTer achieves the best performance. For instance, it achieves 48.45% in HOTA, and 47.66% in IDF1.

## Citation

```bibtex
@misc{li2024lamot,
  title={LaMOT: Language-Guided Multi-Object Tracking},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2406.08324}
}
```

- arXiv: 2406.08324