Deart Eval

---
name: deart-eval
description: Evaluates object detection and pose classification capabilities on historical European paintings. Probes a model's ability to recognize culturally heritage-specific entities and human-like poses in artistic contexts rather than natural photographs. Use when the user wants to benchmark on DEArt, or asks about evaluating this task. Reports mAP@0.5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.01226
  bibtex_key: [REDACTED]
  confidence: high
---

# deart-eval

> DEArt: Dataset of European Art — Reshetnikov et al. (2022) (arXiv:2211.01226, 2022)

## What this evaluates

Evaluates object detection and pose classification capabilities on historical European paintings. Probes a model's ability to recognize culturally heritage-specific entities and human-like poses in artistic contexts rather than natural photographs.

## Datasets

- **DEArt** — total 15000; splits: train (10500), val (2250), test (2250)

## Metrics

- `mAP@0.5` **(primary)** — range: [0, 1]
  - Mean average precision at an Intersection over Union (IoU) threshold of 0.5, averaged across all 69 object classes.
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall for pose classification across 12 classes, reported as both macro and weighted averages.

## Input / output format

**Input**: RGB images of European paintings (XII–XVIII century) with bounding box annotations for 69 object classes and pose labels for human-like objects.

**Output**: Bounding box coordinates, class labels for detected objects, and pose classification labels for human-like objects.

## Scoring recipe

```python
def compute_mAP_at_05(predictions, gold, iou_thresh=0.5):
    aps = []
    for cls in classes:
        cls_preds = [p for p in predictions if p['class'] == cls]
        cls_gold = [g for g in gold if g['class'] == cls]
        ap = average_precision(cls_preds, cls_gold, iou_thresh)
        aps.append(ap)
    return mean(aps)

def compute_f1(predictions, gold):
    return f1_score(gold, predictions, average='weighted')
```

## Common pitfalls

- The dataset is highly unbalanced, causing minority pose classes (e.g., 'ride', 'fall') to severely drag down overall F1 scores.
- Models pretrained on natural image datasets (MS COCO, PASCAL VOC) suffer significant precision drops on artistic styles compared to domain-specific training.
- Complex or non-iconic images can act as noise and contaminate training if the model lacks capacity to capture visual variability across many classes.

## Evidence (verbatim from paper)

> Given that our dataset is in Pascal VOC format, we chose AP@0.5 per class and mAP@0.5 as evaluation metrics. ... Evaluation over all 12 classes shows F1=0.471, with weighted F1=0.89.

## Citation

```bibtex
@misc{reshetnikov2022deart,
  title={DEArt: Dataset of European Art},
  author={Reshetnikov et al. (2022)},
  year={2022},
  note={arXiv:2211.01226}
}
```

- arXiv: 2211.01226