Hasper Eval

---
name: hasper-eval
description: Evaluates the ability of computer vision models to classify hand shadow puppet silhouettes into one of 15 distinct categories. It probes feature extraction robustness, particularly for rotationally asymmetric and visually similar silhouettes under varying lighting and motion dynamics. Use when the user wants to benchmark on HaSPeR, or asks about evaluating this task. Reports top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.10360
  bibtex_key: raiyan2024hasper
  confidence: high
---

# hasper-eval

> HaSPeR: An Image Repository for Hand Shadow Puppet Recognition — Raiyan et al. (2024) (arXiv:2408.10360, 2024)

## What this evaluates

Evaluates the ability of computer vision models to classify hand shadow puppet silhouettes into one of 15 distinct categories. It probes feature extraction robustness, particularly for rotationally asymmetric and visually similar silhouettes under varying lighting and motion dynamics.

## Datasets

- **HaSPeR** — total 15000; splits: train (-1), val (-1), test (-1); repo https://github.com/Starscream-11813/HaSPeR

## Metrics

- `top-1 accuracy` **(primary)** — range: percent
  - Fraction of correctly predicted classes out of the total number of samples.
- `top-k accuracy (k=2,3)` — range: percent
  - Fraction of samples where the true class appears within the top k highest predicted probabilities.
- `Precision` — range: [0, 1]
  - Ratio of true positive predictions to all positive predictions for each class.
- `Recall` — range: [0, 1]
  - Ratio of true positive predictions to all actual positives for each class.
- `F1-score` — range: [0, 1]
  - Harmonic mean of Precision and Recall.

## Input / output format

**Input**: Resized RGB image of a hand shadow puppet silhouette.

**Output**: Single class label from 15 predefined categories.

## Scoring recipe

```python
def compute_metrics(predictions, labels, k=1):
    correct = sum(1 for p, l in zip(predictions, labels) if p == l)
    accuracy = correct / len(labels)
    # Top-k accuracy, precision, recall, F1 computed via standard sklearn.metrics
    return accuracy
```

## Common pitfalls

- Dataset exhibits slight class imbalance, particularly in professional source clips.
- Silhouettes are rotationally asymmetric and incongruent, requiring specific augmentations like random rotation and affine transforms.
- High inter-class similarity (e.g., Crab vs. Bird/Moose/Rabbit) leads to frequent confusion matrix errors.

## Evidence (verbatim from paper)

> We use top-$k$ validation accuracy values (with $k\=1,2,3$), Precision, Recall, and F1-score as evaluation metrics to perform comparative analyses among the aforementioned models.

## Citation

```bibtex
@misc{raiyan2024hasper,
  title={HaSPeR: An Image Repository for Hand Shadow Puppet Recognition},
  author={Raiyan et al. (2024)},
  year={2024},
  note={arXiv:2408.10360}
}
```

- arXiv: 2408.10360