Learn Framework Eval

---
name: learn-framework-eval
description: Evaluates a unified framework for multi-task domain adaptation few-shot learning across image classification, object detection, and video classification. It probes the model's ability to adapt to new domains and scale label budgets incrementally from 1-shot to full dataset size. Use when the user wants to benchmark on DomainNet, Office-Home, Office31, Pool and Car, xView, UCF101, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.16275
  bibtex_key: ravichandran2024learn
  confidence: medium
---

# learn-framework-eval

> LEARN: A Unified Framework for Multi-Task Domain Adapt Few-Shot Learning — Ravichandran et al. (2024) (arXiv:2412.16275, 2024)

## What this evaluates

Evaluates a unified framework for multi-task domain adaptation few-shot learning across image classification, object detection, and video classification. It probes the model's ability to adapt to new domains and scale label budgets incrementally from 1-shot to full dataset size.

## Datasets

- **DomainNet** — total ?; splits: train (-1)
- **Office-Home** — total ?; splits: train (-1)
- **Office31** — total ?; splits: train (-1)
- **Pool and Car** — total ?; splits: train (-1)
- **xView** — total ?; splits: train (-1)
- **UCF101** — total ?; splits: train (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Top-1 classification accuracy, calculated as the number of correctly predicted classes divided by the total number of samples.
- `mAP` — range: [0, 1]
  - Mean Average Precision for object detection, averaging precision across all classes and recall thresholds.

## Input / output format

**Input**: Images or video frames with associated class labels or bounding boxes, sampled according to configurable n-shot or label budget constraints.

**Output**: Predicted class labels or bounding boxes with confidence scores.

## Scoring recipe

```python
def compute_accuracy(preds, gold):
    correct = sum(1 for p, g in zip(preds, gold) if p == g)
    return correct / len(gold)

def compute_map(preds, gold):
    # Standard mAP calculation over IoU thresholds
    return mean_average_precision(preds, gold)
```

## Common pitfalls

- The evaluation supports incremental label budgets (1-shot to full N) in a single run, which may be mistaken for separate independent experiments.
- Qualitative analysis notes potential false positives that may actually be missing ground truth, requiring careful manual verification of detection metrics.

## Evidence (verbatim from paper)

> Table 2 provides an overview of the results across the different tasks, algorithms, and datasets for different incremental label budgets ranging from 1-shot to the full size of the training set. ... DETReg and CutLER are able to achieve a mAP of nearly 0.5 on the PoolCar dataset with only 2 labels per class as shown in [Table 2]. ... PACMAC has the best accuracy with the accuracy score ranging between 48.8% to 56.6% for 1-shot through 10-shot tasks.

## Citation

```bibtex
@misc{ravichandran2024learn,
  title={LEARN: A Unified Framework for Multi-Task Domain Adapt Few-Shot Learning},
  author={Ravichandran et al. (2024)},
  year={2024},
  note={arXiv:2412.16275}
}
```

- arXiv: 2412.16275