Medic Eval

---
name: medic-eval
description: Evaluates image classification models on disaster-related social media images across four interdependent tasks: disaster type, informativeness, humanitarian relevance, and damage severity. It tests both single-task and multi-task learning capabilities, including multiclass and multilabel classification settings. Use when the user wants to benchmark on MEDIC, or asks about evaluating this task. Reports weighted F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2108.12828
  bibtex_key: alam2021medic
  confidence: high
---

# medic-eval

> MEDIC: A Multi-Task Learning Dataset for Disaster Image Classification — Alam et al. (2021) (arXiv:2108.12828, 2021)

## What this evaluates

Evaluates image classification models on disaster-related social media images across four interdependent tasks: disaster type, informativeness, humanitarian relevance, and damage severity. It tests both single-task and multi-task learning capabilities, including multiclass and multilabel classification settings.

## Datasets

- **MEDIC** — total 71198; splits: train (-1), dev (-1), test (-1); repo https://github.com/firojalam/medic

## Metrics

- `weighted F1-score` **(primary)** — range: percent
  - Weighted average of per-class F1-scores, where each class's F1 is weighted by its support (number of true instances in the dataset).
- `micro F1-score` — range: percent
  - Micro-averaged F1 computed by aggregating true positives, false positives, and false negatives across all classes before calculating precision, recall, and F1.
- `humming loss` — range: other
  - A loss function for multilabel classification that penalizes incorrect label assignments based on the Hamming distance between predicted and true label sets.

## Input / output format

**Input**: Raw social media images paired with ground-truth annotations for up to four tasks: disaster type (multiclass), informativeness (binary), humanitarian relevance (multiclass), and damage severity (multiclass). Multilabel variants also provide multiple active labels per task.

**Output**: Predicted class labels or probability distributions for each task. Multiclass tasks use softmax outputs; multilabel tasks use sigmoid outputs.

## Scoring recipe

```python
def compute_weighted_f1(y_true, y_pred, classes):
    f1s = []
    for c in classes:
        tp = sum(1 for yt, yp in zip(y_true, y_pred) if yt == c and yp == c)
        fp = sum(1 for yt, yp in zip(y_true, y_pred) if yt != c and yp == c)
        fn = sum(1 for yt, yp in zip(y_true, y_pred) if yt == c and yp != c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append((f1, sum(1 for yt in y_true if yt == c)))
    return sum(f * w for f, w in f1s) / sum(w for _, w in f1s)
```

## Common pitfalls

- The dataset exhibits class imbalance, but the authors note that the majority-class baseline performs similarly across tasks, suggesting imbalance does not drastically skew results.
- Multilabel metrics (micro F1, humming loss) are reported separately from multiclass metrics (weighted F1), making direct cross-setting comparisons invalid.
- Multi-task learning experiments use equal task weights (w_i=1) and hard parameter sharing, which may not reflect optimal task balancing strategies.

## Evidence (verbatim from paper)

> To measure the performance of each classifier and for each task setting, we use weighted average precision (P), recall (R), and F1-score (F1), which are widely used in the literature. For the multilabel experiments we computed micro average precision (P), recall (R), F1-score (F1) and humming loss, which are commonly used metrics [89, 90].

## Citation

```bibtex
@misc{alam2021medic,
  title={MEDIC: A Multi-Task Learning Dataset for Disaster Image Classification},
  author={Alam et al. (2021)},
  year={2021},
  note={arXiv:2108.12828}
}
```

- arXiv: 2108.12828