C2f Chart Eval

---
name: c2f-chart-eval
description: Evaluates a model's ability to classify chart types from images using a coarse-to-fine curriculum learning approach. It measures performance on broad and fine-grained chart categories to assess hierarchical classification capabilities. Use when the user wants to benchmark on ICPR 2022 UB Unitec PMC Dataset, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.04683
  bibtex_key: [REDACTED]
  confidence: high
---

# c2f-chart-eval

> C2F-CHART: A Curriculum Learning Approach to Chart Classification — Shaheen et al. (2024) (arXiv:2409.04683, 2024)

## What this evaluates

Evaluates a model's ability to classify chart types from images using a coarse-to-fine curriculum learning approach. It measures performance on broad and fine-grained chart categories to assess hierarchical classification capabilities.

## Datasets

- **ICPR 2022 UB Unitec PMC Dataset** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `F1-score` **(primary)** — range: percent
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Reported as a percentage.
- `Precision` — range: percent
  - Ratio of correctly predicted positive observations to the total predicted positives.
- `Recall` — range: percent
  - Ratio of correctly predicted positive observations to all observations in the actual class.

## Input / output format

**Input**: RGB image of a chart.

**Output**: Predicted chart type class label.

## Scoring recipe

```python
def compute_macro_f1(preds, gold):
    classes = set(gold) | set(preds)
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1_scores.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    return sum(f1_scores) / len(f1_scores) * 100
```

## Common pitfalls

- The best-performing level-1 checkpoint does not guarantee the best level-2 performance; a combinatorial search over multiple checkpoints is required.
- Model ensembling (averaging logits) outperforms model souping (averaging weights) for this task, contrary to some prior assumptions.

## Evidence (verbatim from paper)

> We benchmarked our results on the testing dataset, called ICPR 2022 UB Unitec PMC Dataset, and compared them with previous work. ... our testing precision, recall, and F1-score demonstrate superior performance to all competition participants and Swin-Chart.

## Citation

```bibtex
@misc{shaheen2024c2fchart,
  title={C2F-CHART: A Curriculum Learning Approach to Chart Classification},
  author={Shaheen et al. (2024)},
  year={2024},
  note={arXiv:2409.04683}
}
```

- arXiv: 2409.04683