Doc Img Class Retrieval Eval

---
name: doc-img-class-retrieval-eval
description: Evaluates the ability of deep convolutional neural networks and hand-crafted feature methods to classify scanned document images into predefined categories and retrieve semantically similar documents from a large corpus. Use when the user wants to benchmark on SmallTobacco, BigTobacco, or asks about evaluating this task. Reports classification accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1502.07058
  bibtex_key: harley2015docimg
  confidence: high
---

# doc-img-class-retrieval-eval

> Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval — Harley et al. (2015) (arXiv:1502.07058, 2015)

## What this evaluates

Evaluates the ability of deep convolutional neural networks and hand-crafted feature methods to classify scanned document images into predefined categories and retrieve semantically similar documents from a large corpus.

## Datasets

- **SmallTobacco** — total 3482; splits: train (800), val (200), test (2482)
- **BigTobacco** — total 400000; splits: train (320000), val (40000), test (40000)

## Metrics

- `classification accuracy` **(primary)** — range: [0, 1]
  - The fraction of test images correctly assigned to their true category label.
- `mAP@10` — range: [0, 1]
  - Mean Average Precision at rank 10. AP = sum_{k=1}^{10} (P(k) * rel(k)) / number of relevant documents. mAP is the average AP across all test queries.

## Input / output format

**Input**: Scanned document images, typically resized to 227x227 pixels for CNN processing, or converted to bag-of-words feature vectors.

**Output**: Predicted class label (classification) or a ranked list of training set images (retrieval).

## Scoring recipe

```python
# Classification Accuracy
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = correct / len(gold_labels)

# mAP@10
aps = []
for query, gold_label in zip(test_queries, test_labels):
    ranked_docs = get_ranked_docs(query)[:10]
    rel_count = 0
    ap = 0.0
    for k, doc in enumerate(ranked_docs, 1):
        if doc.label == gold_label:
            rel_count += 1
            ap += rel_count / k
    ap /= max(rel_count, 1)
    aps.append(ap)
mAP10 = sum(aps) / len(aps)
```

## Common pitfalls

- SmallTobacco results report the median performance over 10 random train/val/test splits, not the mean.
- Retrieval relevance is strictly defined by matching the document's class label, ignoring other visual or semantic similarities.
- Feature vectors larger than 128 dimensions are compressed via PCA and L2-normalized before retrieval, which can alter distance-based rankings.

## Evidence (verbatim from paper)

> The SmallTobacco dataset was split as in the related work: 800 images were used for training, 200 for validation, and the remainder for testing. Since this is a small dataset, 10 random splits in those proportions were created; results reflect the median performance from those splits. In the case of retrieval, the median was selected based on mean average precision at the 10th retrieval (mAP@10). ... Retrieval was measured using mean average precision (mAP). Average precision computes the average value of precision as a function of recall on some interval. Formally, the discrete version of this metric is given by AP = sum_{k=1}^n (P(k) * rel(k)) / number of relevant documents... Mean average precision is simply the average precision summed over all queries, divided by the number of queries.

## Citation

```bibtex
@misc{harley2015docimg,
  title={Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval},
  author={Harley et al. (2015)},
  year={2015},
  note={arXiv:1502.07058}
}
```

- arXiv: 1502.07058