Open Set Recognition Eval

---
name: open-set-recognition-eval
description: This evaluation probes a model's ability to correctly classify instances into known classes while accurately detecting and rejecting instances from unknown, unseen classes. It measures both outlier detection capability via ROC analysis and multi-class recognition performance including an explicit unknown category. Use when the user wants to benchmark on MNIST, MS Challenge, Android Genome, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1802.04365
  bibtex_key: hassen2018learning
  confidence: high
---

# open-set-recognition-eval

> Learning a Neural-network-based Representation for Open Set Recognition — Hassen et al. (2018) (arXiv:1802.04365, 2018)

## What this evaluates

This evaluation probes a model's ability to correctly classify instances into known classes while accurately detecting and rejecting instances from unknown, unseen classes. It measures both outlier detection capability via ROC analysis and multi-class recognition performance including an explicit unknown category.

## Datasets

- **MNIST** — total ?; splits: train (-1), val (-1), test (-1)
- **MS Challenge** — total 10260; splits: train (-1), val (-1), test (-1)
- **Android Genome** — total 986; splits: train (-1), val (-1), test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, computed using the model's outlier score to calculate True Positive Rate and False Positive Rate at varying thresholds. Reported up to 100% FPR and 10% FPR.
- `F-Score` — range: [0, 1]
  - Harmonic mean of precision and recall. Calculated per class (K known classes + 1 unknown class) and then macro-averaged across all K+1 classes.

## Input / output format

**Input**: Raw images (MNIST) or extracted function call graph features represented as adjacency matrices/edge frequency vectors (malware datasets).

**Output**: A continuous outlier score for detection tasks, or a discrete class label (one of K known classes or unknown) for recognition tasks, determined by comparing the outlier score to a threshold.

## Scoring recipe

```python
# Outlier Detection (AUC)
scores = model.compute_outlier_score(X_test)
y_true = [1 if x in unknown_classes else 0 for x in X_test]
auc = roc_auc_score(y_true, scores) # computed up to 100% or 10% FPR

# Open Set Recognition (F-Score)
preds = []
for x, score in zip(X_test, scores):
    if score > threshold:
        preds.append("unknown")
    else:
        preds.append(model.predict_class(x))
f1s = []
for label in known_classes + ["unknown"]:
    p, r, f1, _ = precision_recall_fscore_support(y_true, preds, labels=[label])
    f1s.append(f1)
avg_f1 = sum(f1s) / len(f1s)
```

## Common pitfalls

- The threshold for classifying an instance as unknown must be tuned using only known-class validation data, not the full test set.
- AUC should be reported at restricted FPR ranges (e.g., ≤10%) for practical relevance, as 100% FPR AUC can mask poor performance at low false positive rates.
- Malware datasets require specific feature extraction (FCG adjacency matrices) and class filtering (≥40 samples) to ensure valid train/val/test splits.

## Evidence (verbatim from paper)

> We use average precision, recall and f-score metrics to evaluate open set recognition performance and t-test for statistical significance. Precision, recall and f-score are first calculated for each of the K known class labels and the one “unknown” label. Then the average overall the K+1 classes is calculated.

## Citation

```bibtex
@misc{hassen2018learning,
  title={Learning a Neural-network-based Representation for Open Set Recognition},
  author={Hassen et al. (2018)},
  year={2018},
  note={arXiv:1802.04365}
}
```

- arXiv: 1802.04365