Local Prompt Ood Eval

---
name: local-prompt-ood-eval
description: This evaluation probes a vision-language model's ability to distinguish in-distribution images from out-of-distribution samples using few-shot prompt tuning. It specifically tests fine-grained regional outlier detection by measuring how well the model separates known classes from diverse OOD datasets and semantically similar near-OOD subsets. Use when the user wants to benchmark on ImageNet-1K & OOD combination, or asks about evaluating this task. Reports FPR95.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.04796
  bibtex_key: zeng2024localprompt
  confidence: high
---

# local-prompt-ood-eval

> Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detection — Zeng et al. (2024) (arXiv:2409.04796, 2024)

## What this evaluates

This evaluation probes a vision-language model's ability to distinguish in-distribution images from out-of-distribution samples using few-shot prompt tuning. It specifically tests fine-grained regional outlier detection by measuring how well the model separates known classes from diverse OOD datasets and semantically similar near-OOD subsets.

## Datasets

- **ImageNet-1K & OOD combination** — total ?; splits: train (-1), test (-1)

## Metrics

- `FPR95` **(primary)** — range: percent
  - False positive rate when the true positive rate is fixed at 95%. Lower is better.
- `AUROC` — range: percent
  - Area under the receiver operating characteristic curve plotting true positive rate against false positive rate. Higher is better.
- `ID accuracy` — range: percent
  - Top-1 classification accuracy on the in-distribution test set. Higher is better.

## Input / output format

**Input**: Image patches fed into a frozen CLIP-Base/16 vision encoder, combined with learnable local prompts and frozen global prompts.

**Output**: OOD score (similarity between image and class prompts) for each class, and the predicted class label (argmax of ID scores).

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, id_preds, id_golds):
    # predictions: OOD scores, gold_labels: 0 for ID, 1 for OOD
    tpr, fpr, _ = roc_curve(gold_labels, predictions)
    fpr95 = fpr[tpr >= 0.95][0]
    auroc = auc(fpr, tpr) * 100  # Convert to percent
    id_acc = sum(p == g for p, g in zip(id_preds, id_golds)) / len(id_golds) * 100
    return {'FPR95': fpr95, 'AUROC': auroc, 'ID accuracy': id_acc}
```

## Common pitfalls

- OOD datasets are pooled together for evaluation but reported separately and averaged; mixing them up skews results.
- FPR95 is reported as a percentage (0–100) in tables, not a probability (0–1), which can cause confusion when comparing with other works.
- Few-shot tuning uses only a subset of the training set (4 or 16 shots per class), so results are not directly comparable to zero-shot or full-tuning baselines without noting the shot count.

## Evidence (verbatim from paper)

> Evaluation metrics. We report the following metrics for evaluation: (1) the area under the receiver operating characteristic curve (AUROC); (2) false positive rate of OOD samples when true positive rate of ID samples is 95% (FPR95); (3) in-distribution data classification accuracy (ID accuracy).

## Citation

```bibtex
@misc{zeng2024localprompt,
  title={Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detection},
  author={Zeng et al. (2024)},
  year={2024},
  note={arXiv:2409.04796}
}
```

- arXiv: 2409.04796