P Bench Eval

---
name: p-bench-eval
description: Evaluates multimodal large language models' ability to recognize and respond to specific individuals in images using in-context learning. It probes robustness to complex scenes (multiple people, augmentations) and the capability to correctly reject unanswerable queries. Use when the user wants to benchmark on P-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.07113
  bibtex_key: pi2024personalized
  confidence: high
---

# p-bench-eval

> Personalized Visual Instruction Tuning — Pi et al. (2024) (arXiv:2410.07113, 2024)

## What this evaluates

Evaluates multimodal large language models' ability to recognize and respond to specific individuals in images using in-context learning. It probes robustness to complex scenes (multiple people, augmentations) and the capability to correctly reject unanswerable queries.

## Datasets

- **P-Bench** — total ?; splits: test (-1); repo https://github.com/sterzhang/PVIT

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correct responses across multiple-choice questions and description tasks. For unanswerable queries, it measures the rejection rate (correctly identifying that the question cannot be answered).

## Input / output format

**Input**: Image(s) of a target individual and/or scene, paired with text prompts containing multiple-choice questions or open-ended description requests.

**Output**: Model-generated text response, either a selected multiple-choice option or a descriptive sentence.

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if gold == 'unanswerable':
        correct += 1 if pred == 'unanswerable' else 0
    else:
        correct += 1 if pred == gold else 0
    total += 1
return (correct / total) * 100
```

## Common pitfalls

- Models frequently hallucinate answers for unanswerable questions instead of correctly rejecting them, which drastically lowers robustness scores.
- Performance degrades significantly when scene images contain multiple individuals or use augmentations, requiring careful stratified evaluation across complexity levels.

## Evidence (verbatim from paper)

> In the figure on the right, we illustrate the evaluation accuracy after training with various amounts of data. Specifically, the horizontal axis indicates the number of data units, and each unit contains 8000 samples. We observe clear performance boost when scaling up the training dataset.

## Citation

```bibtex
@misc{pi2024personalized,
  title={Personalized Visual Instruction Tuning},
  author={Pi et al. (2024)},
  year={2024},
  note={arXiv:2410.07113}
}
```

- arXiv: 2410.07113