Osbench Eval

---
name: osbench-eval
description: Evaluates subject-driven image generation and manipulation capabilities, specifically testing identity consistency, prompt adherence, and background preservation across single- and multi-subject scenarios. Use when the user wants to benchmark on OSBench, or asks about evaluating this task. Reports Overall (Generation).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.08294
  bibtex_key: liu2025opensubject
  confidence: high
---

# osbench-eval

> OpenSubject: Leveraging Video-Derived Identity and Diversity Priors for Subject-driven Image Generation and Manipulation — Liu et al. (2025) (arXiv:2512.08294, 2025)

## What this evaluates

Evaluates subject-driven image generation and manipulation capabilities, specifically testing identity consistency, prompt adherence, and background preservation across single- and multi-subject scenarios.

## Datasets

- **OSBench** — total 240; splits: test (240); repo https://github.com/LAW1223/OpenSubject

## Metrics

- `Prompt Adherence (PA)` — range: [0, 10]
  - VLM judge (GPT-4.1) assigns a 0–10 score based on attribute, count, and relation compliance with the prompt.
- `Identity Fidelity (IF)` — range: [0, 10]
  - VLM judge assigns a 0–10 score measuring consistency with the subject across provided reference images.
- `Overall (Generation)` **(primary)** — range: [0, 10]
  - Geometric mean of PA and IF: sqrt(PA * IF).
- `Manipulation Fidelity (MF)` — range: [0, 10]
  - VLM judge assigns a 0–10 score measuring match between edited regions and referenced subject(s).
- `Background Consistency (BC)` — range: [0, 10]
  - VLM judge assigns a 0–10 score measuring stability of non-edited regions.
- `Overall (Manipulation)` — range: [0, 10]
  - Geometric mean of MF and BC: sqrt(MF * BC).

## Input / output format

**Input**: Reference image(s) and an open-ended text prompt (generation); reference image(s), target scene image, and edit instruction (manipulation).

**Output**: A single synthesized or edited image.

## Scoring recipe

```python
judge = GPT4.1()
scores = {}
for task in [single_gen, multi_gen, single_manip, multi_manip]:
    for sample in task.samples:
        output = model.generate(sample.ref_images, sample.prompt)
        scores[task] = judge.score(output, sample, rubric=task.rubric, scale=10)
gen_overall = (scores['PA'] * scores['IF']) ** 0.5
manip_overall = (scores['MF'] * scores['BC']) ** 0.5
return gen_overall, manip_overall
```

## Common pitfalls

- VLM judges may conflate aesthetic quality with strict prompt adherence or identity fidelity.
- Geometric mean heavily penalizes models that score high on one dimension but low on another, potentially masking balanced performance.
- Rubricized prompts for GPT-4.1 are not publicly detailed, making exact score reproduction difficult.

## Evidence (verbatim from paper)

> Following instruction-based assessment methods (e.g., VIEScore, OmniContext), we use a strong VLM judge (GPT-4.1) to assign 0–10 scores with rubricized prompts and independent criteria. For generation tasks, we report Prompt Adherence (PA) (attribute/count/relation compliance), Identity Fidelity (IF) (consistency with the subject across provided references), and Overall (geometric mean of PA and IF). For manipulation tasks, we report Manipulation Fidelity (MF) (match between edited regions and the referenced subject(s)), Background Consistency (BC) (stability of non-edited regions), and Overall (geometric mean of MF and BC).

## Citation

```bibtex
@misc{liu2025opensubject,
  title={OpenSubject: Leveraging Video-Derived Identity and Diversity Priors for Subject-driven Image Generation and Manipulation},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2512.08294}
}
```

- arXiv: 2512.08294