Factual Scene Graph Parsing Eval

---
name: factual-scene-graph-parsing-eval
description: This benchmark evaluates a model's ability to parse natural language captions into structured scene graphs that faithfully represent described visual elements. It probes compositional generalization and output consistency by testing parsers on both standard and length-constrained splits, measuring how well generated graph structures align with human-annotated ground truth. Use when the user wants to benchmark on FACTUAL, or asks about evaluating this task. Reports SPICE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.17497
  bibtex_key: li2023factual
  confidence: high
---

# factual-scene-graph-parsing-eval

> FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing — Li et al. (2023) (arXiv:2305.17497, 2023)

## What this evaluates

This benchmark evaluates a model's ability to parse natural language captions into structured scene graphs that faithfully represent described visual elements. It probes compositional generalization and output consistency by testing parsers on both standard and length-constrained splits, measuring how well generated graph structures align with human-annotated ground truth.

## Datasets

- **FACTUAL** — total 40369; splits: train (37861), val (1000), test (1508); repo https://github.com/zhuang-li/FACTUAL

## Metrics

- `SPICE` **(primary)** — range: [0, 1]
  - Computes the F-score based on the overlap of sub-components (objects, {object, attribute} tuples, and {object, predicate, object} triples) between the candidate and ground truth scene graphs.
- `Exact Set Match` — range: [0, 1]
  - Calculates the percentage of instances where the set of parsed fact strings exactly matches the ground truth fact strings, ignoring the order of facts.
- `SoftSPICE` — range: [0, 1]
  - Embedding-based graph similarity that encodes graph sub-components using Sentence-BERT and computes the average cosine similarity between candidate and reference component embeddings.

## Input / output format

**Input**: Natural language captions describing image regions.

**Output**: Structured scene graphs represented as sets of triples (object, predicate, object) and tuples (object, attribute), formatted according to the target annotation scheme (e.g., FACTUAL-SG, CDP-SG, or VG-SG).

## Scoring recipe

```python
def exact_set_match(pred_graph, gold_graph):
    pred_facts = set(str(f) for f in pred_graph)
    gold_facts = set(str(f) for f in gold_graph)
    if not gold_facts: return 0.0
    return len(pred_facts & gold_facts) / len(gold_facts)

def spice_score(pred_graph, gold_graph):
    pred_comps = extract_components(pred_graph)
    gold_comps = extract_components(gold_graph)
    matches = len(pred_comps & gold_comps)
    p = matches / max(len(pred_comps), 1)
    r = matches / max(len(gold_comps), 1)
    return 2 * p * r / (p + r) if (p + r) > 0 else 0.0
```

## Common pitfalls

- SPICE-Parser uses a dependency parser trained on general domain text rather than the specific dataset's annotation scheme, causing systematic mismatches with ground truth graphs.
- Exact Set Match ignores fact ordering, potentially masking structural parsing errors where correct facts are arranged in an invalid graph topology.
- Human evaluation of faithfulness and consistency relies on a small panel of three students, introducing subjectivity and limited statistical power.

## Evidence (verbatim from paper)

> Following Schuster et al. (2015); Wang et al. (2018); Choi et al. (2022), we evaluate scene graph parsers utilizing the SPICE metric (Anderson et al. (2016)). The SPICE F-score measures the similarity between the candidate and ground truth graph representations extracted from captions by the parsers. In addition, we also employ the Exact Set Match metric (Yu et al. (2019)), which assesses the accuracy of the parsers by determining whether the strings of the parsed facts match the ground truth facts while disregarding the order of the facts.

## Citation

```bibtex
@misc{li2023factual,
  title={FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2305.17497}
}
```

- arXiv: 2305.17497