Scene Graph Modification Eval

---
name: scene-graph-modification-eval
description: Evaluates a model's ability to modify a source scene graph into a target scene graph conditioned on a natural language query. It probes incremental structure expansion, joint node-edge prediction, and the preservation of unmodified graph components during editing. Use when the user wants to benchmark on User Generated, MSCOCO, GCC, RSICD, or asks about evaluating this task. Reports Graph-level accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2209.09093
  bibtex_key: hu2022scenegraphmodification
  confidence: high
---

# scene-graph-modification-eval

> Scene Graph Modification as Incremental Structure Expanding — Hu et al. (2022) (arXiv:2209.09093, 2022)

## What this evaluates

Evaluates a model's ability to modify a source scene graph into a target scene graph conditioned on a natural language query. It probes incremental structure expansion, joint node-edge prediction, and the preservation of unmodified graph components during editing.

## Datasets

- **User Generated** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/THU-BPM/SGM
- **MSCOCO** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/THU-BPM/SGM
- **GCC** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/THU-BPM/SGM
- **RSICD** — total 10000; splits: train (8000), dev (1000), test (1000); repo https://github.com/THU-BPM/SGM

## Metrics

- `Node F1` — range: percent
  - Standard F1 score computed over predicted versus ground-truth node types/labels across the entire graph. Precision and recall are calculated based on exact node type matching.
- `Edge F1` — range: percent
  - Standard F1 score computed over predicted versus ground-truth edge relations. Precision and recall are calculated based on exact relation type matching between node pairs.
- `Graph-level accuracy` **(primary)** — range: percent
  - Exact string match between the entire predicted scene graph and the target scene graph. The prediction is counted as correct only if every node and edge exactly matches the ground truth.

## Input / output format

**Input**: A natural language query and a source scene graph.

**Output**: A target scene graph consisting of predicted nodes and edges.

## Scoring recipe

```python
def compute_metrics(pred_graph, gold_graph):
    pred_nodes = set(pred_graph.nodes)
    gold_nodes = set(gold_graph.nodes)
    node_f1 = f1_score(gold_nodes, pred_nodes)
    pred_edges = set(pred_graph.edges)
    gold_edges = set(gold_graph.edges)
    edge_f1 = f1_score(gold_edges, pred_edges)
    graph_acc = 1.0 if pred_graph == gold_graph else 0.0
    return node_f1, edge_f1, graph_acc
```

## Common pitfalls

- Graph-level accuracy requires a strict exact structural match, not just semantic equivalence or partial overlap.
- Edge prediction is significantly harder than node prediction, especially on datasets with high data sparsity or large graph sizes.
- Evaluating on full graph regeneration baselines versus incremental modification baselines yields different results; the protocol assumes modification-aware evaluation.

## Evidence (verbatim from paper)

> Following Weber et al. ([2021](#bib.bib52 "")), we use three automatic metrics for the evaluation, including node-level and edge-level F1 score, and graph-level accuracy. Graph-level accuracy is computed based on exact string match, which requires the generated scene graph to be identical to the target scene graph for a correct prediction.

## Citation

```bibtex
@misc{hu2022scenegraphmodification,
  title={Scene Graph Modification as Incremental Structure Expanding},
  author={Hu et al. (2022)},
  year={2022},
  note={arXiv:2209.09093}
}
```

- arXiv: 2209.09093