Ssg Generation Eval

---
name: ssg-generation-eval
description: Evaluates a model's ability to generate structured, human-centric scene graphs from images by jointly predicting verb predicates and fine-grained semantic role-value pairs for persons and objects. It probes multi-concurrent action understanding, affordance reasoning, and structured visual representation learning. Use when the user wants to benchmark on SSG dataset, Action Genome dataset, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.22829
  bibtex_key: sugandhika2024situational
  confidence: high
---

# ssg-generation-eval

> Situational Scene Graph for Structured Human-centric Situation Understanding — Chinthani Sugandhika et al. (2024) (arXiv:2410.22829, 2024)

## What this evaluates

Evaluates a model's ability to generate structured, human-centric scene graphs from images by jointly predicting verb predicates and fine-grained semantic role-value pairs for persons and objects. It probes multi-concurrent action understanding, affordance reasoning, and structured visual representation learning.

## Datasets

- **SSG dataset** — total 25500; splits: test (-1); repo https://github.com/LUNAProject22/SSG
- **Action Genome dataset** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted verb predicates under the PREDcls (with constraint) setting. For VQA subtasks, it is the percentage of correctly selected multiple-choice options.
- `value` — range: percent
  - Proportion of semantic roles where the model correctly predicts at least one role-value pair for that role.
- `value-all` — range: percent
  - Proportion of semantic roles where the model correctly predicts all associated role-value pairs for that role.
- `value-two` — range: percent
  - Proportion of semantic roles where the model correctly predicts at least two role-value pairs for that role.
- `role-based accuracy` — range: percent
  - Average of per-role exact-match accuracy scores across all semantic roles.

## Input / output format

**Input**: RGB image (video frame). For VQA/reasoning subtasks, input includes the image plus a multiple-choice question (and optionally a text-formatted SSG graph).

**Output**: Structured prediction consisting of verb predicates and semantic role-value pairs for persons and objects. For VQA, a single selected option from multiple-choice options.

## Scoring recipe

```python
def score_ssg(preds, gold):
    # preds, gold: dicts mapping entity/role to predicted/gold values
    # 1. Verb Predicate Accuracy (PREDcls)
    verb_acc = sum(1 for p, g in zip(preds['verbs'], gold['verbs']) if p == g) / len(gold['verbs'])
    
    # 2. Value metrics per role
    role_values = {}
    for role in gold['roles']:
        g_vals = set(gold['roles'][role])
        p_vals = set(preds['roles'].get(role, []))
        correct_count = len(g_vals & p_vals)
        role_values[role] = correct_count
        
    # value: >=1 correct, value-two: >=2 correct, value-all: == all
    n_roles = len(role_values)
    val = sum(1 for c in role_values.values() if c >= 1) / n_roles
    val_two = sum(1 for c in role_values.values() if c >= 2) / n_roles
    val_all = sum(1 for c, total in zip(role_values.values(), gold['roles'].values()) if c == len(total)) / n_roles
    
    # role-based accuracy: avg of per-role exact match accuracy
    role_acc = sum(1 for c, total in zip(role_values.values(), gold['roles'].values()) if c == len(total)) / n_roles
    
    return {'verb_acc': verb_acc, 'value': val, 'value-two': val_two, 'value-all': val_all, 'role_based_acc': role_acc}
```

## Common pitfalls

- The dataset contains only a single annotation per record, unlike prior works with multiple annotations, requiring adapted thresholding for value metrics.
- SRV classification uses a top-1 verb setting: if the predicted verb is incorrect, all associated semantic role-values are automatically marked incorrect.
- Object and person SRV classifications assume ground-truth bounding boxes are known (no object detection stage is performed).

## Evidence (verbatim from paper)

> For the verb predicate classification task, we use the metric accuracy (Acc) *[[77]]* under ”with constraint” predicate classification (PREDcls)*[[27]]* setting. Further, since semantic role-value classification is similar to the semantic role labelling in conventional verb-based situation frames, we adapt existing metrics value and value-all *[[46]]* along with a new metric called value-two. However, unlike in *[[77], [46]]*, we have only single annotation per record. Therefore, we assess the value by determining whether the model accurately predicts at least one role-value for a given role out of all the roles associated with that particular semantic entity. The value-all metric evaluates if the model can accurately predict all the semantic role-values, while the value-two metric checks if at least two semantic role-values are correctly predicted out of all the roles.

## Citation

```bibtex
@misc{sugandhika2024situational,
  title={Situational Scene Graph for Structured Human-centric Situation Understanding},
  author={Chinthani Sugandhika et al. (2024)},
  year={2024},
  note={arXiv:2410.22829}
}
```

- arXiv: 2410.22829