Hsri Social Reasoning Eval

---
name: hsri-social-reasoning-eval
description: Evaluates foundational models' ability to detect social errors and competencies, identify specific social attributes, reason about sequential interaction flow (pre/post conditions), and generate rationales and corrective actions in human-robot interaction scenarios. Use when the user wants to benchmark on HSRI, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.13898
  bibtex_key: lee2025hsri
  confidence: high
---

# hsri-social-reasoning-eval

> The Human Robot Social Interaction (HSRI) Dataset: Benchmarking Foundational Models' Social Reasoning — Lee et al. (2025) (arXiv:2504.13898, 2025)

## What this evaluates

Evaluates foundational models' ability to detect social errors and competencies, identify specific social attributes, reason about sequential interaction flow (pre/post conditions), and generate rationales and corrective actions in human-robot interaction scenarios.

## Datasets

- **HSRI** — total 440; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The proportion of instances where the model's extracted answer exactly matches or contains the ground-truth answer option. Used across all eight tasks.
- `macro-F1` — range: [0, 1]
  - The unweighted average of F1 scores computed per class. Used for multi-class and multi-label classification tasks to account for class imbalance.
- `Partial Match (PM)` — range: [0, 1]
  - The proportion of instances where the model correctly predicts at least one of the true multi-label attributes, used specifically for the social attribute identification task.

## Input / output format

**Input**: Contextual information consisting of a transcript and either 15 video frames (for image-based models) or raw video (for video-based models), combined with a task-specific question.

**Output**: A model-generated token sequence O, which is post-processed using an LLM with Pydantic to coerce the output into the expected structured answer options.

## Scoring recipe

```python
def compute_score(predictions, gold):
    correct = 0
    for pred, g in zip(predictions, gold):
        # Post-process pred to extract answer option if needed
        # Check if gold answer is a contiguous subsequence of pred
        if g in pred:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- The correctness function checks for contiguous subsequence containment (A ⊆ O) rather than exact string matching, which can yield false positives if the model outputs conversational filler containing the answer.
- Post-processing requires an external LLM with Pydantic to extract answers, introducing latency and potential extraction failures that are not captured by the raw model output.
- Partial Match (PM) only requires one correct label in multi-label tasks, which can inflate performance metrics when the attribute space is large or imbalanced.

## Evidence (verbatim from paper)

> For the task of social attribute identification, we evaluate with accuracy and macro-F1 (F1) scores. As there can be more than a single attribute label associated with a sample, we further report Partial Match (PM) to evaluate the proportion of instances where the model correctly predicts at least one of the true labels.

## Citation

```bibtex
@misc{lee2025hsri,
  title={The Human Robot Social Interaction (HSRI) Dataset: Benchmarking Foundational Models' Social Reasoning},
  author={Lee et al. (2025)},
  year={2025},
  note={arXiv:2504.13898}
}
```

- arXiv: 2504.13898