His Bench Eval

---
name: his-bench-eval
description: Evaluates a model's ability to perform 3D human-in-scene multimodal understanding through open-ended question answering. It probes capabilities in activity recognition, spatial relationship reasoning, and human-object interaction analysis within dynamic 3D environments. Use when the user wants to benchmark on HIS-Bench, or asks about evaluating this task. Reports HIS-Bench score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.12955
  bibtex_key: zhao2025hisgpt
  confidence: high
---

# his-bench-eval

> HIS-GPT: Towards 3D Human-In-Scene Multimodal Understanding — Jiahe Zhao et al. (2025) (arXiv:2503.12955, 2025)

## What this evaluates

Evaluates a model's ability to perform 3D human-in-scene multimodal understanding through open-ended question answering. It probes capabilities in activity recognition, spatial relationship reasoning, and human-object interaction analysis within dynamic 3D environments.

## Datasets

- **HIS-Bench** — total ?; splits: test (-1); repo https://github.com/ZJHTerry18/HumanInScene

## Metrics

- `HIS-Bench score` **(primary)** — range: percent
  - Each open-ended answer is evaluated by GPT-4 on a scale of 0 to 2. Scores are summed across 50 questions per task, yielding a maximum of 100 per task. Sub-metrics track Activity (Act.), Spatial Relationship (Spa.), and Human-Object Interaction (HoI.).

## Input / output format

**Input**: 3D scene mesh and sequential human motion data (provided as point clouds or rendered video frames), paired with open-ended natural language questions about human-scene interactions.

**Output**: Open-ended natural language text answers.

## Scoring recipe

```python
def score_his_bench(predictions, gold_questions):
    task_scores = []
    current_task_scores = []
    for pred, question in zip(predictions, gold_questions):
        score = call_gpt4_evaluator(question, pred)  # Returns 0, 1, or 2
        current_task_scores.append(score)
        if len(current_task_scores) == 50:
            task_scores.append(sum(current_task_scores))
            current_task_scores = []
    return task_scores
```

## Common pitfalls

- Open-ended answers require LLM-as-a-judge evaluation, which can introduce bias or inconsistency compared to exact-match metrics.
- Baselines often render 3D data to 2D videos, losing critical depth and occlusion information that the proposed model handles natively.
- GPT-4 scoring thresholds (0-2) depend heavily on the specific prompt template used, making cross-paper comparisons difficult without identical prompts.

## Evidence (verbatim from paper)

> Considering that HIS-Bench consists of open-ended questions, we use GPT-4 as an automatic evaluator to assess answer correctness. Following[[15]], we prompt GPT-4 to assign a score between 0 and 2 for each answer. Since each task in HIS-Bench consists of 50 questions, the full score for each task is 100.

## Citation

```bibtex
@misc{zhao2025hisgpt,
  title={HIS-GPT: Towards 3D Human-In-Scene Multimodal Understanding},
  author={Jiahe Zhao et al. (2025)},
  year={2025},
  note={arXiv:2503.12955}
}
```

- arXiv: 2503.12955