Mmoral Bench Eval

---
name: mmoral-bench-eval
description: Evaluates large vision-language models' ability to interpret panoramic dental X-rays. It probes fine-grained anatomical recognition, pathology detection, and clinical report generation across multiple question types. Use when the user wants to benchmark on MMOral-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.09254
  bibtex_key: [REDACTED]
  confidence: high
---

# mmoral-bench-eval

> Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis — Jing Hao et al. (2025) (arXiv:2509.09254, 2025)

## What this evaluates

Evaluates large vision-language models' ability to interpret panoramic dental X-rays. It probes fine-grained anatomical recognition, pathology detection, and clinical report generation across multiple question types.

## Datasets

- **MMOral-Bench** — total 20563; splits: test (-1); repo https://github.com/isbrycee/OralGPT

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered instances out of the total number of instances in a given category (e.g., Teeth, Patho, Overall).

## Input / output format

**Input**: Panoramic X-ray image paired with a text question or instruction.

**Output**: Text response. Closed-ended questions expect short/selected answers; open-ended questions expect free-form text or clinical reports.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize(p) == normalize(g))
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Proprietary models frequently refuse to answer due to safety/ethical filters rather than inability.
- Open-ended VQA tasks consistently yield much lower scores than closed-ended tasks across nearly all models.
- Models show strong bias toward coarse anatomical structures (e.g., Jaw) while struggling with fine-grained details (e.g., Teeth, Pathology).

## Evidence (verbatim from paper)

> Current universal medical LVLMs achieve less than 40% average accuracy on MMOral-Bench, with HealthGPT-XL32 attaining peak performance at 39.59%.

## Citation

```bibtex
@misc{hao2025towardsbetterdentalai,
  title={Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis},
  author={Jing Hao et al. (2025)},
  year={2025},
  note={arXiv:2509.09254}
}
```

- arXiv: 2509.09254