Siqa Eval

---
name: siqa-eval
description: Evaluates multimodal large language models on their ability to understand and assess the quality of scientific images. It probes two distinct capabilities: factual reasoning about scientific content (SIQA-U) and alignment with human expert judgments on perceptual and knowledge dimensions (SIQA-S). Use when the user wants to benchmark on SIQA, or asks about evaluating this task. Reports accuracy, SRCC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.06700
  bibtex_key: li2026siqa
  confidence: high
---

# siqa-eval

> SIQA: Toward Reliable Scientific Image Quality Assessment — Li et al. (2026) (arXiv:2603.06700, 2026)

## What this evaluates

Evaluates multimodal large language models on their ability to understand and assess the quality of scientific images. It probes two distinct capabilities: factual reasoning about scientific content (SIQA-U) and alignment with human expert judgments on perceptual and knowledge dimensions (SIQA-S).

## Datasets

- **SIQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted multiple-choice options (A–D) across four scientific quality dimensions and three question types.
- `SRCC` **(primary)** — range: [-1, 1]
  - Spearman Rank Correlation Coefficient measuring the monotonic relationship between predicted scores and human ratings.
- `PLCC` — range: [-1, 1]
  - Pearson Linear Correlation Coefficient measuring the linear relationship between predicted scores and human ratings.

## Input / output format

**Input**: For SIQA-U: an image paired with a multiple-choice question (options A–D). For SIQA-S: an image paired with a prompt requesting a quality rating across five levels (Bad, Poor, Fair, Good, Excellent).

**Output**: For SIQA-U: a single character (A, B, C, or D), with extra text ignored if the answer is clearly identifiable. For SIQA-S: logits for the five quality tokens, used to compute a continuous predicted score.

## Scoring recipe

```python
# SIQA-U
acc = sum(1 for pred, gold in zip(predictions, golds) if pred.strip()[0] == gold) / len(golds)

# SIQA-S
scores_pred = []
for logits in predictions:
    probs = softmax(logits)
    levels = [1, 2, 3, 4, 5]
    scores_pred.append(sum(p * l for p, l in zip(probs, levels)))
srcc = spearmanr(scores_pred, golds).correlation
plcc = pearsonr(scores_pred, golds).correlation
```

## Common pitfalls

- Models may output verbose text; the protocol strictly requires extracting a clearly identifiable single character (A–D), otherwise marking it incorrect.
- SIQA-S evaluation is restricted to models with publicly available weights or API logprob support, as it requires access to token-level logits for the five rating levels.
- Fine-tuning on SIQA-S can artificially inflate correlation scores through label fitting, masking deficits in actual scientific reasoning measured by SIQA-U.

## Evidence (verbatim from paper)

> These predicted scores are then used to compute the SRCC and Pearson Linear Correlation Coefficient (PLCC) for both the perception and knowledge dimensions.

## Citation

```bibtex
@misc{li2026siqa,
  title={SIQA: Toward Reliable Scientific Image Quality Assessment},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2603.06700}
}
```

- arXiv: 2603.06700