Redis Qa Eval

---
name: redis-qa-eval
description: Evaluates large language models' ability to answer questions about rare diseases, including diagnosis, symptoms, causes, and related properties. It probes the models' medical knowledge retrieval and reasoning capabilities in a specialized, low-resource domain. Use when the user wants to benchmark on ReDis-QA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.08422
  bibtex_key: wang2024redisqa
  confidence: high
---

# redis-qa-eval

> Assessing and Enhancing Large Language Models in Rare Disease Question-answering — Wang et al. (2024) (arXiv:2408.08422, 2024)

## What this evaluates

Evaluates large language models' ability to answer questions about rare diseases, including diagnosis, symptoms, causes, and related properties. It probes the models' medical knowledge retrieval and reasoning capabilities in a specialized, low-resource domain.

## Datasets

- **ReDis-QA** — total 1360; splits: test (-1); repo https://github.com/guanchuwang/redis-bench

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total number of instances. Calculated as the number of correct predictions divided by the total number of questions, multiplied by 100.

## Input / output format

**Input**: Natural language question about a rare disease (e.g., symptoms, causes, diagnosis, related disorders), provided via instruction-tuned prompts.

**Output**: Natural language answer to the question.

## Scoring recipe

```python
correct_count = 0
for pred, gold in zip(predictions, gold_answers):
    if normalize(pred) == normalize(gold):
        correct_count += 1
accuracy = (correct_count / len(gold_answers)) * 100
```

## Common pitfalls

- Model performance varies significantly across different disease property categories (e.g., symptoms vs. diagnosis), so reporting only aggregate accuracy may mask domain-specific weaknesses.
- The evaluation relies on specific instruction-tuned prompts detailed in Appendix 1, meaning results are highly sensitive to prompt formatting and system instructions not fully described in the main text.

## Evidence (verbatim from paper)

> The evaluation metric is the accuracy on the ReDis-QA dataset. ... Figure 2 (b) illustrates the accuracy of LLMs in percentage, with accuracy for each subset of properties displayed separately.

## Citation

```bibtex
@misc{wang2024redisqa,
  title={Assessing and Enhancing Large Language Models in Rare Disease Question-answering},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2408.08422}
}
```

- arXiv: 2408.08422