Rjua Qa Eval

---
name: rjua-qa-eval
description: This benchmark evaluates large language models' ability to perform medical logical reasoning and urological disease diagnosis. It probes the model's capacity to handle complex, real-world clinical scenarios involving subjective patient queries and multi-disease comorbidity reasoning. Use when the user wants to benchmark on RJUA-QA, or asks about evaluating this task. Reports F1 score (diagnosis & advice).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.09785
  bibtex_key: lyu2023rjuaqa
  confidence: high
---

# rjua-qa-eval

> RJUA-QA: A Comprehensive QA Dataset for Urology — Lyu et al. (2023) (arXiv:2312.09785, 2023)

## What this evaluates

This benchmark evaluates large language models' ability to perform medical logical reasoning and urological disease diagnosis. It probes the model's capacity to handle complex, real-world clinical scenarios involving subjective patient queries and multi-disease comorbidity reasoning.

## Datasets

- **RJUA-QA** — total 2132; splits: test (2132); repo https://github.com/alipay/RJU_Ant_QA

## Metrics

- `F1 score (diagnosis & advice)` **(primary)** — range: [0, 1]
  - F1 = 2 × (P × R) / (P + R), where P = TP / (TP + FP) and R = TP / (TP + FN). The final accuracy is a weighted sum of the F1 scores for diagnosis and advice, using weights of 2/3 and 1/3 respectively.
- `Rouge-L` — range: [0, 1]
  - Computes precision, recall, and F1 based on the longest common subsequence (LCS) between the generated response and the reference text.

## Input / output format

**Input**: Patient queries describing subjective symptoms and clinical scenarios.

**Output**: Diagnostic responses containing predicted diseases and examination advice.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    f1_diag = f1_score(predictions['disease'], gold['disease'])
    f1_adv = f1_score(predictions['advice'], gold['advice'])
    final_acc = (2/3) * f1_diag + (1/3) * f1_adv
    rouge_l = rouge_l_score(predictions['response'], gold['response'])
    return {'final_accuracy': final_acc, 'rouge_l': rouge_l}
```

## Common pitfalls

- F1 is computed separately for diagnosis and advice, then combined with a fixed 2:1 weight ratio rather than a single unified F1.
- Rouge-L relies on longest common subsequence matching, which may penalize semantically correct but lexically varied medical advice.
- The protocol does not specify how multi-label comorbidity predictions are aggregated for F1 calculation (e.g., micro vs. macro averaging).

## Evidence (verbatim from paper)

> The F1 score is utilized to measure the accuracy for LLMs’ diagnosis and treatment. According to Table 1, F1 score is is formulated as: F1=2×(P×R)/(P+R), where P=TP/(TP+FP) denotes the precision and R=TP/(TP+FN) denotes the recall. A weighted sum of F1 score for diagnosis and advice is adopted to obtain the final accuracy, i.e., 2/3 for diagnosis and 1/3 for advice. To evaluate the overall quality of the LLMs’ responses, Rouge-L is exploited to calculate the longest common sub-sequence (LCS) between the generation and reference.

## Citation

```bibtex
@misc{lyu2023rjuaqa,
  title={RJUA-QA: A Comprehensive QA Dataset for Urology},
  author={Lyu et al. (2023)},
  year={2023},
  note={arXiv:2312.09785}
}
```

- arXiv: 2312.09785