Bihopr Eval

---
name: bihopr-eval
description: This benchmark evaluates large language models on multi-hop, multi-answer reasoning tasks within the biomedical domain. It probes the model's ability to perform step-by-step inference over biomedical knowledge graphs and generate multiple valid answers for one-to-many-to-many relationships. Use when the user wants to benchmark on BioHopR, or asks about evaluating this task. Reports Embedding-Based Precision.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.22240
  bibtex_key: kim2025bihopr
  confidence: high
---

# bihopr-eval

> BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain — Kim et al. (2025) (arXiv:2505.22240, 2025)

## What this evaluates

This benchmark evaluates large language models on multi-hop, multi-answer reasoning tasks within the biomedical domain. It probes the model's ability to perform step-by-step inference over biomedical knowledge graphs and generate multiple valid answers for one-to-many-to-many relationships.

## Datasets

- **BioHopR** — total ?; splits: test (-1)

## Metrics

- `Embedding-Based Precision` **(primary)** — range: percent
  - Precision is computed using BioLORD-2023-C embeddings. For a prediction embedding p and ground truth answer embeddings {a_1,...,a_n}, compute cosine similarity cos(p, a_i). If max_i cos(p, a_i) > τ (where τ=0.9), the prediction is a true positive. Precision = |True Positives| / |Predicted Responses|. Reported separately for 1-hop (Prec_HOP1) and 2-hop (Prec_HOP2) tasks.

## Input / output format

**Input**: A natural language question requiring 1-hop or 2-hop reasoning over biomedical knowledge graphs, expecting multiple valid answers.

**Output**: A single text string containing the predicted answer(s).

## Scoring recipe

```python
embeddings = BioLORD_2023_C()
pred_emb = embeddings.encode(prediction)
gt_embs = [embeddings.encode(a) for a in ground_truth_answers]
sims = [cosine_similarity(pred_emb, g) for g in gt_embs]
max_sim = max(sims)
if max_sim > 0.9:
    is_correct = True
else:
    is_correct = False
precision = sum(is_correct for all predictions) / len(predictions)
```

## Common pitfalls

- The high similarity threshold (τ=0.9) strictly prioritizes precision, meaning semantically close but not highly similar answers are penalized.
- Using the maximum cosine similarity across multiple ground truth answers can mask partial correctness if one answer is very close but others are not.
- Evaluation is strictly zero-shot with temperature=0, which may disadvantage models that benefit from sampling or few-shot prompting.

## Evidence (verbatim from paper)

> The precision (Prec) is computed using the cosine similarity between the predicted response and the ground truth answer list, leveraging BioLORD-2023-C embeddings. Let p denote the embedding of the predicted response and {a_1,a_2,...,a_n} denote the embeddings of the ground truth answers. ... If the maximum cosine similarity across all ground truth answers satisfies: max_i cos(p,a_i) > τ, then the prediction is considered a true positive. The precision (Prec) is then calculated as: Prec = |True Positives| / |Predicted Responses|. We use τ=0.9 for BioLORD-2023-C embeddings after a grid search of threshold values from 0.5 to 0.9, which led an optimal setting with 0.9.

## Citation

```bibtex
@misc{kim2025bihopr,
  title={BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain},
  author={Kim et al. (2025)},
  year={2025},
  note={arXiv:2505.22240}
}
```

- arXiv: 2505.22240