Teleqna Eval

---
name: teleqna-eval
description: Evaluates large language models' domain-specific knowledge in telecommunications, covering general terminology, research concepts, and complex technical standards. It also benchmarks model performance against human telecom professionals under strict no-search conditions. Use when the user wants to benchmark on TeleQnA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.15051
  bibtex_key: maatouk2023teleqna
  confidence: high
---

# teleqna-eval

> TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge — Maatouk et al. (2023) (arXiv:2310.15051, 2023)

## What this evaluates

Evaluates large language models' domain-specific knowledge in telecommunications, covering general terminology, research concepts, and complex technical standards. It also benchmarks model performance against human telecom professionals under strict no-search conditions.

## Datasets

- **TeleQnA** — total 10000; splits: test (10000); repo https://github.com/netop-team/TeleQnA

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of questions for which the model selected the option marked as correct in the dataset. Calculated as (number of correct predictions / total number of questions) * 100.

## Input / output format

**Input**: Multiple-choice questions with predefined options. In context-augmented experiments, questions are paired with the top-3 most similar 500-word segments from technical standards documents based on embedding similarity.

**Output**: The model must select exactly one option from the provided choices for each question.

## Scoring recipe

```python
correct = 0
for q, options, gold_idx in dataset:
    pred_idx = model.predict(q, options)
    if pred_idx == gold_idx:
        correct += 1
accuracy = (correct / len(dataset)) * 100
```

## Common pitfalls

- Querying questions in large batches (e.g., B=50) reduces accuracy compared to single questions due to increased topic diversity and context dilution.
- Performance heavily depends on providing external context; without top-3 relevant document segments, accuracy on standards questions drops significantly.
- Human baseline requires strict enforcement of a no-search/no-external-reference policy to ensure a fair comparison.

## Evidence (verbatim from paper)

> As a performance measure, we define the accuracy as the percentage of questions for which the entity at hand selected the option marked as correct in the dataset.

## Citation

```bibtex
@misc{maatouk2023teleqna,
  title={TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge},
  author={Maatouk et al. (2023)},
  year={2023},
  note={arXiv:2310.15051}
}
```

- arXiv: 2310.15051