Teleoracle Eval

---
name: teleoracle-eval
description: Evaluates domain-specific question answering and retrieval-augmented generation capabilities in telecommunications. Probes a model's ability to accurately answer multiple-choice questions about 3GPP standards and adhere to retrieved context without relying on generalized prior knowledge. Use when the user wants to benchmark on TeleQnA, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.02617
  bibtex_key: alabbasi2024teleoracle
  confidence: high
---

# teleoracle-eval

> TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Network — Alabbasi et al. (2024) (arXiv:2411.02617, 2024)

## What this evaluates

Evaluates domain-specific question answering and retrieval-augmented generation capabilities in telecommunications. Probes a model's ability to accurately answer multiple-choice questions about 3GPP standards and adhere to retrieved context without relying on generalized prior knowledge.

## Datasets

- **TeleQnA** — total 12000; splits: train (10000), test (2000)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Calculated as the percentage of correctly answered questions out of the total test set. (Correct Predictions / Total Questions) * 100.
- `Faithfulness` — range: percent
  - Measures the extent to which the model’s response is informed by the retrieved context. Higher scores indicate stronger alignment between the context and the generated output.

## Input / output format

**Input**: A multiple-choice question (with up to 5 options) accompanied by retrieved context chunks (top 15 from hybrid search + reranker, extended via SelfExtend to fill the context window).

**Output**: A single selected answer option (1–5) corresponding to the correct choice.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

def compute_faithfulness(predictions, retrieved_contexts):
    # Measures alignment between response and retrieved context [24]
    # Returns percentage score based on external faithfulness evaluator
    pass
```

## Common pitfalls

- Larger general-purpose models often misinterpret specialized telecom terminology by relying on generalized prior knowledge instead of the retrieved context.
- Models may fail to adhere to strict multiple-choice output formats, requiring careful prompting or fine-tuning to extract the correct option.
- Semantic chunking creates variable-length chunks, necessitating a secondary retrieval step to fill the context window and avoid information loss.

## Evidence (verbatim from paper)

> We use the Faithfulness metric to quantify this effect[[24]]. This metric measures the extent to which the model’s response is informed by the retrieved context. Higher faithfulness indicates stronger alignment between the context and the generated output.

## Citation

```bibtex
@misc{alabbasi2024teleoracle,
  title={TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Network},
  author={Alabbasi et al. (2024)},
  year={2024},
  note={arXiv:2411.02617}
}
```

- arXiv: 2411.02617