Netops Eval

---
name: netops-eval
description: Evaluates pre-trained LLMs' networking operations (NetOps) knowledge and reasoning across five technical sub-domains and two languages. It probes both multiple-choice comprehension and open-ended generation capabilities in a domain-specific context. Use when the user wants to benchmark on NetEval, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.05557
  bibtex_key: miao2023netops
  confidence: high
---

# netops-eval

> An Empirical Study of NetOps Capability of Pre-Trained Large Language Models — Miao et al. (2023) (arXiv:2309.05557, 2023)

## What this evaluates

Evaluates pre-trained LLMs' networking operations (NetOps) knowledge and reasoning across five technical sub-domains and two languages. It probes both multiple-choice comprehension and open-ended generation capabilities in a domain-specific context.

## Datasets

- **NetEval** — total 5732; splits: development (5), validation (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The proportion of correctly answered questions out of the total evaluated. For multiple-choice questions, it is 1 if the predicted option matches the gold option, else 0. For open-ended questions, it relies on rule-based extraction and exact-match or string similarity against the gold answer.

## Input / output format

**Input**: Question text. For multiple-choice items, the four options (A-D) are provided. For open-ended items, only the question or fill-in-the-blank prompt is provided.

**Output**: For multiple-choice: a single letter (A, B, C, or D). For open-ended: a generated text string representing the answer or command.

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, golds):
    if gold_type == 'mcq':
        if pred.strip().upper() == gold.strip().upper():
            correct += 1
    else:
        if extract_and_normalize(pred) == extract_and_normalize(gold):
            correct += 1
    total += 1
return correct / total if total > 0 else 0.0
```

## Common pitfalls

- LLMs may guess on multiple-choice questions without possessing actual NetOps knowledge.
- Open-ended generation requires longer context/output, significantly increasing inference cost and latency.
- The full test set is not publicly released; users must submit requests to access evaluation results.

## Evidence (verbatim from paper)

> The collected evaluation set includes 5,269 multiple-choice questions. ... achieving a human-level accuracy in NetOps certification exam questions is still one of the fundamental capabilities we expect LLMs to have.

## Citation

```bibtex
@misc{miao2023netops,
  title={An Empirical Study of NetOps Capability of Pre-Trained Large Language Models},
  author={Miao et al. (2023)},
  year={2023},
  note={arXiv:2309.05557}
}
```

- arXiv: 2309.05557