Openapi Completion Eval

---
name: openapi-completion-eval
description: Evaluates an LLM's ability to perform code infilling for OpenAPI specifications by predicting masked sections of API definitions. It probes semantic understanding of API structure, syntax correctness, and the model's robustness to varying context sizes and prompt formats. Use when the user wants to benchmark on masked OpenAPI definitions, or asks about evaluating this task. Reports correctness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.15729
  bibtex_key: petryshyn2024optimizing
  confidence: high
---

# openapi-completion-eval

> Optimizing Large Language Models for OpenAPI Code Completion — Petryshyn et al. (2024) (arXiv:2405.15729, 2024)

## What this evaluates

Evaluates an LLM's ability to perform code infilling for OpenAPI specifications by predicting masked sections of API definitions. It probes semantic understanding of API structure, syntax correctness, and the model's robustness to varying context sizes and prompt formats.

## Datasets

- **masked OpenAPI definitions** — total ?; splits: (unstated)

## Metrics

- `correctness` **(primary)** — range: percent
  - Percentage of OpenAPI definitions that are correctly completed to match the expected ground truth.
- `validity` — range: percent
  - Percentage of generated completions that parse as syntactically valid OpenAPI specifications.

## Input / output format

**Input**: A partial OpenAPI definition with a `<<MASK>>` marker indicating the location of the missing code, provided with a prefix and suffix context.

**Output**: The completed OpenAPI definition text filling the masked region.

## Scoring recipe

```python
correct_count = 0
valid_count = 0
for pred, gold in zip(predictions, golds):
    if pred == gold:
        correct_count += 1
    if is_valid_openapi(pred):
        valid_count += 1
correctness = (correct_count / len(predictions)) * 100
validity = (valid_count / len(predictions)) * 100
```

## Common pitfalls

- Manual evaluation of GitHub Copilot introduces human latency and error, making automated comparison difficult and inflating generation time measurements.
- Fine-tuned models exhibit strong context-size bias, significantly underperforming when inference context is smaller than the training context.
- SPM infilling format often requires token healing at inference time, which can artificially lower validity scores if not handled.

## Evidence (verbatim from paper)

> The correctness rate, which is the primary target metric in this research, demonstrates a much smoother trend with a significant increase up to the context size of 4096 tokens. Following the completion procedure described, GitHub Copilot correctly completed 29% of the OpenAPI definitions. 68% of the completed definitions were valid.

## Citation

```bibtex
@misc{petryshyn2024optimizing,
  title={Optimizing Large Language Models for OpenAPI Code Completion},
  author={Petryshyn et al. (2024)},
  year={2024},
  note={arXiv:2405.15729}
}
```

- arXiv: 2405.15729