Chatgpt Nlp Eval

---
name: chatgpt-nlp-eval
description: Evaluates ChatGPT's few-shot generation and reasoning capabilities across multiple NLP tasks, including question answering, commonsense reasoning, natural language inference, and sentiment analysis. It probes the model's ability to follow task-specific formalizations, leverage retrieved demonstrations, and mitigate hallucination through self-verification. Use when the user wants to benchmark on SQuADv2, TQA, MRQA-OOD, CSQA, StrategyQA, RTE, CommitmentBank, SST-2, IMDB, Yelp, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.09719
  bibtex_key: sun2023pushing
  confidence: high
---

# chatgpt-nlp-eval

> Pushing the Limits of ChatGPT on NLP Tasks — Sun et al. (2023) (arXiv:2306.09719, 2023)

## What this evaluates

Evaluates ChatGPT's few-shot generation and reasoning capabilities across multiple NLP tasks, including question answering, commonsense reasoning, natural language inference, and sentiment analysis. It probes the model's ability to follow task-specific formalizations, leverage retrieved demonstrations, and mitigate hallucination through self-verification.

## Datasets

- **SQuADv2** — total ?; splits: test (-1)
- **TQA** — total ?; splits: test (-1)
- **MRQA-OOD** — total ?; splits: test (-1)
- **CSQA** — total ?; splits: test (-1)
- **StrategyQA** — total ?; splits: test (-1)
- **RTE** — total ?; splits: test (-1)
- **CommitmentBank** — total ?; splits: test (-1)
- **SST-2** — total ?; splits: test (-1)
- **IMDB** — total ?; splits: test (-1)
- **Yelp** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The proportion of instances where the model's generated output exactly matches the ground truth label or answer substring.

## Input / output format

**Input**: Task-specific prompts: for QA, context split into indexed sentences concatenated with a question; for NLI, premise and hypothesis; for commonsense, question with multiple-choice options; for sentiment, input text. Demonstrations are retrieved and appended.

**Output**: Model generates text following task formalization: e.g., '# (index) answer' for QA; 'yes' or 'no' for each NLI relation; selected option letter/text for others; sentiment keyword for sentiment analysis.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return correct / len(golds)
```

## Common pitfalls

- ChatGPT's output must strictly adhere to the prompt's expected format (e.g., index + substring for QA); parsing failures occur if the model deviates.
- Performance heavily depends on the demonstration retrieval method; using a fine-tuned retriever vs. random/SimCSE causes significant accuracy drops.
- Hallucination in reasoning tasks can produce plausible but incorrect answers, requiring the self-verification step to correct before final scoring.

## Evidence (verbatim from paper)

> We use accuracy in three-class (e.g., entail, contrast, neural) as the evaluation metric.

## Citation

```bibtex
@misc{sun2023pushing,
  title={Pushing the Limits of ChatGPT on NLP Tasks},
  author={Sun et al. (2023)},
  year={2023},
  note={arXiv:2306.09719}
}
```

- arXiv: 2306.09719