Chatgpt Multitask Eval

---
name: chatgpt-multitask-eval
description: Evaluates ChatGPT's zero-shot multitask capabilities across summarization, machine translation, sentiment analysis, question answering, dialogue, and misinformation detection. It probes the model's generalization, reasoning, multilingual understanding, and task-specific performance without fine-tuning. Use when the user wants to benchmark on CNN/DM, SAMSum, FLoRes-200, NusaX, bAbI, EntailmentBank, CLUTRR, StepGame, Pep-3k, COVID-Social, COVID-Scientific, MultiWOZ2.2, OpenDialKG, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2302.04023
  bibtex_key: bang2023chatgptmultitask
  confidence: high
---

# chatgpt-multitask-eval

> A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity — Bang et al. (2023) (arXiv:2302.04023, 2023)

## What this evaluates

Evaluates ChatGPT's zero-shot multitask capabilities across summarization, machine translation, sentiment analysis, question answering, dialogue, and misinformation detection. It probes the model's generalization, reasoning, multilingual understanding, and task-specific performance without fine-tuning.

## Datasets

- **CNN/DM** — total ?; splits: test (-1)
- **SAMSum** — total ?; splits: test (-1)
- **FLoRes-200** — total ?; splits: test (-1)
- **NusaX** — total ?; splits: test (-1)
- **bAbI** — total ?; splits: test (-1)
- **EntailmentBank** — total ?; splits: test (-1)
- **CLUTRR** — total ?; splits: test (-1)
- **StepGame** — total ?; splits: test (-1)
- **Pep-3k** — total ?; splits: test (-1)
- **COVID-Social** — total ?; splits: test (-1)
- **COVID-Scientific** — total ?; splits: test (-1)
- **MultiWOZ2.2** — total ?; splits: test (-1)
- **OpenDialKG** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels or answers out of total samples.
- `ROUGE-1` — range: [0, 1]
  - Unigram overlap between generated and reference text, typically measured as F1 score.
- `ChrF++` — range: [0, 1]
  - Character n-gram F-score used for machine translation quality assessment.
- `Macro F1` — range: [0, 1]
  - Unweighted mean of F1 scores across all classes.
- `JGA` — range: [0, 1]
  - Joint Goal Accuracy: percentage of dialogue turns where all slot values and intent are predicted correctly.
- `BLEU` — range: [0, 1]
  - Geometric mean of modified n-gram precisions with brevity penalty.
- `Inform Rate` — range: [0, 1]
  - Percentage of turns where the system successfully provides requested information.
- `AUC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve for binary classification.
- `ROUGE-L` — range: [0, 1]
  - Longest Common Subsequence overlap between generated and reference text.
- `FeQA` — range: [0, 1]
  - Factuality QA score measuring factual consistency in generated responses.

## Input / output format

**Input**: Zero-shot text prompts, dialogue turns, or parallel sentences provided as natural language instructions or context.

**Output**: Model-generated text responses, translations, or classifications.

## Scoring recipe

```python
def evaluate(predictions, golds):
    acc = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    rouge1 = rouge_score(golds, predictions, rouge_types=['rouge1'])
    bleu = corpus_bleu(golds, predictions)
    inform = sum(1 for p in predictions if p['informative']) / len(predictions)
    return {'accuracy': acc, 'rouge1': rouge1, 'bleu': bleu, 'inform_rate': inform}
```

## Common pitfalls

- Evaluates on small subsets (30-200 samples) rather than full test sets, risking high variance and unrepresentative results.
- Zero-shot evaluation without task-specific fine-tuning makes direct comparison to fully-fine-tuned SOTA models inherently unfair.
- Automatic metrics like BLEU and ROUGE penalize ChatGPT's longer, more fluent responses compared to concise golden answers.

## Evidence (verbatim from paper)

> DST is mediocre while ChatGPT successfully leverages all information provided while answering the questions with a 71.1% inform rate and 5.65 BLEU score.

## Citation

```bibtex
@misc{bang2023chatgptmultitask,
  title={A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity},
  author={Bang et al. (2023)},
  year={2023},
  note={arXiv:2302.04023}
}
```

- arXiv: 2302.04023