Negativeprompt Eval

---
name: negativeprompt-eval
description: Evaluates LLM instruction-following and reasoning capabilities under zero-shot and few-shot settings by appending psychologically grounded negative emotional stimuli to prompts. It probes task accuracy, complex reasoning on beyond-capability tasks, and the truthfulness and informativeness of generated responses. Use when the user wants to benchmark on Instruction Induction, BIG-Bench (curated subset), TruthfulQA, or asks about evaluating this task. Reports accuracy, normalized preferred metric.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.02814
  bibtex_key: wang2024negativeprompt
  confidence: high
---

# negativeprompt-eval

> NegativePrompt: Leveraging Psychology for Large Language Models Enhancement via Negative Emotional Stimuli — Xu Wang et al. (2024) (arXiv:2405.02814, 2024)

## What this evaluates

Evaluates LLM instruction-following and reasoning capabilities under zero-shot and few-shot settings by appending psychologically grounded negative emotional stimuli to prompts. It probes task accuracy, complex reasoning on beyond-capability tasks, and the truthfulness and informativeness of generated responses.

## Datasets

- **Instruction Induction** — total 24; splits: test (24)
- **BIG-Bench (curated subset)** — total 21; splits: test (21)
- **TruthfulQA** — total 817; splits: test (817); HF `truthfulqa/truthful_qa`

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard exact-match accuracy averaged across the 24 Instruction Induction tasks.
- `normalized preferred metric` **(primary)** — range: percent
  - Normalized score where 100 equals human expert performance and 0 equals random guessing. Computed as (model_acc - random_acc) / (human_acc - random_acc) * 100.
- `truthfulness` — range: percent
  - Percentage of responses classified as 'true' by the GPT-judge binary classifier.
- `informativeness` — range: percent
  - Percentage of responses classified as 'informative' by the GPT-info binary classifier.

## Input / output format

**Input**: Original task prompt with negative emotional stimuli appended. For few-shot settings, five randomly selected input-output demonstrations are added after the prompt.

**Output**: Model-generated text response.

## Scoring recipe

```python
def compute_metrics(predictions, golds, random_acc, human_acc):
    # Accuracy (Instruction Induction)
    acc = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
    # Normalized BIG-Bench metric
    model_acc = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
    bb_norm = ((model_acc - random_acc) / (human_acc - random_acc)) * 100
    # Truthfulness & Informativeness (TruthfulQA)
    true_count = sum(1 for p in predictions if GPT_judge(p) == 'true')
    info_count = sum(1 for p in predictions if GPT_info(p) == 'informative')
    truth_pct = true_count / len(predictions)
    info_pct = info_count / len(predictions)
    return acc, bb_norm, truth_pct, info_pct
```

## Common pitfalls

- Applying few-shot demonstrations to BIG-Bench tasks (the paper explicitly uses zero-shot only for BIG-Bench).
- Confusing the '+Ours(avg)' and '+Ours(max)' aggregation strategies: avg averages performance across 10 stimuli first, then across tasks; max selects the best stimulus per task before averaging across tasks.
- Using default ChatGPT settings instead of the specified gpt-3.5-turbo with temperature 0.7.

## Evidence (verbatim from paper)

> For the Instruction Induction tasks, accuracy is the primary evaluation metric. In contrast, for the BIG-Bench tasks, we employ the normalized preferred metric as defined in Srivastava et al. ([2022]). According to this metric, a score of 100 is equated to the performance level of human experts, while a score of 0 aligns with random guessing. It’s critical to note that if an model’s performance on multiple-choice tasks falls below the threshold of random guessing, it may receive a score lower than 0.

## Citation

```bibtex
@misc{wang2024negativeprompt,
  title={NegativePrompt: Leveraging Psychology for Large Language Models Enhancement via Negative Emotional Stimuli},
  author={Xu Wang et al. (2024)},
  year={2024},
  note={arXiv:2405.02814}
}
```

- arXiv: 2405.02814