Eicap Bench Eval

---
name: eicap-bench-eval
description: Evaluates large language models' emotional intelligence (EI) capabilities across a four-layer taxonomy: emotional tracking, cause inference, appraisal, and emotionally appropriate response generation. It probes fine-grained subcategories including cultural sensitivity, valence judgment, and uncertainty calibration using multi-turn conversational contexts. Use when the user wants to benchmark on EICap-Bench, or asks about evaluating this task. Reports macro-average accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.06196
  bibtex_key: nazar2025eicap
  confidence: high
---

# eicap-bench-eval

> EICAP: Deep Dive in Assessment and Enhancement of Large Language Models in Emotional Intelligence through Multi-Turn Conversations — Nazar et al. (2025) (arXiv:2508.06196, 2025)

## What this evaluates

Evaluates large language models' emotional intelligence (EI) capabilities across a four-layer taxonomy: emotional tracking, cause inference, appraisal, and emotionally appropriate response generation. It probes fine-grained subcategories including cultural sensitivity, valence judgment, and uncertainty calibration using multi-turn conversational contexts.

## Datasets

- **EICap-Bench** — total ?; splits: test (-1)

## Metrics

- `macro-average accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly answered multiple-choice questions across all EI subcategories, averaged across dimensions. Reported as overall accuracy and delta accuracy (ΔAcc in percentage points) relative to a baseline after fine-tuning.

## Input / output format

**Input**: Multi-turn conversational context or dialogue prompt requiring emotional intelligence reasoning, presented as a multiple-choice question.

**Output**: Model selects the correct option from the provided choices (MCQ format).

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)
```

## Common pitfalls

- Delta accuracy (ΔAcc) is reported with statistical significance testing (10,000-sample paired bootstrap with Benjamini-Hochberg FDR correction), so raw point differences without confidence intervals or significance markers should not be interpreted as reliable.
- The benchmark evaluates multi-turn dialogues, but the MCQ format abstracts away generation quality; accuracy only measures option selection, not the quality or appropriateness of generated emotional responses.
- Narrow-slice LoRA fine-tuning on general corpora (UltraChat) often degrades foundation-layer EI performance, indicating that domain-aligned, task-annotated data is required for stable improvements.

## Evidence (verbatim from paper)

> Figure[2] summarizes zero-shot performance of three open-source LLM families Gemma-9B, LLaMA-3-8B, and Qwen-2.5-7B in both Base and Instruct variants on EICap-Bench. The evaluation spans fine-grained subcategories across all EI layers defined in the EICap taxonomy. Results show that Qwen-2.5-7B consistently outperforms the other models in macro-average accuracy across multiple EI dimensions, particularly in emotional inference and uncertainty calibration.

## Citation

```bibtex
@misc{nazar2025eicap,
  title={EICAP: Deep Dive in Assessment and Enhancement of Large Language Models in Emotional Intelligence through Multi-Turn Conversations},
  author={Nazar et al. (2025)},
  year={2025},
  note={arXiv:2508.06196}
}
```

- arXiv: 2508.06196