Civl Eval

---
name: civl-eval
description: This benchmark evaluates multimodal complaint analysis by measuring a model's ability to jointly process multi-turn textual dialogues and accompanying images to classify fine-grained aspects and severity levels of customer grievances. It probes cross-modal alignment, multi-label classification, and robustness to class imbalance and subjective tone variations. Use when the user wants to benchmark on CIViL, or asks about evaluating this task. Reports macro F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.14693
  bibtex_key: singh2025valor
  confidence: high
---

# civl-eval

> Talk, Snap, Complain: Validation-Aware Multimodal Expert Framework for Fine-Grained Customer Grievances — Rishu Kumar Singh et al. (2025) (arXiv:2511.14693, 2025)

## What this evaluates

This benchmark evaluates multimodal complaint analysis by measuring a model's ability to jointly process multi-turn textual dialogues and accompanying images to classify fine-grained aspects and severity levels of customer grievances. It probes cross-modal alignment, multi-label classification, and robustness to class imbalance and subjective tone variations.

## Datasets

- **CIViL** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/sarmistha-D/VALOR

## Metrics

- `Accuracy` — range: [0, 1]
  - Proportion of correctly predicted labels out of total instances. Computed independently for Aspect Classification (ACD) and Severity Classification (SD) tasks.
- `macro F1-score` **(primary)** — range: [0, 1]
  - Unweighted mean of the F1-score for each class, treating all classes equally regardless of frequency. Computed independently for ACD and SD tasks.

## Input / output format

**Input**: Multi-turn customer complaint dialogues containing both textual messages and associated images (visual evidence).

**Output**: Multi-label classification predictions for two dimensions: Aspect Classification (ACD) and Severity Classification (SD).

## Scoring recipe

```python
from sklearn.metrics import accuracy_score, f1_score

def compute_metrics(y_true, y_pred):
    # y_true and y_pred are binary multi-label matrices (n_samples, n_classes)
    acc = accuracy_score(y_true, y_pred)
    f1 = f1_score(y_true, y_pred, average='macro')
    return {'accuracy': acc, 'macro_f1': f1}
```

## Common pitfalls

- Class imbalance: Over-representation of dominant aspects (e.g., 'software') and underrepresentation of others (e.g., 'price') can skew predictions and reduce generalization to low-frequency categories.
- Subjective severity interpretation: Variability in user tone or emotionally neutral expressions can lead models to underestimate or misclassify severity levels.
- Multi-aspect recognition: Complex complaints involving multiple issues require the model to correctly map distinct aspect-severity pairs rather than predicting a single dominant label.

## Evidence (verbatim from paper)

> Evaluation is performed using Accuracy and macro F1-score, computed independently for both ACD and SD tasks to provide a comprehensive assessment of model performance across the two fine-grained complaint dimensions.

## Citation

```bibtex
@misc{singh2025valor,
  title={Talk, Snap, Complain: Validation-Aware Multimodal Expert Framework for Fine-Grained Customer Grievances},
  author={Rishu Kumar Singh et al. (2025)},
  year={2025},
  note={arXiv:2511.14693}
}
```

- arXiv: 2511.14693