Pariksha Eval

---
name: pariksha-eval
description: Evaluates multilingual and multi-cultural LLM performance across 10 Indic languages using culturally nuanced prompts. It measures model quality via pairwise comparisons (Elo ratings) and direct assessment scores, while also analyzing human-LLM evaluator agreement and various biases (position, verbosity, self-bias). Use when the user wants to benchmark on PARIKSHA, or asks about evaluating this task. Reports Elo rating, Direct Assessment score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.15053
  bibtex_key: watts2024pariksha
  confidence: high
---

# pariksha-eval

> PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data — Watts et al. (2024) (arXiv:2406.15053, 2024)

## What this evaluates

Evaluates multilingual and multi-cultural LLM performance across 10 Indic languages using culturally nuanced prompts. It measures model quality via pairwise comparisons (Elo ratings) and direct assessment scores, while also analyzing human-LLM evaluator agreement and various biases (position, verbosity, self-bias).

## Datasets

- **PARIKSHA** — total 90000; splits: test (-1)

## Metrics

- `Elo rating` **(primary)** — range: other
  - Pairwise comparison metric based on the MLE Elo rating method. Models are ranked by win/loss/tie rates in head-to-head battles across languages.
- `Direct Assessment score` **(primary)** — range: [0, 1]
  - Average score across all query-response pairs for a model, evaluating metrics like Linguistic Acceptability (LA) and Task Quality (TQ).
- `Fleiss Kappa ($\kappa$)` — range: [-1, 1]
  - Measures inter-annotator agreement for categorical items, calculated at a per-datapoint level for both human-human and human-LLM evaluations.
- `Kendall Tau ($\tau$)` — range: [-1, 1]
  - Rank correlation coefficient measuring agreement between human and LLM-evaluator leaderboards.

## Input / output format

**Input**: Pairwise: prompt + response A + response B. Direct Assessment: prompt + response A. Safety: prompt + response.

**Output**: Pairwise: 'A', 'B', or 'tie'. Direct Assessment: numerical score for Linguistic Acceptability (LA) and Task Quality (TQ), plus hallucination flag. Safety: binary problematic/not problematic.

## Scoring recipe

```python
def compute_elo(wins, losses, ties, total_games):
    win_rate = (wins + 0.5 * ties) / total_games
    # MLE Elo rating update based on win_rate
    return elo_rating

def compute_da_score(scores):
    return sum(scores) / len(scores)

def compute_fleiss_kappa(raters, categories):
    # Standard Fleiss Kappa formula over per-datapoint annotations
    return kappa

def compute_kendall_tau(rankings_h, rankings_l):
    return kendalltau(rankings_h, rankings_l)
```

## Common pitfalls

- LLM evaluators pick fewer ties and are more decisive than humans, inflating win rates and skewing leaderboard rankings.
- Direct assessment yields significantly lower human-LLM agreement than pairwise, especially for culturally nuanced prompts and low-resource Indic languages like Bengali and Odia.
- Heuristic toxicity word-matching underperforms LLM evaluators due to limited stem forms and inability to catch contextual toxicity.

## Evidence (verbatim from paper)

> We compute the Percentage Agreement (PA) and Fleiss Kappa ($\kappa$) score which are calculated at a per-datapoint level as well as the general agreement between the leaderboards using Kendall’s Tau ($\tau$).

## Citation

```bibtex
@misc{watts2024pariksha,
  title={PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data},
  author={Watts et al. (2024)},
  year={2024},
  note={arXiv:2406.15053}
}
```

- arXiv: 2406.15053