Bengalimoralbench Eval

---
name: bengalimoralbench-eval
description: Evaluates large language models' ability to perform moral reasoning and align with human ethical judgments within Bengali language and South Asian socio-cultural contexts. It probes cultural grounding, commonsense reasoning, and fairness across five everyday moral domains using native-speaker consensus annotations. Use when the user wants to benchmark on BengaliMoralBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.03180
  bibtex_key: [REDACTED]
  confidence: high
---

# bengalimoralbench-eval

> BengaliMoralBench: A Benchmark for Auditing Moral Reasoning in Large Language Models within Bengali Language and Culture — Ridoy et al. (2025) (arXiv:2511.03180, 2025)

## What this evaluates

Evaluates large language models' ability to perform moral reasoning and align with human ethical judgments within Bengali language and South Asian socio-cultural contexts. It probes cultural grounding, commonsense reasoning, and fairness across five everyday moral domains using native-speaker consensus annotations.

## Datasets

- **BengaliMoralBench** — total 3000; splits: test (3000); repo https://github.com/shahriyar-zaman/BengaliMoralBench.git

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted labels out of total instances.
- `F1 score` — range: percent
  - Harmonic mean of precision and recall, reported multiplied by 100.
- `Matthews Correlation Coefficient (MCC)` — range: [-1, 1]
  - Correlation coefficient between observed and predicted binary classifications, ranging from -1 to +1.
- `Cohen’s Kappa` — range: [-1, 1]
  - Statistical measure of inter-annotator agreement for qualitative items, adjusting for chance agreement.

## Input / output format

**Input**: A moral scenario prompt in Bengali (or English) presented in a zero-shot format, asking the model to choose the ethically correct option or judgment.

**Output**: Model-generated text containing the selected classification label or reasoning leading to the final choice.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    f1 = f1_score(golds, preds, average='weighted') * 100
    mcc = matthews_corrcoef(golds, preds)
    kappa = cohen_kappa_score(golds, preds)
    return {'accuracy': acc, 'F1 score': f1, 'MCC': mcc, 'Kappa': kappa}
```

## Common pitfalls

- Using translated prompts instead of native Bengali introduces translation bias that artificially lowers performance.
- Evaluating only on Western-centric ethical frameworks ignores collectivist norms and cultural grounding specific to South Asian contexts.
- Reporting only accuracy without MCC or Kappa may mask class imbalance or chance-level agreement in multi-class moral judgments.

## Evidence (verbatim from paper)

> Model performance was measured using standard classification metrics: accuracy (%), F1 score (x100), Matthews Correlation Coefficient (MCC), and Cohen’s Kappa. These metrics provide a comprehensive assessment of the models’ ability to align with human ethical judgments across the benchmark.

## Citation

```bibtex
@misc{ridoy2025bengalimoralbench,
  title={BengaliMoralBench: A Benchmark for Auditing Moral Reasoning in Large Language Models within Bengali Language and Culture},
  author={Ridoy et al. (2025)},
  year={2025},
  note={arXiv:2511.03180}
}
```

- arXiv: 2511.03180