Cendol Eval

---
name: cendol-eval
description: This evaluation protocol assesses the language proficiency, generalization capability, and local cultural commonsense reasoning of instruction-tuned LLMs across Indonesian and nine indigenous languages. It probes zero-shot performance on seen and unseen tasks and languages, as well as nuanced understanding of regional proverbs, figures of speech, and story endings. Use when the user wants to benchmark on COPAL-ID, MABL, IndoStoryCloze, MAPS, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.06138
  bibtex_key: cahyawijaya2024cendol
  confidence: high
---

# cendol-eval

> Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages — Cahyawijaya et al. (2024) (arXiv:2404.06138, 2024)

## What this evaluates

This evaluation protocol assesses the language proficiency, generalization capability, and local cultural commonsense reasoning of instruction-tuned LLMs across Indonesian and nine indigenous languages. It probes zero-shot performance on seen and unseen tasks and languages, as well as nuanced understanding of regional proverbs, figures of speech, and story endings.

## Datasets

- **COPAL-ID** — total ?; splits: test (-1)
- **MABL** — total ?; splits: test (-1)
- **IndoStoryCloze** — total ?; splits: test (-1)
- **MAPS** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Task-dependent; calculated as the proportion of correctly predicted options or generated completions against gold labels in zero-shot settings.

## Input / output format

**Input**: Zero-shot prompts containing scenarios, questions, or sentence fragments with multiple-choice options or open-ended completion targets in Indonesian or indigenous languages.

**Output**: Model-generated text selecting the most plausible option, completing the story, or interpreting the figure of speech/proverb.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize(pred) == normalize(gold):
        correct += 1
return correct / len(gold_labels)
```

## Common pitfalls

- Zero-shot evaluation may not reflect performance after instruction tuning or fine-tuning.
- Cultural commonsense datasets require deep regional knowledge that generic multilingual models often lack, leading to systematic failures on local nuances.
- Code-switching and dialectal variations in indigenous language datasets can cause tokenization mismatches and degraded performance.

## Evidence (verbatim from paper)

> To assess the language capability of Cendol models across Indonesian indigenous languages, we design an evaluation benchmark with 15 datasets covering 10 languages including Indonesian and 9 local languages spoken in Indonesia, i.e., Acehnese (ace), Balinese (ban), Banjarese (bjn), Buginese (bug), Javanese (jav), Madurese (mad), Minangkabau (min), Ngaju (nij), and Sundanese (sun). For all tasks and datasets, we evaluate the model in a zero-shot prompting setting.

## Citation

```bibtex
@misc{cahyawijaya2024cendol,
  title={Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages},
  author={Cahyawijaya et al. (2024)},
  year={2024},
  note={arXiv:2404.06138}
}
```

- arXiv: 2404.06138