Miners Eval

---
name: miners-eval
description: Evaluates multilingual language models as semantic retrievers across 200+ languages without fine-tuning. It probes bitext mining, retrieval-augmented classification, and in-context learning classification to assess cross-lingual and code-switching capabilities. Use when the user wants to benchmark on MINERS (includes NusaX), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.07424
  bibtex_key: [REDACTED]
  confidence: medium
---

# miners-eval

> MINERS: Multilingual Language Models as Semantic Retrievers — Winata et al. (2024) (arXiv:2406.07424, 2024)

## What this evaluates

Evaluates multilingual language models as semantic retrievers across 200+ languages without fine-tuning. It probes bitext mining, retrieval-augmented classification, and in-context learning classification to assess cross-lingual and code-switching capabilities.

## Datasets

- **MINERS (includes NusaX)** — total ?; splits: test (-1); repo https://github.com/gentaiscool/miners

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted class labels. In retrieval-based classification, final predictions are aggregated via majority voting over the retrieved context examples.
- `precision` — range: percent
  - Proportion of correctly predicted positive instances among all instances predicted as positive in the classification tasks.

## Input / output format

**Input**: Bitext: sentence pairs in different languages. Retrieval-based classification: a query sentence and a candidate document pool. ICL classification: a query sentence plus a few-shot context of retrieved examples with ground-truth labels.

**Output**: Bitext: ranked list of retrieved sentence pairs. Classification: predicted class label (determined by majority voting over retrieved examples or direct generation by the LM).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels) * 100

def majority_vote(retrieved_labels):
    from collections import Counter
    return Counter(retrieved_labels).most_common(1)[0][0]
```

## Common pitfalls

- Performance is highly sensitive to the number of retrieved documents (k), with larger k consistently boosting accuracy via majority voting.
- Code-switching (CS) and cross-lingual code-switching (XL CS) settings are significantly more challenging than monolingual or standard cross-lingual settings.
- Commercial API models may benefit from prior exposure to benchmark datasets, potentially inflating their reported performance.

## Evidence (verbatim from paper)

> The inclusion of few-shot context significantly improves the generative LM’s precision in predicting class labels, leading to enhancements. In retrieval-based classification, a larger $k$ offers more contextual examples, leading to more precise label predictions through majority voting.

## Citation

```bibtex
@misc{winata2024miners,
  title={MINERS: Multilingual Language Models as Semantic Retrievers},
  author={Winata et al. (2024)},
  year={2024},
  note={arXiv:2406.07424}
}
```

- arXiv: 2406.07424