Pralekha Eval

---
name: pralekha-eval
description: Evaluates cross-lingual document alignment (CLDA) techniques by measuring chunk/sentence-level alignment accuracy (intrinsic) and the resulting document-level machine translation quality (extrinsic) across English and 11 Indic languages. Use when the user wants to benchmark on Pralekha, or asks about evaluating this task. Reports F1 Score, DocCOMET.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.19096
  bibtex_key: suryanarayanan2024pralekha
  confidence: high
---

# pralekha-eval

> Pralekha: Cross-Lingual Document Alignment for Indic Languages — Sanjay Suryanarayanan et al. (2024) (arXiv:2411.19096, 2024)

## What this evaluates

Evaluates cross-lingual document alignment (CLDA) techniques by measuring chunk/sentence-level alignment accuracy (intrinsic) and the resulting document-level machine translation quality (extrinsic) across English and 11 Indic languages.

## Datasets

- **Pralekha** — total ?; splits: train (-1), val (1000), test (1000); repo https://github.com/AI4Bharat/Pralekha

## Metrics

- `Precision` — range: [0, 1]
  - Ratio of correctly predicted alignment links to all predicted links.
- `Recall` — range: [0, 1]
  - Ratio of correctly predicted alignment links to all gold alignment links.
- `F1 Score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (P * R) / (P + R).
- `DocCOMET` **(primary)** — range: [0, 1]
  - Reference-based neural MT quality metric using the Unbabel/wmt22-comet-da model, extended to incorporate document-level context.
- `ChrF` — range: percent
  - Character-level F-score based on n-gram overlaps between reference and hypothesis.

## Input / output format

**Input**: Parallel document pairs in English and an Indic language, with 50% unalignable noise documents injected for intrinsic evaluation. For extrinsic evaluation, aligned document pairs are used as training data for fine-tuning LLMs.

**Output**: Intrinsic: predicted alignment links between text chunks/sentences/documents. Extrinsic: translated document text generated via greedy decoding (max 4,096 tokens).

## Scoring recipe

```python
def compute_alignment_metrics(pred_links, gold_links):
    tp = len(pred_links & gold_links)
    fp = len(pred_links - gold_links)
    fn = len(gold_links - pred_links)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1

def score_mt(hypothesis, reference):
    comet_score = run_unbabel_wmt22_comet_da(hypothesis, reference)
    chrf_score = compute_chrf_ngram_overlap(hypothesis, reference)
    return comet_score, chrf_score
```

## Common pitfalls

- Omitting the 50% injection of unalignable noise documents from Sangraha Unverified, which artificially inflates precision and misrepresents real-world corpus conditions.
- Using a DAC threshold other than 0.1, which the paper explicitly notes is calibrated to balance precision and recall based on appendix analysis.
- Evaluating extrinsic performance without fine-tuning bilingual models for each specific English↔Indic language pair, as cross-lingual transfer without language-specific tuning degrades results.

## Evidence (verbatim from paper)

> To assess the intrinsic performance of various document alignment methods, we evaluate three key metrics: precision, recall, and F1 score. ... To evaluate document-level MT performance, we use DocCOMET, an extension of COMET that incorporates document-level context for improved quality assessment. Alongside COMET, we also report ChrF, a character-level F-score metric based on n-gram overlaps.

## Citation

```bibtex
@misc{suryanarayanan2024pralekha,
  title={Pralekha: Cross-Lingual Document Alignment for Indic Languages},
  author={Sanjay Suryanarayanan et al. (2024)},
  year={2024},
  note={arXiv:2411.19096}
}
```

- arXiv: 2411.19096