Nep Glu Eval

---
name: nep-glu-eval
description: Evaluates encoder-based language models on core Nepali natural language understanding tasks, including named entity recognition, part-of-speech tagging, text classification, and categorical pair similarity. Use when the user wants to benchmark on Nep-gLUE, or asks about evaluating this task. Reports Nep-gLUE Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.15734
  bibtex_key: thapa2024development
  confidence: high
---

# nep-glu-eval

> Development of Pre-Trained Transformer-based Models for the Nepali Language — Thapa et al. (2024) (arXiv:2411.15734, 2024)

## What this evaluates

Evaluates encoder-based language models on core Nepali natural language understanding tasks, including named entity recognition, part-of-speech tagging, text classification, and categorical pair similarity.

## Datasets

- **Nep-gLUE** — total ?; splits: test (-1)

## Metrics

- `Nep-gLUE Score` **(primary)** — range: percent
  - Macro-average of task-specific accuracy scores across four subtasks: NER, POS tagging, text classification (CC), and categorical pair similarity (CPS).

## Input / output format

**Input**: Tokenized text sequences or sentence pairs for NLU tasks (NER, POS, classification, similarity).

**Output**: Task-specific predictions: entity labels, POS tags, class labels, or similarity scores.

## Scoring recipe

```python
def nep_glu_score(predictions, golds):
    task_scores = []
    for task in ['NER', 'POS', 'CC', 'CPS']:
        task_scores.append(accuracy(predictions[task], golds[task]))
    return sum(task_scores) / len(task_scores)
```

## Common pitfalls

- Hyperparameter tuning is task-specific (3-10 epochs, learning rates from 1e-5 to 5e-5); best model is selected per task on the test set, not via validation.
- The aggregate Nep-gLUE score masks per-task performance; a model can score high overall while underperforming on specific linguistic phenomena.

## Evidence (verbatim from paper)

> We used the Nepali Language Evaluation Benchmark, or Nep-gLUE Timilsina et al. ([2022]). It consists of four tasks, including Named Entity Recognition (NER), Part-of-Speech (POS) Tagging, text classification, and categorical pair similarity... Our models outperformed all existing models across all tasks, scoring 95.60 on Nep-gLUE Timilsina et al. ([2022]) benchmark

## Citation

```bibtex
@misc{thapa2024development,
  title={Development of Pre-Trained Transformer-based Models for the Nepali Language},
  author={Thapa et al. (2024)},
  year={2024},
  note={arXiv:2411.15734}
}
```

- arXiv: 2411.15734