Hiner Ner Eval

---
name: hiner-ner-eval
description: This benchmark evaluates a model's ability to perform Named Entity Recognition (NER) on Hindi text. It probes the model's capacity to identify and classify entity spans (e.g., Person, Location, Organization, and others) in a language characterized by free word order, lack of capitalization, and spelling variations. Use when the user wants to benchmark on HiNER, or asks about evaluating this task. Reports F1-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2204.13743
  bibtex_key: murthy2022hiner
  confidence: high
---

# hiner-ner-eval

> HiNER: A Large Hindi Named Entity Recognition Dataset — Murthy et al. (2022) (arXiv:2204.13743, 2022)

## What this evaluates

This benchmark evaluates a model's ability to perform Named Entity Recognition (NER) on Hindi text. It probes the model's capacity to identify and classify entity spans (e.g., Person, Location, Organization, and others) in a language characterized by free word order, lack of capitalization, and spelling variations.

## Datasets

- **HiNER** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `F1-Score` **(primary)** — range: percent
  - Entity-level F1 score computed via exact match of BIO-tagged spans. Precision is the fraction of predicted entities that are correct, Recall is the fraction of gold entities that are predicted, and F1 is their harmonic mean. Reported as Micro, Macro, and Weighted averages across 11 entity tags.

## Input / output format

**Input**: Tokenized Hindi sentences formatted with I-O-B (BIO) sequence labels.

**Output**: Sequence of BIO tags (e.g., B-PER, I-LOC, O) corresponding to each input token.

## Scoring recipe

```python
from seqeval.metrics import f1_score
# predictions and gold are lists of lists of BIO tags per sentence
micro_f1 = f1_score(gold, predictions, average='micro')
macro_f1 = f1_score(gold, predictions, average='macro')
weighted_f1 = f1_score(gold, predictions, average='weighted')
```

## Common pitfalls

- Hindi lacks capitalization and has free word order, making entity boundary detection significantly harder than in English.
- The dataset supports 11 entity tags, but many baselines collapse them to 3 (Person, Location, Organization); results must be reported for both configurations.
- Evaluation uses entity-level exact matching via Seqeval, not token-level accuracy, so partial span matches are not counted as correct.

## Evidence (verbatim from paper)

> We use the I-O-B encoding as input format for model training and report the results using Seqeval (Nakayama, 2018) generate evaluation statistics. Table 5: Test Set F1-Score of various pre-trained LMs on our HiNER dataset. This table reports a mean F1-score and its standard deviation over 5 runs.

## Citation

```bibtex
@misc{murthy2022hiner,
  title={HiNER: A Large Hindi Named Entity Recognition Dataset},
  author={Murthy et al. (2022)},
  year={2022},
  note={arXiv:2204.13743}
}
```

- arXiv: 2204.13743