Cv 18 Ner Eval

---
name: cv-18-ner-eval
description: Evaluates end-to-end and cascaded named entity recognition from Arabic speech. It probes a model's ability to jointly transcribe spoken Arabic and predict fine-grained entity types (21 categories) using BIO-style tagging, as well as extract entity spans and values. Use when the user wants to benchmark on CV-18 NER, or asks about evaluating this task. Reports CoER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.02209
  bibtex_key: saidi2026cv18ner
  confidence: high
---

# cv-18-ner-eval

> CV-18 NER: Augmented Common Voice for Named Entity Recognition from Arabic Speech — Saidi et al. (2026) (arXiv:2604.02209, 2026)

## What this evaluates

Evaluates end-to-end and cascaded named entity recognition from Arabic speech. It probes a model's ability to jointly transcribe spoken Arabic and predict fine-grained entity types (21 categories) using BIO-style tagging, as well as extract entity spans and values.

## Datasets

- **CV-18 NER** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `CoER` **(primary)** — range: percent
  - Concept Error Rate. Computed as (S+D+I)/N, where S, D, I are substitutions, deletions, and insertions, and N is the number of reference entity labels. Evaluates correctness of entity labels (e.g., B-PERS, I-ORG) independently of lexical content.
- `CVER` — range: percent
  - Concept-Value Error Rate. Computed as (S+D+I)/N over tag-value pairs. Evaluates both entity type and entity span by comparing extracted pairs from predicted and reference transcripts.
- `WER` — range: percent
  - Word Error Rate. Computed as (S+D+I)/N over words, after removing special entity tokens from model outputs.
- `Micro-averaged F1-score` — range: percent
  - Standard micro-averaged F1 computed over token-level BIO predictions for text-based NER systems.

## Input / output format

**Input**: Audio signals resampled to 16 kHz for speech models; normalized text transcriptions (punctuation/diacritics removed, Eastern numerals transliterated) for text-based NER models.

**Output**: For speech models: concatenated transcription with entity tags as single atomic special tokens (e.g., B-PERS, I-ORG). For text models: BIO-style token-level labels aligned with input sentences.

## Scoring recipe

```python
def compute_error_rate(pred_tokens, ref_tokens):
    S, D, I = edit_distance_operations(pred_tokens, ref_tokens)
    N = len(ref_tokens)
    return (S + D + I) / N

def compute_coer(pred_tags, ref_tags):
    return compute_error_rate(pred_tags, ref_tags)

def compute_cver(pred_pairs, ref_pairs):
    return compute_error_rate(pred_pairs, ref_pairs)

# WER is computed on word tokens after stripping entity tags.
# F1 is computed via standard micro-averaging over token labels.
```

## Common pitfalls

- Transcription errors in ASR directly degrade downstream NER performance, but lower WER does not guarantee better CoER/CVER due to differences in error types.
- Entity tags are treated as single atomic tokens in the vocabulary; they must be removed before computing WER to avoid skewing results.
- Performance drops sharply for low-frequency entity types (e.g., PRODUCT, LAW) compared to high-frequency ones (e.g., LANGUAGE, GPE).

## Evidence (verbatim from paper)

> Named entity recognition performance is evaluated using Concept Error Rate (CoER) and Concept-Value Error Rate (CVER), which are adapted from spoken language understanding evaluation. WER, CoER, and CVER follow the standard error-rate formulation: Error Rate = (S+D+I)/N, where S, D, and I denote substitutions, deletions, and insertions, and N is the number of reference units. CoER evaluates the correctness of entity labels only (e.g., B-PERS, I-ORG), independently of the lexical content. CVER evaluates both entity type and entity span by comparing tag–value pairs extracted from the annotated transcripts.

## Citation

```bibtex
@misc{saidi2026cv18ner,
  title={CV-18 NER: Augmented Common Voice for Named Entity Recognition from Arabic Speech},
  author={Saidi et al. (2026)},
  year={2026},
  note={arXiv:2604.02209}
}
```

- arXiv: 2604.02209