Enterprise Benchmarks Eval

---
name: enterprise-benchmarks-eval
description: Evaluates LLMs on domain-specific enterprise tasks across finance, legal, climate, and cybersecurity. It probes capabilities like numerical reasoning, named entity recognition, document relevance ranking, and long-document summarization using real-world industry data. Use when the user wants to benchmark on Earnings Call Transcripts, News Headline, Credit Risk Assessment (NER), KPI-Edgar, FiNER-139, Opinion-based QA (FiQA), Sentiment Analysis (FiQA SA), Insurance QA, ConvFinQA, Financial Text Summarization (EDT), or asks about evaluating this task. Reports Weighted F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.12857
  bibtex_key: zhang2024enterprisebenchmarks
  confidence: high
---

# enterprise-benchmarks-eval

> Enterprise Benchmarks for Large Language Model Evaluation — Bing Zhang et al. (2024) (arXiv:2410.12857, 2024)

## What this evaluates

Evaluates LLMs on domain-specific enterprise tasks across finance, legal, climate, and cybersecurity. It probes capabilities like numerical reasoning, named entity recognition, document relevance ranking, and long-document summarization using real-world industry data.

## Datasets

- **Earnings Call Transcripts** — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
- **News Headline** — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
- **Credit Risk Assessment (NER)** — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
- **KPI-Edgar** — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
- **FiNER-139** — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
- **Opinion-based QA (FiQA)** — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
- **Sentiment Analysis (FiQA SA)** — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
- **Insurance QA** — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
- **ConvFinQA** — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
- **Financial Text Summarization (EDT)** — total 303893; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark

## Metrics

- `Weighted F1` **(primary)** — range: [0, 1]
  - Weighted average of precision and recall across classes, accounting for class imbalance.
- `Entity F1` — range: [0, 1]
  - F1 score computed at the entity level, matching predicted spans and labels to gold standard annotations.
- `Adj F1` — range: [0, 1]
  - Adjusted F1 score for joint NER and relation extraction tasks, penalizing incorrect attribute linking.
- `RR@10` — range: [0, 1]
  - Reciprocal Rank at top 10; 1/k where k is the position of the first relevant document in the ranked list.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly predicted exact numerical values or class labels out of total instances.
- `Rouge-L` — range: [0, 1]
  - Longest Common Subsequence (LCS) based F1 score measuring overlap between generated summary and reference.

## Input / output format

**Input**: Domain-specific text, documents, or tables provided with N-shot prompts (1, 5, 10, or 20 examples) tailored to the task type (classification, NER, QA, ranking, or summarization).

**Output**: Task-specific predictions: class labels, entity spans with tags, ranked document lists, exact numerical values, or abstractive summaries.

## Scoring recipe

```python
def compute_metric(predictions, gold, metric_name):
    if metric_name in ['Weighted F1', 'Entity F1', 'Adj F1']:
        return f1_score(gold, predictions, average='weighted')
    elif metric_name == 'RR@10':
        ranks = [i for i, p in enumerate(predictions[:10]) if p in gold]
        return 1.0 / (ranks[0] + 1) if ranks else 0.0
    elif metric_name == 'Accuracy':
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif metric_name == 'Rouge-L':
        return rouge_l_score(predictions, gold)
    return 0.0
```

## Common pitfalls

- Using generic zero-shot prompts instead of the specified N-shot templates (1, 5, 10, or 20-shot) which are critical for domain adaptation.
- Confusing numerical NER with standard text NER, as financial/XBRL tags depend heavily on context rather than token surface form.
- Ignoring the strict dataset curation rule: benchmarks exclude any dataset without reference answers or with fewer than 100 test cases.

## Evidence (verbatim from paper)

> | Exact Value Match | Chain of Numeric Reasoning (ConvFinQA)Chen et al. ([2022]) | Multi-turn conversational finance question answering data for exploring the chain of numerical reasoning | 1-shot | Accuracy |

## Citation

```bibtex
@misc{zhang2024enterprisebenchmarks,
  title={Enterprise Benchmarks for Large Language Model Evaluation},
  author={Bing Zhang et al. (2024)},
  year={2024},
  note={arXiv:2410.12857}
}
```

- arXiv: 2410.12857