M3finmeeting Eval

---
name: m3finmeeting-eval
description: Probes long-context financial meeting understanding across three languages (EN, ZH, JA) and 11 GICS sectors. It evaluates a model's ability to condense lengthy transcripts into structured summaries, extract relevant question-answer pairs, and localize precise answers within designated sections while ignoring noise. Use when the user wants to benchmark on M3FinMeeting, or asks about evaluating this task. Reports compression ratio.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.02510
  bibtex_key: zhu2025m3finmeeting
  confidence: high
---

# m3finmeeting-eval

> M$^3$FinMeeting: A Multilingual, Multi-Sector, and Multi-Task Financial Meeting Understanding Evaluation Dataset — Zhu et al. (2025) (arXiv:2506.02510, 2025)

## What this evaluates

Probes long-context financial meeting understanding across three languages (EN, ZH, JA) and 11 GICS sectors. It evaluates a model's ability to condense lengthy transcripts into structured summaries, extract relevant question-answer pairs, and localize precise answers within designated sections while ignoring noise.

## Datasets

- **M3FinMeeting** — total 600; splits: test (600); repo https://github.com/aliyun/qwen-dianjin

## Metrics

- `compression ratio` **(primary)** — range: other
  - Ratio of source document length to reference summary length, computed at both token-level and sentence-level following Koh et al. (2022).

## Input / output format

**Input**: Long-context transcribed financial meeting text (tokenized via tiktoken cl100k_base) in English, Chinese, or Japanese, ranging from ~3.5K to >25K tokens. For QA tasks, context is restricted to a designated section with other sections acting as noise.

**Output**: Summarization: sequentially concatenated section summaries. QA Pair Extraction: structured pairs of questions and their corresponding answers. QA: direct answer string localized to the designated section.

## Scoring recipe

```python
def compute_compression_ratio(source_text, ref_text):
    src_tokens = tokenize(source_text)
    ref_tokens = tokenize(ref_text)
    src_sents = split_sentences(source_text)
    ref_sents = split_sentences(ref_text)
    token_ratio = len(src_tokens) / len(ref_tokens)
    sent_ratio = len(src_sents) / len(ref_sents)
    return token_ratio, sent_ratio
```

## Common pitfalls

- ASR transcripts require manual correction before evaluation; using raw ASR output introduces noise that unfairly penalizes models.
- QA tasks restrict context to a single designated section, with other sections acting as deliberate noise to test precise localization rather than global retrieval.
- Compression ratio is a length-based metric following Koh et al. (2022), not a semantic similarity score like ROUGE or BERTScore.

## Evidence (verbatim from paper)

> Given transcribed speech documents and their reference summaries, we follow Koh et al. (2022) to compute the compression ratio of a source document length against its reference summary length at both token-level and sentence-level.

## Citation

```bibtex
@misc{zhu2025m3finmeeting,
  title={M$^3$FinMeeting: A Multilingual, Multi-Sector, and Multi-Task Financial Meeting Understanding Evaluation Dataset},
  author={Zhu et al. (2025)},
  year={2025},
  note={arXiv:2506.02510}
}
```

- arXiv: 2506.02510