Mfmdqwen Eval

---
name: mfmdqwen-eval
description: Evaluates large language models on multilingual financial misinformation detection across nine tasks in English, Chinese, Greek, and Bengali. It probes the model's ability to identify false or misleading financial claims, handle numerical sensitivity and reversed causality, and generalize across diverse linguistic settings. Use when the user wants to benchmark on MFMDBench, or asks about evaluating this task. Reports Macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.18272
  bibtex_key: liu2026mfmdqwen
  confidence: high
---

# mfmdqwen-eval

> MFMDQwen: Multilingual Financial Misinformation Detection Based on Large Language Model — Liu et al. (2026) (arXiv:2604.18272, 2026)

## What this evaluates

Evaluates large language models on multilingual financial misinformation detection across nine tasks in English, Chinese, Greek, and Bengali. It probes the model's ability to identify false or misleading financial claims, handle numerical sensitivity and reversed causality, and generalize across diverse linguistic settings.

## Datasets

- **MFMDBench** — total ?; splits: test (-1); repo https://github.com/lzw108/FMD

## Metrics

- `Accuracy` — range: [0, 1]
  - Standard classification accuracy, calculated as the number of correct predictions divided by the total number of predictions.
- `Macro-F1` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score, computed as the unweighted mean of the F1 scores for each class, treating all classes equally regardless of support.

## Input / output format

**Input**: Multilingual textual financial claims or news snippets (English, Chinese, Greek, or Bengali), often accompanied by evidence or context.

**Output**: Binary classification label indicating whether the input contains financial misinformation.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    macro_f1 = 0.0
    for label in unique_labels:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == label and g == label)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == label and g != label)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != label and g == label)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
        macro_f1 += f1
    macro_f1 /= len(unique_labels)
    return accuracy, macro_f1
```

## Common pitfalls

- Models with strict safety mechanisms may refuse to answer or output irrelevant responses, artificially lowering F1 scores.
- Reasoning-capable models may underperform non-reasoning counterparts of the same size due to overthinking or generating incorrect intermediate steps.
- Performance on low-resource languages (e.g., Bengali) may appear near-saturated due to limited dataset size and reduced task complexity rather than true model capability.

## Evidence (verbatim from paper)

> We uses metrics such as Accuracy, Macro-F1 for misinformation detection evaluation. In the following analysis, we primarily focus on the F1 score.

## Citation

```bibtex
@misc{liu2026mfmdqwen,
  title={MFMDQwen: Multilingual Financial Misinformation Detection Based on Large Language Model},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2604.18272}
}
```

- arXiv: 2604.18272