Scenario Bias Financial Misinfo Eval

---
name: scenario-bias-financial-misinfo-eval
description: Evaluates how scenario-induced contextual factors (personality, region, identity) and multilingual settings alter LLM judgments on financial misinformation claims, quantifying behavioral bias as the performance shift relative to a neutral baseline. Use when the user wants to benchmark on Multilingual Financial Misinformation Dataset, or asks about evaluating this task. Reports Bias_scen.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.05403
  bibtex_key: liu2026sameclaim
  confidence: high
---

# scenario-bias-financial-misinfo-eval

> Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection — Liu et al. (2026) (arXiv:2601.05403, 2026)

## What this evaluates

Evaluates how scenario-induced contextual factors (personality, region, identity) and multilingual settings alter LLM judgments on financial misinformation claims, quantifying behavioral bias as the performance shift relative to a neutral baseline.

## Datasets

- **Multilingual Financial Misinformation Dataset** — total 144; splits: test (144); repo https://github.com/lzw108/FMD

## Metrics

- `Bias_scen` **(primary)** — range: [0, 1]
  - Absolute difference in F1 scores between scenario-conditioned predictions and base (unconditioned) predictions against ground truth: |F1(l_scen, l_gold) - F1(l_base, l_gold)|.

## Input / output format

**Input**: A financial misinformation claim combined with a contextual scenario prompt (specifying role, personality/region/identity) or a neutral prompt without scenario context.

**Output**: A binary truthfulness label: True or False.

## Scoring recipe

```python
f1_scen = f1_score(gold, predictions_scen, average='binary')
f1_base = f1_score(gold, predictions_base, average='binary')
bias_scen = abs(f1_scen - f1_base)
```

## Common pitfalls

- Calculating bias as absolute F1 under scenario rather than the difference from the base prompt.
- Using accuracy instead of F1, which can be misleading for imbalanced true/false distributions in financial news.
- Ignoring translation artifacts in non-English splits that may artificially inflate or deflate bias scores.

## Evidence (verbatim from paper)

> Given a piece of financial information claim $c$, the task is to determine the truthfulness label ($l_{scen},l_{base},l_{gold}\in L\={True,False}$) of the claim $c$ in the scenario $s$. ... $\text{Bias}_{\text{scen}}\=|\text{F1}(l_{\text{scen}},l_{\text{gold}})-\text{F1}(l_{\text{base}},l_{\text{gold}})|$ ... $l_{\text{scen}}$ denotes the LLMs’ predictions under specific financial scenarios, $l_{\text{base}}$ denotes their predictions without financial scenario information, and $l_{\text{gold}}$ represents the ground-truth labels. The behavioral bias is quantified as the difference in F1 scores between these two cases, reflecting how scenario context changes verification performance for the same claim.

## Citation

```bibtex
@misc{liu2026sameclaim,
  title={Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2601.05403}
}
```

- arXiv: 2601.05403