Mined Eval

---
name: mined-eval
description: Probes large multimodal models' temporal awareness and time-sensitive knowledge across six dimensions: cognition, awareness, trustworthiness, understanding, reasoning, and robustness. It evaluates how well models recall, reason about, and reject outdated or misaligned temporal facts in multimodal queries. Use when the user wants to benchmark on Mined, or asks about evaluating this task. Reports Cover Exact Match (CEM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.19457
  bibtex_key: jiang2025mined
  confidence: high
---

# mined-eval

> MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Models — Jiang et al. (2025) (arXiv:2510.19457, 2025)

## What this evaluates

Probes large multimodal models' temporal awareness and time-sensitive knowledge across six dimensions: cognition, awareness, trustworthiness, understanding, reasoning, and robustness. It evaluates how well models recall, reason about, and reject outdated or misaligned temporal facts in multimodal queries.

## Datasets

- **Mined** — total ?; splits: test (-1)

## Metrics

- `Cover Exact Match (CEM)` **(primary)** — range: [0, 1]
  - CEM is 1 if the model's output is a subset of the ground truth, else 0. Capacity is the average CEM across all subtasks.

## Input / output format

**Input**: Image and text prompt (varies across four configurations: “Question”, “Generalization Question”, “Image”, and “Generalization Image”) conveying time-sensitive knowledge queries.

**Output**: Text response from the model.

## Scoring recipe

```python
def compute_cem(model_output, ground_truth):
    return 1 if set(model_output.split()) <= set(ground_truth.split()) else 0

def compute_capacity(outputs, truths):
    return sum(compute_cem(o, t) for o, t in zip(outputs, truths)) / len(truths)
```

## Common pitfalls

- Strict exact-match (subset) requirement means minor phrasing differences yield a score of 0.
- Prompt Agreement averages scores across four prompt variations per instance, which can obscure model sensitivity to specific phrasing.
- Temporal misalignment context (especially past dates) significantly degrades performance, particularly for smaller open-source models.

## Evidence (verbatim from paper)

> In the evaluation of all subtasks, the model is considered to have correctly responded to the time-sensitive knowledge only when its output exactly matches the corresponding ground truth. Therefore, we evaluate the model’s outputs using Cover Exact Match (CEM)*(Xu et al., [2023])* score for each subtask. The model’s capacity in this dimension is defined as the average CEM score across all subtasks. CEM requires matching model’s outputs with ground truth.

## Citation

```bibtex
@misc{jiang2025mined,
  title={MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Models},
  author={Jiang et al. (2025)},
  year={2025},
  note={arXiv:2510.19457}
}
```

- arXiv: 2510.19457