Marca Eval

---
name: marca-eval
description: Evaluates LLMs' ability to perform multilingual web search and extract multiple entities from search results. It probes task decomposition, cross-lingual retrieval, and evidence aggregation under different agentic interaction frameworks. Use when the user wants to benchmark on MARCA, or asks about evaluating this task. Reports Checklist Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.14448
  bibtex_key: almeida2026marca
  confidence: high
---

# marca-eval

> MARCA: A Checklist-Based Benchmark for Multilingual Web Search — Almeida et al. (2026) (arXiv:2604.14448, 2026)

## What this evaluates

Evaluates LLMs' ability to perform multilingual web search and extract multiple entities from search results. It probes task decomposition, cross-lingual retrieval, and evidence aggregation under different agentic interaction frameworks.

## Datasets

- **MARCA** — total 52; splits: test (52); repo https://github.com/maritaca-ai/MARCA

## Metrics

- `Checklist Accuracy` **(primary)** — range: [0, 1]
  - The fraction of ground-truth checklist items (entities/facts) correctly identified and attributed in the model's final answer relative to the total number of items in the checklist. Computed per question and averaged across the dataset.

## Input / output format

**Input**: A multi-entity information-seeking question in English or Portuguese. The model interacts with a web search engine either directly (Basic framework) or via an orchestrator delegating to subagents (Orchestrator framework).

**Output**: A final natural language answer containing a list of entities/facts that should match the predefined checklist for the question.

## Scoring recipe

```python
def compute_checklist_accuracy(predictions, gold_checklists):
    correct = 0
    total = 0
    for pred, checklist in zip(predictions, gold_checklists):
        pred_entities = extract_entities_from_answer(pred)
        matched = len(set(pred_entities) & set(checklist))
        correct += matched
        total += len(checklist)
    return correct / total
```

## Common pitfalls

- Models may implicitly translate Portuguese queries to English, missing region-specific content that only exists in Portuguese.
- Orchestration overhead can hurt performance for strong models that already handle complex queries in a single context.
- Checklist accuracy measures coverage/completeness, not necessarily factual correctness of individual retrieved snippets.

## Evidence (verbatim from paper)

> Table 1 reports Checklist Accuracy (mean ± run-level standard deviation) across languages and inference frameworks for 14 models.

## Citation

```bibtex
@misc{almeida2026marca,
  title={MARCA: A Checklist-Based Benchmark for Multilingual Web Search},
  author={Almeida et al. (2026)},
  year={2026},
  note={arXiv:2604.14448}
}
```

- arXiv: 2604.14448