Co Sight Eval

---
name: co-sight-eval
description: Evaluates long-horizon agentic reasoning, tool-augmented decision making, and factual grounding under conflict-aware verification. It probes how well systems can audit divergent reasoning steps, maintain structured knowledge, and produce accurate answers across multi-hop and interdisciplinary tasks. Use when the user wants to benchmark on GAIA, HLE, Chinese-SimpleQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.21557
  bibtex_key: zhang2025cosight
  confidence: high
---

# co-sight-eval

> Co-Sight: Enhancing LLM-Based Agents via Conflict-Aware Meta-Verification and Trustworthy Reasoning with Structured Facts — Hongwei Zhang et al. (2025) (arXiv:2510.21557, 2025)

## What this evaluates

Evaluates long-horizon agentic reasoning, tool-augmented decision making, and factual grounding under conflict-aware verification. It probes how well systems can audit divergent reasoning steps, maintain structured knowledge, and produce accurate answers across multi-hop and interdisciplinary tasks.

## Datasets

- **GAIA** — total 300; splits: test (300)
- **HLE** — total ?; splits: test (-1)
- **Chinese-SimpleQA** — total 3000; splits: test (3000)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of questions where the model's final answer exactly matches the ground truth.
- `pass@N` — range: percent
  - Reports success if any candidate among N generated trajectories matches the ground truth.

## Input / output format

**Input**: Benchmark questions/tasks (text or multimodal) requiring tool-augmented reasoning and multi-step retrieval.

**Output**: Final answer or selection for each question.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

def compute_pass_at_n(predictions_per_question, gold):
    correct = 0
    for preds, g in zip(predictions_per_question, gold):
        if g in preds:
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Confusing framework-level improvements with backbone model capabilities (e.g., HLE results must be compared against both the framework and the isolated Gemini 2.5 Pro baseline).
- Assuming full-reasoning-chain re-verification is used; the protocol explicitly audits only divergent/conflict nodes to save compute.
- Treating pass@N as equivalent to structured verification; pass@N simply rewards any correct sample in an ensemble, whereas CAMV uses a fixed audit budget across disagreements.

## Evidence (verbatim from paper)

> The oracle-style pass@N reports success if any candidate among N matches the ground truth. As shown in Table 1, the score of CAMV rises from 88.3% to 91.2% when N=1→2, surpassing pass@N under this small-ensemble setting. This improvement indicates that conflict-aware auditing effectively recovers and recombines partial micro-inferences that single trajectories alone fail to consolidate.

## Citation

```bibtex
@misc{zhang2025cosight,
  title={Co-Sight: Enhancing LLM-Based Agents via Conflict-Aware Meta-Verification and Trustworthy Reasoning with Structured Facts},
  author={Hongwei Zhang et al. (2025)},
  year={2025},
  note={arXiv:2510.21557}
}
```

- arXiv: 2510.21557