Gca Tool Use Eval

---
name: gca-tool-use-eval
description: Evaluates an LLM's ability to use region-specific climate tools in a multi-step agentic pipeline. It probes structured tool invocation, argument schema adherence, step-wise reasoning, and end-to-end answer accuracy on Gulf-focused climate queries. Use when the user wants to benchmark on GCA-DS, or asks about evaluating this task. Reports AnsAcc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.12306
  bibtex_key: sheikh2026gcaframework
  confidence: high
---

# gca-tool-use-eval

> GCA Framework: A Gulf-Grounded Dataset and Agentic Pipeline for Climate Decision Support — Muhammad Umer Sheikh et al. (2026) (arXiv:2604.12306, 2026)

## What this evaluates

Evaluates an LLM's ability to use region-specific climate tools in a multi-step agentic pipeline. It probes structured tool invocation, argument schema adherence, step-wise reasoning, and end-to-end answer accuracy on Gulf-focused climate queries.

## Datasets

- **GCA-DS** — total 200000; splits: unspecified (-1)

## Metrics

- `ToolAcc` — range: percent
  - Percentage of steps where the model selects the correct tool from the available tool signatures.
- `ArgAcc` — range: percent
  - Percentage of steps where the model provides correct argument names and adheres to the required schema fields.
- `InstAcc` — range: percent
  - Percentage of steps where the model correctly follows the instruction and formatting requirements for that step.
- `SummAcc` — range: percent
  - Percentage of steps where the model's step-conditioned summary is correct given the actual tool outputs.
- `AnsAcc` **(primary)** — range: percent
  - Percentage of queries where the final answer is accurate after executing the model's predicted tool trace.
- `AnsAcc+I` — range: percent
  - AnsAcc variant that enables image generation during response composition to support visual explanations of temporal trends.

## Input / output format

**Input**: A user query paired with available tool signatures. For evaluation, instances also include a gold tool-usage trace for reference during step-by-step scoring.

**Output**: Structured tool calls (following a specific format with tool name and schema-compliant arguments) and a final synthesized answer. In step-by-step mode, outputs are evaluated per step against gold traces.

## Scoring recipe

```python
def evaluate(predictions, golds):
    step_scores = {'InstAcc': 0, 'ToolAcc': 0, 'ArgAcc': 0, 'SummAcc': 0}
    for p_step, g_step in zip(predictions.steps, golds.steps):
        step_scores['InstAcc'] += int(p_step.format_valid)
        step_scores['ToolAcc'] += int(p_step.tool_name == g_step.tool_name)
        step_scores['ArgAcc'] += int(p_step.args == g_step.args)
        step_scores['SummAcc'] += int(p_step.summary_correct)
    ans_acc = int(predictions.final_answer == golds.final_answer)
    return {k: v/len(predictions)*100 for k,v in step_scores.items()}, ans_acc*100
```

## Common pitfalls

- Format errors in tool-call structure prevent execution entirely, artificially deflating end-to-end accuracy.
- Argument errors (missing or incorrect schema fields) yield invalid tool runs that propagate downstream.
- Models may prematurely answer directly (N/A error) instead of invoking required tools, bypassing necessary grounding.

## Evidence (verbatim from paper)

> We report results in two complementary evaluation modes (Table 3). Step-by-step mode evaluates each step against gold traces: InstAcc (instruction-following for the step), ToolAcc (correct tool choice), ArgAcc (correct argument names/schema fields), and SummAcc (step-conditioned summary correctness given tool outputs). End-to-end mode evaluates the full execution outcome: AnsAcc measures final answer accuracy after executing the predicted tool trace, and AnsAcc+I enables image generation during response composition (useful when answers require visual explanation of temporal trends).

## Citation

```bibtex
@misc{sheikh2026gcaframework,
  title={GCA Framework: A Gulf-Grounded Dataset and Agentic Pipeline for Climate Decision Support},
  author={Muhammad Umer Sheikh et al. (2026)},
  year={2026},
  note={arXiv:2604.12306}
}
```

- arXiv: 2604.12306