Evaluation And Webui

---
name: evaluation-and-webui
description: "Configure and troubleshoot FlashRAG evaluation metrics, prediction
  parsing, result validation, and WebUI/chat/evaluate setup without launching
  services by default."
disable-model-invocation: true
metadata:
  disco-role: operating
license: MIT
---

# FlashRAG Evaluation and WebUI

Use this sub-skill when the task is about FlashRAG evaluation scores, metric selection, parsed predictions, evaluation output validation, or WebUI chat/evaluate configuration triage.

## Route by Task

- **Evaluation metrics**: Use `references/evaluation-api.md` to choose `metrics`, fill `metric_setting`, understand output keys, and validate `Evaluator` behavior.
- **Prediction parsing**: Use `references/evaluation-api.md` before scoring outputs from SelfAsk, IRCoT, first-line answers, or Gaokao multiple-choice generations.
- **WebUI setup triage**: Use `references/webui.md` to inspect Gradio UI modules, saved YAML configs, chat/evaluate inputs, and index/model configuration requirements.
- **Failure diagnosis**: Use `references/troubleshooting.md` for missing metric packages, Chinese/English metric mismatches, LLMJudge setup, malformed predictions, WebUI imports, and port/service caveats.
- **Metric inventory**: Run `skills/flashrag/sub-skills/evaluation-and-webui/scripts/inspect_metrics.py` against an installed or source checkout to list implemented metric classes without launching generators, retrievers, or WebUI services.

## Safe Defaults

- Do not launch the WebUI, start model servers, call external APIs, or load large models unless the user explicitly asks.
- Prefer static inspection, config preview, and small dataset/sample checks before full evaluation.
- Keep evaluation responsibilities separate from data/config setup, pipeline execution, and model backend selection; route those to the corresponding FlashRAG sub-skills.

## Minimal Evaluation Checklist

1. Confirm dataset objects contain the fields required by selected metrics: usually `pred`, `golden_answers`, `choices`, and optionally `retrieval_result`, `prompt`, `question`, `subject`, or `question_type`.
2. Ensure `metrics` names match implemented `metric_name` values, lowercased by `Evaluator`.
3. Fill `metric_setting` for metrics that require it, especially retrieval top-k, BLEU options, tokenizers, or LLMJudge settings.
4. Apply the correct prediction parser before exact-match style metrics when a pipeline emits reasoning plus a final answer.
5. Validate both aggregate result keys and per-item evaluation scores when `save_intermediate_data` is enabled.

## References and Scripts

- `references/evaluation-api.md` — evaluator lifecycle, available metrics, parser behavior, metric config, and validation patterns.
- `references/webui.md` — WebUI module map, chat/evaluate flow, config save/load, and non-launch triage steps.
- `references/troubleshooting.md` — targeted fixes for dependency, metric, prediction, and WebUI failures.
- `scripts/inspect_metrics.py` — bundled at `skills/flashrag/sub-skills/evaluation-and-webui/scripts/inspect_metrics.py`; static AST metric lister with optional safe import mode.