Evaluation

---
name: evaluation
description: "Run and interpret PointLLM Objaverse and ModelNet40 evaluations,
  including local result validation, OpenAI-assisted judging, and traditional
  caption metrics."
disable-model-invocation: true
metadata:
  disco-role: operating
license: CC BY-NC-SA 4.0
---

# PointLLM evaluation

Use this skill after an inference run has produced a JSON result file. It
covers benchmark generation and scoring, not model loading or raw dataset
installation. Keep those concerns in the inference and data skills. Use this
skill to choose a task, check the result contract, run the appropriate judge,
and interpret the output without hiding invalid responses or metric caveats.

## Route the evaluation

1. Identify the result kind before spending compute or API budget:
   - Objaverse open-vocabulary classification: `open-free-form-classification`.
   - Objaverse object captioning: `object-captioning`.
   - ModelNet40 close-set classification: `modelnet-close-set-classification`.
   - Caption-only local comparison: traditional metrics.
2. Validate the inference JSON locally first:
   `python scripts/validate_results_json.py RESULTS.json --kind ...`.
   This bundled checker makes no API calls and downloads no models.
3. Confirm that `prompt` and each result's `ground_truth`, `model_output`, and
   `object_id` match the selected task. ModelNet rows additionally need the
   integer `ground_truth` and `label_name`.
4. Preserve the original inference JSON. Write judge output beside it, using a
   distinct `_evaluated_<model>.json` or `_evaluated_traditional.json` name.

## Generate benchmark results

The project evaluation entry points expose the following stable contracts:

- Objaverse classification/captioning: select `--task_type classification` or
  `captioning`, with classification prompt index 0 or 1 and captioning index 2.
- ModelNet40: select `--prompt_index 0` or `1`; keep `--shuffle False` because
  the dataset index is used as the object ID.
- The generated file contains `prompt` and `results`. Objaverse rows use a
  string object ID and annotation text; ModelNet rows use an index, numeric
  class, generated text, and class name.

Generation needs the model, tokenizer, CUDA placement, and data paths supplied
by the sibling skills. Do not improvise a second result format: downstream
judges and resume logic key on `object_id`.

## Run OpenAI-assisted scoring

1. Obtain explicit approval for external API calls and a cost ceiling. Set
   `OPENAI_API_KEY` in the process environment; never place it in JSON, a
   prompt, a skill file, or shell history committed to a project.
2. Choose the exact `--eval_type` and a supported `--model_type`; see
   `references/cli-reference.md` for commands and filenames.
3. Prefer `--parallel --num_workers N` only after checking rate limits and the
   budget. A smaller worker count is safer for first validation.
4. The evaluator retries selected OpenAI rate-limit, service-unavailable, and
   timeout errors with exponential backoff, but non-retryable errors stop the
   run. It records prompt/completion tokens and estimated `GPT_cost`.
5. On Ctrl+C or an exception, leave the generated
   `<output>_processed_temp.json` in place. Re-run the same command in the
   same output directory to resume by `object_id`; do not edit or merge the
   temporary file manually. A successful completion writes the final JSON and
   removes the temporary file.

Treat `GPT_cost` as an estimate based on the source's hard-coded historical
price table, not a billing statement. Recheck provider pricing before a large
run. Never use a guessed or empty credential to probe the API.

## Interpret outputs

- Open classification reports `accuracy`, counts, invalid responses, token
  totals, cost, and per-row `gpt_cls_result` (`T`, `F`, or `INVALID`).
- ModelNet reports `accuracy`, `clean_accuracy`, and
  `invalid_correct_predictions`; an invalid judge answer is randomly assigned
  a valid category by the evaluator, so it can coincidentally count as correct.
  Report both headline and clean values and the invalid count.
- Captioning reports `average_score` on a 0--100 scale, `total_score`, invalid
  count, token totals, cost, and per-row `gpt_score` (0--100 or -1).
- Traditional evaluation reports BLEU-1/2/3/4, ROUGE-1/2/L, METEOR,
  Sentence-BERT, and SimCSE scores. These are supplementary: README guidance
  warns that BLEU, ROUGE, and METEOR favor short captions and weakly represent
  semantic accuracy/diversity.

Do not call an invalid classification response a genuine model error without
separately reporting the invalid rate. Do not compare traditional metrics to
GPT scores as if they were on the same scale or measuring the same construct.

## Failure handling and handoff

- Import/dependency failures: verify the pinned environment and optional
  evaluator dependencies before retrying; see the troubleshooting reference.
- Data/config failures: hand the missing annotation, point-cloud, or ModelNet
  configuration issue to the data sibling rather than patching a result file.
- API failures: stop when the credential, model entitlement, rate limit, or
  cost ceiling is unclear; resume only from a trusted temp file.
- Output failures: run the bundled validator, inspect duplicate/missing object
  IDs, and retain invalid rows and raw judge text for auditability.

A complete handoff names the benchmark, prompt index, inference file, scoring
mode/model, worker count, final and temporary output paths, counts, invalid
responses, token/cost totals, and unresolved caveats. Record skipped API or
full-benchmark checks explicitly.