Forge Manufacturing Eval

---
name: forge-manufacturing-eval
description: Evaluates multimodal large language models on fine-grained manufacturing tasks, including workpiece verification, surface defect inspection, and assembly verification. It probes the models' ability to combine visual grounding with domain-specific knowledge to identify anomalies or classify conditions. Use when the user wants to benchmark on FORGE, or asks about evaluating this task. Reports exact-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.07413
  bibtex_key: jian2026forge
  confidence: high
---

# forge-manufacturing-eval

> FORGE:Fine-grained Multimodal Evaluation for Manufacturing Scenarios — Jian et al. (2026) (arXiv:2604.07413, 2026)

## What this evaluates

Evaluates multimodal large language models on fine-grained manufacturing tasks, including workpiece verification, surface defect inspection, and assembly verification. It probes the models' ability to combine visual grounding with domain-specific knowledge to identify anomalies or classify conditions.

## Datasets

- **FORGE** — total ?; splits: test (-1); repo https://github.com/AI4Manufacturing/FORGE

## Metrics

- `exact-match accuracy` **(primary)** — range: percent
  - Percentage of test cases where the model's predicted label exactly matches the ground-truth label. Predicted labels are extracted from free-form model responses and compared directly to the correct answer.

## Input / output format

**Input**: Multimodal input consisting of 2D images or 3-view point cloud renderings of manufacturing components, accompanied by a task-specific query. Depending on the evaluation setting, the input may also include reference images of normal assemblies/surfaces or in-context demonstration examples.

**Output**: Free-form text response containing a selected option (e.g., a letter A-F for MCQs) or a surface condition classification label.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        pred_label = extract_label(pred) # Extract letter/label from free-form text
        if pred_label == gold:
            correct += 1
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Models may struggle to correctly format the required letter or label in their free-form generation, leading to false negatives in exact-match scoring.
- The evaluation distinguishes between coarse-grained (wrong workpiece type) and fine-grained (subtle model number variations) difficulties, which require different levels of domain knowledge.
- Performance varies significantly across evaluation settings (Zero-Shot vs. Ref-Cond vs. ICD), so results must be reported per setting rather than aggregated blindly.

## Evidence (verbatim from paper)

> We adopt exact-match accuracy as the evaluation metric. For each test case, the model's predicted MCQ letter is extracted from its free-form response and compared with the ground-truth label. Accuracy is computed as the percentage of cases where the prediction exactly matches the correct answer.

## Citation

```bibtex
@misc{jian2026forge,
  title={FORGE:Fine-grained Multimodal Evaluation for Manufacturing Scenarios},
  author={Jian et al. (2026)},
  year={2026},
  note={arXiv:2604.07413}
}
```

- arXiv: 2604.07413