Bee 8b Mllm Eval

---
name: bee-8b-mllm-eval
description: Evaluates the visual reasoning, factual accuracy, OCR, chart understanding, and mathematical capabilities of fully open multimodal large language models (MLLMs) against a comprehensive suite of established benchmarks. The protocol tests the model's ability to process images and text prompts, generate responses in a thinking mode, and achieve high scores across general VQA, document/chart analysis, and complex math/reasoning tasks. Use when the user wants to benchmark on Bee-8B Evaluation Benchmarks, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.13795
  bibtex_key: zhang2025bee
  confidence: high
---

# bee-8b-mllm-eval

> Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMs — Yi Zhang et al. (arXiv:2510.13795, 2025)

## What this evaluates

Evaluates the visual reasoning, factual accuracy, OCR, chart understanding, and mathematical capabilities of fully open multimodal large language models (MLLMs) against a comprehensive suite of established benchmarks. The protocol tests the model's ability to process images and text prompts, generate responses in a thinking mode, and achieve high scores across general VQA, document/chart analysis, and complex math/reasoning tasks.

## Datasets

- **Bee-8B Evaluation Benchmarks** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions or normalized score based on benchmark-specific rubrics. Some benchmarks (e.g., DocVQA) use LLM-based judging with Qwen3-32B instead of exact string matching.

## Input / output format

**Input**: Multimodal input consisting of an image and a text prompt/question.

**Output**: Text response (answer or step-by-step reasoning trace).

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        # Use exact match or LLM-judge depending on benchmark
        if is_correct(pred, gold_ans):
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Some benchmarks use LLM-as-a-judge (e.g., DocVQA) instead of exact string matching.
- Evaluation is run in 'thinking mode' with a 16,384 token output limit.
- Scores are reported as percentages; some benchmarks have different normalization schemes.

## Evidence (verbatim from paper)

> Its most significant advantages are observed in factual accuracy and complex multi-step reasoning, directly reflecting the strengths of our Honey-Data-15M dataset. We evaluated our model using a customized VLMEvalKit*(Duan et al., [2024])*... Our model was evaluated with the thinking mode and a maximum response length of 16,384 tokens.

## Citation

```bibtex
@misc{zhang2025bee,
  title={Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMs},
  author={Yi Zhang et al.},
  year={2025},
  note={arXiv:2510.13795}
}
```

- arXiv: 2510.13795