Monkey Jump Eval

---
name: monkey-jump-eval
description: Evaluates the performance and parameter/memory/throughput efficiency of a gradient-free MoE-style PEFT routing mechanism across text, image, and video benchmarks compared to standard and MoE-PEFT baselines. Use when the user wants to benchmark on 47 Benchmarks (Text/Image/Video), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.06356
  bibtex_key: prottasha2026monkeyjump
  confidence: high
---

# monkey-jump-eval

> Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning — Prottasha et al. (2026) (arXiv:2601.06356, 2026)

## What this evaluates

Evaluates the performance and parameter/memory/throughput efficiency of a gradient-free MoE-style PEFT routing mechanism across text, image, and video benchmarks compared to standard and MoE-PEFT baselines.

## Datasets

- **47 Benchmarks (Text/Image/Video)** — total 153000; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted labels or tokens compared to ground truth. Reported as mean ± standard deviation over 5 independent runs.
- `throughput` — range: it/s
  - Number of iterations processed per second during training or inference.
- `peak_gpu_memory` — range: GB
  - Maximum GPU memory allocated during forward/backward passes.

## Input / output format

**Input**: Text prompts (Llama-3-8B-Instruct), images, or video frames (LLaVA-OneVision-Qwen2-7B) depending on modality.

**Output**: Class labels or generated text responses.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

def compute_efficiency(model, dataloader, device):
    params = sum(p.numel() for p in model.parameters() if p.requires_grad)
    peak_mem = torch.cuda.max_memory_allocated(device) / 1e9
    start = time.time()
    for batch in dataloader:
        model(batch)
    throughput = len(dataloader) / (time.time() - start)
    return params, peak_mem, throughput
```

## Common pitfalls

- Results are averaged over 5 runs with standard deviation reported; single-run scores may vary significantly.
- Efficiency metrics (memory, throughput) are hardware-specific (H100 GPU) and may not generalize to other architectures.
- MJ variants reuse existing adapters rather than adding new experts, so total model size remains nearly identical to baselines despite different routing.

## Evidence (verbatim from paper)

> On CS&QA, MJLoRAFA achieves the highest accuracy (66.16%), outperforming MoRE (66.09%) and MixLoRA (66.03%). ... MJ achieves 1.5–2× faster training. MJ-Propulsion reaches 5.94 it/s throughput and completes training in 5.0 minutes, compared to 3.02–3.83 it/s and 7.7–9.4 minutes for MoE-PEFT methods.

## Citation

```bibtex
@misc{prottasha2026monkeyjump,
  title={Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning},
  author={Prottasha et al. (2026)},
  year={2026},
  note={arXiv:2601.06356}
}
```

- arXiv: 2601.06356