Microg 4m Eval

---
name: microg-4m-eval
description: Evaluates video-based human action recognition, temporal captioning, and visual question answering in microgravity environments. It probes a model's ability to generalize to orientation-invariant motion, floating objects, and lack of ground contact where terrestrial models typically fail. Use when the user wants to benchmark on MicroG-4M, or asks about evaluating this task. Reports mAP@0.5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.02845
  bibtex_key: wen2025microg4m
  confidence: high
---

# microg-4m-eval

> Go Beyond Earth: Understanding Human Actions and Scenes in Microgravity Environments — Wen et al. (2025) (arXiv:2506.02845, 2025)

## What this evaluates

Evaluates video-based human action recognition, temporal captioning, and visual question answering in microgravity environments. It probes a model's ability to generalize to orientation-invariant motion, floating objects, and lack of ground contact where terrestrial models typically fail.

## Datasets

- **MicroG-4M** — total 4759; splits: train (3331), val (475), test (953); repo https://github.com/LEI-QI-233/HAR-in-Space

## Metrics

- `mAP@0.5` **(primary)** — range: [0, 100]
  - Mean Average Precision at Intersection-over-Union threshold 0.5, macro-averaged over all action classes. Measures detection accuracy per category.
- `F1-score` — range: [0, 100]
  - Macro-averaged F1 score computed per class and then averaged across all action categories.
- `CIDEr` — range: [0, 100]
  - Standard lexical metric for caption/QA evaluation, rescaled to a 0–100 range for consistency.
- `S-BERT` — range: [0, 100]
  - Cosine similarity between Sentence-BERT embeddings of predicted and reference texts, rescaled to 0–100.
- `S-VQA` — range: [0, 100]
  - Semantic equivalence metric for VQA answers, computed as cosine similarity between Sentence-BERT embeddings of predicted and reference answers.

## Input / output format

**Input**: Video clips (typically sampled within a 3-second window, frame counts vary by model) for HAR and captioning; video clips paired with natural language questions for VQA.

**Output**: For HAR: predicted action class labels with bounding boxes/timestamps. For captioning: descriptive text captions. For VQA: natural language answers.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task):
    if task == 'HAR':
        class_aps = []
        for cls in classes:
            pred_boxes = [p for p in predictions if p.label == cls]
            gold_boxes = [g for g in golds if g.label == cls]
            class_aps.append(calculate_ap(pred_boxes, gold_boxes, iou_thresh=0.5))
        return sum(class_aps) / len(class_aps) * 100
    elif task in ['captioning', 'VQA']:
        scores = {}
        scores['CIDEr'] = cider_score(predictions, golds) * 100
        scores['S-BERT'] = cosine_similarity(sbert(predictions), sbert(golds)) * 100
        if task == 'VQA':
            scores['S-VQA'] = cosine_similarity(sbert(predictions), sbert(golds)) * 100
        return scores
```

## Common pitfalls

- Models pretrained on Earth datasets (e.g., Kinetics, AVA) degrade significantly due to gravity-dependent priors (orientation, support/contact) rather than architectural limitations.
- Lexical metrics (BLEU-4, CIDEr) drop sharply due to domain-specific vocabulary and paraphrasing, while semantic metrics (S-BERT, S-VQA) remain higher; relying solely on lexical overlap misrepresents model capability.
- Increasing input frame density within a fixed time window does not consistently improve performance; semantic salience extraction is more critical than temporal redundancy in microgravity.

## Evidence (verbatim from paper)

> Our evaluation metrics include mAP@0.5, F1 score, recall, and AUROC, all calculated using the macro method. Among these, mAP@0.5 is the primary metric for measuring average detection accuracy per category and thus comprehensively evaluating the model’s action recognition performance in a microgravity environment.

## Citation

```bibtex
@misc{wen2025microg4m,
  title={Go Beyond Earth: Understanding Human Actions and Scenes in Microgravity Environments},
  author={Wen et al. (2025)},
  year={2025},
  note={arXiv:2506.02845}
}
```

- arXiv: 2506.02845