Mobclip Downstream Eval

---
name: mobclip-downstream-eval
description: Evaluates geospatial representation models on 11 regression tasks spanning social, economic, and natural domains across multiple spatial scales (point, grid, county, city). It probes the model's ability to capture complex human-centric and environmental patterns using fused multimodal embeddings rather than relying solely on geographic coordinates. Use when the user wants to benchmark on MobCLIP Downstream Benchmark (11 tasks), or asks about evaluating this task. Reports R^2.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.01297
  bibtex_key: wen2025mobclip
  confidence: high
---

# mobclip-downstream-eval

> MobCLIP: Learning General-purpose Geospatial Representation at Scale — Wen et al. (2025) (arXiv:2506.01297, 2025)

## What this evaluates

Evaluates geospatial representation models on 11 regression tasks spanning social, economic, and natural domains across multiple spatial scales (point, grid, county, city). It probes the model's ability to capture complex human-centric and environmental patterns using fused multimodal embeddings rather than relying solely on geographic coordinates.

## Datasets

- **MobCLIP Downstream Benchmark (11 tasks)** — total ?; splits: test (-1); repo https://github.com/ylzhouchris/MobCLIP

## Metrics

- `R^2` **(primary)** — range: other (typically (-∞, 1])
  - Coefficient of determination measuring the proportion of variance in the target variable explained by the model. Calculated as 1 - (SS_res / SS_tot), where SS_res is the sum of squared residuals and SS_tot is the total sum of squares.

## Input / output format

**Input**: Region-level embeddings (128-dimensional) derived from H3 grid cells. For point tasks, the embedding of the containing grid is used. For administrative tasks (county/city), the mean embedding of all constituent grids is computed. Baselines receive 2D coordinates.

**Output**: Continuous scalar prediction for each downstream task (e.g., population density, education years, energy consumption in tonnes of standard coal equivalent).

## Scoring recipe

```python
def compute_r2(y_true, y_pred):
    y_mean = sum(y_true) / len(y_true)
    ss_res = sum((y - yp)**2 for y, yp in zip(y_true, y_pred))
    ss_tot = sum((y - y_mean)**2 for y in y_true)
    return 1.0 - (ss_res / ss_tot)
# Report mean R^2 over 10 independent train/test runs with standard deviation
```

## Common pitfalls

- Downstream models (LightGBM/Ridge) are trained on fixed embeddings without end-to-end fine-tuning, which may not fully leverage the representation quality.
- Two key datasets (ENE, COS) are proprietary and not publicly available, preventing exact reproduction of those specific task results.
- Baseline models are adapted to H3 grid centroids to match the spatial unit, potentially introducing coordinate misalignment not present in their original evaluations.

## Evidence (verbatim from paper)

> Model performances are measured by $R^{2}$, and the standard deviation over ten repeated trainings are marked in brackets below.

## Citation

```bibtex
@misc{wen2025mobclip,
  title={MobCLIP: Learning General-purpose Geospatial Representation at Scale},
  author={Wen et al. (2025)},
  year={2025},
  note={arXiv:2506.01297}
}
```

- arXiv: 2506.01297