Ch4 Detection Intensity Eval

---
name: ch4-detection-intensity-eval
description: Evaluates ensemble machine learning models for binary detection of fugitive methane emissions and continuous prediction of their tracer concentration intensity using meteorological data. Use when the user wants to benchmark on HYSPLIT-generated Methane Emission Dataset, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.10879
  bibtex_key: majumder2023ensemble
  confidence: high
---

# ch4-detection-intensity-eval

> Development and Evaluation of Ensemble Learning-based Environmental Methane Detection and Intensity Prediction Models — Majumder et al. (2023) (arXiv:2312.10879, 2023)

## What this evaluates

Evaluates ensemble machine learning models for binary detection of fugitive methane emissions and continuous prediction of their tracer concentration intensity using meteorological data.

## Datasets

- **HYSPLIT-generated Methane Emission Dataset** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Calculated as (TP + TN) / (TP + FP + TN + FN), representing the proportion of correctly classified instances.
- `F1-Score` — range: [0, 1]
  - Calculated as TP / (TP + (FP + FN) / 2), balancing precision and recall for the positive class.
- `MCC` — range: [-1, 1]
  - Mathew's Correlation Coefficient: ((TP * TN) - (FP * FN)) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)).
- `AUC ROC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, integrating True Positive Rate over False Positive Rate.
- `R²` — range: (-inf, 1]
  - Coefficient of determination: 1 - (SSR / SST), where SSR is sum of squared residuals and SST is total sum of squares.
- `RMSE` — range: [0, inf)
  - Root Mean Squared Error: sqrt(1/N * sum((y_true - y_pred)^2)).

## Input / output format

**Input**: 10 meteorological parameters (e.g., wind speed, temperature, humidity, pressure, water vapor, heat flux). Features are standardized for regression models.

**Output**: Binary 'Leakage' label (0/1) for detection task; continuous tracer concentration value for intensity prediction task.

## Scoring recipe

```python
def score_classification(y_true, y_pred):
    tp = sum((y_pred == 1) & (y_true == 1))
    tn = sum((y_pred == 0) & (y_true == 0))
    fp = sum((y_pred == 1) & (y_true == 0))
    fn = sum((y_pred == 0) & (y_true == 1))
    acc = (tp + tn) / (tp + tn + fp + fn)
    f1 = tp / (tp + (fp + fn) / 2)
    return acc, f1

def score_regression(y_true, y_pred):
    ssr = sum((y_true - y_pred) ** 2)
    sst = sum((y_true - mean(y_true)) ** 2)
    r2 = 1 - (ssr / sst)
    rmse = sqrt(mean((y_true - y_pred) ** 2))
    return r2, rmse
```

## Common pitfalls

- Ground truth labels are synthetically generated using the HYSPLIT dispersion model rather than real-world sensor measurements, which may limit field generalizability.
- The paper does not report the exact number of samples or the train/val/test split ratios, making exact replication of dataset size impossible.
- The final ensemble uses weighted averaging of base models, but the specific weights assigned to each base model are not disclosed in the text.

## Evidence (verbatim from paper)

> For performance evaluation of the classifier models, we used four performance metrics, (i) accuracy (ii) F1-Score, (iii) Mathew's Correlation Coefficient (MCC), and (iv) area under the receiver operating characteristics curve (AUC ROC). ... Our final ensemble model achieved an accuracy of 97.2% on the test dataset and the other performance metrics, such as F1 score, Precision, Recall, MCC, and AUC ROC are 0.972, 0.949, 0.997, 0.945, and 0.995.

## Citation

```bibtex
@misc{majumder2023ensemble,
  title={Development and Evaluation of Ensemble Learning-based Environmental Methane Detection and Intensity Prediction Models},
  author={Majumder et al. (2023)},
  year={2023},
  note={arXiv:2312.10879}
}
```

- arXiv: 2312.10879