Deepecmp Eval

---
name: deepecmp-eval
description: Binary classification of protein sequences to determine if they are extracellular matrix (ECM) proteins. It evaluates the model's ability to handle class imbalance and generalize across different species and feature extraction methods. Use when the user wants to benchmark on benchmark dataset, independent dataset, ECMPride dataset, or asks about evaluating this task. Reports balanced accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.03689
  bibtex_key: ghafoor2021deepecmp
  confidence: high
---

# deepecmp-eval

> DeepECMP: Predicting Extracellular Matrix Proteins using Deep Learning — Ghafoor et al. (2021) (arXiv:2110.03689, 2021)

## What this evaluates

Binary classification of protein sequences to determine if they are extracellular matrix (ECM) proteins. It evaluates the model's ability to handle class imbalance and generalize across different species and feature extraction methods.

## Datasets

- **benchmark dataset** — total ?; splits: test (-1)
- **independent dataset** — total ?; splits: test (-1)
- **ECMPride dataset** — total ?; splits: test (-1)

## Metrics

- `balanced accuracy` **(primary)** — range: percent
  - Calculated as the average of sensitivity (recall for the positive class) and specificity (recall for the negative class): (Sensitivity + Specificity) / 2. Used as the headline metric to rank performance across methods.
- `accuracy` — range: percent
  - Overall proportion of correct predictions: (TP + TN) / (TP + TN + FP + FN).
- `sensitivity` — range: percent
  - True positive rate: TP / (TP + FN).
- `specificity` — range: percent
  - True negative rate: TN / (TN + FP).

## Input / output format

**Input**: Protein sequence string (provided as FASTA files or via UniProtKB API), internally converted to a 100-dimensional vector using ProtVec.

**Output**: Binary classification label (ECM protein vs non-ECM protein). The ensemble outputs predictions from 11 feed-forward neural networks, and the majority vote determines the final label.

## Scoring recipe

```python
def balanced_accuracy(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    specificity = tn / (tn + fp) if (tn + fp) > 0 else 0.0
    return (sensitivity + specificity) / 2 * 100
```

## Common pitfalls

- Using raw accuracy on highly imbalanced datasets will overestimate performance; balanced accuracy is required for fair comparison.
- Performance varies significantly across the benchmark, independent, and ECMPride datasets due to differences in species coverage and feature extraction (e.g., PSSM vs ProtVec).
- The ensemble voting mechanism (majority vote across 11 FNNs) must be implemented exactly as described; averaging probabilities or using a single model yields different results.

## Evidence (verbatim from paper)

> DeepECMP reached $83.6\%$ balanced accuracy on the benchmark dataset (Table 2), $71.0\%$ balanced accuracy on the independent dataset (Table 3) and $77.5\%$ balanced accuracy on the ECMPride dataset (Table 4). By using balanced accuracy as the metric to rank performance, on the benchmark dataset DeepECMP ranked 4/7, on the independent dataset 3/7 and 3/3 on the ECMPride dataset.

## Citation

```bibtex
@misc{ghafoor2021deepecmp,
  title={DeepECMP: Predicting Extracellular Matrix Proteins using Deep Learning},
  author={Ghafoor et al. (2021)},
  year={2021},
  note={arXiv:2110.03689}
}
```

- arXiv: 2110.03689