Bearllm Fault Diagnosis Eval

---
name: bearllm-fault-diagnosis-eval
description: Evaluates a multimodal LLM framework's ability to perform bearing fault diagnosis, anomaly detection, and maintenance recommendation using vibration signals and textual prompts. It probes cross-condition generalization and zero-shot transfer across diverse industrial bearing datasets. Use when the user wants to benchmark on MBHM, JUST, IMS, CWRU, XJTU, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.11281
  bibtex_key: peng2024bearllm
  confidence: high
---

# bearllm-fault-diagnosis-eval

> BearLLM: A Prior Knowledge-Enhanced Bearing Health Management Framework with Unified Vibration Signal Representation — Haotian Peng et al. (2024) (arXiv:2408.11281, 2024)

## What this evaluates

Evaluates a multimodal LLM framework's ability to perform bearing fault diagnosis, anomaly detection, and maintenance recommendation using vibration signals and textual prompts. It probes cross-condition generalization and zero-shot transfer across diverse industrial bearing datasets.

## Datasets

- **MBHM** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/SIA-IDE/BearLLM
- **JUST** — total ?; splits: test (-1)
- **IMS** — total ?; splits: test (-1)
- **CWRU** — total ?; splits: test (-1)
- **XJTU** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly classified fault types or correctly identified tasks relative to the total number of test instances. Calculated as (number of correct predictions / total number of predictions) × 100.

## Input / output format

**Input**: Paired vibration signal segments (time-domain or frequency-aligned) and natural language task prompts (e.g., fault diagnosis, anomaly detection, maintenance recommendation).

**Output**: Natural language text describing the fault type, severity, or maintenance action; or fault class labels for baseline comparison.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Class imbalance in datasets like IMS skews overall accuracy, masking poor performance on minority fault classes.
- Zero-shot generalization tests may suffer from domain shift if the training MBHM dataset lacks representative condition coverage.
- Severity underestimation in zero-shot settings can occur when the model lacks fine-tuning on specific fault severity distributions.

## Evidence (verbatim from paper)

> On the JUST dataset, our method achieves an accuracy of 90.22% without any fine-tuning.

## Citation

```bibtex
@misc{peng2024bearllm,
  title={BearLLM: A Prior Knowledge-Enhanced Bearing Health Management Framework with Unified Vibration Signal Representation},
  author={Haotian Peng et al. (2024)},
  year={2024},
  note={arXiv:2408.11281}
}
```

- arXiv: 2408.11281