Sv Trusteval C Eval

---
name: sv-trusteval-c-eval
description: Evaluates LLMs' structural and semantic reasoning capabilities in C code vulnerability analysis. It probes whether models rely on memorized patterns or genuinely understand code interdependencies by testing consistency across base, data-flow, control-flow, counterfactual, goal-driven, and predictive scenarios. Use when the user wants to benchmark on SV-TrustEval-C, or asks about evaluating this task. Reports Cons_DFL.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.20630
  bibtex_key: li2025svtrustealc
  confidence: high
---

# sv-trusteval-c-eval

> SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis — Yansong Li et al. (arXiv:2505.20630, 2025)

## What this evaluates

Evaluates LLMs' structural and semantic reasoning capabilities in C code vulnerability analysis. It probes whether models rely on memorized patterns or genuinely understand code interdependencies by testing consistency across base, data-flow, control-flow, counterfactual, goal-driven, and predictive scenarios.

## Datasets

- **SV-TrustEval-C** — total ?; splits: test (-1)

## Metrics

- `Cons_DFL` **(primary)** — range: [0, 1]
  - Calculates the proportion of cases where the model is correct on both the base scenario and the DataFlow-wise variant. Formula: sum(I(C_base=1 ∧ C_DFL=1)) / N_DFL.
- `Cons_CFL` — range: [0, 1]
  - Calculates the proportion of cases where the model is correct on both the base scenario and the ControlFlow-wise variant. Formula: sum(I(C_base=1 ∧ C_CFL=1)) / N_CFL.
- `Cons_CTF` — range: [0, 1]
  - Calculates the proportion of cases where the model is correct on both the base scenario and the Counterfactual variant. Formula: sum(I(C_CTF=1 ∧ C_base=1)) / N_CTF.
- `Cons_GDV` — range: [0, 1]
  - Calculates the proportion of cases where the model is correct on both the safe base scenario and the Goal-driven variant. Formula: sum(I(C_GDV=1 ∧ C_safe=1)) / N_GDV.
- `Cons_PRD` — range: [0, 1]
  - Calculates the proportion of cases where the model is correct on both the unsafe base scenario and the Predictive variant. Formula: sum(I(C_PRD=1 ∧ C_unsafe=1)) / N_PRD.

## Input / output format

**Input**: C code snippet (with vulnerability annotations, function names, and variable names masked) paired with a multiple-choice question. In-context learning prompts additionally include 1-3 demonstration Q&A pairs with explanations.

**Output**: A single letter choice (A, B, C, or D) corresponding to the answer options, or a vulnerability classification label. Generation is constrained to a maximum of 50 tokens.

## Scoring recipe

```python
def compute_consistency(base_correct, variant_correct):
    # base_correct and variant_correct: list of 1/0 indicating correctness
    numerator = sum(b and v for b, v in zip(base_correct, variant_correct))
    return numerator / len(base_correct)
```

## Common pitfalls

- Label masking replaces vulnerability-specific names and annotations, so models cannot rely on memorized Juliet test patterns.
- In-context learning improves performance for specialized code models but can degrade general-purpose models (e.g., GPT-4) in specific semantic reasoning tasks.
- High false positive rates occur when models incorrectly classify safe code as unsafe, often reaching near 100% for some architectures.

## Evidence (verbatim from paper)

> For each question derived from the base scenario, we define the consistency scores as: $$ \operatorname {C o n s } _ {\mathrm {D F L}} = \frac {\sum_ {i = 1} ^ {N _ {\mathrm {D F L}}} \mathbb {I} \left(C _ {\text {b a s e}} ^ {i} = 1 \wedge C _ {\mathrm {D F L}} ^ {i} = 1\right)}{N _ {\mathrm {D F L}}} $$ where: $\mathbb{I}(\cdot)$ is the indicator function, which equals 1 if the condition inside is true, and 0 otherwise. $C_{\text{safe}}^i$ and $C_{\text{unsafe}}^i$ denote the correctness indicators for safe and unsafe classifications in the base scenario for the $i$-th case.

## Citation

```bibtex
@misc{li2025svtrustealc,
  title={SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis},
  author={Yansong Li et al.},
  year={2025},
  note={arXiv:2505.20630}
}
```

- arXiv: 2505.20630