Enhancing LLM-Based Vulnerability Detection via Representation-Aware Logical Ensemble

What it claims

RALE combines small fine-tuned code models (CodeT5 220M, CodeBERT 125M, CodeGPT 124M), each trained on one view of a C/C++ function: raw source, AST or CFG. It merges their binary vulnerable/non-vulnerable predictions with explicit logical rules and majority voting. The authors report 78% F1 for the ensemble against at most 73% for any single model. They state that the rules are designed to reduce missed vulnerabilities while staying robust on non-vulnerable code. The "LLMs" here are encoder, encoder-decoder and decoder models of 124M to 220M parameters, not large chat models.

How it works

The data is MegaVul, which has 17,380 vulnerable and 322,168 non-vulnerable functions from 992 open-source repositories, covering 169 CWE types. Joern generates the AST and CFG graphs. Clang generates a second set of ASTs. Clang CFGs are not used because Clang needs a full translation unit, and the functions are isolated fragments.

Graphs are converted to adjacency lists. Each representation has two variants:

• Complete keeps operation names and code snippets.
• Structural strips identifiers and literals and keeps only node types and topology.

Source code has comments and preprocessor directives removed. The setup crosses seven representations with three models, giving 21 experiments.

Each model is fine-tuned separately for classification:

• CodeBERT gets a classification head.
• CodeT5 uses only its encoder.
• CodeGPT classifies from the last non-padding token.

Early stopping ends training as soon as validation loss rises. The ensemble is not trained jointly. It aggregates the independent predictions.

flowchart LR
  F[C function] --> S[Source code]
  F --> A[AST: Joern / Clang]
  F --> C[CFG: Joern only]
  S --> M1[Fine-tuned model]
  A --> M2[Fine-tuned model]
  C --> M3[Fine-tuned model]
  M1 --> R[Logical rules + majority vote]
  M2 --> R
  M3 --> R
  R --> O[Vulnerable / not]

Evidence

The headline is 78% F1 for RALE against 73% for the best single-representation model. The authors also claim up to 78% accuracy and gains in precision and recall. A traditional static analysis tool is included as a baseline, but the excerpt does not name it or give its scores.

Data is split 88% train, 6% validation and 6% test, after balancing to 50/50 vulnerable and non-vulnerable. All training and inference ran on an Intel i7 (12th gen) with a GTX 1660 (6 GB VRAM) and 32 GB RAM. The setup is reproducible on modest hardware.

Limits

The 50/50 undersampling departs from real-world distributions, where the authors cite about 5% vulnerable. Reported accuracy and precision therefore do not reflect deployment at realistic base rates. The authors acknowledge this.

The fetched text cuts off at Table 2. The per-model and per-CWE results are not visible, and neither are the actual logical rules or the comparison with standard voting. Which representation suits which vulnerability class is asserted but not shown in the excerpt.

AST and CFG inputs are filtered by token-length bounds, so very long functions are excluded. The excerpt mentions no code or model release.