When LLM Architectures Meet Smart Contract Vulnerability Detection: a Replication Study

What it is

An October 2026 replication of Chen et al.'s study of LLM-based smart contract vulnerability detection. The paper keeps Chen et al.'s protocol fixed and swaps in models from other architecture families. It asks whether Chen et al.'s high-recall, low-precision finding on GPT models is a property of LLMs in general or of the dense Transformer design.

How it works

The paper reuses Chen et al.'s setup: the smartbugs-curated benchmark, the nine DASP Top 10 vulnerability categories, and a two-step prompt pipeline. A detection prompt ("You are a vulnerability detector for a smart contract... Think step by step, carefully") produces free-form output. A separate "semantic analyzer" session then normalizes that output into 0/1 labels per category, for example "Reentrancy: 1". Chen et al. split the steps because constraining the output format inside the detection prompt hurt performance for most vulnerability types. They checked the normalizer on 50 contracts, and 47 matched the original detection output. Chen et al. ran at temperature 0.7 on gpt-3.5-turbo-0613, gpt-4o-2024-05-13 and gpt-4-0613. The replication holds prompt, dataset, evaluation and temperature fixed.

Models tested:

• GPT-4 as the exact-replication baseline.
• Gemma-3 (dense Transformer).
• DeepSeek-V2-Lite and Qwen-3.5 (Mixture-of-Experts).
• RWKV-7 (RNN-like).
• A State Space Model, described as an attempted preliminary evaluation.

Findings

• The exact replication on GPT-4 confirms Chen et al., including high recall with low precision.
• Gemma-3 shows the same qualitative profile as GPT-4, despite roughly two orders of magnitude fewer parameters.
• DeepSeek-V2-Lite and RWKV-7 are more conservative: higher specificity, substantially lower recall.
• Qwen-3.5 and DeepSeek-V2-Lite are both Mixture-of-Experts but behave differently, so architecture family alone does not predict the detection profile.

What to do

For an engineer choosing a model for contract triage, do not assume a smaller or non-GPT model will have the same false-positive and miss rates as GPT-4. Measure recall and precision on your own contracts per model. A model with GPT-4-like behavior suits a wide first pass that a human or tool then filters. A conservative model like DeepSeek-V2-Lite or RWKV-7 gives fewer false alarms but misses more.

Caveats

• The authors state the design does not isolate architecture as a cause, because the models also differ in scale, training data, tokenization, tuning and cut-off.
• The fetched text is cut off before the methods and results sections. It has no precision or recall figures and no State Space Model outcome, so the size of the differences is not available here.
• The benchmark is smartbugs-curated only, and the prompt was optimized on GPT-4 by Chen et al., which may favor GPT-family models.