When LLM Architectures Meet Smart Contract Vulnerability Detection: a Replication Study

What it claims

Iuliano, De Vito and Di Nucci (University of Salerno, October 2026) re-run Chen et al.'s GPT-based smart contract vulnerability detection protocol unchanged on GPT-4 and on Gemma-3, DeepSeek-V2-Lite, Qwen-3.5 and RWKV-7. GPT-4 reproduces Chen et al.'s high recall with low, unstable precision. Gemma-3 shows the same qualitative profile despite roughly 100x fewer parameters. DeepSeek-V2-Lite (MoE) and RWKV-7 (RNN-like) are much more conservative: they have higher specificity and substantially lower recall. Qwen-3.5, also MoE, does not match DeepSeek-V2-Lite. The authors conclude that architectural family alone does not predict the detection profile.

How it works

The protocol comes from Chen et al., who tested gpt-3.5-turbo-0613, gpt-4o-2024-05-13 and gpt-4-0613 through the OpenAI API at temperature 0.7, with other parameters at defaults. The benchmark is the SmartBugs curated dataset. The replication holds the prompt, dataset, evaluation procedure and temperature fixed across models.

Detection uses two prompts in separate sessions:

1. A detector prompt: "You are a vulnerability detector for a smart contract. Here are nine common vulnerabilities. [VULS]. Check the following smart contract for the above vulnerabilities. Think step by step, carefully. The input is [INPUT]." [VULS] lists the names and aliases of nine DASP Top 10 categories.
2. A "semantic analyzer" prompt converts the free-text answer into per-category binary labels, such as "Reentrancy: 1".

Chen et al. split detection from formatting because constraining the output format inside the detection prompt lowered performance on most vulnerability types. GPT-4 generated the prompt itself. It refined drafts iteratively, and the authors checked them by hand on 20 contracts. On 50 sampled contracts, the normalized output was fully consistent with the detector output in 47 cases.

Evidence

The fetched text ends partway through the description of the reference work. Precision, recall and specificity figures, per-category results and model sizes are not in it. The only comparisons available are the qualitative ones from the abstract and introduction.

Limits

• The authors state the design does not isolate architecture as the cause. The models also differ in scale, training data, tokenization, tuning and cutoff.
• The State Space Model evaluation is described as "attempted" and "preliminary".
• Each architectural family has one or two models. The MoE family has two, and they disagree with each other.
• Chen et al.'s GPT snapshots are scheduled for deprecation by OpenAI, which limits how long the baseline can be re-run.
• The prompt was tuned with GPT-4 on GPT-style output. Results for other models therefore reflect a prompt that was never adapted to them.
• The formatting step is itself an LLM call, and it was verified on only 50 contracts.