SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence

What it is

SecLog-Bench (arXiv 2610.31000v1, 22 October 2026, submitted to IEEE) tests whether an LLM can say how far an attack has progressed from multi-source logs, and whether its reasoning holds up. It does not test detection. Every input is already attack-related. The evaluation set has 216 label-balanced, human-reviewed evidence windows.

How it works

Each window is a slice of an attack-behavior chain, not a fixed time slice or log count. The model sees host audit logs, application logs and network traffic, de-identified and compressed. It must output one of four labels plus a reasoning chain in which every claim cites indexed evidence units.

• PREPARATION: request arrival, probing or trigger conditions, before a server-side handler, runtime or process visibly handles the attack input.
• EXECUTION: visible server-side handling plus a corresponding attack-related operation.
• RESULT: observable consequences attributable to the attack.
• STAGE UNCLOSED (SU): the window is part of the attack chain, but the visible evidence cannot support any concrete stage. It is not a benign class.

When several stages are visible, the label is the most advanced one the evidence supports. Labels depend on the evidence inside the window, not on the global kill-chain or ATT&CK position.

Data comes from a Docker-based simulation platform with 29 scenarios: real CVE exploits, Sandworm-style simulations and multi-hop attacks. Cross-source alignment uses auxiliary indices recorded at runtime: log-writer identity, request or processing-unit boundaries, traffic anchors, and NAT or connection aliases. These are needed because time granularity and entity names differ across sources, and container networking or NAT breaks endpoint matching. The indices do not supply stage labels.

Reference reasoning chains come from an AI annotator and an AI reviewer, then human confirmation. The reference chain and label are hidden from the model and used only after inference.

Scoring has three parts:

• Stage prediction: accuracy and Macro-F1.
• Reasoning quality: SRP, SCC and SRF1.
• Prediction–reasoning consistency: the four cells TT, TF, FT and FF.

The paper also describes the post-hoc reasoning evaluation as AI-assisted.

Results

Eight LLMs were tested. The best reached 44.44% accuracy, 45.07% Macro-F1 and 54.87% SRF1. The authors report that correct labels can come with flawed reasoning, and that reasoning quality often does not track label accuracy. They conclude that label accuracy alone overstates capability.

Caveats

• The evaluation set is small (216 windows), so differences between close models may be noise. The page does not report confidence intervals.
• Reasoning scores depend on AI-assisted grading against references that AI drafted first, though humans confirmed them.
• The inputs are scaffolded, de-identified and compressed windows from simulated attacks. The results say little about raw production logs.
• The fetched text ends partway through the method section. Model names, per-model scores, the full scenario list and any code or data release are not visible, so check the paper before relying on them.