SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence

What it claims

SecLog-Bench frames post-detection log analysis as window-level attack-stage reasoning. A model receives a multi-source evidence window that is already known to be attack-related. It must output one of four labels and a reasoning chain in which every substantive claim cites indexed evidence units. The labels are PREPARATION, EXECUTION, RESULT and STAGE UNCLOSED (SU). The evaluation set has 216 label-balanced, human-reviewed windows. Eight LLMs were scored on it. The best model reached 44.44% accuracy, 45.07% Macro-F1 and 54.87% Stage Reasoning F1 (SRF1).

How it works

The 29 scenarios include real CVE exploits, Sandworm-style simulations and multi-hop attacks. They run in Docker-built isolated network topologies. Each run collects host audit logs, application logs and network traffic at the same time. Audit logs form a "core audit sequence." Application and network records attach to it through auxiliary indices recorded at runtime: log-writer identity, request or processing-unit boundaries, traffic anchors, and NAT or conntrack aliases. These indices exist because one HTTP request can map to many syscalls, and container NAT breaks naive endpoint matching. The approach follows AutoLabel.

Labels describe what is visible inside the window, not the global kill-chain or ATT&CK position. When cues for several stages appear, the most advanced supported stage wins. EXECUTION requires server-side handling plus a matching attack operation. RESULT also requires observable consequences. SU means the window is part of the attack chain but its evidence supports no stage. SU is not a benign class.

flowchart LR
  E["Window evidence E(W)"] --> C["Label-blind de-id + compression"]
  C --> X["Model view X(W)"]
  X --> M[Evaluated LLM]
  E --> A[AI annotator]
  A --> R[AI review]
  R --> H[Human confirmation]
  H --> Ref["Reference chain (withheld)"]
  M --> P[Post-hoc eval]
  Ref --> P

Reference chains have four parts: scope, facts, cross-source relations and judgment. An AI annotator drafts them, an AI reviewer prioritises risky cases, and a human confirms each one. The model never sees the label or the reference chain.

Evidence

The benchmark uses three groups of metrics. Accuracy and Macro-F1 score the stage prediction. SRP, SCC and SRF1 score reasoning quality against the reference chain. A TT/TF/FT/FF matrix scores whether prediction and reasoning agree. The authors report that correct labels often come with flawed reasoning. They also report that reasoning quality is frequently misaligned with label accuracy, and conclude that label-only scoring overstates capability. All of these results use scaffolded evidence, meaning pre-aligned and compressed windows rather than raw logs.

Limits

• The fetched text stops at Fig. 2. The names of the eight models, per-model and per-label results, and the definitions of SRP and SCC are not shown.
• Reasoning scoring is "AI-assisted post-hoc evaluation." The page does not name the judge model or report its agreement with humans.
• The page gives no code, data download, licence or release statement.
• All logs are simulated in Docker. The task assumes detection has already happened, so it does not test finding attacks in benign traffic.
• The 216-window evaluation set is small.
• The paper is v1, dated 22 October 2026, and submitted to IEEE without peer review.