Grok 4.7 for Offensive Security: Orchestration Matters | XBOW

What it is

XBOW's evaluation of an early Grok 4.7 candidate against Grok 4.6 across two offensive-security benchmarks: a production exploit-crafting harness (50 benchmarks × 10 repetitions) and a broader agentic pentest across three large targets, three workload sizes, and four experimental systems (two using xAI's Build, two using unrelated harnesses). The summary already carries the headline (68 vs 42 true positives in Build, the vanished stuck-in-planning failure mode, the atomic-step shift). What it omits is the reasoning-effort finding and the full baseline numbers.

What to do

The actionable engineering takeaway for anyone running Grok 4.7:

• Stop paying for xhigh reasoning. XBOW found xhigh was not distinguishable at all from high reasoning — they double-checked the setup. There is a real point of diminishing returns on reasoning effort. Prior Grok evaluations had shown high reasoning to be the most efficient mode; that advantage no longer extends further up the scale.
• Do not drop to low reasoning for large jobs. low was much less effective for token budgets over 1 million.
• Match the orchestrator to the model. If you run Grok 4.7 for offensive/agentic work, use Build-based orchestration. Outside Build, both exploit crafting (fixed iteration budget) and full pentest performance moved slightly down relative to 4.6 — the improvement is entirely conditional on orchestration.

Details the summary omits

• The external (non-Build) baseline was 72 true-positive findings on a typical run. So the story is: Build 4.6 = 42, external = 72, Build 4.7 = 68. Build closed most of the gap to the external harness rather than exceeding it.
• Behavioral cause: 4.7 produces shorter completions and prefers short shell commands over longer Python scripts — small atomic actions that get fast feedback in Build's act/observe/decide loop. Overall vocabulary and response patterns are otherwise similar to 4.6.
• Efficiency: even measured by token rather than by action, 4.7 showed no clear advantage over 4.6 in the exploit harness; it often needed more iterations, each doing less work.

Caveats

Tested on an early 4.7 candidate, not a final release. XBOW explicitly cannot say how 4.7 was trained; the Build affinity is inferred from behavior. They frame 4.7 as still below "Mythos / GPT Astra / GPT-6" tiers — vendor-relative claims not independently benchmarked here.