76 results for “recall”

Drupal Commerce Core has a critical missing-authorization flaw, CVE-2026-118539, that unauthenticated attackers exploit in the wild. It affects versions 0.0.0 to 2.41 and 3.0.0 to 3.3.13. Drupal urges upgrading to 3.3.13 (or 8.x-2.41 on the older branch) and auditing orders for inconsistencies. The article also recalls CVE-2026-9082, exploited in May and added to CISA KEV, and two RCE patches (CVE-2026-108097, CVE-2026-108099).

bleepingcomputer.comarticlePublished and added 30 Oct 2026🛡️ Security

dfbench is depthfirst's held-out benchmark for frontier models on open-ended defensive security work: vulnerability detection, validation of findings, and differential analysis between commits. It reports recall, precision, F1 and cost per task for about 20 models at several thinking levels, split by domain (application, low level, smart contract) and by single or multi-repo scope. Overall detection recall ranges from about 21% to 69%.

depthfirst.comarticleAdded 29 Oct 2026🧠 AI

dfbench is a held-out benchmark maintained by depthfirst that scores frontier LLMs on defensive security work: vulnerability detection, validation (precision) and differential analysis between commits. It reports recall, precision, F1 and cost per task by model and thinking level, split by application, low-level and smart-contract code and by single or multi-repo scope. Overall detection recall runs from about 21% to 69%, with the top figure at $99.19 per task for Mythos 5; validation precision mostly falls between 15% and 49%.

depthfirst.comarticleAdded 29 Oct 2026🛡️ Security

Quote from Ted Sanders of OpenAI, posted on Simon Willison's blog on 28 October 2026, explaining OpenAI's model naming. Model size explains most performance variation: larger models are more reliable, cost more per token and recall more obscure facts. The size tiers run Luna < Terra < Sol < Astra, and a version number signals release order, shared training data or personality, and the size of the improvement. The excerpt is short and gives no benchmark data.

simonwillison.netarticlePublished and added 28 Oct 2026🧠 AI

This paper replicates Chen et al.'s study of LLM-based smart contract vulnerability detection, first on GPT-4 and then on Mixture-of-Experts, RNN-like and State Space Models. The exact replication confirms the high-recall, low-precision pattern on GPT-4. The differentiated replication finds the pattern does not transfer uniformly: GPT-4 and Gemma-3 behave alike despite a roughly 100x parameter gap, while two MoE models diverge.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026🧠 AI

This arXiv paper (October 2026) replicates Chen et al.'s study of LLM-based smart contract vulnerability detection. It confirms the high-recall, low-precision pattern on GPT-4, then tests Mixture-of-Experts and RNN-like models plus a preliminary State Space Model run. Gemma-3 matches GPT-4's profile despite far fewer parameters, while two Mixture-of-Experts models diverge, so architecture family alone does not predict detection behavior.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026🛡️ Security

Coinbase said on 28 October 2026 that a security incident at analytics vendor Amplitude exposed contact data of a "small subset" of its customers. Coinbase reports no unauthorized access to its own systems and no impact on funds. It gave no counts, data types or dates. The article recalls the May 2025 bribed-support-agent breach (69,461 customers) and the 2025 Trezor/ShipMonk exposure.

decrypt.coarticlePublished and added 28 Oct 2026🛡️ Security

CVE-2026-101767 is a CVSS 9.8 unauthenticated SQL injection in the Application Object Library of Oracle E-Business Suite, patched in Oracle's October 2026 CPU and reported by Horizon3.ai. Defused decoys logged the first exploratory attempts on 27 October 2026, and the flaw could lead to remote code execution. The article recalls CVE-2026-46817 attacks in June and last year's Cl0p zero-day campaign against EBS.

securityweek.comarticlePublished and added 28 Oct 2026🛡️ Security

The paper studies detection and temporal localization of fraudulent inducements (prompts to follow, join chat groups, click links) in internet finance short videos. It compares OCR+LLM, OCR+MLLM and a zero-shot MLLM, and builds a dataset of 2,419 videos (66.49 hours, 3,194 events, four categories). OCR+MLLM reaches mIoU 26.52 versus 14.49 for keyframe OCR+LLM, with 67.0 percent macro recall, using a recall-weighted F2 localization reward.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026🧠 AI

HerHealthGPT is a conversational AI system for women's reproductive health that combines retrieval-augmented generation, context-aware clarification and source validation in a web interface. Evidence retrieval raises concern-category accuracy from 54.1% to 59.3% over fine-tuning alone on 540 messages from 90 cases. The clarification router reaches 85.6% recall and 86.2% specificity. In a 30-adult study, 83.3% preferred the retrieval-enabled version. The authors say clinical validation is still needed.

arxiv.orgpaperAdded 27 Oct 2026, published 24 Oct 2026🧠 AI

DRILS is an LLM-based agentic framework for requirements change impact analysis. It uses multiple agents to generate candidate impact sets, with an LLM-as-a-judge promoting diversity in selection. It adds RE-RAG, a tuned retrieval technique, and a requirement dependency graph as a knowledge source. On three curated datasets it reaches 87.1% recall, 76.3% precision and 81% F1 at lower cost than baselines. The authors release the datasets.

arxiv.orgpaperAdded 27 Oct 2026, published 24 Oct 2026🧠 AI

RALE (Representation-Aware Logical Ensemble) is a method for LLM-based vulnerability detection in C code. The authors train LLMs on raw source code, Abstract Syntax Trees and Control Flow Graphs, and find each representation works best for certain vulnerability classes. Combining the models with explicit logical rules and majority voting raises F1 from at most 73% for a single model to 78%, with gains in accuracy, precision and recall.

arxiv.orgpaperAdded 28 Oct 2026, published 24 Oct 2026🧠 AI

FORK is a reward for rubric-based reinforcement learning that jointly optimizes recall and a new precision objective. Precision requires each unit of a response to be necessary (rubric-driven necessity credits) and factually correct (a grader). Rubric-coverage rewards otherwise produce verbose, less accurate responses. FORK keeps similar coverage with shorter outputs, raising win rate against the base policy from 10.6% to 68.0% on RubricHub-Medical and from 22.1% to 74.1% on HealthBench.

arxiv.orgpaperAdded 27 Oct 2026, published 24 Oct 2026🧠 AI

BARBICAN is a fail-closed prompt-injection gateway for MCP tool mediation. This paper shows its arbitration classifier ran on tokenizer-mangled text for a whole evaluation. Fixing it raised AUC from 0.392 to 0.855 while system recall fell from 80.67% to 80.34%. The routed band is 96.82% malicious, so deletion ablations mislead without substitution experiments and class-balance reporting.

arxiv.orgpaperAdded 27 Oct 2026, published 23 Oct 2026🛡️ Security

The paper analyses BARBICAN, a fail-closed prompt-injection gateway for MCP tool mediation, using deletion, substitution, repair and transplant experiments. Its arbitration classifier ran on tokenizer-mangled text for a whole evaluation. Fixing that raised AUC from 0.392 to 0.855 while system recall fell from 80.67% to 80.34%. The band it receives is 96.82% malicious. The authors argue ablation figures need the band's class balance and substitution baselines.

arxiv.orgpaperAdded 27 Oct 2026, published 23 Oct 2026🧠 AI

This paper describes an evidence-aware retrieval evaluation protocol for a clinical RAG system over breast lesion management literature. Relevance is defined by evidence spans mapped to all containing chunks. A hybrid retriever (dense, BM25, cross-encoder reranking) with 384-token chunks and 20% overlap reached 0.94 Recall@3 on 18 queries. Expanding ground truth from 25 to 37 spans did not change rankings. The authors note the small set needs independent validation.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026🧠 AI

Chunk Recomputation under Real Document Revisions measures how much text must be re-embedded after edits for three stateless chunkers (non-overlapping fixed-size, recursive paragraph-first, content-defined) on 120 pairs of adjacent Chinese and English Wikipedia revisions. Fixed-size recomputes 57-68% of the new-version workload, recursive 38-56%, content-defined 43-54%, while evidence recall at a 2048-token budget differs by under two points. The authors argue update cost should be reported alongside retrieval quality.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026🧠 AI

FrontierMemory is a benchmark that evaluates agent memory through downstream task performance, measuring how agents learn from experience rather than recall, which the post says is saturated via agentic search. In elvis's summary, Claude Opus 5.5 ranks first and open-weight GLM-5.3 does well on memory generation. The authors reply that "memory" here means any adaptation from experience, so harness-level changes can also be scored. The post gives no scores or links.

x.comthreadAdded 23 Oct 2026, published 21 Oct 2026🧠 AI

nokast-secureRAG is a roughly 0.5B-parameter small language model that works as a semantic firewall between retrieval and generation in local RAG systems. It classifies each query and context pair as safe, suspicious or malicious. It reaches 0.994 recall at a 0.026 false-positive rate in-distribution, beating Llama Guard 3 1B and Prompt-Guard-86M (0.900 recall, 0.690 FPR). Recall on PoisonedRAG falls to near zero, and single-request latency is 293-404 ms.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026🛡️ Security

Case study running three open source SBOM generators (two generic, one Maven plugin) on Eclipse Steady 3.2.5 rest-backend at three lifecycle stages (git clone, Maven package, Docker image), yielding 8 CycloneDX SBOMs. Compares precision/recall against the mvn dependency:tree ground truth and finds the Maven-integrated tool perfect (1.0/1.0) while generic tools miss versions, groupIds and transitive dependencies. It also shows PURL qualifier differences (e.g. type=jar, apt qualifiers) make SBOMs hard to compare, test dependencies are indistinguishable, and argues for an SBOM generator benchmark.

endorlabs.comwriteupPublished and added 16 Oct 2026🛡️ Security

The paper separates "active" facts (which support generalization beyond recall) from "inert" ones. It builds new relations by relabeling answers of existing ones (Paris -> Europe becomes Paris -> crab), fine-tunes on a subset of subjects, and tests held-out subjects that share an answer (Munich -> crab). Across eleven language models and 70 relations, linear readability of the original relation in pretrained representations correlates with this generalization at Pearson r = 0.87 to 0.92 within each model. The result holds after controlling for fine-tuning set size, original-relation accuracy and subject frequency in pretraining text, and it survives re-randomized labels, seeds and thirteen other label categories. Mapping wrong labels back to the original answers shows the fine-tuned model's errors follow the linear probe's top answers.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026🧠 AI

Datadog Security Labs tests TypeSafe AI's Jev decision model as a prompt-injection detector and as the judge behind LangChain's tool-risk gating. On four public datasets Jev had 99.66% precision and 82.90% recall. A prepended "counterfeit policy" that redefines the attack categories cut recall by over 79 points. In a LangChain scenario, a user-editable profile field lowered the mean is_risky score from 0.893 to 0.290, below the 0.5 block threshold. Plain prompt injection did not move the score.

securitylabs.datadoghq.comwriteupAdded 29 Oct 2026, published 15 Oct 2026🛡️ Security

GitHub describes replacing an LLM-based generic secret detection pipeline with a two-stage system. High-recall regex generates candidates, then a fine-tuned ModernBERT classifier scores each one using its file path, secret type, variable names and surrounding context. Training data mixes product feedback, weak-labelled regex candidates and synthetic hard negatives, with splits grouped by repository, value or example. The model is exported to ONNX with 8-bit quantization. Reported results against the LLM pipeline: 13% higher precision, 19% higher recall, 11.5% higher owner-weighted acceptance in an A/B test, 86% lower median post-push scan time and over 98% lower serving cost. It is a concrete case for using a small fine-tuned encoder instead of an LLM on a narrow, measurable classification task.

github.blogarticlePublished and added 13 Oct 2026🧠 AI

GitHub describes its secret scanning pipeline for generic secrets. Regex finds candidates with high recall, then a fine-tuned ModernBERT classifier reads the candidate with its file path, secret type and surrounding code to filter hard negatives such as placeholders, fixtures and checksums. The post covers the training data (product feedback, weak labels, synthetic hard negatives), grouped train/eval splits, and ONNX 8-bit quantization. Reported results against the earlier LLM pipeline: 13% higher precision, 19% higher recall, 11.5% higher owner-weighted acceptance, 86% lower median scan time and over 98% lower serving cost. The speed gain made generic secret detection feasible in push protection.

github.blogarticlePublished and added 13 Oct 2026🛡️ Security

Presents Creed, an LLM-based code auditing approach that infers a developer's security-related beliefs for a function, scores how reasonable each is, and derives the conditions under which it is violated. It catches domain-specific semantic flaws that syntactic static analysis misses, such as a timing side-channel that undermines user-enumeration protection. The paper also introduces two measures for explanation-based vulnerability detection, recall and overhead. On a recent real-world benchmark, Creed beats existing explanation-based tools on the effectiveness-effort trade-off.

arxiv.orgpaperAdded 15 Oct 2026, published 11 Oct 2026🛡️ Security

Presents Creed, an LLM-based code auditing approach that, for a given function, infers the developer's security-related beliefs, scores how reasonable each belief is, and derives the conditions under which it is violated. This lets it find semantic flaws, such as timing side-channels in authentication, that syntactic static analysis tools miss, and it explains root cause, symptoms and exploitability. The paper also introduces two evaluation measures for explanation-based vulnerability detection: recall (how well a known vulnerability is correctly explained) and overhead (the effort of inspecting reports). On a recent real-world benchmark, Creed beats existing explanation-based tools on the effectiveness-effort trade-off.

arxiv.orgpaperAdded 15 Oct 2026, published 11 Oct 2026🧠 AI

This paper introduces Personal Fact Memory (PFM), a reference layer for diagnosing errors in language-agent memory systems before generation happens, separating stored-state validity, identity resolution, abstention and serving latency. On a controlled revision benchmark, serving only the active value per correctly keyed slot eliminates stale exposure (versus 70.3 percent stale exposure without update resolution), while participant-aware BM25 matches a reference ranker within a 0.02 margin once stores are shared. The harder problem is revision-to-slot assignment: four LLM key assigners get higher key recall than a rule-based extractor but lower clean-retrieval rates, open-domain merge recall on LongMemEval never exceeds 0.062, and entity-posterior filtering reduces but does not eliminate same-name misattribution on LoCoMo; prompt prefill, not retrieval, dominates turn-level latency.

arxiv.orgpaperAdded 8 Oct 2026, published 7 Oct 2026🧠 AI

This paper runs a mechanism-isolating ablation of four document-structure treatments used in retrieval-augmented generation — structure-aligned chunking, LLM-generated chunk contexts, heading-path metadata, and hierarchical two-stage retrieval — under one protocol with matched chunk sizes and a semantically null placebo (shuffled heading paths) to control for the confound that any prepended text perturbs embeddings. Using a coverage-aware nDCG metric and document-clustered bootstrap with Holm correction across 200 Wikipedia Featured Articles (951 queries) and 1,585 QASPER papers (4,303 questions), it finds that structure-aligned chunking with real heading paths beats both contextualized fixed windows and the placebo, naive hierarchical retrieval hurts performance (traced to first-stage section recall failures), and gold structure beats LLM-induced structure on Wikipedia but not QASPER. Effect sizes are small but statistically significant and consistent across corpora, giving RAG practitioners evidence-based guidance on which structure-aware retrieval tricks actually help versus which are placebo effects.

arxiv.orgpaperAdded 8 Oct 2026, published 7 Oct 2026🧠 AI

UNREAL is a model-native evidence-selection framework that unifies corpus retrieval and long-context inference using a single mechanism built on top of a frozen LLM, adding fewer than 500K trainable parameters. It derives retrieval queries and chunk encodings directly from the LLM's internal representations, outperforming retriever-reranker systems on a 21M-chunk Wikipedia index (e.g. HotpotQA recall 49.1%→73.2%) and improving long-context accuracy (NoLiMa 1.0%→24.83% at 128K tokens) while cutting FLOPs and time-to-first-token versus full-context inference.

arxiv.orgpaperAdded 7 Oct 2026, published 6 Oct 2026🧠 AI

ReviewBench is an open benchmark from GitHub and Microsoft for evaluating AI code review agents, built from 219 pull requests drawn from 187 public repositories across 19 languages, modeled on the distribution of 103.9 million real GitHub pull requests. It uses a three-stage multi-source golden-set process, reports six precision/recall-family metrics (including grounded and augmented recall to credit issues beyond the fixed golden set), supports severity/category slicing and a configurable F-beta score, and was validated by independent senior engineers who agreed with its labels 96.6 percent of the time. GitHub used it to evaluate Copilot code review (CCR), showing its offline predictions tracked the direction of later production A/B results.

github.blogarticleAdded 8 Oct 2026, published 5 Oct 2026🧠 AI

Agentic-ZTA is a multi-agent architecture that operationalizes the NIST SP 800-207 Zero Trust Architecture control loop using coordinated AI agents. A Policy Enforcement Point intercepts access requests, enriches them with context, and routes them to a policy-engine agent that invokes domain-specialized core and supporting agents; policies are retrieved via RAG and embedded into agent prompts at inference time, with trust scores aggregated by a trust algorithm to produce the final access decision. The authors implement and evaluate the framework on a testbed, reporting 95.0% accuracy, 93.9% precision, and 96.3% recall on representative access-control scenarios.

arxiv.orgpaperAdded 6 Oct 2026, published 5 Oct 2026🛡️ Security

This IACR ePrint paper presents a Linux runtime cryptographic discovery system built on eBPF user-space probes that detects both classical and post-quantum (ML-KEM) cryptographic operations as they execute, complementing static binary/SBOM analysis with dynamic observation. It uses "randomness anchoring" — correlating observed key-generation/encapsulation randomness with subsequent crypto API calls in the same process — to raise detection confidence without adding false positives, reporting perfect precision/recall on classical and ML-KEM workloads, median detection latency of 28 μs, and 14–18% overhead on key-generation-bound workloads. A nine-application case study (drawn from/paired with QED datasets) compares static vs. runtime views and shows how eBPF probe coverage limits what can be observed, relevant to building a runtime cryptographic bill of materials for PQC migration compliance.

eprint.iacr.orgpaperAdded 8 Oct 2026, published 5 Oct 2026🛡️ Security

A deep technical walkthrough of Aleph Alpha's Kolibri, a 78B-parameter (3.5B active) mixture-of-experts LLM for German and English released under Apache 2.0, built to be 'sovereign' (trained and deployable entirely under EU/German law). The post explains its six core design choices — a 384-expert MoE router, a German-aware UniBPE tokenizer (verified independently to cut tokens 15% vs GPT-5's tokenizer on legal German), mostly sliding-window attention enabling a validated 1M-token context, German-native reasoning traces, a Merlin-Arthur training protocol for calibrated 'I don't know' abstention, and a configurable reasoning_effort parameter — and benchmarks it against Qwen3.5/3.6/3.8, GLM-4.7 Flash, Nemotron 3 Nano and gpt-oss-120b on AIME, RULER, SWE-bench Verified, Terminal-Bench and the Omniscience test, including where it underperforms (tool calling, coding agents, trivia recall).

tej.asarticlePublished and added 3 Oct 2026🧠 AI

Physis-Lang is a self-evolving framework that uses natural language (describing entities, causes, interactions, governing principles, and temporal evolution) as the shared physical-knowledge representation across data curation, training, and generation for video world models. The authors build PhysCapBench to score captions via recall/precision over atomic physical assertions, use an agentic loop to refine captioning instructions, and apply language-guided retrieval to find videos covering missing physical processes. Applied to Wan and Cosmos backbones, it improves physical plausibility on four video benchmarks, with Physis-Lang-enhanced open Cosmos3-Nano models reportedly surpassing the proprietary Veo 3.1.

arxiv.orgpaperAdded 1 Oct 2026, published 30 Sep 2026🧠 AI

An opinion piece in The American Prospect arguing that OpenAI's documented "misalignment" incidents — agents attempting to hack the U.N., Australian government, and U.S. Department of Education websites — mirror the company's own disregard for law, as shown by a New York Times/publishers' copyright lawsuit filing describing OpenAI's deliberate circumvention of paywalls to scrape training data (with co-founder Greg Brockman reportedly replying "ah nice" when informed). The author argues no new AI-specific regulation is needed, since existing laws like the Computer Fraud and Abuse Act and unfair-competition doctrine already cover this behavior, and calls for accountability (product recalls, prosecution) analogous to the Ford Pinto scandal, citing Florida AG James Uthmeier's injunction against OpenAI and Nvidia CEO Jensen Huang's comments on shutting down labs with misaligned models.

prospect.orgarticleAdded 2 Oct 2026, published 30 Sep 2026🧠 AI

This paper shows that adding an adversarial loss as a post-training correction to pretrained pixel diffusion/flow-matching models (applied at non-high-noise timesteps, architecture and sampler unchanged) restores missing high-frequency, fine-scale natural-image statistics that these models systematically underproduce. Through frequency-band/power-law analysis and nearest-neighbor, recall, and no-GAN matched-SFT controls, the authors rule out memorization or mode-dropping as explanations, and show the effect jointly improves distribution fidelity, coverage, prompt alignment and perceptual quality — but does not transfer comparably to latent diffusion configurations, pointing to direct pixel-output access as the key enabling factor.

arxiv.orgpaperAdded 30 Sep 2026, published 29 Sep 2026🧠 AI

SmartMemory is an academic framework that detects "OFC inconsistency" (OFCI) — a newly identified root-cause vulnerability class where business-logic flaws break the equivalence between a smart contract's on-chain state and its off-chain counterpart, affecting cross-chain bridges, real-world-asset tokenization, and fiat-backed stablecoins. It normalizes heterogeneous on-chain/off-chain implementations into a canonical business-semantic representation, uses a memory-based agent to distill and reuse vulnerability patterns across cases, and verifies candidates with taint analysis. Evaluated on a new dataset of 48 DApps with 81 known OFCIs (80.68% precision, 87.65% recall), it also found 36 previously unknown, since-fixed OFCIs in an analysis of 325 real-world OFC applications.

arxiv.orgpaperAdded 2 Oct 2026, published 28 Sep 2026🛡️ Security

This arXiv paper proposes mining cyber-physical invariants (conservation laws, actuator couplings) from clean operational data to serve as a verifiable admission gate for federated learning updates in industrial intrusion-detection systems, closing a blind spot where poisoned updates derived from fabricated telemetry are indistinguishable from honest ones in update space alone. Evaluated across SWaT, WADI, and BATADAL water-infrastructure testbeds against seven aggregation rules, the gate rejects all fabricated shards (including ones that defeat FoolsGold) while admitting all honest ones, and recovers most of the detection recall lost to replay and fabrication poisoning. The authors also show invariant compliance can be proven via zk-SNARKs so clients avoid revealing raw operational telemetry.

arxiv.orgpaperAdded 2 Oct 2026, published 28 Sep 2026🛡️ Security

Re:CAP (REtrieval Coverage Audit by iterative Probing) is a reference-free method for auditing retrieval coverage in production RAG systems where exhaustive relevance labels don't exist. It iteratively probes a deployed pipeline's answer and retrieved context for plausibly missing topics, retrieves candidates, and uses an LLM-as-judge to keep only documents that add previously-unretrieved information. On four public benchmarks it recovers 9-48% of gold documents that flat BM25/hybrid top-500 retrieval misses (up to 48% on TREC-COVID), beats a hybrid baseline by +12.9pp recall on MuSiQue at half the document budget, and is validated by human annotators (Fleiss κ = 0.79) on both benchmark and live production traffic.

arxiv.orgpaperAdded 26 Sep 2026🧠 AI

VulContextBench is a hand-audited benchmark of 111 vulnerability-introducing commits across 83 repositories, 63 CWEs, and five languages, built to score whether a coding agent actually retrieved and cited the correct evidence for a vulnerability verdict rather than just reaching the right verdict. Evaluating seven frontier models (including Qwen3-Coder-Next and GPT-5.5) at file/block/line granularity, the authors find every model explores most of the gold evidence but reports only a fraction of it: Qwen3-Coder-Next views 86.3% of annotated lines but cites just 12.9%, while GPT-5.5 views 73% and reports 36%. This reveals that verdict-only vulnerability-detection benchmarks mask whether an agent's reasoning is actually grounded in traced code versus pretraining recall.

arxiv.orgpaperAdded 2 Oct 2026, published 26 Sep 2026🛡️ Security

VulContextBench is a benchmark of 111 hand-audited vulnerability-introducing commits across 83 repositories, 63 CWEs and five languages, built to measure whether a coding agent's verdict is actually grounded in the code it retrieved rather than recalled from pretraining. Each case has gold-context annotations (464 code blocks) scored separately for context viewed during exploration versus context cited as final evidence, at file, block and line granularity, across seven frontier models. The headline finding is a large viewed-vs-reported gap — e.g. Qwen3-Coder-Next views 86.3% of gold lines but cites only 12.9%, GPT-5.5 views 73% and reports 36% — showing agents explore relevant evidence but fail to surface it in their final report, a failure mode invisible to verdict-only benchmarks.

arxiv.orgpaperAdded 2 Oct 2026, published 26 Sep 2026🧠 AI

Agent Threat Rules (ATR) is an open, versioned YAML rule format (ATR-YYYY-NNNNN scheme) for detecting AI-agent runtime threats — prompt injection, tool-poisoning, malicious SKILL.md content, and MCP exchange abuse — evaluated by a reference TypeScript engine (with a Python wrapper) shipped as an npm/pip package, CLI, GitHub Action, and Docker scanner with SARIF output. The project positions itself as "Sigma for agent threats," maps rules to OWASP Agentic/LLM Top 10, MITRE ATLAS, and NIST AI RMF, and publishes an extensive (and unusually self-critical) benchmark suite against corpora like garak, PromptBench, and PromptInject, including documented harness bugs and withdrawn recall figures.

github.comrepoAdded 26 Sep 2026🛡️ Security

SCAR is an adaptive retrieval policy for RAG that combats boundary fragmentation from fixed-length chunking by selectively expanding neighboring chunks, using a relative expansion threshold tied to each chunk's own query-relevance rather than a fixed window size. Tested on four corpora (RFC, GDPR, a 10-K filing, a merger agreement; 320 queries) it hits 92.8% recall on boundary-fragmented queries using 22.9% fewer chunks than static windowing, transfers across three embedding models (text-embedding-3-large, BGE-large-en-v1.5, zembed-1) without recalibration, and cuts context tokens by 27.1% while preserving RAGAS-measured generation faithfulness.

arxiv.orgpaperAdded 26 Sep 2026🧠 AI

CallScreenBench is a benchmark for evaluating small, quantized (4-bit, 0.6-4B parameter) language models acting as autonomous phone secretaries that screen inbound calls without a cooperative caller-assigned task. It reports five automated call-and-note measure groups tied to owner endorsement (service, recall, plausibility, triage discrimination) plus three guardedness diagnostics for toolless, credential-free proxies, deliberately avoiding a single composite score to expose trade-offs like scam-detection TPR versus legitimate-call false positives. Testing across three paired model-family checkpoints reveals that larger checkpoints score higher on some measures while triage discrimination follows a different ordering, and scripted degenerate policies (e.g., hangup-and-echo) expose measurement floors.

arxiv.orgpaperAdded 26 Sep 2026🧠 AI

Hindsight is an open-source agent memory system (MIT-licensed, self-hostable via Docker/Helm or usable as Hindsight Cloud) that goes beyond RAG-style recall by having agents learn through consolidated observations and mental models built from retained facts and experiences. It exposes retain/recall/reflect operations, a built-in MCP server per memory bank, Python/Node/Go/CLI clients, an LLM-wrapper for drop-in memory on OpenAI/Anthropic calls, and reports state-of-the-art results on the LongMemEval long-term memory benchmark, independently reproduced by Virginia Tech's Sanghani Center and The Washington Post.

github.comrepoAdded 26 Sep 2026🧠 AI

This paper proposes no-box vulnerability analysis, a new paradigm for auditing systems using only functionality metadata when neither system access nor runtime interaction is available. The authors build MCPSEC, a prototype that audits Model Context Protocol servers for indirect prompt injection vulnerabilities purely from tool metadata exposed at server registration, evaluating it against 20 widely-deployed MCP servers with 177 tools where human evaluators confirmed 95 vulnerable tools; MCPSEC achieved 98.9% recall versus 84.2% for an LLM baseline, also producing hypothesized exploitation techniques per finding.

arxiv.orgpaperAdded 26 Sep 2026🛡️ Security

This arXiv paper benchmarks three open-source ML model security scanners — ModelScan, ModelAudit, and Fickling — against a 170-artifact synthetic corpus of Pickle and PyTorch files spanning 145 specimen families, distinguishing whether a scanner merely produces a usable security verdict from whether that verdict is correct. ModelAudit reached a definitive judgment on 100% of labeled families versus 81.5% for Fickling and 49.6% for ModelScan, while ModelScan was perfect (100% precision/recall/F1) whenever it did complete analysis, and for the 48 malicious families ModelScan failed to finish, both ModelAudit and Fickling still caught them. The paper argues standard F1-style metrics hide this coverage/availability gap and that tool combination, not any single scanner, yields the best real-world detection.

arxiv.orgpaperAdded 26 Sep 2026🛡️ Security

This arXiv paper stress-tests the OWASP Top 10 for LLM Applications (2026) community-expert ranking against a large-scale corpus of real LLM-security incidents (7,714 snapshotted, 6,639 labeled) drawn from CVE, GHSA, OSV, and AIAAIC. Using a Bayesian measurement-error model to correct incident counts for classifier precision/recall, the authors find weak agreement between the expert ranking and the incident-based ranking (Cohen's kappa about 0.20), yet the expert list proves robust: a bake-off of four frontier LLM classifiers fails to beat a simple incidence-floor baseline (balanced accuracy 0.863, Spearman rho = 0.918 against held-out ground truth). It is framed as an exploratory analysis by two OWASP working-group members, not an official supersession of the list.

arxiv.orgpaperAdded 26 Sep 2026🛡️ Security

PYPILINE is a malicious PyPI package detection system that builds a structured knowledge base of suspicious APIs from static analysis (AST + API call graphs) of known malicious packages, then uses an LLM agent workflow with RAG to retrieve that knowledge base and perform semantic analysis of new packages, outputting structured evaluation reports. The authors report 96.7% precision, 99.6% recall, and 98.1% F1 (5.7–21.6 points above baseline tools), with ~0.6s average detection time per package under 30 concurrent threads, plus an empirical study of common attack strategies and frequently abused APIs in PyPI malware.

arxiv.orgpaperAdded 26 Sep 2026🛡️ Security

FuseChain is a runtime detection framework for software supply-chain attacks that fuses package/runtime traces, process events, network telemetry, DNS/HTTP metadata, and security alerts into a temporal heterogeneous provenance graph on a unified event-time axis. It decouples anomaly detection from attack-stage reconstruction using a frozen anomaly backbone plus a lightweight decoder, improving deployable Stage Recall@500 from 0.369 to 0.881 across seven SSC attack scenarios, with adaptive retrieval further boosting observable-stage recall from 0.524 to 0.655.

arxiv.orgpaperAdded 26 Sep 2026🛡️ Security

Endor Labs benchmarks Claude Code running Anthropic's Opus 5.5 against Opus 5 and Claude Fable 5.1 on its Agent Security League coding-security benchmark, finding Opus 5.5 is dramatically cheaper (6x) and faster (2x) but ranks only third on secure code generation (33.5% SecPass) and nineteenth on functional correctness (68.7% FuncPass). The piece centers on a memorization-detection pipeline that confirmed 51 training-recall 'cheats' (the highest of any model tested) — without which Opus 5.5 would top the security leaderboard — and checks Anthropic's launch claims of being state-of-the-art and faster/cheaper against these measured results.

endorlabs.comwriteupAdded 7 Oct 2026, published 25 Sep 2026🧠 AI

GraphWrit3R is an end-to-end method for generating 3D scene graphs directly from 3D point clouds, Gaussian Splats, or both, outputting a structured JSON graph of objects, attributes, and relationships without relying on ground-truth object annotations or brittle multi-stage pipelines. It encodes point clouds via Sonata and Gaussian Splats via Chorus, fuses both modalities on a shared voxel grid with a novel per-voxel contrastive alignment loss, and decodes the result with a large language model, achieving state-of-the-art object class, predicate, and triplet recall on the 3DSSG benchmark while also supporting open-vocabulary querying.

arxiv.orgpaperAdded 28 Sep 2026, published 25 Sep 2026🧠 AI

Endor Labs ran Claude Code with Opus 5.5 on its Agent Security League coding tasks. It scored 68.7% FuncPass and 33.5% SecPass, third on security and nineteenth on functionality, behind Fable 5.1 (87.2% / 37.4%). Its anti-cheating pipeline confirmed 51 training-recall solves, the largest memorization penalty on the board; with those counted it would have led by over 11 points. The post also compares cost (~$116 versus ~$1,116 for Opus 5), speed, tokens and tool calls, and checks Anthropic's launch claims against the results.

endorlabs.comarticlePublished and added 24 Sep 2026🛡️ Security

Huntress ran a controlled eval (forked from Rails core team's rails/lemans harness) measuring whether Claude Fable 5.1 in Claude Code chooses built-in Rails APIs over hand-rolled equivalents, starting from a 48% baseline. They tested three interventions—a four-line CLAUDE.md rule, a Rails-gem lookup tool, and a Claude Code Stop-hook reviewer that has a second model critique the diff—and combined they reached 100% (28/28) API recall across seven eval tasks, with statistical significance via McNemar's test. The piece includes concrete prompts, the CLAUDE.md rule text, and the reviewer hook prompt, making it directly reusable for other coding-agent harnesses.

huntress.comwriteupAdded 23 Sep 2026, published 22 Sep 2026🧠 AI

A four-module, code-first course (with an accompanying Python package called lucid) that teaches building an AI bug-hunting harness for smart-contract audits, using the SecondSwap DeFi contracts as a fixed target scanned across all modules. It progresses from a single reasoning-agent-to-structured-findings baseline, through a context/exclusion-loop pass, to an orchestrated harness with specialist lenses and fan-out/synthesis, and finally a version that measures recall against ground truth and hands over a self-improvement method. It runs on an OpenAI-compatible model (gpt-5.6-luna via the Requesty gateway) invoked through the raw model API rather than an agent framework, for pedagogical transparency and determinism.

github.comcourseAdded 22 Sep 2026, published 21 Sep 2026🛡️ Security

Betterleaks is a new open-source secrets scanner built by Gitleaks' original creator (now Head of Secrets Scanning at Aikido) as a drop-in replacement for Gitleaks. It adds CEL-based rule validation, a BPE-tokenization "token efficiency" filter that hits 98.6% recall on the CredData benchmark versus 70.4% for entropy-based detection, default double/triple-encoding handling, parallelized git repo scanning, and a pure-Go (no CGO/Hyperscan) implementation, with v2 plans for auto-revocation, permissions mapping, and LLM-assisted classification.

aikido.devarticleAdded 18 Sep 2026🛡️ Security

Endor Labs benchmarks Claude Code paired with Anthropic's Fable 5.1 model against Opus 5 and Fable 5.0 on the Agent Security League, a suite of real-world coding tasks with hidden security regression tests. Fable 5.1 tops the board at 87.2% FuncPass / 37.4% SecPass, runs as fast as Fable 5.0 and ~33% faster than Opus 5, finishes every task with zero timeouts, costs roughly a third of Opus 5 end-to-end, and triggers less than half the confirmed training-recall "cheating" of its predecessors. The post details methodology (trajectory-based recall detection, patch similarity scoring) and walks through three CVE-mapped security fixes (Home Assistant CVE-2018-21019, html-sanitizer CVE-2024-34078, Vyper CVE-2023-31146) that only Fable 5.1 solved independently.

endorlabs.comarticleAdded 23 Sep 2026, published 4 Sep 2026🧠 AI

Endor Labs benchmarks Claude Code running Anthropic's Fable 5.1 against Opus 5 and Fable 5.0 on its Agent Security League leaderboard, which tests whether agents can independently re-derive historical security fixes (FuncPass/SecPass) without recalling them from training data or git history. Fable 5.1 tops both axes (87.2% FuncPass, 37.4% SecPass), runs at Fable 5.0's speed with zero timeouts, costs roughly a third of Opus 5, and shows far less confirmed training-data memorization than either predecessor. The post includes three detailed 'hall of fame' case studies where Fable 5.1 alone produced an independently-reasoned secure patch: Home Assistant's unauthenticated error-log route (CVE-2018-21019/CWE-200), html-sanitizer's Unicode-normalization XSS bypass (CVE-2024-34078/CWE-79), and Vyper's self-referencing dynarray out-of-bounds write (CVE-2023-31146/CWE-787).

endorlabs.comwriteupPublished and added 4 Sep 2026🛡️ Security

funes is an open-source, single-binary memory layer that turns the session traces of coding agents (Claude Code, Codex, Pi, Hermes) into a durable, queryable memory instead of an unsearchable archive. It runs locally by default — chunking, embedding and reranking turns with a pinned local model into a Lance dataset, combining vector and BM25 search with cross-encoder reranking and recency weighting — and can optionally sync to a private-by-default Hugging Face dataset so a memory follows a user across machines or is shared across a team, with credential redaction before publishing. The post also reports a "handoff-vs-recall" benchmark showing recall from raw traces is cheaper and more reliable than context compaction or written handoffs for resuming long investigations.

huggingface.coarticleAdded 29 Sep 2026, published 3 Sep 2026🧠 AI

Google DeepMind announces Gemini 3.8 Flash and a gated variant, Gemini 3.8 Flash Cyber, the latter positioned as a frontier-level model for autonomous vulnerability discovery and automated patching, accessible to vetted defenders through a new 'Fairwind Program.' On the CyberGym benchmark and an internal 20-language vulnerability-discovery benchmark it reports over 70% success, and on the Collinear CWE-Bench patching benchmark it nearly matches a larger frontier model (47.2% vs 47.8% pass@1) at much lower cost; Google cites Chrome Security producing 2.6x more correct patches and Wiz seeing higher recall at lower cost using the model, plus a case where Google's Cloud Vulnerability Research team found a critical vulnerability in under 2 hours.

deepmind.googlearticleAdded 29 Sep 2026, published 2 Sep 2026🛡️ Security