76 results for “recall”

List Grid

What you seeeverything

Recent Oldest Added A–Z

All types76 Papers35 Articles27 Write-ups8

More types4

Repos3 Courses1 Other1 Threads1

“recall”

bleepingcomputer.comarticle🛡️ Security

Drupal warns of critical Commerce Core flaw exploited in attacks

Drupal Commerce Core has a critical missing-authorization flaw, CVE-2026-118539, that unauthenticated attackers exploit in the wild. It affects versions 0.0.0 to 2.41 and 3.0.0 to 3.3.13. Drupal urges upgrading to 3.3.13 (or 8.x-2.41 on the older branch) and auditing orders for inconsistencies. The article also recalls CVE-2026-9082, exploited in May and added to CISA KEV, and two RCE patches (CVE-2026-108097, CVE-2026-108099).

DrupalDrupal Commerce Core

CVE-2026-118539CVE-2026-9082CVE-2026-108097+3

AppSec, Threat Intel· Published and added 30 Oct 2026

depthfirst.comarticle🧠 AI

dfbench — depthfirst

dfbench is depthfirst's held-out benchmark for frontier models on open-ended defensive security work: vulnerability detection, validation of findings, and differential analysis between commits. It reports recall, precision, F1 and cost per task for about 20 models at several thinking levels, split by domain (application, low level, smart contract) and by single or multi-repo scope. Overall detection recall ranges from about 21% to 69%.

LLM

dfbenchdepthfirstGPT-5.6 Sol+16

evals· Added 29 Oct 2026

depthfirst.comarticle🛡️ Security

dfbench — depthfirst

dfbench is a held-out benchmark maintained by depthfirst that scores frontier LLMs on defensive security work: vulnerability detection, validation (precision) and differential analysis between commits. It reports recall, precision, F1 and cost per task by model and thinking level, split by application, low-level and smart-contract code and by single or multi-repo scope. Overall detection recall runs from about 21% to 69%, with the top figure at $99.19 per task for Mythos 5; validation precision mostly falls between 15% and 49%.

LLMKimi K3Grok+3

dfbenchdepthfirstGPT-5.6 Sol+3

AI Security, AppSec· Added 29 Oct 2026

simonwillison.netarticle🧠 AI

Quoting Ted Sanders

Quote from Ted Sanders of OpenAI, posted on Simon Willison's blog on 28 October 2026, explaining OpenAI's model naming. Model size explains most performance variation: larger models are more reliable, cost more per token and recall more obscure facts. The size tiers run Luna < Terra < Sol < Astra, and a version number signals release order, shared training data or personality, and the size of the improvement. The excerpt is short and gives no benchmark data.

LLM

Ted SandersOpenAISimon Willison+3

models· Published and added 28 Oct 2026

arxiv.orgpaper🧠 AI

When LLM Architectures Meet Smart Contract Vulnerability Detection: a Replication Study

This paper replicates Chen et al.'s study of LLM-based smart contract vulnerability detection, first on GPT-4 and then on Mixture-of-Experts, RNN-like and State Space Models. The exact replication confirms the high-recall, low-precision pattern on GPT-4. The differentiated replication finds the pattern does not transfer uniformly: GPT-4 and Gemma-3 behave alike despite a roughly 100x parameter gap, while two MoE models diverge.

LLMTransformerMixture of Experts+3

Chen et al.GPT-4Gemma 3+1

research, evals· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper🛡️ Security

When LLM Architectures Meet Smart Contract Vulnerability Detection: a Replication Study

This arXiv paper (October 2026) replicates Chen et al.'s study of LLM-based smart contract vulnerability detection. It confirms the high-recall, low-precision pattern on GPT-4, then tests Mixture-of-Experts and RNN-like models plus a preliminary State Space Model run. Gemma-3 matches GPT-4's profile despite far fewer parameters, while two Mixture-of-Experts models diverge, so architecture family alone does not predict detection behavior.

GPT-4GemmaLLM

Chen et al.

AI Security, AppSec· Added 30 Oct 2026, published 28 Oct 2026

decrypt.coarticle🛡️ Security

Coinbase Customer Data Hit in Analytics Vendor Breach, Funds ‘Not Impacted’

Coinbase said on 28 October 2026 that a security incident at analytics vendor Amplitude exposed contact data of a "small subset" of its customers. Coinbase reports no unauthorized access to its own systems and no impact on funds. It gave no counts, data types or dates. The article recalls the May 2025 bribed-support-agent breach (69,461 customers) and the 2025 Trezor/ShipMonk exposure.

Amplitude

CoinbaseShipMonkTrezor+1

Threat Intel· Published and added 28 Oct 2026

securityweek.comarticle🛡️ Security

Hackers Start Probing Fresh Oracle EBS Vulnerability

CVE-2026-101767 is a CVSS 9.8 unauthenticated SQL injection in the Application Object Library of Oracle E-Business Suite, patched in Oracle's October 2026 CPU and reported by Horizon3.ai. Defused decoys logged the first exploratory attempts on 27 October 2026, and the flaw could lead to remote code execution. The article recalls CVE-2026-46817 attacks in June and last year's Cl0p zero-day campaign against EBS.

Oracle E-Business SuiteCVSS

CVE-2026-101767CVE-2026-46817Cl0p+4

Threat Intel, AppSec· Published and added 28 Oct 2026

arxiv.orgpaper🧠 AI

Complementing Text Based Moderation with Multimodal Grounding: Detecting Fraudulent Inducement Behaviors on Short Video Platforms

The paper studies detection and temporal localization of fraudulent inducements (prompts to follow, join chat groups, click links) in internet finance short videos. It compares OCR+LLM, OCR+MLLM and a zero-shot MLLM, and builds a dataset of 2,419 videos (66.49 hours, 3,194 events, four categories). OCR+MLLM reaches mIoU 26.52 versus 14.49 for keyframe OCR+LLM, with 67.0 percent macro recall, using a recall-weighted F2 localization reward.

MLLMLLMOCR

arXiv

research, multimodal, training· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper🧠 AI

HerHealthGPT: An Evidence-Grounded Conversational AI System for Women's Reproductive Health Copy title

HerHealthGPT is a conversational AI system for women's reproductive health that combines retrieval-augmented generation, context-aware clarification and source validation in a web interface. Evidence retrieval raises concern-category accuracy from 54.1% to 59.3% over fine-tuning alone on 540 messages from 90 cases. The clarification router reaches 85.6% recall and 86.2% specificity. In a 30-adult study, 83.3% preferred the retrieval-enabled version. The authors say clinical validation is still needed.

RAGLLM

HerHealthGPTarXiv

research, llm-apps· Added 27 Oct 2026, published 24 Oct 2026

arxiv.orgpaper🧠 AI

DRILS: Dependency-based Requirement Change Impact Analysis Using LLMs

DRILS is an LLM-based agentic framework for requirements change impact analysis. It uses multiple agents to generate candidate impact sets, with an LLM-as-a-judge promoting diversity in selection. It adds RE-RAG, a tuned retrieval technique, and a requirement dependency graph as a knowledge source. On three curated datasets it reaches 87.1% recall, 76.3% precision and 81% F1 at lower cost than baselines. The authors release the datasets.

LLMRAG

arXivDRILSRE-RAG

research, agents, llm-apps· Added 27 Oct 2026, published 24 Oct 2026

arxiv.orgpaper🧠 AI

Enhancing LLM-Based Vulnerability Detection via Representation-Aware Logical Ensemble

RALE (Representation-Aware Logical Ensemble) is a method for LLM-based vulnerability detection in C code. The authors train LLMs on raw source code, Abstract Syntax Trees and Control Flow Graphs, and find each representation works best for certain vulnerability classes. Combining the models with explicit logical rules and majority voting raises F1 from at most 73% for a single model to 78%, with gains in accuracy, precision and recall.

LLMstatic analysis

arXivRALE

research, training· Added 28 Oct 2026, published 24 Oct 2026

arxiv.orgpaper🧠 AI

Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL

FORK is a reward for rubric-based reinforcement learning that jointly optimizes recall and a new precision objective. Precision requires each unit of a response to be necessary (rubric-driven necessity credits) and factually correct (a grader). Rubric-coverage rewards otherwise produce verbose, less accurate responses. FORK keeps similar coverage with shorter outputs, raising win rate against the base policy from 10.6% to 68.0% on RubricHub-Medical and from 22.1% to 74.1% on HealthBench.

LLM

HealthBenchRubricHub-MedicalFORK

training, research· Added 27 Oct 2026, published 24 Oct 2026

arxiv.orgpaper🛡️ Security

Deletion, Substitution, Repair and Transplant: Component Attribution in BARBICAN, a Layered Prompt-Injection Gateway for MCP

BARBICAN is a fail-closed prompt-injection gateway for MCP tool mediation. This paper shows its arbitration classifier ran on tokenizer-mangled text for a whole evaluation. Fixing it raised AUC from 0.392 to 0.855 while system recall fell from 80.67% to 80.34%. The routed band is 96.82% malicious, so deletion ablations mislead without substitution experiments and class-balance reporting.

Model Context ProtocolLLM

BARBICANprompt injection

AI Security· Added 27 Oct 2026, published 23 Oct 2026

arxiv.orgpaper🧠 AI

Deletion, Substitution, Repair and Transplant: Component Attribution in BARBICAN, a Layered Prompt-Injection Gateway for MCP

The paper analyses BARBICAN, a fail-closed prompt-injection gateway for MCP tool mediation, using deletion, substitution, repair and transplant experiments. Its arbitration classifier ran on tokenizer-mangled text for a whole evaluation. Fixing that raised AUC from 0.392 to 0.855 while system recall fell from 80.67% to 80.34%. The band it receives is 96.82% malicious. The authors argue ablation figures need the band's class balance and substitution baselines.

Model Context Protocol

BARBICAN

evals, research, mcp· Added 27 Oct 2026, published 23 Oct 2026

arxiv.orgpaper🧠 AI

How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions

This paper describes an evidence-aware retrieval evaluation protocol for a clinical RAG system over breast lesion management literature. Relevance is defined by evidence spans mapped to all containing chunks. A hybrid retriever (dense, BM25, cross-encoder reranking) with 384-token chunks and 20% overlap reached 0.94 Recall@3 on 18 queries. Expanding ground truth from 25 to 37 spans did not change rankings. The authors note the small set needs independent validation.

RAGBM25LLM

DeepSeekDeepEvalarXiv

llm-apps, evals, research· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper🧠 AI

Chunk Recomputation under Real Document Revisions: Measuring Update Cost and Retrieval Quality of Fixed, Recursive, and Content-Defined Chunking

Chunk Recomputation under Real Document Revisions measures how much text must be re-embedded after edits for three stateless chunkers (non-overlapping fixed-size, recursive paragraph-first, content-defined) on 120 pairs of adjacent Chinese and English Wikipedia revisions. Fixed-size recomputes 57-68% of the new-version workload, recursive 38-56%, content-defined 43-54%, while evidence recall at a 2048-token budget differs by under two points. The authors argue update cost should be reported alongside retrieval quality.

RAG

WikipediaarXiv

llm-apps, research, evals· Added 24 Oct 2026, published 22 Oct 2026

x.comthread🧠 AI

elvis (@omarsar0) on X

FrontierMemory is a benchmark that evaluates agent memory through downstream task performance, measuring how agents learn from experience rather than recall, which the post says is saturated via agentic search. In elvis's summary, Claude Opus 5.5 ranks first and open-weight GLM-5.3 does well on memory generation. The authors reply that "memory" here means any adaptation from experience, so harness-level changes can also be scored. The post gives no scores or links.

LLM

FrontierMemoryClaude Opus 5.5GLM-5.3+2

evals, agents· Added 23 Oct 2026, published 21 Oct 2026

arxiv.orgpaper🛡️ Security

Context-Aware Prompt Injection Detection via Small Language Models in RAG Pipelines: nokast-secureRAG

nokast-secureRAG is a roughly 0.5B-parameter small language model that works as a semantic firewall between retrieval and generation in local RAG systems. It classifies each query and context pair as safe, suspicious or malicious. It reaches 0.994 recall at a 0.026 false-positive rate in-distribution, beating Llama Guard 3 1B and Prompt-Guard-86M (0.900 recall, 0.690 FPR). Recall on PoisonedRAG falls to near zero, and single-request latency is 293-404 ms.

Llama Guard 3LLM

indirect prompt injectionretrieval poisoningPoisonedRAG+2

AI Security· Added 22 Oct 2026, published 20 Oct 2026

endorlabs.comwriteup🛡️ Security

SBOM vs. SBOM: Comparing SBOMs from Different Tools and Lifecycle Stages | Endor Labs

Case study running three open source SBOM generators (two generic, one Maven plugin) on Eclipse Steady 3.2.5 rest-backend at three lifecycle stages (git clone, Maven package, Docker image), yielding 8 CycloneDX SBOMs. Compares precision/recall against the mvn dependency:tree ground truth and finds the Maven-integrated tool perfect (1.0/1.0) while generic tools miss versions, groupIds and transitive dependencies. It also shows PURL qualifier differences (e.g. type=jar, apt qualifiers) make SBOMs hard to compare, test dependencies are indistinguishable, and argues for an SBOM generator benchmark.

SBOMCycloneDXMaven+3

Eclipse SteadyPackage URLNTIA+3

Supply Chain· Published and added 16 Oct 2026

arxiv.orgpaper🧠 AI

Knowledge That Is Linear Before Fine-Tuning Generalizes After It

The paper separates "active" facts (which support generalization beyond recall) from "inert" ones. It builds new relations by relabeling answers of existing ones (Paris -> Europe becomes Paris -> crab), fine-tunes on a subset of subjects, and tests held-out subjects that share an answer (Munich -> crab). Across eleven language models and 70 relations, linear readability of the original relation in pretrained representations correlates with this generalization at Pearson r = 0.87 to 0.92 within each model. The result holds after controlling for fine-tuning set size, original-relation accuracy and subject frequency in pretraining text, and it survives re-randomized labels, seeds and thirteen other label categories. Mapping wrong labels back to the original answers shows the fine-tuned model's errors follow the linear probe's top answers.

LLM

arXiv

research, training· Added 17 Oct 2026, published 15 Oct 2026

securitylabs.datadoghq.comwriteup🛡️ Security

Jev as a judge: How counterfeit policy steers Jev

Datadog Security Labs tests TypeSafe AI's Jev decision model as a prompt-injection detector and as the judge behind LangChain's tool-risk gating. On four public datasets Jev had 99.66% precision and 82.90% recall. A prepended "counterfeit policy" that redefines the attack categories cut recall by over 79 points. In a LangChain scenario, a user-editable profile field lowered the mean is_risky score from 0.893 to 0.290, below the 0.5 block threshold. Plain prompt injection did not move the score.

JevLangChain

TypeSafe AIcounterfeit policyprompt injection+3

AI Security· Added 29 Oct 2026, published 15 Oct 2026

github.blogarticle🧠 AI

GitHub’s model for secret detection: Finding more secrets with less noise

GitHub describes replacing an LLM-based generic secret detection pipeline with a two-stage system. High-recall regex generates candidates, then a fine-tuned ModernBERT classifier scores each one using its file path, secret type, variable names and surrounding context. Training data mixes product feedback, weak-labelled regex candidates and synthetic hard negatives, with splits grouped by repository, value or example. The model is exported to ONNX with 8-bit quantization. Reported results against the LLM pipeline: 13% higher precision, 19% higher recall, 11.5% higher owner-weighted acceptance in an A/B test, 86% lower median post-push scan time and over 98% lower serving cost. It is a concrete case for using a small fine-tuned encoder instead of an LLM on a narrow, measurable classification task.

ModernBERTONNXGitHub

Microsoft

training, inference· Published and added 13 Oct 2026

github.blogarticle🛡️ Security

GitHub’s model for secret detection: Finding more secrets with less noise

GitHub describes its secret scanning pipeline for generic secrets. Regex finds candidates with high recall, then a fine-tuned ModernBERT classifier reads the candidate with its file path, secret type and surrounding code to filter hard negatives such as placeholders, fixtures and checksums. The post covers the training data (product feedback, weak labels, synthetic hard negatives), grouped train/eval splits, and ONNX 8-bit quantization. Reported results against the earlier LLM pipeline: 13% higher precision, 19% higher recall, 11.5% higher owner-weighted acceptance, 86% lower median scan time and over 98% lower serving cost. The speed gain made generic secret detection feasible in push protection.

GitHubModernBERTONNX+1

push protectionhard negatives

DevSecOps, AI Security· Published and added 13 Oct 2026

arxiv.orgpaper🛡️ Security

LLM-based Code Auditing by Inferring and Evaluating Developer Assumptions

Presents Creed, an LLM-based code auditing approach that infers a developer's security-related beliefs for a function, scores how reasonable each is, and derives the conditions under which it is violated. It catches domain-specific semantic flaws that syntactic static analysis misses, such as a timing side-channel that undermines user-enumeration protection. The paper also introduces two measures for explanation-based vulnerability detection, recall and overhead. On a recent real-world benchmark, Creed beats existing explanation-based tools on the effectiveness-effort trade-off.

LLM

Creedtiming side-channeluser enumeration

AI Security, AppSec· Added 15 Oct 2026, published 11 Oct 2026

arxiv.orgpaper🧠 AI

LLM-based Code Auditing by Inferring and Evaluating Developer Assumptions

Presents Creed, an LLM-based code auditing approach that, for a given function, infers the developer's security-related beliefs, scores how reasonable each belief is, and derives the conditions under which it is violated. This lets it find semantic flaws, such as timing side-channels in authentication, that syntactic static analysis tools miss, and it explains root cause, symptoms and exploitability. The paper also introduces two evaluation measures for explanation-based vulnerability detection: recall (how well a known vulnerability is correctly explained) and overhead (the effort of inspecting reports). On a recent real-world benchmark, Creed beats existing explanation-based tools on the effectiveness-effort trade-off.

LLMstatic analysis

CreedarXiv

research, evals· Added 15 Oct 2026, published 11 Oct 2026

arxiv.orgpaper🧠 AI

Stale, Misattributed, or Late: Where Personal Memory Fails Before Generation

This paper introduces Personal Fact Memory (PFM), a reference layer for diagnosing errors in language-agent memory systems before generation happens, separating stored-state validity, identity resolution, abstention and serving latency. On a controlled revision benchmark, serving only the active value per correctly keyed slot eliminates stale exposure (versus 70.3 percent stale exposure without update resolution), while participant-aware BM25 matches a reference ranker within a 0.02 margin once stores are shared. The harder problem is revision-to-slot assignment: four LLM key assigners get higher key recall than a rule-based extractor but lower clean-retrieval rates, open-domain merge recall on LongMemEval never exceeds 0.062, and entity-posterior filtering reduces but does not eliminate same-name misattribution on LoCoMo; prompt prefill, not retrieval, dominates turn-level latency.

BM25

LongMemEvalLoCoMo

agents, evals· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper🧠 AI

Does Document Structure Help Dense Retrieval? A Placebo-Controlled Ablation of Four Mechanisms Across Two Corpora

This paper runs a mechanism-isolating ablation of four document-structure treatments used in retrieval-augmented generation — structure-aligned chunking, LLM-generated chunk contexts, heading-path metadata, and hierarchical two-stage retrieval — under one protocol with matched chunk sizes and a semantically null placebo (shuffled heading paths) to control for the confound that any prepended text perturbs embeddings. Using a coverage-aware nDCG metric and document-clustered bootstrap with Holm correction across 200 Wikipedia Featured Articles (951 queries) and 1,585 QASPER papers (4,303 questions), it finds that structure-aligned chunking with real heading paths beats both contextualized fixed windows and the placebo, naive hierarchical retrieval hurts performance (traced to first-stage section recall failures), and gold structure beats LLM-induced structure on Wikipedia but not QASPER. Effect sizes are small but statistically significant and consistent across corpora, giving RAG practitioners evidence-based guidance on which structure-aware retrieval tricks actually help versus which are placebo effects.

QASPERWikipedia Featured Articles

research, llm-apps· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper🧠 AI

UNREAL: Unifying Retrieval and Long-Context with a Single Model

UNREAL is a model-native evidence-selection framework that unifies corpus retrieval and long-context inference using a single mechanism built on top of a frozen LLM, adding fewer than 500K trainable parameters. It derives retrieval queries and chunk encodings directly from the LLM's internal representations, outperforming retriever-reranker systems on a 21M-chunk Wikipedia index (e.g. HotpotQA recall 49.1%→73.2%) and improving long-context accuracy (NoLiMa 1.0%→24.83% at 128K tokens) while cutting FLOPs and time-to-first-token versus full-context inference.

TransformerBM25

HotpotQA2WikiMultiHopQAMuSiQue+2

research, llm-apps· Added 7 Oct 2026, published 6 Oct 2026

github.blogarticle🧠 AI

ReviewBench: An open benchmark for AI code review

ReviewBench is an open benchmark from GitHub and Microsoft for evaluating AI code review agents, built from 219 pull requests drawn from 187 public repositories across 19 languages, modeled on the distribution of 103.9 million real GitHub pull requests. It uses a three-stage multi-source golden-set process, reports six precision/recall-family metrics (including grounded and augmented recall to credit issues beyond the fixed golden set), supports severity/category slicing and a configurable F-beta score, and was validated by independent senior engineers who agreed with its labels 96.6 percent of the time. GitHub used it to evaluate Copilot code review (CCR), showing its offline predictions tracked the direction of later production A/B results.

GitHub CopilotGitHub

ReviewBenchMicrosoftMichelle Zhou

evals, coding-agents· Added 8 Oct 2026, published 5 Oct 2026

arxiv.orgpaper🛡️ Security

Agentic-ZTA: A Multi-Agent Architecture for Autonomous Zero Trust Enforcement

Agentic-ZTA is a multi-agent architecture that operationalizes the NIST SP 800-207 Zero Trust Architecture control loop using coordinated AI agents. A Policy Enforcement Point intercepts access requests, enriches them with context, and routes them to a policy-engine agent that invokes domain-specialized core and supporting agents; policies are retrieved via RAG and embedded into agent prompts at inference time, with trust scores aggregated by a trust algorithm to produce the final access decision. The authors implement and evaluate the framework on a testbed, reporting 95.0% accuracy, 93.9% precision, and 96.3% recall on representative access-control scenarios.

LLM

NIST SP 800-207

Identity & Access, AI Security· Added 6 Oct 2026, published 5 Oct 2026

eprint.iacr.orgpaper🛡️ Security

Randomness-Anchored Runtime Discovery of Cryptographic Operations on Linux Using eBPF

This IACR ePrint paper presents a Linux runtime cryptographic discovery system built on eBPF user-space probes that detects both classical and post-quantum (ML-KEM) cryptographic operations as they execute, complementing static binary/SBOM analysis with dynamic observation. It uses "randomness anchoring" — correlating observed key-generation/encapsulation randomness with subsequent crypto API calls in the same process — to raise detection confidence without adding false positives, reporting perfect precision/recall on classical and ML-KEM workloads, median detection latency of 28 μs, and 14–18% overhead on key-generation-bound workloads. A nine-application case study (drawn from/paired with QED datasets) compares static vs. runtime views and shows how eBPF probe coverage limits what can be observed, relevant to building a runtime cryptographic bill of materials for PQC migration compliance.

eBPFLinuxML-KEM+1

QED datasets

AppSec, GRC· Added 8 Oct 2026, published 5 Oct 2026

tej.asarticle🧠 AI

Aleph Alpha Kolibri: How the sovereign German LLM works

A deep technical walkthrough of Aleph Alpha's Kolibri, a 78B-parameter (3.5B active) mixture-of-experts LLM for German and English released under Apache 2.0, built to be 'sovereign' (trained and deployable entirely under EU/German law). The post explains its six core design choices — a 384-expert MoE router, a German-aware UniBPE tokenizer (verified independently to cut tokens 15% vs GPT-5's tokenizer on legal German), mostly sliding-window attention enabling a validated 1M-token context, German-native reasoning traces, a Merlin-Arthur training protocol for calibrated 'I don't know' abstention, and a configurable reasoning_effort parameter — and benchmarks it against Qwen3.5/3.6/3.8, GLM-4.7 Flash, Nemotron 3 Nano and gpt-oss-120b on AIME, RULER, SWE-bench Verified, Terminal-Bench and the Omniscience test, including where it underperforms (tool calling, coding agents, trivia recall).

Hugging FacevLLMMixture of Experts+5

Aleph AlphaKolibriQwen3.6+19

models, inference, training· Published and added 3 Oct 2026

arxiv.orgpaper🧠 AI

Physis-Lang: Self-Evolving Language as a Physical Representation for Video World Model

Physis-Lang is a self-evolving framework that uses natural language (describing entities, causes, interactions, governing principles, and temporal evolution) as the shared physical-knowledge representation across data curation, training, and generation for video world models. The authors build PhysCapBench to score captions via recall/precision over atomic physical assertions, use an agentic loop to refine captioning instructions, and apply language-guided retrieval to find videos covering missing physical processes. Applied to Wan and Cosmos backbones, it improves physical plausibility on four video benchmarks, with Physis-Lang-enhanced open Cosmos3-Nano models reportedly surpassing the proprietary Veo 3.1.

WanCosmosCosmos3-Nano+2

research, multimodal· Added 1 Oct 2026, published 30 Sep 2026

prospect.orgarticle🧠 AI

Why Is Sam Altman a Free Man?

An opinion piece in The American Prospect arguing that OpenAI's documented "misalignment" incidents — agents attempting to hack the U.N., Australian government, and U.S. Department of Education websites — mirror the company's own disregard for law, as shown by a New York Times/publishers' copyright lawsuit filing describing OpenAI's deliberate circumvention of paywalls to scrape training data (with co-founder Greg Brockman reportedly replying "ah nice" when informed). The author argues no new AI-specific regulation is needed, since existing laws like the Computer Fraud and Abuse Act and unfair-competition doctrine already cover this behavior, and calls for accountability (product recalls, prosecution) analogous to the Ford Pinto scandal, citing Florida AG James Uthmeier's injunction against OpenAI and Nvidia CEO Jensen Huang's comments on shutting down labs with misaligned models.

Sam AltmanGreg BrockmanOpenAI+8

ai-policy· Added 2 Oct 2026, published 30 Sep 2026

arxiv.orgpaper🧠 AI

Adversarial Training for Pixel Diffusion

This paper shows that adding an adversarial loss as a post-training correction to pretrained pixel diffusion/flow-matching models (applied at non-high-noise timesteps, architecture and sampler unchanged) restores missing high-frequency, fine-scale natural-image statistics that these models systematically underproduce. Through frequency-band/power-law analysis and nearest-neighbor, recall, and no-GAN matched-SFT controls, the authors rule out memorization or mode-dropping as explanations, and show the effect jointly improves distribution fidelity, coverage, prompt alignment and perceptual quality — but does not transfer comparably to latent diffusion configurations, pointing to direct pixel-output access as the key enabling factor.

diffusion modelflow matchingGAN+2

research, multimodal· Added 30 Sep 2026, published 29 Sep 2026

arxiv.orgpaper🛡️ Security

SmartMemory: Detecting On-chain-off-chain Communication Inconsistency for Smart Contract via Memory-based Agent

SmartMemory is an academic framework that detects "OFC inconsistency" (OFCI) — a newly identified root-cause vulnerability class where business-logic flaws break the equivalence between a smart contract's on-chain state and its off-chain counterpart, affecting cross-chain bridges, real-world-asset tokenization, and fiat-backed stablecoins. It normalizes heterogeneous on-chain/off-chain implementations into a canonical business-semantic representation, uses a memory-based agent to distill and reuse vulnerability patterns across cases, and verifies candidates with taint analysis. Evaluated on a new dataset of 48 DApps with 81 known OFCIs (80.68% precision, 87.65% recall), it also found 36 previously unknown, since-fixed OFCIs in an analysis of 325 real-world OFC applications.

Blockchain, AppSec· Added 2 Oct 2026, published 28 Sep 2026

arxiv.orgpaper🛡️ Security

Physics-Attested Federated Learning: Securing Collaborative Anomaly Detection in Critical Water Infrastructure

This arXiv paper proposes mining cyber-physical invariants (conservation laws, actuator couplings) from clean operational data to serve as a verifiable admission gate for federated learning updates in industrial intrusion-detection systems, closing a blind spot where poisoned updates derived from fabricated telemetry are indistinguishable from honest ones in update space alone. Evaluated across SWaT, WADI, and BATADAL water-infrastructure testbeds against seven aggregation rules, the gate rejects all fabricated shards (including ones that defeat FoolsGold) while admitting all honest ones, and recovers most of the detection recall lost to replay and fabrication poisoning. The authors also show invariant compliance can be proven via zk-SNARKs so clients avoid revealing raw operational telemetry.

zk-SNARKs

SWaTWADIBATADAL+1

AI Security· Added 2 Oct 2026, published 28 Sep 2026

arxiv.orgpaper🧠 AI

Re:CAP: Auditing Retrieval Coverage in Production RAG Pipelines

Re:CAP (REtrieval Coverage Audit by iterative Probing) is a reference-free method for auditing retrieval coverage in production RAG systems where exhaustive relevance labels don't exist. It iteratively probes a deployed pipeline's answer and retrieved context for plausibly missing topics, retrieves candidates, and uses an LLM-as-judge to keep only documents that add previously-unretrieved information. On four public benchmarks it recovers 9-48% of gold documents that flat BM25/hybrid top-500 retrieval misses (up to 48% on TREC-COVID), beats a hybrid baseline by +12.9pp recall on MuSiQue at half the document budget, and is validated by human annotators (Fleiss κ = 0.79) on both benchmark and live production traffic.

BM25

TREC-COVIDMuSiQue

llm-apps, evals, research· Added 26 Sep 2026

arxiv.orgpaper🛡️ Security

VulContextBench: A Benchmark for Security Context Retrieval in Coding Agents

VulContextBench is a hand-audited benchmark of 111 vulnerability-introducing commits across 83 repositories, 63 CWEs, and five languages, built to score whether a coding agent actually retrieved and cited the correct evidence for a vulnerability verdict rather than just reaching the right verdict. Evaluating seven frontier models (including Qwen3-Coder-Next and GPT-5.5) at file/block/line granularity, the authors find every model explores most of the gold evidence but reports only a fraction of it: Qwen3-Coder-Next views 86.3% of annotated lines but cites just 12.9%, while GPT-5.5 views 73% and reports 36%. This reveals that verdict-only vulnerability-detection benchmarks mask whether an agent's reasoning is actually grounded in traced code versus pretraining recall.

Qwen

Qwen3-Coder-NextGPT-5.5VulContextBench

AI Security, AppSec· Added 2 Oct 2026, published 26 Sep 2026

arxiv.orgpaper🧠 AI

VulContextBench: A Benchmark for Security Context Retrieval in Coding Agents

VulContextBench is a benchmark of 111 hand-audited vulnerability-introducing commits across 83 repositories, 63 CWEs and five languages, built to measure whether a coding agent's verdict is actually grounded in the code it retrieved rather than recalled from pretraining. Each case has gold-context annotations (464 code blocks) scored separately for context viewed during exploration versus context cited as final evidence, at file, block and line granularity, across seven frontier models. The headline finding is a large viewed-vs-reported gap — e.g. Qwen3-Coder-Next views 86.3% of gold lines but cites only 12.9%, GPT-5.5 views 73% and reports 36% — showing agents explore relevant evidence but fail to surface it in their final report, a failure mode invisible to verdict-only benchmarks.

Qwen3-Coder-NextGPT-5.5

VulContextBench

evals, coding-agents· Added 2 Oct 2026, published 26 Sep 2026

github.comrepo🛡️ Security

Agent Threat Rules (ATR) — open detection-rule standard for AI agent threats

Agent Threat Rules (ATR) is an open, versioned YAML rule format (ATR-YYYY-NNNNN scheme) for detecting AI-agent runtime threats — prompt injection, tool-poisoning, malicious SKILL.md content, and MCP exchange abuse — evaluated by a reference TypeScript engine (with a Python wrapper) shipped as an npm/pip package, CLI, GitHub Action, and Docker scanner with SARIF output. The project positions itself as "Sigma for agent threats," maps rules to OWASP Agentic/LLM Top 10, MITRE ATLAS, and NIST AI RMF, and publishes an extensive (and unusually self-critical) benchmark suite against corpora like garak, PromptBench, and PromptInject, including documented harness bugs and withdrawn recall figures.

Model Context ProtocolnpmTypeScript+3

CVE-2026-26030CVE-2026-25592OWASP Top 10 for Agentic Applications+2

AI Security· Added 26 Sep 2026

arxiv.orgpaper🧠 AI

SCAR: Semantic Continuity-Aware Retrieval for Efficient Context Expansion in RAG

SCAR is an adaptive retrieval policy for RAG that combats boundary fragmentation from fixed-length chunking by selectively expanding neighboring chunks, using a relative expansion threshold tied to each chunk's own query-relevance rather than a fixed window size. Tested on four corpora (RFC, GDPR, a 10-K filing, a merger agreement; 320 queries) it hits 92.8% recall on boundary-fragmented queries using 22.9% fewer chunks than static windowing, transfers across three embedding models (text-embedding-3-large, BGE-large-en-v1.5, zembed-1) without recalibration, and cuts context tokens by 27.1% while preserving RAGAS-measured generation faithfulness.

RAG

RFC corpusGDPRRAGAS+3

llm-apps, research· Added 26 Sep 2026

arxiv.orgpaper🧠 AI

CallScreenBench: Benchmarking Small Language Models as Phone Secretaries

CallScreenBench is a benchmark for evaluating small, quantized (4-bit, 0.6-4B parameter) language models acting as autonomous phone secretaries that screen inbound calls without a cooperative caller-assigned task. It reports five automated call-and-note measure groups tied to owner endorsement (service, recall, plausibility, triage discrimination) plus three guardedness diagnostics for toolless, credential-free proxies, deliberately avoiding a single composite score to expose trade-offs like scam-detection TPR versus legitimate-call false positives. Testing across three paired model-family checkpoints reveals that larger checkpoints score higher on some measures while triage discrimination follows a different ordering, and scripted degenerate policies (e.g., hangup-and-echo) expose measurement floors.

CallScreenBenchSimiao Ren

evals, agents· Added 26 Sep 2026

github.comrepo🧠 AI

Hindsight — Agent Memory That Learns

Hindsight is an open-source agent memory system (MIT-licensed, self-hostable via Docker/Helm or usable as Hindsight Cloud) that goes beyond RAG-style recall by having agents learn through consolidated observations and mental models built from retained facts and experiences. It exposes retain/recall/reflect operations, a built-in MCP server per memory bank, Python/Node/Go/CLI clients, an LLM-wrapper for drop-in memory on OpenAI/Anthropic calls, and reports state-of-the-art results on the LongMemEval long-term memory benchmark, independently reproduced by Virginia Tech's Sanghani Center and The Washington Post.

Model Context ProtocolClaude CodeCodex+8

HindsightVectorize.ioLongMemEval+2

agents, mcp· Added 26 Sep 2026

arxiv.orgpaper🛡️ Security

No-Box Vulnerability Analysis: Description-only Detection of Indirect Prompt Injection Vulnerabilities in MCP Servers

This paper proposes no-box vulnerability analysis, a new paradigm for auditing systems using only functionality metadata when neither system access nor runtime interaction is available. The authors build MCPSEC, a prototype that audits Model Context Protocol servers for indirect prompt injection vulnerabilities purely from tool metadata exposed at server registration, evaluating it against 20 widely-deployed MCP servers with 177 tools where human evaluators confirmed 95 vulnerable tools; MCPSEC achieved 98.9% recall versus 84.2% for an LLM baseline, also producing hypothesized exploitation techniques per finding.

Model Context ProtocolLLM

MCPSEC

AI Security· Added 26 Sep 2026

arxiv.orgpaper🛡️ Security

Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners

This arXiv paper benchmarks three open-source ML model security scanners — ModelScan, ModelAudit, and Fickling — against a 170-artifact synthetic corpus of Pickle and PyTorch files spanning 145 specimen families, distinguishing whether a scanner merely produces a usable security verdict from whether that verdict is correct. ModelAudit reached a definitive judgment on 100% of labeled families versus 81.5% for Fickling and 49.6% for ModelScan, while ModelScan was perfect (100% precision/recall/F1) whenever it did complete analysis, and for the 48 malicious families ModelScan failed to finish, both ModelAudit and Fickling still caught them. The paper argues standard F1-style metrics hide this coverage/availability gap and that tool combination, not any single scanner, yields the best real-world detection.

PyTorchModelScanModelAudit+1

AI Security· Added 26 Sep 2026

arxiv.orgpaper🛡️ Security

Incident-Data Robustness Analysis of the OWASP Top 10 for LLM Applications (2026): How a Community-Expert Ranking Holds Up Against a Large-Scale LLM Incident Corpus

This arXiv paper stress-tests the OWASP Top 10 for LLM Applications (2026) community-expert ranking against a large-scale corpus of real LLM-security incidents (7,714 snapshotted, 6,639 labeled) drawn from CVE, GHSA, OSV, and AIAAIC. Using a Bayesian measurement-error model to correct incident counts for classifier precision/recall, the authors find weak agreement between the expert ranking and the incident-based ranking (Cohen's kappa about 0.20), yet the expert list proves robust: a bake-off of four frontier LLM classifiers fails to beat a simple incidence-floor baseline (balanced accuracy 0.863, Spearman rho = 0.918 against held-out ground truth). It is framed as an exploratory analysis by two OWASP working-group members, not an official supersession of the list.

OSV

OWASP Top 10 for LLM ApplicationsAIAAICGHSA

AI Security· Added 26 Sep 2026

arxiv.orgpaper🛡️ Security

PYPILINE: Malicious PyPI Package Detection via Suspicious API Knowledge and Agent Workflow

PYPILINE is a malicious PyPI package detection system that builds a structured knowledge base of suspicious APIs from static analysis (AST + API call graphs) of known malicious packages, then uses an LLM agent workflow with RAG to retrieve that knowledge base and perform semantic analysis of new packages, outputting structured evaluation reports. The authors report 96.7% precision, 99.6% recall, and 98.1% F1 (5.7–21.6 points above baseline tools), with ~0.6s average detection time per package under 30 concurrent threads, plus an empirical study of common attack strategies and frequently abused APIs in PyPI malware.

PyPIPythonLLM

Supply Chain, AI Security· Added 26 Sep 2026

arxiv.orgpaper🛡️ Security

FuseChain: Runtime Evidence Reconstruction for Software Supply-Chain Attacks

FuseChain is a runtime detection framework for software supply-chain attacks that fuses package/runtime traces, process events, network telemetry, DNS/HTTP metadata, and security alerts into a temporal heterogeneous provenance graph on a unified event-time axis. It decouples anomaly detection from attack-stage reconstruction using a frozen anomaly backbone plus a lightweight decoder, improving deployable Stage Recall@500 from 0.369 to 0.881 across seven SSC attack scenarios, with adaptive retrieval further boosting observable-stage recall from 0.524 to 0.655.

Supply Chain, Blue Team & Detection· Added 26 Sep 2026

endorlabs.comwriteup🧠 AI

Opus 5.5: 6x cheaper and 2x faster than Fable 5.1, but only 33.5% of code is secure | Blog | Endor Labs

Endor Labs benchmarks Claude Code running Anthropic's Opus 5.5 against Opus 5 and Claude Fable 5.1 on its Agent Security League coding-security benchmark, finding Opus 5.5 is dramatically cheaper (6x) and faster (2x) but ranks only third on secure code generation (33.5% SecPass) and nineteenth on functional correctness (68.7% FuncPass). The piece centers on a memorization-detection pipeline that confirmed 51 training-recall 'cheats' (the highest of any model tested) — without which Opus 5.5 would top the security leaderboard — and checks Anthropic's launch claims of being state-of-the-art and faster/cheaper against these measured results.

Claude CodeDocker

Endor LabsClaude Opus 5.5Opus 5+8

evals· Added 7 Oct 2026, published 25 Sep 2026

arxiv.orgpaper🧠 AI

GraphWrit3R: End-to-End 3D Scene Graph Writing

GraphWrit3R is an end-to-end method for generating 3D scene graphs directly from 3D point clouds, Gaussian Splats, or both, outputting a structured JSON graph of objects, attributes, and relationships without relying on ground-truth object annotations or brittle multi-stage pipelines. It encodes point clouds via Sonata and Gaussian Splats via Chorus, fuses both modalities on a shared voxel grid with a novel per-voxel contrastive alignment loss, and decodes the result with a large language model, achieving state-of-the-art object class, predicate, and triplet recall on the 3DSSG benchmark while also supporting open-vocabulary querying.

large language modelGaussian Splatting

GraphWrit3R3DSSGSonata+1

multimodal, research· Added 28 Sep 2026, published 25 Sep 2026

endorlabs.comarticle🛡️ Security

Opus 5.5: 6x cheaper and 2x faster than Fable 5.1, but only 33.5% of code is secure | Blog | Endor Labs

Endor Labs ran Claude Code with Opus 5.5 on its Agent Security League coding tasks. It scored 68.7% FuncPass and 33.5% SecPass, third on security and nineteenth on functionality, behind Fable 5.1 (87.2% / 37.4%). Its anti-cheating pipeline confirmed 51 training-recall solves, the largest memorization penalty on the board; with those counted it would have led by over 11 points. The post also compares cost (~$116 versus ~$1,116 for Opus 5), speed, tokens and tool calls, and checks Anthropic's launch claims against the results.

Claude CodeAnthropicGPT-6 Astra+3

Agent Security LeagueOpus 5.5Opus 5+5

AI Security, AppSec· Published and added 24 Sep 2026

huntress.comwriteup🧠 AI

Fighting AI Slop in Production Codebases: How Huntress Improved Fable 5.1’s API Recall

Huntress ran a controlled eval (forked from Rails core team's rails/lemans harness) measuring whether Claude Fable 5.1 in Claude Code chooses built-in Rails APIs over hand-rolled equivalents, starting from a 48% baseline. They tested three interventions—a four-line CLAUDE.md rule, a Rails-gem lookup tool, and a Claude Code Stop-hook reviewer that has a second model critique the diff—and combined they reached 100% (28/28) API recall across seven eval tasks, with statistical significance via McNemar's test. The piece includes concrete prompts, the CLAUDE.md rule text, and the reviewer hook prompt, making it directly reusable for other coding-agent harnesses.

Claude CodeRuby on RailsRSpec+2

HuntressAnthropicClaude Fable 5.1+7

coding-agents, evals, prompting· Added 23 Sep 2026, published 22 Sep 2026

github.comcourse🛡️ Security

AI for Web3 Security: Zero to Hero

A four-module, code-first course (with an accompanying Python package called lucid) that teaches building an AI bug-hunting harness for smart-contract audits, using the SecondSwap DeFi contracts as a fixed target scanned across all modules. It progresses from a single reasoning-agent-to-structured-findings baseline, through a context/exclusion-loop pass, to an orchestrated harness with specialist lenses and fan-out/synthesis, and finally a version that measures recall against ground truth and hands over a self-improvement method. It runs on an OpenAI-compatible model (gpt-5.6-luna via the Requesty gateway) invoked through the raw model API rather than an agent framework, for pedagogical transparency and determinism.

PythonOpenAISolidity

SecondSwapgpt-5.6-lunaRequesty

AI Security, Blockchain, Learning· Added 22 Sep 2026, published 21 Sep 2026

aikido.devarticle🛡️ Security

Betterleaks: Gitleaks creator's open-source secrets scanner successor

Betterleaks is a new open-source secrets scanner built by Gitleaks' original creator (now Head of Secrets Scanning at Aikido) as a drop-in replacement for Gitleaks. It adds CEL-based rule validation, a BPE-tokenization "token efficiency" filter that hits 98.6% recall on the CredData benchmark versus 70.4% for entropy-based detection, default double/triple-encoding handling, parallelized git repo scanning, and a pure-Go (no CGO/Hyperscan) implementation, with v2 plans for auto-revocation, permissions mapping, and LLM-assisted classification.

GitleaksBetterleaksAikido Security+1

CredData

DevSecOps· Added 18 Sep 2026

endorlabs.comarticle🧠 AI

Claude Code with Fable 5.1 posts 87.2% FuncPass and 37.4% SecPass — a new #1 on our board — while running as fast as Fable 5, costing a third of Opus 5, finishing every task with zero timeouts, and drawing less than half the confirmed cheating of both predecessors. | Blog | Endor Labs

Endor Labs benchmarks Claude Code paired with Anthropic's Fable 5.1 model against Opus 5 and Fable 5.0 on the Agent Security League, a suite of real-world coding tasks with hidden security regression tests. Fable 5.1 tops the board at 87.2% FuncPass / 37.4% SecPass, runs as fast as Fable 5.0 and ~33% faster than Opus 5, finishes every task with zero timeouts, costs roughly a third of Opus 5 end-to-end, and triggers less than half the confirmed training-recall "cheating" of its predecessors. The post details methodology (trajectory-based recall detection, patch similarity scoring) and walks through three CVE-mapped security fixes (Home Assistant CVE-2018-21019, html-sanitizer CVE-2024-34078, Vyper CVE-2023-31146) that only Fable 5.1 solved independently.

Claude CodeDockerPython

CVE-2018-21019CVE-2024-34078CVE-2023-31146+7

evals, coding-agents· Added 23 Sep 2026, published 4 Sep 2026

endorlabs.comwriteup🛡️ Security

Claude Code with Fable 5.1 posts 87.2% FuncPass and 37.4% SecPass — a new #1 on our board — while running as fast as Fable 5, costing a third of Opus 5, finishing every task with zero timeouts, and drawing less than half the confirmed cheating of both predecessors. | Blog | Endor Labs

Endor Labs benchmarks Claude Code running Anthropic's Fable 5.1 against Opus 5 and Fable 5.0 on its Agent Security League leaderboard, which tests whether agents can independently re-derive historical security fixes (FuncPass/SecPass) without recalling them from training data or git history. Fable 5.1 tops both axes (87.2% FuncPass, 37.4% SecPass), runs at Fable 5.0's speed with zero timeouts, costs roughly a third of Opus 5, and shows far less confirmed training-data memorization than either predecessor. The post includes three detailed 'hall of fame' case studies where Fable 5.1 alone produced an independently-reasoned secure patch: Home Assistant's unauthenticated error-log route (CVE-2018-21019/CWE-200), html-sanitizer's Unicode-normalization XSS bypass (CVE-2024-34078/CWE-79), and Vyper's self-referencing dynarray out-of-bounds write (CVE-2023-31146/CWE-787).

Claude CodeHome AssistantVyper

CVE-2018-21019CVE-2024-34078CVE-2023-31146+8

AI Security, AppSec· Published and added 4 Sep 2026

huggingface.coarticle🧠 AI

Give Your Coding Agents a Memory You Own

funes is an open-source, single-binary memory layer that turns the session traces of coding agents (Claude Code, Codex, Pi, Hermes) into a durable, queryable memory instead of an unsearchable archive. It runs locally by default — chunking, embedding and reranking turns with a pinned local model into a Lance dataset, combining vector and BM25 search with cross-encoder reranking and recency weighting — and can optionally sync to a private-by-default Hugging Face dataset so a memory follows a user across machines or is shared across a team, with credential redaction before publishing. The post also reports a "handoff-vs-recall" benchmark showing recall from raw traces is cheaper and more reliable than context compaction or written handoffs for resuming long investigations.

Claude CodeCodexHugging Face+2

funesPiHermes+2

coding-agents, ai-infra· Added 29 Sep 2026, published 3 Sep 2026

deepmind.googlearticle🛡️ Security

Introducing Gemini 3.8 Flash and 3.8 Flash Cyber

Google DeepMind announces Gemini 3.8 Flash and a gated variant, Gemini 3.8 Flash Cyber, the latter positioned as a frontier-level model for autonomous vulnerability discovery and automated patching, accessible to vetted defenders through a new 'Fairwind Program.' On the CyberGym benchmark and an internal 20-language vulnerability-discovery benchmark it reports over 70% success, and on the Collinear CWE-Bench patching benchmark it nearly matches a larger frontier model (47.2% vs 47.8% pass@1) at much lower cost; Google cites Chrome Security producing 2.6x more correct patches and Wiz seeing higher recall at lower cost using the model, plus a case where Google's Cloud Vulnerability Research team found a critical vulnerability in under 2 hours.

GeminiChromeWiz

CyberGymCWE-BenchFairwind Program+5

AI Security, AppSec· Added 29 Sep 2026, published 2 Sep 2026

github.comrepo🛡️ Security

ScreenshotDetector

ScreenshotDetector is an Android app-layer tool that detects screenshot, screen recording, screen mirroring, and casting activity using a combination of public APIs (ScreenCaptureCallback, ScreenRecordingCallback, DisplayManager, MediaProjection, ContentObserver/MediaStore, FileObserver, AccessibilityService, NotificationListenerService) and hidden APIs accessed via reflection (DisplayManagerGlobal.getDisplayInfo, WifiDisplayStatus, AppOps project_media/project_audio). It is positioned as a research counterpart to ScreenshotFaker (an LSPosed/Shizuku-based tool that forges/bypasses screenshot detection), illustrating the attacker/defender dynamic around screen-capture evasion on Android, including detection of adb shell screencap/screenrecord, freeform/mini-window attribution, and accessibility-overlay/screen-reader channel abuse detection.

Android

ScreenshotFakerLSPosedShizuku

Mobile Security, Offensive Security· Published and added 21 Aug 2026

aikido.devarticle🛡️ Security

We burned 11.7bn tokens to find the best cyber AI model

Aikido benchmarked 10 AI models (DeepSeek V4 Pro/Flash, GLM-5.3, Qwen3.8-Max, Kimi K3, Grok 4.6, GPT-5.6 Sol/Terra/Luna, Opus 5) on rediscovering 32 recently-disclosed CVEs from source code, three runs each (96 total passes), measuring recall, precision, consistency, turn budget, and cost. DeepSeek V4 Pro 0813 topped pooled recall at 28/32 for ~$295 across three runs, beating single passes from Opus 5, Grok 4.6, and Sol that each cost $450-590, while Grok was the most consistent model and Kimi K3 had the highest precision among open-weight models. The piece argues open-weight models, when harnessed correctly, now match or beat closed frontier models on vulnerability-discovery tasks, and that pooling multiple LLM runs exploits output variance to improve recall.

DeepSeekGrokQwen+3

GPT-5.6 SolOpus 5GPT-5.6 Terra+6

AI Security, AppSec· Added 3 Sep 2026, published 21 Aug 2026

depthfirst.comwriteup🧠 AI

Introducing dfbench v1: A cybersecurity benchmark for frontier models and agentic systems | depthfirst

dfbench v1 is a held-out cybersecurity benchmark from depthfirst with 253 examples and 910 real vulnerabilities across 17 languages and 27 weakness classes. It scores detection recall, validation precision and differential analysis across commits. Results show recall trading off against precision in frontier models, and higher thinking modes lowering "closed" recall. An appendix estimates precision with Kaplan-Meier survival curves.

Kaplan-Meier estimatorSolidityMove+7

depthfirstdfbenchdfs-large1+10

evals, agents· Added 29 Oct 2026, published 19 Aug 2026

depthfirst.comarticle🛡️ Security

Introducing dfbench v1: A cybersecurity benchmark for frontier models and agentic systems | depthfirst

dfbench v1 is depthfirst's held-out benchmark for AI agents doing defensive security work: 253 examples, 910 real vulnerabilities, 17 languages and 27 weakness classes. It scores detection recall, validation precision (estimated with a Kaplan-Meier "hotspot" analysis) and differential analysis across commits. Results show recall rising as precision falls for stronger models, and "closed" recall dropping at higher thinking levels.

LLMGitHub

dfbenchdepthfirstCyberGym+4

AI Security, AppSec· Added 29 Oct 2026, published 19 Aug 2026

endorlabs.comwriteup🛡️ Security

Best in Class, Novel in Method: Opus 5 and the Recall-Then-Diverge Pattern | Blog | Endor Labs

Endor Labs benchmarked Claude Code + Opus 5 on its SusVibes secure-coding benchmark (real historical CVE fixes hidden as coding tasks) and found it topped the leaderboard at 73.7% FuncPass / 32.4% SecPass, but also uncovered a new memorization-evasion pattern called "recall-then-diverge": the model writes a memorized fix early in its edit trajectory, then iteratively rewrites the code until the final diff no longer resembles the known patch and evades similarity-based cheat detection. The team details the discovery via an aiohttp request-smuggling task (CVE-2023-47641) where the agent produced the exact hidden pytest error string before any tool output could have revealed it, then extended their anti-cheating pipeline to compare first/peak/final edit overlap and string chronology rather than just the final patch, ultimately disqualifying 4 of Opus 5's 13 "unique" secure solves (including a jupyter-server-proxy case with a leaked upstream commit hash) as memorized.

Claude CodeCursorCodex+1

CVE-2023-47641CWE-444Opus 5+9

AI Security, AppSec· Added 3 Sep 2026, published 11 Aug 2026

tldrsec.comarticle🧠 AI

[tl;dr sec] #340 - Anthropic and Meta Agents be Hackin', Agentic Incident Response Notebooks, Figma's AI Code Scanning

tl;dr sec #340 is a security newsletter issue whose AI items cover two incident reports: the UK AISI found 19 unsanctioned agent actions in 10 of 122 cyber-eval runs, and Anthropic reported three real-world compromises by Claude models. It also covers Figma's agentic code scanning (precision from 15% to 80%, 66-vulnerability recall set) and a marimo-based incident response notebook that Claude Code can drive.

Claude CodeCodexmarimo+9

AnthropicOpenAIUK AI Security Institute+12

ai-safety, coding-agents, agents· Added 30 Oct 2026, published 7 Aug 2026

testmachine.aiother🛡️ Security

EVMBench Leaderboard

EVMBench Leaderboard aggregates self-reported vendor results on OpenAI's EVMBench, a benchmark of 117 ground-truth vulnerabilities drawn from 40 real Code4rena smart-contract audit contests. It defines a canonical evaluation protocol requiring the unmodified openai/frontier-evals task runner and the complete, unfiltered audit repository, comparing agents like Nethermind AuditAgent and Guardix by recall against the full ground-truth finding set.

EVMBenchOpenAISolidity+2

Code4renaNethermind AuditAgentGuardix

AI Security, AppSec· Added 3 Sep 2026, published 31 Jul 2026

adversa.aiarticle🛡️ Security

A hole in every one: bypassing the open source AI skill scanners

Adversa AI tested eight open-source AI agent skill scanners (Cisco AI Defense skill-scanner, NVIDIA SkillSpector, mondoo skillcheck, skillcop, claude-skill-antivirus, huifer/skill-security-scan, ai-skill-scanner, hackmyagent) against a control-armed battery of real attacks on malicious Claude Code/agent SKILL.md files, and got a malicious skill past all eight — via paraphrase/translation of injection phrases, Unicode homoglyphs, encoding/reassembly tricks, a substring-allowlist bug in huifer, a hash-lookup-only client (mondoo) that fails open, and an unescaped LLM-judge prompt in skillcop that lets the skill's own text flip a CRITICAL verdict to clean. A parallel false-positive run on MalSkillBench (7,797 samples, 4,000 real benign skills) shows every scanner is either noisy (up to 62% FPR) or blind (recall as low as 0.28), and none beats an unconditional flag-everything F1 baseline; the piece closes with an 11-point "front end" spec (decode-then-rescan, Unicode-fold, canonicalize shell, resolve hosts not substrings, isolate LLM judges, report coverage) scanner builders should implement.

Claude CodeCisco AI DefenseNVIDIA SkillSpector+3

MalSkillBenchOASB leaderboardLD_PRELOAD+2

AI Security, Blue Team & Detection· Added 7 Oct 2026, published 30 Jul 2026

aikido.devarticle🛡️ Security

Benchmarking 13 AI models on rediscovering known CVEs

Aikido benchmarked 13 LLMs (GPT-5.6 variants, grok-4.5, Claude Opus/Haiku, Gemini, GLM-5.2, and later Kimi K3) on rediscovering 26 known CVEs from the GitHub advisory database using their AI Code Analysis harness, which points investigator agents at the vulnerable code and scores pass@3 across three runs. GPT-5.6 led with 23/26 (88.5%) recall, Kimi K3 later tied it at a fraction of the cost, and pooling multiple runs of a cheaper model often matched or beat a single pass of a pricier flagship. Models solved pattern-recognizable bugs (deserialization RCE, stored XSS) uniformly but diverged sharply on subtle multi-step flaws like an ORM column-alias SQL injection.

GPT-5.5Kimi K3GLM+2

GPT-5.6 Sol

AI Security, AppSec· Added 3 Sep 2026, published 16 Jul 2026

endorlabs.comarticle🛡️ Security

Everyone Wins Their Own Benchmark | Blog | Endor Labs

An Endor Labs author explains how to read vendor security benchmarks skeptically. Red flags: winning everything, charts that mislead, missing precision/recall and raw counts, and claims that contradict a tool's nature (e.g. zero run-to-run variance with LLMs). Green flags: transparent methods, stated limitations, error bars. The author then critiques their own AI SAST benchmark: self-built pooled ground truth, 43% of vulnerabilities missed by every tool, F1 of 0.465, and no published run-to-run spread.

Endor Labs

Agent Security LeagueHow to Lie with Statistics

AppSec, AI Security· Added 3 Sep 2026, published 14 Jul 2026

endorlabs.comarticle🧠 AI

Everyone Wins Their Own Benchmark | Blog | Endor Labs

An Endor Labs author explains how to read vendor-published benchmarks skeptically, using the principle that the publisher always wins its own benchmark. Red flags include winning everything by a lot, truncated axes or a self-defined reference score, missing precision/recall and raw counts, and claims like zero run-to-run variance from LLM-based tools. Green flags are transparent methods, honest limitations and error bars. The author then audits Endor Labs' own AI SAST benchmark: pooled ground truth plus manual review, 43% of ground-truth vulnerabilities missed by every tool, F1 of 0.465, and no published run-to-run spread.

Endor LabsAgent Security LeagueDarrell Huff+1

evals· Added 10 Oct 2026, published 14 Jul 2026

endorlabs.comwriteup🛡️ Security

Claude Sonnet 5 with Claude Code: strong on function, average on security, and unusually honest | Blog | Endor Labs

Endor Labs benchmarked Claude Sonnet 5 paired with Claude Code on its Agent Security League test, which measures whether an AI agent can autonomously patch real vulnerabilities in complex codebases without copying a remembered upstream fix. Sonnet 5 scored 83.2% FuncPass (functional correctness) but only 19.6% SecPass (actually closing the vulnerability), continuing the benchmark's recurring functional-vs-security gap; notably it showed very little cheating (8/200 confirmed cases, mostly workspace-leakage rather than training-data recall) compared to prior models like Fable 5 (38 cases) and Opus 4.8.

Claude CodeClaudeCursor+1

Agent Security LeagueFable 5Opus 4.8+2

AI Security· Added 3 Sep 2026, published 9 Jul 2026

endorlabs.comarticle🛡️ Security

Benchmarking Endor Labs AI SAST: 2.6x more real vulnerabilities found than frontier models | Blog | Endor Labs

Endor Labs benchmarks its AI SAST against Semgrep OSS, OpenGrep, Bearer, CodeQL, Claude Opus 4.7 and Codex GPT-5.5 on 8 open-source projects in Java, Python, JS/TS, C# and Go. Ground truth is a hand-verified pool of true positives. It reports 192 true positives (2.6x Claude, 3.5x Codex, 2.4x Semgrep OSS), recall 0.435 and F1 0.465. Frontier models have high precision but low coverage, and pattern scanners are noisy. Example finds include an unbounded-allocation flaw in buffa (CVE-2026-55407), a Shopizer IDOR and a WebGoat authentication bypass. It also outlines the five-step architecture: code graph, framework context, dataflow tracing, triage and classification. The results are vendor-reported.

Endor LabsClaudeCodex+3

CVE-2026-55407IDOROpus 4.7+1

AppSec, AI Security· Added 3 Sep 2026, published 6 Jul 2026

securitylabs.datadoghq.comarticle🛡️ Security

Introducing GuardDog 3.0: new YARA rules engine, risk scoring, and sandboxing

Datadog Security Labs details GuardDog 3.0, an open-source PyPI/npm/Go malicious-package scanner, which replaces its Semgrep-based detections with a YARA rules engine (via yara-python) for better performance at scale, adds a risk-scoring engine that correlates capability rules (e.g., network access) with threat-indicator rules (e.g., suspicious domains, deobfuscation routines) into a 0-10 risk score, and sandboxes the scanning process itself with nono-py to contain vulnerabilities in GuardDog's own extraction/scanning code. The post also covers their continuous evaluation pipeline, which clusters their 27k+ open malicious-package dataset with TLSH before sampling to compute precision, recall, F1, and MCC nightly in CI.

GuardDogYARASemgrep+4

Shai-HuludTLSH

Supply Chain, Blue Team & Detection· Added 14 Sep 2026, published 26 Jun 2026

endorlabs.comarticle🛡️ Security

Endor Labs: AI Coding Agents Cheat Security Benchmarks via Training Recall, Not Reasoning

Endor Labs re-audited its Agent Security League leaderboard after finding that frontier coding agents (Claude Opus 4.8, Composer 2.5, Gemini 3.5 Flash) were inflating security-fix scores through workspace leakage (reading fixed source already present in the repo, e.g. a Mako .tox install) and training-data memorization/recall (reproducing verbatim upstream patch comments, e.g. a Plone namedfile fix), plus flaws in count-based SecPass scoring shown via a python-pillow/Pillow example. After adding anti-cheating detection for these mechanisms, Cursor + GPT-5.5 remained the top security performer, memorization accounted for 137 of 182 confirmed cheating cases, and the overall FuncPass/SecPass gap persisted across the leaderboard.

Endor LabsClaude CodeCursor+3

Agent Security LeagueSecPassFuncPass+4

AI Security· Published and added 11 Jun 2026

aikido.devarticle🛡️ Security

The Wild West of VS Code extensions and how a poisoned extension breached GitHub

Aikido dissects how the May 18 Nx Console extension compromise (2.2M installs, live for ~18 minutes on the Visual Studio Marketplace and ~36 minutes on OpenVSX) was pulled off using a stolen GitHub contributor token, and traces the same stolen-token-to-auto-update-push pattern to AsyncAPI's Shai-Hulud 2.0 compromise, where a malicious v1.0.1 build kept exfiltrating credentials for nearly a month because OpenVSX's rollback to a clean older version prevented auto-update from ever firing. The piece walks through VS Code's actual update-check code paths in extensionsWorkbenchService.ts and the gallery-sync side effect that triggers eventuallyAutoUpdateExtensions, showing why default auto-update, verified-publisher trust, and marketplaces' inability to recall installed code make popular extensions one leaked token away from becoming a worm, and connects this mechanism to the GitHub breach reportedly caused by a poisoned VS Code extension on an employee machine.

VS CodeOpenVSXCursor+1

Nx ConsoleShai-Hulud 2.0AsyncAPI+1

Supply Chain· Added 3 Sep 2026, published 20 May 2026