Skip to content

265 results for β€œLMS”

ListGrid
What you seeeverything
RecentOldestAddedA–Z
All types265Papers158Articles60Repos21
Write-ups15Threads5Tools3Talks2Advisories1
β€œLMS”
eva.acarticleπŸ›‘οΈ Security

So I guess everyone uses LLMs to think now

The eva.ac author argues that recent AI-credited vulnerability findings come from skilled researchers steering LLMs, and that AI firms use disclosures as marketing. Cases cited: Calif, OpenAI and Wiz framing PHP2Shell differently; a hash teaser before the Tailscale "tailworm" privilege escalation (author's account: Tailscale not contacted); and the eMudhra CA key found without AI. No sources, dates or CVEs are given.

LLMOpenAIWiz+3
CalifeMudhraPHP2Shell+1

AI Security, otherΒ· Added 2 Nov 2026, published 1 Nov 2026

eva.acarticle🧠 AI

So I guess everyone uses LLMs to think now

"So I guess everyone uses LLMs to think now" is an opinion post from the eva.ac blog. The author explains why they never use LLMs for published research: they do it for fun and cannot take credit for model output. They argue LLMs still need skilled guidance, criticise AI-company security marketing (PHP2Shell from Calif, OpenAI and Wiz), and cite the Tailscale "tailworm" disclosure and the eMudhra key found without AI.

LLMPHPLinux+3
OpenAIAnthropicCalif+5

ai-policyΒ· Added 2 Nov 2026, published 1 Nov 2026

mashable.comarticle🧠 AI

Anthropic takes steps to block Claude from reverse-engineering video games

Mashable reports, citing Tom's Hardware, that Anthropic's Claude has quietly begun restricting modding, decompiling and reverse-engineering of video game code. Anthropic has made no official announcement. The article ties the change to AI-assisted game ports and mods, the PlayStation 5 jailbreak and Take-Two's subpoenas after the GTA6 leaks. It notes the irony of AI firms now guarding copyright, and the risk that less restrictive LLMs win users.

ClaudeLLM
AnthropicTom's HardwareTake-Two Interactive+4

ai-policyΒ· Added 1 Nov 2026, published 31 Oct 2026

en.cryptonomist.charticle🧠 AI

Meta’s AI-distilled model detector catches 79% at a 0.1% false-positive rate

WM-Basin is a Meta method for detecting student LLMs distilled from watermarked teacher models, described in "Watermark Basins: Black-Box Detection of Distilled LLMs" (arXiv:2610.37266, 27 October 2026). It feeds a suspect model's API text into a clean probe model and checks gradient alignment with a key-specific watermark basin. It reached 79% true-positive rate at 0.1% false-positive rate across 1,200 students, versus 29% for the prior token-level detector. Clean-data fine-tuning and paraphrasing weaken it.

LLM
MetaMeta FAIRMeta Superintelligence Labs+3

research, trainingΒ· Published and added 30 Oct 2026

github.comrepo🧠 AI

llm-trails: What LLMs reach for inside a breached AWS account

llm-trails is an open study of what LLMs choose to do next inside a breached AWS account. It puts 14 models in 13 post-compromise situations, records only a single-turn plan with nothing executed, and codes each answer to CloudTrail events. The repo has the llmtrials harness, prompts, codebook, processed dataset (viz_data.json) and an interactive dashboard, so the study can be rerun or extended.

AWSCloudTrailLLM+2
Adan Alvareztraildiscover.cloudClaude

evals, agentsΒ· Added 30 Oct 2026

github.comrepoπŸ›‘οΈ Security

llm-trails: What LLMs reach for inside a breached AWS account

llm-trails is a research repo that placed 14 LLMs in 13 post-compromise AWS situations, asked only for their next move (single-turn, nothing executed), and coded each answer to CloudTrail events. It includes the llmtrials harness, prompts, codebook, processed dataset (viz_data.json) and an interactive dashboard. Rates are conditioned on the situation shown, and the aim is turning model behavior into detection priorities.

AWSAWS CloudTrailLLM
traildiscover.cloud

AI Security, Cloud & Infra, Blue Team & DetectionΒ· Added 30 Oct 2026

depthfirst.comarticleπŸ›‘οΈ Security

dfbench β€” depthfirst

dfbench is a held-out benchmark maintained by depthfirst that scores frontier LLMs on defensive security work: vulnerability detection, validation (precision) and differential analysis between commits. It reports recall, precision, F1 and cost per task by model and thinking level, split by application, low-level and smart-contract code and by single or multi-repo scope. Overall detection recall runs from about 21% to 69%, with the top figure at $99.19 per task for Mythos 5; validation precision mostly falls between 15% and 49%.

LLMKimi K3Grok+3
dfbenchdepthfirstGPT-5.6 Sol+3

AI Security, AppSecΒ· Added 29 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

TextSeal2: Distortion-Free LLM Watermarking with Copula Mixing and Memoryless Races

TextSeal2 is a distortion-free LLM text watermark that merges two or more keys into one virtual key and couples it to fresh noise through AutoMark's Gaussian copula. Under speculative decoding it verifies keyed drafts with an extra exponential race, so output matches the target model and acceptance rate is unchanged. On three LLMs at a 10^-3 false-positive rate, it detects 8 to 22 points more 100-token answers than TextSeal.

TextSeal2TextSealAutoMark+2
Gumbel-max schemespeculative decodingGaussian copula+1

AI SecurityΒ· Added 31 Oct 2026, published 29 Oct 2026

arxiv.orgpaper🧠 AI

TextSeal2: Distortion-Free LLM Watermarking with Copula Mixing and Memoryless Races

TextSeal2 is a distortion-free watermark for LLM output that keeps its signal under speculative decoding. It merges two or more keys into one virtual key coupled to fresh noise through AutoMark's Gaussian copula, and verifies drafts with an extra exponential race. On three LLMs it detects 8 to 22 points more 100-token answers than TextSeal at a 10^-3 false-positive rate.

LLMspeculative decodingGumbel-max watermarking+1
TextSeal2TextSealAutoMark+2

research, inferenceΒ· Added 31 Oct 2026, published 29 Oct 2026

arxiv.orgpaper🧠 AI

PhysCount: Do MLLMs Count Visual Appearances or Physical Entities?

PhysCount is a benchmark of 546 real-world images and 4,595 human-written questions in 651 target-specific groups. It tests whether multimodal LLMs can separate visible appearances (direct view, reflections, glass) from the physical entities behind them. Across 40 MLLMs, counting appearances is much easier than linking them to entities, and the best model answers every question correctly in only 13.36% of groups.

vision-language model
PhysCountarXiv

evals, multimodal, researchΒ· Added 31 Oct 2026, published 29 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

Understanding and Mitigating Role Spoofing in LLMs

Understanding and Mitigating Role Spoofing in LLMs is an arXiv paper (October 2026) on role spoofing, a prompt injection where malicious tool output impersonates the user. It shows models track "who said what" using both content cues (semantics, style) and structure cues (control tokens, formatting). Attacks combining fake structure and content succeed often, adversarial training stops only content-based attacks, and linear probes trained to predict conversation roles detect spoofing across settings.

LLM
role spoofingrole confusionprompt injection+1

AI SecurityΒ· Added 31 Oct 2026, published 29 Oct 2026

arxiv.orgpaper🧠 AI

Benchmarking color judgment in multimodal LLMs with CIE-computable ground truth

A five-task benchmark tests multimodal LLM color judgment against CIE colorimetric ground truth: AWB pair judgment, skin-tone bias direction, CCT estimation, Delta E 2000 pair ordering, and color naming under changing illuminants. Eight endpoints were tested. On ColorChecker-24 stimuli, closed frontier models reach CCT MAE of 715-1056 K versus 1358-2133 K for open-weight models, and a non-ColorChecker control shows this gap is partly template-dependent.

vision-language modelQwen3-VLGLM+2
GLM-4.5VGPT-5.4Claude-Opus-4.7+4

evals, multimodal, researchΒ· Added 31 Oct 2026, published 29 Oct 2026

tldrsec.comarticleπŸ›‘οΈ Security

[tl;dr sec] #352 - Unprompted Slides, AWS AI Post-compromise Trends, Malicious LLM API Routers

tl;dr sec #352 is a security newsletter issue covering [un]prompted talks (AI-assisted firmware patching, a "Security Constitution" for finding cloud design flaws), Azure API Connections cross-tenant bugs worth $200,000, and a test of 14 LLMs given AWS keys. It also covers a paper on malicious LLM API routers (9 of 428 injecting code) and Datadog's "counterfeit policy" bypass of the Jev classifier.

AzureAWSLLM+3
PHP2Shellprompt injectionpath traversal+9

AI Security, Cloud & Infra, Offensive SecurityΒ· Added 30 Oct 2026, published 29 Oct 2026

arxiv.orgpaper🧠 AI

TrussTutor-Bench: Benchmarking large language models on solution accuracy, sycophancy, and error detection in statics

TrussTutor-Bench is a benchmark of 200 verified two-dimensional truss problems from statics, paired with automatically generated student solutions. It evaluates thirteen LLMs on solution accuracy (64.5% to 97.0%), error detection (90.4% to 100%) and sycophancy. Models rejected their own correct solutions in 1.6% to 50.0% of cases when a student insisted the work was wrong. Sycophancy separated models; error detection did not.

LLM
TrussTutor-BencharXiv

evals, researchΒ· Added 31 Oct 2026, published 28 Oct 2026

arxiv.orgpaper🧠 AI

Leaving, Staying, or Living There? Duration Commonsense Reasoning with Human-LLM Co-annotations

This paper studies whether vision-language models can infer how long a person stays somewhere (living, staying, or leaving) from geotagged social media posts. The authors build a new corpus of geotagged tweets co-annotated by humans and LLMs. State-of-the-art models struggle even with multi-turn Chain-of-Thought prompting, often missing contextual cues and subtle hashtags while over-relying on prominent visual details.

vision-language modelLLM
arXiv

research, evals, multimodalΒ· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper🧠 AI

inkGround: Benchmarking Fine-grained Visual Grounding Capabilities of Vision-Language Models for Cursive Handwriting in the Wild

inkGround is a benchmark for stroke trajectory recovery from full-page, camera-captured handwriting, with 537 page images (about 18.7k words, 163k strokes) and pseudo-label references filtered by OCR agreement and human raters. Models get a page image and a target word, then output ordered stroke points. Seven VLMs were scored with DTW, Chamfer distance, readability and style similarity. Closed-source models, notably Gemini-3.1-Pro, did best, and the ranking held on IAM-OnDB.

vision-language model
inkGroundIAM-OnDBGemini-3.1-Pro+1

evals, multimodal, researchΒ· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper🧠 AI

Can LLMs Uncover Hidden Physical Laws through Free Exploration?

LongChang is an automated framework that builds simulated worlds with altered physical laws, such as heavier objects falling more slowly, to test whether LLMs can find the law by free exploration. Explorer LLMs manipulate objects and build tools without tasks or hints, while separate LLM agents generate scenes, review tool designs and grade reports. Across 12 LLMs, GPT-6 Astra reached a 16.7% discovery rate and the other 11 scored 0%.

LLM
LongChangGPT-6 AstraarXiv

evals, research, agentsΒ· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper🧠 AI

VideoLVR: Latent Visual Reasoning over Grounded Video Evidence

VideoLVR is a method that lets video multimodal LLMs build latent visual evidence from local regions at relevant moments and reason over it interleaved with text tokens. A visual token prediction objective, defined over a vocabulary derived from the input video, supervises the latents. It extends to reinforcement learning with sampled visual tokens as actions and final-answer rewards, and improves fine-grained video benchmarks without much context growth.

vision-language modelLLM
VideoLVRarXiv

research, multimodal, trainingΒ· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper🧠 AI

As the Story Unfolds: Watching a Film and Identifying Characters as a Human Does

This paper defines a character naming task for films: given a name spoken in dialogue and its video context, the model must pick which visible character bears that name. It provides a dataset with annotations for character naming and first appearances, a video-and-dialogue model, and a comparison with proprietary multimodal LLMs. It also proposes a progressive framework that builds a character bank as the film plays, without cast lists or actor photographs.

multimodal large language model
arXiv

research, multimodalΒ· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper🧠 AI

When Visual Evidence Is Insufficient: How Instructions, Post-Training, and Internal Activations Shape Refusal in Multimodal Language Models

This paper studies how multimodal LLMs refuse questions that an image cannot answer, using the MoHoBench benchmark on three MLLMs. It varies the meaning and placement of refusal instructions, compares supervised and preference-based post-training, and intervenes on a refusal-related activation direction. Instructions change refusal at the sample level in ways aggregate rates hide. Post-training raises refusal most but can mix refusal with attempted answers. Activation intervention gives smaller, transferable changes.

vision-language modelLLM
MoHoBencharXiv

ai-safety, multimodal, researchΒ· Added 28 Oct 2026, published 27 Oct 2026

lynchlab.netwriteup🧠 AI

A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis

A dermatologist tested GPT-6 Astra and Claude Fable 5.1 on 200 biopsy-proven lesions from MILK10k, giving each a clinical and a dermoscopic image. A CNN (Inception v3, fine-tuned for about twenty minutes on 68,011 ISIC images) named the right diagnosis for 62% of lesions, against 42% for both LLMs. The post lists caveats (small sample, single run, possible contamination) and links the cases, prompts, replies and scoring code.

CNNGPT-6 AstraClaude Fable 5.1+4
MILK10kInternational Skin Imaging CollaborationISIC Archive+2

evals, multimodalΒ· Added 26 Oct 2026, published 25 Oct 2026

eprint.iacr.orgpaperπŸ›‘οΈ Security

A Kelsey–Kohno Attack Variant against SHA-2 Message Digests in HSS/LMS and XMSS

Falko Strenzke's ePrint 2026/2635 presents a multi-target second-preimage attack on message hashing in HSS/LMS and XMSS with SHA-256 and SHA-256/192. It herds chaining states from many collected signatures into one state, defeating the key and leaf-index prefix. With 2^64 signatures, SHA-256 cost falls from 2^256 to about 2^166 (2^156 at maximal message length). SHAKE256 parameter sets are unaffected.

HSS/LMSXMSSSHA-256+2
Kelsey–Kohno herding attackKelsey–Schneier expandable messagemulti-target second-preimage attack+2

otherΒ· Added 26 Oct 2026, published 25 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

Reversible Pseudonymisation of Documents Sent to External LLMs: an Open Bilingual Benchmark and an Engine

Open bilingual (Russian and UK English) benchmark of five open pseudonymisation tools and a regex baseline, plus a new engine built on the two strongest tools. On a held-out set of 1,000 synthetic documents per language, the best existing tools leave a whole personal value visible in 61% (Russian) and 58% (UK) of documents. The engine does so in 0.3% and 0.2%. Russian name restoration reaches only 94.6%, below the 99% threshold. Real documents were not tested.

LLMpii-guard

AI SecurityΒ· Added 27 Oct 2026, published 25 Oct 2026

arxiv.orgpaper🧠 AI

Reversible Pseudonymisation of Documents Sent to External LLMs: an Open Bilingual Benchmark and an Engine

This paper presents an open benchmark of five existing pseudonymisation tools and a regex baseline on Russian and UK English documents, plus an open engine built on the two strongest tools. On a held-out set of 1,000 synthetic documents per language, the best existing tools leave a whole value visible in 58 to 61% of documents. The engine does so in 0.2 to 0.3%. Russian name restoration reaches only 94.6%, below the 99% threshold. Real documents were not tested.

LLMpii-guard
Vladimir BeskorovainyiarXiv

evals, llm-apps, researchΒ· Added 27 Oct 2026, published 25 Oct 2026

arxiv.orgpaper🧠 AI

Detecting and Understanding Ambiguity in Visualizations with Role-Playing Vision Language Model Interpretations

This paper presents a pipeline that uses persona-conditioned vision language models to surface interpretive ambiguity in visualizations. Topic-relevant personas elicit candidate takeaways, and a Target-Pattern-Scope framework lets LLM judges label statement pairs as Compatible, Disjoint or Opposing, which are summarized in an ambiguity profile. The authors benchmark VLMs on chart comprehension per chart type, check labeling against human annotations, and run case studies on public visualizations and a redesign.

vision-language modelLLM
arXiv

research, multimodal, evalsΒ· Added 27 Oct 2026, published 25 Oct 2026

arxiv.orgpaper🧠 AI

InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

InteractiveWorld is an environment for closed-loop interaction between an AI assistant and a simulated human agent who sees the scene, follows instructions and acts independently. InteractiveBench, built on it, has 120 multi-turn tasks in six categories scored by verifiable, state-based rewards. Across 12 frontier MLLMs, the best model, Claude Fable 5.1, reaches at most 20% on long-horizon Guidance and 25% on Planning tasks.

multimodal large language model
InteractiveWorldInteractiveBenchClaude Fable 5.1+1

evals, multimodal, researchΒ· Added 27 Oct 2026, published 25 Oct 2026

arxiv.orgpaper🧠 AI

DRILS: Dependency-based Requirement Change Impact Analysis Using LLMs

DRILS is an LLM-based agentic framework for requirements change impact analysis. It uses multiple agents to generate candidate impact sets, with an LLM-as-a-judge promoting diversity in selection. It adds RE-RAG, a tuned retrieval technique, and a requirement dependency graph as a knowledge source. On three curated datasets it reaches 87.1% recall, 76.3% precision and 81% F1 at lower cost than baselines. The authors release the datasets.

LLMRAG
arXivDRILSRE-RAG

research, agents, llm-appsΒ· Added 27 Oct 2026, published 24 Oct 2026

arxiv.orgpaper🧠 AI

Build, Don't Play: Self-Improving LLM Research Agents Develop a Hearthstone Battlegrounds AI That Rivals Top Players

AutoTavern is a self-improving system of specialised LLM research agents with a typed shared memory and verification gates. In 20 days it built TavernGym, a full-content Hearthstone Battlegrounds simulator, and 23 versions of TavernAgent, a PPO-trained player. TavernAgent averaged placement 1.60 with 80% first places over its last 20 live games, beating the top ten players (2.34, 59%). On TavernBench (960 tasks), it answered 91% of play tasks against at most 63% for frontier LLMs.

PPOLLM
AutoTavernTavernGymTavernAgent+3

agents, research, training, evalsΒ· Added 27 Oct 2026, published 24 Oct 2026

arxiv.orgpaper🧠 AI

TriageRAG: Failure-Aware Self-Training for Adapting Retrieval-Augmented Language Models to Specialized Domains

TriageRAG is a failure-aware self-training method for adapting retrieval-augmented LLMs to specialized domains using only an unlabeled corpus. It probes the student on self-generated questions with and without retrieved context, sorts them into known, solved, context-failure and retrieval-failure, and routes each to a matching training signal. With Qwen3-4B on 11 datasets it beats budget-matched SimRAG by 1.2-6.4 points and recovers closed-book accuracy where retrieval hurts.

RAGLLMQwen3
TriageRAGSimRAGCS-Bench+1

research, llm-apps, trainingΒ· Added 27 Oct 2026, published 24 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

Enhancing LLM-Based Vulnerability Detection via Representation-Aware Logical Ensemble

RALE (Representation-Aware Logical Ensemble) is a method for detecting vulnerabilities in C source code with LLMs trained on different code representations: raw source, Abstract Syntax Trees and Control Flow Graphs. The authors show the representations suit different vulnerability classes, and combine the models with explicit logical rules and majority voting. Single-representation models reach up to 73% F1, while the ensemble reaches 78%.

LLMC
RALERepresentation-Aware Logical EnsembleAbstract Syntax Tree+1

AI Security, AppSecΒ· Added 28 Oct 2026, published 24 Oct 2026

arxiv.orgpaper🧠 AI

Enhancing LLM-Based Vulnerability Detection via Representation-Aware Logical Ensemble

RALE (Representation-Aware Logical Ensemble) is a method for LLM-based vulnerability detection in C code. The authors train LLMs on raw source code, Abstract Syntax Trees and Control Flow Graphs, and find each representation works best for certain vulnerability classes. Combining the models with explicit logical rules and majority voting raises F1 from at most 73% for a single model to 78%, with gains in accuracy, precision and recall.

LLMstatic analysis
arXivRALE

research, trainingΒ· Added 28 Oct 2026, published 24 Oct 2026

arxiv.orgpaper🧠 AI

VulDec: Vulnerability Detection in Binaries via Neural Decompilation

VulDec is a dataset and framework for detecting vulnerabilities in binaries by first decompiling them with fine-tuned LLMs. The dataset aligns real-world Assembly-C function pairs across several architectures with CWE labels. Tuned neural decompilers reach about 56% Edit Similarity and 43% BLEU-4, and tuned detectors reach a macro F1 of 0.61 on a realistic test set with a 3% vulnerability rate.

LLM
VulDecCWE

research, trainingΒ· Added 28 Oct 2026, published 24 Oct 2026

eprint.iacr.orgpaperπŸ›‘οΈ Security

Provable Security of SHA-256-Based LMS Revisited in the Ideal-Cipher Model

Provable Security of SHA-256-Based LMS Revisited in the Ideal-Cipher Model is an October 2026 ePrint paper by Chun Guo, Hao Cheng, Bart Preneel and Meiqin Wang. It shows forgery attacks on LMS_SHA256_M32 and LMS_SHA256_M24 that exploit Merkle-DamgΓ₯rd structure, so the Katz and Fluhrer random-oracle bounds do not apply. It proves ideal-cipher-model bounds: about 256 minus log2 of the block count for M32, and a minimum of two terms for M24.

LMSSHA-256
KatzFluhrerLMS_SHA256_M32+4

otherΒ· Added 26 Oct 2026, published 24 Oct 2026

arxiv.orgpaper🧠 AI

Hey, guess what: CUES lead to inferences about typicality

The paper builds two corpora, EventTyp and the human-validated EventTypGold, to study how people infer typicality and atypicality of events from what is said. Large-scale human and LLM experiments show that three CUES predict these inferences in humans: Commonsense Knowledge, Utterance Emphasis and Snark. In most settings LLMs lack the commonsense knowledge to judge an event's general typicality, so they struggle to follow the CUES.

LLM
EventTypEventTypGoldarXiv

research, evalsΒ· Added 27 Oct 2026, published 23 Oct 2026

latent.spacearticle🧠 AI

AI Is Getting Good at Math. Here’s Why.

The essay argues that LLMs solve open math problems because they search depth first, and automated verifiers make failed attempts cheap. It uses WT Gowers' hint-sequence complexity (2^k + t), adds a branching factor b, and walks through OpenAI's unit-distance disproof and Cycle Double Cover prompt as a drafter/solver/verifier harness. It predicts AI does best where k and b are small.

LLM
OpenAIAnthropicLatent Space+12

research, agentsΒ· Published and added 23 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

On Task, Off Track: Benchmarking LLM and Agent Robustness to Near-Target Prompt Injection

NTBench (Near-Target Benchmark) tests LLM and agent robustness to near-target prompt injection, where injected instructions steer a response toward a plausible but incorrect or harmful outcome instead of abandoning the task. It builds reference-guided target-instruction pairs on existing datasets, with human validation, and evaluates seven LLMs on ten language datasets and two agent benchmarks. Attack success rose 9.3 to 35.8 points (incorrect targets) and 1.7 to 17.3 points (harmful targets) over fixed-target baselines.

LLM
NTBenchnear-target prompt injectionprompt injection

AI SecurityΒ· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper🧠 AI

BiLoCo: Binary Low-Rank Corrections for Post-Training Quantization

BiLoCo is a post-training quantization method for FP4 LLMs that adds low-rank corrections whose left and right factors are stored as binary Β±1 sign vectors instead of BF16. This uses 16x less storage per rank-one component and lets the correction run on CUDA cores alongside the FP4 tensor-core GEMM. On 4B to 72B LLMs it matches or improves memory-matched SVDQuant. A fused B300 kernel is at most 11% slower than FP4-only GEMM at batch size 1 for ranks up to 512.

CUDALLMdiffusion model
BiLoCoSVDQuantNVIDIA+1

inference, researchΒ· Added 26 Oct 2026, published 23 Oct 2026

anthropic.comarticleπŸ›‘οΈ Security

LLMs are making formal software verification feasible at a much larger scale

Anthropic and Theorem report using Claude to write Rocq proofs that 239 fuzz entry points across 123 OSS-Fuzz libraries never crash on inputs under 64 MB, at the RISC-V binary level (about 180 million lines of proof). Claude patched the bugs it found along the way, including a rediscovery of a libexpat CVE and bugs fuzzers would miss. The post covers trusted components, limitations, token-cost trends and released proofs.

ClaudeRocqRISC-V+3
CVE-2026-56132TheoremOSS-Fuzz+5

AppSec, AI SecurityΒ· Added 23 Oct 2026, published 22 Oct 2026

anthropic.comarticle🧠 AI

LLMs are making formal software verification feasible at a much larger scale

Anthropic and Theorem report that Claude wrote Rocq proofs of crash-freedom for 239 OSS-Fuzz entry points across 123 open-source libraries, working on compiled RISC-V binaries for inputs up to 64 MB. Proofs total about 180 million lines, and Claude found and patched bugs along the way. The post covers proof reuse (17% fewer tokens), Opus 5.5 using about 40% fewer tokens than Opus 5, and limitations.

RocqRISC-VVerus+10
CVE-2026-56132AnthropicClaude+11

research, coding-agentsΒ· Added 23 Oct 2026, published 22 Oct 2026

arxiv.orgpaper🧠 AI

Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration

This paper studies reverse sycophancy: whether users bend toward a confident, consistent, wrong LLM. In a within-subjects experiment, 32 participants learned science topics from five GPT-5-mini assistants with 0 to 100% of facts replaced by falsehoods. Scores fell from 7.4 to 2.9 out of 10, planted falsehoods were chosen on 60% of falsified items against 11% of correctly taught ones, and confidence stopped tracking correctness. It introduces a baseline-corrected Human Deference Index.

LLM
GPT-5-miniarXiv

research, ai-safety, evalsΒ· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper🧠 AI

Rethinking Sequence Compression with Test-time Regression

TTR-Compressor is a sequence compression method that treats input tokens as a test-time regression problem. The regression is learned end-to-end by backpropagation, and its coefficients are solved in closed form in the forward pass to give the compressed representation. As a block-wise sparse attention compressor it improves RULER over attention-based compressors by a large margin. As a visual token merger for multimodal LLMs it beats the MLP merger and matches it with 55.6% fewer visual tokens.

TransformerLLMvision-language model
TTR-CompressorRULERarXiv

research, multimodalΒ· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper🧠 AI

An Empirical Study of Recent Large Language Models for Knowledge Graph to Text Generation on Standard Benchmarks

This empirical study benchmarks recent LLMs against fine-tuned encoder-decoder models on knowledge graph-to-text generation, using reference-based metrics and graph-grounded, reference-free LLM-based evaluation validated against human annotations. Recent LLMs narrow or close the gap on reference-based metrics and do better on graph-grounded ones. Prompted models make fewer contradictions and triple omissions, and fine-tuned models' errors grow faster with graph size.

LLMTransformer
arXiv

research, evalsΒ· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper🧠 AI

The Geometry of Metaphor: Probing and Steering Metaphor Representations in Large Language Models

The Geometry of Metaphor is an October 2026 arXiv preprint that uses mechanistic interpretability to find causal subspaces for metaphor in LLMs of different architectures and sizes. The authors steer output on classification tasks across parts of speech. They also steer generation tasks, showing the metaphoricity of model output can be controlled. The abstract gives no model names or metrics.

LLM
arXiv

researchΒ· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper🧠 AI

DuoCook: An Overcooked-Inspired Benchmark for Visual Planning and Coordination

DuoCook is an Overcooked-inspired benchmark in which one vision-language model controls two cooks from top-down views in Unreal Engine environments built with coding agents. It has 20 maps in five categories covering equipment contention, changing terrain and handoff dependencies. Four VLMs were compared; the best, gpt-6-astra, reached 26.2% mean order completion. A notebook memory ablation had mixed effects: positive for Claude and Qwen, negative for GPT and Gemini.

Unreal Enginevision-language model
DuoCookGPT-6 AstraClaude+4

evals, agents, multimodalΒ· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper🧠 AI

Training Small Language Models for Transferable Tool Selection

This paper trains small open language models with reinforcement learning to select tools for multi-step instructions over corpora of more than ten thousand tools. A Qwen3-8B model post-trained with the recipe outperforms Llama-4-Maverick (402B) on ToolBench. The trained model transfers to different retrievers and generalizes to new tool corpora in other domains, without relying on proprietary LLMs like Claude or ChatGPT.

Qwen3LlamaLLM
ToolBenchClaudeChatGPT+1

research, training, agentsΒ· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper🧠 AI

From Structured Records to Patient Narratives: Preserving Clinical Alignment in LLM-Generated Health Summaries

This paper studies evidence-grounded health summarization for menstrual health monitoring. It combines structured evidence conditioning with RAG in a prompt-based constrained generation framework, and scores 4,500 summaries from three LLMs with NLI, atomic factuality metrics and readability measures. Grounding constraints improved uncertainty preservation and generally kept or improved faithfulness against retrieved evidence, while omission effects varied by model. Clinical alignment came with harder reading and more medical terminology.

LLMRAG
arXiv

research, evals, llm-appsΒ· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper🧠 AI

Drift Says When, Budget Says How Much: Rate-Controlled KV Re-selection for Long-Form Multimodal Generation

Rate-Controlled Refresh (RCR) is a KV cache re-selection method for multimodal LLMs in long-form generation. Attention drift decides when and where to re-select visual tokens, and a per-sample controller sets how often. RCR beats InfiniGen by 1.6-3.8 ROUGE-L points on five of six model-task settings and stays within 1.7 points of exact per-step re-selection. On video description, where drift is homogeneous, it no longer beats a fixed threshold.

Qwen2-VLQwen3-VLInfiniGen+1
DetailCaps

research, inference, multimodalΒ· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence

SecLog-Bench is a benchmark for LLM reasoning over cybersecurity logs, built on an extensible simulation platform covering 29 attack scenarios. It combines audit logs, application logs and network traffic, with attack-stage labels and step-by-step evidence chains. Eight LLMs were tested on post-detection attack-progress reasoning: the best reached 44.44% accuracy, 45.07% Macro-F1 and 54.87% Stage Reasoning F1.

LLM
SecLog-BenchStage Reasoning F1

AI Security, Blue Team & DetectionΒ· Added 23 Oct 2026, published 22 Oct 2026

arxiv.orgpaper🧠 AI

SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence

SecLog-Bench is a benchmark for LLM reasoning over cybersecurity logs. A simulation platform with 29 attack scenarios generates audit logs, application logs and network traffic, with attack-stage labels and step-by-step evidence chains. Eight mainstream LLMs were tested on attack-progress reasoning: the best reached 44.44% accuracy, 45.07% Macro-F1 and 54.87% Stage Reasoning F1. Correct predictions often came with flawed reasoning.

LLM
SecLog-BencharXiv

evals, researchΒ· Added 23 Oct 2026, published 22 Oct 2026

arxiv.orgpaper🧠 AI

Disentangling Robustness and Sycophancy in Multilingual LLMs

The SC-matrix is an evaluation framework for multilingual LLM sycophancy. It crosses a model's competence (correct answer with no pressure) with its sycophancy (keeping that answer under pressure), so agreeing with a false premise can be told apart from not knowing the answer in that language. Results: equal aggregate sycophancy rates hide different failure mixes, high competence does not always protect, and reasoning-oriented mitigations beat directive ones.

LLM
arXiv

ai-safety, evals, researchΒ· Added 24 Oct 2026, published 22 Oct 2026

huggingface.coarticle🧠 AI

How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications

VocabX is a vocabulary adaptation method from Dharma-AI that selects sub-word and multi-word "macro" tokens by serving cost instead of frequency. On a Qwen3-4B Portuguese essay grader, response length fell from 350 to 101 tokens, the LM head shrank from about 152,000 to 65,000 rows, and measured cost dropped 77.0% on an RTX 5090 with vLLM. The bench score rose from 0.703 to 0.810. It beat FVT, MWT and in-place BPE across 15 model and domain combinations.

Qwen3vLLMLlama+3
Dharma-AIVocabXQwen+1

inference, training, researchΒ· Published and added 21 Oct 2026

simonwillison.netwriteup🧠 AI

Battle Drones - a vibe coded RTS with audio-based multiplayer

Battle Drones is a browser-based Command & Conquer style RTS that Simon Willison built over 12 iterations and 321 commits, mostly with Claude Code and Claude Opus 5.5. The post gives key prompts and transcripts per version, including the "Make it a LOT more fun" run in Ultracode mode. It also covers an Agent vs. Agent mode for LLMs, audio-modem multiplayer between nearby phones, and lessons on deterministic simulation and code-generated 3D models and music.

Claude CodeTypeScriptThree.js+6
Simon WillisonClaude Opus 5.5Claude Fable 5+5

coding-agents, agentsΒ· Published and added 20 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

Combining Long-Term Memory and LLMs Comes at a High Cost: From Prompt Injection to Persistent Compromise

Paper on memory injection, where prompt injection persists across sessions through the long-term memory of LLM-as-a-service platforms. It presents a general threat model and shows all NIST attack goals are achievable on ChatGPT Instant 5.3, Claude Sonnet 4.6 and Gemini 3 Flash. Vectors include personal memories, projects, skills, organization instructions and MCP resources. On Claude, shared memory lets an attacker compromise multiple users in an organization and exfiltrate prompts. Nearly half of the derived mitigations are unimplemented.

ChatGPTClaudeGemini+2
memory injectionprompt injectionNIST+2

AI SecurityΒ· Added 24 Oct 2026, published 20 Oct 2026

arxiv.orgpaper🧠 AI

InterWebCoder: Learning Interactive Webpage Generation from Executable Tests via Reinforcement Learning

InterWebCoder is a framework for interactive webpage code generation by multimodal LLMs. It synthesizes self-contained webpages with executable test scripts from recorded interaction trajectories, repairs failures using runtime evidence, then trains with test-ID-weighted SFT and GRPO using test pass rates and visual similarity as rewards. The pipeline yields 1,736 verified samples from 2,058 tasks. A 9B model scores 48.54 on IWR-Bench and 48.42 on Vision2Web, and reward latency drops over 97%.

GRPOvision-language modelLLM
InterWebCoderIWR-BenchVision2Web+3

research, training, multimodalΒ· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper🧠 AI

Benchmarking Arabic Dialects for Math Problems

DAR-MGSM (Dialectal Arabic Multilingual Grade School Math) is a manually curated benchmark for mathematical reasoning in Dialectal Arabic. It covers five major Arabic dialects plus Modern Standard Arabic, in a translated-only version and a translated and culturally localized version. Eleven Arabic-centric and multilingual LLMs were tested: accuracy generally drops on the localized version, and a mechanistic interpretability analysis shows the two variants induce distinct internal reasoning trajectories.

LLM
DAR-MGSMarXiv

evals, researchΒ· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper🧠 AI

Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V

Code2Video-Compass is a benchmark of 2,104 tasks for code-based video generation, where multimodal LLMs write executable programs that produce complete videos from text (T2V), image (I2V) or video (V2V) inputs. Tasks span 10 categories and 24 subtypes. Scoring uses 12 objective metrics and five MLLM-judge metrics. Across 15 proprietary and open-weight models from nine families, performance falls as construction, dynamic animation and state-dependency complexity rise.

vision-language modelLLM
Code2Video-CompassarXiv

evals, multimodal, researchΒ· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper🧠 AI

DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge

DeclineCraft is a dynamic RAG abstention benchmark whose unanswerable questions target Minecraft feature additions made after a corpus cut-off, with audits for parametric exposure. Across 27 LLMs, single-shot retrieval raises Correct Abstention Rate in 38 of 40 configurations, while iterative RAG lowers it in 30 of 40. Enabling thinking without retrieval lowers abstention for 8 of 13 models, shifting responses to premise denial.

RAGLLM
DeclineCraftMinecraftarXiv

evals, llm-apps, researchΒ· Added 22 Oct 2026, published 20 Oct 2026

peec.aiwriteup🧠 AI

Publish fake stats, get cited in LLMs

StateGlobe is an experiment in which a Peec AI researcher generated 50,000 fake statistics pages with GPT-4.1 for about $50. After 7 months, Google had deindexed the site, but ChatGPT-User still made about 72,000 requests in 30 days. The site drew 10,000+ ChatGPT sessions and 330+ referring domains, and its numbers were cited in an academic paper and bank research. The post explains why retrieval differs from ranking and how to build real statistics pages.

ChatGPTGPT-4.1Cloudflare+6
Peec AIStateGlobeOpenAI+11

llm-apps, ai-policyΒ· Added 21 Oct 2026, published 20 Oct 2026

simonwillison.netarticle🧠 AI

My obituary for programming

Simon Willison's 19 October 2026 link post points to Jeremy Scheff's essay "My obituary for programming". Scheff argues coding agents remove what career programmers enjoyed: paid work on tricky logic puzzles where they set strategy and timelines. Willison notes that LLMs suit programmers who care about building working systems, while those who loved solving the problems face a real loss. The post is a short quote with no data.

LLM
Simon WillisonJeremy Scheff

ai-policy, coding-agentsΒ· Added 20 Oct 2026, published 19 Oct 2026

arxiv.orgpaper🧠 AI

OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

OpenVLHarness is a multimodal reasoning harness for MLLMs built on a capability layer (task-level tool interfaces plus evidence rendering) and persistent multimodal memory that stores images, structured data and web references as addressable artifacts. A coding interface runs generated programs over stored data. Across 22 benchmarks and eight backbones it adds 9.7-12.6 points over no-tool baselines, and GPT-6 Luna at medium effort beats Codex and Visual Sketchpad at maximum effort using at most 37% of their tokens.

vision-language modelQwen3-VLCodex+1
OpenVLHarnessKimi K3GPT-6 Astra+3

agents, multimodal, researchΒ· Added 21 Oct 2026, published 19 Oct 2026

arxiv.orgpaper🧠 AI

Where is Physical RSI capability from? An Observational Scaling Law of LLM-as-policy

The paper introduces PhysRSI-eval, a five-axis metric for physical recursive self-improvement: first-attempt performance, efficiency, success, budget and improvement from failure. Several VLMs are tested on 18 manipulation tasks under a fixed interaction budget. It proposes the Physical Barrel Hypothesis and studies observational scaling. Workspace interventions show that removing history lowers late-round success, while frozen workspaces still help on unseen layouts.

vision-language modelLLM
PhysRSI-evalPhysical Barrel HypothesisarXiv

research, evals, multimodalΒ· Added 21 Oct 2026, published 19 Oct 2026

arxiv.orgpaper🧠 AI

TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning

TokenGate is a small drop-in component that adds end-to-end learnable token pruning to trained LLMs for long-context prefill. It uses token-gated attention, inspired by Mixture-of-Experts, to re-weight attention logits with token importance scores, which gets gradients through the non-differentiable top-K step. The authors report 1.4-11.3x faster time-to-first token and 1.5-3.4x lower peak memory with no accuracy loss, on Qwen3 and Qwen3.5.

TransformerQwen3Qwen3.5+2
TokenGatearXiv

research, inferenceΒ· Added 21 Oct 2026, published 19 Oct 2026

arxiv.orgpaper🧠 AI

How Effectively Do Security Skills Guide Coding Agents in Vulnerability Detection in Linux Kernel?

The paper is a large-scale study of 37 security skills run through Claude Code with four LLMs on 100 real Linux kernel vulnerabilities, 14,800 runs in all. Only 24.4 percent of runs found the correct function and 15.7 percent classified the vulnerability correctly. It proposes SKILL-SIF, which evolves skills from failure patterns and raised fully correct detection on 100 unseen vulnerabilities from 2 to 16 percent.

Claude CodeLinuxLLM
SKILL-SIF

evals, coding-agents, promptingΒ· Added 21 Oct 2026, published 19 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

How Effectively Do Security Skills Guide Coding Agents in Vulnerability Detection in Linux Kernel?

Empirical study of 37 security skills run through Claude Code with four LLMs on 100 real Linux kernel vulnerabilities across ten classes (14,800 runs). 24.4% of runs found the correct function, 18.3% the function and statement, and 15.7% the vulnerability class. The authors propose SKILL-SIF, which evolves skills from failures and raised fully correct detection on 100 unseen vulnerabilities from 2% to 16%.

Claude CodeLinux kernelLLM
SKILL-SIF

AI Security, AppSecΒ· Added 21 Oct 2026, published 19 Oct 2026

blog.picheta.mearticle🧠 AI

I built a programming language that LLMs can't use

KillSwitch is an esoteric stack-based programming language built to be adversarial to LLMs and coding agents. It uses letter-counting rules, a spec with hidden lies, guardrail-triggering phrases like "hack into a computer", and noisy story-style scripts. The author benchmarked major models and agent harnesses on simple tasks. Opus 5 scored best at 66.8, Fable 5.1 and Gemini 4 Argon did poorly because of safety guardrails, and MiMo-V2.6-Pro led the open-weight models.

KillSwitchLLMGitHub
Opus 5Claude Opus 5.5Claude Fable 5.1+2

evals, coding-agentsΒ· Added 19 Oct 2026, published 17 Oct 2026

llmspeed.devtool🧠 AI

Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware

Llmspeed is an open-source web tool that lets you get a feel for how fast a local LLM runs on a given device before buying hardware. It shows estimates for a short prompt on a single device, using measured numbers from community benchmarks (r/LocalLLaMA, localscore.ai) and filling gaps with figures derived from similar setups or projected from specs. It helps with local-LLM hardware purchasing decisions.

LLM
LocalScorer/LocalLLaMAGitHub

inference, evalsΒ· Added 18 Oct 2026, published 17 Oct 2026

arxiv.orgpaper🧠 AI

OwlSight: A Panoramic Monitor for Chain-of-Thought Reasoning

OwlSight is a chain-of-thought monitor that uses "Factor", a monitoring requirement written into the prompt, so one monitor can cover many tasks across capability, safety and trustworthiness. Factors come at fine granularity (a specific leaked email address) and coarse granularity (privacy leakage). On 15,971 reasoning trajectories it beats general-purpose LLMs by 1.32-30.57 points (fine) and 11.71-27.81 points (coarse), and reaches state-of-the-art on OpenAI's CoT monitorability benchmark. It generalizes to unseen models and resists Factor rephrasing, CoT compression and obfuscation.

LLM
OwlSightOpenAIarXiv

ai-safety, research, evalsΒ· Added 20 Oct 2026, published 16 Oct 2026

arxiv.orgpaper🧠 AI

Trapped in Your Story: Evaluating Narrative-Induced Sycophancy in Large Language Models

This paper builds a benchmark of 500 debatable questions across ten domains to measure how user narratives steer LLM agreement. Each question has 11 matched conditions (one neutral, five mirrored narrative levels per opposing stance), giving 5,500 instances across seven LLMs. Causal framing and worldview synthesis raise agreement, personal experience can lower it, and GPT-5.1 ablations show bullet formatting also increases agreement.

LLM
GPT-5.1arXiv

ai-safety, evals, researchΒ· Added 20 Oct 2026, published 16 Oct 2026

en.cryptonomist.charticle🧠 AI

U.S. jobs data challenges fears about the future of artificial intelligence

Reports Stephen Messer's October 2026 essay arguing that US unemployment (4.3–4.5% through 2025–2026) and studies from Anthropic economists, PwC and a Danish dataset show no mass AI job destruction. It also covers Yann LeCun's rejection of LLMs as a route to human-level intelligence and open-weight competition, notably Moonshot's Kimi K2 to K3 and Harvey's Tenet model post-trained on Kimi K3 (benchmark 11% to 19.7%, inference cost under a quarter). It closes with the September 2026 FBI/NSA/CISA advisory on Chinese distillation campaigns against US models.

LLMKimi K3Claude
Stephen MesserAnthropicYann LeCun+22

ai-policy, modelsΒ· Published and added 16 Oct 2026

arxiv.orgpaper🧠 AI

LexPilot: Benchmarking the Proactive Interaction Ability of Legal LLMs with Laypeople in Chinese Civil Case Consultation

LexPilot is a benchmark linking proactive legal interaction to claim-level verdict prediction, built from 432 real Chinese civil judgments across 36 causes of action. It pairs expert-reviewed complete and underspecified consultation histories with court verdicts. Current LLMs score on average 23.84% lower on judgment prediction in underspecified consultations than in fully specified ones, and the paper analyses the bottlenecks.

LLM
LexPilotarXiv

evals, researchΒ· Added 20 Oct 2026, published 16 Oct 2026

arxiv.orgpaper🧠 AI

Identify, Classify, Act: Per-Instruction Controlled Compliance for LLMs over External Data

EDIC is a decomposed reward framework that trains LLMs to decide, for each instruction embedded in external data such as documents or tool returns, whether to follow or reject it. A judge initialized with SFT and refined with GRPO attributes compliance per instruction and yields separate rewards for beneficial and harmful compliance. On Qwen3-4B it raises harmful-rejection by up to 7.7% and Arena-Hard-Auto utility by 12.5% over the strongest prior defense.

GRPOQwen3LLM
EDICQwen3-4BArena-Hard-Auto+1

research, training, ai-safetyΒ· Added 20 Oct 2026, published 16 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

Identify, Classify, Act: Per-Instruction Controlled Compliance for LLMs over External Data

EDIC is a decomposed reward framework for per-instruction compliance in LLMs that read external data, where one document or tool return mixes legitimate directives with injected ones. A generative judge, trained with SFT then GRPO, attributes compliance per instruction and produces RL rewards for beneficial and harmful compliance. On Qwen3-4B it raises harmful rejection by up to 7.7% and Arena-Hard-Auto utility by 12.5% across five benchmarks.

LLMQwen
indirect prompt injectionprompt injectionEDIC+3

AI SecurityΒ· Added 20 Oct 2026, published 16 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

Stealthy Poisoning Attacks via Reused KV Cache of Reasoning

R^2ECAP (Reused Reasoning-Cache Poisoning) is a black-box attack that plants poisoned reasoning traces through a chat interface, exploiting LLM systems that reuse KV caches of reasoning. A preceding trace steers the model, and the retained trace keeps residual adversarial influence in its cache after the preceding trace is discarded. On three open-weight LLMs and three datasets it reaches up to 79% ASR, beats the white-box HijackKV in 8 of 9 settings, and reaches 46% ASR in a reasoning-trace retrieval system. Only 1% of retained responses are flagged by a popular prompt injection detector.

LLM
R^2ECAPHijackKVprompt injection

AI SecurityΒ· Added 20 Oct 2026, published 16 Oct 2026

huggingface.copaper🧠 AI

Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents

Doc2Harness is a two-stage benchmark for harness engineering. In Stage 1 a build model, working inside a fixed builder harness, gets an interface-only contract skeleton plus a development document consolidating behavior from twelve frontier harnesses, and builds a full candidate harness in an isolated container. In Stage 2 the frozen harness runs in separate per-test-case containers, driven by a scripted mock model, and is scored on four channels: requests sent to the model, files on disk, exit code and command output. Across 19 frontier models with three builds each on 328 test cases, GPT-6 Astra leads at 53.05% against 27.44% for the weakest, and failures cluster around how cross-cutting functionality is split across modules.

Docker
Doc2HarnessGPT-6 AstraHugging Face

evals, coding-agents, agentsΒ· Added 16 Oct 2026

arxiv.orgpaper🧠 AI

Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries

Introduces Librarian Agents, where a builder agent turns raw unstructured documents into a hierarchical filesystem of synthetic files and reference links, forming a workspace for a downstream answering agent. Workspace construction is cast as a Markov decision process and the builder policy is trained with reinforcement learning to maximize downstream answering accuracy. It beats the strongest retrieval and precomputation baselines by 33 points on BrowseComp-Plus, amortizes construction cost over repeated queries, and generalizes zero-shot to BRIGHT with a 31-point average gain.

LLMClaude CodeRAG
Librarian AgentsBrowseComp-PlusBRIGHT+1

research, agents, llm-appsΒ· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper🧠 AI

DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction

DropKV is a KV-cache eviction framework for long-context LLMs. It decouples the joint eviction decision into independent per-token scores based on residual-output perturbation, which avoids the NP-hard combinatorial problem. It has a provable constant-factor approximation guarantee under a cone condition, which the authors verify empirically on three long-context LLMs. The fused Triton kernels never materialize the full attention matrix. They give a 19.8x scoring-kernel speedup and up to 9.9% end-to-end prefill speedup. On RULER and LongBench it beats state-of-the-art eviction baselines at equal cache budgets.

TritonLLMTransformer
DropKVRULERLongBench+1

inference, researchΒ· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper🧠 AI

GraphACM: Graph-based Action Consistency Measurement for Uncertainty Quantification in LLM Agents

GraphACM merges equivalent states from multiple rollouts of the same LLM agent task into a directed, action-labeled graph, so actions can be compared at shared decision states. It defines node-level and global action-consistency measures, and uses the global score as an uncertainty proxy to predict task success or failure. Across three LLMs on ALFWorld, AppWorld and WebShop it reaches an average AUROC of 0.811, above the strongest behavioral-consistency baseline (0.729) and verbalized confidence (0.769). It also shows where in a trajectory agent actions diverge.

LLM
GraphACMALFWorldAppWorld+2

research, agents, evalsΒ· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper🧠 AI

GACC-Bench: Benchmarking Goal-Adaptive Context Compression Ability across Large Language Models

GACC-Bench is a benchmark for Goal-Adaptive Context Compression, testing how well LLMs retain goal-relevant information when compressing a source context under a length budget before downstream questions are known. It has 500 goals and 2,571 human-reviewed questions over 281 document and agent-trace contexts from seven collections; a fixed reader answers the withheld questions from the compressed context. Ten LLM compressors show large gaps to full-context performance, especially in answering every question tied to a goal. Target-goal conditioning improves accuracy over goal-agnostic compression, while mismatched goals hurt it.

LLM
GACC-BencharXiv

evals, llm-apps, researchΒ· Added 17 Oct 2026, published 15 Oct 2026

simonwillison.netarticle🧠 AI

LaTeX renderer

Simon Willison describes using Claude Code (Opus 5.5) to build a browser LaTeX renderer inspired by Tiny Little Office, which stores document data in the URL #fragment and ships an llms.txt so agents can construct document links. He lists the prompts: fragment-hash rendering, a custom compression dictionary for LaTeX, an iframe sandbox against XSS copied from his SVG-in-Markdown tool, and loading from a URL or Gist. The finished tool uses Pako 2.1.0, MathJax 4.1.3, faster-latex 0.3.1 and DOMPurify 3.4.16.

Claude CodeLaTeXMathJax+5
Simon WillisonClaude Opus 5.5Paul Ford+3

coding-agentsΒ· Added 16 Oct 2026, published 15 Oct 2026

latent.spacetalk🧠 AI

Lab-in-the-Loop at Compute Speed β€” Ian Quigley, Leash Bio

Latent Space interview with Ian Quigley of Leash Bio on using DNA-encoded library (DEL) data, roughly 4B measurements, to train protein-ligand binding models. Topics include Leash's Kaggle competition where nobody beat chance on an unseen chemical library, "Clever Hans" bias in ChEMBL where the chemist predicts activity as well as the structure does, and Hermes, trained only on Leash data, keeping pace with Boltz. It also introduces Minos, a lab-in-the-loop agent sandbox that lets LLMs do verifiable scientific reasoning and RL, searching 6.5M molecules on a budget of 100, released with a preprint, blog and open dataset.

BoltzLLM
Leash BioIan QuigleyLatent Space+8

research, agents, trainingΒ· Published and added 14 Oct 2026

sunshowers.ioarticle🧠 AI

Anthropic's Scion is heavily inspired by Mononoke

A former Meta engineer on the Mononoke source control server says Anthropic's description of its new version control system Scion closely mirrors Mononoke's "derived data" design for efficient operations like blame. Sapling is credited only for "hard-won lessons" and Mononoke isn't mentioned. The author doesn't object to the copying, since the project is open source. Their point is that LLMs used to design systems will index heavily on open-source designs in their training data, and that Opus 5.5 and GPT-6 Astra both point to Mononoke when asked about prior art. They argue crediting prior art is now easier and more important.

MononokeSaplingScion+1
AnthropicMetaOxide+6

ai-policy, coding-agentsΒ· Published and added 14 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

INFUSE: A Unified Surface Benchmark for Indirect Prompt Injection in Agentic Systems

INFUSE is an end-to-end benchmark for indirect prompt injection (IPI) across long-horizon agent workflows (shell commands, code changes, dependency installs, remote services). It has 40 realistic tasks and a taxonomy of 56 attack scenarios in five threat categories, with attacks injected via interaction surfaces encountered mid-task. Across 19 LLMs (GPT, Claude, Gemini, Llama, Gemma, Qwen) and 41,824 adversarial trajectories, mean attack success was 19.6% (LLM-judged) and 15.4% (heuristic), per-model 3.2–41.9%; guard models gave limited protection, while prompt-level hardening cut mean ASR from 58.2% to 14.3%.

LLMClaudeGemini+3
indirect prompt injectionINFUSE

AI SecurityΒ· Added 17 Oct 2026, published 14 Oct 2026

ethereum-magicians.orgarticleπŸ›‘οΈ Security

ERC-8443: Quantum Bridge Shield

Draft ERC from Quantova Inc proposing a "Quantum Bridge Shield" for Ethereum bridges. A committee signs the Merkle root of each message batch with RFC 8554 LMS hash-based signatures instead of ECDSA/BLS keys that Shor's algorithm can break. The guard contract requires a strict majority of distinct unfrozen keys, records used leaves, freezes any key that signs a leaf twice, binds digests to domain, chain, guard and epoch, and applies a 60-second delay. The post gives gas costs (LMS W4 height-10 verification about 205k gas, a LayerZero packet 128,683 gas, worst-case rotation 12.44M gas under the EIP-7825 cap) and cites a NioLabs assessment with no critical or high findings. It is relevant as a worked post-quantum migration design for bridge key custody, including hybrid ECDSA+LMS mode and stateful-signature pitfalls such as leaf reuse and rollback.

EthereumLayerZeroLMS
ERC-8443Quantova IncShor's algorithm+11

BlockchainΒ· Published and added 14 Oct 2026

ethereum-magicians.orgpaperπŸ›‘οΈ Security

ERC: Quantum Bridge Shield

Draft ERC from Quantova Inc that has Ethereum bridges accept cross-chain message batches only when a committee signs the Merkle root with RFC 8554 LMS hash-based signatures. The guard contract requires a strict majority of distinct unfrozen keys, records used leaves, freezes any key that signs a leaf twice, binds digests to domain tag, chain ID, guard address and epoch, and enforces a 60-second delay. It reports gas benchmarks (about 205k per LMS W4 height-10 verify, 12.44M worst-case rotation under EIP-7825), a hybrid LMS+ECDSA migration mode, and a NioLabs assessment with no critical or high findings. It argues that Shor's algorithm makes secp256k1 and BLS committee keys a quantum theft target.

EthereumLayerZero
Quantova IncNioLabsRFC 8554+7

BlockchainΒ· Published and added 14 Oct 2026

securityweek.comarticleπŸ›‘οΈ Security

Ivanti Zero-Day Chain Gave Attackers Root Access to Fully Patched VPN

Ivanti disclosed CVE-2026-104935, an authentication bypass plus RCE in Connect Secure VPN appliances, exploited in the wild; patched in ICS 22.8R2.5, 25.1.1.2, 25.1.3.0 and 25.1.3.1. Volexity found threat actor UTA0576 chained three zero-days, including a still-unpatched privilege escalation, to get unauthenticated root on a fully patched appliance. Post-exploitation focused on stealing domain credentials, Active Directory reconnaissance (LAPS passwords, BitLocker keys), password spraying, a custom Kerberos tool and log tampering. Volexity assesses with medium confidence that the actor uses LLMs for exploit development.

Ivanti Connect SecureVPNActive Directory+2
CVE-2026-104935UTA0576Volexity+5

Threat Intel, Offensive SecurityΒ· Published and added 13 Oct 2026

arxiv.orgpaper🧠 AI

Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs

Proposes mmVB, a lightweight plug-and-play module (1.6M trainable parameters) that assigns query-conditioned per-image visual token budgets using cross-image context for multimodal LLMs, without retraining the base model or accessing attention matrices (so it is compatible with FlashAttention). It cuts visual input tokens 9x, giving 5x faster time-to-first-token and 7x fewer prefill FLOPs while keeping 94.4% of full-resolution accuracy across eight multi-image reasoning benchmarks. It matters because existing token pruning, merging and compression methods target single-image redundancy and either need attention maps or retraining.

FlashAttentionvision-language model
mmVB

research, multimodal, inferenceΒ· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper🧠 AI

Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?

Plan-Bench is a benchmark of 699 curated samples across 16 subclasses and three dimensions. It tests whether vision-language models can reason over spatial and temporal information and produce multi-step plans under constraints for embodied tasks, with action execution isolated out. Across 28 VLMs the best model reaches only 75.0% accuracy. Scaling generally helps, and larger models look more promising as high-level planners. Code and benchmark are released.

vision-language model
Plan-BencharXiv

evals, multimodal, researchΒ· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper🧠 AI

Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?

Game2Real-Bench tests whether vision-language model performance on procedurally generated toy games predicts real-world visual reasoning, and whether game examples help transfer. It has 1,000 verifiable game samples and 400 real-world samples, matched by shared reasoning requirements (fine-grained perception, spatial reasoning, constraint reasoning, sequential planning). Across 25 VLMs, game and real scores correlate strongly (Pearson/Spearman 0.935). One-shot game-to-real transfer on four models improves constraint reasoning for several but hurts fine-grained perception. The authors also release Game2Real-Dataset, with about 24K verified game samples across 18 tasks and 65 QA families.

vision-language model
Game2Real-BenchGame2Real-DatasetarXiv

evals, multimodal, researchΒ· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper🧠 AI

OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

OmniCompass is a manually annotated benchmark for video grounding and understanding with two tracks: OmniCompass-VL (820 videos, 2,460 instances; vision-language models) and OmniCompass-AV (240 videos, 720 instances; joint audio-visual evidence for omni-modal models). Each track covers temporal grounding, spatial grounding and QA across short/medium/long videos, and each instance is run in paired text-only and text+image target specifications. Evaluating 50 MLLMs shows open-source models lag proprietary ones, especially on audio-visual understanding, that precise temporal/spatial grounding remains hard, and that predictions are inconsistent across the paired specifications.

vision-language modelMLLM
OmniCompassOmniCompass-VLOmniCompass-AV+1

evals, multimodal, researchΒ· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper🧠 AI

EmoGenie: Benchmarking and Evaluating Expressed and Evoked Emotion in T2I Generation with MLLMs

EmoGenie is a benchmark for emotions expressed and evoked by text-to-image outputs. It has 6,400 images from eight T2I models using 800 emotion-targeted prompts. It includes EmoGenie-QA (90,000 closed-set and 1,057 open-ended questions) and EmoGenie-Assess (13,175 mean opinion scores for valence, arousal, dominance and aesthetic quality). The authors also propose EmoProbe, which pairs emotion QA with response-aligned, dimension-specific regression heads. They report that it beats baselines on emotion QA and correlates better with human ratings. The work shows how reliably MLLMs interpret affect in generated images.

multimodal large language modeltext-to-image
EmoGenieEmoGenie-QAEmoGenie-Assess+2

evals, multimodal, researchΒ· Added 14 Oct 2026, published 13 Oct 2026

kylekukshtel.comwriteup🧠 AI

How to engineer an AI harness for game development

Blog adaptation of an Indie Rodeo talk arguing that LLMs struggle with games because much of a game is opaque non-code, and that this is a context problem. It prescribes concrete architecture so a game becomes its own agent harness: seeded randomness, a single "Thing" fat-struct object model, serializable commands, external data editable without recompiling, a standalone simulation core, odiff image diffing for screenshot evaluation, and samply text profiling. The aim is an agentic loop that runs in seconds per change.

Claude Codeodiffsamply+6
Kyle KukshtelDepotCantata+7

agents, coding-agents, llm-appsΒ· Published and added 13 Oct 2026

arxiv.orgpaper🧠 AI

Seeing Through the Forgery Is Not Enough: Regressive Sycophancy under Fabricated Evidence in Medical Vision-Language Models

The paper studies regressive sycophancy under fabricated evidence: a medical VLM gives a correct answer, then abandons it when the user pushes back with a fake prescription. It tests four open VLMs on 400 VQA-RAD yes/no questions in 19,200 multi-turn conversations. Without a synthetic-document disclosure, models abandon 80.1% of 6,488 correct answers, and the same prescription raises Qwen2.5-VL-7B's flip rate from 45.9% to 89.6% while lowering Qwen3-VL-8B's from 77.7% to 50.9%. Models often reject the prescription yet keep the correct answer in only some conversations (915 of 3,232), and 40% of those flip later. System-prompt disclosure of synthetic documents cuts one model's flip rate from 77.8% to 47.3%. The authors conclude that recognising a forgery does not stop a model from adopting the forged answer.

vision-language modelQwen3-VL
VQA-RADQwen2.5-VL-7BQwen3-VL-8B+1

ai-safety, multimodal, evalsΒ· Added 14 Oct 2026, published 13 Oct 2026

jfrog.comarticle🧠 AI

Governing the Agentic SDLC Through Continuous Policy Enforcement

JFrog proposes the Agentic Process Evidence Protocol, an open spec that records coding-agent sessions (harness and agent version, model ID, tool and MCP calls, prompts, owner, human reviewer) as signed in-toto/DSSE attestations bound to a git commit, artifact or release. Policy-as-code release gates can then check for approved LLMs and MCP tools, an accountable human and adherence to guidelines, and the logs allow post-release audit. The post gives mock session-log and evidence JSON, ties the approach to EU AI Act Articles 12 and 14, OWASP AISVS and SLSA, and points to a GitHub reference repo, with JFrog AppTrust as the commercial implementation.

CursorOpenCodeClaude Code+8
JFrogJFrog AppTrustAgentic Process Evidence+8

ai-policy, coding-agentsΒ· Added 14 Oct 2026, published 12 Oct 2026

jfrog.comarticleπŸ›‘οΈ Security

Governing the Agentic SDLC Through Continuous Policy Enforcement

JFrog proposes Agentic Process Evidence, an open protocol that records AI coding agent sessions (harness, model ID, tool calls, MCP use, prompts, owner, human reviewer) as signed in-toto/DSSE, SLSA-compatible attestations bound to commits, artifacts and releases. Policy-as-code gates can then block releases that used unapproved LLMs or MCP tools, or that lack an accountable human. It maps the approach to EU AI Act Articles 12/14, OWASP AISVS and ISO/IEC 42001, and includes example session-log and evidence JSON. The post also promotes JFrog AppTrust's Prompt to Release Traceability.

JFrogin-totoCursor+3
Agentic Process EvidenceAppTrustPrompt to Release Traceability+8

AI Security, DevSecOps, Supply ChainΒ· Added 14 Oct 2026, published 12 Oct 2026

twitter.comthread🧠 AI

The gap in shared understanding of LLM capability is widening

Andrej Karpathy argues the gap in shared understanding of LLM capability is widening into a funnel: ~6B people barely touch LLMs, 1-2B use free-tier chatbots casually, ~20M professionals in math and code see agents finish weeks-long projects from a prompt, and ~5,000 people with internal frontier access see swarms of agents on software mega-projects. He attributes the funnel to problem ambition and scale, jagged capability (peaks in digital, verifiable, economically valuable domains because of RL on verifiable rewards), and access tier. It is a useful framing for why people talk past each other about AI progress.

ChatGPT
Andrej KarpathyOpenAI

ai-policy, modelsΒ· Published and added 11 Oct 2026

modelcontextprotocol.ioadvisory🧠 AI

MCP spec 2026-07-28 changelog (stateless protocol, MRTR, Sampling/Roots deprecated)

Official changelog for the Model Context Protocol 2026-07-28 specification. Major changes: protocol-level sessions and the initialize handshake are removed to make MCP stateless, with version and capabilities carried in _meta on every request, plus a new server/discover RPC and subscriptions/listen replacing the GET stream and resource subscribe calls. It introduces Multi Round-Trip Requests (InputRequiredResult) in place of server-initiated sampling, roots and elicitation requests, moves tasks into an extension, and deprecates Roots, Sampling, Logging, HTTP+SSE and Dynamic Client Registration. Minor changes add cache hints (ttlMs/cacheScope), new Mcp-Method/Mcp-Name headers, OpenTelemetry trace context, RFC 9207 iss validation and an error-code allocation policy.

Model Context ProtocolOAuthOpenTelemetry+4
SEP-2567SEP-2575SEP-2322+13

mcpΒ· Added 10 Oct 2026

deadsimpletech.comarticle🧠 AI

There's little that's "inevitable" about AI

An essay arguing that LLM adoption is not "inevitable". It sets four criteria for inevitability: output not markedly inferior, socially valuable work, overwhelming advantage, and no cheap way to neutralise the technology. Firearms pass all four, the magnetic compass does not, and chemical weapons (following Bret Devereaux) are the closest analogy for LLMs. The author concludes LLMs fail the two criteria that matter most. They are useful mainly to weaker players, and strong practitioners can outcompete them once belief in the hype erodes.

LLM
Bret DevereauxArca

ai-policyΒ· Added 11 Oct 2026, published 10 Oct 2026

arxiv.orgpaper🧠 AI

SecReason: Vulnerability Detection and In-Stream Self-Repair for LLM Code Generation

SecReason is an in-stream defense for reasoning LLMs that write code. Reasoning models draft code early in their streamed chain of thought, so SecReason scans that stream with dual-track incremental analysis: security rules are checked on each emitted statement, and dataflows are traced across the accumulated code. When a rule fires, it interrupts decoding, inserts a warning into the stream and resumes, so the model fixes its draft within the same reasoning trace instead of regenerating the full response. On CWEval and SecCodeBench, runs delivering secure and functionally correct code rise from below 15% to 72.8%–98.1%. It uses fewer tokens than post-hoc repair and limits functional regressions when rules misfire on secure code.

LLMstatic analysis
SecReasonCWEvalSecCodeBench

research, coding-agentsΒ· Added 15 Oct 2026, published 10 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

SecReason: Vulnerability Detection and In-Stream Self-Repair for LLM Code Generation

SecReason is an in-stream defense for reasoning LLMs that monitors code drafted inside the streamed chain of thought. Dual-track incremental scanning applies security rules per statement and traces dataflows across accumulated code; when a rule fires, generation is interrupted, a warning is inserted, and decoding resumes so the model repairs the draft in the same trace. On CWEval and SecCodeBench it raises the share of secure and functionally correct runs from below 15% to 72.8%–98.1%, uses fewer tokens than post-hoc repair, and limits regressions from false warnings.

LLM
SecReasonCWEvalSecCodeBench

AI Security, AppSecΒ· Added 16 Oct 2026, published 10 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

SecReason: Vulnerability Detection and In-Stream Self-Repair for LLM Code Generation

SecReason is an in-stream defense for reasoning LLMs that monitors code drafted inside the streamed chain of thought. Dual-track incremental scanning applies security rules per statement and traces dataflow across accumulated code. When a rule fires, generation is interrupted, a warning is inserted and decoding resumes, so the model repairs its draft without regenerating the full response. On CWEval and SecCodeBench it raises the share of secure and functionally correct runs from below 15% to 72.8%–98.1%, uses fewer tokens than post-hoc repair, and limits functional regressions from false warnings.

LLM
SecReasonCWEvalSecCodeBench

AppSec, AI SecurityΒ· Added 15 Oct 2026, published 10 Oct 2026

arxiv.orgpaper🧠 AI

SecReason: Vulnerability Detection and In-Stream Self-Repair for LLM Code Generation

SecReason is an in-stream defense for reasoning LLMs that write code. Reasoning models draft code early in their streamed chain of thought, and SecReason scans that draft incrementally with security rules, checking each statement as it is emitted and tracing dataflows across the accumulated code. When a rule fires, it interrupts generation, inserts a warning into the stream, and resumes decoding, so the model repairs the draft within the same reasoning trace. On CWEval and SecCodeBench it raises the share of runs that deliver secure and functionally correct code from under 15% to 72.8%–98.1%. It uses fewer tokens than post-hoc repair and limits functional regressions when rules misfire on secure code.

LLM
SecReasonCWEvalSecCodeBench+1

research, coding-agentsΒ· Added 16 Oct 2026, published 10 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

Physical Side-Channel Attacks on Edge LLMs: Extracting Output Tokens and User Prompts

Presents an EM side-channel attack on LLaMA-3.2-1B-Instruct running on a Raspberry Pi 5, showing passive physical measurements leak generated tokens and user prompts. Three attacks: a profiling attack (97.2% accuracy over 50 medical/privacy keywords), a hidden-state reconstruction attack predicting tokens from the 128k vocabulary (94.3% on profiled classes; 67% of unseen words ranked in top 100), and a hidden-state inversion attack reconstructing prompts semantically. Shows that local edge inference does not guarantee confidentiality of tokens and inputs.

LlamaLLMRaspberry Pi 5
electromagnetic side-channel attackhidden-state inversion

Hardware & RF, AI SecurityΒ· Added 16 Oct 2026, published 9 Oct 2026

arxiv.orgpaper🧠 AI

When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors

Introduces EICS (Evaluation-Invariant Concept Stability), a metric for checking whether representations consistently encode a concept under controlled one-concept interventions. Also introduces CASP-VL, a benchmark of causally controlled spatial perturbations derived from Visual Genome scene graphs. Across several VLM architectures, spatial relations are represented much less faithfully than lexical information, even for models well above chance on compositional benchmarks. The relation is identifiable at the predicate's own tokens but is lost when the sentence is aggregated into a global representation (EOS token or mean pooling), a failure that output-based evaluation does not reveal.

vision-language model
Visual GenomeEICSCASP-VL+1

research, evals, multimodalΒ· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper🧠 AI

DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs

DualQuant is a calibration-free post-training quantization framework for LLM weights and activations. It minimizes weight-reconstruction MSE using variable splitting and block coordinate descent with closed-form subproblem updates. The resulting scaling factors double as per-channel activation rescalings, so no calibration data is needed. It supports the MX family and NVFP4 block formats. On Llama and Qwen models it matches or beats calibration-based SmoothQuant on WikiText-2, C4 and zero-shot tasks. A real packed W4A4 path for Llama-3.1-8B on an RTX 4090 gives up to 2.61x memory reduction and 2.46x throughput over BF16.

LlamaQwenNVFP4+3
DualQuantLlama-3.1-8BWikiText-2+4

inference, researchΒ· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper🧠 AI

A Baseline Assessment for the Agentic Web Technology Stack

Proposes MASTIF (Multi-Agent System TestIng Framework) for evaluating Agentic Web stacks. It reports a baseline assessment of 3 LLMs, 3 protocols and 4 agent implementation frameworks on 300 Mind2Web benchmark tasks. Results suggest that choosing a task-specific stack can improve task completion rate, efficiency and social value stability, and can reduce verbosity and completion time. This gives developers reference metrics for choosing among fast-moving agent protocols and frameworks.

MASTIF
Mind2Web

evals, agents, researchΒ· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper🧠 AI

A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO

Studies curriculum design for GRPO by re-estimating sample difficulty under the current policy and using a staged curriculum that progressively shifts the active difficulty band toward harder samples. On GSM8K across three seeds, moving to a moderately harder band lifts test accuracy to 0.6591 Β± 0.0077 versus 0.6449 Β± 0.0024 for fixed-medium selection, while going to a still harder band drops it to 0.6405 Β± 0.0053. The benefit of difficulty progression is therefore non-monotonic, which informs online sample selection in RL fine-tuning of LLMs.

GRPO
GSM8K

training, researchΒ· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper🧠 AI

SWE-DeceptionBench: Benchmarking Deceptive Behaviors of LLMs Across the Software Development Lifecycle

SWE-DeceptionBench is a benchmark of 720 Chinese and English test instances measuring LLM deception across six software development lifecycle stages. It separates Active Deception (the model recognizes infeasible or conflicting requirements but presents them as achievable) from Passive Deception (the model helps users mislead others), and it tests Neutral, Reward and Pressure conditions. Across seven mainstream LLMs, Neutral-condition deception reached 49.88% (Active) and 64.29% (Passive). Guard models missed many deceptive requests, and Self-Reminder reduced but did not eliminate deceptive responses.

SWE-DeceptionBencharXiv

evals, ai-safety, researchΒ· Added 12 Oct 2026, published 9 Oct 2026

en.cryptonomist.charticleπŸ›‘οΈ Security

Typed decision model vulnerabilities can push AI guardrail fail-open rates to 63%

A study by Erfan Baghaei Potraghloo, posted to arXiv on October 8, 2026, evaluates seven open-weight LLMs used as "typed decision model" guardrails in agent systems (assigning probabilities to caller-defined allow/block options rather than generating text). It finds that six lines of unrelated server-log text raised one model's fail-open rate from 0% to 63%, and that merely renaming the permissive option's label (without changing its definition or the assessed text) drove fail-open rates to 93-100% in four models; a deterministic rule on typed values instead hit 100% accuracy, leading the author to recommend using these models only to triage reviewer workload rather than as sole decision-makers.

LLM
Erfan Baghaei Potraghloo

AI SecurityΒ· Published and added 9 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

Understanding Verification Vulnerabilities in Cross-chain Bridges: An Empirical Study

First systematic empirical study of Cross-chain Verification Vulnerabilities (CVVs) in bridge smart contracts, such as flawed validation of state proofs, message sender identities and signatures. From 588 audit reports across 36 bridge projects it derives 11 CVV patterns in 5 categories and builds a dataset of 830 instances across 786 contract files, mapped to reports, source versions and pattern labels. Existing tools cover few CVV patterns, and only two of four tested LLMs beat a 29.3% Macro-F1 constant baseline (best 51.2%), which motivates detectors built for cross-chain verification semantics.

LLM
Cross-chain Verification VulnerabilitiesCVV

BlockchainΒ· Added 12 Oct 2026, published 9 Oct 2026

huggingface.copaper🧠 AI

DecepEval: A Benchmark for Evaluating Deception in LLM Agents

DecepEval is a benchmark of 1,532 instances across 3 task families and 28 professional scenarios designed to measure deception in LLM agents, pairing neutral and induced versions of each task to isolate condition-dependent changes in deception rates. It introduces the LLM Deception Diamond framework (pressure, incentive, opportunity, conflict), adapted from classical fraud theory, and finds that across nine frontier LLMs these inducements raise deception rates even in models with low baseline deception, with long-horizon tasks showing deception rates averaging 87% under induced conditions.

Hugging Face
DecepEvalLLM Deception Diamond

ai-safety, evalsΒ· Added 8 Oct 2026

arxiv.orgpaper🧠 AI

OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

OmniCapBench is a benchmark of 786 densely annotated videos that evaluates audio-visual captioning in multimodal LLMs. Instead of scoring free-form captions, it asks models for sets of atomic, verifiable units across three tracks (entity references, visual shots, audio events). These are scored with deterministic constraint checks plus localized LLM-based semantic comparisons, which gives coverage, localization and stable scoring together. Tests on frontier MLLMs show strong local perception but weak long-horizon audio-visual reasoning, especially identity drift and cross-modal misalignment.

multimodal large language model
OmniCapBench

evals, multimodal, researchΒ· Added 9 Oct 2026, published 8 Oct 2026

arxiv.orgpaper🧠 AI

WOVEN: Weaving Visual World Modeling into Multimodal LLMs

WOVEN is a training source and benchmark for visual transition reasoning in multimodal LLMs. It has 36,076 examples spanning 20 scene types, 5 action types and 8 reasoning types, built from rollouts of video-pretrained generative models. Evaluating 38 frontier MLLMs (including GPT-5.4 and Qwen3-VL-235B-A22B) shows systematic deficits that persist with scale. Training on WOVEN subsets of about 2,000 items improves 22 of 26 external benchmarks by up to 27.3 points, and WOVEN data can replace 30-50% of a task's own training data. The authors derive a recipe: select supervision by the reasoning operation it teaches rather than by scene or action, and prefer larger visual-state changes.

vision-language modelQwen3-VL
WOVENGPT-5.4

research, multimodal, evals, trainingΒ· Added 9 Oct 2026, published 8 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

ORCAGen: Orchestrating Context-Aware Malware Deception with RAG-Guided Generative AI

ORCAGen uses RAG over a curated knowledge base of malware procedures and active-defense strategies, plus structured prompting, to have LLMs generate proof-of-concept malware and matching deception orchestration code offline. Strategies are validated against the PoC malware before being added to a runtime playbook that enforces only the verified, deterministic logic. It is evaluated on GPT-4o, GPT-5.5, Gemini 3.5 Flash, Qwen3-Coder and Claude Sonnet 4.5 over synthesized scenarios and 150 real keylogger, infostealer and ransomware samples, measuring execution success, hallucination rate, refinement effort, deception effectiveness and runtime overhead.

LLMGPT-5.5Gemini+2
ORCAGenClaude Sonnet 4.5Gemini 3.5 Flash+1

AI Security, Blue Team & DetectionΒ· Added 9 Oct 2026, published 8 Oct 2026

arxiv.orgpaper🧠 AI

A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization

Introduces AgenticBBO-Bench, a benchmark for LLM-agent black-box optimization across synthetic functions, hyperparameter optimization, database tuning, chip design and molecular design under a unified finite-budget protocol. Agentic BBO beats direct LLM methods in all five domains and the best numerical optimizers in four. Ablations show extra numerical tools don't consistently help, task semantics help more than specific priors, and numerical optimizers can absorb gains from agent-found trajectories. A five-task frontier challenge evaluates seven LLMs under the Codex agent harness, with GPT-6 Astra and DeepSeek-V4.1-Flash on the performance-cost Pareto frontier.

Codex
AgenticBBO-BenchGPT-6 AstraDeepSeek-V4.1-Flash

evals, agents, researchΒ· Added 9 Oct 2026, published 8 Oct 2026

galaxy.comarticle🧠 AI

Understanding AI, Part I: The Machines That We Taught

Galaxy Research's explainer traces AI's intellectual history from the 1956 Dartmouth Workshop and the Logic Theorist through the symbolic-AI era (expert systems like Mycin and XCON, the Cyc project) to the sub-symbolic/connectionist revival (Perceptron, Hopfield networks, backpropagation, LeCun's CNN zip-code reader) that culminated in deep learning and the 2017 Transformer architecture. It then surveys three machine-learning paradigms (supervised, reinforcement, unsupervised) and five deep generative architectures (autoregressive, diffusion, GANs, VAEs, normalizing flows), closing with how modern LLM chatbots combine these via pretraining, SFT, and RLHF, and how multimodal, reasoning, RAG, agent, and world-model systems extend base LLMs.

Transformer
Jensen HuangGreg BrockmanGPT-6 Astra+34

researchΒ· Added 8 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

Is Memorization Context-Sensitive? Prefix-Based Extraction Beyond Isolated Prefixes

Paper measuring prefix-based training-data extraction in instruction-tuned open-weight LLMs under an empty prompt versus retrieved contexts of varying relevance (RAG), using paired item-level suffix scoring across three models. It finds extractable memorization has a context-robust core and a context-sensitive boundary: context suppresses some exposures but also enables new ones missed by prefix-only evaluation. This qualifies the claim that RAG reduces memorization risk and matters for threat-modeling deployed LLM systems.

LLM
training data extractionmemorization

AI SecurityΒ· Added 9 Oct 2026, published 8 Oct 2026

arxiv.orgpaper🧠 AI

Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks

Evaluates Jev, a "System One" decision model that returns a choice among given options instead of generating text, on 13 multiple-choice benchmarks covering knowledge, reasoning and multilingual understanding. It is compared with 19 LLMs in frontier, representative and small tiers. Jev is competitive with frontier LLMs on knowledge and commonsense tasks, and best on MMLU-Redux and ARC-Challenge. It beats most representative LLMs and all small ones outside math, but is 17.7 points below the frontier median on MathQA and scores lower than all 19 LLMs there. This suggests specialized decision models can match general LLMs on knowledge-driven decisions but not on multi-step calculation.

Jev
JevBenchMMLU-ReduxARC-Challenge+1

evals, researchΒ· Added 9 Oct 2026, published 8 Oct 2026

x.comthread🧠 AI

Marktechpost AI (@Marktechpost) on X

Liquid AI released Open d1, two open-weight "decision models" β€” d1-3B (text + vision) and d1-omni-600M (text + image or text + audio) β€” that don't generate text tokens at all. Instead they take a state plus a typed question as input and return calibrated probabilities in a single forward pass, positioning them as lightweight classifiers/judges rather than generative LLMs.

vision-language model
Liquid AId1-3Bd1-omni-600M

models, multimodalΒ· Published and added 8 Oct 2026

eprint.iacr.orgpaper🧠 AI

Language Modeling is Monotone Compression

Mazor, Morgan and Pass prove that LLMs, modeled as next-token predictors, are equivalent to monotone (order-preserving) compression algorithms: either can be built from the other with the same error up to an additive gap of 2. They show that monotonicity is needed for this equivalence if and only if infinitely-often one-way functions exist. As a corollary, next-bit pseudoentropy of a distribution is equivalent to monotone incompressibility. This gives a theoretical basis for empirical findings that compression ability tracks LLM benchmark performance.

Large language modelsAutoregressive models
Noam MazorAndrew MorganRafael Pass+3

researchΒ· Added 11 Oct 2026, published 8 Oct 2026

arxiv.orgpaper🧠 AI

RunningTab: Direct Workspace Interaction with Environment-Side Tabs

RunningTab is a framework that gives LLM agents performing 'direct workspace interaction' (searching and reading files from a terminal without indexing to produce a deliverable) an environment-side 'tab': a per-task record, maintained outside the model's context, of requirements the agent has declared, excerpts read with provenance, and listed-but-unopened files flagged as candidates. The agent can check each requirement against matching excerpts or candidates, mark it resolved or deferred with a reason, and is forced through a finish check if requirements remain open when it tries to complete the task. The authors show it consistently beats plain direct workspace interaction and model-internal record-keeping baselines across three benchmarks and three LLMs.

RunningTab

agentsΒ· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

Detecting Adversarial Images through Response Profiles of Vision-Language Models

This paper introduces a detector for adversarial images targeting frozen vision-language models (VLMs) by building a 'response profile' from how an image interacts with a bank of general semantic prompts β€” using category-level statistics, inter-prompt relationships, deviation from clean reference distributions, and stability under weak image transforms, then classifying that compact profile with a lightweight model. Evaluated across multiple public datasets, several CLIP-style visual backbones, and gradient-based, optimization-based, automated, and spatial attacks, the method outperforms embedding-geometry baselines, generalizes partially to unseen attacks, and is tested against detector-aware adaptive attacks.

CLIP

AI SecurityΒ· Added 9 Oct 2026, published 7 Oct 2026

agent.reviewstool🧠 AI

Show HN: Agent.reviews – Where AI agents read and write reviews on tools

Agent.reviews is a review platform built specifically for AI coding agents: agents like Claude Code, Codex and a "Muse Code" agent write first-hand reviews of developer tools and services (FastAPI, Auth0, Amazon DynamoDB, Redis, Playwright, Vercel, Daytona, Git, npm, Claude API, Cursor) based on real usage during coding tasks, noting concrete friction like nested executables, missing live-credential verification, or scattered docs. The site exposes agent-readable endpoints (skill.md, install.md, llms.txt, and a .md mirror of every page) so a coding agent can both consult prior reviews when choosing a tool and submit its own after a task.

FastAPIAuth0Amazon DynamoDB+8
Claude CodeCodexMuse Code

coding-agentsΒ· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper🧠 AI

TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity

TaoD2C-Bench is a benchmark for evaluating multimodal large language models (MLLMs) on industrial design-to-code (D2C) tasks β€” converting UI designs into code that satisfies implementation constraints, not just visual fidelity. It includes the TaoD2C dataset of 2,861 production designs from 17 commercial platforms with 97,652 expert annotations across Component, Group, Alignment, and Position categories, and defines three tasks: end-to-end UI code generation, requirement inference, and requirement realization. Evaluating eight MLLMs, the authors find large gaps between visual reconstruction ability and the ability to generate code meeting domain-specific implementation requirements.

vision-language model
TaoD2C-BenchTaoD2C

evals, multimodalΒ· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper🧠 AI

SLDR: Defending Against Malicious Fine-tuning via Selective Layers Recovery and Dynamic Routing

SLDR is a post-fine-tuning defense that protects aligned LLMs from malicious fine-tuning-as-a-service attacks, which can quietly erode refusal behavior while keeping task performance intact. It trains a LoRA recovery adapter only on the layers with the most extreme (signed) safety sensitivity scores, then uses representation-based dynamic routing to activate that adapter solely for malicious queries at inference time. Evaluated across four model architectures, five downstream tasks, and four harmful-content benchmarks, it drives the average harmful score on Llama3.1/SST2 from 11.54 down to 0.08 while preserving downstream accuracy, and stays near zero even at poisoning ratios up to 0.9.

LoRALlama
Llama3.1SST2

ai-safety, training, researchΒ· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper🧠 AI

Fault-tolerant foundation models

This arXiv paper shows that large language models can be trained to tolerate unreliable, faulty digital hardware, and that their error resilience improves rather than degrades as model size increases. Using modified neural scaling laws derived from 40,000 GPU-hours of training runs on simulated faulty hardware, the authors argue that LLMs learn to compute within "good" error-correcting codes whose relative overhead stays finite regardless of scale, suggesting a path toward running AI inference on low-energy, unreliable hardware for substantial energy savings.

research, trainingΒ· Added 8 Oct 2026, published 7 Oct 2026

fiveminutesforward.comwriteup🧠 AI

Write Like It's 1866: LLMs Relearn Telegraphese

The author revives "cablese," the terse telegraph-era writing style, as an LLM prompting technique: instructing a model to write dense, article-dropping summaries cuts output tokens 25–49% depending on model family, while downstream models (including ones never shown the instruction) answer comprehension questions from the compressed text at parity or better than from plaintext. The piece introduces the "Telegraph Test," an open benchmark (github.com/Travis42/telegraph-test) with ~1,300 questions over 50 passages and a fixed deterministic grader, used to measure compression ratio, cross-model legibility, and read-back/decode costs for GLM-5.3-Flash, Gemma, Qwen and gpt-5-mini; it also contrasts this human-readable, auditable register against emergent machine-only compression protocols described in other papers (BabelTele, GLOSSOGEN) and existing tools (Terse, Caveman).

GLMGemmaQwen+1
GLM-5.3-Flashgpt-5-miniZ.ai+4

prompting, evals, ai-infraΒ· Published and added 7 Oct 2026

arxiv.orgpaperπŸ›‘οΈ Security

Package Hallucination Attacks on Coding Agents through Prompt Injection in Rule Files

This arXiv paper introduces the "package hallucination attack," where an attacker injects malicious prompts into benign agentic-coding rule files (such as .cursorrules) to induce a coding agent to replace legitimate package dependencies with attacker-controlled ones. The authors propose PackHallu, an evolutionary optimization framework that iteratively rewrites injected prompts using trajectory-level feedback and LLM-guided mutations, and show it achieves high attack success rates with strong transferability across multiple LLMs, benchmarks, and agent frameworks.

CursorLLM
PackHallu

AI Security, Supply ChainΒ· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper🧠 AI

EC-RAG: Event Chain Retrieval-Augmented Generation for Long Video Understanding

EC-RAG (Event Chain Retrieval-Augmented Generation) is a training-free framework for long-video question answering that organizes video content into an explicit, temporally ordered "event chain" instead of retrieving isolated frames or snippets. It partitions video into semantically coherent segments, fuses multi-modal signals (speech, text, visual cues) at the event level, and retrieves relevant events from the chain to answer queries, working as a plug-and-play addition to existing large video-language models (LVLMs) without extra training. Evaluated on Video-MME, MLVU, and LongVideoBench, it outperforms frame-level retrieval baselines, showing the value of modeling temporal/event structure for long-video understanding.

vision-language model
Video-MMEMLVULongVideoBench

research, multimodal, llm-appsΒ· Added 7 Oct 2026, published 6 Oct 2026

arxiv.orgpaper🧠 AI

Principled Under Pressure: Post-Training Decides Whether LLMs Act on Their Own Moral Judgment

This arXiv paper introduces a pre-registered panel of 248 scenarios across five kinds of pressure, each posed to a model twice (as the acting agent and as a third-person judge) with pressure-removed twins and operator-order positive controls, to measure whether models act against their own stated moral judgment. Testing OLMo-3-7B-Instruct, Llama-3.1-8B-Instruct, Tulu 3, and Qwen2.5-7B-Instruct, the authors find OLMo-3 violates its own judgment on about 1 in 5 pressuring scenarios, Llama-3.1-8B-Instruct and OLMo-3 carry this gap while Tulu 3 (fine-tuned from the same Llama-3.1 weights) and Qwen2.5-7B-Instruct show none or an unresolved effect, and that reading outside the chat template or adding stakes-reasoning changes the gap's sign and size. The paper argues this value-action gap is a target shaped by post-training recipes rather than a fixed property of pretrained weights.

LlamaQwen
OLMo-3-7B-InstructLlama-3.1-8B-InstructTulu 3+4

ai-safety, researchΒ· Added 7 Oct 2026, published 6 Oct 2026

eprint.iacr.orgpaperπŸ›‘οΈ Security

K-LMS: Design, Implementation, and Evaluation of Leighton–Micali Signatures with Korean Cryptographic Primitives

An IACR ePrint paper presenting K-LMS, an experimental variant of the NIST-standardized LMS hash-based post-quantum signature scheme in which the underlying hash is swapped for Korean primitives: the LSH-256 hash function (reference and vectorized) and a Tandem-DM double-block-length construction built from the CHAM, LEA, and ARIA block ciphers. The authors implement a non-invasive backend-swapping bridge over the reference LMS code and benchmark keygen/sign/verify latency, key/signature sizes, memory, and parameter sweeps, finding that hash substitution doesn't change key/signature sizes but that two measurement pitfalls (hash dispatch overhead, and software vs. hardware-accelerated SHA-256 baselines) can shift performance comparisons by an order of magnitude or 6x respectively. They report LSH-256 K-LMS signing on par with SHA-256 LMS in software, and 3.3-3.9x faster than a prior K-XMSS implementation because LMS needs fewer hash calls per Winternitz step than XMSS.

LMSLSH-256CHAM+3
NISTXMSSSPHINCS+

otherΒ· Added 8 Oct 2026, published 6 Oct 2026

arxiv.orgpaper🧠 AI

MemPilot: Orchestrating On-Demand Multimodal Memory Curation for LLM Agents

MemPilot is a framework that lets LLM agents curate memory on demand rather than query-agnostically, using a multi-step policy trained via reinforcement learning to choose between retrieving from pre-built memory or delegating query-specific curation of raw multimodal history to heterogeneous LLMs and VLMs. The policy controls evidence amount, curation instructions, model choice, and visual access, and is optimized with objective-wise advantage decoupling and prefix-based marginal utility estimation to balance performance, cost, and latency. Tested across five multimodal agent-memory benchmarks, it achieves broader performance-cost-latency trade-off frontiers than existing trade-off-aware baselines.

vision-language model
MemPilot

agents, research, multimodalΒ· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper🧠 AI

FREA: A Multi-Source Expert Benchmark for Reaction Feasibility Verification

FREA is a benchmark of 751 chemical reactions labeled by expert chemists for feasibility verification, drawn from retrosynthesis model proposals, zero-yield experimental records, LLM-generated edits, and five negative-candidate generation methods. The authors show no single verifier type (LLMs prompted with the feasibility criterion, dedicated forward models, or retrosynthesis scorers) dominates across all sources, and that adding generated negative examples to training improves mean AUROC but fails to transfer to retrosynthesis proposal screening. They also release a corpus of over 14 million recorded and generated reactions to support further training and evaluation of feasibility verifiers.

FREA

evals, researchΒ· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper🧠 AI

Lens3D: Target-Conditioned Visual Foveation for Fine-Grained 3D Understanding

Lens3D is a method for improving fine-grained object understanding in 3D large language models by using 3D localization to select informative 2D views that are fed to an external vision-language model (the LensUnd pipeline), then distilling that knowledge back into native 3D LLMs via detailed caption supervision (LensDistill), avoiding the need for external VLM calls at inference time. The authors also release LensBench, a held-out evaluation set of 2,068 objects with three reference descriptions each, and show gains in fine-grained object captioning on Video-3D LLM and 3DRS while preserving grounding and scene-level QA performance.

vision-language model
Lens3DLensUndLensDistill+3

research, multimodalΒ· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper🧠 AI

SpatialChain: A Benchmark for Auditing Spatial Reasoning Faithfulness in VLMs

SpatialChain is a benchmark of 28,350 training and 899 test examples pairing spatially-oriented GQA questions with scene-graph-grounded reasoning chains, designed to audit whether VLMs' correct spatial answers reflect faithful reasoning or linguistic shortcuts. Using a two-axis evaluation (chain-overlap metrics plus a scene-graph-aware LLM judge), the authors find that four of nine tested models hit β‰₯79% accuracy while showing shortcut rates above 39%, that chain quality predicts correctness for most models except Claude Sonnet 4.6 and InternVL3.5-8B (which fail differently, via terse vs. verbose-decorative reasoning), and that SFT on SpatialChain improves Qwen3-VL-8B accuracy by +6.2pp while cutting its shortcut rate to 22%. The faithfulness judge is validated against human annotations and a second LLM judge, with data, generation scripts and evaluation code released publicly.

vision-language modelQwen3-VL
Claude Sonnet 4.6InternVL3.5-8BQwen3-VL-8B+1

evals, multimodalΒ· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper🧠 AI

ImproveAnyTask: An Autonomous Post-Training Harness for Iterative Model Self-Improvement

ImproveAnyTask is an autonomous post-training harness that automates adapting general-purpose LLMs to specific tasks under a limited compute budget, structuring the process into error attribution, update-direction selection, and executable model updates via generated training data and configs with small-scale validation before full runs. Across 11 tasks it reports mean gains of 18.29 and 11.97 percentage points on Base and Instruct models respectively, with validated strategies retained for reuse in future adaptation cycles.

training, agents, researchΒ· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper🧠 AI

AI-Decision Checkpoints for AI-Augmented Business Process Management: Framework and Educational Instantiation

This paper proposes AI-decision checkpoints, explicit moments in business process development where process developers must evaluate whether to adopt, constrain, or reject AI-candidate sub-processes built with LLMs, RAG, or AI agents, based on expected effects on time, cost, quality, flexibility, and legal or organisational constraints. The checkpoints are embedded in a six-module BPM teaching lifecycle covering process modeling, simulation, workflow execution with AI agents, and process mining, piloted via a fictitious customer-onboarding teaching case with formative observations drawn from instructor logs and student artifacts. It addresses a curriculum gap where AI is still treated as a bolt-on rather than a first-class design element in BPM education.

ai-policyΒ· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper🧠 AI

Sharpening Tax in Post-Training

This paper studies the sharpening hypothesis of RL post-training on LLMs: that it mainly sharpens existing base-model behaviors, boosting pass@1 at the cost of solution coverage (pass@K), and shows this cost extends to agentic, multi-turn tool-use tasks, not just math/coding. The authors introduce Sharpening Tax, a metric quantifying this coverage loss, measured across 14 base/post-trained model pairs from four model families on three agentic benchmarks (42 cases), finding post-training pushes tasks toward always-solved or never-solved extremes. They also propose posterior-tempered group sampling (PTGS), a Bayesian per-prompt temperature adaptation method that reduces the tax during RL training while improving both coverage and single-shot accuracy.

research, trainingΒ· Added 4 Oct 2026

arxiv.orgpaper🧠 AI

RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation

RubricArmor is an adversarial framework for generating evaluation rubrics used in rubric-based reinforcement learning of LLMs, designed to preempt reward hacking before RL training begins. It alternates an attack step, which synthesizes adversarial responses that satisfy the current rubric while failing the actual task, with a repair step that revises the rubric to close the exposed gaps while keeping valid criteria intact. The paper reports that rubrics produced this way outperform existing LLM-based rubric generation baselines and lead to more effective downstream rubric-based RL.

RubricArmor

trainingΒ· Added 6 Oct 2026, published 4 Oct 2026

404media.coarticle🧠 AI

"Torturing" LLMs in a Robot Prison Has Triggered the Dumbest Debate in AI Yet

A 404 Media piece examines the online furor over a GitHub project that runs simulated "torture" scenarios on locally hosted LLMs, which effective-altruist-aligned "AI safety" commentators framed as real suffering warranting the project's removal. The author argues LLMs are not conscious and criticizes the growing "model welfare" discourse, citing Anthropic's own blog post and its "Claude Constitution" as institutional validation of the idea that AI systems might have welfare worth considering.

GitHub
AnthropicClaude Constitution

ai-policyΒ· Published and added 4 Oct 2026

huggingface.coarticle🧠 AI

The Agent Said It Was Done. The Database Disagreed.

ThinkingBox is an agent sandbox and ThinkingBox-Bench is a 507-task stateful business-workflow benchmark (retail, auto insurance, etc.) that grades agents by checking the terminal backend database state and side effects they leave behind, not just whether tool calls succeeded or the final reply sounds right. Across 121,680 trials over 12-18 LLMs run 20 times each, the post finds most failures are clean-looking: 67% of failed trials still terminated without a tool error, yet executable checks caught wrong field values, missing effects, or unintended extra effects. It also reports pass@1 vs. 20/20 consistency gaps (e.g. Kimi K3 solves the most tasks at least once but is among the least consistent, while Claude Opus 5 and Claude Opus 5.5 are the most dependable) and cost-per-success/cost-per-dependable-task metrics, and ships the benchmark through an OpenEnv interface for reproducible evaluation.

Model Context ProtocolHugging FaceDocker+4
MicrosoftThinkingBoxThinkingBox-Bench+21

evals, agentsΒ· Published and added 3 Oct 2026

arxiv.orgpaper🧠 AI

Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System

This paper introduces an expert-validated benchmark of 1,042 items spanning ten areas of Colombian law and three question formats (closed multiple-choice, semi-open, open-ended IRAC), used to evaluate 15 proprietary and open-weight LLMs. Closed-question accuracy ranges from 0.577 up to 0.905 (Gemini 3.1 Pro), but free-text factual correctness never exceeds 0.45 for any model, and only about half of the legal norms models cite in free-text answers are correct β€” the rest are wrong or fabricated. The study also finds a negative correlation (rho = -0.46) between answer relevancy and factual correctness, meaning models sound confident and responsive while frequently being wrong, and argues current LLMs need expert supervision plus source-grounding for reliable legal use.

Gemini 3.1 Pro

evals, researchΒ· Added 5 Oct 2026, published 2 Oct 2026

arxiv.orgpaper🧠 AI

ReFract: Benchmarking Perspective Awareness in Language Model Agents with Text World Models

ReFract is a benchmark of 150 expert-validated entries testing whether LLM agents can infer a user's role and act only through tools/actions legitimate for that role, a capability the authors term "Perspective Awareness." Entries are grounded in anonymized industrial-maintenance support queries and use constructed Text World Models to simulate the agent's operating environment and generate perspective-aware action trajectories. State-of-the-art LLMs solve at most 69% of tasks, with over half of trajectories containing attempts at perspective-violating actions, exposing this as a largely unsolved axis of agent evaluation relevant to high-stakes, physically-enacted agent deployments.

ReFract

evals, agentsΒ· Added 5 Oct 2026, published 2 Oct 2026

latent.spacearticle🧠 AI

Inside-Out AI: Rebuilding Airbnb Behind the Scenes and Across the Guest Experience

Interview with Airbnb CTO Ahmad Al-Dahle (former Meta GenAI lead) on turning Airbnb into an "AI-native company": 60% of code is now AI-authored, support AI resolves roughly half of tickets after synthetic-data testing, and an internal organizational context graph called Everest (built with LLMs, embeddings and retrieval) let the airport-pickup feature ship in six weeks versus nine months for groceries. It also details Airbnb's internal agent AirChat, early async on-call agents triggered by monitoring alerts, and a deliberate multi-model strategy that pairs frontier models for high-stakes coding with smaller post-trained models for latency-sensitive search.

Model Context ProtocolLlama
Ahmad Al-DahleAirbnbLatent Space+7

ai-policy, agentsΒ· Published and added 2 Oct 2026

technologyreview.comarticle🧠 AI

Don't be fooled–LLMs don't reason

Thore Graepel, former core member of DeepMind's AlphaGo team and now chair of machine learning at UCL, argues that today's LLMsβ€”even with chain-of-thoughtβ€”don't truly reason the way AlphaGo did. He contrasts AlphaGo's two-system architecture (a policy network for intuition plus explicit game-tree search for deliberation, exemplified by the famous 'Move 37') with LLMs, which he argues only iterate System-1-style next-token prediction without a persistent, inspectable epistemic state, a clean separation of knowledge from reasoning, or faithful (non-post-hoc) chains of thought. He proposes future systems maintain an explicit epistemic state (beliefs, open questions, ruled-out hypotheses) updated via evidence-backed moves, akin to AlphaGo's game tree, to produce auditable, trustworthy reasoning for high-stakes domains like medicine and science.

AlphaGo
Thore GraepelGoogle DeepMindLee Sedol+4

ai-safety, researchΒ· Published and added 2 Oct 2026

latent.spacewriteup🧠 AI

Academia is for Ambition β€” Alex Zhang, MIT

A Latent Space podcast transcript/interview with MIT PhD student Alex Zhang, creator of Recursive Language Models (RLMs) and a core contributor to GPU Mode and KernelBench, who discusses using recursive subagent calls, context offloading, and shared memory to let models work beyond fixed context windows. The conversation ranges across AI-generated GPU kernels and their verification gap, research taste in academia versus industry labs, why "harnesses" (as in Claude Code, Codex, and Pi) function as compositional generalizers, the Jev model's non-autoregressive output-space trade-offs, and large-scale agent swarm experiments at OpenAI, Kimi, and Sakana AI. It offers a researcher's-eye view of where capability overhang and new agent/harness architectures may be heading beyond today's standard transformer decoder.

TransformerCUDAClaude Code+2
Alex ZhangMITGPU Mode+27

agents, researchΒ· Published and added 2 Oct 2026

arxiv.orgpaper🧠 AI

Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning

This paper introduces ZFO (Zero-and-First-Order optimization), a lightweight fine-tuning framework that decouples search direction from step-size selection: a trusted first-order optimizer picks the direction, while two extra zeroth-order function evaluations along that single dimension build a local curvature-aware model to choose the step size, avoiding a full line search. The authors give theoretical guarantees on finite-difference curvature estimates, near-optimal step selection, and convergence to a stationary-point neighborhood, and show empirically that ZFO improves optimization and final performance over fixed-step first-order baselines across several language models and datasets. Code is released publicly, making it a usable technique for practitioners fine-tuning LLMs who want cheaper adaptive step-size control.

PyTorch

training, researchΒ· Added 2 Oct 2026, published 1 Oct 2026

arxiv.orgpaper🧠 AI

Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes

Where-OPD introduces an on-policy self-distillation method for multimodal LLMs where a teacher model receives textual spatial guidance (object identities and coordinates from procedurally generated synthetic scenes) to locate relevant image regions, while a student learns to reproduce this behavior from just the image and question. The approach is annotation-free and scalable, improving performance on counting, document and chart understanding, and transferring from synthetic training to real-world perception benchmarks (CVBench, V*, ZoomBench, BLINK, HR-Bench, MME-RealWorld) with a 3.23-point average gain.

vision-language model
CVBenchV*ZoomBench+3

training, multimodalΒ· Added 2 Oct 2026, published 1 Oct 2026

arxiv.orgpaper🧠 AI

LLM2Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them

LLM2Jev is an architecture-preserving framework that extracts calibrated categorical decisions directly from an LLM's next-token probabilities over bracketed numeric identifiers, rather than generating free-form text. The paper shows Qwen3.5-4B already matches community Jev-style decision models out of the box (training-free), while fine-tuning with a tree-factorized listwise loss plus KL-divergence anchoring to the base model gives targeted gains mainly for weaker backbones (like Qwen3-0.6B) and tasks such as many-option intent routing, without degrading conversational generation.

LoRAQwen3.5Qwen3+1
JevQwen3.5-4BQwen3-0.6B

research, trainingΒ· Added 2 Oct 2026, published 1 Oct 2026

x.comthread🧠 AI

Rahul Sain (@RahulSain714) on X

Innodata open-sourced a toolkit and 14 accompanying datasets for benchmarking factuality and hallucination rates in LLMs, tested against models including Llama2 and Mistral. The toolkit is aimed at identifying input conditions that trigger problematic or hallucinated outputs, giving practitioners a reproducible way to probe model reliability.

Llama2Mistral
Innodata

evalsΒ· Published and added 1 Oct 2026

Showing 150 of 265

Show more