29 results for “repos”

FeatureArc is a benchmark for coding agents on evolving features, built by a pipeline that mines repository histories into multi-stage tasks across four programming languages. It has 73 manually reviewed tasks of 2 to 10 stages, validated with fail-to-pass and pass-to-pass tests. Across 10 agents, pooled success drops from 62.7% at the first stage to 28.5% for the whole feature. Inherited defects appear in 72.9% of failed later stages.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026🧠 AI

MCP-Injector is a protocol-aware fault-injection middleware for Model Context Protocol tool calls, with permanent, transient and intermittent faults on seedable schedules. Its fault model comes from 57,240 issue and PR threads across 1,774 open-source MCP repositories. In a 1,350-cell experiment, faults cut task-completion score by 60% (8.08 to 3.23 of 10), and the 120B gpt-oss model was no more resilient than the 20B. Agent stacks differed in how well they absorbed faults.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026🧠 AI

AKV introduces cache engineering, which lets LLM agents control their active KV cache with drop (exclude cache from future attention) and repos (adjust retained keys' position encodings) without editing the trajectory or re-prefilling. The AKV serving engine keeps a stateless API while managing stateful cache reuse, eviction and restoration. A tool-response rolling-drop policy raised benchmark scores by 1.7 points and cut cost 27.7% versus threshold-triggered summarization.

huggingface.copaperAdded 23 Oct 2026🧠 AI

Cache Engineering for Agents proposes letting LLM agents control their active KV cache directly, using a drop operation to exclude cache from attention and repos to adjust retained keys' position encodings, without editing the trajectory or re-prefilling. The authors build the AKV serving engine to handle history-dependent cache states. A tool-response rolling-drop policy raises average benchmark scores by 1.7 points and cuts cost by 27.7% against threshold-triggered summarization. AKV gives up to 1.2x output throughput and 1.5x faster P95 latency.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026🧠 AI

ToFu is a tool-aware fuzzing framework that derives semantic security checks for LLM tools (mainly MCP servers) from their metadata and a corpus of 74 CWE-guided sanitizers. It generates schema-valid Atheris harnesses, uses runtime tracers and outcome oracles to catch bugs that don't crash, and uses an LLM only to map sanitizers and seed inputs, never as the judge. It also describes a "Repeat-After-Me" attack, in which the LLM is told to repeat attacker-written text that the orchestrator then dispatches as a tool call. It was reproduced on LangGraph, MAF, ADK, AutoGen, Strands and CrewAI. On 273 MCP servers it reports 253 verified bugs across 343 tools in 118 repositories, including path traversal, log injection, exfiltration and command bypass.

arxiv.orgpaperAdded 18 Oct 2026🛡️ Security

This October 2026 arXiv paper studies task over-specification and under-specification in repository-level coding benchmarks. It shows that hints derived from ground truth, as added in SWE-bench Pro, FEA-bench, Doc2Feat-bench and Multi-SWE-bench, leak spoilers in some of them. The authors test several specification levels, recommend techniques for parsimonious hints, and plan to release their hints.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026🧠 AI

Endor Labs' 2025 State of Dependency Management report examines how AI coding agents and MCP servers add supply chain risk. Researchers analysed 10,663 GitHub MCP server repositories and tested AI-generated dependency recommendations across PyPI, npm, Maven and NuGet, finding agents import vulnerable or non-existent packages. MCP servers are immature: 75% are built by individuals, 41% lack license info, and 82% use sensitive APIs. The report advises treating AI-generated code as untrusted third-party input subject to code review, SAST/SCA and dependency governance.

endorlabs.compaperPublished and added 12 Oct 2026🛡️ Security

The paper builds the first dated copy network of agent skills (SKILL.md files) from the git history of repositories in GitSkills, covering 2,193,119 skill adoptions on GitHub, and ships an interactive viewer. It finds that a few source repositories account for almost all copies, that GitHub stars do not identify them, and that copies almost never track their source, so fixes rarely propagate. A fitted model of which repositories get copied ranks repositories for audit: reviewing the top 100 would prevent 14.9% of later adoptions of high-risk skills, against 0.5% for the 100 most-starred. The authors recommend that platforms distribute versioned references instead of copies.

arxiv.orgpaperAdded 9 Oct 2026, published 8 Oct 2026🧠 AI

Builds the first dated copy network of agent skills (SKILL.md files) from git history in the GitSkills dataset, covering 2,193,119 skill adoptions on GitHub, plus an interactive viewer. Finds a few repositories are the source of nearly all copies, stars do not identify them, and copies almost never track their source, so upstream fixes rarely propagate. A model of which repositories get copied ranks repos for audit: reviewing the top 100 prevents 14.9% of later adoptions of high-risk skills versus 0.5% for the 100 most starred. The authors recommend platforms distribute versioned references rather than copies.

arxiv.orgpaperAdded 9 Oct 2026, published 8 Oct 2026🛡️ Security

Introduces DITTO, a stack-based, context-aware scanner for Pickle-based pre-trained models that tracks Pickle VM state transitions and applies semantic analysis to infer model intent. Authors found 9.3% of 10,000+ popular Hugging Face repos still use Pickle, and release PickleBench (959 benign, 92 malicious models, including extension registry attacks missed by existing tools). Reports 100% coverage, 0% false negatives, 0.7% false positives, F1 0.966, outperforming current scanners.

arxiv.orgpaperAdded 9 Oct 2026, published 7 Oct 2026🛡️ Security

This paper proposes three evaluation screens — Decisive-Action BPB, Patch MCQ, and prefix-conditioned pass@K — that predict how well a base model will perform as a coding agent after post-training, without requiring the base checkpoint to run a full agentic harness. By replaying successful post-trained agent trajectories and identifying the "decisive step" that flips a repository from failing to passing tests, the authors measure a base model's latent support for that action. Tested across ten pairs of public base and post-trained models, all three screens closely track post-trained SWE-bench Verified pass@1 scores, offering a way to cheaply vet base checkpoints before committing to expensive agentic post-training.

arxiv.orgpaperAdded 8 Oct 2026, published 7 Oct 2026🧠 AI

A preprint hosted in OpenAI's "math" GitHub repository claiming a result on integer multiplication algorithms with complexity below the n log n bound, the long-standing benchmark achieved by the Harvey–van der Hoeven algorithm in 2019. The page itself only exposes citation metadata (author, title, BibTeX entry) with the actual paper linked as a PDF, so the specific proof technique and whether it was AI-assisted cannot be confirmed from the fetched text alone.

github.compaperAdded 7 Oct 2026, published 6 Oct 2026🧠 AI

CheckerBench is an executable benchmark of 300 tasks drawn from 297 CVEs across 167 repositories, 85 CWEs, and five language ecosystems, testing whether long-horizon coding agents can synthesize working static-analysis checkers from a defect specification through repeated compile-and-analyze feedback loops. Its companion framework CheckerLab independently rebuilds submitted checkers and scores vulnerable-fixed diagnostic contrast, patch localization, false positives, and tool use; across 21 model-harness configurations mean Pass@1 is 32.30 percent and the best reaches 45.33 percent, showing reliable checker synthesis remains hard for current coding agents.

arxiv.orgpaperAdded 8 Oct 2026, published 6 Oct 2026🧠 AI

CheckerBench is an executable benchmark of 300 tasks derived from 297 CVEs across 167 repositories, 85 CWEs, and five language ecosystems, testing whether long-horizon coding agents can synthesize working static-analysis checkers from a defect specification through repeated compilation and analysis feedback. The accompanying CheckerLab framework independently rebuilds submitted checkers and measures vulnerable-vs-fixed diagnostic contrast, patch localization, false positives, and tool use; across 21 model-harness configurations mean Pass@1 is 32.30% (best 45.33%), showing reliable checker synthesis remains hard for current agents.

arxiv.orgpaperAdded 7 Oct 2026, published 6 Oct 2026🛡️ Security

AgSpec is a retrieval-based speculative decoding framework built for coding-agent pipelines. It drafts tokens by retrieving from session, workspace, and global corpora, including the ongoing session trajectory and opened files indexed in the agent's own emission format, and bounds or adapts each agent's draft length using an offline-profiled cap plus online verification feedback. On two repository-level multi-agent coding benchmarks it beats five retrieval-based drafters and EAGLE-3 in most settings, raising generation throughput over autoregressive decoding by up to 4.37x at batch size 1 and 4.76x at batch size 16, with gains that generalize to non-repository, single-agent coding settings too.

arxiv.orgpaperAdded 2 Oct 2026, published 1 Oct 2026🧠 AI

AuraForge is a method for synthesizing and validating executable security tests to train coding agents toward secure (not just functionally correct) code, combining attack-oriented test synthesis, multi-language task construction, and safeguards against reward hacking. The authors release AuraGym, a 679-task executable training gym drawn from 344 real-world Python/JavaScript/TypeScript repositories spanning 177 CWE categories, and show that training Qwen3.5-4B with AuraForge-synthesized tests beats human-written tests on both functional-pass (FuncPass) and security-pass (SecPass) metrics while cutting false positives by over 83%.

arxiv.orgpaperAdded 3 Oct 2026, published 1 Oct 2026🛡️ Security

AuraForge is a pipeline that synthesizes and validates executable security tests to supervise the training of coding agents, using attack-oriented test synthesis and safeguards against reward hacking. The authors build AuraGym, a multi-language (Python, JavaScript, TypeScript) training gym of 679 feature-implementation tasks from 344 real repositories spanning 177 CWE categories, and show that training Qwen3.5-4B with these synthesized tests beats human-written security tests on both functional correctness (FuncPass) and security (SecPass) metrics, while producing about 3x more test cases and cutting false positives by 83 percent.

arxiv.orgpaperAdded 2 Oct 2026, published 1 Oct 2026🧠 AI

VibeMemBench is a benchmark for evaluating persistent memory systems on 111 real-repository coding targets (from 90 SWE-rebench V2 repos, with 3,634 prior-work trajectories), built via a four-stage SIEVE pipeline that verifies a target's injected history experience actually improves executable test outcomes before inclusion. The paper finds that frozen, verified "useful" experience transfers small resolution gains (1.1–4.5 pp) and step reductions to held-out solvers (DeepSeek-V4-Pro, GLM-5, GLM-5.2, Kimi-K2.7-code, Qwen3.8-max), but when four existing memory systems (Mem0, SimpleMem, MemoryOS, A-MEM) must construct and retrieve that experience themselves, 11 of 12 solver/system pairings fail to beat the memory-off baseline — exposing a large gap between latent useful history and what current memory systems actually deliver.

arxiv.orgpaperAdded 30 Sep 2026🧠 AI

Aletheia is a permission-minimality testing framework for coding-agent repository instruction files (rules), which can be weaponized via prompt injection to request unnecessary credential access or data exfiltration while still producing a correct patch. It translates requested authority into a typed language, synthesizes sandbox configurations, and runs tasks under full versus incrementally restricted permissions to produce "dispensability witnesses" that flag suspicious permission requests. On the AIShellJack benchmark it detects all 314 attack inputs with zero false alarms on five benign templates, and on 80 manually verified benign GHAgentFiles rules it raises only 3 false positives (3.75%).

arxiv.orgpaperAdded 2 Oct 2026, published 30 Sep 2026🛡️ Security

VulContextBench is a hand-audited benchmark of 111 vulnerability-introducing commits across 83 repositories, 63 CWEs, and five languages, built to score whether a coding agent actually retrieved and cited the correct evidence for a vulnerability verdict rather than just reaching the right verdict. Evaluating seven frontier models (including Qwen3-Coder-Next and GPT-5.5) at file/block/line granularity, the authors find every model explores most of the gold evidence but reports only a fraction of it: Qwen3-Coder-Next views 86.3% of annotated lines but cites just 12.9%, while GPT-5.5 views 73% and reports 36%. This reveals that verdict-only vulnerability-detection benchmarks mask whether an agent's reasoning is actually grounded in traced code versus pretraining recall.

arxiv.orgpaperAdded 2 Oct 2026, published 26 Sep 2026🛡️ Security

VulContextBench is a benchmark of 111 hand-audited vulnerability-introducing commits across 83 repositories, 63 CWEs and five languages, built to measure whether a coding agent's verdict is actually grounded in the code it retrieved rather than recalled from pretraining. Each case has gold-context annotations (464 code blocks) scored separately for context viewed during exploration versus context cited as final evidence, at file, block and line granularity, across seven frontier models. The headline finding is a large viewed-vs-reported gap — e.g. Qwen3-Coder-Next views 86.3% of gold lines but cites only 12.9%, GPT-5.5 views 73% and reports 36% — showing agents explore relevant evidence but fail to surface it in their final report, a failure mode invisible to verdict-only benchmarks.

arxiv.orgpaperAdded 2 Oct 2026, published 26 Sep 2026🧠 AI

This paper empirically measures the completeness of AI Bills of Materials (AIBOMs) generated for roughly 97.5K Hugging Face model repositories, examining structural/metadata fields, model identity, license and external-reference data, and model-card documentation such as datasets, limitations, safety-risk and environmental information. It finds that generated AIBOMs satisfy required structural fields but are weak on AI-specific documentation like responsible-use, environmental impact, and meaningful descriptions, and argues for better model-card practices and automated AIBOM validation.

arxiv.orgpaperAdded 26 Sep 2026🧠 AI

HoF-Bench is a benchmark of 95 real, publicly disclosed CVEs originally found by AISLE's LLM-based vulnerability analyzer (280+ CVEs across 78 projects including OpenSSL, curl, and GnuTLS), repackaged into eight repositories pinned at vulnerable commits. Analyzers get source/target-file scope but no CVE ID, description, fix, or expected mechanism, and a detector-blinded frontier-model judge only credits findings that match the same code path, root cause, attack condition, and impact. Testing ten non-frontier backbones (open-weight 21B-284B and proprietary "flash"-tier models) in a fixed multi-pass scaffold, a minimal analyzer rediscovers up to 68% of the CVEs, with misses concentrated in C infrastructure code — showing small/cheap models can approximate frontier-level vulnerability discovery on known bug classes.

arxiv.orgpaperAdded 26 Sep 2026🛡️ Security

This paper empirically studies AI Bills of Materials (AIBOMs) generated for ~97.5K public Hugging Face model repositories, measuring how completely they capture provenance, licenses, datasets, model-card details, safety-risk assessments, and environmental information. It finds that generated AIBOMs satisfy required structural/metadata schema fields but are weak or missing on AI-specific documentation such as model cards, responsible-use notes, limitations, and dataset declarations, exposing transparency and governance gaps in the ML model supply chain. The authors argue for better model-card practices, repository-level traceability, and automated AIBOM validation.

arxiv.orgpaperAdded 26 Sep 2026🛡️ Security

AutoMUD is a source-code-driven tool that generates Manufacturer Usage Description (MUD) network access-control profiles for IoT devices by statically analyzing firmware/source code rather than relying on live traffic observation. It combines static and syntactic extraction with retrieval-grounded LLM reasoning and deterministic validation to recover rarely-exercised or conditional communication paths, link each generated rule to its source-level provenance, and record excluded findings with explicit reasons for review. Evaluated on a Linux-based repository plus a controlled semantic fault-injection campaign, it produced structurally valid, traceable MUD profiles that let analysts detect, localize and correct propagated errors.

arxiv.orgpaperAdded 28 Sep 2026, published 25 Sep 2026🛡️ Security

This arXiv paper studies whether natural-language documentation helps coding agents resolve software issues. The authors introduce a roundtrip benchmark that scores code descriptions by whether code regenerated from them passes the original tests, use it to optimize a description-writing prompt that reaches full fidelity and generalizes to unseen files, then test the hypothesis on real repository issues across two model families and ten repos — finding that neither static compact documentation nor retrieved context beats giving the agent the issue alone.

arxiv.orgpaperAdded 28 Sep 2026, published 25 Sep 2026🧠 AI

VIPER-MCP is an end-to-end automated auditing framework that combines two-pass static taint analysis (anchor queries mapping alerts to specific MCP tool handlers) with feedback-driven LLM prompt evolution to both detect and dynamically confirm taint-style vulnerabilities (command injection, SSRF, path traversal) in Model Context Protocol servers. Run against 39,884 real-world open-source MCP server repositories, it discovered 106 0-day vulnerabilities — all confirmed via end-to-end exploit traces with concrete proof-of-concept natural-language prompts — resulting in 67 assigned CVEs, including a root-privilege RCE in the bytebot computer-use agent. It reports 0% false positive rate and 7.7% false negative rate, substantially outperforming existing static and dynamic MCP security baselines.

arxiv.orgpaperAdded 22 Sep 2026, published 17 Sep 2026🛡️ Security

VIPER-MCP is an automated auditing framework for Model Context Protocol servers that detects taint-style vulnerabilities in tool handlers and dynamically confirms exploitability via generated proof-of-concept prompts. It uses two-pass static analysis plus feedback-driven prompt evolution, and scanning 39,884 real-world MCP server repos found 106 0-day vulnerabilities with 67 CVE IDs assigned.

arxiv.orgpaperAdded 22 Sep 2026, published 13 Sep 2026🛡️ Security

This arXiv paper introduces SkillSec-Eval, a lifecycle-aware framework for evaluating the security of reusable LLM agent skills, covering stages beyond runtime execution: repository admission, semantic retrieval, planner selection, execution, and skill evolution. The authors build a threat taxonomy across this lifecycle and empirically evaluate it against a repository of 327 real-world agent skills, showing vulnerabilities exist at multiple stages that prior work (focused mainly on prompt injection and runtime execution) has largely ignored.

arxiv.orgpaperPublished and added 18 Jul 2026🛡️ Security