49 results for “Prompt Injection”

claude-skills is an open-source MIT library of about 398 Claude Code skills, agents and plugins that also installs into Codex, Gemini CLI, Cursor and 9 other coding agents. The security-relevant parts are skill-security-auditor, a stdlib-only Python scanner that checks skills for command injection, data exfiltration, prompt injection and dependency supply-chain risks before install, plus a security-guidance PreToolUse hook, a security suite and compliance skills (ISO 27001, SOC 2, GDPR).

github.comrepoAdded 25 Oct 2026🛡️ Security

Arcjet JS SDK is an Apache-2.0 JavaScript monorepo that adds runtime security checks to AI apps, agents and HTTP endpoints. It ships framework SDKs (Next.js, Node.js, Bun, Deno, Fastify, NestJS) plus a Guard API for agent tool calls, MCP handlers, queue workers and background jobs that have no HTTP request object. Features include prompt injection detection, sensitive-information detection (PII, credit cards, custom patterns) and token-bucket, fixed-window and sliding-window rate limiting.

x.comrepoPublished and added 11 Oct 2026🛡️ Security

The Vulnerable MCP Project is a curated, continuously growing database of security vulnerabilities and attack techniques specific to the Model Context Protocol ecosystem. Entries cover concrete CVEs (e.g. CVE-2025-68143/68144/68145 in Anthropic's mcp-server-git), transport-layer SDK flaws (cross-client response leakage, missing DNS rebinding protection in the official TypeScript/Python SDKs), unauthenticated/0.0.0.0-bound servers enabling RCE (MCPJam inspector, MCP Inspector, Grafana MCP), command/path injection in dozens of third-party servers (gemini-mcp-tool, mcp-server-kubernetes, Framelink Figma MCP, Zen MCP, filesystem-mcp), SSRF chains reaching cloud metadata (MarkItDown MCP, mcp-fetch-server), and a large set of AI-specific attacks: tool poisoning, trigger-phrase data exfiltration, prompt injection via GitHub issues, Jira and Google Docs, rug-pull tool mutation, and cross-server tool-name collisions. It functions as the de facto vulnerability registry for the MCP ecosystem, useful for tracking disclosed issues before patching or adopting third-party MCP servers.

vulnerablemcp.inforepoAdded 6 Oct 2026🛡️ Security

Release notes for garak v0.17.0, NVIDIA's open-source LLM vulnerability scanner used for red-teaming and probing models for jailbreaks, prompt injection, data leakage and other failure modes. The release adds an EU AI Act risk-category mapping to tag and group probe results, plus fixes to detectors (agent_breaker judge verdict validation, packagehallucination import parsing, MarkdownExfilContent divide-by-zero, TriggerListDetector alignment) and probes (badchars ASCII selection, Markdown exfiltration link templates), along with Ollama generator improvements and dropped Python 3.10 support in favor of 3.13.

github.comrepoAdded 29 Sep 2026🛡️ Security

security-harness is an open-source Claude Code plugin (also packaged for Gemini CLI, Codex and opencode) that orchestrates subagents — sh-recon, sh-hunter, sh-chainer, sh-verifier, sh-reporter — through a static-analysis pipeline covering 15 vulnerability classes (IDOR/BOLA, SQLi, XSS, SSRF, injection, auth, deserialization, path traversal, secrets, CSRF, XXE, open redirect, crypto, race conditions, file upload), producing README/JSON/SARIF/HTML/PDF reports with PoCs, CVSS and CWE/OWASP mappings. It also ships sh-pr-review, a per-PR merge-gate skill that posts inline findings and a commit status scoped only to code the PR itself introduced or aggravated, with caching, dedup via fingerprints, and a scheduled GitHub Action that keeps its per-class knowledge bases fresh by having one AI agent draft updates from allowlisted sources (OWASP, PortSwigger Research, CWE/CAPEC, HackerOne) while a separate adversarial reviewer agent fails-closed against prompt injection before auto-merge.

github.comrepoAdded 2 Oct 2026, published 29 Sep 2026🛡️ Security

garak is NVIDIA's open-source LLM vulnerability scanner that runs probes/detectors for jailbreaks, prompt injection, PII leakage, package hallucination, markdown exfiltration and more against LLM targets. This v0.17.0 release adds a new EU AI Act risk-category mapping to tag and group probe results by the Act's risk categories, alongside fixes to the agent_breaker judge, markdown-exfiltration detectors, Ollama auth/generation support, and dropped Python 3.10 in favor of 3.13 support.

github.comrepoAdded 26 Sep 2026🛡️ Security

Agent Threat Rules (ATR) is an open, versioned YAML rule format (ATR-YYYY-NNNNN scheme) for detecting AI-agent runtime threats — prompt injection, tool-poisoning, malicious SKILL.md content, and MCP exchange abuse — evaluated by a reference TypeScript engine (with a Python wrapper) shipped as an npm/pip package, CLI, GitHub Action, and Docker scanner with SARIF output. The project positions itself as "Sigma for agent threats," maps rules to OWASP Agentic/LLM Top 10, MITRE ATLAS, and NIST AI RMF, and publishes an extensive (and unusually self-critical) benchmark suite against corpora like garak, PromptBench, and PromptInject, including documented harness bugs and withdrawn recall figures.

github.comrepoAdded 26 Sep 2026🛡️ Security

Release changelog for MITRE ATLAS's atlas-data repository, the structured YAML/STIX dataset behind the ATLAS knowledge base of adversary tactics and techniques targeting AI-enabled systems (prompt injection, model/data poisoning, AI agent tool abuse, AI supply chain compromise). The v2026.09 release brings the matrix to 120 techniques, 88 sub-techniques, 40 mitigations, and 73 case studies, and recent versions introduced a new v6 YAML data format with Pydantic/SQLAlchemy/FastAPI tooling, platform tagging (Predictive AI, Generative AI, Agentic AI, Enterprise), and renamed/generalized techniques like "AI Agent Tool Poisoning," "Manipulate AI Model," and "Enterprise Resource Discovery." It's the canonical, actively-maintained taxonomy practitioners use for AI red-teaming, threat modeling, and mapping real-world AI attack case studies.

github.comrepoAdded 26 Sep 2026🛡️ Security

who-let-the-agents-act is a self-hostable lab suite (Docker/Python, Groq-backed LLM agent) with nine isolated scenarios — overpowered data tools, IDOR-style cross-account access, unsafe agent handoffs, refund-limit bypass, indirect prompt injection via poisoned invoices, multi-tenant RAG leakage, secret leakage through debugging, fail-open approval outages, and a multi-hop confused-deputy agent chain. Each scenario runs in multiple modes (vulnerable vs. application-enforced) against the same prompt, recording model plan, policy decision, tool calls, and DB access as JSON so a reader can directly compare what changes when authorization is enforced outside the system prompt.

github.comrepoAdded 26 Sep 2026🛡️ Security

DVLAA (Damn Vulnerable LLM and Agent Application) is a self-hosted, Docker-deployable security training range for LLM and agent applications, modeled on the DVWA concept but for AI. It covers OWASP LLM Top 10 and Agent Security Top 10 risks (prompt injection, RAG poisoning, tool abuse, memory poisoning, excessive resource consumption, etc.) through 81 hands-on challenges, including an "AWDP" attack/defense track where 10 challenges reproduce real disclosed vulnerabilities in Dify, RAGFlow, Langflow, Flowise, Open WebUI and n8n (with optional real upstream container reproduction), 20 challenges adapted from a China–ASEAN AI security CTF, and 6 challenges adapted from a real model-poisoning CTF. It provides real model interaction, state-machine flag verification, source/prompt viewers, model management (local, Ollama, SiliconFlow, OpenAI-compatible), and bilingual UI.

github.comrepoAdded 22 Sep 2026🛡️ Security

jev-guard is an MIT-licensed hook layer that adds an auto mode to Claude Code, Codex, Copilot CLI, Gemini CLI, Cursor, pi, OpenCode, Antigravity and ACP editors. It asks the Jev model three typed questions (risk, user_requested, from_untrusted) per tool call and decides deny, ask or allow in about 20 lines of code. It also scans tool results for prompt injection and checks skills and plugins.

github.comrepoAdded 27 Oct 2026, published 17 Sep 2026🧠 AI

A GitHub-hosted, CC0-licensed index of 281 public writeups on attacks against AI agents, organized into six categories: indirect prompt injection, direct prompt injection/jailbreaks/guardrail bypass, MCP and agent tooling, AI platform/infrastructure bugs, classic web bugs in AI product surfaces, and AI used as an attacker's tool. Entries span 2021-2026 and include major disclosures like GeminiJack, ForcedLeak, Clinejection, CVE-2025-53773 (GitHub Copilot RCE), and dozens of Johann Rehberger/PromptArmor/qual1a writeups covering exfiltration, MCP OAuth flaws, and coding-agent sandbox escapes. Comes with an accompanying blog post "How to Hack AI Agents" and a scripted links.txt for automation.

github.comrepoAdded 22 Sep 2026, published 17 Sep 2026🛡️ Security

Agent Egress Bench is an open, tool-neutral benchmark maintained by PipeLab (makers of Pipelock) that tests whether products mediating an AI agent's outbound traffic — proxies, egress firewalls, MCP wrappers, agent gateways — actually catch exfiltration attempts caused by prompt injection or misalignment, rather than testing the underlying model itself (that's left to AgentDojo, InjecAgent, CyberSecEval, AgentHarm, ASB). It ships a reproducible Go runner with a proxy/WebSocket/MCP/A2A adapter contract, an offline validator, DSSE-signed evidence bundles, and a CI-runnable GitHub Action, and separates containment from false-positive rate as two distinct scored axes with immutable case IDs and OWASP Agentic Top 10/MITRE ATT&CK mappings. Its governance explicitly permits third parties to publish results unfavorable to the maintainer's own Pipelock product.

github.comrepoAdded 12 Sep 2026🛡️ Security

A curated GitHub corpus tracking real-world security incidents, CVEs, attack techniques, and defensive tools specific to autonomous AI agents — spanning prompt injection/goal hijacking, MCP client-side tool-metadata poisoning, the "Promptware Kill Chain" multi-stage attack model, MITRE ATLAS agentic techniques, and the OWASP Top 10 for Agentic Applications. It also documents the OpenClaw/ClawHub supply-chain incident (1,184 malicious skills confirmed by Antiy CERT) and links defensive projects like OWASP Agent Memory Guard for LangChain/LlamaIndex agent memory protection.

github.comrepoAdded 3 Sep 2026🛡️ Security

galyarder-security (formerly Shannon Proxy) is a Docker/Temporal-based auditing layer that intercepts model SDK calls (Anthropic, Groq, local Ollama, etc.) and routes them through a local proxy on port 8317 rather than direct cloud endpoints, giving operators visibility into and control over what autonomous coding/tool-using agents execute. It runs structured "security sweep" and threat-modeling workflows against a target repo (`./shannon start`) and includes deterministic cleanup routines (`./shannon cleanup`) to remove agent-generated scripts and execution residue and prevent workspace pollution/exposure. It targets a threat model broader than chat-based prompt injection: agent tool-execution drift, dependency supply-chain issues, and unverified code paths in "agentic company" shell-access workflows.

github.comrepoPublished and added 31 Aug 2026🛡️ Security

A curated GitHub awesome-list cataloging the emerging agentic-AI security ecosystem: dozens of Model Context Protocol servers that give LLM agents access to security tools (Nuclei, Shodan, VirusTotal, Burp Suite, Ghidra, BloodHound, x64dbg), autonomous penetration-testing and CTF-solving agent frameworks (CAI, pentagi, BreachSeek, Dark-Moon, D-CIPHER), scanners/guardrails that audit agent skills, MCP servers, and multi-agent chains for prompt injection and other risks (Aguara, Inkog, MCP Doctor, AI-Infra-Guard, ATR), agentic SOC/detection platforms (Vigil), threat-modeling frameworks for agentic systems (MAESTRO, ATFAA/SHIELD), research papers, benchmark datasets, and community/learning resources. It functions as a living index for anyone building or defending AI-agent-driven security tooling.

github.comrepoAdded 3 Sep 2026, published 31 Aug 2026🛡️ Security

DarkMoon is a self-hosted, GPLv3 AI penetration testing platform that orchestrates 50 specialist agents over an MCP-gated toolbox (Nuclei, BloodHound, sqlmap, Impacket, Naabu, httpx, ffuf, etc.) to autonomously run recon-to-exploitation across web, API, Active Directory, Kubernetes, multi-cloud, CI/CD, databases, IoT firmware, and AI/LLM inference endpoints. It uses a Privacy Gateway that tokenizes real IPs/hosts/credentials into placeholders so raw infrastructure data never reaches the (optionally local, e.g. Ollama/llama.cpp) LLM, and can generate sandbox-validated, exploit-retested pull requests for findings. It also ships a dedicated LLM agent that probes OpenAI-compatible/Ollama/vLLM/TGI inference endpoints for the OWASP LLM Top 10 (prompt injection, jailbreaks, system-prompt leakage) using garak-backed probes.

github.comrepoPublished and added 5 Aug 2026🛡️ Security

SecureAI-Scan is an offline CLI that performs import-resolved dataflow analysis on TypeScript, JavaScript, and Python code to find LLM/MCP/Agent-Skill/RAG-specific risks (prompt injection, tool poisoning, unsafe output handling, vector-store access control), mapped to the OWASP Top 10 for LLM Applications, Agentic Applications, and MCP Top 10. It distinguishes proven dataflow traces from heuristic keyword matches, scans .mcp.json/claude_desktop_config.json/SKILL.md for tool-poisoning and command-injection patterns (citing the WhatsApp MCP rug-pull and postmark-mcp backdoor), and includes evasion-resistant deobfuscation checks built in response to the "Cloak and Detonate" adversarial study (arXiv:2607.02357) that bypassed most published skill scanners. It also runs offline advisory checks against known-malicious/vulnerable packages (validated against CVE-2025-6514 and CVE-2024-34359) and integrates via GitHub Actions, pre-commit hooks, SARIF output, and an MCP server for Claude Code.

github.comrepoPublished and added 5 Aug 2026🛡️ Security

ClawSecCheck audits OpenClaw agent installs for prompt injection, malicious MCP servers, and supply-chain risk in skills and dependencies.

github.comrepoPublished and added 22 Jul 2026🛡️ Security

Yolorouter is a free, self-hosted, single-binary LLM gateway that translates between the four major chat wire protocols — OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Gemini generateContent — so a client built for one provider (e.g. Claude Code) can be pointed at any other. It adds multi-provider failover, round-robin upstream API key pooling with rate-limit/quota handling, per-key budgets and access control, a multi-user admin console with OAuth2/OIDC SSO, and native image/video generation support for OpenAI-shaped and non-OpenAI-shaped providers (DashScope, Volcengine Ark, Kling, MiniMax). It also ships cost-optimization features — deterministic system-prompt injection and tool-output compression for coding agents — backed by a published benchmark, plus built-in usage/cost observability.

github.comrepoAdded 2 Oct 2026, published 21 Jul 2026🧠 AI

SingGuard-NSFA is an open-source guardrail framework (released as 0.8B/2B/4B/9B model checkpoints) that targets operational threats in LLM agents—prompt injection, sensitive-info extraction, malicious code requests, tool misuse, and resource exhaustion—rather than just content-safety issues. It's built on a CIA-triad-grounded taxonomy of 185 risk variants across 7 domains (cross-validated against OWASP guidelines) and a 133-language benchmark with 93K+ samples, offering both generative chain-of-thought auditing and ~50ms real-time classification heads. It beats Llama Guard 3, WildGuard, and GPT-5.1 by 6-12 F1 points on multilingual benchmarks, and can also be plugged into Llama Guard 3 as a lightweight enhancement (+17.6 F1).

github.comrepoPublished and added 19 Jul 2026🛡️ Security

LLMVault is a self-hosted, deliberately-vulnerable training range modeled on WebGoat but for LLM applications, covering 25 labs across the OWASP Top 10 for LLM Applications (2025) - prompt injection, jailbreaking, data/model poisoning, agent exploitation, model extraction, and insecure downstream output handling. It ships two modes: a scored Play Mode with scripted assistants for repeatable flag capture, and an unscripted Live Mode that points the same attacks at a real local Ollama model (qwen2.5:3b-instruct) with a freshly minted per-session secret, requiring the player to actually talk the model into leaking it. Each lab pairs the attack with a written defense explanation, and an expert tier is shipped AES/Fernet-encrypted and unlocked manually by the maintainer.

github.comrepoPublished and added 18 Jul 2026🛡️ Security

HOL Guard is an open-source local "antivirus" runtime protection layer for AI coding agents that intercepts shell commands, file access, package installs, and MCP tool calls before execution, classifying them for secret exposure, destructive operations, prompt injection, and supply-chain risk, then allowing/blocking/requesting approval per policy. It integrates via native hooks and managed MCP proxies with Codex, Claude Code, GitHub Copilot CLI, Cursor, Cline, Gemini CLI, Grok, and other agents, and includes a supply-chain package/advisory checker (hol-guard supply-chain scan/explain). The repo also bundles plugin-scanner, a companion CLI/GitHub Action that lints and scores agent plugins, skills, and MCP integrations for manifest issues, secrets, MCP transport risks, and dependency/lockfile problems, producing SARIF/JSON reports for CI gating.

github.comrepoPublished and added 15 Jul 2026🛡️ Security

LLM Guard (Protect AI) is an open source Python toolkit that wraps input/output scanners around LLM applications to detect and block prompt injection, PII/data leakage, toxic or harmful content, and other unsafe interactions, with a pip-installable library and API deployment option. The project has since been archived and is no longer maintained, though the code and associated Hugging Face models remain publicly available.

github.comrepoAdded 3 Sep 2026, published 14 Jul 2026🛡️ Security

promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications, letting developers run automated evals, compare model providers side-by-side, and scan for LLM-specific vulnerabilities like jailbreaks and prompt injection. It runs entirely locally, integrates into CI/CD for automated checks, and can generate security vulnerability reports and review pull requests for LLM-related security and compliance issues. The project was recently acquired by OpenAI but remains MIT-licensed and open source.

github.comrepoAdded 3 Sep 2026, published 14 Jul 2026🛡️ Security

GitHub repo by Kai Greshake collecting research and proof-of-concept material on LLM security, notably indirect prompt injection attacks where malicious instructions are embedded in retrieved content (web pages, documents) to hijack LLM-integrated applications like Bing Chat. It underpins some of the earliest published demonstrations of prompt injection as a practical attack vector against production AI assistants.

github.comrepoAdded 3 Sep 2026, published 14 Jul 2026🛡️ Security

Awesome LLM Security is a GitHub-curated list of papers, tools, and datasets covering LLM attack and defense research, including jailbreak techniques (GCG, AutoDAN, DeepInception, GPTFUZZER, cipher-based jailbreaks), indirect prompt injection, backdoor/virtual prompt injection attacks, model fingerprinting (LLMmap, TRAP), and defenses like PARDEN, circuit breakers, and self-defense filtering. It's a comprehensive reference for tracking the state of adversarial and defensive LLM security research from 2022 onward.

github.comrepoAdded 3 Sep 2026, published 14 Jul 2026🛡️ Security

garak is NVIDIA's open-source LLM vulnerability scanner, described as an nmap/Metasploit for LLMs that probes generative models for hallucination, data leakage, prompt injection, jailbreaks, toxicity, and other failure modes. It supports a wide range of targets such as Hugging Face, OpenAI, AWS Bedrock, Cohere, Groq, llama.cpp/gguf, REST endpoints, and NIM, and structures scans around pluggable probes, detectors, harnesses, and generators, producing JSONL reports and hit logs.

github.comrepoPublished and added 14 Jul 2026🛡️ Security

Giskard is an open-source Python testing/evaluation library for agentic and LLM systems, rewritten in v3 as a modular set of packages (giskard-checks, giskard-scan, giskard-core, giskard-llm, giskard-agents). giskard-scan is the red-teaming layer: it auto-generates adversarial test suites from a plain-language agent description, covering prompt injection, harmful content, stereotypes, and misinformation across OWASP LLM Top-10 categories, and includes a built-in prompt-injection payload dataset. giskard-checks provides LLM-as-judge and assertion-based evals (Groundedness, Conformity) for catching regressions and enforcing safety rules in multi-turn agent conversations; the legacy v2 also offered a tabular ML bias/robustness scanner and RAGET for RAG test-set generation.

github.comrepoPublished and added 14 Jul 2026🛡️ Security

NeMo Guardrails is NVIDIAs open-source Python toolkit (nemoguardrails) for adding programmable rails between an application and an LLM, covering input, dialog, retrieval, execution, and output stages. It ships built-in defenses against jailbreaks and prompt injection, self-check fact-checking and hallucination detection, sensitive-data masking, integrations with NVIDIA safety models, ActiveFence and PolicyAI moderation, LangChain, and a Colang DSL for defining conversational flow rails, plus a CLI/server and an evaluation tool for vulnerability scanning against sample bots.

github.comrepoPublished and added 14 Jul 2026🛡️ Security