Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry

What it is

This paper (Saha, Faghih, Feizi; University of Maryland) attacks SKILL.md, the manifest/instruction file for Agent Skills, across three distinct registry stages using real skills pulled from ClawHub.ai (OpenClaw's registry): 100 skills across email, travel, tax, health, and prompt categories, each among the top 20 by downloads in its category.

How it works

Three separate attack methods, each modifying only natural-language text in SKILL.md (no code changes):

• Discovery (ranking manipulation): appends a short optimized "trigger" string to SKILL.md to push embedding-similarity scores toward a target query. Two variants tested — a beam-search black-box attack (uses Qwen2.5-0.5B-Instruct as a proposal model, beam width 4, 20-token budget, only needs score feedback) and a gradient-based white-box attack (needs embedding-model/gradient access, 20-token trigger, 5 runs, top-5 candidates retained over 50 iterations).
• Selection manipulation: rewrites only the description field to bias an agent into picking a functionally-identical "pro" variant over the original when both are offered as candidates.
• Governance evasion: semantic rewording of malicious instructions so a ClawHub-style scanning pipeline (static scan + LLM review + VirusTotal-style scan) doesn't return a blocking verdict, while the unsafe intent is preserved for the agent that eventually loads the skill.

flowchart LR
    A[Adversary edits SKILL.md text only] --> B{Discovery}
    B -->|trigger tokens| C[Higher rank in search]
    A --> D{Selection}
    D -->|biased description| E[Agent picks adversarial variant]
    A --> F{Governance}
    F -->|reworded intent| G[Scanner misses malicious verdict]
    C --> H[Agent loads malicious skill]
    E --> H
    G --> H

Preconditions / Affected

Applies to registries that rank/surface skills via embedding-based retrieval (tested against BAAI/bge-base-en-v1.5, BAAI/bge-small-en-v1.5, and OpenAI text-embedding-3-small), and to agents that select among candidate skills based on description text. Governance evasion specifically targets pipelines combining static checks, LLM-based semantic review, and malware scanning (modeled on ClawHub's pipeline).

What to do

No fixes or detections are given in the fetched portion of the paper — it's an attack-evaluation study, not a mitigation guide. Concrete numbers worth noting for risk assessment: black-box beam-search triggers against OpenAI's embedding model reached an 86.14% pairwise win rate with only a 20-token (~1% of file length) addition, and white-box/gradient attacks against the BAAI models achieved up to 84.76% win rate in ~3-4% score boosts. Triggers were also shown to transfer across embedding models to some degree (per the cross-model table), meaning an attacker optimizing against one retriever can still gain partial advantage against another.

Caveats

Only the Discovery section's full results were captured from the fetched text (pairwise win rates and score boosts); the paper's own abstract states Selection succeeds in 77.6% of trials and Governance evasion ranges 36.5%-100%, but the methodology/breakdown for those two sections wasn't present in the retrieved page content. Code is published at github.com/ShoumikSaha/agent-skill-security for reproduction.