Skip to content

Back to the catalog

SECTOR (@sector_sol) on X
x.comthread

SECTOR (@sector_sol) on X

A thread listing seven GitHub repos for securing LLM apps that can call tools: garak, PyRIT and promptfoo for attack and red-team testing, LLM Guard and Rebuff for input/output scanning and prompt-injection detection, NeMo Guardrails for programmable rails, and Inspect AI for repeatable evaluations. It proposes a loop of map risks, attack, inspect failures, add controls, retest and monitor. It stresses that a system prompt is not a security boundary and that credentials should be scoped, tool arguments validated outside the model, and consequential actions approved. It also warns that Rebuff is archived.

TL;DR Thread listing 7 repos (garak, PyRIT, promptfoo, LLM Guard, Rebuff, NeMo Guardrails, Inspect AI) for red-teaming and guarding tool-calling LLM apps before production.
Open original ↗
Document

What it is

A seven-repo checklist for LLM apps that can call tools. The repos are NVIDIA/garak, Azure/PyRIT, promptfoo, protectai LLM Guard, protectai Rebuff, NVIDIA-NeMo Guardrails and UK Government's Inspect AI. The thread's claim is that a bad answer becomes a different problem once the model has permission to execute.

What to do

The thread splits the tools by lifecycle stage, which is the part most useful for planning:

flowchart LR
  A[Prelaunch] --> B[garak]
  A --> C[promptfoo]
  A --> D[Inspect AI]
  E[Runtime] --> F[Input/output checks]
  E --> G[Guardrails]
  E --> H[Tool-side permission checks]
  • Prelaunch: garak, promptfoo and Inspect AI. Rerun them whenever the setup changes. promptfoo is the one pitched for CI, turning red-team checks into repeatable tests. PyRIT is pitched for structured campaigns, as opposed to "testing five prompts and calling it secure".
  • Runtime: input/output scanning (LLM Guard, which the thread says catches exposed secrets, malicious URLs and suspicious instructions), guardrails (NeMo), and permission checks on the tool side.
  • Tool boundary: keep credentials scoped, validate tool arguments outside the model, and require approval for consequential actions. If the model gets confused, the app should deny the action rather than execute it and apologize afterward.

Caveats

  • The thread calls NeMo Guardrails "a layer, not a replacement for tool permissions".
  • Rebuff is described as a prototype with an archived repo, so it is not a maintained production shield.
  • It gives no commands, versions, benchmarks or coverage claims for any tool, so it works as a pointer list, not as evidence of effectiveness.

Technologies named here

Techniques named here

Related in AI Security