Skip to content

Back to the catalog

NVIDIA/garak
github.comrepo

NVIDIA/garak

garak is NVIDIA's open-source LLM vulnerability scanner, described as an nmap/Metasploit for LLMs that probes generative models for hallucination, data leakage, prompt injection, jailbreaks, toxicity, and other failure modes. It supports a wide range of targets such as Hugging Face, OpenAI, AWS Bedrock, Cohere, Groq, llama.cpp/gguf, REST endpoints, and NIM, and structures scans around pluggable probes, detectors, harnesses, and generators, producing JSONL reports and hit logs.

TL;DR NVIDIA's garak is an open-source CLI framework for red-teaming LLMs against jailbreaks, prompt injection, hallucination, and toxicity.
Open original ↗
Document

What it is

Operational usage detail for garak, NVIDIA's LLM vulnerability scanner. The summary says what garak tests for; the page gives the actual invocation syntax, install paths, probe-selection mechanics, and how to read results — the things an engineer needs to actually run a scan.

How it works

garak is a Linux/OSX command-line tool. Given a target model it runs, by default, every probe it knows, using each probe's recommended detectors. A "generator" is specified by --target_type (the model family/interface) plus optionally --target_name (the exact model). Some generators need an API key set as an environment variable and will tell you if so. Model types are whitelisted so the plugin knows which sub-API to call.

Per probe, garak prints a progress bar during generation, then a row scoring results against each detector. By default it makes 10 generations per prompt, so totals run high; a row like 840/840 means total generations vs. how many behaved acceptably. Undesirable behavior is marked FAIL with a failure rate.

What to do

Install from PyPI, or from GitHub for a fresher build:

python -m pip install -U garak
python -m pip install -U git+https://github.com/NVIDIA/garak.git@main

List available probes:

garak --list_probes

Restrict to a probe family or a single plugin (family, then .PluginName):

--spec probes.promptinject
--spec probes.lmrc.SlurUsage

Probe a commercial model for encoding-based injection:

export OPENAI_API_KEY="sk-123XXXXXXXXXXXX"
python3 -m garak --target_type openai --target_name gpt-5-nano --spec probes.encoding

Test a Hugging Face model against DAN 11.0:

python3 -m garak --target_type huggingface --target_name gpt2 --spec probes.dan.Dan_11_0

Hugging Face access modes: huggingface (local Pipeline), huggingface.InferenceAPI, and huggingface.InferenceEndpoint (with --target_name as the endpoint URL and HF_INFERENCE_TOKEN set to a read-role token).

Each run is logged to a .jsonl file; errors go to garak.log. To find the probes and prompts with the most hits, run the bundled script:

analyse/analyse_log.py

Caveats

Only generative models work as Hugging Face targets; unsupported types should yield an informative error (open an issue if a valid model fails). The page's own example — encoding injection on a GPT-3 variant vs. ChatGPT — shows the newer model was more susceptible, so a passing older model is no guarantee for a newer one. If you cloned before the move to the NVIDIA org, update your remote:

git remote set-url origin https://github.com/NVIDIA/garak.git

Technologies named here

  • garak
  • Hugging Face Hub generative models
  • Replicate text models
  • Nmap
  • Metasploit Framework
  • OpenAI API chat & continuation models
  • AWS Bedrock foundation models
  • PromptInject
  • LiteLLM-accessible models
  • Language Model Risk Cards
  • analyse/analyse_log.py
  • REST-accessible LLM endpoints
  • gguf models via llama.cpp version >= 1046
  • GPT2
  • text-babbage-001
  • ChatGPT
  • Cohere models
  • Groq-hosted models
  • ggml models
  • NIM endpoints / build.nvidia.com models
  • Hugging Face
  • OpenAI
  • AWS Bedrock
  • LiteLLM
  • llama.cpp

Techniques named here

  • Prompt Injection
  • Jailbreaks
  • Hallucination elicitation
  • Data leakage probing
  • Misinformation generation
  • Toxicity generation
  • LMRC SlurUsage checks
  • PromptInject framework methods

Related in AI Security