Skip to content

Back to the catalog

Giskard-AI/giskard-oss
github.comrepo

Giskard-AI/giskard-oss

Giskard is an open-source Python testing/evaluation library for agentic and LLM systems, rewritten in v3 as a modular set of packages (giskard-checks, giskard-scan, giskard-core, giskard-llm, giskard-agents). giskard-scan is the red-teaming layer: it auto-generates adversarial test suites from a plain-language agent description, covering prompt injection, harmful content, stereotypes, and misinformation across OWASP LLM Top-10 categories, and includes a built-in prompt-injection payload dataset. giskard-checks provides LLM-as-judge and assertion-based evals (Groundedness, Conformity) for catching regressions and enforcing safety rules in multi-turn agent conversations; the legacy v2 also offered a tabular ML bias/robustness scanner and RAGET for RAG test-set generation.

TL;DR Open-source Python library that red-teams LLM agents for prompt injection and OWASP LLM Top-10 vulnerabilities and provides LLM-as-judge evals for RAG/agent quality.
Open original ↗
Document

What it is

The GitHub README for Giskard's v3 rewrite — a modular set of Python packages (giskard-core, giskard-llm, giskard-agents as pulled-in foundations; giskard-checks and giskard-scan as the user-facing layers) for testing LLMs, black-box agents, and multi-step pipelines. This document carries the practical details the catalog summary omits: install matrix, version requirements, the v2→v3 API migration map, and telemetry.

Preconditions / Affected

  • Requires Python 3.12+.
  • LLM-as-judge checks (Groundedness, Conformity, LLMJudge) and scan generators both need a provider extra installed plus the matching API key. Default judge model: openai/gpt-4o-mini.
  • v2 remains installable but is no longer actively maintained. Only the legacy tabular/ML scan stays v2-only and is not planned for v3.

What to do

Install per the surface you need:

pip install giskard           # checks (+ agents, llm, core)
pip install "giskard[scan]"   # + vulnerability / quality scan
pip install "giskard[openai]" # provider SDK for LLM judges / generators

Migrating from v2 — the API mapping the page states:

  • v2 LLM scan → vulnerability_scan
  • RAGET (RAG eval test-set generation) → quality_scan with KnowledgeBase

For the v2-only tabular bias/robustness scan, pin the old line:

pip install "giskard[llm]>2,<3"

Telemetry is on by default (optional aggregated analytics via giskard-core; the page states no prompts or outputs are sent). Opt out:

export DO_NOT_TRACK=1
export GISKARD_TELEMETRY_DISABLED=1

Set these (or the same keys in a working-directory .env) before import to avoid creating ~/.giskard/id; setting them later still stops further sends but the ID file may already exist.

Minimal scan invocation takes a callable target plus a plain-language description and language list:

await vulnerability_scan(
    target=my_agent,
    description="A customer support chatbot for an e-commerce platform.",
    languages=["en"],
)

Caveats

  • The page notes a naming trap: giskard.agents.Generator (an LLM client for workflows/judges) is not the same as giskard.checks input generators (LLMGenerator, which synthesize user messages).
  • v3 can wrap ML models as targets for giskard-checks/giskard-scan, but the auto-detecting tabular detector suite, the giskard.testing ML test suite, and the Giskard Hub are v2-only with no v3 port planned.

Technologies named here

Techniques named here

  • Prompt Injection
  • Red-teaming / adversarial test generation
  • OWASP LLM Top-10 threat category coverage
  • Harmful content, stereotype, and misinformation probing
  • LLM-as-judge evaluation
  • Groundedness / RAG evaluation
  • Bias, performance, and robustness detection

Related in AI Security