How to evaluate ADLC security solutions: A buyer's checklist | Endor Labs

What it is

A vendor-authored buyer's checklist from Endor Labs for evaluating security tools in the agentic development lifecycle (ADLC). The page defines two risks: the code AI agents produce, and the actions they take (package installs, CI edits, MCP and tool calls). It is written by a vendor, and the product examples are Endor Labs' own (AI SAST, AURI, OPA-based policies, upgrade impact analysis).

How it works

The page gives tests to run on your own code rather than vendor claims.

• Logic flaws: Seed or pick a flaw where untrusted input crosses three to five architectural layers (route handler, service, data access), such as IDOR. Ask each vendor to find it and show the full source-to-sink path.
• Reachability: Ask for the call path for a specific CVE in your repo, from first-party code through the dependency and into container OS packages. A tool that only says the package is present leaves triage to you.
• Evidence over percentages: Ask which codebases the false positive and false negative rates were measured on, who set ground truth, whether false negatives were measured, and whether you can reproduce it. For each finding, require data flow, call path and written classification reasoning.
• Look-alike test: SHA-1 inside ASN.1 DER encoding is the page's example of a safe pattern that naive scanners flag as weak crypto.
• Repeatability: Scan the same commit two or three times unchanged. Wording variation is acceptable, but findings that appear and disappear are not. Also ask which parts of the analysis are deterministic (dataflow, reachability) and which use model reasoning (triage, logic flaws).
• Agent actions: Ask whether the tool can inventory agents, models, MCP servers, skills and hooks (including unapproved ones). Ask whether it can evaluate an action before execution and allow, block or log it, and whether it can intercept a package install before it lands. Ask whether it can keep agents from secrets and whether one policy point covers every agent.

What to do

Scope first: agents and IDEs in use, risky languages, repo structure, weekly PR volume and agent share, where agents have write or execute access, and required CI/CD and ticketing integrations. AppSec and platform engineering should write the scorecard together.

Then run a 2 to 4 week proof of concept:

1. Pick 3 to 5 repos, including one with complex authorization, one with a large dependency tree, one shipped as a container, and repos where agents write much of the code.
2. Build ground truth before any tool runs: multi-layer logic flaws, reachable and unreachable open source vulnerabilities, live and revoked secrets in history, plus safe look-alikes.
3. Pin commits and run every tool on the same code with equal configuration effort, including the same real or replayed PRs.
4. Score on one sheet (the page's table did not come through in the fetched text), then have both teams review together.

Workflow checks: incremental PR scans that report only new findings, scan time per PR versus full scan, one PR scan covering code, dependencies and secrets, and policy that blocks only on high-confidence findings. Check fix quality against your test suite, including how the vendor predicts upgrade breakage.

Governance questions to get in writing: where code is processed, which model providers are used, retention and deletion, whether customer code trains any model, data residency, and audit logs. The page suggests mapping items to NIST SSDF (SP 800-218).

Red flags: demos only on the vendor's sample app, unexplained accuracy figures, findings without data flow or call path, no answer on which analysis uses a model, "agent support" that is an after-the-fact report, and fixes that are a link to a CVE page.

Caveats

• Public suites such as the OWASP Benchmark are small, well known and may be in model training data, so use them only as a supplement.
• The one benchmark cited is Endor Labs' own: AI SAST across 8 real-world projects in Java, Python, JavaScript, TypeScript, C# and Go found 192 real vulnerabilities, roughly 3x what frontier models found. The page gives no further methodology detail, so treat it as an example of method, not an independent result.
• The page says its product examples are reference points, not the only valid design.