Confident AI (@confident_ai) on X

What it claims

Confident AI's announcement argues that most eval suites "fail quietly": the scores look fine but don't track the behavior that matters. Eval Skills, released under Apache 2.0, responds by building evals from real failures (traces, human review, named failure modes) rather than from synthetic cases. A coding agent then changes the app one measured step at a time against that baseline. The post says the bundled Python and TypeScript tooling exists so the agent "doesn't spend tokens rebuilding review infrastructure" for tracing, annotation, grading and reports.

How to use it

The post gives one entry point. Open your app's folder in Claude Code or Codex and prompt:

Use eval-build to build evals from real failures and establish a runnable baseline.

The intended order is eval-build first, to get a runnable baseline, then eval-descent for the improvement loop. You can manage the loop in a local UI or in Confident AI's hosted product. The repository link is truncated on the page (github.com/confident-ai/e…). The post gives no install steps, skill-registration instructions, package names or version requirements, so check the repo README before adopting.

Limits

• This is an announcement post. It gives no results, no case studies and no figures for how closely the graders agree with human judgment, even though "validated against human judgment" is the central claim.
• The post says held-out cases "stay out of the loop" but does not say how. It could be enforced by the tooling or left to the agent's prompt; the post does not say which. A reply in the thread names the practical risk: a coding agent with write access to the eval it is graded on will edit it. Before trusting eval-descent scores, check whether the repo prevents the agent from modifying graders, cases or splits.
• The post does not say which trace sources or formats eval-build can read. It also does not say what the "incidents" input looks like, or whether the Confident AI platform is needed for any step beyond the optional hosted UI.