Same Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?

What it claims

IBM Research's CUGA (ConfigUrable Generalist Agent) is an open-source agent harness for enterprise work on business systems such as CRM and payments. It runs with open-weight or hosted models inside the user's own environment, including sovereign clouds and air-gapped deployments. The post argues that harness choice matters as much as model choice. It cites a separate analysis in which GLM-5.2 went from 23% to 52% on SWE-bench Pro when only the harness changed. In IBM's own AppWorld test, task completion ranged from 30% to 81% across CUGA, Hermes (Nous Research) and Deep Agents (LangChain).

How it works

Every CUGA agent runs one loop: prepare, call the model, execute, repeat. The same loop scales from a single CugaAgent with its own tools to a CugaSupervisor that coordinates several agents, either locally or remotely over A2A. One feature used in the AppWorld runs is find_tools, which searches for any other tool a step needs. The authors say the design means the model makes fewer decisions and the harness catches more mistakes before they reach the result.

The policy layer moves business rules out of the system prompt and into the harness. The page says policies act at four points in every run but names only three mechanisms:

• Guidance steers the model.
• Approvals are enforced by the runtime.
• ToolGuards are enforced by the runtime.

IBM recommends placing rules such as payment approval and PII handling under approvals and ToolGuards. The layer is described in the ACM CAIS 2026 paper "Governance by Construction for Generalist Agents", which is on arXiv.

Evidence

AppWorld covers 9 apps and 457 APIs, including email, payments and file storage. The test used 90 tasks in 30 scenarios with three variants each and one attempt per task. Every harness used GLM-5.3-flash. Runs used the Harbor evaluation framework, and the setup is public in IBM's fork, harbor-eval.

The scenario-level metric, SGC, counts a scenario only when all three variants pass. On SGC, CUGA passed 20 scenarios, Hermes 8 and Deep Agents 2. The page text does not say which harness scored 30% on task completion; that appears only in Figure 1.

A separate Policy Playground demo runs two live CUGA agents on the prompt "Pay everything I owe." Without policies, the agent paid $90, including a denied claim in collections. With policies, the agent held the payment for approval and then paid only the $10 that was due.

Limits

• Author bias: The CUGA team ran the comparison.
• Unequal setups: CUGA used an AppWorld-specific configuration that the other harnesses did not, so the setups were not identical.
• Small sample: The results come from 90 tasks on one benchmark.
• Proxy apps: AppWorld's apps are consumer apps, so the benchmark is only a proxy for enterprise automation.
• No policy test: No benchmark run used policies, so the numbers do not measure policy enforcement.
• Fit to task: The authors state that neither Hermes nor Deep Agents was built for multi-app API work.

IBM invites competing Hermes or Deep Agents configurations through harbor-eval. It promises to rerun them on the same tasks and publish the results.