Interpretable Meta-Policies for Neural Decision Making

What it claims

NEXUS (Neural EXecution Under Symbols) is a hierarchical RL method from TU Darmstadt, hessian.AI and DFKI (Emunds, Blüml, Delfosse, Kersting; arXiv, 27 October 2026). It extends Parallelised Q-Networks (PQN, Gallici et al., 2024) to options. A meta-policy picks among neural skills, and each skill has its own object-centric reward. The authors argue that symbolic policies over atomic actions explode into many rules. Their Seaquest example shows nine Prolog-style rules for fire/left/right/up/down. NEXUS replaces these with three skill-level rules: fight_enemy, rescue_diver and surface.

How it works

Each step has three stages. Object-centric state (object, x, y, orientation, value) is extracted from pixels. A meta-policy selects a skill. The skill's Q-network picks the low-level action. Exploration is epsilon-greedy at both levels.

The skill MDPs share states, actions and transitions and differ only in reward. Each skill gets a separate Q-network trained on its own skill reward. The meta Q-network trains on the environment reward. Training is off-policy, so every skill learns from trajectories produced by whichever skill is active. All Q-functions are updated toward lambda-returns. Because the method builds on PQN, it has no replay buffer or target network and relies on many vectorised environments plus normalisation. LLMs propose the skill set, the skill reward functions and the meta-policy rules (the paper's Appendix F, not included in the fetched text).

The three meta-policy variants differ only in skill selection:

flowchart TD
    S[Object-centric state] --> V{Meta-policy variant}
    V -->|Neural| N[argmax meta Q over all skills]
    V -->|Symbolic| R[Single matching rule, conditions disjoint]
    V -->|NeSy| M[Rules give active-skill mask]
    M --> Q[argmax meta Q, inactive skills set to -inf]
    N --> K[Selected neural skill acts]
    R --> K
    Q --> K

• Symbolic. This variant needs pairwise-disjoint rule conditions, so every decision traces to exactly one rule. It has no learned meta Q-function.
• Neuro-symbolic. Several conditions may fire at once. A masked argmax over the learned meta Q-values resolves the tie. The motivating case is a low-oxygen submarine with a nearby enemy, where hand-written priorities would grow complicated.
• Neural. This is plain hierarchical PQN. The skills stay disentangled, but the reason a skill was chosen is not visible.

Limits

The fetched text stops at the start of the experimental section. It gives no return figures, baselines, environment list beyond Seaquest, compute or training cost. The claims of being competitive with deep RL and harder to reward-hack cannot be checked from this page.

The method depends on an object extractor and hand-written or LLM-generated per-skill reward functions. The symbolic variant also requires mutually exclusive conditions. The authors themselves describe a weakness of that variant: it handles situations where several skills apply poorly, and that weakness is the reason the NeSy variant exists. The page does not mention a code release or licence.