[BUG] Pilot can falsely complete embodied tasks without execution evidence

[enkerewpo]

Problem

Pilot can mark an embodied task as complete based on the planning model's own task_update=done response even when a required physical action was never planned or executed. This allows the robot to claim that it reached a destination or performed an operation without corresponding RTDL or environment evidence.

Observed incident

The user task was to identify a drink bottle, travel to the elevator, and report the result.

The recorded task trace shows:

1. Pilot queried Scene's goal_room capability and received a reachable elevator target pose.
2. No Navigation capability call or navigation RTDL node was subsequently created.
3. The robot did not move.
4. The model returned task_update.status = done with zero capability calls, and Pilot accepted the task as complete.
5. An earlier execution of the same instruction successfully followed goal_room -> navigation -> pose observation -> speech, so the missing navigation step is stochastic rather than an unsupported capability.

This rules out Navigation or Executor dropping a dispatched call: there was no navigation call to drop. The failure is a false-success / premature-termination decision in the Pilot harness.

Raw provider chain-of-thought is neither available nor an appropriate correctness mechanism. The useful debugging evidence is the structured model output, RTDL plan, Executor events, capability results, and fresh environment observations.

Current design gap

Pilot currently lets the same model both plan the task and approve its completion. A prompt tells the model to ground room goals through Navigation, but prompt-only policy did not prevent this failure. Plan completion and user-task completion are also different concepts: Executor can authoritatively report whether an emitted plan finished, but that does not prove that all obligations in the original user request were covered by any plan.

A navigation-specific check such as if goal_room was called, require navigation would address this example but would not solve the general problem. Equivalent failures can occur for manipulation, perception, speech, memory, device control, or multi-step tasks.

Investigation summary

Recent evaluations show that a one-shot LLM judge is not sufficient as the sole completion authority:

• From Confident Closing to Silent Failure finds false success to be common and reports weak performance from ordinary LLM judges; it recommends direct trajectory/environment consistency checks for consequential tasks.
• AgentRewardBench finds that no evaluated LLM judge is reliable across environments and documents judges accepting confident but incorrect agent reasoning.
• Inner Monologue, REFLECT, and Code-as-Monitor support closed-loop post-action observation and task-conditioned verification for robots, but learned verification remains dependent on perception quality.
• Goal-Autopilot and Proof of Execution support an external completion state machine and verifiable execution records. Their guarantees still depend on whether the original goal was translated into complete verification obligations.

The practical direction is therefore a hybrid: use a model for semantic supervision and evidence selection, while the harness enforces only generic evidence integrity and lifecycle invariants.

Proposed direction

This is an initial design direction, not a frozen API:

1. At task admission, an independent Supervisor compiles the raw user goal into a frozen CompletionContract containing atomic obligations, acceptable evidence descriptions, freshness requirements, and dependencies. A separate clean-context coverage review should detect omitted obligations.
2. Planner may emit propose_done, but may not directly transition the task to a terminal success state.
3. Executor emits generic execution receipts and/or a fan-out RTDL event stream carrying task revision, plan/node identity, capability identity, request/result, timestamps, causal linkage, and terminal state.
4. The Supervisor receives the original goal, frozen completion contract, authoritative receipts, and fresh environment state. It should not be anchored by Planner's claim that the task is already complete. It may actively call read-only capabilities discovered through Atlas to obtain missing evidence.
5. The Supervisor returns verified_done, continue with missing obligations, blocked, or waiting_user.
6. A model-independent harness floor checks only generic facts: cited receipts exist, belong to the current task revision, are fresh, have completed, and have no unresolved dependencies. Domain semantics remain with the Supervisor; Pilot must not hardcode Navigation, manipulation, or other capability-specific sequences.

For the reported trace, goal_room would be evidence that a destination was resolved, not evidence that the robot arrived. The Supervisor should reject completion as insufficient and request further execution or observation without any goal_room -> navigation rule in Pilot.

Acceptance criteria

• A Planner response cannot directly produce terminal success; success requires an independent verification verdict.
• Replaying the reported trace is rejected because required physical completion evidence is absent.
• Replaying the earlier successful trace can be accepted using its execution receipts and fresh state observations.
• Alternate valid plans remain acceptable; verification is based on satisfied task obligations, not a hardcoded capability sequence.
• blocked, waiting_user, cancellation, and failure are distinct from successful completion.
• Verification decisions record the completion-contract revision and exact evidence IDs used.
• Tests cover missing actions, failed actions, stale observations, confident false completion text, successful alternate trajectories, and Supervisor uncertainty.

Open questions

• Whether Supervisor is a separate Robonix system service or an isolated role within Pilot.
• How Executor should expose durable/fan-out execution receipts in addition to its current caller-owned stream.
• Which read-only observation capabilities are safe for active verification.
• When high-risk physical tasks should require a second verifier or human escalation.
• How to benchmark false-success rate, false rejection rate, verification latency, and additional model/tool cost.