Remote delegated task never completes when the remote thread has an orphaned queued run (lineage shows running forever) #22651

Summary

A task delegated to a linked environment (delegate_task with remote) stays running in the parent forever when the remote thread's run fails and a later run is left queued behind it. The parent's Lineage panel then shows "N running" with nothing running, and task_status reports status: running, workState: working indefinitely. Observed on v0.1.1-preview.20261018.3327: two remote tasks stuck for 45 h.

Repro (observed)

1. Delegate a task to a linked environment. The remote run 1 fails (here: the provider gave up after repeated API errors).
2. Before run 1 ends, t3_thread_send a follow-up to the remote thread: run 2 is created queued.
3. Run 2 never starts (it was not held and not resumed after the remote server restarted). Thread state: run 1 failed, run 2 queued with startedAt: null, no active run.
4. The parent task never completes.

Direct check from the parent with t3_thread_wait on the remote thread, run 1, 5 s timeout:

• untilDelegatedWorkSettled: true → {status: "failed", timedOut: true}
• without the flag → {status: "failed", timedOut: false}

Cause

• RemoteDelegation.followOnce (apps/server/src/peer/RemoteDelegation.ts, ~L594 at the tag) waits with untilDelegatedWorkSettled: true and a 10-minute timeout, and on timedOut returns false, so follow repeats forever. There is no fallback to the selected run's terminal status and no retry cap.
• ThreadManagementService.waitForThread (~L894-915) only reports settled when delegatedTaskProgress returns result_available.
• delegatedTaskProgress (packages/provider-core/src/server/subagentProjection.ts ~L212) treats any non-terminal run as active unless it is queued && queueHeld === true. The orphaned run 2 is queued and not held, so the thread is "working" forever even though the selected run is terminal and nothing will start run 2.

Second defect on the remote side: after a failed run, recovery is supposed to hold queued runs (ProviderRuntimeRecoveryService ~L334, Orchestrator ~L1343) so they are ignored by progress; here the queued run survived a server restart unheld and unstarted.

Suggested fix

• In followOnce: when the wait times out but the selected run is terminal and the remote thread has no active run (and its only pending runs are queued with no start), complete the task from that run's result instead of looping.
• In delegatedTaskProgress: do not count a queued run as active when the latest executed run failed and the queue is not being started.
• Make queued-run recovery after restart either start or hold the run; never leave it unheld and unscheduled.

Impact

Lineage "running" count is wrong, task_status never reaches a terminal state, and anything that waits on the task (follow timers, "wait" mode delegation) hangs.