Skip to content

fix: runtime follow-ups from the ROS 2 Host/Slave test - #384

Merged
Xuwznln merged 6 commits into
devfrom
fix/runtime-followups
Nov 19, 2026
Merged

Xuwznln merged 6 commits into
devfrom
fix/runtime-followups

Conversation

@Xuwznln

@Xuwznln Xuwznln commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Why

Testing #383 (scheduling authority + Host + ROS 2 Slave) turned up four runtime bugs; fixing them turned up two more. One commit per fix.

Changes

  1. fix(ros2): resolve string property annotations for ROS type mapping
    • Symptom: virtual_transfer_pump took down the whole ROS 2 Slave runtime (message_converter.py:237: 'str' object has no attribute '__dict__').
    • Cause: the driver module uses from __future__ import annotations, so the property's return type float reached the ROS wrapper as the string "float". The wrapper then tried to map that string to a ROS action type.
    • Fix: annotations are resolved with typing.get_type_hints. A type that cannot be mapped now raises the intended ValueError instead of an AttributeError.
  2. fix(ros2): run native actions with an empty optional vessel
    • Symptom: stir without a vessel failed with KeyError 'id'. A specified vessel failed too: the Host copied the single Resource field field-by-field and lost the uuid, so the device received {}.
    • Fix, Host side: a single Resource field is now JSON-encoded in data, as list elements already were.
    • Fix, device side: an empty message with no id or name means "not specified", and the driver gets None, as on the HostLink backend.
  3. fix(hostlink): fail a dispatch to an offline Slave before the start
    • Symptom: the Host consumed the start permit and only then found the Slave unreachable, so the job ended as execution_unknown. A few seconds later the same dispatch failed instead with a misleading "unknown device" KeyError.
    • Fix: the Host checks the Slave's connection before the start boundary (in send_goal, and again right before the permit is requested). If the Slave is offline it raises ExecutionNotStarted (category transport), which goes through the normal action-error decision, as fix(ros2): Host/Slave networking, offline reporting, restart and shutdown #383 does on ROS 2.
  4. fix(scheduler): state the actual cause in execution_unknown decisions
    • Symptom: the reconciliation decision said "进程重启时该动作正在执行" for every cause, including a Slave going offline.
    • Fix: that wording is kept for the three real restart reasons, now named constants. Every other cause gets "无法确认该动作(attempt N)是否已在设备上完成:".
  5. fix(hostlink): keep a request lost with its connection unknown
    • Symptom: when a Slave's connection closed with a request pending, the Host resolved the request itself as a Slave failure (RemoteError). A Slave killed mid-action therefore ended as an ordinary failure that could be retried at once, although the Slave may have run the action.
    • Fix: pending requests now fail with a LinkError, so the job ends as execution_unknown.
  6. fix(hostlink): settle a cancel-requested job as canceled when its Slave disconnects

Verification

Unit tests: every fix has tests that fail on the base and pass with the fix:

  • test_device_node_wrapper.py;
  • msgs/test_mapping.py, msgs/test_basic.py and msgs/test_conversion.py;
  • test_device_node_contract.py;
  • hostlink/test_host_node.py, which covers three fixes;
  • test_host_restart_reconciliation.py.
Gate Result
tests/client/host, without ROS · with ROS 2 Jazzy 298 passed, 6 skipped (ROS-only) · 382 passed
tests/server · tests/protocol · tests/app 1572 passed, 14 skipped · 38 passed · 269 passed
After rebasing on dev and adding commit 6: tests/client/host/hostlink · reconciliation and Edge tests 191 passed, 6 skipped · 32 passed
README demo e2e, harness unchanged: hostlink single · hostlink split 7/7 · 7/7
README demo e2e, harness unchanged: ros2 single · ros2 split 7/7 (125 s) · 7/7 (173 s)

Real runs:

  • Transfer pump: starts on a ROS 2 Slave and its actions succeed next to another device.
  • stir: succeeds without a vessel (absent, {}, null, "") and with a specified beaker, which arrives with its uuid.
  • Offline HostLink dispatch: a dispatch to the device of a killed HostLink Slave raises ExecutionNotStarted and opens a normal action-error decision. The start permit is never requested.
  • Decision texts:
    • Host child killed mid-action: still the restart wording.
    • ROS 2 or HostLink Slave killed mid-action: the actual cause.

Behavior changes

  • Specified vessels now reach native ROS 2 actions. Before, they arrived empty. A non-dict value in a single Resource field now fails on the Host before the start.
  • A dispatch to an offline HostLink Slave's device raises ExecutionNotStarted instead of a KeyError, also in test mode.
  • A Slave lost mid-action now leads to reconciliation instead of a retryable failure. If a cancel was already requested, the job ends canceled.

Found, not fixed here

  • Unsupported-backend devices: a device whose class does not support the ROS 2 backend is skipped silently by the ROS 2 Host and crashes the ROS 2 Slave (from reading the code). The HostLink backend reports such devices as unavailable routes.
  • Failing constructors: a device whose constructor raises still takes down its runtime on both backends. The Host child crash-loops; a Slave exits 1.
  • SIGTERM exit code: a HostLink Slave exits 143 on SIGTERM; the ROS 2 Slave exits 0 since fix(ros2): Host/Slave networking, offline reporting, restart and shutdown #383.
  • Stray pump action: the transfer pump still gets an automatically created set_max_velocity ROS action whose goal fields do not match the driver. It is not listed among the device's actions.

virtual_transfer_pump could not start on a ROS 2 Slave. The device
wrapper read the property getter's raw __annotations__, which are
strings in modules using "from __future__ import annotations" (every
built-in virtual device does). The pump's max_velocity property has a
set_max_velocity method that the registry lists as
auto-set_max_velocity, so the wrapper mapped the string "float" to a
ROS action type; compare_model_fields() then read "float".__dict__ and
the AttributeError ended the whole Slave runtime before any device was
announced.

- Resolve the getter's return annotation with typing.get_type_hints()
  in its own module; keep the string form when a name cannot be
  resolved.
- compare_model_fields() treats an input without fields as different,
  so an unsupported or unresolved type raises the documented ValueError
  instead of AttributeError.

A device whose construction raises still stops the runtime. HostLink
does the same for a Host or a Slave (it only isolates devices whose
class does not support the backend), so ROS 2 keeps that behavior.
A native ROS action whose optional Resource field was left empty, such
as stir on a virtual stirrer without a vessel, failed with KeyError
'id': the device decoded the empty Resource message to {} and looked
the resource up by id. HostLink treats None, "" and {} as "not
specified" and passes None to the driver, and the documentation says
that an empty vessel only simulates the step.

The Host lost a specified vessel as well: convert_to_ros_msg() copied a
nested Resource field field by field, so the reference's uuid never
reached the device, which only decodes Resource.data. A specified
vessel failed with the same KeyError, and treating {} as unset alone
would have dropped it silently.

- Encode a single Resource field like a sequence<Resource> element:
  the whole reference as JSON in data. Unset values (None, "", {})
  become an empty Resource().
- On the device, an empty Resource message (no id, no name, empty data)
  means "not specified": no lookup, the driver gets None. A message
  that carries an id is still looked up, so an older sender fails
  visibly instead of being treated as unset.
The HostLink Host entered the start boundary, consuming the execution
permit, before it called a remote device. When the device's Slave was
already disconnected, the call then failed with "hostlink device
offline" without sending anything, and the job ended as
execution_unknown without error_info, opened a reconciliation decision
and kept the device lock. The Host refreshes its device snapshot every
0.5 s, so a dispatch right after a Slave was killed hit this; a later
dispatch failed with a misleading "unknown device" KeyError.

- In send_goal(), and again right before the permit is requested,
  check the Slave's connection and heartbeat directly. When the Host
  can tell that the Slave is offline, raise ExecutionNotStarted like
  the ROS 2 Host: a definite failure with error_info (category
  transport) that takes the normal action-error decision.
- Failures after the start boundary stay execution_unknown.
The reconciliation decision for an execution_unknown attempt always
said that the action was running when the process restarted, followed
by the uncertainty reason. Slave liveness and HostLink transport losses
also end attempts as execution_unknown, so operators were told about a
restart that never happened.

Keep the restart wording only for restart reasons (authority restart,
Host process restart, a Host report without a reason). For any other
reason the text says that the outcome could not be confirmed and names
the recorded reason. The restart reasons become named constants so the
scheduler can recognize them.
When a Slave's connection closed while a Host request was pending, the
Host resolved the request itself with {"ok": false, "error":
"connection closed"}, which surfaced as RemoteError, the error for a
failure the Slave reported. A Slave killed in the middle of an action
therefore ended the job as an ordinary device failure that could be
retried at once, although the Slave may have run the action.

Fail pending requests with a LinkError instead, so the job ends as
execution_unknown and waits for reconciliation.
…ve disconnects

Failing a pending request with LinkError when its connection closes
turned a cancel-requested job into execution_unknown. Before that
change such a job ended as canceled, and the ROS 2 HostNode settles a
cancel-requested job on an offline device as canceled too: the Slave is
gone, so the action cannot keep running.

Keep that outcome for cancel-requested jobs. A job without a cancel
request still ends as execution_unknown, because the Slave may have run
it.
@Xuwznln
Xuwznln merged commit be20923 into dev Nov 19, 2026
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant