Skip to content

Port the link watchdog - #101

Open
jamiecounsell wants to merge 2 commits into
attermann:masterfrom
zerokelvin-dev:link-watchdog
Open

jamiecounsell wants to merge 2 commits into
attermann:masterfrom
zerokelvin-dev:link-watchdog

Conversation

@jamiecounsell

@jamiecounsell jamiecounsell commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The link watchdog was still marked as TODO. Without it, a link request with no reply stayed pending forever, initiators never sent keepalives, and dead links were never found stale or closed.

Solution

I ported the reference watchdog as one pass per link, run from Transport::jobs() for each pending and active link:

The reference has sent LINKCLOSE on a stale close since at least RNS 0.9.6, so this matches it. The commented-out Python in Link.cpp came from an older version. The extra last_outbound keepalive trigger from the first version of this PR is gone.

The watchdog reads three values that nothing was setting, so this sets them as the reference does:

  • keepalive and stale_time, from the RTT when the link becomes active (__update_keepalive()). Adds KEEPALIVE_MIN / KEEPALIVE_MAX; KEEPALIVE keeps its value (360).
  • last_keepalive, in had_outbound(true) (L652-L655).
  • last_proof: PacketReceipt::validate_link_proof() now records it on the link, as the reference does (Packet.py L447). This is a one-line behavior change outside the watchdog. Without it, a link whose only inbound traffic is proofs would go stale.

The watchdog runs after the job loop because it sends packets (keepalives, LINKCLOSE), and Transport::outbound() waits for the job loop to finish. jobs() now also collects pending links, so unanswered link requests time out.

Compatibility

  • No change to wire behavior vs. the Python reference implementation: keepalives and LINKCLOSE are sent when the reference sends them (links above)
  • Any divergence from the reference is behind an RNS_* preprocessor guard (off by default) (none)
  • No breaking public API change (adds Link::update_keepalive(), Link::last_proof(double), Type::Link::KEEPALIVE_MIN / KEEPALIVE_MAX)
  • No board- or variant-specific code; any platform-specific code is unavoidable, isolated, and explained below

Testing

New native tests in test_transport:

  • test_link_watchdog_pending_timeout: a link request to a destination that never answers stays PENDING before the establishment timeout and closes with TIMEOUT after it, through Reticulum::loop(). Fails on master (the link stays PENDING).
  • test_link_watchdog_stale_then_close: an active link stays ACTIVE while traffic arrives, goes STALE after stale_time without it, stays STALE during the grace period, then closes with TIMEOUT.

A new interop scenario test_interop/run_link_keepalive.sh (C++ initiator, Python RNS responder) was added in its own commit:

  1. The C++ side opens a link and leaves it idle for 22s (longer than the 10s stale time for local link). Python checks it answered at least 3 keepalives and the link is still ACTIVE.
  2. We then drop everything on the link except LINKCLOSE. The C++ side should go stale, close with TIMEOUT, and send LINKCLOSE, which Python receives as INITIATOR_CLOSED.

Run against Python RNS 1.3.8:

[python] link established: d1fadd3024d840ae5b92c53f3d27fb74
[python] answered keepalive 1 at 5.9 s
[python] answered keepalive 2 at 10.9 s
[python] answered keepalive 3 at 15.9 s
[python] answered keepalive 4 at 21.0 s
[python] held 22 s with 4 keepalives answered; going silent
[python] link closed after 36.0 s: teardown_reason=INITIATOR_CLOSED
[python] SUCCESS initiator closed the link 13.9 s after Python went silent
[cpp] link established: d1fadd3024d840ae5b92c53f3d27fb74, rtt 0.019 s
[cpp] link went stale after 30.9 s
[cpp] link closed after 36.0 s: teardown_reason=TIMEOUT

The last answered keepalive was at ~21s, so the link went stale 10s later (stale_time) and closed 5s after that (the grace period). On master the same test fails: no keepalives go out, and Python closes the link after ~15s.

Platforms tested (and how):

  • native (CMake): Linux aarch64, GCC 14.2, C++17. All 19 test suites pass.
  • ESP32 (board: ThinkNode M9, ESP32-S3): this branch ran in SolarOS against Python RNS nodes over LoRa, exchanging LXMF messages.
  • nRF52: not tested on hardware; CI builds wiscore_rak4631.
  • Interop with Python RNS (test_interop/): run_link_keepalive.sh and run_link_lifecycle.sh both pass against RNS 1.3.8 on Linux aarch64. The senders run with transport disabled, so they were built with -DRNS_PERSIST_PATHS=0 -DRNS_PERSIST_KNOWN_DESTINATIONS=0 -DRNS_PERSIST_HASHLIST=0.

@attermann

Copy link
Copy Markdown
Owner

Thanks for this, and for the detailed write-up and the on-hardware testing against Python RNS. Porting the link watchdog fills a real gap, and running it after the job loop to avoid the outbound() deadlock makes sense.

Before this can go in, a few things need sorting out:

  1. The contribution policy asks that behavior differing from the Python reference be guarded and documented (section 5.4). Two places differ:

    • The STALE→CLOSED transition now sends a LINKCLOSE packet, which the reference doesn't do at that point.
    • The ACTIVE check also fires on last_outbound + keepalive.

    Please either match the reference (including links to relevant reference Python code) or put these behind an RNS_* guard that's off by default.

  2. PacketReceipt::validate_link_proof now validates proofs for real (it was if (false)) and records last_proof. That looks like the right fix, but it's a separate behavior change from the watchdog. Please call it out explicitly in the description, or split it into its own PR.

  3. The PR is stacked on Validate link proofs #99, so please rebase onto master once that lands. The branch also predates the latest release, so a rebase would be good to confirm nothing has shifted underneath it.

  4. A native test for the pending-timeout and stale→close paths would be very welcome.

jamiecounsell and others added 2 commits November 12, 2026 13:15
The link watchdog was still marked as TODO. Without it, a link request
with no reply stayed pending forever, initiators never sent keepalives,
and dead links were never found stale or closed.

The watchdog is now one pass of the reference loop per link, run from
Transport::jobs() for each pending and active link, after the job loop
next to the resource watchdogs (it sends keepalives and link closes, and
outbound() waits for the job loop). Where the reference sleeps, the pass
compares times instead:

- PENDING and HANDSHAKE links time out after the establishment timeout.
- ACTIVE links: when nothing has come in for the keepalive interval, an
  initiator sends a keepalive (at most one per interval), and a link
  with nothing in for stale_time goes STALE.
- STALE links send LINKCLOSE and close once the grace period has passed:
  rtt * keepalive_timeout_factor + STALE_GRACE, capped at
  WATCHDOG_MAX_SLEEP, as the reference sleeps.

The watchdog reads three values that nothing was setting, so this sets
them as the reference does:

- the keepalive interval, derived from the RTT when a link comes up
  (adds KEEPALIVE_MIN/KEEPALIVE_MAX; KEEPALIVE keeps its value)
- the time the last keepalive was sent
- the time of the last proof received on the link: validate_link_proof()
  now records it on the link, as the reference does

test_transport checks that an unanswered link request times out through
Reticulum::loop(), and that an active link goes stale and then closes
after the grace period.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbF8zYqrDkMpUrin37AMCv
A C++ initiator opens a link to a Python destination and leaves it idle
for 22 s, longer than the 10 s stale time on a local link. Python checks
that it answered at least three keepalives and that the link is still
active. Python then drops everything on the link except LINKCLOSE, and
the C++ side has to go stale, close the link with TIMEOUT and send
LINKCLOSE, which Python sees as INITIATOR_CLOSED.

Without the link watchdog, the C++ side sends no keepalives and Python
closes the idle link after 15 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbF8zYqrDkMpUrin37AMCv
@jamiecounsell

Copy link
Copy Markdown
Contributor Author
  • Rebased onto master
  • LINKCLOSE on a stale close matches the reference, which has sent it since at least 0.9.6; the links are in the description. The extra last_outbound trigger is gone.
  • The proof validation itself went in with Validate link proofs #99; the remaining last_proof line is now called out in the description.
  • Added some native tests for pending-timeout and stale to close transition, as well as a test_interop keepalive scenario

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants