feat(control-lane): signed control→instance lane — Recycle and DeleteSpace in another instance's mesh by rbuergi · Pull Request #5785 · Systemorph/MeshWeaver

rbuergi · 2026-09-27T05:52:44Z

What

The core half of a signed control→instance lane, so governed Hosting/InstanceActions filed on the control instance can execute targeted mesh operations inside another instance's mesh. The first operations are Recycle and DeleteSpace; the first use is deleting memex-cloud's stranded MeshWeaver samples partition (refs #5228).

Design and owner commands: new doc Doc/Architecture/ControlLane.

Request (ControlLaneRequest) names the target deployment, the operation (open vocabulary: Recycle, DeleteSpace), target and confirmation, dry-run flag, the approved plan digest, requester, approver and a validity window. It is HMAC-signed with the target deployment's own key. That is the announcement-key slot in reverse: Hosting:PlatformWebhookSecret:{deployment} on control and ControlLane:Key on the target. The fleet secret is never used.
Target (ControlLaneReceiver, POST /api/control-lane) checks, in order:
1. the lane is armed;
2. the signature verifies;
3. the body is a well-formed request;
4. it names this deployment;
5. its expiry and lifetime are valid (at most 15 minutes);
6. operation, shape and approval are acceptable;
7. it is single use: the ledger node Admin/ControlLane/{id} is created before anything runs, so a replay's create fails.
Each check has its own status code. The target then computes its own plan as system and refuses a real run whose digest is not the approved one. Every step goes to the ledger and is reported back signed to the control inbox.
Operations are RecycleOperation (hub.RecycleNode from the off-router execution hub) and DeleteSpaceOperation. DeleteSpaceOperation runs over SpaceDeletion, which is Plugins' in-mesh DeleteSpace engine moved into core so the in-process action and the lane run the same code.
Control half (ControlLaneClient): signs only with the deployment's own key, never the fleet secret or a child equal to it. It verifies the target's signed acceptance and each report, including the action-plan/v1 digest.
DeploymentContent.ControlLaneKeySecret is the record's claim on the key.

Why core: every portal image carries it without the Hosting package. It is security code that a remote request invokes as system, so it must be compiled and reviewed rather than in-mesh. It also needs only core surfaces (the recycle cascade, the partition teardown, the framework deletes).

Nothing is armed anywhere until ControlLane:Key and Hosting:Deployment are mounted.

Verification

dotnet build -c Release -warnaserror, 0 warnings: MeshWeaver.Graph, MeshWeaver.Deployment.Contract, Memex.Portal.Shared, MeshWeaver.Graph.Test, MeshWeaver.Documentation.Test.
ControlLaneTest: 12/12 passed, over two meshes (control and target on MonolithMeshTestBase).
- Happy paths, end to end: DeleteSpace (dry run, then the approved plan; the space is gone; target ledger reads Done) and Recycle (dry run, then approved; a fresh activation answered).
- Negatives, each touching nothing: bad signature (wrong key, fleet key, another deployment's key, unsigned), replay (409), expired and over-long lifetime (410), plan mismatch (Refused, space intact), wrong deployment (403), missing approver, unknown operation, sloppy confirmation, protected partition.
- Pure checks: arming, control-key selection, record binding, the report/request discriminator, report digest drift, and the action-plan/v1 preimage.
- Negative control: with the signature and digest checks disabled, ABadSignature… and APlanThatIsNotTheApprovedOne… go red (2/11 failed); restored after.
MeshWeaver.Documentation.Test: 649/649 passed. The MergeBounded and router-origin guards caught two sites, and both are fixed.

Cross-repo

Pairs-with: none — nothing public is removed; types and one init property are only added.
Implementers: none — no member is added to an existing interface (IControlLaneOperation, IControlLaneReportSink and IControlLaneTransport are new).
Mirror-sync: none — no i18n key is added or changed.

The Plugins half follows once this is sealed: the portal maps /api/control-lane, Hosting/InstanceAction routes remote Recycle/DeleteSpace through the lane, the inbox watcher folds control-lane-report, and the in-mesh DeleteSpaceRunner delegates to SpaceDeletion.

🤖 Generated with Claude Code

…teSpace executed IN another instance's mesh The control instance could reach its own mesh (the inbox, instance→control) and the cluster (the operator), but never another instance's mesh — so Recycle and DeleteSpace refused cross-instance, and memex-cloud, which does not run Hosting, could not be operated at all. Core half (Doc/Architecture/ControlLane): - ControlLaneRequest/Report/Plan envelopes, HMAC-signed with the TARGET's own per-deployment key; the request and the report are told apart by a discriminator that must be PRESENT in the body. - ControlLaneReceiver (target): armed only by ControlLane:Key + Hosting:Deployment, refuses a lane key equal to any inbox secret; checks signature → version/id → deployment → validity window (≤15 min, 2 min skew) → operation/shape/approval → single use (the ledger node Admin/ControlLane/{id} is CREATED before anything runs; a second create is the replay). - The target computes its own plan as system and REFUSES a real run whose plan digest is not the approved one; every step is written to the ledger and reported back signed to the control inbox. - Operations: RecycleOperation (hub.RecycleNode from the off-router execution hub) and DeleteSpaceOperation over SpaceDeletion — the space-deletion engine moved from Plugins' in-mesh DeleteSpaceRunner so both paths run the SAME reads, plan and deletes. - ControlLaneClient (control): signs only with Hosting:PlatformWebhookSecret:{deployment}, never the fleet secret or a child equal to it; verifies the target's signed acceptance and its reports (including that the reported digest is the one this code computes — action-plan/v1). - POST /api/control-lane in Memex.Portal.Shared; AddControlLane() registered on every portal. - DeploymentContent.ControlLaneKeySecret: the record's claim on the key (the binding). - ControlLaneTest: two meshes (control + target) on MonolithMeshTestBase — dry run → approved DeleteSpace and Recycle end to end, plus bad signature / fleet key / other deployment's key, replay, expired, over-long lifetime, plan mismatch, wrong deployment, unknown operation, protected partition; negative control (checks disabled) turns the two security cases red. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

github-actions · 2026-09-27T05:58:06Z

Test Results (shard 0)

1 files 1 suites 2m 40s ⏱️
348 tests 348 ✅ 0 💤 0 ❌
352 runs 352 ✅ 0 💤 0 ❌

Results for commit 41913aa.

♻️ This comment has been updated with latest results.

github-actions · 2026-09-27T05:58:11Z

Test Results (shard 3)

460 tests 460 ✅ 55s ⏱️
3 suites 0 💤
3 files 0 ❌

Results for commit 41913aa.

♻️ This comment has been updated with latest results.

Copilot

Copilot review overview

🔵 Needs a closer look

Unresolved critical safety and correctness findings remain in destructive operations, mesh-state reads, and request handling.

Review effort: Lite
Findings: 8 · 6 · 1

Open (15)

Execute the exact approved dependency address set · New
Refuse incomplete dependency networks before disposal · New
Reject duplicate operation registrations · New
Treat null partition coverage as unknown · New
Fail closed when store existence is unknown · New
Use authoritative root data for deletion checks · New
Use authoritative streams for record discovery · New
Do not treat empty deletion completion as success · New
Move body reads behind the HTTP I/O pool · New
Sanitize RequestedBy and ApprovedBy in dispose reasons · New
Use authoritative node streams for target resolution · New
Fail when the control-lane ledger cannot be updated · New
Require AccessService for system-scoped deletion posts · New
Use authoritative streams for test existence checks · New
Add German translation for ControlLaneKeySecret · New

What changed in this PR

Adds a signed control-to-instance lane for governed Recycle and DeleteSpace operations, with HMAC authentication, plan digests, auditing, deployment keys, and portal integration.

Changes:

Adds control-lane protocol, admission, client, receiver, and operation infrastructure.
Adds space-deletion logic and deployment configuration.
Adds tests and architecture documentation.

File	Summary
`test/MeshWeaver.Graph.Test/ControlLaneTest.cs`	End-to-end control-lane tests
`src/MeshWeaver.Graph/ControlLane/SpaceDeletion.cs`	Space inventory and deletion engine
`src/MeshWeaver.Graph/ControlLane/MeshReading.cs`	Mesh query-reading abstraction
`src/MeshWeaver.Graph/ControlLane/IControlLaneOperation.cs`	Operation contract
`src/MeshWeaver.Graph/ControlLane/ControlLaneReceiver.cs`	Target receiver, ledger, and execution
`src/MeshWeaver.Graph/ControlLane/ControlLaneOperations.cs`	Recycle and DeleteSpace implementations
`src/MeshWeaver.Graph/ControlLane/ControlLaneKeys.cs`	Key validation and binding
`src/MeshWeaver.Graph/ControlLane/ControlLaneExtensions.cs`	Dependency-injection registration
`src/MeshWeaver.Graph/ControlLane/ControlLaneEnvelopes.cs`	Wire envelopes and plan digests
`src/MeshWeaver.Graph/ControlLane/ControlLaneClient.cs`	Signed transport and report verification
`src/MeshWeaver.Graph/ControlLane/ControlLaneAdmission.cs`	Request validation and admission
`src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateAnnouncementKey.md`	Related-key documentation
`src/MeshWeaver.Documentation/Data/Architecture/ControlLane.md`	Control-lane architecture documentation
`src/MeshWeaver.Documentation/Data/Architecture.md`	Architecture index entry
`src/MeshWeaver.Deployment.Contract/DeploymentRecordExtensions.cs`	Deployment key helper
`src/MeshWeaver.Deployment.Contract/DeploymentContent.cs`	Deployment key property
`memex/Memex.Portal.Shared/MemexConfiguration.cs`	Lane registration
`memex/Memex.Portal.Shared/Api/ControlLaneEndpoints.cs`	HTTP endpoint adapter

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

rbuergi · 2026-09-27T06:05:54Z

+            ("Recycle",
+                $"hub.RecycleNode(\"{node.Path}\", reason) as system — "
+                + (isNodeType
+                    ? $"a NodeType: the dispose cascades to {network!.Addresses.Count} address(es) in its dependency network"
+                    : $"a node address (nodeType {node.NodeType ?? "none"}): only this address, no cascade"),
+                false),
+        };
+        var notes = network is { IsComplete: false }
+            ? new[] { $"⚠️ {network.Incomplete.Count} enumeration leg(s) of the dependency network could not be read: "
+                      + string.Join(" | ", network.Incomplete) }
+            : [];
+        var plan = ControlLanePlan.Of(Operation, request.Deployment, steps, notes);
+        return new ControlLanePreparation(plan, () =>
+            SpaceDeletion.AsSystem(hub, () => hub.RecycleNode(node.Path, RecycleBudget, ControlLaneText.ReasonLine(request)))


Fixed in a30ac2d. The plan now binds the EXACT address set: the digest of the sorted dependency-network addresses is in the step's command, and the addresses are listed in the notes. Right before the dispose, the run derives the network again and refuses unless it is complete and has the same set digest. Nothing is disposed in that case.

rbuergi · 2026-09-27T06:05:55Z

+                    if (network is { IsComplete: false })
+                        throw new InvalidOperationException(
+                            $"the recycle of '{node.Path}' is INCOMPLETE: {network.Incomplete.Count} enumeration leg(s) of its dependency network "
+                            + $"could not be read ({string.Join(" | ", network.Incomplete)}); the addresses that WERE derived were recycled");


Fixed in a30ac2d. An incomplete network is now refused in Prepare, so no plan is reported and nothing is disposed. The execute path cannot reach RecycleNode with an incomplete network: it is re-derived and checked before the dispose.

rbuergi · 2026-09-27T06:05:56Z

+        this.operations = operations
+            .GroupBy(o => o.Operation, StringComparer.Ordinal)
+            .ToImmutableDictionary(g => g.Key, g => g.First(), StringComparer.Ordinal);


Fixed in a30ac2d. ControlLaneReceiver now throws at construction when two executors claim one operation, naming both types. Pinned by TwoExecutorsClaimingOneOperation_AreRefusedAtConstruction.

rbuergi · 2026-09-27T06:05:58Z

+    public bool IsAnswer => Answered && Fault is null && SilentProviders.Count == 0 && Partitions is not { Count: 0 };
+
+    /// <summary>Why the rows are not an answer; null when they are. Pure.</summary>
+    public string? WhyNotAnAnswer =>
+        Fault is not null ? $"the read failed — {Fault}"
+        : !Answered ? "the query completed without a complete frame"
+        : SilentProviders.Count > 0
+            ? $"provider(s) {string.Join(", ", SilentProviders)} completed without answering and were counted as empty, so the frame is a floor"
+        : Partitions is { Count: 0 } ? "the store read from NO partition — zero rows over nothing is not an answer"
+        : null;


Kept on purpose, for parity with the engine this ports. MeshWeaver.Plugins' InstanceActionControlPlane.IndexReading.IsAnswer has the same three-way contract: null coverage is UNKNOWN and said, never a floor; an empty non-null list is a floor. The in-process DeleteSpace/Recycle already run on it. Treating null as a floor would make every lane operation refuse on providers that do not report coverage (the in-memory store among them). Two things stop an unknown-coverage read from turning into a silent destructive act here. First, a zero-row space is REFUSED (IsGone → 'nothing left to delete'), never reported as a deletion. Second, the real run re-plans from a fresh read, and its digest must match the approved one. Changing the contract should happen in one place for both paths, not only in the lane.

rbuergi · 2026-09-27T06:05:59Z

+    public int TotalRows => Tables.Sum(t => t.Rows);
+
+    /// <summary>True when nothing of the space is left that a deletion removes. Pure.</summary>
+    public bool IsGone => TotalRows == 0 && Grants.Count == 0 && GitSync.Count == 0 && !RecordExists && StoreExists != true;


Same contract as the in-process engine this ports. IsGone never drives a deletion: it drives a REFUSAL ('there is nothing left … and no provider that can say whether a store exists'), so an unknown store cannot be reported as a completed deletion. After a run, an unknown store is recorded as Store: not reported in the audit, never as dropped. The plan's store step is included whenever StoreExists != false, so unknown plans the drop too. Unknown is therefore never read as 'gone'.

rbuergi · 2026-09-27T06:06:06Z

+        var query = $"path:{target} limit:1";
+        return SpaceDeletion.AsSystem(hub, () => MeshReading.Read(mesh, query))
+            .SelectMany(reading =>
+            {
+                var node = reading.Rows.FirstOrDefault(r => string.Equals(r.Path, target, StringComparison.Ordinal));
+                if (node is null)
+                    return Observable.Throw<ControlLanePreparation>(new InvalidOperationException(reading.IsAnswer
+                        ? $"there is no node at '{target}' — the index was asked ({query}) and answered without listing it. "
+                          + "Nothing was recycled: a dispose to an address with no node is a no-op, and a no-op is never reported as a recycle"
+                        : $"could NOT ESTABLISH whether '{target}' exists — {reading.WhyNotAnAnswer}. Nothing was recycled, "
+                          + "and this is NOT a statement that the target is absent"));


Same as the RecycleRunner resolution this mirrors: index for existence, and a stale negative REFUSES rather than acting. A stale NodeType classification cannot widen the blast radius unseen, for two reasons. The exact network set is now bound, and the run re-derives it and refuses on any change. Reading the node's content from the stream after the listing is a reasonable tightening for the shared resolution, but it is a follow-up for both paths.

rbuergi · 2026-09-27T06:06:07Z

+                var record = node.ContentAs<ControlLaneRecord>(execution.JsonSerializerOptions);
+                return record is null
+                    ? node
+                    : node with
+                    {
+                        Content = record with
+                        {
+                            Status = status ?? record.Status,
+                            Plan = plan ?? record.Plan,
+                            PlanDigest = plan?.Digest() ?? record.PlanDigest,
+                            UpdatedAt = DateTimeOffset.UtcNow,
+                            Log = record.Log.Add(line),
+                        },
+                    };


Fixed in a30ac2d. An unreadable ledger record now throws, so the run stops and fails. It is no longer a silent no-op update.

rbuergi · 2026-09-27T06:06:10Z

+            var reading = await AsSystem(Mesh, () => MeshReading.Read(MeshQuery, $"path:{path} limit:1"));
+            reading.IsAnswer.Should().BeTrue(reading.WhyNotAnAnswer ?? "");
+            return reading.Rows.Any(r => r.Path == path);


These reads come after the target reported Done (or a refusal). Done is only reported after the engine's own verification inventory read the space as gone. The helper also requires IsAnswer, so a floor fails the test rather than passing it. path: with limit:1 is the listing-shaped existence read; there is no content read here.

rbuergi · 2026-09-27T06:06:11Z

+    [Description("Key Vault secret NAME of the key the control lane signs requests to this deployment with")]
+    public string? ControlLaneKeySecret { get; init; }


DeploymentContent carries no [Translation] on any of its properties, and MeshWeaver.Deployment.Contract references no localization assembly. Adding the first one for a single field would add a dependency to a contract assembly. The record form's German labels are a whole-record gap, not something this field introduces.

github-actions · 2026-09-27T06:01:29Z

Test Results (shard 1)

1 690 tests 1 690 ✅ 4m 14s ⏱️
2 suites 0 💤
2 files 0 ❌

Results for commit 41913aa.

♻️ This comment has been updated with latest results.

github-actions · 2026-09-27T06:03:57Z

Test Results (shard 4)

3 files 3 suites 6m 34s ⏱️
2 171 tests 2 171 ✅ 0 💤 0 ❌
2 172 runs 2 172 ✅ 0 💤 0 ❌

Results for commit 41913aa.

♻️ This comment has been updated with latest results.

github-actions · 2026-09-27T06:04:43Z

Test Results (shard 2)

762 tests 571 ✅ 7m 15s ⏱️
3 suites 191 💤
3 files 0 ❌

Results for commit 41913aa.

♻️ This comment has been updated with latest results.

…fuse incomplete ones before disposing, one executor per operation, sanitised identities, no silent empty delete or unreadable ledger Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…(memexcloud-), so a teardown by prefix finds it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

github-actions · 2026-09-27T06:43:55Z

Test Results (shard 5)

5 files 5 suites 14m 45s ⏱️
4 253 tests 4 251 ✅ 2 💤 0 ❌
4 257 runs 4 255 ✅ 2 💤 0 ❌

Results for commit 41913aa.

♻️ This comment has been updated with latest results.

github-actions · 2026-09-27T06:44:23Z

Test Results

17 files 17 suites 36m 26s ⏱️
9 684 tests 9 491 ✅ 193 💤 0 ❌
9 693 runs 9 500 ✅ 193 💤 0 ❌

Results for commit 41913aa.

♻️ This comment has been updated with latest results.

…ySecret Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Copilot AI lite review requested due to automatic review settings September 27, 2026 05:52

meshweaver-cloud Bot enabled auto-merge September 27, 2026 05:52

Copilot started reviewing on behalf of rbuergi September 27, 2026 05:53 View session

Copilot AI reviewed Sep 27, 2026

View reviewed changes

rbuergi and others added 2 commits September 27, 2026 08:05

fix(control-lane): answer review — bind the exact recycle network, re…

a30ac2d

…fuse incomplete ones before disposing, one executor per operation, sanitised identities, no silent empty delete or unreadable ledger Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs(control-lane): the vault object carries the deployment's prefix …

7ab169e

…(memexcloud-), so a teardown by prefix finds it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs(control-lane): the renderer parity table names WithControlLaneKe…

41913aa

…ySecret Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

meshweaver-cloud Bot added this pull request to the merge queue Sep 27, 2026

Merged via the queue into main with commit 021c117 Sep 27, 2026
43 checks passed

		[Description("Key Vault secret NAME of the key the control lane signs requests to this deployment with")]
		public string? ControlLaneKeySecret { get; init; }

Conversation

rbuergi commented Sep 27, 2026

What

Verification

Cross-repo

Uh oh!

github-actions Bot commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (shard 0)

Uh oh!

github-actions Bot commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (shard 3)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (shard 1)

Uh oh!

github-actions Bot commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (shard 4)

Uh oh!

github-actions Bot commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (shard 2)

Uh oh!

github-actions Bot commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (shard 5)

Uh oh!

github-actions Bot commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Sep 27, 2026 •

edited

Loading

github-actions Bot commented Sep 27, 2026 •

edited

Loading

github-actions Bot commented Sep 27, 2026 •

edited

Loading

github-actions Bot commented Sep 27, 2026 •

edited

Loading

github-actions Bot commented Sep 27, 2026 •

edited

Loading

github-actions Bot commented Sep 27, 2026 •

edited

Loading

github-actions Bot commented Sep 27, 2026 •

edited

Loading