Repository navigation
feat(observability): make the sandbox event stream verifiable — sequence numbers and a hash chain #3817
Description
User Story
As an operator who must show an auditor or incident reviewer what a sandboxed agent did,
I want the sandbox event stream OpenShell produces to prove its own completeness,
so that a removed, lost or reordered event is detectable instead of silent.
Problem Statement
In #2762 the direction was that OpenShell should "produce a durable source of OCSF events that something else can slurp." Today, whatever slurps that source cannot tell whether it received all of it.
The sandbox event stream has no sequence numbers, no chaining between events, and no record of events lost between supervisor and gateway. Timestamps are not a substitute: in a live capture, 7 of 75 adjacent lines ran backwards in time because gateway and sandbox lines interleave.
Impact / Why This Matters
I ran OpenShell 0.1.2 (Docker driver) on a GitHub-hosted runner: one sandbox under the default policy, then under the quickstart policy. Enforcement behaved exactly as documented: the GitHub API was denied then allowed, example.com denied throughout, every denial carried a reason, the policy reload was logged with its full hash, and the stream was held outside the agent's reach.
But removing the six DENIED lines from that capture produces a log no downstream check can distinguish from the original. The runtime blocked the requests; the record cannot show later that they happened.
Today users must trust that nothing between the gateway and their SIEM, and nothing in retention or handling afterwards, dropped or removed an event. For audits, incident review and EU AI Act Article 12 record-keeping, the question is not only "was it blocked?" but "can the record prove it is complete?" Right now it cannot.
Capture (raw output, policy used, install script checksum, converter): https://github.com/MattyIceMatrix/vlc-1/tree/main/examples/third-party/openshell-live
Proposed Design
From the consumer's side, holding only the exported event stream:
- Every event from a sandbox carries a gapless per-sandbox sequence number, so a missing event shows up as a gap.
- Every event is linked to the one before it (for example, a digest over the event plus the previous event's digest), so deleting, inserting or reordering any event is detectable offline, without trusting the transport or the store.
- When events are lost (buffer full, restart), the stream contains an explicit record saying how many and over what interval, instead of nothing.
- Optionally, each allow/deny event names the hash of the policy it was decided under, not only the CONFIG:LOADED event, so a decision stays tied to its policy even if events are reordered.
Enforcement and export scope are unchanged; this makes the durable source trustworthy for whatever consumes it.
Acceptance Criteria
- Every event from a sandbox carries a gapless sequence number
- Deleting, inserting or reordering any event is detectable from the stream alone
- Event loss produces an explicit, counted drop record
- (Optional) allow/deny events carry the active policy hash
Alternatives Considered
- Rely on the SIEM or log store for integrity: protects the data after it arrives, but cannot show that everything arrived, and loss before ingestion stays invisible.
- Sign the export at the gateway: proves the file came from the gateway, but a signed file missing events still verifies; signing without sequence or chaining does not reveal gaps.
- Use timestamps to spot gaps: unreliable, since gateway and sandbox lines interleave out of order in practice.
Per-event sequence plus chaining is the smallest change that lets any consumer verify completeness independently.
Reference, if a concrete target helps: VLC-1, an open, public-domain (CC0) spec and conformance checker for verifiable log completeness, https://doi.org/10.5281/zenodo.22728393. Disclosure: I maintain it, and I'm happy to test a change against the checker.
Agent Investigation
No response
Checklist
- I've reviewed existing issues and the architecture docs
- This is a design proposal, not a "please build this" request
Update, 2026-10-18: design refined in this thread
This section folds in what was refined in the comments and in the related issues. The original text above is unchanged.
Completeness
- A gapless sequence number on every event. The comments here proposed counting the supervisor and the gateway separately, which also explains why timestamps interleave out of order in the live capture. Per feat(observability): export an atomic sandbox governance evidence bundle #2745, the number is best stamped at
ocsf_emit!, so the window for undetectable suppression is as small as the architecture allows. - Drop records that state the sequence range they cover, so a counted loss can be told apart from an event removed later. (Today the OTLP export emits an
openshell.ocsf.gaprecord with a dropped count and the time of the first loss, per the maintainer on Correlating sandbox OCSF events with agent outputs for audit use #4658. A range is what is still missing.) - First and last sequence number recorded at the start and end of each sandbox's stream (lifecycle events), so a missing stream or a truncated one is visible.
Integrity
- A hash chain over the sequenced events. It can be built by the consumer on top of the sequence; chaining at the source is stronger and can come later.
Boundary
- A gapless sequence shows nothing was removed after a number was assigned. An event suppressed before
ocsf_emit!leaves no gap and no trace, so a clean range means "verified over the covered range", not proof that everything that happened was exported. feat(observability): export an atomic sandbox governance evidence bundle #2745 proposes a three-valued verifier result (verified complete over the range / provable gap / not established); I think the same distinction belongs here.
Related, tracked elsewhere
- Policy generation on each decision: feat(observability): record the policy generation on allow and deny decisions #3840.
- Signed teardown statement for a sandbox: feat(gateway): signed teardown statement for a sandbox, with a pluggable signer. #4421.
- Correlation, completeness and integrity questions for audit use: Correlating sandbox OCSF events with agent outputs for audit use #4658.
- Who authorized a policy and delegation lineage: out of scope here. A separate issue is suggested, linking this one and feat(policy): revalidate running subagents when ancestor authority changes #3808.
Known gaps noted by the maintainer (#4658)
- Local JSONL and stderr output can drop events without recording that they did.
- Records lost during shutdown produce no gap record.
Acceptance criteria (updated)
- Every event from a sandbox carries a gapless sequence number
- Drop records state the sequence range they cover
- Each stream records its first and last sequence number
- Deleting, inserting or reordering any event is detectable from the stream alone
- A removed event is reported as a provable gap, and an interval in which nothing was emitted is reported as not established, not complete (cases proposed in feat(observability): export an atomic sandbox governance evidence bundle #2745)
- (Optional) allow/deny events carry the active policy hash or generation
Credit: the per-sandbox metadata.sequence stamped at ocsf_emit! and the three-valued verifier result come from #2745 (@lbelyaev, @imran-siddique); lifecycle bounds and ranged drop records from @rakeshgohel01; the audit workflow from @certanum.
I'll re-run the same live capture and post a before/after score when any of this lands.
Activity
THIRD-PARTY: link the OpenShell finding to NVIDIA/OpenShell#3817
rakeshgohel01 commentedon Sep 29, 2026
+1, with two additions and one related item. This matters most for regulated deployments where decision records have to hold up in review after the fact.
In #2745 (since closed), @lbelyaev and @imran-siddique converged on a per-sandbox
metadata.sequencestamped atocsf_emit!, so the only case a consumer can't check is an event suppressed before emit. That placement seems worth carrying over here. It also fits the scope set on #2762: OpenShell produces the durable source, and signing and export stay with the consumer.First addition: a per-sandbox sequence shows a gap inside a stream, but not a stream that never arrived or one cut off after its last delivered event. The gateway knows every sandbox it created. A lifecycle event at creation and at teardown, carrying that sandbox's first and final sequence numbers, would let a consumer check that it holds every stream it should and that each one ends where it really ended.
Second addition: drop records should carry the sequence range they cover, so a counted loss and a removed event look different to a verifier.
On the optional item, tying each decision to its policy: it doesn't depend on sequencing. The engine already knows which policy generation evaluated each request, but most decision events don't record it. Transparent TCP allows keep it in
unmapped, and JSON-RPC decisions only mention it in the message text. I'll open it as a separate issue. I have a patch that records the generation on each decision and on the policy load event that carries the hash. That's one step short of the hash on every event, but it lets a consumer join a decision to its policy without timestamps.MattyIceMatrix commentedon Sep 29, 2026
Thanks both, this is a better design than my original ask.
The two proposals fit together. shleder's per-emitter counters (supervisor vs gateway) also explain what the live capture showed: 7 of 75 adjacent lines out of timestamp order, because the two streams interleave on arrival. And rakesh's lifecycle events with first/last sequence, plus drop records carrying the sequence range they cover, are what let a verifier tell a counted loss from a removed event and a complete stream from a truncated one.
On where the hash chain lives: if OpenShell emits the per-emitter sequence, the lifecycle bounds and ranged drop records at the source, a consumer can build the chain and signature on top, which keeps within the #2762 scope. Chaining in the supervisor is stronger but can come later.
@rakeshgohel01, thanks for opening the policy-generation item. When any of this lands I'll re-run the same live capture (branch capture/openshell-ocsf in MattyIceMatrix/vlc-1) and post a before/after score here: the full log against the log with the DENIED lines removed, so there's an independent check that the deletion becomes visible
bba-git commentedon Sep 30, 2026
The design as it stands after the three comments answers two of the three questions a reviewer asks of this stream.
The third question is the one #2025 and #3808 define (
child_policy <= parent_delegable_view <= parent_boundary <= human_admin_maximum_policy) and #2109 already hashes (candidate/delta hashes on approvals). None of that reaches the exported events, so a consumer holding a complete, chained stream still cannot show that a denied or allowed request traces to a human-approved grant.One addition keeps it inside the #2762 scope, with no new chain:
parentreference: the hash of the grant or approval it was admitted under.That gives a consumer three offline checks over the same bytes: completeness (VLC-1), decision-to-policy (the generation patch), policy-to-principal (a recursive parent walk). Revocation under #3808 is then a visible break in the walk rather than a missing line.
Reference, if a concrete shape helps: VAL, an Apache-2.0 spec whose lineage invariant is exactly rule 3 (every non-root block carries a
parent_assignment_hashrecursing to a human-signed root) with an offline verifier that re-derives it from exported bytes, https://github.com/val-protocol/verifiable-authorization-lineage. It is transport- and storage-agnostic, so it fits behind OCSF export as written here. Disclosure: I maintain it.github-actions commentedon Oct 15, 2026
This issue has had no activity for 14 days and is now marked stale. It may be closed in 14 days if there is no further activity. Comment or remove the state:stale label to keep it open.
certanum commentedon Oct 17, 2026
Adding an audit workflow from regulated use (clinical research), as
requested in #4658.
An independent reviewer receives an agent output, possibly months after the
run, together with the exported OCSF records for the sandbox that produced
it. To accept those records as evidence, the reviewer needs to establish:
loss is explicitly counted with the sequence range it covers, so a
counted loss can be told apart from an event removed later.
The signed teardown statement in feat(gateway): signed teardown statement for a sandbox, with a pluggable signer. #4421 would cover this.
trust the operator or the system the logs were exported to. Retention
periods in this domain are long, and the reviewer is often a separate
party.
Sequence numbers with drop records stating their range, plus a hash chain
anchored by a signed teardown statement, would meet 1–3. Point 4 is the one
that decides whether exported logs can serve as audit evidence.
Certanum Engineering
MattyIceMatrix commentedon Oct 18, 2026
Thanks @certanum, @bba-git, @rakeshgohel01. I read the maintainer's answer on #4658 as well, so here is one summary of where this stands and what I think belongs in this issue.
What exists
OTLP export on main emits an
openshell.ocsf.gaprecord with a dropped count and the time of the first loss (per the maintainer on #4658; I haven't read the code). The known gaps the maintainer listed matter for audit use: local JSONL and stderr can drop silently, and records lost at shutdown leave no gap record.What #3817 still needs for completeness
Integrity (a chain, built at the source or by the consumer) follows from 1 to 3. Chaining at the source is stronger and can come later.
Scope (#4421)
As described on #4658, a signed teardown statement binding the sandbox, image, policy, how it ended and the last sequence number would make point 3 checkable. Its last-sequence field is what ties it to the stream.
Independence
The maintainer says the trust boundary is, for now, expected to be the system the logs are exported to. For an auditor that means completeness, integrity and scope can be checked offline, but independence of the operator can't: a chain and a signature from the same operator show consistency, not independence. A signature adds independence only if the signer is outside the operator's control.
Authorization lineage (@bba-git's third question)
A fair gap, and I'd keep it out of this issue's acceptance criteria. It depends on the sequence landing first. If delegation or grant events are added later, they should be ordinary events in the same sequenced stream, so the completeness check covers them without a second mechanism. I'd suggest a separate issue linking this one and #3808. To be precise about my side: VLC-1 checks completeness of the delivered log and that decisions are bound to the policy that produced them. It doesn't check who authorized a policy or whether a child stayed inside its parent's delegable view. I haven't reviewed the VAL verifier.
Next step on my side
When any of this lands I'll re-run the same live capture (branch
capture/openshell-ocsfin MattyIceMatrix/vlc-1) and post a before/after score: the full log against the log with the DENIED lines removed, so the deletion is shown to become visible.Disclosure: I maintain VLC-1 (CC0 spec, MIT checker): https://doi.org/10.5281/zenodo.22728393
MattyIceMatrix commentedon Oct 18, 2026
Quick follow-up, and thank you all. This thread has taken my original ask from "add sequence numbers" to a much better design than I started with, and it's been a pleasure to see it come together.
@certanum, your audit workflow is the clearest statement of the requirement so far. @rakeshgohel01, the first/last sequence idea is what makes a truncated stream visible. @bba-git, the third question is a good one, and I think it deserves its own issue so it gets proper attention.
If it would help anyone, I'm happy to test a branch against the checker, or to write up the live capture as a small before/after example for the docs. Just tag me.
Thanks again.
certanum commentedon Oct 18, 2026
@MattyIceMatrix -Thank you, and thanks for pulling the thread together so clearly.
Agreed on all points, and especially on independence: for audit use, a
chain and signature from the operator alone show consistency, not
independence. That is the requirement we would most like to see addressed
in the long run, even if it sits outside this issue.
We'll follow #4421 and the sequence work here, and are happy to describe the
reviewer's side of the workflow in more detail if that helps.
Certanum Engineering