Repository navigation
spike: TXE runs world state + AVM fully in-process (Slice C) - #24642
Closed
charlielye wants to merge 17 commits into
Closed
charlielye wants to merge 17 commits into
charlielye wants to merge 17 commits into
Conversation
Replace the in-process NAPI AVM with an out-of-process bb-avm-sim binary reached over IPC. The simulator drives a pool of bb-avm-sim processes (AvmSimulatorPool) behind an AvmExecutor that also runs the CDB server answering the C++ AVM's contract-data callbacks; per-fork work goes through AvmExecutor.forFork. The pool prewarms one process on spawn so the first simulate() runs at steady-state cost. Also adds the bench_compare script + bench-regression skill for PR perf checks.
The public-execution wiring threaded a concrete `AvmExecutor` (pool + CDB
server) through the node, prover-node, checkpoint builder, and TXE, and each
call site called `forFork(forkId, contractsDB, timestamp)` to get a
fork-bound `ForkedAvmSimulator`. That leaked the IPC transport (the CDB
server, its fork-id routing key, and the register/unregister dance) into
every consumer of public simulation.
Thread a single transport-agnostic `AvmSimulator` interface instead, mirroring
the pre-IPC shape where the factory threaded `(fork, contractsDB)` and called
an in-process simulate. `simulate` now takes an `AvmContractsDBContext`
({contractsDB, forkId, timestamp}); the implementation owns the CDB
register -> run -> unregister lifecycle. The pool (`AvmSimulatorPool`) folds in
the CDB server and is the sole concrete `AvmSimulator`, spawned only at
program entrypoints. `AvmExecutor`, `forFork`, and `ForkedAvmSimulator` are
gone.
`PublicTxSimulatorBase` gains `contractsDB`; consumers (PublicProcessorFactory,
prover-node, checkpoint builder, TXE synchronizer) hold the `AvmSimulator`
interface and pass the DBs, deriving forkId from the fork's revision.
…rrency Two leaks/bugs surfaced by a full-CI process-tree capture of the AVM cutover: 1. TXE session leak. `TXESession.dispose()` closed the world state and LMDB but never stopped the synchronizer's AVM pool, so every test orphaned its prewarmed bb-avm-sim process — and, because that process kept a live connection to the WSDB server, pinned the WSDB alive too. Over a run this accumulated to hundreds of undisposed process-triples. Dispose the pool (`closeIpc()`) first, so the bb-avm-sim processes exit and release their WSDB connections before the WSDB is closed. 2. Pool overshoot under concurrency. `AvmSimulatorPool` bumped `createdCount` only after the `await` that spawns a process, so concurrent checkouts could all observe `createdCount < maxSize` and each spawn, growing the pool past maxSize (a pool was seen holding maxSize+1 processes in CI). Reserve the count synchronously at the top of `createSlot`, before the spawn, and roll it back if the spawn fails, so the maxSize guard is exclusive.
Each TXE test spins up its own tiny, short-lived world state — dozens concurrently — and each spawned aztec-wsdb was sized to one thread per core (min(16, cpus)), used for BOTH the tree-op pool AND the IPC dispatch pool, so a 16-core box gave ~32 threads per wsdb. The trees are cache-sized (256MB map, tiny state), so those threads buy nothing and multiply into thousands across a test run. Thread a `threads` option through IpcWorldState.spawn -> NativeWorldStateService .tmp (default unchanged: min(16, cpus)), and have `ephemeral()` — used only by the TXE — request 1. That yields a single tree-op thread; the C++ IPC dispatch pool still floors at 2 (max(2, threads)), which is the minimum that avoids the dispatch/tree-pool deadlock. Net: ~32 -> ~3 threads per TXE wsdb, production paths untouched.
…+ services Adds `ipc_codegen/in_process.hpp` (`dispatch_sync`) to the C++ template dir the codegen copies into every generated service. It drives a service's generated `make_<prefix>_handler(ctx)` synchronously across a direct function-call boundary (NAPI/FFI) instead of a socket — the in-process counterpart to the `<prefix>_ipc_server.hpp` socket serve loop. The concrete entry point (a NAPI method or named FFI symbol) is instantiated by the host per service, deliberately not a single global `ipc_ffi_entry`, so multiple services can be co-hosted in one process without symbol collisions. First building block of the TXE in-process spike (run the AVM + world state in the TXE's own process, no bb-avm-sim/aztec-wsdb child processes).
Carves the AVM simulator's dispatch + command handlers out of the bb-avm-sim executable into a transport-agnostic `avm_sim_ffi` library, and adds a plain-C ABI (`avm_ffi.h`: avm_create_ipc / avm_call / avm_destroy) that runs a simulation in-process via the codegen `dispatch_sync` primitive — no socket. The bb-avm-sim executable now just links the library and wraps its dispatch in the socket serve loop (out-of-process path unchanged). Any in-process host — a small NAPI wrapper generated up the stack, pointed at this library — is the other consumer. Barretenberg itself stays free of any Node/NAPI knowledge; the ABI is plain C. avm_execute.hpp gains explicit-specialization declarations for the AvmRequest handlers so both the executable and the library link against them rather than implicitly instantiating the definition-less primary template. Builds green (AVM=ON): libavm_sim_ffi.a exports the three C symbols and bb-avm-sim relinks onto it.
A ~90-line node-addon-api wrapper (`InProcessAvm`) over barretenberg's avm_sim_ffi C ABI. `new InProcessAvm(wsdbPath, cdbPath)` → `avm_create_ipc`; `.call(requestFrame)` → `avm_call` (dispatch_sync, in-process, no subprocess); `.destroy()` → `avm_destroy`. Generic byte-in/byte-out `call`; the only AVM-specific bit is the constructor's dep wiring. Builds standalone with barretenberg's zig toolchain (ABI match) and links its static archives into the .node — no shared lib, no PIC issues (bb is built -fPIC). Barretenberg stays Node/NAPI-free; this wrapper is the in-process consumer of its C ABI, mirroring bb-avm-sim as the out-of-process consumer. Verified: builds, loads in Node, exposes InProcessAvm. This proves the whole native chain (bb C ABI → linked .node → loadable) — the spike's riskiest part. The generalized form (codegen emits this given a lib path) is the follow-up; TS backend + InProcessAvmSimulator + TXE wiring come next.
Wires the in-process AVM through to a working round-trip, behind the existing AvmSimulator interface so nothing downstream changes. - NAPI wrapper `call` now runs avm_call on a worker thread and returns a Promise (was synchronous). Mandatory even for Slice A: the CDB server runs on the JS event loop, so a synchronous AVM call on that thread would block the loop and deadlock against its own CDB requests. Off-thread frees the loop to serve CDB. - `NapiAvmSimulator implements AvmSimulator` — sibling of AvmSimulatorPool: owns the same CDB reverse channel (CdbIpcServer + per-simulate fork registration) and drives the generated AvmService via a NapiBackend (IpcClientAsync over the addon), so the in-process path runs behind the codegen'd API, not bespoke. Only the transport differs: an in-process worker thread vs a child process. - CdbIpcServer.ready() so the addon's synchronous connect waits for the socket to bind. - TXESynchronizer + PublicProcessorTestEnv select the in-process backend behind TXE_IN_PROCESS / AVM_INPROCESS_NODE; the pool remains the default. Verified: the public-processor deployments test (deploy + public calls + CDB callbacks + block-cache revert) passes with AVM_INPROCESS_NODE set — identical results to the subprocess pool, no bb-avm-sim child process. Slice A done.
Adds the generic host_call(target, bytes) reverse channel so an in-process AVM can reach contract data via a direct host callback instead of a CDB socket: - avm_ffi.h gains `avm_host_call_fn` (the native twin of the wasm host_call import — same (target, req, resp) contract) and `avm_create_hostcall`. - HostCallContractDB implements ContractDBInterface over that proxy, speaking the exact same CDB wire protocol/framing as CdbIpcContractDB — only the transport differs (host callback vs socket). - AvmRequest is decoupled from the concrete CDB: it now holds a ContractDBInterface& plus a set_fork_id closure (fork routing is a transport concern, not part of the DB interface), so the socket and host_call CDBs both slot in. bb-avm-sim's out-of-process path is updated to match and still links. Builds green (AVM=ON). NAPI TSFN delivery of host_call + the TS host router (→ CdbIpcServer dispatch) come next; wasm reuses the same (target, bytes) shape.
Completes the generic host_call(target, bytes) reverse channel so the in-process AVM reaches contract data by calling straight into the host, dropping the CDB socket: - host_call callback gains a `void* ctx` (standard C-callback pattern) so the NAPI wrapper can route it to the owning instance's ThreadSafeFunction. - NAPI wrapper: `new InProcessAvm(wsdbPath, onHostCall)` builds a TSFN over the JS onHostCall; the AVM's worker-thread host_call blocks on a std::promise the TSFN callback fulfils — handling onHostCall returning a Promise (the CDB read is async). This is the native twin of the wasm host_call import; same (target, bytes) contract, so the future wasm build reuses the router. - CdbIpcServer can run socketless (listenSocket=false) and exposes handle(bytes) = its generated dispatch; NapiAvmSimulator routes host_call target CDB→handle. registerFork/unregisterFork unchanged. Verified: the public-processor deployments test (deploy + public calls + CDB lookups) passes with the AVM in-process AND no CDB socket. Only the wsdb socket remains between the in-process AVM and the host (Slice C).
…ice C) The generated C++ IPC client was hardwired to a socket (make_client by path suffix). Add a `Transport` (byte request→response function) and a constructor that takes one, routing the per-command `send<>` through it. The path constructor is unchanged — it now just installs a socket-backed Transport — so every existing consumer (CdbIpcContractDB, WsdbIpcMerkleDB, bb-avm-sim) is unaffected. This is the enabler for an in-process AVM to reach a co-hosted world-state server directly (C++→C++) instead of over a socket: WsdbIpcMerkleDB can wrap a WsdbIpcClient built over an in-process transport that drives the wsdb dispatch of a shared WorldState — the last step to zero child processes for the TXE (Slice C). Mirrors the zig/rust clients, which are already backend-parameterized. Builds green (AVM=ON): cdb + wsdb clients regenerate with the injectable transport and all consumers relink; the socket path is untouched.
WsdbScheduler took an `ipc::IpcServer&` but used it for exactly one thing: the `has_pending_request()` check that gates the inline fast path. Replace it with an injected `std::function<bool()>` predicate, so the scheduler (read-batch / write-barrier per-fork ordering + thread pool) is transport-agnostic — unchanged behaviour, but no longer tied to the socket server. This lets the wsdb dispatch run behind a plain C ABI: the socket server passes its `has_pending_request()`, and an in-process (FFI/NAPI) host passes its own predicate. First step toward running wsdb in-process (shared WorldState with the in-process AVM) via the same generalized-backend model as the AVM, rather than reverting to a pre-wire binding. aztec-wsdb relinks; the socket path is unchanged.
Mirror avm_sim_ffi: move the WorldState, per-fork scheduler, and generated dispatch behind a plain-C ABI (wsdb_create / wsdb_call / wsdb_destroy) in a new wsdb_ffi static library. wsdb_call is async — it decodes and schedules synchronously but delivers the response through a callback, so the socket server's concurrency is preserved. aztec-wsdb becomes a thin socket adapter that feeds wire frames into wsdb_call; an in-process host (NAPI wrapper, or a co-linked AVM sharing the WorldState) drives the same ABI without a socket.
The Slice C AVM shape: no sockets. World state is reached through a synchronous wsdb byte transport (avm_wsdb_call_fn — the host drives a co-hosted WorldState's dispatch, so leaf reads stay C++<->C++ and never bounce through JS), and contract data through the existing host_call proxy. The AVM sees a plain WsdbIpcClient built via its Transport ctor and does not know world state is in-process. avm_sim_ffi stays free of any wsdb_ffi / world_state dependency, so bb-avm-sim is unaffected.
Add an InProcessWsdb ObjectWrap over the wsdb_ffi C ABI: it owns a WorldState in-process and exposes call()/destroy() so a generated WsdbService (AsyncApi) drives it exactly like the spawned aztec-wsdb backend, no child process. Add a third InProcessAvm constructor — (inProcessWsdb, onHostCall) — that co-hosts the AVM against the SAME WorldState via avm_create_inprocess: world-state reads go C++<->C++ through a synchronous wsdb trampoline (never bouncing to JS), contract data via host_call. The scheduler's inline fast path is disabled (always-pending predicate) since in-process requests arrive on multiple threads. Together: a TXE session with zero child processes.
…ce C) Wire the TS side so TXE_IN_PROCESS runs both the world state and the AVM in one process sharing a single WorldState, with zero child processes: - world-state: loosen IpcWorldState to drive any AsyncApi (a spawned WsdbService or an in-process NAPI backend); getIpcPath guards on absence. Add NativeWorldStateService.fromWsdbBackend(IpcClientAsync) and buildInProcessWsdbOptions (the in-process twin of getWsdbExtraArgs). - simulator: createInProcessWsdb + NapiWsdbBackend (drive the wsdb AsyncApi over the addon's InProcessWsdb), and NapiAvmSimulator.spawnCoHosted, which hands that same InProcessWsdb to the AVM so both share one WorldState. - txe: TXESynchronizer.createInProcess() creates the shared InProcessWsdb, builds the world state over a NapiWsdbBackend, and co-hosts the AVM. Session teardown already disposes the AVM before closing the world state. TXE_IN_PROCESS now means full in-process (was AVM-only).
Capture the gap between the Slice C spike and a shippable feature: the concurrency model + lifetime safety the spike papered over (highest risk), build/packaging/distribution, the codegen generalization, and config/rollout/ measurement — with a priority order and a note that #3 (codegen) only pays off if the mechanism is reused.
charlielye
force-pushed
the
cl/ipc-5-avm-cutover
branch
5 times, most recently
from
July 17, 2026 10:34
38d53fe to
96082e3
Compare
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Spike: TXE runs world state + AVM fully in-process (zero child processes)
Draft / spike — proves feasibility and surfaces the edges; productionization is a follow-up.
The AVM cutover made public execution an out-of-process
bb-avm-simservice and world state anaztec-wsdbservice. Right for production nodes; heavy for the TXE, where each of hundreds of Noir tests spins up its own(aztec-wsdb + bb-avm-sim + cdb)process-triple. Pre-cutover the TXE ran both in-process.This spike makes where a service runs a backend choice expressed through the codegen service model, and has the TXE choose to run both the AVM and world state in one process, sharing a single
WorldState— recovering the pre-cutover topology while production keeps the out-of-process backends unchanged. SetTXE_IN_PROCESS=1.How it fits the service model
wsdb_ffi,avm_sim_ffi). The out-of-process executables (aztec-wsdb,bb-avm-sim) and a small NAPI wrapper are both thin consumers of the same dispatch — the socket server just feeds wire frames into the FFI entry.create/call/destroyis the symmetric backend lifecycle (process for IPC, thread/instance for NAPI).host_call(target, bytes)is the generic reverse channel (same contract as the wasmonHostCall): for NAPI it reaches TS-resident services (the CDB). World state stays C++↔C++ (a synchronous byte transport), so leaf reads never bounce through JS.Shape
wsdb_ffi:wsdb_create/wsdb_call(async) /wsdb_destroy;aztec-wsdbbecomes a thin socket adapter over it.avm_create_inprocess: the AVM reaches world state via a sync wsdb byte-callback and contract data viahost_call.avm_sim_ffistays free ofworld_state, sobb-avm-simis unaffected.avm_inprocessaddon co-hostsInProcessWsdb+ a co-hostingInProcessAvm(inProcessWsdb, onHostCall)— one sharedWorldState.NativeWorldStateService.fromWsdbBackend+NapiWsdbBackend;NapiAvmSimulator.spawnCoHosted;TXESynchronizer.createInProcess().Verification
aztec-wsdb/bb-avm-simstill build and run.public_immutableread + init,nft transfer_in_publicwrite) with noaztec-wsdband nobb-avm-simchild processes.Non-goals (spike)
Production node in-process;
MemoryMerkleDB/ no-disk world state; folding the hand-written NAPI wrapper into the codegen; browser/wasm backend.