Repository navigation
atlas-core: core search library (Atlas epic #5350 step 2, #5352) - #5372
Merged
Merged
Conversation
![developing-today[bot]](https://s.lequ.pw/_assets/images/gravatars/gravatar-user-420.png){kind=avatar}
The search pipeline as a library, a semantics-exact port of the production Python atlas_search.py, building on step 1's atlas-embed (#5351): - Filters + sensitivity ceilings (shareable < personal < local-only < secret-bearing; default personal; NULL never passes — fail-closed), enforced inside branch construction (filter-before-rank) on both backends. - FTS branch + vector branch, RRF fusion k=10, pool 100/branch defaults; envelope rendering (flat, grouped, get-chunk) byte-equal to the Python's. - All three vector backends behind the SearchBackend trait: exact scan (default; the production path), index fetch-to-fill (depth doubles to a 3200 cap until the pool survives the filter), index fixed-fetch. The default flips only at #5356 from A/B evidence. - Turso over the HTTPS /v2/pipeline API, read-only token flow (env -> Bitwarden -> 20h cache -> turso CLI mint); the library never issues writes. Local mode against SQLite publish files with temp-view schema normalization and vector side stores joined by content hash (never chunk_id). - Query embedding via atlas-embed at the library boundary: documents prompt-free, queries get the "query: " prefix; the embedder session is &mut self, so AtlasEmbedder serializes on a mutex — one resident session per process by design. Verification (step gate): - 16 unit tests (RRF decompositions, ceiling clamping, filter-before-rank, fetch-to-fill escalation vs a recording fake, round6/Python float parity). - 4 parity tests against fixtures recorded from the production Python (gen_core_fixtures.py): all 12 search envelopes and 2 get-chunk envelopes byte-equal, Turso branch SQL byte-for-byte, allowed-ids / index-fill / index-fixed call sequences exact. - Live smoke (examples/live_smoke.rs, read-only, production corpus): 6 of 6 diverse questions — Rust scan-path fused top-10 identical to the deployed Python CLI's. - cargo test --workspace green, clippy 0 warnings, fmt clean.
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 11 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This was referenced Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 2 of the Atlas binary/service epic (#5350): the core search library, per #5352 and design §4/§5. Builds on step 1's merged
atlas-embed(#5351, Gate A bit-identity).What
crates/atlas-core— the search pipeline as a library, a semantics-exact port of the production Pythonatlas_search.py:"query: "prompt is applied at this library boundary) → FTS branch + vector branch → RRF fusion (k=10, pool 100/branch defaults) → envelope (flat / grouped-by-session / get-chunk), rendered byte-equal to the Python's.shareable < personal < local-only < secret-bearing, defaultpersonal, NULL sensitivity never passes (fail-closed) — mirroring--max-sensitivityexactly.vector_distance_cosover the allowed set — production path, default), index fetch-to-fill (depth doubles to a 3200 cap until ≥pool survivors), index fixed-fetch. Which index strategy becomes default is decided at Atlas step 6: golden-parity gate (Gate B) + cutover #5356 from the A/B — nothing hard-defaulted here./v2/pipelineAPI with the read-only token flow (env → Bitwarden → 20h cache →tursoCLI mint); the library never issues writes. Local mode against SQLite publish files: temp-view schema normalization (labels/sensitivity from the file or an attached sibling publish copy), side stores joined by content hash.atlas-protocrate in this step: the design assigns the shared protocol types to step 4 (Atlas step 4: serve mode — job protocol, worker pool, in-memory temp store #5354), where the serve protocol shapes them; the library's envelope isserde_json::Valueuntil then.Gate evidence
round()parity for rendered floats.fixtures/gen_core_fixtures.py→core-expectations.json): all 12 search envelopes and both get-chunk envelopes byte-equal; Turso branch SQL byte-for-byte across 4 filter shapes;allowed_ids/ index-fill / index-fixed call sequences exact against a recording transport. The fixture corpus's canary (secret-bearing chunk holding the query's exact vector) is absent at the default ceiling and lands at vector rank 1 under a spelled-out secret-bearing ceiling.crates/atlas-core/examples/live_smoke.rs, read-only, production corpus, scan backend): 6 of 6 diverse questions — Rust fused top-10 identical to the deployed Python CLI's:cargo test --workspacegreen (atlas-embed untouched),cargo clippy --workspace --all-targets0 warnings,cargo fmt --checkclean.Notes for steps 3–4
Session::runtakes&mut self:AtlasEmbedderserializes queries on a mutex around ONE resident session (~1.6 GB RSS). Serve mode (Atlas step 4: serve mode — job protocol, worker pool, in-memory temp store #5354) should hold oneAtlasEmbedderper process (or a small pool behind the same shape) — never an embedder per request; embedding is the serialization point, branch execution is I/O-parallel.libstdc++.so.6+libz.so.1on the loader path (observed with the v3 venv's libonnxruntime) — Atlas step 5: nix packaging (pkgs/atlas-search) + amd deploy #5355's nix closure must carry both; noted in the example's docs.Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.