Skip to content

atlas-search: step 3 - CLI mode at Python surface parity (#5353) - #5377

Merged
dezren39 merged 1 commit into
mainfrom
atlas-step3-cli
Nov 4, 2026
Merged

dezren39 merged 1 commit into
mainfrom
atlas-step3-cli

Conversation

@dezren39

@dezren39 dezren39 commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Step 3 of the Atlas binary program (epic #5350), closing #5353: the atlas-search binary crate — the one-shot CLI at Python surface parity, built on atlas-core (step 2, #5372) and atlas-embed (step 1, #5367). Core semantics untouched.

What was built

  • search + get-chunk subcommands (clap) mirroring atlas_search.py's argparse surface: same flags, defaults, output envelope, --text rendering, and exit behavior (die() fatals → atlas-search: <msg> on stderr, exit 2; get-chunk not-found → exit 1; backend/transport failures → exit 1 with the message instead of a traceback).
  • Standalone by default and complete: direct Turso via atlas-core's read-only token chain, or --backend local against a publish file / side stores via LocalBackend. One lazily-loaded embedder per process — FTS-only search and get-chunk never touch the model; vector/hybrid without a model/ORT fail loudly (design §4 degradation parity).
  • Output is byte-identical to the Python: a hand-rolled CPython-compatible JSON writer (json.dumps(indent=1, ensure_ascii=False), CPython float repr — serde_json prints 0.000087 where CPython prints 8.7e-05) and a line-for-line render_text port.
  • --via-service (Drew Q3): prefer-with-loud-fallback implemented now — probes the service base URL (--service-url / ATLAS_SEARCH_URL / ATLAS_SEARCH_VIA=service, default http://127.0.0.1:18760), and on any failure prints the warning to stderr and records it in the envelope notes, then serves standalone. The job-protocol client lands behind this seam in step 4 (Atlas step 4: serve mode — job protocol, worker pool, in-memory temp store #5354); no machine detection in the binary (install-context defaults are Atlas step 5: nix packaging (pkgs/atlas-search) + amd deploy #5355 packaging).

Flag parity (Python → Rust)

Python flag Rust Notes
query / chunk_id positionals mirrored
--k, --pool, --mode, --kind, --topic, --project, --client, --since, --until, --max-sensitivity, --group-by-session, --expand, --with-message, --text mirrored identical defaults + validation messages (via atlas-core)
--backend, --db, --vecdb mirrored auto = token chain → turso, else local (same order as the Python)
--model (env ATLAS_SEARCH_MODEL), --vec-table (env ATLAS_VEC_TABLE) mirrored env fallbacks preserved via clap
--trunc, --vector-backend mirrored scan default; index modes ride atlas-core
--embedder mirrored, with a declared divergence flag + provenance label identical; a fastembed embedder or a non-Snowflake --model cannot be served by atlas-embed (Snowflake ST path only) and fails loudly for vector/hybrid, pointing at the Python CLI — never a silent wrong-model embedding. FTS/get-chunk unaffected.
— (new) --via-service, --service-url step 3 addition per the design; no Python counterpart

Other declared divergences: help formatting is clap's (flags/defaults identical); default_db_path's script-relative fallback resolves relative to the binary (only reachable when the canonical path is absent); ATLAS_SEARCH_QVEC_FILE is a test-only hook substituting a fixed query vector — the same monkeypatch the Python fixture generator applies to embed_query.

Verification (laptop, rustup 1.99.0, sccache wrapper disabled — the step-1 E0514 trap)

  • Fixture parity: all 12 recorded search cases + both get-chunk cases from fixtures/core-expectations.json run through the compiled binary against the fixture DB — envelopes equal, and stdout byte-equal to CPython's own json.dumps rendering of the recorded envelopes (14/14). --text goldens were generated by the Python's render_text itself and match byte-for-byte. Not-found exit code + stderr pinned by test.
  • Live smoke (read-only production Turso, release binary, ATLAS_ORT_LIB = the v3 venv's libonnxruntime.so.1.31.1 + gcc-lib/zlib on the loader path): 4 questions, top-10s (chunk ids, scores, rank_sources) identical to the deployed Python CLI on all 4. Cold one-shot latency: Rust 13.0–16.4s vs Python 17.2–25.6s in the same battery (both dominated by model load; Python baseline ~35s colder).
  • Via-service fallback: --via-service against the (not yet existing) service → stderr atlas-search: warning: --via-service: service at http://127.0.0.1:18760 unreachable (io: Connection refused (os error 111)); falling back to standalone, envelope note recorded, results identical to the standalone run.
  • Degradation: FTS + get-chunk succeed with no model env; vector mode without ORT exits 2 with the loud message; local backend exercised against the v2 publish copy.
  • cargo test --workspace (29 tests incl. 7 new), cargo clippy --workspace --all-targets, cargo fmt --check — all clean.

Notes for later steps

  • Step 4 (Atlas step 4: serve mode — job protocol, worker pool, in-memory temp store #5354): replace service::attempt's probe with the atlas-proto submit/page client; the fallback contract around it (stderr warning + envelope note + standalone) does not change.
  • Step 5 (Atlas step 5: nix packaging (pkgs/atlas-search) + amd deploy #5355): the binary resolves the model from ATLAS_MODEL_DIR → ~/.local/state/atlas-search/model/<rev> → HF cache snapshot, and ORT from ATLAS_ORT_LIB → ~/.local/state/atlas-search/ort/libonnxruntime.so; the packaged build must instead resolve ONNX Runtime from its nix closure, which must carry libstdc++.so.6 + libz.so.1 (the dlopen loader-path lesson from opencode#1554 — verified live here with the venv's runtime plus the store gcc-lib/zlib pair).

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

The atlas-search binary crate: search + get-chunk subcommands over
atlas-core/atlas-embed, mirroring the Python CLI flag-for-flag
(flags, defaults, output envelope, --text rendering, exit codes).
Standalone by default (direct Turso via the read-only token flow,
or --backend local against a publish file); the query embedder
loads lazily so FTS-only search and get-chunk work with no model
present, and vector/hybrid fail loudly without one.

--via-service implements the prefer-with-loud-fallback contract
now (stderr warning + envelope note, standalone results); the
job-protocol client lands behind the same seam in step 4 (#5354).

Output is byte-identical to the Python: a CPython-compatible JSON
writer (indent=1, repr float formatting) and a line-for-line
render_text port, proven by fixture parity — all 12 recorded
search cases + both get-chunk cases byte-equal through the binary,
plus --text goldens generated by the Python renderer itself.
@dezren39
dezren39 requested a review from a team as a code owner November 4, 2026 09:52
@dezren39
dezren39 merged commit 9128ee8 into main Nov 4, 2026
1 check failed
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-04T09:53:39.588737Z 5cb3b6e PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Explain Change →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant