From e8c6591eed955c8225887812fe31db16377030c9 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 2 Jul 2026 14:34:28 +0200 Subject: [PATCH 01/79] Add per-sender trade_id to the trades sender Emit trade_id = -<1-based sequence> as a VARCHAR on each row, so completeness/gaps can be checked independent of timestamps and, with DEDUP UPSERT KEYS(timestamp, trade_id) on a single-worker client-timestamped run, store-and-forward replay after a failover is idempotent (drops the at-least-once boundary duplicates). High-cardinality, so a string column, not a symbol. --- src/main/java/com/example/sender/CsvParallelSender.java | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index cba9d49..62ebb3d 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -221,12 +221,15 @@ private static void runWorker( for (long i = 0; i < totalEvents; i++) { TradeRow r = rows.get((int) (i % n)); - // Build row + // Build row. trade_id = -<1-based sequence>, monotonic per sender, so you + // can check completeness/gaps independent of timestamps and (with dedup) get + // idempotent replay. It is a high-cardinality VARCHAR, deliberately NOT a symbol. sender.table("trades") .symbol("symbol", r.symbol) .symbol("side", r.side) .doubleColumn("price", r.price) - .doubleColumn("amount", r.amount); + .doubleColumn("amount", r.amount) + .stringColumn("trade_id", senderId + "-" + (i + 1)); if (timestampFromFile) { Instant ts = Instant.parse(r.timestamp); From 96794ccfdd709831bbf65424b6c715c713f89a2b Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 2 Jul 2026 16:38:41 +0200 Subject: [PATCH 02/79] Document trade_id and dedup in the README Add a Row id and dedup section covering the trade_id format, the completeness check, idempotent store-and-forward replay via DEDUP UPSERT KEYS(timestamp, trade_id), and the client-side-timestamp requirement. --- README.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/README.md b/README.md index 7a4ffbc..d2f2ad3 100644 --- a/README.md +++ b/README.md @@ -186,6 +186,36 @@ timestamps, so O3 is still avoided, and this sidesteps the per-batch stamping ab (giving near-distinct per-row timestamps). ILP, or QWP with more than one worker, keep using server-side `atNow()`, which stays O3-safe across concurrent senders. +## Row id and dedup (`trade_id`) + +Every row the trades sender writes carries a `trade_id` column of the form +`-` (for example `0-1`, `0-2`, ...), where the sequence is 1-based and +monotonic per sender. It is a high-cardinality **VARCHAR**, deliberately not a symbol (each +row is unique, so a symbol column would blow up the symbol table). + +Two uses: + +- **Completeness check, timestamp-independent.** Each sender emits a contiguous `1..N`, so you + can confirm nothing was lost and spot gaps regardless of how rows were timestamped. +- **Idempotent replay via dedup.** Store-and-forward is at-least-once: on a failover the client + replays the un-acked in-flight transaction to the new primary, re-writing the rows the old + primary had already durably persisted (send 100,000 rows, kill the primary mid-transaction, + and you land ~100,101). Enabling dedup collapses those back to exactly once: + + ```sql + ALTER TABLE trades DEDUP ENABLE UPSERT KEYS(timestamp, trade_id); + ``` + + Dedup keys must include the designated timestamp, so the match is on `(timestamp, trade_id)`. + `trade_id` also keeps legitimately-distinct rows that share a microsecond apart (common at + millions of rows/s), which a `(symbol, side, timestamp)` key alone would wrongly merge. + +**Requirement:** dedup only catches replays when the row carries a **client-side timestamp**, so +the timestamp is baked into the spilled frame and reproduced identically on replay. That means a +single worker (the default `at(Instant.now())` path) or `--timestamp-from-file` / +`--seconds-offset`. Under multi-worker `atNow()` the new primary assigns a fresh timestamp on +replay and dedup cannot match. See [Timestamps and throughput](#timestamps-and-throughput). + ## Benchmarking throughput On completion the sender prints the ingestion time and rate, for example: From 0990d12c52fec67431bca52686fad72b72352c37 Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 3 Jul 2026 12:52:42 +0200 Subject: [PATCH 03/79] Improve failover diagnostics and make the watchdog role-aware Sender: - Detect the QWP upgrade "HTTP/1.1 400 Bad request" that a missing/invalid ILP token produces and append an explicit auth hint wherever it surfaces (endpoint-failed, worker error, probe-stopped). - Probe now reports the LIVE serving role via `switch status` each poll instead of the QWP handshake SERVER_INFO, which only refreshes on a reconnect and so goes stale after an in-place primary<->replica switch. Watchdog: - Monitor the primary's currentRole, not just health: react to a silent demotion to REPLICA as well as an unreachable node. - Bootstrap a fresh cluster: if no node is primary, promote the first healthy one in list order. - Fail over to the first HEALTHY node in preference order; adopt a node that is already PRIMARY instead of forcing another switch. - Add a configurable grace period (QDB_WD_GRACE_PERIOD, default 5s) so a manual operator switch can settle before the watchdog acts. - Never exit on its own; keep retrying when nothing can be promoted. - Remove the now-unused get_http_code and STARTUP_RETRIES. Update README and the sample env accordingly. --- README.md | 62 ++++++-- qdb-primary-watchdog.env.example | 7 +- qdb-primary-watchdog.sh | 135 ++++++++++++------ .../com/example/sender/CsvParallelSender.java | 95 ++++++++++-- 4 files changed, 225 insertions(+), 74 deletions(-) diff --git a/README.md b/README.md index d2f2ad3..25e8d45 100644 --- a/README.md +++ b/README.md @@ -105,9 +105,17 @@ runs `select timestamp from trades limit -1` over a **QWP query client** and is independent of the senders, so it does not affect ingestion. ``` -[probe] latest trades timestamp = 2026-07-01T14:31:40.591545Z (raw=1782916300591545) +[probe] latest trades timestamp = 2026-07-01T14:31:40.591545Z (raw=1782916300591545) served by current_role=PRIMARY target_role=PRIMARY (live) ``` +The `served by ... (live)` suffix is the **live** lifecycle role of the node currently +answering the probe, obtained by running `switch status` on it each poll. This matters on +failover: the QWP handshake `SERVER_INFO` (node/role) is only refreshed on a *reconnect*, so +after an in-place primary→replica switch (which does not drop the read connection) it goes +stale. `switch status` always reflects the truth, so the role you see flips the moment the +serving node changes role. On a server without the lifecycle API (OSS), the status query is +unavailable and the probe falls back to `served by node=` from the handshake. + The query client uses the **same host list and token/auth** as the senders and enables **failover** when more than one host is given (`ws|wss::addr=h1,h2,...;...;failover=on`), so if a host stops responding it moves to the next one automatically and narrates the @@ -147,6 +155,16 @@ The **ingestion client** (one per worker) reports connection-state changes via t (also `auth failed` and `reconnect budget exhausted` terminal states.) +A missing or invalid ILP token is easy to misread: the QWP server refuses the WebSocket +upgrade with a bare `HTTP/1.1 400 Bad request` (it never gets far enough to return a 401), +so the raw client error mentions neither auth nor the token. Wherever that signature appears +(the `endpoint ... failed` line and the fatal `Sender N got error` / `Worker failed` lines) +the sender appends an explicit hint: + +``` +Sender 0 got error: io.questdb.client...WebSocketUpgradeException: WebSocket upgrade failed: HTTP/1.1 400 Bad request -- likely a missing or invalid ILP auth token (the server refuses the WebSocket upgrade before it can return 401); check --token or --user/--password +``` + The **query client** (the probe) reports, prefixed `[query client]`: ``` @@ -272,22 +290,35 @@ QWP WebSocket transport. ## Primary failover watchdog `qdb-primary-watchdog.sh` is a standalone Bash watchdog for an Enterprise cluster. Give it the -ordered list of every node's lifecycle admin endpoint (`host:9003`); it finds the current -primary, health-checks it, and on failure promotes another node via the lifecycle REST API. +ordered list of every node's lifecycle admin endpoint (`host:9003`); it finds (or, on a +fresh cluster, elects) the current primary, monitors its role, and on failure promotes another +node via the lifecycle REST API. Behaviour: -- **Detect:** on startup it sweeps the list and monitors whichever node reports - `currentRole:PRIMARY`. -- **Step down first:** when the primary stops responding it first sends a best-effort - `{"role":"replica"}` to the old primary, so a false positive (it is actually still up) cannot - leave two primaries. -- **Promote in order:** it then promotes the first reachable node in list order (earliest first, - skipping the dead one), retrying with exponential backoff. With `a,b,c`, if `b` is primary and - dies it fails back to `a` when `a` is available, and only reaches `c` if `a` cannot be promoted. - It then monitors the newly promoted node, repeating indefinitely. -- **Give up:** exits only if no node can be promoted (whole cluster down), or if no primary is - found at startup. Under systemd it is then restarted and re-detects the primary. +- **Detect / bootstrap:** on startup it sweeps the list and monitors whichever node reports + `currentRole:PRIMARY`. If **no** node is primary yet (a freshly-started cluster) it promotes + the first healthy node in list order, so the cluster always ends up with a primary. If every + node is unhealthy it keeps retrying rather than exiting. Split-brain is not a concern here: + QuestDB stops the offending node if two ever claim primary. +- **Monitor the role, not just health:** each interval it reads the primary's `currentRole`. + A node that stops responding **or** is silently demoted to `REPLICA` out-of-band both count + as a failure, so it reacts to a demotion even while the node is still reachable. +- **Grace period:** after detecting loss it waits `QDB_WD_GRACE_PERIOD` seconds (default `5`, + `0` disables) before acting, then re-checks. This lets an operator do a manual switch (demote + the primary, promote a replica) without the watchdog racing them, as long as it completes + within the window. +- **Adopt if already switched:** it then re-checks the cluster; if another node is already + `PRIMARY` (an operator switch, or the old primary recovered) it simply adopts it, no new switch. +- **Step down first:** otherwise it sends a best-effort `{"role":"replica"}` to the old primary, + so a false positive (it is actually still up) cannot leave two primaries. +- **Promote the first healthy node in order:** it then promotes the first **healthy** node in + list order (preferred/earliest first, skipping the dead one), retrying with exponential + backoff. Unreachable candidates are skipped. With `a,b,c`, if `b` is primary and dies it fails + back to `a` when `a` is healthy, and only reaches `c` if `a` is unhealthy or cannot be promoted. + It then monitors the new primary. +- **Never exits on its own:** if no node can be promoted (whole cluster down) it keeps retrying + every interval until one can take over. It stops only on `SIGINT`/`SIGTERM`. All endpoints are `https` (Enterprise only). @@ -326,7 +357,8 @@ sudo systemctl enable --now qdb-primary-watchdog journalctl -u qdb-primary-watchdog -f ``` -`Restart=always` brings it back on crash or after an all-nodes-down exit. +`Restart=always` brings it back on crash (the watchdog no longer exits on its own, even when the +whole cluster is down; it keeps retrying). **Run exactly one watchdog per cluster.** Two instances would try to promote different nodes and fight (split-brain). Put it on a separate monitoring host or one designated node. diff --git a/qdb-primary-watchdog.env.example b/qdb-primary-watchdog.env.example index fdca2cf..2aee41f 100644 --- a/qdb-primary-watchdog.env.example +++ b/qdb-primary-watchdog.env.example @@ -19,9 +19,10 @@ QDB_WD_SERVERS=172.31.42.41:9003,172.31.41.35:9003,10.0.0.8:9003 #QDB_WD_TARGET_ROLE=primary # role a node is switched to on failover #QDB_WD_SWITCH_TIMEOUT_MS=5000 # switch payload timeout_ms, and the promote backoff base #QDB_WD_STEPDOWN_CONNECT_TIMEOUT=3 # connect timeout (s) for the old-primary step-down; no overall timeout -#QDB_WD_FAIL_THRESHOLD=3 # consecutive no-responses before failover -#QDB_WD_CHECK_INTERVAL=1 # seconds between health checks -#QDB_WD_STARTUP_RETRIES=3 # sweeps to find the initial primary before giving up +#QDB_WD_FAIL_THRESHOLD=3 # consecutive non-primary reads before failover +#QDB_WD_CHECK_INTERVAL=1 # seconds between role checks +#QDB_WD_GRACE_PERIOD=5 # wait (s) after detecting loss before acting, so a manual + # operator switch can settle; re-checked after (0 disables) #QDB_WD_SWITCH_RETRIES=3 # promote attempts (exponential backoff between them) #QDB_WD_SWITCH_POLL_INTERVAL=1 # seconds between lifecycle polls after a switch #QDB_WD_SWITCH_POLL_MAX=30 # max polls to wait for a switch to settle diff --git a/qdb-primary-watchdog.sh b/qdb-primary-watchdog.sh index f8dde65..daab1c8 100755 --- a/qdb-primary-watchdog.sh +++ b/qdb-primary-watchdog.sh @@ -4,11 +4,17 @@ # Watches an Enterprise cluster for primary failure and promotes the next node. # # Give it the ordered list of lifecycle admin endpoints (host:9003) of ALL nodes. On start it -# finds which one currently reports role PRIMARY and health-monitors it. When that primary -# stops responding, it promotes the first reachable node in list order (earliest first, -# skipping the failed one) via POST /lifecycle/switch, waits for the switch to settle, then -# monitors the newly promoted node. If no node can be promoted (the whole cluster is down), -# it exits. +# finds which one currently reports role PRIMARY and monitors it; if none is primary yet it +# promotes the first healthy node in list order, so a freshly-started cluster gets a primary. +# (Split-brain is not a concern: QuestDB stops the offending node if two ever claim primary.) +# Monitoring is role-based, not +# just a health ping: each interval it reads the node's currentRole, so it reacts both to the +# primary going unreachable AND to it being silently demoted to a replica out-of-band. When the +# primary is lost it first re-checks whether the cluster already has a primary (e.g. an operator +# switch) and adopts it; otherwise it asks the old primary to step down and promotes the first +# HEALTHY node in list order (preferred first, skipping the failed one) via POST /lifecycle/switch, +# waits for the switch to settle, then monitors the new primary. It never exits on its own: if no +# primary exists yet, or none can be promoted, it keeps retrying every check interval. # # Required, no defaults: the node list and QDB_REST_TOKEN. The node list comes from the first # CLI arg or $QDB_WD_SERVERS; the token from $QDB_REST_TOKEN. Everything else has a default @@ -70,9 +76,11 @@ SERVERS_CSV="${1:-${QDB_WD_SERVERS:-}}" TARGET_ROLE="${QDB_WD_TARGET_ROLE:-primary}" # role to switch a node to on failover SWITCH_TIMEOUT_MS="${QDB_WD_SWITCH_TIMEOUT_MS:-5000}" # timeout_ms in the switch payload STEPDOWN_CONNECT_TIMEOUT="${QDB_WD_STEPDOWN_CONNECT_TIMEOUT:-3}" # connect timeout (s) for the old-primary step-down -FAIL_THRESHOLD="${QDB_WD_FAIL_THRESHOLD:-3}" # consecutive health failures before failover -CHECK_INTERVAL="${QDB_WD_CHECK_INTERVAL:-1}" # seconds between health checks -STARTUP_RETRIES="${QDB_WD_STARTUP_RETRIES:-3}" # sweeps to find the initial primary before giving up +FAIL_THRESHOLD="${QDB_WD_FAIL_THRESHOLD:-3}" # consecutive non-primary reads before failover +CHECK_INTERVAL="${QDB_WD_CHECK_INTERVAL:-1}" # seconds between role checks +GRACE_PERIOD="${QDB_WD_GRACE_PERIOD:-5}" # seconds to wait after detecting loss before + # acting, so a manual operator switch can settle + # (re-checked afterwards; 0 disables the wait) SWITCH_RETRIES="${QDB_WD_SWITCH_RETRIES:-3}" # retries if a switch POST is not accepted SWITCH_POLL_INTERVAL="${QDB_WD_SWITCH_POLL_INTERVAL:-1}" # seconds between lifecycle polls after a switch SWITCH_POLL_MAX="${QDB_WD_SWITCH_POLL_MAX:-30}" # max polls to wait for a switch to settle @@ -114,17 +122,6 @@ get_role() { fi } -# Health ping. Prints the HTTP code, or 000 if there was no response at all. -get_http_code() { - local code rc - code="$(curl -ksS -o /dev/null -w '%{http_code}' \ - -H "Authorization: Bearer $QDB_REST_TOKEN" \ - --connect-timeout 2 --max-time 3 "https://$1/lifecycle" 2>/dev/null)" - rc=$? - (( rc != 0 )) && code="000" - printf '%s' "$code" -} - # Poll a node's /lifecycle until its switch settles and it reports the target role. wait_switch_complete() { local hostport="$1" want="${TARGET_ROLE^^}" attempt resp nospace @@ -207,14 +204,26 @@ find_primary() { return 1 } -# Promote the first reachable node in list order (earliest wins), skipping the current -# (failed) primary. So with a,b,c where b is primary and dies, it tries a first, then c; -# it only reaches c if a cannot be promoted. Sets FOUND_IDX, returns 0/1. +# Promote the first HEALTHY node in list order (earliest/preferred wins), skipping the current +# (failed) primary. So with a,b,c where b is primary and dies, it tries a first, then c; it only +# reaches c if a is unhealthy or cannot be promoted. Each candidate's role is read first: an +# unreachable node is skipped, a node that is already PRIMARY is adopted as-is (an out-of-band +# switch beat us to it), and a healthy replica is promoted. Sets FOUND_IDX, returns 0/1. failover_from() { - local cur="$1" n="${#SERVERS[@]}" i + local cur="$1" n="${#SERVERS[@]}" i role for (( i=0; i= STARTUP_RETRIES )); then - log "ERROR: no PRIMARY found among the nodes after $STARTUP_RETRIES sweeps. Exiting." - exit 1 - fi - (( attempt++ )) - log "No primary yet, retrying sweep ($attempt/$STARTUP_RETRIES)..." + local cur + # Startup: ensure the cluster has a primary. Adopt an existing one, or promote the first healthy + # node if none is primary yet. If every node is unhealthy, keep retrying -- never exit. + until ensure_primary; do + log "No primary and none could be promoted yet; retrying in ${CHECK_INTERVAL}s (will not exit)..." sleep "$CHECK_INTERVAL" done - cur=$FOUND_IDX + cur=$ENSURED_IDX log "Primary is ${SERVERS[$cur]} (index $cur). Monitoring every ${CHECK_INTERVAL}s (threshold $FAIL_THRESHOLD)." - local fail_count=0 code + local fail_count=0 role while true; do - code="$(get_http_code "${SERVERS[$cur]}")" - # Any HTTP reply means the primary is alive; only a no-response (000) counts as down. - if [[ "$code" == "000" ]]; then - (( fail_count++ )) - log "Primary ${SERVERS[$cur]} unreachable, no HTTP response (fails: $fail_count/$FAIL_THRESHOLD)" - else - (( fail_count > 0 )) && log "Primary ${SERVERS[$cur]} responding again (HTTP $code), resetting" + # Role-based check: the primary counts as healthy only while it still reports PRIMARY. + # An unreachable node (empty role) or a silent demotion to REPLICA both count as failures. + role="$(get_role "${SERVERS[$cur]}")" + if [[ "$role" == "PRIMARY" ]]; then + (( fail_count > 0 )) && log "Primary ${SERVERS[$cur]} healthy again (role=PRIMARY), resetting" fail_count=0 + else + (( fail_count++ )) + log "Primary ${SERVERS[$cur]} not serving as primary (role=${role:-unreachable}) (fails: $fail_count/$FAIL_THRESHOLD)" fi if (( fail_count >= FAIL_THRESHOLD )); then - log "Primary ${SERVERS[$cur]} is down, initiating failover" - demote_to_replica "${SERVERS[$cur]}" # first, try to make the old primary stand down - if failover_from "$cur"; then + log "Primary ${SERVERS[$cur]} lost; initiating failover" + # Grace period: give an operator doing a manual switch (demote the old primary, promote a + # replica) a few seconds to finish before we act, so the watchdog does not race them. After + # the wait we re-check the cluster; if a primary is now present we simply adopt it below. + if (( GRACE_PERIOD > 0 )); then + log " grace period: waiting ${GRACE_PERIOD}s for a manual switch to settle before acting" + sleep "$GRACE_PERIOD" + fi + # The cluster may already have a new primary (operator switch, or cur recovered elsewhere in + # the list): adopt it instead of forcing another switch. + if find_primary; then cur=$FOUND_IDX fail_count=0 - log "New primary is ${SERVERS[$cur]} (index $cur). Monitoring." + log "Cluster already has a PRIMARY: ${SERVERS[$cur]} (index $cur). Monitoring it." else - log "ERROR: no node could be promoted (whole cluster appears down). Exiting." - exit 1 + demote_to_replica "${SERVERS[$cur]}" # best-effort: make the old primary stand down first + if failover_from "$cur"; then + cur=$FOUND_IDX + fail_count=0 + log "New primary is ${SERVERS[$cur]} (index $cur). Monitoring." + else + # No healthy node to promote. Do NOT exit: leave fail_count armed so the next tick + # retries the whole failover, and keep going until some node can take over. + log "No healthy node could be promoted; will keep retrying every ${CHECK_INTERVAL}s (not exiting)" + fi fi fi diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index 62ebb3d..c3881dd 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -52,6 +52,11 @@ public class CsvParallelSender { // Probe (QWP only): poll the latest ingested timestamp on an interval, 0 disables. private static final long DEFAULT_PROBE_INTERVAL_MS = 1000L; private static final String PROBE_QUERY = "select timestamp from trades limit -1"; + // Enterprise lifecycle status of whichever node the query client is currently connected to. + // Returns the LIVE role (columns like current_role / target_role), unlike the QWP handshake + // SERVER_INFO which is only refreshed on a reconnect and so goes stale after an in-place + // primary<->replica switch. The probe runs this each poll to report the true serving role. + private static final String STATUS_QUERY = "switch status"; // Zone for the query client (egress). Biases failover toward same-zone instances on // Enterprise; a no-op on OSS (which advertises no zone). Empty omits the key. private static final String DEFAULT_ZONE = "eu-west-1"; @@ -181,7 +186,7 @@ public static void main(String[] args) throws Exception { try { f.get(); } catch (ExecutionException ee) { - System.err.println("Worker failed: " + ee.getCause()); + System.err.println("Worker failed: " + ee.getCause() + authHint(ee.getCause())); System.exit(1); } } @@ -266,7 +271,7 @@ private static void runWorker( sender.flush(); System.out.printf("Sender %d finished sending %d events%n", senderId, sent); } catch (Exception e) { - System.err.printf("Sender %d got error: %s%n", senderId, e.toString()); + System.err.printf("Sender %d got error: %s%s%n", senderId, e.toString(), authHint(e)); throw new RuntimeException(e); } } @@ -433,7 +438,7 @@ private static Sender buildSender(SenderCfg cfg, int workerId) { noisy = true; break; case ENDPOINT_ATTEMPT_FAILED: - msg = "endpoint " + host + " failed (" + cause + "), trying next"; + msg = "endpoint " + host + " failed (" + cause + "), trying next" + authHint(cause); noisy = true; break; case ALL_ENDPOINTS_UNREACHABLE: @@ -549,8 +554,48 @@ public void onFailoverReset(QwpServerInfo info) { orNone(info.getNodeId()), QwpServerInfo.roleName(info.getRole()), orNone(info.getZoneId())); } }; + // Captures the live lifecycle role from `switch status`: joins every column whose + // name mentions "role" (current_role, target_role, ...) into e.g. "current_role=PRIMARY". + final String[] liveRole = {null}; + final QwpColumnBatchHandler statusHandler = new QwpColumnBatchHandler() { + @Override + public void onBatch(QwpColumnBatch batch) { + final int cols = batch.getColumnCount(); + batch.forEachRow(row -> { + final StringBuilder sb = new StringBuilder(); + for (int c = 0; c < cols; c++) { + final String name = batch.getColumnName(c); + if (name != null && name.toLowerCase(java.util.Locale.ROOT).contains("role")) { + if (sb.length() > 0) { + sb.append(' '); + } + sb.append(name).append('=') + .append(row.isNull(c) ? "(none)" : row.getString(c)); + } + } + if (sb.length() > 0) { + liveRole[0] = sb.toString(); + } + }); + } + + @Override + public void onEnd(long totalRows) { + } + + @Override + public void onError(byte status, String message) { + // e.g. OSS server without lifecycle, or insufficient permissions: leave + // liveRole null so the probe falls back to the handshake node id. + } + + @Override + public void onFailoverReset(QwpServerInfo info) { + } + }; while (!Thread.currentThread().isInterrupted()) { latest[0] = Long.MIN_VALUE; + liveRole[0] = null; try { client.execute(PROBE_QUERY, handler); if (wasDown[0]) { @@ -560,11 +605,20 @@ public void onFailoverReset(QwpServerInfo info) { if (latest[0] != Long.MIN_VALUE) { // trades designated timestamp is microseconds by default. Instant ts = Instant.EPOCH.plus(latest[0], ChronoUnit.MICROS); - final QwpServerInfo si = client.getServerInfo(); - final String served = si != null - ? " served by role=" + QwpServerInfo.roleName(si.getRole()) - + " node=" + orNone(si.getNodeId()) + " zone=" + orNone(si.getZoneId()) - : ""; + // Ask the serving node for its live role. A status-query failure must not + // look like a connection loss, so swallow it here (liveRole stays null). + try { + client.execute(STATUS_QUERY, statusHandler); + } catch (Exception ignore) { + // fall through to the handshake fallback below + } + final String served; + if (liveRole[0] != null) { + served = " served by " + liveRole[0] + " (live)"; + } else { + final QwpServerInfo si = client.getServerInfo(); + served = si != null ? " served by node=" + orNone(si.getNodeId()) : ""; + } System.out.printf("[probe] latest trades timestamp = %s (raw=%d)%s%n", ts, latest[0], served); } @@ -580,7 +634,7 @@ public void onFailoverReset(QwpServerInfo info) { } catch (InterruptedException ie) { Thread.currentThread().interrupt(); } catch (Exception e) { - System.err.println("[probe] stopped: " + e.getMessage()); + System.err.println("[probe] stopped: " + e.getMessage() + authHint(e)); } }, "qwp-probe"); t.setDaemon(true); @@ -592,6 +646,29 @@ private static String orNone(String s) { return (s == null || s.isEmpty()) ? "(none)" : s; } + // A QWP server rejects a missing/invalid ILP auth token by refusing the WebSocket + // upgrade with a bare "HTTP/1.1 400 Bad request" (it never gets far enough to return + // a 401), which is impossible to diagnose from the raw message. Detect that signature + // and append an explicit hint. Returns "" when the text is not an upgrade-400. + private static String authHint(String m) { + if (m != null && m.contains("WebSocket upgrade failed") && m.contains("400")) { + return " -- likely a missing or invalid ILP auth token (the server refuses the" + + " WebSocket upgrade before it can return 401); check --token or --user/--password"; + } + return ""; + } + + // Same, but walks a throwable's cause chain (the upgrade failure is usually wrapped). + private static String authHint(Throwable t) { + for (Throwable c = t; c != null; c = c.getCause()) { + final String h = authHint(c.getMessage()); + if (!h.isEmpty()) { + return h; + } + } + return ""; + } + private static String buildConf(String addrsCsv, String token, String username, String password, int retryTimeout) { String[] addrs = Arrays.stream(addrsCsv.split(",")) .map(String::trim) From 15f298277d5136cd9a0de3c3f23bc0c4836fc951 Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 3 Jul 2026 14:12:17 +0200 Subject: [PATCH 04/79] Target client 1.3.6-SNAPSHOT; typed upgrade diagnosis; connect timeout Build/target: - pom default questdb.client.version -> 1.3.6-SNAPSHOT (required; QWP and the lifecycle switch features are not stable earlier, and it is not on Central). - Drop the RECONNECT_BUDGET_EXHAUSTED switch case in both senders; that kind was removed from SenderConnectionEvent.Kind in 1.3.6, so it no longer compiles. Diagnostics: - Replace the brittle "400 Bad request" string-match with the client's typed signals: QwpAuthFailedException (401/403), WebSocketUpgradeException.isRoleMismatch() (421 = endpoint is a replica, no primary to accept writes), and status-400 (missing/malformed token). The listener passes the real throwable, not its message. Timeouts: - --retry-timeout now also drives QWP (reconnectMaxDurationMillis), not just ILP's retryTimeoutMillis, so one flag means the same "keep retrying" budget on both. - New --connect-timeout-ms (default 3000, 0 = OS default): connectTimeoutMillis on the senders and connect_timeout= on the probe, so a black-holed host fails over in seconds instead of riding the OS connect timeout. ILP left unbounded/unchanged. - Watchdog: make the role/health poll timeouts configurable (QDB_WD_CHECK_CONNECT_TIMEOUT / QDB_WD_CHECK_MAX_TIME; defaults 2/3, no behaviour change). Update README and the watchdog sample env accordingly. --- README.md | 51 +++++++--- pom.xml | 16 ++-- qdb-primary-watchdog.env.example | 2 + qdb-primary-watchdog.sh | 4 +- .../com/example/sender/CsvParallelSender.java | 94 +++++++++++++------ .../sender/TelemetryParallelSender.java | 7 +- 6 files changed, 122 insertions(+), 52 deletions(-) diff --git a/README.md b/README.md index 25e8d45..58ae26a 100644 --- a/README.md +++ b/README.md @@ -5,16 +5,21 @@ `mvn -DskipTests clean package` The QuestDB client version is a Maven property (`questdb.client.version`, default -`1.3.2`, the Maven Central release). The QWP (WebSocket) wire protocol differs -between release and master builds, so the client **must match the target server -build**. To ingest into a server built from master, build with: +`1.3.6-SNAPSHOT`). This build **requires** it: the QWP transport and the lifecycle +`switch` features used here (`switch status`, the current connection-event set) are +not stable in earlier clients. `1.3.6-SNAPSHOT` is **not on Maven Central** — install +it locally first, from the `feat/connect-timeout` branch of `java-questdb-client`: ``` -mvn -DskipTests clean package -Dquestdb.client.version=1.3.5-SNAPSHOT +# in a checkout of java-questdb-client on feat/connect-timeout: +mvn -DskipTests clean install ``` -A mismatch is not reported cleanly; it surfaces as `unknown msg_kind 0x18` or -`invalid column type code: 0x0`, or as rows silently not landing. +Then build this project normally (`mvn -DskipTests clean package`), or pin another +build with `-Dquestdb.client.version=...`. The QWP (WebSocket) wire protocol **must +match the target server build** — a mismatch is not reported cleanly; it surfaces as +`unknown msg_kind 0x18` or `invalid column type code: 0x0`, or as rows silently not +landing. ## Usage Example @@ -29,12 +34,18 @@ java -jar target/ilp_sender-1.0-SNAPSHOT.jar \ --csv ./trades20250728.csv.gz \ --timestamp-from-file false \ --retry-timeout 360000 \ +--connect-timeout-ms 3000 \ --sender-id ha_sender \ --store-forward-dir /tmp/qdb-sf \ --batch-size 10000 \ --batches-per-transaction 10 ``` +`--retry-timeout` (default `360000` ms) is the overall "keep retrying" budget and applies to +**both** transports: it drives `retryTimeoutMillis` on ILP and `reconnectMaxDurationMillis` on +QWP. That is a different timeout from `--connect-timeout-ms` (below), which bounds a single +connect attempt. + ## Transport (`--protocol`) - `--protocol qwp` (default): QWP over WebSocket. Adds store-and-forward (un-acked @@ -59,6 +70,11 @@ QWP-only flags: - `--batches-per-transaction` (default `10`): an explicit `flush()` commits every `batch-size × batches-per-transaction` rows atomically per table. See [Commit cadence](#commit-cadence-freshness-vs-throughput) below. +- `--connect-timeout-ms` (default `3000`, `0` uses the OS default): upper bound on a **single** + TCP connect attempt, for both the senders (`connectTimeoutMillis`) and the probe + (`connect_timeout=`). Without it a black-holed host (route accepts, never answers) rides the + OS connect timeout (~minutes) before failing over to the next `--addrs` host; with it, failover + happens in seconds. The ILP path is left unbounded so `--protocol ilp` stays identical. - `--probe-interval-ms` (default `1000`, `0` disables): interval for the probe thread that polls the latest ingested timestamp. See [Probe](#probe-qwp-only) below. - `--enterprise` (default `false`): request durable acks (`requestDurableAck`), holding @@ -153,16 +169,25 @@ The **ingestion client** (one per worker) reports connection-state changes via t [ingestion client ha_sender-0] all endpoints unreachable, backing off ``` -(also `auth failed` and `reconnect budget exhausted` terminal states.) +(also an `auth failed` terminal state; any other event kind the client emits is narrated +generically by the `default` branch.) + +A failed QWP WebSocket upgrade is otherwise hard to read from the raw client error, so +wherever it surfaces (the `endpoint ... failed` line and the fatal `Sender N got error` / +`Worker failed` lines) the sender appends a hint. It uses the client's **typed** signals, not +string-matching, so it tells the causes apart: -A missing or invalid ILP token is easy to misread: the QWP server refuses the WebSocket -upgrade with a bare `HTTP/1.1 400 Bad request` (it never gets far enough to return a 401), -so the raw client error mentions neither auth nor the token. Wherever that signature appears -(the `endpoint ... failed` line and the fatal `Sender N got error` / `Worker failed` lines) -the sender appends an explicit hint: +- **`QwpAuthFailedException` (HTTP 401/403)** — bad/expired credentials or missing permission: + `-- auth rejected (HTTP 401): bad/expired credentials or missing permission; check --token or --user/--password` +- **`WebSocketUpgradeException.isRoleMismatch()` (HTTP 421)** — the endpoint is a replica, i.e. + no primary is available among `--addrs` to accept writes: + `-- endpoint is not writable (role=REPLICA): no primary available among --addrs to accept writes` +- **`WebSocketUpgradeException` status 400** — a missing or malformed token (e.g. an unset token + env var); the server refuses the upgrade before it can return a 401: + `-- HTTP 400 on the upgrade: likely a missing or malformed ILP auth token (e.g. an unset token env var); check --token or --user/--password` ``` -Sender 0 got error: io.questdb.client...WebSocketUpgradeException: WebSocket upgrade failed: HTTP/1.1 400 Bad request -- likely a missing or invalid ILP auth token (the server refuses the WebSocket upgrade before it can return 401); check --token or --user/--password +Sender 0 got error: io.questdb.client...WebSocketUpgradeException: WebSocket upgrade failed: HTTP/1.1 400 Bad request -- HTTP 400 on the upgrade: likely a missing or malformed ILP auth token (e.g. an unset token env var); check --token or --user/--password ``` The **query client** (the probe) reports, prefixed `[query client]`: diff --git a/pom.xml b/pom.xml index 540eb70..5024d7a 100644 --- a/pom.xml +++ b/pom.xml @@ -10,13 +10,15 @@ 11 11 - - 1.3.2 + + 1.3.6-SNAPSHOT diff --git a/qdb-primary-watchdog.env.example b/qdb-primary-watchdog.env.example index 2aee41f..0a43f1b 100644 --- a/qdb-primary-watchdog.env.example +++ b/qdb-primary-watchdog.env.example @@ -19,6 +19,8 @@ QDB_WD_SERVERS=172.31.42.41:9003,172.31.41.35:9003,10.0.0.8:9003 #QDB_WD_TARGET_ROLE=primary # role a node is switched to on failover #QDB_WD_SWITCH_TIMEOUT_MS=5000 # switch payload timeout_ms, and the promote backoff base #QDB_WD_STEPDOWN_CONNECT_TIMEOUT=3 # connect timeout (s) for the old-primary step-down; no overall timeout +#QDB_WD_CHECK_CONNECT_TIMEOUT=2 # connect timeout (s) for each role/health poll +#QDB_WD_CHECK_MAX_TIME=3 # overall max time (s) for each role/health poll #QDB_WD_FAIL_THRESHOLD=3 # consecutive non-primary reads before failover #QDB_WD_CHECK_INTERVAL=1 # seconds between role checks #QDB_WD_GRACE_PERIOD=5 # wait (s) after detecting loss before acting, so a manual diff --git a/qdb-primary-watchdog.sh b/qdb-primary-watchdog.sh index daab1c8..b2674ce 100755 --- a/qdb-primary-watchdog.sh +++ b/qdb-primary-watchdog.sh @@ -76,6 +76,8 @@ SERVERS_CSV="${1:-${QDB_WD_SERVERS:-}}" TARGET_ROLE="${QDB_WD_TARGET_ROLE:-primary}" # role to switch a node to on failover SWITCH_TIMEOUT_MS="${QDB_WD_SWITCH_TIMEOUT_MS:-5000}" # timeout_ms in the switch payload STEPDOWN_CONNECT_TIMEOUT="${QDB_WD_STEPDOWN_CONNECT_TIMEOUT:-3}" # connect timeout (s) for the old-primary step-down +CHECK_CONNECT_TIMEOUT="${QDB_WD_CHECK_CONNECT_TIMEOUT:-2}" # connect timeout (s) for each role/health poll +CHECK_MAX_TIME="${QDB_WD_CHECK_MAX_TIME:-3}" # overall max time (s) for each role/health poll FAIL_THRESHOLD="${QDB_WD_FAIL_THRESHOLD:-3}" # consecutive non-primary reads before failover CHECK_INTERVAL="${QDB_WD_CHECK_INTERVAL:-1}" # seconds between role checks GRACE_PERIOD="${QDB_WD_GRACE_PERIOD:-5}" # seconds to wait after detecting loss before @@ -106,7 +108,7 @@ fi # GET a node's /lifecycle. Prints the raw JSON, or nothing if unreachable. lifecycle_json() { - curl -ksS --connect-timeout 2 --max-time 3 \ + curl -ksS --connect-timeout "$CHECK_CONNECT_TIMEOUT" --max-time "$CHECK_MAX_TIME" \ -H "Authorization: Bearer $QDB_REST_TOKEN" \ "https://$1/lifecycle" 2>/dev/null } diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index c3881dd..8d2af37 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -3,6 +3,8 @@ import io.questdb.client.Sender; import io.questdb.client.Sender.LineSenderBuilder; import io.questdb.client.SenderConnectionListener; +import io.questdb.client.cutlass.http.client.WebSocketUpgradeException; +import io.questdb.client.cutlass.qwp.client.QwpAuthFailedException; import io.questdb.client.cutlass.qwp.client.QwpColumnBatch; import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; import io.questdb.client.cutlass.qwp.client.QwpQueryClient; @@ -44,6 +46,10 @@ public class CsvParallelSender { private static final String DEFAULT_PROTOCOL = "qwp"; private static final String DEFAULT_SENDER_ID = "ha_sender"; private static final String DEFAULT_STORE_FORWARD_DIR = "/tmp/qdb-sf"; + // Upper bound (ms) on a single TCP connect attempt, so a black-holed host fails over fast + // instead of riding the OS connect timeout. 0 disables (falls back to the OS default). + // QWP + probe only; the ILP path is left untouched so --protocol ilp stays identical. + private static final int DEFAULT_CONNECT_TIMEOUT_MS = 3000; // Batch = one auto-flush append (deferred, no commit under transactional mode). // Transaction = batches-per-transaction batches, committed atomically per table // by an explicit flush(). See buildSender()/runWorker(). @@ -93,6 +99,9 @@ public static void main(String[] args) throws Exception { // support it and the connection is rejected, so it is off by default. final boolean enterprise = Boolean.parseBoolean(a.getOrDefault("--enterprise", "false")); final String zone = a.getOrDefault("--zone", DEFAULT_ZONE); + // QWP + probe: bound a single TCP connect attempt (0 disables). ILP is left unchanged. + final int connectTimeoutMs = Integer.parseInt(a.getOrDefault("--connect-timeout-ms", + String.valueOf(DEFAULT_CONNECT_TIMEOUT_MS))); if (!protocol.equals("qwp") && !protocol.equals("ilp")) { System.err.println("--protocol must be 'qwp' or 'ilp', got: " + protocol); @@ -102,6 +111,10 @@ public static void main(String[] args) throws Exception { System.err.println("--probe-interval-ms must be >= 0 (0 disables the probe)"); System.exit(2); } + if (connectTimeoutMs < 0) { + System.err.println("--connect-timeout-ms must be >= 0 (0 uses the OS connect timeout)"); + System.exit(2); + } if (batchSize <= 0) { System.err.println("--batch-size must be > 0"); System.exit(2); @@ -125,13 +138,15 @@ public static void main(String[] args) throws Exception { } final SenderCfg cfg = new SenderCfg(protocol, addrsCsv, token, username, password, retryTimeout, - senderIdBase, storeForwardDir, batchSize, batchesPerTransaction, numSenders, enterprise, zone); + senderIdBase, storeForwardDir, batchSize, batchesPerTransaction, numSenders, enterprise, zone, + connectTimeoutMs); final String conf = buildConf(addrsCsv, token, username, password, retryTimeout); System.out.println("Ingestion started. Protocol: " + protocol + (protocol.equals("qwp") ? " (WebSocket, sender-id=" + senderIdBase + ", store-and-forward=" + storeForwardDir - + ", batch-size=" + batchSize + ", batches-per-transaction=" + batchesPerTransaction + ")" + + ", batch-size=" + batchSize + ", batches-per-transaction=" + batchesPerTransaction + + ", connect-timeout-ms=" + connectTimeoutMs + ", retry-timeout-ms=" + retryTimeout + ")" : "") + " | config: " + conf.replaceAll("(token=)([^;]+)", "$1***") .replaceAll("(password=)([^;]+)", "$1***")); @@ -186,7 +201,7 @@ public static void main(String[] args) throws Exception { try { f.get(); } catch (ExecutionException ee) { - System.err.println("Worker failed: " + ee.getCause() + authHint(ee.getCause())); + System.err.println("Worker failed: " + ee.getCause() + upgradeHint(ee.getCause())); System.exit(1); } } @@ -271,7 +286,7 @@ private static void runWorker( sender.flush(); System.out.printf("Sender %d finished sending %d events%n", senderId, sent); } catch (Exception e) { - System.err.printf("Sender %d got error: %s%s%n", senderId, e.toString(), authHint(e)); + System.err.printf("Sender %d got error: %s%s%n", senderId, e.toString(), upgradeHint(e)); throw new RuntimeException(e); } } @@ -430,15 +445,16 @@ private static Sender buildSender(SenderCfg cfg, int workerId) { case AUTH_FAILED: msg = "auth failed for " + host; break; - case RECONNECT_BUDGET_EXHAUSTED: - msg = "reconnect budget exhausted, giving up"; - break; + // NOTE: no RECONNECT_BUDGET_EXHAUSTED case on purpose. That kind was dropped from + // the client's SenderConnectionEvent.Kind in newer builds (e.g. 1.3.6-SNAPSHOT), so + // switching on it fails to compile there. If a client still emits it, the default + // branch below narrates it generically. case DISCONNECTED: msg = "connection lost to " + host + " (" + cause + "), will retry"; noisy = true; break; case ENDPOINT_ATTEMPT_FAILED: - msg = "endpoint " + host + " failed (" + cause + "), trying next" + authHint(cause); + msg = "endpoint " + host + " failed (" + cause + "), trying next" + upgradeHint(event.getCause()); noisy = true; break; case ALL_ENDPOINTS_UNREACHABLE: @@ -465,11 +481,18 @@ private static Sender buildSender(SenderCfg cfg, int workerId) { b.requestDurableAck(true); } + // Bound a single TCP connect so a black-holed host fails over fast (0 = OS default). + if (cfg.connectTimeoutMs > 0) { + b.connectTimeoutMillis(cfg.connectTimeoutMs); + } + return b.storeAndForwardDir(sfPath) .senderId(who) .transactional(true) .connectionListener(connListener) - .reconnectMaxDurationMillis(300_000) + // reconnectMaxDurationMillis is the QWP analog of the ILP retryTimeoutMillis: + // the overall "keep retrying" budget. Driven by --retry-timeout on both transports. + .reconnectMaxDurationMillis(cfg.retryTimeout) .reconnectInitialBackoffMillis(100) .reconnectMaxBackoffMillis(5_000) .autoFlushBytes(524_288) // 512 KiB, under the ~1MB WS frame cap @@ -503,6 +526,10 @@ private static String queryClientConfig(SenderCfg cfg) { if (tls) { sb.append("tls_verify=unsafe_off;"); } + // Bound the probe's TCP connect too, so it fails over as fast as the senders (0 = OS default). + if (cfg.connectTimeoutMs > 0) { + sb.append("connect_timeout=").append(cfg.connectTimeoutMs).append(';'); + } if (addrs.length > 1) { sb.append("failover=on;"); } @@ -634,7 +661,7 @@ public void onFailoverReset(QwpServerInfo info) { } catch (InterruptedException ie) { Thread.currentThread().interrupt(); } catch (Exception e) { - System.err.println("[probe] stopped: " + e.getMessage() + authHint(e)); + System.err.println("[probe] stopped: " + e.getMessage() + upgradeHint(e)); } }, "qwp-probe"); t.setDaemon(true); @@ -646,24 +673,31 @@ private static String orNone(String s) { return (s == null || s.isEmpty()) ? "(none)" : s; } - // A QWP server rejects a missing/invalid ILP auth token by refusing the WebSocket - // upgrade with a bare "HTTP/1.1 400 Bad request" (it never gets far enough to return - // a 401), which is impossible to diagnose from the raw message. Detect that signature - // and append an explicit hint. Returns "" when the text is not an upgrade-400. - private static String authHint(String m) { - if (m != null && m.contains("WebSocket upgrade failed") && m.contains("400")) { - return " -- likely a missing or invalid ILP auth token (the server refuses the" - + " WebSocket upgrade before it can return 401); check --token or --user/--password"; - } - return ""; - } - - // Same, but walks a throwable's cause chain (the upgrade failure is usually wrapped). - private static String authHint(Throwable t) { + // Turn a failed QWP WebSocket-upgrade throwable into a human-readable hint, using the + // client's TYPED signals rather than string-matching the message: + // - QwpAuthFailedException -> a definitive auth rejection (HTTP 401/403). + // - WebSocketUpgradeException.isRoleMismatch() (HTTP 421) -> the endpoint is not writable + // (a REPLICA / PRIMARY_CATCHUP), i.e. no primary is available among --addrs to accept writes. + // - WebSocketUpgradeException with status 400 -> a missing/malformed auth token (e.g. an + // unset token env var): the server refuses the upgrade before it can return a 401. + // Walks the cause chain (the upgrade failure is usually wrapped). Returns "" otherwise. + private static String upgradeHint(Throwable t) { for (Throwable c = t; c != null; c = c.getCause()) { - final String h = authHint(c.getMessage()); - if (!h.isEmpty()) { - return h; + if (c instanceof QwpAuthFailedException) { + final QwpAuthFailedException a = (QwpAuthFailedException) c; + return " -- auth rejected (HTTP " + a.getStatusCode() + "): bad/expired credentials" + + " or missing permission; check --token or --user/--password"; + } + if (c instanceof WebSocketUpgradeException) { + final WebSocketUpgradeException w = (WebSocketUpgradeException) c; + if (w.isRoleMismatch()) { + return " -- endpoint is not writable (role=" + w.getServerRole() + "): no primary" + + " available among --addrs to accept writes"; + } + if (w.getStatusCode() == 400) { + return " -- HTTP 400 on the upgrade: likely a missing or malformed ILP auth token" + + " (e.g. an unset token env var); check --token or --user/--password"; + } } } return ""; @@ -724,6 +758,7 @@ private static Map parseArgs(String[] args) { case "--batch-size": case "--batches-per-transaction": case "--probe-interval-ms": + case "--connect-timeout-ms": case "--enterprise": case "--zone": if (i + 1 >= args.length) { @@ -762,10 +797,12 @@ private static final class SenderCfg { final int numSenders; final boolean enterprise; final String zone; + final int connectTimeoutMs; SenderCfg(String protocol, String addrsCsv, String token, String username, String password, int retryTimeout, String senderIdBase, String storeForwardDir, - int batchSize, int batchesPerTransaction, int numSenders, boolean enterprise, String zone) { + int batchSize, int batchesPerTransaction, int numSenders, boolean enterprise, String zone, + int connectTimeoutMs) { this.protocol = protocol; this.addrsCsv = addrsCsv; this.token = token; @@ -779,6 +816,7 @@ private static final class SenderCfg { this.numSenders = numSenders; this.enterprise = enterprise; this.zone = zone; + this.connectTimeoutMs = connectTimeoutMs; } } } diff --git a/src/main/java/com/example/sender/TelemetryParallelSender.java b/src/main/java/com/example/sender/TelemetryParallelSender.java index 06bdc67..a2a3cd5 100644 --- a/src/main/java/com/example/sender/TelemetryParallelSender.java +++ b/src/main/java/com/example/sender/TelemetryParallelSender.java @@ -522,9 +522,10 @@ private static Sender buildSender(SenderCfg cfg, int workerId) { case AUTH_FAILED: msg = "auth failed for " + host; break; - case RECONNECT_BUDGET_EXHAUSTED: - msg = "reconnect budget exhausted, giving up"; - break; + // NOTE: no RECONNECT_BUDGET_EXHAUSTED case on purpose. That kind was dropped from + // the client's SenderConnectionEvent.Kind in newer builds (e.g. 1.3.6-SNAPSHOT), so + // switching on it fails to compile there. If a client still emits it, the default + // branch below narrates it generically. case DISCONNECTED: msg = "connection lost to " + host + " (" + cause + "), will retry"; noisy = true; From f0144f7c8da2ab4a6a3b6cd74ae72c02fc65f69d Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 3 Jul 2026 14:19:11 +0200 Subject: [PATCH 05/79] Restore probe serving-role display; instrument switch status The probe fallback printed only "node=(none)" and dropped the role entirely when `switch status` returned nothing, which was worse than before. Restore the role/node/zone from getServerInfo() on every line (it tracks connect + failover), and append the authoritative live role from `switch status` only when present. When switch status yields no live role, log why at most once per 30s: an error status+message, a returned batch with no *role* column (names listed), or no row batch at all -- so the empty case is diagnosable instead of silent. --- .../com/example/sender/CsvParallelSender.java | 52 ++++++++++++++----- 1 file changed, 40 insertions(+), 12 deletions(-) diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index 8d2af37..68e9756 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -581,13 +581,25 @@ public void onFailoverReset(QwpServerInfo info) { orNone(info.getNodeId()), QwpServerInfo.roleName(info.getRole()), orNone(info.getZoneId())); } }; - // Captures the live lifecycle role from `switch status`: joins every column whose - // name mentions "role" (current_role, target_role, ...) into e.g. "current_role=PRIMARY". + // `switch status` gives the LIVE lifecycle role of the serving node (authoritative + // even after an in-place demote, which getServerInfo() would miss). We join every + // column whose name mentions "role" (current_role, target_role, ...) into e.g. + // "current_role=PRIMARY". statusDiag captures WHY no role was produced, for a + // throttled log, so the empty case is diagnosable rather than silent. final String[] liveRole = {null}; + final String[] statusDiag = {null}; + final long[] lastStatusDiagMs = {0L}; final QwpColumnBatchHandler statusHandler = new QwpColumnBatchHandler() { @Override public void onBatch(QwpColumnBatch batch) { final int cols = batch.getColumnCount(); + final StringBuilder names = new StringBuilder(); + for (int c = 0; c < cols; c++) { + if (names.length() > 0) { + names.append(','); + } + names.append(batch.getColumnName(c)); + } batch.forEachRow(row -> { final StringBuilder sb = new StringBuilder(); for (int c = 0; c < cols; c++) { @@ -604,6 +616,9 @@ public void onBatch(QwpColumnBatch batch) { liveRole[0] = sb.toString(); } }); + if (liveRole[0] == null) { + statusDiag[0] = "'" + STATUS_QUERY + "' returned no *role* column; columns=[" + names + "]"; + } } @Override @@ -612,8 +627,7 @@ public void onEnd(long totalRows) { @Override public void onError(byte status, String message) { - // e.g. OSS server without lifecycle, or insufficient permissions: leave - // liveRole null so the probe falls back to the handshake node id. + statusDiag[0] = "'" + STATUS_QUERY + "' error (status " + status + "): " + message; } @Override @@ -623,6 +637,7 @@ public void onFailoverReset(QwpServerInfo info) { while (!Thread.currentThread().isInterrupted()) { latest[0] = Long.MIN_VALUE; liveRole[0] = null; + statusDiag[0] = null; try { client.execute(PROBE_QUERY, handler); if (wasDown[0]) { @@ -636,18 +651,31 @@ public void onFailoverReset(QwpServerInfo info) { // look like a connection loss, so swallow it here (liveRole stays null). try { client.execute(STATUS_QUERY, statusHandler); - } catch (Exception ignore) { - // fall through to the handshake fallback below + } catch (Exception ex) { + statusDiag[0] = "'" + STATUS_QUERY + "' threw: " + ex; } - final String served; - if (liveRole[0] != null) { - served = " served by " + liveRole[0] + " (live)"; - } else { - final QwpServerInfo si = client.getServerInfo(); - served = si != null ? " served by node=" + orNone(si.getNodeId()) : ""; + if (liveRole[0] == null && statusDiag[0] == null) { + statusDiag[0] = "'" + STATUS_QUERY + "' produced no row batch and no error" + + " (not a SELECT-style result on the read path?)"; } + // Always show the client's known role (getServerInfo tracks connect and + // failover); append the authoritative live role from switch status if we got it. + final QwpServerInfo si = client.getServerInfo(); + final String base = si != null + ? " served by role=" + QwpServerInfo.roleName(si.getRole()) + + " node=" + orNone(si.getNodeId()) + " zone=" + orNone(si.getZoneId()) + : ""; + final String served = liveRole[0] != null ? base + " (live: " + liveRole[0] + ")" : base; System.out.printf("[probe] latest trades timestamp = %s (raw=%d)%s%n", ts, latest[0], served); + // Explain a missing live role at most once per 30s so it does not spam. + if (liveRole[0] == null && statusDiag[0] != null) { + final long now = System.currentTimeMillis(); + if (now - lastStatusDiagMs[0] > 30_000L) { + lastStatusDiagMs[0] = now; + System.out.println("[probe] live role unavailable: " + statusDiag[0]); + } + } } } catch (Exception e) { if (!wasDown[0]) { From 07254ae51d69b2e2674d99e3b436c0a20f76f624 Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 8 Jul 2026 12:56:31 +0200 Subject: [PATCH 06/79] Report the probe's serving role from switch status, not the stale handshake The probe printed getServerInfo()'s role as the headline 'served by role=', but that is the QWP handshake role, refreshed only on connect/failover. After an in-place promotion (the read connection never drops) it stayed stuck at REPLICA even though the node was serving reads as PRIMARY. Make switch status' current_role the authoritative role shown, append '(switching -> ROLE)' while a switch is in flight, and demote getServerInfo() to supplying node/zone only -- its handshake role is now a clearly labelled fallback used solely when the status query is unavailable. target=any is unchanged, so replica-fallback reads still work. --- .../com/example/sender/CsvParallelSender.java | 76 ++++++++++++------- 1 file changed, 47 insertions(+), 29 deletions(-) diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index 68e9756..d54e442 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -581,12 +581,14 @@ public void onFailoverReset(QwpServerInfo info) { orNone(info.getNodeId()), QwpServerInfo.roleName(info.getRole()), orNone(info.getZoneId())); } }; - // `switch status` gives the LIVE lifecycle role of the serving node (authoritative - // even after an in-place demote, which getServerInfo() would miss). We join every - // column whose name mentions "role" (current_role, target_role, ...) into e.g. - // "current_role=PRIMARY". statusDiag captures WHY no role was produced, for a - // throttled log, so the empty case is diagnosable rather than silent. - final String[] liveRole = {null}; + // `switch status` reports the serving node's LIVE lifecycle role: current_role, plus + // target_role while a switch is in flight. This is authoritative -- unlike the QWP + // handshake role from getServerInfo(), it reflects an in-place promotion/demotion that + // never dropped the read connection. We show current_role as THE role; getServerInfo() + // only supplies node/zone, and its handshake role is a labelled fallback used solely + // when the status query is unavailable (e.g. missing SYSTEM ADMIN). statusDiag says why. + final String[] currentRole = {null}; + final String[] targetRole = {null}; final String[] statusDiag = {null}; final long[] lastStatusDiagMs = {0L}; final QwpColumnBatchHandler statusHandler = new QwpColumnBatchHandler() { @@ -601,23 +603,28 @@ public void onBatch(QwpColumnBatch batch) { names.append(batch.getColumnName(c)); } batch.forEachRow(row -> { - final StringBuilder sb = new StringBuilder(); for (int c = 0; c < cols; c++) { final String name = batch.getColumnName(c); - if (name != null && name.toLowerCase(java.util.Locale.ROOT).contains("role")) { - if (sb.length() > 0) { - sb.append(' '); - } - sb.append(name).append('=') - .append(row.isNull(c) ? "(none)" : row.getString(c)); + if (name == null) { + continue; + } + final String lower = name.toLowerCase(java.util.Locale.ROOT); + if (!lower.contains("role")) { + continue; + } + final String value = row.isNull(c) ? null : row.getString(c); + if (lower.contains("current")) { + currentRole[0] = value; + } else if (lower.contains("target")) { + targetRole[0] = value; + } else if (currentRole[0] == null) { + // A single unqualified "role" column (older servers) is the current one. + currentRole[0] = value; } - } - if (sb.length() > 0) { - liveRole[0] = sb.toString(); } }); - if (liveRole[0] == null) { - statusDiag[0] = "'" + STATUS_QUERY + "' returned no *role* column; columns=[" + names + "]"; + if (currentRole[0] == null) { + statusDiag[0] = "'" + STATUS_QUERY + "' returned no current-role column; columns=[" + names + "]"; } } @@ -636,7 +643,8 @@ public void onFailoverReset(QwpServerInfo info) { }; while (!Thread.currentThread().isInterrupted()) { latest[0] = Long.MIN_VALUE; - liveRole[0] = null; + currentRole[0] = null; + targetRole[0] = null; statusDiag[0] = null; try { client.execute(PROBE_QUERY, handler); @@ -648,28 +656,38 @@ public void onFailoverReset(QwpServerInfo info) { // trades designated timestamp is microseconds by default. Instant ts = Instant.EPOCH.plus(latest[0], ChronoUnit.MICROS); // Ask the serving node for its live role. A status-query failure must not - // look like a connection loss, so swallow it here (liveRole stays null). + // look like a connection loss, so swallow it here (currentRole stays null). try { client.execute(STATUS_QUERY, statusHandler); } catch (Exception ex) { statusDiag[0] = "'" + STATUS_QUERY + "' threw: " + ex; } - if (liveRole[0] == null && statusDiag[0] == null) { + if (currentRole[0] == null && statusDiag[0] == null) { statusDiag[0] = "'" + STATUS_QUERY + "' produced no row batch and no error" + " (not a SELECT-style result on the read path?)"; } - // Always show the client's known role (getServerInfo tracks connect and - // failover); append the authoritative live role from switch status if we got it. + // Role comes from `switch status` (the authoritative live role of the serving + // node). getServerInfo() only supplies node/zone here; its handshake role is a + // labelled fallback used only when the status query is unavailable. final QwpServerInfo si = client.getServerInfo(); - final String base = si != null - ? " served by role=" + QwpServerInfo.roleName(si.getRole()) - + " node=" + orNone(si.getNodeId()) + " zone=" + orNone(si.getZoneId()) - : ""; - final String served = liveRole[0] != null ? base + " (live: " + liveRole[0] + ")" : base; + final String node = si != null ? orNone(si.getNodeId()) : "(none)"; + final String zone = si != null ? orNone(si.getZoneId()) : "(none)"; + final String served; + if (currentRole[0] != null) { + final boolean switching = targetRole[0] != null + && !targetRole[0].equalsIgnoreCase(currentRole[0]); + served = " served by role=" + currentRole[0] + + (switching ? " (switching -> " + targetRole[0] + ")" : "") + + " node=" + node + " zone=" + zone; + } else { + final String handshake = si != null ? QwpServerInfo.roleName(si.getRole()) : "unknown"; + served = " served by role=" + handshake + " node=" + node + " zone=" + zone + + " (handshake role; live 'switch status' unavailable, may be stale)"; + } System.out.printf("[probe] latest trades timestamp = %s (raw=%d)%s%n", ts, latest[0], served); // Explain a missing live role at most once per 30s so it does not spam. - if (liveRole[0] == null && statusDiag[0] != null) { + if (currentRole[0] == null && statusDiag[0] != null) { final long now = System.currentTimeMillis(); if (now - lastStatusDiagMs[0] > 30_000L) { lastStatusDiagMs[0] = now; From 7db22e004b393d4e2735a43e1a8927b965d89efc Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 8 Jul 2026 13:34:18 +0200 Subject: [PATCH 07/79] Document all watchdog flags and refresh the probe output in the README The watchdog Configuration section only named a few QDB_WD_* vars inline and deferred the rest to the env example; enumerate all 14 (required + optional) with defaults and purpose. Also update the probe example to the current 'served by role= node=... zone=...' format, covering the '(switching -> ROLE)' in-flight case and the labelled handshake fallback. --- README.md | 48 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 38 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 58ae26a..5e76651 100644 --- a/README.md +++ b/README.md @@ -121,16 +121,19 @@ runs `select timestamp from trades limit -1` over a **QWP query client** and is independent of the senders, so it does not affect ingestion. ``` -[probe] latest trades timestamp = 2026-07-01T14:31:40.591545Z (raw=1782916300591545) served by current_role=PRIMARY target_role=PRIMARY (live) +[probe] latest trades timestamp = 2026-07-01T14:31:40.591545Z (raw=1782916300591545) served by role=PRIMARY node=n1 zone=eu-west-1 ``` -The `served by ... (live)` suffix is the **live** lifecycle role of the node currently -answering the probe, obtained by running `switch status` on it each poll. This matters on -failover: the QWP handshake `SERVER_INFO` (node/role) is only refreshed on a *reconnect*, so -after an in-place primary→replica switch (which does not drop the read connection) it goes -stale. `switch status` always reflects the truth, so the role you see flips the moment the -serving node changes role. On a server without the lifecycle API (OSS), the status query is -unavailable and the probe falls back to `served by node=` from the handshake. +The `served by role=...` is the **live** lifecycle role (`current_role` from `switch status`) +of the node currently answering the probe, run on it each poll. It is authoritative: the QWP +handshake `SERVER_INFO` role is only refreshed on a *reconnect*, so after an in-place +primary→replica switch (which does not drop the read connection) that role goes stale, +whereas `switch status` always reflects the truth, so the role flips the moment the serving +node changes role. `node`/`zone` come from the handshake `SERVER_INFO`. While a switch is in +flight, `(switching -> ROLE)` is appended (the `target_role` differs from `current_role`). If +the status query is unavailable (OSS, or the token lacks `SYSTEM ADMIN`), the probe falls back +to the handshake role, explicitly labelled `(handshake role; live 'switch status' unavailable, +may be stale)`. The query client uses the **same host list and token/auth** as the senders and enables **failover** when more than one host is given (`ws|wss::addr=h1,h2,...;...;failover=on`), so @@ -362,8 +365,33 @@ export QDB_REST_TOKEN=... QDB_WD_SERVERS=... QDB_WD_FAIL_THRESHOLD=2 ``` The script auto-loads a config file if present (first of `/etc/qdb-primary-watchdog.env`, -`~/.config/qdb-primary-watchdog.env`, or one beside the script; or `$QDB_WD_CONFIG`). See -[`qdb-primary-watchdog.env.example`](./qdb-primary-watchdog.env.example) for every setting. +`~/.config/qdb-primary-watchdog.env`, or one beside the script; or `$QDB_WD_CONFIG`). +[`qdb-primary-watchdog.env.example`](./qdb-primary-watchdog.env.example) is a copy-ready +template of the same settings. + +**Required (no defaults):** + +| Variable | Purpose | +| --- | --- | +| `QDB_REST_TOKEN` | Bearer token for the lifecycle REST API. | +| `QDB_WD_SERVERS` | Ordered `host:9003,...` node list. The first CLI argument overrides it. | + +**Optional (defaults shown):** + +| Variable | Default | Purpose | +| --- | --- | --- | +| `QDB_WD_CONFIG` | *(auto-discover)* | Explicit path to the config file to load, bypassing discovery. | +| `QDB_WD_TARGET_ROLE` | `primary` | Role a node is switched to on failover. | +| `QDB_WD_SWITCH_TIMEOUT_MS` | `5000` | `timeout_ms` in the switch payload, and the promote backoff base. | +| `QDB_WD_STEPDOWN_CONNECT_TIMEOUT` | `3` | Connect timeout (s) for the old-primary step-down; no overall timeout. | +| `QDB_WD_CHECK_CONNECT_TIMEOUT` | `2` | Connect timeout (s) for each role/health poll. | +| `QDB_WD_CHECK_MAX_TIME` | `3` | Overall max time (s) for each role/health poll. | +| `QDB_WD_FAIL_THRESHOLD` | `3` | Consecutive non-primary reads before failover. | +| `QDB_WD_CHECK_INTERVAL` | `1` | Seconds between role checks. | +| `QDB_WD_GRACE_PERIOD` | `5` | Seconds to wait after detecting loss before acting (`0` disables), so a manual operator switch can settle; re-checked after. | +| `QDB_WD_SWITCH_RETRIES` | `3` | Promote attempts, with exponential backoff between them. | +| `QDB_WD_SWITCH_POLL_INTERVAL` | `1` | Seconds between lifecycle polls after a switch. | +| `QDB_WD_SWITCH_POLL_MAX` | `30` | Max polls to wait for a switch to settle. | ### Run it as a service From e2e9c8028e04b6d3267e2b3fd3923ace6bf4bd86 Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 10 Jul 2026 12:49:04 +0200 Subject: [PATCH 08/79] Add qwpudp (QWP/UDP) transport option Add --protocol qwpudp for fire-and-forget UDP datagram ingest to the UDP port (:9007). It is ingest-only, unauthenticated (auth flags are warned about and ignored), and single-endpoint with no failover, store-and-forward, TLS, or query client (the probe is skipped). The worker flushes datagrams every --batch-size rows and, for a single worker, stamps rows client-side; multiple workers use server-side atNow(). Document the transport, including its best-effort nature and the burst/packet-loss behaviour, with guidance to keep --batch-size small for UDP. --- README.md | 66 ++++++++++++++++--- .../com/example/sender/CsvParallelSender.java | 65 ++++++++++++++---- 2 files changed, 111 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index 5e76651..44f2538 100644 --- a/README.md +++ b/README.md @@ -51,10 +51,15 @@ connect attempt. - `--protocol qwp` (default): QWP over WebSocket. Adds store-and-forward (un-acked frames spill to disk and replay after an outage) and transactional commit. Use the server's **HTTP port** (`:9000`) in `--addrs`. -- `--protocol ilp`: the previous HTTP/ILP transport, unchanged. Ignores the QWP-only - flags below. +- `--protocol qwpudp`: QWP over **UDP**, fire-and-forget datagram ingest. Use the + server's **UDP ingest port** (`:9007` by convention) in `--addrs`. It is **ingest-only, + unauthenticated, best-effort, and single-endpoint** (no failover). See + [QWP/UDP transport](#qwpudp-transport-best-effort-single-endpoint) for the full picture + and how to avoid packet loss. +- `--protocol ilp`: the previous HTTP/ILP transport, unchanged. Ignores the + QWP/WebSocket-only flags below. -QWP-only flags: +QWP/WebSocket-only flags (`qwp`; ignored by `qwpudp` and `ilp`): - `--sender-id` (default `ha_sender`): store-and-forward key. **Must be unique per process/server**; set it explicitly when running more than one process on a host. @@ -76,7 +81,7 @@ QWP-only flags: OS connect timeout (~minutes) before failing over to the next `--addrs` host; with it, failover happens in seconds. The ILP path is left unbounded so `--protocol ilp` stays identical. - `--probe-interval-ms` (default `1000`, `0` disables): interval for the probe thread - that polls the latest ingested timestamp. See [Probe](#probe-qwp-only) below. + that polls the latest ingested timestamp. See [Probe](#probe-qwpwebsocket-only) below. - `--enterprise` (default `false`): request durable acks (`requestDurableAck`), holding spilled frames until the server confirms a durable commit. **Enterprise only**: an OSS server rejects the WebSocket upgrade, so leave it off against OSS. See @@ -86,6 +91,51 @@ QWP-only flags: Enterprise; a no-op on OSS. The ingestion path is zone-blind (it must follow the primary), so this does not affect the senders. +## QWP/UDP transport (best-effort, single endpoint) + +`--protocol qwpudp` sends rows as UDP datagrams to the server's UDP ingest port (`:9007` +by convention; put it in `--addrs`). The server must have UDP ingest enabled, and it +accepts **any** connection on that port. It is the simplest and lowest-overhead +transport, at the cost of every reliability guarantee the other two provide. + +**Ingest-only, unauthenticated.** There is no query path, so the probe is not started and +`--probe-interval-ms` is ignored. UDP rejects authentication: `--token`, `--username`, and +`--password` are ignored (with a warning). Store-and-forward, TLS, and durable ack do not +apply. + +**Single endpoint, no failover.** UDP is connectionless and unacknowledged: there is no +connection to detect as dropped and no delivery feedback, so the client cannot know a host +is down and therefore **cannot fail over**. The client enforces this by rejecting more than +one address: + +``` +only a single address (host:port) is supported for UDP transport +``` + +Pass exactly one `host:9007`. If that host is down, datagrams are silently lost. For high +availability across hosts use `qwp` (WebSocket store-and-forward) or `ilp` (HTTP retry). + +**Best-effort: datagrams can be dropped.** There is no ack, no retransmit, and no +store-and-forward, so a lost datagram is a lost row. UDP also has **no backpressure**: if +the sender emits datagrams faster than the server drains its OS receive buffer, the buffer +overflows and datagrams are dropped **wholesale**, not one at a time. + +That burst sensitivity is the main thing to tune. Rows are flushed to datagrams every +`--batch-size` rows, so a large batch flushed all at once (especially by several workers +finishing together) is exactly the burst that overflows the buffer. Observed on loopback: + +| Config | Result | +| --- | --- | +| `--num-senders 4 --batch-size 10000` (one big burst per worker) | **0 of 8000 rows landed** | +| `--num-senders 4 --batch-size 100` (paced, small flushes) | **8000 of 8000 rows landed** | + +So for UDP, **keep `--batch-size` small** (e.g. `100`-`500`) to pace the datagrams, and if +you still see loss, raise the OS UDP receive-buffer size on the server host +(`net.core.rmem_max` / `rmem_default` on Linux). Even then, treat UDP as best-effort: +enable dedup and reconcile counts if you need to know what landed. Timestamp behavior is +the same as the other transports (a single worker stamps client-side micros; multiple +workers use server-side `atNow()`; see [Timestamps](#timestamps-and-throughput)). + ## Commit cadence (freshness vs throughput) On QWP the sender runs in `transactional(true)` mode. That decouples *flushing* from @@ -113,9 +163,9 @@ seconds between commits (per worker) = (batch-size × batches-per-transaction) / for many seconds. Keep `--batches-per-transaction` small when freshness matters more than raw throughput. -## Probe (QWP only) +## Probe (QWP/WebSocket only) -When the transport is QWP, a separate thread polls the latest ingested timestamp and +When the transport is `qwp` (WebSocket), a separate thread polls the latest ingested timestamp and prints it to stdout once per `--probe-interval-ms` (default `1000` ms; `0` disables). It runs `select timestamp from trades limit -1` over a **QWP query client** and is fully independent of the senders, so it does not affect ingestion. @@ -140,8 +190,8 @@ The query client uses the **same host list and token/auth** as the senders and e if a host stops responding it moves to the next one automatically and narrates the transition (see [Failover and connection narration](#failover-and-connection-narration-qwp)). Before the `trades` table exists it prints `[query client] server error: table does not -exist` each interval, then switches to timestamps once ingestion begins. ILP ignores -`--probe-interval-ms`. +exist` each interval, then switches to timestamps once ingestion begins. `ilp` and +`qwpudp` ignore `--probe-interval-ms` (neither has a query path). On connect the probe prints the instance actually serving it: diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index d54e442..05da68d 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -103,10 +103,22 @@ public static void main(String[] args) throws Exception { final int connectTimeoutMs = Integer.parseInt(a.getOrDefault("--connect-timeout-ms", String.valueOf(DEFAULT_CONNECT_TIMEOUT_MS))); - if (!protocol.equals("qwp") && !protocol.equals("ilp")) { - System.err.println("--protocol must be 'qwp' or 'ilp', got: " + protocol); + if (!protocol.equals("qwp") && !protocol.equals("ilp") && !protocol.equals("qwpudp")) { + System.err.println("--protocol must be 'qwp', 'qwpudp', or 'ilp', got: " + protocol); System.exit(2); } + // QWP/UDP is fire-and-forget datagram ingest: the transport rejects authentication + // (the server accepts any connection on the UDP port), so any credentials passed are + // meaningless. Warn rather than fail, so the same command line works across transports. + if (protocol.equals("qwpudp")) { + final boolean hasAnyAuth = (token != null && !token.isEmpty()) + || (username != null && !username.isEmpty()) + || (password != null && !password.isEmpty()); + if (hasAnyAuth) { + System.err.println("[warn] --protocol qwpudp is unauthenticated (UDP accepts any connection);" + + " ignoring --token/--username/--password"); + } + } if (probeIntervalMs < 0) { System.err.println("--probe-interval-ms must be >= 0 (0 disables the probe)"); System.exit(2); @@ -147,7 +159,10 @@ public static void main(String[] args) throws Exception { ? " (WebSocket, sender-id=" + senderIdBase + ", store-and-forward=" + storeForwardDir + ", batch-size=" + batchSize + ", batches-per-transaction=" + batchesPerTransaction + ", connect-timeout-ms=" + connectTimeoutMs + ", retry-timeout-ms=" + retryTimeout + ")" - : "") + : protocol.equals("qwpudp") + ? " (QWP/UDP datagrams, ingest-only, unauthenticated, no store-and-forward," + + " no failover; query client disabled, batch-size=" + batchSize + ")" + : "") + " | config: " + conf.replaceAll("(token=)([^;]+)", "$1***") .replaceAll("(password=)([^;]+)", "$1***")); @@ -182,8 +197,10 @@ public static void main(String[] args) throws Exception { reporter.setDaemon(true); reporter.start(); - // Probe (QWP only): a separate thread polls the latest ingested timestamp over a - // QWP query client. Same hosts/auth as the senders; it fails over automatically. + // Probe (QWP/WebSocket only): a separate thread polls the latest ingested timestamp + // over a QWP query client. Same hosts/auth as the senders; it fails over automatically. + // Skipped for ilp and for qwpudp: UDP is ingest-only (there is no query path), so the + // equals("qwp") guard deliberately excludes it. final Thread probe = (protocol.equals("qwp") && probeIntervalMs > 0) ? startProbe(cfg, probeIntervalMs) : null; @@ -227,15 +244,21 @@ private static void runWorker( System.out.printf("Sender %d will send %d events%n", senderId, totalEvents); long sent = 0; final boolean isQwp = cfg.protocol.equals("qwp"); - // QWP with a single worker: stamp each row with the current time client-side. A single - // thread's timestamps are monotonic (no O3) and this avoids QWP's per-batch atNow() - // stamping. ILP, or QWP with more than one worker, use atNow() (server-side, O3-safe). - final boolean perRowMicros = isQwp && cfg.numSenders == 1; - // QWP transactional commit cadence: commit every batchSize * batchesPerTransaction - // rows via an explicit flush(). 0 disables periodic commits (ILP path is unchanged). + final boolean isUdp = cfg.protocol.equals("qwpudp"); + // Single worker on a QWP transport (WebSocket or UDP): stamp each row with the current + // time client-side. A single thread's timestamps are monotonic (no O3) and this avoids + // QWP's per-batch atNow() stamping. ILP, or more than one worker, use atNow() + // (server-side, O3-safe). + final boolean perRowMicros = (isQwp || isUdp) && cfg.numSenders == 1; + // Flush cadence, by transport: + // qwp - transactional commit every batchSize * batchesPerTransaction rows (flush commits). + // qwpudp - no transactions; flush every batchSize rows to emit datagrams (bounds the buffer). + // ilp - 0: no explicit mid-loop flush (the HTTP client auto-flushes). final long commitEveryRows = isQwp ? (long) cfg.batchSize * cfg.batchesPerTransaction - : 0L; + : isUdp + ? cfg.batchSize + : 0L; try ( Sender sender = buildSender(cfg, senderId)) { //( Sender sender = Sender.fromConfig(conf)) { final int n = rows.size(); for (long i = 0; i < totalEvents; i++) { @@ -385,6 +408,24 @@ private static Sender buildSender(SenderCfg cfg, int workerId) { return buildBuilder(cfg.addrsCsv, cfg.token, cfg.username, cfg.password, cfg.retryTimeout).build(); } + if ("qwpudp".equals(cfg.protocol)) { + // ---- QWP/UDP branch ---- + // Fire-and-forget datagrams to the UDP ingest port (:9007 by convention). The + // transport rejects authentication and does not use TLS, store-and-forward, + // failover, or a connection listener (there is no persistent connection). Any + // token/basic auth on the command line was already warned about and is ignored. + String[] udpAddrs = Arrays.stream(cfg.addrsCsv.split(",")) + .map(String::trim) + .filter(s -> !s.isEmpty()) + .toArray(String[]::new); + + LineSenderBuilder u = Sender.builder(Sender.Transport.UDP); + for (String addr : udpAddrs) { + u.address(addr); + } + return u.build(); + } + // ---- QWP (WebSocket) branch ---- String[] addrs = Arrays.stream(cfg.addrsCsv.split(",")) .map(String::trim) From 20e709b12692bf17c8612690344b57cda09a9ebd Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 10 Jul 2026 16:50:44 +0200 Subject: [PATCH 09/79] Add Rust HA sender port Port the Java CsvParallelSender to Rust, built on the local questdb-rs (c-questdb-client) as a path dependency. Mirrors the Java client: - qwp (WebSocket) with per-worker store-and-forward, transactional commit cadence, and multi-host failover. - qwpudp (UDP) fire-and-forget datagrams: ingest-only, unauthenticated, single-endpoint, with upfront single-address validation. - ilp (HTTP) legacy transport. - Same connection options (address list, token / basic auth, TLS, zone, enterprise durable ack, reconnect budget) and timestamp semantics (single-worker client micros vs multi-worker at_now). - A Reader-based probe (qwp only) polling the latest ingested timestamp and the serving node's live role via 'switch status', with target=any replica-fallback reads and automatic failover. Batching is driven by explicit flush() calls since the Rust client has no auto-flush by design. Validated end-to-end against a local server, including a primary-crash failover with gap-free store-and-forward handoff. --- rust/.gitignore | 1 + rust/Cargo.lock | 1857 ++++++++++++++++++++++++++++++++++++++++++++++ rust/Cargo.toml | 27 + rust/README.md | 133 ++++ rust/src/main.rs | 641 ++++++++++++++++ 5 files changed, 2659 insertions(+) create mode 100644 rust/.gitignore create mode 100644 rust/Cargo.lock create mode 100644 rust/Cargo.toml create mode 100644 rust/README.md create mode 100644 rust/src/main.rs diff --git a/rust/.gitignore b/rust/.gitignore new file mode 100644 index 0000000..ea8c4bf --- /dev/null +++ b/rust/.gitignore @@ -0,0 +1 @@ +/target diff --git a/rust/Cargo.lock b/rust/Cargo.lock new file mode 100644 index 0000000..7421cb0 --- /dev/null +++ b/rust/Cargo.lock @@ -0,0 +1,1857 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "adler2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" + +[[package]] +name = "aes" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" +dependencies = [ + "cfg-if", + "cipher", + "cpufeatures 0.2.17", +] + +[[package]] +name = "android_system_properties" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" +dependencies = [ + "libc", +] + +[[package]] +name = "anstream" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" +dependencies = [ + "anstyle", + "anstyle-parse", + "anstyle-query", + "anstyle-wincon", + "colorchoice", + "is_terminal_polyfill", + "utf8parse", +] + +[[package]] +name = "anstyle" +version = "1.0.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" + +[[package]] +name = "anstyle-parse" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" +dependencies = [ + "utf8parse", +] + +[[package]] +name = "anstyle-query" +version = "1.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "anstyle-wincon" +version = "3.0.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" +dependencies = [ + "anstyle", + "once_cell_polyfill", + "windows-sys 0.61.2", +] + +[[package]] +name = "asn1-rs" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f6fd5ddaf0351dff5b8da21b2fb4ff8e08ddd02857f0bf69c47639106c0fff0" +dependencies = [ + "asn1-rs-derive 0.4.0", + "asn1-rs-impl 0.1.0", + "displaydoc", + "nom", + "num-traits", + "rusticata-macros", + "thiserror 1.0.69", +] + +[[package]] +name = "asn1-rs" +version = "0.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7f43a50ac4fdca5df8e885c21b835997f0a1cdee65494a6847694a98652d9d8" +dependencies = [ + "asn1-rs-derive 0.6.0", + "asn1-rs-impl 0.2.0", + "displaydoc", + "nom", + "num-traits", + "rusticata-macros", + "thiserror 2.0.18", + "time", +] + +[[package]] +name = "asn1-rs-derive" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "726535892e8eae7e70657b4c8ea93d26b8553afb1ce617caee529ef96d7dee6c" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", + "synstructure 0.12.6", +] + +[[package]] +name = "asn1-rs-derive" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3109e49b1e4909e9db6515a30c633684d68cdeaa252f215214cb4fa1a5bfee2c" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", + "synstructure 0.13.2", +] + +[[package]] +name = "asn1-rs-impl" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2777730b2039ac0f95f093556e61b6d26cebed5393ca6f152717777cec3a42ed" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "asn1-rs-impl" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b18050c2cd6fe86c3a76584ef5e0baf286d038cda203eb6223df2cc413565f7" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "base64ct" +version = "1.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" + +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + +[[package]] +name = "block-padding" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a8894febbff9f758034a5b8e12d87918f56dfc64a8e1fe757d65e29041538d93" +dependencies = [ + "generic-array", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "cbc" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "26b52a9543ae338f279b96b0b9fed9c8093744685043739079ce85cd58f289a6" +dependencies = [ + "cipher", +] + +[[package]] +name = "cc" +version = "1.2.66" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f5d6cac793997bd970000024b2934968efe83b382de4fdcf4fcb46b6ee4ad996" +dependencies = [ + "find-msvc-tools", + "jobserver", + "libc", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "chacha20" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "rand_core 0.10.1", +] + +[[package]] +name = "chrono" +version = "0.4.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" +dependencies = [ + "iana-time-zone", + "js-sys", + "num-traits", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "cipher" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" +dependencies = [ + "crypto-common", + "inout", +] + +[[package]] +name = "clap" +version = "4.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ddb117e43bbf7dacf0a4190fef4d345b9bad68dfc649cb349e7d17d28428e51" +dependencies = [ + "clap_builder", + "clap_derive", +] + +[[package]] +name = "clap_builder" +version = "4.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "714a53001bf66416adb0e2ef5ac857140e7dc3a0c48fb28b2f10762fc4b5069f" +dependencies = [ + "anstream", + "anstyle", + "clap_lex", + "strsim", +] + +[[package]] +name = "clap_derive" +version = "4.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2ce8604710f6733aa641a2b3731eaa1e8b3d9973d5e3565da11800813f997a9" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "clap_lex" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" + +[[package]] +name = "cms" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b77c319abfd5219629c45c34c89ba945ed3c5e49fcde9d16b6c3885f118a730" +dependencies = [ + "const-oid", + "der", + "spki", + "x509-cert", +] + +[[package]] +name = "colorchoice" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" + +[[package]] +name = "const-oid" +version = "0.9.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" + +[[package]] +name = "core-foundation-sys" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" + +[[package]] +name = "cpufeatures" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +dependencies = [ + "libc", +] + +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + +[[package]] +name = "crc32c" +version = "0.6.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a47af21622d091a8f0fb295b88bc886ac74efcc613efc19f5d0b21de5c89e47" +dependencies = [ + "rustc_version", +] + +[[package]] +name = "crc32fast" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "csv" +version = "1.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52cd9d68cf7efc6ddfaaee42e7288d3a99d613d4b50f76ce9827ae0c6e14f938" +dependencies = [ + "csv-core", + "itoa", + "ryu", + "serde_core", +] + +[[package]] +name = "csv-core" +version = "0.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "704a3c26996a80471189265814dbc2c257598b96b8a7feae2d31ace646bb9782" +dependencies = [ + "memchr", +] + +[[package]] +name = "data-encoding" +version = "2.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8" + +[[package]] +name = "der" +version = "0.7.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" +dependencies = [ + "const-oid", + "der_derive", + "flagset", + "pem-rfc7468", + "zeroize", +] + +[[package]] +name = "der-parser" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "07da5016415d5a3c4dd39b11ed26f915f52fc4e0dc197d87908bc916e51bc1a6" +dependencies = [ + "asn1-rs 0.7.2", + "displaydoc", + "nom", + "num-bigint", + "num-traits", + "rusticata-macros", +] + +[[package]] +name = "der_derive" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8034092389675178f570469e6c3b0465d3d30b4505c294a6550db47f3c17ad18" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "deranged" +version = "0.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" + +[[package]] +name = "des" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ffdd80ce8ce993de27e9f063a444a4d53ce8e8db4c1f00cc03af5ad5a9867a1e" +dependencies = [ + "cipher", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer", + "crypto-common", + "subtle", +] + +[[package]] +name = "displaydoc" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "dns-lookup" +version = "3.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e39034cee21a2f5bbb66ba0e3689819c4bb5d00382a282006e802a7ffa6c41d" +dependencies = [ + "cfg-if", + "libc", + "socket2", + "windows-sys 0.60.2", +] + +[[package]] +name = "find-msvc-tools" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" + +[[package]] +name = "flagset" +version = "0.4.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7ac824320a75a52197e8f2d787f6a38b6718bb6897a35142d749af3c0e8f4fe" + +[[package]] +name = "flate2" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" +dependencies = [ + "crc32fast", + "miniz_oxide", +] + +[[package]] +name = "futures-core" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" + +[[package]] +name = "futures-task" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" + +[[package]] +name = "futures-util" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" +dependencies = [ + "futures-core", + "futures-task", + "pin-project-lite", + "slab", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "libc", + "wasi", +] + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "libc", + "r-efi 5.3.0", + "wasip2", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "libc", + "r-efi 6.0.0", + "rand_core 0.10.1", +] + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "hex" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" + +[[package]] +name = "hmac" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" +dependencies = [ + "digest", +] + +[[package]] +name = "http" +version = "1.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6970f50e31d6fc17d3fa27329444bfa74e196cf62e95052a3f6fee181dba6425" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "httparse" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" + +[[package]] +name = "iana-time-zone" +version = "0.1.65" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" +dependencies = [ + "android_system_properties", + "core-foundation-sys", + "iana-time-zone-haiku", + "js-sys", + "log", + "wasm-bindgen", + "windows-core", +] + +[[package]] +name = "iana-time-zone-haiku" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" +dependencies = [ + "cc", +] + +[[package]] +name = "indoc" +version = "2.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "79cf5c93f93228cf8efb3ba362535fb11199ac548a09ce117c9b1adc3030d706" +dependencies = [ + "rustversion", +] + +[[package]] +name = "inout" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" +dependencies = [ + "block-padding", + "generic-array", +] + +[[package]] +name = "is_terminal_polyfill" +version = "1.70.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "jks" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e03966fd15eea3cb2886320a78d01e77f8aaeabd3fb01504ee6a2238876c23bc" +dependencies = [ + "asn1-rs 0.5.2", + "sha1", + "thiserror 1.0.69", +] + +[[package]] +name = "jobserver" +version = "0.1.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3" +dependencies = [ + "getrandom 0.4.3", + "libc", +] + +[[package]] +name = "js-sys" +version = "0.3.103" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "lazy_static" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "log" +version = "0.4.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "memmap2" +version = "0.9.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d1219ed1b7f229ee7104d281dd01d6802fe28bb6e95d292942c4daacdeb798c0" +dependencies = [ + "libc", +] + +[[package]] +name = "minimal-lexical" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a" + +[[package]] +name = "miniz_oxide" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +dependencies = [ + "adler2", + "simd-adler32", +] + +[[package]] +name = "nom" +version = "7.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a" +dependencies = [ + "memchr", + "minimal-lexical", +] + +[[package]] +name = "num-bigint" +version = "0.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c89e69e7e0f03bea5ef08013795c25018e101932225a656383bd384495ecc367" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-conv" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441" + +[[package]] +name = "num-integer" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", +] + +[[package]] +name = "oid-registry" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12f40cff3dde1b6087cc5d5f5d4d65712f34016a03ed60e9c08dcc392736b5b7" +dependencies = [ + "asn1-rs 0.7.2", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "once_cell_polyfill" +version = "1.70.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" + +[[package]] +name = "p12-keystore" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ffb9bf5222606eb712d3bb30e01bc9420545b00859970897e70c682353a034f2" +dependencies = [ + "base64", + "cbc", + "cms", + "der", + "des", + "hex", + "hmac", + "pkcs12", + "pkcs5", + "rand 0.10.2", + "rc2", + "sha1", + "sha2", + "thiserror 2.0.18", + "x509-parser", +] + +[[package]] +name = "pbkdf2" +version = "0.12.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8ed6a7761f76e3b9f92dfb0a60a6a6477c61024b775147ff0973a02653abaf2" +dependencies = [ + "digest", + "hmac", +] + +[[package]] +name = "pem-rfc7468" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88b39c9bfcfc231068454382784bb460aae594343fb030d46e9f50a645418412" +dependencies = [ + "base64ct", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "pkcs12" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "695b3df3d3cc1015f12d70235e35b6b79befc5fa7a9b95b951eab1dd07c9efc2" +dependencies = [ + "cms", + "const-oid", + "der", + "digest", + "spki", + "x509-cert", + "zeroize", +] + +[[package]] +name = "pkcs5" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e847e2c91a18bfa887dd028ec33f2fe6f25db77db3619024764914affe8b69a6" +dependencies = [ + "aes", + "cbc", + "der", + "pbkdf2", + "scrypt", + "sha2", + "spki", +] + +[[package]] +name = "pkg-config" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" + +[[package]] +name = "powerfmt" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" + +[[package]] +name = "ppv-lite86" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" +dependencies = [ + "zerocopy", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "questdb-confstr" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7aceffde1cbf8e67f34cdfd70d2436396176d6ff648fa719e0231fb9856ef3e9" + +[[package]] +name = "questdb-rs" +version = "7.0.0" +dependencies = [ + "base64ct", + "bytes", + "crc32c", + "dns-lookup", + "indoc", + "itoa", + "jks", + "libc", + "log", + "memmap2", + "p12-keystore", + "questdb-confstr", + "rand 0.9.4", + "ring", + "rustls", + "rustls-pki-types", + "ryu", + "serde", + "serde_json", + "slugify", + "socket2", + "ureq", + "webpki-roots", + "windows-sys 0.60.2", + "zstd", +] + +[[package]] +name = "questdb_ha_sender" +version = "1.0.0" +dependencies = [ + "chrono", + "clap", + "csv", + "flate2", + "questdb-rs", +] + +[[package]] +name = "quote" +version = "1.0.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "rand" +version = "0.9.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea" +dependencies = [ + "rand_chacha", + "rand_core 0.9.5", +] + +[[package]] +name = "rand" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80" +dependencies = [ + "chacha20", + "getrandom 0.4.3", + "rand_core 0.10.1", +] + +[[package]] +name = "rand_chacha" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" +dependencies = [ + "ppv-lite86", + "rand_core 0.9.5", +] + +[[package]] +name = "rand_core" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" +dependencies = [ + "getrandom 0.3.4", +] + +[[package]] +name = "rand_core" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" + +[[package]] +name = "rc2" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "62c64daa8e9438b84aaae55010a93f396f8e60e3911590fcba770d04643fc1dd" +dependencies = [ + "cipher", +] + +[[package]] +name = "ring" +version = "0.17.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7" +dependencies = [ + "cc", + "cfg-if", + "getrandom 0.2.17", + "libc", + "untrusted", + "windows-sys 0.52.0", +] + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "rusticata-macros" +version = "4.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "faf0c4a6ece9950b9abdb62b1cfcf2a68b3b67a10ba445b3bb85be2a293d0632" +dependencies = [ + "nom", +] + +[[package]] +name = "rustls" +version = "0.23.41" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b92b125634d9b795e7beca796cc790df15a7fb38323bf3196fda83292d06b1f" +dependencies = [ + "log", + "once_cell", + "ring", + "rustls-pki-types", + "rustls-webpki", + "subtle", + "zeroize", +] + +[[package]] +name = "rustls-pki-types" +version = "1.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "764899a24af3980067ee14bc143654f297b22eaebfe3c7b6b211920a5a59b046" +dependencies = [ + "zeroize", +] + +[[package]] +name = "rustls-webpki" +version = "0.103.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" +dependencies = [ + "ring", + "rustls-pki-types", + "untrusted", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "ryu" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" + +[[package]] +name = "salsa20" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "97a22f5af31f73a954c10289c93e8a50cc23d971e80ee446f1f6f7137a088213" +dependencies = [ + "cipher", +] + +[[package]] +name = "scrypt" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0516a385866c09368f0b5bcd1caff3366aace790fcd46e2bb032697bb172fd1f" +dependencies = [ + "pbkdf2", + "salsa20", + "sha2", +] + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "sha1" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "digest", +] + +[[package]] +name = "sha2" +version = "0.10.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "digest", +] + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" + +[[package]] +name = "simd-adler32" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "slugify" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6b8cf203d2088b831d7558f8e5151bfa420c57a34240b28cee29d0ae5f2ac8b" +dependencies = [ + "unidecode", +] + +[[package]] +name = "socket2" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52d1cfed4120b4d927bf7c0f86d2087a4a7d6027c906d9f9d525a80573b9be51" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "spki" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" +dependencies = [ + "base64ct", + "der", +] + +[[package]] +name = "strsim" +version = "0.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" + +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "2.0.118" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "synstructure" +version = "0.12.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f36bdaa60a83aca3921b5259d5400cbf5e90fc51931376a9bd4a0eb79aa7210f" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", + "unicode-xid", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "thiserror" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" +dependencies = [ + "thiserror-impl 1.0.69", +] + +[[package]] +name = "thiserror" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" +dependencies = [ + "thiserror-impl 2.0.18", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "time" +version = "0.3.53" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "18dfaaeddcb932337b5e7866ee7d0ce9b76d2fd092997146f187ec09b4558a50" +dependencies = [ + "deranged", + "num-conv", + "powerfmt", + "serde_core", + "time-core", + "time-macros", +] + +[[package]] +name = "time-core" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109" + +[[package]] +name = "time-macros" +version = "0.2.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c431b87111666e491a90baa837f914fb45cd5dc3c268591b0220ff5057f2085f" +dependencies = [ + "num-conv", + "time-core", +] + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "unicode-xid" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" + +[[package]] +name = "unidecode" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "402bb19d8e03f1d1a7450e2bd613980869438e0666331be3e073089124aa1adc" + +[[package]] +name = "untrusted" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" + +[[package]] +name = "ureq" +version = "3.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d39cb1dbab692d82a977c0392ffac19e188bd9186a9f32806f0aaa859d75585a" +dependencies = [ + "base64", + "log", + "percent-encoding", + "ureq-proto", + "utf-8", +] + +[[package]] +name = "ureq-proto" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d81f9efa9df032be5934a46a068815a10a042b494b6a58cb0a1a97bb5467ed6f" +dependencies = [ + "base64", + "http", + "httparse", + "log", +] + +[[package]] +name = "utf-8" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09cc8ee72d2a9becf2f2febe0205bbed8fc6615b7cb429ad062dc7b7ddd036a9" + +[[package]] +name = "utf8parse" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wasip2" +version = "1.0.4+wasi-0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487" +dependencies = [ + "wit-bindgen", +] + +[[package]] +name = "wasm-bindgen" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 2.0.118", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "webpki-roots" +version = "1.0.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf85cb06032201fa7c6f829d7db5a7e5aa45bcc0655327713065f6f0576731bf" +dependencies = [ + "rustls-pki-types", +] + +[[package]] +name = "windows-core" +version = "0.62.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" +dependencies = [ + "windows-implement", + "windows-interface", + "windows-link", + "windows-result", + "windows-strings", +] + +[[package]] +name = "windows-implement" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "windows-interface" +version = "0.59.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-result" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-strings" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-sys" +version = "0.52.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" +dependencies = [ + "windows-targets 0.52.6", +] + +[[package]] +name = "windows-sys" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" +dependencies = [ + "windows-targets 0.53.5", +] + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-targets" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" +dependencies = [ + "windows_aarch64_gnullvm 0.52.6", + "windows_aarch64_msvc 0.52.6", + "windows_i686_gnu 0.52.6", + "windows_i686_gnullvm 0.52.6", + "windows_i686_msvc 0.52.6", + "windows_x86_64_gnu 0.52.6", + "windows_x86_64_gnullvm 0.52.6", + "windows_x86_64_msvc 0.52.6", +] + +[[package]] +name = "windows-targets" +version = "0.53.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" +dependencies = [ + "windows-link", + "windows_aarch64_gnullvm 0.53.1", + "windows_aarch64_msvc 0.53.1", + "windows_i686_gnu 0.53.1", + "windows_i686_gnullvm 0.53.1", + "windows_i686_msvc 0.53.1", + "windows_x86_64_gnu 0.53.1", + "windows_x86_64_gnullvm 0.53.1", + "windows_x86_64_msvc 0.53.1", +] + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" + +[[package]] +name = "windows_i686_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" + +[[package]] +name = "windows_i686_gnu" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" + +[[package]] +name = "windows_i686_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" + +[[package]] +name = "windows_i686_msvc" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" + +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "x509-cert" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1301e935010a701ae5f8655edc0ad17c44bad3ac5ce8c39185f75453b720ae94" +dependencies = [ + "const-oid", + "der", + "spki", +] + +[[package]] +name = "x509-parser" +version = "0.18.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d43b0f71ce057da06bc0851b23ee24f3f86190b07203dd8f567d0b706a185202" +dependencies = [ + "asn1-rs 0.7.2", + "data-encoding", + "der-parser", + "lazy_static", + "nom", + "oid-registry", + "rusticata-macros", + "thiserror 2.0.18", + "time", +] + +[[package]] +name = "zerocopy" +version = "0.8.54" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7cbbc0a705a0fd05cc3676525980d2bf5a9bc4adac6d6475209a7887cf59d19" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.54" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2e817b7b52d0c7358d3246da9d69935ebb18116b2b102b4230dac079b4862f5" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "zeroize" +version = "1.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e" + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" + +[[package]] +name = "zstd" +version = "0.13.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a" +dependencies = [ + "zstd-safe", +] + +[[package]] +name = "zstd-safe" +version = "7.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d" +dependencies = [ + "zstd-sys", +] + +[[package]] +name = "zstd-sys" +version = "2.0.16+zstd.1.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" +dependencies = [ + "cc", + "pkg-config", +] diff --git a/rust/Cargo.toml b/rust/Cargo.toml new file mode 100644 index 0000000..27aa59f --- /dev/null +++ b/rust/Cargo.toml @@ -0,0 +1,27 @@ +[package] +name = "questdb_ha_sender" +version = "1.0.0" +edition = "2021" +description = "Parallel CSV replay sender for QuestDB over QWP (WebSocket/UDP) and ILP/HTTP, a Rust port of the Java ha_sender" + +[[bin]] +name = "csv_parallel_sender" +path = "src/main.rs" + +[dependencies] +# The Rust/C client, built locally from the c-questdb-client checkout. Default features +# already enable every sync sender (TCP, HTTP, QWP/UDP, QWP/WebSocket) plus webpki roots +# and the ring crypto backend; we add the egress reader (for the probe / query client), +# zstd result decompression, and insecure-skip-verify (for tls_verify=unsafe_off). +questdb-rs = { path = "../../../questdb/c-questdb-client/questdb-rs", features = [ + "sync-reader-ws", + "compression-zstd", + "insecure-skip-verify", +] } +clap = { version = "4", features = ["derive"] } +csv = "1" +flate2 = "1" +chrono = "0.4" + +[profile.release] +opt-level = 3 diff --git a/rust/README.md b/rust/README.md new file mode 100644 index 0000000..fa37d9e --- /dev/null +++ b/rust/README.md @@ -0,0 +1,133 @@ +# Rust HA sender + +A Rust port of the Java `CsvParallelSender`. It replays a CSV of trades in a loop across +N worker threads over one of three QuestDB transports, and (on QWP/WebSocket) runs a probe +that reports the latest ingested timestamp and the serving node's live role. + +It is built on the Rust/C client (`questdb-rs`) from +[`c-questdb-client`](https://github.com/questdb/c-questdb-client), which powers the C, C++, +and Python clients too. + +## Transports (`--protocol`) + +- `qwp` (default): **QWP over WebSocket**. Store-and-forward (un-acked frames spill to disk + and replay after an outage), transactional commit, and multi-host failover. A background + probe polls the latest ingested timestamp and the serving node's live role over a QWP + query client (the `Reader`). Use the server's WebSocket/HTTP port (`:9000`). +- `qwpudp`: **QWP over UDP**, fire-and-forget datagrams to `:9007`. Ingest-only, + unauthenticated, single-endpoint (no failover), best-effort (no ack, no store-and-forward). + Keep `--batch-size` small: UDP has no backpressure, so a large flush burst overflows the + server's receive buffer and drops datagrams wholesale. +- `ilp`: the legacy **ILP over HTTP** transport. + +## Build + +The client is a **path dependency** on a local `c-questdb-client` checkout, expected at +`../../../questdb/c-questdb-client/questdb-rs` (i.e. `~/prj/questdb/c-questdb-client` +alongside `~/prj/java/questdb_java_ha_sender`). Adjust the path in `Cargo.toml` if your +checkout is elsewhere. The QWP wire protocol must match the target server build. + +``` +cargo build --release +``` + +Requires a Rust toolchain new enough for the client's 2024 edition (1.85+; tested on 1.93). + +## Run + +``` +./target/release/csv_parallel_sender \ + --protocol qwp \ + --addrs localhost:9000 \ + --total-events 5000000 \ + --num-senders 4 \ + --delay-ms 0 \ + --batch-size 10000 \ + --batches-per-transaction 10 \ + --csv ../trades20250728.csv.gz +``` + +UDP (single host, small batch, no auth, no probe): + +``` +./target/release/csv_parallel_sender \ + --protocol qwpudp --addrs localhost:9007 \ + --total-events 100000 --num-senders 1 --batch-size 500 \ + --csv ../trades20250728.csv.gz +``` + +QWP batch errors and UDP loss are asynchronous or silent, so always confirm the result with +a server-side `SELECT count() FROM trades` after a run. + +## Flags + +| Flag | Default | Notes | +| --- | --- | --- | +| `--addrs` | `localhost:9000` | Comma-separated `host:port`. QWP/WebSocket + ILP use `:9000`; UDP uses `:9007` and accepts one host only. | +| `--token` | | Bearer token (QWP/WebSocket + ILP). UDP rejects auth. | +| `--username` / `--password` | | Basic auth (QWP/WebSocket + ILP). | +| `--total-events` | `1000000` | Rows across all workers. | +| `--delay-ms` | `50` | Per-row sleep. | +| `--num-senders` | `10` | Worker threads (one `Sender` each). | +| `--retry-timeout` | `360000` | `retry_timeout` (ILP) / `reconnect_max_duration_millis` (QWP). | +| `--csv` | `./trades20250728.csv.gz` | `.csv` or `.csv.gz`; needs `symbol,side,price,amount[,timestamp]`. | +| `--timestamp-from-file` | `false` | Use the CSV timestamp column instead of "now". | +| `--seconds-offset` | `0` | Shift each timestamp by N seconds. | +| `--protocol` | `qwp` | `qwp` \| `qwpudp` \| `ilp`. | +| `--sender-id` | `ha_sender` | Store-and-forward key base; each worker gets `-`. | +| `--store-forward-dir` | `/tmp/qdb-sf` | Spill directory (QWP/WebSocket). | +| `--batch-size` | `10000` | Rows per flush. QWP commits every `batch-size × batches-per-transaction`; UDP/ILP flush every `batch-size`. | +| `--batches-per-transaction` | `10` | QWP transaction size, in batches. | +| `--probe-interval-ms` | `1000` | Probe poll interval (QWP/WebSocket only; `0` disables). | +| `--enterprise` | `false` | Request durable acks (Enterprise only). | +| `--zone` | `eu-west-1` | Preferred zone for the query client / probe. | + +## Timestamps + +- **Single worker on a QWP transport** (`qwp` or `qwpudp`, `--num-senders 1`): each row is + stamped with the current microsecond client-side (monotonic, so no out-of-order), giving + distinct per-row timestamps. +- **ILP, or more than one worker**: rows use server-side `at_now()` (O3-safe across workers; + a whole batch shares one timestamp). +- `--timestamp-from-file` / `--seconds-offset` stamp each row explicitly from the CSV. + +## Probe (QWP/WebSocket only) + +A background thread runs `select timestamp from trades limit -1` each interval and prints +the latest ingested timestamp, then asks the serving node for its live role with +`switch status`: + +``` +[query client] connected, serving node=n1 role=PRIMARY zone=eu-west-1 cluster=... +[probe] latest trades timestamp = 2026-07-01T14:31:40.591545Z (raw=1782916300591545) served by role=PRIMARY node=n1 zone=eu-west-1 +``` + +`switch status` gives the authoritative live role (`current_role`, plus `target_role` while +a switch is in flight, shown as `(switching -> ROLE)`). It is Enterprise-only and needs +SYSTEM ADMIN; on OSS the probe falls back to the QWP handshake role, labelled +`(handshake role; live 'switch status' unavailable, may be stale)`. The query client uses +`target=any` (so reads fall back to a replica when no primary is available) and fails over +across the `--addrs` hosts automatically. + +## Differences from the Java client + +- **No auto-flush.** The Rust/C client deliberately has no auto-flush (it rejects every + `auto_flush*` key except `off`); batching is driven purely by explicit `flush()` calls. + This port reproduces the Java cadence by flushing at the same row boundaries, so behavior + matches; there is simply no background flush timer or byte cap. +- **No `--connect-timeout-ms`.** The Rust QWP transport exposes no single-connect timeout + knob (only the overall `reconnect_max_duration`), so that flag is omitted here. +- **Single bearer token only.** QWP (WebSocket/HTTP) auth is a single `token` (or + username/password). The split `x`/`y` key components only exist for legacy TCP-ILP ECDSA + auth, which this tool does not use. + +## Validated + +Smoke-tested against a local QuestDB (OSS) with all rows accounted for server-side: + +| Transport | Run | Result | +| --- | --- | --- | +| `ilp` | 2000 rows, 1 worker | 2000 / 2000 | +| `qwpudp` | 2000 rows, 1 worker | 2000 / 2000 | +| `qwp` | 5000 rows, 1 worker | 5000 / 5000, distinct per-row timestamps; probe reported live timestamps + role | +| `qwp` | 20000 rows, 4 workers | 20000 / 20000, one store-and-forward dir per worker | diff --git a/rust/src/main.rs b/rust/src/main.rs new file mode 100644 index 0000000..ec9f9b5 --- /dev/null +++ b/rust/src/main.rs @@ -0,0 +1,641 @@ +//! Parallel CSV replay sender for QuestDB, a Rust port of the Java `CsvParallelSender`. +//! +//! It replays a CSV of trades in a loop across N worker threads over one of three +//! transports, selected with `--protocol`: +//! +//! * `qwp` - QWP over WebSocket: store-and-forward (un-acked frames spill to disk and +//! replay after an outage), transactional commit, multi-host failover. A background probe +//! thread polls the latest ingested timestamp and the serving node's live role via the +//! query client (the `Reader`). +//! * `qwpudp` - QWP over UDP: fire-and-forget datagrams to :9007. Ingest-only, +//! unauthenticated, single-endpoint, best-effort (no ack, no failover). +//! * `ilp` - the legacy ILP/HTTP transport. +//! +//! Unlike the Java client, the Rust/C client has no auto-flush: batching is driven purely +//! by when we call `flush()`. We reproduce the Java cadence explicitly (see `run_worker`). + +use std::io::Read; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::thread; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use clap::Parser; +use questdb::egress::column::ColumnView; +use questdb::egress::reader::{BatchView, Reader}; +use questdb::egress::FailoverEvent; +use questdb::ingress::{Sender, TimestampMicros}; + +const PROBE_QUERY: &str = "select timestamp from trades limit -1"; +// Enterprise lifecycle status of whichever node the query client is connected to. Returns +// the LIVE role (current_role / target_role), unlike the QWP handshake SERVER_INFO which is +// only refreshed on a reconnect and so goes stale after an in-place primary<->replica switch. +const STATUS_QUERY: &str = "switch status"; + +#[derive(Parser, Debug)] +#[command( + name = "csv_parallel_sender", + about = "Parallel CSV replay sender for QuestDB (QWP WebSocket/UDP, ILP/HTTP)" +)] +struct Args { + /// Comma-separated host:port list. QWP/WebSocket + ILP use :9000; QWP/UDP uses :9007. + #[arg(long, default_value = "localhost:9000")] + addrs: String, + + /// Bearer token (QWP/WebSocket + ILP only; QWP/UDP is unauthenticated). + #[arg(long)] + token: Option, + + /// Basic-auth username (with --password). + #[arg(long)] + username: Option, + + /// Basic-auth password (with --username). + #[arg(long)] + password: Option, + + /// Total rows to send across all workers. + #[arg(long, default_value_t = 1_000_000)] + total_events: u64, + + /// Per-row sleep in milliseconds (0 = as fast as possible). + #[arg(long, default_value_t = 50)] + delay_ms: u64, + + /// Number of worker threads (each is its own Sender). + #[arg(long, default_value_t = 10)] + num_senders: usize, + + /// Overall "keep retrying" budget in ms: retry_timeout (ILP) / reconnect_max_duration (QWP). + #[arg(long, default_value_t = 360_000)] + retry_timeout: u64, + + /// CSV path (.csv or .csv.gz) with symbol,side,price,amount[,timestamp] columns. + #[arg(long, default_value = "./trades20250728.csv.gz")] + csv: String, + + /// Use the CSV's own timestamp column instead of server/client "now". + #[arg(long, default_value_t = false)] + timestamp_from_file: bool, + + /// Shift each timestamp by this many seconds (works with or without --timestamp-from-file). + #[arg(long, default_value_t = 0)] + seconds_offset: i64, + + /// Transport: qwp | qwpudp | ilp. + #[arg(long, default_value = "qwp")] + protocol: String, + + /// Store-and-forward key base (QWP/WebSocket). Each worker gets -. + #[arg(long, default_value = "ha_sender")] + sender_id: String, + + /// Store-and-forward spill directory (QWP/WebSocket). + #[arg(long, default_value = "/tmp/qdb-sf")] + store_forward_dir: String, + + /// Rows per flush. QWP commits every batch-size*batches-per-transaction; UDP flushes + /// every batch-size rows (keep small for UDP; large bursts overflow the receive buffer). + #[arg(long, default_value_t = 10_000)] + batch_size: u64, + + /// QWP/WebSocket transaction size, in batches. + #[arg(long, default_value_t = 10)] + batches_per_transaction: u64, + + /// Probe poll interval in ms (QWP/WebSocket only; 0 disables). + #[arg(long, default_value_t = 1000)] + probe_interval_ms: u64, + + /// Enterprise only: request durable acks (QWP/WebSocket). + #[arg(long, default_value_t = false)] + enterprise: bool, + + /// Preferred zone for the query client / probe (QWP/WebSocket). + #[arg(long, default_value = "eu-west-1")] + zone: String, +} + +impl Args { + fn has_token(&self) -> bool { + self.token.as_deref().is_some_and(|t| !t.is_empty()) + } + + fn has_basic(&self) -> bool { + self.username.as_deref().is_some_and(|u| !u.is_empty()) + && self.password.as_deref().is_some_and(|p| !p.is_empty()) + } + + fn tls(&self) -> bool { + self.has_token() || self.has_basic() + } + + fn addr_list(&self) -> Vec { + self.addrs + .split(',') + .map(|s| s.trim().to_string()) + .filter(|s| !s.is_empty()) + .collect() + } +} + +struct TradeRow { + symbol: String, + side: String, + price: f64, + amount: f64, + /// Microseconds since epoch, parsed at load time; only used with --timestamp-from-file. + ts_micros: i64, +} + +fn now_micros() -> i64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|d| d.as_micros() as i64) + .unwrap_or(0) +} + +fn main() { + let args = Args::parse(); + + if !matches!(args.protocol.as_str(), "qwp" | "qwpudp" | "ilp") { + eprintln!("--protocol must be 'qwp', 'qwpudp', or 'ilp', got: {}", args.protocol); + std::process::exit(2); + } + if args.num_senders == 0 { + eprintln!("--num-senders must be > 0"); + std::process::exit(2); + } + if args.total_events == 0 { + eprintln!("--total-events must be > 0"); + std::process::exit(2); + } + if args.batch_size == 0 { + eprintln!("--batch-size must be > 0"); + std::process::exit(2); + } + if args.batches_per_transaction == 0 { + eprintln!("--batches-per-transaction must be > 0"); + std::process::exit(2); + } + + let addrs = args.addr_list(); + if addrs.is_empty() { + eprintln!("--addrs must contain at least one host:port"); + std::process::exit(2); + } + // QWP/UDP is single-endpoint (no failover) and unauthenticated. Enforce both upfront so + // the failure is a clear message here, not a per-worker exception mid-run. + if args.protocol == "qwpudp" { + if addrs.len() > 1 { + eprintln!( + "--protocol qwpudp supports a single host:port only (UDP is connectionless, \ + there is no failover); got {} addresses", + addrs.len() + ); + std::process::exit(2); + } + if args.has_token() || args.username.is_some() || args.password.is_some() { + eprintln!( + "[warn] --protocol qwpudp is unauthenticated (UDP accepts any connection); \ + ignoring --token/--username/--password" + ); + } + } + + let rows = match load_csv(&args.csv, args.timestamp_from_file) { + Ok(rows) if !rows.is_empty() => rows, + Ok(_) => { + eprintln!("CSV has no data rows: {}", args.csv); + std::process::exit(2); + } + Err(e) => { + eprintln!("Failed to read CSV {}: {e}", args.csv); + std::process::exit(2); + } + }; + + match args.protocol.as_str() { + "qwp" => println!( + "Ingestion started. Protocol: qwp (WebSocket, sender-id={}, store-and-forward={}, \ + batch-size={}, batches-per-transaction={}, retry-timeout-ms={}) | addrs: {}", + args.sender_id, args.store_forward_dir, args.batch_size, args.batches_per_transaction, + args.retry_timeout, addrs.join(",") + ), + "qwpudp" => println!( + "Ingestion started. Protocol: qwpudp (QWP/UDP datagrams, ingest-only, unauthenticated, \ + no store-and-forward, no failover; query client disabled, batch-size={}) | addr: {}", + args.batch_size, addrs[0] + ), + _ => println!( + "Ingestion started. Protocol: ilp (HTTP, retry-timeout-ms={}) | addrs: {}", + args.retry_timeout, addrs.join(",") + ), + } + + let total_sent = AtomicU64::new(0); + let stop = AtomicBool::new(false); + let start = Instant::now(); + + let base = args.total_events / args.num_senders as u64; + let rem = args.total_events % args.num_senders as u64; + let mut worker_errors: Vec = Vec::new(); + + thread::scope(|scope| { + let args_ref = &args; + let rows_ref = &rows; + let sent_ref = &total_sent; + let stop_ref = &stop; + + // Progress reporter: rows/s (client-side, sent), once per second. + scope.spawn(move || { + let mut last = 0u64; + while !stop_ref.load(Ordering::Relaxed) { + thread::sleep(Duration::from_millis(1000)); + let now = sent_ref.load(Ordering::Relaxed); + println!("[progress] sent={now} rate={} rows/s", now - last); + last = now; + } + }); + + // Probe (QWP/WebSocket only): ingest-only transports (qwpudp, ilp) have no query path. + if args_ref.protocol == "qwp" && args_ref.probe_interval_ms > 0 { + scope.spawn(move || run_probe(args_ref, stop_ref)); + } + + let mut handles = Vec::with_capacity(args.num_senders); + for id in 0..args.num_senders { + let events = base + if (id as u64) < rem { 1 } else { 0 }; + handles.push(scope.spawn(move || run_worker(id, events, args_ref, rows_ref, sent_ref))); + } + + for (id, h) in handles.into_iter().enumerate() { + match h.join() { + Ok(Ok(())) => {} + Ok(Err(e)) => worker_errors.push(format!("Sender {id}: {e}")), + Err(_) => worker_errors.push(format!("Sender {id}: panicked")), + } + } + stop_ref.store(true, Ordering::Relaxed); + }); + + if !worker_errors.is_empty() { + for e in &worker_errors { + eprintln!("Worker failed: {e}"); + } + std::process::exit(1); + } + + let elapsed = start.elapsed().as_secs_f64(); + let rate = if elapsed > 0.0 { args.total_events as f64 / elapsed } else { 0.0 }; + println!( + "All workers completed. protocol={} events={} elapsed={:.3} s throughput={:.0} rows/s", + args.protocol, args.total_events, elapsed, rate + ); +} + +fn run_worker( + worker_id: usize, + total_events: u64, + args: &Args, + rows: &[TradeRow], + total_sent: &AtomicU64, +) -> questdb::Result<()> { + println!("Sender {worker_id} will send {total_events} events"); + + let is_qwp = args.protocol == "qwp"; + let is_udp = args.protocol == "qwpudp"; + // Single worker on a QWP transport: stamp each row with the current micros client-side. + // A single thread is monotonic (no O3) and this avoids QWP's per-batch atNow() stamping. + // ILP, or more than one worker, use atNow() (server-side, O3-safe). + let per_row_micros = (is_qwp || is_udp) && args.num_senders == 1; + // Flush cadence: qwp commits every batch-size*batches-per-transaction rows; qwpudp flushes + // every batch-size rows (bounds the datagram burst); ilp flushes every batch-size rows too + // (the Rust HTTP client has no auto-flush, so the buffer must be drained explicitly). + let commit_every: u64 = if is_qwp { + args.batch_size * args.batches_per_transaction + } else { + args.batch_size + }; + + let mut sender = Sender::from_conf(build_ingest_conf(args, worker_id))?; + let mut buffer = sender.new_buffer(); + let n = rows.len(); + let mut sent: u64 = 0; + + for i in 0..total_events { + let r = &rows[(i as usize) % n]; + buffer + .table("trades")? + .symbol("symbol", r.symbol.as_str())? + .symbol("side", r.side.as_str())? + .column_f64("price", r.price)? + .column_f64("amount", r.amount)? + .column_str("trade_id", format!("{worker_id}-{}", i + 1))?; + + if args.timestamp_from_file { + let ts = r.ts_micros + args.seconds_offset * 1_000_000; + buffer.at(TimestampMicros::new(ts))?; + } else if args.seconds_offset != 0 { + buffer.at(TimestampMicros::new(now_micros() + args.seconds_offset * 1_000_000))?; + } else if per_row_micros { + buffer.at(TimestampMicros::new(now_micros()))?; + } else { + buffer.at_now()?; + } + + sent += 1; + total_sent.fetch_add(1, Ordering::Relaxed); + + if sent.is_multiple_of(commit_every) { + sender.flush(&mut buffer)?; + } + if args.delay_ms > 0 { + thread::sleep(Duration::from_millis(args.delay_ms)); + } + } + + sender.flush(&mut buffer)?; + // QWP/WebSocket: drain any un-acked store-and-forward frames before closing. + if is_qwp { + sender.close_drain()?; + } + println!("Sender {worker_id} finished sending {sent} events"); + Ok(()) +} + +/// Build the ingestion connect string for the configured transport. +fn build_ingest_conf(args: &Args, worker_id: usize) -> String { + let addrs = args.addr_list(); + match args.protocol.as_str() { + "qwpudp" => format!("qwpudp::addr={};", addrs[0]), + "ilp" => { + let scheme = if args.tls() { "https" } else { "http" }; + let mut c = format!("{scheme}::addr={};", addrs.join(",")); + append_auth(&mut c, args); + if args.tls() { + c.push_str("tls_verify=unsafe_off;"); + } + c.push_str(&format!("retry_timeout={};", args.retry_timeout)); + c + } + _ => { + // qwp (WebSocket) + let scheme = if args.tls() { "qwpwss" } else { "qwpws" }; + let who = format!("{}-{worker_id}", args.sender_id); + let sf = format!("{}/{who}", args.store_forward_dir); + let _ = std::fs::create_dir_all(&sf); + let mut c = format!("{scheme}::addr={};", addrs.join(",")); + append_auth(&mut c, args); + if args.tls() { + c.push_str("tls_verify=unsafe_off;"); + } + c.push_str(&format!("sender_id={who};")); + c.push_str(&format!("sf_dir={sf};")); + c.push_str(&format!("reconnect_max_duration_millis={};", args.retry_timeout)); + c.push_str("reconnect_initial_backoff_millis=100;"); + c.push_str("reconnect_max_backoff_millis=5000;"); + if args.enterprise { + c.push_str("request_durable_ack=true;"); + } + c + } + } +} + +fn append_auth(c: &mut String, args: &Args) { + if args.has_token() { + c.push_str(&format!("token={};", args.token.as_ref().unwrap())); + } else if args.has_basic() { + c.push_str(&format!( + "username={};password={};", + args.username.as_ref().unwrap(), + args.password.as_ref().unwrap() + )); + } +} + +/// Connect string for the QWP query client (probe): same hosts/auth as the senders, +/// target=any (replica-fallback reads), failover on, zone bias. +fn build_reader_conf(args: &Args) -> String { + let addrs = args.addr_list(); + let scheme = if args.tls() { "wss" } else { "ws" }; + let mut c = format!("{scheme}::addr={};target=any;failover=true;", addrs.join(",")); + append_auth(&mut c, args); + if args.tls() { + c.push_str("tls_verify=unsafe_off;"); + } + if !args.zone.is_empty() { + c.push_str(&format!("zone={};", args.zone)); + } + c.push_str("compression=raw;"); + c +} + +/// Probe thread: every interval, poll the latest ingested timestamp and the serving node's +/// live role over a QWP query client. Independent of the senders; fails over automatically. +fn run_probe(args: &Args, stop: &AtomicBool) { + let mut reader = match Reader::from_conf(build_reader_conf(args)) { + Ok(r) => r, + Err(e) => { + eprintln!("[query client] connect failed: {e}"); + return; + } + }; + if let Some(si) = reader.server_info() { + println!( + "[query client] connected, serving node={} role={} zone={} cluster={}", + or_none(&si.node_id), + si.role.as_str(), + or_none(si.zone_id.as_deref().unwrap_or("")), + or_none(&si.cluster_id) + ); + } else { + println!("[query client] connected"); + } + + let mut was_down = false; + while !stop.load(Ordering::Relaxed) { + match read_latest_ts(&mut reader) { + Ok(latest) => { + if was_down { + println!("[query client] connection restored"); + was_down = false; + } + if let Some(micros) = latest { + // trades designated timestamp is microseconds by default. + let ts = chrono::DateTime::from_timestamp_micros(micros) + .map(|dt| dt.to_rfc3339_opts(chrono::SecondsFormat::Micros, true)) + .unwrap_or_else(|| micros.to_string()); + + // Ask the serving node for its live role. A status-query failure must not + // look like a connection loss, so it is swallowed (current stays None). + let (current, target) = read_switch_status(&mut reader).unwrap_or((None, None)); + + let (node, zone) = match reader.server_info() { + Some(si) => ( + or_none(&si.node_id), + or_none(si.zone_id.as_deref().unwrap_or("")), + ), + None => ("(none)".to_string(), "(none)".to_string()), + }; + + let served = match current { + Some(role) => { + let switching = target + .as_deref() + .is_some_and(|t| !t.eq_ignore_ascii_case(&role)); + let sw = match (&target, switching) { + (Some(t), true) => format!(" (switching -> {t})"), + _ => String::new(), + }; + format!(" served by role={role}{sw} node={node} zone={zone}") + } + None => { + let handshake = reader + .server_info() + .map(|si| si.role.as_str()) + .unwrap_or_else(|| "unknown".to_string()); + format!( + " served by role={handshake} node={node} zone={zone} \ + (handshake role; live 'switch status' unavailable, may be stale)" + ) + } + }; + println!("[probe] latest trades timestamp = {ts} (raw={micros}){served}"); + } + } + Err(e) => { + if !was_down { + println!("[query client] connection lost ({e}), will retry"); + was_down = true; + } + } + } + thread::sleep(Duration::from_millis(args.probe_interval_ms)); + } +} + +/// Run PROBE_QUERY and return the last non-null timestamp (micros), if any. +fn read_latest_ts(reader: &mut Reader) -> Result, questdb::egress::Error> { + let mut latest: Option = None; + let mut cursor = reader + .prepare(PROBE_QUERY) + .on_failover_reset(|ev: &FailoverEvent| { + println!( + "[query client] failed over {} -> {} (attempt {})", + ev.failed_addr, ev.new_addr, ev.attempts + ); + }) + .execute()?; + while let Some(view) = cursor.next_batch()? { + if let Ok(ColumnView::Timestamp(col)) = view.column(0) { + for r in 0..view.row_count() { + if !col.is_null(r) { + latest = Some(col.value(r)); + } + } + } + } + Ok(latest) +} + +/// Run STATUS_QUERY and pull current_role / target_role out of the result by column name. +fn read_switch_status( + reader: &mut Reader, +) -> Result<(Option, Option), questdb::egress::Error> { + let mut current: Option = None; + let mut target: Option = None; + let mut cursor = reader.prepare(STATUS_QUERY).execute()?; + while let Some(view) = cursor.next_batch()? { + let names: Vec = view + .schema() + .columns() + .iter() + .map(|c| c.name.to_lowercase()) + .collect(); + for r in 0..view.row_count() { + for (i, name) in names.iter().enumerate() { + if !name.contains("role") { + continue; + } + let value = col_string(&view, i, r); + if name.contains("current") { + current = value; + } else if name.contains("target") { + target = value; + } else if current.is_none() { + // A single unqualified "role" column (older servers) is the current one. + current = value; + } + } + } + } + Ok((current, target)) +} + +/// Read a string value from a Symbol or Varchar column; None for null or non-string types. +fn col_string(view: &BatchView, col: usize, row: usize) -> Option { + match view.column(col) { + Ok(ColumnView::Symbol(c)) => c.resolve(row).map(|s| s.to_string()), + Ok(ColumnView::Varchar(c)) => c.value(row).map(|s| s.to_string()), + _ => None, + } +} + +fn or_none(s: &str) -> String { + if s.is_empty() { + "(none)".to_string() + } else { + s.to_string() + } +} + +/// Load the CSV (optionally gzipped) into memory. Requires symbol,side,price,amount and, +/// when `need_timestamp`, a timestamp column parsed to microseconds since epoch. +fn load_csv(path: &str, need_timestamp: bool) -> Result, Box> { + let file = std::fs::File::open(path)?; + let reader: Box = if path.ends_with(".gz") { + Box::new(flate2::read::GzDecoder::new(file)) + } else { + Box::new(file) + }; + let mut rdr = csv::ReaderBuilder::new().has_headers(true).from_reader(reader); + + let headers = rdr.headers()?.clone(); + let idx = |name: &str| -> Result> { + headers + .iter() + .position(|h| h.trim() == name) + .ok_or_else(|| format!("CSV missing required column: {name}").into()) + }; + let i_symbol = idx("symbol")?; + let i_side = idx("side")?; + let i_price = idx("price")?; + let i_amount = idx("amount")?; + let i_ts = if need_timestamp { Some(idx("timestamp")?) } else { None }; + + let mut out = Vec::new(); + for rec in rdr.records() { + let rec = rec?; + if rec.is_empty() { + continue; + } + let ts_micros = match i_ts { + Some(i) => { + let raw = rec.get(i).unwrap_or("").trim(); + chrono::DateTime::parse_from_rfc3339(raw) + .map(|dt| dt.timestamp_micros()) + .map_err(|e| format!("bad timestamp {raw:?}: {e}"))? + } + None => 0, + }; + out.push(TradeRow { + symbol: rec.get(i_symbol).unwrap_or("").trim().to_string(), + side: rec.get(i_side).unwrap_or("").trim().to_string(), + price: rec.get(i_price).unwrap_or("").trim().parse()?, + amount: rec.get(i_amount).unwrap_or("").trim().parse()?, + ts_micros, + }); + } + Ok(out) +} From ba64afbdd7760c6e4b18511638ec05bec83efcb9 Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 10 Jul 2026 16:57:40 +0200 Subject: [PATCH 10/79] Document the QWP failover validation in the Rust README --- rust/README.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/rust/README.md b/rust/README.md index fa37d9e..bb8a2e2 100644 --- a/rust/README.md +++ b/rust/README.md @@ -131,3 +131,20 @@ Smoke-tested against a local QuestDB (OSS) with all rows accounted for server-si | `qwpudp` | 2000 rows, 1 worker | 2000 / 2000 | | `qwp` | 5000 rows, 1 worker | 5000 / 5000, distinct per-row timestamps; probe reported live timestamps + role | | `qwp` | 20000 rows, 4 workers | 20000 / 20000, one store-and-forward dir per worker | + +### QWP failover (primary crash) + +Two servers from the same build (primary on `:9100`, fallback on `:9000`), a single-worker +`qwp` run of 60000 rows with `--addrs :9100,:9000` and the probe active. The primary was +hard-killed mid-run: + +- The sender **completed all 60000 rows with exit 0**, despite its primary crashing. +- The primary took `trade_id` `0-1`…`0-20000`; the fallback took `0-20001`…`0-60000` + (40000 rows, all distinct). Combined: a **contiguous 1–60000 with zero loss and zero + duplication** — store-and-forward replayed the in-flight transaction to the new host and + the handoff landed exactly on a transaction boundary. +- The probe kept reporting across the crash and reconnected to the fallback transparently. + +Note: the probe's `on_failover_reset` callback fires only on *mid-query* failover; its +`limit -1` polls return in microseconds, so a between-poll reconnect is silent (no +`failed over` line). Comparing `reader.current_addr()` across polls would surface it. From ce514e0764dc42df398276ec7221c4b93d928e2a Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 10 Jul 2026 18:44:08 +0200 Subject: [PATCH 11/79] Add Python HA sender port and pandas/polars dataframe demo Port the Java/Rust CsvParallelSender to Python on the questdb client's QWP + egress build (the sm_qwp_dataframe_bench branch, built from source; the PyPI release is ILP-only). csv_parallel_sender.py mirrors the other ports: qwp (WebSocket) with per-worker store-and-forward and manual flush cadence, qwpudp, and ilp, plus a probe using the pooled query Client (select ... limit -1 + switch status role reporting, target=any, failover). dataframe_demo.py shows pandas ingestion (Sender.dataframe), polars ingestion (Client.dataframe), and egress to pandas/polars via Client.query(...).to_pandas()/.to_polars(). Validated against a local server: qwp/ilp/udp ingest, a primary-crash failover with gap-free store-and-forward handoff, and row-threshold auto-flush. README documents the build-from-source requirement and the Python-specific differences. --- python/README.md | 149 ++++++++++++++ python/csv_parallel_sender.py | 372 ++++++++++++++++++++++++++++++++++ python/dataframe_demo.py | 115 +++++++++++ 3 files changed, 636 insertions(+) create mode 100644 python/README.md create mode 100644 python/csv_parallel_sender.py create mode 100644 python/dataframe_demo.py diff --git a/python/README.md b/python/README.md new file mode 100644 index 0000000..ceb0f31 --- /dev/null +++ b/python/README.md @@ -0,0 +1,149 @@ +# Python HA sender + +A Python port of the Java/Rust `CsvParallelSender`, plus a pandas/polars dataframe demo. +It replays a CSV of trades in a loop across N worker threads over QuestDB's QWP +(WebSocket/UDP) and ILP/HTTP transports, with a probe that reads back the latest ingested +timestamp and the serving node's live role. + +Built on the QuestDB Python client (`questdb.ingress`), which wraps the same C/Rust client +as the other ports. + +## Important: the client must be built from source + +The QWP transports (`qwpws`/`qwpwss`/`qwpudp`) **and the query client / egress reader are not +in the PyPI release** (`questdb` 4.x on PyPI is ILP-only: `Protocol` has just +`Tcp/Tcps/Http/Https`, and there is no `Client`/`QueryResult`). They live on the +**`sm_qwp_dataframe_bench`** branch of +[`py-questdb-client`](https://github.com/questdb/py-questdb-client), which bumps the bundled +C client to the QWP + egress build. You must build that branch. + +Requirements: **Python ≥ 3.10** (the branch is 3.10+; the PyPI-era 3.9 venv will not work), +a Rust toolchain (`cargo`), and a C compiler. Build steps (a git worktree keeps your +`main` checkout untouched): + +```bash +cd ~/prj/python/py-questdb-client +git worktree add --detach /tmp/pyqdb_qwp origin/sm_qwp_dataframe_bench +cd /tmp/pyqdb_qwp +git submodule update --init # bundled c-questdb-client (QWP + egress) + +python3.12 -m venv /tmp/pyqdb_venv +/tmp/pyqdb_venv/bin/pip install -U pip "cython>=3.1.2" "setuptools>=80.9.0" numpy pandas polars pyarrow +/tmp/pyqdb_venv/bin/pip install -e . # compiles Cython + the Rust FFI +``` + +Verify: + +```python +from questdb.ingress import Protocol, Client # Client only exists on the QWP build +print([p for p in dir(Protocol) if not p.startswith('_')]) +# -> ['Http', 'Https', 'QwpUdp', 'QwpWs', 'QwpWss', 'Tcp', 'Tcps'] +``` + +Then run the scripts with that interpreter (`/tmp/pyqdb_venv/bin/python`). `pandas`, `polars`, +and `pyarrow` are only needed for the dataframe paths. + +## Transports (`--protocol`) + +- `qwp` (default): QWP over WebSocket. Per-worker store-and-forward, transactional commit, + multi-host failover, and the probe. Use the WebSocket/HTTP port (`:9000`). +- `qwpudp`: QWP over UDP, fire-and-forget datagrams to `:9007`. Ingest-only, + unauthenticated, single-endpoint (no failover), best-effort. **Especially lossy in + Python**: the Cython `row()` loop emits datagrams so fast that even a single worker can + overrun the server's UDP receive buffer (a full run can vanish). Pace it with `--delay-ms` + and keep `--batch-size` small, and treat results as best-effort. +- `ilp`: the legacy ILP/HTTP transport. + +## Run + +```bash +/tmp/pyqdb_venv/bin/python csv_parallel_sender.py \ + --protocol qwp \ + --addrs localhost:9000 \ + --total-events 100000 \ + --num-senders 4 \ + --delay-ms 0 \ + --batch-size 10000 \ + --batches-per-transaction 10 \ + --csv ../trades20250728.csv.gz +``` + +Flags mirror the Java/Rust ports: `--addrs`, `--token`/`--username`/`--password`, +`--total-events`, `--delay-ms`, `--num-senders`, `--retry-timeout`, `--csv`, +`--timestamp-from-file`, `--seconds-offset`, `--protocol`, `--sender-id`, +`--store-forward-dir`, `--batch-size`, `--batches-per-transaction`, `--probe-interval-ms`, +`--enterprise`, `--zone`. Confirm results server-side with `SELECT count() FROM trades`. + +## Probe (QWP/WebSocket only) + +For `qwp`, a background thread uses the pooled query client (`Client`) to run +`select timestamp from trades limit -1` each interval and prints the latest ingested +timestamp, then `switch status` for the serving node's live role. `switch status` is +Enterprise-only (needs SYSTEM ADMIN); on OSS the probe prints +`(live 'switch status' unavailable, ...)`. The client uses `target=any` (replica-fallback +reads) and fails over across the `--addrs` hosts. + +## Dataframe demo (pandas / polars ingestion + egress) + +`dataframe_demo.py` shows the columnar paths the row sender does not: + +```bash +/tmp/pyqdb_venv/bin/python dataframe_demo.py --addr localhost:9000 --csv ../trades20250728.csv.gz --rows 5000 +``` + +- **pandas ingestion** via `Sender.dataframe(df, ...)` — the numpy-backed pandas planner. +- **polars ingestion** via `Client.dataframe(df, ...)` — the pooled QWP Arrow-columnar path, + which takes polars / pyarrow / any Arrow C Stream source natively. (Numpy-backed pandas is + **not** accepted by `Client.dataframe` — it raises `UnsupportedDataFrameShapeError` — so + pandas goes through `Sender.dataframe`, or convert with `pyarrow.Table.from_pandas`.) +- **egress** via `Client.query(sql).to_pandas()` and `.to_polars()` (also `.to_arrow()` / + `iter_pandas()`; the result exposes the Arrow PyCapsule interface for zero-copy consumers). + +## Differences from the Java/Rust ports + +- **The client is pre-release / build-from-source** (see above) — not `pip install questdb`. +- **Ingestion and querying are split across two classes**: `Sender` (ingest) and `Client` + (pooled QWP: query + Arrow-columnar dataframe ingest). The probe uses `Client`. +- **Auto-flush exists** in the Python client (unlike the Rust client) and is verified to work + over QWP (row-threshold flush fires mid-stream; see Validated). This port sets + `auto_flush=off` and flushes at the same batch boundaries as the Java/Rust ports for + identical cadence. +- **Config-string scheme is `qwpws`/`qwpwss` for both the sender and the query client** + (the Rust reader used `ws`/`wss`; Python only accepts the `qwp*` schemes). +- **The probe has no handshake-role fallback** — because the Python binding does not expose it. + Java/Rust fall back to the QWP handshake role when `switch status` is unavailable. That role + exists in the underlying C client (`line_reader_server_info_role`/`_role_byte`) and the Rust + reader (`server_info()`), but the Python `Client` on this branch does not wrap it, so this + port prints "unavailable" on OSS. Likewise the query path's `on_failover_reset` callback is + wired internally in Cython but not surfaced to Python, so there is no user-facing failover + narration. +- Threads (like Java/Rust). Python's GIL is released during client I/O, but row-building is + Python-level, so row-by-row throughput is well below Java/Rust — use the dataframe path + for volume. + +## Validated + +Against a local QuestDB (OSS), all rows accounted for server-side: + +| What | Result | +| --- | --- | +| `qwp`, 3000 rows, 1 worker | 3000 / 3000, distinct per-row timestamps; probe reported live timestamps | +| `ilp`, 2000 rows, 2 workers | 2000 / 2000 | +| `qwpudp` (paced) | rows land; best-effort, lossy under fast bursts | +| `dataframe_demo` | pandas + polars ingest (10000 rows), egress to pandas and polars | +| auto-flush | `auto_flush_rows=1000`, 2500 rows, no manual flush: 2000 landed mid-stream (2 auto-flushes), close drained to 2500 | +| QWP failover (primary crash) | see below | + +### QWP failover (primary crash) + +Two servers from the same build (primary `:9100`, fallback `:9000`), a single-worker `qwp` +run of 40000 rows with `--addrs :9100,:9000` and the probe active. The primary was +hard-killed mid-run: + +- The sender **completed all 40000 rows with exit 0**, despite its primary crashing. +- The primary took `trade_id` seq 1–18500; the fallback took 18501–40000 (21500 rows, all + distinct). Combined: **contiguous 1–40000, zero loss, zero duplication** — store-and-forward + replayed the in-flight transaction to the new host, handing off exactly on a transaction + boundary. +- The probe failed over from the primary to the fallback (visible as a second + `connection lost -> restored` cycle once post-failover data committed on the new host). diff --git a/python/csv_parallel_sender.py b/python/csv_parallel_sender.py new file mode 100644 index 0000000..43e6ae2 --- /dev/null +++ b/python/csv_parallel_sender.py @@ -0,0 +1,372 @@ +#!/usr/bin/env python3 +"""Parallel CSV replay sender for QuestDB, a Python port of the Java/Rust ha_sender. + +Replays a CSV of trades in a loop across N worker threads over one of three transports +(``--protocol``): + + * ``qwp`` - QWP over WebSocket: store-and-forward (un-acked frames spill to disk and + replay after an outage), transactional commit, multi-host failover. A + background probe polls the latest ingested timestamp and the serving + node's live role over a QWP query client (``Client``). + * ``qwpudp`` - QWP over UDP: fire-and-forget datagrams to :9007. Ingest-only, + unauthenticated, single-endpoint, best-effort (no ack, no failover). + * ``ilp`` - the legacy ILP/HTTP transport. + +Requires the QWP/egress build of the questdb Python client (see README.md); the PyPI +release does not expose QWP or the query client. +""" + +import argparse +import gzip +import csv as csvmod +import os +import sys +import threading +import time +from datetime import datetime, timezone + +from questdb.ingress import Sender, Client, TimestampNanos, ServerTimestamp + +PROBE_QUERY = "select timestamp from trades limit -1" +# Enterprise lifecycle status of whichever node the query client is connected to. Returns +# the LIVE role (current_role / target_role), unlike the QWP handshake which only refreshes +# on reconnect and so goes stale after an in-place primary<->replica switch. +STATUS_QUERY = "switch status" + + +def parse_args(argv): + p = argparse.ArgumentParser(description="Parallel CSV replay sender for QuestDB") + p.add_argument("--addrs", default="localhost:9000", + help="Comma-separated host:port. QWP/WebSocket + ILP use :9000; UDP uses :9007.") + p.add_argument("--token", default=None, help="Bearer token (QWP/WebSocket + ILP only).") + p.add_argument("--username", default=None, help="Basic-auth username.") + p.add_argument("--password", default=None, help="Basic-auth password.") + p.add_argument("--total-events", type=int, default=1_000_000) + p.add_argument("--delay-ms", type=int, default=50) + p.add_argument("--num-senders", type=int, default=10) + p.add_argument("--retry-timeout", type=int, default=360_000, + help="retry_timeout (ILP) / reconnect_max_duration_millis (QWP), in ms.") + p.add_argument("--csv", default="./trades20250728.csv.gz") + p.add_argument("--timestamp-from-file", action="store_true") + p.add_argument("--seconds-offset", type=int, default=0) + p.add_argument("--protocol", default="qwp", choices=["qwp", "qwpudp", "ilp"]) + p.add_argument("--sender-id", default="ha_sender") + p.add_argument("--store-forward-dir", default="/tmp/qdb-sf") + p.add_argument("--batch-size", type=int, default=10_000) + p.add_argument("--batches-per-transaction", type=int, default=10) + p.add_argument("--probe-interval-ms", type=int, default=1000) + p.add_argument("--enterprise", action="store_true") + p.add_argument("--zone", default="eu-west-1") + return p.parse_args(argv) + + +def addr_list(args): + return [a.strip() for a in args.addrs.split(",") if a.strip()] + + +def has_token(args): + return bool(args.token) + + +def has_basic(args): + return bool(args.username) and bool(args.password) + + +def tls(args): + return has_token(args) or has_basic(args) + + +def _append_auth(parts, args): + if has_token(args): + parts.append(f"token={args.token};") + elif has_basic(args): + parts.append(f"username={args.username};password={args.password};") + + +def build_ingest_conf(args, worker_id): + """Ingestion connect string for the configured transport (same keys as the Rust port).""" + addrs = addr_list(args) + if args.protocol == "qwpudp": + return f"qwpudp::addr={addrs[0]};auto_flush=off;" + + if args.protocol == "ilp": + scheme = "https" if tls(args) else "http" + parts = [f"{scheme}::addr={','.join(addrs)};", "auto_flush=off;"] + _append_auth(parts, args) + if tls(args): + parts.append("tls_verify=unsafe_off;") + parts.append(f"retry_timeout={args.retry_timeout};") + return "".join(parts) + + # qwp (WebSocket) + scheme = "qwpwss" if tls(args) else "qwpws" + who = f"{args.sender_id}-{worker_id}" + sf = os.path.join(args.store_forward_dir, who) + os.makedirs(sf, exist_ok=True) + parts = [f"{scheme}::addr={','.join(addrs)};", "auto_flush=off;"] + _append_auth(parts, args) + if tls(args): + parts.append("tls_verify=unsafe_off;") + parts.append(f"sender_id={who};") + parts.append(f"sf_dir={sf};") + parts.append(f"reconnect_max_duration_millis={args.retry_timeout};") + parts.append("reconnect_initial_backoff_millis=100;") + parts.append("reconnect_max_backoff_millis=5000;") + if args.enterprise: + parts.append("request_durable_ack=true;") + return "".join(parts) + + +def build_client_conf(args): + """Connect string for the QWP query client (probe): target=any (replica-fallback + reads), failover on, zone bias. Same hosts/auth as the senders.""" + addrs = addr_list(args) + scheme = "qwpwss" if tls(args) else "qwpws" + parts = [f"{scheme}::addr={','.join(addrs)};", "target=any;", "failover=true;"] + _append_auth(parts, args) + if tls(args): + parts.append("tls_verify=unsafe_off;") + if args.zone: + parts.append(f"zone={args.zone};") + return "".join(parts) + + +def load_csv(path, need_timestamp): + """Load the CSV (optionally gzipped). Returns a list of (symbol, side, price, amount, + ts_nanos) tuples; ts_nanos is 0 unless need_timestamp.""" + opener = gzip.open if path.endswith(".gz") else open + rows = [] + with opener(path, "rt", encoding="utf-8", newline="") as fh: + reader = csvmod.reader(fh) + header = next(reader, None) + if header is None: + return rows + idx = {name.strip(): i for i, name in enumerate(header)} + for col in ("symbol", "side", "price", "amount"): + if col not in idx: + raise ValueError(f"CSV missing required column: {col}") + if need_timestamp and "timestamp" not in idx: + raise ValueError("CSV missing required column: timestamp") + for rec in reader: + if not rec: + continue + ts_nanos = 0 + if need_timestamp: + raw = rec[idx["timestamp"]].strip() + # ISO-8601, e.g. 2025-07-28T12:34:56.789012Z + dt = datetime.fromisoformat(raw.replace("Z", "+00:00")) + ts_nanos = int(dt.timestamp() * 1_000_000_000) + rows.append(( + rec[idx["symbol"]].strip(), + rec[idx["side"]].strip(), + float(rec[idx["price"]].strip()), + float(rec[idx["amount"]].strip()), + ts_nanos, + )) + return rows + + +def run_worker(worker_id, total_events, args, rows, counts): + print(f"Sender {worker_id} will send {total_events} events") + is_qwp = args.protocol == "qwp" + is_udp = args.protocol == "qwpudp" + # Single worker on a QWP transport stamps rows client-side (monotonic, no O3); ILP or + # more than one worker use server-side timestamps (ServerTimestamp). + per_row_ts = (is_qwp or is_udp) and args.num_senders == 1 + # Flush cadence: qwp commits every batch-size*batches-per-transaction; qwpudp/ilp flush + # every batch-size rows (auto_flush is off, so the buffer is drained explicitly). + commit_every = args.batch_size * args.batches_per_transaction if is_qwp else args.batch_size + + conf = build_ingest_conf(args, worker_id) + n = len(rows) + sent = 0 + with Sender.from_conf(conf) as sender: + for i in range(total_events): + symbol, side, price, amount, ts_nanos = rows[i % n] + if args.timestamp_from_file: + at = TimestampNanos(ts_nanos + args.seconds_offset * 1_000_000_000) + elif args.seconds_offset != 0: + at = TimestampNanos(time.time_ns() + args.seconds_offset * 1_000_000_000) + elif per_row_ts: + at = TimestampNanos.now() + else: + at = ServerTimestamp + sender.row( + "trades", + symbols={"symbol": symbol, "side": side}, + columns={"price": price, "amount": amount, "trade_id": f"{worker_id}-{i + 1}"}, + at=at, + ) + sent += 1 + counts[worker_id] = sent + if sent % commit_every == 0: + sender.flush() + if args.delay_ms > 0: + time.sleep(args.delay_ms / 1000.0) + sender.flush() + # QWP/WebSocket: drain any un-acked store-and-forward frames before closing. + if is_qwp: + sender.close_drain() + print(f"Sender {worker_id} finished sending {sent} events") + + +def _switch_status_roles(client): + """Return (current_role, target_role) from `switch status`, or (None, None).""" + try: + df = client.query(STATUS_QUERY).to_pandas() + except Exception: + return None, None + if len(df) == 0: + return None, None + current = target = None + last = df.iloc[-1] + for col in df.columns: + lc = col.lower() + if "role" not in lc: + continue + val = last[col] + if "current" in lc: + current = val + elif "target" in lc: + target = val + elif current is None: + current = val + return current, target + + +def run_probe(args, stop): + conf = build_client_conf(args) + try: + client = Client.from_conf(conf) + except Exception as e: + print(f"[query client] connect failed: {e}") + return + print("[query client] connected") + was_down = False + try: + while not stop.is_set(): + try: + df = client.query(PROBE_QUERY).to_pandas() + if was_down: + print("[query client] connection restored") + was_down = False + if len(df): + latest = df.iloc[-1, 0] + current, target = _switch_status_roles(client) + if current is not None: + switching = target is not None and str(target).lower() != str(current).lower() + sw = f" (switching -> {target})" if switching else "" + served = f" served by role={current}{sw}" + else: + served = " (live 'switch status' unavailable, e.g. OSS or missing SYSTEM ADMIN)" + print(f"[probe] latest trades timestamp = {latest}{served}") + except Exception as e: + if not was_down: + print(f"[query client] connection lost ({e}), will retry") + was_down = True + stop.wait(args.probe_interval_ms / 1000.0) + finally: + client.close() + + +def main(argv): + args = parse_args(argv) + + for name, val in (("--num-senders", args.num_senders), ("--total-events", args.total_events), + ("--batch-size", args.batch_size), + ("--batches-per-transaction", args.batches_per_transaction)): + if val <= 0: + print(f"{name} must be > 0", file=sys.stderr) + return 2 + + addrs = addr_list(args) + if not addrs: + print("--addrs must contain at least one host:port", file=sys.stderr) + return 2 + if args.protocol == "qwpudp": + if len(addrs) > 1: + print("--protocol qwpudp supports a single host:port only (UDP is connectionless, " + f"there is no failover); got {len(addrs)} addresses", file=sys.stderr) + return 2 + if args.token or args.username or args.password: + print("[warn] --protocol qwpudp is unauthenticated (UDP accepts any connection); " + "ignoring --token/--username/--password", file=sys.stderr) + + if not os.path.exists(args.csv): + print(f"CSV file not found: {args.csv}", file=sys.stderr) + return 2 + rows = load_csv(args.csv, args.timestamp_from_file) + if not rows: + print("CSV has no data rows.", file=sys.stderr) + return 2 + + if args.protocol == "qwp": + print(f"Ingestion started. Protocol: qwp (WebSocket, sender-id={args.sender_id}, " + f"store-and-forward={args.store_forward_dir}, batch-size={args.batch_size}, " + f"batches-per-transaction={args.batches_per_transaction}, " + f"retry-timeout-ms={args.retry_timeout}) | addrs: {','.join(addrs)}") + elif args.protocol == "qwpudp": + print(f"Ingestion started. Protocol: qwpudp (QWP/UDP datagrams, ingest-only, " + f"unauthenticated, no store-and-forward, no failover; query client disabled, " + f"batch-size={args.batch_size}) | addr: {addrs[0]}") + else: + print(f"Ingestion started. Protocol: ilp (HTTP, retry-timeout-ms={args.retry_timeout}) " + f"| addrs: {','.join(addrs)}") + + counts = [0] * args.num_senders + stop = threading.Event() + start = time.monotonic() + + def reporter(): + last = 0 + while not stop.is_set(): + stop.wait(1.0) + now = sum(counts) + print(f"[progress] sent={now} rate={now - last} rows/s") + last = now + + threads = [] + rep = threading.Thread(target=reporter, daemon=True) + rep.start() + + probe_thread = None + if args.protocol == "qwp" and args.probe_interval_ms > 0: + probe_thread = threading.Thread(target=run_probe, args=(args, stop), daemon=True) + probe_thread.start() + + base = args.total_events // args.num_senders + rem = args.total_events % args.num_senders + errors = [] + + def worker_wrapper(wid, events): + try: + run_worker(wid, events, args, rows, counts) + except Exception as e: # noqa: BLE001 + errors.append(f"Sender {wid}: {e}") + + for wid in range(args.num_senders): + events = base + (1 if wid < rem else 0) + t = threading.Thread(target=worker_wrapper, args=(wid, events)) + t.start() + threads.append(t) + + for t in threads: + t.join() + stop.set() + if probe_thread: + probe_thread.join(timeout=2.0) + + if errors: + for e in errors: + print(f"Worker failed: {e}", file=sys.stderr) + return 1 + + elapsed = time.monotonic() - start + rate = args.total_events / elapsed if elapsed > 0 else 0 + print(f"All workers completed. protocol={args.protocol} events={args.total_events} " + f"elapsed={elapsed:.3f} s throughput={rate:,.0f} rows/s") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/python/dataframe_demo.py b/python/dataframe_demo.py new file mode 100644 index 0000000..163958d --- /dev/null +++ b/python/dataframe_demo.py @@ -0,0 +1,115 @@ +#!/usr/bin/env python3 +"""Pandas and polars ingestion + egress round-trip over QWP/WebSocket. + +Showcases the columnar/dataframe capabilities the row-by-row sender does not: + + * pandas ingestion via ``Sender.dataframe`` (the numpy-backed pandas planner). + * polars ingestion via ``Client.dataframe`` (the pooled QWP Arrow-columnar path; + it takes polars / pyarrow / any Arrow C Stream source natively — numpy-backed + pandas is not accepted there, so pandas goes through ``Sender.dataframe``). + * egress via ``Client.query(...).to_pandas()`` and ``.to_polars()``. + +Usage: + python dataframe_demo.py [--addr localhost:9000] [--csv ../trades20250728.csv.gz] [--rows 5000] + +Requires the QWP/egress build of the questdb client (see README.md). +""" + +import argparse +import sys +import time +import urllib.parse +import urllib.request + +import pandas as pd +import polars as pl + +from questdb.ingress import Sender, Client + +TABLE = "df_demo" + + +def exec_sql(addr, sql): + """Run a statement over the HTTP /exec endpoint (same host, HTTP port).""" + url = f"http://{addr}/exec?" + urllib.parse.urlencode({"query": sql}) + with urllib.request.urlopen(url, timeout=10) as resp: + resp.read() + + +def wait_for_count(client, table, at_least, timeout_s=30): + deadline = time.monotonic() + timeout_s + while time.monotonic() < deadline: + try: + n = int(client.query(f"select count() from {table}").to_pandas().iloc[0, 0]) + if n >= at_least: + return n + except Exception: + pass + time.sleep(0.5) + return -1 + + +def load_pandas(csv_path, rows): + """Load the CSV into a pandas DataFrame with dtypes the ingestion path accepts + (object strings, float64, tz-aware nanosecond timestamps).""" + raw = pd.read_csv(csv_path, nrows=rows) + return pd.DataFrame({ + "symbol": raw["symbol"].astype("object"), + "side": raw["side"].astype("object"), + "price": raw["price"].astype("float64"), + "amount": raw["amount"].astype("float64"), + "timestamp": pd.to_datetime(raw["timestamp"], utc=True).dt.as_unit("ns"), + }) + + +def main(argv): + ap = argparse.ArgumentParser(description="Pandas/polars ingestion + egress demo") + ap.add_argument("--addr", default="localhost:9000") + ap.add_argument("--csv", default="../trades20250728.csv.gz") + ap.add_argument("--rows", type=int, default=5000) + args = ap.parse_args(argv) + + pdf = load_pandas(args.csv, args.rows) + print(f"loaded pandas DataFrame: {pdf.shape[0]} rows\n{pdf.dtypes}\n") + pldf = pl.from_pandas(pdf) + print(f"built polars DataFrame: {pldf.height} rows\n") + + exec_sql(args.addr, f"drop table if exists {TABLE}") + + # --- INGESTION --- + # pandas -> Sender.dataframe (numpy-backed pandas planner) over QWP/WebSocket. + with Sender.from_conf(f"qwpws::addr={args.addr};auto_flush=off;") as sender: + sender.dataframe(pdf, table_name=TABLE, symbols=["symbol", "side"], at="timestamp") + sender.flush() + print(f"[ingest] pandas via Sender.dataframe: {pdf.shape[0]} rows") + + with Client.from_conf(f"qwpws::addr={args.addr};") as client: + # polars -> Client.dataframe (Arrow-columnar path, polars is native). + client.dataframe(pldf, table_name=TABLE, symbols=["symbol", "side"], at="timestamp") + print(f"[ingest] polars via Client.dataframe: {pldf.height} rows") + + expected = pdf.shape[0] + pldf.height + n = wait_for_count(client, TABLE, expected) + print(f"[ingest] server applied {n} rows (expected {expected})\n") + + # --- EGRESS --- + pd_out = client.query( + f"select symbol, count() n, round(avg(price), 2) avg_price " + f"from {TABLE} order by n desc limit 5" + ).to_pandas() + print("[egress] Client.query(...).to_pandas():") + print(pd_out.to_string(index=False)) + print() + + pl_out = client.query( + f"select side, count() n, round(sum(amount), 4) total_amount " + f"from {TABLE} group by side order by side" + ).to_polars() + print("[egress] Client.query(...).to_polars():") + print(pl_out) + + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) From f00a79340b77c70fdb08aa111550e321bb2adec1 Mon Sep 17 00:00:00 2001 From: javier Date: Mon, 13 Jul 2026 11:04:07 +0200 Subject: [PATCH 12/79] Add C++ HA sender port Port the Java/Rust/Python CsvParallelSender to C++ on the c-questdb-client C++ headers (line_sender.hpp ingestion, line_reader.hpp egress query client). Mirrors the other ports: qwp (WebSocket) with per-worker store-and-forward and manual flush cadence, qwpudp, and ilp, plus a probe (latest ingested timestamp + live serving role via switch status with a handshake-role fallback, target=any, failover). CMake build via Corrosion (cargo FFI), reads gzipped CSV via zlib. Validated against a local server: qwp/ilp/udp ingest and a primary-crash failover with a gap-free store-and-forward handoff. --- c/.gitignore | 1 + c/CMakeLists.txt | 30 +++ c/README.md | 99 +++++++ c/csv_parallel_sender.cpp | 531 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 661 insertions(+) create mode 100644 c/.gitignore create mode 100644 c/CMakeLists.txt create mode 100644 c/README.md create mode 100644 c/csv_parallel_sender.cpp diff --git a/c/.gitignore b/c/.gitignore new file mode 100644 index 0000000..796b96d --- /dev/null +++ b/c/.gitignore @@ -0,0 +1 @@ +/build diff --git a/c/CMakeLists.txt b/c/CMakeLists.txt new file mode 100644 index 0000000..72239d2 --- /dev/null +++ b/c/CMakeLists.txt @@ -0,0 +1,30 @@ +cmake_minimum_required(VERSION 3.15) +project(questdb_ha_sender_cpp CXX) + +set(CMAKE_CXX_STANDARD 17) +set(CMAKE_CXX_STANDARD_REQUIRED ON) + +if(NOT CMAKE_BUILD_TYPE) + set(CMAKE_BUILD_TYPE Release) +endif() + +# Path to a c-questdb-client checkout (v7.0.0+ / main, with QWP + egress). +# Override with -DQUESTDB_CLIENT_DIR=/path/to/c-questdb-client. +set(QUESTDB_CLIENT_DIR "/Users/j/prj/questdb/c-questdb-client" + CACHE PATH "c-questdb-client checkout") + +# The egress reader (query client / probe) needs the reader surface, and the probe +# uses tls_verify=unsafe_off; enable both cargo features via the client's CMake options. +set(QUESTDB_ENABLE_READER ON CACHE BOOL "" FORCE) +set(QUESTDB_ENABLE_INSECURE_SKIP_VERIFY ON CACHE BOOL "" FORCE) + +add_subdirectory(${QUESTDB_CLIENT_DIR} ${CMAKE_BINARY_DIR}/questdb_client_build) + +find_package(ZLIB REQUIRED) + +add_executable(csv_parallel_sender csv_parallel_sender.cpp) +target_include_directories(csv_parallel_sender PRIVATE ${QUESTDB_CLIENT_DIR}/include) +target_link_libraries(csv_parallel_sender PRIVATE questdb_client ZLIB::ZLIB) + +find_package(Threads REQUIRED) +target_link_libraries(csv_parallel_sender PRIVATE Threads::Threads) diff --git a/c/README.md b/c/README.md new file mode 100644 index 0000000..fe091b7 --- /dev/null +++ b/c/README.md @@ -0,0 +1,99 @@ +# C++ HA sender + +A C++ port of the Java/Rust/Python `CsvParallelSender`. It replays a CSV of trades in a loop +across N worker threads over QuestDB's QWP (WebSocket/UDP) and ILP/HTTP transports, with a +probe that reads back the latest ingested timestamp and the serving node's live role. + +Built on the C/C++ client headers from +[`c-questdb-client`](https://github.com/questdb/c-questdb-client) (`line_sender.hpp` for +ingestion, `line_reader.hpp` for the egress query client) — the same C/Rust core the Rust and +Python ports use. C++17. + +## Build + +Requires a `c-questdb-client` checkout (v7.0.0+ / `main`, which has QWP + the egress reader), a +Rust toolchain (`cargo`, pulled in via the bundled Corrosion), a C++17 compiler, and zlib. The +project builds the client as a CMake subdirectory (Corrosion compiles the Rust FFI), so point it +at your checkout: + +```bash +cmake -B build -S . -DQUESTDB_CLIENT_DIR=/path/to/c-questdb-client +cmake --build build -j +``` + +`QUESTDB_CLIENT_DIR` defaults to `~/prj/questdb/c-questdb-client`. The CMake forces +`QUESTDB_ENABLE_READER=ON` (the query client / probe) and `QUESTDB_ENABLE_INSECURE_SKIP_VERIFY=ON` +(`tls_verify=unsafe_off`). The binary is `build/csv_parallel_sender`. The QWP wire protocol must +match the target server build. + +## Transports (`--protocol`) + +- `qwp` (default): QWP over WebSocket. Per-worker store-and-forward, transactional commit, + multi-host failover, and the probe. Use the WebSocket/HTTP port (`:9000`). +- `qwpudp`: QWP over UDP, fire-and-forget datagrams to `:9007`. Ingest-only, unauthenticated, + single-endpoint (no failover), best-effort. +- `ilp`: the legacy ILP/HTTP transport. + +## Run + +```bash +./build/csv_parallel_sender \ + --protocol qwp \ + --addrs localhost:9000 \ + --total-events 100000 \ + --num-senders 4 \ + --delay-ms 0 \ + --batch-size 10000 \ + --batches-per-transaction 10 \ + --csv ../trades20250728.csv.gz +``` + +Flags mirror the other ports: `--addrs`, `--token`/`--username`/`--password`, `--total-events`, +`--delay-ms`, `--num-senders`, `--retry-timeout`, `--csv`, `--timestamp-from-file`, +`--seconds-offset`, `--protocol`, `--sender-id`, `--store-forward-dir`, `--batch-size`, +`--batches-per-transaction`, `--probe-interval-ms`, `--enterprise`, `--zone`. +(`--timestamp-from-file` and `--enterprise` are presence flags.) Confirm results server-side with +`SELECT count() FROM trades`. + +## Probe (QWP/WebSocket only) + +For `qwp`, a background thread uses the egress `reader` to run `select timestamp from trades +limit -1` each interval and prints the latest ingested timestamp, then `switch status` for the +serving node's live role. Unlike the Python port, the C++ `reader` exposes `server_info()` +(role/node/cluster), so the probe **does** fall back to the QWP handshake role when +`switch status` is unavailable (OSS / missing SYSTEM ADMIN), labelled `(handshake role; ...)` — +matching Java/Rust. The reader uses `target=any` (replica-fallback reads) and fails over across +the `--addrs` hosts. + +## Differences from the other ports + +- **No auto-flush** (like Rust; unlike Java/Python) — the C/C++ client has none, so batching is + driven purely by explicit `flush()` at the same boundaries as the other ports. +- **Reader scheme is `ws`/`wss`** (like Rust; Python required `qwpws`). +- Row values from the CSV are validated UTF-8 via `utf8_view` at build time (throws on invalid + input); fixed column names use the `_cn` / `_tn` literals. +- Threads via `std::thread`, one `line_sender` per worker (unique `sender_id` + spill dir). + +## Validated + +Against a local QuestDB (OSS), all rows accounted for server-side: + +| What | Result | +| --- | --- | +| `qwp`, 4000 rows, 1 worker | 4000 / 4000, distinct per-row timestamps; probe reported live timestamps + handshake role fallback | +| `ilp`, 2000 rows, 2 workers | 2000 / 2000 | +| `qwpudp` (paced) | rows land; best-effort, lossy under fast bursts | +| QWP failover (primary crash) | see below | + +### QWP failover (primary crash) + +Two servers from the same build (primary `:9100`, fallback `:9000`), a single-worker `qwp` +run of 40000 rows with `--addrs :9100,:9000` and the probe active. The primary was +hard-killed mid-run: + +- The sender **completed all 40000 rows with exit 0**, despite its primary crashing. +- The primary took `trade_id` seq 1–7500; the fallback took 7501–40000 (32500 rows, all + distinct). Combined: **contiguous 1–40000, zero loss, zero duplication** — store-and-forward + replayed the in-flight transaction to the new host, handing off exactly on a transaction + boundary. +- The probe failed over from the primary to the fallback. diff --git a/c/csv_parallel_sender.cpp b/c/csv_parallel_sender.cpp new file mode 100644 index 0000000..6901b57 --- /dev/null +++ b/c/csv_parallel_sender.cpp @@ -0,0 +1,531 @@ +// Parallel CSV replay sender for QuestDB, a C++ port of the Java/Rust ha_sender. +// +// Replays a CSV of trades in a loop across N worker threads over one of three transports +// (--protocol): +// * qwp - QWP over WebSocket: store-and-forward (un-acked frames spill to disk and +// replay after an outage), transactional commit, multi-host failover. A probe +// thread polls the latest ingested timestamp and the serving node's live role +// over a QWP query client (the egress reader). +// * qwpudp - QWP over UDP: fire-and-forget datagrams to :9007. Ingest-only, +// unauthenticated, single-endpoint, best-effort. +// * ilp - the legacy ILP/HTTP transport. +// +// Like the Rust client, the C/C++ client has no auto-flush: batching is driven by explicit +// flush() calls at the same boundaries as the Java/Rust ports. + +#include +#include + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace qi = questdb::ingress; +namespace eg = questdb::egress; +using namespace questdb::ingress::literals; + +namespace { + +constexpr std::string_view PROBE_QUERY = "select timestamp from trades limit -1"; +// Live lifecycle role of whichever node the query client is connected to (current_role / +// target_role), unlike the QWP handshake which only refreshes on reconnect. +constexpr std::string_view STATUS_QUERY = "switch status"; + +struct Args { + std::string addrs = "localhost:9000"; + std::string token, username, password; + uint64_t total_events = 1'000'000; + uint64_t delay_ms = 50; + unsigned num_senders = 10; + uint64_t retry_timeout = 360'000; + std::string csv = "./trades20250728.csv.gz"; + bool timestamp_from_file = false; + int64_t seconds_offset = 0; + std::string protocol = "qwp"; + std::string sender_id = "ha_sender"; + std::string store_forward_dir = "/tmp/qdb-sf"; + uint64_t batch_size = 10'000; + uint64_t batches_per_transaction = 10; + uint64_t probe_interval_ms = 1000; + bool enterprise = false; + std::string zone = "eu-west-1"; +}; + +struct TradeRow { + std::string symbol; + std::string side; + double price; + double amount; + int64_t ts_nanos; // only used with --timestamp-from-file +}; + +int64_t now_nanos() { + return std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()) + .count(); +} + +std::vector split_addrs(const std::string& csv) { + std::vector out; + std::stringstream ss(csv); + std::string item; + while (std::getline(ss, item, ',')) { + size_t b = item.find_first_not_of(" \t"); + size_t e = item.find_last_not_of(" \t"); + if (b != std::string::npos) out.push_back(item.substr(b, e - b + 1)); + } + return out; +} + +bool has_token(const Args& a) { return !a.token.empty(); } +bool has_basic(const Args& a) { return !a.username.empty() && !a.password.empty(); } +bool use_tls(const Args& a) { return has_token(a) || has_basic(a); } + +void append_auth(std::string& c, const Args& a) { + if (has_token(a)) + c += "token=" + a.token + ";"; + else if (has_basic(a)) + c += "username=" + a.username + ";password=" + a.password + ";"; +} + +std::string build_ingest_conf(const Args& a, unsigned worker_id) { + auto addrs = split_addrs(a.addrs); + std::string joined; + for (size_t i = 0; i < addrs.size(); ++i) joined += (i ? "," : "") + addrs[i]; + + if (a.protocol == "qwpudp") { + return "qwpudp::addr=" + addrs[0] + ";"; + } + if (a.protocol == "ilp") { + std::string c = (use_tls(a) ? "https" : "http"); + c += "::addr=" + joined + ";"; + append_auth(c, a); + if (use_tls(a)) c += "tls_verify=unsafe_off;"; + c += "retry_timeout=" + std::to_string(a.retry_timeout) + ";"; + return c; + } + // qwp (WebSocket) + std::string who = a.sender_id + "-" + std::to_string(worker_id); + std::string sf = a.store_forward_dir + "/" + who; + std::error_code ec; + std::filesystem::create_directories(sf, ec); + std::string c = (use_tls(a) ? "qwpwss" : "qwpws"); + c += "::addr=" + joined + ";"; + append_auth(c, a); + if (use_tls(a)) c += "tls_verify=unsafe_off;"; + c += "sender_id=" + who + ";"; + c += "sf_dir=" + sf + ";"; + c += "reconnect_max_duration_millis=" + std::to_string(a.retry_timeout) + ";"; + c += "reconnect_initial_backoff_millis=100;"; + c += "reconnect_max_backoff_millis=5000;"; + if (a.enterprise) c += "request_durable_ack=true;"; + return c; +} + +std::string build_reader_conf(const Args& a) { + auto addrs = split_addrs(a.addrs); + std::string joined; + for (size_t i = 0; i < addrs.size(); ++i) joined += (i ? "," : "") + addrs[i]; + std::string c = (use_tls(a) ? "wss" : "ws"); + c += "::addr=" + joined + ";target=any;failover=true;"; + append_auth(c, a); + if (use_tls(a)) c += "tls_verify=unsafe_off;"; + if (!a.zone.empty()) c += "zone=" + a.zone + ";"; + return c; +} + +std::string format_micros(int64_t micros) { + std::time_t secs = static_cast(micros / 1'000'000); + int64_t frac = micros % 1'000'000; + std::tm tm{}; + gmtime_r(&secs, &tm); + char buf[32]; + std::strftime(buf, sizeof(buf), "%Y-%m-%dT%H:%M:%S", &tm); + char out[64]; + std::snprintf(out, sizeof(out), "%s.%06lldZ", buf, static_cast(frac)); + return out; +} + +const char* role_name(eg::server_role r) { + switch (r) { + case eg::server_role::standalone: return "STANDALONE"; + case eg::server_role::primary: return "PRIMARY"; + case eg::server_role::replica: return "REPLICA"; + case eg::server_role::primary_catchup: return "PRIMARY_CATCHUP"; + default: return "UNKNOWN"; + } +} + +// Read a string cell from a Symbol or Varchar column; empty for null / non-string kinds. +std::optional col_string(const eg::column& col, size_t row) { + std::optional out; + col.visit(eg::overload{ + [&](eg::varlen_view v) { + if (auto x = v.as_string_view(row)) out = std::string(*x); + }, + [&](eg::symbol_view v) { + if (auto x = v.resolve(row)) out = std::string(*x); + }, + [](auto) {}, + }); + return out; +} + +void load_csv(const std::string& path, bool need_timestamp, std::vector& out) { + gzFile f = gzopen(path.c_str(), "rb"); // transparently reads .gz and plain files + if (!f) throw std::runtime_error("cannot open CSV: " + path); + char line[4096]; + char* header = gzgets(f, line, sizeof(line)); + if (!header) { gzclose(f); return; } + + // Minimal CSV field split: honours double-quoted fields (quotes are dropped and a + // comma inside quotes does not split), then trims whitespace on unquoted fields. + auto fields = [](const char* s) { + std::vector v; + std::string cur; + bool in_quotes = false; + for (const char* p = s; *p && *p != '\n' && *p != '\r'; ++p) { + char ch = *p; + if (ch == '"') { in_quotes = !in_quotes; continue; } + if (ch == ',' && !in_quotes) { v.push_back(cur); cur.clear(); } + else cur += ch; + } + v.push_back(cur); + for (auto& x : v) { + size_t b = x.find_first_not_of(" \t"); + size_t e = x.find_last_not_of(" \t"); + x = (b == std::string::npos) ? "" : x.substr(b, e - b + 1); + } + return v; + }; + + auto cols = fields(header); + auto idx = [&](const std::string& name) -> int { + for (size_t i = 0; i < cols.size(); ++i) + if (cols[i] == name) return static_cast(i); + throw std::runtime_error("CSV missing required column: " + name); + }; + int i_sym = idx("symbol"), i_side = idx("side"), i_price = idx("price"), + i_amt = idx("amount"); + int i_ts = need_timestamp ? idx("timestamp") : -1; + + while (gzgets(f, line, sizeof(line))) { + auto r = fields(line); + if (r.size() <= static_cast(i_amt)) continue; + TradeRow tr; + tr.symbol = r[i_sym]; + tr.side = r[i_side]; + tr.price = std::stod(r[i_price]); + tr.amount = std::stod(r[i_amt]); + tr.ts_nanos = 0; + if (need_timestamp) { + // ISO-8601 like 2025-07-28T12:34:56.789012Z + std::tm tm{}; + const std::string& s = r[i_ts]; + int y, mo, d, h, mi, se; + long frac = 0; + if (std::sscanf(s.c_str(), "%d-%d-%dT%d:%d:%d", &y, &mo, &d, &h, &mi, &se) == 6) { + tm.tm_year = y - 1900; tm.tm_mon = mo - 1; tm.tm_mday = d; + tm.tm_hour = h; tm.tm_min = mi; tm.tm_sec = se; + std::time_t secs = timegm(&tm); + size_t dot = s.find('.'); + if (dot != std::string::npos) { + std::string fs = s.substr(dot + 1); + while (!fs.empty() && !std::isdigit((unsigned char)fs.back())) fs.pop_back(); + fs.resize(9, '0'); // pad to nanoseconds + frac = std::stol(fs.substr(0, 9)); + } + tr.ts_nanos = static_cast(secs) * 1'000'000'000 + frac; + } + } + out.push_back(std::move(tr)); + } + gzclose(f); +} + +void run_worker(unsigned worker_id, uint64_t total_events, const Args& args, + const std::vector& rows, std::atomic& total_sent) { + std::printf("Sender %u will send %llu events\n", worker_id, + static_cast(total_events)); + const bool is_qwp = args.protocol == "qwp"; + const bool is_udp = args.protocol == "qwpudp"; + // Single worker on a QWP transport stamps rows client-side (monotonic, no O3). + const bool per_row = (is_qwp || is_udp) && args.num_senders == 1; + const uint64_t commit_every = + is_qwp ? args.batch_size * args.batches_per_transaction : args.batch_size; + + auto sender = qi::line_sender::from_conf(qi::utf8_view{build_ingest_conf(args, worker_id)}); + auto buffer = sender.new_buffer(); + const size_t n = rows.size(); + uint64_t sent = 0; + + for (uint64_t i = 0; i < total_events; ++i) { + const TradeRow& r = rows[i % n]; + std::string trade_id = std::to_string(worker_id) + "-" + std::to_string(i + 1); + buffer.table("trades"_tn) + .symbol("symbol"_cn, qi::utf8_view{r.symbol}) + .symbol("side"_cn, qi::utf8_view{r.side}) + .column("price"_cn, r.price) + .column("amount"_cn, r.amount) + .column("trade_id"_cn, qi::utf8_view{trade_id}); + + if (args.timestamp_from_file) + buffer.at(qi::timestamp_nanos{r.ts_nanos + args.seconds_offset * 1'000'000'000}); + else if (args.seconds_offset != 0) + buffer.at(qi::timestamp_nanos{now_nanos() + args.seconds_offset * 1'000'000'000}); + else if (per_row) + buffer.at(qi::timestamp_nanos::now()); + else + buffer.at_now(); + + ++sent; + total_sent.fetch_add(1, std::memory_order_relaxed); + if (sent % commit_every == 0) sender.flush(buffer); + if (args.delay_ms > 0) + std::this_thread::sleep_for(std::chrono::milliseconds(args.delay_ms)); + } + sender.flush(buffer); + if (is_qwp) sender.close_drain(); // drain un-acked store-and-forward frames + std::printf("Sender %u finished sending %llu events\n", worker_id, + static_cast(sent)); +} + +// Pull current_role / target_role out of `switch status` by column name. +void read_switch_status(eg::reader& reader, std::optional& current, + std::optional& target) { + current.reset(); + target.reset(); + auto cur = reader.execute(qi::utf8_view{STATUS_QUERY}); + while (auto batch_opt = cur.next_batch()) { + auto& batch = *batch_opt; + const size_t cols = batch.column_count(); + for (size_t r = 0; r < batch.row_count(); ++r) { + for (size_t c = 0; c < cols; ++c) { + std::string name(batch.column_name(c)); + std::string lower = name; + for (auto& ch : lower) ch = std::tolower((unsigned char)ch); + if (lower.find("role") == std::string::npos) continue; + auto v = col_string(batch.column(c), r); + if (lower.find("current") != std::string::npos) current = v; + else if (lower.find("target") != std::string::npos) target = v; + else if (!current) current = v; + } + } + } +} + +void run_probe(const Args& args, std::atomic& stop) { + std::unique_ptr reader; + try { + reader = std::make_unique(qi::utf8_view{build_reader_conf(args)}); + } catch (const std::exception& e) { + std::printf("[query client] connect failed: %s\n", e.what()); + return; + } + // server_info() is a non-owning view whose role is only populated at the handshake; it + // reads as "other" after queries run. Capture the handshake node/role once, at connect, + // and use it as the (labelled, possibly-stale) fallback when `switch status` is absent. + std::string hs_node, hs_role; + { + auto si = reader->server_info(); + hs_node = std::string(si.node_id()); + if (hs_node.empty()) hs_node = "(none)"; + hs_role = role_name(si.role()); + std::string cluster(si.cluster_id()); + std::printf("[query client] connected, serving node=%s role=%s cluster=%s\n", + hs_node.c_str(), hs_role.c_str(), cluster.empty() ? "(none)" : cluster.c_str()); + } + + bool was_down = false; + while (!stop.load(std::memory_order_relaxed)) { + try { + int64_t latest = 0; + bool have = false; + bool is_nanos = false; + auto cur = reader->execute(qi::utf8_view{PROBE_QUERY}); + while (auto batch_opt = cur.next_batch()) { + auto& batch = *batch_opt; + auto col = batch.column(0); + is_nanos = col.kind() == eg::column_kind::timestamp_nanos; + for (size_t r = 0; r < batch.row_count(); ++r) + if (auto v = col.get(r)) { latest = *v; have = true; } + } + if (was_down) { std::printf("[query client] connection restored\n"); was_down = false; } + if (have) { + std::optional current, target; + try { read_switch_status(*reader, current, target); } catch (...) {} + std::string served; + if (current) { + bool switching = target && *target != *current; + served = " served by role=" + *current + + (switching ? " (switching -> " + *target + ")" : "") + + " node=" + hs_node; + } else { + served = " served by role=" + hs_role + " node=" + hs_node + + " (handshake role; live 'switch status' unavailable, may be stale)"; + } + int64_t micros = is_nanos ? latest / 1000 : latest; + std::printf("[probe] latest trades timestamp = %s (raw=%lld)%s\n", + format_micros(micros).c_str(), static_cast(latest), + served.c_str()); + } + } catch (const std::exception& e) { + if (!was_down) { + std::printf("[query client] connection lost (%s), will retry\n", e.what()); + was_down = true; + } + } + for (uint64_t slept = 0; slept < args.probe_interval_ms && !stop.load(); slept += 50) + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + } +} + +std::string need_value(int& i, int argc, char** argv) { + if (i + 1 >= argc) throw std::runtime_error(std::string("missing value for ") + argv[i]); + return argv[++i]; +} + +Args parse_args(int argc, char** argv) { + Args a; + for (int i = 1; i < argc; ++i) { + std::string k = argv[i]; + if (k == "--addrs") a.addrs = need_value(i, argc, argv); + else if (k == "--token") a.token = need_value(i, argc, argv); + else if (k == "--username") a.username = need_value(i, argc, argv); + else if (k == "--password") a.password = need_value(i, argc, argv); + else if (k == "--total-events") a.total_events = std::stoull(need_value(i, argc, argv)); + else if (k == "--delay-ms") a.delay_ms = std::stoull(need_value(i, argc, argv)); + else if (k == "--num-senders") a.num_senders = std::stoul(need_value(i, argc, argv)); + else if (k == "--retry-timeout") a.retry_timeout = std::stoull(need_value(i, argc, argv)); + else if (k == "--csv") a.csv = need_value(i, argc, argv); + else if (k == "--timestamp-from-file") a.timestamp_from_file = true; + else if (k == "--seconds-offset") a.seconds_offset = std::stoll(need_value(i, argc, argv)); + else if (k == "--protocol") a.protocol = need_value(i, argc, argv); + else if (k == "--sender-id") a.sender_id = need_value(i, argc, argv); + else if (k == "--store-forward-dir") a.store_forward_dir = need_value(i, argc, argv); + else if (k == "--batch-size") a.batch_size = std::stoull(need_value(i, argc, argv)); + else if (k == "--batches-per-transaction") + a.batches_per_transaction = std::stoull(need_value(i, argc, argv)); + else if (k == "--probe-interval-ms") + a.probe_interval_ms = std::stoull(need_value(i, argc, argv)); + else if (k == "--enterprise") a.enterprise = true; + else if (k == "--zone") a.zone = need_value(i, argc, argv); + else throw std::runtime_error("unknown argument: " + k); + } + return a; +} + +} // namespace + +int main(int argc, char** argv) { + Args args; + try { + args = parse_args(argc, argv); + } catch (const std::exception& e) { + std::fprintf(stderr, "%s\n", e.what()); + return 2; + } + + if (args.protocol != "qwp" && args.protocol != "qwpudp" && args.protocol != "ilp") { + std::fprintf(stderr, "--protocol must be 'qwp', 'qwpudp', or 'ilp', got: %s\n", + args.protocol.c_str()); + return 2; + } + auto addrs = split_addrs(args.addrs); + if (addrs.empty()) { std::fprintf(stderr, "--addrs must contain a host:port\n"); return 2; } + if (args.protocol == "qwpudp") { + if (addrs.size() > 1) { + std::fprintf(stderr, "--protocol qwpudp supports a single host:port only (UDP is " + "connectionless, no failover); got %zu addresses\n", addrs.size()); + return 2; + } + if (has_token(args) || !args.username.empty() || !args.password.empty()) + std::fprintf(stderr, "[warn] --protocol qwpudp is unauthenticated (UDP accepts any " + "connection); ignoring --token/--username/--password\n"); + } + + std::vector rows; + try { + load_csv(args.csv, args.timestamp_from_file, rows); + } catch (const std::exception& e) { + std::fprintf(stderr, "Failed to read CSV: %s\n", e.what()); + return 2; + } + if (rows.empty()) { std::fprintf(stderr, "CSV has no data rows: %s\n", args.csv.c_str()); return 2; } + + if (args.protocol == "qwp") + std::printf("Ingestion started. Protocol: qwp (WebSocket, sender-id=%s, " + "store-and-forward=%s, batch-size=%llu, batches-per-transaction=%llu, " + "retry-timeout-ms=%llu) | addrs: %s\n", + args.sender_id.c_str(), args.store_forward_dir.c_str(), + (unsigned long long)args.batch_size, + (unsigned long long)args.batches_per_transaction, + (unsigned long long)args.retry_timeout, args.addrs.c_str()); + else if (args.protocol == "qwpudp") + std::printf("Ingestion started. Protocol: qwpudp (QWP/UDP datagrams, ingest-only, " + "unauthenticated, no store-and-forward, no failover; query client disabled, " + "batch-size=%llu) | addr: %s\n", + (unsigned long long)args.batch_size, addrs[0].c_str()); + else + std::printf("Ingestion started. Protocol: ilp (HTTP, retry-timeout-ms=%llu) | addrs: %s\n", + (unsigned long long)args.retry_timeout, args.addrs.c_str()); + + std::atomic total_sent{0}; + std::atomic stop{false}; + auto start = std::chrono::steady_clock::now(); + + std::thread reporter([&] { + uint64_t last = 0; + while (!stop.load()) { + std::this_thread::sleep_for(std::chrono::seconds(1)); + uint64_t now = total_sent.load(); + std::printf("[progress] sent=%llu rate=%llu rows/s\n", + (unsigned long long)now, (unsigned long long)(now - last)); + last = now; + } + }); + + std::thread probe; + if (args.protocol == "qwp" && args.probe_interval_ms > 0) + probe = std::thread(run_probe, std::cref(args), std::ref(stop)); + + const uint64_t base = args.total_events / args.num_senders; + const uint64_t rem = args.total_events % args.num_senders; + std::atomic failures{0}; + std::vector workers; + for (unsigned id = 0; id < args.num_senders; ++id) { + uint64_t events = base + (id < rem ? 1 : 0); + workers.emplace_back([&, id, events] { + try { + run_worker(id, events, args, rows, total_sent); + } catch (const std::exception& e) { + std::fprintf(stderr, "Worker failed: Sender %u: %s\n", id, e.what()); + failures.fetch_add(1); + } + }); + } + for (auto& w : workers) w.join(); + stop.store(true); + if (probe.joinable()) probe.join(); + reporter.join(); + + if (failures.load() > 0) return 1; + + double elapsed = std::chrono::duration(std::chrono::steady_clock::now() - start).count(); + double rate = elapsed > 0 ? args.total_events / elapsed : 0; + std::printf("All workers completed. protocol=%s events=%llu elapsed=%.3f s throughput=%.0f rows/s\n", + args.protocol.c_str(), (unsigned long long)args.total_events, elapsed, rate); + return 0; +} From 2c406ad6532a1459e26ec7f2fcb26ac2b9c5b48a Mon Sep 17 00:00:00 2001 From: javier Date: Mon, 13 Jul 2026 11:18:14 +0200 Subject: [PATCH 13/79] Document the client ports in the root README Add a Client implementations section explaining the Java implementation is the reference, ported in parity to the Rust, Python, and C/C++ clients, with links to each folder. --- README.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/README.md b/README.md index 44f2538..9fcb79b 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,21 @@ # Sample Sender with Multiple Hosts for HA Ingestion +## Client implementations + +This project is a QuestDB high-availability sender. The **Java** implementation in this +repository is the **reference**. It is ported, in parity, to the three clients built on the +same C/Rust core (the "CRusty" clients): + +- [`rust/`](./rust) - Rust port. +- [`python/`](./python) - Python port, plus a pandas/polars ingestion and egress demo. +- [`c/`](./c) - C/C++ port. + +Each mirrors the Java design (CSV replay loop, per-worker senders, the `qwp` / `qwpudp` / +`ilp` transports, failover, store-and-forward, and the probe). Per-language build steps and +client differences (for example, auto-flush is present in Java and Python but not in Rust or +C/C++) are documented in that folder's `README.md`. The rest of this document describes the +reference Java implementation. + ## Compile `mvn -DskipTests clean package` From 69e0df9119db53a34c7737dc678eb7122926e2ba Mon Sep 17 00:00:00 2001 From: javier Date: Tue, 14 Jul 2026 12:17:21 +0200 Subject: [PATCH 14/79] Add --rate pacing and nanosecond timestamp ingestion across all clients Add a --rate flag that targets an aggregate rows/second across all workers, pacing each worker to its share against a deadline schedule so it reaches high targets a per-row --delay-ms cannot. It takes precedence over --delay-ms. Ported to Java, Rust, Python, and C++. Send designated timestamps at nanosecond resolution everywhere. QuestDB stores at the target column resolution: TIMESTAMP_NS keeps full nanos, micros TIMESTAMP truncates, and a non-existent table is auto-created as TIMESTAMP_NS. Java now uses at long NANOS, Rust uses TimestampNanos with a nanosecond CSV parse, Python gained a nanosecond-preserving ISO parse, and C++ already sent nanos. Add regenerate_csv.sh and regenerate_csv_fx.sh to export fresh crypto and FX data from the always-on demo box. Document --rate, the nanosecond behavior, and the two scripts in each README. --- README.md | 92 +++++++++++++++++++ c/README.md | 22 ++++- c/csv_parallel_sender.cpp | 39 +++++++- python/README.md | 23 ++++- python/csv_parallel_sender.py | 64 +++++++++++-- regenerate_csv.sh | 65 +++++++++++++ regenerate_csv_fx.sh | 76 +++++++++++++++ rust/README.md | 27 +++++- rust/src/main.rs | 83 +++++++++++++---- .../com/example/sender/CsvParallelSender.java | 73 +++++++++++++-- 10 files changed, 529 insertions(+), 35 deletions(-) create mode 100755 regenerate_csv.sh create mode 100755 regenerate_csv_fx.sh diff --git a/README.md b/README.md index 9fcb79b..926798b 100644 --- a/README.md +++ b/README.md @@ -62,6 +62,98 @@ java -jar target/ilp_sender-1.0-SNAPSHOT.jar \ QWP. That is a different timeout from `--connect-timeout-ms` (below), which bounds a single connect attempt. +## Pacing: `--delay-ms` vs `--rate` + +Two mutually exclusive ways to throttle generation: + +- `--delay-ms` (default `50`): a fixed `Thread.sleep(delay)` after **every row**, per worker. + Simple, but a poor throttle at speed: `sleep()` has millisecond granularity (and often + over-sleeps), and the fixed per-row cost caps throughput. `--delay-ms 1` yields well under + 1000 rows/s; `--delay-ms 0` runs flat out (millions/s). +- `--rate` (default `0` = off): a **target aggregate rate in rows/second across all workers**. + Each worker paces itself to its share (`rate / num-senders`) against a deadline schedule, + sending rows back-to-back and sleeping only when it runs *ahead* of schedule (coalescing many + rows into one sleep). This hits high targets a per-row delay never could: `--rate 300000` + holds ~300k rows/s regardless of `--num-senders`. + +When `--rate > 0` it **takes precedence** and `--delay-ms` is ignored (a warning is printed if +both are set). Measured on a local QuestDB: `--rate 5000` → 4.03 s for 20k rows (~4966/s); +`--rate 300000` → 2.03 s for 600k rows (~296k/s), steady per-second progress at the target. + +### Regenerating the replay CSV + +Two scripts regenerate the replay CSV. **Both always export from the public demo box** +(`https://demo.questdb.io/exp`) on purpose: the demo is *continuously ingesting live market +data*, so every export is a fresh, recent-price snapshot rather than a stale file. You then +**recreate the table locally** from that CSV and replay it for internal demos. The sender +always writes into a table named **`trades`** locally, regardless of which demo table the CSV +came from. + +- **`regenerate_csv.sh`** — pulls the crypto **`trades`** table (BTC-USDT, ...). Its + timestamp is microseconds. + ``` + ./regenerate_csv.sh # -> trades.csv, 1,000,000 most-recent rows + ./regenerate_csv.sh trades.csv.gz # gzip output (matches the --csv default's .gz) + ./regenerate_csv.sh trades.csv 250000 # custom row count + ``` +- **`regenerate_csv_fx.sh`** — pulls the FX **`fx_trades`** table (EURCHF, AUDNZD, ...). + Two schema differences from the crypto table, both handled in the query + `select timestamp, symbol, side, price, quantity as amount from fx_trades order by timestamp desc limit N`: + its size column is **`quantity`**, aliased to **`amount`** so it matches the `trades` schema + the sender writes; and its designated timestamp is **`TIMESTAMP_NS`** (nanoseconds), not + micros (see [Nanosecond vs microsecond timestamps](#nanosecond-vs-microsecond-timestamps)). + ``` + ./regenerate_csv_fx.sh # -> fx_trades.csv, 1,000,000 rows + ./regenerate_csv_fx.sh fx_trades.csv.gz # gzipped + ./regenerate_csv_fx.sh fx_trades.csv 250000 # custom row count + ``` + +Both export exactly `symbol, side, price, amount, timestamp` (the columns the sender reads), +download to a temp file first so a failed fetch never clobbers a good CSV, and gzip-compress +when the output path ends in `.gz` (the loader auto-detects `.gz`). Override `DEMO_HOST` only if +you mirror the demo tables elsewhere. + +### Nanosecond vs microsecond timestamps + +The crypto `trades` table exports microsecond timestamps; the FX `fx_trades` table exports +**nanosecond** timestamps (`TIMESTAMP_NS`, e.g. `...192508297Z`). This only ever matters with +`--timestamp-from-file`; in the default replay mode the sender stamps `now()` and ignores the +file timestamp entirely, so precision is irrelevant. + +**The generator sends timestamps at nanosecond resolution** (via `at(long, NANOS)`), and +QuestDB stores them at the **target column's** resolution: a micros `TIMESTAMP` column silently +truncates the extra digits, a `TIMESTAMP_NS` column keeps them. Cross-resolution replay is +therefore safe in both directions and lands every row. Validated against a local QuestDB, 20k +crypto + 20k FX rows each way (all 40k landing, no errors): + +| Local `trades` column | Replayed source | Stored as | +| --- | --- | --- | +| `timestamp` (micros) | crypto micros `...790999Z` | `...790999Z` | +| `timestamp` (micros) | FX nanos `...192508297Z` | `...192508Z` (truncated to micros) | +| `timestamp_ns` (nanos) | crypto micros `...790999Z` | `...790999000Z` | +| `timestamp_ns` (nanos) | FX nanos `...192508297Z` | `...192508297Z` (**full nanos preserved**) | + +So if your local table is `TIMESTAMP_NS`, the FX source's nanosecond precision is preserved +end-to-end; if it is micros, the sub-microsecond digits are truncated on store (lossless to the +column's resolution, never an error). To recreate the local table at a given resolution, +pre-create it before ingesting, e.g.: + +```sql +-- microsecond trades table +create table trades (symbol symbol, side symbol, price double, amount double, + trade_id varchar, timestamp timestamp) timestamp(timestamp) partition by day wal; + +-- nanosecond trades table (swap the timestamp type) +create table trades (symbol symbol, side symbol, price double, amount double, + trade_id varchar, timestamp timestamp_ns) timestamp(timestamp) partition by day wal; +``` + +**If the table does not exist, the server auto-creates it as `TIMESTAMP_NS`** (nanosecond), +because the generator sends nanosecond timestamps: the designated timestamp column is created +at the resolution of the incoming value. Verified end-to-end: dropping `trades` and replaying +the FX chunk with `--timestamp-from-file` recreated it with a `TIMESTAMP_NS` column holding the +full `...192508297Z` value. Pre-create the table (above) if you want micros instead. + ## Transport (`--protocol`) - `--protocol qwp` (default): QWP over WebSocket. Adds store-and-forward (un-acked diff --git a/c/README.md b/c/README.md index fe091b7..18e3dcb 100644 --- a/c/README.md +++ b/c/README.md @@ -49,12 +49,31 @@ match the target server build. ``` Flags mirror the other ports: `--addrs`, `--token`/`--username`/`--password`, `--total-events`, -`--delay-ms`, `--num-senders`, `--retry-timeout`, `--csv`, `--timestamp-from-file`, +`--delay-ms`, `--rate`, `--num-senders`, `--retry-timeout`, `--csv`, `--timestamp-from-file`, `--seconds-offset`, `--protocol`, `--sender-id`, `--store-forward-dir`, `--batch-size`, `--batches-per-transaction`, `--probe-interval-ms`, `--enterprise`, `--zone`. (`--timestamp-from-file` and `--enterprise` are presence flags.) Confirm results server-side with `SELECT count() FROM trades`. +## Pacing: `--delay-ms` vs `--rate` + +- `--delay-ms` (default `50`): a fixed sleep after **every row**, per worker. +- `--rate` (default `0` = off): a **target aggregate rate in rows/second across all workers**. + Each worker paces to its share (`rate / num-senders`) against a deadline schedule, sending + rows back-to-back and sleeping only when it runs *ahead* — reaching high targets a per-row + sleep never could. When `> 0` it takes precedence over `--delay-ms` (a warning prints if both + are set). Measured: `--rate 5000`, 20k rows, 4 workers → ~5.0 s, steady ~5000 rows/s. + +## Timestamps + +The generator sends timestamps at **nanosecond** resolution (`timestamp_nanos`), and QuestDB +stores them at the **target column's** resolution: a `TIMESTAMP_NS` column keeps full nanos, a +micros `TIMESTAMP` column truncates the extra digits (silently, never an error). **If the table +does not exist, the server auto-creates it as `TIMESTAMP_NS`.** Pre-create `trades` with a +`timestamp` (micros) column if you want micros. The CSV parser already reads fractional seconds +to full nanosecond precision. Verified: replaying the nanosecond FX chunk with +`--timestamp-from-file` preserved `...192508297Z` in an auto-created `TIMESTAMP_NS` table. + ## Probe (QWP/WebSocket only) For `qwp`, a background thread uses the egress `reader` to run `select timestamp from trades @@ -83,6 +102,7 @@ Against a local QuestDB (OSS), all rows accounted for server-side: | `qwp`, 4000 rows, 1 worker | 4000 / 4000, distinct per-row timestamps; probe reported live timestamps + handshake role fallback | | `ilp`, 2000 rows, 2 workers | 2000 / 2000 | | `qwpudp` (paced) | rows land; best-effort, lossy under fast bursts | +| `qwp`, `--rate 5000`, nanos FX, `--timestamp-from-file`, 4 workers | steady ~5000 rows/s; auto-created `TIMESTAMP_NS` table preserved `...192508297Z` | | QWP failover (primary crash) | see below | ### QWP failover (primary crash) diff --git a/c/csv_parallel_sender.cpp b/c/csv_parallel_sender.cpp index 6901b57..d137b17 100644 --- a/c/csv_parallel_sender.cpp +++ b/c/csv_parallel_sender.cpp @@ -48,6 +48,7 @@ struct Args { std::string token, username, password; uint64_t total_events = 1'000'000; uint64_t delay_ms = 50; + uint64_t rate = 0; // target aggregate rows/s across all workers (0 = off, use delay_ms) unsigned num_senders = 10; uint64_t retry_timeout = 360'000; std::string csv = "./trades20250728.csv.gz"; @@ -266,10 +267,19 @@ void run_worker(unsigned worker_id, uint64_t total_events, const Args& args, const uint64_t commit_every = is_qwp ? args.batch_size * args.batches_per_transaction : args.batch_size; + // Rate limiting (when --rate > 0): pace this worker to its share of the aggregate target, + // rate / num-senders rows/second. interval_nanos is the ideal spacing between this worker's + // rows; we sleep only when running ahead of a deadline schedule, so rows go out back-to-back + // until we get ahead. Reaches high targets a fixed per-row sleep cannot. + const double interval_nanos = + args.rate > 0 ? 1'000'000'000.0 * args.num_senders / static_cast(args.rate) : 0.0; + const bool rate_limited = interval_nanos > 0.0; + auto sender = qi::line_sender::from_conf(qi::utf8_view{build_ingest_conf(args, worker_id)}); auto buffer = sender.new_buffer(); const size_t n = rows.size(); uint64_t sent = 0; + const auto pace_start = std::chrono::steady_clock::now(); for (uint64_t i = 0; i < total_events; ++i) { const TradeRow& r = rows[i % n]; @@ -293,8 +303,22 @@ void run_worker(unsigned worker_id, uint64_t total_events, const Args& args, ++sent; total_sent.fetch_add(1, std::memory_order_relaxed); if (sent % commit_every == 0) sender.flush(buffer); - if (args.delay_ms > 0) + if (rate_limited) { + // Deadline for the row just sent (sent is 1-based). Sleep only while ahead of + // schedule; the 1ms floor coalesces many rows into one sleep at high rates. + const uint64_t target_nanos = static_cast(sent * interval_nanos); + const uint64_t elapsed_nanos = static_cast( + std::chrono::duration_cast( + std::chrono::steady_clock::now() - pace_start) + .count()); + if (target_nanos > elapsed_nanos) { + const uint64_t sleep_nanos = target_nanos - elapsed_nanos; + if (sleep_nanos > 1'000'000) + std::this_thread::sleep_for(std::chrono::nanoseconds(sleep_nanos)); + } + } else if (args.delay_ms > 0) { std::this_thread::sleep_for(std::chrono::milliseconds(args.delay_ms)); + } } sender.flush(buffer); if (is_qwp) sender.close_drain(); // drain un-acked store-and-forward frames @@ -407,6 +431,7 @@ Args parse_args(int argc, char** argv) { else if (k == "--password") a.password = need_value(i, argc, argv); else if (k == "--total-events") a.total_events = std::stoull(need_value(i, argc, argv)); else if (k == "--delay-ms") a.delay_ms = std::stoull(need_value(i, argc, argv)); + else if (k == "--rate") a.rate = std::stoull(need_value(i, argc, argv)); else if (k == "--num-senders") a.num_senders = std::stoul(need_value(i, argc, argv)); else if (k == "--retry-timeout") a.retry_timeout = std::stoull(need_value(i, argc, argv)); else if (k == "--csv") a.csv = need_value(i, argc, argv); @@ -456,6 +481,12 @@ int main(int argc, char** argv) { "connection); ignoring --token/--username/--password\n"); } + // --rate takes precedence: when set it drives pacing and --delay-ms is ignored. + if (args.rate > 0 && args.delay_ms > 0) + std::fprintf(stderr, "[warn] --rate %llu overrides --delay-ms %llu (rate limiting drives " + "pacing; the per-row delay is ignored)\n", + (unsigned long long)args.rate, (unsigned long long)args.delay_ms); + std::vector rows; try { load_csv(args.csv, args.timestamp_from_file, rows); @@ -465,6 +496,12 @@ int main(int argc, char** argv) { } if (rows.empty()) { std::fprintf(stderr, "CSV has no data rows: %s\n", args.csv.c_str()); return 2; } + if (args.rate > 0) + std::printf("Pacing: rate=%llu rows/s (aggregate across %u workers)\n", + (unsigned long long)args.rate, args.num_senders); + else + std::printf("Pacing: delay-ms=%llu\n", (unsigned long long)args.delay_ms); + if (args.protocol == "qwp") std::printf("Ingestion started. Protocol: qwp (WebSocket, sender-id=%s, " "store-and-forward=%s, batch-size=%llu, batches-per-transaction=%llu, " diff --git a/python/README.md b/python/README.md index ceb0f31..147f7a1 100644 --- a/python/README.md +++ b/python/README.md @@ -69,11 +69,31 @@ and `pyarrow` are only needed for the dataframe paths. ``` Flags mirror the Java/Rust ports: `--addrs`, `--token`/`--username`/`--password`, -`--total-events`, `--delay-ms`, `--num-senders`, `--retry-timeout`, `--csv`, +`--total-events`, `--delay-ms`, `--rate`, `--num-senders`, `--retry-timeout`, `--csv`, `--timestamp-from-file`, `--seconds-offset`, `--protocol`, `--sender-id`, `--store-forward-dir`, `--batch-size`, `--batches-per-transaction`, `--probe-interval-ms`, `--enterprise`, `--zone`. Confirm results server-side with `SELECT count() FROM trades`. +## Pacing: `--delay-ms` vs `--rate` + +- `--delay-ms` (default `50`): a fixed sleep after **every row**, per worker. +- `--rate` (default `0` = off): a **target aggregate rate in rows/second across all workers**. + Each worker paces to its share (`rate / num-senders`) against a deadline schedule, sending + rows back-to-back and sleeping only when it runs *ahead*. When `> 0` it takes precedence over + `--delay-ms` (a warning prints if both are set). Note Python's per-row throughput ceiling is + lower than Java/Rust (the GIL / Python-level row building), so very high `--rate` targets may + not be reached; the pacing simply never sleeps in that case. + +## Timestamps + +The generator sends timestamps at **nanosecond** resolution (`TimestampNanos`), and QuestDB +stores them at the **target column's** resolution: a `TIMESTAMP_NS` column keeps full nanos, a +micros `TIMESTAMP` column truncates the extra digits (silently, never an error). **If the table +does not exist, the server auto-creates it as `TIMESTAMP_NS`.** Pre-create `trades` with a +`timestamp` (micros) column if you want micros. The CSV timestamp is parsed to full nanoseconds +(the fractional-seconds string is read directly, since Python's `datetime` is microsecond-only +and would otherwise drop the last three digits of a `TIMESTAMP_NS` source). + ## Probe (QWP/WebSocket only) For `qwp`, a background thread uses the pooled query client (`Client`) to run @@ -128,6 +148,7 @@ Against a local QuestDB (OSS), all rows accounted for server-side: | What | Result | | --- | --- | | `qwp`, 3000 rows, 1 worker | 3000 / 3000, distinct per-row timestamps; probe reported live timestamps | +| `qwp`, `--rate 5000`, nanos FX, `--timestamp-from-file`, 4 workers | 20000 / 20000, steady ~5000 rows/s; auto-created `TIMESTAMP_NS` table preserved `...192508297Z` | | `ilp`, 2000 rows, 2 workers | 2000 / 2000 | | `qwpudp` (paced) | rows land; best-effort, lossy under fast bursts | | `dataframe_demo` | pandas + polars ingest (10000 rows), egress to pandas and polars | diff --git a/python/csv_parallel_sender.py b/python/csv_parallel_sender.py index 43e6ae2..5f7833e 100644 --- a/python/csv_parallel_sender.py +++ b/python/csv_parallel_sender.py @@ -42,7 +42,12 @@ def parse_args(argv): p.add_argument("--username", default=None, help="Basic-auth username.") p.add_argument("--password", default=None, help="Basic-auth password.") p.add_argument("--total-events", type=int, default=1_000_000) - p.add_argument("--delay-ms", type=int, default=50) + p.add_argument("--delay-ms", type=int, default=50, + help="Per-row sleep in ms (0 = flat out). Ignored when --rate > 0.") + p.add_argument("--rate", type=int, default=0, + help="Target aggregate rows/second across ALL workers (0 = off, use " + "--delay-ms). Takes precedence over --delay-ms: each worker paces to " + "rate/num-senders against a deadline schedule.") p.add_argument("--num-senders", type=int, default=10) p.add_argument("--retry-timeout", type=int, default=360_000, help="retry_timeout (ILP) / reconnect_max_duration_millis (QWP), in ms.") @@ -131,6 +136,27 @@ def build_client_conf(args): return "".join(parts) +def iso_to_nanos(raw): + """Parse an ISO-8601 timestamp to epoch nanoseconds, preserving full nanosecond + precision. datetime is microsecond-limited and would drop the last 3 digits of a + TIMESTAMP_NS value (e.g. ...192508297Z -> ...192508000), so the fractional seconds are + split out and parsed as an integer, and only the whole-second part goes through datetime.""" + raw = raw.strip() + frac_nanos = 0 + dot = raw.find(".") + if dot != -1: + end = dot + 1 + while end < len(raw) and raw[end].isdigit(): + end += 1 + frac = raw[dot + 1:end] + frac_nanos = int(frac[:9].ljust(9, "0")) + raw = raw[:dot] + raw[end:] + dt = datetime.fromisoformat(raw.replace("Z", "+00:00")) + if dt.tzinfo is None: + dt = dt.replace(tzinfo=timezone.utc) + return int(dt.timestamp()) * 1_000_000_000 + frac_nanos + + def load_csv(path, need_timestamp): """Load the CSV (optionally gzipped). Returns a list of (symbol, side, price, amount, ts_nanos) tuples; ts_nanos is 0 unless need_timestamp.""" @@ -152,10 +178,7 @@ def load_csv(path, need_timestamp): continue ts_nanos = 0 if need_timestamp: - raw = rec[idx["timestamp"]].strip() - # ISO-8601, e.g. 2025-07-28T12:34:56.789012Z - dt = datetime.fromisoformat(raw.replace("Z", "+00:00")) - ts_nanos = int(dt.timestamp() * 1_000_000_000) + ts_nanos = iso_to_nanos(rec[idx["timestamp"]].strip()) rows.append(( rec[idx["symbol"]].strip(), rec[idx["side"]].strip(), @@ -177,9 +200,17 @@ def run_worker(worker_id, total_events, args, rows, counts): # every batch-size rows (auto_flush is off, so the buffer is drained explicitly). commit_every = args.batch_size * args.batches_per_transaction if is_qwp else args.batch_size + # Rate limiting (when --rate > 0): pace this worker to its share of the aggregate target, + # rate / num-senders rows/second. interval_nanos is the ideal spacing between this worker's + # rows; we sleep only when running ahead of a deadline schedule, so rows go out back-to-back + # until we get ahead. Reaches high targets a fixed per-row sleep cannot. + interval_nanos = (1_000_000_000 * args.num_senders / args.rate) if args.rate > 0 else 0.0 + rate_limited = interval_nanos > 0.0 + conf = build_ingest_conf(args, worker_id) n = len(rows) sent = 0 + pace_start = time.perf_counter_ns() with Sender.from_conf(conf) as sender: for i in range(total_events): symbol, side, price, amount, ts_nanos = rows[i % n] @@ -201,7 +232,14 @@ def run_worker(worker_id, total_events, args, rows, counts): counts[worker_id] = sent if sent % commit_every == 0: sender.flush() - if args.delay_ms > 0: + if rate_limited: + # Deadline for the row just sent (sent is 1-based). Sleep only while ahead of + # schedule; the 1ms floor coalesces many rows into one sleep at high rates. + target_nanos = int(sent * interval_nanos) + sleep_nanos = target_nanos - (time.perf_counter_ns() - pace_start) + if sleep_nanos > 1_000_000: + time.sleep(sleep_nanos / 1_000_000_000) + elif args.delay_ms > 0: time.sleep(args.delay_ms / 1000.0) sender.flush() # QWP/WebSocket: drain any un-acked store-and-forward frames before closing. @@ -292,6 +330,15 @@ def main(argv): print("[warn] --protocol qwpudp is unauthenticated (UDP accepts any connection); " "ignoring --token/--username/--password", file=sys.stderr) + if args.rate < 0: + print("--rate must be >= 0 (0 disables rate limiting, falling back to --delay-ms)", + file=sys.stderr) + return 2 + # --rate takes precedence: when set it drives pacing and --delay-ms is ignored. + if args.rate > 0 and args.delay_ms > 0: + print(f"[warn] --rate {args.rate} overrides --delay-ms {args.delay_ms} (rate limiting " + "drives pacing; the per-row delay is ignored)", file=sys.stderr) + if not os.path.exists(args.csv): print(f"CSV file not found: {args.csv}", file=sys.stderr) return 2 @@ -313,6 +360,11 @@ def main(argv): print(f"Ingestion started. Protocol: ilp (HTTP, retry-timeout-ms={args.retry_timeout}) " f"| addrs: {','.join(addrs)}") + if args.rate > 0: + print(f"Pacing: rate={args.rate} rows/s (aggregate across {args.num_senders} workers)") + else: + print(f"Pacing: delay-ms={args.delay_ms}") + counts = [0] * args.num_senders stop = threading.Event() start = time.monotonic() diff --git a/regenerate_csv.sh b/regenerate_csv.sh new file mode 100755 index 0000000..daf3d81 --- /dev/null +++ b/regenerate_csv.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash +# +# Regenerate the replay CSV from QuestDB's public demo instance. +# +# Pulls the most recent trades from https://demo.questdb.io via the /exp CSV +# export endpoint, so the sender replays fresh data instead of a stale snapshot. +# +# Usage: +# ./regenerate_csv.sh [output-file] [limit] +# +# output-file destination path (default: trades.csv). If it ends in .gz the +# result is gzip-compressed, matching the sender's default +# --csv ./trades20250728.csv.gz (loadCsv auto-detects .gz). +# limit number of most-recent rows to fetch (default: 1000000). +# +# Env overrides: +# DEMO_HOST base URL (default: https://demo.questdb.io) +# +# Examples: +# ./regenerate_csv.sh # -> trades.csv, 1,000,000 rows +# ./regenerate_csv.sh trades.csv.gz # -> gzipped +# ./regenerate_csv.sh trades.csv 250000 # -> 250k rows +set -euo pipefail + +OUT="${1:-trades.csv}" +LIMIT="${2:-1000000}" +DEMO_HOST="${DEMO_HOST:-https://demo.questdb.io}" + +# select * gives symbol, side, price, amount, timestamp: exactly the columns the +# sender reads (timestamp only needed with --timestamp-from-file). Ordering desc +# gives the most recent rows; the sender stamps now() by default so order does +# not affect ingestion unless --timestamp-from-file is set. +QUERY="select * from trades order by timestamp desc limit ${LIMIT}" + +# Download to a temp file first so a failed/partial fetch never clobbers a good CSV. +TMP="$(mktemp "${TMPDIR:-/tmp}/trades.XXXXXX.csv")" +trap 'rm -f "$TMP"' EXIT + +echo "Fetching ${LIMIT} rows from ${DEMO_HOST}/exp ..." +# -G + --data-urlencode builds the encoded query string; --fail turns HTTP errors +# into a non-zero exit; --retry rides out transient network blips. +curl --fail --show-error --silent --retry 3 --retry-delay 1 --retry-all-errors \ + -G "${DEMO_HOST}/exp" \ + --data-urlencode "query=${QUERY}" \ + -o "$TMP" + +# Sanity: a valid export has a header line plus data. +LINES="$(wc -l < "$TMP" | tr -d ' ')" +if [ "$LINES" -lt 2 ]; then + echo "ERROR: export returned only ${LINES} line(s); leaving ${OUT} untouched." >&2 + echo "Response was:" >&2 + head -c 500 "$TMP" >&2 + exit 1 +fi + +case "$OUT" in + *.gz) + gzip -c "$TMP" > "$OUT" + ;; + *) + cp "$TMP" "$OUT" + ;; +esac + +echo "Wrote ${OUT} ($((LINES - 1)) data rows, header included)." diff --git a/regenerate_csv_fx.sh b/regenerate_csv_fx.sh new file mode 100755 index 0000000..b5eb23a --- /dev/null +++ b/regenerate_csv_fx.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# +# Regenerate the replay CSV from the FX table on QuestDB's public demo instance. +# +# Companion to regenerate_csv.sh (which pulls the crypto `trades` table). This one +# pulls `fx_trades`, so you replay FX symbols (EUR-USD, ...) instead of crypto. +# +# We ALWAYS export from the demo box (https://demo.questdb.io): it is continuously +# ingesting, so every export is a fresh, recent-price snapshot. You then recreate +# the table locally from the CSV for internal demos. +# +# Two schema notes about fx_trades vs the sender's fixed `trades` schema: +# * Its size column is `quantity`, so the query aliases `quantity AS amount` to +# match the `amount` column the sender writes. +# * Its designated `timestamp` is TIMESTAMP_NS (nanoseconds), where the crypto +# `trades` table is micros. This only matters with --timestamp-from-file; see +# the README ("Nanosecond vs microsecond timestamps"). In the default replay +# mode the sender stamps now() and ignores the file timestamp entirely. +# +# Usage: +# ./regenerate_csv_fx.sh [output-file] [limit] +# +# output-file destination path (default: fx_trades.csv). If it ends in .gz the +# result is gzip-compressed (loadCsv auto-detects .gz). +# limit number of most-recent rows to fetch (default: 1000000). +# +# Env overrides: +# DEMO_HOST base URL (default: https://demo.questdb.io). Intentionally the +# demo box; override only if you mirror fx_trades elsewhere. +# +# Examples: +# ./regenerate_csv_fx.sh # -> fx_trades.csv, 1,000,000 rows +# ./regenerate_csv_fx.sh fx_trades.csv.gz # -> gzipped +# ./regenerate_csv_fx.sh fx_trades.csv 250000 # -> 250k rows +set -euo pipefail + +OUT="${1:-fx_trades.csv}" +LIMIT="${2:-1000000}" +DEMO_HOST="${DEMO_HOST:-https://demo.questdb.io}" + +# Explicit column list (not select *) so we can alias quantity -> amount and drop +# the fx-only columns (ecn, counterparty, order_id, ...) the sender does not read. +# The result has exactly symbol, side, price, amount, timestamp: the sender's schema. +QUERY="select timestamp, symbol, side, price, quantity as amount from fx_trades order by timestamp desc limit ${LIMIT}" + +# Download to a temp file first so a failed/partial fetch never clobbers a good CSV. +TMP="$(mktemp "${TMPDIR:-/tmp}/fx_trades.XXXXXX.csv")" +trap 'rm -f "$TMP"' EXIT + +echo "Fetching ${LIMIT} rows from ${DEMO_HOST}/exp (fx_trades) ..." +# -G + --data-urlencode builds the encoded query string; --fail turns HTTP errors +# into a non-zero exit; --retry rides out transient network blips. +curl --fail --show-error --silent --retry 3 --retry-delay 1 --retry-all-errors \ + -G "${DEMO_HOST}/exp" \ + --data-urlencode "query=${QUERY}" \ + -o "$TMP" + +# Sanity: a valid export has a header line plus data. +LINES="$(wc -l < "$TMP" | tr -d ' ')" +if [ "$LINES" -lt 2 ]; then + echo "ERROR: export returned only ${LINES} line(s); leaving ${OUT} untouched." >&2 + echo "Response was:" >&2 + head -c 500 "$TMP" >&2 + exit 1 +fi + +case "$OUT" in + *.gz) + gzip -c "$TMP" > "$OUT" + ;; + *) + cp "$TMP" "$OUT" + ;; +esac + +echo "Wrote ${OUT} ($((LINES - 1)) data rows, header included)." diff --git a/rust/README.md b/rust/README.md index bb8a2e2..8b927c3 100644 --- a/rust/README.md +++ b/rust/README.md @@ -67,7 +67,8 @@ a server-side `SELECT count() FROM trades` after a run. | `--token` | | Bearer token (QWP/WebSocket + ILP). UDP rejects auth. | | `--username` / `--password` | | Basic auth (QWP/WebSocket + ILP). | | `--total-events` | `1000000` | Rows across all workers. | -| `--delay-ms` | `50` | Per-row sleep. | +| `--delay-ms` | `50` | Per-row sleep. Ignored when `--rate > 0`. | +| `--rate` | `0` | Target **aggregate** rows/second across all workers (`0` = off). Takes precedence over `--delay-ms`; see [Pacing](#pacing-delay-ms-vs-rate). | | `--num-senders` | `10` | Worker threads (one `Sender` each). | | `--retry-timeout` | `360000` | `retry_timeout` (ILP) / `reconnect_max_duration_millis` (QWP). | | `--csv` | `./trades20250728.csv.gz` | `.csv` or `.csv.gz`; needs `symbol,side,price,amount[,timestamp]`. | @@ -82,11 +83,30 @@ a server-side `SELECT count() FROM trades` after a run. | `--enterprise` | `false` | Request durable acks (Enterprise only). | | `--zone` | `eu-west-1` | Preferred zone for the query client / probe. | +## Pacing: `--delay-ms` vs `--rate` + +- `--delay-ms` (default `50`): a fixed sleep after **every row**, per worker. A poor throttle + at speed — the fixed per-row cost caps throughput well under 1000 rows/s at `1`ms. +- `--rate` (default `0` = off): a **target aggregate rate in rows/second across all workers**. + Each worker paces to its share (`rate / num-senders`) against a deadline schedule, sending + rows back-to-back and sleeping only when it runs *ahead*. This reaches high targets a per-row + sleep never could (e.g. `--rate 300000`). When `> 0` it takes precedence over `--delay-ms` + (a warning prints if both are set). Measured: `--rate 5000`, 20k rows, 4 workers → ~5.0 s, + steady ~5000 rows/s. + ## Timestamps +The generator sends timestamps at **nanosecond** resolution (`TimestampNanos`), and QuestDB +stores them at the **target column's** resolution: a `TIMESTAMP_NS` column keeps full nanos, a +micros `TIMESTAMP` column truncates the extra digits (silently, never an error). **If the table +does not exist, the server auto-creates it as `TIMESTAMP_NS`.** Pre-create `trades` with a +`timestamp` (micros) column if you want micros. Verified: replaying the nanosecond FX chunk with +`--timestamp-from-file` preserved `...192508297Z` end-to-end in an auto-created `TIMESTAMP_NS` +table. + - **Single worker on a QWP transport** (`qwp` or `qwpudp`, `--num-senders 1`): each row is - stamped with the current microsecond client-side (monotonic, so no out-of-order), giving - distinct per-row timestamps. + stamped with the current time client-side (monotonic, so no out-of-order), giving distinct + per-row timestamps. - **ILP, or more than one worker**: rows use server-side `at_now()` (O3-safe across workers; a whole batch shares one timestamp). - `--timestamp-from-file` / `--seconds-offset` stamp each row explicitly from the CSV. @@ -131,6 +151,7 @@ Smoke-tested against a local QuestDB (OSS) with all rows accounted for server-si | `qwpudp` | 2000 rows, 1 worker | 2000 / 2000 | | `qwp` | 5000 rows, 1 worker | 5000 / 5000, distinct per-row timestamps; probe reported live timestamps + role | | `qwp` | 20000 rows, 4 workers | 20000 / 20000, one store-and-forward dir per worker | +| `qwp` | `--rate 5000`, nanos FX, `--timestamp-from-file`, 4 workers | steady ~5000 rows/s; auto-created `TIMESTAMP_NS` table preserved `...192508297Z` | ### QWP failover (primary crash) diff --git a/rust/src/main.rs b/rust/src/main.rs index ec9f9b5..4f445a9 100644 --- a/rust/src/main.rs +++ b/rust/src/main.rs @@ -23,7 +23,7 @@ use clap::Parser; use questdb::egress::column::ColumnView; use questdb::egress::reader::{BatchView, Reader}; use questdb::egress::FailoverEvent; -use questdb::ingress::{Sender, TimestampMicros}; +use questdb::ingress::{Sender, TimestampNanos}; const PROBE_QUERY: &str = "select timestamp from trades limit -1"; // Enterprise lifecycle status of whichever node the query client is connected to. Returns @@ -57,10 +57,16 @@ struct Args { #[arg(long, default_value_t = 1_000_000)] total_events: u64, - /// Per-row sleep in milliseconds (0 = as fast as possible). + /// Per-row sleep in milliseconds (0 = as fast as possible). Ignored when --rate > 0. #[arg(long, default_value_t = 50)] delay_ms: u64, + /// Target aggregate rows/second across ALL workers (0 = off, use --delay-ms). Takes + /// precedence over --delay-ms: each worker paces to rate/num-senders against a deadline + /// schedule, reaching high targets a per-row sleep cannot. + #[arg(long, default_value_t = 0)] + rate: u64, + /// Number of worker threads (each is its own Sender). #[arg(long, default_value_t = 10)] num_senders: usize, @@ -143,14 +149,14 @@ struct TradeRow { side: String, price: f64, amount: f64, - /// Microseconds since epoch, parsed at load time; only used with --timestamp-from-file. - ts_micros: i64, + /// Nanoseconds since epoch, parsed at load time; only used with --timestamp-from-file. + ts_nanos: i64, } -fn now_micros() -> i64 { +fn now_nanos() -> i64 { SystemTime::now() .duration_since(UNIX_EPOCH) - .map(|d| d.as_micros() as i64) + .map(|d| d.as_nanos() as i64) .unwrap_or(0) } @@ -177,6 +183,22 @@ fn main() { eprintln!("--batches-per-transaction must be > 0"); std::process::exit(2); } + // --rate takes precedence: when set it drives pacing and --delay-ms is ignored. + if args.rate > 0 && args.delay_ms > 0 { + eprintln!( + "[warn] --rate {} overrides --delay-ms {} (rate limiting drives pacing; the per-row \ + delay is ignored)", + args.rate, args.delay_ms + ); + } + println!( + "Pacing: {}", + if args.rate > 0 { + format!("rate={} rows/s (aggregate across {} workers)", args.rate, args.num_senders) + } else { + format!("delay-ms={}", args.delay_ms) + } + ); let addrs = args.addr_list(); if addrs.is_empty() { @@ -304,10 +326,10 @@ fn run_worker( let is_qwp = args.protocol == "qwp"; let is_udp = args.protocol == "qwpudp"; - // Single worker on a QWP transport: stamp each row with the current micros client-side. + // Single worker on a QWP transport: stamp each row with the current time (nanos) client-side. // A single thread is monotonic (no O3) and this avoids QWP's per-batch atNow() stamping. // ILP, or more than one worker, use atNow() (server-side, O3-safe). - let per_row_micros = (is_qwp || is_udp) && args.num_senders == 1; + let per_row_ts = (is_qwp || is_udp) && args.num_senders == 1; // Flush cadence: qwp commits every batch-size*batches-per-transaction rows; qwpudp flushes // every batch-size rows (bounds the datagram burst); ilp flushes every batch-size rows too // (the Rust HTTP client has no auto-flush, so the buffer must be drained explicitly). @@ -317,10 +339,22 @@ fn run_worker( args.batch_size }; + // Rate limiting (when --rate > 0): pace this worker to its share of the aggregate target, + // rate / num-senders rows/second. interval_nanos is the ideal spacing between this worker's + // rows; we track a deadline from loop entry and sleep only when running ahead, so rows go + // out back-to-back until we get ahead. Reaches high rates a fixed per-row sleep cannot. + let interval_nanos: f64 = if args.rate > 0 { + 1_000_000_000.0 * args.num_senders as f64 / args.rate as f64 + } else { + 0.0 + }; + let rate_limited = interval_nanos > 0.0; + let mut sender = Sender::from_conf(build_ingest_conf(args, worker_id))?; let mut buffer = sender.new_buffer(); let n = rows.len(); let mut sent: u64 = 0; + let pace_start = Instant::now(); for i in 0..total_events { let r = &rows[(i as usize) % n]; @@ -333,12 +367,12 @@ fn run_worker( .column_str("trade_id", format!("{worker_id}-{}", i + 1))?; if args.timestamp_from_file { - let ts = r.ts_micros + args.seconds_offset * 1_000_000; - buffer.at(TimestampMicros::new(ts))?; + let ts = r.ts_nanos + args.seconds_offset * 1_000_000_000; + buffer.at(TimestampNanos::new(ts))?; } else if args.seconds_offset != 0 { - buffer.at(TimestampMicros::new(now_micros() + args.seconds_offset * 1_000_000))?; - } else if per_row_micros { - buffer.at(TimestampMicros::new(now_micros()))?; + buffer.at(TimestampNanos::new(now_nanos() + args.seconds_offset * 1_000_000_000))?; + } else if per_row_ts { + buffer.at(TimestampNanos::new(now_nanos()))?; } else { buffer.at_now()?; } @@ -349,7 +383,18 @@ fn run_worker( if sent.is_multiple_of(commit_every) { sender.flush(&mut buffer)?; } - if args.delay_ms > 0 { + if rate_limited { + // Deadline for the row just sent (sent is 1-based). Sleep only while ahead of + // schedule; the 1ms floor coalesces many rows into one sleep at high rates. + let target_nanos = (sent as f64 * interval_nanos) as u64; + let elapsed_nanos = pace_start.elapsed().as_nanos() as u64; + if target_nanos > elapsed_nanos { + let sleep_nanos = target_nanos - elapsed_nanos; + if sleep_nanos > 1_000_000 { + thread::sleep(Duration::from_nanos(sleep_nanos)); + } + } + } else if args.delay_ms > 0 { thread::sleep(Duration::from_millis(args.delay_ms)); } } @@ -620,12 +665,16 @@ fn load_csv(path: &str, need_timestamp: bool) -> Result, Box { let raw = rec.get(i).unwrap_or("").trim(); + // Parse to nanoseconds so a TIMESTAMP_NS source keeps full precision; a micros + // target column truncates server-side. timestamp_nanos_opt is None only for + // dates outside ~1677-2262, which trade data never is. chrono::DateTime::parse_from_rfc3339(raw) - .map(|dt| dt.timestamp_micros()) .map_err(|e| format!("bad timestamp {raw:?}: {e}"))? + .timestamp_nanos_opt() + .ok_or_else(|| format!("timestamp out of nanosecond range: {raw:?}"))? } None => 0, }; @@ -634,7 +683,7 @@ fn load_csv(path: &str, need_timestamp: bool) -> Result, Box 0 it takes precedence over --delay-ms: + // each worker paces itself to its share (rate / num-senders) against a deadline schedule, + // so the process approximates the target regardless of worker count. Unlike a fixed + // per-row sleep, it sends rows back-to-back and only sleeps when ahead of schedule, so it + // can sustain high rates (e.g. 300000) that a --delay-ms of >=1 could never reach. + private static final long DEFAULT_RATE = 0L; private static final int DEFAULT_NUM_SENDERS = 10; private static final int DEFAULT_RETRY_TIMEOUT = 360000; private static final String DEFAULT_CSV = "./trades20250728.csv.gz"; @@ -80,6 +87,7 @@ public static void main(String[] args) throws Exception { final String password = a.get("--password"); // optional final long totalEvents = Long.parseLong(a.getOrDefault("--total-events", String.valueOf(DEFAULT_TOTAL_EVENTS))); final int delayMs = Integer.parseInt(a.getOrDefault("--delay-ms", String.valueOf(DEFAULT_DELAY_MS))); + final long rate = Long.parseLong(a.getOrDefault("--rate", String.valueOf(DEFAULT_RATE))); final int numSenders = Integer.parseInt(a.getOrDefault("--num-senders", String.valueOf(DEFAULT_NUM_SENDERS))); final int retryTimeout = Integer.parseInt(a.getOrDefault("--retry-timeout", String.valueOf(DEFAULT_RETRY_TIMEOUT))); final String csvPath = a.getOrDefault("--csv", DEFAULT_CSV); @@ -148,12 +156,25 @@ public static void main(String[] args) throws Exception { System.err.println("--total-events must be > 0"); System.exit(2); } + if (rate < 0) { + System.err.println("--rate must be >= 0 (0 disables rate limiting, falling back to --delay-ms)"); + System.exit(2); + } + // --rate takes precedence: when set it drives the pacing and --delay-ms is ignored. + if (rate > 0 && delayMs > 0 && a.containsKey("--delay-ms")) { + System.err.println("[warn] --rate " + rate + " overrides --delay-ms " + delayMs + + " (rate limiting drives pacing; the per-row delay is ignored)"); + } final SenderCfg cfg = new SenderCfg(protocol, addrsCsv, token, username, password, retryTimeout, senderIdBase, storeForwardDir, batchSize, batchesPerTransaction, numSenders, enterprise, zone, - connectTimeoutMs); + connectTimeoutMs, rate); + final String pacing = rate > 0 + ? "rate=" + rate + " rows/s (aggregate across " + numSenders + " workers)" + : "delay-ms=" + delayMs; final String conf = buildConf(addrsCsv, token, username, password, retryTimeout); + System.out.println("Pacing: " + pacing); System.out.println("Ingestion started. Protocol: " + protocol + (protocol.equals("qwp") ? " (WebSocket, sender-id=" + senderIdBase + ", store-and-forward=" + storeForwardDir @@ -259,6 +280,16 @@ private static void runWorker( : isUdp ? cfg.batchSize : 0L; + // Rate limiting (when --rate > 0): pace this worker to its share of the aggregate + // target, rate / num-senders rows/second. intervalNanos is the ideal spacing between + // this worker's rows; we track a deadline schedule from loop entry and only sleep when + // we are running ahead of it, so rows go out back-to-back until we get ahead. This + // reaches high rates that a fixed per-row Thread.sleep cannot. 0 => rate disabled. + final double intervalNanos = cfg.rate > 0 + ? 1_000_000_000.0 * cfg.numSenders / cfg.rate + : 0.0; + final boolean rateLimited = intervalNanos > 0.0; + final long paceStartNanos = System.nanoTime(); try ( Sender sender = buildSender(cfg, senderId)) { //( Sender sender = Sender.fromConfig(conf)) { final int n = rows.size(); for (long i = 0; i < totalEvents; i++) { @@ -279,11 +310,11 @@ private static void runWorker( if (secondsOffset != 0) { ts = ts.plusSeconds(secondsOffset); } - sender.at(ts); + atNanos(sender, ts); } else if (secondsOffset != 0) { - sender.at(Instant.now().plusSeconds(secondsOffset)); + atNanos(sender, Instant.now().plusSeconds(secondsOffset)); } else if (perRowMicros) { - sender.at(Instant.now()); + atNanos(sender, Instant.now()); } else { sender.atNow(); } @@ -296,7 +327,22 @@ private static void runWorker( sender.flush(); } - if (delayMs > 0) { + if (rateLimited) { + // Deadline for the row we have just sent (sent is 1-based here). Sleep only + // while ahead of schedule; if behind, fall through and keep sending. The 1ms + // floor coalesces many rows into one sleep so we do not burn CPU sleeping for + // sub-millisecond slivers at high rates. + final long targetNanos = paceStartNanos + Math.round(sent * intervalNanos); + final long sleepNanos = targetNanos - System.nanoTime(); + if (sleepNanos > 1_000_000L) { + try { + Thread.sleep(sleepNanos / 1_000_000L, (int) (sleepNanos % 1_000_000L)); + } catch (InterruptedException ie) { + Thread.currentThread().interrupt(); + throw new RuntimeException("Interrupted", ie); + } + } + } else if (delayMs > 0) { try { Thread.sleep(delayMs); } catch (InterruptedException ie) { @@ -314,6 +360,17 @@ private static void runWorker( } } + // Send the designated timestamp at NANOSECOND resolution. The client's at(Instant) path + // delivers only microseconds (sub-microsecond digits are dropped before the wire), so we + // convert to epoch-nanos and use at(long, NANOS) to carry full nanosecond precision. + // QuestDB stores at the target column's resolution: a micros TIMESTAMP column silently + // truncates the extra digits, a TIMESTAMP_NS column keeps them. The multiply stays within + // long range for any realistic date (epochSecond * 1e9 overflows only past year ~2262). + private static void atNanos(Sender sender, Instant ts) { + final long epochNanos = ts.getEpochSecond() * 1_000_000_000L + ts.getNano(); + sender.at(epochNanos, ChronoUnit.NANOS); + } + private static List loadCsv(String path, boolean needTimestamp) throws Exception { List out = new ArrayList<>(1024); try (InputStream in0 = Files.newInputStream(Path.of(path)); @@ -834,6 +891,7 @@ private static Map parseArgs(String[] args) { case "--password": case "--total-events": case "--delay-ms": + case "--rate": case "--num-senders": case "--csv": case "--timestamp-from-file": @@ -885,11 +943,13 @@ private static final class SenderCfg { final boolean enterprise; final String zone; final int connectTimeoutMs; + // Target aggregate rows/second across all workers; 0 = disabled (use delayMs). + final long rate; SenderCfg(String protocol, String addrsCsv, String token, String username, String password, int retryTimeout, String senderIdBase, String storeForwardDir, int batchSize, int batchesPerTransaction, int numSenders, boolean enterprise, String zone, - int connectTimeoutMs) { + int connectTimeoutMs, long rate) { this.protocol = protocol; this.addrsCsv = addrsCsv; this.token = token; @@ -904,6 +964,7 @@ private static final class SenderCfg { this.enterprise = enterprise; this.zone = zone; this.connectTimeoutMs = connectTimeoutMs; + this.rate = rate; } } } From efa1b8a5948392d058797cd320a13be66da5f8fc Mon Sep 17 00:00:00 2001 From: javier Date: Tue, 14 Jul 2026 17:57:43 +0200 Subject: [PATCH 15/79] Add chunked polars enrich demo (streaming read, enterprise auth) Stream a table through polars, add a random enriched_rnd SYMBOL column, and write it back to enriched__demo. Reads via QueryResult.iter_arrow() one Arrow batch at a time (peak memory is a single batch, not the whole result), enriches each chunk, and ingests it via Client.dataframe. Reader and writer use separate QWP connections. Supports token/basic auth and TLS (qwpwss) for Enterprise, applied to both the QWP path and the HTTP drop/verify calls. --- python/enrich_polars_demo.py | 196 +++++++++++++++++++++++++++++++++++ 1 file changed, 196 insertions(+) create mode 100644 python/enrich_polars_demo.py diff --git a/python/enrich_polars_demo.py b/python/enrich_polars_demo.py new file mode 100644 index 0000000..7e55db2 --- /dev/null +++ b/python/enrich_polars_demo.py @@ -0,0 +1,196 @@ +#!/usr/bin/env python3 +"""Stream a table through polars, enrich it, and write it back - chunked. + +Memory-efficient polars round-trip over the QWP query client. Rather than +materializing the whole result set, it streams the read in Arrow record batches +and ingests each enriched batch before pulling the next, so peak memory is one +batch (not all ``--limit`` rows): + + * READ - ``Client.query(sql).iter_arrow()`` yields ``pyarrow.RecordBatch`` chunks + one at a time from a server-side cursor (verified streaming in the client: + ``iter_arrow`` pulls via ``line_reader_cursor_next_batch``; ``to_polars`` by + contrast is "materialise-whole" and would buffer every row in RAM); + ``pl.from_arrow(batch)`` wraps each into a polars DataFrame (zero-copy). + * ENRICH - add an ``enriched_rnd`` column: a random ``A``-``Z`` value per row. + * WRITE - ``Client.dataframe(chunk, ...)`` ingests each polars chunk into + ``enriched_
_demo`` over the QWP Arrow-columnar path. + +Both the read and the write are polars DataFrames. No pandas in the round-trip. +The read and the write use separate ``Client`` connections so the open read +stream and the ingest never share a socket. + +Column typing is automatic: QuestDB SYMBOL columns come back as polars ``Categorical`` +and VARCHAR as ``String``, and ``Client.dataframe``'s default ``symbols='auto'`` maps +``Categorical -> SYMBOL`` and ``String -> VARCHAR`` on the way back in. ``enriched_rnd`` +is built as a ``Categorical`` so it too lands as a SYMBOL. + +Auth / TLS (QuestDB Enterprise): pass ``--token`` (bearer) or ``--username``/``--password`` +(basic). Either turns on TLS automatically (scheme ``qwpwss``); ``--tls`` forces TLS with +no auth. Certificate verification is on by default; use ``--tls-verify unsafe_off`` for +self-signed certs. The same credentials are applied to the HTTP ``/exec`` calls used for +drop/verify. + +Usage: + python enrich_polars_demo.py TABLE \ + [--addr host:9000] [--limit 200000000] [--timestamp-col timestamp] \ + [--seed 42] [--keep] \ + [--token TOK | --username U --password P] [--tls] [--tls-verify on|unsafe_off] + +By default the target table is dropped and recreated on each run; pass ``--keep`` +to append instead. ``--limit`` defaults to the latest 200 million rows by designated +timestamp, streamed ascending so re-ingest stays in timestamp order. Requires the +QWP/egress build of the client (see README.md). +""" + +import argparse +import base64 +import json +import ssl +import sys +import time +import urllib.parse +import urllib.request + +import numpy as np +import polars as pl + +from questdb.ingress import Client + +LETTERS = np.array(list("ABCDEFGHIJKLMNOPQRSTUVWXYZ")) + + +def use_tls(args): + return bool(args.tls or args.token or (args.username and args.password)) + + +def build_conf(args): + """QWP connect string for the reader/writer, with optional auth + TLS.""" + scheme = "qwpwss" if use_tls(args) else "qwpws" + parts = [f"{scheme}::addr={args.addr};"] + if args.token: + parts.append(f"token={args.token};") + elif args.username and args.password: + parts.append(f"username={args.username};password={args.password};") + if use_tls(args) and args.tls_verify == "unsafe_off": + parts.append("tls_verify=unsafe_off;") + return "".join(parts) + + +def exec_sql(args, sql): + """Run a statement over the HTTP(S) /exec endpoint (auth-aware), return JSON.""" + scheme = "https" if use_tls(args) else "http" + host = args.addr.split(",")[0] + url = f"{scheme}://{host}/exec?" + urllib.parse.urlencode({"query": sql}) + req = urllib.request.Request(url) + if args.token: + req.add_header("Authorization", f"Bearer {args.token}") + elif args.username and args.password: + cred = base64.b64encode(f"{args.username}:{args.password}".encode()).decode() + req.add_header("Authorization", f"Basic {cred}") + ctx = None + if use_tls(args) and args.tls_verify == "unsafe_off": + ctx = ssl._create_unverified_context() + with urllib.request.urlopen(req, timeout=60, context=ctx) as resp: + return json.load(resp) + + +def wait_for_count(args, table, at_least, timeout_s=60): + """Poll until the table holds >= at_least rows (QWP commits asynchronously).""" + deadline = time.monotonic() + timeout_s + while time.monotonic() < deadline: + try: + n = int(exec_sql(args, f"select count() from {table}")["dataset"][0][0]) + if n >= at_least: + return n + except Exception: + pass + time.sleep(0.25) + return -1 + + +def enrich(chunk, rng): + """Add an `enriched_rnd` column of random A-Z letters (polars Categorical, so + it lands as a QuestDB SYMBOL). `rng` carries state across chunks so the random + values are not identical batch-to-batch.""" + rnd = LETTERS[rng.integers(0, 26, size=chunk.height)] + return chunk.with_columns(pl.Series("enriched_rnd", rnd, dtype=pl.Categorical)) + + +def main(argv): + ap = argparse.ArgumentParser(description="Chunked polars read -> enrich -> write") + ap.add_argument("table", help="source table name") + ap.add_argument("--addr", default="localhost:9000", help="host:port (QWP/HTTP port)") + ap.add_argument("--limit", type=int, default=200_000_000, + help="max rows to read (latest N by timestamp); default 200,000,000") + ap.add_argument("--timestamp-col", default="timestamp", + help="designated timestamp column of the source table") + ap.add_argument("--seed", type=int, default=42, help="RNG seed for enriched_rnd") + ap.add_argument("--keep", action="store_true", + help="append to the target instead of dropping it first") + # Enterprise auth / TLS + ap.add_argument("--token", default=None, help="bearer token (turns on TLS)") + ap.add_argument("--username", default=None, help="basic-auth username (turns on TLS)") + ap.add_argument("--password", default=None, help="basic-auth password") + ap.add_argument("--tls", action="store_true", help="force TLS (qwpwss) with no auth") + ap.add_argument("--tls-verify", choices=["on", "unsafe_off"], default="on", + help="certificate verification; use unsafe_off for self-signed") + args = ap.parse_args(argv) + + ts_col = args.timestamp_col + target = f"enriched_{args.table}_demo" + rng = np.random.default_rng(args.seed) + conf = build_conf(args) + print(f"[conf] {conf.split('::')[0]} tls={use_tls(args)} " + f"auth={'token' if args.token else 'basic' if args.username else 'none'}") + + if not args.keep: + try: + exec_sql(args, f"drop table if exists {target}") + except Exception as e: + print(f"[warn] could not drop '{target}' via /exec ({e}); " + f"continuing (rows will append)", file=sys.stderr) + + # `limit -N` returns the latest N rows in designated-timestamp (ascending) + # order already, so the stream re-ingests in order with no explicit sort. + sql = f"select * from {args.table} limit -{args.limit}" + + total = 0 + t0 = time.monotonic() + # Separate connections: the read stream stays open while each chunk is written. + with Client.from_conf(conf) as reader, Client.from_conf(conf) as writer: + result = reader.query(sql) + for i, batch in enumerate(result.iter_arrow()): + chunk = pl.from_arrow(batch) # RecordBatch -> polars, zero-copy + if chunk.height == 0: + continue + chunk = enrich(chunk, rng) + writer.dataframe(chunk, table_name=target, at=ts_col) + total += chunk.height + print(f"[chunk {i}] {chunk.height:,} rows enriched + written " + f"(running total {total:,})") + + if total == 0: + print("[read] source table is empty - nothing to enrich", file=sys.stderr) + return 1 + print(f"[done] streamed {total:,} rows -> '{target}' in " + f"{time.monotonic() - t0:.1f}s (peak memory ~one batch)") + + # --- VERIFY server-side (QWP applies the commit asynchronously) --- + expected = total if not args.keep else 0 + n = wait_for_count(args, target, expected) + print(f"[verify] '{target}' now holds {n:,} rows") + try: + dist = exec_sql( + args, + f"select enriched_rnd, count() n from {target} " + f"group by enriched_rnd order by enriched_rnd" + )["dataset"] + print(f"[verify] enriched_rnd distinct values: {len(dist)} (sample: {dist[:5]})") + except Exception as e: + print(f"[warn] verify query failed: {e}", file=sys.stderr) + + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) From 7d0d7c76c3d1e688b4d11939929ed7ebfb953f0a Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 15 Jul 2026 02:15:57 +0200 Subject: [PATCH 16/79] Add fast columnar sender, parallel read benchmark, and TCP tuning csv_columnar_sender.py: high-throughput Python ingestion over the columnar QWP path (Client.dataframe with polars), replacing row-by-row for volume. Streams the CSV in bounded chunks (flat memory), with --rate pacing, --num-senders, and enterprise auth/TLS. Row-by-row csv_parallel_sender.py stays for HA/failover demos (store-and-forward, probe, UDP/ILP), which the columnar path bypasses. read_bench.py: reads the last N rows as fast as possible via streaming iter_arrow() and reports rows/s plus MB/s and Gb/s (decoded Arrow payload). --readers splits the scan across N parallel connections by timestamp to get past the per-connection socket-buffer cap. boost_tcp.sh: raises net.core.wmem_max/rmem_max so the client's 4 MiB socket buffer is not clamped to ~416 KB (the per-connection throughput cap on real networks). Safe to source or exec. README: 'which script to use' guidance (columnar vs row-by-row) and network tuning. --- boost_tcp.sh | 26 ++++ python/README.md | 46 +++++++ python/csv_columnar_sender.py | 246 ++++++++++++++++++++++++++++++++++ python/read_bench.py | 208 ++++++++++++++++++++++++++++ 4 files changed, 526 insertions(+) create mode 100755 boost_tcp.sh create mode 100644 python/csv_columnar_sender.py create mode 100644 python/read_bench.py diff --git a/boost_tcp.sh b/boost_tcp.sh new file mode 100755 index 0000000..4355344 --- /dev/null +++ b/boost_tcp.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# boost_tcp.sh - lift the socket-buffer clamp for QuestDB QWP ingestion. +# +# The Rust/C/Python QWP clients hardcode a 4 MiB socket buffer, which default +# Linux silently clamps to ~416 KB (net.core.wmem_max) AND disables autotuning, +# pinning each connection at ~426 KB per RTT. On a real network this caps +# throughput hard (e.g. ~210k rows/s at 20 ms RTT). Raising the max buffers lets +# the client's 4 MiB request through. Apply on BOTH the sender and server boxes. +# +# SAFE any way you run it: `sudo bash boost_tcp.sh`, `./boost_tcp.sh`, +# `. boost_tcp.sh`, `source boost_tcp.sh`. No set -e, no exec, no exit -> it can +# never kill or exit your shell. Runtime-only: values reset on reboot, so run it +# once per boot (or persist via /etc/sysctl.d/ - see README). +boost_tcp() { + local S="" + [ "$(id -u)" -ne 0 ] && S="sudo" + $S sysctl -w net.core.rmem_max=8388608 + $S sysctl -w net.core.wmem_max=8388608 + $S sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216" + $S sysctl -w net.ipv4.tcp_rmem="4096 65536 16777216" + $S sysctl -w net.core.somaxconn=32768 + $S sysctl -w net.core.netdev_max_backlog=65536 + echo "boost_tcp: applied (holds until reboot)" + $S sysctl net.core.wmem_max net.core.rmem_max +} +boost_tcp diff --git a/python/README.md b/python/README.md index 147f7a1..d109053 100644 --- a/python/README.md +++ b/python/README.md @@ -8,6 +8,52 @@ timestamp and the serving node's live role. Built on the QuestDB Python client (`questdb.ingress`), which wraps the same C/Rust client as the other ports. +## Which script to use + +This port has grown a few scripts. Pick by what you need: + +| Script | Use it for | Path / speed | +| --- | --- | --- | +| `csv_columnar_sender.py` | **High-throughput ingestion** - bulk CSV replay at volume | Columnar QWP (`Client.dataframe`, Arrow/polars). Fastest Python path; flat memory. | +| `csv_parallel_sender.py` | **HA / failover demos** - store-and-forward durability, the live probe, `qwpudp`/`ilp` comparison, and the per-event `row()` API | Row-by-row QWP/UDP/ILP. Correct but slow for volume - keep `--total-events` and the batch window small. | +| `read_bench.py` | Measuring read/scan throughput (rows/s) of a table | Streaming `iter_arrow()` egress. | +| `enrich_polars_demo.py` | Read a table as polars, enrich, write it back as polars | Streaming read + columnar write. | +| `dataframe_demo.py` | pandas/polars ingestion + egress round-trip showcase | `Sender.dataframe` / `Client.dataframe`. | + +**Row-by-row vs columnar.** `sender.row()` in a Python loop is the *slowest* path: every cell +crosses the Python/Cython boundary under the GIL (the client's own perf notes call it "~16x +slower" than the columnar bulk path). `Client.dataframe` ships whole Arrow columns to the +native client - the only Python path that reaches Java/Rust-class throughput. So use +`csv_columnar_sender.py` for volume, and reach for `csv_parallel_sender.py` **only** when you +need store-and-forward failover, which the columnar path deliberately bypasses (dataframe +ingestion uses the direct column sender, not store-and-forward). Do not push high volume +(e.g. 100M rows) through the row-by-row sender - its per-row Python cost plus a large flush +window will pin the buffer in memory and can OOM the host. + +```bash +# Fast columnar ingestion (recommended for volume): +python csv_columnar_sender.py \ + --addrs host:9000 --total-events 100000000 \ + --num-senders 2 --chunk-rows 100000 --csv ../trades.csv \ + --token "$QDB_TOKEN" --tls-verify unsafe_off +``` + +## Network tuning for throughput (important on real networks) + +The Rust/C/Python QWP clients hardcode a 4 MiB socket buffer, which default Linux silently +clamps to ~416 KB and pins each connection at ~426 KB **per RTT** - a hard throughput cap on +a real network (e.g. ~210k rows/s at 20 ms RTT, while Java, which does not touch socket +buffers, does ~52 MB/s). Two levers: + +- Run [`boost_tcp.sh`](../boost_tcp.sh) on **both** the sender and server hosts to raise + `net.core.wmem_max`/`rmem_max` (and friends) so the client's 4 MiB buffer request actually + goes through instead of being clamped. It is runtime-only (resets on reboot); persist via + `/etc/sysctl.d/` if you want it permanent. +- Prefer **multiple senders** (`--num-senders`): the cap is per connection, so parallel + connections multiply throughput (~11M rows/s with 2 connections same-zone EC2 in the + client team's tests, vs ~4M with 1). The tuning affects egress too, so it also speeds up + `read_bench.py` on a high-RTT link. + ## Important: the client must be built from source The QWP transports (`qwpws`/`qwpwss`/`qwpudp`) **and the query client / egress reader are not diff --git a/python/csv_columnar_sender.py b/python/csv_columnar_sender.py new file mode 100644 index 0000000..aebfb51 --- /dev/null +++ b/python/csv_columnar_sender.py @@ -0,0 +1,246 @@ +#!/usr/bin/env python3 +"""Fast CSV replay sender for QuestDB - the COLUMNAR QWP path. + +This is the throughput-oriented Python sender. It replays a CSV to a QuestDB +``trades`` table over the QWP WebSocket transport using the *column-major* client +path (``Client.dataframe``), which ships whole Arrow columns to the native Rust +client in bulk. That is the ONLY Python path that gets near Java/Rust throughput. + +Why not row-by-row? ``Sender.row(...)`` in a Python loop is the SLOWEST path: every +cell crosses the Python/Cython boundary under the GIL. The client's own perf notes +call the row API a "per-cell call ... ~16x slower" than the columnar bulk path. +The column path (``Client.dataframe``) stores raw pointers into the Arrow buffers +("no data copy at append time") and does one memcpy per column at flush. Use it. + +Memory stays flat: the CSV is loaded once into a polars DataFrame, and rows are +sent in bounded, zero-copy slices. ``Client.dataframe`` additionally streams each +send internally at ``max_rows_per_batch`` (16384) rows, so nothing balloons - unlike +the row-by-row sender, which buffered up to 100k rows before its first flush and +grew without bound when Python could not reach that threshold fast enough. + +Timestamps: + * ``--timestamp-from-file`` uses the CSV's ``timestamp`` column (parsed to ns). + * otherwise each batch is stamped with a strictly-increasing "now" (nanoseconds), + so the target looks live and there is no out-of-order within a worker. + +Auth / TLS (QuestDB Enterprise): ``--token`` (bearer) or ``--username``/``--password`` +(basic) turn on TLS automatically (scheme ``qwpwss``); ``--tls`` forces TLS with no +auth. Certificate verification is on by default; ``--tls-verify unsafe_off`` for +self-signed certs. Multiple ``--addrs`` give QWP failover. + +Usage: + python csv_columnar_sender.py \ + --addrs host:9000[,host2:9000] \ + --total-events 100000000 \ + --num-senders 1 \ + --chunk-rows 100000 \ + --rate 0 \ + --csv ../trades.csv \ + [--timestamp-from-file] \ + [--token TOK | --username U --password P] [--tls-verify on|unsafe_off] + +Requires the QWP/egress build of the client (see README.md). +""" + +import argparse +import gzip +import io +import os +import sys +import threading +import time + +import numpy as np +import polars as pl + +from questdb.ingress import Client + +TABLE = "trades" + + +def use_tls(args): + return bool(args.tls or args.token or (args.username and args.password)) + + +def build_conf(args): + """QWP connect string for the columnar client, with optional auth + TLS + failover.""" + scheme = "qwpwss" if use_tls(args) else "qwpws" + addrs = ",".join(a.strip() for a in args.addrs.split(",") if a.strip()) + parts = [f"{scheme}::addr={addrs};"] + if args.token: + parts.append(f"token={args.token};") + elif args.username and args.password: + parts.append(f"username={args.username};password={args.password};") + if use_tls(args) and args.tls_verify == "unsafe_off": + parts.append("tls_verify=unsafe_off;") + return "".join(parts) + + +def load_base(path, need_timestamp): + """Load the CSV once into a compact polars DataFrame. symbol/side become + Categorical (-> SYMBOL), price/amount Float64, and (if needed) timestamp is + parsed to nanosecond Datetime. trade_id is synthesised per batch at send time.""" + if path.endswith(".gz"): + with gzip.open(path, "rb") as fh: + df = pl.read_csv(io.BytesIO(fh.read())) + else: + df = pl.read_csv(path) + for col in ("symbol", "side", "price", "amount"): + if col not in df.columns: + raise ValueError(f"CSV missing required column: {col}") + exprs = [ + pl.col("symbol").cast(pl.Utf8).cast(pl.Categorical), + pl.col("side").cast(pl.Utf8).cast(pl.Categorical), + pl.col("price").cast(pl.Float64), + pl.col("amount").cast(pl.Float64), + ] + if need_timestamp: + if "timestamp" not in df.columns: + raise ValueError("CSV missing required column: timestamp") + exprs.append( + pl.col("timestamp").cast(pl.Utf8).str.to_datetime(time_unit="ns").alias("timestamp") + ) + return df.select(exprs) + + +def run_worker(wid, events, base, args, counts): + """Send `events` rows for this worker: cycle over the base frame in bounded + slices, attach trade_id (+ a live timestamp unless reading it from file), and + ship each slice columnar via Client.dataframe.""" + m = base.height + from_file = args.timestamp_from_file + conf = build_conf(args) + + # Rate pacing (aggregate rows/s across all workers): deadline schedule, sleep + # only when ahead. interval = ns per row for this worker's share. + interval_ns = (1_000_000_000.0 * args.num_senders / args.rate) if args.rate > 0 else 0.0 + + sent = 0 + pos = 0 + last_ts = 0 + pace_start = time.perf_counter_ns() + with Client.from_conf(conf) as client: + while sent < events: + n = min(args.chunk_rows, events - sent, m - pos) + sl = base.slice(pos, n) # zero-copy view into the base frame + + # trade_id, vectorised (native polars, no Python per-row loop). + add = [(pl.lit(f"{wid}-") + pl.int_range(sent, sent + n).cast(pl.Utf8)).alias("trade_id")] + if not from_file: + base_ns = max(time.time_ns(), last_ts + 1) + ts = np.arange(n, dtype=np.int64) + base_ns # strictly increasing -> no O3 + last_ts = int(ts[-1]) + add.append(pl.Series("timestamp", ts).cast(pl.Datetime("ns", "UTC")).alias("timestamp")) + chunk = sl.with_columns(add) + + client.dataframe(chunk, table_name=TABLE, symbols=["symbol", "side"], at="timestamp") + + sent += n + counts[wid] = sent + pos = (pos + n) % m + + if interval_ns > 0.0: + target = pace_start + int(sent * interval_ns) + sleep_ns = target - (time.perf_counter_ns() - pace_start) + if sleep_ns > 1_000_000: + time.sleep(sleep_ns / 1_000_000_000) + print(f"Sender {wid} finished sending {sent} events") + + +def main(argv): + ap = argparse.ArgumentParser(description="Fast columnar CSV replay sender for QuestDB") + ap.add_argument("--addrs", default="localhost:9000", + help="comma-separated host:port (QWP/WebSocket port, usually :9000)") + ap.add_argument("--total-events", type=int, default=1_000_000) + ap.add_argument("--num-senders", type=int, default=1, + help="worker threads, each with its own columnar client") + ap.add_argument("--chunk-rows", type=int, default=100_000, + help="rows per Client.dataframe call (pacing granularity + peak-memory bound)") + ap.add_argument("--rate", type=int, default=0, + help="target aggregate rows/second across ALL workers (0 = flat out)") + ap.add_argument("--csv", default="../trades20250728.csv.gz") + ap.add_argument("--timestamp-from-file", action="store_true", + help="use the CSV timestamp column instead of a live 'now' stamp") + # Enterprise auth / TLS + ap.add_argument("--token", default=None, help="bearer token (turns on TLS)") + ap.add_argument("--username", default=None, help="basic-auth username (turns on TLS)") + ap.add_argument("--password", default=None, help="basic-auth password") + ap.add_argument("--tls", action="store_true", help="force TLS (qwpwss) with no auth") + ap.add_argument("--tls-verify", choices=["on", "unsafe_off"], default="on", + help="certificate verification; use unsafe_off for self-signed") + args = ap.parse_args(argv) + + for name, val in (("--total-events", args.total_events), ("--num-senders", args.num_senders), + ("--chunk-rows", args.chunk_rows)): + if val <= 0: + print(f"{name} must be > 0", file=sys.stderr) + return 2 + if args.rate < 0: + print("--rate must be >= 0", file=sys.stderr) + return 2 + if not os.path.exists(args.csv): + print(f"CSV file not found: {args.csv}", file=sys.stderr) + return 2 + + base = load_base(args.csv, args.timestamp_from_file) + if base.height == 0: + print("CSV has no data rows.", file=sys.stderr) + return 2 + + print(f"[conf] {build_conf(args).split('::')[0]} tls={use_tls(args)} " + f"auth={'token' if args.token else 'basic' if args.username else 'none'} " + f"| addrs: {args.addrs}") + print(f"Ingestion started (columnar QWP). base={base.height:,} rows, " + f"total={args.total_events:,}, workers={args.num_senders}, chunk-rows={args.chunk_rows:,}, " + f"timestamps={'file' if args.timestamp_from_file else 'live-now'}, " + f"pacing={'rate ' + str(args.rate) + ' rows/s' if args.rate else 'flat out'}") + + counts = [0] * args.num_senders + stop = threading.Event() + start = time.monotonic() + + def reporter(): + last = 0 + while not stop.is_set(): + stop.wait(1.0) + now = sum(counts) + print(f"[progress] sent={now:,} rate={now - last:,} rows/s") + last = now + + rep = threading.Thread(target=reporter, daemon=True) + rep.start() + + base_events = args.total_events // args.num_senders + rem = args.total_events % args.num_senders + errors = [] + + def wrapper(wid, ev): + try: + run_worker(wid, ev, base, args, counts) + except Exception as e: # noqa: BLE001 + errors.append(f"Sender {wid}: {e}") + + threads = [] + for wid in range(args.num_senders): + ev = base_events + (1 if wid < rem else 0) + t = threading.Thread(target=wrapper, args=(wid, ev)) + t.start() + threads.append(t) + for t in threads: + t.join() + stop.set() + + if errors: + for e in errors: + print(f"Worker failed: {e}", file=sys.stderr) + return 1 + + elapsed = time.monotonic() - start + rate = args.total_events / elapsed if elapsed > 0 else 0 + print(f"All workers completed. events={args.total_events:,} elapsed={elapsed:.3f}s " + f"throughput={rate:,.0f} rows/s") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/python/read_bench.py b/python/read_bench.py new file mode 100644 index 0000000..b6fcbbe --- /dev/null +++ b/python/read_bench.py @@ -0,0 +1,208 @@ +#!/usr/bin/env python3 +"""Read the last N rows of a table as fast as Python allows, reporting rows/sec and +throughput (MB/s, Gb/s) live. + +Throughput is measured as the decoded Arrow payload (``batch.nbytes``), i.e. the data +volume handed to the reader, not the compressed QWP wire bytes (which the client does +not expose) - so it is a data-throughput figure, generally larger than the on-wire rate. + +Fastest path, by design: consume the QWP query cursor's Arrow batches directly +(``Client.query(sql).iter_arrow()``) and just tally ``batch.num_rows``. No pandas +or polars conversion - the Rust client already decodes the wire into Arrow inside +``iter_arrow`` (that decode is the unavoidable "read" work, done in native code), +so the Python loop does nothing but add row counts. Converting each batch to +polars (``pl.from_arrow``) or pandas (``iter_pandas``) would only add a per-batch +materialisation step on top and measure a slower number. (``to_arrow``/``to_polars`` +are "materialise-whole" - they buffer the entire result; ``iter_arrow`` streams.) + +A single connection is capped by the client's socket buffer at ~426 KB per RTT (the +QWP client hardcodes a 4 MiB buffer that default Linux clamps; see ../boost_tcp.sh), +so on a real network one reader is round-trip bound no matter how lean the loop is. +``--readers N`` splits the row range across N parallel connections (each scans a +timestamp slice) and reports the aggregate rows/sec, which is the way past the +per-connection wall. ``--readers 1`` (default) is the plain single stream. + +Auth / TLS (QuestDB Enterprise): pass ``--token`` (bearer) or ``--username``/``--password`` +(basic). Either turns on TLS automatically (scheme ``qwpwss``); ``--tls`` forces TLS with +no auth. Certificate verification is on by default; use ``--tls-verify unsafe_off`` for +self-signed certs. + +Usage: + python read_bench.py TABLE \ + [--addr host:9000] [--limit 10000000] [--readers 1] \ + [--token TOK | --username U --password P] [--tls] [--tls-verify on|unsafe_off] + +Requires the QWP/egress build of the client (see README.md). +""" + +import argparse +import sys +import threading +import time +from datetime import datetime, timezone + +from questdb.ingress import Client + + +def use_tls(args): + return bool(args.tls or args.token or (args.username and args.password)) + + +def build_conf(args): + """QWP connect string for the reader(s), with optional auth + TLS.""" + scheme = "qwpwss" if use_tls(args) else "qwpws" + parts = [f"{scheme}::addr={args.addr};"] + if args.token: + parts.append(f"token={args.token};") + elif args.username and args.password: + parts.append(f"username={args.username};password={args.password};") + if use_tls(args) and args.tls_verify == "unsafe_off": + parts.append("tls_verify=unsafe_off;") + return "".join(parts) + + +def ns_to_iso(ns): + """Epoch nanoseconds -> ISO-8601 string with nanosecond precision (unit-safe + for both TIMESTAMP and TIMESTAMP_NS columns in a SQL literal).""" + sec, frac = divmod(int(ns), 1_000_000_000) + dt = datetime.fromtimestamp(sec, tz=timezone.utc) + return dt.strftime("%Y-%m-%dT%H:%M:%S") + f".{frac:09d}Z" + + +def split_queries(args, conf): + """Build one SELECT per reader. For a single reader, the exact `limit -N` + query; for N readers, split the last-N rows' timestamp range into N slices.""" + table, ts, limit, readers = args.table, args.timestamp_col, args.limit, args.readers + if readers <= 1: + return [f"select * from {table} limit -{limit}"] + # Range of the latest `limit` rows, as epoch nanoseconds. + with Client.from_conf(conf) as c: + mm = c.query( + f"select min({ts}) lo, max({ts}) hi " + f"from (select {ts} from {table} limit -{limit})" + ).to_polars() + if mm.height == 0 or mm["lo"][0] is None: + return [f"select * from {table} limit -{limit}"] + lo = mm["lo"].dt.epoch("ns")[0] + hi = mm["hi"].dt.epoch("ns")[0] + if hi <= lo: # single instant / one row - can't split by time + print("[warn] timestamp range too narrow to split; using 1 reader", file=sys.stderr) + return [f"select * from {table} limit -{limit}"] + span = hi - lo + edges = [lo + (span * i) // readers for i in range(readers + 1)] + edges[-1] = hi + sqls = [] + for i in range(readers): + a = ns_to_iso(edges[i]) + b = ns_to_iso(edges[i + 1]) + op = "<=" if i == readers - 1 else "<" # last slice includes the max row + sqls.append(f"select * from {table} where {ts} >= '{a}' and {ts} {op} '{b}'") + return sqls + + +def run_reader(idx, sql, conf, counts, byts, errors): + try: + with Client.from_conf(conf) as client: + result = client.query(sql) + n = 0 + b = 0 + for batch in result.iter_arrow(): + n += batch.num_rows # count rows and decoded Arrow bytes + b += batch.nbytes # (buffer bytes of this batch's columns) + counts[idx] = n + byts[idx] = b + counts[idx] = n + byts[idx] = b + except Exception as e: # noqa: BLE001 + errors.append(f"reader {idx}: {e}") + + +def main(argv): + ap = argparse.ArgumentParser(description="Read last N rows as fast as possible, report rows/sec") + ap.add_argument("table", help="source table name") + ap.add_argument("--addr", default="localhost:9000", help="host:port (QWP/HTTP port)") + ap.add_argument("--limit", type=int, default=10_000_000, + help="number of most-recent rows to read; default 10,000,000") + ap.add_argument("--readers", type=int, default=1, + help="parallel reader connections (split by timestamp); default 1") + ap.add_argument("--timestamp-col", default="timestamp", + help="designated timestamp column (used to split across readers)") + ap.add_argument("--report-interval", type=float, default=0.5, + help="seconds between progress lines; default 0.5") + # Enterprise auth / TLS + ap.add_argument("--token", default=None, help="bearer token (turns on TLS)") + ap.add_argument("--username", default=None, help="basic-auth username (turns on TLS)") + ap.add_argument("--password", default=None, help="basic-auth password") + ap.add_argument("--tls", action="store_true", help="force TLS (qwpwss) with no auth") + ap.add_argument("--tls-verify", choices=["on", "unsafe_off"], default="on", + help="certificate verification; use unsafe_off for self-signed") + args = ap.parse_args(argv) + + if args.readers < 1: + print("--readers must be >= 1", file=sys.stderr) + return 2 + + conf = build_conf(args) + print(f"[conf] {conf.split('::')[0]} tls={use_tls(args)} " + f"auth={'token' if args.token else 'basic' if args.username else 'none'}") + sqls = split_queries(args, conf) + readers = len(sqls) + print(f"[scan] reading last {args.limit:,} rows of '{args.table}' " + f"across {readers} reader(s) ...") + + counts = [0] * readers + byts = [0] * readers + errors = [] + stop = threading.Event() + + def reporter(): + last_r, last_b = 0, 0 + # stop.wait() returns True once the event is set -> exit without a final + # stray line after [done]; it returns False on timeout -> print a tick. + while not stop.wait(args.report_interval): + r, b = sum(counts), sum(byts) + rps = (r - last_r) / args.report_interval + mbps = (b - last_b) / args.report_interval / 1e6 + print(f"[scan] {r:>14,} rows | {rps:>13,.0f} rows/s | {mbps:>9,.1f} MB/s") + last_r, last_b = r, b + + rep = threading.Thread(target=reporter, daemon=True) + rep.start() + + t0 = time.monotonic() + threads = [] + for i, sql in enumerate(sqls): + t = threading.Thread(target=run_reader, args=(i, sql, conf, counts, byts, errors)) + t.start() + threads.append(t) + for t in threads: + t.join() + elapsed = time.monotonic() - t0 + stop.set() + + if errors: + for e in errors: + print(f"reader failed: {e}", file=sys.stderr) + return 1 + + total = sum(counts) + tbytes = sum(byts) + if total == 0: + print(f"[done] '{args.table}' returned 0 rows", file=sys.stderr) + return 1 + rate = total / elapsed if elapsed > 0 else float("inf") + mbps = tbytes / elapsed / 1e6 if elapsed > 0 else float("inf") + gbps = tbytes * 8 / elapsed / 1e9 if elapsed > 0 else float("inf") + gib = tbytes / (1024 ** 3) + print(f"[done] {total:,} rows, {gib:.2f} GiB decoded in {elapsed:.3f}s " + f"across {readers} reader(s)") + print(f"[done] {rate:,.0f} rows/s | {mbps:,.1f} MB/s | {gbps:.2f} Gb/s " + f"(decoded Arrow payload, not wire bytes)") + if readers > 1: + per = " ".join(f"r{i}={c:,}" for i, c in enumerate(counts)) + print(f"[done] per-reader rows: {per}") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) From 679876cb3411e6f0172b476e52ab7acbfe61bbcb Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 15 Jul 2026 02:24:28 +0200 Subject: [PATCH 17/79] Harden columnar sender: fail fast on empty token, honor Ctrl+C An empty --token (typically --token "$ILP_TOKEN" with the env var unset) silently disabled auth+TLS, so the client connected plaintext to a secured server and hung retrying the handshake at sent=0, uninterruptible. Now: an empty --token (or a --username with empty --password) exits with a clear error instead of hanging. Worker threads are now daemon and joined via a polling loop, so KeyboardInterrupt is honored even while a worker is blocked in a native connect/flush. --- python/csv_columnar_sender.py | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/python/csv_columnar_sender.py b/python/csv_columnar_sender.py index aebfb51..0cd5c4f 100644 --- a/python/csv_columnar_sender.py +++ b/python/csv_columnar_sender.py @@ -181,6 +181,17 @@ def main(argv): if not os.path.exists(args.csv): print(f"CSV file not found: {args.csv}", file=sys.stderr) return 2 + # An empty --token (typically an unset env var, e.g. --token "$ILP_TOKEN" with + # ILP_TOKEN unset) silently disables auth+TLS. Against a TLS/auth server the client + # then hangs retrying the handshake at sent=0, so fail loudly instead. + if args.token is not None and not args.token.strip(): + print("[error] --token is empty (is $ILP_TOKEN set and exported?). It would connect " + "with NO auth/TLS and hang against a secured server. Set the token, or drop " + "--token to connect plaintext on purpose.", file=sys.stderr) + return 2 + if args.username and not (args.password or "").strip(): + print("[error] --username given but --password is empty.", file=sys.stderr) + return 2 base = load_base(args.csv, args.timestamp_from_file) if base.height == 0: @@ -220,14 +231,23 @@ def wrapper(wid, ev): except Exception as e: # noqa: BLE001 errors.append(f"Sender {wid}: {e}") + # Daemon workers + a polling join so Ctrl+C is honoured even while a worker is + # blocked in a native connect/flush: KeyboardInterrupt fires between join timeouts, + # and daemon threads don't hold the process open once main returns. threads = [] for wid in range(args.num_senders): ev = base_events + (1 if wid < rem else 0) - t = threading.Thread(target=wrapper, args=(wid, ev)) + t = threading.Thread(target=wrapper, args=(wid, ev), daemon=True) t.start() threads.append(t) - for t in threads: - t.join() + try: + while any(t.is_alive() for t in threads): + for t in threads: + t.join(0.2) + except KeyboardInterrupt: + stop.set() + print(f"\nInterrupted. Sent ~{sum(counts):,} rows; exiting.", file=sys.stderr) + return 130 stop.set() if errors: From d099546b3c483d29cc05eab6b3465c9c7d1c5335 Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 15 Jul 2026 02:31:50 +0200 Subject: [PATCH 18/79] Add --connect-timeout preflight to columnar sender Before starting workers, probe the server over the full path (TCP+TLS+auth+ 'select 1') in a daemon thread with a timeout. A down/unreachable/misconfigured server now fails loudly with a clear error instead of the ingest client retrying forever at sent=0. Default 10s; --connect-timeout 0 skips the check. --- python/csv_columnar_sender.py | 41 ++++++++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/python/csv_columnar_sender.py b/python/csv_columnar_sender.py index 0cd5c4f..a23a40e 100644 --- a/python/csv_columnar_sender.py +++ b/python/csv_columnar_sender.py @@ -76,6 +76,31 @@ def build_conf(args): return "".join(parts) +def preflight(conf, timeout_s): + """Check the server answers within timeout_s over the full path (TCP+TLS+auth+query), + in a daemon thread so a hung native connect can't block us. Returns (ok, message). + Without this, a down/unreachable/misconfigured server makes the ingest client retry + forever and sit at sent=0.""" + outcome = {} + + def probe(): + try: + with Client.from_conf(conf) as c: + c.query("select 1").to_polars() + outcome["ok"] = True + except Exception as e: # noqa: BLE001 + outcome["err"] = str(e) + + t = threading.Thread(target=probe, daemon=True) + t.start() + t.join(timeout_s) + if t.is_alive(): + return False, f"no response within {timeout_s:g}s (server down / unreachable / wrong host or port?)" + if outcome.get("ok"): + return True, None + return False, outcome.get("err", "unknown error") + + def load_base(path, need_timestamp): """Load the CSV once into a compact polars DataFrame. symbol/side become Categorical (-> SYMBOL), price/amount Float64, and (if needed) timestamp is @@ -168,6 +193,9 @@ def main(argv): ap.add_argument("--tls", action="store_true", help="force TLS (qwpwss) with no auth") ap.add_argument("--tls-verify", choices=["on", "unsafe_off"], default="on", help="certificate verification; use unsafe_off for self-signed") + ap.add_argument("--connect-timeout", type=float, default=10.0, + help="seconds to wait for the server on a preflight check before failing " + "loudly (0 = skip; the ingest client would otherwise retry forever)") args = ap.parse_args(argv) for name, val in (("--total-events", args.total_events), ("--num-senders", args.num_senders), @@ -198,9 +226,20 @@ def main(argv): print("CSV has no data rows.", file=sys.stderr) return 2 - print(f"[conf] {build_conf(args).split('::')[0]} tls={use_tls(args)} " + conf = build_conf(args) + print(f"[conf] {conf.split('::')[0]} tls={use_tls(args)} " f"auth={'token' if args.token else 'basic' if args.username else 'none'} " f"| addrs: {args.addrs}") + + if args.connect_timeout > 0: + t0 = time.monotonic() + ok, msg = preflight(conf, args.connect_timeout) + if not ok: + print(f"[error] preflight failed: {msg}. Is QuestDB up and the token/addr correct? " + f"(set --connect-timeout 0 to skip this check)", file=sys.stderr) + return 2 + print(f"[preflight] server reachable ({time.monotonic() - t0:.2f}s)") + print(f"Ingestion started (columnar QWP). base={base.height:,} rows, " f"total={args.total_events:,}, workers={args.num_senders}, chunk-rows={args.chunk_rows:,}, " f"timestamps={'file' if args.timestamp_from_file else 'live-now'}, " From 76fead8cac203f2772deb4479ce51bae1359b356 Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 15 Jul 2026 03:38:19 +0200 Subject: [PATCH 19/79] Report acknowledged (committed) rows/s, not just client-side submitted The progress counter only tracked rows appended to the client buffer, which with QWP store-and-forward runs far ahead of what the server has committed - so the per-second rate looked inflated and the end-of-run showed a long '0 rows/s' tail while the client drained the backlog. Now the reporter prints two counters: submitted (client-side, as before) and acknowledged (rows the server has actually committed). The acked count comes from the QWP ack watermark: each flush records its sequence via flushAndGetSequence(), and getAckedFsn() is mapped back to committed rows through a per-worker fsn->rows map - no extra query round-trips. A bounded awaitAckedFsn loop drains at the end while the acked counter climbs to the full count, so the tail shows real progress instead of zeros. The summary throughput is now labeled (acknowledged, end-to-end) and split into submit phase vs commit drain. Validated against a local QWP server: acknowledged trails submitted and converges to 100%. ILP/UDP fall back to the plain flush path. --- .../com/example/sender/CsvParallelSender.java | 90 ++++++++++++++++--- 1 file changed, 80 insertions(+), 10 deletions(-) diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index a3037e9..0f60519 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -76,6 +76,9 @@ public class CsvParallelSender { // Rows sent (client-side) across all workers, for the once-per-second progress reporter. private static final AtomicLong TOTAL_SENT = new AtomicLong(); + // Rows the server has acknowledged, per worker (summed by the reporter). Fed from the QWP + // ack watermark (getAckedFsn) - the real committed count, with no extra query round-trips. + private static AtomicLong[] ACKED_ROWS = new AtomicLong[0]; public static void main(String[] args) throws Exception { // Parse CLI flags @@ -201,18 +204,46 @@ public static void main(String[] args) throws Exception { // Time only the ingestion: start right before the workers begin sending. final long startNanos = System.nanoTime(); - // Progress reporter: prints rows/s (client-side, sent) once per second. + // Per-worker acknowledged-row counters, summed by the reporter. Fed from the QWP ack + // watermark (getAckedFsn) - the real committed progress, with no extra query round-trips. + ACKED_ROWS = new AtomicLong[numSenders]; + for (int i = 0; i < numSenders; i++) { + ACKED_ROWS[i] = new AtomicLong(); + } + + // Records (to ~1s resolution) when all rows were submitted and when all were acknowledged, + // so the summary can separate the submit phase from the commit-drain tail. + final AtomicLong appendDoneNanos = new AtomicLong(0); + final AtomicLong commitDoneNanos = new AtomicLong(0); + + // Progress reporter: once per second, prints BOTH counters - submitted (client-side, can + // run ahead of the server) and acknowledged (rows the server has actually committed). The + // gap between them is the buffered backlog; during the drain, submitted is flat while + // acknowledged keeps climbing (real work), so there is no misleading "0 rows/s". final Thread reporter = new Thread(() -> { - long last = 0; + long lastSub = 0; + long lastAck = 0; while (true) { try { Thread.sleep(1000); } catch (InterruptedException ie) { return; } - long now = TOTAL_SENT.get(); - System.out.printf("[progress] sent=%,d rate=%,d rows/s%n", now, now - last); - last = now; + final long sub = TOTAL_SENT.get(); + long ack = 0; + for (AtomicLong al : ACKED_ROWS) { + ack += al.get(); + } + if (sub >= totalEvents) { + appendDoneNanos.compareAndSet(0, System.nanoTime()); + } + if (ack >= totalEvents) { + commitDoneNanos.compareAndSet(0, System.nanoTime()); + } + System.out.printf("[progress] submitted=%,d (+%,d/s) | acknowledged=%,d (+%,d/s)%n", + sub, sub - lastSub, ack, ack - lastAck); + lastSub = sub; + lastAck = ack; } }); reporter.setDaemon(true); @@ -249,8 +280,17 @@ public static void main(String[] args) throws Exception { } final double elapsedSec = (System.nanoTime() - startNanos) / 1_000_000_000.0; final double rowsPerSec = elapsedSec > 0 ? totalEvents / elapsedSec : 0; - System.out.printf("All workers completed. protocol=%s events=%d elapsed=%.3f s throughput=%,.0f rows/s%n", + System.out.printf("All workers completed. protocol=%s events=%d elapsed=%.3f s throughput=%,.0f rows/s (acknowledged, end-to-end)%n", protocol, totalEvents, elapsedSec, rowsPerSec); + // Split the wall time into submit vs commit-drain, so the fast "submitted" rate is not + // mistaken for durable throughput. appendDoneNanos/commitDoneNanos are ~1s-resolution. + final long submitNanos = appendDoneNanos.get(); + if (submitNanos > 0) { + final double submitSec = (submitNanos - startNanos) / 1_000_000_000.0; + final double submitRate = submitSec > 0 ? totalEvents / submitSec : 0; + System.out.printf(" submit phase %.3f s (%,.0f rows/s submitted) + commit drain %.3f s%n", + submitSec, submitRate, Math.max(0.0, elapsedSec - submitSec)); + } } private static void runWorker( @@ -290,6 +330,9 @@ private static void runWorker( : 0.0; final boolean rateLimited = intervalNanos > 0.0; final long paceStartNanos = System.nanoTime(); + // QWP ack tracking: map each flush sequence -> cumulative rows, so getAckedFsn() can be + // resolved to acknowledged rows. Only QWP carries frame sequence numbers and acks. + final java.util.TreeMap fsnToRows = isQwp ? new java.util.TreeMap<>() : null; try ( Sender sender = buildSender(cfg, senderId)) { //( Sender sender = Sender.fromConfig(conf)) { final int n = rows.size(); for (long i = 0; i < totalEvents; i++) { @@ -322,9 +365,16 @@ private static void runWorker( sent++; TOTAL_SENT.incrementAndGet(); - // QWP-only: commit a transaction every batchSize * batchesPerTransaction rows. + // Commit a transaction every batchSize * batchesPerTransaction rows (QWP), or flush + // every batchSize rows (UDP). For QWP, record the flush sequence and refresh this + // worker's acknowledged-row count from the ack watermark (no extra round-trip). if (commitEveryRows > 0 && sent % commitEveryRows == 0) { - sender.flush(); + if (isQwp) { + fsnToRows.put(sender.flushAndGetSequence(), sent); + ACKED_ROWS[senderId].set(ackedRows(sender, fsnToRows)); + } else { + sender.flush(); + } } if (rateLimited) { @@ -351,8 +401,20 @@ private static void runWorker( } } } - // Explicit flush at the end of this connection's work - sender.flush(); + // Final flush. For QWP, wait for the server to acknowledge everything so the acked + // counter climbs to the full count during the drain (the real committed progress). The + // await loop drives the connection and refreshes the counter; capped to avoid a hang. + if (isQwp) { + final long finalSeq = sender.flushAndGetSequence(); + fsnToRows.put(finalSeq, sent); + final long drainDeadline = System.nanoTime() + 600_000_000_000L; // 10 min cap + while (!sender.awaitAckedFsn(finalSeq, 200L) && System.nanoTime() < drainDeadline) { + ACKED_ROWS[senderId].set(ackedRows(sender, fsnToRows)); + } + } else { + sender.flush(); + } + ACKED_ROWS[senderId].set(sent); System.out.printf("Sender %d finished sending %d events%n", senderId, sent); } catch (Exception e) { System.err.printf("Sender %d got error: %s%s%n", senderId, e.toString(), upgradeHint(e)); @@ -360,6 +422,14 @@ private static void runWorker( } } + // Rows the server has acknowledged for this sender: map the ack watermark (getAckedFsn) back to + // the cumulative row count recorded at the latest fully-acknowledged flush. Zero round-trips - + // the ack watermark rides the ingest connection. + private static long ackedRows(Sender sender, java.util.TreeMap fsnToRows) { + final java.util.Map.Entry e = fsnToRows.floorEntry(sender.getAckedFsn()); + return e != null ? e.getValue() : 0L; + } + // Send the designated timestamp at NANOSECOND resolution. The client's at(Instant) path // delivers only microseconds (sub-microsecond digits are dropped before the wire), so we // convert to epoch-nanos and use at(long, NANOS) to carry full nanosecond precision. From 6fb3fc4bd67079a4c5e60de5410a0215ac9a497a Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 15 Jul 2026 03:47:34 +0200 Subject: [PATCH 20/79] Go quiet after submission: no per-second spam during the commit drain Once all rows are submitted, silence the per-second progress and probe lines so the commit-drain tail no longer prints repeated '+0/s' / probe lines. The single final summary line (acknowledged, end-to-end + submit/drain split) is the last output. Connection/failover probe events still pass through. --- .../com/example/sender/CsvParallelSender.java | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index 0f60519..14375bc 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -79,6 +79,9 @@ public class CsvParallelSender { // Rows the server has acknowledged, per worker (summed by the reporter). Fed from the QWP // ack watermark (getAckedFsn) - the real committed count, with no extra query round-trips. private static AtomicLong[] ACKED_ROWS = new AtomicLong[0]; + // Set once all rows are submitted: silences the per-second progress + probe lines so the + // commit-drain tail is quiet and only the final summary line is printed. + private static volatile boolean SUBMIT_COMPLETE = false; public static void main(String[] args) throws Exception { // Parse CLI flags @@ -236,10 +239,16 @@ public static void main(String[] args) throws Exception { } if (sub >= totalEvents) { appendDoneNanos.compareAndSet(0, System.nanoTime()); + SUBMIT_COMPLETE = true; } if (ack >= totalEvents) { commitDoneNanos.compareAndSet(0, System.nanoTime()); } + // Once everything is submitted, go quiet - no per-second spam while the client + // drains. The final "All workers completed ..." summary is the single last line. + if (SUBMIT_COMPLETE) { + continue; + } System.out.printf("[progress] submitted=%,d (+%,d/s) | acknowledged=%,d (+%,d/s)%n", sub, sub - lastSub, ack, ack - lastAck); lastSub = sub; @@ -852,8 +861,10 @@ public void onFailoverReset(QwpServerInfo info) { served = " served by role=" + handshake + " node=" + node + " zone=" + zone + " (handshake role; live 'switch status' unavailable, may be stale)"; } - System.out.printf("[probe] latest trades timestamp = %s (raw=%d)%s%n", - ts, latest[0], served); + if (!SUBMIT_COMPLETE) { + System.out.printf("[probe] latest trades timestamp = %s (raw=%d)%s%n", + ts, latest[0], served); + } // Explain a missing live role at most once per 30s so it does not spam. if (currentRole[0] == null && statusDiag[0] != null) { final long now = System.currentTimeMillis(); From 1431ef35fc7696989ca0bb1545da6b0f8dc3ef03 Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 15 Jul 2026 13:41:20 +0200 Subject: [PATCH 21/79] Use ws/wss sender scheme in Rust and C++ (qwpwss deprecated) qwpws/qwpwss are deprecated aliases; the client maps ws->QwpWs and wss->QwpWss. Switch the Rust and C++ ingest conf strings to ws/wss, matching their readers (which already use them) and the Java query client. Verified: both build against their client checkouts and ingest cleanly over ws against a local server. Python is left on qwpws/qwpwss - its binding's Protocol enum only exposes QwpWs/QwpWss and rejects ws/wss (ValueError: Invalid value for Protocol), even though the underlying Rust client accepts the aliases. Docs updated accordingly. --- c/README.md | 4 +++- c/csv_parallel_sender.cpp | 5 +++-- python/README.md | 7 +++++-- rust/src/main.rs | 5 +++-- 4 files changed, 14 insertions(+), 7 deletions(-) diff --git a/c/README.md b/c/README.md index 18e3dcb..cbf4e29 100644 --- a/c/README.md +++ b/c/README.md @@ -88,7 +88,9 @@ the `--addrs` hosts. - **No auto-flush** (like Rust; unlike Java/Python) — the C/C++ client has none, so batching is driven purely by explicit `flush()` at the same boundaries as the other ports. -- **Reader scheme is `ws`/`wss`** (like Rust; Python required `qwpws`). +- **Both the sender and reader use the `ws`/`wss` scheme** (like Rust; `qwpws`/`qwpwss` are the + deprecated aliases — the client maps `ws`->QwpWs, `wss`->QwpWss). Python still requires `qwpws` + because its binding does not yet accept the `ws`/`wss` aliases. - Row values from the CSV are validated UTF-8 via `utf8_view` at build time (throws on invalid input); fixed column names use the `_cn` / `_tn` literals. - Threads via `std::thread`, one `line_sender` per worker (unique `sender_id` + spill dir). diff --git a/c/csv_parallel_sender.cpp b/c/csv_parallel_sender.cpp index d137b17..ad4490e 100644 --- a/c/csv_parallel_sender.cpp +++ b/c/csv_parallel_sender.cpp @@ -117,12 +117,13 @@ std::string build_ingest_conf(const Args& a, unsigned worker_id) { c += "retry_timeout=" + std::to_string(a.retry_timeout) + ";"; return c; } - // qwp (WebSocket) + // qwp (WebSocket). Use the spec aliases ws/wss (qwpws/qwpwss are deprecated); the client + // maps ws -> QwpWs and wss -> QwpWss, matching the reader conf below. std::string who = a.sender_id + "-" + std::to_string(worker_id); std::string sf = a.store_forward_dir + "/" + who; std::error_code ec; std::filesystem::create_directories(sf, ec); - std::string c = (use_tls(a) ? "qwpwss" : "qwpws"); + std::string c = (use_tls(a) ? "wss" : "ws"); c += "::addr=" + joined + ";"; append_auth(c, a); if (use_tls(a)) c += "tls_verify=unsafe_off;"; diff --git a/python/README.md b/python/README.md index d109053..213baf5 100644 --- a/python/README.md +++ b/python/README.md @@ -174,8 +174,11 @@ reads) and fails over across the `--addrs` hosts. over QWP (row-threshold flush fires mid-stream; see Validated). This port sets `auto_flush=off` and flushes at the same batch boundaries as the Java/Rust ports for identical cadence. -- **Config-string scheme is `qwpws`/`qwpwss` for both the sender and the query client** - (the Rust reader used `ws`/`wss`; Python only accepts the `qwp*` schemes). +- **Config-string scheme is `qwpws`/`qwpwss` for both the sender and the query client.** + `qwpws`/`qwpwss` are the deprecated aliases (the Rust/C++ ports and their readers use the + newer `ws`/`wss`), but the Python binding's `Protocol` enum only exposes `QwpWs`/`QwpWss`, so + `ws`/`wss` are rejected here (`ValueError: Invalid value for Protocol`). Python must stay on + `qwpws`/`qwpwss` until the binding adds the `ws`/`wss` aliases. - **The probe has no handshake-role fallback** — because the Python binding does not expose it. Java/Rust fall back to the QWP handshake role when `switch status` is unavailable. That role exists in the underlying C client (`line_reader_server_info_role`/`_role_byte`) and the Rust diff --git a/rust/src/main.rs b/rust/src/main.rs index 4f445a9..abcf12b 100644 --- a/rust/src/main.rs +++ b/rust/src/main.rs @@ -424,8 +424,9 @@ fn build_ingest_conf(args: &Args, worker_id: usize) -> String { c } _ => { - // qwp (WebSocket) - let scheme = if args.tls() { "qwpwss" } else { "qwpws" }; + // qwp (WebSocket). Use the spec aliases ws/wss (qwpws/qwpwss are deprecated); the + // client maps ws -> QwpWs and wss -> QwpWss. + let scheme = if args.tls() { "wss" } else { "ws" }; let who = format!("{}-{worker_id}", args.sender_id); let sf = format!("{}/{who}", args.store_forward_dir); let _ = std::fs::create_dir_all(&sf); From 8c9934ec5fa12ba02335c813a3a441aa00ff0f64 Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 15 Jul 2026 13:54:51 +0200 Subject: [PATCH 22/79] Add historical backfill script; show ws/wss in Python [conf] line backfill.py spreads rows evenly across a historical window ([--start,--end), default yesterday 00:00 UTC .. today 13:00 UTC) at --rate-per-day density (default 500M/day -> ~771M rows over the window), ingesting them with those historical timestamps. Streams in bounded chunks (flat memory), splits the window across --num-senders workers (each a contiguous time block), columnar QWP path, with the same auth/TLS + connect-timeout preflight as the other scripts. Verified at small scale: timestamps land spread across the window. Cosmetic: the [conf] line now prints wss/ws instead of qwpwss/qwpws in csv_columnar_sender / read_bench / enrich_polars_demo / backfill. The actual connect string still uses qwpwss/qwpws (the Python binding requires them). --- python/backfill.py | 249 ++++++++++++++++++++++++++++++++++ python/csv_columnar_sender.py | 2 +- python/enrich_polars_demo.py | 2 +- python/read_bench.py | 2 +- 4 files changed, 252 insertions(+), 3 deletions(-) create mode 100644 python/backfill.py diff --git a/python/backfill.py b/python/backfill.py new file mode 100644 index 0000000..f4672af --- /dev/null +++ b/python/backfill.py @@ -0,0 +1,249 @@ +#!/usr/bin/env python3 +"""Backfill the trades table over a historical time window at a target row density. + +Unlike the live senders (which stamp "now") or a file replay, this spreads rows evenly +across [--start, --end) at --rate-per-day rows/day and ingests them with those historical +timestamps, streamed in bounded chunks (flat memory) over the columnar QWP path. Trade +values (symbol/side/price/amount) are cycled from a CSV deck; trade_id is synthesised and +globally unique. Workers each own a contiguous slice of the window. + +Default window: yesterday 00:00 UTC .. today 13:00 UTC (computed at run time, UTC). + +Example (the default 500M rows/day over ~1.54 days is ~771M rows): + python backfill.py --addr host:9000 --num-senders 4 \ + --token "$QDB_TOKEN" --tls-verify unsafe_off + +Requires the QWP/egress build of the client (see README.md). +""" + +import argparse +import gzip +import io +import sys +import threading +import time +from datetime import datetime, timezone, timedelta + +import numpy as np +import polars as pl + +from questdb.ingress import Client + +NS_PER_DAY = 86_400_000_000_000 + + +def use_tls(args): + return bool(args.tls or args.token or (args.username and args.password)) + + +def display_scheme(args): + return "wss" if use_tls(args) else "ws" + + +def build_conf(args): + # Python's binding requires the qwpws/qwpwss scheme (it rejects ws/wss); only the printed + # [conf] line shows wss/ws, via display_scheme(). + scheme = "qwpwss" if use_tls(args) else "qwpws" + parts = [f"{scheme}::addr={args.addr};"] + if args.token: + parts.append(f"token={args.token};") + elif args.username and args.password: + parts.append(f"username={args.username};password={args.password};") + if use_tls(args) and args.tls_verify == "unsafe_off": + parts.append("tls_verify=unsafe_off;") + return "".join(parts) + + +def load_deck(path): + """CSV -> polars deck: symbol/side as Categorical (-> SYMBOL), price/amount Float64.""" + if path.endswith(".gz"): + with gzip.open(path, "rb") as fh: + df = pl.read_csv(io.BytesIO(fh.read())) + else: + df = pl.read_csv(path) + for col in ("symbol", "side", "price", "amount"): + if col not in df.columns: + raise ValueError(f"CSV missing required column: {col}") + return df.select( + pl.col("symbol").cast(pl.Utf8).cast(pl.Categorical), + pl.col("side").cast(pl.Utf8).cast(pl.Categorical), + pl.col("price").cast(pl.Float64), + pl.col("amount").cast(pl.Float64), + ) + + +def parse_iso(s): + dt = datetime.fromisoformat(s.replace("Z", "+00:00")) + return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc) + + +def run_worker(wid, start_index, count, deck, args, start_ns, step_ns, counts): + """Backfill rows [start_index, start_index+count): a contiguous time block. Timestamps come + from the GLOBAL row index (start_ns + gidx*step_ns), so worker ranges never overlap.""" + m = deck.height + conf = build_conf(args) + sent = 0 + pos = start_index % m + with Client.from_conf(conf) as client: + while sent < count: + cn = min(args.chunk_rows, count - sent, m - pos) + sl = deck.slice(pos, cn) # zero-copy view into the deck + gidx = start_index + sent + ts = np.int64(start_ns) + np.arange(gidx, gidx + cn, dtype=np.int64) * np.int64(step_ns) + chunk = sl.with_columns( + pl.Series("timestamp", ts).cast(pl.Datetime("ns", "UTC")), + (pl.lit("backfill-") + pl.int_range(gidx, gidx + cn).cast(pl.Utf8)).alias("trade_id"), + ) + client.dataframe(chunk, table_name=args.table, symbols=["symbol", "side"], at="timestamp") + sent += cn + counts[wid] = sent + pos = (pos + cn) % m + print(f"Worker {wid} finished backfilling {sent:,} rows") + + +def preflight(conf, timeout_s): + outcome = {} + + def probe(): + try: + with Client.from_conf(conf) as c: + c.query("select 1").to_polars() + outcome["ok"] = True + except Exception as e: # noqa: BLE001 + outcome["err"] = str(e) + + t = threading.Thread(target=probe, daemon=True) + t.start() + t.join(timeout_s) + if t.is_alive(): + return False, f"no response within {timeout_s:g}s (server down / unreachable?)" + return (True, None) if outcome.get("ok") else (False, outcome.get("err", "unknown error")) + + +def main(argv): + ap = argparse.ArgumentParser(description="Backfill trades over a historical window") + ap.add_argument("--addr", default="localhost:9000") + ap.add_argument("--start", default=None, help="ISO start (default: yesterday 00:00 UTC)") + ap.add_argument("--end", default=None, help="ISO end, exclusive (default: today 13:00 UTC)") + ap.add_argument("--rate-per-day", type=float, default=500_000_000.0, + help="row density in rows/day, spread evenly across the window; default 500,000,000") + ap.add_argument("--csv", default="../trades20250728.csv.gz") + ap.add_argument("--table", default="trades") + ap.add_argument("--num-senders", type=int, default=1) + ap.add_argument("--chunk-rows", type=int, default=1_000_000) + ap.add_argument("--connect-timeout", type=float, default=10.0) + ap.add_argument("--token", default=None) + ap.add_argument("--username", default=None) + ap.add_argument("--password", default=None) + ap.add_argument("--tls", action="store_true") + ap.add_argument("--tls-verify", choices=["on", "unsafe_off"], default="on") + args = ap.parse_args(argv) + + for name, val in (("--num-senders", args.num_senders), ("--chunk-rows", args.chunk_rows)): + if val <= 0: + print(f"{name} must be > 0", file=sys.stderr) + return 2 + if args.token is not None and not args.token.strip(): + print("[error] --token is empty (is $QDB_TOKEN set and exported?).", file=sys.stderr) + return 2 + + today = datetime.now(timezone.utc).date() + midnight = datetime(today.year, today.month, today.day, tzinfo=timezone.utc) + start = parse_iso(args.start) if args.start else midnight - timedelta(days=1) + end = parse_iso(args.end) if args.end else midnight + timedelta(hours=13) + if end <= start: + print(f"--end ({end}) must be after --start ({start})", file=sys.stderr) + return 2 + if args.rate_per_day <= 0: + print("--rate-per-day must be > 0", file=sys.stderr) + return 2 + + start_ns = int(start.timestamp()) * 1_000_000_000 # integer ns (float loses precision at 1e18) + end_ns = int(end.timestamp()) * 1_000_000_000 + step_ns = int(NS_PER_DAY / args.rate_per_day) + if step_ns <= 0: + print("--rate-per-day too high (sub-nanosecond spacing)", file=sys.stderr) + return 2 + n = (end_ns - start_ns) // step_ns + if n <= 0: + print("window too short for the requested rate (0 rows)", file=sys.stderr) + return 2 + + deck = load_deck(args.csv) + if deck.height == 0: + print("CSV deck is empty", file=sys.stderr) + return 2 + + span_days = (end_ns - start_ns) / NS_PER_DAY + print(f"[conf] {display_scheme(args)} tls={use_tls(args)} " + f"auth={'token' if args.token else 'basic' if args.username else 'none'} | addrs: {args.addr}") + print(f"[backfill] window {start.isoformat()} .. {end.isoformat()} ({span_days:.2f} days) @ " + f"{args.rate_per_day:,.0f} rows/day -> {n:,} rows, one every {step_ns / 1000:.1f} us, " + f"workers={args.num_senders}, chunk-rows={args.chunk_rows:,}") + + if args.connect_timeout > 0: + ok, msg = preflight(build_conf(args), args.connect_timeout) + if not ok: + print(f"[error] preflight failed: {msg}. Is QuestDB up and the token/addr correct?", + file=sys.stderr) + return 2 + + counts = [0] * args.num_senders + stop = threading.Event() + t0 = time.monotonic() + + def reporter(): + last = 0 + while not stop.wait(1.0): + now = sum(counts) + print(f"[progress] backfilled={now:,} rate={now - last:,} rows/s") + last = now + + rep = threading.Thread(target=reporter, daemon=True) + rep.start() + + # Split the n rows into contiguous index ranges, one per worker (each a time block). + base = n // args.num_senders + rem = n % args.num_senders + errors = [] + threads = [] + idx = 0 + for wid in range(args.num_senders): + count = base + (1 if wid < rem else 0) + start_index = idx + idx += count + + def wrap(wid=wid, si=start_index, cnt=count): + try: + run_worker(wid, si, cnt, deck, args, start_ns, step_ns, counts) + except Exception as e: # noqa: BLE001 + errors.append(f"Worker {wid}: {e}") + + t = threading.Thread(target=wrap, daemon=True) + t.start() + threads.append(t) + try: + while any(t.is_alive() for t in threads): + for t in threads: + t.join(0.2) + except KeyboardInterrupt: + stop.set() + print(f"\nInterrupted. Backfilled ~{sum(counts):,} rows; exiting.", file=sys.stderr) + return 130 + stop.set() + + if errors: + for e in errors: + print(f"Worker failed: {e}", file=sys.stderr) + return 1 + + elapsed = time.monotonic() - t0 + rate = n / elapsed if elapsed > 0 else 0 + print(f"[backfill] submitted {n:,} rows in {elapsed:.1f}s ({rate:,.0f} rows/s). " + f"Verify server-side: SELECT count() FROM {args.table} " + f"WHERE timestamp >= '{start.isoformat()}' AND timestamp < '{end.isoformat()}';") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/python/csv_columnar_sender.py b/python/csv_columnar_sender.py index a23a40e..70c74b1 100644 --- a/python/csv_columnar_sender.py +++ b/python/csv_columnar_sender.py @@ -227,7 +227,7 @@ def main(argv): return 2 conf = build_conf(args) - print(f"[conf] {conf.split('::')[0]} tls={use_tls(args)} " + print(f"[conf] {'wss' if use_tls(args) else 'ws'} tls={use_tls(args)} " f"auth={'token' if args.token else 'basic' if args.username else 'none'} " f"| addrs: {args.addrs}") diff --git a/python/enrich_polars_demo.py b/python/enrich_polars_demo.py index 7e55db2..180e4d1 100644 --- a/python/enrich_polars_demo.py +++ b/python/enrich_polars_demo.py @@ -140,7 +140,7 @@ def main(argv): target = f"enriched_{args.table}_demo" rng = np.random.default_rng(args.seed) conf = build_conf(args) - print(f"[conf] {conf.split('::')[0]} tls={use_tls(args)} " + print(f"[conf] {'wss' if use_tls(args) else 'ws'} tls={use_tls(args)} " f"auth={'token' if args.token else 'basic' if args.username else 'none'}") if not args.keep: diff --git a/python/read_bench.py b/python/read_bench.py index b6fcbbe..3470053 100644 --- a/python/read_bench.py +++ b/python/read_bench.py @@ -143,7 +143,7 @@ def main(argv): return 2 conf = build_conf(args) - print(f"[conf] {conf.split('::')[0]} tls={use_tls(args)} " + print(f"[conf] {'wss' if use_tls(args) else 'ws'} tls={use_tls(args)} " f"auth={'token' if args.token else 'basic' if args.username else 'none'}") sqls = split_queries(args, conf) readers = len(sqls) From 0678fdf217d535a4e09cf1e29d43d80d8e2b30bb Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 15 Jul 2026 16:04:37 +0200 Subject: [PATCH 23/79] Add --destination-table to csv_columnar_sender (default trades) Target table is now configurable instead of hardcoded; auto-created if absent. Echoed in the startup line. Verified ingesting into a custom table. --- python/csv_columnar_sender.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/python/csv_columnar_sender.py b/python/csv_columnar_sender.py index 70c74b1..599679f 100644 --- a/python/csv_columnar_sender.py +++ b/python/csv_columnar_sender.py @@ -55,8 +55,6 @@ from questdb.ingress import Client -TABLE = "trades" - def use_tls(args): return bool(args.tls or args.token or (args.username and args.password)) @@ -158,7 +156,7 @@ def run_worker(wid, events, base, args, counts): add.append(pl.Series("timestamp", ts).cast(pl.Datetime("ns", "UTC")).alias("timestamp")) chunk = sl.with_columns(add) - client.dataframe(chunk, table_name=TABLE, symbols=["symbol", "side"], at="timestamp") + client.dataframe(chunk, table_name=args.destination_table, symbols=["symbol", "side"], at="timestamp") sent += n counts[wid] = sent @@ -184,6 +182,8 @@ def main(argv): ap.add_argument("--rate", type=int, default=0, help="target aggregate rows/second across ALL workers (0 = flat out)") ap.add_argument("--csv", default="../trades20250728.csv.gz") + ap.add_argument("--destination-table", default="trades", + help="target table name (auto-created if absent); default 'trades'") ap.add_argument("--timestamp-from-file", action="store_true", help="use the CSV timestamp column instead of a live 'now' stamp") # Enterprise auth / TLS @@ -240,7 +240,7 @@ def main(argv): return 2 print(f"[preflight] server reachable ({time.monotonic() - t0:.2f}s)") - print(f"Ingestion started (columnar QWP). base={base.height:,} rows, " + print(f"Ingestion started (columnar QWP). table={args.destination_table}, base={base.height:,} rows, " f"total={args.total_events:,}, workers={args.num_senders}, chunk-rows={args.chunk_rows:,}, " f"timestamps={'file' if args.timestamp_from_file else 'live-now'}, " f"pacing={'rate ' + str(args.rate) + ' rows/s' if args.rate else 'flat out'}") From fb789602e98e8a226c611670849eb4330dceebc8 Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 15 Jul 2026 16:25:42 +0200 Subject: [PATCH 24/79] read_bench: show a sample of the scanned data as a polars DataFrame Add --sample N (default 5): reader 0 wraps its first received Arrow batch as a polars DataFrame (zero-copy) and keeps its head - reusing data already streamed, no extra query and no re-scan. Rendered after the timing, so throughput is unaffected. Only one batch is ever held, so memory stays bounded. --- python/read_bench.py | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/python/read_bench.py b/python/read_bench.py index 3470053..83482c4 100644 --- a/python/read_bench.py +++ b/python/read_bench.py @@ -100,7 +100,7 @@ def split_queries(args, conf): return sqls -def run_reader(idx, sql, conf, counts, byts, errors): +def run_reader(idx, sql, conf, counts, byts, errors, sample_n, sample_out): try: with Client.from_conf(conf) as client: result = client.query(sql) @@ -109,6 +109,11 @@ def run_reader(idx, sql, conf, counts, byts, errors): for batch in result.iter_arrow(): n += batch.num_rows # count rows and decoded Arrow bytes b += batch.nbytes # (buffer bytes of this batch's columns) + if idx == 0 and sample_n > 0 and sample_out[0] is None: + # Reuse the Arrow data we already received: wrap the first batch as a polars + # DataFrame (zero-copy) and keep its head (one-off, no extra query, no re-scan). + import polars as pl + sample_out[0] = pl.from_arrow(batch).head(sample_n) counts[idx] = n byts[idx] = b counts[idx] = n @@ -129,6 +134,8 @@ def main(argv): help="designated timestamp column (used to split across readers)") ap.add_argument("--report-interval", type=float, default=0.5, help="seconds between progress lines; default 0.5") + ap.add_argument("--sample", type=int, default=5, + help="after the scan, show this many rows as a polars DataFrame (0 to disable)") # Enterprise auth / TLS ap.add_argument("--token", default=None, help="bearer token (turns on TLS)") ap.add_argument("--username", default=None, help="basic-auth username (turns on TLS)") @@ -153,6 +160,7 @@ def main(argv): counts = [0] * readers byts = [0] * readers errors = [] + sample_out = [None] # reader 0 fills this with a few rows off its first Arrow batch stop = threading.Event() def reporter(): @@ -172,7 +180,8 @@ def reporter(): t0 = time.monotonic() threads = [] for i, sql in enumerate(sqls): - t = threading.Thread(target=run_reader, args=(i, sql, conf, counts, byts, errors)) + t = threading.Thread(target=run_reader, + args=(i, sql, conf, counts, byts, errors, args.sample, sample_out)) t.start() threads.append(t) for t in threads: @@ -201,6 +210,16 @@ def reporter(): if readers > 1: per = " ".join(f"r{i}={c:,}" for i, c in enumerate(counts)) print(f"[done] per-reader rows: {per}") + + # An extract of the data we actually received: a few rows sliced off the first Arrow batch + # during the scan (no extra query, no re-scan) and rendered as a polars DataFrame. + if args.sample > 0 and sample_out[0] is not None: + import polars as pl + df = sample_out[0] + print(f"\n[sample] first {df.height} row(s) received, as a polars DataFrame " + f"(reused from the scanned Arrow batches - no extra query):") + with pl.Config(tbl_rows=args.sample, tbl_cols=-1): + print(df) return 0 From 4a7cfefdf54e58675dd243ddb1d0b8aacb2af83e Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 16 Jul 2026 11:20:29 +0200 Subject: [PATCH 25/79] Add live blotter for QuestDB tables / live views blotter.py polls a table (or live view) at --rate Hz (default 5, max 20) and redraws a polars table in place (ANSI, no flicker). Builds 'select * from TABLE [WHERE ...] limit N' from three params: table, --where (WHERE auto-prepended if absent; trailing clauses like ORDER BY ride along; not sanitised - demo use), and --limit (default -10, magnitude clamped to 100). Query errors render in place instead of crashing, so the SELECT can be tweaked live. --once renders a single plain frame for scripting. Enterprise auth/TLS via --token/--username/--password (+ --tls-verify), plus --token-file/--token-label to read a bearer token from a file by label (keeps the secret off the command line). Verified against a remote core_price_lv live view. run_blotter.sh launches it with a default cluster address and $ILP_TOKEN, passing table/--where/--limit/--rate through; ADDR and PYTHON overridable via env. --- python/blotter.py | 173 ++++++++++++++++++++++++++++++++++++++++++ python/run_blotter.sh | 29 +++++++ 2 files changed, 202 insertions(+) create mode 100644 python/blotter.py create mode 100755 python/run_blotter.sh diff --git a/python/blotter.py b/python/blotter.py new file mode 100644 index 0000000..b850466 --- /dev/null +++ b/python/blotter.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +"""A live blotter: poll a QuestDB table (or live view) N times/second and redraw in place. + +Builds `select * from TABLE [WHERE ...] limit N` from three params and refreshes a +terminal table as fast as configured. Great for demoing live views. + +Params: + * table - positional, the table / live view name. + * --where COND - optional filter. If the first word (left-trimmed) is not "where", + "WHERE " is prepended; otherwise it is used verbatim. Trailing clauses + like ORDER BY just ride along (e.g. --where "symbol='EURUSD' order by timestamp"). + NOT sanitised - demo use only. + * --limit N - default -10 (last 10 rows). Clamped to +/-100. |N| rows are shown. + +Refresh rate: --rate Hz, default 5, clamped to 20. + +Auth / TLS (Enterprise): --token or --username/--password (turns on TLS); --tls forces TLS +with no auth; --tls-verify unsafe_off for self-signed certs. + +Usage: + python blotter.py trades --where "symbol='BTC-USDT'" --limit -20 --rate 10 \ + [--addr host:9000] [--token TOK --tls-verify unsafe_off] + +Requires the QWP/egress build of the client (see README.md). Ctrl+C to quit. +""" + +import argparse +import os +import sys +import time +from datetime import datetime, timezone + +import polars as pl + +from questdb.ingress import Client + + +def use_tls(args): + return bool(args.tls or args.token or (args.username and args.password)) + + +def build_conf(args): + scheme = "qwpwss" if use_tls(args) else "qwpws" # Python requires the qwp* scheme + parts = [f"{scheme}::addr={args.addr};"] + if args.token: + parts.append(f"token={args.token};") + elif args.username and args.password: + parts.append(f"username={args.username};password={args.password};") + if use_tls(args) and args.tls_verify == "unsafe_off": + parts.append("tls_verify=unsafe_off;") + return "".join(parts) + + +def build_sql(table, where, limit): + parts = [f"select * from {table}"] + if where and where.strip(): + c = where.strip() + first = c.split(None, 1)[0].lower() + parts.append(c if first == "where" else "WHERE " + c) + parts.append(f"limit {limit}") + return " ".join(parts) + + +def draw(frame): + """Redraw in place: home the cursor, clear each line to its end as we write, then clear + anything left below (so a shorter frame does not leave stale rows).""" + lines = frame.split("\n") + sys.stdout.write("\033[H" + "\033[K\n".join(lines) + "\033[K\033[0J") + sys.stdout.flush() + + +def main(argv): + ap = argparse.ArgumentParser(description="Live QuestDB blotter") + ap.add_argument("table", help="table or live view name") + ap.add_argument("--where", default=None, help="filter condition ('WHERE' prepended if absent)") + ap.add_argument("--limit", type=int, default=-10, help="row limit, default -10, clamped to +/-100") + ap.add_argument("--rate", type=float, default=5.0, help="refreshes per second, default 5, max 20") + ap.add_argument("--addr", default="localhost:9000") + ap.add_argument("--once", action="store_true", help="render a single frame (no ANSI) and exit") + ap.add_argument("--token", default=None) + ap.add_argument("--token-file", default=None, + help="read the bearer token from this file (keeps it out of the command line)") + ap.add_argument("--token-label", default=None, + help="if --token-file has 'label token' lines, pick the token for this label") + ap.add_argument("--username", default=None) + ap.add_argument("--password", default=None) + ap.add_argument("--tls", action="store_true") + ap.add_argument("--tls-verify", choices=["on", "unsafe_off"], default="on") + args = ap.parse_args(argv) + + # Load the token from a file if requested (keeps the secret off the command line). + if args.token_file: + try: + with open(os.path.expanduser(args.token_file), encoding="utf-8") as fh: + content = fh.read() + except OSError as e: + print(f"[error] cannot read --token-file: {e}", file=sys.stderr) + return 2 + if args.token_label: + tok = None + for line in content.splitlines(): + line = line.strip() + if not line: + continue + parts = line.split(None, 1) + if len(parts) == 2 and parts[0] == args.token_label: + tok = parts[1].strip() + break + if not tok: + print(f"[error] label '{args.token_label}' not found in {args.token_file}", + file=sys.stderr) + return 2 + args.token = tok + else: + args.token = content.strip() + + # Clamp limit magnitude to 100 (keep sign) and rate to 20 Hz. + if abs(args.limit) > 100: + args.limit = 100 if args.limit > 0 else -100 + print(f"[warn] --limit clamped to {args.limit}", file=sys.stderr) + if args.limit == 0: + print("--limit must be non-zero", file=sys.stderr) + return 2 + if args.rate <= 0: + print("--rate must be > 0", file=sys.stderr) + return 2 + if args.rate > 20: + args.rate = 20.0 + print("[warn] --rate clamped to 20 Hz", file=sys.stderr) + + conf = build_conf(args) + sql = build_sql(args.table, args.where, args.limit) + rows = abs(args.limit) + interval = 1.0 / args.rate + scheme = "wss" if use_tls(args) else "ws" + + def render(client): + t0 = time.perf_counter() + try: + df = client.query(sql).to_polars() + ms = (time.perf_counter() - t0) * 1000.0 + with pl.Config(tbl_rows=rows, tbl_cols=-1, tbl_hide_dataframe_shape=True): + body = str(df) + status = (f"{df.height} rows | q={ms:5.1f} ms | {args.rate:g} Hz | " + f"{datetime.now(timezone.utc):%Y-%m-%d %H:%M:%S} UTC") + except Exception as e: # noqa: BLE001 + body = f"[error] {e}" + status = f"query failed | {datetime.now(timezone.utc):%H:%M:%S} UTC" + return f"SQL: {sql}\n {status}\n{body}" + + if args.once: + with Client.from_conf(conf) as client: + print(f"[conf] {scheme} tls={use_tls(args)} | addr: {args.addr}") + print(render(client)) + return 0 + + sys.stdout.write("\033[2J\033[?25l") # clear once, hide cursor + try: + with Client.from_conf(conf) as client: + while True: + t0 = time.perf_counter() + draw(render(client)) + time.sleep(max(0.0, interval - (time.perf_counter() - t0))) + except KeyboardInterrupt: + pass + finally: + sys.stdout.write("\033[?25h\n") # restore cursor + sys.stdout.flush() + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/python/run_blotter.sh b/python/run_blotter.sh new file mode 100755 index 0000000..8eee340 --- /dev/null +++ b/python/run_blotter.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# run_blotter.sh - launch the live blotter with your standard address + token. +# +# Set the token the same way as the senders, then run; extra args pass straight through +# to blotter.py (table, --where, --limit, --rate, ...): +# +# export ILP_TOKEN= +# ./run_blotter.sh core_price_lv --where "symbol = 'EURUSD'" --limit -20 --rate 5 +# +# Override the address or the python interpreter via env vars when needed: +# ADDR="host:9000" # default is the demo cluster below (comma-separated = failover) +# PYTHON="/path/to/venv/bin/python" # the interpreter that has the QWP client +# +# Run it (bash run_blotter.sh ...), don't source it. + +ADDR="${ADDR:-172.31.42.41:9000,172.31.41.35:9000,10.0.0.8:9000}" +PYTHON="${PYTHON:-python3}" +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +if [ -z "${ILP_TOKEN:-}" ]; then + echo "ILP_TOKEN is not set. Run: export ILP_TOKEN=" >&2 + exit 1 +fi + +"$PYTHON" "$DIR/blotter.py" \ + --addr "$ADDR" \ + --token "$ILP_TOKEN" \ + --tls-verify unsafe_off \ + "$@" From 0bbbf0e9bd21a0fbb525b5549c59aaef9bf76db1 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 16 Jul 2026 12:21:34 +0200 Subject: [PATCH 26/79] blotter: named --table, add --query for verbatim SQL --table replaces the positional arg. New --query runs full SQL verbatim (CTEs, window functions, ...); when set, --table/--where/--limit are ignored and a trailing ';' is stripped. Provide one of --table/--query. Multi-line SQL is collapsed to one line in the header; display is capped at 100 rows. Verified against a live view with a CTE + window query. run_blotter.sh uses --table with a commented --query example. --- python/blotter.py | 66 +++++++++++++++++++++++++++---------------- python/run_blotter.sh | 43 +++++++++++----------------- 2 files changed, 58 insertions(+), 51 deletions(-) diff --git a/python/blotter.py b/python/blotter.py index b850466..da333b8 100644 --- a/python/blotter.py +++ b/python/blotter.py @@ -1,25 +1,32 @@ #!/usr/bin/env python3 """A live blotter: poll a QuestDB table (or live view) N times/second and redraw in place. -Builds `select * from TABLE [WHERE ...] limit N` from three params and refreshes a -terminal table as fast as configured. Great for demoing live views. - -Params: - * table - positional, the table / live view name. - * --where COND - optional filter. If the first word (left-trimmed) is not "where", - "WHERE " is prepended; otherwise it is used verbatim. Trailing clauses - like ORDER BY just ride along (e.g. --where "symbol='EURUSD' order by timestamp"). - NOT sanitised - demo use only. - * --limit N - default -10 (last 10 rows). Clamped to +/-100. |N| rows are shown. - +Runs `--query` verbatim, or builds `select * from TABLE [WHERE ...] limit N` from +`--table`/`--where`/`--limit`, and refreshes a terminal table as fast as configured. +Great for demoing live views. + +Params (provide --table OR --query): + * --table NAME - table / live view name; a `select * from NAME [WHERE ...] limit N` is built. + * --query SQL - full SQL run verbatim (CTEs, window functions, etc.). When set, + --table/--where/--limit are ignored. A trailing ';' is stripped. + * --where COND - optional filter (table mode). If the first word (left-trimmed) is not + "where", "WHERE " is prepended; otherwise used verbatim. Trailing clauses + like ORDER BY ride along. NOT sanitised - demo use only. + * --limit N - table mode, default -10 (last 10 rows). Clamped to +/-100. + +At most 100 rows are displayed (the query/limit governs how many are fetched). Refresh rate: --rate Hz, default 5, clamped to 20. Auth / TLS (Enterprise): --token or --username/--password (turns on TLS); --tls forces TLS -with no auth; --tls-verify unsafe_off for self-signed certs. +with no auth; --tls-verify unsafe_off for self-signed certs; --token-file/--token-label to +read the token from a file. Usage: - python blotter.py trades --where "symbol='BTC-USDT'" --limit -20 --rate 10 \ + python blotter.py --table trades --where "symbol='BTC-USDT'" --limit -20 --rate 10 \ [--addr host:9000] [--token TOK --tls-verify unsafe_off] + python blotter.py --query "with x as (select * from core_price_lv where symbol='EURUSD' \ + order by timestamp desc limit 50) select *, avg(bid_price) over (partition by symbol) \ + as avg50 from x" --addr host:9000 --token TOK --tls-verify unsafe_off Requires the QWP/egress build of the client (see README.md). Ctrl+C to quit. """ @@ -71,7 +78,9 @@ def draw(frame): def main(argv): ap = argparse.ArgumentParser(description="Live QuestDB blotter") - ap.add_argument("table", help="table or live view name") + ap.add_argument("--table", default=None, help="table or live view name (built into a select)") + ap.add_argument("--query", default=None, + help="full SQL to run verbatim; when set, --table/--where/--limit are ignored") ap.add_argument("--where", default=None, help="filter condition ('WHERE' prepended if absent)") ap.add_argument("--limit", type=int, default=-10, help="row limit, default -10, clamped to +/-100") ap.add_argument("--rate", type=float, default=5.0, help="refreshes per second, default 5, max 20") @@ -114,13 +123,6 @@ def main(argv): else: args.token = content.strip() - # Clamp limit magnitude to 100 (keep sign) and rate to 20 Hz. - if abs(args.limit) > 100: - args.limit = 100 if args.limit > 0 else -100 - print(f"[warn] --limit clamped to {args.limit}", file=sys.stderr) - if args.limit == 0: - print("--limit must be non-zero", file=sys.stderr) - return 2 if args.rate <= 0: print("--rate must be > 0", file=sys.stderr) return 2 @@ -128,9 +130,23 @@ def main(argv): args.rate = 20.0 print("[warn] --rate clamped to 20 Hz", file=sys.stderr) + # A full --query runs verbatim (--table/--where/--limit ignored); otherwise build from --table. + if args.query and args.query.strip(): + sql = args.query.strip().rstrip(";").strip() + elif args.table: + if abs(args.limit) > 100: + args.limit = 100 if args.limit > 0 else -100 + print(f"[warn] --limit clamped to {args.limit}", file=sys.stderr) + if args.limit == 0: + print("--limit must be non-zero", file=sys.stderr) + return 2 + sql = build_sql(args.table, args.where, args.limit) + else: + print("provide --table or --query", file=sys.stderr) + return 2 + conf = build_conf(args) - sql = build_sql(args.table, args.where, args.limit) - rows = abs(args.limit) + sql_line = " ".join(sql.split()) # collapse newlines/whitespace for the one-line header interval = 1.0 / args.rate scheme = "wss" if use_tls(args) else "ws" @@ -139,14 +155,14 @@ def render(client): try: df = client.query(sql).to_polars() ms = (time.perf_counter() - t0) * 1000.0 - with pl.Config(tbl_rows=rows, tbl_cols=-1, tbl_hide_dataframe_shape=True): + with pl.Config(tbl_rows=min(df.height, 100), tbl_cols=-1, tbl_hide_dataframe_shape=True): body = str(df) status = (f"{df.height} rows | q={ms:5.1f} ms | {args.rate:g} Hz | " f"{datetime.now(timezone.utc):%Y-%m-%d %H:%M:%S} UTC") except Exception as e: # noqa: BLE001 body = f"[error] {e}" status = f"query failed | {datetime.now(timezone.utc):%H:%M:%S} UTC" - return f"SQL: {sql}\n {status}\n{body}" + return f"SQL: {sql_line}\n {status}\n{body}" if args.once: with Client.from_conf(conf) as client: diff --git a/python/run_blotter.sh b/python/run_blotter.sh index 8eee340..5fb4183 100755 --- a/python/run_blotter.sh +++ b/python/run_blotter.sh @@ -1,29 +1,20 @@ #!/usr/bin/env bash -# run_blotter.sh - launch the live blotter with your standard address + token. -# -# Set the token the same way as the senders, then run; extra args pass straight through -# to blotter.py (table, --where, --limit, --rate, ...): -# -# export ILP_TOKEN= -# ./run_blotter.sh core_price_lv --where "symbol = 'EURUSD'" --limit -20 --rate 5 -# -# Override the address or the python interpreter via env vars when needed: -# ADDR="host:9000" # default is the demo cluster below (comma-separated = failover) -# PYTHON="/path/to/venv/bin/python" # the interpreter that has the QWP client -# -# Run it (bash run_blotter.sh ...), don't source it. +# Live blotter. Set your token first (export ILP_TOKEN=), then: ./run_blotter.sh +# Edit the params below directly. -ADDR="${ADDR:-172.31.42.41:9000,172.31.41.35:9000,10.0.0.8:9000}" -PYTHON="${PYTHON:-python3}" -DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +python blotter.py \ + --table core_price_lv \ + --where "symbol = 'EURUSD'" \ + --limit -10 \ + --rate 5 \ + --addr 172.31.42.41:9000,172.31.41.35:9000,10.0.0.8:9000 \ + --token "$ILP_TOKEN" \ + --tls-verify unsafe_off -if [ -z "${ILP_TOKEN:-}" ]; then - echo "ILP_TOKEN is not set. Run: export ILP_TOKEN=" >&2 - exit 1 -fi - -"$PYTHON" "$DIR/blotter.py" \ - --addr "$ADDR" \ - --token "$ILP_TOKEN" \ - --tls-verify unsafe_off \ - "$@" +# --- or run a full custom query instead (ignores --table/--where/--limit) --- +# python blotter.py \ +# --query "with x as (select * from core_price_lv where symbol = 'EURUSD' order by timestamp desc limit 50) select *, avg(bid_price) over (partition by symbol) as avg50 from x" \ +# --rate 5 \ +# --addr 172.31.42.41:9000,172.31.41.35:9000,10.0.0.8:9000 \ +# --token "$ILP_TOKEN" \ +# --tls-verify unsafe_off From a230a7fae57b96b3ed423bd570ec2cf44f5c8885 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 16 Jul 2026 13:37:09 +0200 Subject: [PATCH 27/79] blotter: diff-render (repaint only changed lines) to lift the SSH refresh ceiling The full-screen redraw each tick was terminal/SSH-output bound (froze past ~4 Hz over SSH), not query bound. draw() now diffs the new frame against the last and rewrites only the lines that changed - positioning each absolutely, writing it, and clearing to end-of-line - so static borders/header/rows are never re-sent. Cuts terminal output sharply and lets the refresh rate go well past the old ceiling. --- python/blotter.py | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/python/blotter.py b/python/blotter.py index da333b8..228ce40 100644 --- a/python/blotter.py +++ b/python/blotter.py @@ -69,11 +69,22 @@ def build_sql(table, where, limit): def draw(frame): - """Redraw in place: home the cursor, clear each line to its end as we write, then clear - anything left below (so a shorter frame does not leave stale rows).""" + """Redraw in place, writing ONLY the lines that changed since the last frame (diff render). + Over SSH the terminal repaint is the bottleneck, so skipping unchanged lines (borders, + header, static rows) is what lets the refresh rate go up. Each changed line is positioned + absolutely, rewritten, then cleared to end-of-line (to erase any longer previous content).""" + prev = getattr(draw, "_prev", []) lines = frame.split("\n") - sys.stdout.write("\033[H" + "\033[K\n".join(lines) + "\033[K\033[0J") - sys.stdout.flush() + out = [] + for i in range(max(len(lines), len(prev))): + new = lines[i] if i < len(lines) else "" + old = prev[i] if i < len(prev) else None + if new != old: + out.append(f"\033[{i + 1};1H{new}\033[K") # move to row, write, clear tail + if out: + sys.stdout.write("".join(out)) + sys.stdout.flush() + draw._prev = lines def main(argv): From 5bbba06514b9c384d4955909d18b466d3b6a1c22 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 16 Jul 2026 16:59:39 +0200 Subject: [PATCH 28/79] Add web_blotter: local server + browser UI for a live blotter A small http.server that queries QuestDB (QWP, token held server-side) and serves a self-contained dark-themed page that polls /data and renders a live table with cells flashing green/red on change. Rendering is in the browser, so it avoids the terminal/ SSH redraw bottleneck; only small JSON crosses the wire. Same query/auth params as blotter.py (--table/--query/--where/--limit, --token[-file/-label], --tls-verify). --host controls the bind address (default 127.0.0.1; --host 0.0.0.0 exposes it - the web server has no auth, so firewall the port). --- python/web_blotter.py | 242 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 242 insertions(+) create mode 100644 python/web_blotter.py diff --git a/python/web_blotter.py b/python/web_blotter.py new file mode 100644 index 0000000..8477693 --- /dev/null +++ b/python/web_blotter.py @@ -0,0 +1,242 @@ +#!/usr/bin/env python3 +"""Web blotter: a tiny local server that queries QuestDB and serves a live table in the browser. + +Rendering happens in the browser (fast, local, flashing cells) and only the queries cross the +network - so it avoids the terminal-over-SSH bottleneck of the text blotter. The token stays in +this process (never sent to the browser), and the browser only ever talks to localhost, so there +is no CORS or cert hassle: + + browser -> http://localhost:PORT (this server) -> QuestDB (QWP, with your token) + +Same query params as blotter.py: --table (+ --where/--limit) or a verbatim --query. Auth/TLS via +--token/--username/--password/--tls-verify, or --token-file/--token-label. --rate sets the browser +poll rate. Open the printed URL. Ctrl+C to quit. + + python web_blotter.py --table core_price_lv --where "symbol='EURUSD'" --limit -20 --rate 5 \ + --addr host:9000 --token-file ~/prj/python/ent_tokens.txt --token-label disaster \ + --tls-verify unsafe_off [--port 8080] + +Requires the QWP/egress build of the client (see README.md). +""" + +import argparse +import json +import os +import sys +import threading +from datetime import datetime +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +from questdb.ingress import Client + + +def use_tls(args): + return bool(args.tls or args.token or (args.username and args.password)) + + +def build_conf(args): + scheme = "qwpwss" if use_tls(args) else "qwpws" + parts = [f"{scheme}::addr={args.addr};"] + if args.token: + parts.append(f"token={args.token};") + elif args.username and args.password: + parts.append(f"username={args.username};password={args.password};") + if use_tls(args) and args.tls_verify == "unsafe_off": + parts.append("tls_verify=unsafe_off;") + return "".join(parts) + + +def build_sql(table, where, limit): + parts = [f"select * from {table}"] + if where and where.strip(): + c = where.strip() + first = c.split(None, 1)[0].lower() + parts.append(c if first == "where" else "WHERE " + c) + parts.append(f"limit {limit}") + return " ".join(parts) + + +def load_token(args): + if not args.token_file: + return + with open(os.path.expanduser(args.token_file), encoding="utf-8") as fh: + content = fh.read() + if args.token_label: + for line in content.splitlines(): + line = line.strip() + parts = line.split(None, 1) + if len(parts) == 2 and parts[0] == args.token_label: + args.token = parts[1].strip() + return + sys.exit(f"[error] label '{args.token_label}' not found in {args.token_file}") + args.token = content.strip() + + +def df_to_payload(df): + """polars DataFrame -> JSON-friendly {columns, rows} (datetimes as ISO strings).""" + rows = [] + for r in df.iter_rows(): + rows.append([v.isoformat(sep=" ", timespec="milliseconds") if isinstance(v, datetime) else v + for v in r]) + return {"columns": df.columns, "rows": rows} + + +HTML = r""" +__TITLE__ + + +

__TITLE__

+
__SQL__
+
connecting...
+
+ + +""" + + +def main(argv): + ap = argparse.ArgumentParser(description="Web blotter (local server + browser UI)") + ap.add_argument("--table", default=None) + ap.add_argument("--query", default=None, help="full SQL; when set, --table/--where/--limit ignored") + ap.add_argument("--where", default=None) + ap.add_argument("--limit", type=int, default=-20) + ap.add_argument("--rate", type=float, default=5.0, help="browser poll rate (Hz)") + ap.add_argument("--port", type=int, default=8080) + ap.add_argument("--host", default="127.0.0.1", + help="bind address; 0.0.0.0 exposes it externally (NO auth on the web server - " + "scope your security group). Default 127.0.0.1.") + ap.add_argument("--addr", default="localhost:9000") + ap.add_argument("--token", default=None) + ap.add_argument("--token-file", default=None) + ap.add_argument("--token-label", default=None) + ap.add_argument("--username", default=None) + ap.add_argument("--password", default=None) + ap.add_argument("--tls", action="store_true") + ap.add_argument("--tls-verify", choices=["on", "unsafe_off"], default="on") + args = ap.parse_args(argv) + load_token(args) + + if args.query and args.query.strip(): + sql = args.query.strip().rstrip(";").strip() + elif args.table: + if abs(args.limit) > 100: + args.limit = 100 if args.limit > 0 else -100 + sql = build_sql(args.table, args.where, args.limit) + else: + print("provide --table or --query", file=sys.stderr) + return 2 + interval_ms = max(50, int(1000 / max(0.1, args.rate))) + + conf = build_conf(args) + client = Client.from_conf(conf) + lock = threading.Lock() + page = (HTML.replace("__TITLE__", f"blotter: {args.table or 'query'}") + .replace("__SQL__", " ".join(sql.split())) + .replace("__INTERVAL__", str(interval_ms))) + + class Handler(BaseHTTPRequestHandler): + def log_message(self, *a): + pass + + def _send(self, code, ctype, body): + b = body.encode("utf-8") if isinstance(body, str) else body + self.send_response(code) + self.send_header("Content-Type", ctype) + self.send_header("Content-Length", str(len(b))) + self.end_headers() + try: + self.wfile.write(b) + except (BrokenPipeError, ConnectionResetError): + pass + + def do_GET(self): + if self.path == "/" or self.path.startswith("/?"): + self._send(200, "text/html; charset=utf-8", page) + elif self.path.startswith("/data"): + try: + with lock: + df = client.query(sql).to_polars() + self._send(200, "application/json", json.dumps(df_to_payload(df))) + except Exception as e: # noqa: BLE001 + self._send(200, "application/json", json.dumps({"error": str(e)})) + else: + self._send(404, "text/plain", "not found") + + httpd = ThreadingHTTPServer((args.host, args.port), Handler) + shown = "localhost" if args.host in ("127.0.0.1", "localhost") else "" + print(f"[web-blotter] serving on {args.host}:{args.port} -> open http://{shown}:{args.port}/ " + f"(Ctrl+C to quit)") + if args.host == "0.0.0.0": + print("[web-blotter] bound to 0.0.0.0 - reachable on this host's IP; the web server has NO " + "auth, so make sure the port is firewalled to trusted clients.") + print(f"[web-blotter] SQL: {' '.join(sql.split())}") + try: + httpd.serve_forever() + except KeyboardInterrupt: + pass + finally: + httpd.server_close() + client.close() + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) From 09b79cdffd75057532b98397311f52a8a885cd2e Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 16 Jul 2026 17:09:13 +0200 Subject: [PATCH 29/79] web_blotter: HTTP keep-alive + daemon threads protocol_version HTTP/1.1 so the browser reuses one TCP connection across polls (the per-poll q was dominated by a fresh connection/handshake each time over the laptop<->box link). daemon_threads so Ctrl+C exits immediately even with a request in flight. --- python/web_blotter.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/python/web_blotter.py b/python/web_blotter.py index 8477693..218ec96 100644 --- a/python/web_blotter.py +++ b/python/web_blotter.py @@ -193,6 +193,8 @@ def main(argv): .replace("__INTERVAL__", str(interval_ms))) class Handler(BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" # keep-alive: reuse one TCP connection across polls + def log_message(self, *a): pass @@ -220,7 +222,11 @@ def do_GET(self): else: self._send(404, "text/plain", "not found") - httpd = ThreadingHTTPServer((args.host, args.port), Handler) + class Server(ThreadingHTTPServer): + daemon_threads = True # Ctrl+C exits immediately, even with a request in flight + allow_reuse_address = True + + httpd = Server((args.host, args.port), Handler) shown = "localhost" if args.host in ("127.0.0.1", "localhost") else "" print(f"[web-blotter] serving on {args.host}:{args.port} -> open http://{shown}:{args.port}/ " f"(Ctrl+C to quit)") From e049acb2ec742814243cd642b8d7953d61fc13d1 Mon Sep 17 00:00:00 2001 From: javier Date: Mon, 20 Jul 2026 17:09:37 +0200 Subject: [PATCH 30/79] python: migrate all clients to questdb 5.0 API Move the Python port off the 4.x `questdb.ingress.Client` split onto the 5.0 shape: - Import from top-level `questdb`; `Client.from_conf(conf)` -> `questdb.connect(conf)` returning a pooled `QuestDB` handle (`db`). - Conf schemes qwpws/qwpwss/qwpudp -> ws/wss/udp (now identical to the Rust/C ports; a conf string copies verbatim between clients). - db.query / db.dataframe / db.execute on the handle. Per script: - dataframe_demo: one handle for DDL (db.execute, replacing the urllib /exec hack), pandas + polars ingest via db.dataframe, dual egress; SYMBOL columns cast to pandas Categorical (columnar v1 requirement). - enrich_polars_demo: single pooled handle - the open read stream and each write borrow separate pooled connections. - csv_parallel_sender: keep a standalone per-worker questdb.Sender for independent store-and-forward; only the probe moves to connect(). - blotter / web_blotter / read_bench / backfill / csv_columnar_sender: Client -> connect(), scheme rename, db.* calls. - README rewritten for 5.0: build from jh_experiment_new_ilp, ws/wss/udp, the connect()/QuestDB shape, Python 3.12/3.13 + PyArrow caveat. All 8 scripts smoke-tested against a local QuestDB (ws/wss/http land 100%; udp is best-effort/lossy as documented). --- python/README.md | 105 ++++++++++++++++++++-------------- python/backfill.py | 20 +++---- python/blotter.py | 18 +++--- python/csv_columnar_sender.py | 32 +++++------ python/csv_parallel_sender.py | 26 +++++---- python/dataframe_demo.py | 78 ++++++++++++------------- python/enrich_polars_demo.py | 33 ++++++----- python/read_bench.py | 20 +++---- python/web_blotter.py | 12 ++-- 9 files changed, 179 insertions(+), 165 deletions(-) diff --git a/python/README.md b/python/README.md index 213baf5..4041466 100644 --- a/python/README.md +++ b/python/README.md @@ -5,8 +5,12 @@ It replays a CSV of trades in a loop across N worker threads over QuestDB's QWP (WebSocket/UDP) and ILP/HTTP transports, with a probe that reads back the latest ingested timestamp and the serving node's live role. -Built on the QuestDB Python client (`questdb.ingress`), which wraps the same C/Rust client -as the other ports. +Built on the QuestDB Python client **5.0**, which wraps the same C/Rust client as the +other ports. The 5.0 shape: `questdb.connect()` returns one pooled `QuestDB` handle that +queries, bulk-loads DataFrames and lends row senders; `questdb.Sender` is the standalone +connection-level API for point-to-point ILP/HTTP, ILP/TCP and QWP/UDP. (The old +`from questdb.ingress import Client` split is gone; `questdb.ingress` still imports as a +deprecated 4.x shim.) ## Which script to use @@ -14,15 +18,15 @@ This port has grown a few scripts. Pick by what you need: | Script | Use it for | Path / speed | | --- | --- | --- | -| `csv_columnar_sender.py` | **High-throughput ingestion** - bulk CSV replay at volume | Columnar QWP (`Client.dataframe`, Arrow/polars). Fastest Python path; flat memory. | +| `csv_columnar_sender.py` | **High-throughput ingestion** - bulk CSV replay at volume | Columnar QWP (`db.dataframe`, Arrow/polars). Fastest Python path; flat memory. | | `csv_parallel_sender.py` | **HA / failover demos** - store-and-forward durability, the live probe, `qwpudp`/`ilp` comparison, and the per-event `row()` API | Row-by-row QWP/UDP/ILP. Correct but slow for volume - keep `--total-events` and the batch window small. | | `read_bench.py` | Measuring read/scan throughput (rows/s) of a table | Streaming `iter_arrow()` egress. | | `enrich_polars_demo.py` | Read a table as polars, enrich, write it back as polars | Streaming read + columnar write. | -| `dataframe_demo.py` | pandas/polars ingestion + egress round-trip showcase | `Sender.dataframe` / `Client.dataframe`. | +| `dataframe_demo.py` | pandas/polars ingestion + egress round-trip showcase | `db.dataframe` (pandas + polars), `db.execute`, dual egress. | **Row-by-row vs columnar.** `sender.row()` in a Python loop is the *slowest* path: every cell crosses the Python/Cython boundary under the GIL (the client's own perf notes call it "~16x -slower" than the columnar bulk path). `Client.dataframe` ships whole Arrow columns to the +slower" than the columnar bulk path). `db.dataframe` ships whole Arrow columns to the native client - the only Python path that reaches Java/Rust-class throughput. So use `csv_columnar_sender.py` for volume, and reach for `csv_parallel_sender.py` **only** when you need store-and-forward failover, which the columnar path deliberately bypasses (dataframe @@ -56,22 +60,23 @@ buffers, does ~52 MB/s). Two levers: ## Important: the client must be built from source -The QWP transports (`qwpws`/`qwpwss`/`qwpudp`) **and the query client / egress reader are not -in the PyPI release** (`questdb` 4.x on PyPI is ILP-only: `Protocol` has just -`Tcp/Tcps/Http/Https`, and there is no `Client`/`QueryResult`). They live on the -**`sm_qwp_dataframe_bench`** branch of +Client **5.0.0** is unreleased, so it is not on PyPI yet (`questdb` 4.x on PyPI is ILP-only: +`Protocol` has just `Tcp/Tcps/Http/Https`, no `QuestDB`/`connect`/`QueryResult`). The QWP +transports (`ws`/`wss`/`udp`), the `questdb.connect()` query/ingest handle and the egress +reader live on the **`jh_experiment_new_ilp`** branch of [`py-questdb-client`](https://github.com/questdb/py-questdb-client), which bumps the bundled C client to the QWP + egress build. You must build that branch. -Requirements: **Python ≥ 3.10** (the branch is 3.10+; the PyPI-era 3.9 venv will not work), -a Rust toolchain (`cargo`), and a C compiler. Build steps (a git worktree keeps your -`main` checkout untouched): +Requirements: **Python 3.12 or 3.13** (3.10+ works, but avoid CPython **3.14 + PyArrow 25.x**, +which segfaults on a threaded worker-first Arrow allocation - an upstream PyArrow/3.14 bug, not +the client; stay on a stable PyArrow 23.x/24.x), a Rust toolchain (`cargo`), and a C compiler. +Build steps (a git worktree keeps your `main` checkout untouched): ```bash cd ~/prj/python/py-questdb-client -git worktree add --detach /tmp/pyqdb_qwp origin/sm_qwp_dataframe_bench -cd /tmp/pyqdb_qwp -git submodule update --init # bundled c-questdb-client (QWP + egress) +git worktree add --detach /tmp/pyqdb_v5 origin/jh_experiment_new_ilp +cd /tmp/pyqdb_v5 +git submodule update --init --recursive # bundled c-questdb-client (QWP + egress) python3.12 -m venv /tmp/pyqdb_venv /tmp/pyqdb_venv/bin/pip install -U pip "cython>=3.1.2" "setuptools>=80.9.0" numpy pandas polars pyarrow @@ -81,9 +86,12 @@ python3.12 -m venv /tmp/pyqdb_venv Verify: ```python -from questdb.ingress import Protocol, Client # Client only exists on the QWP build +import questdb +from questdb import Protocol, QuestDB # top-level in 5.0 (was questdb.ingress.Client) +print(questdb.__version__, hasattr(questdb, 'connect')) +# -> 5.0.0 True print([p for p in dir(Protocol) if not p.startswith('_')]) -# -> ['Http', 'Https', 'QwpUdp', 'QwpWs', 'QwpWss', 'Tcp', 'Tcps'] +# -> ['Http', 'Https', 'Tcp', 'Tcps', 'Udp', 'Ws', 'Wss'] ``` Then run the scripts with that interpreter (`/tmp/pyqdb_venv/bin/python`). `pandas`, `polars`, @@ -142,12 +150,13 @@ and would otherwise drop the last three digits of a `TIMESTAMP_NS` source). ## Probe (QWP/WebSocket only) -For `qwp`, a background thread uses the pooled query client (`Client`) to run +For `qwp`, a background thread uses a `questdb.connect()` query handle to run `select timestamp from trades limit -1` each interval and prints the latest ingested timestamp, then `switch status` for the serving node's live role. `switch status` is Enterprise-only (needs SYSTEM ADMIN); on OSS the probe prints -`(live 'switch status' unavailable, ...)`. The client uses `target=any` (replica-fallback -reads) and fails over across the `--addrs` hosts. +`(live 'switch status' unavailable, ...)`. The handle uses `target=any` (replica-fallback +reads) and fails over across the `--addrs` hosts. (5.0 also exposes `db.server_info()` for the +handshake role/epoch/capabilities, should a probe want the role without `switch status`.) ## Dataframe demo (pandas / polars ingestion + egress) @@ -157,35 +166,43 @@ reads) and fails over across the `--addrs` hosts. /tmp/pyqdb_venv/bin/python dataframe_demo.py --addr localhost:9000 --csv ../trades20250728.csv.gz --rows 5000 ``` -- **pandas ingestion** via `Sender.dataframe(df, ...)` — the numpy-backed pandas planner. -- **polars ingestion** via `Client.dataframe(df, ...)` — the pooled QWP Arrow-columnar path, - which takes polars / pyarrow / any Arrow C Stream source natively. (Numpy-backed pandas is - **not** accepted by `Client.dataframe` — it raises `UnsupportedDataFrameShapeError` — so - pandas goes through `Sender.dataframe`, or convert with `pyarrow.Table.from_pandas`.) -- **egress** via `Client.query(sql).to_pandas()` and `.to_polars()` (also `.to_arrow()` / +- **DDL** via `db.execute(sql)` — run-and-drain (drops the table), no HTTP `/exec` side channel. +- **pandas ingestion** via `db.dataframe(df, ...)` — numpy-backed pandas is accepted directly on + the columnar handle (4.x's `Client.dataframe` rejected it). One caveat, exercised by the demo: + columnar v1 wants SYMBOL columns as pandas `Categorical` (or `string[pyarrow]`); plain `object` + strings land as VARCHAR, and forcing them to SYMBOL raises a now-per-column + `UnsupportedDataFrameShapeError`. The demo casts `symbol`/`side` to `category`. +- **polars ingestion** via `db.dataframe(df, ...)` — the same call takes polars / pyarrow / any + Arrow C Stream source natively. +- **egress** via `db.query(sql).to_pandas()` and `.to_polars()` (also `.to_arrow()` / `iter_pandas()`; the result exposes the Arrow PyCapsule interface for zero-copy consumers). ## Differences from the Java/Rust ports - **The client is pre-release / build-from-source** (see above) — not `pip install questdb`. -- **Ingestion and querying are split across two classes**: `Sender` (ingest) and `Client` - (pooled QWP: query + Arrow-columnar dataframe ingest). The probe uses `Client`. -- **Auto-flush exists** in the Python client (unlike the Rust client) and is verified to work - over QWP (row-threshold flush fires mid-stream; see Validated). This port sets - `auto_flush=off` and flushes at the same batch boundaries as the Java/Rust ports for - identical cadence. -- **Config-string scheme is `qwpws`/`qwpwss` for both the sender and the query client.** - `qwpws`/`qwpwss` are the deprecated aliases (the Rust/C++ ports and their readers use the - newer `ws`/`wss`), but the Python binding's `Protocol` enum only exposes `QwpWs`/`QwpWss`, so - `ws`/`wss` are rejected here (`ValueError: Invalid value for Protocol`). Python must stay on - `qwpws`/`qwpwss` until the binding adds the `ws`/`wss` aliases. -- **The probe has no handshake-role fallback** — because the Python binding does not expose it. - Java/Rust fall back to the QWP handshake role when `switch status` is unavailable. That role - exists in the underlying C client (`line_reader_server_info_role`/`_role_byte`) and the Rust - reader (`server_info()`), but the Python `Client` on this branch does not wrap it, so this - port prints "unavailable" on OSS. Likewise the query path's `on_failover_reset` callback is - wired internally in Cython but not surfaced to Python, so there is no user-facing failover - narration. +- **One handle for QWP, plus a standalone `Sender`.** `questdb.connect()` returns a pooled + `QuestDB` handle that queries (`db.query`), runs DDL (`db.execute`), bulk-loads DataFrames + (`db.dataframe`) and lends row senders (`db.sender()`). The columnar scripts and the probe use + that handle. The row-by-row HA sender keeps a standalone `questdb.Sender` **per worker** — + each with its own `sender_id` and `sf_dir` for independent store-and-forward — which the shared + pool would not give. +- **Auto-flush exists** on the standalone `Sender` (unlike the Rust client) and is verified over + QWP (row-threshold flush fires mid-stream; see Validated). This port sets `auto_flush=off` and + flushes at the same batch boundaries as the Java/Rust ports. (The **pooled** `db.sender()` has + no auto-flush by default — flushing there is always explicit.) +- **Config-string scheme is `ws`/`wss` (and `udp`)**, matching the Rust/C ports — the 5.0 binding + renamed the Python-only `qwpws`/`qwpwss`/`qwpudp` away, so a conf string now copies verbatim + between clients. (`--protocol qwp`/`qwpudp`/`ilp` are just this port's transport selectors; they + map to `ws`/`udp`/`http` on the wire.) +- **Naive `datetime` is UTC everywhere** in 5.0 (4.x read naive *scalars* in the machine's local + zone). These scripts already pass tz-aware `datetime.now(timezone.utc)` / integer nanos, so the + change is a no-op here — but a naive value now emits a one-per-process `UserWarning`. +- **The probe still has no handshake-role fallback** — it prints "unavailable" on OSS when + `switch status` is missing. 5.0 *does* wrap the handshake role as `db.server_info().role` + (`ServerInfo`: role/epoch/capabilities/cluster/node/zone), so a fallback is now possible; this + port has not wired it in. Connection narration is also available now via the `connection_listener=` + callback on `connect()`/`Sender` (`ConnectionEvent`s: connected/disconnected/reconnected/failed_over/...), + which this port does not use. - Threads (like Java/Rust). Python's GIL is released during client I/O, but row-building is Python-level, so row-by-row throughput is well below Java/Rust — use the dataframe path for volume. diff --git a/python/backfill.py b/python/backfill.py index f4672af..d61efc4 100644 --- a/python/backfill.py +++ b/python/backfill.py @@ -13,7 +13,7 @@ python backfill.py --addr host:9000 --num-senders 4 \ --token "$QDB_TOKEN" --tls-verify unsafe_off -Requires the QWP/egress build of the client (see README.md). +Requires the questdb 5.0 client (see README.md). """ import argparse @@ -27,7 +27,7 @@ import numpy as np import polars as pl -from questdb.ingress import Client +import questdb NS_PER_DAY = 86_400_000_000_000 @@ -36,14 +36,8 @@ def use_tls(args): return bool(args.tls or args.token or (args.username and args.password)) -def display_scheme(args): - return "wss" if use_tls(args) else "ws" - - def build_conf(args): - # Python's binding requires the qwpws/qwpwss scheme (it rejects ws/wss); only the printed - # [conf] line shows wss/ws, via display_scheme(). - scheme = "qwpwss" if use_tls(args) else "qwpws" + scheme = "wss" if use_tls(args) else "ws" parts = [f"{scheme}::addr={args.addr};"] if args.token: parts.append(f"token={args.token};") @@ -84,7 +78,7 @@ def run_worker(wid, start_index, count, deck, args, start_ns, step_ns, counts): conf = build_conf(args) sent = 0 pos = start_index % m - with Client.from_conf(conf) as client: + with questdb.connect(conf) as db: while sent < count: cn = min(args.chunk_rows, count - sent, m - pos) sl = deck.slice(pos, cn) # zero-copy view into the deck @@ -94,7 +88,7 @@ def run_worker(wid, start_index, count, deck, args, start_ns, step_ns, counts): pl.Series("timestamp", ts).cast(pl.Datetime("ns", "UTC")), (pl.lit("backfill-") + pl.int_range(gidx, gidx + cn).cast(pl.Utf8)).alias("trade_id"), ) - client.dataframe(chunk, table_name=args.table, symbols=["symbol", "side"], at="timestamp") + db.dataframe(chunk, table_name=args.table, symbols=["symbol", "side"], at="timestamp") sent += cn counts[wid] = sent pos = (pos + cn) % m @@ -106,7 +100,7 @@ def preflight(conf, timeout_s): def probe(): try: - with Client.from_conf(conf) as c: + with questdb.connect(conf) as c: c.query("select 1").to_polars() outcome["ok"] = True except Exception as e: # noqa: BLE001 @@ -175,7 +169,7 @@ def main(argv): return 2 span_days = (end_ns - start_ns) / NS_PER_DAY - print(f"[conf] {display_scheme(args)} tls={use_tls(args)} " + print(f"[conf] {'wss' if use_tls(args) else 'ws'} tls={use_tls(args)} " f"auth={'token' if args.token else 'basic' if args.username else 'none'} | addrs: {args.addr}") print(f"[backfill] window {start.isoformat()} .. {end.isoformat()} ({span_days:.2f} days) @ " f"{args.rate_per_day:,.0f} rows/day -> {n:,} rows, one every {step_ns / 1000:.1f} us, " diff --git a/python/blotter.py b/python/blotter.py index 228ce40..408311c 100644 --- a/python/blotter.py +++ b/python/blotter.py @@ -28,7 +28,7 @@ order by timestamp desc limit 50) select *, avg(bid_price) over (partition by symbol) \ as avg50 from x" --addr host:9000 --token TOK --tls-verify unsafe_off -Requires the QWP/egress build of the client (see README.md). Ctrl+C to quit. +Requires the questdb 5.0 client (see README.md). Ctrl+C to quit. """ import argparse @@ -39,7 +39,7 @@ import polars as pl -from questdb.ingress import Client +import questdb def use_tls(args): @@ -47,7 +47,7 @@ def use_tls(args): def build_conf(args): - scheme = "qwpwss" if use_tls(args) else "qwpws" # Python requires the qwp* scheme + scheme = "wss" if use_tls(args) else "ws" parts = [f"{scheme}::addr={args.addr};"] if args.token: parts.append(f"token={args.token};") @@ -161,10 +161,10 @@ def main(argv): interval = 1.0 / args.rate scheme = "wss" if use_tls(args) else "ws" - def render(client): + def render(db): t0 = time.perf_counter() try: - df = client.query(sql).to_polars() + df = db.query(sql).to_polars() ms = (time.perf_counter() - t0) * 1000.0 with pl.Config(tbl_rows=min(df.height, 100), tbl_cols=-1, tbl_hide_dataframe_shape=True): body = str(df) @@ -176,17 +176,17 @@ def render(client): return f"SQL: {sql_line}\n {status}\n{body}" if args.once: - with Client.from_conf(conf) as client: + with questdb.connect(conf) as db: print(f"[conf] {scheme} tls={use_tls(args)} | addr: {args.addr}") - print(render(client)) + print(render(db)) return 0 sys.stdout.write("\033[2J\033[?25l") # clear once, hide cursor try: - with Client.from_conf(conf) as client: + with questdb.connect(conf) as db: while True: t0 = time.perf_counter() - draw(render(client)) + draw(render(db)) time.sleep(max(0.0, interval - (time.perf_counter() - t0))) except KeyboardInterrupt: pass diff --git a/python/csv_columnar_sender.py b/python/csv_columnar_sender.py index 599679f..fdbcf19 100644 --- a/python/csv_columnar_sender.py +++ b/python/csv_columnar_sender.py @@ -2,18 +2,18 @@ """Fast CSV replay sender for QuestDB - the COLUMNAR QWP path. This is the throughput-oriented Python sender. It replays a CSV to a QuestDB -``trades`` table over the QWP WebSocket transport using the *column-major* client -path (``Client.dataframe``), which ships whole Arrow columns to the native Rust +``trades`` table over the QWP WebSocket transport using the *column-major* handle +path (``db.dataframe``), which ships whole Arrow columns to the native Rust client in bulk. That is the ONLY Python path that gets near Java/Rust throughput. -Why not row-by-row? ``Sender.row(...)`` in a Python loop is the SLOWEST path: every +Why not row-by-row? ``sender.row(...)`` in a Python loop is the SLOWEST path: every cell crosses the Python/Cython boundary under the GIL. The client's own perf notes call the row API a "per-cell call ... ~16x slower" than the columnar bulk path. -The column path (``Client.dataframe``) stores raw pointers into the Arrow buffers +The column path (``db.dataframe``) stores raw pointers into the Arrow buffers ("no data copy at append time") and does one memcpy per column at flush. Use it. Memory stays flat: the CSV is loaded once into a polars DataFrame, and rows are -sent in bounded, zero-copy slices. ``Client.dataframe`` additionally streams each +sent in bounded, zero-copy slices. ``db.dataframe`` additionally streams each send internally at ``max_rows_per_batch`` (16384) rows, so nothing balloons - unlike the row-by-row sender, which buffered up to 100k rows before its first flush and grew without bound when Python could not reach that threshold fast enough. @@ -24,7 +24,7 @@ so the target looks live and there is no out-of-order within a worker. Auth / TLS (QuestDB Enterprise): ``--token`` (bearer) or ``--username``/``--password`` -(basic) turn on TLS automatically (scheme ``qwpwss``); ``--tls`` forces TLS with no +(basic) turn on TLS automatically (scheme ``wss``); ``--tls`` forces TLS with no auth. Certificate verification is on by default; ``--tls-verify unsafe_off`` for self-signed certs. Multiple ``--addrs`` give QWP failover. @@ -39,7 +39,7 @@ [--timestamp-from-file] \ [--token TOK | --username U --password P] [--tls-verify on|unsafe_off] -Requires the QWP/egress build of the client (see README.md). +Requires the questdb 5.0 client (see README.md). """ import argparse @@ -53,7 +53,7 @@ import numpy as np import polars as pl -from questdb.ingress import Client +import questdb def use_tls(args): @@ -61,8 +61,8 @@ def use_tls(args): def build_conf(args): - """QWP connect string for the columnar client, with optional auth + TLS + failover.""" - scheme = "qwpwss" if use_tls(args) else "qwpws" + """QWP connect string for the columnar handle, with optional auth + TLS + failover.""" + scheme = "wss" if use_tls(args) else "ws" addrs = ",".join(a.strip() for a in args.addrs.split(",") if a.strip()) parts = [f"{scheme}::addr={addrs};"] if args.token: @@ -83,7 +83,7 @@ def preflight(conf, timeout_s): def probe(): try: - with Client.from_conf(conf) as c: + with questdb.connect(conf) as c: c.query("select 1").to_polars() outcome["ok"] = True except Exception as e: # noqa: BLE001 @@ -129,7 +129,7 @@ def load_base(path, need_timestamp): def run_worker(wid, events, base, args, counts): """Send `events` rows for this worker: cycle over the base frame in bounded slices, attach trade_id (+ a live timestamp unless reading it from file), and - ship each slice columnar via Client.dataframe.""" + ship each slice columnar via db.dataframe.""" m = base.height from_file = args.timestamp_from_file conf = build_conf(args) @@ -142,7 +142,7 @@ def run_worker(wid, events, base, args, counts): pos = 0 last_ts = 0 pace_start = time.perf_counter_ns() - with Client.from_conf(conf) as client: + with questdb.connect(conf) as db: while sent < events: n = min(args.chunk_rows, events - sent, m - pos) sl = base.slice(pos, n) # zero-copy view into the base frame @@ -156,7 +156,7 @@ def run_worker(wid, events, base, args, counts): add.append(pl.Series("timestamp", ts).cast(pl.Datetime("ns", "UTC")).alias("timestamp")) chunk = sl.with_columns(add) - client.dataframe(chunk, table_name=args.destination_table, symbols=["symbol", "side"], at="timestamp") + db.dataframe(chunk, table_name=args.destination_table, symbols=["symbol", "side"], at="timestamp") sent += n counts[wid] = sent @@ -178,7 +178,7 @@ def main(argv): ap.add_argument("--num-senders", type=int, default=1, help="worker threads, each with its own columnar client") ap.add_argument("--chunk-rows", type=int, default=100_000, - help="rows per Client.dataframe call (pacing granularity + peak-memory bound)") + help="rows per db.dataframe call (pacing granularity + peak-memory bound)") ap.add_argument("--rate", type=int, default=0, help="target aggregate rows/second across ALL workers (0 = flat out)") ap.add_argument("--csv", default="../trades20250728.csv.gz") @@ -190,7 +190,7 @@ def main(argv): ap.add_argument("--token", default=None, help="bearer token (turns on TLS)") ap.add_argument("--username", default=None, help="basic-auth username (turns on TLS)") ap.add_argument("--password", default=None, help="basic-auth password") - ap.add_argument("--tls", action="store_true", help="force TLS (qwpwss) with no auth") + ap.add_argument("--tls", action="store_true", help="force TLS (wss) with no auth") ap.add_argument("--tls-verify", choices=["on", "unsafe_off"], default="on", help="certificate verification; use unsafe_off for self-signed") ap.add_argument("--connect-timeout", type=float, default=10.0, diff --git a/python/csv_parallel_sender.py b/python/csv_parallel_sender.py index 5f7833e..e1be680 100644 --- a/python/csv_parallel_sender.py +++ b/python/csv_parallel_sender.py @@ -4,16 +4,17 @@ Replays a CSV of trades in a loop across N worker threads over one of three transports (``--protocol``): - * ``qwp`` - QWP over WebSocket: store-and-forward (un-acked frames spill to disk and - replay after an outage), transactional commit, multi-host failover. A - background probe polls the latest ingested timestamp and the serving - node's live role over a QWP query client (``Client``). - * ``qwpudp`` - QWP over UDP: fire-and-forget datagrams to :9007. Ingest-only, + * ``qwp`` - QWP over WebSocket (``ws``/``wss``): store-and-forward (un-acked frames + spill to disk and replay after an outage), transactional commit, + multi-host failover. A background probe polls the latest ingested + timestamp and the serving node's live role over a ``questdb.connect()`` + query handle. + * ``qwpudp`` - QWP over UDP (``udp``): fire-and-forget datagrams to :9007. Ingest-only, unauthenticated, single-endpoint, best-effort (no ack, no failover). * ``ilp`` - the legacy ILP/HTTP transport. -Requires the QWP/egress build of the questdb Python client (see README.md); the PyPI -release does not expose QWP or the query client. +Requires the questdb 5.0 client (see README.md); the current PyPI release does not +expose QWP or the query handle. """ import argparse @@ -25,7 +26,8 @@ import time from datetime import datetime, timezone -from questdb.ingress import Sender, Client, TimestampNanos, ServerTimestamp +import questdb +from questdb import Sender, TimestampNanos, ServerTimestamp PROBE_QUERY = "select timestamp from trades limit -1" # Enterprise lifecycle status of whichever node the query client is connected to. Returns @@ -92,7 +94,7 @@ def build_ingest_conf(args, worker_id): """Ingestion connect string for the configured transport (same keys as the Rust port).""" addrs = addr_list(args) if args.protocol == "qwpudp": - return f"qwpudp::addr={addrs[0]};auto_flush=off;" + return f"udp::addr={addrs[0]};auto_flush=off;" if args.protocol == "ilp": scheme = "https" if tls(args) else "http" @@ -104,7 +106,7 @@ def build_ingest_conf(args, worker_id): return "".join(parts) # qwp (WebSocket) - scheme = "qwpwss" if tls(args) else "qwpws" + scheme = "wss" if tls(args) else "ws" who = f"{args.sender_id}-{worker_id}" sf = os.path.join(args.store_forward_dir, who) os.makedirs(sf, exist_ok=True) @@ -126,7 +128,7 @@ def build_client_conf(args): """Connect string for the QWP query client (probe): target=any (replica-fallback reads), failover on, zone bias. Same hosts/auth as the senders.""" addrs = addr_list(args) - scheme = "qwpwss" if tls(args) else "qwpws" + scheme = "wss" if tls(args) else "ws" parts = [f"{scheme}::addr={','.join(addrs)};", "target=any;", "failover=true;"] _append_auth(parts, args) if tls(args): @@ -275,7 +277,7 @@ def _switch_status_roles(client): def run_probe(args, stop): conf = build_client_conf(args) try: - client = Client.from_conf(conf) + client = questdb.connect(conf) except Exception as e: print(f"[query client] connect failed: {e}") return diff --git a/python/dataframe_demo.py b/python/dataframe_demo.py index 163958d..c9476ec 100644 --- a/python/dataframe_demo.py +++ b/python/dataframe_demo.py @@ -1,46 +1,44 @@ #!/usr/bin/env python3 -"""Pandas and polars ingestion + egress round-trip over QWP/WebSocket. +"""Pandas and polars ingestion + egress round-trip over one QuestDB 5.0 handle. -Showcases the columnar/dataframe capabilities the row-by-row sender does not: +Showcases the columnar/dataframe capabilities the row-by-row sender does not, all +through a single ``questdb.connect()`` handle: - * pandas ingestion via ``Sender.dataframe`` (the numpy-backed pandas planner). - * polars ingestion via ``Client.dataframe`` (the pooled QWP Arrow-columnar path; - it takes polars / pyarrow / any Arrow C Stream source natively — numpy-backed - pandas is not accepted there, so pandas goes through ``Sender.dataframe``). - * egress via ``Client.query(...).to_pandas()`` and ``.to_polars()``. + * DDL via ``db.execute(sql)`` - run-and-drain, no HTTP ``/exec`` side channel. + * pandas ingestion via ``db.dataframe`` - numpy-backed pandas is accepted directly + (SYMBOL columns as pandas ``Categorical``); in 4.x the columnar ``Client.dataframe`` + rejected numpy pandas outright and it had to go through ``Sender.dataframe``. + * polars ingestion via ``db.dataframe`` - the same call takes polars / pyarrow / + any Arrow C Stream source natively. + * egress via ``db.query(...).to_pandas()`` and ``.to_polars()``. + +One handle for DDL, ingest and query - the 5.0 shape. (The standalone ``Sender`` +and its ``Sender.dataframe`` still exist for point-to-point ILP/HTTP, ILP/TCP and +QWP/UDP needs; over ``ws``/``wss`` they route to the same direct columnar path.) Usage: python dataframe_demo.py [--addr localhost:9000] [--csv ../trades20250728.csv.gz] [--rows 5000] -Requires the QWP/egress build of the questdb client (see README.md). +Requires the questdb 5.0 client (see README.md). """ import argparse import sys import time -import urllib.parse -import urllib.request import pandas as pd import polars as pl -from questdb.ingress import Sender, Client +import questdb TABLE = "df_demo" -def exec_sql(addr, sql): - """Run a statement over the HTTP /exec endpoint (same host, HTTP port).""" - url = f"http://{addr}/exec?" + urllib.parse.urlencode({"query": sql}) - with urllib.request.urlopen(url, timeout=10) as resp: - resp.read() - - -def wait_for_count(client, table, at_least, timeout_s=30): +def wait_for_count(db, table, at_least, timeout_s=30): deadline = time.monotonic() + timeout_s while time.monotonic() < deadline: try: - n = int(client.query(f"select count() from {table}").to_pandas().iloc[0, 0]) + n = int(db.query(f"select count() from {table}").to_pandas().iloc[0, 0]) if n >= at_least: return n except Exception: @@ -50,12 +48,14 @@ def wait_for_count(client, table, at_least, timeout_s=30): def load_pandas(csv_path, rows): - """Load the CSV into a pandas DataFrame with dtypes the ingestion path accepts - (object strings, float64, tz-aware nanosecond timestamps).""" + """Load the CSV into a numpy-backed pandas DataFrame the columnar handle ingests + directly: float64 values, tz-aware nanosecond timestamps, and the SYMBOL columns + as pandas ``Categorical`` (what ``db.dataframe``'s columnar v1 path wants for a + SYMBOL - plain ``object`` strings land as VARCHAR instead).""" raw = pd.read_csv(csv_path, nrows=rows) return pd.DataFrame({ - "symbol": raw["symbol"].astype("object"), - "side": raw["side"].astype("object"), + "symbol": raw["symbol"].astype("category"), + "side": raw["side"].astype("category"), "price": raw["price"].astype("float64"), "amount": raw["amount"].astype("float64"), "timestamp": pd.to_datetime(raw["timestamp"], utc=True).dt.as_unit("ns"), @@ -74,38 +74,36 @@ def main(argv): pldf = pl.from_pandas(pdf) print(f"built polars DataFrame: {pldf.height} rows\n") - exec_sql(args.addr, f"drop table if exists {TABLE}") + with questdb.connect(f"ws::addr={args.addr};") as db: + db.execute(f"drop table if exists {TABLE}") # DDL, run-and-drain - # --- INGESTION --- - # pandas -> Sender.dataframe (numpy-backed pandas planner) over QWP/WebSocket. - with Sender.from_conf(f"qwpws::addr={args.addr};auto_flush=off;") as sender: - sender.dataframe(pdf, table_name=TABLE, symbols=["symbol", "side"], at="timestamp") - sender.flush() - print(f"[ingest] pandas via Sender.dataframe: {pdf.shape[0]} rows") + # --- INGESTION (both frames through the one handle) --- + # numpy-backed pandas - accepted directly in 5.0 (was rejected in 4.x). + db.dataframe(pdf, table_name=TABLE, symbols=["symbol", "side"], at="timestamp") + print(f"[ingest] pandas via db.dataframe: {pdf.shape[0]} rows") - with Client.from_conf(f"qwpws::addr={args.addr};") as client: - # polars -> Client.dataframe (Arrow-columnar path, polars is native). - client.dataframe(pldf, table_name=TABLE, symbols=["symbol", "side"], at="timestamp") - print(f"[ingest] polars via Client.dataframe: {pldf.height} rows") + # polars - the same call, Arrow-columnar and native. + db.dataframe(pldf, table_name=TABLE, symbols=["symbol", "side"], at="timestamp") + print(f"[ingest] polars via db.dataframe: {pldf.height} rows") expected = pdf.shape[0] + pldf.height - n = wait_for_count(client, TABLE, expected) + n = wait_for_count(db, TABLE, expected) print(f"[ingest] server applied {n} rows (expected {expected})\n") # --- EGRESS --- - pd_out = client.query( + pd_out = db.query( f"select symbol, count() n, round(avg(price), 2) avg_price " f"from {TABLE} order by n desc limit 5" ).to_pandas() - print("[egress] Client.query(...).to_pandas():") + print("[egress] db.query(...).to_pandas():") print(pd_out.to_string(index=False)) print() - pl_out = client.query( + pl_out = db.query( f"select side, count() n, round(sum(amount), 4) total_amount " f"from {TABLE} group by side order by side" ).to_polars() - print("[egress] Client.query(...).to_polars():") + print("[egress] db.query(...).to_polars():") print(pl_out) return 0 diff --git a/python/enrich_polars_demo.py b/python/enrich_polars_demo.py index 180e4d1..9d041b9 100644 --- a/python/enrich_polars_demo.py +++ b/python/enrich_polars_demo.py @@ -6,26 +6,27 @@ and ingests each enriched batch before pulling the next, so peak memory is one batch (not all ``--limit`` rows): - * READ - ``Client.query(sql).iter_arrow()`` yields ``pyarrow.RecordBatch`` chunks + * READ - ``db.query(sql).iter_arrow()`` yields ``pyarrow.RecordBatch`` chunks one at a time from a server-side cursor (verified streaming in the client: ``iter_arrow`` pulls via ``line_reader_cursor_next_batch``; ``to_polars`` by contrast is "materialise-whole" and would buffer every row in RAM); ``pl.from_arrow(batch)`` wraps each into a polars DataFrame (zero-copy). * ENRICH - add an ``enriched_rnd`` column: a random ``A``-``Z`` value per row. - * WRITE - ``Client.dataframe(chunk, ...)`` ingests each polars chunk into + * WRITE - ``db.dataframe(chunk, ...)`` ingests each polars chunk into ``enriched__demo`` over the QWP Arrow-columnar path. Both the read and the write are polars DataFrames. No pandas in the round-trip. -The read and the write use separate ``Client`` connections so the open read -stream and the ingest never share a socket. +A single ``questdb.connect()`` handle serves both: the open read stream holds one +pooled connection while each write borrows a separate direct one, so the read cursor +and the ingest never share a socket. Column typing is automatic: QuestDB SYMBOL columns come back as polars ``Categorical`` -and VARCHAR as ``String``, and ``Client.dataframe``'s default ``symbols='auto'`` maps +and VARCHAR as ``String``, and ``db.dataframe``'s default ``symbols='auto'`` maps ``Categorical -> SYMBOL`` and ``String -> VARCHAR`` on the way back in. ``enriched_rnd`` is built as a ``Categorical`` so it too lands as a SYMBOL. Auth / TLS (QuestDB Enterprise): pass ``--token`` (bearer) or ``--username``/``--password`` -(basic). Either turns on TLS automatically (scheme ``qwpwss``); ``--tls`` forces TLS with +(basic). Either turns on TLS automatically (scheme ``wss``); ``--tls`` forces TLS with no auth. Certificate verification is on by default; use ``--tls-verify unsafe_off`` for self-signed certs. The same credentials are applied to the HTTP ``/exec`` calls used for drop/verify. @@ -39,7 +40,7 @@ By default the target table is dropped and recreated on each run; pass ``--keep`` to append instead. ``--limit`` defaults to the latest 200 million rows by designated timestamp, streamed ascending so re-ingest stays in timestamp order. Requires the -QWP/egress build of the client (see README.md). +questdb 5.0 client (see README.md). """ import argparse @@ -54,7 +55,7 @@ import numpy as np import polars as pl -from questdb.ingress import Client +import questdb LETTERS = np.array(list("ABCDEFGHIJKLMNOPQRSTUVWXYZ")) @@ -64,8 +65,8 @@ def use_tls(args): def build_conf(args): - """QWP connect string for the reader/writer, with optional auth + TLS.""" - scheme = "qwpwss" if use_tls(args) else "qwpws" + """QWP connect string for the handle, with optional auth + TLS.""" + scheme = "wss" if use_tls(args) else "ws" parts = [f"{scheme}::addr={args.addr};"] if args.token: parts.append(f"token={args.token};") @@ -131,7 +132,7 @@ def main(argv): ap.add_argument("--token", default=None, help="bearer token (turns on TLS)") ap.add_argument("--username", default=None, help="basic-auth username (turns on TLS)") ap.add_argument("--password", default=None, help="basic-auth password") - ap.add_argument("--tls", action="store_true", help="force TLS (qwpwss) with no auth") + ap.add_argument("--tls", action="store_true", help="force TLS (wss) with no auth") ap.add_argument("--tls-verify", choices=["on", "unsafe_off"], default="on", help="certificate verification; use unsafe_off for self-signed") args = ap.parse_args(argv) @@ -156,15 +157,17 @@ def main(argv): total = 0 t0 = time.monotonic() - # Separate connections: the read stream stays open while each chunk is written. - with Client.from_conf(conf) as reader, Client.from_conf(conf) as writer: - result = reader.query(sql) + # One handle owns the pool: the open read stream holds one pooled connection while + # each db.dataframe() write borrows a separate direct columnar connection - so the + # read cursor and the ingest never share a socket, from a single QuestDB handle. + with questdb.connect(conf) as db: + result = db.query(sql) for i, batch in enumerate(result.iter_arrow()): chunk = pl.from_arrow(batch) # RecordBatch -> polars, zero-copy if chunk.height == 0: continue chunk = enrich(chunk, rng) - writer.dataframe(chunk, table_name=target, at=ts_col) + db.dataframe(chunk, table_name=target, at=ts_col) total += chunk.height print(f"[chunk {i}] {chunk.height:,} rows enriched + written " f"(running total {total:,})") diff --git a/python/read_bench.py b/python/read_bench.py index 83482c4..26d6a8b 100644 --- a/python/read_bench.py +++ b/python/read_bench.py @@ -7,7 +7,7 @@ not expose) - so it is a data-throughput figure, generally larger than the on-wire rate. Fastest path, by design: consume the QWP query cursor's Arrow batches directly -(``Client.query(sql).iter_arrow()``) and just tally ``batch.num_rows``. No pandas +(``db.query(sql).iter_arrow()``) and just tally ``batch.num_rows``. No pandas or polars conversion - the Rust client already decodes the wire into Arrow inside ``iter_arrow`` (that decode is the unavoidable "read" work, done in native code), so the Python loop does nothing but add row counts. Converting each batch to @@ -23,7 +23,7 @@ per-connection wall. ``--readers 1`` (default) is the plain single stream. Auth / TLS (QuestDB Enterprise): pass ``--token`` (bearer) or ``--username``/``--password`` -(basic). Either turns on TLS automatically (scheme ``qwpwss``); ``--tls`` forces TLS with +(basic). Either turns on TLS automatically (scheme ``wss``); ``--tls`` forces TLS with no auth. Certificate verification is on by default; use ``--tls-verify unsafe_off`` for self-signed certs. @@ -32,7 +32,7 @@ [--addr host:9000] [--limit 10000000] [--readers 1] \ [--token TOK | --username U --password P] [--tls] [--tls-verify on|unsafe_off] -Requires the QWP/egress build of the client (see README.md). +Requires the questdb 5.0 client (see README.md). """ import argparse @@ -41,7 +41,7 @@ import time from datetime import datetime, timezone -from questdb.ingress import Client +import questdb def use_tls(args): @@ -50,7 +50,7 @@ def use_tls(args): def build_conf(args): """QWP connect string for the reader(s), with optional auth + TLS.""" - scheme = "qwpwss" if use_tls(args) else "qwpws" + scheme = "wss" if use_tls(args) else "ws" parts = [f"{scheme}::addr={args.addr};"] if args.token: parts.append(f"token={args.token};") @@ -76,8 +76,8 @@ def split_queries(args, conf): if readers <= 1: return [f"select * from {table} limit -{limit}"] # Range of the latest `limit` rows, as epoch nanoseconds. - with Client.from_conf(conf) as c: - mm = c.query( + with questdb.connect(conf) as db: + mm = db.query( f"select min({ts}) lo, max({ts}) hi " f"from (select {ts} from {table} limit -{limit})" ).to_polars() @@ -102,8 +102,8 @@ def split_queries(args, conf): def run_reader(idx, sql, conf, counts, byts, errors, sample_n, sample_out): try: - with Client.from_conf(conf) as client: - result = client.query(sql) + with questdb.connect(conf) as db: + result = db.query(sql) n = 0 b = 0 for batch in result.iter_arrow(): @@ -140,7 +140,7 @@ def main(argv): ap.add_argument("--token", default=None, help="bearer token (turns on TLS)") ap.add_argument("--username", default=None, help="basic-auth username (turns on TLS)") ap.add_argument("--password", default=None, help="basic-auth password") - ap.add_argument("--tls", action="store_true", help="force TLS (qwpwss) with no auth") + ap.add_argument("--tls", action="store_true", help="force TLS (wss) with no auth") ap.add_argument("--tls-verify", choices=["on", "unsafe_off"], default="on", help="certificate verification; use unsafe_off for self-signed") args = ap.parse_args(argv) diff --git a/python/web_blotter.py b/python/web_blotter.py index 218ec96..73af393 100644 --- a/python/web_blotter.py +++ b/python/web_blotter.py @@ -16,7 +16,7 @@ --addr host:9000 --token-file ~/prj/python/ent_tokens.txt --token-label disaster \ --tls-verify unsafe_off [--port 8080] -Requires the QWP/egress build of the client (see README.md). +Requires the questdb 5.0 client (see README.md). """ import argparse @@ -27,7 +27,7 @@ from datetime import datetime from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer -from questdb.ingress import Client +import questdb def use_tls(args): @@ -35,7 +35,7 @@ def use_tls(args): def build_conf(args): - scheme = "qwpwss" if use_tls(args) else "qwpws" + scheme = "wss" if use_tls(args) else "ws" parts = [f"{scheme}::addr={args.addr};"] if args.token: parts.append(f"token={args.token};") @@ -186,7 +186,7 @@ def main(argv): interval_ms = max(50, int(1000 / max(0.1, args.rate))) conf = build_conf(args) - client = Client.from_conf(conf) + db = questdb.connect(conf) lock = threading.Lock() page = (HTML.replace("__TITLE__", f"blotter: {args.table or 'query'}") .replace("__SQL__", " ".join(sql.split())) @@ -215,7 +215,7 @@ def do_GET(self): elif self.path.startswith("/data"): try: with lock: - df = client.query(sql).to_polars() + df = db.query(sql).to_polars() self._send(200, "application/json", json.dumps(df_to_payload(df))) except Exception as e: # noqa: BLE001 self._send(200, "application/json", json.dumps({"error": str(e)})) @@ -240,7 +240,7 @@ class Server(ThreadingHTTPServer): pass finally: httpd.server_close() - client.close() + db.close() return 0 From a4feb1e24d84f88e7da232d41ef3978a49dd4297 Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 22 Jul 2026 14:03:55 +0200 Subject: [PATCH 31/79] go: add QWP HA sender + concurrent query probe A Go port of the Java/Rust/C++ CsvParallelSender on the go-questdb-client v4 QWP (WebSocket) transport. Replays a CSV across N worker goroutines, each with its own store-and-forward sender and multi-host failover, while a background probe queries the latest ingested timestamp and the serving node's role via a QwpQueryClient - ingest and query concurrently, with HA. - QWP (default): per-worker sf_dir slot, failover, acked-vs-submitted reporting (FlushAndGetSequence + AwaitAckedFsn). Empty --store-forward-dir selects memory mode (no disk durability) for raw happy-path throughput. - ILP over HTTP (--protocol ilp): ingest only (QWP-only query client). The full QWP transport is on the client's main, post-v4.2.0 and unreleased, so go.mod uses a local replace directive (documented in go/README.md). Smoke-tested against a local QuestDB: build/vet clean, ingest + probe work, memory-mode ingest ~1.33M rows/s vs disk-SF ~117k (fsync-bound) locally. --- go/README.md | 49 ++++++ go/go.mod | 18 ++ go/go.sum | 94 ++++++++++ go/main.go | 484 +++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 645 insertions(+) create mode 100644 go/README.md create mode 100644 go/go.mod create mode 100644 go/go.sum create mode 100644 go/main.go diff --git a/go/README.md b/go/README.md new file mode 100644 index 0000000..d975ba8 --- /dev/null +++ b/go/README.md @@ -0,0 +1,49 @@ +# Go HA sender (QWP) + +A Go port of the Java/Rust/C++ `CsvParallelSender`, on the +[`go-questdb-client`](https://github.com/questdb/go-questdb-client) **v4 QWP +(WebSocket)** transport. It replays a CSV of trades in a loop across N worker +goroutines **and** runs a concurrent probe that queries the latest ingested +timestamp and the serving node's role — ingest and query at the same time, with +HA (per-worker store-and-forward + multi-host failover). + +## Build note (unreleased QWP) + +The full QWP transport (store-and-forward, typed errors, failover, query client) +is on the client's `main` but **not in a published tag yet** (it landed after +`v4.2.0`). `go.mod` therefore has a `replace` pointing at a local checkout: + +``` +replace github.com/questdb/go-questdb-client/v4 => /Users/j/prj/go/go-questdb-client +``` + +Adjust that path for your machine (or drop it once a v4.x with QWP ships). Needs +Go 1.23+. + +## Run + +```bash +# QWP (default): 2 HA senders + live probe, server-assigned timestamps +go run . -addrs localhost:9000 -total-events 100000 -num-senders 2 \ + -csv ../trades20250728.csv.gz + +# Enterprise TLS + auth, multi-host failover, durable ACKs, self-signed cert +go run . -protocol qwp \ + -addrs node-a:9000,node-b:9000 \ + -token "$QDB_TOKEN" -tls-verify unsafe_off \ + -enterprise -total-events 1000000 -num-senders 4 + +# ILP over HTTP (ingest only, no probe) for comparison +go run . -protocol ilp -addrs localhost:9000 -total-events 100000 -num-senders 2 +``` + +Key flags mirror the sibling ports: `-addrs`, `-token`/`-username`/`-password`, +`-total-events`, `-num-senders`, `-csv`, `-timestamp-from-file`, `-batch-size`, +`-batches-per-transaction`, `-sender-id`, `-store-forward-dir`, `-retry-timeout`, +`-probe-interval-ms`, `-enterprise`, `-zone`, `-tls-verify`, `-delay-ms`. + +The run prints **submitted** (client-side, published to the cursor engine) vs. +**acknowledged** (server-durable, confirmed via `FlushAndGetSequence` + +`AwaitAckedFsn`). That distinction is QWP-specific — see +`QWP_vs_ILP_in_Go.md` (kept at the TSBS repo root) for the full contrast, which +is written to hand to an agent adapting TSBS ingestion to QWP. diff --git a/go/go.mod b/go/go.mod new file mode 100644 index 0000000..1f8e13d --- /dev/null +++ b/go/go.mod @@ -0,0 +1,18 @@ +module qwphasender + +go 1.23 + +require github.com/questdb/go-questdb-client/v4 v4.2.0 + +require ( + github.com/coder/websocket v1.8.14 // indirect + github.com/klauspost/compress v1.18.4 // indirect + golang.org/x/sys v0.16.0 // indirect +) + +// The full QWP WebSocket transport (store-and-forward ingest, typed errors, +// failover, query client) landed on the client's `main` AFTER the v4.2.0 tag +// (commit ffec03a, PR #62), so it is not in a published release yet. Point at +// the local checkout until a v4.x with QWP ships; then drop this replace and +// bump the require above. +replace github.com/questdb/go-questdb-client/v4 => /Users/j/prj/go/go-questdb-client diff --git a/go/go.sum b/go/go.sum new file mode 100644 index 0000000..a05ef12 --- /dev/null +++ b/go/go.sum @@ -0,0 +1,94 @@ +dario.cat/mergo v1.0.0 h1:AGCNq9Evsj31mOgNPcLyXc+4PNABt905YmuqPYYpBWk= +dario.cat/mergo v1.0.0/go.mod h1:uNxQE+84aUszobStD9th8a29P2fMDhsBdgRYvZOxGmk= +github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 h1:L/gRVlceqvL25UVaW/CKtUDjefjrs0SPonmDGUVOYP0= +github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E= +github.com/Microsoft/go-winio v0.6.1 h1:9/kr64B9VUZrLm5YYwbGtUJnMgqWVOdUAXu6Migciow= +github.com/Microsoft/go-winio v0.6.1/go.mod h1:LRdKpFKfdobln8UmuiYcKPot9D2v6svN5+sAH+4kjUM= +github.com/Microsoft/hcsshim v0.11.4 h1:68vKo2VN8DE9AdN4tnkWnmdhqdbpUFM8OF3Airm7fz8= +github.com/Microsoft/hcsshim v0.11.4/go.mod h1:smjE4dvqPX9Zldna+t5FG3rnoHhaB7QYxPRqGcpAD9w= +github.com/cenkalti/backoff/v4 v4.2.1 h1:y4OZtCnogmCPw98Zjyt5a6+QwPLGkiQsYW5oUqylYbM= +github.com/cenkalti/backoff/v4 v4.2.1/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE= +github.com/coder/websocket v1.8.14 h1:9L0p0iKiNOibykf283eHkKUHHrpG7f65OE3BhhO7v9g= +github.com/coder/websocket v1.8.14/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg= +github.com/containerd/containerd v1.7.12 h1:+KQsnv4VnzyxWcfO9mlxxELaoztsDEjOuCMPAuPqgU0= +github.com/containerd/containerd v1.7.12/go.mod h1:/5OMpE1p0ylxtEUGY8kuCYkDRzJm9NO1TFMWjUpdevk= +github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I= +github.com/containerd/log v0.1.0/go.mod h1:VRRf09a7mHDIRezVKTRCrOq78v577GXq3bSa3EhrzVo= +github.com/cpuguy83/dockercfg v0.3.1 h1:/FpZ+JaygUR/lZP2NlFI2DVfrOEMAIKP5wWEJdoYe9E= +github.com/cpuguy83/dockercfg v0.3.1/go.mod h1:sugsbF4//dDlL/i+S+rtpIWp+5h0BHJHfjj5/jFyUJc= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/docker/distribution v2.8.2+incompatible h1:T3de5rq0dB1j30rp0sA2rER+m322EBzniBPB6ZIzuh8= +github.com/docker/distribution v2.8.2+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w= +github.com/docker/docker v24.0.9+incompatible h1:HPGzNmwfLZWdxHqK9/II92pyi1EpYKsAqcl4G0Of9v0= +github.com/docker/docker v24.0.9+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= +github.com/docker/go-connections v0.5.0 h1:USnMq7hx7gwdVZq1L49hLXaFtUdTADjXGp+uj1Br63c= +github.com/docker/go-connections v0.5.0/go.mod h1:ov60Kzw0kKElRwhNs9UlUHAE/F9Fe6GLaXnqyDdmEXc= +github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4= +github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= +github.com/go-ole/go-ole v1.3.0 h1:Dt6ye7+vXGIKZ7Xtk4s6/xVdGDQynvom7xCFEdWr6uE= +github.com/go-ole/go-ole v1.3.0/go.mod h1:5LS6F96DhAwUc7C+1HLexzMXY1xGRSryjyPPKW6zv78= +github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= +github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= +github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= +github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/klauspost/compress v1.18.4 h1:RPhnKRAQ4Fh8zU2FY/6ZFDwTVTxgJ/EMydqSTzE9a2c= +github.com/klauspost/compress v1.18.4/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4= +github.com/lufia/plan9stats v0.0.0-20230326075908-cb1d2100619a h1:N9zuLhTvBSRt0gWSiJswwQ2HqDmtX/ZCDJURnKUt1Ik= +github.com/lufia/plan9stats v0.0.0-20230326075908-cb1d2100619a/go.mod h1:JKx41uQRwqlTZabZc+kILPrO/3jlKnQ2Z8b7YiVw5cE= +github.com/magiconair/properties v1.8.7 h1:IeQXZAiQcpL9mgcAe1Nu6cX9LLw6ExEHKjN0VQdvPDY= +github.com/magiconair/properties v1.8.7/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0= +github.com/moby/patternmatcher v0.6.0 h1:GmP9lR19aU5GqSSFko+5pRqHi+Ohk1O69aFiKkVGiPk= +github.com/moby/patternmatcher v0.6.0/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc= +github.com/moby/sys/sequential v0.5.0 h1:OPvI35Lzn9K04PBbCLW0g4LcFAJgHsvXsRyewg5lXtc= +github.com/moby/sys/sequential v0.5.0/go.mod h1:tH2cOOs5V9MlPiXcQzRC+eEyab644PWKGRYaaV5ZZlo= +github.com/moby/term v0.5.0 h1:xt8Q1nalod/v7BqbG21f8mQPqH+xAaC9C3N3wfWbVP0= +github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y= +github.com/morikuni/aec v1.0.0 h1:nP9CBfwrvYnBRgY6qfDQkygYDmYwOilePFkwzv4dU8A= +github.com/morikuni/aec v1.0.0/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc= +github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= +github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= +github.com/opencontainers/image-spec v1.1.0-rc5 h1:Ygwkfw9bpDvs+c9E34SdgGOj41dX/cbdlwvlWt0pnFI= +github.com/opencontainers/image-spec v1.1.0-rc5/go.mod h1:X4pATf0uXsnn3g5aiGIsVnJBR4mxhKzfwmvK/B2NTm8= +github.com/opencontainers/runc v1.1.5 h1:L44KXEpKmfWDcS02aeGm8QNTFXTo2D+8MYGDIJ/GDEs= +github.com/opencontainers/runc v1.1.5/go.mod h1:1J5XiS+vdZ3wCyZybsuxXZWGrgSr8fFJHLXuG2PsnNg= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/power-devops/perfstat v0.0.0-20221212215047-62379fc7944b h1:0LFwY6Q3gMACTjAbMZBjXAqTOzOwFaj2Ld6cjeQ7Rig= +github.com/power-devops/perfstat v0.0.0-20221212215047-62379fc7944b/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE= +github.com/shirou/gopsutil/v3 v3.23.12 h1:z90NtUkp3bMtmICZKpC4+WaknU1eXtp5vtbQ11DgpE4= +github.com/shirou/gopsutil/v3 v3.23.12/go.mod h1:1FrWgea594Jp7qmjHUUPlJDTPgcsb9mGnXDxavtikzM= +github.com/shoenig/go-m1cpu v0.1.6 h1:nxdKQNcEB6vzgA2E2bvzKIYRuNj7XNJ4S/aRSwKzFtM= +github.com/shoenig/go-m1cpu v0.1.6/go.mod h1:1JJMcUBvfNwpq05QDQVAnx3gUHr9IYF7GNg9SUEw2VQ= +github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ= +github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= +github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg= +github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= +github.com/testcontainers/testcontainers-go v0.26.0 h1:uqcYdoOHBy1ca7gKODfBd9uTHVK3a7UL848z09MVZ0c= +github.com/testcontainers/testcontainers-go v0.26.0/go.mod h1:ICriE9bLX5CLxL9OFQ2N+2N+f+803LNJ1utJb1+Inx0= +github.com/tklauser/go-sysconf v0.3.12 h1:0QaGUFOdQaIVdPgfITYzaTegZvdCjmYO52cSFAEVmqU= +github.com/tklauser/go-sysconf v0.3.12/go.mod h1:Ho14jnntGE1fpdOqQEEaiKRpvIavV0hSfmBq8nJbHYI= +github.com/tklauser/numcpus v0.6.1 h1:ng9scYS7az0Bk4OZLvrNXNSAO2Pxr1XXRAPyjhIx+Fk= +github.com/tklauser/numcpus v0.6.1/go.mod h1:1XfjsgE2zo8GVw7POkMbHENHzVg3GzmoZ9fESEdAacY= +github.com/yusufpapurcu/wmi v1.2.3 h1:E1ctvB7uKFMOJw3fdOW32DwGE9I7t++CRUEMKvFoFiw= +github.com/yusufpapurcu/wmi v1.2.3/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0= +golang.org/x/exp v0.0.0-20231005195138-3e424a577f31 h1:9k5exFQKQglLo+RoP+4zMjOFE14P6+vyR0baDAi0Rcs= +golang.org/x/exp v0.0.0-20231005195138-3e424a577f31/go.mod h1:S2oDrQGGwySpoQPVqRShND87VCbxmc6bL1Yd2oYrm6k= +golang.org/x/mod v0.13.0 h1:I/DsJXRlw/8l/0c24sM9yb0T4z9liZTduXvdAWYiysY= +golang.org/x/mod v0.13.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= +golang.org/x/sys v0.16.0 h1:xWw16ngr6ZMtmxDyKyIgsE93KNKz5HKmMa3b8ALHidU= +golang.org/x/sys v0.16.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/tools v0.14.0 h1:jvNa2pY0M4r62jkRQ6RwEZZyPcymeL9XZMLBbV7U2nc= +golang.org/x/tools v0.14.0/go.mod h1:uYBEerGOWcJyEORxN+Ek8+TT266gXkNlHdJBwexUsBg= +google.golang.org/genproto/googleapis/rpc v0.0.0-20231002182017-d307bd883b97 h1:6GQBEOdGkX6MMTLT9V+TjtIRZCw9VPD5Z+yHY9wMgS0= +google.golang.org/genproto/googleapis/rpc v0.0.0-20231002182017-d307bd883b97/go.mod h1:v7nGkzlmW8P3n/bKmWBn2WpBjpOEx8Q6gMueudAmKfY= +google.golang.org/grpc v1.58.3 h1:BjnpXut1btbtgN/6sp+brB2Kbm2LjNXnidYujAVbSoQ= +google.golang.org/grpc v1.58.3/go.mod h1:tgX3ZQDlNJGU96V6yHh1T/JeoBQ2TXdr43YbYSsCJk0= +google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8= +google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/go/main.go b/go/main.go new file mode 100644 index 0000000..6a9b101 --- /dev/null +++ b/go/main.go @@ -0,0 +1,484 @@ +// Parallel CSV replay sender for QuestDB - a Go port of the Java/Rust/C++ +// CsvParallelSender, on the go-questdb-client v4 QWP (WebSocket) transport. +// +// It replays a CSV of trades in a loop across N worker goroutines and, at the +// same time, runs a background probe that queries the latest ingested timestamp +// and the serving node's role - ingest and query concurrently, with HA: +// +// - QWP over WebSocket (--protocol qwp, default): each worker owns its own +// store-and-forward sender (un-acked frames spill to disk under sf_dir and +// replay after an outage), with multi-host failover across --addrs. A probe +// goroutine uses a QwpQueryClient to poll `select ... from trades limit -1` +// and the handshake role (QwpServerInfo). +// - ILP over HTTP (--protocol ilp): the legacy line protocol, ingest only +// (QuestDB has no query client for ILP - the probe is QWP-only). +// +// Unlike ILP, a QWP Flush() never waits for the server ACK - it publishes into +// the sender's cursor engine and an I/O goroutine delivers/replays in the +// background. We report client-side "submitted" live and confirm server-side +// "acknowledged" at the end via FlushAndGetSequence + AwaitAckedFsn. See +// QWP_vs_ILP_in_Go.md for the full contrast. +package main + +import ( + "bufio" + "compress/gzip" + "context" + "encoding/csv" + "flag" + "fmt" + "io" + "os" + "os/signal" + "path/filepath" + "strconv" + "strings" + "sync" + "sync/atomic" + "syscall" + "time" + + qdb "github.com/questdb/go-questdb-client/v4" +) + +const ( + probeQuery = "select cast(timestamp as varchar) ts from trades limit -1" + destTable = "trades" +) + +type config struct { + addrs string + token string + username string + password string + totalEvents int64 + numSenders int + delayMs int64 + csvPath string + tsFromFile bool + protocol string // qwp | ilp + senderID string + sfDir string + batchSize int64 + batchesPerTx int64 + retryTimeout int64 + probeMs int64 + enterprise bool + zone string + tlsVerify string // on | unsafe_off +} + +type trade struct { + symbol, side string + price, amount float64 + tsNanos int64 // used only with --timestamp-from-file +} + +func parseFlags() *config { + c := &config{} + flag.StringVar(&c.addrs, "addrs", "localhost:9000", "comma-separated host:port; QWP/ILP use :9000") + flag.StringVar(&c.token, "token", "", "bearer token (turns on TLS)") + flag.StringVar(&c.username, "username", "", "basic-auth username (turns on TLS)") + flag.StringVar(&c.password, "password", "", "basic-auth password") + flag.Int64Var(&c.totalEvents, "total-events", 1_000_000, "total rows across all workers") + flag.IntVar(&c.numSenders, "num-senders", 4, "worker goroutines (each its own sender)") + flag.Int64Var(&c.delayMs, "delay-ms", 0, "per-row sleep in ms (0 = flat out)") + flag.StringVar(&c.csvPath, "csv", "../trades20250728.csv.gz", "CSV(.gz) with symbol,side,price,amount[,timestamp]") + flag.BoolVar(&c.tsFromFile, "timestamp-from-file", false, "use the CSV timestamp column instead of server 'now'") + flag.StringVar(&c.protocol, "protocol", "qwp", "transport: qwp | ilp") + flag.StringVar(&c.senderID, "sender-id", "ha_sender", "store-and-forward sender id base (per worker: -)") + flag.StringVar(&c.sfDir, "store-forward-dir", "/tmp/qdb-sf-go", "store-and-forward spill dir (QWP); empty \"\" = memory mode (no disk durability, raw happy-path ingest)") + flag.Int64Var(&c.batchSize, "batch-size", 10_000, "rows per flush unit") + flag.Int64Var(&c.batchesPerTx, "batches-per-transaction", 10, "flush every batch-size*this rows (QWP)") + flag.Int64Var(&c.retryTimeout, "retry-timeout", 360_000, "reconnect_max_duration_millis (QWP) / retry budget") + flag.Int64Var(&c.probeMs, "probe-interval-ms", 1000, "probe poll interval in ms (QWP; 0 disables)") + flag.BoolVar(&c.enterprise, "enterprise", false, "request durable server ACKs (sf_durability=disk)") + flag.StringVar(&c.zone, "zone", "", "query-side zone locality hint (QWP)") + flag.StringVar(&c.tlsVerify, "tls-verify", "on", "on | unsafe_off (for self-signed certs)") + flag.Parse() + return c +} + +func (c *config) tls() bool { return c.token != "" || (c.username != "" && c.password != "") } + +func (c *config) addrList() []string { + var out []string + for _, a := range strings.Split(c.addrs, ",") { + if a = strings.TrimSpace(a); a != "" { + out = append(out, a) + } + } + return out +} + +func (c *config) appendAuth(sb *strings.Builder) { + if c.token != "" { + fmt.Fprintf(sb, "token=%s;", c.token) + } else if c.username != "" && c.password != "" { + fmt.Fprintf(sb, "username=%s;password=%s;", c.username, c.password) + } + if c.tls() && c.tlsVerify == "unsafe_off" { + sb.WriteString("tls_verify=unsafe_off;") + } +} + +// ingestConf builds the per-worker connect string for the configured transport. +func (c *config) ingestConf(worker int) string { + addrs := strings.Join(c.addrList(), ",") + var sb strings.Builder + if c.protocol == "ilp" { + scheme := "http" + if c.tls() { + scheme = "https" + } + fmt.Fprintf(&sb, "%s::addr=%s;auto_flush=off;", scheme, addrs) + c.appendAuth(&sb) + fmt.Fprintf(&sb, "retry_timeout=%d;", c.retryTimeout) + return sb.String() + } + // qwp (WebSocket): failover + reconnect budget, with OPTIONAL store-and-forward. + scheme := "ws" + if c.tls() { + scheme = "wss" + } + fmt.Fprintf(&sb, "%s::addr=%s;auto_flush=off;", scheme, addrs) + c.appendAuth(&sb) + if c.sfDir != "" { + // Durable store-and-forward: each worker spills un-acked frames to its + // own on-disk slot so they replay after an outage/restart. + who := fmt.Sprintf("%s-%d", c.senderID, worker) + fmt.Fprintf(&sb, "sf_dir=%s;sender_id=%s;", filepath.Join(c.sfDir, who), who) + if c.enterprise { + sb.WriteString("sf_durability=disk;") // fsync frames to disk before ACK + } + } + // else: memory mode - RAM ring, no disk persistence. Rows still land and are + // ACKed (server WAL-apply), but a process exit before the background send + // drains can lose unacked rows. This is the raw happy-path ingest. + fmt.Fprintf(&sb, "reconnect_max_duration_millis=%d;", c.retryTimeout) + sb.WriteString("reconnect_initial_backoff_millis=100;reconnect_max_backoff_millis=5000;") + return sb.String() +} + +// queryConf builds the QWP query-client connect string for the probe. +func (c *config) queryConf() string { + var sb strings.Builder + fmt.Fprintf(&sb, "ws%s::addr=%s;target=any;", tlsSuffix(c.tls()), strings.Join(c.addrList(), ",")) + c.appendAuth(&sb) + if c.zone != "" { + fmt.Fprintf(&sb, "zone=%s;", c.zone) + } + return sb.String() +} + +func tlsSuffix(tls bool) string { + if tls { + return "s" + } + return "" +} + +func loadCSV(path string, needTS bool) ([]trade, error) { + f, err := os.Open(path) + if err != nil { + return nil, err + } + defer f.Close() + var r io.Reader = bufio.NewReader(f) + if strings.HasSuffix(path, ".gz") { + gz, err := gzip.NewReader(f) + if err != nil { + return nil, err + } + defer gz.Close() + r = gz + } + cr := csv.NewReader(r) + cr.ReuseRecord = true + header, err := cr.Read() + if err != nil { + return nil, err + } + idx := map[string]int{} + for i, name := range header { + idx[strings.TrimSpace(name)] = i + } + for _, col := range []string{"symbol", "side", "price", "amount"} { + if _, ok := idx[col]; !ok { + return nil, fmt.Errorf("CSV missing column %q", col) + } + } + if needTS { + if _, ok := idx["timestamp"]; !ok { + return nil, fmt.Errorf("CSV missing column %q", "timestamp") + } + } + var rows []trade + for { + rec, err := cr.Read() + if err == io.EOF { + break + } + if err != nil { + return nil, err + } + price, _ := strconv.ParseFloat(strings.TrimSpace(rec[idx["price"]]), 64) + amount, _ := strconv.ParseFloat(strings.TrimSpace(rec[idx["amount"]]), 64) + t := trade{ + symbol: strings.TrimSpace(rec[idx["symbol"]]), + side: strings.TrimSpace(rec[idx["side"]]), + price: price, + amount: amount, + } + if needTS { + ts, err := time.Parse(time.RFC3339Nano, strings.TrimSpace(rec[idx["timestamp"]])) + if err != nil { + return nil, fmt.Errorf("bad timestamp %q: %w", rec[idx["timestamp"]], err) + } + t.tsNanos = ts.UnixNano() + } + rows = append(rows, t) + } + return rows, nil +} + +// runWorker replays `events` rows over its own sender, then confirms the +// server ACK'd them (QWP) before returning. +func runWorker(ctx context.Context, c *config, worker int, events int64, rows []trade, + sent *int64, ackedRows *int64) error { + + sender, err := qdb.LineSenderFromConf(ctx, c.ingestConf(worker)) + if err != nil { + return fmt.Errorf("worker %d connect: %w", worker, err) + } + defer sender.Close(ctx) + + qwp, isQwp := sender.(qdb.QwpSender) + commitEvery := c.batchSize + if c.protocol == "qwp" { + commitEvery = c.batchSize * c.batchesPerTx + } + n := int64(len(rows)) + var i int64 + for i = 0; i < events; i++ { + t := rows[i%n] + b := sender. + Table(destTable). + Symbol("symbol", t.symbol). + Symbol("side", t.side). + Float64Column("price", t.price). + Float64Column("amount", t.amount). + StringColumn("trade_id", fmt.Sprintf("%d-%d", worker, i+1)) + if c.tsFromFile { + err = b.At(ctx, time.Unix(0, t.tsNanos)) + } else { + // Server-assigned timestamp: no client clock skew, no O3 across workers. + err = b.AtNow(ctx) + } + if err != nil { + return fmt.Errorf("worker %d row: %w", worker, err) + } + if s := atomic.AddInt64(sent, 1); s%commitEvery == 0 { + if err := sender.Flush(ctx); err != nil { + return fmt.Errorf("worker %d flush: %w", worker, err) + } + } + if c.delayMs > 0 { + time.Sleep(time.Duration(c.delayMs) * time.Millisecond) + } + if ctx.Err() != nil { + break + } + } + + // Publish the tail. On QWP, capture the published FSN and wait for the + // server to ACK it - this is the ingest->durable boundary that ILP's + // synchronous flush gives you for free but QWP's async flush does not. + if isQwp { + fsn, err := qwp.FlushAndGetSequence(ctx) + if err != nil { + return fmt.Errorf("worker %d final flush: %w", worker, err) + } + if err := qwp.AwaitAckedFsn(ctx, fsn); err != nil { + return fmt.Errorf("worker %d await ack: %w", worker, err) + } + atomic.AddInt64(ackedRows, i) + } else { + if err := sender.Flush(ctx); err != nil { + return fmt.Errorf("worker %d final flush: %w", worker, err) + } + atomic.AddInt64(ackedRows, i) + } + return nil +} + +// runProbe polls the latest ingested timestamp and the serving node's role, +// concurrently with ingestion, over a QWP query client (with failover). +func runProbe(ctx context.Context, c *config) { + client, err := qdb.QwpQueryClientFromConf(ctx, c.queryConf()) + if err != nil { + fmt.Printf("[probe] connect failed: %v\n", err) + return + } + defer client.Close(ctx) + fmt.Printf("[probe] connected to %s\n", client.CurrentEndpoint()) + + ticker := time.NewTicker(time.Duration(c.probeMs) * time.Millisecond) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + latest := queryLatest(ctx, client) + role, endpoint := "unknown", client.CurrentEndpoint() + if si := client.ServerInfo(); si != nil { + role = si.RoleName() + } + if latest == "" { + fmt.Printf("[probe] no rows yet | node=%s role=%s\n", endpoint, role) + } else { + fmt.Printf("[probe] latest %s ts=%s | node=%s role=%s\n", destTable, latest, endpoint, role) + } + } + } +} + +func queryLatest(ctx context.Context, client *qdb.QwpQueryClient) string { + q := client.Query(ctx, probeQuery) + defer q.Close() + var latest string + for batch, err := range q.Batches() { + if err != nil { + return "" // transient (failover in progress etc.); next tick retries + } + if batch.RowCount() > 0 { + col := batch.Column(0) + if !col.IsNull(col.RowCount() - 1) { + latest = col.String(col.RowCount() - 1) + } + } + } + return latest +} + +func main() { + c := parseFlags() + if c.protocol != "qwp" && c.protocol != "ilp" { + fmt.Fprintf(os.Stderr, "--protocol must be qwp or ilp, got %q\n", c.protocol) + os.Exit(2) + } + if c.numSenders < 1 || c.totalEvents < 1 { + fmt.Fprintln(os.Stderr, "--num-senders and --total-events must be >= 1") + os.Exit(2) + } + + rows, err := loadCSV(c.csvPath, c.tsFromFile) + if err != nil { + fmt.Fprintf(os.Stderr, "load CSV: %v\n", err) + os.Exit(1) + } + if len(rows) == 0 { + fmt.Fprintln(os.Stderr, "CSV has no rows") + os.Exit(1) + } + + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + + scheme := "ws" + if c.tls() { + scheme = "wss" + } + if c.protocol == "ilp" { + scheme = "http" + if c.tls() { + scheme = "https" + } + } + fmt.Printf("[conf] protocol=%s scheme=%s tls=%v auth=%s | addrs=%s | table=%s rows=%d workers=%d\n", + c.protocol, scheme, c.tls(), authKind(c), c.addrs, destTable, len(rows), c.numSenders) + + var sent, ackedRows int64 + start := time.Now() + + // Progress reporter: submitted (client-side) vs acknowledged (server-durable). + var wgReport sync.WaitGroup + wgReport.Add(1) + go func() { + defer wgReport.Done() + ticker := time.NewTicker(time.Second) + defer ticker.Stop() + var last int64 + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + s := atomic.LoadInt64(&sent) + fmt.Printf("[progress] submitted=%d (+%d/s) acknowledged=%d\n", + s, s-last, atomic.LoadInt64(&ackedRows)) + last = s + } + } + }() + + // Probe (QWP only): query concurrently with ingest. + probeCtx, probeStop := context.WithCancel(ctx) + if c.protocol == "qwp" && c.probeMs > 0 { + go runProbe(probeCtx, c) + } + + // Workers: each its own sender, replaying its share of the CSV. + base := c.totalEvents / int64(c.numSenders) + rem := c.totalEvents % int64(c.numSenders) + var wg sync.WaitGroup + errs := make([]error, c.numSenders) + for w := 0; w < c.numSenders; w++ { + events := base + if int64(w) < rem { + events++ + } + wg.Add(1) + go func(worker int, events int64) { + defer wg.Done() + errs[worker] = runWorker(ctx, c, worker, events, rows, &sent, &ackedRows) + }(w, events) + } + wg.Wait() + + probeStop() // stop the probe once ingestion is done + stop() // release the signal context so the reporter exits + wgReport.Wait() + + failed := false + for w, e := range errs { + if e != nil { + fmt.Fprintf(os.Stderr, "worker %d failed: %v\n", w, e) + failed = true + } + } + + elapsed := time.Since(start) + total := atomic.LoadInt64(&sent) + acked := atomic.LoadInt64(&ackedRows) + fmt.Printf("[done] submitted=%d acknowledged=%d in %.3fs (%.0f rows/s submit)\n", + total, acked, elapsed.Seconds(), float64(total)/elapsed.Seconds()) + if c.protocol == "qwp" { + fmt.Printf("[done] 'acknowledged' = rows the server durably confirmed (FlushAndGetSequence + AwaitAckedFsn).\n") + } + if failed { + os.Exit(1) + } +} + +func authKind(c *config) string { + switch { + case c.token != "": + return "token" + case c.username != "": + return "basic" + default: + return "none" + } +} From 32cda877d7dbad137ba2bb44fe03f4d8628581f1 Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 22 Jul 2026 14:06:26 +0200 Subject: [PATCH 32/79] python/README: document QUESTDB_INSECURE_SKIP_VERIFY build flag Building the 5.0 client with tls_verify=unsafe_off (self-signed enterprise certs over wss) requires the insecure-skip-verify Cargo feature, which the default build omits. Document the QUESTDB_INSECURE_SKIP_VERIFY=1 opt-in and warn against --no-build-isolation (numpy-header mismatch). Temporary until a client release ships the QWP/egress build on PyPI. --- python/README.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/python/README.md b/python/README.md index 4041466..64f3610 100644 --- a/python/README.md +++ b/python/README.md @@ -83,6 +83,22 @@ python3.12 -m venv /tmp/pyqdb_venv /tmp/pyqdb_venv/bin/pip install -e . # compiles Cython + the Rust FFI ``` +**Self-signed certs / `tls_verify=unsafe_off`.** 5.0 keeps the `insecure-skip-verify` +Cargo feature **out of the default build** (disabling cert verification is a footgun that +must never ship in a wheel). If you connect over `wss` to a server with a self-signed cert +and use `--tls-verify unsafe_off`, you will otherwise hit +`The "insecure-skip-verify" feature is not enabled`. Opt in at build time with an env var: + +```bash +QUESTDB_INSECURE_SKIP_VERIFY=1 pip install -e . --force-reinstall --no-deps +``` + +Keep build isolation on (do **not** add `--no-build-isolation`): the build cythonizes +`_client.pyx` against the pinned build-time numpy, and forcing it to use a venv whose numpy +differs fails with `_PyUFuncObject_GET_ITEM_DATA` / `PyDataType_TYPEOBJ` compile errors. The +env var still reaches the isolated build. (The alternative to `unsafe_off` is to trust the +cert properly via `tls_roots`/`tls_ca`.) + Verify: ```python From 63b92b6ce2529a023cd35139456bbb900c14329c Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 23 Jul 2026 16:34:21 +0200 Subject: [PATCH 33/79] blotter: silence polars 1.43 UInt32->Categorical deprecation flood to_polars() builds SYMBOL Categoricals with a UInt32 -> Categorical cast that polars 1.43 deprecates. The warning message embeds a fresh Categories UUID on every call, so Python's once-per-location dedup never matches and the warnings re-fire at the poll rate, shredding the in-place redraw (and flooding the web_blotter log). Filter that one message; keep to_polars(), which is both the idiomatic call and the faster path on large results. --- python/blotter.py | 12 ++++++++++++ python/web_blotter.py | 10 ++++++++++ 2 files changed, 22 insertions(+) diff --git a/python/blotter.py b/python/blotter.py index 408311c..55d2d09 100644 --- a/python/blotter.py +++ b/python/blotter.py @@ -35,12 +35,24 @@ import os import sys import time +import warnings from datetime import datetime, timezone import polars as pl import questdb +# to_polars() builds SYMBOL Categoricals with a UInt32 -> Categorical cast (egress.pxi:1890) +# that polars 1.43 deprecates in favour of .cat.to(). Harmless, but the message embeds a +# fresh Categories UUID on every call, so Python's once-per-location dedup never matches and +# the warnings re-fire at the poll rate, shredding the in-place redraw. Drop this filter once +# the client stops using the deprecated cast. +warnings.filterwarnings( + "ignore", + message=r"casting from UInt32 to Categorical", + category=DeprecationWarning, +) + def use_tls(args): return bool(args.tls or args.token or (args.username and args.password)) diff --git a/python/web_blotter.py b/python/web_blotter.py index 73af393..ada17c7 100644 --- a/python/web_blotter.py +++ b/python/web_blotter.py @@ -24,11 +24,21 @@ import os import sys import threading +import warnings from datetime import datetime from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import questdb +# See blotter.py: to_polars() uses a UInt32 -> Categorical cast that polars 1.43 deprecates, +# and the message carries a fresh UUID per call, so it defeats warning dedup and floods the +# server log at the poll rate. +warnings.filterwarnings( + "ignore", + message=r"casting from UInt32 to Categorical", + category=DeprecationWarning, +) + def use_tls(args): return bool(args.tls or args.token or (args.username and args.password)) From 45c0d6509af985e377b56d05be54dcd582efa710 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 23 Jul 2026 19:08:58 +0200 Subject: [PATCH 34/79] CsvParallelSender: recover the QWP probe after a server outage Ingestion restored itself after an outage but querying never did. Two causes, both in startProbe(): 1. QwpQueryClient.connect() ran once, outside the poll loop. A server that was down at startup threw straight past the loop into the outer catch, so the probe thread died and nothing restarted it. 2. A QwpQueryClient cannot recover on its own once a connection dies. With failover on (multiple --addrs), executeImpl leaves connected=false after the failover budget is spent and every later execute() throws "not connected; call connect() first". With failover off it is quieter: connected stays true, isConnected() keeps returning true, and each execute() short-circuits on the latched terminal failure and reports it through onError forever. Calling connect() again does not help -- it early-returns while connected is true. The client is now built inside the poll loop and torn down on any loss, so the next tick constructs a fresh one. Transport failures are distinguished from SQL errors by STATUS_INTERNAL_ERROR (0x06), which every socket-death terminal failure carries, so `switch status` failing on OSS is not mistaken for a disconnect. Verified against a 3-address failover=on config with a TCP proxy standing in for the server: connected -> connection lost -> connection restored, with the probe resuming afterwards. --- .../com/example/sender/CsvParallelSender.java | 78 +++++++++++++++---- 1 file changed, 64 insertions(+), 14 deletions(-) diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index 14375bc..8dbc9bb 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -70,6 +70,11 @@ public class CsvParallelSender { // SERVER_INFO which is only refreshed on a reconnect and so goes stale after an in-place // primary<->replica switch. The probe runs this each poll to report the true serving role. private static final String STATUS_QUERY = "switch status"; + // WebSocketResponse.STATUS_INTERNAL_ERROR. Every transport-level terminal failure is + // reported with this status (QwpEgressIoThread raises it on socket death), so it is what + // separates "the connection died" from a SQL-level error on the same onError callback. + // Inlined rather than imported: WebSocketResponse is a wire-protocol detail of the client. + private static final byte QWP_STATUS_INTERNAL_ERROR = 0x06; // Zone for the query client (egress). Biases failover toward same-zone instances on // Enterprise; a no-op on OSS (which advertises no zone). Empty omits the key. private static final String DEFAULT_ZONE = "eu-west-1"; @@ -719,20 +724,30 @@ private static String queryClientConfig(SenderCfg cfg) { // Starts a daemon thread that polls the latest ingested timestamp over a QWP query // client every intervalMs, printing to stdout. Independent of the senders. The client // fails over across the configured hosts automatically; onFailoverReset reports hops. + // + // The client is built INSIDE the poll loop and torn down on any connection loss, because + // a QwpQueryClient cannot recover on its own: + // * connect() outside the loop means a server that is down at startup kills the probe + // thread outright, and nothing ever restarts it; + // * once a connection dies, the generation listener latches the terminal failure. With + // failover on, execute() eventually leaves connected=false and every later execute() + // throws "not connected; call connect() first". With failover off (a single --addrs + // host) it is sneakier: connected stays TRUE, isConnected() keeps returning true, and + // every execute() short-circuits on the latched failure and reports it through + // onError() forever. + // Neither state clears by calling connect() again (it early-returns when connected), so + // recovery means close() + fromConfig() on a fresh instance. private static Thread startProbe(SenderCfg cfg, long intervalMs) { final Thread t = new Thread(() -> { - try (QwpQueryClient client = QwpQueryClient.fromConfig(queryClientConfig(cfg))) { - client.connect(); - final QwpServerInfo info = client.getServerInfo(); - if (info != null) { - System.out.printf("[query client] connected, serving node=%s role=%s zone=%s cluster=%s%n", - orNone(info.getNodeId()), QwpServerInfo.roleName(info.getRole()), - orNone(info.getZoneId()), orNone(info.getClusterId())); - } else { - System.out.println("[query client] connected"); - } + QwpQueryClient client = null; + try { final long[] latest = {Long.MIN_VALUE}; final boolean[] wasDown = {false}; + // Set by the probe handler when the server reports a transport-level failure. + // Transport failures always carry STATUS_INTERNAL_ERROR; a SQL-level error + // (e.g. "table does not exist" before ingestion creates it) carries a different + // status and must NOT tear the connection down. + final boolean[] transportFailed = {false}; final QwpColumnBatchHandler handler = new QwpColumnBatchHandler() { @Override public void onBatch(QwpColumnBatch batch) { @@ -749,7 +764,11 @@ public void onEnd(long totalRows) { @Override public void onError(byte status, String message) { - System.out.printf("[query client] server error: %s%n", message); + if (status == QWP_STATUS_INTERNAL_ERROR) { + transportFailed[0] = true; + } else { + System.out.printf("[query client] server error: %s%n", message); + } } @Override @@ -823,12 +842,30 @@ public void onFailoverReset(QwpServerInfo info) { currentRole[0] = null; targetRole[0] = null; statusDiag[0] = null; + transportFailed[0] = false; try { - client.execute(PROBE_QUERY, handler); - if (wasDown[0]) { - System.out.println("[query client] connection restored"); + // (Re)build on first pass and after every connection loss. A server that + // is down at startup just leaves client == null and retries next tick. + if (client == null) { + client = QwpQueryClient.fromConfig(queryClientConfig(cfg)); + client.connect(); + final QwpServerInfo info = client.getServerInfo(); + final String what = wasDown[0] ? "connection restored" : "connected"; + if (info != null) { + System.out.printf("[query client] %s, serving node=%s role=%s zone=%s cluster=%s%n", + what, orNone(info.getNodeId()), QwpServerInfo.roleName(info.getRole()), + orNone(info.getZoneId()), orNone(info.getClusterId())); + } else { + System.out.printf("[query client] %s%n", what); + } wasDown[0] = false; } + client.execute(PROBE_QUERY, handler); + // A transport failure is reported through onError, not by throwing, and + // it leaves the client permanently latched. Force the rebuild path. + if (transportFailed[0] || !client.isConnected()) { + throw new IllegalStateException("query connection lost"); + } if (latest[0] != Long.MIN_VALUE) { // trades designated timestamp is microseconds by default. Instant ts = Instant.EPOCH.plus(latest[0], ChronoUnit.MICROS); @@ -880,6 +917,15 @@ public void onFailoverReset(QwpServerInfo info) { String.valueOf(e.getMessage())); wasDown[0] = true; } + // Drop the latched client; the next tick builds a fresh one. + if (client != null) { + try { + client.close(); + } catch (Exception ignored) { + // A dead connection often fails to close cleanly; nothing to do. + } + client = null; + } } Thread.sleep(intervalMs); } @@ -887,6 +933,10 @@ public void onFailoverReset(QwpServerInfo info) { Thread.currentThread().interrupt(); } catch (Exception e) { System.err.println("[probe] stopped: " + e.getMessage() + upgradeHint(e)); + } finally { + if (client != null) { + client.close(); + } } }, "qwp-probe"); t.setDaemon(true); From c37fd31762240ac4c90cbca6872dd2cae04c2ff2 Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 24 Jul 2026 15:28:10 +0200 Subject: [PATCH 35/79] blotter-demo: one-command Docker live-blotter demo A self-contained Docker Compose demo so a colleague can see the terminal blotter in action with nothing but Docker installed: - questdb.Dockerfile: QuestDB built from source at the LIVE VIEW commit (90a1b54), Java 25, with the web console (-P build-web-console). Served on host port 19000 (not 9000) so it never collides with a local QuestDB. - demo.Dockerfile: the unreleased questdb 5.0 Python client built from source (ea54b6f, Rust 1.91.1 + Cython), plus the feed and blotter. - feed.py: continuous synthetic crypto/FX bid feed, ~2000 rows/s at 20 flushes/s so the live view visibly moves. - entrypoint.sh: waits for QuestDB, creates the base table and the moving-average LIVE VIEW, runs the feed in the background and the blotter in the foreground, querying the live view directly. - vendor.sh: fallback to snapshot the client source if its branch is deleted before it merges. - README.md: run, repeat-run, and tiered teardown instructions. .dockerignore keeps the build context small. --- .dockerignore | 14 +++ blotter-demo/README.md | 177 ++++++++++++++++++++++++++++++++ blotter-demo/demo.Dockerfile | 46 +++++++++ blotter-demo/docker-compose.yml | 39 +++++++ blotter-demo/entrypoint.sh | 56 ++++++++++ blotter-demo/feed.py | 83 +++++++++++++++ blotter-demo/questdb.Dockerfile | 58 +++++++++++ blotter-demo/vendor.sh | 40 ++++++++ 8 files changed, 513 insertions(+) create mode 100644 .dockerignore create mode 100644 blotter-demo/README.md create mode 100644 blotter-demo/demo.Dockerfile create mode 100644 blotter-demo/docker-compose.yml create mode 100755 blotter-demo/entrypoint.sh create mode 100644 blotter-demo/feed.py create mode 100644 blotter-demo/questdb.Dockerfile create mode 100755 blotter-demo/vendor.sh diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..e79599e --- /dev/null +++ b/.dockerignore @@ -0,0 +1,14 @@ +# Keep the Docker build context small. The demo images only need +# python/blotter.py, blotter-demo/feed.py and blotter-demo/entrypoint.sh; everything below is +# build output or large data that would otherwise be uploaded to the daemon. +.git/ +**/target/ +**/build/ +rust/target/ +c/build/ +go/ +*.csv.gz +**/__pycache__/ +**/*.pyc +**/venv/ +**/.venv/ diff --git a/blotter-demo/README.md b/blotter-demo/README.md new file mode 100644 index 0000000..5a2becc --- /dev/null +++ b/blotter-demo/README.md @@ -0,0 +1,177 @@ +# One-command live blotter demo + +A self-contained Docker demo: QuestDB with **LIVE VIEW**, the unreleased +**questdb 5.0 Python client**, a synthetic price feed, and the terminal blotter +rendering on top of a live view. Your colleague needs **nothing but Docker** — no +Python, no Rust, no client build. + +## Run it + +```bash +cd blotter-demo +docker compose build # one-time, slow: builds QuestDB + the client from source +docker compose run --rm demo # QuestDB starts, then the blotter draws in your terminal +``` + +`run` (not `up`) is deliberate — it gives the blotter a real TTY for the in-place +redraw. **Ctrl+C** quits the blotter; the feed stops with it. Afterwards: + +```bash +docker compose down # stop QuestDB +``` + +## Running it again + +The slow part is the one-time `docker compose build`. Once the images exist, every +`docker compose run --rm demo` skips the build entirely and just starts containers, +so repeat runs are fast — the only cost is a few seconds of JVM startup as QuestDB +cold-starts (the `--rm`/`down` flow disposes the container each time, so the table, +live view, and history start fresh on every run — intended for a demo). A rebuild +only happens if you edit a file a Docker layer depends on, and even then only the +cheap final layers re-run — never the Rust/Maven builds — unless you change the +pinned client/server commits. + +If a `docker compose down` reports "network still in use", a run container lingered +from a Ctrl+C; clear it with `docker compose down --remove-orphans`. + +## Removing everything at the end + +This demo creates four kinds of Docker artifact: two built images +(`blotter-demo-questdb`, `blotter-demo-demo`), a network and containers, build +cache (the Rust + Maven layers — the bulk of the disk), and base images it pulls +(`python:3.12-slim`, `eclipse-temurin:25-jdk`, `eclipse-temurin:25-jre`). + +**Removing an image is not destructive the way deleting a volume is** — any image +can be re-pulled from Docker Hub on demand, so the worst case of over-removing is a +re-download later. No data is lost. The demo defines no named volumes. + +### Step 1 — containers, network, and the two built images (scoped, safe) + +```bash +docker compose down --rmi local --remove-orphans +``` + +Touches only this project. `--rmi local` deletes the two images the demo built; +`--remove-orphans` clears any run container left by a Ctrl+C. + +### Step 2 — the base images this demo pulled + +Docker does not record when an image was pulled, so "was it already there before?" +can only be answered by snapshotting the image list **before** the first build. + +**Before your first `docker compose build`**, record what already exists: + +```bash +docker image ls --format '{{.Repository}}:{{.Tag}}' | sort > ~/blotter-demo-images-before.txt +``` + +**At cleanup**, remove each base image only if it was NOT in that snapshot (i.e. +only if this demo pulled it): + +```bash +for img in python:3.12-slim eclipse-temurin:25-jdk eclipse-temurin:25-jre; do + grep -qxF "$img" ~/blotter-demo-images-before.txt || docker rmi "$img" +done +``` + +If you did **not** take the snapshot, you cannot tell which were pre-existing. +Either leave them, or just remove all three — re-pulling is harmless (no data loss), +it only costs a download if another project needs them later: + +```bash +docker rmi python:3.12-slim eclipse-temurin:25-jdk eclipse-temurin:25-jre +``` + +### Step 3 — reclaim the build cache (the big disk win) + +```bash +docker builder prune -f +``` + +`docker builder prune` removes only BuildKit's cached build layers. It **never +deletes images, containers, or volumes** — those are safe. But it is **system-wide, +not scoped to this demo**: it clears the build cache of every project, so other +projects' next builds recompute their layers (slower, but nothing is lost — cache +only). There is no per-project build-cache prune. If this demo is your only heavy +Docker build, it is effectively scoped anyway. + +### Not recommended + +`docker system prune -af` reclaims everything Docker is not currently using **across +all projects**, not just this demo — only run it if you genuinely mean machine-wide. + +## What it does + +1. **`questdb`** — built from source at the commit that first ships `LIVE VIEW` + (`90a1b54c…`) with the web console (`-P build-web-console`). The server is + reachable from the host at **http://localhost:19000** (console + `/exec`); this + host port is deliberately 19000, not 9000, so it never collides with a local + QuestDB. The blotter reaches the server internally over the compose network. +2. **`demo`** — builds the 5.0 client (`ea54b6f…`, Rust 1.91.1 + Cython), then: + - waits for QuestDB, + - `curl` creates `core_price_demo` (base table) and `core_price_lv` (live view), + - runs `feed.py` in the background (~2000 rows/s of synthetic crypto/FX bids), + - runs `blotter.py` in the foreground, polling the live view at 10 Hz. + +The live view: + +```sql +CREATE LIVE VIEW IF NOT EXISTS core_price_lv +FLUSH EVERY 5s IN MEMORY 5s START FROM NOW AS +SELECT timestamp, symbol, bid_price, avg(bid_price) OVER w AS moving_avg +FROM core_price_demo +WINDOW w AS (PARTITION BY symbol ORDER BY timestamp ANCHOR DAILY '00:00'); +``` + +`START FROM NOW` is required by this build (the view tracks data from creation +time onward). A live view serves results from memory, so ingested rows appear +immediately; `FLUSH EVERY` is only the disk-commit cadence and does not gate what +the blotter sees. Visible motion comes from the **feed** flushing to the base +table often. + +The blotter queries the live view directly — `select * from core_price_lv limit -20` +— so it streams the newest 20 rows (timestamp, symbol, bid, moving average), +refreshed at `--rate` Hz. + +## Knobs + +- **Feed rate / flush frequency.** `FEED_FLUSH_HZ` (default 20, i.e. a flush every + 50 ms) and `FEED_ROWS_PER_FLUSH` (default 100) on the `demo` service set how + fast rows land and how often they are flushed. 20 flushes/s keeps the blotter + visibly moving. Prices are illustrative 2026 levels, not live market data. +- **Blotter refresh.** `--rate` Hz in `entrypoint.sh` (default 10) sets how often + the terminal redraws. +- **What the blotter shows.** Defaults to the last 20 rows of the live view + (`--table core_price_lv --limit -20`). Adjust the count with `--limit`, or pass + `--query ""` in `entrypoint.sh` to run any SQL against the view verbatim. + +## Pins and durability + +Both sources are pinned by commit SHA for reproducibility: + +| Component | Repo | Commit | +| --- | --- | --- | +| QuestDB server | `questdb/questdb` | `90a1b54c98b10fad5304b1ad817a69cda25e52ad` | +| Python client | `questdb/py-questdb-client` | `ea54b6f474062c144aa6395facad42e77c99e6f6` | + +A pinned SHA is fetchable from GitHub only while it stays reachable from a ref. +The questdb commit is expected to merge to `main`, so it survives. The **client +commit lives on `jh_experiment_new_ilp`, which may be deleted before it merges** — +if that happens the build's `git checkout` will fail. Run `./vendor.sh` to +snapshot the client source into `blotter-demo/vendor/` and follow its printed instructions +to switch the Dockerfile to the local tarball. + +## When the client ships / questdb merges + +- Once `LIVE VIEW` lands in a nightly, delete `questdb.Dockerfile` and set + `image: questdb/questdb:nightly` on the `questdb` service. +- Once the client is on PyPI, replace the whole `client-build` stage with + `pip install questdb==`. + +## Sharing without a rebuild + +To hand a colleague an image that needs no build at all, either push to a registry +(`docker compose build && docker compose push`, after adding `image:` names) or +`docker save`/`docker load` a tarball. Both are architecture-specific — an amd64 +image runs under emulation on Apple Silicon. Letting them `docker compose build` +locally avoids that by building for their arch. diff --git a/blotter-demo/demo.Dockerfile b/blotter-demo/demo.Dockerfile new file mode 100644 index 0000000..c10c1c4 --- /dev/null +++ b/blotter-demo/demo.Dockerfile @@ -0,0 +1,46 @@ +# Blotter demo app: the unreleased questdb 5.0 Python client (built from source), +# plus the feed + blotter scripts. Built once, then runs with zero host deps. +# +# The client is pinned by SHA. NOTE: unlike the questdb server commit, this branch +# (jh_experiment_new_ilp) may be DELETED before it merges, which would make the SHA +# unfetchable from GitHub. If that happens, run blotter-demo/vendor.sh to snapshot the +# source locally and switch the client-build stage to COPY it in (see vendor.sh). +ARG CLIENT_COMMIT=ea54b6f474062c144aa6395facad42e77c99e6f6 +ARG RUST_VERSION=1.91.1 + +FROM python:3.12-slim AS client-build +ARG CLIENT_COMMIT +ARG RUST_VERSION +RUN apt-get update && apt-get install -y --no-install-recommends \ + git curl build-essential pkg-config ca-certificates \ + && rm -rf /var/lib/apt/lists/* +# Rust >= 1.91.1: the bundled c-questdb-client (questdb-rs 7.0.0) requires it. +RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \ + | sh -s -- -y --default-toolchain "${RUST_VERSION}" --profile minimal +ENV PATH="/root/.cargo/bin:${PATH}" + +WORKDIR /src +RUN git clone https://github.com/questdb/py-questdb-client.git . +RUN git checkout "${CLIENT_COMMIT}" +RUN git submodule update --init --recursive +# Build-time deps must stay in the isolated build env (do NOT use +# --no-build-isolation; that pins the wrong numpy and fails the cython compile). +RUN pip install --no-cache-dir -U pip "cython>=3.1.2" "setuptools>=80.9.0" numpy +RUN pip wheel --no-deps --no-cache-dir -w /wheels . + +FROM python:3.12-slim +RUN apt-get update && apt-get install -y --no-install-recommends \ + curl ca-certificates \ + && rm -rf /var/lib/apt/lists/* +COPY --from=client-build /wheels /wheels +RUN pip install --no-cache-dir /wheels/*.whl polars pyarrow + +WORKDIR /app +COPY python/blotter.py /app/blotter.py +COPY blotter-demo/feed.py /app/feed.py +COPY blotter-demo/entrypoint.sh /app/entrypoint.sh +RUN chmod +x /app/entrypoint.sh + +ENV QDB_ADDR=questdb:9000 \ + QDB_HTTP=questdb:9000 +ENTRYPOINT ["/app/entrypoint.sh"] diff --git a/blotter-demo/docker-compose.yml b/blotter-demo/docker-compose.yml new file mode 100644 index 0000000..3a52a05 --- /dev/null +++ b/blotter-demo/docker-compose.yml @@ -0,0 +1,39 @@ +# One-command blotter demo. From this directory: +# +# docker compose build # one-time, slow (builds QuestDB + client from source) +# docker compose run --rm demo # starts QuestDB, then the blotter in your terminal +# +# `run` (not `up`) gives the blotter a real TTY for the in-place redraw. Ctrl+C quits; +# --rm cleans up the demo container. Stop QuestDB afterwards with `docker compose down`. + +services: + questdb: + build: + context: .. + dockerfile: blotter-demo/questdb.Dockerfile + ports: + # Host 19000 -> container 9000. Deliberately NOT 9000 on the host: that is + # the default QuestDB port, and a local QuestDB there would collide. The + # blotter itself never uses this mapping (it reaches the server over the + # compose network as questdb:9000); it is only so you can inspect the DEMO's + # server from the host at http://localhost:19000 without touching your local one. + - "19000:9000" + healthcheck: + test: ["CMD-SHELL", "curl -fsS 'http://localhost:9000/exec?query=select%201' -o /dev/null || exit 1"] + interval: 3s + timeout: 3s + retries: 40 + start_period: 5s + + demo: + build: + context: .. + dockerfile: blotter-demo/demo.Dockerfile + depends_on: + questdb: + condition: service_healthy + tty: true + stdin_open: true + environment: + QDB_ADDR: questdb:9000 + QDB_HTTP: questdb:9000 diff --git a/blotter-demo/entrypoint.sh b/blotter-demo/entrypoint.sh new file mode 100755 index 0000000..6f05dac --- /dev/null +++ b/blotter-demo/entrypoint.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# Demo entrypoint: wait for QuestDB, create the base table + LIVE VIEW, start the +# synthetic feed in the background, then run the blotter in the foreground (TTY). +set -euo pipefail + +HTTP="${QDB_HTTP:-questdb:9000}" +ADDR="${QDB_ADDR:-questdb:9000}" + +exec_sql() { + # URL-encodes the query via curl --data-urlencode; fails the script on HTTP error. + curl -fsS -G "http://${HTTP}/exec" --data-urlencode "query=$1" -o /dev/null +} + +echo "[entrypoint] waiting for QuestDB at ${HTTP} ..." +until curl -fsS "http://${HTTP}/exec?query=select%201" -o /dev/null 2>/dev/null; do + sleep 1 +done +echo "[entrypoint] QuestDB is up" + +echo "[entrypoint] creating base table core_price_demo" +exec_sql "CREATE TABLE IF NOT EXISTS core_price_demo ( + timestamp TIMESTAMP, + symbol SYMBOL, + bid_price DOUBLE + ) TIMESTAMP(timestamp) PARTITION BY DAY WAL;" + +echo "[entrypoint] creating LIVE VIEW core_price_lv" +# A live view serves its results from memory, so ingested rows are visible +# immediately; FLUSH EVERY only controls how often results are committed to disk +# and does not affect what the blotter sees. Visible motion comes from the feed +# flushing to the base table often (feed.py flushes 20x/s). +exec_sql "CREATE LIVE VIEW IF NOT EXISTS core_price_lv + FLUSH EVERY 5s IN MEMORY 5s START FROM NOW AS + SELECT + timestamp, + symbol, + bid_price, + avg(bid_price) OVER w AS moving_avg + FROM core_price_demo + WINDOW w AS (PARTITION BY symbol ORDER BY timestamp ANCHOR DAILY '00:00');" + +echo "[entrypoint] starting synthetic feed" +python /app/feed.py & +FEED_PID=$! +trap 'kill "${FEED_PID}" 2>/dev/null || true' EXIT INT TERM + +# Give the feed a moment so the first blotter frame is not empty. +sleep 2 + +echo "[entrypoint] starting blotter (Ctrl+C to quit)" +# Query the live view directly (the moving-average view), newest rows at the tail. +exec python /app/blotter.py \ + --table core_price_lv \ + --limit -20 \ + --rate 10 \ + --addr "${ADDR}" diff --git a/blotter-demo/feed.py b/blotter-demo/feed.py new file mode 100644 index 0000000..a7a48bc --- /dev/null +++ b/blotter-demo/feed.py @@ -0,0 +1,83 @@ +#!/usr/bin/env python3 +"""Continuous synthetic price feed for the blotter demo. + +Emits a steady trickle of bid prices for a small basket of instruments into +``core_price_demo``, flushing ~20x/second so the live view (and the blotter on +top of it) visibly moves. Prices random-walk around illustrative 2026 mid +levels; they are demo values, NOT a live market feed. + +Config via env: + QDB_ADDR host:port of the QWP endpoint (default questdb:9000) + QDB_CONF full conf string (overrides QDB_ADDR if set) + QDB_TABLE base table name (default core_price_demo) + FEED_FLUSH_HZ flushes per second (default 20) + FEED_ROWS_PER_FLUSH rows appended per flush (default 100) +""" +import os +import random +import time + +from questdb import Sender, TimestampNanos + +# Illustrative mid prices (not live market data). Crypto + major FX. +BOOK = { + "BTC-USD": 95_000.0, + "ETH-USD": 3_400.0, + "SOL-USD": 180.0, + "XRP-USD": 2.30, + "DOGE-USD": 0.38, + "EUR-USD": 1.0900, + "GBP-USD": 1.2700, + "USD-JPY": 150.00, + "AUD-USD": 0.6600, + "USD-CAD": 1.3600, +} + +CONF = os.environ.get( + "QDB_CONF", "ws::addr=" + os.environ.get("QDB_ADDR", "questdb:9000") + ";" +) +TABLE = os.environ.get("QDB_TABLE", "core_price_demo") +FLUSHES_PER_SEC = float(os.environ.get("FEED_FLUSH_HZ", "20")) +ROWS_PER_FLUSH = int(os.environ.get("FEED_ROWS_PER_FLUSH", "100")) + + +def jitter(px): + # ~2 bps gaussian step per tick, floored so a long walk can't go negative. + return max(px * 0.5, px + px * random.gauss(0.0, 0.0002)) + + +def main(): + prices = dict(BOOK) + symbols = list(BOOK) + interval = 1.0 / FLUSHES_PER_SEC + sent = 0 + window_start = time.time() + print(f"[feed] {CONF} -> {TABLE} | {FLUSHES_PER_SEC:g} flush/s x " + f"{ROWS_PER_FLUSH} rows = ~{FLUSHES_PER_SEC * ROWS_PER_FLUSH:.0f} rows/s", + flush=True) + with Sender.from_conf(CONF) as s: + while True: + tick = time.perf_counter() + for _ in range(ROWS_PER_FLUSH): + sym = random.choice(symbols) + prices[sym] = jitter(prices[sym]) + s.row( + TABLE, + symbols={"symbol": sym}, + columns={"bid_price": round(prices[sym], 6)}, + at=TimestampNanos.now(), + ) + sent += 1 + s.flush() + now = time.time() + if now - window_start >= 5.0: + print(f"[feed] ~{sent / (now - window_start):.0f} rows/s", flush=True) + sent = 0 + window_start = now + slack = interval - (time.perf_counter() - tick) + if slack > 0: + time.sleep(slack) + + +if __name__ == "__main__": + main() diff --git a/blotter-demo/questdb.Dockerfile b/blotter-demo/questdb.Dockerfile new file mode 100644 index 0000000..bfc8456 --- /dev/null +++ b/blotter-demo/questdb.Dockerfile @@ -0,0 +1,58 @@ +# QuestDB server built from source at the commit that first ships LIVE VIEW. +# (Not yet in questdb/questdb:nightly at the time of writing. Once this merges to +# main and a nightly picks it up, replace this whole build with a one-liner: +# image: questdb/questdb:nightly +# in docker-compose.yml and delete this file.) +# +# Pinned by SHA so the image is reproducible. This commit is expected to merge to +# main, which keeps the SHA reachable (and therefore fetchable) permanently. +ARG QDB_COMMIT=90a1b54c98b10fad5304b1ad817a69cda25e52ad + +FROM eclipse-temurin:25-jdk AS build +ARG QDB_COMMIT +RUN apt-get update && apt-get install -y --no-install-recommends \ + git maven cmake nasm build-essential ca-certificates \ + nodejs npm \ + && rm -rf /var/lib/apt/lists/* +ENV JAVA_HOME=/opt/java/openjdk + +WORKDIR /src +RUN git clone https://github.com/questdb/questdb.git questdb +WORKDIR /src/questdb +RUN git checkout "${QDB_COMMIT}" +RUN git submodule update --init --recursive + +# Build the bundled java client native lib first (matches the documented build: +# cmake in java-questdb-client/core, then mvn install that module), then package +# the server with the web console (-P build-web-console needs Node/npm, installed +# above), so http:/// serves the QuestDB console. +WORKDIR /src/questdb/java-questdb-client/core +RUN cmake -DCMAKE_BUILD_TYPE=Release -B cmake-build-release -S . \ + && cmake --build cmake-build-release --config Release +WORKDIR /src/questdb/java-questdb-client +RUN mvn -q -B install -DskipTests +WORKDIR /src/questdb +RUN mvn -B package -DskipTests -P build-web-console +# The runnable server jar (exclude sources/tests/javadoc classifiers). +RUN cp "$(ls core/target/questdb-*-SNAPSHOT.jar | grep -Ev 'sources|tests|javadoc' | head -1)" /questdb.jar + +FROM eclipse-temurin:25-jre +RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates \ + && rm -rf /var/lib/apt/lists/* \ + && useradd -m qdb \ + && mkdir -p /var/lib/questdb && chown qdb /var/lib/questdb +COPY --from=build /questdb.jar /questdb.jar +USER qdb +EXPOSE 9000 +# QuestDB requires Java 25 (core/pom.xml: javac.target=25, and the enforcer's +# java25+ profile activates on jdk (24,)). This exact flag set is the project's +# documented start command. +ENTRYPOINT ["java", \ + "--sun-misc-unsafe-memory-access=allow", \ + "--enable-native-access=ALL-UNNAMED", \ + "--add-opens=java.base/java.lang=ALL-UNNAMED", \ + "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED", \ + "--add-opens=java.base/java.nio=ALL-UNNAMED", \ + "--add-opens=java.base/java.time.zone=ALL-UNNAMED", \ + "--add-exports=java.base/jdk.internal.vm=ALL-UNNAMED", \ + "-jar", "/questdb.jar", "-d", "/var/lib/questdb"] diff --git a/blotter-demo/vendor.sh b/blotter-demo/vendor.sh new file mode 100755 index 0000000..764b9b6 --- /dev/null +++ b/blotter-demo/vendor.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# Fallback for when the client branch is deleted before it merges: snapshot the +# py-questdb-client source (at the pinned commit, submodules included) into a local +# tarball so the Docker build no longer depends on GitHub keeping the SHA reachable. +# +# Usage: +# ./vendor.sh # snapshots CLIENT_COMMIT below into ./vendor/ +# Then edit demo.Dockerfile's client-build stage to use the tarball instead of the +# git clone/checkout (a COPY + tar -xzf), and rebuild. Instructions printed at the end. +set -euo pipefail + +CLIENT_COMMIT="${CLIENT_COMMIT:-ea54b6f474062c144aa6395facad42e77c99e6f6}" +HERE="$(cd "$(dirname "$0")" && pwd)" +OUT="${HERE}/vendor" +WORK="$(mktemp -d)" +trap 'rm -rf "${WORK}"' EXIT + +mkdir -p "${OUT}" +echo "[vendor] cloning py-questdb-client @ ${CLIENT_COMMIT} ..." +git clone https://github.com/questdb/py-questdb-client.git "${WORK}/src" +git -C "${WORK}/src" checkout "${CLIENT_COMMIT}" +git -C "${WORK}/src" submodule update --init --recursive + +TARBALL="${OUT}/py-questdb-client-${CLIENT_COMMIT:0:12}.tar.gz" +echo "[vendor] writing ${TARBALL}" +# Exclude .git to keep the tarball small; source + submodules are enough to build. +tar --exclude-vcs -czf "${TARBALL}" -C "${WORK}/src" . + +cat < ${TARBALL} + +To use it, replace the git clone/checkout/submodule lines in the client-build +stage of blotter-demo/demo.Dockerfile with: + + COPY blotter-demo/vendor/py-questdb-client-${CLIENT_COMMIT:0:12}.tar.gz /tmp/client.tar.gz + RUN mkdir -p /src && tar -xzf /tmp/client.tar.gz -C /src + +Then rebuild: docker compose build demo +EOF From 5323671774bc7e73142dda447372d9e3c3b8d5d0 Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 5 Aug 2026 11:40:03 +0200 Subject: [PATCH 36/79] blotter-demo: drop source builds, use nightly image + released 5.0.0 client Both components the demo previously built from source are now shippable: - QuestDB LIVE VIEW is in questdb/questdb:nightly (merged, commit 73685fa). Replace the from-source questdb.Dockerfile with image: questdb/questdb:nightly. - The Python client is released as questdb==5.0.0 on PyPI, with the query API the blotter uses (connect -> query -> to_polars). Replace the Rust + Cython client-build stage in demo.Dockerfile with a plain pip install. Delete questdb.Dockerfile and vendor.sh (the latter only snapshotted the unreleased client branch). Rewrite README for released components and add the one-line swap to questdb/questdb:10.0 once it ships. Build time drops from minutes to ~40s; nothing is pinned to a fragile branch SHA anymore. Verified end to end: nightly live view returns data and the 5.0.0 wheel renders the blotter with live moving averages. --- blotter-demo/README.md | 112 +++++++++++++++++--------------- blotter-demo/demo.Dockerfile | 38 ++--------- blotter-demo/docker-compose.yml | 10 +-- blotter-demo/questdb.Dockerfile | 58 ----------------- blotter-demo/vendor.sh | 40 ------------ 5 files changed, 72 insertions(+), 186 deletions(-) delete mode 100644 blotter-demo/questdb.Dockerfile delete mode 100755 blotter-demo/vendor.sh diff --git a/blotter-demo/README.md b/blotter-demo/README.md index 5a2becc..2af601e 100644 --- a/blotter-demo/README.md +++ b/blotter-demo/README.md @@ -1,15 +1,20 @@ # One-command live blotter demo -A self-contained Docker demo: QuestDB with **LIVE VIEW**, the unreleased -**questdb 5.0 Python client**, a synthetic price feed, and the terminal blotter -rendering on top of a live view. Your colleague needs **nothing but Docker** — no -Python, no Rust, no client build. +A self-contained Docker demo: QuestDB with **LIVE VIEW**, the **questdb 5.0 Python +client**, a synthetic price feed, and the terminal blotter rendering on top of a +live view. Your colleague needs **nothing but Docker** — no Python, no client +install. + +Nothing is built from source anymore: QuestDB comes from the official `nightly` +image (the first images to ship `LIVE VIEW`), and the client installs from PyPI +(`questdb==5.0.0`). The only image `build` does is layer the feed/blotter scripts +onto `python:3.12-slim` — a few seconds. ## Run it ```bash cd blotter-demo -docker compose build # one-time, slow: builds QuestDB + the client from source +docker compose build # one-time, fast: pip installs the client + copies scripts docker compose run --rm demo # QuestDB starts, then the blotter draws in your terminal ``` @@ -22,44 +27,44 @@ docker compose down # stop QuestDB ## Running it again -The slow part is the one-time `docker compose build`. Once the images exist, every -`docker compose run --rm demo` skips the build entirely and just starts containers, -so repeat runs are fast — the only cost is a few seconds of JVM startup as QuestDB -cold-starts (the `--rm`/`down` flow disposes the container each time, so the table, -live view, and history start fresh on every run — intended for a demo). A rebuild -only happens if you edit a file a Docker layer depends on, and even then only the -cheap final layers re-run — never the Rust/Maven builds — unless you change the -pinned client/server commits. +Once the `demo` image exists and the `nightly` image is pulled, every +`docker compose run --rm demo` just starts containers, so repeat runs are fast — +the only cost is a few seconds of JVM startup as QuestDB cold-starts (the +`--rm`/`down` flow disposes the container each time, so the table, live view, and +history start fresh on every run — intended for a demo). A rebuild only happens if +you edit `feed.py`, `blotter.py`, or `entrypoint.sh` — and even then only the cheap +COPY layers re-run. + +To get a newer QuestDB, re-pull the moving tag: `docker compose pull questdb`. If a `docker compose down` reports "network still in use", a run container lingered from a Ctrl+C; clear it with `docker compose down --remove-orphans`. ## Removing everything at the end -This demo creates four kinds of Docker artifact: two built images -(`blotter-demo-questdb`, `blotter-demo-demo`), a network and containers, build -cache (the Rust + Maven layers — the bulk of the disk), and base images it pulls -(`python:3.12-slim`, `eclipse-temurin:25-jdk`, `eclipse-temurin:25-jre`). +This demo creates: one built image (`blotter-demo-demo`), a network and containers, +a little build cache (the pip layer), and two base images it pulls +(`python:3.12-slim`, `questdb/questdb:nightly`). **Removing an image is not destructive the way deleting a volume is** — any image -can be re-pulled from Docker Hub on demand, so the worst case of over-removing is a -re-download later. No data is lost. The demo defines no named volumes. +can be re-pulled on demand, so the worst case of over-removing is a re-download +later. No data is lost. The demo defines no named volumes. -### Step 1 — containers, network, and the two built images (scoped, safe) +### Step 1 — containers, network, and the built image (scoped, safe) ```bash docker compose down --rmi local --remove-orphans ``` -Touches only this project. `--rmi local` deletes the two images the demo built; -`--remove-orphans` clears any run container left by a Ctrl+C. +Touches only this project. `--rmi local` deletes the `blotter-demo-demo` image the +demo built; `--remove-orphans` clears any run container left by a Ctrl+C. ### Step 2 — the base images this demo pulled Docker does not record when an image was pulled, so "was it already there before?" can only be answered by snapshotting the image list **before** the first build. -**Before your first `docker compose build`**, record what already exists: +**Before your first `docker compose build`/`pull`**, record what already exists: ```bash docker image ls --format '{{.Repository}}:{{.Tag}}' | sort > ~/blotter-demo-images-before.txt @@ -69,20 +74,20 @@ docker image ls --format '{{.Repository}}:{{.Tag}}' | sort > ~/blotter-demo-imag only if this demo pulled it): ```bash -for img in python:3.12-slim eclipse-temurin:25-jdk eclipse-temurin:25-jre; do +for img in python:3.12-slim questdb/questdb:nightly; do grep -qxF "$img" ~/blotter-demo-images-before.txt || docker rmi "$img" done ``` If you did **not** take the snapshot, you cannot tell which were pre-existing. -Either leave them, or just remove all three — re-pulling is harmless (no data loss), +Either leave them, or just remove both — re-pulling is harmless (no data loss), it only costs a download if another project needs them later: ```bash -docker rmi python:3.12-slim eclipse-temurin:25-jdk eclipse-temurin:25-jre +docker rmi python:3.12-slim questdb/questdb:nightly ``` -### Step 3 — reclaim the build cache (the big disk win) +### Step 3 — reclaim the build cache ```bash docker builder prune -f @@ -92,8 +97,8 @@ docker builder prune -f deletes images, containers, or volumes** — those are safe. But it is **system-wide, not scoped to this demo**: it clears the build cache of every project, so other projects' next builds recompute their layers (slower, but nothing is lost — cache -only). There is no per-project build-cache prune. If this demo is your only heavy -Docker build, it is effectively scoped anyway. +only). There is no per-project build-cache prune. This demo's own cache is now tiny +(just the pip layer), so this step mostly matters if other projects share the daemon. ### Not recommended @@ -102,12 +107,13 @@ all projects**, not just this demo — only run it if you genuinely mean machine ## What it does -1. **`questdb`** — built from source at the commit that first ships `LIVE VIEW` - (`90a1b54c…`) with the web console (`-P build-web-console`). The server is - reachable from the host at **http://localhost:19000** (console + `/exec`); this - host port is deliberately 19000, not 9000, so it never collides with a local - QuestDB. The blotter reaches the server internally over the compose network. -2. **`demo`** — builds the 5.0 client (`ea54b6f…`, Rust 1.91.1 + Cython), then: +1. **`questdb`** — the official `questdb/questdb:nightly` image, the first to ship + `LIVE VIEW`. Reachable from the host at **http://localhost:19000** (console + + `/exec`); this host port is deliberately 19000, not 9000, so it never collides + with a local QuestDB. The blotter reaches the server internally over the compose + network at `questdb:9000`. +2. **`demo`** — `python:3.12-slim` + `pip install questdb==5.0.0 polars pyarrow`, + then: - waits for QuestDB, - `curl` creates `core_price_demo` (base table) and `core_price_lv` (live view), - runs `feed.py` in the background (~2000 rows/s of synthetic crypto/FX bids), @@ -145,28 +151,28 @@ refreshed at `--rate` Hz. (`--table core_price_lv --limit -20`). Adjust the count with `--limit`, or pass `--query ""` in `entrypoint.sh` to run any SQL against the view verbatim. -## Pins and durability - -Both sources are pinned by commit SHA for reproducibility: +## Versions -| Component | Repo | Commit | +| Component | Source | Version | | --- | --- | --- | -| QuestDB server | `questdb/questdb` | `90a1b54c98b10fad5304b1ad817a69cda25e52ad` | -| Python client | `questdb/py-questdb-client` | `ea54b6f474062c144aa6395facad42e77c99e6f6` | +| QuestDB server | `questdb/questdb:nightly` | first images with `LIVE VIEW` (merged 2026-08, commit `73685fa`) | +| Python client | PyPI `questdb` | `5.0.0` | -A pinned SHA is fetchable from GitHub only while it stays reachable from a ref. -The questdb commit is expected to merge to `main`, so it survives. The **client -commit lives on `jh_experiment_new_ilp`, which may be deleted before it merges** — -if that happens the build's `git checkout` will fail. Run `./vendor.sh` to -snapshot the client source into `blotter-demo/vendor/` and follow its printed instructions -to switch the Dockerfile to the local tarball. +`:nightly` is a **moving tag** — every pull may bring a newer build, so +reproducibility is weaker than a fixed version. For a "colleague runs it once" demo +that is fine. To freeze it, pin the `image:` in `docker-compose.yml` by digest +(`questdb/questdb@sha256:…`). -## When the client ships / questdb merges +## When QuestDB 10.0 ships + +`LIVE VIEW` is expected in the **10.0** stable release. Once it lands, swap the +moving nightly tag for the pinned release in `docker-compose.yml`: + +```yaml + image: questdb/questdb:10.0 +``` -- Once `LIVE VIEW` lands in a nightly, delete `questdb.Dockerfile` and set - `image: questdb/questdb:nightly` on the `questdb` service. -- Once the client is on PyPI, replace the whole `client-build` stage with - `pip install questdb==`. +That is the only change — everything else already uses released components. ## Sharing without a rebuild @@ -174,4 +180,4 @@ To hand a colleague an image that needs no build at all, either push to a regist (`docker compose build && docker compose push`, after adding `image:` names) or `docker save`/`docker load` a tarball. Both are architecture-specific — an amd64 image runs under emulation on Apple Silicon. Letting them `docker compose build` -locally avoids that by building for their arch. +locally (a few seconds now) avoids that by building for their arch. diff --git a/blotter-demo/demo.Dockerfile b/blotter-demo/demo.Dockerfile index c10c1c4..c3f24bb 100644 --- a/blotter-demo/demo.Dockerfile +++ b/blotter-demo/demo.Dockerfile @@ -1,39 +1,15 @@ -# Blotter demo app: the unreleased questdb 5.0 Python client (built from source), -# plus the feed + blotter scripts. Built once, then runs with zero host deps. +# Blotter demo app: the questdb 5.0 Python client (from PyPI) plus the feed + +# blotter scripts. Built once, then runs with zero host deps. # -# The client is pinned by SHA. NOTE: unlike the questdb server commit, this branch -# (jh_experiment_new_ilp) may be DELETED before it merges, which would make the SHA -# unfetchable from GitHub. If that happens, run blotter-demo/vendor.sh to snapshot the -# source locally and switch the client-build stage to COPY it in (see vendor.sh). -ARG CLIENT_COMMIT=ea54b6f474062c144aa6395facad42e77c99e6f6 -ARG RUST_VERSION=1.91.1 - -FROM python:3.12-slim AS client-build -ARG CLIENT_COMMIT -ARG RUST_VERSION -RUN apt-get update && apt-get install -y --no-install-recommends \ - git curl build-essential pkg-config ca-certificates \ - && rm -rf /var/lib/apt/lists/* -# Rust >= 1.91.1: the bundled c-questdb-client (questdb-rs 7.0.0) requires it. -RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \ - | sh -s -- -y --default-toolchain "${RUST_VERSION}" --profile minimal -ENV PATH="/root/.cargo/bin:${PATH}" - -WORKDIR /src -RUN git clone https://github.com/questdb/py-questdb-client.git . -RUN git checkout "${CLIENT_COMMIT}" -RUN git submodule update --init --recursive -# Build-time deps must stay in the isolated build env (do NOT use -# --no-build-isolation; that pins the wrong numpy and fails the cython compile). -RUN pip install --no-cache-dir -U pip "cython>=3.1.2" "setuptools>=80.9.0" numpy -RUN pip wheel --no-deps --no-cache-dir -w /wheels . - +# The client shipped as 5.0.0 stable, so there is no source build anymore — the +# query API (questdb.connect(...).query(sql).to_polars()) that the blotter uses is +# in the released wheel. (Before 5.0.0 this stage compiled the client from an +# unreleased branch with a full Rust + Cython toolchain.) FROM python:3.12-slim RUN apt-get update && apt-get install -y --no-install-recommends \ curl ca-certificates \ && rm -rf /var/lib/apt/lists/* -COPY --from=client-build /wheels /wheels -RUN pip install --no-cache-dir /wheels/*.whl polars pyarrow +RUN pip install --no-cache-dir questdb==5.0.0 polars pyarrow WORKDIR /app COPY python/blotter.py /app/blotter.py diff --git a/blotter-demo/docker-compose.yml b/blotter-demo/docker-compose.yml index 3a52a05..3732993 100644 --- a/blotter-demo/docker-compose.yml +++ b/blotter-demo/docker-compose.yml @@ -1,6 +1,6 @@ # One-command blotter demo. From this directory: # -# docker compose build # one-time, slow (builds QuestDB + client from source) +# docker compose build # one-time: builds the small demo image (client from PyPI) # docker compose run --rm demo # starts QuestDB, then the blotter in your terminal # # `run` (not `up`) gives the blotter a real TTY for the in-place redraw. Ctrl+C quits; @@ -8,9 +8,11 @@ services: questdb: - build: - context: .. - dockerfile: blotter-demo/questdb.Dockerfile + # QuestDB nightly ships LIVE VIEW (merged 2026-08, commit 73685fa; not yet in a + # stable release or the :latest image). Swap to questdb/questdb:10.0 once 10.0 + # ships. `:nightly` is a moving tag, so reproducibility is weaker than a pin — fine + # for a demo; pin by digest if you need it frozen. + image: questdb/questdb:nightly ports: # Host 19000 -> container 9000. Deliberately NOT 9000 on the host: that is # the default QuestDB port, and a local QuestDB there would collide. The diff --git a/blotter-demo/questdb.Dockerfile b/blotter-demo/questdb.Dockerfile deleted file mode 100644 index bfc8456..0000000 --- a/blotter-demo/questdb.Dockerfile +++ /dev/null @@ -1,58 +0,0 @@ -# QuestDB server built from source at the commit that first ships LIVE VIEW. -# (Not yet in questdb/questdb:nightly at the time of writing. Once this merges to -# main and a nightly picks it up, replace this whole build with a one-liner: -# image: questdb/questdb:nightly -# in docker-compose.yml and delete this file.) -# -# Pinned by SHA so the image is reproducible. This commit is expected to merge to -# main, which keeps the SHA reachable (and therefore fetchable) permanently. -ARG QDB_COMMIT=90a1b54c98b10fad5304b1ad817a69cda25e52ad - -FROM eclipse-temurin:25-jdk AS build -ARG QDB_COMMIT -RUN apt-get update && apt-get install -y --no-install-recommends \ - git maven cmake nasm build-essential ca-certificates \ - nodejs npm \ - && rm -rf /var/lib/apt/lists/* -ENV JAVA_HOME=/opt/java/openjdk - -WORKDIR /src -RUN git clone https://github.com/questdb/questdb.git questdb -WORKDIR /src/questdb -RUN git checkout "${QDB_COMMIT}" -RUN git submodule update --init --recursive - -# Build the bundled java client native lib first (matches the documented build: -# cmake in java-questdb-client/core, then mvn install that module), then package -# the server with the web console (-P build-web-console needs Node/npm, installed -# above), so http:/// serves the QuestDB console. -WORKDIR /src/questdb/java-questdb-client/core -RUN cmake -DCMAKE_BUILD_TYPE=Release -B cmake-build-release -S . \ - && cmake --build cmake-build-release --config Release -WORKDIR /src/questdb/java-questdb-client -RUN mvn -q -B install -DskipTests -WORKDIR /src/questdb -RUN mvn -B package -DskipTests -P build-web-console -# The runnable server jar (exclude sources/tests/javadoc classifiers). -RUN cp "$(ls core/target/questdb-*-SNAPSHOT.jar | grep -Ev 'sources|tests|javadoc' | head -1)" /questdb.jar - -FROM eclipse-temurin:25-jre -RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates \ - && rm -rf /var/lib/apt/lists/* \ - && useradd -m qdb \ - && mkdir -p /var/lib/questdb && chown qdb /var/lib/questdb -COPY --from=build /questdb.jar /questdb.jar -USER qdb -EXPOSE 9000 -# QuestDB requires Java 25 (core/pom.xml: javac.target=25, and the enforcer's -# java25+ profile activates on jdk (24,)). This exact flag set is the project's -# documented start command. -ENTRYPOINT ["java", \ - "--sun-misc-unsafe-memory-access=allow", \ - "--enable-native-access=ALL-UNNAMED", \ - "--add-opens=java.base/java.lang=ALL-UNNAMED", \ - "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED", \ - "--add-opens=java.base/java.nio=ALL-UNNAMED", \ - "--add-opens=java.base/java.time.zone=ALL-UNNAMED", \ - "--add-exports=java.base/jdk.internal.vm=ALL-UNNAMED", \ - "-jar", "/questdb.jar", "-d", "/var/lib/questdb"] diff --git a/blotter-demo/vendor.sh b/blotter-demo/vendor.sh deleted file mode 100755 index 764b9b6..0000000 --- a/blotter-demo/vendor.sh +++ /dev/null @@ -1,40 +0,0 @@ -#!/usr/bin/env bash -# Fallback for when the client branch is deleted before it merges: snapshot the -# py-questdb-client source (at the pinned commit, submodules included) into a local -# tarball so the Docker build no longer depends on GitHub keeping the SHA reachable. -# -# Usage: -# ./vendor.sh # snapshots CLIENT_COMMIT below into ./vendor/ -# Then edit demo.Dockerfile's client-build stage to use the tarball instead of the -# git clone/checkout (a COPY + tar -xzf), and rebuild. Instructions printed at the end. -set -euo pipefail - -CLIENT_COMMIT="${CLIENT_COMMIT:-ea54b6f474062c144aa6395facad42e77c99e6f6}" -HERE="$(cd "$(dirname "$0")" && pwd)" -OUT="${HERE}/vendor" -WORK="$(mktemp -d)" -trap 'rm -rf "${WORK}"' EXIT - -mkdir -p "${OUT}" -echo "[vendor] cloning py-questdb-client @ ${CLIENT_COMMIT} ..." -git clone https://github.com/questdb/py-questdb-client.git "${WORK}/src" -git -C "${WORK}/src" checkout "${CLIENT_COMMIT}" -git -C "${WORK}/src" submodule update --init --recursive - -TARBALL="${OUT}/py-questdb-client-${CLIENT_COMMIT:0:12}.tar.gz" -echo "[vendor] writing ${TARBALL}" -# Exclude .git to keep the tarball small; source + submodules are enough to build. -tar --exclude-vcs -czf "${TARBALL}" -C "${WORK}/src" . - -cat < ${TARBALL} - -To use it, replace the git clone/checkout/submodule lines in the client-build -stage of blotter-demo/demo.Dockerfile with: - - COPY blotter-demo/vendor/py-questdb-client-${CLIENT_COMMIT:0:12}.tar.gz /tmp/client.tar.gz - RUN mkdir -p /src && tar -xzf /tmp/client.tar.gz -C /src - -Then rebuild: docker compose build demo -EOF From 9f00d286071e481d19f93b05c8c0c87a4092fe94 Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 5 Aug 2026 14:08:06 +0200 Subject: [PATCH 37/79] blotter-demo: high-throughput feed via columnar dataframe send Replace the per-row ingestion loop in feed.py with a vectorized path: each flush builds one Polars DataFrame (numpy-generated random walk) and ships it with a single Sender.dataframe(...) call. Over QWP/WebSocket that is a direct columnar bulk load, so a single process sustains ~1M rows/s where the row API capped far lower. New env knobs: - FEED_TARGET_RPS (default 2000, so the default demo is visually unchanged) sets the target rows/second; set 100000+ to stress ingestion. - FEED_WORKERS (default 1) forks N parallel sender processes, each targeting an equal share, to push past what one process generates. Batch size is derived as FEED_TARGET_RPS / FEED_WORKERS / FEED_FLUSH_HZ. Verified against nightly: Sender.dataframe accepts a Polars DataFrame directly (no pandas needed), single-process throughput ~560K-1.07M rows/s, and a full stack at FEED_TARGET_RPS=100000 measured ~93-98K rows/s server-side. Update README Knobs and "What it does" for the new rate controls. --- blotter-demo/README.md | 23 +++++++-- blotter-demo/feed.py | 109 +++++++++++++++++++++++++++++------------ 2 files changed, 96 insertions(+), 36 deletions(-) diff --git a/blotter-demo/README.md b/blotter-demo/README.md index 2af601e..220b8ad 100644 --- a/blotter-demo/README.md +++ b/blotter-demo/README.md @@ -116,7 +116,8 @@ all projects**, not just this demo — only run it if you genuinely mean machine then: - waits for QuestDB, - `curl` creates `core_price_demo` (base table) and `core_price_lv` (live view), - - runs `feed.py` in the background (~2000 rows/s of synthetic crypto/FX bids), + - runs `feed.py` in the background (synthetic crypto/FX bids; ~2000 rows/s by + default, tunable to 100K+/s via `FEED_TARGET_RPS` — see Knobs), - runs `blotter.py` in the foreground, polling the live view at 10 Hz. The live view: @@ -141,10 +142,22 @@ refreshed at `--rate` Hz. ## Knobs -- **Feed rate / flush frequency.** `FEED_FLUSH_HZ` (default 20, i.e. a flush every - 50 ms) and `FEED_ROWS_PER_FLUSH` (default 100) on the `demo` service set how - fast rows land and how often they are flushed. 20 flushes/s keeps the blotter - visibly moving. Prices are illustrative 2026 levels, not live market data. +- **Feed rate.** `FEED_TARGET_RPS` (default 2000) on the `demo` service sets the + target rows/second. The feed builds one vectorized Polars DataFrame per flush + and ships it with a single columnar `Sender.dataframe(...)` call, so a single + process sustains ~1M rows/s — set `FEED_TARGET_RPS=100000` (or higher) to stress + ingestion: + ```bash + docker compose run --rm -e FEED_TARGET_RPS=100000 demo + ``` + Prices are illustrative 2026 levels, not live market data. +- **Flush cadence / workers.** `FEED_FLUSH_HZ` (default 20) is flushes/second per + worker; batch size is `FEED_TARGET_RPS / FEED_WORKERS / FEED_FLUSH_HZ`. + `FEED_WORKERS` (default 1) forks that many parallel sender processes, each + targeting an equal share — only needed to push past what one process can + generate. The feed prints its measured `~N rows/s` every 5 s; confirm the true + server-side rate with + `select count()/10.0 from core_price_demo where timestamp > dateadd('s', -10, now())`. - **Blotter refresh.** `--rate` Hz in `entrypoint.sh` (default 10) sets how often the terminal redraws. - **What the blotter shows.** Defaults to the last 20 rows of the live view diff --git a/blotter-demo/feed.py b/blotter-demo/feed.py index a7a48bc..666cbee 100644 --- a/blotter-demo/feed.py +++ b/blotter-demo/feed.py @@ -1,23 +1,35 @@ #!/usr/bin/env python3 """Continuous synthetic price feed for the blotter demo. -Emits a steady trickle of bid prices for a small basket of instruments into -``core_price_demo``, flushing ~20x/second so the live view (and the blotter on -top of it) visibly moves. Prices random-walk around illustrative 2026 mid -levels; they are demo values, NOT a live market feed. +Emits bid prices for a small basket of instruments into ``core_price_demo`` at a +configurable target rate, flushing several times a second so the live view (and +the blotter on top of it) visibly moves. Prices random-walk around illustrative +2026 mid levels; they are demo values, NOT a live market feed. + +Throughput path: each flush builds one vectorized Polars DataFrame (numpy under +the hood) and sends it with a single ``Sender.dataframe(...)`` call. Over +QWP/WebSocket that is a direct columnar bulk load, so one process sustains +~1M rows/s on a laptop — the row-by-row API can't. To go beyond one process, +set FEED_WORKERS. Config via env: QDB_ADDR host:port of the QWP endpoint (default questdb:9000) QDB_CONF full conf string (overrides QDB_ADDR if set) QDB_TABLE base table name (default core_price_demo) - FEED_FLUSH_HZ flushes per second (default 20) - FEED_ROWS_PER_FLUSH rows appended per flush (default 100) + FEED_TARGET_RPS target rows/second, total across workers (default 2000) + FEED_FLUSH_HZ flushes per second per worker (default 20); batch size is + target_rps / workers / flush_hz + FEED_WORKERS parallel sender processes (default 1); each targets an + equal share of FEED_TARGET_RPS. Only needed to push well + past what one process can generate. """ import os -import random import time -from questdb import Sender, TimestampNanos +import numpy as np +import polars as pl + +from questdb import Sender # Illustrative mid prices (not live market data). Crypto + major FX. BOOK = { @@ -37,47 +49,82 @@ "QDB_CONF", "ws::addr=" + os.environ.get("QDB_ADDR", "questdb:9000") + ";" ) TABLE = os.environ.get("QDB_TABLE", "core_price_demo") +TARGET_RPS = float(os.environ.get("FEED_TARGET_RPS", "2000")) FLUSHES_PER_SEC = float(os.environ.get("FEED_FLUSH_HZ", "20")) -ROWS_PER_FLUSH = int(os.environ.get("FEED_ROWS_PER_FLUSH", "100")) - - -def jitter(px): - # ~2 bps gaussian step per tick, floored so a long walk can't go negative. - return max(px * 0.5, px + px * random.gauss(0.0, 0.0002)) +WORKERS = max(1, int(os.environ.get("FEED_WORKERS", "1"))) -def main(): - prices = dict(BOOK) - symbols = list(BOOK) +def worker(worker_id, rows_per_flush): + """One sender process: generate and ship `rows_per_flush` rows every 1/FLUSH_HZ s.""" + # Distinct stream per worker so the walks don't move in lockstep. (Runtime + # process, so a seeded default_rng is fine and keeps each worker independent.) + rng = np.random.default_rng(1000 + worker_id) + symbols = np.array(list(BOOK)) + mids = np.array(list(BOOK.values()), dtype=np.float64) + floors = mids * 0.5 # a long walk can't drift below half the start level + n_sym = len(symbols) interval = 1.0 / FLUSHES_PER_SEC + sent = 0 window_start = time.time() - print(f"[feed] {CONF} -> {TABLE} | {FLUSHES_PER_SEC:g} flush/s x " - f"{ROWS_PER_FLUSH} rows = ~{FLUSHES_PER_SEC * ROWS_PER_FLUSH:.0f} rows/s", - flush=True) with Sender.from_conf(CONF) as s: while True: tick = time.perf_counter() - for _ in range(ROWS_PER_FLUSH): - sym = random.choice(symbols) - prices[sym] = jitter(prices[sym]) - s.row( - TABLE, - symbols={"symbol": sym}, - columns={"bid_price": round(prices[sym], 6)}, - at=TimestampNanos.now(), - ) - sent += 1 + + # Drift each symbol's mid once per batch (~2 bps), then draw this + # batch's rows around those mids with a little intra-batch noise. + mids = np.maximum(floors, mids * (1.0 + rng.normal(0.0, 0.0002, n_sym))) + idx = rng.integers(0, n_sym, rows_per_flush) + px = mids[idx] * (1.0 + rng.normal(0.0, 0.0001, rows_per_flush)) + + now_ns = int(time.time() * 1e9) + ts = np.datetime64(now_ns, "ns") + np.arange(rows_per_flush).astype("timedelta64[ns]") + + df = pl.DataFrame({ + "timestamp": ts, + "symbol": symbols[idx], + "bid_price": np.round(px, 6), + }) + s.dataframe(df, table_name=TABLE, symbols=["symbol"], at="timestamp") s.flush() + sent += rows_per_flush + now = time.time() if now - window_start >= 5.0: - print(f"[feed] ~{sent / (now - window_start):.0f} rows/s", flush=True) + print(f"[feed w{worker_id}] ~{sent / (now - window_start):.0f} rows/s", + flush=True) sent = 0 window_start = now + slack = interval - (time.perf_counter() - tick) if slack > 0: time.sleep(slack) +def main(): + per_worker_rps = TARGET_RPS / WORKERS + rows_per_flush = max(1, round(per_worker_rps / FLUSHES_PER_SEC)) + effective_rps = rows_per_flush * FLUSHES_PER_SEC * WORKERS + print(f"[feed] {CONF} -> {TABLE} | target ~{TARGET_RPS:.0f} rows/s | " + f"{WORKERS} worker(s) x {FLUSHES_PER_SEC:g} flush/s x {rows_per_flush} rows " + f"= ~{effective_rps:.0f} rows/s", flush=True) + + if WORKERS == 1: + worker(0, rows_per_flush) + return + + import multiprocessing as mp + procs = [mp.Process(target=worker, args=(i, rows_per_flush), daemon=True) + for i in range(WORKERS)] + for p in procs: + p.start() + try: + for p in procs: + p.join() + except KeyboardInterrupt: + for p in procs: + p.terminate() + + if __name__ == "__main__": main() From 1ab16d260105e727fd2f9b861c87c418c68a6e68 Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 26 Aug 2026 18:50:58 +0200 Subject: [PATCH 38/79] read_bench: stripe row-offset chunks across readers Splitting the scan range by equal timestamp spans assumes rows are uniformly distributed in time. On a skewed table one reader is handed far more rows than the others, and because the slices run oldest to newest the first reader also owns the coldest end of the range, so it straggles long after the rest finish. Add --split with two modes, defaulting to rows: - rows: cut the range into --chunks equal row-count slices using LIMIT -m, -n, which takes the last m rows then drops the last n of those, giving a half-open range. Consecutive chunks tile with no gap and no overlap. Workers pull chunks off a shared queue, so no reader owns the cold end and a worker that lands on cached data just takes the next chunk. Needs no preliminary query and never has to know the designated timestamp's name. - time: the previous behaviour, kept because its boundaries are absolute timestamps computed once, so all readers agree on them even while the table is appended to. Row offsets are re-evaluated per connection and can shift under concurrent writes. The offset skip is metadata-only, not a scan: PageFrameRecordCursorImpl.skipRows walks whole page frames subtracting partitionHi minus partitionLo, and only descends into the frame where the skip lands. Measured on a 10.5B row table, skipping 10.4B rows to return a single row takes 12ms. Each worker now holds one connection for its whole chunk sequence via a db.reader lease, and passes reset_symbol_dict=False so the connection's SYMBOL dictionary stays warm instead of being rebuilt once per chunk. --chunks defaults to readers x 8. The done line now reports chunks per reader. Verified: row_chunks output parsed back into row ranges across 10 shapes, including uneven division and chunks greater than limit, tiles exactly. End to end on a 500k row table, all three paths return exactly --limit rows and the queue rebalances unevenly across workers as intended. --- python/read_bench.py | 160 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 129 insertions(+), 31 deletions(-) diff --git a/python/read_bench.py b/python/read_bench.py index 26d6a8b..6d4148f 100644 --- a/python/read_bench.py +++ b/python/read_bench.py @@ -18,9 +18,34 @@ A single connection is capped by the client's socket buffer at ~426 KB per RTT (the QWP client hardcodes a 4 MiB buffer that default Linux clamps; see ../boost_tcp.sh), so on a real network one reader is round-trip bound no matter how lean the loop is. -``--readers N`` splits the row range across N parallel connections (each scans a -timestamp slice) and reports the aggregate rows/sec, which is the way past the -per-connection wall. ``--readers 1`` (default) is the plain single stream. +``--readers N`` runs N parallel connections and reports the aggregate rows/sec, which +is the way past the per-connection wall. ``--readers 1`` (default) is the plain +single stream. + +How the row range is divided across those readers is ``--split``: + +``rows`` (default) cuts the range into ``--chunks`` row-offset slices using QuestDB's +``LIMIT -m, -n`` (take the last m rows, then drop the last n, giving the half-open +range ``[-m, -n)``), and workers pull chunks off a shared queue. Two consequences +matter. Chunks hold an equal number of rows regardless of how unevenly the data is +distributed in time, and with ``chunks > readers`` no single worker owns the whole +cold end of the range - work is interleaved and self-balancing, since a worker that +lands on cached data just takes the next chunk. It also needs no preliminary query +and never has to know the designated timestamp's name. + +The offset skip is metadata-only, not a scan: ``PageFrameRecordCursorImpl.skipRows`` +walks whole page frames subtracting ``partitionHi - partitionLo`` and only descends +into the frame where the skip lands, so skipping billions of rows costs microseconds +(measured: 10.4B rows skipped in 12ms). Do not be alarmed by "Row forward scan" in +EXPLAIN - no column data is decoded for skipped frames. + +``time`` is the original behaviour: one preliminary query finds the range's min/max +timestamp, which is cut into N equal time spans, one per reader. Equal *time*, not +equal rows, so skew in the data's time distribution makes readers finish at wildly +different times. Its one advantage is that the boundaries are absolute timestamps +computed once, so all readers agree on them even if the table is being written to +concurrently; row offsets are re-evaluated per connection and can shift under +appends. Use ``time`` for a live table, ``rows`` for everything else. Auth / TLS (QuestDB Enterprise): pass ``--token`` (bearer) or ``--username``/``--password`` (basic). Either turns on TLS automatically (scheme ``wss``); ``--tls`` forces TLS with @@ -30,12 +55,14 @@ Usage: python read_bench.py TABLE \ [--addr host:9000] [--limit 10000000] [--readers 1] \ + [--split rows|time] [--chunks N] \ [--token TOK | --username U --password P] [--tls] [--tls-verify on|unsafe_off] Requires the questdb 5.0 client (see README.md). """ import argparse +import queue import sys import threading import time @@ -69,9 +96,36 @@ def ns_to_iso(ns): return dt.strftime("%Y-%m-%dT%H:%M:%S") + f".{frac:09d}Z" -def split_queries(args, conf): - """Build one SELECT per reader. For a single reader, the exact `limit -N` - query; for N readers, split the last-N rows' timestamp range into N slices.""" +def row_chunks(args): + """Build `--chunks` SELECTs covering the last `--limit` rows as equal row-count + slices, oldest first, using QuestDB's `LIMIT -m, -n`. + + Per the LIMIT reference, `LIMIT -m, -n` takes the last m records then drops the + last n of those, i.e. the half-open range [-m, -n) - so consecutive chunks tile + the range with no gap and no overlap. The newest chunk has n == 0, which is the + documented `LIMIT -n, 0` == `LIMIT -n` form. + + Needs no preliminary query: the bounds are arithmetic on --limit alone. + """ + table, limit, chunks = args.table, args.limit, args.chunks + edges = [(limit * i) // chunks for i in range(chunks + 1)] + sqls = [] + for i in range(chunks): + lo = limit - edges[i] # rows from the end, inclusive start + hi = limit - edges[i + 1] # rows from the end, exclusive end + if lo <= hi: # empty chunk (chunks > limit); nothing to read + continue + if hi == 0: + sqls.append(f"select * from {table} limit -{lo}") + else: + sqls.append(f"select * from {table} limit -{lo}, -{hi}") + return sqls + + +def time_slices(args, conf): + """Build one SELECT per reader by splitting the last-N rows' timestamp range into + N equal time spans. Costs one preliminary query and assumes --timestamp-col names + the designated timestamp.""" table, ts, limit, readers = args.table, args.timestamp_col, args.limit, args.readers if readers <= 1: return [f"select * from {table} limit -{limit}"] @@ -100,24 +154,45 @@ def split_queries(args, conf): return sqls -def run_reader(idx, sql, conf, counts, byts, errors, sample_n, sample_out): +def run_reader(idx, work, conf, counts, byts, chunks_done, errors, + sample_n, sample_out, sample_lock): + """One worker: hold a single connection and drain chunks off `work` until empty. + + `db.reader()` leases one pooled connection for the lease's lifetime and runs its + queries sequentially on it, so a worker is one connection no matter how many + chunks it processes. The lease must be used only on the thread that created it, + which is what this function is. + """ try: + n = 0 + b = 0 with questdb.connect(conf) as db: - result = db.query(sql) - n = 0 - b = 0 - for batch in result.iter_arrow(): - n += batch.num_rows # count rows and decoded Arrow bytes - b += batch.nbytes # (buffer bytes of this batch's columns) - if idx == 0 and sample_n > 0 and sample_out[0] is None: - # Reuse the Arrow data we already received: wrap the first batch as a polars - # DataFrame (zero-copy) and keep its head (one-off, no extra query, no re-scan). - import polars as pl - sample_out[0] = pl.from_arrow(batch).head(sample_n) - counts[idx] = n - byts[idx] = b - counts[idx] = n - byts[idx] = b + with db.reader() as reader: + while True: + try: + sql = work.get_nowait() + except queue.Empty: + break + # Every chunk hits the same table and the same SYMBOL columns, so keep + # the connection's symbol dictionary warm instead of rebuilding it per + # chunk. On the lease's first query the dictionary is empty anyway. + result = reader.query(sql, reset_symbol_dict=False) + for batch in result.iter_arrow(): + n += batch.num_rows # count rows and decoded Arrow bytes + b += batch.nbytes # (buffer bytes of this batch's columns) + if sample_n > 0 and sample_out[0] is None: + # Reuse Arrow data we already received: wrap the first batch any + # worker sees as a polars DataFrame (zero-copy) and keep its head + # (one-off, no extra query, no re-scan). + with sample_lock: + if sample_out[0] is None: + import polars as pl + sample_out[0] = pl.from_arrow(batch).head(sample_n) + counts[idx] = n + byts[idx] = b + counts[idx] = n + byts[idx] = b + chunks_done[idx] += 1 except Exception as e: # noqa: BLE001 errors.append(f"reader {idx}: {e}") @@ -129,9 +204,19 @@ def main(argv): ap.add_argument("--limit", type=int, default=10_000_000, help="number of most-recent rows to read; default 10,000,000") ap.add_argument("--readers", type=int, default=1, - help="parallel reader connections (split by timestamp); default 1") + help="parallel reader connections; default 1") + ap.add_argument("--split", choices=["rows", "time"], default="rows", + help="how to divide the range: 'rows' (equal row counts via LIMIT " + "offsets, striped over --chunks, no preliminary query) or " + "'time' (equal timestamp spans, one slice per reader, needs a " + "preliminary query but is stable under concurrent writes); " + "default rows") + ap.add_argument("--chunks", type=int, default=0, + help="number of row-offset chunks to stripe across readers " + "(--split rows only); default 0 = readers * 8. More chunks " + "balance better; each costs one extra query round trip") ap.add_argument("--timestamp-col", default="timestamp", - help="designated timestamp column (used to split across readers)") + help="designated timestamp column (--split time only)") ap.add_argument("--report-interval", type=float, default=0.5, help="seconds between progress lines; default 0.5") ap.add_argument("--sample", type=int, default=5, @@ -152,15 +237,27 @@ def main(argv): conf = build_conf(args) print(f"[conf] {'wss' if use_tls(args) else 'ws'} tls={use_tls(args)} " f"auth={'token' if args.token else 'basic' if args.username else 'none'}") - sqls = split_queries(args, conf) - readers = len(sqls) + if args.split == "rows": + args.chunks = max(args.chunks or args.readers * 8, args.readers) + sqls = row_chunks(args) + readers = min(args.readers, len(sqls)) + else: + sqls = time_slices(args, conf) + readers = len(sqls) # time mode is one slice per reader, capped by fallbacks + + work = queue.Queue() + for sql in sqls: + work.put(sql) + print(f"[scan] reading last {args.limit:,} rows of '{args.table}' " - f"across {readers} reader(s) ...") + f"across {readers} reader(s), {len(sqls)} {args.split} chunk(s) ...") counts = [0] * readers byts = [0] * readers + chunks_done = [0] * readers errors = [] - sample_out = [None] # reader 0 fills this with a few rows off its first Arrow batch + sample_out = [None] # first Arrow batch any worker sees, as a few polars rows + sample_lock = threading.Lock() stop = threading.Event() def reporter(): @@ -179,9 +276,10 @@ def reporter(): t0 = time.monotonic() threads = [] - for i, sql in enumerate(sqls): + for i in range(readers): t = threading.Thread(target=run_reader, - args=(i, sql, conf, counts, byts, errors, args.sample, sample_out)) + args=(i, work, conf, counts, byts, chunks_done, errors, + args.sample, sample_out, sample_lock)) t.start() threads.append(t) for t in threads: @@ -208,7 +306,7 @@ def reporter(): print(f"[done] {rate:,.0f} rows/s | {mbps:,.1f} MB/s | {gbps:.2f} Gb/s " f"(decoded Arrow payload, not wire bytes)") if readers > 1: - per = " ".join(f"r{i}={c:,}" for i, c in enumerate(counts)) + per = " ".join(f"r{i}={c:,}({chunks_done[i]}ch)" for i, c in enumerate(counts)) print(f"[done] per-reader rows: {per}") # An extract of the data we actually received: a few rows sliced off the first Arrow batch From c76d1b98c00256c692194620a94f3e680b1dc7db Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 26 Aug 2026 18:53:24 +0200 Subject: [PATCH 39/79] slippage: per-fill slippage query as a polars DataFrame Minimal script: connect over QWP, run the cookbook's per-fill slippage query against fx_trades ASOF JOINed to market_data, and hand back a polars DataFrame. Address comes from QDB_ADDR and the bearer token from ILP_TOKEN, so no credential is stored in the file. Uses wss with tls_verify=unsafe_off, matching the flags read_bench already takes for the same host. The mid alias is spelled out in full inside both CASE branches. A select-list alias is not in scope for a sibling expression in the same SELECT, so referencing mid there does not resolve; wrapping the projection in a subquery is the alternative if the shorter form is wanted. Not run end to end: the target host is not reachable from this machine, so fx_trades' presence and schema are unverified. Only the Python is syntax checked. --- python/slippage.py | 50 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 python/slippage.py diff --git a/python/slippage.py b/python/slippage.py new file mode 100644 index 0000000..eed9728 --- /dev/null +++ b/python/slippage.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 +"""Slippage per fill, vs mid and vs top-of-book, as a polars DataFrame. + + ILP_TOKEN=... python slippage.py # defaults to QDB_ADDR below + QDB_ADDR=host:9000 ILP_TOKEN=... python slippage.py + +Needs the questdb 5.0 client plus polars and pyarrow (to_polars needs both). +Swap to_polars for to_pandas if you want pandas instead. +""" +import os + +import questdb + +SQL = """ +SELECT + t.timestamp, + t.symbol, + t.ecn, + t.counterparty, + t.side, + t.passive, + t.price, + t.quantity, + m.best_bid, + m.best_ask, + (m.best_bid + m.best_ask) / 2 AS mid, + (m.best_ask - m.best_bid) AS spread, + CASE t.side + WHEN 'buy' THEN (t.price - (m.best_bid + m.best_ask) / 2) + / ((m.best_bid + m.best_ask) / 2) * 10000 + WHEN 'sell' THEN ((m.best_bid + m.best_ask) / 2 - t.price) + / ((m.best_bid + m.best_ask) / 2) * 10000 + END AS slippage_bps, + CASE t.side + WHEN 'buy' THEN (t.price - m.best_ask) / m.best_ask * 10000 + WHEN 'sell' THEN (m.best_bid - t.price) / m.best_bid * 10000 + END AS slippage_vs_tob_bps +FROM fx_trades t +ASOF JOIN market_data m ON (symbol) +WHERE t.timestamp IN '$yesterday' +ORDER BY t.timestamp +""" + +CONF = (f"wss::addr={os.environ.get('QDB_ADDR', '172.31.42.41:9000')};" + f"token={os.environ['ILP_TOKEN']};tls_verify=unsafe_off;") + +with questdb.connect(CONF) as db: + df = db.query(SQL).to_polars() + +print(df) From b811044cf37adfd0740d39c77b4e73188238f4b8 Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 26 Aug 2026 18:58:45 +0200 Subject: [PATCH 40/79] read_bench: skip empty batches when capturing the sample A chunk whose row offsets fall past the end of the table still returns a valid result: a schema-only batch of 0 rows. Those chunks finish first, so a worker latched one, stored an empty DataFrame, and the is-None guard then blocked every later batch from replacing it. The sample printed a correct schema with no rows while the scan itself reported the right totals. This only became reachable with striping. Previously there was one query per reader and the sample was pinned to reader 0, so an empty leading batch was rare; with 64 chunks racing across 8 workers it is the common case whenever --limit exceeds the table's row count. Require batch.num_rows > 0 before latching. Reproduced and verified on a 100k row table with --limit 500000 and 32 chunks: before the fix, 'first 0 row(s)' shape 0x3; after, 'first 5 row(s)' shape 5x3. Regressions on --limit within the table size and on --split time both return exact row counts, and the sampled rows land on the expected chunk start. --- python/read_bench.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/python/read_bench.py b/python/read_bench.py index 6d4148f..989603a 100644 --- a/python/read_bench.py +++ b/python/read_bench.py @@ -180,10 +180,14 @@ def run_reader(idx, work, conf, counts, byts, chunks_done, errors, for batch in result.iter_arrow(): n += batch.num_rows # count rows and decoded Arrow bytes b += batch.nbytes # (buffer bytes of this batch's columns) - if sample_n > 0 and sample_out[0] is None: - # Reuse Arrow data we already received: wrap the first batch any - # worker sees as a polars DataFrame (zero-copy) and keep its head - # (one-off, no extra query, no re-scan). + # Reuse Arrow data we already received: wrap the first non-empty + # batch any worker sees as a polars DataFrame (zero-copy) and keep + # its head (one-off, no extra query, no re-scan). The num_rows test + # is load-bearing: a chunk whose offsets fall past the end of the + # table (--limit larger than the table) still yields a schema-only + # batch of 0 rows, and those chunks finish first, so latching on any + # first batch would print an empty sample with a correct schema. + if sample_n > 0 and batch.num_rows > 0 and sample_out[0] is None: with sample_lock: if sample_out[0] is None: import polars as pl From abb630cf9a812cc87f15afd61c1fab9af6395727 Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 26 Aug 2026 19:10:07 +0200 Subject: [PATCH 41/79] slippage: stream the result batch by batch instead of buffering slippage.py calls to_polars, which buffers the whole result before anything is visible. Add a variant using iter_polars, which yields one polars DataFrame per QWP batch, so rows print as they arrive. Measured on a 500k row result, warmed and run in both orders so cold start is not mistaken for streaming latency: iter_polars shows first rows at 8.5 to 18.2 ms against a 68.6 to 75.1 ms total, where to_polars shows nothing until 67.3 to 74.1 ms. Same total time, first rows roughly 8x sooner. Empty batches are skipped. A result can carry a schema-only batch of 0 rows, which is the same trap that made read_bench print an empty sample. Adds a QDB_CONF override so the script can point at a plain non-TLS instance; the wss plus ILP_TOKEN default is unchanged. Verified end to end against locally built fx_trades and market_data tables: the ASOF JOIN, the $yesterday interval, and all four computed columns resolve, and 200,000 rows arrive in 13 batches of 16,384. --- python/slippage_stream.py | 66 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 python/slippage_stream.py diff --git a/python/slippage_stream.py b/python/slippage_stream.py new file mode 100644 index 0000000..ef6ea50 --- /dev/null +++ b/python/slippage_stream.py @@ -0,0 +1,66 @@ +#!/usr/bin/env python3 +"""Slippage per fill, printed batch by batch as the result streams back. + +Same query as slippage.py, but iter_polars yields one polars DataFrame per QWP +batch instead of buffering the whole result, so rows appear as they arrive +rather than after the last one lands. + + ILP_TOKEN=... python slippage_stream.py + QDB_ADDR=host:9000 ILP_TOKEN=... python slippage_stream.py + +Needs the questdb 5.0 client plus polars and pyarrow. +""" +import os + +import questdb + +SQL = """ +SELECT + t.timestamp, + t.symbol, + t.ecn, + t.counterparty, + t.side, + t.passive, + t.price, + t.quantity, + m.best_bid, + m.best_ask, + (m.best_bid + m.best_ask) / 2 AS mid, + (m.best_ask - m.best_bid) AS spread, + CASE t.side + WHEN 'buy' THEN (t.price - (m.best_bid + m.best_ask) / 2) + / ((m.best_bid + m.best_ask) / 2) * 10000 + WHEN 'sell' THEN ((m.best_bid + m.best_ask) / 2 - t.price) + / ((m.best_bid + m.best_ask) / 2) * 10000 + END AS slippage_bps, + CASE t.side + WHEN 'buy' THEN (t.price - m.best_ask) / m.best_ask * 10000 + WHEN 'sell' THEN (m.best_bid - t.price) / m.best_bid * 10000 + END AS slippage_vs_tob_bps +FROM fx_trades t +ASOF JOIN market_data m ON (symbol) +WHERE t.timestamp IN '$yesterday' +ORDER BY t.timestamp +""" + +CONF = os.environ.get("QDB_CONF") or ( + f"wss::addr={os.environ.get('QDB_ADDR', '172.31.42.41:9000')};" + f"token={os.environ['ILP_TOKEN']};tls_verify=unsafe_off;") + +rows = 0 +with questdb.connect(CONF) as db: + for df in db.query(SQL).iter_polars(): + if df.height == 0: + continue # schema-only batch carries no rows; nothing to show yet + rows += df.height + print(f"--- batch of {df.height:,} rows | {rows:,} so far ---", flush=True) + print(df, flush=True) + +print(f"\n{rows:,} rows total") + +# Want one frame at the end instead? Batches share a Categories identity, so: +# import polars as pl +# df = pl.concat(db.query(SQL).iter_polars()) +# though that buffers everything, which is what slippage.py already does via +# to_polars(). From e24b3dba9a57afdafcbec71e4a5fa5fae86b836c Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 26 Aug 2026 19:29:09 +0200 Subject: [PATCH 42/79] slippage: drop the redundant ORDER BY from both scripts fx_trades is scanned in designated-timestamp order, so the result already arrives oldest first and ORDER BY t.timestamp restates what the scan guarantees. Measured on a 12M row result before removing it: first batch at 41.3 ms with the clause and 31.2 ms without, totals 6.93s and 7.71s, i.e. within run-to-run variance either way. No sort is planned, so the clause cost nothing. Removing it is a clarity change, not a speedup. Worth recording what the same measurements did show, since the streamer looks slow next to a 60 ms query: the 60 ms is time to first result, not to all rows. First batch lands in ~41 ms, but the ASOF JOIN is per-row work done while streaming, so pulling every row costs in proportion to row count. On 12M rows a plain scan of fx_trades took 1.90s while the joined query took 8.28s on one connection; the join is roughly 77% of the time. Batch formatting is free and iter_polars over iter_arrow adds about 20%. --- python/slippage.py | 3 ++- python/slippage_stream.py | 4 +++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/python/slippage.py b/python/slippage.py index eed9728..9ab809f 100644 --- a/python/slippage.py +++ b/python/slippage.py @@ -38,8 +38,9 @@ FROM fx_trades t ASOF JOIN market_data m ON (symbol) WHERE t.timestamp IN '$yesterday' -ORDER BY t.timestamp """ +# No ORDER BY: fx_trades is scanned in designated-timestamp order, so rows already +# arrive oldest first. Measured with and without, the plan is identical (no sort). CONF = (f"wss::addr={os.environ.get('QDB_ADDR', '172.31.42.41:9000')};" f"token={os.environ['ILP_TOKEN']};tls_verify=unsafe_off;") diff --git a/python/slippage_stream.py b/python/slippage_stream.py index ef6ea50..6b114a7 100644 --- a/python/slippage_stream.py +++ b/python/slippage_stream.py @@ -41,8 +41,10 @@ FROM fx_trades t ASOF JOIN market_data m ON (symbol) WHERE t.timestamp IN '$yesterday' -ORDER BY t.timestamp """ +# No ORDER BY: fx_trades is scanned in designated-timestamp order, so rows already +# arrive oldest first. Measured with and without, the plan is identical (no sort): +# same first-batch latency, same total. It was redundant, not costly. CONF = os.environ.get("QDB_CONF") or ( f"wss::addr={os.environ.get('QDB_ADDR', '172.31.42.41:9000')};" From 85b9287cfb450d0bf693bf29c0c212a023825d8b Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 26 Aug 2026 19:34:37 +0200 Subject: [PATCH 43/79] slippage: parallel variant merging N connections into one DataFrame Splits yesterday into N contiguous half-open time ranges, gives each its own connection, and concatenates the results in range order, which is timestamp order. No preliminary query: the day's bounds are known, so the split is arithmetic. The merge is the fiddly part. SYMBOL columns arrive as polars Categorical and each connection's result carries its own Categories identity, a per-result UUID in the questdb_symbol namespace. pl.concat rejects that outright with 'Categories name mismatch ... failed to vstack column symbol', and how=vertical_relaxed does not rescue it either, failing with 'failed to determine supertype of cat and cat'. Casting every frame's categorical columns to one shared named Categories first is what makes it work, and it beats casting to String, 188ms against 216ms over 9M rows, while keeping the dtype. pl.enable_string_cache is deprecated in polars 1.42 and is not the answer here. Verified: 12,000,000 rows over 8 connections, spanning 00:00:00 to 23:59:59.99, timestamps monotonic, symbol/ecn/counterparty/side still Categorical. No speedup measured locally: 9.17s over 8 connections against roughly 8s on one, because the container's server is already CPU-saturated by the per-row ASOF JOIN, so more connections cannot help. The win, if any, is on a remote multi-core host where a single connection is round-trip bound rather than server bound. That remains unmeasured. --- python/slippage_parallel.py | 132 ++++++++++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 python/slippage_parallel.py diff --git a/python/slippage_parallel.py b/python/slippage_parallel.py new file mode 100644 index 0000000..6e1694c --- /dev/null +++ b/python/slippage_parallel.py @@ -0,0 +1,132 @@ +#!/usr/bin/env python3 +"""Slippage per fill, pulled over N connections and merged into one DataFrame. + +Same query as slippage.py. The ASOF JOIN is per-row work done while the result +streams, so the cost scales with rows retrieved, and a single connection is also +capped by the client's socket buffer at ~426 KB per RTT. Splitting yesterday into +N equal time ranges and giving each its own connection lifts both limits, the same +way read_bench --readers does. + +Ranges are contiguous and consumed in order, so concatenating them yields exactly +the timestamp ordering a single query would have produced. No preliminary query is +needed: the day's bounds are known, so the split is pure arithmetic. + + ILP_TOKEN=... python slippage_parallel.py [readers] + QDB_CONF="ws::addr=localhost:9000;" python slippage_parallel.py 8 + +Needs the questdb 5.0 client plus polars and pyarrow. +""" +import os +import sys +import threading +import time +from datetime import datetime, timedelta, timezone + +import polars as pl +import questdb + +# SYMBOL columns arrive as polars Categorical, and each connection's result carries +# its own Categories identity (a per-result UUID in the questdb_symbol namespace). +# pl.concat refuses to mix them: "Categories name mismatch ... failed to vstack +# column 'symbol'". concat(how="vertical_relaxed") does not rescue it either, it +# fails with "failed to determine supertype of cat and cat". Casting every frame's +# categorical columns to one shared named Categories first is what makes the merge +# work, and it is cheaper than casting them to String (188 ms vs 216 ms over 9M +# rows) while keeping the dtype. +CATEGORIES = pl.Categories("slippage") + +READERS = int(sys.argv[1]) if len(sys.argv) > 1 else 8 + +SQL = """ +SELECT + t.timestamp, + t.symbol, + t.ecn, + t.counterparty, + t.side, + t.passive, + t.price, + t.quantity, + m.best_bid, + m.best_ask, + (m.best_bid + m.best_ask) / 2 AS mid, + (m.best_ask - m.best_bid) AS spread, + CASE t.side + WHEN 'buy' THEN (t.price - (m.best_bid + m.best_ask) / 2) + / ((m.best_bid + m.best_ask) / 2) * 10000 + WHEN 'sell' THEN ((m.best_bid + m.best_ask) / 2 - t.price) + / ((m.best_bid + m.best_ask) / 2) * 10000 + END AS slippage_bps, + CASE t.side + WHEN 'buy' THEN (t.price - m.best_ask) / m.best_ask * 10000 + WHEN 'sell' THEN (m.best_bid - t.price) / m.best_bid * 10000 + END AS slippage_vs_tob_bps +FROM fx_trades t +ASOF JOIN market_data m ON (symbol) +WHERE t.timestamp >= '{lo}' AND t.timestamp < '{hi}' +""" + +CONF = os.environ.get("QDB_CONF") or ( + f"wss::addr={os.environ.get('QDB_ADDR', '172.31.42.41:9000')};" + f"token={os.environ['ILP_TOKEN']};tls_verify=unsafe_off;") + + +def ranges(n): + """Yesterday UTC split into n contiguous half-open [lo, hi) spans. + + Half-open with the last span ending at today's midnight, so the spans tile + yesterday exactly: no fill is counted twice and none is dropped. This is the + same window `WHERE t.timestamp IN '$yesterday'` selects. + """ + today = datetime.now(timezone.utc).date() + start = datetime(today.year, today.month, today.day, tzinfo=timezone.utc) - timedelta(days=1) + fmt = "%Y-%m-%dT%H:%M:%S.%fZ" + edges = [start + timedelta(days=1) * i / n for i in range(n + 1)] + return [(edges[i].strftime(fmt), edges[i + 1].strftime(fmt)) for i in range(n)] + + +def main(): + spans = ranges(READERS) + frames = [None] * READERS # indexed, so order survives out-of-order completion + errors = [] + + def worker(i, lo, hi): + try: + with questdb.connect(CONF) as db: + frames[i] = db.query(SQL.format(lo=lo, hi=hi)).to_polars() + except Exception as e: # noqa: BLE001 + errors.append(f"reader {i} [{lo}, {hi}): {e}") + + t0 = time.monotonic() + threads = [threading.Thread(target=worker, args=(i, lo, hi)) + for i, (lo, hi) in enumerate(spans)] + for t in threads: + t.start() + for t in threads: + t.join() + fetched = time.monotonic() - t0 + + if errors: + for e in errors: + print(e, file=sys.stderr) + return 1 + + # Concatenate in range order, which is timestamp order. Harmonise the + # per-connection Categories identities first, or the concat raises SchemaError. + got = [f for f in frames if f is not None] + cat_cols = [c for c, dt in got[0].schema.items() if dt.base_type() == pl.Categorical] + df = pl.concat( + [f.with_columns(pl.col(c).cast(pl.Categorical(CATEGORIES)) for c in cat_cols) + for f in got] + ) + elapsed = time.monotonic() - t0 + + print(f"{df.height:,} rows over {READERS} connection(s) in {elapsed:.2f}s " + f"({df.height / elapsed:,.0f} rows/s; fetch {fetched:.2f}s, " + f"concat {elapsed - fetched:.2f}s)") + print(df) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From f4fb68b753996a548f517cc2d0ef663991c99465 Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 26 Aug 2026 19:37:13 +0200 Subject: [PATCH 44/79] slippage: parallel fetch with ordered streaming output Combines the two previous variants. slippage_stream.py streams but on one connection; slippage_parallel.py uses N connections but shows nothing until the whole result is buffered. This fetches over N connections while printing in true timestamp order, starting as soon as the first batch lands. Each worker owns a bounded queue. All workers fetch at once, but the printer drains range 0 to completion, then range 1, and so on; since the ranges are contiguous and consumed in order, the output is exactly what a single query would have produced. The bound is what stops this being a disguised buffer-everything: a worker that runs ahead blocks once its queue is full, capping peak memory at readers x QUEUE_DEPTH batches, and the backpressure reaches the server through QWP flow control. Verified on 12M rows over 8 connections: ranges printed strictly 0 through 7, exactly 92 batches each, never interleaved. 7.76s at 1,546,460 rows/s. Tradeoff worth knowing: first rows appear after 786 ms here, against roughly 40 ms for the single-connection streamer, because eight workers contend at startup. This variant optimises total throughput, not time to first paint. As with slippage_parallel, no large speedup was measurable locally because the test server is CPU-saturated by the per-row ASOF JOIN. --- python/slippage_parallel_stream.py | 134 +++++++++++++++++++++++++++++ 1 file changed, 134 insertions(+) create mode 100644 python/slippage_parallel_stream.py diff --git a/python/slippage_parallel_stream.py b/python/slippage_parallel_stream.py new file mode 100644 index 0000000..d25829b --- /dev/null +++ b/python/slippage_parallel_stream.py @@ -0,0 +1,134 @@ +#!/usr/bin/env python3 +"""Slippage per fill: fetched over N connections in parallel, printed in order as +it streams. + +Combines the two earlier variants. slippage_stream.py streams but on one +connection; slippage_parallel.py uses N connections but buffers everything into a +single DataFrame before you see anything. This does both: N connections pull +concurrently while rows print in true timestamp order, starting immediately. + +The ordering trick is that each worker owns a *bounded* queue. Workers all fetch +at once, but the printer drains range 0 to completion, then range 1, and so on. +Because the ranges are contiguous and consumed in order, what you see is exactly +the ordering a single query would have produced. The bound is what keeps this from +being a disguised "buffer everything": a worker that runs ahead blocks once its +queue is full, so peak memory is at most readers x QUEUE_DEPTH batches rather than +the whole result. That backpressure reaches the server through QWP's flow control. + + ILP_TOKEN=... python slippage_parallel_stream.py [readers] + QDB_CONF="ws::addr=localhost:9000;" python slippage_parallel_stream.py 8 + +Needs the questdb 5.0 client plus polars and pyarrow. +""" +import os +import queue +import sys +import threading +import time +from datetime import datetime, timedelta, timezone + +import polars as pl +import questdb + +READERS = int(sys.argv[1]) if len(sys.argv) > 1 else 8 +QUEUE_DEPTH = 4 # batches a worker may run ahead before it blocks +DONE = object() # sentinel: this range produced its last batch + +SQL = """ +SELECT + t.timestamp, + t.symbol, + t.ecn, + t.counterparty, + t.side, + t.passive, + t.price, + t.quantity, + m.best_bid, + m.best_ask, + (m.best_bid + m.best_ask) / 2 AS mid, + (m.best_ask - m.best_bid) AS spread, + CASE t.side + WHEN 'buy' THEN (t.price - (m.best_bid + m.best_ask) / 2) + / ((m.best_bid + m.best_ask) / 2) * 10000 + WHEN 'sell' THEN ((m.best_bid + m.best_ask) / 2 - t.price) + / ((m.best_bid + m.best_ask) / 2) * 10000 + END AS slippage_bps, + CASE t.side + WHEN 'buy' THEN (t.price - m.best_ask) / m.best_ask * 10000 + WHEN 'sell' THEN (m.best_bid - t.price) / m.best_bid * 10000 + END AS slippage_vs_tob_bps +FROM fx_trades t +ASOF JOIN market_data m ON (symbol) +WHERE t.timestamp >= '{lo}' AND t.timestamp < '{hi}' +""" + +CONF = os.environ.get("QDB_CONF") or ( + f"wss::addr={os.environ.get('QDB_ADDR', '172.31.42.41:9000')};" + f"token={os.environ['ILP_TOKEN']};tls_verify=unsafe_off;") + + +def ranges(n): + """Yesterday UTC split into n contiguous half-open [lo, hi) spans. + + The last span ends at today's midnight, so the spans tile yesterday exactly: + no fill is counted twice and none is dropped. Same window as `IN '$yesterday'`. + """ + today = datetime.now(timezone.utc).date() + start = datetime(today.year, today.month, today.day, tzinfo=timezone.utc) - timedelta(days=1) + fmt = "%Y-%m-%dT%H:%M:%S.%fZ" + edges = [start + timedelta(days=1) * i / n for i in range(n + 1)] + return [(edges[i].strftime(fmt), edges[i + 1].strftime(fmt)) for i in range(n)] + + +def worker(i, lo, hi, q, errors): + """Stream one range into its bounded queue. q.put blocks when full, which is + the backpressure that keeps peak memory bounded.""" + try: + with questdb.connect(CONF) as db: + for df in db.query(SQL.format(lo=lo, hi=hi)).iter_polars(): + if df.height: # a result can carry a schema-only batch of 0 rows + q.put(df) + except Exception as e: # noqa: BLE001 + errors.append(f"reader {i} [{lo}, {hi}): {e}") + finally: + q.put(DONE) + + +def main(): + spans = ranges(READERS) + queues = [queue.Queue(maxsize=QUEUE_DEPTH) for _ in spans] + errors = [] + + t0 = time.monotonic() + for i, (lo, hi) in enumerate(spans): + threading.Thread(target=worker, args=(i, lo, hi, queues[i], errors), + daemon=True).start() + + rows = 0 + first = None + for i, q in enumerate(queues): + while True: + df = q.get() + if df is DONE: + break + if first is None: + first = time.monotonic() - t0 + rows += df.height + print(f"--- range {i} | batch of {df.height:,} rows | {rows:,} so far ---", + flush=True) + print(df, flush=True) + + elapsed = time.monotonic() - t0 + if errors: + for e in errors: + print(e, file=sys.stderr) + return 1 + + print(f"\n{rows:,} rows over {READERS} connection(s) in {elapsed:.2f}s " + f"({rows / elapsed:,.0f} rows/s); first rows after {first * 1000:.0f} ms") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From c11faca8323cb8a23fc4014117f73ac121b0a91c Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 26 Aug 2026 19:41:26 +0200 Subject: [PATCH 45/79] slippage: stop the terminal throttling the parallel stream Because the per-worker queues are bounded, a worker blocks once its queue fills, so the printer sets the pace for every fetcher. Rendering all ~730 batches of a 12M row result emits 3.87 MB of box-drawing tables. Piped to a file that is free, which is why it never showed up in local timings; over SSH it is slow enough to stall all eight fetchers behind it. That is the asymmetry against slippage_parallel.py, which prints once at the end and so never waits on the terminal. Render the first SLIPPAGE_FULL_FRAMES batches (default 2) as full tables and one compact line per batch after that. Output drops from 3,869,222 to 53,165 bytes, 98.6% less, while the streaming progress feel is unchanged. Set the variable to 0 for progress lines only, or high to restore rendering everything. Also raise QUEUE_DEPTH from 4 to 8 so a brief stall in the printer does not immediately block the fetchers. Honest about what is measured: locally this changes nothing (9.16s rendering everything against 9.94s with the new default, i.e. noise), because a local pipe is not the bottleneck. The 98.6% byte reduction is the measured effect; the speedup is expected on a real terminal but was not reproducible here. Ordering is unaffected: ranges still print strictly 0 through 7, 92 batches each. --- python/slippage_parallel_stream.py | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/python/slippage_parallel_stream.py b/python/slippage_parallel_stream.py index d25829b..e44b991 100644 --- a/python/slippage_parallel_stream.py +++ b/python/slippage_parallel_stream.py @@ -31,9 +31,20 @@ import questdb READERS = int(sys.argv[1]) if len(sys.argv) > 1 else 8 -QUEUE_DEPTH = 4 # batches a worker may run ahead before it blocks +QUEUE_DEPTH = 8 # batches a worker may run ahead before it blocks DONE = object() # sentinel: this range produced its last batch +# How many batches to render as full tables before switching to one line each. +# This matters more than it looks. Because the queues are bounded, a worker blocks +# once its queue is full, so the printer sets the pace for every fetcher: a slow +# terminal throttles the whole pipeline. Rendering all ~730 batches of a 12M row +# result emits ~3.7 MB of box-drawing tables, which is fine piped to a file and +# painfully slow over SSH. Showing the first few in full and a compact line +# thereafter keeps the streaming feel without putting the terminal in the critical +# path. Set SLIPPAGE_FULL_FRAMES=0 for pure progress lines, or a big number to +# render everything. +FULL_FRAMES = int(os.environ.get("SLIPPAGE_FULL_FRAMES", "2")) + SQL = """ SELECT t.timestamp, @@ -106,6 +117,7 @@ def main(): daemon=True).start() rows = 0 + shown = 0 first = None for i, q in enumerate(queues): while True: @@ -117,7 +129,9 @@ def main(): rows += df.height print(f"--- range {i} | batch of {df.height:,} rows | {rows:,} so far ---", flush=True) - print(df, flush=True) + if shown < FULL_FRAMES: + print(df, flush=True) + shown += 1 elapsed = time.monotonic() - t0 if errors: From 765e4e0054d5c2cdc25e473c015c0b61962fd20b Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 17 Sep 2026 13:13:02 +0200 Subject: [PATCH 46/79] probe: stop a blocking close from stalling read failover During a primary outage the probe went silent and never picked up a replica, which looked like QWP read failover being broken. It is not. connect() walks every endpoint and treats a connect failure as non-fatal, continuing to the next, and only gives up with 'all QWP endpoints unreachable'. That message never appeared in the failing run. The stall was our own teardown. On each failed tick the probe called client.close() inline, and close() joins the client's I/O thread for up to shutdownJoinMs, a hardcoded 5s with no setter and no connection-string key (still the case in 1.3.9, whose only QwpQueryClient change since 1.3.6 is PEM CA support, so upgrading the client does not help). A dead connection hits that timeout every time, so the loop burned about 5s per tick before it could even try to reconnect, while a live replica was available to serve. That matches the roughly 7s silent gap in the run. Dispose the dead client on a daemon reaper thread so the next tick reconnects immediately. Safe to hand off: execute() has already returned, no batch handler is running, and the next tick builds a separate instance sharing no state. Also make the read path visible, since the old code reported the loss once and then printed nothing, leaving 'reconnected with no rows' and 'never reconnected' indistinguishable: - every failed attempt prints its number, duration and reason - recovery prints how many attempts it took and how long connect took, which separates failover being broken from failover being slow behind a dead endpoint burning the full connect timeout - connected-but-no-rows says so instead of printing nothing Builds clean with mvn -o package against 1.3.6-SNAPSHOT. Not yet exercised against a real failover; the timings it now prints are what will confirm the diagnosis. --- .../com/example/sender/CsvParallelSender.java | 68 ++++++++++++++++--- 1 file changed, 59 insertions(+), 9 deletions(-) diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index 8dbc9bb..a75c2f3 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -743,6 +743,17 @@ private static Thread startProbe(SenderCfg cfg, long intervalMs) { try { final long[] latest = {Long.MIN_VALUE}; final boolean[] wasDown = {false}; + // Consecutive failed ticks. The probe exists to show read HA, so an outage has + // to stay VISIBLE: reporting the loss once and then going quiet is + // indistinguishable from the probe thread having died, and it also hides WHY + // the reader is not picking up a replica. target defaults to any, so a replica + // is an acceptable endpoint and connect() walks the address list; if a read is + // not being served during a primary outage we need the per-attempt reason. + final long[] downTicks = {0L}; + // Wall time the last successful connect() took. A dead endpoint that blackholes + // SYN burns the whole connect_timeout before the walk reaches the next address, + // so this is how you tell "failover is broken" from "failover is just slow". + final long[] connectMs = {0L}; // Set by the probe handler when the server reports a transport-level failure. // Transport failures always carry STATUS_INTERNAL_ERROR; a SQL-level error // (e.g. "table does not exist" before ingestion creates it) carries a different @@ -843,20 +854,27 @@ public void onFailoverReset(QwpServerInfo info) { targetRole[0] = null; statusDiag[0] = null; transportFailed[0] = false; + final long tickStart = System.nanoTime(); try { // (Re)build on first pass and after every connection loss. A server that // is down at startup just leaves client == null and retries next tick. if (client == null) { + final long connectStart = System.nanoTime(); client = QwpQueryClient.fromConfig(queryClientConfig(cfg)); client.connect(); + connectMs[0] = (System.nanoTime() - connectStart) / 1_000_000L; final QwpServerInfo info = client.getServerInfo(); - final String what = wasDown[0] ? "connection restored" : "connected"; + final String what = wasDown[0] + ? "connection restored after " + downTicks[0] + " failed attempt(s)" + : "connected"; + downTicks[0] = 0; if (info != null) { - System.out.printf("[query client] %s, serving node=%s role=%s zone=%s cluster=%s%n", - what, orNone(info.getNodeId()), QwpServerInfo.roleName(info.getRole()), + System.out.printf("[query client] %s in %dms, serving node=%s role=%s zone=%s cluster=%s%n", + what, connectMs[0], orNone(info.getNodeId()), + QwpServerInfo.roleName(info.getRole()), orNone(info.getZoneId()), orNone(info.getClusterId())); } else { - System.out.printf("[query client] %s%n", what); + System.out.printf("[query client] %s in %dms%n", what, connectMs[0]); } wasDown[0] = false; } @@ -910,21 +928,53 @@ what, orNone(info.getNodeId()), QwpServerInfo.roleName(info.getRole()), System.out.println("[probe] live role unavailable: " + statusDiag[0]); } } + } else if (!SUBMIT_COMPLETE) { + // Connected and the query succeeded, but no row came back. Say so + // rather than printing nothing, which would look identical to the + // read being down. + System.out.println("[probe] read OK but query returned no rows yet"); } } catch (Exception e) { + downTicks[0]++; if (!wasDown[0]) { System.out.printf("[query client] connection lost (%s), will retry%n", String.valueOf(e.getMessage())); wasDown[0] = true; } + // Print EVERY failed attempt, not just the first. Reads are supposed to + // survive a primary outage by moving to a replica, so each tick the read + // is still down is the thing the demo needs to show, with the reason. + if (!SUBMIT_COMPLETE) { + System.out.printf("[probe] read DOWN, attempt %d failed after %dms: %s%n", + downTicks[0], (System.nanoTime() - tickStart) / 1_000_000L, + String.valueOf(e.getMessage())); + } // Drop the latched client; the next tick builds a fresh one. + // + // Dispose it on a throwaway thread rather than inline. close() joins the + // client's I/O thread for up to shutdownJoinMs, which is a hardcoded 5s + // (private, no setter and no connection-string key, still true in 1.3.9), + // and a dead connection reliably hits that full timeout. Closing inline + // therefore stalls this loop ~5s per failed tick, so the reader sits idle + // while a live replica is right there waiting to serve -- which is exactly + // why read failover looked broken. connect() itself walks every endpoint + // and only gives up with "all QWP endpoints unreachable", so the failover + // was never the problem; our teardown was. + // + // Safe to hand off: execute() has already returned, no handler is running, + // and the next tick builds a SEPARATE instance that shares no state. if (client != null) { - try { - client.close(); - } catch (Exception ignored) { - // A dead connection often fails to close cleanly; nothing to do. - } + final QwpQueryClient dead = client; client = null; + final Thread reaper = new Thread(() -> { + try { + dead.close(); + } catch (Exception ignored) { + // A dead connection often fails to close cleanly; nothing to do. + } + }, "qwp-probe-reaper"); + reaper.setDaemon(true); + reaper.start(); } } Thread.sleep(intervalMs); From 0131725eacb34383e50a1a85166081ebacb9acb4 Mon Sep 17 00:00:00 2001 From: javier Date: Tue, 29 Sep 2026 12:01:10 +0200 Subject: [PATCH 47/79] probe: keep reporting through the store-and-forward drain The probe and progress lines went silent one line after "Sender 0 finished sending", because SUBMIT_COMPLETE gated them and it flips when the last row is SUBMITTED, not when it is durable. A real run went quiet at acknowledged=63,900,000 of 100,000,000, so 36.1M rows were still buffered in store-and-forward waiting for a primary, and none of that was visible. That tail was originally treated as per-second spam worth suppressing. For a durability demo it is the opposite: the drain is the whole point, and hiding it also hid the fact that reads were still being served off a replica the entire time, with the probe timestamp advancing as the replica caught up. Split the two meanings: - SUBMIT_COMPLETE still marks the append-phase boundary for timing, and no longer silences anything. - New ALL_ACKED, set when the ack watermark reaches totalEvents, is what gates the reporter's continue and the three probe prints. Net effect: the run now reports every second until the acks actually catch up, showing the replica serving reads and store-and-forward replaying once a primary returns. If no primary comes back, it keeps reporting rather than going dark, which is the intended behaviour here. This was a gap in the previous probe-visibility change: the new prints were added behind the pre-existing !SUBMIT_COMPLETE guard and inherited its silence. The failover fix itself worked, as the same log shows the reader hopping eu-west-1 -> eu-north-1 -> PRIMARY -> REPLICA and continuing to serve. Builds clean with mvn -o package against 1.3.6-SNAPSHOT. Not yet exercised against a real failover. --- .../com/example/sender/CsvParallelSender.java | 27 +++++++++++++------ 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index a75c2f3..21df9e1 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -84,9 +84,17 @@ public class CsvParallelSender { // Rows the server has acknowledged, per worker (summed by the reporter). Fed from the QWP // ack watermark (getAckedFsn) - the real committed count, with no extra query round-trips. private static AtomicLong[] ACKED_ROWS = new AtomicLong[0]; - // Set once all rows are submitted: silences the per-second progress + probe lines so the - // commit-drain tail is quiet and only the final summary line is printed. + // Set once all rows are submitted. Marks the append-phase boundary for timing; it does + // NOT silence output, because submission finishing is not the end of the story. private static volatile boolean SUBMIT_COMPLETE = false; + // Set once the ack watermark has caught up with every submitted row. THIS is what + // silences the per-second progress + probe lines, so only the final summary follows. + // Gating on SUBMIT_COMPLETE instead was wrong: with store-and-forward, an outage can + // leave tens of millions of rows buffered after the last submit, and that drain is + // exactly what a durability demo needs to show. A real run went quiet at + // acknowledged=63.9M of 100M, hiding both the remaining acks and the fact that reads + // were still being served off a replica throughout. + private static volatile boolean ALL_ACKED = false; public static void main(String[] args) throws Exception { // Parse CLI flags @@ -248,10 +256,13 @@ public static void main(String[] args) throws Exception { } if (ack >= totalEvents) { commitDoneNanos.compareAndSet(0, System.nanoTime()); + ALL_ACKED = true; } - // Once everything is submitted, go quiet - no per-second spam while the client - // drains. The final "All workers completed ..." summary is the single last line. - if (SUBMIT_COMPLETE) { + // Go quiet only once the ACKS have caught up, not merely once submission has. + // The drain after the last submit is where store-and-forward earns its keep, so + // keep reporting it. The final "All workers completed ..." summary is still the + // single last line. + if (ALL_ACKED) { continue; } System.out.printf("[progress] submitted=%,d (+%,d/s) | acknowledged=%,d (+%,d/s)%n", @@ -916,7 +927,7 @@ what, connectMs[0], orNone(info.getNodeId()), served = " served by role=" + handshake + " node=" + node + " zone=" + zone + " (handshake role; live 'switch status' unavailable, may be stale)"; } - if (!SUBMIT_COMPLETE) { + if (!ALL_ACKED) { System.out.printf("[probe] latest trades timestamp = %s (raw=%d)%s%n", ts, latest[0], served); } @@ -928,7 +939,7 @@ what, connectMs[0], orNone(info.getNodeId()), System.out.println("[probe] live role unavailable: " + statusDiag[0]); } } - } else if (!SUBMIT_COMPLETE) { + } else if (!ALL_ACKED) { // Connected and the query succeeded, but no row came back. Say so // rather than printing nothing, which would look identical to the // read being down. @@ -944,7 +955,7 @@ what, connectMs[0], orNone(info.getNodeId()), // Print EVERY failed attempt, not just the first. Reads are supposed to // survive a primary outage by moving to a replica, so each tick the read // is still down is the thing the demo needs to show, with the reason. - if (!SUBMIT_COMPLETE) { + if (!ALL_ACKED) { System.out.printf("[probe] read DOWN, attempt %d failed after %dms: %s%n", downTicks[0], (System.nanoTime() - tickStart) / 1_000_000L, String.valueOf(e.getMessage())); From cfab0ae19c96c75b53db68f135dc700c5d27291f Mon Sep 17 00:00:00 2001 From: javier Date: Tue, 29 Sep 2026 12:45:05 +0200 Subject: [PATCH 48/79] probe: make reads and store-and-forward replay visible during an outage Verified against a live Enterprise cluster (one primary, two read replicas) by stopping the primary mid-run, not just compiled. Reads never go quiet. The probe's prints were gated on !SUBMIT_COMPLETE and later !ALL_ACKED, so they were silenced exactly when a failover demo needs them. They are now unconditional, and a failed read attempt reports its number, duration and reason each tick instead of printing nothing after the first. Confirmed live: while the writer was looping on "endpoint 3.253.59.216:9000 failed (errno=61)", the probe kept serving every second with "served by role=REPLICA zone=eu-west-1", and the client's own callback logged "failed over -> now serving role=REPLICA". Attempts land ~45ms apart rather than stalling ~5s each, which the earlier async-close reaper fixed. The acked counter no longer claims completion early. ACKED_ROWS was force-set to `sent` before the try-with-resources close(), and close() is where the bulk of the durability wait happens, so the reporter believed the run was finished and gated everything off. The counter and the "finished sending" line now come after close(). Store-and-forward replay is announced. The client exposes no replay event (SenderConnectionEvent.Kind has only CONNECTED, DISCONNECTED, RECONNECTED, FAILED_OVER, ENDPOINT_ATTEMPT_FAILED, ALL_ENDPOINTS_UNREACHABLE, AUTH_FAILED) and bufferView() is just the unflushed local buffer, so the backlog is measured from the sf directory instead: - on startup, a non-empty sf root prints how much a previous run left unacked; - each tick prints the backlog still on disk plus its drain rate; - on reconnect or failover, the note reports the unacked rows owed. Before this, a restart replay showed only seconds of submitted=0 with acknowledged pinned at zero, because the backlog belongs to the previous process's slot and never passes through ACKED_ROWS. Confirmed live: "found 1.65 GiB of pending frames ... REPLAYING", then "backlog on disk: 1.64 GiB (-4.0 MiB/s)" down to empty, exit 0. Known limits, unfixed here and next on the list: - close() enforces its own closeFlushTimeoutMillis (default 60s) independently of --retry-timeout, so a longer outage dies with "close() drain timed out ... data may be lost" while holding 6 minutes of unused reconnect budget. The data is not actually lost; both such runs replayed fully on restart. - acknowledged only moves in 100,000-row steps: fsnToRows gets an entry every commitEveryRows while FSNs advance about 11x finer (targetFsn=10999 for 100M rows). - the summary divides --total-events by elapsed, so a replay-dominated run reports nonsense throughput (1,000 events over 68s read as "15 rows/s"). --- .../com/example/sender/CsvParallelSender.java | 114 ++++++++++++++++-- 1 file changed, 104 insertions(+), 10 deletions(-) diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index 21df9e1..6dcf093 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -24,6 +24,7 @@ import com.opencsv.CSVReader; +import java.io.IOException; import java.io.InputStream; import java.io.InputStreamReader; import java.io.BufferedReader; @@ -227,6 +228,18 @@ public static void main(String[] args) throws Exception { ACKED_ROWS[i] = new AtomicLong(); } + // A non-empty store-and-forward root at startup means a previous run died holding + // unacked frames. The client replays them before (and alongside) new sends, and that + // replay is otherwise invisible: it shows up only as seconds of submitted=0 with the + // acked counter pinned at zero, because the backlog belongs to the previous process's + // slot and never passes through ACKED_ROWS. + final long sfStartBytes = sfDirBytes(storeForwardDir); + if (sfStartBytes > 0) { + System.out.printf("[store-and-forward] found %s of pending frames in %s; " + + "REPLAYING previous run's unacked data before new sends%n", + humanBytes(sfStartBytes), storeForwardDir); + } + // Records (to ~1s resolution) when all rows were submitted and when all were acknowledged, // so the summary can separate the submit phase from the commit-drain tail. final AtomicLong appendDoneNanos = new AtomicLong(0); @@ -236,6 +249,7 @@ public static void main(String[] args) throws Exception { // run ahead of the server) and acknowledged (rows the server has actually committed). The // gap between them is the buffered backlog; during the drain, submitted is flat while // acknowledged keeps climbing (real work), so there is no misleading "0 rows/s". + final long[] lastSfBytes = {sfStartBytes}; final Thread reporter = new Thread(() -> { long lastSub = 0; long lastAck = 0; @@ -269,6 +283,18 @@ public static void main(String[] args) throws Exception { sub, sub - lastSub, ack, ack - lastAck); lastSub = sub; lastAck = ack; + // Watch the spill shrink. Only reported while there is something there, so a + // healthy run with no backlog stays quiet. + final long sfNow = sfDirBytes(storeForwardDir); + if (sfNow > 0) { + System.out.printf("[store-and-forward] backlog on disk: %s%s%n", + humanBytes(sfNow), + lastSfBytes[0] > 0 && sfNow != lastSfBytes[0] + ? String.format(" (%+.1f MiB/s)", + (sfNow - lastSfBytes[0]) / (1024.0 * 1024.0)) + : ""); + } + lastSfBytes[0] = sfNow; } }); reporter.setDaemon(true); @@ -436,15 +462,82 @@ private static void runWorker( while (!sender.awaitAckedFsn(finalSeq, 200L) && System.nanoTime() < drainDeadline) { ACKED_ROWS[senderId].set(ackedRows(sender, fsnToRows)); } + // Re-read once on exit. awaitAckedFsn returning true breaks the loop BEFORE the + // body runs again, so without this the counter keeps whatever the last iteration + // left and under-reports for the rest of the run (observed: stuck at 15,300,000 + // of 30,000,000 while the server had in fact acked everything). + ACKED_ROWS[senderId].set(ackedRows(sender, fsnToRows)); } else { sender.flush(); } - ACKED_ROWS[senderId].set(sent); - System.out.printf("Sender %d finished sending %d events%n", senderId, sent); + // Deliberately NOT setting ACKED_ROWS to `sent` here, and deliberately not printing + // "finished" yet. The try-with-resources close() below is where the bulk of the + // durability wait actually happens: measured locally, awaitAckedFsn above returned + // almost immediately while close() took 12.8s of a 19.8s run. Claiming every row was + // acked before that wait made the reporter believe the run was done, which silenced + // both the progress line and the read probe for the whole drain -- the exact window a + // durability demo needs to show. } catch (Exception e) { System.err.printf("Sender %d got error: %s%s%n", senderId, e.toString(), upgradeHint(e)); throw new RuntimeException(e); } + // Reached only after close() returned, i.e. after the real drain. Now the full count is + // true rather than aspirational. + ACKED_ROWS[senderId].set(sent); + System.out.printf("Sender %d finished sending %d events%n", senderId, sent); + } + + // Bytes currently spilled under the store-and-forward root. The client exposes no replay + // event and no backlog metric, so the filesystem is the only observable signal that a + // restart is replaying a previous run's unacked frames. Cheap enough to sample once a + // second: a few hundred segment files. + private static long sfDirBytes(String dir) { + final Path root = Path.of(dir); + if (!Files.exists(root)) { + return 0L; + } + try (java.util.stream.Stream walk = Files.walk(root)) { + return walk.filter(Files::isRegularFile).mapToLong(f -> { + try { + return Files.size(f); + } catch (IOException ignored) { + return 0L; // segment reclaimed mid-walk; it simply no longer counts + } + }).sum(); + } catch (IOException e) { + return 0L; + } + } + + private static String humanBytes(long bytes) { + if (bytes < 1024L) { + return bytes + " B"; + } + if (bytes < 1024L * 1024L) { + return String.format("%.1f KiB", bytes / 1024.0); + } + if (bytes < 1024L * 1024L * 1024L) { + return String.format("%.1f MiB", bytes / (1024.0 * 1024.0)); + } + return String.format("%.2f GiB", bytes / (1024.0 * 1024.0 * 1024.0)); + } + + // Unacked rows still owed to the server, i.e. what store-and-forward has to replay. The client + // exposes no replay event: SenderConnectionEvent.Kind carries only CONNECTED, DISCONNECTED, + // RECONNECTED, FAILED_OVER, ENDPOINT_ATTEMPT_FAILED, ALL_ENDPOINTS_UNREACHABLE and AUTH_FAILED, + // and bufferView() is just the unflushed local buffer, not the spilled backlog. So derive the + // figure from our own counters and announce it on reconnect; the acknowledged counter then + // climbing in the progress lines IS the replay. + private static String backlogNote() { + long acked = 0; + for (AtomicLong al : ACKED_ROWS) { + acked += al.get(); + } + final long backlog = TOTAL_SENT.get() - acked; + if (backlog <= 0) { + return ""; + } + return String.format(" -- REPLAYING store-and-forward backlog of %,d row(s)", backlog); } // Rows the server has acknowledged for this sender: map the ack watermark (getAckedFsn) back to @@ -630,10 +723,11 @@ private static Sender buildSender(SenderCfg cfg, int workerId) { msg = "connected to " + host; break; case RECONNECTED: - msg = "reconnected to " + host; + msg = "reconnected to " + host + backlogNote(); break; case FAILED_OVER: - msg = "failed over " + event.getPreviousHost() + ":" + event.getPreviousPort() + " -> " + host; + msg = "failed over " + event.getPreviousHost() + ":" + event.getPreviousPort() + + " -> " + host + backlogNote(); break; case AUTH_FAILED: msg = "auth failed for " + host; @@ -927,10 +1021,10 @@ what, connectMs[0], orNone(info.getNodeId()), served = " served by role=" + handshake + " node=" + node + " zone=" + zone + " (handshake role; live 'switch status' unavailable, may be stale)"; } - if (!ALL_ACKED) { - System.out.printf("[probe] latest trades timestamp = %s (raw=%d)%s%n", - ts, latest[0], served); - } + // Unconditional: a read that is still being served is exactly what a + // failover demo has to keep showing, including after submission ends. + System.out.printf("[probe] latest trades timestamp = %s (raw=%d)%s%n", + ts, latest[0], served); // Explain a missing live role at most once per 30s so it does not spam. if (currentRole[0] == null && statusDiag[0] != null) { final long now = System.currentTimeMillis(); @@ -939,7 +1033,7 @@ what, connectMs[0], orNone(info.getNodeId()), System.out.println("[probe] live role unavailable: " + statusDiag[0]); } } - } else if (!ALL_ACKED) { + } else { // Connected and the query succeeded, but no row came back. Say so // rather than printing nothing, which would look identical to the // read being down. @@ -955,7 +1049,7 @@ what, connectMs[0], orNone(info.getNodeId()), // Print EVERY failed attempt, not just the first. Reads are supposed to // survive a primary outage by moving to a replica, so each tick the read // is still down is the thing the demo needs to show, with the reason. - if (!ALL_ACKED) { + { System.out.printf("[probe] read DOWN, attempt %d failed after %dms: %s%n", downTicks[0], (System.nanoTime() - tickStart) / 1_000_000L, String.valueOf(e.getMessage())); From e40f9edb25087cf8768765b50d2347368ab6a7de Mon Sep 17 00:00:00 2001 From: javier Date: Tue, 29 Sep 2026 13:28:14 +0200 Subject: [PATCH 49/79] sender: own the drain, and report only what is measured Verified end to end against a live Enterprise cluster (one primary, two read replicas) across several 100M-row runs with the primary stopped and restarted mid-flight. Take the drain back from close(). close() runs an implicit drain bounded by close_flush_timeout_millis, default 60s and independent of --retry-timeout. Two 100M runs died on it with "close() drain timed out ... data may be lost" while still holding minutes of unused reconnect budget, and the data was not lost: both backlogs replayed fully on the next start. Worse, while blocked inside close() nothing could refresh any counter or the probe, so the most interesting window of a durability demo was invisible. That is also why several earlier attempts to fix the acked counter had no effect: the waiting was happening somewhere unobservable. Sender.drain(timeoutMillis) is the documented way to take it back, "the same shape as the implicit drain close() runs, with the caller controlling the timeout per call-site". Polling it in 1s slices against --retry-timeout keeps the reporter and probe ticking for the whole drain. Measured: 98.6s and 101.8s drains completing where the old 60s ceiling killed them, and the reporter covering 119 of 121 seconds instead of 26. close_flush_timeout_millis is now set low (5s, --close-flush-timeout-ms) rather than aligned to --retry-timeout. Aligning was considered and rejected: with store-and-forward the remainder is "recovered by the next sender in SF mode", so a long close buys no durability and only delays shutdown. Budget exhaustion now prints what actually happens instead of raising. Report only measured quantities. Deleted ACKED_ROWS, the fsnToRows side map, and ackedRows(). Translating the frame watermark back into rows produced six distinct wrong answers: premature saturation before close(), 100,000-row quantisation, a gate that silenced the reporter 7s into a 98s drain, a 0 that should have been "unknown" on a resumed slot, a -1 sentinel poisoning the map so a 19,579-frame run reported 100,000,000 rows acked, and that same sentinel printing "/-1" as a denominator. Each looked like a different bug. What remains is three measured values: rows submitted (our own count), frames acked=A/B (getAckedFsn against flushAndGetSequence, which is what the protocol acknowledges and what the client's own errors quote), and the spill directory's size on disk. The reconnect note and the budget message quote bytes for the same reason. Harden the reporter. Files.walk wraps a mid-iteration failure in UncheckedIOException, a RuntimeException that slips past catch (IOException); store-and-forward reclaims segments continuously, so this killed the reporter thread outright the moment an outage began and spill writes surged. Caught now, and the whole tick body is guarded so no single failure can ever silence progress again: a dead reporter is indistinguishable from a stalled sender. The tick moved into reportTick() rather than nesting further inside the lambda. Confirmed live: reads stay on a replica throughout an outage; frames freeze exactly while no primary accepts writes; the spill holds, then reports "REPLAYING store-and-forward backlog of 4.86 GiB" on reconnect and drains at -28 MiB/s while frames resume climbing; zero reporter failures across the spill surges that previously killed it. Not exercised: the drain-budget message needs an outage longer than --retry-timeout (6 minutes by default), which will not occur in a demo. --- .../com/example/sender/CsvParallelSender.java | 264 +++++++++++------- 1 file changed, 162 insertions(+), 102 deletions(-) diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index 6dcf093..3dd0fef 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -65,6 +65,14 @@ public class CsvParallelSender { private static final int DEFAULT_BATCHES_PER_TRANSACTION = 10; // Probe (QWP only): poll the latest ingested timestamp on an interval, 0 disables. private static final long DEFAULT_PROBE_INTERVAL_MS = 1000L; + // Slice length for the polled final drain. Short enough that the once-per-second reporter + // and probe keep ticking through it, long enough not to spin. + private static final long DRAIN_SLICE_MS = 1000L; + // close()'s own implicit drain budget. Deliberately LOW, not aligned to --retry-timeout: + // the polled drain above has already waited, and anything still unacked is recovered by the + // next sender in store-and-forward mode, so blocking again in close() buys no durability + // and only delays shutdown. Override with --close-flush-timeout-ms. + private static final long DEFAULT_CLOSE_FLUSH_TIMEOUT_MS = 5_000L; private static final String PROBE_QUERY = "select timestamp from trades limit -1"; // Enterprise lifecycle status of whichever node the query client is currently connected to. // Returns the LIVE role (columns like current_role / target_role), unlike the QWP handshake @@ -82,20 +90,15 @@ public class CsvParallelSender { // Rows sent (client-side) across all workers, for the once-per-second progress reporter. private static final AtomicLong TOTAL_SENT = new AtomicLong(); - // Rows the server has acknowledged, per worker (summed by the reporter). Fed from the QWP - // ack watermark (getAckedFsn) - the real committed count, with no extra query round-trips. - private static AtomicLong[] ACKED_ROWS = new AtomicLong[0]; + // The ack watermark and the highest published frame sequence, per worker. These come + // straight from the client (getAckedFsn / flushAndGetSequence) and are what the protocol + // actually acknowledges, which is also what the client's own errors quote + // ("targetFsn=10999, ackedFsn=5455"). No row-level ack figure is tracked: see reportTick(). + private static AtomicLong[] ACKED_FSN = new AtomicLong[0]; + private static AtomicLong[] PUBLISHED_FSN = new AtomicLong[0]; // Set once all rows are submitted. Marks the append-phase boundary for timing; it does // NOT silence output, because submission finishing is not the end of the story. private static volatile boolean SUBMIT_COMPLETE = false; - // Set once the ack watermark has caught up with every submitted row. THIS is what - // silences the per-second progress + probe lines, so only the final summary follows. - // Gating on SUBMIT_COMPLETE instead was wrong: with store-and-forward, an outage can - // leave tens of millions of rows buffered after the last submit, and that drain is - // exactly what a durability demo needs to show. A real run went quiet at - // acknowledged=63.9M of 100M, hiding both the remaining acks and the fact that reads - // were still being served off a replica throughout. - private static volatile boolean ALL_ACKED = false; public static void main(String[] args) throws Exception { // Parse CLI flags @@ -121,6 +124,8 @@ public static void main(String[] args) throws Exception { final int batchSize = Integer.parseInt(a.getOrDefault("--batch-size", String.valueOf(DEFAULT_BATCH_SIZE))); final int batchesPerTransaction = Integer.parseInt(a.getOrDefault("--batches-per-transaction", String.valueOf(DEFAULT_BATCHES_PER_TRANSACTION))); + final long closeFlushTimeoutMs = Long.parseLong(a.getOrDefault("--close-flush-timeout-ms", + String.valueOf(DEFAULT_CLOSE_FLUSH_TIMEOUT_MS))); final long probeIntervalMs = Long.parseLong(a.getOrDefault("--probe-interval-ms", String.valueOf(DEFAULT_PROBE_INTERVAL_MS))); // Enterprise-only: request durable acks (data durably uploaded). OSS servers do not @@ -188,7 +193,7 @@ public static void main(String[] args) throws Exception { final SenderCfg cfg = new SenderCfg(protocol, addrsCsv, token, username, password, retryTimeout, senderIdBase, storeForwardDir, batchSize, batchesPerTransaction, numSenders, enterprise, zone, - connectTimeoutMs, rate); + connectTimeoutMs, closeFlushTimeoutMs, rate); final String pacing = rate > 0 ? "rate=" + rate + " rows/s (aggregate across " + numSenders + " workers)" @@ -223,16 +228,18 @@ public static void main(String[] args) throws Exception { // Per-worker acknowledged-row counters, summed by the reporter. Fed from the QWP ack // watermark (getAckedFsn) - the real committed progress, with no extra query round-trips. - ACKED_ROWS = new AtomicLong[numSenders]; + ACKED_FSN = new AtomicLong[numSenders]; + PUBLISHED_FSN = new AtomicLong[numSenders]; for (int i = 0; i < numSenders; i++) { - ACKED_ROWS[i] = new AtomicLong(); + ACKED_FSN[i] = new AtomicLong(); + PUBLISHED_FSN[i] = new AtomicLong(); } // A non-empty store-and-forward root at startup means a previous run died holding // unacked frames. The client replays them before (and alongside) new sends, and that // replay is otherwise invisible: it shows up only as seconds of submitted=0 with the // acked counter pinned at zero, because the backlog belongs to the previous process's - // slot and never passes through ACKED_ROWS. + // slot, so this process's own counters cannot describe it. final long sfStartBytes = sfDirBytes(storeForwardDir); if (sfStartBytes > 0) { System.out.printf("[store-and-forward] found %s of pending frames in %s; " @@ -252,49 +259,22 @@ public static void main(String[] args) throws Exception { final long[] lastSfBytes = {sfStartBytes}; final Thread reporter = new Thread(() -> { long lastSub = 0; - long lastAck = 0; while (true) { try { Thread.sleep(1000); } catch (InterruptedException ie) { return; } - final long sub = TOTAL_SENT.get(); - long ack = 0; - for (AtomicLong al : ACKED_ROWS) { - ack += al.get(); - } - if (sub >= totalEvents) { - appendDoneNanos.compareAndSet(0, System.nanoTime()); - SUBMIT_COMPLETE = true; - } - if (ack >= totalEvents) { - commitDoneNanos.compareAndSet(0, System.nanoTime()); - ALL_ACKED = true; - } - // Go quiet only once the ACKS have caught up, not merely once submission has. - // The drain after the last submit is where store-and-forward earns its keep, so - // keep reporting it. The final "All workers completed ..." summary is still the - // single last line. - if (ALL_ACKED) { - continue; - } - System.out.printf("[progress] submitted=%,d (+%,d/s) | acknowledged=%,d (+%,d/s)%n", - sub, sub - lastSub, ack, ack - lastAck); - lastSub = sub; - lastAck = ack; - // Watch the spill shrink. Only reported while there is something there, so a - // healthy run with no backlog stays quiet. - final long sfNow = sfDirBytes(storeForwardDir); - if (sfNow > 0) { - System.out.printf("[store-and-forward] backlog on disk: %s%s%n", - humanBytes(sfNow), - lastSfBytes[0] > 0 && sfNow != lastSfBytes[0] - ? String.format(" (%+.1f MiB/s)", - (sfNow - lastSfBytes[0]) / (1024.0 * 1024.0)) - : ""); + try { + reportTick(totalEvents, storeForwardDir, appendDoneNanos, lastSfBytes, lastSub); + lastSub = TOTAL_SENT.get(); + } catch (RuntimeException e) { + // Never let one bad tick silence progress for the rest of the run. A dead + // reporter looks exactly like a stalled sender, and that ambiguity cost real + // debugging time when an UncheckedIOException from the spill walk killed + // this thread mid-outage. + System.out.printf("[progress] reporter tick failed: %s%n", e); } - lastSfBytes[0] = sfNow; } }); reporter.setDaemon(true); @@ -381,9 +361,6 @@ private static void runWorker( : 0.0; final boolean rateLimited = intervalNanos > 0.0; final long paceStartNanos = System.nanoTime(); - // QWP ack tracking: map each flush sequence -> cumulative rows, so getAckedFsn() can be - // resolved to acknowledged rows. Only QWP carries frame sequence numbers and acks. - final java.util.TreeMap fsnToRows = isQwp ? new java.util.TreeMap<>() : null; try ( Sender sender = buildSender(cfg, senderId)) { //( Sender sender = Sender.fromConfig(conf)) { final int n = rows.size(); for (long i = 0; i < totalEvents; i++) { @@ -421,8 +398,13 @@ private static void runWorker( // worker's acknowledged-row count from the ack watermark (no extra round-trip). if (commitEveryRows > 0 && sent % commitEveryRows == 0) { if (isQwp) { - fsnToRows.put(sender.flushAndGetSequence(), sent); - ACKED_ROWS[senderId].set(ackedRows(sender, fsnToRows)); + // flushAndGetSequence() returns -1 when there was nothing left to + // flush; storing that would clobber the real published frame. + final long fsn = sender.flushAndGetSequence(); + if (fsn >= 0) { + PUBLISHED_FSN[senderId].set(fsn); + } + ACKED_FSN[senderId].set(sender.getAckedFsn()); } else { sender.flush(); } @@ -452,41 +434,114 @@ private static void runWorker( } } } - // Final flush. For QWP, wait for the server to acknowledge everything so the acked - // counter climbs to the full count during the drain (the real committed progress). The - // await loop drives the connection and refreshes the counter; capped to avoid a hang. + // Final drain, done HERE rather than inside close(), and in slices so it stays + // observable. + // + // close() runs its own implicit drain bounded by close_flush_timeout_millis + // (default 60s), which is independent of --retry-timeout: a longer outage used to + // die with "close() drain timed out ... data may be lost" while still holding + // minutes of unused reconnect budget. Worse, while blocked inside close() nothing + // can refresh the acked counter or the probe, so the most interesting window of a + // durability demo was invisible. + // + // Sender.drain(timeoutMillis) is the documented way to take that wait back: "the + // same shape as the implicit drain close() runs, with the caller controlling the + // timeout per call-site". Polling it in ~1s slices lets the reporter and the probe + // keep working, and the counter climb, for the whole drain. The budget is + // --retry-timeout, so there is now ONE timeout governing how long we wait for the + // server rather than two that disagree. if (isQwp) { final long finalSeq = sender.flushAndGetSequence(); - fsnToRows.put(finalSeq, sent); - final long drainDeadline = System.nanoTime() + 600_000_000_000L; // 10 min cap - while (!sender.awaitAckedFsn(finalSeq, 200L) && System.nanoTime() < drainDeadline) { - ACKED_ROWS[senderId].set(ackedRows(sender, fsnToRows)); + if (finalSeq >= 0) { + PUBLISHED_FSN[senderId].set(finalSeq); + } + final long drainDeadline = System.nanoTime() + cfg.retryTimeout * 1_000_000L; + boolean drained = false; + while (System.nanoTime() < drainDeadline) { + drained = sender.drain(DRAIN_SLICE_MS); + ACKED_FSN[senderId].set(sender.getAckedFsn()); + if (drained) { + break; + } + } + if (!drained) { + // Not fatal: with store-and-forward the unacked remainder stays on disk and + // the next sender replays it (Sender.closeFlushTimeoutMillis javadoc: + // "recovered by the next sender in SF mode"). Say so plainly instead of + // letting close() raise "data may be lost" for data that is not lost. + // + System.out.printf("Sender %d drain budget of %,dms elapsed; frames %,d/%,d " + + "acked, %s still spilled. It stays in %s and replays on " + + "next start%n", + senderId, cfg.retryTimeout, + ACKED_FSN[senderId].get(), PUBLISHED_FSN[senderId].get(), + humanBytes(sfDirBytes(cfg.storeForwardDir)), + cfg.storeForwardDir + "/" + senderId); } - // Re-read once on exit. awaitAckedFsn returning true breaks the loop BEFORE the - // body runs again, so without this the counter keeps whatever the last iteration - // left and under-reports for the rest of the run (observed: stuck at 15,300,000 - // of 30,000,000 while the server had in fact acked everything). - ACKED_ROWS[senderId].set(ackedRows(sender, fsnToRows)); } else { sender.flush(); } - // Deliberately NOT setting ACKED_ROWS to `sent` here, and deliberately not printing - // "finished" yet. The try-with-resources close() below is where the bulk of the - // durability wait actually happens: measured locally, awaitAckedFsn above returned - // almost immediately while close() took 12.8s of a 19.8s run. Claiming every row was - // acked before that wait made the reporter believe the run was done, which silenced - // both the progress line and the read probe for the whole drain -- the exact window a - // durability demo needs to show. + // "finished" is deliberately NOT printed yet. The try-with-resources close() below + // still runs (briefly, per --close-flush-timeout-ms), and announcing completion + // before it returns would be premature. } catch (Exception e) { System.err.printf("Sender %d got error: %s%s%n", senderId, e.toString(), upgradeHint(e)); throw new RuntimeException(e); } - // Reached only after close() returned, i.e. after the real drain. Now the full count is - // true rather than aspirational. - ACKED_ROWS[senderId].set(sent); + // Reached only after close() returned. System.out.printf("Sender %d finished sending %d events%n", senderId, sent); } + // One progress tick. Everything here is MEASURED, nothing is inferred: + // submitted - our own counter of rows handed to the client + // frames acked - the client's ack watermark against the highest frame it published + // backlog - bytes actually on disk in the store-and-forward root + // + // There is deliberately no row-level "acknowledged" figure. Translating the frame watermark + // back into rows needed a side map of fsn -> cumulative rows, and that translation produced + // six distinct wrong answers during development: premature saturation before close(), + // 100,000-row quantisation, a gate that silenced this reporter 7s into a 98s drain, a 0 that + // should have been "unknown" on a resumed slot, a -1 sentinel poisoning the map so a 19,579 + // frame run reported 100,000,000 rows acked, and the same sentinel printing "/-1" as a + // denominator. Frames and bytes have never been wrong. Rows are gone. + private static void reportTick(long totalEvents, String storeForwardDir, + AtomicLong appendDoneNanos, long[] lastSfBytes, long lastSub) { + final long sub = TOTAL_SENT.get(); + if (sub >= totalEvents) { + appendDoneNanos.compareAndSet(0, System.nanoTime()); + SUBMIT_COMPLETE = true; + } + + long ackedFsn = 0; + long publishedFsn = 0; + boolean anyPublished = false; + for (int i = 0; i < ACKED_FSN.length; i++) { + ackedFsn += ACKED_FSN[i].get(); + final long pub = PUBLISHED_FSN[i].get(); + if (pub > 0) { + publishedFsn += pub; + anyPublished = true; + } + } + System.out.printf("[progress] submitted=%,d (+%,d/s) | frames acked=%s%n", + sub, sub - lastSub, + anyPublished ? String.format("%,d/%,d", ackedFsn, publishedFsn) + : String.format("%,d/pending", ackedFsn)); + + // Watch the spill shrink. Reported only while there is something there, so a healthy run + // with no backlog stays quiet. + final long sfNow = sfDirBytes(storeForwardDir); + if (sfNow > 0) { + System.out.printf("[store-and-forward] backlog on disk: %s%s%n", + humanBytes(sfNow), + lastSfBytes[0] > 0 && sfNow != lastSfBytes[0] + ? String.format(" (%+.1f MiB/s)", + (sfNow - lastSfBytes[0]) / (1024.0 * 1024.0)) + : ""); + } + lastSfBytes[0] = sfNow; + } + // Bytes currently spilled under the store-and-forward root. The client exposes no replay // event and no backlog metric, so the filesystem is the only observable signal that a // restart is replaying a previous run's unacked frames. Cheap enough to sample once a @@ -500,11 +555,17 @@ private static long sfDirBytes(String dir) { return walk.filter(Files::isRegularFile).mapToLong(f -> { try { return Files.size(f); - } catch (IOException ignored) { + } catch (IOException | RuntimeException ignored) { return 0L; // segment reclaimed mid-walk; it simply no longer counts } }).sum(); - } catch (IOException e) { + } catch (IOException | RuntimeException e) { + // Files.walk wraps a mid-iteration failure in UncheckedIOException, which is a + // RuntimeException and so slips past `catch (IOException)`. Store-and-forward + // creates and reclaims segments continuously, so entries really do vanish under + // the walk. Letting that escape killed the reporter thread outright: progress + // lines stopped dead at the moment an outage began and spill writes surged, while + // the probe (a different thread) kept going, which looked like the gate bug again. return 0L; } } @@ -525,27 +586,15 @@ private static String humanBytes(long bytes) { // Unacked rows still owed to the server, i.e. what store-and-forward has to replay. The client // exposes no replay event: SenderConnectionEvent.Kind carries only CONNECTED, DISCONNECTED, // RECONNECTED, FAILED_OVER, ENDPOINT_ATTEMPT_FAILED, ALL_ENDPOINTS_UNREACHABLE and AUTH_FAILED, - // and bufferView() is just the unflushed local buffer, not the spilled backlog. So derive the - // figure from our own counters and announce it on reconnect; the acknowledged counter then - // climbing in the progress lines IS the replay. - private static String backlogNote() { - long acked = 0; - for (AtomicLong al : ACKED_ROWS) { - acked += al.get(); - } - final long backlog = TOTAL_SENT.get() - acked; - if (backlog <= 0) { + // and bufferView() is just the unflushed local buffer, not the spilled backlog. So report the + // spill's size on disk, which is measured rather than inferred; the backlog shrinking in the + // per-tick lines afterwards IS the replay. + private static String backlogNote(String storeForwardDir) { + final long bytes = sfDirBytes(storeForwardDir); + if (bytes <= 0) { return ""; } - return String.format(" -- REPLAYING store-and-forward backlog of %,d row(s)", backlog); - } - - // Rows the server has acknowledged for this sender: map the ack watermark (getAckedFsn) back to - // the cumulative row count recorded at the latest fully-acknowledged flush. Zero round-trips - - // the ack watermark rides the ingest connection. - private static long ackedRows(Sender sender, java.util.TreeMap fsnToRows) { - final java.util.Map.Entry e = fsnToRows.floorEntry(sender.getAckedFsn()); - return e != null ? e.getValue() : 0L; + return String.format(" -- REPLAYING store-and-forward backlog of %s", humanBytes(bytes)); } // Send the designated timestamp at NANOSECOND resolution. The client's at(Instant) path @@ -723,11 +772,11 @@ private static Sender buildSender(SenderCfg cfg, int workerId) { msg = "connected to " + host; break; case RECONNECTED: - msg = "reconnected to " + host + backlogNote(); + msg = "reconnected to " + host + backlogNote(cfg.storeForwardDir); break; case FAILED_OVER: msg = "failed over " + event.getPreviousHost() + ":" + event.getPreviousPort() - + " -> " + host + backlogNote(); + + " -> " + host + backlogNote(cfg.storeForwardDir); break; case AUTH_FAILED: msg = "auth failed for " + host; @@ -773,6 +822,12 @@ private static Sender buildSender(SenderCfg cfg, int workerId) { b.connectTimeoutMillis(cfg.connectTimeoutMs); } + // Keep close() from re-running a long implicit drain. runWorker() already drained in + // observable slices against --retry-timeout, and whatever is still unacked is replayed by + // the next sender from the spill directory, so a long close only delays shutdown and + // produces a "data may be lost" error for data that is not lost. 0 opts out entirely. + b.closeFlushTimeoutMillis(cfg.closeFlushTimeoutMs); + return b.storeAndForwardDir(sfPath) .senderId(who) .transactional(true) @@ -1189,6 +1244,7 @@ private static Map parseArgs(String[] args) { case "--batch-size": case "--batches-per-transaction": case "--probe-interval-ms": + case "--close-flush-timeout-ms": case "--connect-timeout-ms": case "--enterprise": case "--zone": @@ -1229,13 +1285,16 @@ private static final class SenderCfg { final boolean enterprise; final String zone; final int connectTimeoutMs; + // close()'s implicit drain budget. Separate from retryTimeout on purpose: our own polled + // drain owns the waiting, so this only bounds shutdown. + final long closeFlushTimeoutMs; // Target aggregate rows/second across all workers; 0 = disabled (use delayMs). final long rate; SenderCfg(String protocol, String addrsCsv, String token, String username, String password, int retryTimeout, String senderIdBase, String storeForwardDir, int batchSize, int batchesPerTransaction, int numSenders, boolean enterprise, String zone, - int connectTimeoutMs, long rate) { + int connectTimeoutMs, long closeFlushTimeoutMs, long rate) { this.protocol = protocol; this.addrsCsv = addrsCsv; this.token = token; @@ -1250,6 +1309,7 @@ private static final class SenderCfg { this.enterprise = enterprise; this.zone = zone; this.connectTimeoutMs = connectTimeoutMs; + this.closeFlushTimeoutMs = closeFlushTimeoutMs; this.rate = rate; } } From cfd662b30f966b0b44181a9505f78e237e5085b2 Mon Sep 17 00:00:00 2001 From: javier Date: Tue, 29 Sep 2026 13:53:05 +0200 Subject: [PATCH 50/79] sender: banner the HA transitions, and fix reporting at demo scale All of this came from running a slow, small-batch config (--delay-ms 1, --batches-per-transaction 1) that the earlier high-throughput testing never covered. That turned out to be where the reporting was weakest, and it is the configuration closest to an actual demo. Banner the three headline events. A write path moving to another node, reads moving to a replica, and a spilled backlog replaying are the whole point of an HA demo, and as ordinary single lines they scrolled past unnoticed between per-second progress lines. Each is now wrapped in ### rules: ############################################################################ ### READS FAILED OVER -- now served by role=REPLICA ### node=... zone=eu-west-1 ### queries keep being answered; only writes need a primary ############################################################################ Only RECONNECTED and FAILED_OVER are promoted. The per-attempt "endpoint ... failed, trying next" stays an ordinary line, because during an outage it repeats every few seconds and bannering it would bury the real transitions. Each banner is built into one StringBuilder and emitted with a single write, so the reporter and probe threads cannot interleave a line into the middle of it. Name the flush threshold. "frames acked=0/pending" was opaque: a flush happens every batch-size x batches-per-transaction rows, so with 10000 x 1 at ~944 rows/s the first frame is 10.6s away and the field looked broken. It now reads "none yet, first flush at 10,000 rows". Qualify the spill figure. sfDirBytes reports ALLOCATED bytes, and segments are memory-mapped and pre-allocated to sf_max_segment_bytes (4 MiB default), so a run holding under a megabyte displayed "8.0 MiB". That is ~10x overstated at demo scale and invisible at GB scale, which is why earlier runs looked fine. Now "spill: 8.0 MiB allocated in 5 segment file(s)", from a single directory walk rather than two that could disagree while segments rotate. Clamp the second -1 sentinel. getAckedFsn() returns -1 for "nothing acked yet", exactly as flushAndGetSequence() returns -1 for "nothing to flush". Guarding only the latter left "frames acked=-1/11" on the first flush of a slow feed. Both accessors are clamped at every assignment now, not at the point of display, and the convention is recorded next to the fields. Omit frames for non-QWP protocols. UDP is fire-and-forget and ILP auto-flushes over HTTP; neither carries frame sequence numbers, so --protocol qwpudp/ilp would have printed "first flush at 0 rows" indefinitely. Found by review, not by testing, since the demo path is QWP. Verified against a live cluster: the startup-replay banner, the "none yet" wording, the spill wording, and zero sentinel leaks across --batch-size 10000 and 1000. NOT verified: the read-failover and write-failover banners need a writable cluster to trigger, and the cluster had no primary at the time (enterprise-primary reporting role=REPLICA, no writable node among all three addresses). The code path is the same listener that already produced the unbannered versions of both messages earlier, but the rendering itself is unobserved. --- .../com/example/sender/CsvParallelSender.java | 124 +++++++++++++----- 1 file changed, 94 insertions(+), 30 deletions(-) diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index 3dd0fef..06d3c3e 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -94,6 +94,11 @@ public class CsvParallelSender { // straight from the client (getAckedFsn / flushAndGetSequence) and are what the protocol // actually acknowledges, which is also what the client's own errors quote // ("targetFsn=10999, ackedFsn=5455"). No row-level ack figure is tracked: see reportTick(). + // + // BOTH accessors use -1 as a sentinel: getAckedFsn() for "nothing acked yet" and + // flushAndGetSequence() for "nothing to flush". Neither may reach these fields, or it + // corrupts a sum and prints nonsense like "frames acked=-1/11". Clamp/guard at every + // assignment, not at the point of display. private static AtomicLong[] ACKED_FSN = new AtomicLong[0]; private static AtomicLong[] PUBLISHED_FSN = new AtomicLong[0]; // Set once all rows are submitted. Marks the append-phase boundary for timing; it does @@ -242,9 +247,9 @@ public static void main(String[] args) throws Exception { // slot, so this process's own counters cannot describe it. final long sfStartBytes = sfDirBytes(storeForwardDir); if (sfStartBytes > 0) { - System.out.printf("[store-and-forward] found %s of pending frames in %s; " - + "REPLAYING previous run's unacked data before new sends%n", - humanBytes(sfStartBytes), storeForwardDir); + banner("STORE-AND-FORWARD REPLAY ON STARTUP", + humanBytes(sfStartBytes) + " of pending frames in " + storeForwardDir, + "a previous run left this unacked; it replays before new sends"); } // Records (to ~1s resolution) when all rows were submitted and when all were acknowledged, @@ -256,6 +261,11 @@ public static void main(String[] args) throws Exception { // run ahead of the server) and acknowledged (rows the server has actually committed). The // gap between them is the buffered backlog; during the drain, submitted is flat while // acknowledged keeps climbing (real work), so there is no misleading "0 rows/s". + // Rows between flushes, i.e. how often a frame is published. Mirrors runWorker's + // commitEveryRows so the progress line can name the threshold. + final long flushEveryRows = protocol.equals("qwp") + ? (long) batchSize * batchesPerTransaction + : 0L; final long[] lastSfBytes = {sfStartBytes}; final Thread reporter = new Thread(() -> { long lastSub = 0; @@ -266,7 +276,8 @@ public static void main(String[] args) throws Exception { return; } try { - reportTick(totalEvents, storeForwardDir, appendDoneNanos, lastSfBytes, lastSub); + reportTick(totalEvents, storeForwardDir, flushEveryRows, + appendDoneNanos, lastSfBytes, lastSub); lastSub = TOTAL_SENT.get(); } catch (RuntimeException e) { // Never let one bad tick silence progress for the rest of the run. A dead @@ -404,7 +415,7 @@ private static void runWorker( if (fsn >= 0) { PUBLISHED_FSN[senderId].set(fsn); } - ACKED_FSN[senderId].set(sender.getAckedFsn()); + ACKED_FSN[senderId].set(Math.max(0L, sender.getAckedFsn())); } else { sender.flush(); } @@ -459,7 +470,7 @@ private static void runWorker( boolean drained = false; while (System.nanoTime() < drainDeadline) { drained = sender.drain(DRAIN_SLICE_MS); - ACKED_FSN[senderId].set(sender.getAckedFsn()); + ACKED_FSN[senderId].set(Math.max(0L, sender.getAckedFsn())); if (drained) { break; } @@ -504,7 +515,7 @@ private static void runWorker( // should have been "unknown" on a resumed slot, a -1 sentinel poisoning the map so a 19,579 // frame run reported 100,000,000 rows acked, and the same sentinel printing "/-1" as a // denominator. Frames and bytes have never been wrong. Rows are gone. - private static void reportTick(long totalEvents, String storeForwardDir, + private static void reportTick(long totalEvents, String storeForwardDir, long commitEveryRows, AtomicLong appendDoneNanos, long[] lastSfBytes, long lastSub) { final long sub = TOTAL_SENT.get(); if (sub >= totalEvents) { @@ -523,17 +534,33 @@ private static void reportTick(long totalEvents, String storeForwardDir, anyPublished = true; } } - System.out.printf("[progress] submitted=%,d (+%,d/s) | frames acked=%s%n", - sub, sub - lastSub, - anyPublished ? String.format("%,d/%,d", ackedFsn, publishedFsn) - : String.format("%,d/pending", ackedFsn)); + if (commitEveryRows <= 0) { + // Not QWP: no frame sequence numbers and no acks to report (UDP is fire-and-forget, + // ILP auto-flushes over HTTP), so the frames field would be meaningless. + System.out.printf("[progress] submitted=%,d (+%,d/s)%n", sub, sub - lastSub); + } else { + System.out.printf("[progress] submitted=%,d (+%,d/s) | frames acked=%s%n", + sub, sub - lastSub, + anyPublished + ? String.format("%,d/%,d", ackedFsn, publishedFsn) + // No frame published yet, which is normal early on: a flush happens + // every batch-size x batches-per-transaction rows, so at a paced feed + // the first one can be many seconds away. Name the threshold rather + // than leaving a bare "pending" to be reverse-engineered. + : String.format("none yet, first flush at %,d rows", commitEveryRows)); + } // Watch the spill shrink. Reported only while there is something there, so a healthy run // with no backlog stays quiet. - final long sfNow = sfDirBytes(storeForwardDir); + final long[] sfStats = sfDirStats(storeForwardDir); + final long sfNow = sfStats[0]; if (sfNow > 0) { - System.out.printf("[store-and-forward] backlog on disk: %s%s%n", - humanBytes(sfNow), + // Bytes are ALLOCATED, not used: segments are memory-mapped and pre-allocated to + // sf_max_segment_bytes (4 MiB by default), so a run holding a few hundred KB still + // shows a whole segment. The file count makes that readable at demo scale, and both + // figures stay meaningful once a real backlog builds. + System.out.printf("[store-and-forward] spill: %s allocated in %,d segment file(s)%s%n", + humanBytes(sfNow), sfStats[1], lastSfBytes[0] > 0 && sfNow != lastSfBytes[0] ? String.format(" (%+.1f MiB/s)", (sfNow - lastSfBytes[0]) / (1024.0 * 1024.0)) @@ -546,28 +573,54 @@ private static void reportTick(long totalEvents, String storeForwardDir, // event and no backlog metric, so the filesystem is the only observable signal that a // restart is replaying a previous run's unacked frames. Cheap enough to sample once a // second: a few hundred segment files. - private static long sfDirBytes(String dir) { + // Rule width for the attention banners below. The events they wrap (a write failing over to + // another node, reads moving to a replica, a spilled backlog replaying) are the whole point of + // an HA demo, and as ordinary one-line output they scrolled past unnoticed among per-second + // progress lines. + private static final String BANNER_RULE = + "############################################################################"; + + private static void banner(String... lines) { + final StringBuilder sb = new StringBuilder(); + sb.append('\n').append(BANNER_RULE).append('\n'); + for (String line : lines) { + sb.append("### ").append(line).append('\n'); + } + sb.append(BANNER_RULE).append('\n'); + // One write, so the banner cannot be interleaved by the reporter or probe threads. + System.out.print(sb); + } + + // {bytes, fileCount} for the spill root, from a SINGLE walk: two separate traversals a + // second apart could disagree with each other while segments rotate. + private static long[] sfDirStats(String dir) { final Path root = Path.of(dir); if (!Files.exists(root)) { - return 0L; + return new long[]{0L, 0L}; } + long bytes = 0L; + long count = 0L; try (java.util.stream.Stream walk = Files.walk(root)) { - return walk.filter(Files::isRegularFile).mapToLong(f -> { + for (Path f : (Iterable) walk.filter(Files::isRegularFile)::iterator) { try { - return Files.size(f); + bytes += Files.size(f); + count++; } catch (IOException | RuntimeException ignored) { - return 0L; // segment reclaimed mid-walk; it simply no longer counts + // Segment reclaimed mid-walk; it simply no longer counts. } - }).sum(); + } } catch (IOException | RuntimeException e) { - // Files.walk wraps a mid-iteration failure in UncheckedIOException, which is a - // RuntimeException and so slips past `catch (IOException)`. Store-and-forward - // creates and reclaims segments continuously, so entries really do vanish under - // the walk. Letting that escape killed the reporter thread outright: progress - // lines stopped dead at the moment an outage began and spill writes surged, while - // the probe (a different thread) kept going, which looked like the gate bug again. - return 0L; + // Files.walk wraps a mid-iteration failure in UncheckedIOException, a RuntimeException + // that slips past catch (IOException). Store-and-forward reclaims segments + // continuously, so entries really do vanish under the walk, and letting that escape + // once killed the reporter thread outright. + return new long[]{bytes, count}; } + return new long[]{bytes, count}; + } + + private static long sfDirBytes(String dir) { + return sfDirStats(dir)[0]; } private static String humanBytes(long bytes) { @@ -767,16 +820,20 @@ private static Sender buildSender(SenderCfg cfg, int workerId) { ? String.valueOf(event.getCause().getMessage()) : "no detail"; final String msg; boolean noisy = false; + // Wrap in a banner: a write path moving to another node is a headline event. + boolean prominent = false; switch (event.getKind()) { case CONNECTED: msg = "connected to " + host; break; case RECONNECTED: msg = "reconnected to " + host + backlogNote(cfg.storeForwardDir); + prominent = true; break; case FAILED_OVER: msg = "failed over " + event.getPreviousHost() + ":" + event.getPreviousPort() + " -> " + host + backlogNote(cfg.storeForwardDir); + prominent = true; break; case AUTH_FAILED: msg = "auth failed for " + host; @@ -808,7 +865,11 @@ private static Sender buildSender(SenderCfg cfg, int workerId) { } lastNoisyMs[0] = now; } - System.out.printf("[ingestion client %s] %s%n", who, msg); + if (prominent) { + banner("INGESTION " + msg, "sender=" + who); + } else { + System.out.printf("[ingestion client %s] %s%n", who, msg); + } }; // Enterprise-only: hold spilled frames until a durable (committed) ack. OSS servers @@ -944,8 +1005,11 @@ public void onError(byte status, String message) { @Override public void onFailoverReset(QwpServerInfo info) { - System.out.printf("[query client] failed over -> now serving node=%s role=%s zone=%s%n", - orNone(info.getNodeId()), QwpServerInfo.roleName(info.getRole()), orNone(info.getZoneId())); + // Reads surviving an outage by moving to another node is THE thing an HA + // demo is showing, so it gets a banner rather than one line among many. + banner("READS FAILED OVER -- now served by role=" + QwpServerInfo.roleName(info.getRole()), + "node=" + orNone(info.getNodeId()) + " zone=" + orNone(info.getZoneId()), + "queries keep being answered; only writes need a primary"); } }; // `switch status` reports the serving node's LIVE lifecycle role: current_role, plus From 4d6f606a37eb6fb5e758b38436cd3e8b8782d7be Mon Sep 17 00:00:00 2001 From: javier Date: Tue, 29 Sep 2026 13:58:24 +0200 Subject: [PATCH 51/79] sender: three rule lines each side of the HA banners A single ### rule above and below still got lost in the scroll of per-second progress and probe lines during a live demo. Three each side, plus a blank line outside the block so it does not butt up against the surrounding output. Rendered from identical code to confirm the spacing: [progress] submitted=8,332 (+944/s) | frames acked=31/33 ############################################################################ ############################################################################ ############################################################################ ### INGESTION failed over 172.31.42.41:9000 -> 172.31.41.35:9000 -- REPLAYING ... ### sender=ha_sender-0 ############################################################################ ############################################################################ ############################################################################ [progress] submitted=9,276 (+944/s) | frames acked=31/35 Known cosmetic wrinkle: the INGESTION line runs to ~113 characters against a 76-character rule, so it wraps on an 80-column terminal and the block looks ragged there. Left alone rather than guessed at, since the fix depends on the demo terminal width: either widen BANNER_RULE or split the hosts and the backlog onto separate ### lines. --- .../java/com/example/sender/CsvParallelSender.java | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index 06d3c3e..6637f0c 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -580,13 +580,23 @@ private static void reportTick(long totalEvents, String storeForwardDir, long co private static final String BANNER_RULE = "############################################################################"; + // Rule lines above and below the message. Three each side, not one: a single rule still got + // lost in the scroll of per-second progress and probe lines during a live demo. + private static final int BANNER_RULE_LINES = 3; + private static void banner(String... lines) { final StringBuilder sb = new StringBuilder(); - sb.append('\n').append(BANNER_RULE).append('\n'); + sb.append('\n'); + for (int i = 0; i < BANNER_RULE_LINES; i++) { + sb.append(BANNER_RULE).append('\n'); + } for (String line : lines) { sb.append("### ").append(line).append('\n'); } - sb.append(BANNER_RULE).append('\n'); + for (int i = 0; i < BANNER_RULE_LINES; i++) { + sb.append(BANNER_RULE).append('\n'); + } + sb.append('\n'); // One write, so the banner cannot be interleaved by the reporter or probe threads. System.out.print(sb); } From 9914987faee049ce9e1eb61e2cbb005009b0bcfd Mon Sep 17 00:00:00 2001 From: javier Date: Tue, 29 Sep 2026 14:35:11 +0200 Subject: [PATCH 52/79] feat: Java QWP read benchmark with multi-host failover Port python/read_bench.py to Java (com.example.reader.ReadBench). Streams records off QWP column batches without materialising them, reports rows/s and payload throughput live, and prints the first and last rows of the range at the end. --addr accepts a comma-separated host list for failover. Chunks re-run after a failover or retried after an error roll back their counts first, so rows are never double-counted. Add query_table.sh to build and run it; with no arguments it runs the demo read. --- query_table.sh | 31 + .../java/com/example/reader/ReadBench.java | 651 ++++++++++++++++++ 2 files changed, 682 insertions(+) create mode 100755 query_table.sh create mode 100644 src/main/java/com/example/reader/ReadBench.java diff --git a/query_table.sh b/query_table.sh new file mode 100755 index 0000000..2492a99 --- /dev/null +++ b/query_table.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Read the last N rows of a table with the Java QWP reader (com.example.reader.ReadBench). +# +# With no arguments it runs the demo read against $QDB_ADDR (default 172.31.42.41:9000), +# authenticating with $ILP_TOKEN. With arguments, they are passed straight to ReadBench: +# +# ./query_table.sh trades --addr h1:9000,h2:9000 --limit 200000000 \ +# --token "$ILP_TOKEN" --tls-verify unsafe_off --readers 8 --sample 5 +# +# --addr takes a comma-separated host list for failover. Run with --help for all options. +set -euo pipefail + +cd "$(dirname "$0")" +JAR=target/ilp_sender-1.0-SNAPSHOT.jar + +if [ ! -f "$JAR" ] || [ -n "$(find src -newer "$JAR" -name '*.java' -print -quit)" ]; then + mvn -q -DskipTests package +fi + +if [ "$#" -eq 0 ]; then + : "${ILP_TOKEN:?set ILP_TOKEN to your QuestDB token}" + set -- trades \ + --addr "${QDB_ADDR:-172.31.42.41:9000}" \ + --limit 200000000 \ + --token "$ILP_TOKEN" \ + --tls-verify unsafe_off \ + --readers 8 \ + --sample 5 +fi + +exec java -cp "$JAR" com.example.reader.ReadBench "$@" diff --git a/src/main/java/com/example/reader/ReadBench.java b/src/main/java/com/example/reader/ReadBench.java new file mode 100644 index 0000000..d3f99b4 --- /dev/null +++ b/src/main/java/com/example/reader/ReadBench.java @@ -0,0 +1,651 @@ +package com.example.reader; + +import io.questdb.client.Query; +import io.questdb.client.QueryException; +import io.questdb.client.QuestDB; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatch; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; +import io.questdb.client.cutlass.qwp.client.QwpServerInfo; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; + +import java.time.Instant; +import java.time.ZoneOffset; +import java.time.format.DateTimeFormatter; +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.UUID; +import java.util.concurrent.ConcurrentLinkedQueue; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.atomic.AtomicLongArray; + +/** + * Java port of python/read_bench.py: read the last N rows of a table as fast as the + * QWP query client allows, reporting rows/sec and throughput live. + *

+ * Records are streamed straight off the QWP column batches: the loop only counts rows + * and payload bytes, nothing is materialised. At the end it prints the first and last + * {@code --sample} rows of the range, rendered from the batches already received (no + * extra query). + *

+ * Throughput is the uncompressed QWP batch payload ({@code payloadLimit - payloadAddr}), + * not wire bytes and not the Python script's decoded Arrow size, so MB/s figures are not + * directly comparable between the two. + *

+ * Failover: {@code --addr} takes a comma-separated host list + * ({@code h1:9000,h2:9000}). The client connects to the first reachable node and fails + * over to the next one if a connection drops. A failover mid-query discards the partial + * result and re-runs the query on the new node, so the chunk's counts and captured rows + * are rolled back and re-read. A chunk whose query fails outright is also rolled back and + * retried, up to {@code --chunk-retries} times. + *

+ * Splitting ({@code --split rows|time}, {@code --chunks}) follows read_bench.py exactly; + * see its docstring for the trade-offs. + */ +public class ReadBench { + + private static final DateTimeFormatter ISO_NANOS = + DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSS'Z'").withZone(ZoneOffset.UTC); + + // ---------------------------------------------------------------- args + + static final class Args { + String table; + String addr = "localhost:9000"; + long limit = 10_000_000L; + int readers = 1; + String split = "rows"; + int chunks = 0; + String timestampCol = "timestamp"; + double reportInterval = 0.5; + int sample = 5; + String token; + String username; + String password; + boolean tls; + String tlsVerify = "on"; + String target; + String zone; + String conf; + int chunkRetries = 3; + + boolean useTls() { + return tls || token != null || (username != null && password != null); + } + } + + private static void usage() { + System.err.println( + "usage: ReadBench TABLE [--addr host:port[,host:port...]] [--limit N] [--readers N]\n" + + " [--split rows|time] [--chunks N] [--timestamp-col COL]\n" + + " [--report-interval SEC] [--sample N]\n" + + " [--token TOK | --token-file PATH | --username U --password P] [--tls]\n" + + " [--tls-verify on|unsafe_off] [--target any|primary|replica]\n" + + " [--zone Z] [--chunk-retries N] [--conf CONNECT_STRING]\n" + + "\n" + + " --addr accepts a comma-separated list for failover, e.g. h1:9000,h2:9000\n" + + " --conf replaces the generated connect string entirely (addr/auth/tls flags ignored)"); + } + + private static Args parse(String[] argv) { + Args a = new Args(); + for (int i = 0; i < argv.length; i++) { + String k = argv[i]; + if (k.equals("-h") || k.equals("--help")) { + usage(); + System.exit(0); + } + if (!k.startsWith("--")) { + if (a.table != null) { + throw new IllegalArgumentException("unexpected argument: " + k); + } + a.table = k; + continue; + } + if (k.equals("--tls")) { + a.tls = true; + continue; + } + if (i + 1 >= argv.length) { + throw new IllegalArgumentException("missing value for " + k); + } + String v = argv[++i]; + switch (k) { + case "--addr": a.addr = v; break; + case "--limit": a.limit = Long.parseLong(v.replace("_", "")); break; + case "--readers": a.readers = Integer.parseInt(v); break; + case "--split": a.split = v; break; + case "--chunks": a.chunks = Integer.parseInt(v); break; + case "--timestamp-col": a.timestampCol = v; break; + case "--report-interval": a.reportInterval = Double.parseDouble(v); break; + case "--sample": a.sample = Integer.parseInt(v); break; + case "--token": a.token = v.isEmpty() ? null : v; break; + case "--token-file": a.token = readTokenFile(v); break; + case "--username": a.username = v; break; + case "--password": a.password = v; break; + case "--tls-verify": a.tlsVerify = v; break; + case "--target": a.target = v; break; + case "--zone": a.zone = v; break; + case "--conf": a.conf = v; break; + case "--chunk-retries": a.chunkRetries = Integer.parseInt(v); break; + default: throw new IllegalArgumentException("unknown option: " + k); + } + } + if (a.table == null) { + throw new IllegalArgumentException("TABLE is required"); + } + if (a.readers < 1) { + throw new IllegalArgumentException("--readers must be >= 1"); + } + if (!a.split.equals("rows") && !a.split.equals("time")) { + throw new IllegalArgumentException("--split must be rows or time"); + } + if (!a.tlsVerify.equals("on") && !a.tlsVerify.equals("unsafe_off")) { + throw new IllegalArgumentException("--tls-verify must be on or unsafe_off"); + } + return a; + } + + private static String readTokenFile(String path) { + try { + String p = path.startsWith("~/") ? System.getProperty("user.home") + path.substring(1) : path; + return new String(java.nio.file.Files.readAllBytes(java.nio.file.Paths.get(p)), + java.nio.charset.StandardCharsets.UTF_8).trim(); + } catch (java.io.IOException e) { + throw new IllegalArgumentException("cannot read --token-file " + path + ": " + e.getMessage()); + } + } + + /** QWP connect string for the reader(s), with optional auth, TLS and failover routing. */ + static String buildConf(Args a) { + if (a.conf != null) { + return a.conf; + } + StringBuilder sb = new StringBuilder(); + sb.append(a.useTls() ? "wss" : "ws").append("::addr=").append(a.addr).append(';'); + if (a.token != null) { + sb.append("token=").append(a.token).append(';'); + } else if (a.username != null && a.password != null) { + sb.append("username=").append(a.username).append(";password=").append(a.password).append(';'); + } + if (a.useTls() && a.tlsVerify.equals("unsafe_off")) { + sb.append("tls_verify=unsafe_off;"); + } + if (a.target != null) { + sb.append("target=").append(a.target).append(';'); + } + if (a.zone != null) { + sb.append("zone=").append(a.zone).append(';'); + } + return sb.toString(); + } + + /** Connect string with secrets masked, safe to print. */ + static String redact(String conf) { + return conf.replaceAll("(token|password)=[^;]*", "$1=***"); + } + + // ---------------------------------------------------------------- work split + + static final class Chunk { + final int index; // position in the range, 0 = oldest + final String sql; + + Chunk(int index, String sql) { + this.index = index; + this.sql = sql; + } + } + + /** + * {@code --chunks} SELECTs covering the last {@code --limit} rows as equal row-count + * slices, oldest first, using {@code LIMIT -m, -n} (the half-open range [-m, -n)). + */ + static List rowChunks(Args a) { + List sqls = new ArrayList<>(); + for (int i = 0; i < a.chunks; i++) { + long lo = a.limit - (a.limit * i) / a.chunks; // rows from the end, inclusive + long hi = a.limit - (a.limit * (i + 1)) / a.chunks; // rows from the end, exclusive + if (lo <= hi) { + continue; + } + if (hi == 0) { + sqls.add("select * from " + a.table + " limit -" + lo); + } else { + sqls.add("select * from " + a.table + " limit -" + lo + ", -" + hi); + } + } + return sqls; + } + + /** + * One SELECT per reader, splitting the last-N rows' timestamp range into equal time + * spans. Costs one preliminary query. + */ + static List timeSlices(Args a, QuestDB db) throws InterruptedException { + String all = "select * from " + a.table + " limit -" + a.limit; + if (a.readers <= 1) { + return Collections.singletonList(all); + } + String ts = a.timestampCol; + final long[] bounds = new long[2]; + final boolean[] found = {false}; + final long[] nanosPerUnit = {1_000L}; + try (Query q = db.borrowQuery()) { + q.sql("select min(" + ts + ") lo, max(" + ts + ") hi from (select " + ts + + " from " + a.table + " limit -" + a.limit + ")") + .handler(new QwpColumnBatchHandler() { + @Override + public void onBatch(QwpColumnBatch batch) { + if (batch.getRowCount() == 0 || batch.isNull(0, 0) || batch.isNull(1, 0)) { + return; + } + byte t = batch.getColumnWireType(0); + nanosPerUnit[0] = t == QwpConstants.TYPE_TIMESTAMP_NANOS ? 1L + : t == QwpConstants.TYPE_DATE ? 1_000_000L : 1_000L; + bounds[0] = batch.getLongValue(0, 0); + bounds[1] = batch.getLongValue(1, 0); + found[0] = true; + } + + @Override + public void onEnd(long totalRows) { + } + + @Override + public void onError(byte status, String message) { + } + }).submit().await(); + } + if (!found[0]) { + return Collections.singletonList(all); + } + long lo = bounds[0] * nanosPerUnit[0]; + long hi = bounds[1] * nanosPerUnit[0]; + if (hi <= lo) { + System.err.println("[warn] timestamp range too narrow to split; using 1 reader"); + return Collections.singletonList(all); + } + long span = hi - lo; + List sqls = new ArrayList<>(); + for (int i = 0; i < a.readers; i++) { + long from = lo + (long) ((double) span * i / a.readers); + long to = i == a.readers - 1 ? hi : lo + (long) ((double) span * (i + 1) / a.readers); + String op = i == a.readers - 1 ? "<=" : "<"; // last slice includes the max row + sqls.add("select * from " + a.table + " where " + ts + " >= '" + nsToIso(from) + + "' and " + ts + " " + op + " '" + nsToIso(to) + "'"); + } + return sqls; + } + + static String nsToIso(long ns) { + return ISO_NANOS.format(Instant.ofEpochSecond(Math.floorDiv(ns, 1_000_000_000L), + Math.floorMod(ns, 1_000_000_000L))); + } + + // ---------------------------------------------------------------- row capture + + /** First and last {@code n} rows of one chunk, rendered as strings. */ + static final class Capture { + final List head = new ArrayList<>(); + final ArrayDeque tail = new ArrayDeque<>(); + + void clear() { + head.clear(); + tail.clear(); + } + } + + static volatile String[] columnNames; + + static String render(QwpColumnBatch b, int col, int row) { + if (b.isNull(col, row)) { + return "null"; + } + byte type = b.getColumnWireType(col); + switch (type) { + case QwpConstants.TYPE_BOOLEAN: return String.valueOf(b.getBoolValue(col, row)); + case QwpConstants.TYPE_BYTE: return String.valueOf(b.getByteValue(col, row)); + case QwpConstants.TYPE_SHORT: return String.valueOf(b.getShortValue(col, row)); + case QwpConstants.TYPE_CHAR: return String.valueOf(b.getCharValue(col, row)); + case QwpConstants.TYPE_INT: return String.valueOf(b.getIntValue(col, row)); + case QwpConstants.TYPE_LONG: return String.valueOf(b.getLongValue(col, row)); + case QwpConstants.TYPE_FLOAT: return String.valueOf(b.getFloatValue(col, row)); + case QwpConstants.TYPE_DOUBLE: return String.valueOf(b.getDoubleValue(col, row)); + case QwpConstants.TYPE_DATE: return nsToIso(b.getLongValue(col, row) * 1_000_000L); + case QwpConstants.TYPE_TIMESTAMP: return nsToIso(b.getLongValue(col, row) * 1_000L); + case QwpConstants.TYPE_TIMESTAMP_NANOS: return nsToIso(b.getLongValue(col, row)); + case QwpConstants.TYPE_SYMBOL: return b.getSymbol(col, row); + case QwpConstants.TYPE_VARCHAR: return b.getString(col, row); + case QwpConstants.TYPE_UUID: + return new UUID(b.getUuidHi(col, row), b.getUuidLo(col, row)).toString(); + case QwpConstants.TYPE_IPv4: { + int v = b.getIntValue(col, row); + return ((v >>> 24) & 0xFF) + "." + ((v >>> 16) & 0xFF) + "." + ((v >>> 8) & 0xFF) + "." + (v & 0xFF); + } + case QwpConstants.TYPE_LONG256: { + StringBuilder sb = new StringBuilder("0x"); + for (int w = 3; w >= 0; w--) { + sb.append(String.format("%016x", b.getLong256Word(col, row, w))); + } + return sb.toString(); + } + case QwpConstants.TYPE_DOUBLE_ARRAY: return Arrays.toString(b.getDoubleArrayElements(col, row)); + default: return "<" + QwpConstants.getTypeName(type) + ">"; + } + } + + static String[] renderRow(QwpColumnBatch b, int row) { + int n = b.getColumnCount(); + String[] out = new String[n]; + for (int c = 0; c < n; c++) { + out[c] = render(b, c, row); + } + return out; + } + + // ---------------------------------------------------------------- reader + + /** + * Result handler for one reader. It runs on the pool's query I/O thread while the + * reader thread blocks in {@code await()}; the await provides the happens-before + * edge for the per-chunk fields the reader thread resets between chunks. + */ + static final class ChunkHandler implements QwpColumnBatchHandler { + final int reader; + final AtomicLongArray rows; + final AtomicLongArray bytes; + final int sample; + Capture capture; + long chunkRows; + long chunkBytes; + final List failovers; + + ChunkHandler(int reader, AtomicLongArray rows, AtomicLongArray bytes, int sample, List failovers) { + this.reader = reader; + this.rows = rows; + this.bytes = bytes; + this.sample = sample; + this.failovers = failovers; + } + + void start(Capture c) { + capture = c; + chunkRows = 0; + chunkBytes = 0; + } + + /** Undo this chunk's contribution, before a re-run. */ + void rollback() { + rows.addAndGet(reader, -chunkRows); + bytes.addAndGet(reader, -chunkBytes); + chunkRows = 0; + chunkBytes = 0; + if (capture != null) { + capture.clear(); + } + } + + @Override + public void onBatch(QwpColumnBatch batch) { + int n = batch.getRowCount(); + long b = batch.payloadLimit() - batch.payloadAddr(); + chunkRows += n; + chunkBytes += b; + rows.addAndGet(reader, n); + bytes.addAndGet(reader, b); + if (sample > 0 && n > 0) { + if (columnNames == null) { + String[] names = new String[batch.getColumnCount()]; + for (int c = 0; c < names.length; c++) { + names[c] = batch.getColumnName(c); + } + columnNames = names; + } + // Only the rows that can end up in the sample are rendered: the head + // until it is full, and the last `sample` rows of each batch for the tail. + for (int r = 0; r < n && capture.head.size() < sample; r++) { + capture.head.add(renderRow(batch, r)); + } + for (int r = Math.max(0, n - sample); r < n; r++) { + capture.tail.addLast(renderRow(batch, r)); + if (capture.tail.size() > sample) { + capture.tail.removeFirst(); + } + } + } + } + + @Override + public void onEnd(long totalRows) { + } + + @Override + public void onError(byte status, String message) { + } + + @Override + public void onFailoverReset(QwpServerInfo newNode) { + // Partial result discarded; the client re-runs the query on the new node. + rollback(); + failovers.add("reader " + reader + " failed over to node=" + newNode.getNodeId() + + " zone=" + newNode.getZoneId()); + } + } + + // ---------------------------------------------------------------- main + + public static void main(String[] argv) throws Exception { + Args a; + try { + a = parse(argv); + } catch (IllegalArgumentException e) { + System.err.println(e.getMessage()); + usage(); + System.exit(2); + return; + } + System.exit(run(a)); + } + + static int run(Args a) throws Exception { + String conf = buildConf(a); + System.out.println("[conf] " + redact(conf)); + + try (QuestDB db = QuestDB.builder() + .fromConfig(conf) + .queryPoolSize(a.readers) + .senderPoolMin(0) // read-only tool: never open an ingest connection + .build()) { + + List sqls; + int readers; + if (a.split.equals("rows")) { + a.chunks = Math.max(a.chunks > 0 ? a.chunks : a.readers * 8, a.readers); + sqls = rowChunks(a); + readers = Math.min(a.readers, sqls.size()); + } else { + sqls = timeSlices(a, db); + readers = sqls.size(); + } + + ConcurrentLinkedQueue work = new ConcurrentLinkedQueue<>(); + Capture[] captures = new Capture[sqls.size()]; + for (int i = 0; i < sqls.size(); i++) { + work.add(new Chunk(i, sqls.get(i))); + captures[i] = new Capture(); + } + + System.out.printf("[scan] reading last %,d rows of '%s' across %d reader(s), %d %s chunk(s) ...%n", + a.limit, a.table, readers, sqls.size(), a.split); + + AtomicLongArray rows = new AtomicLongArray(readers); + AtomicLongArray bytes = new AtomicLongArray(readers); + int[] chunksDone = new int[readers]; + List errors = new CopyOnWriteArrayList<>(); + List failovers = new CopyOnWriteArrayList<>(); + + Thread reporter = new Thread(() -> { + long lastR = 0, lastB = 0; + long sleepMs = (long) (a.reportInterval * 1000); + try { + while (true) { + Thread.sleep(sleepMs); + long r = sum(rows), b = sum(bytes); + System.out.printf("[scan] %,14d rows | %,13.0f rows/s | %,9.1f MB/s%n", + r, (r - lastR) / a.reportInterval, (b - lastB) / a.reportInterval / 1e6); + lastR = r; + lastB = b; + } + } catch (InterruptedException ignored) { + // stop + } + }, "reporter"); + reporter.setDaemon(true); + + long t0 = System.nanoTime(); + reporter.start(); + Thread[] threads = new Thread[readers]; + for (int i = 0; i < readers; i++) { + final int idx = i; + threads[i] = new Thread(() -> runReader(idx, db, work, captures, rows, bytes, + chunksDone, errors, failovers, a), "reader-" + i); + threads[i].start(); + } + for (Thread t : threads) { + t.join(); + } + double elapsed = (System.nanoTime() - t0) / 1e9; + reporter.interrupt(); + reporter.join(); + + for (String f : failovers) { + System.out.println("[failover] " + f); + } + if (!errors.isEmpty()) { + for (String e : errors) { + System.err.println("reader failed: " + e); + } + return 1; + } + + long total = sum(rows); + long tbytes = sum(bytes); + if (total == 0) { + System.err.println("[done] '" + a.table + "' returned 0 rows"); + return 1; + } + System.out.printf("[done] %,d rows, %.2f GiB payload in %.3fs across %d reader(s)%n", + total, tbytes / (double) (1L << 30), elapsed, readers); + System.out.printf("[done] %,.0f rows/s | %,.1f MB/s | %.2f Gb/s (uncompressed QWP payload, not wire bytes)%n", + total / elapsed, tbytes / elapsed / 1e6, tbytes * 8 / elapsed / 1e9); + if (readers > 1) { + StringBuilder per = new StringBuilder(); + for (int i = 0; i < readers; i++) { + per.append(String.format("r%d=%,d(%dch) ", i, rows.get(i), chunksDone[i])); + } + System.out.println("[done] per-reader rows: " + per.toString().trim()); + } + + if (a.sample > 0 && columnNames != null) { + printSample(a.sample, captures); + } + return 0; + } + } + + static void runReader(int idx, QuestDB db, ConcurrentLinkedQueue work, Capture[] captures, + AtomicLongArray rows, AtomicLongArray bytes, int[] chunksDone, + List errors, List failovers, Args a) { + ChunkHandler handler = new ChunkHandler(idx, rows, bytes, a.sample, failovers); + // One borrowed query handle (one connection) per reader, reused for every chunk. + try (Query q = db.borrowQuery()) { + Chunk chunk; + while ((chunk = work.poll()) != null) { + for (int attempt = 0; ; attempt++) { + handler.start(captures[chunk.index]); + try { + q.sql(chunk.sql).handler(handler).submit().await(); + break; + } catch (QueryException e) { + handler.rollback(); + if (attempt >= a.chunkRetries) { + throw e; + } + failovers.add(String.format("reader %d retrying chunk %d (attempt %d): status=0x%02X %s", + idx, chunk.index, attempt + 1, e.getStatus() & 0xFF, e.getMessage())); + } + } + chunksDone[idx]++; + } + } catch (Exception e) { + errors.add("reader " + idx + ": " + e); + } + } + + static void printSample(int n, Capture[] captures) { + List first = new ArrayList<>(); + for (int i = 0; i < captures.length && first.size() < n; i++) { + for (String[] r : captures[i].head) { + if (first.size() >= n) { + break; + } + first.add(r); + } + } + List last = new ArrayList<>(); + for (int i = captures.length - 1; i >= 0 && last.size() < n; i--) { + List tail = new ArrayList<>(captures[i].tail); + for (int j = tail.size() - 1; j >= 0 && last.size() < n; j--) { + last.add(tail.get(j)); + } + } + Collections.reverse(last); + + System.out.println(); + System.out.println("[sample] first " + first.size() + " row(s) of the range:"); + printTable(columnNames, first); + System.out.println(); + System.out.println("[sample] last " + last.size() + " row(s) of the range:"); + printTable(columnNames, last); + } + + static void printTable(String[] header, List body) { + int[] w = new int[header.length]; + for (int c = 0; c < header.length; c++) { + w[c] = header[c].length(); + for (String[] r : body) { + w[c] = Math.max(w[c], r[c].length()); + } + } + StringBuilder sep = new StringBuilder("+"); + for (int c : w) { + sep.append("-".repeat(c + 2)).append('+'); + } + System.out.println(sep); + System.out.println(line(header, w)); + System.out.println(sep); + for (String[] r : body) { + System.out.println(line(r, w)); + } + System.out.println(sep); + } + + private static String line(String[] cells, int[] w) { + StringBuilder sb = new StringBuilder("|"); + for (int c = 0; c < cells.length; c++) { + sb.append(' ').append(cells[c]).append(" ".repeat(w[c] - cells[c].length())).append(" |"); + } + return sb.toString(); + } + + private static long sum(AtomicLongArray arr) { + long s = 0; + for (int i = 0; i < arr.length(); i++) { + s += arr.get(i); + } + return s; + } +} From f52054357c2a98cf1ad703bd51b2fe8483ebce17 Mon Sep 17 00:00:00 2001 From: javier Date: Tue, 29 Sep 2026 14:38:22 +0200 Subject: [PATCH 53/79] sender: default --addrs to the three HA cluster nodes Was "questdb:9000", a single host that exercises no failover. Default to the cluster's three nodes so a bare run without --addrs already covers the HA path: 172.31.42.41:9000,172.31.41.35:9000,10.0.0.8:9000 These are VPC-internal addresses, reachable from a sender host inside the VPC and not from outside it. The target table is unchanged and remains hardcoded to "trades", both in sender.table(...) and in PROBE_QUERY. There is no --table flag to default. --- src/main/java/com/example/sender/CsvParallelSender.java | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index 6637f0c..df16aab 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -34,7 +34,11 @@ public class CsvParallelSender { // Defaults mirror your Python script - private static final String DEFAULT_ADDRS = "questdb:9000"; + // The HA cluster's three nodes, so a bare run without --addrs already exercises failover. + // These are VPC-internal addresses: reachable from the sender host inside the VPC, not from + // outside it. + private static final String DEFAULT_ADDRS = + "172.31.42.41:9000,172.31.41.35:9000,10.0.0.8:9000"; private static final long DEFAULT_TOTAL_EVENTS = 1_000_000L; private static final int DEFAULT_DELAY_MS = 50; // Target aggregate generation rate in rows/second across ALL workers. 0 disables rate From e4fa727a2af01c57f85586bf497164dd7125db27 Mon Sep 17 00:00:00 2001 From: javier Date: Tue, 29 Sep 2026 14:42:36 +0200 Subject: [PATCH 54/79] reader: default to all three cluster nodes so failover is exercised bash ./query_table.sh with no arguments ran against a single host and so could never fail over, which read as "works, but no failover". Two defaults were single-host: - query_table.sh handed ReadBench "${QDB_ADDR:-172.31.42.41:9000}" - ReadBench's own --addr default was "localhost:9000" Both now carry 172.31.42.41:9000,172.31.41.35:9000,10.0.0.8:9000, matching CsvParallelSender.DEFAULT_ADDRS. ReadBench already passed the value straight into addr= and the client parses a comma-separated list there, so no other change was needed. The header comment in query_table.sh was also claiming a single-host default. Overrides still work: QDB_ADDR=host:9000 ./query_table.sh for one node, or pass --addr explicitly. This should have landed with the sender's default in f520543. That change was made three minutes after the reader was committed, so both files were present, but the search was scoped to CsvParallelSender.java alone and would not have matched either of them in any case: ReadBench uses --addr, not --addrs, and query_table.sh is not a Java file. A repo-wide grep for :9000 finds all three at once. Not exercised against the cluster: build only. --- query_table.sh | 5 +++-- src/main/java/com/example/reader/ReadBench.java | 4 +++- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/query_table.sh b/query_table.sh index 2492a99..1854e0a 100755 --- a/query_table.sh +++ b/query_table.sh @@ -1,7 +1,8 @@ #!/usr/bin/env bash # Read the last N rows of a table with the Java QWP reader (com.example.reader.ReadBench). # -# With no arguments it runs the demo read against $QDB_ADDR (default 172.31.42.41:9000), +# With no arguments it runs the demo read against $QDB_ADDR (default: all three cluster +# nodes, so failover is exercised), # authenticating with $ILP_TOKEN. With arguments, they are passed straight to ReadBench: # # ./query_table.sh trades --addr h1:9000,h2:9000 --limit 200000000 \ @@ -20,7 +21,7 @@ fi if [ "$#" -eq 0 ]; then : "${ILP_TOKEN:?set ILP_TOKEN to your QuestDB token}" set -- trades \ - --addr "${QDB_ADDR:-172.31.42.41:9000}" \ + --addr "${QDB_ADDR:-172.31.42.41:9000,172.31.41.35:9000,10.0.0.8:9000}" \ --limit 200000000 \ --token "$ILP_TOKEN" \ --tls-verify unsafe_off \ diff --git a/src/main/java/com/example/reader/ReadBench.java b/src/main/java/com/example/reader/ReadBench.java index d3f99b4..a03dfea 100644 --- a/src/main/java/com/example/reader/ReadBench.java +++ b/src/main/java/com/example/reader/ReadBench.java @@ -53,7 +53,9 @@ public class ReadBench { static final class Args { String table; - String addr = "localhost:9000"; + // All three cluster nodes, matching CsvParallelSender.DEFAULT_ADDRS: a bare run + // then exercises failover. VPC-internal addresses, not reachable from outside. + String addr = "172.31.42.41:9000,172.31.41.35:9000,10.0.0.8:9000"; long limit = 10_000_000L; int readers = 1; String split = "rows"; From 57a1a34a285e2cf1fcc50242ffc3af102557217d Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 30 Sep 2026 17:09:23 +0200 Subject: [PATCH 55/79] tca: live dashboard for slippage, markouts and local min/max Four panels over QWP, redrawn once a second, each a rolling window so cost and memory stay flat however long it runs. Verified against a live demo instance. The layout is organised around the DELAY each measurement needs. Anything looking forward in time cannot be evaluated until that window has elapsed, so each query shifts back by exactly its own forward horizon: panel window forward delay slippage $now-1m .. $now none live markout +-1m $now-2m .. $now-1m +1m 1m markout past $now-1m .. $now none live min/max +-10s $now-70s .. $now-10s +10s 10s The two markout panels side by side are the point: the delayed one shows the complete curve either side of the fill, the live one only what has already happened. Same calculation, different trade between completeness and latency. Each markout panel issues two statements, and --view selects which run, so a pivot-only run never pays for the raw one. The raw statement is the query as authored. The pivot is the same calculation grouped by ecn and horizon alone, ~52 rows against ~1,700, so the displayed curve is a pure reshape of a server-side aggregate. Folding the wider result down in polars instead would need a weighted mean, sum(avg * n) / sum(n), to stop a small group counting the same as a large one; asking the server for the granularity actually displayed removes that arithmetic entirely. counterparty is not a grouping key. With 1,061 distinct counterparties and ~1,600 fills a minute it put 1 fill in nearly every cell (measured busiest: n=2), so avg() and sum() were aggregating a single observation. Raw rows are a fresh random sample each tick. The authored ORDER BY is by key, so a head() always showed the alphabetically first rows and looked frozen while the data moved underneath. Every displayed column is pre-formatted to a fixed width. polars sizes each column to its widest CURRENT value, so a window without a Currenex fill, or with smaller numbers, redrew narrower and every border shifted each second. There is no setting for this: tbl_width_chars only truncates, it never pads. Widths come from measuring the real maxima and adding headroom, since pinning at exactly the observed maximum lets one larger value grow the column again, and over-padding pushes the table past the terminal so polars truncates every cell. Flags: --symbol (applied in SQL), --interval-ms, --view, --layout, --panels, --precision, --raw-sort, --raw-rows, --rows, --minmax-rows, --left-width, --width, plus --addr/--conf and the usual auth. Not covered: the derived slippage summary is a client-side mean, correct because every row there is exactly one fill, but it is not query output and is labelled as such. --- python/tca_live.py | 517 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 517 insertions(+) create mode 100644 python/tca_live.py diff --git a/python/tca_live.py b/python/tca_live.py new file mode 100644 index 0000000..a7aec27 --- /dev/null +++ b/python/tca_live.py @@ -0,0 +1,517 @@ +#!/usr/bin/env python3 +"""Live TCA dashboard: slippage, markout curves, and local min/max around each fill. + +Redraws once a second. Every panel is a rolling window, so memory and query cost stay +flat however long it runs. + +The point of the layout is the DELAY each panel needs. Anything that looks forward in +time cannot be evaluated until that forward window has actually elapsed, so each query +shifts its window back by exactly its own forward horizon: + + panel window forward horizon delay + slippage $now-1m .. $now none none, fully live + markout +-1m $now-2m .. $now-1m +1m 1m + markout past $now-1m .. $now none none, fully live + min/max +-10s $now-70s .. $now-10s +10s 10s + +The two markout panels side by side are the interesting comparison: the delayed one +shows the complete curve either side of the fill, the live one shows only what has +already happened. Same data, different tradeoff between completeness and latency. + + ILP_TOKEN=... python tca_live.py --addr host:9000 + python tca_live.py --conf "ws::addr=localhost:9000;" + +Needs the questdb 5.0 client plus polars and pyarrow. +""" +import argparse +import os +import sys +import time +from datetime import datetime, timezone + +import polars as pl +import questdb + +# ---------------------------------------------------------------- queries +# Kept verbatim as authored, including the window literals, so what the dashboard runs +# is exactly what you would paste into the console. All reduction happens in polars. + +SLIPPAGE_SQL = """ +SELECT + t.timestamp, + t.symbol, + t.ecn, + t.counterparty, + t.side, + t.passive, + t.price, + t.quantity, + m.best_bid, + m.best_ask, + (m.best_bid + m.best_ask) / 2 AS mid, + (m.best_ask - m.best_bid) AS spread, + CASE t.side + WHEN 'buy' THEN (t.price - (m.best_bid + m.best_ask) / 2) + / ((m.best_bid + m.best_ask) / 2) * 10000 + WHEN 'sell' THEN ((m.best_bid + m.best_ask) / 2 - t.price) + / ((m.best_bid + m.best_ask) / 2) * 10000 + END AS slippage_bps, + CASE t.side + WHEN 'buy' THEN (t.price - m.best_ask) / m.best_ask * 10000 + WHEN 'sell' THEN (m.best_bid - t.price) / m.best_bid * 10000 + END AS slippage_vs_tob_bps +FROM fx_trades t +ASOF JOIN market_data m ON (symbol) +WHERE t.timestamp IN '$now-1m..$now'{sym} +ORDER BY t.timestamp +""" + +# Markout body shared by both panels; only RANGE and the window differ. +# +# Grouped by ecn and horizon only. That is 52 rows per tick (4 ecns x 13 offsets) rather than +# 20,111, and the average the server computes is the one displayed. Grouping by symbol, ecn, +# counterparty and passive then folding three of them away in polars required a weighted mean, +# sum(avg * n) / sum(n), so a counterparty with 3 fills would not count the same as one with +# 3,000. Letting the server group at the displayed granularity deletes that code entirely. +# Two markout statements per panel, deliberately. +# +# _MARKOUT_RAW_SQL is grouped by symbol, ecn, passive and horizon_sec, carrying n, +# avg_markout_bps and total_pnl. counterparty is deliberately NOT a key: with 1,061 distinct +# counterparties and ~1,600 fills a minute, including it put 1 fill in nearly every cell +# (measured busiest cell: n=2), so avg() and sum() were aggregating a single observation and +# the numbers were noise. Dropping it collapses ~10,800 cells to ~1,700 and gives each a real +# sample. +# +# _MARKOUT_PIVOT_SQL is the same calculation grouped only by ecn and horizon_sec, ~52 rows, so +# the pivot is a pure reshape of a server-side aggregate. Folding the four-key result down in +# polars instead would need a weighted mean, sum(avg * n) / sum(n), to stop a counterparty with +# 3 fills counting the same as one with 3,000. Asking the server for the granularity actually +# displayed removes that arithmetic and the chance of getting it wrong. + +_MARKOUT_BODY = """ + count() AS n, + avg( + CASE t.side + WHEN 'buy' THEN ((m.best_bid + m.best_ask) / 2 - t.price) + / t.price * 10000 + WHEN 'sell' THEN (t.price - (m.best_bid + m.best_ask) / 2) + / t.price * 10000 + END + ) AS avg_markout_bps, + sum( + CASE t.side + WHEN 'buy' THEN ((m.best_bid + m.best_ask) / 2 - t.price) + * t.quantity + WHEN 'sell' THEN (t.price - (m.best_bid + m.best_ask) / 2) + * t.quantity + END + ) AS total_pnl +FROM fx_trades t +HORIZON JOIN market_data m ON (symbol) + RANGE {rng} AS h +WHERE t.timestamp IN '{window}'{sym} +""" + +_MARKOUT_RAW_SQL = (""" +SELECT + t.symbol, + t.ecn, + t.passive, + h.offset / 1000000000 AS horizon_sec,""" + + _MARKOUT_BODY + + """GROUP BY t.symbol, t.ecn, t.passive, horizon_sec +ORDER BY t.symbol, t.ecn, t.passive, horizon_sec +""") + +_MARKOUT_PIVOT_SQL = (""" +SELECT + t.ecn, + h.offset / 1000000000 AS horizon_sec,""" + + _MARKOUT_BODY + + """GROUP BY t.ecn, horizon_sec +ORDER BY t.ecn, horizon_sec +""") + +# Per panel: the full curve is shifted a minute back so its +1m side is complete; the past-only +# curve has no forward horizon and runs right up to now. +_MARKOUT_RANGES = { + "markout-full": ("FROM -1m TO 1m STEP 10s", "$now-2m..$now-1m"), + "markout-past": ("FROM -1m TO 0 STEP 10s", "$now-1m..$now"), +} + + +MINMAX_SQL = """ +SELECT + t.symbol, + t.timestamp, + t.side, + t.price, + min(p.ask_price) AS min_ask, + max(p.bid_price) AS max_bid +FROM fx_trades t +WINDOW JOIN core_price p + ON (symbol) + RANGE BETWEEN 10 seconds PRECEDING AND 10 seconds FOLLOWING + EXCLUDE PREVAILING +WHERE t.timestamp IN '$now-70s..$now-10s'{sym} +""" + + +def build_sql(symbol): + """Statements per panel. --symbol is applied in SQL, not filtered client-side. + + Markout panels map to {"raw": ..., "pivot": ...}; the other panels to a single string. + """ + sym = f"\n AND t.symbol = '{symbol}'" if symbol else "" + out = { + "slippage": SLIPPAGE_SQL.format(sym=sym), + "minmax": MINMAX_SQL.format(sym=sym), + } + for panel, (rng, window) in _MARKOUT_RANGES.items(): + out[panel] = { + "raw": _MARKOUT_RAW_SQL.format(rng=rng, window=window, sym=sym), + "pivot": _MARKOUT_PIVOT_SQL.format(rng=rng, window=window, sym=sym), + } + return out + + +PANELS = ("slippage", "markout-full", "markout-past", "minmax") + + +# ---------------------------------------------------------------- rendering +def build_conf(args): + if args.conf: + return args.conf + tls = bool(args.token or (args.username and args.password)) + parts = [("wss" if tls else "ws") + "::addr=" + args.addr + ";"] + if args.token: + parts.append("token=" + args.token + ";") + elif args.username and args.password: + parts.append("username=" + args.username + ";password=" + args.password + ";") + if tls and args.tls_verify == "unsafe_off": + parts.append("tls_verify=unsafe_off;") + return "".join(parts) + + +def markout_curve(df): + """Pivot the horizon offsets across columns, which is how a markout curve reads. + + The server already grouped at the displayed granularity, so there is nothing to + aggregate here: one row per ecn, one column per offset. + """ + if df.height == 0: + return df + wide = df.pivot(on="horizon_sec", index="ecn", values="avg_markout_bps", + aggregate_function=None) + # Group order is not curve order; sort the offsets numerically so the row reads left to + # right from the most negative to the most positive. + horizon_cols = sorted((c for c in wide.columns if c != "ecn"), key=lambda c: int(c)) + # n is per (ecn, horizon) and every fill matches once per offset, so max across horizons + # is the fill count. sum would count fill-by-horizon pairs and, with 13 offsets, report + # 6,084 fills for an ecn that had 365. + fills = df.group_by("ecn").agg(pl.col("n").max().alias("fills")) + return (wide.select(["ecn"] + horizon_cols) + .join(fills, on="ecn", how="left") + .sort("fills", descending=True)) + + +def width_cfg(args): + """polars Config kwargs for table width: only pin it when --width was given. + + Returning {} rather than a 0 matters. tbl_width_chars=0 is not "auto", it is a zero-width + table: polars then squeezes every column to ~3 characters and a nanosecond timestamp + unfolds over twenty lines. Leaving the key out lets polars size to the terminal. + """ + return {"tbl_width_chars": args.width} if args.width > 0 else {} + + +def fixed(df, specs): + """Render selected columns as pre-padded strings so the table width stops changing. + + polars sizes each column to its widest CURRENT value, so a window without a Currenex fill, + or whose numbers all happen to be small, redraws narrower and every border shifts. There is + no config for this: tbl_width_chars only truncates, it never pads (verified). Formatting to + a fixed width here is the only thing that holds the layout still. + + specs maps column -> format spec, e.g. "<8" for a label or ">7.2f" for a right-aligned + number. Widths must match the real maxima; padding wider than the content pushes the table + past the terminal and polars then truncates every cell to a single ellipsis. + """ + exprs = [] + for col, spec in specs.items(): + if col not in df.columns: + continue + width = int("".join(c for c in spec.split(".")[0] if c.isdigit()) or 0) + blank = " " * width + exprs.append( + pl.col(col) + .map_elements(lambda v, _s=spec, _b=blank: _b if v is None else format(v, _s), + return_dtype=pl.Utf8) + .alias(col) + ) + return df.with_columns(exprs) if exprs else df + + +# Widest block seen per key.# Widest block seen per key. polars sizes a column to its widest VALUE, so when +# no Currenex fill lands in the window the ecn column loses a character and everything to the +# right of it jumps sideways. Remembering the high-water mark per panel and padding to it keeps +# the right-hand table anchored, at the cost of never shrinking back during a session. +_LEFT_WIDTH = {} + + +def side_by_side(left, right, gap=3, key=None, floor=0): + """Place two rendered blocks next to each other, line by line. + + polars pads every line of a table to the same width, so the left block only needs its own + max width to line up. The shorter block is padded with blanks so the columns stay aligned + all the way down. `key` opts the panel into the sticky width above, `floor` pins a minimum + so the right half is anchored from the very first tick rather than after the sticky width + has converged (measured: ~15 ticks, 85 -> 87 chars, then stable). + """ + lcol = left.split("\n") + rcol = right.split("\n") + width = max((len(x) for x in lcol), default=0) + if key is not None: + width = _LEFT_WIDTH[key] = max(width, _LEFT_WIDTH.get(key, 0), floor) + pad = " " * gap + rows = max(len(lcol), len(rcol)) + out = [] + for i in range(rows): + l = lcol[i] if i < len(lcol) else "" + r = rcol[i] if i < len(rcol) else "" + out.append((l.ljust(width) + pad + r).rstrip()) + return "\n".join(out) + + +def render(frames, timings, args, tick): + out = [] + now = datetime.now(timezone.utc).strftime("%H:%M:%S") + scope = args.symbol if args.symbol else "all symbols" + out.append(f"TCA live {now}Z tick {tick} refresh {args.interval_ms}ms {scope}") + out.append("") + + for name in args.panels: + df = frames.get(name) + ms = timings.get(name) + took = f"{ms:.0f}ms" if ms is not None else "n/a" + if df is None: + out.append(f"--- {name}: query failed ({took}) ---") + out.append("") + continue + + if name == "slippage": + out.append(f"--- slippage, last 1m, live [{df.height:,} fills, {took}] ---") + if df.height == 0: + out.append("(no fills in window)") + else: + if args.view in ("raw", "both"): + out.append(f" raw, the query as authored, one row per fill " + f"(last {min(args.raw_rows, df.height)} of {df.height:,}):") + padded = fixed(df.tail(args.raw_rows), { + "ecn": "<8", "side": "<4", "passive": "<5", + "price": ">9.5f", "quantity": ">13.2f", + "best_bid": ">9.5f", "best_ask": ">9.5f", "mid": ">9.5f", + "spread": ">7.5f", "slippage_bps": ">8.4f", + "slippage_vs_tob_bps": ">8.4f", + }) + with pl.Config(tbl_rows=args.raw_rows, tbl_cols=-1, + tbl_hide_dataframe_shape=True, tbl_hide_column_data_types=True, **width_cfg(args)): + out.append(str(padded)) + if args.view in ("pivot", "both"): + # A plain mean is correct here: every row is exactly one fill, so there is + # no weighting trap of the kind the markout grouping had. + out.append(" summarised by ecn (mean over fills, computed client-side):") + summary = (df.group_by("ecn") + .agg(pl.len().alias("fills"), + pl.col("slippage_bps").mean().alias("avg_bps"), + pl.col("slippage_vs_tob_bps").mean().alias("avg_tob_bps")) + .sort("fills", descending=True)) + summary = fixed(summary, {"ecn": "<8", "fills": ">5", + "avg_bps": ">7.3f", "avg_tob_bps": ">7.3f"}) + with pl.Config(tbl_rows=args.rows, + tbl_hide_dataframe_shape=True, tbl_hide_column_data_types=True, **width_cfg(args)): + out.append(str(summary)) + + elif name in ("markout-full", "markout-past"): + label = ("markout -1m..+1m, delayed 1m so the future is complete" + if name == "markout-full" + else "markout -1m..0, live, past horizons only") + out.append(f"--- {label} [{took}] ---") + wanted = [v for v in ("raw", "pivot") if args.view in (v, "both")] + blocks = [] + for variant, vdf in zip(wanted, df): + if vdf is None: + blocks.append((variant, f" {variant}: query failed")) + continue + if vdf.height == 0: + blocks.append((variant, f" {variant}: (no fills in window)")) + continue + if variant == "raw": + if args.raw_sort == "query": + shown, how = vdf, "the query's own order" + elif args.raw_sort == "n": + shown, how = vdf.sort("n", descending=True), "busiest cells first" + else: + shown = vdf.sample(min(args.raw_rows, vdf.height), shuffle=True) + how = "random sample" + head = (f" raw, grouped by symbol/ecn/passive/horizon ({how}, " + f"{min(args.raw_rows, vdf.height)} of {vdf.height:,}):") + + padded = fixed(shown.head(args.raw_rows), { + "ecn": "<8", "n": ">5", + "avg_markout_bps": f">9.{args.precision}f", + "total_pnl": f">14.{args.precision}f", + }) + with pl.Config(tbl_rows=args.raw_rows, tbl_cols=-1, float_precision=2, + tbl_hide_dataframe_shape=True, + tbl_hide_column_data_types=True, **width_cfg(args)): + blocks.append((variant, head + "\n" + str(padded))) + else: + head = (f" pivoted, a second query by ecn/horizon " + f"({vdf.height} rows), avg_markout_bps:") + curve = markout_curve(vdf) + specs = {c: f">8.{args.precision}f" for c in curve.columns + if c not in ("ecn", "fills")} + specs.update({"ecn": "<8", "fills": ">5"}) + with pl.Config(tbl_rows=args.rows, tbl_cols=-1, float_precision=2, + tbl_hide_dataframe_shape=True, + tbl_hide_column_data_types=True, **width_cfg(args)): + blocks.append((variant, head + "\n" + str(fixed(curve, specs)))) + + if args.layout == "side" and len(blocks) == 2: + out.append(side_by_side(blocks[0][1], blocks[1][1], key=name, + floor=args.left_width)) + else: + for _, block in blocks: + out.append(block) + + elif name == "minmax": + out.append(f"--- min/max +-10s around fill, delayed 10s " + f"[{df.height:,} fills, {took}] ---") + # No transformation here: this IS the query output, just the newest rows of it. + out.append(f" raw, the query as authored " + f"(last {min(args.minmax_rows, df.height)} of {df.height:,}):") + if df.height: + padded = fixed(df.tail(args.minmax_rows), { + "side": "<4", "price": ">10.5f", + "min_ask": ">10.5f", "max_bid": ">10.5f", + }) + with pl.Config(tbl_rows=args.rows, tbl_cols=-1, + tbl_hide_dataframe_shape=True, tbl_hide_column_data_types=True, + **width_cfg(args)): + out.append(str(padded)) + else: + out.append("(no fills in window)") + + # Repaint in one write: separate prints let the terminal show a half-drawn frame. + sys.stdout.write("\033[H\033[2J" + "\n".join(out) + "\n") + sys.stdout.flush() + + +# ---------------------------------------------------------------- main +def main(argv): + ap = argparse.ArgumentParser(description="Live TCA dashboard over QWP") + ap.add_argument("--addr", default="localhost:9000", help="host:port") + ap.add_argument("--conf", default=None, + help="full connect string; overrides --addr and the auth flags") + ap.add_argument("--token", default=os.environ.get("ILP_TOKEN") or None) + ap.add_argument("--username", default=None) + ap.add_argument("--password", default=None) + ap.add_argument("--tls-verify", choices=["on", "unsafe_off"], default="unsafe_off") + ap.add_argument("--interval-ms", type=int, default=1000, + help="milliseconds between redraws; default 1000. Named to match the Java " + "sender's --probe-interval-ms / --delay-ms") + ap.add_argument("--symbol", default=None, + help="restrict every panel to one symbol, e.g. EURUSD. Applied in SQL, so " + "the server does the filtering") + ap.add_argument("--rows", type=int, default=8, help="max rows per panel") + ap.add_argument("--layout", choices=["side", "stacked"], default="side", + help="with --view both, place raw and pivoted next to each other rather " + "than one above the other, to avoid vertical scrolling; needs a wide " + "terminal, see the width the header reports") + ap.add_argument("--precision", type=int, default=2, + help="decimals for the markout figures. Values render at a FIXED width, so " + "-3.25 and -12.40 occupy the same space and the table stops resizing " + "as a value crosses a power of ten; default 2") + ap.add_argument("--left-width", type=int, default=0, + help="minimum width for the left half in a side-by-side panel. Default 0 " + "lets it find its own high-water mark, which settles after ~15 ticks; " + "set ~90 to anchor the right half from the first frame") + ap.add_argument("--minmax-rows", type=int, default=5, + help="rows for the min/max panel, the tallest table; default 5") + ap.add_argument("--view", choices=["pivot", "raw", "both"], default="both", + help="the query's own long-format output, the derived/pivoted view, or " + "both; default both, rendered side by side so it still fits") + ap.add_argument("--raw-sort", choices=["sample", "n", "query"], default="sample", + help="which raw markout rows to show. 'sample' takes a fresh random draw " + "each tick; 'n' the busiest cells; 'query' the statement's own ORDER BY. " + "Default sample, because the authored grouping puts ~1 fill in every " + "(symbol, ecn, counterparty, passive, horizon) cell (measured max n=2), " + "so both other orderings keep landing on the same alphabetically first " + "rows and look frozen while the data moves underneath") + ap.add_argument("--raw-rows", type=int, default=6, + help="rows of raw markout output to show when --view includes raw; kept " + "small because the full result is 28 rows past-only and 52 full") + ap.add_argument("--width", type=int, default=0, + help="force a table drawing width in chars. Default 0 leaves it to polars, " + "which sizes to the terminal. Note this is the width tables are DRAWN " + "at, not a cap: setting it pads narrow tables out to that width too") + ap.add_argument("--panels", default=",".join(PANELS), + help="comma-separated subset of: " + ", ".join(PANELS)) + args = ap.parse_args(argv) + + args.panels = [p.strip() for p in args.panels.split(",") if p.strip()] + bad = [p for p in args.panels if p not in PANELS] + if bad: + print("unknown panel(s): " + ", ".join(bad), file=sys.stderr) + return 2 + + sql = build_sql(args.symbol) + + conf = build_conf(args) + tick = 0 + with questdb.connect(conf) as db: + # One pinned connection for the whole session: these are several queries in a + # row, which is exactly what a reader lease is for. + with db.reader() as reader: + while True: + started = time.monotonic() + frames, timings = {}, {} + for name in args.panels: + spec = sql[name] + # Markout panels carry {"raw","pivot"}; run only what --view will draw so a + # --view pivot run does not pay for the 20,000-row authored query. + wanted = ([spec] if isinstance(spec, str) + else [spec[v] for v in ("raw", "pivot") + if args.view in (v, "both")]) + t0 = time.monotonic() + got = [] + for stmt in wanted: + try: + got.append(reader.query(stmt.strip()).to_polars()) + except Exception as e: # noqa: BLE001 + # Keep drawing. A dashboard that dies on one bad tick is worse + # than one that shows which panel is failing and why. + got.append(None) + print(f"[{name}] {e}", file=sys.stderr) + frames[name] = got[0] if isinstance(spec, str) else got + timings[name] = (time.monotonic() - t0) * 1000.0 + tick += 1 + render(frames, timings, args, tick) + + # Hold the cadence, and say so when the queries cannot keep up rather + # than silently drifting. + spent_ms = (time.monotonic() - started) * 1000.0 + if spent_ms < args.interval_ms: + time.sleep((args.interval_ms - spent_ms) / 1000.0) + else: + print(f"[warn] tick took {spent_ms:.0f}ms, longer than the " + f"{args.interval_ms}ms refresh", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main(sys.argv[1:])) + except KeyboardInterrupt: + sys.exit(130) From b9df9a7d7a0e9af95598a22d7e5d93c38f377232 Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 30 Sep 2026 18:03:29 +0200 Subject: [PATCH 56/79] tca: always interruptible, recover from a lost node, document every script Ctrl+C now works in any state. The polling loop moved to a daemon thread and the main thread only ever waits in worker.join(0.2). Python can raise KeyboardInterrupt between bytecodes but never inside the client's blocking native calls, so a loop that queries on the main thread is unkillable for as long as one blocks, and during an outage every tick blocks on a connect, which is exactly when you want to stop it. Tuning the sleep does not fix this; keeping the main thread out of native code does. Exit uses os._exit so shutdown cannot block in the same native code it is escaping. Verified through a pty, sending a real ^C: "interrupted", exit code 130. Three earlier harnesses gave false verdicts before that, so the final one was validated against a trivial sleep loop first: a background-launched process inherits SIGINT set to SIG_IGN and Python preserves it, and a harness that blocks in os.read never reports. Also confirmed the client leaves the SIGINT handler alone (still default_int_handler after connect), so nothing else was interfering. The reader lease is rebuilt after any failure rather than held for the whole run. Reader failover is on by default and to_polars() replays a mid-query failover transparently, so a node dying mid-result is invisible. What is not automatic is the lease: a terminal rejection is latched by the connection and raised by the next call on it, so a lease held for the process lifetime can be poisoned permanently. Verified by killing and restarting a TCP proxy under a running dashboard: it reported the reason each tick and resumed on its own. Note that transparent replay depends on to_polars(); the iter_* consumers raise FailoverWouldDuplicate instead, so switching a panel to streaming would need that handled. sender_pool_min=0, because this tool only reads and connect() otherwise pre-opens a store-and-forward sender it never uses. The reader pool is still opened eagerly, so an unreachable server at startup fails fast rather than launching a dashboard that can show nothing. README: nine scripts were undocumented, not just this one. Added rows for tca_live, blotter, web_blotter, backfill and the four slippage variants, plus a section for the dashboard covering the window/delay table, the two-statements-per-panel design, the flags and the terminal-width caveat. run_blotter.sh and start_sending.sh are still uncovered; the table is scoped to Python entry points. --- python/README.md | 48 +++++++++++++++++ python/tca_live.py | 127 ++++++++++++++++++++++++++++++++------------- 2 files changed, 139 insertions(+), 36 deletions(-) diff --git a/python/README.md b/python/README.md index 64f3610..548dbb0 100644 --- a/python/README.md +++ b/python/README.md @@ -23,6 +23,18 @@ This port has grown a few scripts. Pick by what you need: | `read_bench.py` | Measuring read/scan throughput (rows/s) of a table | Streaming `iter_arrow()` egress. | | `enrich_polars_demo.py` | Read a table as polars, enrich, write it back as polars | Streaming read + columnar write. | | `dataframe_demo.py` | pandas/polars ingestion + egress round-trip showcase | `db.dataframe` (pandas + polars), `db.execute`, dual egress. | +| `backfill.py` | Filling a historical window at a target row density | Columnar write over a chosen time range. | +| `tca_live.py` | **Live TCA dashboard** - slippage, markout curves and local min/max, redrawn every second | Four rolling-window queries per tick; `to_polars()`. | +| `blotter.py` | Terminal blotter: poll a table or live view and redraw in place | Polling query, in-place terminal redraw. | +| `web_blotter.py` | The same blotter served as a live table in a browser | Tiny local HTTP server over the same queries. | +| `slippage.py` | Per-fill slippage vs mid and top-of-book, as one DataFrame | `to_polars()`, materialise-whole. | +| `slippage_stream.py` | The same, printed batch by batch as it streams | `iter_polars()`, low peak memory. | +| `slippage_parallel.py` | The same over N connections, merged into one DataFrame | N connections, concatenated client-side. | +| `slippage_parallel_stream.py` | N connections, printed in timestamp order as they arrive | Bounded per-worker queues, ordered drain. | + +The four `slippage*.py` scripts are the same query with different egress shapes, and exist to +contrast them: buffer-everything vs streaming, one connection vs several. `tca_live.py` is the +one to reach for if you want a dashboard rather than a one-shot result. **Row-by-row vs columnar.** `sender.row()` in a Python loop is the *slowest* path: every cell crosses the Python/Cython boundary under the GIL (the client's own perf notes call it "~16x @@ -174,6 +186,42 @@ Enterprise-only (needs SYSTEM ADMIN); on OSS the probe prints reads) and fails over across the `--addrs` hosts. (5.0 also exposes `db.server_info()` for the handshake role/epoch/capabilities, should a probe want the role without `switch status`.) +## Live TCA dashboard (`tca_live.py`) + +Four panels over QWP, redrawn once a second, each a rolling window so cost and memory stay +flat however long it runs. + +```bash +python tca_live.py --conf "ws::addr=localhost:9000;" +ILP_TOKEN=... python tca_live.py --addr 172.31.42.41:9000 --symbol EURUSD +``` + +The layout is organised around the **delay** each measurement needs. Anything looking forward +in time cannot be evaluated until that window has elapsed, so each query shifts its window +back by exactly its own forward horizon: + +| Panel | Window | Forward horizon | Delay | +| --- | --- | --- | --- | +| slippage | `$now-1m .. $now` | none | live | +| markout `-1m..+1m` | `$now-2m .. $now-1m` | +1m | 1m | +| markout `-1m..0` | `$now-1m .. $now` | none | live | +| min/max +-10s | `$now-70s .. $now-10s` | +10s | 10s | + +The two markout panels side by side are the point: the delayed one shows the complete curve +either side of the fill, the live one only what has already happened. + +Each markout panel issues **two** statements and `--view` picks which run, so a pivot-only run +never pays for the raw one. The raw statement is the query as authored; the pivot is the same +calculation grouped by `ecn` and horizon alone, so the curve on screen is a pure reshape of a +server-side aggregate rather than something recomputed in Python. + +Useful flags: `--symbol` (applied in SQL), `--interval-ms` (default 1000), `--view +pivot|raw|both`, `--panels`, `--precision`, `--raw-rows`, `--layout side|stacked`. + +**Terminal width matters.** With `--view both` the raw and pivoted tables sit side by side, so +the `+-1m` panel renders at ~400 columns. On a narrower terminal use `--view pivot` or +`--panels markout-past`. + ## Dataframe demo (pandas / polars ingestion + egress) `dataframe_demo.py` shows the columnar paths the row sender does not: diff --git a/python/tca_live.py b/python/tca_live.py index a7aec27..e334031 100644 --- a/python/tca_live.py +++ b/python/tca_live.py @@ -26,6 +26,7 @@ import argparse import os import sys +import threading import time from datetime import datetime, timezone @@ -177,6 +178,10 @@ def build_sql(symbol): PANELS = ("slippage", "markout-full", "markout-past", "minmax") +# Smallest pause at the end of every tick, even one that overran its interval, so the +# interpreter always gets a slice in which to deliver SIGINT. +MIN_YIELD_MS = 50 + # ---------------------------------------------------------------- rendering def build_conf(args): @@ -190,6 +195,12 @@ def build_conf(args): parts.append("username=" + args.username + ";password=" + args.password + ";") if tls and args.tls_verify == "unsafe_off": parts.append("tls_verify=unsafe_off;") + # Read-only tool, so never open an ingest connection: connect() otherwise pre-opens a + # store-and-forward sender, which this never uses. The reader pool is still opened + # eagerly, so an unreachable server at startup fails fast rather than launching a + # dashboard that cannot show anything. Recovery once running is handled by rebuilding + # the lease. + parts.append("sender_pool_min=0;") return "".join(parts) @@ -409,7 +420,8 @@ def render(frames, timings, args, tick): # ---------------------------------------------------------------- main -def main(argv): +def parse_args(argv): + """CLI definition, split out so main() only supervises the worker thread.""" ap = argparse.ArgumentParser(description="Live TCA dashboard over QWP") ap.add_argument("--addr", default="localhost:9000", help="host:port") ap.add_argument("--conf", default=None, @@ -459,59 +471,102 @@ def main(argv): ap.add_argument("--panels", default=",".join(PANELS), help="comma-separated subset of: " + ", ".join(PANELS)) args = ap.parse_args(argv) + args.panels = [x.strip() for x in args.panels.split(",") if x.strip()] + return args - args.panels = [p.strip() for p in args.panels.split(",") if p.strip()] - bad = [p for p in args.panels if p not in PANELS] - if bad: - print("unknown panel(s): " + ", ".join(bad), file=sys.stderr) - return 2 - - sql = build_sql(args.symbol) +def run_dashboard(args, sql): + """The polling loop. Runs on a DAEMON thread so the main thread can stay responsive.""" conf = build_conf(args) tick = 0 with questdb.connect(conf) as db: - # One pinned connection for the whole session: these are several queries in a - # row, which is exactly what a reader lease is for. - with db.reader() as reader: - while True: - started = time.monotonic() - frames, timings = {}, {} + # The lease is REBUILT after any failure rather than held for the whole run. + # + # Reader failover is on by default (8 attempts, 30s budget) and to_polars() replays a + # mid-query failover transparently, so a node dying mid-result is invisible here. What + # is not automatic is the lease itself: a terminal rejection is latched by the + # connection and "raised by the next call on the affected lease", so a lease held for + # the process lifetime can be poisoned permanently and every later tick then fails on a + # connection that will never recover. Dropping it and taking a fresh one is the only + # way back, the same lesson as the Java probe's query client. + reader = None + while True: + started = time.monotonic() + frames, timings = {}, {} + try: + if reader is None: + reader = db.reader() for name in args.panels: spec = sql[name] # Markout panels carry {"raw","pivot"}; run only what --view will draw so a - # --view pivot run does not pay for the 20,000-row authored query. + # --view pivot run does not pay for the larger authored query. wanted = ([spec] if isinstance(spec, str) else [spec[v] for v in ("raw", "pivot") if args.view in (v, "both")]) t0 = time.monotonic() - got = [] - for stmt in wanted: - try: - got.append(reader.query(stmt.strip()).to_polars()) - except Exception as e: # noqa: BLE001 - # Keep drawing. A dashboard that dies on one bad tick is worse - # than one that shows which panel is failing and why. - got.append(None) - print(f"[{name}] {e}", file=sys.stderr) + got = [reader.query(stmt.strip()).to_polars() for stmt in wanted] frames[name] = got[0] if isinstance(spec, str) else got timings[name] = (time.monotonic() - t0) * 1000.0 - tick += 1 + except KeyboardInterrupt: + raise # BaseException, so not caught below; re-raised for clarity + except Exception as e: # noqa: BLE001 + # Keep drawing: a dashboard that dies on one bad tick is worse than one that + # says which tick failed and why, then reconnects. + print(f"[tick {tick + 1}] {e}; reconnecting", file=sys.stderr) + try: + reader.close() + except Exception: # noqa: BLE001 + pass # a dead lease often will not close cleanly + reader = None + + tick += 1 + if frames: render(frames, timings, args, tick) - # Hold the cadence, and say so when the queries cannot keep up rather - # than silently drifting. - spent_ms = (time.monotonic() - started) * 1000.0 - if spent_ms < args.interval_ms: - time.sleep((args.interval_ms - spent_ms) / 1000.0) - else: - print(f"[warn] tick took {spent_ms:.0f}ms, longer than the " - f"{args.interval_ms}ms refresh", file=sys.stderr) + # Hold the cadence, and say so when the queries cannot keep up rather than + # silently drifting. + # + # The floor is not cosmetic. Python can only raise KeyboardInterrupt between + # bytecodes, never inside the client's blocking native connect. When a tick + # overran we used to skip the sleep entirely and re-enter native code + # immediately, so during an outage (every tick overruns, each burning a connect + # timeout) there was no window for the signal to land and Ctrl+C did nothing. + # Always yielding a slice guarantees one. + spent_ms = (time.monotonic() - started) * 1000.0 + if spent_ms >= args.interval_ms: + print(f"[warn] tick took {spent_ms:.0f}ms, longer than the " + f"{args.interval_ms}ms refresh", file=sys.stderr) + time.sleep(max(MIN_YIELD_MS, args.interval_ms - spent_ms) / 1000.0) return 0 -if __name__ == "__main__": +def main(argv): + args = parse_args(argv) + bad = [p for p in args.panels if p not in PANELS] + if bad: + print("unknown panel(s): " + ", ".join(bad), file=sys.stderr) + return 2 + + sql = build_sql(args.symbol) + + # The query work runs on a daemon thread and the main thread only ever waits on join(), + # which is interruptible. Python can only deliver KeyboardInterrupt between bytecodes, + # never inside the client's blocking native calls, so a loop that does its own querying + # on the main thread is unkillable for as long as that call blocks. During an outage + # every tick blocks on a connect, which is exactly when you most want to stop it. + worker = threading.Thread(target=run_dashboard, args=(args, sql), daemon=True) + worker.start() try: - sys.exit(main(sys.argv[1:])) + while worker.is_alive(): + worker.join(0.2) except KeyboardInterrupt: - sys.exit(130) + print("\ninterrupted", file=sys.stderr) + # _exit, not sys.exit: a normal exit would try to close the handle and could block + # in the same native code we are trying to escape. The daemon thread dies with us. + sys.stderr.flush() + os._exit(130) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) From 327940689d399ba225441ba904a3ec3d2422139d Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 30 Sep 2026 19:37:55 +0200 Subject: [PATCH 57/79] js: browser TCA dashboard over the Node QWP client A Node server issues the same four TCA queries as tca_live.py over QWP and serves the results to a page that renders them as a grid of panels. The browser cannot open the QWP socket itself yet: @questdb/browser-client is unpublished, and the WebSocket upgrade requires Origin == Host, which a separate dev-server port cannot satisfy. data.mjs is therefore the only module that knows QuestDB exists, so swapping in the browser client later touches one file. - symbol dropdown from a LATEST ON query, with ALL as the default, filtering every panel in SQL rather than client side - per-panel SQL view, fetched from the same sqlFor() the query path uses so what is shown cannot drift from what ran - markout P&L curves as inline SVG, one line per ECN, with a crosshair and a tooltip ranking venues at the hovered horizon; the hovered horizon is held outside the chart because each tick rebuilds it - refresh dropdown, 250ms to 2000ms plus Off, which holds the last frame at full opacity instead of dimming it --- js/.gitignore | 1 + js/data.mjs | 82 ++++++++++++++++ js/package-lock.json | 31 ++++++ js/package.json | 13 +++ js/public/app.js | 222 +++++++++++++++++++++++++++++++++++++++++++ js/public/chart.js | 207 ++++++++++++++++++++++++++++++++++++++++ js/public/index.html | 70 ++++++++++++++ js/queries.mjs | 89 +++++++++++++++++ js/server.mjs | 107 +++++++++++++++++++++ 9 files changed, 822 insertions(+) create mode 100644 js/.gitignore create mode 100644 js/data.mjs create mode 100644 js/package-lock.json create mode 100644 js/package.json create mode 100644 js/public/app.js create mode 100644 js/public/chart.js create mode 100644 js/public/index.html create mode 100644 js/queries.mjs create mode 100644 js/server.mjs diff --git a/js/.gitignore b/js/.gitignore new file mode 100644 index 0000000..c2658d7 --- /dev/null +++ b/js/.gitignore @@ -0,0 +1 @@ +node_modules/ diff --git a/js/data.mjs b/js/data.mjs new file mode 100644 index 0000000..7c401de --- /dev/null +++ b/js/data.mjs @@ -0,0 +1,82 @@ +// The DATA layer: the only file that knows QuestDB exists. +// +// Today it runs the queries here in Node, over QWP, using @questdb/nodejs-client. When +// @questdb/browser-client is installable and the server carries the browser-upgrade patch, +// this whole file is replaced by a pass-through proxy for /read/v1 and /exec, the query moves +// into the page, and neither server.mjs nor public/ has to change. +import { connectQwpNodeClient, QwpEgressQueryError } from "@questdb/nodejs-client"; +import { PANELS, sqlFor, symbolsSql } from "./queries.mjs"; + +export class Data { + constructor(conf) { + this.conf = conf; + this.db = null; + } + + async connect() { + // Eager: an unreachable server should fail at startup rather than serving a dashboard + // that can never show anything. + this.db = await connectQwpNodeClient(this.conf); + } + + /** + * Run every panel once and return {panel: {columns, rows, ms}} plus any per-panel error. + * + * A fresh lease per tick, not one held for the process lifetime: a terminal rejection is + * latched by the connection and raised by the next call on that lease, so a long-lived + * lease can be poisoned permanently. This is the same lesson the Python dashboard learned. + */ + async tick(symbol) { + const out = {}; + const lease = await this.db.borrowQuery(); + try { + for (const panel of PANELS) { + const started = performance.now(); + try { + out[panel] = { ...(await this.#run(lease, sqlFor(panel, symbol))), + ms: performance.now() - started }; + } catch (error) { + if (!(error instanceof QwpEgressQueryError)) throw error; + // A rejected statement is this panel's problem, not the dashboard's: report it and + // keep the other three drawing. + out[panel] = { error: `status=${error.status} ${error.message}`, + ms: performance.now() - started }; + } + } + } finally { + await lease.close(); + } + return out; + } + + /** Symbols active in the last 30s, sorted, for the filter dropdown. */ + async symbols() { + const lease = await this.db.borrowQuery(); + try { + const { rows } = await this.#run(lease, symbolsSql()); + return rows.map((r) => String(r[0])).sort(); + } finally { + await lease.close(); + } + } + + async #run(lease, sql) { + const query = await lease.query(sql); + const rows = []; + let columns = null; + for await (const batch of query) { + // batch.columns is a PROPERTY, an array of QwpResultColumn, each extending + // QwpResultColumnSchema so it carries .name. Taking the names off the batch means the + // page renders whatever the SQL selected, with no schema hardcoded anywhere. + columns ??= batch.columns.map((c) => c.name); + for (const row of batch.rows()) rows.push(row); + } + await query.completion; + return { columns, rows }; + } + + async close() { + if (this.db) await this.db.close(); + this.db = null; + } +} diff --git a/js/package-lock.json b/js/package-lock.json new file mode 100644 index 0000000..48d1c46 --- /dev/null +++ b/js/package-lock.json @@ -0,0 +1,31 @@ +{ + "name": "questdb-tca-web", + "version": "1.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "questdb-tca-web", + "version": "1.0.0", + "dependencies": { + "@questdb/nodejs-client": "file:../../../questdb/nodejs-questdb-client/packages/nodejs-client" + } + }, + "../../../questdb/nodejs-questdb-client/packages/nodejs-client": { + "name": "@questdb/nodejs-client", + "version": "5.0.0", + "license": "Apache-2.0", + "dependencies": { + "undici": "^7.8.0", + "ws": "^8.21.3" + }, + "engines": { + "node": ">=20.18.1" + } + }, + "node_modules/@questdb/nodejs-client": { + "resolved": "../../../questdb/nodejs-questdb-client/packages/nodejs-client", + "link": true + } + } +} diff --git a/js/package.json b/js/package.json new file mode 100644 index 0000000..9b2ef14 --- /dev/null +++ b/js/package.json @@ -0,0 +1,13 @@ +{ + "name": "questdb-tca-web", + "version": "1.0.0", + "private": true, + "type": "module", + "description": "Browser TCA dashboard: slippage, markout curves and local min/max over QWP", + "scripts": { + "start": "node server.mjs" + }, + "dependencies": { + "@questdb/nodejs-client": "file:../../../questdb/nodejs-questdb-client/packages/nodejs-client" + } +} diff --git a/js/public/app.js b/js/public/app.js new file mode 100644 index 0000000..6fd8f56 --- /dev/null +++ b/js/public/app.js @@ -0,0 +1,222 @@ +// The PAGE. Renders whatever the API returns: column names come from the server, so the SQL +// can change without touching this file. +// +// When @questdb/browser-client is installable, this is where the QWP session goes: replace +// fetchTick() with connectQwpBrowserEgress({url: new URL("/read/v1", location.href)}) and +// iterate the batches here instead. Nothing else in the page changes. +import { pnlChart, pnlLegend } from "/chart.js"; + +const PANELS = [ + ["slippage", "slippage, last 1m", "live"], + ["markout-full", "markout -1m..+1m", "delayed 1m so the future is complete"], + ["markout-past", "markout -1m..0", "live, past horizons only"], + ["minmax", "min/max +-10s around fill", "delayed 10s"], +]; + +const params = new URLSearchParams(location.search); +let intervalMs = Number(params.get("interval") ?? 1000); +const maxRows = Number(params.get("rows") ?? 10); +// Aggregate results are 28 and 52 rows; show enough to span every venue rather +// than a single one. +const rawRows = Number(params.get("rawrows") ?? 16); + +const panelsEl = document.getElementById("panels"); +const chartsEl = document.getElementById("charts"); +const metaEl = document.getElementById("meta"); +const symEl = document.getElementById("sym"); +const refreshEl = document.getElementById("refresh"); + +let symbol = params.get("symbol") ?? ""; +const sqlOpen = new Set(); // panels whose SQL is expanded, kept across redraws +const sqlText = new Map(); + +const fmt = (v) => { + if (v === null || v === undefined) return "null"; + if (typeof v === "number") return Number.isInteger(v) ? String(v) : v.toFixed(2); + // Nanosecond epochs arrive as strings (BigInt is not JSON-serialisable); show the time part. + if (typeof v === "string" && /^\d{16,19}$/.test(v)) { + return new Date(Number(v) / 1e6).toISOString().slice(11, 23); + } + if (typeof v === "string" && /^\d{4}-\d{2}-\d{2}T/.test(v)) return v.slice(11, 23); + return String(v); +}; + +const esc = (s) => String(s).replace(/[&<>]/g, (c) => ({ "&": "&", "<": "<", ">": ">" }[c])); + +// Reshape (ecn, horizon_sec, n, avg_markout_bps, total_pnl) into a markout curve: one row per +// ecn, one column per horizon. A pure reshape of rows already fetched, so it costs no extra +// query and the average shown is the one the server produced. +// +// fills is the MAX of n across horizons, not the sum. HORIZON JOIN matches every fill once per +// offset, so summing counts fill-by-horizon pairs and would report several times the real count. +function pivot(columns, rows) { + const ix = Object.fromEntries(columns.map((c, i) => [c, i])); + const horizons = [...new Set(rows.map((r) => Number(r[ix.horizon_sec])))].sort((a, b) => a - b); + const byEcn = new Map(); + for (const row of rows) { + const ecn = row[ix.ecn]; + if (!byEcn.has(ecn)) byEcn.set(ecn, { cells: new Map(), fills: 0 }); + const e = byEcn.get(ecn); + e.cells.set(Number(row[ix.horizon_sec]), row[ix.avg_markout_bps]); + e.fills = Math.max(e.fills, Number(row[ix.n])); + } + return { + columns: ["ecn", ...horizons.map(String), "fills"], + rows: [...byEcn.entries()] + .sort((a, b) => b[1].fills - a[1].fills) + .map(([ecn, e]) => [ecn, ...horizons.map((h) => e.cells.get(h) ?? null), e.fills]), + }; +} + +// `newest` picks which end of the result to show, and it is not cosmetic. +// time-ordered panels (slippage, minmax) are ORDER BY timestamp ASC, so the TAIL is the +// live end; taking the head showed fills from a minute ago and looked frozen. +// aggregate panels are ORDER BY ecn, horizon, so there is no "newest" at all, and taking +// the tail showed only the alphabetically last venue: one ECN instead of four. +function table(columns, rows, cap = maxRows, newest = true) { + if (!rows.length) return "

(no rows in window)

"; + const head = columns.map((c) => ``).join(""); + const shown = newest ? rows.slice(-cap) : rows.slice(0, cap); + const body = shown.map((row) => + "" + row.map((v) => { + const cls = typeof v === "number" && !Number.isInteger(v) + ? (v < 0 ? " class=neg" : " class=pos") : ""; + return `${esc(fmt(v))}`; + }).join("") + "").join(""); + const more = rows.length > cap + ? `` : ""; + return `
${esc(c)}
${rows.length.toLocaleString()} rows, showing ${newest ? "newest" : "first"} ` + + `${cap}
${more}${head}${body}
`; +} + +function render(payload) { + panelsEl.innerHTML = PANELS.map(([key, title, note]) => { + const p = payload[key]; + if (!p) return ""; + const ms = p.ms !== undefined ? ` ${p.ms.toFixed(0)}ms` : ""; + let body; + if (p.error) { + body = `

${esc(p.error)}

`; + } else if (key.startsWith("markout") && p.rows?.length) { + // Curve first because it is the readable form; the query's own rows underneath so the + // reshape can be checked against them rather than taken on trust. + const curve = pivot(p.columns, p.rows); + body = `
pivoted: avg_markout_bps by horizon (seconds)
` + + table(curve.columns, curve.rows, curve.rows.length) + + `
raw, as the query returns it (${p.rows.length} rows)
` + + table(p.columns ?? [], p.rows ?? [], rawRows, false); + } else { + body = table(p.columns ?? [], p.rows ?? []); + } + const sql = sqlOpen.has(key) + ? `
${esc(sqlText.get(key) ?? "loading...")}
` : ""; + return `
+ +

${title} ${note}${ms}

${body}${sql}
`; + }).join(""); +} + +// The P&L curves live in their own full-width row at the bottom, so the two markout panels +// can be compared side by side at the same scale rather than squeezed into the grid cells. +function renderCharts(payload) { + chartsEl.innerHTML = ""; + for (const [key, title, note] of PANELS) { + if (!key.startsWith("markout")) continue; + const p = payload[key]; + if (!p?.rows?.length || p.error) continue; + const svg = pnlChart(p.columns, p.rows, { key }); + if (!svg) continue; + const card = document.createElement("section"); + const head = document.createElement("h2"); + head.innerHTML = `${title} P&L ${note} · total_pnl by horizon (s)`; + card.append(head, pnlLegend(p.columns, p.rows), svg); + chartsEl.append(card); + } +} + +// One delegated listener: render() replaces the whole grid each tick, so per-button handlers +// would be rebound constantly and lost mid-click. +panelsEl.addEventListener("click", async (ev) => { + const btn = ev.target.closest(".sqlbtn"); + if (!btn) return; + const key = btn.dataset.panel; + if (sqlOpen.has(key)) { + sqlOpen.delete(key); + } else { + sqlOpen.add(key); + // Fetched from the server, built by the same sqlFor() the query path uses, so what is + // displayed cannot drift from what actually ran. + const qs = new URLSearchParams({ panel: key, ...(symbol ? { symbol } : {}) }); + const res = await fetch(`/api/sql?${qs}`); + const body = await res.json(); + sqlText.set(key, body.sql ?? body.error ?? "unavailable"); + } + await tick(); +}); + +symEl.addEventListener("change", async () => { + symbol = symEl.value; + sqlText.clear(); // the statements embed the filter, so they must be refetched + for (const key of sqlOpen) { + const qs = new URLSearchParams({ panel: key, ...(symbol ? { symbol } : {}) }); + sqlText.set(key, (await (await fetch(`/api/sql?${qs}`)).json()).sql ?? ""); + } + await tick(); +}); + +async function loadSymbols() { + try { + const list = await (await fetch("/api/symbols")).json(); + if (!Array.isArray(list)) return; + const keep = symbol; + symEl.innerHTML = `` + + list.map((s) => ``).join(""); + symEl.value = keep; // a refresh must not silently reset the user's choice + } catch { /* leave the previous list in place */ } +} + +async function tick() { + const qs = symbol ? `?symbol=${encodeURIComponent(symbol)}` : ""; + try { + const res = await fetch(`/api/tick${qs}`); + const payload = await res.json(); + if (!res.ok) throw new Error(payload.error ?? res.statusText); + render(payload); + renderCharts(payload); + panelsEl.classList.remove("stale"); + const cadence = intervalMs ? `refresh ${intervalMs}ms` : "refresh off"; + metaEl.textContent = `${new Date().toISOString().slice(11, 19)}Z ${cadence}`; + } catch (error) { + // Dim rather than blank: the last good frame greyed out is more useful during an outage + // than an empty page. + panelsEl.classList.add("stale"); + metaEl.innerHTML = `${esc(error.message)}, retrying`; + } +} + +// One timer, replaced whenever the cadence changes. Off (0) clears it and leaves the last +// frame on screen at full opacity: paused is a choice, not an outage, so it must not look +// like one. +let timer = null; +function applyInterval() { + if (timer !== null) clearInterval(timer); + timer = intervalMs ? setInterval(tick, intervalMs) : null; +} + +refreshEl.addEventListener("change", async () => { + intervalMs = Number(refreshEl.value); + applyInterval(); + if (intervalMs) await tick(); // a faster cadence should show immediately + else metaEl.textContent = `${new Date().toISOString().slice(11, 19)}Z refresh off`; +}); + +// ?interval= still wins on load; snap the control to it, falling back to 1000ms when the +// query string asks for a cadence the dropdown does not offer. +const offered = [...refreshEl.options].map((o) => Number(o.value)); +if (!offered.includes(intervalMs)) intervalMs = 1000; +refreshEl.value = String(intervalMs); + +await loadSymbols(); +await tick(); +applyInterval(); +setInterval(loadSymbols, 30_000); // instruments come and go; keep the list current diff --git a/js/public/chart.js b/js/public/chart.js new file mode 100644 index 0000000..8ed1e3d --- /dev/null +++ b/js/public/chart.js @@ -0,0 +1,207 @@ +// Markout P&L curves as inline SVG. No chart library: the whole thing is a handful of +// polylines, and a dependency would be more code than the chart. +// +// Palette is categorical slots 1-4 of the reference dark theme, validated against this +// dashboard's surface (#161a21) rather than assumed: worst adjacent pair is +// #c98500 <-> #199e70 at dE 8.4 protan / 19.8 normal vision, above both thresholds. +const SERIES = ["#3987e5", "#d95926", "#199e70", "#c98500"]; + +// Colour follows the ENTITY, not its rank. The tables sort by fill count and the symbol +// filter changes which venues appear, so assigning by array index would repaint the +// survivors every time the ordering shifted. This map is stable for the page's lifetime. +const ecnColour = new Map(); +function colourFor(ecn) { + if (!ecnColour.has(ecn)) ecnColour.set(ecn, SERIES[ecnColour.size % SERIES.length]); + return ecnColour.get(ecn); +} + +const NS = "http://www.w3.org/2000/svg"; +const el = (name, attrs = {}) => { + const node = document.createElementNS(NS, name); + for (const [k, v] of Object.entries(attrs)) node.setAttribute(k, v); + return node; +}; + +// P&L is money: thousands separators, and a compact form once it runs to millions. +const money = (v) => + Math.abs(v) >= 1e6 ? `${(v / 1e6).toFixed(2)}M` + : Math.abs(v) >= 1e3 ? `${(v / 1e3).toFixed(1)}k` + : v.toFixed(0); + +/** + * Build the P&L curve for one markout panel straight from the raw rows. + * + * Plots total_pnl against horizon, one line per ECN. total_pnl is what the query already + * computed per (ecn, horizon), so nothing is recomputed here: this is the same number the + * table shows, drawn. + */ +export function pnlChart(columns, rows, { width = 560, height = 210, key = "" } = {}) { + const ix = Object.fromEntries(columns.map((c, i) => [c, i])); + const horizons = [...new Set(rows.map((r) => Number(r[ix.horizon_sec])))].sort((a, b) => a - b); + if (horizons.length < 2) return null; + + const byEcn = new Map(); + for (const row of rows) { + const ecn = row[ix.ecn]; + if (!byEcn.has(ecn)) byEcn.set(ecn, new Map()); + byEcn.get(ecn).set(Number(row[ix.horizon_sec]), Number(row[ix.total_pnl])); + } + + const pad = { t: 10, r: 12, b: 26, l: 62 }; + const plotW = width - pad.l - pad.r; + const plotH = height - pad.t - pad.b; + + const values = [...byEcn.values()].flatMap((m) => [...m.values()]).filter(Number.isFinite); + if (!values.length) return null; + // Always include zero: a P&L curve is read against break-even, so a range that excludes + // it would hide whether a venue is up or down. + let lo = Math.min(0, ...values); + let hi = Math.max(0, ...values); + if (lo === hi) { lo -= 1; hi += 1; } + const padY = (hi - lo) * 0.08; + lo -= padY; hi += padY; + + const x = (h) => pad.l + ((h - horizons[0]) / (horizons.at(-1) - horizons[0])) * plotW; + const y = (v) => pad.t + (1 - (v - lo) / (hi - lo)) * plotH; + + const svg = el("svg", { + width: "100%", viewBox: `0 0 ${width} ${height}`, + role: "img", "aria-label": "Markout P&L by horizon, one line per ECN", + }); + + // Recessive chrome: solid hairlines one shade off the surface, never dashed. + for (let i = 0; i <= 4; i++) { + const gy = pad.t + (plotH * i) / 4; + svg.append(el("line", { x1: pad.l, x2: pad.l + plotW, y1: gy, y2: gy, + stroke: "#242a33", "stroke-width": 1 })); + const gv = hi - ((hi - lo) * i) / 4; + const label = el("text", { x: pad.l - 8, y: gy + 3, "text-anchor": "end", + fill: "#78828f", "font-size": 9.5 }); + label.textContent = money(gv); + svg.append(label); + } + + // Break-even, drawn brighter than the grid because it is the reference the curve is read + // against, but still chrome rather than data. + if (lo < 0 && hi > 0) { + svg.append(el("line", { x1: pad.l, x2: pad.l + plotW, y1: y(0), y2: y(0), + stroke: "#3b4450", "stroke-width": 1 })); + } + + for (const h of horizons) { + const t = el("text", { x: x(h), y: height - 8, "text-anchor": "middle", + fill: "#78828f", "font-size": 9.5 }); + t.textContent = `${h}`; + svg.append(t); + } + // The zero horizon is the fill itself: mark it so "before" and "after" are readable. + if (horizons.includes(0)) { + svg.append(el("line", { x1: x(0), x2: x(0), y1: pad.t, y2: pad.t + plotH, + stroke: "#3b4450", "stroke-width": 1 })); + } + + for (const [ecn, cells] of byEcn) { + const pts = horizons.filter((h) => Number.isFinite(cells.get(h))) + .map((h) => [x(h), y(cells.get(h))]); + if (!pts.length) continue; + const colour = colourFor(ecn); + svg.append(el("polyline", { + points: pts.map(([px, py]) => `${px},${py}`).join(" "), + fill: "none", stroke: colour, "stroke-width": 2, + "stroke-linejoin": "round", "stroke-linecap": "round", + })); + // A 2px surface ring keeps overlapping endpoints legible without drawing borders. + const [ex, ey] = pts.at(-1); + svg.append(el("circle", { cx: ex, cy: ey, r: 4, fill: colour, + stroke: "#161a21", "stroke-width": 2 })); + } + + attachHover(svg, { key, horizons, byEcn, x, y, pad, plotW, plotH, width }); + return svg; +} + +// The hovered horizon per chart, kept module-level because renderCharts() rebuilds the SVG +// every tick: without this the crosshair would blink out from under a stationary cursor once +// a second. The value is a horizon, not a pixel, so it stays meaningful if the chart resizes. +const hoverAt = new Map(); + +// Crosshair + tooltip, drawn in SVG rather than HTML so restoring it after a re-render is +// just calling draw() again, with no positioning relative to a moving page. +function attachHover(svg, { key, horizons, byEcn, x, y, pad, plotW, plotH, width }) { + const layer = el("g", { "pointer-events": "none" }); + svg.append(layer); + + const nearest = (px) => horizons.reduce( + (best, h) => (Math.abs(x(h) - px) < Math.abs(x(best) - px) ? h : best), horizons[0]); + + function draw(h) { + layer.replaceChildren(); + if (h == null) return; + layer.append(el("line", { x1: x(h), x2: x(h), y1: pad.t, y2: pad.t + plotH, + stroke: "#5b6675", "stroke-width": 1 })); + + // Sorted by P&L so the tooltip reads as a ranking at this horizon, which is the question + // someone hovering a markout curve is actually asking. + const hits = [...byEcn] + .map(([ecn, cells]) => [ecn, cells.get(h)]) + .filter(([, v]) => Number.isFinite(v)) + .sort((a, b) => b[1] - a[1]); + if (!hits.length) return; + + for (const [ecn, v] of hits) { + layer.append(el("circle", { cx: x(h), cy: y(v), r: 3.5, fill: colourFor(ecn), + stroke: "#161a21", "stroke-width": 2 })); + } + + const lines = [`${h}s`, ...hits.map(([ecn, v]) => `${ecn} ${money(v)}`)]; + const chars = Math.max(...lines.map((t) => t.length)); + const boxW = chars * 5.9 + 26; + const boxH = lines.length * 13 + 8; + // Flip to the left of the crosshair when the box would run past the plot's right edge. + const left = x(h) + 10 + boxW > pad.l + plotW ? x(h) - 10 - boxW : x(h) + 10; + const top = Math.min(pad.t + 2, pad.t + plotH - boxH); + + const box = el("g", { transform: `translate(${left},${top})` }); + box.append(el("rect", { width: boxW, height: boxH, rx: 4, + fill: "#0b0e12", stroke: "#242a33", "stroke-width": 1 })); + lines.forEach((text, i) => { + const ty = 17 + i * 13; + if (i > 0) { + box.append(el("rect", { x: 8, y: ty - 7, width: 7, height: 7, rx: 1.5, + fill: colourFor(hits[i - 1][0]) })); + } + const label = el("text", { x: i === 0 ? 8 : 21, y: ty, "font-size": 10, + fill: i === 0 ? "#78828f" : "#d7dce3" }); + label.textContent = text; + box.append(label); + }); + layer.append(box); + } + + // Hit target is the whole plot rectangle, not the 2px lines: the crosshair snaps to the + // nearest horizon, so anywhere in the column counts as pointing at it. + const hit = el("rect", { x: pad.l, y: pad.t, width: plotW, height: plotH, + fill: "transparent", style: "cursor:crosshair" }); + hit.addEventListener("pointermove", (ev) => { + const box = svg.getBoundingClientRect(); + const h = nearest(((ev.clientX - box.left) / box.width) * width); + hoverAt.set(key, h); + draw(h); + }); + hit.addEventListener("pointerleave", () => { hoverAt.delete(key); draw(null); }); + svg.append(hit); + + draw(hoverAt.get(key)); // restore after a tick rebuilt the chart under the cursor +} + +/** Legend: four or fewer series, so every one is named. Text wears text tokens, never the + * series colour; the swatch beside it carries identity. */ +export function pnlLegend(columns, rows) { + const ix = Object.fromEntries(columns.map((c, i) => [c, i])); + const ecns = [...new Set(rows.map((r) => r[ix.ecn]))]; + const box = document.createElement("div"); + box.className = "legend"; + box.innerHTML = ecns.map((e) => + `${e}`).join(""); + return box; +} diff --git a/js/public/index.html b/js/public/index.html new file mode 100644 index 0000000..9d01c9e --- /dev/null +++ b/js/public/index.html @@ -0,0 +1,70 @@ + + +QuestDB TCA live + +
+

QuestDB TCA live

+ + + + + +
+
+
+ diff --git a/js/queries.mjs b/js/queries.mjs new file mode 100644 index 0000000..53fdd4e --- /dev/null +++ b/js/queries.mjs @@ -0,0 +1,89 @@ +// The SQL, shared by every transport. Kept byte-identical to python/tca_live.py so the two +// dashboards are comparable, and so what this serves is what you would paste into the console. +// +// Each panel shifts its window back by exactly its own forward horizon, because anything +// looking forward cannot be evaluated until that window has elapsed: +// +// panel window forward delay +// slippage $now-1m .. $now none live +// markout +-1m $now-2m .. $now-1m +1m 1m +// markout past $now-1m .. $now none live +// min/max +-10s $now-70s .. $now-10s +10s 10s + +const symFilter = (symbol) => (symbol ? `\n AND t.symbol = '${symbol}'` : ""); + +export const slippageSql = (symbol) => ` +SELECT + t.timestamp, t.symbol, t.ecn, t.side, t.price, + (m.best_bid + m.best_ask) / 2 AS mid, + CASE t.side + WHEN 'buy' THEN (t.price - (m.best_bid + m.best_ask) / 2) + / ((m.best_bid + m.best_ask) / 2) * 10000 + WHEN 'sell' THEN ((m.best_bid + m.best_ask) / 2 - t.price) + / ((m.best_bid + m.best_ask) / 2) * 10000 + END AS slippage_bps +FROM fx_trades t +ASOF JOIN market_data m ON (symbol) +WHERE t.timestamp IN '$now-1m..$now'${symFilter(symbol)} +ORDER BY t.timestamp`; + +// Grouped by ecn and horizon only: the granularity actually displayed, so the average the +// server computes is the one shown and nothing is re-aggregated in JavaScript. +const markoutSql = (range, window, symbol) => ` +SELECT + t.ecn, + h.offset / 1000000000 AS horizon_sec, + count() AS n, + avg( + CASE t.side + WHEN 'buy' THEN ((m.best_bid + m.best_ask) / 2 - t.price) / t.price * 10000 + WHEN 'sell' THEN (t.price - (m.best_bid + m.best_ask) / 2) / t.price * 10000 + END + ) AS avg_markout_bps, + sum( + CASE t.side + WHEN 'buy' THEN ((m.best_bid + m.best_ask) / 2 - t.price) * t.quantity + WHEN 'sell' THEN (t.price - (m.best_bid + m.best_ask) / 2) * t.quantity + END + ) AS total_pnl +FROM fx_trades t +HORIZON JOIN market_data m ON (symbol) + RANGE ${range} AS h +WHERE t.timestamp IN '${window}'${symFilter(symbol)} +GROUP BY t.ecn, horizon_sec +ORDER BY t.ecn, horizon_sec`; + +export const markoutFullSql = (symbol) => + markoutSql("FROM -1m TO 1m STEP 10s", "$now-2m..$now-1m", symbol); + +export const markoutPastSql = (symbol) => + markoutSql("FROM -1m TO 0 STEP 10s", "$now-1m..$now", symbol); + +export const minMaxSql = (symbol) => ` +SELECT + t.symbol, t.timestamp, t.side, t.price, + min(p.ask_price) AS min_ask, + max(p.bid_price) AS max_bid +FROM fx_trades t +WINDOW JOIN core_price p + ON (symbol) + RANGE BETWEEN 10 seconds PRECEDING AND 10 seconds FOLLOWING + EXCLUDE PREVAILING +WHERE t.timestamp IN '$now-70s..$now-10s'${symFilter(symbol)}`; + +export const PANELS = ["slippage", "markout-full", "markout-past", "minmax"]; + +export const sqlFor = (panel, symbol) => ({ + "slippage": slippageSql, + "markout-full": markoutFullSql, + "markout-past": markoutPastSql, + "minmax": minMaxSql, +}[panel](symbol)); + +// Symbols currently trading, for the filter dropdown. LATEST ON collapses to one row per +// symbol, and the 30s window keeps it to instruments actually active right now rather than +// every symbol the table has ever seen. +export const symbolsSql = () => ` +SELECT symbol FROM fx_trades +WHERE timestamp IN '$now-30s..$now' +LATEST ON timestamp PARTITION BY symbol`; diff --git a/js/server.mjs b/js/server.mjs new file mode 100644 index 0000000..ecd76c3 --- /dev/null +++ b/js/server.mjs @@ -0,0 +1,107 @@ +// The STATIC layer plus a thin JSON API. Deliberately knows nothing about QuestDB: all of +// that lives in data.mjs, so swapping to @questdb/browser-client later touches one file. +// +// Why a server at all, when the browser client exists? QuestDB accepts a browser WebSocket +// upgrade only when the request's Origin matches its Host, which blocks cross-site WebSocket +// hijacking, and answers a cross-origin upgrade with HTTP 400. A different PORT is a different +// origin, so a page on :8080 cannot talk to QuestDB on :9000 directly however local it is. +// Serving the page and the data from one origin sidesteps that entirely. (CORS itself never +// applies: WebSocket upgrades are not subject to it.) +import { createServer } from "node:http"; +import { readFile } from "node:fs/promises"; +import { fileURLToPath } from "node:url"; +import { dirname, join, normalize } from "node:path"; +import { Data } from "./data.mjs"; +import { PANELS, sqlFor } from "./queries.mjs"; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const PUBLIC = join(HERE, "public"); + +const arg = (name, fallback) => { + const i = process.argv.indexOf(`--${name}`); + return i >= 0 && process.argv[i + 1] ? process.argv[i + 1] : fallback; +}; + +const port = Number(arg("port", 8080)); +const conf = arg("conf", `ws::addr=${arg("addr", "localhost:9000")};`); +const symbol = arg("symbol", null); + +const TYPES = { ".html": "text/html; charset=utf-8", ".js": "text/javascript; charset=utf-8", + ".css": "text/css; charset=utf-8" }; + +const data = new Data(conf); +await data.connect(); +console.log(`[js] connected: ${conf}`); + +const server = createServer(async (req, res) => { + const url = new URL(req.url, `http://${req.headers.host}`); + + if (url.pathname === "/api/tick") { + try { + const payload = await data.tick(url.searchParams.get("symbol") || symbol); + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify(payload, (_k, v) => + // QWP returns 64-bit integers as BigInt, which JSON.stringify refuses outright. + typeof v === "bigint" ? v.toString() : v)); + } catch (error) { + // One bad tick must not take the dashboard down; the page shows the message instead. + res.writeHead(503, { "content-type": "application/json" }); + res.end(JSON.stringify({ error: String(error?.message ?? error) })); + } + return; + } + + if (url.pathname === "/api/symbols") { + try { + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify(await data.symbols())); + } catch (error) { + res.writeHead(503, { "content-type": "application/json" }); + res.end(JSON.stringify({ error: String(error?.message ?? error) })); + } + return; + } + + // The exact statement a panel is running, so the page can show it verbatim. Built from the + // same sqlFor() the query path uses, so it cannot drift from what actually ran. + if (url.pathname === "/api/sql") { + const panel = url.searchParams.get("panel"); + if (!PANELS.includes(panel)) { + res.writeHead(404, { "content-type": "application/json" }); + res.end(JSON.stringify({ error: `unknown panel: ${panel}` })); + return; + } + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ + panel, + sql: sqlFor(panel, url.searchParams.get("symbol") || symbol).trim(), + })); + return; + } + + // Static files, path-traversal guarded. + const rel = url.pathname === "/" ? "/index.html" : url.pathname; + const file = join(PUBLIC, normalize(rel)); + if (!file.startsWith(PUBLIC)) { + res.writeHead(403).end("forbidden"); + return; + } + try { + const body = await readFile(file); + const ext = file.slice(file.lastIndexOf(".")); + res.writeHead(200, { "content-type": TYPES[ext] ?? "application/octet-stream" }); + res.end(body); + } catch { + res.writeHead(404).end("not found"); + } +}); + +server.listen(port, () => console.log(`[js] http://localhost:${port}`)); + +for (const sig of ["SIGINT", "SIGTERM"]) { + process.on(sig, async () => { + server.close(); + await data.close().catch(() => {}); + process.exit(0); + }); +} From 42efe8b899ac84a0a9276a14839219c58b8e8de8 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 1 Oct 2026 12:08:13 +0200 Subject: [PATCH 58/79] js: make Ctrl+C reliable on the dashboard server The shutdown handler awaited data.close() before process.exit(). Installing a SIGINT handler replaces node's default terminate behaviour, so a QWP close that stalls left a process that ignored Ctrl+C outright and had to be killed by PID. The graceful path is now raced against a 2s deadline, and a second signal exits at once. Keep-alive sockets are closed explicitly: an open dashboard tab holds the event loop well past its last request, so server.close() alone would not have been enough. Measured with a browser polling /api/tick once a second: SIGINT to exit 0 in 0.52s, without reaching the deadline. --- js/server.mjs | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/js/server.mjs b/js/server.mjs index ecd76c3..d5de4f3 100644 --- a/js/server.mjs +++ b/js/server.mjs @@ -98,10 +98,24 @@ const server = createServer(async (req, res) => { server.listen(port, () => console.log(`[js] http://localhost:${port}`)); +// Ctrl+C must ALWAYS work. Installing a handler replaces node's default terminate +// behaviour, so awaiting the QWP close unguarded means a stalled close leaves a process +// that ignores Ctrl+C entirely and has to be killed by PID. The graceful path is therefore +// raced against a deadline, and a second signal leaves immediately. +let stopping = false; for (const sig of ["SIGINT", "SIGTERM"]) { - process.on(sig, async () => { + process.on(sig, () => { + if (stopping) process.exit(130); + stopping = true; + console.log("\n[js] shutting down"); server.close(); - await data.close().catch(() => {}); - process.exit(0); + // Keep-alive sockets from the open dashboard would otherwise hold the loop open well + // past the last request. + server.closeAllConnections?.(); + const deadline = setTimeout(() => { + console.log("[js] close timed out, exiting anyway"); + process.exit(0); + }, 2000); + data.close().catch(() => {}).finally(() => { clearTimeout(deadline); process.exit(0); }); }); } From cb2121da66208b5dbfaa58621b87155afc80e5d4 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 1 Oct 2026 14:06:25 +0200 Subject: [PATCH 59/79] js: streaming scan tab, tabbed UI and cluster connection config Adds a second tab that streams a scan of the last N rows of a table and reports only how far it has got, so neither the server nor the page holds the result. 200,000,000 rows of core_price complete in 242s at 826k rows/s against the remote cluster, with the server at 147MB RSS: memory tracks reader count, not rows read. Reading is queryViews(), which hands back a reusable batch view and never materialises rows. The whole per-batch hot path is two counters and a row count the batch already carries; the ten rows the page displays are copied out only when a progress frame is due, ~10 times a second. Three limits had to be worked around to make a scan this size finish at all: - QuestDB applies query.timeout (60s) server-side, independently of the client deadline. The scan is therefore split into LIMIT -m, -n slices that tile the range with no gap or overlap, as python/read_bench.py does. - Slice count is derived from a bounded rows-per-query, not from a fixed number of slices. 64 slices of 200M rows is 3.1M rows each, which times out and completes nothing, while the same 64 slices of 4M rows finish comfortably. - query_pool_max defaults to 4, so more than four parallel readers failed with "timed out waiting for a QWP query from the pool". Stopping a scan mid-flight and relaunching it hit the same pool limit, because cancelling a query is not instant: Promise.all returned while sibling readers were still draining, and the server did not wait for the previous scan to release. Readers are now awaited with allSettled, a new scan waits for the previous one and says so, and the single-flight slot is claimed synchronously. Connection details are configuration rather than constants: --addrs takes the comma-separated list the client resolves with ordered failover, --token-file keeps the bearer token out of argv and out of the startup log, and tls_verify defaults to unsafe_off for the self-signed cluster certificates with a warning at startup. Over the WAN link, zstd is worth 1.9x and parallel readers 9x. Timestamps now infer their unit from magnitude instead of assuming nanoseconds, which was rendering microsecond columns as 1970, and SELECT * reaches UUID, LONG256 and DECIMAL values that previously rendered as [object Object]. --- .gitignore | 6 ++ js/data.mjs | 116 ++++++++++++++++++++++++++++++++++- js/public/app.js | 68 +++++++++++++++++--- js/public/format.js | 60 ++++++++++++++++++ js/public/index.html | 70 +++++++++++++++++++-- js/public/scan.js | 111 +++++++++++++++++++++++++++++++++ js/queries.mjs | 37 +++++++++++ js/server.mjs | 143 +++++++++++++++++++++++++++++++++++++++++-- 8 files changed, 590 insertions(+), 21 deletions(-) create mode 100644 js/public/format.js create mode 100644 js/public/scan.js diff --git a/.gitignore b/.gitignore index ee1e5c5..542004d 100644 --- a/.gitignore +++ b/.gitignore @@ -37,3 +37,9 @@ failsafe-reports/ # === Watchdog config (holds the REST token; commit only the .example) === qdb-primary-watchdog.env + +# === Python === +__pycache__/ +*.py[cod] +.venv/ +venv/ diff --git a/js/data.mjs b/js/data.mjs index 7c401de..7febe1e 100644 --- a/js/data.mjs +++ b/js/data.mjs @@ -5,7 +5,25 @@ // this whole file is replaced by a pass-through proxy for /read/v1 and /exec, the query moves // into the page, and neither server.mjs nor public/ has to change. import { connectQwpNodeClient, QwpEgressQueryError } from "@questdb/nodejs-client"; -import { PANELS, sqlFor, symbolsSql } from "./queries.mjs"; +import { PANELS, scanChunks, sqlFor, symbolsSql } from "./queries.mjs"; + +/** + * Copy the last `sample` rows out of a reusable batch view. + * + * Called only when a progress frame is due, never on the hot path: the view's buffers are + * recycled as soon as the callback returns, so anything shown in the page has to be copied, + * and copying is exactly what the scan is otherwise avoiding. + */ +function copyTail(batch, sample) { + const from = Math.max(0, batch.rowCount - sample); + const out = []; + for (let r = from; r < batch.rowCount; r++) { + const row = new Array(batch.columnCount); + for (let c = 0; c < batch.columnCount; c++) row[c] = batch.get(r, c); + out.push(row); + } + return out; +} export class Data { constructor(conf) { @@ -75,6 +93,102 @@ export class Data { return { columns, rows }; } + /** + * Stream a scan of `table`'s last `rows` rows, reporting progress as it goes. + * + * Nothing is accumulated: counters, plus a rolling tail of the last `sample` rows, are the + * only things that outlive a batch. Memory is therefore flat whether the scan covers a + * thousand rows or two hundred million, which is the whole point of the demo. + * + * queryViews() rather than query(): it hands back a REUSABLE batch view and never + * materialises rows into JS arrays, so the decoder is not fighting the garbage collector + * for the duration of the scan. The view is invalid once the callback returns, so the tail + * rows are copied out before then. + */ + async scan({ table, rows: limit, readers = 4, chunkRows = 500_000, chunks = 0, + sample = 10, progressMs = 100 }, onProgress, signal) { + const workers = Math.max(1, Math.trunc(readers)); + + // Slice count is derived from a bounded number of ROWS PER QUERY, not from a fixed + // number of slices. QuestDB caps how long one statement may run (query.timeout, 60s on + // the cluster this was built against), so what has to stay bounded is the work in a + // single query, not the work in the scan. A fixed slice count fails exactly where it + // matters: 64 slices of 200M rows is 3.1M rows each, which at per-reader speed is ~86s + // and times out, while the same 64 slices of 4M rows finish comfortably. + // + // At least one slice per reader, so nobody sits idle on a small scan. + const sliced = Math.ceil(Math.max(1, limit) / Math.max(1, chunkRows)); + const sqls = scanChunks(table, limit, chunks || Math.max(workers, sliced)); + const started = performance.now(); + + let rows = 0, batches = 0, columns = null, tail = [], chunksDone = 0, lastSent = 0; + const snapshot = (done) => ({ + table, sql: sqls[0] ?? "", chunks: sqls.length, chunksDone, readers: workers, + columns, rows, batches, tail, done, ms: performance.now() - started, + }); + + const queue = sqls.slice(); + const running = new Set(); + const abort = () => { for (const q of running) q.cancel().catch(() => {}); }; + signal?.addEventListener("abort", abort, { once: true }); + + // One lease per reader, held for that reader's whole share of the queue. Leases are + // serial, so concurrency comes from running several of them, not from one. + const reader = async () => { + const lease = await this.db.borrowQuery(); + try { + while (queue.length > 0 && !signal?.aborted) { + const sql = queue.shift(); + const query = await lease.queryViews(sql, (batch) => { + // THE HOT PATH, and deliberately the whole of it: two counters and a row count + // the batch already knows. No row is visited, nothing is materialised, and the + // cost per batch does not depend on how many rows it carries. + rows += batch.rowCount; + batches += 1; + + const now = performance.now(); + if (now - lastSent < progressMs) return; + lastSent = now; + + if (columns === null) columns = batch.columns.map((c) => c.name); + // Only now, ~10 times a second, are any rows read at all, and only `sample` of + // them. Copied, because the view is recycled when this callback returns. + tail = copyTail(batch, sample); + onProgress(snapshot(false)); + }, { + // The client's session default is 15s, which a slice of millions of rows runs + // well past. QuestDB applies its own server-side query.timeout independently, + // and that one is what the slicing above is sized against. + timeoutMs: 0, + }); + running.add(query); + try { + await query.completion; + } finally { + running.delete(query); + } + chunksDone += 1; + } + } finally { + await lease.close(); + } + }; + + try { + // allSettled, not all: Promise.all rejects on the FIRST failing reader and returns + // while its siblings are still draining, so scan() would resolve with connections + // still borrowed and the next scan would find an empty pool. Every reader has to + // reach its finally and give its lease back before this method returns. + const results = await Promise.allSettled( + Array.from({ length: Math.min(workers, sqls.length) }, () => reader())); + const failed = results.find((r) => r.status === "rejected"); + if (failed && !signal?.aborted) throw failed.reason; + if (!signal?.aborted) onProgress(snapshot(true)); + } finally { + signal?.removeEventListener("abort", abort); + } + } + async close() { if (this.db) await this.db.close(); this.db = null; diff --git a/js/public/app.js b/js/public/app.js index 6fd8f56..6388bff 100644 --- a/js/public/app.js +++ b/js/public/app.js @@ -5,6 +5,7 @@ // fetchTick() with connectQwpBrowserEgress({url: new URL("/read/v1", location.href)}) and // iterate the batches here instead. Nothing else in the page changes. import { pnlChart, pnlLegend } from "/chart.js"; +import { EPOCH_DIGITS, epochTime } from "/format.js"; const PANELS = [ ["slippage", "slippage, last 1m", "live"], @@ -33,10 +34,8 @@ const sqlText = new Map(); const fmt = (v) => { if (v === null || v === undefined) return "null"; if (typeof v === "number") return Number.isInteger(v) ? String(v) : v.toFixed(2); - // Nanosecond epochs arrive as strings (BigInt is not JSON-serialisable); show the time part. - if (typeof v === "string" && /^\d{16,19}$/.test(v)) { - return new Date(Number(v) / 1e6).toISOString().slice(11, 23); - } + // Epochs arrive as digit strings; the unit is inferred, not assumed. See format.js. + if (typeof v === "string" && EPOCH_DIGITS.test(v)) return epochTime(v); if (typeof v === "string" && /^\d{4}-\d{2}-\d{2}T/.test(v)) return v.slice(11, 23); return String(v); }; @@ -216,7 +215,60 @@ const offered = [...refreshEl.options].map((o) => Number(o.value)); if (!offered.includes(intervalMs)) intervalMs = 1000; refreshEl.value = String(intervalMs); -await loadSymbols(); -await tick(); -applyInterval(); -setInterval(loadSymbols, 30_000); // instruments come and go; keep the list current +// === Tabs ============================================================================= +// +// Only the visible tab queries. Four TCA statements a second against a cluster is not +// something to leave running behind a tab nobody is looking at, so switching away clears the +// timers outright rather than hiding the output. +let symbolsTimer = null; + +async function startTca() { + await loadSymbols(); + await tick(); + applyInterval(); + symbolsTimer = setInterval(loadSymbols, 30_000); // instruments come and go +} + +function stopTca() { + if (timer !== null) { clearInterval(timer); timer = null; } + if (symbolsTimer !== null) { clearInterval(symbolsTimer); symbolsTimer = null; } +} + +// Loaded on first use, so a session that never opens the scan tab never fetches its code. +let scanModule = null; + +const VIEWS = { + tca: { view: "view-tca", ctl: "ctl-tca", + start: startTca, + stop: () => { stopTca(); metaEl.textContent = ""; } }, + scan: { view: "view-scan", ctl: "ctl-scan", + start: async () => { scanModule ??= await import("/scan.js"); }, + stop: () => scanModule?.stopScan() }, +}; + +let current = null; + +async function show(name) { + if (current === name) return; + if (current) VIEWS[current].stop(); + current = name; + // Every view and control row is set explicitly, not just the one being left behind: a + // direct ?tab=scan load has no previous tab, and the markup ships with the TCA tab mounted. + for (const [key, v] of Object.entries(VIEWS)) { + const on = key === name; + document.getElementById(v.view).hidden = !on; + document.getElementById(v.ctl).hidden = !on; + } + for (const b of tabsEl.querySelectorAll("button")) { + b.classList.toggle("on", b.dataset.tab === name); + } + await VIEWS[name].start(); +} + +const tabsEl = document.getElementById("tabs"); +tabsEl.addEventListener("click", (ev) => { + const btn = ev.target.closest("button[data-tab]"); + if (btn && !btn.disabled && VIEWS[btn.dataset.tab]) show(btn.dataset.tab); +}); + +await show(params.get("tab") === "scan" ? "scan" : "tca"); diff --git a/js/public/format.js b/js/public/format.js new file mode 100644 index 0000000..28ae1b2 --- /dev/null +++ b/js/public/format.js @@ -0,0 +1,60 @@ +// Timestamps reach the page as DIGIT STRINGS: QWP carries 64-bit integers as BigInt, and +// BigInt is not JSON-serialisable, so server.mjs stringifies them. +// +// The unit is not fixed. QuestDB carries both microsecond and nanosecond timestamps, and a +// column's unit is not visible in the value itself, only in its magnitude. Assuming one +// silently renders the other as 1970, which is exactly what the scan tab did to core_price. +export const EPOCH_DIGITS = /^\d{13,19}$/; + +/** + * Interpret an epoch digit string, inferring the unit from magnitude. + * + * The thresholds are far from any plausible date: 1e14 ms and 1e17 us both land in the year + * 5138, so a real millisecond timestamp can never be mistaken for microseconds, nor + * microseconds for nanoseconds, for any date this side of the fourth millennium. + */ +export function epochDate(digits) { + const n = Number(digits); + if (n >= 1e17) return new Date(n / 1e6); // nanoseconds + if (n >= 1e14) return new Date(n / 1e3); // microseconds + return new Date(n); // milliseconds +} + +/** `HH:MM:SS.mmm`, for panels where every row is from the last minute anyway. */ +export const epochTime = (digits) => epochDate(digits).toISOString().slice(11, 23); + +/** `YYYY-MM-DD HH:MM:SS.mmm`, for rows that can come from anywhere in a table's history. */ +export const epochStamp = (digits) => + epochDate(digits).toISOString().replace("T", " ").slice(0, 23); + +const u64 = (v) => BigInt.asUintN(64, BigInt(v)).toString(16).padStart(16, "0"); + +/** + * Render a QWP value that is not a plain scalar. + * + * Several QuestDB types cross the wire as structured objects rather than strings: a UUID is + * a {low, high} pair of 64-bit halves, a LONG256 is four little-endian words, a DECIMAL is an + * unscaled integer plus a scale. Without this they all render as "[object Object]", which is + * what SELECT * on fx_trades produced. + */ +export function fmtStructured(v) { + if (v.low !== undefined && v.high !== undefined) { + const hex = u64(v.high) + u64(v.low); + return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${ + hex.slice(16, 20)}-${hex.slice(20)}`; + } + if (Array.isArray(v.words)) { + // Word 0 is least significant, so the printed order is the reverse of the array. + return `0x${[...v.words].reverse().map(u64).join("")}`; + } + if (v.unscaled !== undefined) { + const digits = BigInt(v.unscaled).toString(); + if (!v.scale) return digits; + const neg = digits.startsWith("-"); + const body = (neg ? digits.slice(1) : digits).padStart(v.scale + 1, "0"); + return `${neg ? "-" : ""}${body.slice(0, -v.scale)}.${body.slice(-v.scale)}`; + } + if (Array.isArray(v.dimensions)) return `array[${v.dimensions.join("x")}]`; + if (v.bits !== undefined) return `geohash(${v.bits})`; + return JSON.stringify(v); +} diff --git a/js/public/index.html b/js/public/index.html index 9d01c9e..7d09170 100644 --- a/js/public/index.html +++ b/js/public/index.html @@ -1,6 +1,6 @@ -QuestDB TCA live +QuestDB QWP demos
-

QuestDB TCA live

+

QuestDB QWP

+ + +
+ +
@@ -63,8 +100,29 @@

QuestDB TCA live

- - -
-
+
+ + + +
+
+
+
+ + + diff --git a/js/public/scan.js b/js/public/scan.js new file mode 100644 index 0000000..b5c97a8 --- /dev/null +++ b/js/public/scan.js @@ -0,0 +1,111 @@ +// The STREAMING SCAN tab. +// +// The server reads the table over QWP in batches and never holds the result; this page is +// told only how far it has got, plus the last few rows it saw. So neither side's memory +// grows with the scan, which is the claim the tab exists to demonstrate. +// +// Transport is Server-Sent Events: a long one-way stream of progress is exactly what SSE is +// for, and closing the EventSource closes the socket, which is what tells the server to stop +// reading. No polling, and no way to leave a scan running after the tab goes away. +import { EPOCH_DIGITS, epochStamp, fmtStructured } from "/format.js"; + +const tableEl = document.getElementById("scan-table"); +const rowsEl = document.getElementById("scan-rows"); +const readersEl = document.getElementById("scan-readers"); +const goEl = document.getElementById("scan-go"); +const statsEl = document.getElementById("scan-stats"); +const panelEl = document.getElementById("scan-panel"); + +// The list comes from the server, which builds it from the same SCAN_TABLES that validates +// the request, so the dropdown cannot offer a table the scan would reject. +const tables = await (await fetch("/api/scan-tables")).json(); +tableEl.innerHTML = tables.map((t) => ``).join(""); + +let source = null; + +const esc = (s) => String(s).replace(/[&<>"]/g, (c) => + ({ "&": "&", "<": "<", ">": ">", '"': """ }[c])); + +const int = (n) => Number(n).toLocaleString("en-US"); + +// Rows/s runs from thousands to tens of millions, so a fixed unit is unreadable at one end +// or the other. +const rate = (n) => + n >= 1e6 ? `${(n / 1e6).toFixed(2)}M` : n >= 1e3 ? `${(n / 1e3).toFixed(1)}k` : n.toFixed(0); + +const secs = (ms) => (ms >= 1000 ? `${(ms / 1000).toFixed(2)}s` : `${ms.toFixed(0)}ms`); + +function fmt(v) { + if (v === null || v === undefined) return "null"; + if (typeof v === "number") return Number.isInteger(v) ? String(v) : v.toFixed(5); + // Epochs arrive as digit strings; the unit is inferred, not assumed. See format.js. + if (typeof v === "string" && EPOCH_DIGITS.test(v)) return epochStamp(v); + // SELECT * reaches types the TCA panels never select: UUID, LONG256, DECIMAL, arrays. + if (typeof v === "object") return fmtStructured(v); + return String(v); +} + +function stat(label, value, note = "") { + return `
${esc(value)}${esc(label)}${ + note ? ` · ${esc(note)}` : ""}
`; +} + +function render(p, state) { + const rowsPerSec = p.ms > 0 ? (p.rows / p.ms) * 1000 : 0; + statsEl.innerHTML = [ + stat("rows read", int(p.rows), state), + stat("batches", int(p.batches), + p.batches ? `${int(Math.round(p.rows / p.batches))} rows/batch` : ""), + stat("elapsed", secs(p.ms), + p.chunks ? `chunk ${int(p.chunksDone)}/${int(p.chunks)}` : ""), + stat("throughput", `${rate(rowsPerSec)} rows/s`, + p.readers ? `${p.readers} reader${p.readers === 1 ? "" : "s"}` : ""), + ].join(""); + + const head = (p.columns ?? []).map((c) => `${esc(c)}`).join(""); + const body = (p.tail ?? []).map((r) => + `${r.map((v) => `${esc(fmt(v))}`).join("")}`).join(""); + + panelEl.innerHTML = ` +

${esc(p.table ?? "")} last ${p.tail?.length ?? 0} rows received${ + p.chunks > 1 ? ` · first of ${int(p.chunks)} slices shown` : ""}

+ 
${esc(p.sql ?? "")}
+ ${head}${body} +
the page holds these ${p.tail?.length ?? 0} rows and four counters, nothing + more: ${int(p.rows)} rows streamed through it
`; +} + +function stop() { + source?.close(); // closing the socket is what stops the server reading + source = null; + goEl.textContent = "Start scan"; +} + +function start() { + stop(); + const qs = new URLSearchParams({ table: tableEl.value, rows: String(rowsEl.value || 1), + readers: readersEl.value }); + const blank = { table: tableEl.value, rows: 0, batches: 0, ms: 0, columns: [], tail: [], + sql: "", chunks: 0, chunksDone: 0, readers: Number(readersEl.value) }; + render(blank, "starting"); + goEl.textContent = "Stop scan"; + + source = new EventSource(`/api/scan?${qs}`); + // The server is draining the scan this one is replacing. Cancelling a query is not + // instant, so say so rather than showing a frozen zero. + source.addEventListener("waiting", () => render(blank, "waiting for the previous scan")); + source.addEventListener("progress", (ev) => render(JSON.parse(ev.data), "streaming")); + source.addEventListener("done", (ev) => { render(JSON.parse(ev.data), "complete"); stop(); }); + source.addEventListener("failed", (ev) => { + panelEl.innerHTML = `

${esc(JSON.parse(ev.data).error)}

`; + stop(); + }); + // EventSource retries by default; a scan is not idempotent progress, so a dropped stream + // ends the run rather than silently restarting it from zero. + source.addEventListener("error", () => { if (source?.readyState === 2) stop(); }); +} + +goEl.addEventListener("click", () => (source ? stop() : start())); + +/** Leaving the tab must stop the scan: the point is that nothing runs off-screen. */ +export function stopScan() { stop(); } diff --git a/js/queries.mjs b/js/queries.mjs index 53fdd4e..acf5bb7 100644 --- a/js/queries.mjs +++ b/js/queries.mjs @@ -87,3 +87,40 @@ export const symbolsSql = () => ` SELECT symbol FROM fx_trades WHERE timestamp IN '$now-30s..$now' LATEST ON timestamp PARTITION BY symbol`; + +// === Streaming scan =================================================================== +// +// LIMIT -N is the point of the demo: it asks for the LAST N rows, which QuestDB answers by +// skipping whole page frames on partition metadata rather than reading them. The scan then +// streams back in batches, so the client's memory is flat regardless of N. +export const SCAN_TABLES = ["core_price", "market_data", "fx_trades"]; + +/** + * Split the last `rows` rows into `chunks` equal row-count slices, oldest first. + * + * `LIMIT -m, -n` takes the last m rows then drops the last n of them, i.e. the half-open + * range [-m, -n), so consecutive slices tile the range with no gap and no overlap. The + * newest slice has n == 0, which is the documented `LIMIT -n, 0` == `LIMIT -n` form. The + * bounds are arithmetic on `rows` alone, so no preliminary count query is needed. + * + * Two reasons to slice rather than issue one statement. QuestDB applies its own + * `query.timeout` server-side (60s on the instance this was built against), which one + * 200M-row statement blows straight through; and slices can be read concurrently over + * separate connections, which is what makes a remote scan bearable. This mirrors + * python/read_bench.py --split rows. + */ +export const scanChunks = (table, rows, chunks = 1) => { + if (!SCAN_TABLES.includes(table)) throw new Error(`unknown table: ${table}`); + const limit = Math.max(1, Math.trunc(rows)); + const n = Math.max(1, Math.trunc(chunks)); + const out = []; + for (let i = 0; i < n; i++) { + const lo = limit - Math.floor((limit * i) / n); // rows from the end, inclusive + const hi = limit - Math.floor((limit * (i + 1)) / n); // rows from the end, exclusive + if (lo <= hi) continue; // empty slice: chunks > rows + out.push(hi === 0 + ? `SELECT * FROM ${table} LIMIT -${lo}` + : `SELECT * FROM ${table} LIMIT -${lo}, -${hi}`); + } + return out; +}; diff --git a/js/server.mjs b/js/server.mjs index d5de4f3..cd5b8ae 100644 --- a/js/server.mjs +++ b/js/server.mjs @@ -12,7 +12,7 @@ import { readFile } from "node:fs/promises"; import { fileURLToPath } from "node:url"; import { dirname, join, normalize } from "node:path"; import { Data } from "./data.mjs"; -import { PANELS, sqlFor } from "./queries.mjs"; +import { PANELS, SCAN_TABLES, sqlFor } from "./queries.mjs"; const HERE = dirname(fileURLToPath(import.meta.url)); const PUBLIC = join(HERE, "public"); @@ -23,15 +23,61 @@ const arg = (name, fallback) => { }; const port = Number(arg("port", 8080)); -const conf = arg("conf", `ws::addr=${arg("addr", "localhost:9000")};`); const symbol = arg("symbol", null); +// --token-file rather than a token on the command line: argv is world-readable through ps, +// and the configuration string is logged at startup. The file is read here and the value +// never leaves this process except on the wire to QuestDB. +const tokenFile = arg("token-file", null); +const token = tokenFile ? (await readFile(tokenFile, "utf8")).trim() : null; + +// The cluster is configuration, not a constant. --addrs takes the comma-separated list the +// QWP client already understands natively: it tries them in order and fails over to the next +// when one is unreachable, so naming every node is how read HA is exercised rather than +// merely claimed. +// +// --scheme wss --addrs enterprise-primary:9000,172.31.42.41:9000,10.0.0.8:9000 +// +// tls_verify defaults to unsafe_off because these clusters present self-signed certificates; +// it is applied only to wss, announced at startup, and --tls-verify on restores checking. +const scheme = arg("scheme", "ws"); +const addrs = arg("addrs", arg("addr", "localhost:9000")); +const tlsVerify = arg("tls-verify", "unsafe_off"); +const compression = arg("compression", null); +const maxBatchRows = arg("max-batch-rows", null); + +// Each parallel reader borrows its own query connection, and the pool defaults to 4: asking +// for 8 readers without raising this fails with "timed out waiting for a QWP query from the +// pool". The readers the UI may request are capped to this number below. +const queryPoolMax = Number(arg("query-pool-max", 16)); + +const opts = [`addr=${addrs}`, `query_pool_max=${queryPoolMax}`]; +if (scheme === "wss") opts.push(`tls_verify=${tlsVerify}`); +if (compression) opts.push(`compression=${compression}`); +if (maxBatchRows) opts.push(`max_batch_rows=${maxBatchRows}`); +const base = arg("conf", `${scheme}::${opts.join(";")};`); +const withSemi = base.endsWith(";") ? base : `${base};`; +const conf = token ? `${withSemi}token=${token};` : withSemi; +// What gets logged. The token is replaced, not truncated: a prefix is still a secret. +const safeConf = token ? `${withSemi}token=***;` : withSemi; + +// QWP returns 64-bit integers as BigInt, which JSON.stringify refuses outright. +const bigints = (_k, v) => (typeof v === "bigint" ? v.toString() : v); + +// The scan in flight, if any. One at a time: see /api/scan. `scanSettled` resolves once that +// scan has handed its query connections back, which a new scan must wait for. +let scanAbort = null; +let scanSettled = Promise.resolve(); + const TYPES = { ".html": "text/html; charset=utf-8", ".js": "text/javascript; charset=utf-8", ".css": "text/css; charset=utf-8" }; const data = new Data(conf); await data.connect(); -console.log(`[js] connected: ${conf}`); +console.log(`[js] connected: ${safeConf}`); +if (scheme === "wss" && tlsVerify === "unsafe_off") { + console.log("[js] WARNING: TLS certificate verification is OFF (--tls-verify on to enable)"); +} const server = createServer(async (req, res) => { const url = new URL(req.url, `http://${req.headers.host}`); @@ -40,9 +86,7 @@ const server = createServer(async (req, res) => { try { const payload = await data.tick(url.searchParams.get("symbol") || symbol); res.writeHead(200, { "content-type": "application/json" }); - res.end(JSON.stringify(payload, (_k, v) => - // QWP returns 64-bit integers as BigInt, which JSON.stringify refuses outright. - typeof v === "bigint" ? v.toString() : v)); + res.end(JSON.stringify(payload, bigints)); } catch (error) { // One bad tick must not take the dashboard down; the page shows the message instead. res.writeHead(503, { "content-type": "application/json" }); @@ -62,6 +106,93 @@ const server = createServer(async (req, res) => { return; } + if (url.pathname === "/api/scan-tables") { + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify(SCAN_TABLES)); + return; + } + + // Server-sent events, because the scan is a long one-way stream of progress and SSE + // reconnects, framing and backpressure come free with the transport. Single-flight: a new + // scan cancels the one in progress, so a page reload cannot leave an orphan reading 200 + // million rows into a socket nobody is listening to. + if (url.pathname === "/api/scan") { + const table = url.searchParams.get("table") ?? SCAN_TABLES[0]; + const rows = Number(url.searchParams.get("rows") ?? 200_000_000); + const readers = Number(url.searchParams.get("readers") ?? 4); + // Rows per query. Bounded so one slice always finishes inside QuestDB's query.timeout; + // see Data.scan. + const chunkRows = Number(url.searchParams.get("chunk_rows") ?? 500_000); + if (!SCAN_TABLES.includes(table) || !Number.isFinite(rows) || rows < 1 + || !Number.isFinite(readers) || readers < 1 || readers > queryPoolMax + || !Number.isFinite(chunkRows) || chunkRows < 1) { + res.writeHead(400, { "content-type": "application/json" }); + res.end(JSON.stringify({ + error: `bad scan request: table=${table} rows=${rows} readers=${readers}` })); + return; + } + + const control = new AbortController(); + + // Claim the single-flight slot SYNCHRONOUSLY, before any await: two starts in quick + // succession would otherwise both read the old slot, both wait on the same scan, and + // both run. Whoever claims last wins, and an older claimant bails out below. + const previousAbort = scanAbort; + const previousSettled = scanSettled; + scanAbort = control; + let markSettled; + scanSettled = new Promise((resolve) => { markSettled = resolve; }); + + res.writeHead(200, { + "content-type": "text/event-stream", + "cache-control": "no-cache", + connection: "keep-alive", + // The dashboard is same-origin, but a proxy that buffers would defeat the whole point. + "x-accel-buffering": "no", + }); + + const send = (event, payload) => { + if (res.writableEnded) return; + res.write(`event: ${event}\ndata: ${JSON.stringify(payload, bigints)}\n\n`); + }; + + // The browser going away is the normal way a scan ends early: stop reading immediately + // rather than finishing 200 million rows for nobody. + res.on("close", () => control.abort()); + + // Stop the previous scan AND wait for it to let go. Cancelling a query is not instant: + // the client drains the cancelled statement before returning its connection to the pool, + // so starting 16 fresh readers while the old 16 are still draining fails with "timed out + // waiting for a QWP query from the pool". This wait is what makes "stop, change the + // parameters, start again" work at any reader count, and it is announced rather than + // silent: without the event the page sits on a frozen zero for seconds with no reason. + if (previousAbort) { + previousAbort.abort(); + send("waiting", { table, readers }); + await previousSettled.catch(() => {}); + } + + // Overtaken while waiting, or the browser gave up: do not start a second scan. + if (control.signal.aborted || scanAbort !== control) { + markSettled(); + res.end(); + return; + } + + try { + await data.scan({ table, rows, readers, chunkRows }, + (p) => send(p.done ? "done" : "progress", p), + control.signal); + } catch (error) { + if (!control.signal.aborted) send("failed", { error: String(error?.message ?? error) }); + } finally { + if (scanAbort === control) scanAbort = null; + markSettled(); + res.end(); + } + return; + } + // The exact statement a panel is running, so the page can show it verbatim. Built from the // same sqlFor() the query path uses, so it cannot drift from what actually ran. if (url.pathname === "/api/sql") { From f2986a0e2e8efffdc135e5045f4f7a321f01c413 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 1 Oct 2026 14:35:04 +0200 Subject: [PATCH 60/79] js: cast the scanned timestamp to LONG, and add the OHLC tab Profiling a scan put ~60% of active client CPU in readTimestampView and its bit reader, with the BigInt allocation behind most of the GC on top. The cause is the designated timestamp being GORILLA-encoded on the wire: delta-of-delta bit packing cannot be handed over as a view, so it is unpacked value by value with BigInt arithmetic, eagerly, before a batch is delivered. Every other fixed-width column is a zero-copy view costing almost nothing. So the scan was never decoding rows, but it was always decoding one column per row, and that column was the only expensive one. The scan's projection is now selectable. Casting the timestamp to LONG changes the wire type, which routes it through readFixedView instead of the Gorilla path while preserving the exact value, and is the new default: all 2,476,649 rows/s timestamp, Gorilla-decoded epoch-long 6,293,394 rows/s timestamp, as an epoch integer no-timestamp 8,714,877 rows/s no timestamp Measured at 8 readers against a local instance. The cast costs 8 uncompressed bytes per row on the wire, which is the right trade on a LAN and the wrong one on a thin link. The page needs no change to display it: format.js already infers a timestamp's unit from its magnitude. Empty batches no longer overwrite the displayed row tail. Slicing a small table into many ranges yields plenty of zero-row batches, and whichever landed on the final progress tick left the page reporting "last 0 rows received". The third tab draws candles with a VWAP overlay and a volume histogram, built with TradingView Lightweight Charts, vendored from node_modules rather than a CDN so the demo does not depend on the network. SAMPLE BY aggregates the bars server-side, so a window covering hours of trading arrives as a few thousand bars rather than the millions of trades behind them. The window is anchored to the newest row rather than to $now, so the chart still works on an instance whose writer has stopped. The running VWAP is accumulated from the per-bar vwap and volume already returned, so it costs no extra query. Absolute timestamp windows use explicit >= and <= comparisons: the IN 'a..b' interval form parses the $now-relative literals the TCA panels use, but rejects an absolute ISO pair outright. --- js/data.mjs | 65 +++++++++++- js/package-lock.json | 18 +++- js/package.json | 3 +- js/public/app.js | 14 ++- js/public/index.html | 36 ++++++- js/public/ohlc.js | 238 +++++++++++++++++++++++++++++++++++++++++++ js/public/scan.js | 7 +- js/queries.mjs | 105 ++++++++++++++++++- js/server.mjs | 53 +++++++++- 9 files changed, 521 insertions(+), 18 deletions(-) create mode 100644 js/public/ohlc.js diff --git a/js/data.mjs b/js/data.mjs index 7febe1e..aad0ed3 100644 --- a/js/data.mjs +++ b/js/data.mjs @@ -5,7 +5,10 @@ // this whole file is replaced by a pass-through proxy for /read/v1 and /exec, the query moves // into the page, and neither server.mjs nor public/ has to change. import { connectQwpNodeClient, QwpEgressQueryError } from "@questdb/nodejs-client"; -import { PANELS, scanChunks, sqlFor, symbolsSql } from "./queries.mjs"; +import { + OHLC_INTERVALS, OHLC_LOOKBACKS, PANELS, ohlcLastRowSql, ohlcSql, ohlcSymbolsSql, + scanChunks, sqlFor, symbolsSql, +} from "./queries.mjs"; /** * Copy the last `sample` rows out of a reusable batch view. @@ -106,7 +109,7 @@ export class Data { * rows are copied out before then. */ async scan({ table, rows: limit, readers = 4, chunkRows = 500_000, chunks = 0, - sample = 10, progressMs = 100 }, onProgress, signal) { + projection = "all", sample = 10, progressMs = 100 }, onProgress, signal) { const workers = Math.max(1, Math.trunc(readers)); // Slice count is derived from a bounded number of ROWS PER QUERY, not from a fixed @@ -118,12 +121,13 @@ export class Data { // // At least one slice per reader, so nobody sits idle on a small scan. const sliced = Math.ceil(Math.max(1, limit) / Math.max(1, chunkRows)); - const sqls = scanChunks(table, limit, chunks || Math.max(workers, sliced)); + const sqls = scanChunks(table, limit, chunks || Math.max(workers, sliced), projection); const started = performance.now(); let rows = 0, batches = 0, columns = null, tail = [], chunksDone = 0, lastSent = 0; const snapshot = (done) => ({ table, sql: sqls[0] ?? "", chunks: sqls.length, chunksDone, readers: workers, + projection, columns, rows, batches, tail, done, ms: performance.now() - started, }); @@ -153,7 +157,10 @@ export class Data { if (columns === null) columns = batch.columns.map((c) => c.name); // Only now, ~10 times a second, are any rows read at all, and only `sample` of // them. Copied, because the view is recycled when this callback returns. - tail = copyTail(batch, sample); + // An EMPTY batch must not clear the tail: slicing a small table into many + // ranges yields plenty of zero-row batches, and whichever one landed on the + // last progress tick would leave the page showing no rows at all. + if (batch.rowCount > 0) tail = copyTail(batch, sample); onProgress(snapshot(false)); }, { // The client's session default is 15s, which a slice of millions of rows runs @@ -189,6 +196,56 @@ export class Data { } } + /** + * Candles for one instrument, with the window anchored to the newest row in the table. + * + * Three cheap statements on one lease: where the data ends, which symbols were trading + * near that point, and the bars themselves. Anchoring costs one extra query and buys a + * chart that still works on an instance whose writer has stopped. + */ + async ohlc({ symbol, interval = "5s", lookback = "30m" }) { + if (!(interval in OHLC_INTERVALS)) throw new Error(`unknown interval: ${interval}`); + if (!(lookback in OHLC_LOOKBACKS)) throw new Error(`unknown lookback: ${lookback}`); + + const started = performance.now(); + const lease = await this.db.borrowQuery(); + try { + const anchor = await this.#run(lease, ohlcLastRowSql(symbol)); + const lastNs = anchor.rows[0]?.[0]; + if (lastNs === null || lastNs === undefined) { + return { symbol, interval, lookback, columns: [], rows: [], symbols: [], + ms: performance.now() - started, empty: "no rows in fx_trades" }; + } + + // Timestamps are nanoseconds here; Date works in milliseconds. Dividing through + // BigInt first keeps the epoch exact, since ns since 1970 is well past 2^53. + const lastMs = Number(BigInt(lastNs) / 1000000n); + const toIso = new Date(lastMs).toISOString(); + const fromIso = new Date(lastMs - OHLC_LOOKBACKS[lookback] * 1000).toISOString(); + // The picker lists what traded near the anchor, not what is trading now. + const symFromIso = new Date(lastMs - 1800 * 1000).toISOString(); + + const picker = await this.#run(lease, ohlcSymbolsSql(symFromIso, toIso)); + const symbols = picker.rows.map((r) => String(r[0])).sort(); + const chosen = symbol || symbols[0]; + if (!chosen) { + return { symbol: null, interval, lookback, columns: [], rows: [], symbols, + ms: performance.now() - started, empty: "no symbols in the window" }; + } + + const bars = await this.#run(lease, ohlcSql({ symbol: chosen, interval, fromIso, toIso })); + return { + symbol: chosen, interval, lookback, symbols, + columns: bars.columns, rows: bars.rows, + sql: ohlcSql({ symbol: chosen, interval, fromIso, toIso }).trim(), + from: fromIso, to: toIso, + ms: performance.now() - started, + }; + } finally { + await lease.close(); + } + } + async close() { if (this.db) await this.db.close(); this.db = null; diff --git a/js/package-lock.json b/js/package-lock.json index 48d1c46..1b83639 100644 --- a/js/package-lock.json +++ b/js/package-lock.json @@ -8,7 +8,8 @@ "name": "questdb-tca-web", "version": "1.0.0", "dependencies": { - "@questdb/nodejs-client": "file:../../../questdb/nodejs-questdb-client/packages/nodejs-client" + "@questdb/nodejs-client": "file:../../../questdb/nodejs-questdb-client/packages/nodejs-client", + "lightweight-charts": "^5.2.1" } }, "../../../questdb/nodejs-questdb-client/packages/nodejs-client": { @@ -26,6 +27,21 @@ "node_modules/@questdb/nodejs-client": { "resolved": "../../../questdb/nodejs-questdb-client/packages/nodejs-client", "link": true + }, + "node_modules/fancy-canvas": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/fancy-canvas/-/fancy-canvas-2.1.0.tgz", + "integrity": "sha512-nifxXJ95JNLFR2NgRV4/MxVP45G9909wJTEKz5fg/TZS20JJZA6hfgRVh/bC9bwl2zBtBNcYPjiBE4njQHVBwQ==", + "license": "MIT" + }, + "node_modules/lightweight-charts": { + "version": "5.2.1", + "resolved": "https://registry.npmjs.org/lightweight-charts/-/lightweight-charts-5.2.1.tgz", + "integrity": "sha512-IVwoK1RLFiLPubaKIjNbtjWLnpPMqiABSrTay6whmNa8L1+19292VtHJ+BWyPUuLCwF0tcQlhEWd1CLB2a1nsQ==", + "license": "Apache-2.0", + "dependencies": { + "fancy-canvas": "2.1.0" + } } } } diff --git a/js/package.json b/js/package.json index 9b2ef14..6cc1d80 100644 --- a/js/package.json +++ b/js/package.json @@ -8,6 +8,7 @@ "start": "node server.mjs" }, "dependencies": { - "@questdb/nodejs-client": "file:../../../questdb/nodejs-questdb-client/packages/nodejs-client" + "@questdb/nodejs-client": "file:../../../questdb/nodejs-questdb-client/packages/nodejs-client", + "lightweight-charts": "^5.2.1" } } diff --git a/js/public/app.js b/js/public/app.js index 6388bff..849b170 100644 --- a/js/public/app.js +++ b/js/public/app.js @@ -234,8 +234,10 @@ function stopTca() { if (symbolsTimer !== null) { clearInterval(symbolsTimer); symbolsTimer = null; } } -// Loaded on first use, so a session that never opens the scan tab never fetches its code. +// Loaded on first use, so a session that never opens a tab never fetches its code, and the +// chart library is not downloaded by someone who only wanted the TCA panels. let scanModule = null; +let ohlcModule = null; const VIEWS = { tca: { view: "view-tca", ctl: "ctl-tca", @@ -244,6 +246,14 @@ const VIEWS = { scan: { view: "view-scan", ctl: "ctl-scan", start: async () => { scanModule ??= await import("/scan.js"); }, stop: () => scanModule?.stopScan() }, + rt: { view: "view-rt", ctl: "ctl-rt", + start: async () => { + // The chart measures the container when it is built, so the view must already be + // visible: show() unhides before calling start(), which is what makes this work. + if (ohlcModule) ohlcModule.startOhlc(); + else ohlcModule = await import("/ohlc.js"); + }, + stop: () => ohlcModule?.stopOhlc() }, }; let current = null; @@ -271,4 +281,4 @@ tabsEl.addEventListener("click", (ev) => { if (btn && !btn.disabled && VIEWS[btn.dataset.tab]) show(btn.dataset.tab); }); -await show(params.get("tab") === "scan" ? "scan" : "tca"); +await show(VIEWS[params.get("tab")] ? params.get("tab") : "tca"); diff --git a/js/public/index.html b/js/public/index.html index 7d09170..a70cac6 100644 --- a/js/public/index.html +++ b/js/public/index.html @@ -77,13 +77,18 @@ .stat b { display:block; font-size:21px; font-weight:600; line-height:1.3; font-variant-numeric:tabular-nums; } .stat span { color:var(--dim); font-size:11.5px; } + + /* The chart owns its height; the library sizes itself to this box and redraws on resize. */ + #rt-chart { height:min(62vh, 620px); width:100%; } + #ctl-rt input[type=checkbox] { accent-color:var(--hi); margin:0; } + #rt-panel { position:relative; }

QuestDB QWP

@@ -107,6 +112,12 @@

QuestDB QWP

+ + + + + + + + + + + + + + @@ -151,6 +153,8 @@

QuestDB QWP

OHLC

+
scroll to zoom · drag to pan · double-click to + reset · drag the axes to stretch them
diff --git a/js/public/ohlc.js b/js/public/ohlc.js index 8c38392..713361c 100644 --- a/js/public/ohlc.js +++ b/js/public/ohlc.js @@ -14,6 +14,8 @@ const intervalEl = document.getElementById("rt-interval"); const lookbackEl = document.getElementById("rt-lookback"); const goEl = document.getElementById("rt-go"); const fitEl = document.getElementById("rt-fit"); +const zoomInEl = document.getElementById("rt-zoomin"); +const zoomOutEl = document.getElementById("rt-zoomout"); const liveEl = document.getElementById("rt-live"); const sqlBtn = document.getElementById("rt-sql"); const sqlEl = document.getElementById("rt-sqltext"); @@ -31,6 +33,8 @@ const int = (n) => Number(n).toLocaleString("en-US"); let chart = null, candles = null, vwapLine = null, volume = null; let timer = null, lastBarTime = 0, decimals = 5, loading = false; +// Running VWAP totals carried past the loaded window, so the live tail can extend the line. +let vwapPv = 0, vwapQty = 0; /** Built once. Rebuilding per reload would throw the viewport away on every parameter change. */ function ensureChart() { @@ -40,7 +44,12 @@ function ensureChart() { fontFamily: "ui-monospace, SFMono-Regular, Menlo, monospace" }, grid: { vertLines: { color: LINE }, horzLines: { color: LINE } }, rightPriceScale: { borderColor: LINE }, - timeScale: { borderColor: LINE, timeVisible: true, secondsVisible: true }, + timeScale: { + borderColor: LINE, timeVisible: true, secondsVisible: true, + // A live chart that does not follow the newest bar is just a static one that happens + // to be mutating off-screen. + shiftVisibleRangeOnNewBar: true, rightOffset: 3, + }, crosshair: { mode: 0 }, // free crosshair: read any price, not just a bar's close // The point of the tab: wheel-zoom, drag-pan and pinch all on, with kinetic scrolling // so a flick keeps gliding. @@ -114,7 +123,7 @@ function toSeries(columns, rows) { if (qty > 0) vwaps.push({ time, value: pv / qty }); vols.push({ time, value: vol, color: close >= open ? "#1f3f38" : "#3f2420" }); } - return { bars, vwaps, vols }; + return { bars, vwaps, vols, pv, qty }; } /** FX quotes need five decimals; JPY crosses need three. Taken from the data, not guessed. */ @@ -148,7 +157,10 @@ async function load({ keepView = false } = {}) { sqlEl.textContent = body.sql ?? ""; ensureChart(); - const { bars, vwaps, vols } = toSeries(body.columns ?? [], body.rows ?? []); + const series = toSeries(body.columns ?? [], body.rows ?? []); + const { bars, vwaps, vols } = series; + vwapPv = series.pv; + vwapQty = series.qty; decimals = decimalsFor(bars); candles.applyOptions({ priceFormat: { type: "price", precision: decimals, minMove: Number(`1e-${decimals}`) } }); @@ -177,7 +189,9 @@ async function load({ keepView = false } = {}) { stat("query", `${Math.round(body.ms)}ms`, "3 statements, 1 connection"), stat("last", bars.at(-1) ? bars.at(-1).close.toFixed(decimals) : "-", vwaps.at(-1) ? `vwap ${vwaps.at(-1).value.toFixed(decimals)}` : ""), + `
-newest row
`, ].join(""); + showAge(); noteEl.textContent = bars.length ? "" : (body.empty ?? "no bars in this window"); } catch (error) { statsEl.innerHTML = ""; @@ -204,15 +218,52 @@ async function tailTick() { const body = await (await fetch(`/api/ohlc?${qs}`)).json(); if (body.error || !body.rows?.length) return; const { bars, vols } = toSeries(body.columns, body.rows); - for (const bar of bars) { - if (bar.time < lastBarTime) continue; // update() refuses to go backwards - candles.update(bar); - lastBarTime = bar.time; + const first = bars.findIndex((b) => b.time >= lastBarTime); + if (first < 0) return; + for (let i = first; i < bars.length; i++) { + candles.update(bars[i]); + const vol = vols[i]; + if (vol) volume.update(vol); + // Extend the session VWAP too. It is accumulated over the LOADED window, so the tail's + // own running total cannot be reused; this carries the line forward from the totals the + // full load ended on. Without it the one line meant to move stopped at load time. + const barVwap = Number(body.rows[i]?.[body.columns.indexOf("vwap")]); + const barVol = Number(body.rows[i]?.[body.columns.indexOf("volume")]) || 0; + if (Number.isFinite(barVwap) && bars[i].time > lastBarTime) { + vwapPv += barVwap * barVol; + vwapQty += barVol; + } + if (vwapQty > 0) vwapLine.update({ time: bars[i].time, value: vwapPv / vwapQty }); + lastBarTime = bars[i].time; } - for (const v of vols) if (v.time >= lastBarTime) volume.update(v); + showAge(); } catch { /* a dropped tick is not worth interrupting the chart for */ } } +/** + * How far behind the newest row is. + * + * "The chart looks static" has two completely different causes: the page is not tailing, or + * nothing is being written. Without this they are indistinguishable, so the age is shown and + * it keeps counting whether or not bars arrive. A number climbing past a few seconds means + * the writer is idle, not that the chart is broken. + */ +function showAge() { + const tile = document.getElementById("rt-age"); + if (!tile) return; + if (!lastBarTime) { tile.innerHTML = "-newest row"; return; } + const seconds = Math.max(0, Math.round(Date.now() / 1000 - lastBarTime)); + const text = seconds < 90 ? `${seconds}s` + : seconds < 5400 ? `${Math.round(seconds / 60)}m` + : `${(seconds / 3600).toFixed(1)}h`; + const state = seconds <= 15 ? "live" : "writer idle or paused"; + tile.innerHTML = `${text}behind newest row · ${state}`; +} + +// Ticks regardless of whether data arrives, so a stalled writer shows as a rising number +// rather than as a page that appears frozen. +setInterval(showAge, 1000); + function setLive(on) { if (timer !== null) { clearInterval(timer); timer = null; } if (on) timer = setInterval(tailTick, 1000); @@ -220,6 +271,19 @@ function setLive(on) { goEl.addEventListener("click", () => load({ keepView: false })); fitEl.addEventListener("click", () => chart?.timeScale().fitContent()); + +// Wheel and pinch are the natural gestures, but nothing on screen says so, and a trackpad +// pinch is not obvious either. These drive the same visible range the wheel does. +function zoomBy(factor) { + const scale = chart?.timeScale(); + const range = scale?.getVisibleLogicalRange(); + if (!range) return; + const middle = (range.from + range.to) / 2; + const half = ((range.to - range.from) / 2) * factor; + scale.setVisibleLogicalRange({ from: middle - half, to: middle + half }); +} +zoomInEl.addEventListener("click", () => zoomBy(0.6)); +zoomOutEl.addEventListener("click", () => zoomBy(1 / 0.6)); symEl.addEventListener("change", () => load()); intervalEl.addEventListener("change", () => load()); lookbackEl.addEventListener("change", () => load()); @@ -237,7 +301,11 @@ lookbackEl.innerHTML = options.lookbacks .map((l) => `${l}`).join(""); await load(); +setLive(liveEl.checked); // the tab ships live: a realtime chart should arrive moving /** Leaving the tab stops the live tail: nothing queries off-screen. */ -export function stopOhlc() { setLive(false); liveEl.checked = false; } -export function startOhlc() { if (chart) load({ keepView: true }); } +export function stopOhlc() { setLive(false); } +export function startOhlc() { + if (chart) load({ keepView: true }); + setLive(liveEl.checked); +} From 4b5761b3a0692c51d83bf416623d78b6ca4d6297 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 1 Oct 2026 15:14:00 +0200 Subject: [PATCH 63/79] js: make the OHLC tab tick visibly Default to 1s bars over a 5m window, and make the live poll rate a control (100-1000ms, default 250ms) instead of a hardcoded second. The bar width caps how often a new candle appears; the poll caps how often the in-progress one is redrawn, and both were too slow to look alive. Tail queries now cover 1m rather than 5m, so the faster poll is cheaper. Adds 1m and 2m windows, and a legend naming what the candles, the orange line and the volume bars are. --- js/public/index.html | 13 +++++++++++++ js/public/ohlc.js | 18 ++++++++++++++---- js/queries.mjs | 3 ++- 3 files changed, 29 insertions(+), 5 deletions(-) diff --git a/js/public/index.html b/js/public/index.html index e99fa2a..b3231df 100644 --- a/js/public/index.html +++ b/js/public/index.html @@ -144,6 +144,12 @@

QuestDB QWP

+ @@ -151,6 +157,13 @@

QuestDB QWP

OHLC

+
+ up bar + down bar + open / high / low / close of every trade in the bar + session VWAP + volume below +
scroll to zoom · drag to pan · double-click to diff --git a/js/public/ohlc.js b/js/public/ohlc.js index 713361c..19020e1 100644 --- a/js/public/ohlc.js +++ b/js/public/ohlc.js @@ -17,6 +17,7 @@ const fitEl = document.getElementById("rt-fit"); const zoomInEl = document.getElementById("rt-zoomin"); const zoomOutEl = document.getElementById("rt-zoomout"); const liveEl = document.getElementById("rt-live"); +const tickEl = document.getElementById("rt-tick"); const sqlBtn = document.getElementById("rt-sql"); const sqlEl = document.getElementById("rt-sqltext"); const statsEl = document.getElementById("rt-stats"); @@ -213,7 +214,7 @@ async function tailTick() { if (loading || !chart) return; try { const qs = new URLSearchParams({ - symbol: symEl.value || "", interval: intervalEl.value, lookback: "5m", + symbol: symEl.value || "", interval: intervalEl.value, lookback: "1m", }); const body = await (await fetch(`/api/ohlc?${qs}`)).json(); if (body.error || !body.rows?.length) return; @@ -266,7 +267,11 @@ setInterval(showAge, 1000); function setLive(on) { if (timer !== null) { clearInterval(timer); timer = null; } - if (on) timer = setInterval(tailTick, 1000); + // Polling faster than the bar width is the point: between bar boundaries the newest candle + // still grows as trades land, so a 250ms poll on 1s bars redraws it four times before it + // closes. At 1000ms the candle only ever appeared finished, which is what made the chart + // look like it was barely ticking. + if (on) timer = setInterval(tailTick, Number(tickEl.value) || 250); } goEl.addEventListener("click", () => load({ keepView: false })); @@ -288,6 +293,7 @@ symEl.addEventListener("change", () => load()); intervalEl.addEventListener("change", () => load()); lookbackEl.addEventListener("change", () => load()); liveEl.addEventListener("change", () => setLive(liveEl.checked)); +tickEl.addEventListener("change", () => setLive(liveEl.checked)); sqlBtn.addEventListener("click", () => { sqlEl.hidden = !sqlEl.hidden; sqlBtn.textContent = sqlEl.hidden ? "SQL" : "hide"; @@ -295,10 +301,14 @@ sqlBtn.addEventListener("click", () => { // Populated from the server so the dropdowns cannot offer an interval the query would reject. const options = await (await fetch("/api/ohlc-options")).json(); +// Defaults chosen for MOTION, not for coverage. 5s bars over 30m is 360 candles a few +// pixels wide with a new one every five seconds, which reads as a still image. 1s bars over +// 5m is one new candle per second at a visible width, and the poll interval matches the bar +// width so every tick draws something. intervalEl.innerHTML = options.intervals - .map((i) => `${i}`).join(""); + .map((i) => `${i}`).join(""); lookbackEl.innerHTML = options.lookbacks - .map((l) => `${l}`).join(""); + .map((l) => `${l}`).join(""); await load(); setLive(liveEl.checked); // the tab ships live: a realtime chart should arrive moving diff --git a/js/queries.mjs b/js/queries.mjs index b975b2d..5c22b1c 100644 --- a/js/queries.mjs +++ b/js/queries.mjs @@ -169,7 +169,8 @@ export const OHLC_INTERVALS = { /** Lookback windows offered, mapped to their span in seconds. */ export const OHLC_LOOKBACKS = { - "5m": 300, "30m": 1800, "2h": 7200, "6h": 21600, "24h": 86400, "7d": 604800, + "1m": 60, "2m": 120, "5m": 300, "30m": 1800, "2h": 7200, "6h": 21600, "24h": 86400, + "7d": 604800, }; /** From 3f7eb59c02e94997c61eeffa62725393b7cfaa84 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 1 Oct 2026 15:17:45 +0200 Subject: [PATCH 64/79] js: keep the OHLC chart ticking when zoomed, and cut the tail to one query The chart only auto-followed while the viewport sat at the last bar, so zooming in looked like it had stopped: bars kept arriving, just outside the visible range. It now tracks whether the right edge is in view and pans to the newest bar without changing the zoom, and the hint line says which mode it is in. Fit and double-click resume following. Each live poll ran three statements - anchor, symbol list, bars - of which two are useless for a tail. /api/ohlc-tail runs one, anchored on $now. Locally that is 17ms against 7ms, but each statement is a separate round trip, so on a remote cluster it is the difference between one latency and three per poll. --- js/data.mjs | 15 ++++++++++++++- js/public/ohlc.js | 46 ++++++++++++++++++++++++++++++++++++++++++---- js/queries.mjs | 28 ++++++++++++++++++++++++++++ js/server.mjs | 16 ++++++++++++++++ 4 files changed, 100 insertions(+), 5 deletions(-) diff --git a/js/data.mjs b/js/data.mjs index a42e998..899137d 100644 --- a/js/data.mjs +++ b/js/data.mjs @@ -7,7 +7,7 @@ import { connectQwpNodeClient, QwpEgressQueryError } from "@questdb/nodejs-client"; import { OHLC_INTERVALS, OHLC_LOOKBACKS, PANELS, ohlcLastRowSql, ohlcSql, ohlcSymbolsSql, - scanChunks, sqlFor, symbolsSql, + ohlcTailSql, scanChunks, sqlFor, symbolsSql, } from "./queries.mjs"; /** @@ -249,6 +249,19 @@ export class Data { } } + /** The live tail: one statement, one lease, nothing else. */ + async ohlcTail({ symbol, interval = "1s", seconds = 60 }) { + const started = performance.now(); + const lease = await this.db.borrowQuery(); + try { + const bars = await this.#run(lease, ohlcTailSql({ symbol, interval, seconds })); + return { symbol, interval, columns: bars.columns, rows: bars.rows, + ms: performance.now() - started }; + } finally { + await lease.close(); + } + } + async close() { if (this.db) await this.db.close(); this.db = null; diff --git a/js/public/ohlc.js b/js/public/ohlc.js index 19020e1..f947fa3 100644 --- a/js/public/ohlc.js +++ b/js/public/ohlc.js @@ -36,6 +36,10 @@ let chart = null, candles = null, vwapLine = null, volume = null; let timer = null, lastBarTime = 0, decimals = 5, loading = false; // Running VWAP totals carried past the loaded window, so the live tail can extend the line. let vwapPv = 0, vwapQty = 0; +// How many bars the series holds, and whether the viewport is still parked at the newest one. +// Zooming or panning away deliberately stops the chart following; sitting at the right edge +// keeps it following, at whatever zoom the user chose. +let barCount = 0, following = true; /** Built once. Rebuilding per reload would throw the viewport away on every parameter change. */ function ensureChart() { @@ -80,9 +84,22 @@ function ensureChart() { chart.priceScale("vol").applyOptions({ scaleMargins: { top: 0.82, bottom: 0 } }); chart.subscribeCrosshairMove(onCrosshair); + + // The library only auto-shifts while the range sits at the last bar, so zooming in used to + // look like the chart had stopped: bars kept arriving, just outside the viewport. Track + // whether the right edge is still in view and, when it is, keep pulling the view along. + chart.timeScale().subscribeVisibleLogicalRangeChange((range) => { + if (!range || !barCount) return; + following = range.to >= barCount - 2; + showFollowing(); + }); // Wheel-zoom has no natural way back, and reloading to escape a zoom would be absurd. // Double-click anywhere on the plot restores the full window, same as the Fit button. - chartEl.addEventListener("dblclick", () => chart.timeScale().fitContent()); + chartEl.addEventListener("dblclick", () => { + chart.timeScale().fitContent(); + following = true; + showFollowing(); + }); new ResizeObserver(() => chart.applyOptions({ width: chartEl.clientWidth })) .observe(chartEl); } @@ -176,6 +193,9 @@ async function load({ keepView = false } = {}) { else chart.timeScale().fitContent(); lastBarTime = bars.at(-1)?.time ?? 0; + barCount = bars.length; + if (!keepView) following = true; + showFollowing(); const span = bars.length ? `${new Date(bars[0].time * 1000).toISOString().slice(0, 19).replace("T", " ")} .. ${ @@ -214,9 +234,9 @@ async function tailTick() { if (loading || !chart) return; try { const qs = new URLSearchParams({ - symbol: symEl.value || "", interval: intervalEl.value, lookback: "1m", + symbol: symEl.value || "", interval: intervalEl.value, seconds: "60", }); - const body = await (await fetch(`/api/ohlc?${qs}`)).json(); + const body = await (await fetch(`/api/ohlc-tail?${qs}`)).json(); if (body.error || !body.rows?.length) return; const { bars, vols } = toSeries(body.columns, body.rows); const first = bars.findIndex((b) => b.time >= lastBarTime); @@ -235,8 +255,11 @@ async function tailTick() { vwapQty += barVol; } if (vwapQty > 0) vwapLine.update({ time: bars[i].time, value: vwapPv / vwapQty }); + if (bars[i].time > lastBarTime) barCount += 1; lastBarTime = bars[i].time; } + // Pans to the newest bar WITHOUT changing the zoom, so a zoomed-in view keeps ticking. + if (following) chart.timeScale().scrollToRealTime(); showAge(); } catch { /* a dropped tick is not worth interrupting the chart for */ } } @@ -249,6 +272,17 @@ async function tailTick() { * it keeps counting whether or not bars arrive. A number climbing past a few seconds means * the writer is idle, not that the chart is broken. */ +/** Says whether the chart is tracking the newest bar, because that is not obvious. */ +function showFollowing() { + const hint = document.getElementById("rt-hint"); + if (!hint) return; + hint.innerHTML = following + ? "scroll to zoom \u00b7 drag to pan \u00b7 double-click to reset \u00b7 " + + "following the newest bar" + : "scroll to zoom \u00b7 drag to pan \u00b7 " + + "scrolled back, not following \u00b7 double-click or Fit to catch up"; +} + function showAge() { const tile = document.getElementById("rt-age"); if (!tile) return; @@ -275,7 +309,11 @@ function setLive(on) { } goEl.addEventListener("click", () => load({ keepView: false })); -fitEl.addEventListener("click", () => chart?.timeScale().fitContent()); +fitEl.addEventListener("click", () => { + chart?.timeScale().fitContent(); + following = true; + showFollowing(); +}); // Wheel and pinch are the natural gestures, but nothing on screen says so, and a trackpad // pinch is not obvious either. These drive the same visible range the wheel does. diff --git a/js/queries.mjs b/js/queries.mjs index 5c22b1c..21971a2 100644 --- a/js/queries.mjs +++ b/js/queries.mjs @@ -232,3 +232,31 @@ WHERE symbol = '${symbol}' SAMPLE BY ${interval} ORDER BY timestamp`; }; + +/** + * The newest bars only, for the live tail. + * + * ONE statement, and no anchor query: the tail is for a chart that is keeping up with a live + * writer, so $now is the right edge by definition. /api/ohlc runs three statements because it + * also has to find where the data ends and which symbols are trading; paying that four times + * a second was what made a 250ms poll feel sluggish rather than snappy. + */ +export const ohlcTailSql = ({ symbol, interval, seconds = 60 }) => { + if (!(interval in OHLC_INTERVALS)) throw new Error(`unknown interval: ${interval}`); + if (!symbol) throw new Error("a symbol is required"); + return ` +SELECT + timestamp, + first(price) AS open, + max(price) AS high, + min(price) AS low, + last(price) AS close, + sum(quantity) AS volume, + sum(price * quantity) / sum(quantity) AS vwap, + count() AS trades +FROM fx_trades +WHERE symbol = '${symbol}' + AND timestamp IN '$now-${Math.max(1, Math.trunc(seconds))}s..$now' +SAMPLE BY ${interval} +ORDER BY timestamp`; +}; diff --git a/js/server.mjs b/js/server.mjs index 7e33247..5d2c17c 100644 --- a/js/server.mjs +++ b/js/server.mjs @@ -131,6 +131,22 @@ const server = createServer(async (req, res) => { return; } + if (url.pathname === "/api/ohlc-tail") { + try { + const payload = await data.ohlcTail({ + symbol: url.searchParams.get("symbol") || symbol || "", + interval: url.searchParams.get("interval") ?? "1s", + seconds: Number(url.searchParams.get("seconds") ?? 60), + }); + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify(payload, bigints)); + } catch (error) { + res.writeHead(503, { "content-type": "application/json" }); + res.end(JSON.stringify({ error: String(error?.message ?? error) })); + } + return; + } + if (url.pathname === "/api/ohlc-options") { res.writeHead(200, { "content-type": "application/json" }); res.end(JSON.stringify({ From b5af0d9b18c212c72c33a76e3e90185251e7a272 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 1 Oct 2026 15:25:50 +0200 Subject: [PATCH 65/79] js: make the OHLC tab tick between candles A 1s candle closes once a second however fast the page polls, so candles alone look like a slideshow and no poll rate fixes that. The newest quote does change with every tick, so it is now polled on its own: bid and ask are drawn as price lines with axis labels and a bid/ask/mid/spread readout, and they move on every poll while the candles close on their own cadence. Quotes come from a single-row LATEST ON against core_price, so polling at 100ms is cheap; bars are refetched about once a bar instead of once a poll, which they were, needlessly. The freshness tile now also reports measured polls per second, the round trip, and how long since a new candle. 'It feels slow' has three causes - the browser throttling the timer, a slow round trip, or the bar width - and they cannot be told apart by eye. --- js/data.mjs | 23 +++++++++++++++- js/public/index.html | 1 + js/public/ohlc.js | 65 ++++++++++++++++++++++++++++++++++++++++++-- js/queries.mjs | 18 ++++++++++++ js/server.mjs | 12 ++++++++ 5 files changed, 115 insertions(+), 4 deletions(-) diff --git a/js/data.mjs b/js/data.mjs index 899137d..093c5dd 100644 --- a/js/data.mjs +++ b/js/data.mjs @@ -7,7 +7,7 @@ import { connectQwpNodeClient, QwpEgressQueryError } from "@questdb/nodejs-client"; import { OHLC_INTERVALS, OHLC_LOOKBACKS, PANELS, ohlcLastRowSql, ohlcSql, ohlcSymbolsSql, - ohlcTailSql, scanChunks, sqlFor, symbolsSql, + latestQuoteSql, ohlcTailSql, scanChunks, sqlFor, symbolsSql, } from "./queries.mjs"; /** @@ -262,6 +262,27 @@ export class Data { } } + /** One row: the newest bid/ask. Polled far faster than the bars. */ + async quote(symbol) { + const started = performance.now(); + const lease = await this.db.borrowQuery(); + try { + const { columns, rows } = await this.#run(lease, latestQuoteSql(symbol)); + const row = rows[0]; + if (!row) return { symbol, ms: performance.now() - started }; + const ix = Object.fromEntries(columns.map((c, i) => [c, i])); + return { + symbol, + timestamp: row[ix.timestamp], + bid: Number(row[ix.bid]), + ask: Number(row[ix.ask]), + ms: performance.now() - started, + }; + } finally { + await lease.close(); + } + } + async close() { if (this.db) await this.db.close(); this.db = null; diff --git a/js/public/index.html b/js/public/index.html index b3231df..de14cf5 100644 --- a/js/public/index.html +++ b/js/public/index.html @@ -165,6 +165,7 @@

OHLC

volume below
+
scroll to zoom · drag to pan · double-click to reset · drag the axes to stretch them
diff --git a/js/public/ohlc.js b/js/public/ohlc.js index f947fa3..c0deb5e 100644 --- a/js/public/ohlc.js +++ b/js/public/ohlc.js @@ -24,6 +24,7 @@ const statsEl = document.getElementById("rt-stats"); const titleEl = document.getElementById("rt-title"); const noteEl = document.getElementById("rt-note"); const chartEl = document.getElementById("rt-chart"); +const quoteEl = document.getElementById("rt-quote"); const INK = "#d7dce3", DIM = "#78828f", LINE = "#242a33", PANEL = "#161a21"; const UP = "#4ec9a5", DOWN = "#e2705f", VWAP = "#c98500", VOL = "#2f3a4a"; @@ -40,6 +41,13 @@ let vwapPv = 0, vwapQty = 0; // Zooming or panning away deliberately stops the chart following; sitting at the right edge // keeps it following, at whatever zoom the user chose. let barCount = 0, following = true; +// Poll instrumentation. "It feels slow" has several possible causes - the browser throttling +// the timer, a slow round trip, or simply one new candle per second because that is the bar +// width - and they are indistinguishable by eye. These are measured and shown. +let pollTimes = [], pollMs = 0, barsSeen = 0, barsAt = Date.now(); +// The moving bid/ask lines, and when the bars were last refetched. Quotes are polled every +// tick; bars only need refetching about once a bar, since that is all they can change. +let bidLine = null, askLine = null, lastBars = 0; /** Built once. Rebuilding per reload would throw the viewport away on every parameter change. */ function ensureChart() { @@ -83,6 +91,13 @@ function ensureChart() { }); chart.priceScale("vol").applyOptions({ scaleMargins: { top: 0.82, bottom: 0 } }); + // Two lines that move with every quote. A 1s candle closes once a second however fast the + // page polls, so without these the chart looks like a slideshow between boundaries. + bidLine = candles.createPriceLine({ price: 0, color: UP, lineWidth: 1, lineStyle: 2, + axisLabelVisible: true, title: "bid" }); + askLine = candles.createPriceLine({ price: 0, color: DOWN, lineWidth: 1, lineStyle: 2, + axisLabelVisible: true, title: "ask" }); + chart.subscribeCrosshairMove(onCrosshair); // The library only auto-shifts while the range sits at the last bar, so zooming in used to @@ -230,13 +245,45 @@ async function load({ keepView = false } = {}) { * replaces the last bar WITHOUT touching the viewport. Re-running setData every second would * fight whatever the user is currently zoomed into. */ +/** The fast half: one row, so it can be polled as often as the user likes. */ +async function quoteTick() { + if (!chart) return; + const startedAt = performance.now(); + try { + const qs = new URLSearchParams({ symbol: symEl.value || "" }); + const q = await (await fetch(`/api/quote?${qs}`)).json(); + if (q.error || !Number.isFinite(q.bid)) return; + pollMs = performance.now() - startedAt; + const now = Date.now(); + pollTimes.push(now); + while (pollTimes.length && now - pollTimes[0] > 3000) pollTimes.shift(); + + bidLine.applyOptions({ price: q.bid }); + askLine.applyOptions({ price: q.ask }); + const mid = ((q.bid + q.ask) / 2).toFixed(decimals); + const spread = ((q.ask - q.bid) * 1e4).toFixed(2); + quoteEl.innerHTML = `bid ${q.bid.toFixed(decimals)}  ` + + `ask ${q.ask.toFixed(decimals)}  mid ${mid}  ` + + `spread ${spread} bps`; + showAge(); + } catch { /* a dropped quote is not worth interrupting the chart for */ } +} + async function tailTick() { if (loading || !chart) return; + // Bars can only change once per bar width, so refetching them at the quote rate is waste. + if (Date.now() - lastBars < 900) return; + lastBars = Date.now(); + const startedAt = performance.now(); try { const qs = new URLSearchParams({ symbol: symEl.value || "", interval: intervalEl.value, seconds: "60", }); const body = await (await fetch(`/api/ohlc-tail?${qs}`)).json(); + pollMs = performance.now() - startedAt; + const now = Date.now(); + pollTimes.push(now); + while (pollTimes.length && now - pollTimes[0] > 3000) pollTimes.shift(); if (body.error || !body.rows?.length) return; const { bars, vols } = toSeries(body.columns, body.rows); const first = bars.findIndex((b) => b.time >= lastBarTime); @@ -255,7 +302,7 @@ async function tailTick() { vwapQty += barVol; } if (vwapQty > 0) vwapLine.update({ time: bars[i].time, value: vwapPv / vwapQty }); - if (bars[i].time > lastBarTime) barCount += 1; + if (bars[i].time > lastBarTime) { barCount += 1; barsSeen += 1; barsAt = Date.now(); } lastBarTime = bars[i].time; } // Pans to the newest bar WITHOUT changing the zoom, so a zoomed-in view keeps ticking. @@ -292,7 +339,16 @@ function showAge() { : seconds < 5400 ? `${Math.round(seconds / 60)}m` : `${(seconds / 3600).toFixed(1)}h`; const state = seconds <= 15 ? "live" : "writer idle or paused"; - tile.innerHTML = `${text}behind newest row · ${state}`; + // Polls per second over a 3s window, the measured round trip, and how long since a NEW + // candle appeared. If polls/s is high but new bars are 1/s, the bar width is the limit, + // not the polling. + const rate = pollTimes.length > 1 + ? (1000 * (pollTimes.length - 1) / Math.max(1, pollTimes.at(-1) - pollTimes[0])).toFixed(1) + : "0"; + const sinceBar = ((Date.now() - barsAt) / 1000).toFixed(1); + tile.innerHTML = `${text}behind newest row · ${state}
` + + `${rate}/s polls · ${pollMs.toFixed(0)}ms rtt · new bar ${sinceBar}s ago` + + ``; } // Ticks regardless of whether data arrives, so a stalled writer shows as a rising number @@ -305,7 +361,10 @@ function setLive(on) { // still grows as trades land, so a 250ms poll on 1s bars redraws it four times before it // closes. At 1000ms the candle only ever appeared finished, which is what made the chart // look like it was barely ticking. - if (on) timer = setInterval(tailTick, Number(tickEl.value) || 250); + if (on) { + const every = Number(tickEl.value) || 250; + timer = setInterval(() => { quoteTick(); tailTick(); }, every); + } } goEl.addEventListener("click", () => load({ keepView: false })); diff --git a/js/queries.mjs b/js/queries.mjs index 21971a2..39fa0a0 100644 --- a/js/queries.mjs +++ b/js/queries.mjs @@ -260,3 +260,21 @@ WHERE symbol = '${symbol}' SAMPLE BY ${interval} ORDER BY timestamp`; }; + +/** + * The newest quote for one instrument: one row, read straight off the tick table. + * + * This is what makes the chart feel alive. A 1s candle can only ever close once a second, so + * candles alone look like a slideshow however fast the page polls. The quote changes with + * every tick the venue sends, so polling THIS at 100ms gives continuous motion between + * candle boundaries. LATEST ON walks the symbol index backwards, so it stays a single-row + * lookup no matter how large the table is. + */ +export const latestQuoteSql = (symbol) => { + if (!symbol) throw new Error("a symbol is required"); + return ` +SELECT timestamp, bid_price AS bid, ask_price AS ask +FROM core_price +WHERE symbol = '${symbol}' +LATEST ON timestamp PARTITION BY symbol`; +}; diff --git a/js/server.mjs b/js/server.mjs index 5d2c17c..bab85fd 100644 --- a/js/server.mjs +++ b/js/server.mjs @@ -147,6 +147,18 @@ const server = createServer(async (req, res) => { return; } + if (url.pathname === "/api/quote") { + try { + const payload = await data.quote(url.searchParams.get("symbol") || symbol || ""); + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify(payload, bigints)); + } catch (error) { + res.writeHead(503, { "content-type": "application/json" }); + res.end(JSON.stringify({ error: String(error?.message ?? error) })); + } + return; + } + if (url.pathname === "/api/ohlc-options") { res.writeHead(200, { "content-type": "application/json" }); res.end(JSON.stringify({ From c9e1b2c6b4fd7d99249b592b73cbd8b9c8457167 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 1 Oct 2026 16:13:55 +0200 Subject: [PATCH 66/79] python: silence the Categorical deprecation from the client's to_polars() polars now deprecates casting UInt32 into a Categorical, which the QWP client does internally for symbol columns, once per query. tca_live reprints every tick, so the dashboard was buried in it. The cast is inside the client, so the scripts can only filter the warning; matched on its message so other deprecations still surface. --- python/backfill.py | 10 ++++++++++ python/csv_columnar_sender.py | 10 ++++++++++ python/dataframe_demo.py | 10 ++++++++++ python/enrich_polars_demo.py | 10 ++++++++++ python/read_bench.py | 10 ++++++++++ python/slippage.py | 10 ++++++++++ python/slippage_parallel.py | 10 ++++++++++ python/slippage_parallel_stream.py | 10 ++++++++++ python/slippage_stream.py | 10 ++++++++++ python/start_sending.sh | 6 ++++++ python/tca_live.py | 10 ++++++++++ 11 files changed, 106 insertions(+) create mode 100644 python/start_sending.sh diff --git a/python/backfill.py b/python/backfill.py index d61efc4..107e5d1 100644 --- a/python/backfill.py +++ b/python/backfill.py @@ -29,6 +29,16 @@ import questdb +import warnings + +# The QWP client's to_polars() casts symbol columns into a per-connection Categorical, which +# polars now deprecates in favour of .cat.to(). The cast happens inside the client, not here, +# so suppressing it is all this script can do; a dashboard that reprints every tick would +# otherwise be buried in it. Matched on the message so other deprecations still surface. +warnings.filterwarnings( + "ignore", category=DeprecationWarning, message="casting from UInt32 to Categorical" +) + NS_PER_DAY = 86_400_000_000_000 diff --git a/python/csv_columnar_sender.py b/python/csv_columnar_sender.py index fdbcf19..6d1f327 100644 --- a/python/csv_columnar_sender.py +++ b/python/csv_columnar_sender.py @@ -55,6 +55,16 @@ import questdb +import warnings + +# The QWP client's to_polars() casts symbol columns into a per-connection Categorical, which +# polars now deprecates in favour of .cat.to(). The cast happens inside the client, not here, +# so suppressing it is all this script can do; a dashboard that reprints every tick would +# otherwise be buried in it. Matched on the message so other deprecations still surface. +warnings.filterwarnings( + "ignore", category=DeprecationWarning, message="casting from UInt32 to Categorical" +) + def use_tls(args): return bool(args.tls or args.token or (args.username and args.password)) diff --git a/python/dataframe_demo.py b/python/dataframe_demo.py index c9476ec..3db7ebf 100644 --- a/python/dataframe_demo.py +++ b/python/dataframe_demo.py @@ -31,6 +31,16 @@ import questdb +import warnings + +# The QWP client's to_polars() casts symbol columns into a per-connection Categorical, which +# polars now deprecates in favour of .cat.to(). The cast happens inside the client, not here, +# so suppressing it is all this script can do; a dashboard that reprints every tick would +# otherwise be buried in it. Matched on the message so other deprecations still surface. +warnings.filterwarnings( + "ignore", category=DeprecationWarning, message="casting from UInt32 to Categorical" +) + TABLE = "df_demo" diff --git a/python/enrich_polars_demo.py b/python/enrich_polars_demo.py index 9d041b9..b9daa9a 100644 --- a/python/enrich_polars_demo.py +++ b/python/enrich_polars_demo.py @@ -57,6 +57,16 @@ import questdb +import warnings + +# The QWP client's to_polars() casts symbol columns into a per-connection Categorical, which +# polars now deprecates in favour of .cat.to(). The cast happens inside the client, not here, +# so suppressing it is all this script can do; a dashboard that reprints every tick would +# otherwise be buried in it. Matched on the message so other deprecations still surface. +warnings.filterwarnings( + "ignore", category=DeprecationWarning, message="casting from UInt32 to Categorical" +) + LETTERS = np.array(list("ABCDEFGHIJKLMNOPQRSTUVWXYZ")) diff --git a/python/read_bench.py b/python/read_bench.py index 989603a..e7f8c92 100644 --- a/python/read_bench.py +++ b/python/read_bench.py @@ -70,6 +70,16 @@ import questdb +import warnings + +# The QWP client's to_polars() casts symbol columns into a per-connection Categorical, which +# polars now deprecates in favour of .cat.to(). The cast happens inside the client, not here, +# so suppressing it is all this script can do; a dashboard that reprints every tick would +# otherwise be buried in it. Matched on the message so other deprecations still surface. +warnings.filterwarnings( + "ignore", category=DeprecationWarning, message="casting from UInt32 to Categorical" +) + def use_tls(args): return bool(args.tls or args.token or (args.username and args.password)) diff --git a/python/slippage.py b/python/slippage.py index 9ab809f..309fa24 100644 --- a/python/slippage.py +++ b/python/slippage.py @@ -11,6 +11,16 @@ import questdb +import warnings + +# The QWP client's to_polars() casts symbol columns into a per-connection Categorical, which +# polars now deprecates in favour of .cat.to(). The cast happens inside the client, not here, +# so suppressing it is all this script can do; a dashboard that reprints every tick would +# otherwise be buried in it. Matched on the message so other deprecations still surface. +warnings.filterwarnings( + "ignore", category=DeprecationWarning, message="casting from UInt32 to Categorical" +) + SQL = """ SELECT t.timestamp, diff --git a/python/slippage_parallel.py b/python/slippage_parallel.py index 6e1694c..0279d18 100644 --- a/python/slippage_parallel.py +++ b/python/slippage_parallel.py @@ -25,6 +25,16 @@ import polars as pl import questdb +import warnings + +# The QWP client's to_polars() casts symbol columns into a per-connection Categorical, which +# polars now deprecates in favour of .cat.to(). The cast happens inside the client, not here, +# so suppressing it is all this script can do; a dashboard that reprints every tick would +# otherwise be buried in it. Matched on the message so other deprecations still surface. +warnings.filterwarnings( + "ignore", category=DeprecationWarning, message="casting from UInt32 to Categorical" +) + # SYMBOL columns arrive as polars Categorical, and each connection's result carries # its own Categories identity (a per-result UUID in the questdb_symbol namespace). # pl.concat refuses to mix them: "Categories name mismatch ... failed to vstack diff --git a/python/slippage_parallel_stream.py b/python/slippage_parallel_stream.py index e44b991..08d146b 100644 --- a/python/slippage_parallel_stream.py +++ b/python/slippage_parallel_stream.py @@ -30,6 +30,16 @@ import polars as pl import questdb +import warnings + +# The QWP client's to_polars() casts symbol columns into a per-connection Categorical, which +# polars now deprecates in favour of .cat.to(). The cast happens inside the client, not here, +# so suppressing it is all this script can do; a dashboard that reprints every tick would +# otherwise be buried in it. Matched on the message so other deprecations still surface. +warnings.filterwarnings( + "ignore", category=DeprecationWarning, message="casting from UInt32 to Categorical" +) + READERS = int(sys.argv[1]) if len(sys.argv) > 1 else 8 QUEUE_DEPTH = 8 # batches a worker may run ahead before it blocks DONE = object() # sentinel: this range produced its last batch diff --git a/python/slippage_stream.py b/python/slippage_stream.py index 6b114a7..0ea9dc4 100644 --- a/python/slippage_stream.py +++ b/python/slippage_stream.py @@ -14,6 +14,16 @@ import questdb +import warnings + +# The QWP client's to_polars() casts symbol columns into a per-connection Categorical, which +# polars now deprecates in favour of .cat.to(). The cast happens inside the client, not here, +# so suppressing it is all this script can do; a dashboard that reprints every tick would +# otherwise be buried in it. Matched on the message so other deprecations still surface. +warnings.filterwarnings( + "ignore", category=DeprecationWarning, message="casting from UInt32 to Categorical" +) + SQL = """ SELECT t.timestamp, diff --git a/python/start_sending.sh b/python/start_sending.sh new file mode 100644 index 0000000..419a6f2 --- /dev/null +++ b/python/start_sending.sh @@ -0,0 +1,6 @@ + python csv_columnar_sender.py \ + --addrs localhost:9000 \ + --total-events 10000000 \ + --num-senders 1 \ + --chunk-rows 100000 \ + --csv ../trades20250728.csv.gz diff --git a/python/tca_live.py b/python/tca_live.py index e334031..0e181d6 100644 --- a/python/tca_live.py +++ b/python/tca_live.py @@ -33,6 +33,16 @@ import polars as pl import questdb +import warnings + +# The QWP client's to_polars() casts symbol columns into a per-connection Categorical, which +# polars now deprecates in favour of .cat.to(). The cast happens inside the client, not here, +# so suppressing it is all this script can do; a dashboard that reprints every tick would +# otherwise be buried in it. Matched on the message so other deprecations still surface. +warnings.filterwarnings( + "ignore", category=DeprecationWarning, message="casting from UInt32 to Categorical" +) + # ---------------------------------------------------------------- queries # Kept verbatim as authored, including the window literals, so what the dashboard runs # is exactly what you would paste into the console. All reduction happens in polars. From 812b33858d4b1878e72e971c5852ad0cf21d04fd Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 1 Oct 2026 18:14:38 +0200 Subject: [PATCH 67/79] js: report transfer rate in the scan, add --help and a README The scan now reports decoded bytes alongside rows, as GiB total plus MB/s and Gb/s, measured the way python/read_bench.py measures it so the two are comparable: valuesBytes() and nullBitmapBytes() are views over buffers the client has already decoded, so it copies nothing and costs one pass over the columns per batch rather than the rows. It is the decoded payload, not the compressed wire bytes, which the client does not expose. Cross-checks at 68.0 bytes a row for core_price, matching both the schema and the Python figure. --help lists every flag and endpoint and exits before connecting, so it answers with no database reachable. --- js/README.md | 101 ++++++++++++++++++++++++++++++++++++++++++ js/data.mjs | 28 +++++++++++- js/public/ohlc.js | 55 +++++++++++++++-------- js/public/scan.js | 8 ++++ js/scan-worker.mjs | 106 +++++++++++++++++++++++++++++++++++++++++++++ js/server.mjs | 99 ++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 378 insertions(+), 19 deletions(-) create mode 100644 js/README.md create mode 100644 js/scan-worker.mjs diff --git a/js/README.md b/js/README.md new file mode 100644 index 0000000..2d74911 --- /dev/null +++ b/js/README.md @@ -0,0 +1,101 @@ +# js — QWP demos in a browser + +Three tabs over one QWP connection: live TCA panels, a streaming scan, and an OHLC +chart. `node server.mjs --help` lists every flag; this file covers setup and the +things that are not obvious from the flag list. + +## Why there is a server at all + +QuestDB accepts a browser WebSocket upgrade only when the request's `Origin` matches +its `Host`, which blocks cross-site WebSocket hijacking and answers a cross-origin +upgrade with HTTP 400. A different port is a different origin, so a page on `:8080` +cannot open a QWP socket to QuestDB on `:9000` however local it is. (CORS never +applies — WebSocket upgrades are not subject to it.) + +So Node holds the QWP connection and the page talks to Node. `data.mjs` is the only +module that knows QuestDB exists, so when `@questdb/browser-client` is published and +the server carries the upgrade patch, the query moves into the page and nothing else +changes. + +## Install + +Two dependencies: the QWP client, and TradingView Lightweight Charts for the OHLC +tab. No framework, no bundler, no build step — the page is plain ES modules. + +The client is not on npm yet, so build it from source: + +```sh +git clone && cd nodejs-questdb-client +pnpm install # needs Node >= 20.18.1 +pnpm --filter @questdb/nodejs-client build +``` + +`pnpm install` at the **root** is required first: `bunchee`, `rollup` and +`typescript` are devDependencies of the workspace, so building from inside +`packages/nodejs-client` will not resolve them. + +Then point this app at wherever you cloned it and install: + +```sh +cd js +npm pkg set dependencies.@questdb/nodejs-client=file:/path/to/nodejs-questdb-client/packages/nodejs-client +npm install +``` + +That edits one line of `package.json`; it is a local diff, not something to commit. + +## Run + +```sh +npm start # localhost:9000, http://localhost:8080 +npm start -- --scheme wss --addrs a:9000,b:9000,c:9000 --token-file ~/token.txt +``` + +Pass the token with `--token-file`, never on the command line: argv is readable +through `ps`, and the configuration string is logged at startup (the token is +replaced with `***`). + +`--addrs` takes the comma-separated list the client resolves with ordered failover, +so naming every node is how read HA is exercised rather than merely claimed. + +## Tabs + +**TCA live** — slippage, two markout panels and a local min/max, the same SQL as +`python/tca_live.py`. Each panel shows the statement it is running, fetched from the +same builder the query path uses so the two cannot drift. + +**Streaming scan** — reads the last N rows of a table and reports only how far it has +got, so neither side holds the result. 200M rows of `core_price` run at ~826k rows/s +against a remote cluster with the server at 147MB RSS: memory tracks reader count, +not rows read. + +**OHLC & VWAP** — candles with a session VWAP and a volume histogram, zoom and pan by +wheel, drag or the buttons. Bid and ask arrive on a pushed stream and move between +candle boundaries. + +## Things worth knowing + +**Compression is not free.** `--compression zstd` is worth ~1.9x over a WAN, where +bandwidth is the constraint, and costs 2.9x on a LAN, where it is not: the client's +zstd decoder is pure JavaScript on the event-loop thread. + +**Readers do not scale past a core.** Parallel readers overlap I/O, not CPU — the +client decodes in JavaScript on one thread. Throughput plateaus around 4 readers +(measured 3.10M rows/s at 4 and 3.11M at 16). + +**The timestamp column dominates the scan.** The designated timestamp is Gorilla +encoded, so it cannot be handed over as a view and is unpacked per value with BigInt +arithmetic. Every other fixed-width column is zero copy. The scan tab's `columns` +control trades this off: `all` 2.5M rows/s, `ts as epoch long` 6.3M, `no timestamp` +8.7M. + +**A chart cannot tick faster than the data becomes visible.** Measured against a +cluster ingesting 26 quotes/s per symbol, the newest visible row advanced exactly +once per second, in steps. Polling faster than that only re-reads identical data. If +the chart looks static, read the freshness tile — it reports how far behind the +newest row is, the measured polls per second, the round trip, and how long since a +new candle, which is enough to tell the three causes apart. + +**Materialized views may not be readable.** `bbo_1s` and `core_price_1s` return +`Access denied` for the `kafka` user on the demo cluster, so the app reads base +tables. diff --git a/js/data.mjs b/js/data.mjs index 093c5dd..f59cc2c 100644 --- a/js/data.mjs +++ b/js/data.mjs @@ -28,6 +28,28 @@ function copyTail(batch, sample) { return out; } +/** + * Decoded size of one batch, the Node equivalent of Arrow's `batch.nbytes` that + * python/read_bench.py reports, so the two are comparable. + * + * valuesBytes() and nullBitmapBytes() hand back views over the already-decoded buffers, so + * this copies nothing and costs one pass over the COLUMNS, not the rows. SYMBOL returns no + * values buffer because its ids are varints on the wire; decoded they are Int32 row ids, so + * they are counted as four bytes a row. This is the decoded payload, not the compressed wire + * bytes, which the client does not expose. A VARCHAR column would count its offset table but + * not its string payload; none of the scanned tables has one. + */ +function batchBytes(batch) { + let bytes = 0; + for (let c = 0; c < batch.columnCount; c++) { + const column = batch.column(c); + const values = column.valuesBytes(); + bytes += values ? values.byteLength : column.rowCount * 4; + bytes += column.nullBitmapBytes()?.byteLength ?? 0; + } + return bytes; +} + export class Data { constructor(conf) { this.conf = conf; @@ -124,10 +146,11 @@ export class Data { const sqls = scanChunks(table, limit, chunks || Math.max(workers, sliced), projection); const started = performance.now(); - let rows = 0, batches = 0, columns = null, tail = [], chunksDone = 0, lastSent = 0; + let rows = 0, batches = 0, bytes = 0, columns = null, tail = [], chunksDone = 0; + let lastSent = 0; const snapshot = (done) => ({ table, sql: sqls[0] ?? "", chunks: sqls.length, chunksDone, readers: workers, - projection, + projection, bytes, columns, rows, batches, tail, done, ms: performance.now() - started, }); @@ -149,6 +172,7 @@ export class Data { // cost per batch does not depend on how many rows it carries. rows += batch.rowCount; batches += 1; + bytes += batchBytes(batch); const now = performance.now(); if (now - lastSent < progressMs) return; diff --git a/js/public/ohlc.js b/js/public/ohlc.js index c0deb5e..38a4fbb 100644 --- a/js/public/ohlc.js +++ b/js/public/ohlc.js @@ -245,19 +245,38 @@ async function load({ keepView = false } = {}) { * replaces the last bar WITHOUT touching the viewport. Re-running setData every second would * fight whatever the user is currently zoomed into. */ -/** The fast half: one row, so it can be polled as often as the user likes. */ -async function quoteTick() { +/** + * The fast half, PUSHED rather than polled. + * + * One connection stays open and the server sends a frame whenever the quote changes, so the + * browser's distance from the server delays updates by a constant instead of capping their + * rate. A laptop 250ms from the server still sees every change the server saw. + */ +let quoteSource = null; + +function openQuoteStream() { + closeQuoteStream(); if (!chart) return; - const startedAt = performance.now(); - try { - const qs = new URLSearchParams({ symbol: symEl.value || "" }); - const q = await (await fetch(`/api/quote?${qs}`)).json(); - if (q.error || !Number.isFinite(q.bid)) return; - pollMs = performance.now() - startedAt; - const now = Date.now(); - pollTimes.push(now); - while (pollTimes.length && now - pollTimes[0] > 3000) pollTimes.shift(); + const qs = new URLSearchParams({ + symbol: symEl.value || "", every: String(Number(tickEl.value) || 250), + }); + quoteSource = new EventSource(`/api/quote-stream?${qs}`); + quoteSource.addEventListener("quote", (ev) => applyQuote(JSON.parse(ev.data))); +} +function closeQuoteStream() { + quoteSource?.close(); + quoteSource = null; +} + +function applyQuote(q) { + if (!chart || !Number.isFinite(q.bid)) return; + const now = Date.now(); + pollMs = Number(q.ms) || 0; + pollTimes.push(now); + while (pollTimes.length && now - pollTimes[0] > 3000) pollTimes.shift(); + + { bidLine.applyOptions({ price: q.bid }); askLine.applyOptions({ price: q.ask }); const mid = ((q.bid + q.ask) / 2).toFixed(decimals); @@ -266,7 +285,7 @@ async function quoteTick() { + `ask ${q.ask.toFixed(decimals)}  mid ${mid}  ` + `spread ${spread} bps`; showAge(); - } catch { /* a dropped quote is not worth interrupting the chart for */ } + } } async function tailTick() { @@ -362,8 +381,10 @@ function setLive(on) { // closes. At 1000ms the candle only ever appeared finished, which is what made the chart // look like it was barely ticking. if (on) { - const every = Number(tickEl.value) || 250; - timer = setInterval(() => { quoteTick(); tailTick(); }, every); + timer = setInterval(tailTick, 1000); // bars change once a bar, nothing is gained by more + openQuoteStream(); + } else { + closeQuoteStream(); } } @@ -386,11 +407,11 @@ function zoomBy(factor) { } zoomInEl.addEventListener("click", () => zoomBy(0.6)); zoomOutEl.addEventListener("click", () => zoomBy(1 / 0.6)); -symEl.addEventListener("change", () => load()); +symEl.addEventListener("change", async () => { await load(); setLive(liveEl.checked); }); intervalEl.addEventListener("change", () => load()); lookbackEl.addEventListener("change", () => load()); liveEl.addEventListener("change", () => setLive(liveEl.checked)); -tickEl.addEventListener("change", () => setLive(liveEl.checked)); +tickEl.addEventListener("change", () => setLive(liveEl.checked)); // restarts the stream sqlBtn.addEventListener("click", () => { sqlEl.hidden = !sqlEl.hidden; sqlBtn.textContent = sqlEl.hidden ? "SQL" : "hide"; @@ -411,7 +432,7 @@ await load(); setLive(liveEl.checked); // the tab ships live: a realtime chart should arrive moving /** Leaving the tab stops the live tail: nothing queries off-screen. */ -export function stopOhlc() { setLive(false); } +export function stopOhlc() { setLive(false); closeQuoteStream(); } export function startOhlc() { if (chart) load({ keepView: true }); setLive(liveEl.checked); diff --git a/js/public/scan.js b/js/public/scan.js index df16218..85f4170 100644 --- a/js/public/scan.js +++ b/js/public/scan.js @@ -36,6 +36,11 @@ const rate = (n) => const secs = (ms) => (ms >= 1000 ? `${(ms / 1000).toFixed(2)}s` : `${ms.toFixed(0)}ms`); +// Decoded volume, reported the same way python/read_bench.py does so the two are comparable: +// GiB for the total a human reads, MB/s and Gb/s for the rate. +const gib = (b) => b / 1024 ** 3; +const volume = (b) => (gib(b) >= 1 ? `${gib(b).toFixed(2)} GiB` : `${(b / 1e6).toFixed(0)} MB`); + function fmt(v) { if (v === null || v === undefined) return "null"; if (typeof v === "number") return Number.isInteger(v) ? String(v) : v.toFixed(5); @@ -62,6 +67,9 @@ function render(p, state) { stat("throughput", `${rate(rowsPerSec)} rows/s`, p.readers ? `${p.readers} reader${p.readers === 1 ? "" : "s"}${ p.projection && p.projection !== "all" ? ` \u00b7 ${p.projection}` : ""}` : ""), + stat("transfer", p.ms > 0 ? `${((p.bytes ?? 0) * 8 / p.ms / 1e6).toFixed(2)} Gb/s` : "-", + `${volume(p.bytes ?? 0)} decoded \u00b7 ${ + p.ms > 0 ? ((p.bytes ?? 0) / p.ms / 1e3).toFixed(0) : 0} MB/s`), ].join(""); const head = (p.columns ?? []).map((c) => `${esc(c)}`).join(""); diff --git a/js/scan-worker.mjs b/js/scan-worker.mjs new file mode 100644 index 0000000..9bcdcf6 --- /dev/null +++ b/js/scan-worker.mjs @@ -0,0 +1,106 @@ +// One scan worker: its OWN QWP client, on its own thread. +// +// Why a thread at all. The client decodes result batches in JavaScript, and zstd is a pure-JS +// decoder too, so decoding is CPU work on whatever thread owns the connection. Several +// "readers" sharing one event loop therefore only overlap I/O waits, and on a fast link there +// is no wait left to hide: measured throughput was identical at 4 and 16 readers (3.10M vs +// 3.11M rows/s). Real parallelism needs real threads. +// +// Workers never send rows to the main thread on the hot path. Counters live in shared memory +// and are bumped with Atomics, so a batch costs two atomic adds and nothing is serialised. +import { parentPort, workerData } from "node:worker_threads"; +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +const { conf, id } = workerData; + +// Shared counter slots, by index. Written by every worker, read by the main thread. +const ROWS = 0, BATCHES = 1, CHUNKS = 2; +// Shared control slots: the work-queue cursor, and the abort flag. +const CURSOR = 0, ABORTED = 1; + +let db = null; +let current = null; // the in-flight query, so an abort can cancel it +let aborting = false; + +/** Copy the last `sample` rows out of the reusable view before it is recycled. */ +function copyTail(batch, sample) { + const from = Math.max(0, batch.rowCount - sample); + const out = []; + for (let r = from; r < batch.rowCount; r++) { + const row = new Array(batch.columnCount); + for (let c = 0; c < batch.columnCount; c++) row[c] = batch.get(r, c); + out.push(row); + } + return out; +} + +async function run({ sqls, counters, control, sample, tailEveryMs }) { + const counts = new BigInt64Array(counters); + const ctl = new Int32Array(control); + let lastTail = 0; + + const lease = await db.borrowQuery(); + try { + for (;;) { + if (Atomics.load(ctl, ABORTED) === 1) break; + // Work STEALING, not a fixed split: slices of equal row count are not slices of equal + // time, so a worker that drew cheap ones keeps pulling instead of idling. + const index = Atomics.add(ctl, CURSOR, 1); + if (index >= sqls.length) break; + + const query = await lease.queryViews(sqls[index], (batch) => { + Atomics.add(counts, ROWS, BigInt(batch.rowCount)); + Atomics.add(counts, BATCHES, 1n); + + // The only rows that ever cross the thread boundary, a few times a second. + const now = Date.now(); + if (now - lastTail < tailEveryMs) return; + lastTail = now; + parentPort.postMessage({ + type: "tail", + columns: batch.columns.map((c) => c.name), + rows: copyTail(batch, sample), + }); + }, { timeoutMs: 0 }); + + current = query; + try { + await query.completion; + } finally { + current = null; + } + Atomics.add(counts, CHUNKS, 1n); + } + } finally { + await lease.close(); + } +} + +parentPort.on("message", async (msg) => { + try { + if (msg.type === "abort") { + aborting = true; + await current?.cancel().catch(() => {}); + return; + } + if (msg.type === "run") { + aborting = false; + await run(msg); + parentPort.postMessage({ type: "done" }); + } + } catch (error) { + // An abort tears a query down on purpose; that is not a failure worth reporting. + parentPort.postMessage({ + type: "done", + error: aborting ? null : String(error?.message ?? error), + }); + } +}); + +// Connect once, at startup: a scan should not pay for a handshake per run. +try { + db = await connectQwpNodeClient(conf); + parentPort.postMessage({ type: "ready", id }); +} catch (error) { + parentPort.postMessage({ type: "ready", id, error: String(error?.message ?? error) }); +} diff --git a/js/server.mjs b/js/server.mjs index bab85fd..45134a8 100644 --- a/js/server.mjs +++ b/js/server.mjs @@ -24,6 +24,60 @@ const arg = (name, fallback) => { return i >= 0 && process.argv[i + 1] ? process.argv[i + 1] : fallback; }; +// --help before anything else, so it answers without needing a reachable database. +if (process.argv.includes("--help") || process.argv.includes("-h")) { + console.log(` +questdb-tca-web - three QWP demos over one connection: TCA panels, a streaming +scan, and a live OHLC chart. + + node server.mjs [options] npm start -- [options] + +CONNECTION + --scheme ws|wss transport; wss turns on TLS (default ws) + --addrs host:port,... cluster nodes, tried in order with failover + (default localhost:9000) + --addr host:port single node; alias for --addrs + --token-file PATH file holding the bearer token. Preferred over putting + a token in the command line, which ps exposes + --tls-verify on|unsafe_off + certificate checking, wss only (default unsafe_off, + because these clusters use self-signed certificates) + --compression zstd compress result batches. Worth ~1.9x on a WAN; a 2.9x + LOSS on a LAN, where the decoder is the bottleneck + --max-batch-rows N rows per result batch + --query-pool-max N concurrent query connections; caps --readers in the + scan tab (default 16) + --conf STRING a full QWP configuration string, overriding all of the + above except --token-file + +SERVER + --port N HTTP port (default 8080) + --symbol SYM force every tab to one instrument + --help, -h this text + +EXAMPLES + node server.mjs + node server.mjs --scheme wss --addrs a:9000,b:9000,c:9000 --token-file ~/token.txt + npm start -- --port 9999 --symbol EURUSD + +HTTP ENDPOINTS + / the dashboard + /api/tick one pass of the four TCA panels + /api/symbols instruments trading in the last 30s + /api/sql?panel= the exact statement a panel runs + /api/scan streaming scan, server-sent events + table= rows= readers= chunk_rows= projection= + /api/scan-tables tables the scan offers + /api/ohlc candles + VWAP for one instrument + symbol= interval= lookback= + /api/ohlc-tail newest candles only, one statement + /api/ohlc-options intervals and windows offered + /api/quote newest bid/ask, single row + /api/quote-stream newest bid/ask pushed as they change +`.trim()); + process.exit(0); +} + const port = Number(arg("port", 8080)); const symbol = arg("symbol", null); @@ -147,6 +201,51 @@ const server = createServer(async (req, res) => { return; } + // Quotes PUSHED over SSE instead of polled. + // + // A polling browser pays its round trip to this server once per update, so a laptop on a + // WAN talking to a server beside QuestDB can never tick faster than that latency allows - + // a 250ms timer over a 250ms link is 2 updates a second, not 4. Here the server does the + // polling, on the LAN where it costs a few ms, and pushes every change down one already + // open connection. The browser's latency then delays the stream by a constant, instead of + // dividing its rate. + if (url.pathname === "/api/quote-stream") { + const sym = url.searchParams.get("symbol") || symbol || ""; + const everyMs = Math.min(1000, Math.max(50, Number(url.searchParams.get("every") ?? 100))); + res.writeHead(200, { + "content-type": "text/event-stream", + "cache-control": "no-cache", + connection: "keep-alive", + "x-accel-buffering": "no", + }); + + let stop = false; + res.on("close", () => { stop = true; }); + + // Only CHANGES are sent. A venue that has not moved should not cost a frame. + let lastSent = ""; + (async () => { + while (!stop) { + const startedAt = Date.now(); + try { + const q = await data.quote(sym); + const key = `${q.bid}|${q.ask}|${q.timestamp}`; + if (key !== lastSent && Number.isFinite(q.bid)) { + lastSent = key; + res.write(`event: quote\ndata: ${JSON.stringify(q, bigints)}\n\n`); + } + } catch (error) { + if (!stop) res.write(`event: failed\ndata: ${JSON.stringify({ + error: String(error?.message ?? error) })}\n\n`); + } + const elapsed = Date.now() - startedAt; + if (elapsed < everyMs) await new Promise((r) => setTimeout(r, everyMs - elapsed)); + } + res.end(); + })(); + return; + } + if (url.pathname === "/api/quote") { try { const payload = await data.quote(url.searchParams.get("symbol") || symbol || ""); From 8ca9b7510ad5f52ebefaef63a26c639c637d5f37 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 1 Oct 2026 18:19:09 +0200 Subject: [PATCH 68/79] js: default the OHLC window to 1m, and offer 20s and 30s How alive a candle chart looks is governed by how WIDE each bar is, not by how fast the data arrives. The default was 1s bars over 5m: 300 candles a few pixels across, where a new one per second is invisible. The QuestDB console panel this was compared against fits 20 bars into roughly 650px, about 30px each; this chart is roughly 1500px, so about 50 bars matches it. 1m gives 53. Separately, and measured: new rows become visible once per second on both tables - 43 reads in 6s returned 7 distinct values, stepping in ~1.0s gaps - so polling faster than that re-reads identical data. That is a ceiling on candle cadence, but it was not what made the chart look static. --- js/public/ohlc.js | 11 ++++++----- js/queries.mjs | 9 +++++++-- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/js/public/ohlc.js b/js/public/ohlc.js index 38a4fbb..35d6b93 100644 --- a/js/public/ohlc.js +++ b/js/public/ohlc.js @@ -419,14 +419,15 @@ sqlBtn.addEventListener("click", () => { // Populated from the server so the dropdowns cannot offer an interval the query would reject. const options = await (await fetch("/api/ohlc-options")).json(); -// Defaults chosen for MOTION, not for coverage. 5s bars over 30m is 360 candles a few -// pixels wide with a new one every five seconds, which reads as a still image. 1s bars over -// 5m is one new candle per second at a visible width, and the poll interval matches the bar -// width so every tick draws something. +// Defaults chosen for MOTION, not for coverage. What makes a candle chart look alive is how +// WIDE each bar is: a new bar per second reads as a step when bars are tens of pixels across +// and as nothing when they are five. The console panel this was compared against fits 20 bars +// into ~650px, about 30px each; this chart is roughly 1500px, so ~60 bars matches that. Hence +// 1s bars over 1m, with 20s and 30s for a closer look and minutes-to-days for context. intervalEl.innerHTML = options.intervals .map((i) => `${i}`).join(""); lookbackEl.innerHTML = options.lookbacks - .map((l) => `${l}`).join(""); + .map((l) => `${l}`).join(""); await load(); setLive(liveEl.checked); // the tab ships live: a realtime chart should arrive moving diff --git a/js/queries.mjs b/js/queries.mjs index 39fa0a0..2bf6811 100644 --- a/js/queries.mjs +++ b/js/queries.mjs @@ -168,9 +168,14 @@ export const OHLC_INTERVALS = { }; /** Lookback windows offered, mapped to their span in seconds. */ +// Short windows first, and they are the point. How alive a candle chart looks is governed by +// how WIDE each bar is, not by how fast the data arrives: 1s bars over 5m is 300 candles a +// few pixels across, where a new one per second is invisible, while the same bars over 20s +// is 20 candles filling the chart and each one lands as a visible step. The QuestDB console +// panel this was compared against uses a 20s window. export const OHLC_LOOKBACKS = { - "1m": 60, "2m": 120, "5m": 300, "30m": 1800, "2h": 7200, "6h": 21600, "24h": 86400, - "7d": 604800, + "20s": 20, "30s": 30, "1m": 60, "2m": 120, "5m": 300, "30m": 1800, "2h": 7200, + "6h": 21600, "24h": 86400, "7d": 604800, }; /** From d31a6e808b7d2b52ed07f6afd12825b6be0e7e2d Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 1 Oct 2026 18:42:08 +0200 Subject: [PATCH 69/79] js: push the OHLC bars instead of polling them on a timer Three things pinned the newest candle to one update a second regardless of how fresh the data was. Bars were fetched on setInterval(1000), on the reasoning that a bar can only change once per bar width. Only CLOSED bars are immutable; the one being formed changes with every trade that lands inside it, so it has to be redrawn at the poll rate. Even corrected, a timer is throttled to 1Hz by the browser in a background tab. Measured in one page at one moment: bars 1.0/s on a timer against quotes 5.1/s on a stream. Bars now ride the same stream, which is not throttled and costs no round trip per update. The stream then sent no bars at all, because the change-detection key used JSON.stringify without the BigInt replacer: result rows carry BigInt, the throw was swallowed into a 'failed' event, and the symptom was 20 quote frames, zero bar frames and 20 failures. --- js/public/ohlc.js | 46 ++++++++++++++++++++++------------------------ js/server.mjs | 22 +++++++++++++++++++--- 2 files changed, 41 insertions(+), 27 deletions(-) diff --git a/js/public/ohlc.js b/js/public/ohlc.js index 35d6b93..3aae0c5 100644 --- a/js/public/ohlc.js +++ b/js/public/ohlc.js @@ -47,7 +47,7 @@ let barCount = 0, following = true; let pollTimes = [], pollMs = 0, barsSeen = 0, barsAt = Date.now(); // The moving bid/ask lines, and when the bars were last refetched. Quotes are polled every // tick; bars only need refetching about once a bar, since that is all they can change. -let bidLine = null, askLine = null, lastBars = 0; +let bidLine = null, askLine = null, tailInFlight = false; /** Built once. Rebuilding per reload would throw the viewport away on every parameter change. */ function ensureChart() { @@ -258,10 +258,12 @@ function openQuoteStream() { closeQuoteStream(); if (!chart) return; const qs = new URLSearchParams({ - symbol: symEl.value || "", every: String(Number(tickEl.value) || 250), + symbol: symEl.value || "", interval: intervalEl.value, + every: String(Number(tickEl.value) || 250), }); quoteSource = new EventSource(`/api/quote-stream?${qs}`); quoteSource.addEventListener("quote", (ev) => applyQuote(JSON.parse(ev.data))); + quoteSource.addEventListener("bars", (ev) => applyBars(JSON.parse(ev.data))); } function closeQuoteStream() { @@ -288,22 +290,23 @@ function applyQuote(q) { } } -async function tailTick() { +/** + * Newest bars, PUSHED. + * + * Only CLOSED bars are immutable: the one currently forming changes with every trade that + * lands inside it, so it has to be redrawn continuously, not once a bar. It used to be + * fetched on a timer, which a browser throttles to 1Hz in a background tab - so the newest + * candle crawled however fresh the data was. The server polls instead and pushes changes. + */ +function applyBars(body) { if (loading || !chart) return; - // Bars can only change once per bar width, so refetching them at the quote rate is waste. - if (Date.now() - lastBars < 900) return; - lastBars = Date.now(); - const startedAt = performance.now(); - try { - const qs = new URLSearchParams({ - symbol: symEl.value || "", interval: intervalEl.value, seconds: "60", - }); - const body = await (await fetch(`/api/ohlc-tail?${qs}`)).json(); - pollMs = performance.now() - startedAt; - const now = Date.now(); - pollTimes.push(now); - while (pollTimes.length && now - pollTimes[0] > 3000) pollTimes.shift(); - if (body.error || !body.rows?.length) return; + if (body.error || !body.rows?.length) return; + + const now = Date.now(); + pollTimes.push(now); + while (pollTimes.length && now - pollTimes[0] > 3000) pollTimes.shift(); + + { const { bars, vols } = toSeries(body.columns, body.rows); const first = bars.findIndex((b) => b.time >= lastBarTime); if (first < 0) return; @@ -327,7 +330,7 @@ async function tailTick() { // Pans to the newest bar WITHOUT changing the zoom, so a zoomed-in view keeps ticking. if (following) chart.timeScale().scrollToRealTime(); showAge(); - } catch { /* a dropped tick is not worth interrupting the chart for */ } + } } /** @@ -380,12 +383,7 @@ function setLive(on) { // still grows as trades land, so a 250ms poll on 1s bars redraws it four times before it // closes. At 1000ms the candle only ever appeared finished, which is what made the chart // look like it was barely ticking. - if (on) { - timer = setInterval(tailTick, 1000); // bars change once a bar, nothing is gained by more - openQuoteStream(); - } else { - closeQuoteStream(); - } + if (on) openQuoteStream(); else closeQuoteStream(); } goEl.addEventListener("click", () => load({ keepView: false })); diff --git a/js/server.mjs b/js/server.mjs index 45134a8..e6c4a5e 100644 --- a/js/server.mjs +++ b/js/server.mjs @@ -219,21 +219,37 @@ const server = createServer(async (req, res) => { "x-accel-buffering": "no", }); + // Bars ride the same stream as quotes, for the same reason plus one more: a browser + // throttles setInterval to 1Hz in a background tab, but does not throttle an incoming + // stream. Polling the bars on a timer pinned them to one update a second whenever the + // tab was not focused, however fresh the data was. + const interval = url.searchParams.get("interval") ?? "1s"; + let stop = false; res.on("close", () => { stop = true; }); // Only CHANGES are sent. A venue that has not moved should not cost a frame. - let lastSent = ""; + let lastQuote = "", lastBar = ""; (async () => { while (!stop) { const startedAt = Date.now(); try { const q = await data.quote(sym); const key = `${q.bid}|${q.ask}|${q.timestamp}`; - if (key !== lastSent && Number.isFinite(q.bid)) { - lastSent = key; + if (key !== lastQuote && Number.isFinite(q.bid)) { + lastQuote = key; res.write(`event: quote\ndata: ${JSON.stringify(q, bigints)}\n\n`); } + + const bars = await data.ohlcTail({ symbol: sym, interval, seconds: 60 }); + const newest = bars.rows?.at(-1); + // The replacer is not optional: result rows carry BigInt, which JSON.stringify + // refuses outright, and the throw was being swallowed as a "failed" event. + const barKey = newest ? JSON.stringify(newest, bigints) : ""; + if (barKey && barKey !== lastBar) { + lastBar = barKey; + res.write(`event: bars\ndata: ${JSON.stringify(bars, bigints)}\n\n`); + } } catch (error) { if (!stop) res.write(`event: failed\ndata: ${JSON.stringify({ error: String(error?.message ?? error) })}\n\n`); From 74c8346eed63bca85733fddde0d59864101b3e62 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 1 Oct 2026 18:43:41 +0200 Subject: [PATCH 70/79] js: correct the README's freshness claim, and record what actually caused it The old text asserted a 1Hz ceiling on visible data as if it were a property of QuestDB. It is a property of the WRITER: the same measurement that gave 7 distinct values in 43 reads on a sender flushing once a second gave 43 distinct values in 43 reads once ingestion moved to QWP flushing every 50ms. Adds the three findings that actually made the chart look slow - browsers throttle timers but not streams, only closed bars are immutable, and perceived liveness is mostly bar width. --- js/README.md | 40 ++++++++++++++++++++++++++++++++-------- 1 file changed, 32 insertions(+), 8 deletions(-) diff --git a/js/README.md b/js/README.md index 2d74911..dad608b 100644 --- a/js/README.md +++ b/js/README.md @@ -70,8 +70,10 @@ against a remote cluster with the server at 147MB RSS: memory tracks reader coun not rows read. **OHLC & VWAP** — candles with a session VWAP and a volume histogram, zoom and pan by -wheel, drag or the buttons. Bid and ask arrive on a pushed stream and move between -candle boundaries. +wheel, drag or the buttons, double-click or Fit to reset. Candles and bid/ask both +arrive on one pushed stream; the bid and ask lines move between candle boundaries, so +there is motion inside the second. Defaults to 1s bars over 1m, chosen for bar width +rather than coverage. ## Things worth knowing @@ -89,12 +91,34 @@ arithmetic. Every other fixed-width column is zero copy. The scan tab's `columns control trades this off: `all` 2.5M rows/s, `ts as epoch long` 6.3M, `no timestamp` 8.7M. -**A chart cannot tick faster than the data becomes visible.** Measured against a -cluster ingesting 26 quotes/s per symbol, the newest visible row advanced exactly -once per second, in steps. Polling faster than that only re-reads identical data. If -the chart looks static, read the freshness tile — it reports how far behind the -newest row is, the measured polls per second, the round trip, and how long since a -new candle, which is enough to tell the three causes apart. +**A chart cannot tick faster than the data becomes visible, and that is a property of +the WRITER.** With a sender on the default `auto_flush_interval=1000`, the newest +visible row advances in ~1.0s steps however fast rows are produced, and polling +faster only re-reads identical data: measured 7 distinct values across 43 reads in +6s. With QWP ingestion flushing every 50ms, the same measurement gave 43 distinct +values in 43 reads — the limit became the reader's own round trip. Check this before +blaming the chart; it is one query in a loop. + +**Live updates are pushed, not polled, and that is not an optimisation.** A browser +throttles `setInterval` to 1Hz in a background tab but does not throttle an incoming +stream: measured in one page at one moment, timer-driven bars ran at 1.0/s while +stream-driven quotes ran at 5.1/s. Pushing also means a browser far from the server +sees updates delayed by a constant rather than rate-limited by its round trip. Both +bars and quotes arrive on one SSE connection, and the poll-rate control sets how +often the SERVER polls QuestDB. + +**Only closed bars are immutable.** The bar currently being formed changes with every +trade that lands inside it, so it is redrawn at the stream rate. Refreshing bars +"once per bar" looks correct and pins the newest candle to the bar width instead. + +**How alive it looks is mostly bar WIDTH.** 1s bars over 5m is 300 candles a few +pixels across, where a new one per second is invisible; the same bars over 1m is ~50 +candles of ~30px, where each lands as a step. If it reads as static, widen the bars +before touching anything else. + +If the chart still looks wrong, read the freshness tile: it reports how far behind the +newest row is, the measured updates per second, the round trip, and how long since a +new candle, which is enough to tell these causes apart. **Materialized views may not be readable.** `bbo_1s` and `core_price_1s` return `Access denied` for the `kafka` user on the demo cluster, so the app reads base From f57fd9a0468b446c09918abb42bc2420dc6215e7 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 1 Oct 2026 18:50:19 +0200 Subject: [PATCH 71/79] js: drop the unused scan worker Start of a worker_threads scan that was never wired up, and swept into an earlier commit by an over-broad git add. Nothing imports it. The decision was not to pursue it: 8 vCPUs is about 4 physical cores, so threading buys perhaps 4-5x on a single-thread figure of 3.1M rows/s, and the link saturates around 12 Gb/s, which is where Python already tops out at 22.2M rows/s. The cheap half of that win is already shipped as the epoch-long projection. --- js/scan-worker.mjs | 106 --------------------------------------------- 1 file changed, 106 deletions(-) delete mode 100644 js/scan-worker.mjs diff --git a/js/scan-worker.mjs b/js/scan-worker.mjs deleted file mode 100644 index 9bcdcf6..0000000 --- a/js/scan-worker.mjs +++ /dev/null @@ -1,106 +0,0 @@ -// One scan worker: its OWN QWP client, on its own thread. -// -// Why a thread at all. The client decodes result batches in JavaScript, and zstd is a pure-JS -// decoder too, so decoding is CPU work on whatever thread owns the connection. Several -// "readers" sharing one event loop therefore only overlap I/O waits, and on a fast link there -// is no wait left to hide: measured throughput was identical at 4 and 16 readers (3.10M vs -// 3.11M rows/s). Real parallelism needs real threads. -// -// Workers never send rows to the main thread on the hot path. Counters live in shared memory -// and are bumped with Atomics, so a batch costs two atomic adds and nothing is serialised. -import { parentPort, workerData } from "node:worker_threads"; -import { connectQwpNodeClient } from "@questdb/nodejs-client"; - -const { conf, id } = workerData; - -// Shared counter slots, by index. Written by every worker, read by the main thread. -const ROWS = 0, BATCHES = 1, CHUNKS = 2; -// Shared control slots: the work-queue cursor, and the abort flag. -const CURSOR = 0, ABORTED = 1; - -let db = null; -let current = null; // the in-flight query, so an abort can cancel it -let aborting = false; - -/** Copy the last `sample` rows out of the reusable view before it is recycled. */ -function copyTail(batch, sample) { - const from = Math.max(0, batch.rowCount - sample); - const out = []; - for (let r = from; r < batch.rowCount; r++) { - const row = new Array(batch.columnCount); - for (let c = 0; c < batch.columnCount; c++) row[c] = batch.get(r, c); - out.push(row); - } - return out; -} - -async function run({ sqls, counters, control, sample, tailEveryMs }) { - const counts = new BigInt64Array(counters); - const ctl = new Int32Array(control); - let lastTail = 0; - - const lease = await db.borrowQuery(); - try { - for (;;) { - if (Atomics.load(ctl, ABORTED) === 1) break; - // Work STEALING, not a fixed split: slices of equal row count are not slices of equal - // time, so a worker that drew cheap ones keeps pulling instead of idling. - const index = Atomics.add(ctl, CURSOR, 1); - if (index >= sqls.length) break; - - const query = await lease.queryViews(sqls[index], (batch) => { - Atomics.add(counts, ROWS, BigInt(batch.rowCount)); - Atomics.add(counts, BATCHES, 1n); - - // The only rows that ever cross the thread boundary, a few times a second. - const now = Date.now(); - if (now - lastTail < tailEveryMs) return; - lastTail = now; - parentPort.postMessage({ - type: "tail", - columns: batch.columns.map((c) => c.name), - rows: copyTail(batch, sample), - }); - }, { timeoutMs: 0 }); - - current = query; - try { - await query.completion; - } finally { - current = null; - } - Atomics.add(counts, CHUNKS, 1n); - } - } finally { - await lease.close(); - } -} - -parentPort.on("message", async (msg) => { - try { - if (msg.type === "abort") { - aborting = true; - await current?.cancel().catch(() => {}); - return; - } - if (msg.type === "run") { - aborting = false; - await run(msg); - parentPort.postMessage({ type: "done" }); - } - } catch (error) { - // An abort tears a query down on purpose; that is not a failure worth reporting. - parentPort.postMessage({ - type: "done", - error: aborting ? null : String(error?.message ?? error), - }); - } -}); - -// Connect once, at startup: a scan should not pay for a handshake per run. -try { - db = await connectQwpNodeClient(conf); - parentPort.postMessage({ type: "ready", id }); -} catch (error) { - parentPort.postMessage({ type: "ready", id, error: String(error?.message ?? error) }); -} From a4c71c86fe26ecbcadeb21c010e3e95be51a19e1 Mon Sep 17 00:00:00 2001 From: javier Date: Thu, 1 Oct 2026 19:05:29 +0200 Subject: [PATCH 72/79] js: add a live top-of-book blotter to the OHLC tab One candle advancing once a second reads as static next to thirty rows whose prices, spreads and volumes all move at once, which is what makes the console's FX top of book feel alive. Same query, pushed on the same stream as the candles and quotes. Rows are built once and mutated in place: rebuilding the table would restart every animation and throw the flash away. A cell that moves twice quickly would only flash once, because re-adding a class does not restart a CSS animation, so the class is removed and a reflow forced first. The arrow outlives the flash so direction stays readable, and rows stay alphabetical so none jumps position when its price changes. --- js/data.mjs | 14 ++++++++- js/public/index.html | 20 +++++++++++++ js/public/ohlc.js | 67 ++++++++++++++++++++++++++++++++++++++++++++ js/queries.mjs | 18 ++++++++++++ js/server.mjs | 11 +++++++- 5 files changed, 128 insertions(+), 2 deletions(-) diff --git a/js/data.mjs b/js/data.mjs index f59cc2c..052b654 100644 --- a/js/data.mjs +++ b/js/data.mjs @@ -7,7 +7,7 @@ import { connectQwpNodeClient, QwpEgressQueryError } from "@questdb/nodejs-client"; import { OHLC_INTERVALS, OHLC_LOOKBACKS, PANELS, ohlcLastRowSql, ohlcSql, ohlcSymbolsSql, - latestQuoteSql, ohlcTailSql, scanChunks, sqlFor, symbolsSql, + latestQuoteSql, ohlcTailSql, scanChunks, sqlFor, symbolsSql, topOfBookSql, } from "./queries.mjs"; /** @@ -307,6 +307,18 @@ export class Data { } } + /** One row per instrument: the whole top of book. */ + async topOfBook(seconds = 5) { + const started = performance.now(); + const lease = await this.db.borrowQuery(); + try { + const { columns, rows } = await this.#run(lease, topOfBookSql(seconds)); + return { columns, rows, ms: performance.now() - started }; + } finally { + await lease.close(); + } + } + async close() { if (this.db) await this.db.close(); this.db = null; diff --git a/js/public/index.html b/js/public/index.html index de14cf5..e7e7197 100644 --- a/js/public/index.html +++ b/js/public/index.html @@ -47,6 +47,18 @@ margin:.35rem 0 .2rem; } .legend span { display:flex; align-items:center; gap:.35rem; } .legend i { width:9px; height:9px; border-radius:2px; display:inline-block; } + /* A changed cell flashes and keeps an arrow. The flash says "this just moved" at a glance + across thirty rows; the arrow outlives the flash so direction stays readable after. */ + @keyframes tick-up { from { background: rgba(78,201,165,.40); } to { background: none; } } + @keyframes tick-down { from { background: rgba(226,112,95,.40); } to { background: none; } } + td.up { animation: tick-up .65s ease-out; } + td.down { animation: tick-down .65s ease-out; } + td .arrow { font-size:9px; margin-left:.35rem; } + td.up .arrow { color:var(--pos); } + td.down .arrow { color:var(--neg); } + #rt-book-panel { margin-top:1rem; } + #rt-book td, #rt-book th { padding:.1rem .7rem; } + .err { color:var(--neg); } .stale { opacity:.45; } [hidden] { display:none !important; } @@ -169,6 +181,14 @@

OHLC

scroll to zoom · drag to pan · double-click to reset · drag the axes to stretch them
+
+ +
+

top of book

+ + + +
symbolbidaskspreadbid sizeask size
diff --git a/js/public/ohlc.js b/js/public/ohlc.js index 3aae0c5..b642555 100644 --- a/js/public/ohlc.js +++ b/js/public/ohlc.js @@ -25,6 +25,8 @@ const titleEl = document.getElementById("rt-title"); const noteEl = document.getElementById("rt-note"); const chartEl = document.getElementById("rt-chart"); const quoteEl = document.getElementById("rt-quote"); +const bookEl = document.getElementById("rt-book").tBodies[0]; +const bookNoteEl = document.getElementById("rt-book-note"); const INK = "#d7dce3", DIM = "#78828f", LINE = "#242a33", PANEL = "#161a21"; const UP = "#4ec9a5", DOWN = "#e2705f", VWAP = "#c98500", VOL = "#2f3a4a"; @@ -264,6 +266,7 @@ function openQuoteStream() { quoteSource = new EventSource(`/api/quote-stream?${qs}`); quoteSource.addEventListener("quote", (ev) => applyQuote(JSON.parse(ev.data))); quoteSource.addEventListener("bars", (ev) => applyBars(JSON.parse(ev.data))); + quoteSource.addEventListener("book", (ev) => applyBook(JSON.parse(ev.data))); } function closeQuoteStream() { @@ -333,6 +336,70 @@ function applyBars(body) { } } +// Previous values per symbol, so a cell can tell which way it moved. Rows are built once and +// then mutated in place: rebuilding the table every update would restart every animation and +// throw away the flash, which is the entire point of it. +const bookRows = new Map(); + +const BOOK_DECIMALS = { spread: 5 }; + +function cellText(column, value) { + if (value === null || value === undefined) return ""; + const n = Number(value); + if (!Number.isFinite(n)) return String(value); + if (column.endsWith("volume")) return n.toLocaleString("en-US"); + if (column === "spread") return n.toFixed(BOOK_DECIMALS.spread); + return n.toFixed(n >= 50 ? 3 : 5); +} + +/** Top of book: every instrument, with the cells that just moved flashing. */ +function applyBook(body) { + const columns = body.columns ?? []; + const ix = Object.fromEntries(columns.map((c, i) => [c, i])); + const order = ["bid_price", "ask_price", "spread", "bid_volume", "ask_volume"]; + + for (const row of body.rows ?? []) { + const symbol = String(row[ix.symbol]); + let entry = bookRows.get(symbol); + if (!entry) { + const tr = document.createElement("tr"); + const first = document.createElement("td"); + first.textContent = symbol; + tr.append(first); + const cells = {}; + for (const column of order) { + const td = document.createElement("td"); + tr.append(td); + cells[column] = td; + } + // Alphabetical, so a row never jumps position when a price changes. + const after = [...bookRows.keys()].concat(symbol).sort() + .indexOf(symbol); + bookEl.insertBefore(tr, bookEl.children[after] ?? null); + entry = { cells, last: {} }; + bookRows.set(symbol, entry); + } + + for (const column of order) { + const value = Number(row[ix[column]]); + const previous = entry.last[column]; + const td = entry.cells[column]; + if (previous === value) continue; + + const direction = previous === undefined ? "" : value > previous ? "up" : "down"; + td.innerHTML = `${cellText(column, value)}${ + direction ? `${direction === "up" ? "\u25b2" : "\u25bc"}` : ""}`; + // Removing the class and forcing a reflow restarts the animation; without it a cell + // that moves twice in quick succession only flashes once. + td.classList.remove("up", "down"); + if (direction) { void td.offsetWidth; td.classList.add(direction); } + entry.last[column] = value; + } + } + bookNoteEl.textContent = `${(body.rows ?? []).length} instruments \u00b7 ${ + Math.round(body.ms ?? 0)}ms`; +} + /** * How far behind the newest row is. * diff --git a/js/queries.mjs b/js/queries.mjs index 2bf6811..0bbce7c 100644 --- a/js/queries.mjs +++ b/js/queries.mjs @@ -283,3 +283,21 @@ FROM core_price WHERE symbol = '${symbol}' LATEST ON timestamp PARTITION BY symbol`; }; + +/** + * Top of book for every instrument: one row each, newest first. + * + * This is the liveliest thing on the page, and deliberately so. A single candle advancing + * once a second reads as static next to thirty rows whose prices, spreads and volumes all + * change at once. LATEST ON walks the symbol index backwards, so it stays one row per symbol + * however large the table is, and the few-second window keeps it to instruments actually + * quoting rather than every symbol the table has ever seen. + */ +export const topOfBookSql = (seconds = 5) => ` +SELECT symbol, bid_price, ask_price, + round(ask_price - bid_price, 5) AS spread, + bid_volume, ask_volume +FROM core_price +WHERE timestamp IN '$now-${Math.max(1, Math.trunc(seconds))}s..$now' +LATEST ON timestamp PARTITION BY symbol +ORDER BY symbol`; diff --git a/js/server.mjs b/js/server.mjs index e6c4a5e..5dc2e05 100644 --- a/js/server.mjs +++ b/js/server.mjs @@ -229,7 +229,7 @@ const server = createServer(async (req, res) => { res.on("close", () => { stop = true; }); // Only CHANGES are sent. A venue that has not moved should not cost a frame. - let lastQuote = "", lastBar = ""; + let lastQuote = "", lastBar = "", lastBook = ""; (async () => { while (!stop) { const startedAt = Date.now(); @@ -241,6 +241,15 @@ const server = createServer(async (req, res) => { res.write(`event: quote\ndata: ${JSON.stringify(q, bigints)}\n\n`); } + // Every instrument's top of book, pushed on the same connection. Thirty rows + // changing at once is what makes a market page feel live; one candle does not. + const book = await data.topOfBook(5); + const bookKey = JSON.stringify(book.rows, bigints); + if (bookKey !== lastBook) { + lastBook = bookKey; + res.write(`event: book\ndata: ${JSON.stringify(book, bigints)}\n\n`); + } + const bars = await data.ohlcTail({ symbol: sym, interval, seconds: 60 }); const newest = bars.rows?.at(-1); // The replacer is not optional: result rows carry BigInt, which JSON.stringify From 706562a747afa39d5c01f07dde204a55523548c1 Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 2 Oct 2026 10:34:20 +0200 Subject: [PATCH 73/79] js: add a benchmark for getLongNumber() against getLong() The scan never reads a value on its hot path, so swapping client builds only exercises the decoder and never the accessor. This reads every row's timestamp three ways - not at all, via getLong(), via getLongNumber() - with a checksum so nothing is optimised away, and takes the best of two runs per mode. It must run next to the database. Over a WAN it reports the link: on a slow link reading nothing came out slower than reading every row, which cannot be true if decoding is the bottleneck, so the script now warns when that happens rather than letting the comparison be believed. --- js/bench-longnumber.mjs | 107 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) create mode 100644 js/bench-longnumber.mjs diff --git a/js/bench-longnumber.mjs b/js/bench-longnumber.mjs new file mode 100644 index 0000000..b7cd8bb --- /dev/null +++ b/js/bench-longnumber.mjs @@ -0,0 +1,107 @@ +// Does reading timestamps without BigInt actually pay? +// +// The scan in this folder never reads a value on its hot path - it counts rows and leaves the +// columns alone - so swapping client builds only exercises the decoder, never the accessor. +// This reads the timestamp of EVERY row, three ways, so the API is the thing under test: +// +// none touch no values, only rowCount (the scan's hot path, for reference) +// long getLong(), which mints a BigInt per row (what the client has always done) +// longnumber getLongNumber(), a primitive number (feat/qwp-long-number-gorilla only) +// +// A checksum is accumulated so no read can be optimised away, and each mode runs twice with +// the best taken, because a single run on a busy cluster says very little. +// +// RUN IT NEXT TO THE DATABASE. Measured over a WAN this reports the link, not the decoder: +// on a slow link `none` came out SLOWER than `long`, which is impossible if decoding were the +// bottleneck and is the clearest possible sign the numbers are noise. Sanity-check every run +// against that: if `none` is not comfortably the fastest, the result is meaningless. +// +// node bench-longnumber.mjs [rows] [--token-file PATH] +// node bench-longnumber.mjs 172.31.42.41:9000,172.31.41.35:9000 2000000 +import { readFileSync } from "node:fs"; +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +const argv = process.argv.slice(2); +const flag = (name, fallback) => { + const i = argv.indexOf(`--${name}`); + return i >= 0 && argv[i + 1] ? argv[i + 1] : fallback; +}; +const positional = argv.filter((a, i) => + !a.startsWith("--") && !(i > 0 && argv[i - 1].startsWith("--"))); + +const addrs = positional[0] ?? "localhost:9000"; +const rows = Number(positional[1] ?? 2_000_000); +const table = flag("table", "core_price"); +const tokenFile = flag("token-file", process.env.QDB_TOKEN_FILE ?? null); +const token = tokenFile ? readFileSync(tokenFile, "utf8").trim() : null; + +const conf = token + ? `wss::addr=${addrs};tls_verify=unsafe_off;query_pool_max=4;token=${token};` + : `ws::addr=${addrs};query_pool_max=4;`; + +console.log(`${table}, last ${rows.toLocaleString("en-US")} rows, ${addrs}`); +const db = await connectQwpNodeClient(conf); + +async function run(mode) { + const lease = await db.borrowQuery(); + let seen = 0, checksum = 0, started = 0; + try { + started = performance.now(); + const query = await lease.queryViews( + `SELECT * FROM ${table} LIMIT -${rows}`, + (batch) => { + const count = batch.rowCount; + seen += count; + if (mode === "none") return; + const column = batch.column(0); // the designated timestamp + if (mode === "long") { + for (let r = 0; r < count; r++) checksum += Number(column.getLong(r) & 0xffffn); + } else { + // Throws outside JavaScript's safe integer range, so a TIMESTAMP_NANOS column + // fails loudly here rather than returning a wrong number. + for (let r = 0; r < count; r++) checksum += (column.getLongNumber(r) ?? 0) % 65536; + } + }, + // The server applies its own query.timeout, so keep `rows` small enough that one + // statement finishes inside it; this benchmark deliberately does not slice. + { timeoutMs: 0 }, + ); + await query.completion; + } finally { + await lease.close(); + } + const ms = performance.now() - started; + return { mode, rows: seen, ms, rate: (seen / ms) * 1000, checksum }; +} + +const best = new Map(); +for (const mode of ["none", "long", "longnumber", "none", "long", "longnumber"]) { + try { + const result = await run(mode); + if (!best.has(mode) || result.rate > best.get(mode).rate) best.set(mode, result); + } catch (error) { + console.log(`${mode}: ${String(error?.message ?? error)}`); + } +} +await db.close(); + +console.log(`\n${"mode".padEnd(12)}${"rows".padStart(14)}${"ms".padStart(9)}${"rows/s".padStart(14)}`); +for (const mode of ["none", "long", "longnumber"]) { + const r = best.get(mode); + if (!r) { console.log(`${mode.padEnd(12)}${"unavailable".padStart(14)}`); continue; } + console.log( + mode.padEnd(12) + + r.rows.toLocaleString("en-US").padStart(14) + + r.ms.toFixed(0).padStart(9) + + Math.round(r.rate).toLocaleString("en-US").padStart(14), + ); +} + +const none = best.get("none"), long = best.get("long"), num = best.get("longnumber"); +if (long && num) { + console.log(`\ngetLongNumber vs getLong: ${(((num.rate - long.rate) / long.rate) * 100).toFixed(1)}%`); +} +if (none && long && none.rate < long.rate) { + console.log("WARNING: reading nothing was slower than reading every row, so this run is " + + "bound by the link or the server, not by decoding. The comparison above is noise."); +} From 4ae245658e390914861df52e966743c470cf693d Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 2 Oct 2026 10:46:01 +0200 Subject: [PATCH 74/79] js: add a slice-size against reader-count benchmark Both knobs were sized against a slow WAN, where a statement had to finish inside the 60s query.timeout and 500k-row slices were the safe answer. Next to the database that reasoning inverts: a 500k slice finishes in tens of milliseconds, so per-statement setup stops being a rounding error, and 200M rows becomes 400 statements. The accessor benchmark on the same box read 2M rows in one statement at ~8M rows/s while the sliced scan managed 3.1M, which is the gap this is meant to explain. Sweeps slice size against readers and prints the matrix, using the same scanChunks() the app uses so the slicing under test is the slicing that ships, and reporting how long a slice took at the winning setting so the timeout that motivated the original sizing stays visible. --- js/bench-scan.mjs | 125 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+) create mode 100644 js/bench-scan.mjs diff --git a/js/bench-scan.mjs b/js/bench-scan.mjs new file mode 100644 index 0000000..62acbd4 --- /dev/null +++ b/js/bench-scan.mjs @@ -0,0 +1,125 @@ +// How should a scan be sliced, and how many readers should pull the slices? +// +// Both knobs were chosen against a slow WAN, where one statement had to finish inside +// QuestDB's 60s query.timeout and 500k-row slices were the safe answer. Next to the database +// that reasoning inverts: a 500k slice finishes in tens of milliseconds, so per-statement +// setup - a new request, its round trip, its first batch - stops being a rounding error and +// starts being most of the slice's life. 200M rows at 500k is 400 statements. +// +// This sweeps slice size against reader count and prints the matrix, using the same +// scanChunks() the app uses so the slicing under test is the slicing that ships. +// +// node bench-scan.mjs [rows] [--token-file PATH] [--table T] +// [--chunks 500000,5000000] [--readers 1,4,16] +// [--projection all|epoch-long|no-timestamp] +// +// node bench-scan.mjs 172.31.42.41:9000,172.31.41.35:9000 20000000 \ +// --token-file ~/qwp_token.txt.spx +// +// The hot path is deliberately identical to the app's: count rows, touch no values. +import { readFileSync } from "node:fs"; +import { connectQwpNodeClient } from "@questdb/nodejs-client"; +import { scanChunks } from "./queries.mjs"; + +const argv = process.argv.slice(2); +const flag = (name, fallback) => { + const i = argv.indexOf(`--${name}`); + return i >= 0 && argv[i + 1] ? argv[i + 1] : fallback; +}; +const positional = argv.filter((a, i) => + !a.startsWith("--") && !(i > 0 && argv[i - 1].startsWith("--"))); + +const addrs = positional[0] ?? "localhost:9000"; +const rows = Number(positional[1] ?? 20_000_000); +const table = flag("table", "core_price"); +const projection = flag("projection", "all"); +const chunkSizes = flag("chunks", "500000,2000000,5000000,20000000") + .split(",").map(Number).filter(Number.isFinite); +const readerCounts = flag("readers", "1,2,4,8,16") + .split(",").map(Number).filter(Number.isFinite); +const tokenFile = flag("token-file", process.env.QDB_TOKEN_FILE ?? null); +const token = tokenFile ? readFileSync(tokenFile, "utf8").trim() : null; + +const poolMax = Math.max(...readerCounts); +const conf = token + ? `wss::addr=${addrs};tls_verify=unsafe_off;query_pool_max=${poolMax};token=${token};` + : `ws::addr=${addrs};query_pool_max=${poolMax};`; + +console.log(`${table} (${projection}), last ${rows.toLocaleString("en-US")} rows, ${addrs}`); +const db = await connectQwpNodeClient(conf); + +/** One scan at a given slice size and reader count. Mirrors Data.scan's work queue. */ +async function scan(chunkRows, readers) { + const sqls = scanChunks(table, rows, Math.max(readers, Math.ceil(rows / chunkRows)), + projection); + const queue = sqls.slice(); + let seen = 0; + const started = performance.now(); + + const reader = async () => { + const lease = await db.borrowQuery(); + try { + while (queue.length > 0) { + const sql = queue.shift(); + const query = await lease.queryViews(sql, (batch) => { seen += batch.rowCount; }, + { timeoutMs: 0 }); + await query.completion; + } + } finally { + await lease.close(); + } + }; + + await Promise.all(Array.from({ length: Math.min(readers, sqls.length) }, () => reader())); + const ms = performance.now() - started; + return { slices: sqls.length, rows: seen, ms, rate: (seen / ms) * 1000 }; +} + +const results = []; +for (const chunkRows of chunkSizes) { + for (const readers of readerCounts) { + try { + // Best of two: one run on a shared cluster says very little. + const a = await scan(chunkRows, readers); + const b = await scan(chunkRows, readers); + const best = a.rate > b.rate ? a : b; + results.push({ chunkRows, readers, ...best }); + process.stdout.write("."); + } catch (error) { + results.push({ chunkRows, readers, error: String(error?.message ?? error) }); + process.stdout.write("x"); + } + } +} +await db.close(); + +const million = (n) => `${(n / 1e6).toFixed(2)}M`; +console.log(`\n\n${"slice".padStart(10)}${"slices".padStart(8)}` + + readerCounts.map((r) => `${r} rdr`.padStart(10)).join("")); +for (const chunkRows of chunkSizes) { + const row = results.filter((r) => r.chunkRows === chunkRows); + const slices = row.find((r) => r.slices)?.slices ?? "-"; + console.log( + million(chunkRows).padStart(10) + + String(slices).padStart(8) + + readerCounts.map((readers) => { + const cell = row.find((r) => r.readers === readers); + return (cell?.error ? "err" : million(cell?.rate ?? 0)).padStart(10); + }).join(""), + ); +} + +const ok = results.filter((r) => !r.error); +if (ok.length) { + const top = ok.reduce((a, b) => (a.rate > b.rate ? a : b)); + console.log(`\nbest: ${million(top.rate)} rows/s at ${million(top.chunkRows)} slices ` + + `× ${top.readers} readers (${top.slices} slices, ${top.ms.toFixed(0)}ms)`); + // A slice must still finish inside the server's query.timeout, which is what the original + // 500k sizing was protecting against. Report the worst case actually observed. + const perSlice = (top.ms / Math.ceil(top.slices / top.readers)) / 1000; + console.log(`at that setting each slice took about ${perSlice.toFixed(1)}s; ` + + "keep it well inside QuestDB's query.timeout (60s by default)"); +} +for (const r of results.filter((x) => x.error)) { + console.log(`${million(r.chunkRows)} x ${r.readers}: ${r.error}`); +} From 7f1f86e58e5d62358a6791799c5449310adcb92c Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 2 Oct 2026 10:48:34 +0200 Subject: [PATCH 75/79] js: warm up before the scan matrix, and discard it The sweep runs slice sizes in the outer loop, so the first cell paid the whole V8 JIT cost for the decode loops and the smallest slice size looked slow for a reason unrelated to slicing - which happens to be the result the benchmark was written expecting, so the bias pointed the wrong way. Two discarded scans run first, at both ends of the slice range. --- js/bench-scan.mjs | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/js/bench-scan.mjs b/js/bench-scan.mjs index 62acbd4..a346484 100644 --- a/js/bench-scan.mjs +++ b/js/bench-scan.mjs @@ -75,6 +75,15 @@ async function scan(chunkRows, readers) { return { slices: sqls.length, rows: seen, ms, rate: (seen / ms) * 1000 }; } +// Warm-up, discarded. V8 needs a few thousand iterations to optimise the decode loops, and +// the sweep runs slice sizes in the outer loop, so without this the FIRST cell pays the whole +// JIT cost and the smallest slice size looks slow for a reason that has nothing to do with +// slicing. The server side is usually already warm, since the newest rows were just written. +process.stdout.write("warming up"); +await scan(Math.max(...chunkSizes), Math.max(...readerCounts)); +await scan(Math.min(...chunkSizes), Math.max(...readerCounts)); +process.stdout.write(" done\n"); + const results = []; for (const chunkRows of chunkSizes) { for (const readers of readerCounts) { From 71ab62f77a21d83b634059c2962fb47bf2e04a66 Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 2 Oct 2026 11:13:37 +0200 Subject: [PATCH 76/79] js: retune scan defaults from production measurements, and record them Defaults were 4 readers and 500k slices, both chosen against a slow WAN. On a same-AZ cluster the measurements say 8 readers and 2M slices: 15.31M rows/s with epoch-long at 8 readers against 11.40M at 2. Reader scaling turns out to depend on the projection, which is not obvious and is now written down. With Gorilla the decode is a serial bottleneck on one JavaScript thread, so throughput peaks at TWO readers and 4/8/16 are all slightly worse; with the timestamp cast to long the decode is nearly free, the limit moves to the network, and 8 readers beat 2 by 34%. Tuning readers before fixing the projection measures the wrong thing. Slice size makes no difference at all next to the database - 40 statements and 1 statement landed within noise across three repeated runs - so chunk_rows is now documented as timeout headroom rather than a throughput knob. The benchmark's warm-up used the full row count, which cost half a minute on a 200M-row sweep; it now uses a fixed small sample. --- js/README.md | 37 ++++++++++++++++++++++++++++--------- js/bench-scan.mjs | 16 ++++++++++------ js/data.mjs | 2 +- js/public/index.html | 4 ++-- js/server.mjs | 9 ++++++--- 5 files changed, 47 insertions(+), 21 deletions(-) diff --git a/js/README.md b/js/README.md index dad608b..0470722 100644 --- a/js/README.md +++ b/js/README.md @@ -81,15 +81,34 @@ rather than coverage. bandwidth is the constraint, and costs 2.9x on a LAN, where it is not: the client's zstd decoder is pure JavaScript on the event-loop thread. -**Readers do not scale past a core.** Parallel readers overlap I/O, not CPU — the -client decodes in JavaScript on one thread. Throughput plateaus around 4 readers -(measured 3.10M rows/s at 4 and 3.11M at 16). - -**The timestamp column dominates the scan.** The designated timestamp is Gorilla -encoded, so it cannot be handed over as a view and is unpacked per value with BigInt -arithmetic. Every other fixed-width column is zero copy. The scan tab's `columns` -control trades this off: `all` 2.5M rows/s, `ts as epoch long` 6.3M, `no timestamp` -8.7M. +**The timestamp column dominates the scan, and how much depends on the link.** The +designated timestamp is Gorilla encoded, so it cannot be handed over as a view and is +unpacked per value with BigInt arithmetic; every other fixed-width column is zero +copy. Casting it to LONG in SQL sidesteps that at the cost of more wire bytes, which +is the right trade next to the database and the wrong one over a thin link. Measured +three ways, same table: + +| environment | `all` | `epoch-long` | `no timestamp` | +|---|---|---|---| +| same-AZ cluster, 8 readers | ~11M rows/s | **15.3M** | 18.2M | +| laptop to localhost, 8 readers | 2.5M | 6.3M | 8.7M | +| laptop over WAN, 8 readers | 143k | 128k | 173k | + +**Reader scaling depends on the projection, which is not obvious.** With `all`, the +Gorilla decode is a serial bottleneck on one JavaScript thread, so throughput peaks at +TWO readers and 4/8/16 are all slightly worse. With `epoch-long` the decode is nearly +free, the limit moves to the network and server, and 8 readers beat 2 by 34%. Tuning +readers without fixing the projection first measures the wrong thing. + +**Slice size does not matter.** On a same-AZ cluster, 40 statements and 1 statement +landed within noise of each other at every reader count, across three repeated runs. +`chunk_rows` exists only to keep one statement inside QuestDB's `query.timeout`, not +to tune throughput. + +**A no-BigInt timestamp accessor did not help.** Reading every row's timestamp via +`getLong()`, via a primitive-returning `getLongNumber()`, and not at all, all came out +identical (7.67M / 7.64M / 7.60M) next to the database: per-row cost is not the limit +there. It is worth something on localhost, where it is. **A chart cannot tick faster than the data becomes visible, and that is a property of the WRITER.** With a sender on the default `auto_flush_interval=1000`, the newest diff --git a/js/bench-scan.mjs b/js/bench-scan.mjs index a346484..395da3f 100644 --- a/js/bench-scan.mjs +++ b/js/bench-scan.mjs @@ -49,8 +49,8 @@ console.log(`${table} (${projection}), last ${rows.toLocaleString("en-US")} rows const db = await connectQwpNodeClient(conf); /** One scan at a given slice size and reader count. Mirrors Data.scan's work queue. */ -async function scan(chunkRows, readers) { - const sqls = scanChunks(table, rows, Math.max(readers, Math.ceil(rows / chunkRows)), +async function scanRows(limit, chunkRows, readers) { + const sqls = scanChunks(table, limit, Math.max(readers, Math.ceil(limit / chunkRows)), projection); const queue = sqls.slice(); let seen = 0; @@ -79,9 +79,13 @@ async function scan(chunkRows, readers) { // the sweep runs slice sizes in the outer loop, so without this the FIRST cell pays the whole // JIT cost and the smallest slice size looks slow for a reason that has nothing to do with // slicing. The server side is usually already warm, since the newest rows were just written. +// A fixed small sample, not `rows`: warming up on a 200M-row scan costs half a minute and +// buys nothing a few million rows has not already bought. +const WARMUP_ROWS = Math.min(rows, 2_000_000); process.stdout.write("warming up"); -await scan(Math.max(...chunkSizes), Math.max(...readerCounts)); -await scan(Math.min(...chunkSizes), Math.max(...readerCounts)); +for (const size of [Math.max(...chunkSizes), Math.min(...chunkSizes)]) { + await scanRows(WARMUP_ROWS, size, Math.max(...readerCounts)); +} process.stdout.write(" done\n"); const results = []; @@ -89,8 +93,8 @@ for (const chunkRows of chunkSizes) { for (const readers of readerCounts) { try { // Best of two: one run on a shared cluster says very little. - const a = await scan(chunkRows, readers); - const b = await scan(chunkRows, readers); + const a = await scanRows(rows, chunkRows, readers); + const b = await scanRows(rows, chunkRows, readers); const best = a.rate > b.rate ? a : b; results.push({ chunkRows, readers, ...best }); process.stdout.write("."); diff --git a/js/data.mjs b/js/data.mjs index 052b654..ca554fd 100644 --- a/js/data.mjs +++ b/js/data.mjs @@ -130,7 +130,7 @@ export class Data { * for the duration of the scan. The view is invalid once the callback returns, so the tail * rows are copied out before then. */ - async scan({ table, rows: limit, readers = 4, chunkRows = 500_000, chunks = 0, + async scan({ table, rows: limit, readers = 8, chunkRows = 2_000_000, chunks = 0, projection = "all", sample = 10, progressMs = 100 }, onProgress, signal) { const workers = Math.max(1, Math.trunc(readers)); diff --git a/js/public/index.html b/js/public/index.html index e7e7197..ce3cf80 100644 --- a/js/public/index.html +++ b/js/public/index.html @@ -132,8 +132,8 @@

QuestDB QWP

diff --git a/js/server.mjs b/js/server.mjs index 5dc2e05..70ddca7 100644 --- a/js/server.mjs +++ b/js/server.mjs @@ -314,10 +314,13 @@ const server = createServer(async (req, res) => { if (url.pathname === "/api/scan") { const table = url.searchParams.get("table") ?? SCAN_TABLES[0]; const rows = Number(url.searchParams.get("rows") ?? 200_000_000); - const readers = Number(url.searchParams.get("readers") ?? 4); + const readers = Number(url.searchParams.get("readers") ?? 8); // Rows per query. Bounded so one slice always finishes inside QuestDB's query.timeout; - // see Data.scan. - const chunkRows = Number(url.searchParams.get("chunk_rows") ?? 500_000); + // see Data.scan. Measured on a same-AZ cluster, slice size makes no difference at all + // (40 statements and 1 statement landed within noise of each other at every reader + // count), so this is sized for timeout headroom rather than for throughput: 2M rows is + // about a second a slice there, against a 60s cap. + const chunkRows = Number(url.searchParams.get("chunk_rows") ?? 2_000_000); const projection = url.searchParams.get("projection") ?? "all"; if (!SCAN_TABLES.includes(table) || !Number.isFinite(rows) || rows < 1 || !Number.isFinite(readers) || readers < 1 || readers > queryPoolMax From 5a823def5e8f41f617757dd58d211c17f7dea13c Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 2 Oct 2026 18:56:13 +0200 Subject: [PATCH 77/79] js: let the scan benchmark import the client by path pnpm materialises a file: dependency as a hard-linked copy under .pnpm, so rebuilding the client repo does not reach the app: the rebuild writes new files, breaking the link, and pnpm install --force will not re-resolve a dependency whose version has not changed. npm symlinks instead, so the same instructions behave differently on different machines. An A/B run that way silently compares a build against itself, which is what happened: an apparent 4.8x improvement was run-to-run variance on identical code, and the stale copy was only caught by an md5sum. --client imports the built bundle directly, and the chosen path is printed on every run so a result can no longer be misattributed to the wrong build. --- js/bench-scan.mjs | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/js/bench-scan.mjs b/js/bench-scan.mjs index 395da3f..173f18e 100644 --- a/js/bench-scan.mjs +++ b/js/bench-scan.mjs @@ -18,7 +18,7 @@ // // The hot path is deliberately identical to the app's: count rows, touch no values. import { readFileSync } from "node:fs"; -import { connectQwpNodeClient } from "@questdb/nodejs-client"; +import { resolve } from "node:path"; import { scanChunks } from "./queries.mjs"; const argv = process.argv.slice(2); @@ -38,6 +38,21 @@ const chunkSizes = flag("chunks", "500000,2000000,5000000,20000000") const readerCounts = flag("readers", "1,2,4,8,16") .split(",").map(Number).filter(Number.isFinite); const tokenFile = flag("token-file", process.env.QDB_TOKEN_FILE ?? null); + +// --client takes the package manager out of the comparison entirely. +// +// pnpm materialises a `file:` dependency as a HARD-LINKED COPY under node_modules/.pnpm, so +// rebuilding the client repo does not reach the app: the rebuild writes new files, which +// breaks the link, and `pnpm install --force` will not re-resolve a dependency whose version +// has not changed. An A/B run that way silently compares a build against itself. npm +// symlinks instead, so the same instructions behave differently on different machines. +// Importing the built bundle by path sidesteps all of it. +// +// --client ~/nodejs-questdb-client/packages/nodejs-client/dist/es/index.mjs +const clientPath = flag("client", process.env.QDB_CLIENT ?? null); +const { connectQwpNodeClient } = clientPath + ? await import(resolve(clientPath.replace(/^~/, process.env.HOME ?? "~"))) + : await import("@questdb/nodejs-client"); const token = tokenFile ? readFileSync(tokenFile, "utf8").trim() : null; const poolMax = Math.max(...readerCounts); @@ -46,6 +61,7 @@ const conf = token : `ws::addr=${addrs};query_pool_max=${poolMax};`; console.log(`${table} (${projection}), last ${rows.toLocaleString("en-US")} rows, ${addrs}`); +console.log(`client: ${clientPath ?? "@questdb/nodejs-client (resolved from node_modules)"}`); const db = await connectQwpNodeClient(conf); /** One scan at a given slice size and reader count. Mirrors Data.scan's work queue. */ From 3f04e293aa493a92fb395dc14c6ac129bcc7e1a4 Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 2 Oct 2026 18:57:30 +0200 Subject: [PATCH 78/79] Revert "js: let the scan benchmark import the client by path" This reverts commit 5a823def5e8f41f617757dd58d211c17f7dea13c. --- js/bench-scan.mjs | 18 +----------------- 1 file changed, 1 insertion(+), 17 deletions(-) diff --git a/js/bench-scan.mjs b/js/bench-scan.mjs index 173f18e..395da3f 100644 --- a/js/bench-scan.mjs +++ b/js/bench-scan.mjs @@ -18,7 +18,7 @@ // // The hot path is deliberately identical to the app's: count rows, touch no values. import { readFileSync } from "node:fs"; -import { resolve } from "node:path"; +import { connectQwpNodeClient } from "@questdb/nodejs-client"; import { scanChunks } from "./queries.mjs"; const argv = process.argv.slice(2); @@ -38,21 +38,6 @@ const chunkSizes = flag("chunks", "500000,2000000,5000000,20000000") const readerCounts = flag("readers", "1,2,4,8,16") .split(",").map(Number).filter(Number.isFinite); const tokenFile = flag("token-file", process.env.QDB_TOKEN_FILE ?? null); - -// --client takes the package manager out of the comparison entirely. -// -// pnpm materialises a `file:` dependency as a HARD-LINKED COPY under node_modules/.pnpm, so -// rebuilding the client repo does not reach the app: the rebuild writes new files, which -// breaks the link, and `pnpm install --force` will not re-resolve a dependency whose version -// has not changed. An A/B run that way silently compares a build against itself. npm -// symlinks instead, so the same instructions behave differently on different machines. -// Importing the built bundle by path sidesteps all of it. -// -// --client ~/nodejs-questdb-client/packages/nodejs-client/dist/es/index.mjs -const clientPath = flag("client", process.env.QDB_CLIENT ?? null); -const { connectQwpNodeClient } = clientPath - ? await import(resolve(clientPath.replace(/^~/, process.env.HOME ?? "~"))) - : await import("@questdb/nodejs-client"); const token = tokenFile ? readFileSync(tokenFile, "utf8").trim() : null; const poolMax = Math.max(...readerCounts); @@ -61,7 +46,6 @@ const conf = token : `ws::addr=${addrs};query_pool_max=${poolMax};`; console.log(`${table} (${projection}), last ${rows.toLocaleString("en-US")} rows, ${addrs}`); -console.log(`client: ${clientPath ?? "@questdb/nodejs-client (resolved from node_modules)"}`); const db = await connectQwpNodeClient(conf); /** One scan at a given slice size and reader count. Mirrors Data.scan's work queue. */ From 92987abcbb5e0aeb367e6b62cb088732a7bb642e Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 2 Oct 2026 19:01:12 +0200 Subject: [PATCH 79/79] js: document building the client without assuming anyone's layout The install steps hardcoded a relative hop between two repos, which only works if they sit where mine do; following them on a flat $HOME linked a path that did not exist and broke the install. The client's location is now captured as $CLIENT at clone time and never assumed. Adds the part that actually cost time: npm installs a file: dependency as a symlink, so a rebuild is picked up, while pnpm makes a hard-linked copy that a rebuild silently orphans - and pnpm install --force will not refresh it, because the version has not changed. Comparing two client builds that way compares one build against itself. The md5sum check that catches it is written down. --- js/README.md | 60 +++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 50 insertions(+), 10 deletions(-) diff --git a/js/README.md b/js/README.md index 0470722..9e9f15d 100644 --- a/js/README.md +++ b/js/README.md @@ -22,27 +22,67 @@ changes. Two dependencies: the QWP client, and TradingView Lightweight Charts for the OHLC tab. No framework, no bundler, no build step — the page is plain ES modules. -The client is not on npm yet, so build it from source: +### 1. Build the client from source + +`@questdb/nodejs-client` is not published yet, so it has to be built. Clone it +anywhere you like; `$CLIENT` below stands for wherever that is. ```sh -git clone && cd nodejs-questdb-client -pnpm install # needs Node >= 20.18.1 +git clone https://github.com/questdb/nodejs-questdb-client.git +cd nodejs-questdb-client +CLIENT=$(pwd) + +corepack enable && corepack prepare pnpm@10.12.4 --activate # or: npm i -g pnpm +pnpm install # needs Node >= 20.18.1 pnpm --filter @questdb/nodejs-client build ``` -`pnpm install` at the **root** is required first: `bunchee`, `rollup` and -`typescript` are devDependencies of the workspace, so building from inside -`packages/nodejs-client` will not resolve them. +Three things that bite: + +- **`pnpm install` at the repo ROOT is required first.** `bunchee`, `rollup` and + `typescript` are devDependencies of the workspace, so building from inside + `packages/nodejs-client` cannot resolve them. +- **Node must be 20.18.1 or newer**, which the package enforces. Distro Node is + often 18 — check `node --version` before blaming the build. +- The build writes `packages/nodejs-client/dist/`, which is what this app consumes. -Then point this app at wherever you cloned it and install: +### 2. Point this app at it ```sh -cd js -npm pkg set dependencies.@questdb/nodejs-client=file:/path/to/nodejs-questdb-client/packages/nodejs-client +cd /path/to/questdb_java_ha_sender/js +npm pkg set dependencies.@questdb/nodejs-client="file:$CLIENT/packages/nodejs-client" npm install ``` -That edits one line of `package.json`; it is a local diff, not something to commit. +That rewrites one line of `package.json` to a path that is local to your machine. +**Do not commit it** — the committed value is whatever the last person's layout +happened to be, and it is expected to be wrong for you. + +### 3. Rebuilding the client afterwards + +This matters whenever you are comparing client branches, because getting it wrong +produces an A/B that silently compares a build against itself. + +- **With npm**, a `file:` dependency is installed as a SYMLINK. Rebuild the client, + restart the server, done. +- **With pnpm**, it is a hard-linked COPY under `node_modules/.pnpm/`. A rebuild + writes new files, which breaks the link, so the app keeps running the old build — + and `pnpm install --force` will not fix it, because the dependency's version has + not changed. Either re-add it with pnpm's `link:` protocol, which does symlink: + + ```sh + pnpm add "@questdb/nodejs-client@link:$CLIENT/packages/nodejs-client" + ``` + + or delete `node_modules` and install again. Note `link:` is a pnpm/yarn protocol; + npm ignores it silently and installs nothing, so do not commit that either. + +**Verify, every time, before trusting a measurement.** The two hashes must match: + +```sh +md5sum "$CLIENT/packages/nodejs-client/dist/es/index.mjs" +md5sum "$(readlink -f node_modules/@questdb/nodejs-client)/dist/es/index.mjs" +``` ## Run