From e88cea833cdaf6e07da56f1799557a617eaa35de Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Mon, 15 Jun 2026 13:06:24 +0100 Subject: [PATCH 01/25] docs: add EXPIRE ROWS materialized-view documentation Deep-dive concept page (concepts/deep-dive/expire-rows.md) for passthrough materialized views with EXPIRE ROWS: all modes, worked examples, read-filter/cleanup mechanics, NULL/ties/monotonicity semantics, and limitations. Adds an ALTER MATERIALIZED VIEW SET EXPIRE reference page, an EXPIRE ROWS section in CREATE MATERIALIZED VIEW, and sidebar entries. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../concepts/deep-dive/expire-rows.md | 416 ++++++++++++++++++ .../query/sql/alter-mat-view-set-expire.md | 112 +++++ documentation/query/sql/create-mat-view.md | 39 +- documentation/sidebars.js | 2 + 4 files changed, 568 insertions(+), 1 deletion(-) create mode 100644 documentation/concepts/deep-dive/expire-rows.md create mode 100644 documentation/query/sql/alter-mat-view-set-expire.md diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md new file mode 100644 index 0000000000..e2e05478a6 --- /dev/null +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -0,0 +1,416 @@ +--- +title: Expiring rows (EXPIRE ROWS) +sidebar_label: Expiring rows +description: + EXPIRE ROWS is a row-level retention policy for passthrough materialized + views. Keep the latest row per key, the top-N per group, or rows matching a + predicate — recomputed continuously, with expired rows hidden immediately and + reclaimed in the background. +--- + +`EXPIRE ROWS` is a row-level retention policy for +[materialized views](/docs/concepts/materialized-views/). Where +[TTL](/docs/concepts/ttl/) drops whole partitions once they age out, `EXPIRE +ROWS` decides retention **row by row** — keep the latest row per key, the top-N +per group, rows matching a predicate, and so on — and recomputes the result +continuously as the view refreshes. + +Expired rows disappear from query results **immediately**; their on-disk storage +is reclaimed afterwards by a background job. + +:::note + +`EXPIRE ROWS` is **materialized-view-only**, and the view must be a +**passthrough** (non-aggregating) view: `SELECT * FROM base` with no +`SAMPLE BY` / `GROUP BY`. `CREATE TABLE ... EXPIRE ROWS` and aggregating views +are rejected. For base-table retention use [TTL](/docs/concepts/ttl/) or, on +Enterprise, [storage policies](/docs/concepts/storage-policy/). + +::: + +## When to use EXPIRE ROWS + +Reach for `EXPIRE ROWS` on a passthrough materialized view when you want a +continuously-maintained, pruned copy of a base table: + +- **Current-state-per-key tables** — keep only the latest row per device, + symbol, or session (`KEEP LATEST`). +- **Per-group extremes or leaderboards** — keep the highest/lowest value per + group, or the top-N (`KEEP HIGHEST/LOWEST`, `KEEP N`). +- **Rolling row-level windows** — keep rows newer than a cutoff, or matching any + predicate, at finer granularity than TTL's whole-partition drops + (`WHEN predicate`). + +Use [TTL](/docs/concepts/ttl/) instead when partition-granularity, age-based +retention on a base table is enough — it is cheaper and has no passthrough-view +requirement. + +## Requirements + +`EXPIRE ROWS` requires a **passthrough materialized view**: + +- The view query is `SELECT * FROM base`. A column subset and a `WHERE` filter + are allowed; aggregation, `SAMPLE BY`, `GROUP BY`, `LATEST ON`, `DISTINCT`, + `UNION`, and `JOIN` are not — they make the view non-passthrough and are + rejected with + `EXPIRE ROWS is only supported on passthrough (non-aggregating) materialized views`. +- The view inherits the base table's + [designated timestamp](/docs/concepts/designated-timestamp/) and partitioning. + +A passthrough view mirrors its base table 1:1 and refreshes incrementally, so it +is effectively a continuously-maintained replica. `EXPIRE ROWS` prunes that +replica down to the rows you want to keep — without touching the base table. + +## The modes + +Every mode keeps a defined set of rows and expires the rest. A row is expired +only when the rule selects it for removal. + +| Mode | What it keeps | Syntax | +| --------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | +| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | +| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | +| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | +| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | +| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | + +`KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a +window predicate, so the window `WHEN` is the general escape hatch. + +The clause is attached to a passthrough `CREATE MATERIALIZED VIEW` (after the +query, and after `PARTITION BY` if present), or set later with +[`ALTER MATERIALIZED VIEW ... SET EXPIRE ROWS`](/docs/query/sql/alter-mat-view-set-expire/): + +``` +EXPIRE ROWS + { WHEN predicate + | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] + | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } + [ CLEANUP EVERY duration ] +``` + +| Element | Meaning | +| ------------------ | ---------------------------------------------------------------------------------------- | +| `predicate` | Any boolean expression over the view's columns. A row expires when it evaluates `TRUE`. | +| `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp. | +| `ON timestampCol` | Optional; if given it must name the view's designated timestamp. | +| `HIGHEST\|LOWEST` | Keep rows at the max / min of `col` per group (`N` omitted), or the top `N` per group. | +| `CLEANUP EVERY` | How often the background reclamation job runs for this view. Defaults to `1h` if omitted. | + +## Worked examples + +The following walks through every mode on a small fixed dataset so you can see +exactly which rows each policy keeps. + +### Setup + +```questdb-sql title="Base table and sample data" +CREATE TABLE trades ( + symbol SYMBOL, + price DOUBLE, + amount DOUBLE, + ts TIMESTAMP +) TIMESTAMP(ts) PARTITION BY DAY WAL; + +INSERT INTO trades VALUES + ('BTC', 100.0, 1.0, '2024-01-01T10:00:00.000000Z'), + ('BTC', 105.0, 2.0, '2024-01-01T11:00:00.000000Z'), + ('BTC', 102.0, 1.5, '2024-01-02T09:00:00.000000Z'), + ('ETH', 50.0, 3.0, '2024-01-01T10:30:00.000000Z'), + ('ETH', 55.0, 1.0, '2024-01-02T08:00:00.000000Z'); +``` + +:::note + +A materialized view starts an **asynchronous initial refresh** on creation, so +it may briefly return no rows. Check progress with +`SELECT view_name, view_status, base_table_txn, refresh_base_table_txn FROM materialized_views();` +— the view is up to date when `refresh_base_table_txn = base_table_txn`. With +this small dataset that is effectively instant. + +::: + +### Per-row predicate: `WHEN` + +A per-row predicate expires a row when it evaluates `TRUE`. Here, expire small +trades (`amount < 1.5`): + +```questdb-sql title="Expire rows where amount < 1.5" +CREATE MATERIALIZED VIEW trades_sized AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN amount < 1.5; + +SELECT * FROM trades_sized; +``` + +| symbol | price | amount | ts | +| ------ | ----- | ------ | --------------------------- | +| BTC | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | +| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | +| ETH | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | + +The two `amount = 1.0` rows are expired. `amount = 1.5` is kept (`1.5 < 1.5` is +`FALSE`), and any `NULL` amount would be kept too (the comparison is `UNKNOWN`, +not `TRUE` — see [NULLs](#nulls)). + +A predicate on the designated timestamp gives a **rolling retention window**, +re-evaluated on every read so the visible set rolls forward with the clock: + +```questdb-sql title="Keep the last 1 day" +CREATE MATERIALIZED VIEW trades_recent AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN ts < dateadd('d', -1, now()); +``` + +(With the 2024 sample timestamps above, every row is already older than a day +and would be hidden; use recent data to see rows retained.) + +### Keep latest per key: `KEEP LATEST` + +Keep only the most recent row per key — turning the passthrough view into a +live, current-state-per-symbol table: + +```questdb-sql title="Keep the latest row per symbol" +CREATE MATERIALIZED VIEW trades_latest AS ( + SELECT * FROM trades +) EXPIRE ROWS KEEP LATEST PARTITION BY symbol; + +SELECT * FROM trades_latest; +``` + +| symbol | price | amount | ts | +| ------ | ----- | ------ | --------------------------- | +| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | +| ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | + +One row per symbol — the latest by the designated timestamp `ts`. As new trades +arrive, the kept row advances automatically. `PARTITION BY` may list multiple key +columns. You may write `KEEP LATEST ON ts PARTITION BY symbol`, but the `ON` +column must be the view's designated timestamp. + +### Keep extremes per group: `KEEP HIGHEST` / `KEEP LOWEST` + +Keep the rows tied at the group maximum (or minimum) of a column: + +```questdb-sql title="Keep the highest-priced trade per symbol" +CREATE MATERIALIZED VIEW trades_peak AS ( + SELECT * FROM trades +) EXPIRE ROWS KEEP HIGHEST price PARTITION BY symbol; + +SELECT * FROM trades_peak; +``` + +| symbol | price | amount | ts | +| ------ | ----- | ------ | --------------------------- | +| BTC | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | +| ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | + +`KEEP LOWEST price PARTITION BY symbol` keeps the cheapest instead (BTC `100.0`, +ETH `50.0`). All rows **tied** at the extreme are kept, and `NULL`-valued rows +are kept (a `NULL` is never less than the max). + +### Keep top-N per group: `KEEP N HIGHEST` / `KEEP N LOWEST` + +Keep a per-group leaderboard — the `N` highest (or lowest) rows: + +```questdb-sql title="Keep the 2 highest-priced trades per symbol" +CREATE MATERIALIZED VIEW trades_top2 AS ( + SELECT * FROM trades +) EXPIRE ROWS KEEP 2 HIGHEST price PARTITION BY symbol; + +SELECT * FROM trades_top2 ORDER BY symbol, price DESC; +``` + +| symbol | price | amount | ts | +| ------ | ----- | ------ | --------------------------- | +| BTC | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | +| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | +| ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | +| ETH | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | + +BTC keeps its two highest (`105`, `102`) and drops `100`; ETH has only two rows, +so both survive. Ties are broken by the designated timestamp, so the N-th +boundary is deterministic. + +### Window predicate: the escape hatch + +`KEEP HIGHEST/LOWEST` and `KEEP N` are shorthand for window predicates. When you +need a rule they do not cover, write the window predicate directly in `WHEN`. +For example, this is exactly what `KEEP HIGHEST price PARTITION BY symbol` +expands to: + +```questdb-sql title="Equivalent to KEEP HIGHEST, written as a window predicate" +CREATE MATERIALIZED VIEW trades_peak_win AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN price < max(price) OVER (PARTITION BY symbol); +``` + +A row expires when its price is below its symbol's maximum, so only the peak per +symbol survives — the same result as `trades_peak` above. From here you can +express richer rules, for example keeping rows within 5% of the peak +(`WHEN price < 0.95 * max(price) OVER (PARTITION BY symbol)`) or a ranked window +(`WHEN row_number() OVER (PARTITION BY symbol ORDER BY ts DESC) > 100`). + +## How it works + +`EXPIRE ROWS` has two cooperating parts: an authoritative read-time filter and a +best-effort background cleanup. + +### Read-time filter (authoritative) + +Every query against a policied view is transparently rewritten so that only the +kept rows are visible — **immediately, and regardless of whether cleanup has +run**. This is what makes results correct at all times: + +- **Per-row `WHEN`** keeps rows where the predicate is not `TRUE`. QuestDB + filtering is three-valued, so `FALSE` **and** `NULL` are kept (see + [NULLs](#nulls)). +- **`KEEP LATEST`** returns the latest row per key using the designated + timestamp. +- **`KEEP HIGHEST/LOWEST/N` and window `WHEN`** compute the keep-set with a + window function over the whole view. + +Because the filter is applied at query time, a freshly-refreshed row that should +be expired is hidden the moment it lands, and a row that should reappear (under a +time-based predicate) reappears on the next read. + +### Physical cleanup (best-effort) + +A background job reclaims disk for non-active partitions: a fully-expired +partition is removed, and a partially-expired one is compacted down to its +survivors. It runs at the `CLEANUP EVERY` cadence (default `1h`) and is +**best-effort** — the read filter is authoritative, so deferred or skipped +reclamation only affects disk usage, never query results. + +On QuestDB Enterprise, cleanup runs on the **primary only**, but the reclamation +still replicates: the compaction commits are ordinary WAL transactions, so +replicas reclaim the identical rows by applying them. A read-only replica neither +runs the job nor needs to. Disable the job with `cairo.row.expiry.enabled=false` +in `server.conf` (reads stay filtered; only reclamation stops). + +To observe reclamation, compare the physical row count per partition before and +after a sweep: + +```questdb-sql title="Physical rows still on disk per partition" +SELECT name, numRows FROM table_partitions('trades_latest'); +``` + +Reclamation **defers while a view is being refreshed continuously** and resumes +on a quiet sweep. + +## Semantics + +### NULLs + +The keep-set is computed with three-valued logic, so a `NULL` value is never +*less than* a group maximum (the comparison is `UNKNOWN`, not `TRUE`). +Therefore: + +- **`KEEP HIGHEST/LOWEST` and value-based `WHEN`** predicates **keep** rows whose + value is `NULL`. +- **`KEEP LATEST`** uses the designated timestamp, which is never `NULL`. +- **`KEEP N` is the exception.** It ranks rows with `row_number()`, and QuestDB + has no `NULLS LAST`, so where a `NULL` lands is **type-dependent**: under + `DESC` a floating-point `NULL` (NaN) sorts first (kept while there is room in + `N`), while an integer/timestamp `NULL` sorts last (expired first). Use + `KEEP HIGHEST/LOWEST` (no `N`) when every `NULL` must be kept regardless of + type. + +### Ties and determinism + +`KEEP HIGHEST/LOWEST` keeps **all** rows tied at the max/min — deterministic by +construction. `KEEP N` makes the order total by appending the designated +timestamp as a tiebreak, so the N-th boundary is deterministic (pair the base +table with [`DEDUP UPSERT KEYS`](/docs/concepts/deduplication/) if `(col, ts)` +is not already unique). + +### Monotonicity and cleanup safety + +Physical deletion is only safe when expiry is **monotonic**: a row that is +expired now must stay expired forever. All the relative modes (`KEEP LATEST`, +`KEEP HIGHEST/LOWEST`, `KEEP N`) are monotonic by construction, as is a +designated-timestamp predicate such as `WHEN ts < now()` (a row only gets +older). A scalar `WHEN predicate` is arbitrary SQL, so **monotonicity is the +author's responsibility**. + +:::warning + +A non-monotonic predicate such as `WHEN ts > now()` expires *future* rows that +**un-expire** as `now()` advances. The read filter recomputes `now()` on every +read and stays correct, but the cleanup job assumes monotonicity and may +physically delete a row that a later read would otherwise show (recoverable only +by a full refresh). Write `WHEN` predicates that expire things in the **past** +(`ts < now()`) or against fixed thresholds — never rows that the passage of time +will later keep. + +::: + +## Inspecting a policy + +`SHOW CREATE MATERIALIZED VIEW` renders the clause as written: + +```questdb-sql +SHOW CREATE MATERIALIZED VIEW trades_latest; +-- ... EXPIRE ROWS KEEP LATEST PARTITION BY symbol +``` + +The [`materialized_views()`](/docs/query/functions/meta/) function exposes the +policy in the `expire_predicate` and `expire_cleanup_every` columns (both +`NULL` when no policy is set): + +```questdb-sql title="List EXPIRE ROWS policies" +SELECT view_name, expire_predicate, expire_cleanup_every +FROM materialized_views(); +``` + +| view_name | expire_predicate | expire_cleanup_every | +| ------------- | ------------------------------- | -------------------- | +| trades_sized | amount < 1.5 | 1h | +| trades_latest | KEEP LATEST PARTITION BY symbol | 1h | +| trades_top2 | KEEP 2 HIGHEST price ... | 1h | + +## Changing or removing a policy + +Set, change, or drop a policy on an existing passthrough view — see +[`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/): + +```questdb-sql +-- set or replace the policy +ALTER MATERIALIZED VIEW trades_latest SET EXPIRE ROWS KEEP LATEST PARTITION BY symbol; + +-- remove it (keeps all rows again) +ALTER MATERIALIZED VIEW trades_latest DROP EXPIRE; +``` + +`SET EXPIRE ROWS` validates the new policy against the view's columns before +applying it, so an invalid predicate or an unknown column is rejected up front +rather than breaking subsequent reads. + +## Limitations and operational notes + +- **Reads recompute the keep-set.** A relative/window policy computes its + keep-set over the whole physical view on every read. `KEEP LATEST` on an + [indexed](/docs/concepts/deep-dive/indexes/) symbol key is cheap; the window + modes (and non-indexed keep-latest) scan the view. A tighter `CLEANUP EVERY` + keeps the physical residue — and therefore the read cost — small. +- **Cleanup defers under continuous refresh.** Reclamation only proceeds when the + view is quiescent and fully applied, so a view being refreshed continuously + defers reclamation to a quiet sweep. The read filter stays authoritative + meanwhile. +- **`KEEP LATEST [ON ts]`.** The optional `ON ts` is accepted for familiarity but + the view's designated timestamp is always used; naming a different column is + rejected. +- **Non-monotonic `WHEN` predicates are unsupported for cleanup** — see + [monotonicity](#monotonicity-and-cleanup-safety) above. + +## Related documentation + +- [Materialized views](/docs/concepts/materialized-views/) — the view type + `EXPIRE ROWS` runs on +- [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/) — full create + syntax, including the `EXPIRE ROWS` clause +- [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) + — set, change, or drop a policy +- [Time To Live (TTL)](/docs/concepts/ttl/) — partition-level retention by age +- [Storage policy](/docs/concepts/storage-policy/) — graduated partition + lifecycle (Enterprise) diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md new file mode 100644 index 0000000000..8c3d4216b8 --- /dev/null +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -0,0 +1,112 @@ +--- +title: ALTER MATERIALIZED VIEW SET EXPIRE +sidebar_label: SET EXPIRE +description: + ALTER MATERIALIZED VIEW SET EXPIRE ROWS / DROP EXPIRE SQL keyword reference + documentation. +--- + +Sets, replaces, or removes an [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) +row-retention policy on a **passthrough** materialized view. Expired rows are +hidden from queries immediately and reclaimed on disk in the background. + +## Syntax + +``` +ALTER MATERIALIZED VIEW viewName SET EXPIRE ROWS + { WHEN predicate + | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] + | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } + [ CLEANUP EVERY duration ] + +ALTER MATERIALIZED VIEW viewName DROP EXPIRE +``` + +## Parameters + +| Parameter | Description | +| ---------------- | --------------------------------------------------------------------------------- | +| `viewName` | Name of the passthrough materialized view to modify | +| `WHEN predicate` | Per-row (or window) predicate; a row expires when it evaluates `TRUE` | +| `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp | +| `KEEP [N] HIGHEST\|LOWEST col` | Keep the rows at the max/min of `col` per group, or the top `N` | +| `CLEANUP EVERY` | Background reclamation cadence (e.g. `30m`, `1h`). Defaults to `1h` if omitted | + +For the full description of each mode and its semantics, see the +[Expiring rows](/docs/concepts/deep-dive/expire-rows/) concept page. + +## When to use + +- Add a retention policy to a passthrough view created without one +- Switch a view between modes (e.g. from a value predicate to `KEEP LATEST`) +- Tune the `CLEANUP EVERY` cadence +- Remove a policy with `DROP EXPIRE` so the view keeps all rows again + +## How it works + +`SET EXPIRE ROWS` validates the new policy against the view's columns first +(compiling the predicate / checking the key columns), so an invalid predicate or +an unknown column is rejected immediately rather than breaking later reads. Once +set, queries against the view are filtered to the kept rows immediately; physical +reclamation follows in the background. See +[How it works](/docs/concepts/deep-dive/expire-rows/#how-it-works). + +## Examples + +```questdb-sql title="Per-row predicate, with a tighter cleanup cadence" +ALTER MATERIALIZED VIEW trades_mirror + SET EXPIRE ROWS WHEN amount < 1.5 CLEANUP EVERY 30m; +``` + +```questdb-sql title="Keep the latest row per symbol" +ALTER MATERIALIZED VIEW trades_mirror + SET EXPIRE ROWS KEEP LATEST PARTITION BY symbol; +``` + +```questdb-sql title="Keep the highest-priced row per symbol" +ALTER MATERIALIZED VIEW trades_mirror + SET EXPIRE ROWS KEEP HIGHEST price PARTITION BY symbol; +``` + +```questdb-sql title="Keep the 2 highest-priced rows per symbol" +ALTER MATERIALIZED VIEW trades_mirror + SET EXPIRE ROWS KEEP 2 HIGHEST price PARTITION BY symbol; +``` + +```questdb-sql title="Remove the policy" +ALTER MATERIALIZED VIEW trades_mirror DROP EXPIRE; +``` + +## Behavior + +| Aspect | Description | +| ----------------- | ---------------------------------------------------------------------------- | +| Passthrough only | The view must be `SELECT * FROM base`; aggregating views are rejected | +| Validation | The policy is checked against the view's columns before it is applied | +| Immediate effect | Reads are filtered to the kept rows as soon as the policy is set | +| Replication | The policy and the reclamation it drives replicate as normal WAL traffic | + +## Permissions (Enterprise) + +Changing the policy requires the `ALTER MATERIALIZED VIEW` permission: + +```questdb-sql title="Grant alter permission" +GRANT ALTER MATERIALIZED VIEW ON trades_mirror TO user1; +``` + +## Errors + +| Error | Cause | +| ----- | ----- | +| `materialized view does not exist` | View with the specified name doesn't exist | +| `EXPIRE ROWS is only supported on passthrough (non-aggregating) materialized views` | View aggregates (e.g. `SAMPLE BY`) | +| `EXPIRE ROWS KEEP LATEST ON must name the designated timestamp ...` | `ON` names a column other than the designated timestamp | +| `invalid EXPIRE ROWS KEEP LATEST column: ...` | A `PARTITION BY` key column does not exist | +| `permission denied` | Missing `ALTER MATERIALIZED VIEW` permission (Enterprise) | + +## See also + +- [Expiring rows (EXPIRE ROWS) concept](/docs/concepts/deep-dive/expire-rows/) +- [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/) +- [Materialized views concept](/docs/concepts/materialized-views/) +- [ALTER MATERIALIZED VIEW SET TTL](/docs/query/sql/alter-mat-view-set-ttl/) diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 0dd9739535..19593d34cf 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -22,13 +22,16 @@ AS [ ( ] query [ ) ] [ TIMESTAMP ( columnRef ) ] [ PARTITION BY ( YEAR | MONTH | WEEK | DAY | HOUR ) [ TTL n timeUnit ] ] +[ EXPIRE ROWS expirePolicy [ CLEANUP EVERY duration ] ] [ OWNED BY ownerName ] ``` Where: - `interval`: Duration like `1m`, `10m`, `1h`, `1d` - `timeUnit`: `HOURS | DAYS | WEEKS | MONTHS | YEARS` -- `query`: Must contain `SAMPLE BY` or time-based `GROUP BY` +- `query`: Must contain `SAMPLE BY` or time-based `GROUP BY` (except for the + passthrough views used by `EXPIRE ROWS` — see below) +- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON ts] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]` — passthrough views only ## Parameters @@ -43,6 +46,7 @@ Where: | `TIMESTAMP` | Designate timestamp column for the view | | `PARTITION BY` | Partitioning unit for view storage | | `TTL` | Retention period for view data | +| `EXPIRE ROWS` | Row-level retention for passthrough views (see below) | | `OWNED BY` | Assign ownership (Enterprise) | ## Rules and defaults @@ -278,6 +282,39 @@ Time units: `HOURS`, `DAYS`, `WEEKS`, `MONTHS`, `YEARS` The view's TTL is independent of the base table's TTL. See [TTL documentation](/docs/concepts/ttl/) for details. +## EXPIRE ROWS + +Attach a row-level retention policy with `EXPIRE ROWS`. Unlike `TTL` (which drops +whole partitions by age), `EXPIRE ROWS` keeps a defined set of rows — the latest +per key, the top-N per group, or rows matching a predicate — recomputed +continuously as the view refreshes. + +`EXPIRE ROWS` applies **only to passthrough (non-aggregating) views** — the query +is `SELECT * FROM base` with no `SAMPLE BY` / `GROUP BY`: + +```questdb-sql title="Passthrough view that keeps the latest row per symbol" +CREATE MATERIALIZED VIEW trades_latest AS ( + SELECT * FROM trades +) EXPIRE ROWS KEEP LATEST PARTITION BY symbol; +``` + +The clause goes after the query (and after `PARTITION BY` if present): + +``` +EXPIRE ROWS + { WHEN predicate + | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] + | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } + [ CLEANUP EVERY duration ] +``` + +Expired rows are hidden from queries immediately and reclaimed on disk in the +background (`CLEANUP EVERY`, default `1h`). Change or remove a policy with +[`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/). + +See the [Expiring rows](/docs/concepts/deep-dive/expire-rows/) concept page for +all modes, worked examples, and semantics (NULLs, ties, monotonicity). + ## Complete example Putting it all together: diff --git a/documentation/sidebars.js b/documentation/sidebars.js index 9955129082..d9fa245edb 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -311,6 +311,7 @@ module.exports = { "query/sql/alter-mat-view-set-refresh", "query/sql/alter-mat-view-set-refresh-limit", "query/sql/alter-mat-view-set-ttl", + "query/sql/alter-mat-view-set-expire", ], }, { @@ -546,6 +547,7 @@ module.exports = { items: [ "concepts/deep-dive/indexes", "concepts/deep-dive/posting-index", + "concepts/deep-dive/expire-rows", "concepts/deep-dive/interval-scan", "concepts/deep-dive/jit-compiler", "concepts/deep-dive/query-tracing", From 3fd5d8d7c4066784ce8a1b606ebe84efc36211d4 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Tue, 21 Jul 2026 15:35:59 +0200 Subject: [PATCH 02/25] docs: align EXPIRE ROWS pages with the implementation Corrections and additions ported from the core repo's in-repo design doc (docs/row-expiry.md, now removed there): - Aggregating views are ACCEPTED with a logged advisory, not rejected (the previously-cited error message does not exist); reworded the concept note, Requirements, the ALTER behavior and errors tables, and the CREATE section accordingly. - materialized_views() exposes the policy as expire_clause, not expire_predicate. - Monotonicity: list the exact clock shapes the cleanup job proves monotonic (bare clock, clock minus non-negative constant, fixed-unit look-back dateadd) and that calendar units, look-forward offsets, compound arithmetic, and window predicates skip cleanup; the job skips rather than risks deleting rows. - No-policied-chains rule: CREATE rejects a defining query reading a policied view (base or join); SET EXPIRE is rejected with dependent views; corresponding errors added to the ALTER page. - Reserved __qdb_re_keep column name; no line comments inside the clause; CLEANUP EVERY strict grammar (s/m/h/d/w). - Kill switch is read at startup (restart required); failing sweeps back off from 1s up to a 10-minute cap. - Parquet side effect: compacting a partially-expired Parquet partition rewrites it as native storage until re-conversion. Co-Authored-By: Claude Fable 5 --- .../concepts/deep-dive/expire-rows.md | 85 +++++++++++++------ .../query/sql/alter-mat-view-set-expire.md | 22 +++-- documentation/query/sql/create-mat-view.md | 9 +- 3 files changed, 80 insertions(+), 36 deletions(-) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md index e2e05478a6..70cd194159 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -20,11 +20,15 @@ is reclaimed afterwards by a background job. :::note -`EXPIRE ROWS` is **materialized-view-only**, and the view must be a -**passthrough** (non-aggregating) view: `SELECT * FROM base` with no -`SAMPLE BY` / `GROUP BY`. `CREATE TABLE ... EXPIRE ROWS` and aggregating views -are rejected. For base-table retention use [TTL](/docs/concepts/ttl/) or, on -Enterprise, [storage policies](/docs/concepts/storage-policy/). +`EXPIRE ROWS` is **materialized-view-only**: `CREATE TABLE ... EXPIRE ROWS` is +rejected. It is designed for a **passthrough** (non-aggregating) view — `SELECT +* FROM base` with no `SAMPLE BY` / `GROUP BY` — where the view mirrors base +rows 1:1 and reclamation is permanent. An aggregating view is **accepted with a +logged advisory**: reads stay filtered, but a later refresh can regenerate +reclaimed rows from base rows that still exist (see +[Requirements](#requirements)). For base-table retention use +[TTL](/docs/concepts/ttl/) or, on Enterprise, +[storage policies](/docs/concepts/storage-policy/). ::: @@ -47,13 +51,11 @@ requirement. ## Requirements -`EXPIRE ROWS` requires a **passthrough materialized view**: +`EXPIRE ROWS` is designed for a **passthrough materialized view**: - The view query is `SELECT * FROM base`. A column subset and a `WHERE` filter - are allowed; aggregation, `SAMPLE BY`, `GROUP BY`, `LATEST ON`, `DISTINCT`, - `UNION`, and `JOIN` are not — they make the view non-passthrough and are - rejected with - `EXPIRE ROWS is only supported on passthrough (non-aggregating) materialized views`. + keep the view passthrough; aggregation, `SAMPLE BY`, `GROUP BY`, `LATEST ON`, + `DISTINCT`, `UNION`, and `JOIN` make it non-passthrough. - The view inherits the base table's [designated timestamp](/docs/concepts/designated-timestamp/) and partitioning. @@ -61,6 +63,17 @@ A passthrough view mirrors its base table 1:1 and refreshes incrementally, so it is effectively a continuously-maintained replica. `EXPIRE ROWS` prunes that replica down to the rows you want to keep — without touching the base table. +A **non-passthrough (aggregating) view is accepted with a logged advisory** +rather than rejected: the read filter still hides expired rows, but physical +reclamation only sticks when base-table retention is aligned with the expiry +horizon — a later incremental or full refresh can regenerate a reclaimed row +from base rows that still exist. + +A policied view must also stand alone: `CREATE MATERIALIZED VIEW` rejects a +defining query that reads a policied view (as its base or in a join), and +`ALTER ... SET EXPIRE` is rejected on a view that other materialized views +derive from — otherwise those views would copy expired rows on refresh. + ## The modes Every mode keeps a defined set of rows and expires the rest. A row is expired @@ -95,7 +108,7 @@ EXPIRE ROWS | `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp. | | `ON timestampCol` | Optional; if given it must name the view's designated timestamp. | | `HIGHEST\|LOWEST` | Keep rows at the max / min of `col` per group (`N` omitted), or the top `N` per group. | -| `CLEANUP EVERY` | How often the background reclamation job runs for this view. Defaults to `1h` if omitted. | +| `CLEANUP EVERY` | How often the background reclamation job runs for this view: `` with unit `s`/`m`/`h`/`d`/`w`. Defaults to `1h` if omitted. | ## Worked examples @@ -286,7 +299,9 @@ On QuestDB Enterprise, cleanup runs on the **primary only**, but the reclamation still replicates: the compaction commits are ordinary WAL transactions, so replicas reclaim the identical rows by applying them. A read-only replica neither runs the job nor needs to. Disable the job with `cairo.row.expiry.enabled=false` -in `server.conf` (reads stay filtered; only reclamation stops). +in `server.conf` (reads stay filtered; only reclamation stops); the setting is +read at startup, so changing it requires a restart. A failing sweep retries +after one second, doubling the per-view retry gap up to a 10-minute cap. To observe reclamation, compare the physical row count per partition before and after a sweep: @@ -328,20 +343,32 @@ is not already unique). Physical deletion is only safe when expiry is **monotonic**: a row that is expired now must stay expired forever. All the relative modes (`KEEP LATEST`, -`KEEP HIGHEST/LOWEST`, `KEEP N`) are monotonic by construction, as is a -designated-timestamp predicate such as `WHEN ts < now()` (a row only gets -older). A scalar `WHEN predicate` is arbitrary SQL, so **monotonicity is the -author's responsibility**. +`KEEP HIGHEST/LOWEST`, `KEEP N`) are monotonic by construction. A scalar +`WHEN predicate` is arbitrary SQL, so the cleanup job reclaims disk only for +predicates it can **prove** monotonic: + +- clock-free predicates (`WHEN amount < 1.5`), and +- designated-timestamp thresholds of a proven advancing-clock shape: a bare + clock (`ts < now()`), a bare clock minus a non-negative constant + (`ts < now() - 7200000000`), or a fixed-unit look-back `dateadd` on a bare + clock (`ts < dateadd('d', -1, now())`, units `s`/`m`/`h`/`d`/`w` and finer). + +Anything else **skips cleanup**: calendar units (`dateadd('M', -1, now())` — a +month is a variable amount), look-forward offsets (`dateadd('h', 1, now())`), +further clock arithmetic, non-constant offsets, and arbitrary window `WHEN` +predicates. A skipped policy stays correct at read time (the filter recomputes +on every read), but its disk is not reclaimed until the policy is changed to a +proven shape. :::warning A non-monotonic predicate such as `WHEN ts > now()` expires *future* rows that **un-expire** as `now()` advances. The read filter recomputes `now()` on every -read and stays correct, but the cleanup job assumes monotonicity and may -physically delete a row that a later read would otherwise show (recoverable only -by a full refresh). Write `WHEN` predicates that expire things in the **past** -(`ts < now()`) or against fixed thresholds — never rows that the passage of time -will later keep. +read and stays correct, and the cleanup job skips such a policy rather than +risk physically deleting a row a later read must show — at the cost of disk +never being reclaimed for it. Write `WHEN` predicates that expire things in the +**past** or against fixed thresholds — never rows that the passage of time will +later keep. ::: @@ -355,15 +382,15 @@ SHOW CREATE MATERIALIZED VIEW trades_latest; ``` The [`materialized_views()`](/docs/query/functions/meta/) function exposes the -policy in the `expire_predicate` and `expire_cleanup_every` columns (both +policy in the `expire_clause` and `expire_cleanup_every` columns (both `NULL` when no policy is set): ```questdb-sql title="List EXPIRE ROWS policies" -SELECT view_name, expire_predicate, expire_cleanup_every +SELECT view_name, expire_clause, expire_cleanup_every FROM materialized_views(); ``` -| view_name | expire_predicate | expire_cleanup_every | +| view_name | expire_clause | expire_cleanup_every | | ------------- | ------------------------------- | -------------------- | | trades_sized | amount < 1.5 | 1h | | trades_latest | KEEP LATEST PARTITION BY symbol | 1h | @@ -402,6 +429,16 @@ rather than breaking subsequent reads. rejected. - **Non-monotonic `WHEN` predicates are unsupported for cleanup** — see [monotonicity](#monotonicity-and-cleanup-safety) above. +- **Reserved column name.** The window/keep modes compute the keep-set through a + synthetic boolean column named `__qdb_re_keep`; a policy is rejected on a view + that exposes a column with that name. +- **No line comments in the clause.** The clause text is stored verbatim and + embedded into generated SQL, so `--` comments are rejected inside an + `EXPIRE ROWS` clause; terminated block comments (`/* ... */`) are fine. +- **Compacting a Parquet partition rewrites it as native storage.** When cleanup + compacts a *partially*-expired partition held in Parquet, the partition + reverts to native QuestDB storage until the Parquet-conversion job re-converts + it. Reclamation correctness is unaffected. ## Related documentation diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 8c3d4216b8..17367847d4 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -7,8 +7,9 @@ description: --- Sets, replaces, or removes an [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) -row-retention policy on a **passthrough** materialized view. Expired rows are -hidden from queries immediately and reclaimed on disk in the background. +row-retention policy on a materialized view (designed for **passthrough** +views — see the concept page). Expired rows are hidden from queries immediately +and reclaimed on disk in the background. ## Syntax @@ -79,12 +80,13 @@ ALTER MATERIALIZED VIEW trades_mirror DROP EXPIRE; ## Behavior -| Aspect | Description | -| ----------------- | ---------------------------------------------------------------------------- | -| Passthrough only | The view must be `SELECT * FROM base`; aggregating views are rejected | -| Validation | The policy is checked against the view's columns before it is applied | -| Immediate effect | Reads are filtered to the kept rows as soon as the policy is set | -| Replication | The policy and the reclamation it drives replicate as normal WAL traffic | +| Aspect | Description | +| ----------------------- | ---------------------------------------------------------------------------- | +| Passthrough recommended | An aggregating view is accepted with a logged advisory: a later refresh can regenerate reclaimed rows, so align base-table retention with the expiry horizon | +| No dependent views | Rejected when other materialized views derive from this view (they would copy expired rows on refresh) | +| Validation | The policy is checked against the view's columns before it is applied | +| Immediate effect | Reads are filtered to the kept rows as soon as the policy is set | +| Replication | The policy and the reclamation it drives replicate as normal WAL traffic | ## Permissions (Enterprise) @@ -99,9 +101,11 @@ GRANT ALTER MATERIALIZED VIEW ON trades_mirror TO user1; | Error | Cause | | ----- | ----- | | `materialized view does not exist` | View with the specified name doesn't exist | -| `EXPIRE ROWS is only supported on passthrough (non-aggregating) materialized views` | View aggregates (e.g. `SAMPLE BY`) | +| `cannot set an EXPIRE ROWS policy on '...': it is the base of N materialized view(s), which would copy expired rows on refresh` | Other materialized views derive from this view | | `EXPIRE ROWS KEEP LATEST ON must name the designated timestamp ...` | `ON` names a column other than the designated timestamp | | `invalid EXPIRE ROWS KEEP LATEST column: ...` | A `PARTITION BY` key column does not exist | +| `EXPIRE ROWS KEEP / window retention cannot be used on a view with a column named '__qdb_re_keep'` | The view exposes a column named like the reserved keep column | +| `invalid EXPIRE ROWS predicate: ...` | The predicate does not parse, bind, or type-check against the view's columns | | `permission denied` | Missing `ALTER MATERIALIZED VIEW` permission (Enterprise) | ## See also diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 19593d34cf..5c80d36caf 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -31,7 +31,7 @@ Where: - `timeUnit`: `HOURS | DAYS | WEEKS | MONTHS | YEARS` - `query`: Must contain `SAMPLE BY` or time-based `GROUP BY` (except for the passthrough views used by `EXPIRE ROWS` — see below) -- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON ts] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]` — passthrough views only +- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON ts] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]` — designed for passthrough views (see below) ## Parameters @@ -289,8 +289,11 @@ whole partitions by age), `EXPIRE ROWS` keeps a defined set of rows — the late per key, the top-N per group, or rows matching a predicate — recomputed continuously as the view refreshes. -`EXPIRE ROWS` applies **only to passthrough (non-aggregating) views** — the query -is `SELECT * FROM base` with no `SAMPLE BY` / `GROUP BY`: +`EXPIRE ROWS` is designed for **passthrough (non-aggregating) views** — the +query is `SELECT * FROM base` with no `SAMPLE BY` / `GROUP BY`. An aggregating +view is accepted with a logged advisory (a later refresh can regenerate +reclaimed rows). The defining query must not read another policied view, as its +base or in a join: ```questdb-sql title="Passthrough view that keeps the latest row per symbol" CREATE MATERIALIZED VIEW trades_latest AS ( From 6abf656b3f7fe9a077a15600587fcf5b1d0e0847 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Tue, 18 Aug 2026 14:52:55 +0200 Subject: [PATCH 03/25] Update docs to match changes in code --- .../concepts/deep-dive/expire-rows.md | 98 +++++++++++++------ documentation/query/functions/meta.md | 47 +++++++-- .../query/sql/alter-mat-view-set-expire.md | 3 +- documentation/query/sql/create-mat-view.md | 7 +- 4 files changed, 116 insertions(+), 39 deletions(-) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md index 70cd194159..310fe73534 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -4,8 +4,8 @@ sidebar_label: Expiring rows description: EXPIRE ROWS is a row-level retention policy for passthrough materialized views. Keep the latest row per key, the top-N per group, or rows matching a - predicate — recomputed continuously, with expired rows hidden immediately and - reclaimed in the background. + predicate — recomputed continuously, with expired rows hidden immediately, and + reclaimed in the background under the modes that allow it. --- `EXPIRE ROWS` is a row-level retention policy for @@ -15,8 +15,12 @@ ROWS` decides retention **row by row** — keep the latest row per key, the top- per group, rows matching a predicate, and so on — and recomputes the result continuously as the view refreshes. -Expired rows disappear from query results **immediately**; their on-disk storage -is reclaimed afterwards by a background job. +Expired rows disappear from query results **immediately** in every mode. Their +on-disk storage is reclaimed afterwards by a background job under a monotonic +`WHEN` predicate; the relative modes (`KEEP LATEST`, `KEEP HIGHEST/LOWEST`, +`KEEP N`) and window predicates hide rows without freeing disk. See +[The modes](#the-modes) and +[Monotonicity and cleanup safety](#monotonicity-and-cleanup-safety). :::note @@ -79,17 +83,27 @@ derive from — otherwise those views would copy expired rows on refresh. Every mode keeps a defined set of rows and expires the rest. A row is expired only when the rule selects it for removal. -| Mode | What it keeps | Syntax | -| --------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | -| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | -| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | -| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | -| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | -| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | +| Mode | What it keeps | Syntax | Frees disk | +| --------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | --------------------- | +| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when monotonic | +| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | No — read filter only | +| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | +| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | +| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No — read filter only | `KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a window predicate, so the window `WHEN` is the general escape hatch. +The **Frees disk** column is the difference between hiding a row and deleting +it. Every mode hides expired rows from every read, immediately. Only a monotonic +`WHEN` predicate also has those rows deleted from disk by the cleanup job: a +mode whose keep-set depends on the other rows in the view keeps its expired rows +on disk until a full refresh rebuilds the view. A `KEEP LATEST` view therefore +holds a full copy of its base table unless the view's own +[TTL](/docs/concepts/ttl/) bounds it. +`materialized_views().expire_enforcement` reports which of the two a given view +gets; see [Inspecting a policy](#inspecting-a-policy). + The clause is attached to a passthrough `CREATE MATERIALIZED VIEW` (after the query, and after `PARTITION BY` if present), or set later with [`ALTER MATERIALIZED VIEW ... SET EXPIRE ROWS`](/docs/query/sql/alter-mat-view-set-expire/): @@ -295,13 +309,21 @@ survivors. It runs at the `CLEANUP EVERY` cadence (default `1h`) and is **best-effort** — the read filter is authoritative, so deferred or skipped reclamation only affects disk usage, never query results. +The job runs only under a monotonic `WHEN` predicate. It skips `KEEP LATEST`, +`KEEP HIGHEST/LOWEST`, `KEEP N` and window policies entirely: a later refresh +can remove the row those modes currently keep, which promotes an older row back +into the keep-set, and the job cannot reconstruct a row it has already deleted. +Those views accumulate their expired rows on disk. + On QuestDB Enterprise, cleanup runs on the **primary only**, but the reclamation still replicates: the compaction commits are ordinary WAL transactions, so replicas reclaim the identical rows by applying them. A read-only replica neither -runs the job nor needs to. Disable the job with `cairo.row.expiry.enabled=false` -in `server.conf` (reads stay filtered; only reclamation stops); the setting is -read at startup, so changing it requires a restart. A failing sweep retries -after one second, doubling the per-view retry gap up to a 10-minute cap. +runs the job nor needs to. Disable the job with +`cairo.row.expiry.cleanup.enabled=false` in `server.conf` (reads stay filtered; +only reclamation stops — the setting does not disable `EXPIRE ROWS` itself); the +setting is read at startup, so changing it requires a restart. A failing sweep +retries after one second, doubling the per-view retry gap up to a 10-minute +cap. To observe reclamation, compare the physical row count per partition before and after a sweep: @@ -342,10 +364,17 @@ is not already unique). ### Monotonicity and cleanup safety Physical deletion is only safe when expiry is **monotonic**: a row that is -expired now must stay expired forever. All the relative modes (`KEEP LATEST`, -`KEEP HIGHEST/LOWEST`, `KEEP N`) are monotonic by construction. A scalar -`WHEN predicate` is arbitrary SQL, so the cleanup job reclaims disk only for -predicates it can **prove** monotonic: +expired now must stay expired forever. Two separate things can break that. + +The relative and window modes (`KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N`, +window `WHEN`) decide each row's fate by comparing it against the other rows in +the view. A later refresh can remove or replace the row a key currently keeps, +which promotes an older row back into the keep-set — so the cleanup job never +deletes for these modes, whatever their predicate looks like. + +A scalar `WHEN predicate` judges each row on its own, so it is eligible. It is +arbitrary SQL, so the cleanup job reclaims disk only for predicates it can +**prove** monotonic: - clock-free predicates (`WHEN amount < 1.5`), and - designated-timestamp thresholds of a proven advancing-clock shape: a bare @@ -382,19 +411,27 @@ SHOW CREATE MATERIALIZED VIEW trades_latest; ``` The [`materialized_views()`](/docs/query/functions/meta/) function exposes the -policy in the `expire_clause` and `expire_cleanup_every` columns (both -`NULL` when no policy is set): +policy in the `expire_clause`, `expire_cleanup_every` and `expire_enforcement` +columns (all `NULL` when no policy is set): ```questdb-sql title="List EXPIRE ROWS policies" -SELECT view_name, expire_clause, expire_cleanup_every +SELECT view_name, expire_clause, expire_cleanup_every, expire_enforcement FROM materialized_views(); ``` -| view_name | expire_clause | expire_cleanup_every | -| ------------- | ------------------------------- | -------------------- | -| trades_sized | amount < 1.5 | 1h | -| trades_latest | KEEP LATEST PARTITION BY symbol | 1h | -| trades_top2 | KEEP 2 HIGHEST price ... | 1h | +| view_name | expire_clause | expire_cleanup_every | expire_enforcement | +| ------------- | ------------------------------- | -------------------- | ------------------ | +| trades_sized | amount < 1.5 | 1h | FILTER_AND_RECLAIM | +| trades_latest | KEEP LATEST PARTITION BY symbol | 1h | FILTER_ONLY | +| trades_top2 | KEEP 2 HIGHEST price ... | 1h | FILTER_ONLY | + +`expire_enforcement` is the verdict the cleanup job acts on: + +- `FILTER_AND_RECLAIM` — reads hide the expired rows and the job deletes them + from disk. +- `FILTER_ONLY` — reads hide the expired rows and they stay on disk. Every + relative and window policy reports this, as does a `WHEN` predicate that + cannot be proven monotonic. ## Changing or removing a policy @@ -427,8 +464,11 @@ rather than breaking subsequent reads. - **`KEEP LATEST [ON ts]`.** The optional `ON ts` is accepted for familiarity but the view's designated timestamp is always used; naming a different column is rejected. -- **Non-monotonic `WHEN` predicates are unsupported for cleanup** — see - [monotonicity](#monotonicity-and-cleanup-safety) above. +- **Only a monotonic `WHEN` predicate frees disk.** The relative and window + modes hide expired rows on every read but never delete them, and neither does + a `WHEN` predicate that cannot be proven monotonic — see + [monotonicity](#monotonicity-and-cleanup-safety) above. Check a view's verdict + with `materialized_views().expire_enforcement`. - **Reserved column name.** The window/keep modes compute the keep-set through a synthetic boolean column named `__qdb_re_keep`; a policy is rejected on a view that exposes a column with that name. diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index 1c78bfce82..6d0254aa6f 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -262,25 +262,58 @@ Returns a `table` including the following information: - `view_table_dir_name` - view directory name - `invalidation_reason` - message explaining why the view was marked as invalid - `view_status` - view status: 'valid', 'refreshing', or 'invalid' +- `refresh_period_hi` - end of the last refreshed period, for a view defined + with a `REFRESH PERIOD` - `refresh_base_table_txn` - the last base table transaction used to refresh the materialized view - `base_table_txn` - the last committed transaction in the base table -- `refresh_limit_value` - how many units back in time the refresh limit goes +- `refresh_limit` - how many units back in time the refresh limit goes - `refresh_limit_unit` - how long each unit is +- `timer_time_zone` - time zone the refresh timer and period boundaries use - `timer_start` - start date for the scheduled refresh timer -- `timer_interval_value` - how many interval units between each refresh +- `timer_interval` - how many interval units between each refresh - `timer_interval_unit` - how long each unit is +- `period_length` - length of one refresh period (`PERIOD (LENGTH ...)`) +- `period_length_unit` - how long each period-length unit is +- `period_delay` - delay before a completed period is refreshed + (`PERIOD (DELAY ...)`) +- `period_delay_unit` - how long each period-delay unit is +- `refresh_avg_commit_nanos` - moving average of one refresh commit, in + nanoseconds. Held in memory only, so it resets on restart +- `refresh_avg_scan_sample_nanos` - moving average of one refresh iteration's + base-table scan, in nanoseconds. In memory only +- `refresh_avg_scan_range_ts_units` - moving average of the timestamp range one + refresh iteration covers, in the base table's timestamp unit. In memory only +- `refresh_gap_threshold_ts_units` - timestamp gap below which the refresh job + merges two adjacent intervals instead of paying for a second commit. `0` means + merging is disabled +- `expire_clause` - the view's + [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy as written, or + `NULL` when the view has no policy +- `expire_cleanup_every` - how often the cleanup job runs for the policy, or + `NULL` +- `expire_enforcement` - `FILTER_AND_RECLAIM` when the cleanup job frees disk + for the policy, `FILTER_ONLY` when reads hide the expired rows but they stay + on disk, `NULL` when the view has no policy **Examples:** +`materialized_views()` on its own returns every column listed above. The example +below projects a readable subset over three views: an aggregating one with no +retention policy, and two passthrough views with an +[`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy. + ```questdb-sql title="List all materialized views" -materialized_views(); +SELECT view_name, view_status, base_table_name, refresh_base_table_txn, + base_table_txn, expire_clause, expire_cleanup_every, expire_enforcement +FROM materialized_views(); ``` -| view_name | refresh_type | base_table_name | last_refresh_start_timestamp | last_refresh_finish_timestamp | view_sql | view_table_dir_name | invalidation_reason | view_status | refresh_base_table_txn | base_table_txn | refresh_limit_value | refresh_limit_unit | timer_start | timer_interval_value | timer_interval_unit | -|------------------|--------------|-----------------|------------------------------|-------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|---------------------|-------------|------------------------|----------------|---------------------|--------------------|-------------|----------------------|---------------------| -| trades_OHLC_15m | immediate | trades | 2025-05-30T16:40:37.562421Z | 2025-05-30T16:40:37.568800Z | SELECT timestamp, symbol, first(price) AS open, max(price) as high, min(price) as low, last(price) AS close, sum(amount) AS volume FROM trades SAMPLE BY 15m | trades_OHLC_15m~27 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | -| trades_latest_1d | immediate | trades | 2025-05-30T16:40:37.554274Z | 2025-05-30T16:40:37.562049Z | SELECT timestamp, symbol, side, last(price) AS price, last(amount) AS amount, last(timestamp) as latest FROM trades SAMPLE BY 1d | trades_latest_1d~28 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | +| view_name | view_status | base_table_name | refresh_base_table_txn | base_table_txn | expire_clause | expire_cleanup_every | expire_enforcement | +| --------------- | ----------- | --------------- | ---------------------- | -------------- | ------------------------------------ | -------------------- | ------------------ | +| trades_OHLC_15m | valid | trades | 1 | 1 | null | null | null | +| trades_recent | valid | trades | 1 | 1 | timestamp < dateadd('d', -7, now()) | 30m | FILTER_AND_RECLAIM | +| trades_latest | valid | trades | 1 | 1 | KEEP LATEST PARTITION BY symbol | 1h | FILTER_ONLY | ## memory_metrics diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 17367847d4..c02337bdb9 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -9,7 +9,8 @@ description: Sets, replaces, or removes an [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) row-retention policy on a materialized view (designed for **passthrough** views — see the concept page). Expired rows are hidden from queries immediately -and reclaimed on disk in the background. +in every mode, and reclaimed on disk in the background under a monotonic `WHEN` +predicate — the relative and window modes hide rows without freeing disk. ## Syntax diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 5c80d36caf..ee597a5377 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -311,8 +311,11 @@ EXPIRE ROWS [ CLEANUP EVERY duration ] ``` -Expired rows are hidden from queries immediately and reclaimed on disk in the -background (`CLEANUP EVERY`, default `1h`). Change or remove a policy with +Expired rows are hidden from queries immediately in every mode. They are +reclaimed on disk in the background (`CLEANUP EVERY`, default `1h`) under a +monotonic `WHEN` predicate; `KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N` and +window predicates hide rows without freeing disk. Change or remove a policy +with [`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/). See the [Expiring rows](/docs/concepts/deep-dive/expire-rows/) concept page for From e01ce4fc599c2a38fea720abe12b5a1ad53b02d2 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Wed, 19 Aug 2026 10:56:38 +0200 Subject: [PATCH 04/25] Clarified interaction between TTL and EXPIRE ROWS --- .../concepts/deep-dive/expire-rows.md | 32 ++++++++++++++++++- documentation/query/sql/create-mat-view.md | 5 +++ 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md index 310fe73534..63f5310d94 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -100,7 +100,8 @@ it. Every mode hides expired rows from every read, immediately. Only a monotonic mode whose keep-set depends on the other rows in the view keeps its expired rows on disk until a full refresh rebuilds the view. A `KEEP LATEST` view therefore holds a full copy of its base table unless the view's own -[TTL](/docs/concepts/ttl/) bounds it. +[TTL](/docs/concepts/ttl/) bounds it — see +[Combining with TTL](#combining-with-ttl). `materialized_views().expire_enforcement` reports which of the two a given view gets; see [Inspecting a policy](#inspecting-a-policy). @@ -361,6 +362,35 @@ timestamp as a tiebreak, so the N-th boundary is deterministic (pair the base table with [`DEDUP UPSERT KEYS`](/docs/concepts/deduplication/) if `(col, ts)` is not already unique). +### Combining with TTL + +A view can carry a [TTL](/docs/concepts/ttl/) and an `EXPIRE ROWS` policy at the +same time, and the order is fixed: **TTL removes rows from the view first, then +the policy applies to the rows that stay.** TTL drops whole partitions from the +view's own storage as they age out, and the keep-set is computed over what +remains. + +```questdb-sql title="Highest price per symbol, over a 3-day window" +CREATE MATERIALIZED VIEW trades_peak_3d AS ( + SELECT * FROM trades +) PARTITION BY DAY TTL 3 DAYS + EXPIRE ROWS KEEP HIGHEST price PARTITION BY symbol; +``` + +`TTL` goes before `EXPIRE ROWS` in the statement, as it does after any +`PARTITION BY`. + +This view reports the highest price of the **last three days**, so its answer +can go **down** as the window moves: when the day holding a symbol's maximum +ages out, the next-highest price within the window takes over. That is what the +two clauses ask for together — the view is no longer "the highest price ever", +it is "the highest price still retained". The base table is unaffected; it keeps +whatever its own retention settings keep. + +TTL is also the only control that bounds the size of a `KEEP LATEST`, +`KEEP HIGHEST/LOWEST` or `KEEP N` view, since the cleanup job never reclaims +disk for those modes. + ### Monotonicity and cleanup safety Physical deletion is only safe when expiry is **monotonic**: a row that is diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index ee597a5377..c4073b4e24 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -318,6 +318,11 @@ window predicates hide rows without freeing disk. Change or remove a policy with [`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/). +A view can carry both `TTL` and `EXPIRE ROWS`. `TTL` comes first in the +statement and first in effect: it removes rows from the view, and the +`EXPIRE ROWS` policy then applies to the rows that stay — see +[Combining with TTL](/docs/concepts/deep-dive/expire-rows/#combining-with-ttl). + See the [Expiring rows](/docs/concepts/deep-dive/expire-rows/) concept page for all modes, worked examples, and semantics (NULLs, ties, monotonicity). From 6e444360b064e07013f1c70f8b11a24f4fd2c918 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Mon, 24 Aug 2026 15:27:30 +0200 Subject: [PATCH 05/25] Document passthrough views as a view shape Passthrough materialized views only appeared in the docs as a prerequisite of EXPIRE ROWS. The single sentence describing which queries qualify lived in the EXPIRE ROWS concept page, and every example on every page was literally "SELECT * FROM base", so a reader who wanted a maintained subset of a table without a retention policy had no reason to find any of it. Three statements also still said a materialized view query has to aggregate, which is no longer true: - "Materialized views require a `SAMPLE BY` or time-based `GROUP BY` query." in the concepts page - "Must use `SAMPLE BY` or `GROUP BY` with a designated timestamp column" in the same page's Technical reference - "Query must aggregate" in the CREATE MATERIALIZED VIEW rules table Give the shape its own section in the general materialized views concept page: what a passthrough view is, when to reach for one (a narrowed replica of a big table, or a target for EXPIRE ROWS), what it inherits from the base table, and which queries qualify. The inherited-properties part records something documented nowhere before: a passthrough view inherits the base table's symbol indexes, under whatever alias the projection gives the column, so an indexed lookup on the view costs what it costs on the base. An aggregating view never inherits an index, because its rows are not base rows. The query table lists shapes verified by running them, rather than read off isPassthrough(). A column subset, a column alias and a row-local expression such as "price * amount AS notional" are all passthrough. A window function is rejected, but not by either check inside isPassthrough() -- a separate guard reports window function on base table is not supported for materialized views Dropping the designated timestamp, including by an ORDER BY on another column, is rejected with materialized view query is required to have designated timestamp and REFRESH PERIOD is rejected with PERIOD is not supported for non-aggregating (passthrough) materialized views while REFRESH MANUAL and REFRESH EVERY are both accepted. Add a section to the EXPIRE ROWS page on choosing between a WHERE clause in the view's query and an EXPIRE ROWS WHEN predicate. For a deterministic predicate that judges each row on its own the two keep the same rows, and nothing said how to pick between them. A WHERE clause is cheaper on storage, on reads and on writes, because the row is never written. An EXPIRE ROWS policy can be retuned with ALTER instead of forcing a drop and re-create, which re-reads the base table and can lose rows the base no longer holds. A policied view also cannot serve as another view's base. Also correct the claim that JOIN enrichment fails because SAMPLE BY is mandatory. A passthrough view does keep raw rows; it just has to read a single table, so enrichment stays unavailable for a different reason. Co-Authored-By: Claude Opus 5 --- .../concepts/deep-dive/expire-rows.md | 73 ++++++++++++- documentation/concepts/materialized-views.md | 103 +++++++++++++++++- documentation/query/sql/create-mat-view.md | 21 ++-- 3 files changed, 179 insertions(+), 18 deletions(-) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md index 63f5310d94..5a0cdbbee4 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -57,11 +57,13 @@ requirement. `EXPIRE ROWS` is designed for a **passthrough materialized view**: -- The view query is `SELECT * FROM base`. A column subset and a `WHERE` filter - keep the view passthrough; aggregation, `SAMPLE BY`, `GROUP BY`, `LATEST ON`, - `DISTINCT`, `UNION`, and `JOIN` make it non-passthrough. +- The view query keeps view rows 1:1 with base rows — a projection over a single + table, with or without a `WHERE` filter. See + [which queries are passthrough](/docs/concepts/materialized-views/#which-queries-are-passthrough) + for the full rules. - The view inherits the base table's - [designated timestamp](/docs/concepts/designated-timestamp/) and partitioning. + [designated timestamp](/docs/concepts/designated-timestamp/), partitioning and + symbol indexes. A passthrough view mirrors its base table 1:1 and refreshes incrementally, so it is effectively a continuously-maintained replica. `EXPIRE ROWS` prunes that @@ -78,6 +80,67 @@ defining query that reads a policied view (as its base or in a join), and `ALTER ... SET EXPIRE` is rejected on a view that other materialized views derive from — otherwise those views would copy expired rows on refresh. +## `WHERE` filter or `EXPIRE ROWS`? + +A passthrough view can exclude rows in two places: a `WHERE` clause in its +defining query, or an `EXPIRE ROWS WHEN` predicate. For a deterministic, +row-local predicate the two describe the same set of surviving rows, so the +choice is not about what the view contains — it is about whether the rule is +part of the view's *definition* or a *knob you expect to turn*. + +**Default to `WHERE`.** A row a `WHERE` clause excludes is never written, so it +costs nothing at any stage: + +| | `WHERE` in the query | `EXPIRE ROWS WHEN` | +| --- | --- | --- | +| Storage | Row is never written | Row is written, and occupies disk until a sweep reclaims it | +| Read cost | None | The keep-set filter is applied on every read of the view | +| Write cost | None | The cleanup job rewrites partitions on the `CLEANUP EVERY` cadence | +| After a full refresh | Still excluded | Re-materialized from the base, then hidden and swept again | +| Can be another view's base | Yes | No — a policied view is rejected as a base | + +That last row is a hard constraint rather than a preference. If any other view +will read this one, the policy is not available and the predicate has to go in +the `WHERE` clause. + +**Reach for `EXPIRE ROWS` when the cutoff is a knob.** There is no +`ALTER MATERIALIZED VIEW ... AS `, so changing a `WHERE` clause means +dropping the view and re-creating it, which re-materializes it from the base. +Changing a policy is a metadata operation: + +```questdb-sql title="Retuning a retention horizon without a rebuild" +ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-06-01T00:00:00.000000Z'; +ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-07-01T00:00:00.000000Z'; +ALTER MATERIALIZED VIEW trades_recent DROP EXPIRE; +``` + +The rebuild a `WHERE` change forces is not only slow, it can lose data: if the +base table has its own [TTL](/docs/concepts/ttl/), re-creating the view reads a +base that no longer holds everything the view held. A view whose retention +horizon is longer than its base table's cannot afford to be rebuilt, so its +cutoff belongs in a policy. + +Two things only `EXPIRE ROWS` can express at all, whichever way you lean: + +- **Rules that move with the clock.** A view's defining query rejects + non-deterministic functions, so `WHERE ts > dateadd('d', -7, now())` is not + accepted. `EXPIRE ROWS WHEN ts < dateadd('d', -7, now())` is the supported way + to write a rolling window. +- **Rules that compare rows against each other.** `KEEP LATEST`, + `KEEP N HIGHEST/LOWEST` and window predicates have no `WHERE` equivalent — a + `LATEST ON` or a window function in the defining query makes the view + non-passthrough. + +In practice the two compose, and on a passthrough view that is usually the right +shape: the `WHERE` clause fixes what the view is about, and the policy fixes how +long it keeps what it has. + +```questdb-sql title="A filter for the subject, a policy for the horizon" +CREATE MATERIALIZED VIEW trades_btc_recent AS ( + SELECT * FROM trades WHERE symbol = 'BTC' +) EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 1h; +``` + ## The modes Every mode keeps a defined set of rows and expires the rest. A row is expired @@ -514,6 +577,8 @@ rather than breaking subsequent reads. - [Materialized views](/docs/concepts/materialized-views/) — the view type `EXPIRE ROWS` runs on +- [Passthrough views](/docs/concepts/materialized-views/#passthrough-views) — + the non-aggregating view shape `EXPIRE ROWS` is designed for - [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/) — full create syntax, including the `EXPIRE ROWS` clause - [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index cf7ed4ac26..f8f3e7eea8 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -12,6 +12,11 @@ their results at query time, materialized views persist their data to disk, making them particularly efficient for expensive aggregate queries that are run frequently. +Most materialized views aggregate, bucketing base rows with `SAMPLE BY` or a +time-based `GROUP BY`. A view can also be a +[passthrough view](#passthrough-views), which projects base rows one-for-one +instead of summarising them. + ## What are materialized views for? Let's say your application ingests trade data into a table like this: @@ -116,9 +121,10 @@ reads on a smaller, pre-aggregated dataset. ### Not suited for: data enrichment -Materialized views support JOINs, but `SAMPLE BY` (aggregation) is mandatory. -This means you can enrich aggregated results with data from other tables, but -you cannot keep raw (non-aggregated) rows while adding enrichment columns. +Materialized views support JOINs only in an aggregating query. A +[passthrough view](#passthrough-views) keeps raw rows, but it must read a single +table. So neither shape lets you keep raw rows while adding columns from another +table. For example, joining aggregated trades with instrument metadata works: @@ -157,6 +163,89 @@ refresh. Changes to joined tables do not trigger updates. **Coming soon**: We are actively developing a new type of materialized view that will support data enrichment use cases. Stay tuned for updates. +## Passthrough views + +Not every materialized view aggregates. A **passthrough view** projects base +rows one-for-one instead of bucketing them, so the view is a +continuously-maintained copy of its base table, optionally narrowed to a subset +of columns, a subset of rows, or both. Its query has no `SAMPLE BY` and no +time-based `GROUP BY`: + +```questdb-sql title="Passthrough view: a maintained, filtered copy of trades" +CREATE MATERIALIZED VIEW trades_btc AS ( + SELECT timestamp, symbol, price, amount + FROM trades + WHERE symbol = 'BTC' +); +``` + +Reach for one when you want a maintained *subset* of a large table rather than a +summary of it: + +- **A narrowed replica** — one symbol, one tenant, one region, or a handful of + columns out of a wide table, kept current automatically and queried without + the base table's scan cost. +- **A row-level retention target** — a passthrough view is the shape + [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) is built for. Its rows + *are* base rows, so reclaiming one is permanent. + +Passthrough views refresh incrementally like any other materialized view, and +accept the same `REFRESH IMMEDIATE` (the default), `REFRESH MANUAL` and +`REFRESH EVERY` strategies. `REFRESH PERIOD` is rejected: there are no buckets +for a period to align to. + +### What a passthrough view inherits + +A passthrough view takes its shape from the base table instead of from a +`SAMPLE BY` clause: + +- **Designated timestamp and partitioning** come from the base table. The + projection has to keep the designated timestamp; a query that drops it is + rejected with `materialized view query is required to have designated + timestamp`. `PARTITION BY` and `TTL` can still be stated explicitly. +- **Symbol indexes are inherited.** A base column declared `SYMBOL INDEX` stays + indexed in the view, under whatever alias the projection gives it, so an + indexed lookup on the view costs what it costs on the base. An aggregating + view never inherits an index, because its rows are not base rows. + +### Which queries are passthrough + +The rule is that view rows stay 1:1 with base rows. A projection over a single +table qualifies, with or without a filter: + +| Query | | +| ----- | --- | +| `SELECT * FROM trades` | Passthrough | +| `SELECT timestamp, symbol, price FROM trades` | Passthrough — column subset | +| `SELECT timestamp, symbol AS ticker FROM trades` | Passthrough — aliases are fine | +| `SELECT timestamp, price * amount AS notional FROM trades` | Passthrough — a row-local expression | +| `SELECT * FROM trades WHERE symbol = 'BTC'` | Passthrough — a filter only removes rows | +| `... SAMPLE BY 1h`, or `GROUP BY` on a timestamp | Aggregating view | +| `SELECT DISTINCT ...` | Rejected | +| `... LATEST ON timestamp PARTITION BY symbol` | Rejected | +| `JOIN`, `UNION` | Rejected | +| `row_number() OVER (...)` and other window functions | Rejected | +| `LIMIT` | Rejected | +| `ORDER BY` a non-timestamp column | Rejected — the view loses its designated timestamp | + +Everything in the rejected group produces output that depends on rows *other +than* the one being emitted. An incremental refresh sees only the newly-arrived +slice of the base table, so it cannot compute those correctly — a `LIMIT 100` +would admit 100 rows per refresh rather than 100 in total, and a `row_number()` +would restart within each slice. + +A query that is neither passthrough nor aggregating is rejected at creation +time. Most report `materialized view query requires a sampling interval, use +SAMPLE BY or GROUP BY timestamp_floor()`: the query looked like it meant to +aggregate but named no bucket. `LIMIT` and window functions have their own +messages. + +To keep only some of a passthrough view's rows over time — the latest per key, +the top-N per group, or rows matching a predicate — attach an +[`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy. That page also +covers +[when to put a predicate in the view's `WHERE` clause instead](/docs/concepts/deep-dive/expire-rows/#where-filter-or-expire-rows). + ## Creating a materialized view ### Basic syntax @@ -214,7 +303,9 @@ interval: ### The query -Materialized views require a `SAMPLE BY` or time-based `GROUP BY` query. +An aggregating materialized view uses a `SAMPLE BY` or time-based `GROUP BY` +query. (The other shape is a [passthrough view](#passthrough-views), which does +not aggregate; the rules below apply to aggregating views.) **Supported:** @@ -526,7 +617,9 @@ rows. Materialized view queries: -- Must use `SAMPLE BY` or `GROUP BY` with a designated timestamp column +- Must either aggregate with `SAMPLE BY` / `GROUP BY` on a designated timestamp + column, or be a [passthrough](#passthrough-views) projection over a single + table - Must not use `FROM-TO`, `FILL`, or `ALIGN TO FIRST OBSERVATION` - Must not use non-deterministic functions (`now()`, `rnd_uuid4()`) - Must use join conditions compatible with incremental refresh diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index c4073b4e24..135e25e1fd 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -29,8 +29,10 @@ AS [ ( ] query [ ) ] Where: - `interval`: Duration like `1m`, `10m`, `1h`, `1d` - `timeUnit`: `HOURS | DAYS | WEEKS | MONTHS | YEARS` -- `query`: Must contain `SAMPLE BY` or time-based `GROUP BY` (except for the - passthrough views used by `EXPIRE ROWS` — see below) +- `query`: Either an aggregating query with `SAMPLE BY` or a time-based + `GROUP BY`, or a + [passthrough](/docs/concepts/materialized-views/#passthrough-views) projection + over a single table - `expirePolicy`: `WHEN predicate | KEEP LATEST [ON ts] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]` — designed for passthrough views (see below) ## Parameters @@ -42,7 +44,7 @@ Where: | `WITH BASE` | Specify base table (required for JOINs) | | `REFRESH` | Refresh strategy (default: `IMMEDIATE`) | | `DEFERRED` | Skip initial refresh on creation | -| `query` | A `SAMPLE BY` or time-based `GROUP BY` query | +| `query` | An aggregating (`SAMPLE BY` / time-based `GROUP BY`) or [passthrough](/docs/concepts/materialized-views/#passthrough-views) query | | `TIMESTAMP` | Designate timestamp column for the view | | `PARTITION BY` | Partitioning unit for view storage | | `TTL` | Retention period for view data | @@ -53,7 +55,7 @@ Where: | Rule | Description | | ---- | ----------- | -| Query must aggregate | Requires `SAMPLE BY` or `GROUP BY` with designated timestamp | +| Query must aggregate or be passthrough | Either `SAMPLE BY` / `GROUP BY` with a designated timestamp, or a 1:1 [passthrough](/docs/concepts/materialized-views/#passthrough-views) projection over a single table | | Default refresh | `IMMEDIATE` (refreshes after each base table transaction) | | WITH BASE required | Must specify when query contains JOINs | | PARTITION BY sizing | Should be larger than or equal to `SAMPLE BY` interval | @@ -289,11 +291,12 @@ whole partitions by age), `EXPIRE ROWS` keeps a defined set of rows — the late per key, the top-N per group, or rows matching a predicate — recomputed continuously as the view refreshes. -`EXPIRE ROWS` is designed for **passthrough (non-aggregating) views** — the -query is `SELECT * FROM base` with no `SAMPLE BY` / `GROUP BY`. An aggregating -view is accepted with a logged advisory (a later refresh can regenerate -reclaimed rows). The defining query must not read another policied view, as its -base or in a join: +`EXPIRE ROWS` is designed for +[**passthrough (non-aggregating) views**](/docs/concepts/materialized-views/#passthrough-views) +— a projection over a single table, with no `SAMPLE BY` / `GROUP BY`, whose rows +stay 1:1 with the base. An aggregating view is accepted with a logged advisory +(a later refresh can regenerate reclaimed rows). The defining query must not +read another policied view, as its base or in a join: ```questdb-sql title="Passthrough view that keeps the latest row per symbol" CREATE MATERIALIZED VIEW trades_latest AS ( From f3ef9cc142571a3526e53680e5866a956c086341 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Mon, 24 Aug 2026 18:07:05 +0200 Subject: [PATCH 06/25] Steer EXPIRE ROWS WHEN toward clock-based rules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A predicate that looks only at the row's own values, such as EXPIRE ROWS WHEN amount < 1.5 selects the same surviving rows as writing the opposite test in the view's own query: SELECT * FROM trades WHERE amount >= 1.5 The `WHERE` version is cheaper everywhere. The rows it excludes are never copied into the view, so they cost no disk, no read-time filtering, and no rewriting of partitions by the cleanup job. The `EXPIRE ROWS` version writes those rows, hides them on every read, and reclaims them later. The docs presented the two as a roughly even choice and left the one thing only `EXPIRE ROWS WHEN` can do — a cutoff that moves with the clock — as a footnote at the end of the comparison. Someone reading quickly could reasonably conclude that any filter is fine to write as a policy. The comparison section now opens with the rule instead: use `EXPIRE ROWS WHEN` for predicates that involve wall-clock time, and put a predicate that depends only on the row's values in the `WHERE` clause. A rolling window genuinely has no `WHERE` equivalent, because a view's defining query rejects non-deterministic functions and will not accept `now()`. The argument for keeping a fixed, hand-advanced cutoff in a policy is still true and still here, moved into its own subsection so it reads as the exception rather than an equal option. The section keeps its `#where-filter-or-expire-rows` anchor, which `materialized-views.md` links to. Two other notes were added while rewriting. The two forms are not exact opposites where `NULL`s are concerned: `WHERE` keeps a row only when the test is `TRUE`, while `EXPIRE ROWS WHEN` expires a row only when the test is `TRUE`, so a `NULL` amount is dropped by the first and kept by the second. And the worked example that expires small trades now says up front that it is written that way to make the kept rows easy to read off the sample data, not because that is where such a filter belongs. The same guidance was added to the two SQL reference pages, so a reader who lands on one of them without reading the concept page still sees it. The `ALTER MATERIALIZED VIEW SET EXPIRE` page previously opened its examples with `WHEN amount < 1.5`, which demonstrated exactly the shape being discouraged; that example is now a rolling seven-day window. Co-Authored-By: Claude Opus 5 --- .../concepts/deep-dive/expire-rows.md | 93 ++++++++++++------- documentation/concepts/materialized-views.md | 7 +- .../query/sql/alter-mat-view-set-expire.md | 10 +- documentation/query/sql/create-mat-view.md | 7 ++ 4 files changed, 77 insertions(+), 40 deletions(-) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md index 5a0cdbbee4..f2c3a1ab0d 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -45,10 +45,16 @@ continuously-maintained, pruned copy of a base table: symbol, or session (`KEEP LATEST`). - **Per-group extremes or leaderboards** — keep the highest/lowest value per group, or the top-N (`KEEP HIGHEST/LOWEST`, `KEEP N`). -- **Rolling row-level windows** — keep rows newer than a cutoff, or matching any - predicate, at finer granularity than TTL's whole-partition drops +- **Rolling row-level windows** — keep rows newer than a moving cutoff such as + `now() - 7d`, at finer granularity than TTL's whole-partition drops (`WHEN predicate`). +The `WHEN` form earns its keep on predicates that involve **wall-clock time**. A +deterministic predicate — one that depends only on the row's own values — +selects the same rows more cheaply as a `WHERE` clause in the view's defining +query, which never copies the excluded rows into the view at all. See +[`WHERE` filter or `EXPIRE ROWS`?](#where-filter-or-expire-rows). + Use [TTL](/docs/concepts/ttl/) instead when partition-granularity, age-based retention on a base table is enough — it is cheaper and has no passthrough-view requirement. @@ -83,13 +89,22 @@ derive from — otherwise those views would copy expired rows on refresh. ## `WHERE` filter or `EXPIRE ROWS`? A passthrough view can exclude rows in two places: a `WHERE` clause in its -defining query, or an `EXPIRE ROWS WHEN` predicate. For a deterministic, -row-local predicate the two describe the same set of surviving rows, so the -choice is not about what the view contains — it is about whether the rule is -part of the view's *definition* or a *knob you expect to turn*. - -**Default to `WHERE`.** A row a `WHERE` clause excludes is never written, so it -costs nothing at any stage: +defining query, or an `EXPIRE ROWS WHEN` predicate. The dividing line is the +clock. + +**Use `EXPIRE ROWS WHEN` for rules that move with wall-clock time.** A rolling +window cannot be written as a `WHERE` clause at all: a view's defining query +rejects non-deterministic functions, so `WHERE ts > dateadd('d', -7, now())` is +not accepted. `EXPIRE ROWS WHEN ts < dateadd('d', -7, now())` is the supported +way to say "keep the last 7 days" — the read filter re-evaluates `now()` on +every read, so the window rolls forward on its own and the cleanup job reclaims +the disk behind it. This is what the `WHEN` form is for. + +**Put a deterministic predicate in the `WHERE` clause.** A predicate that +depends only on the row's own values — `symbol = 'BTC'`, `amount >= 1.5` — +describes the same surviving rows either way, so the two are near-equivalent in +what the view contains, and `WHERE` is the cheaper of the two at every stage. A +row the `WHERE` clause excludes is never copied into the view: | | `WHERE` in the query | `EXPIRE ROWS WHEN` | | --- | --- | --- | @@ -103,7 +118,25 @@ That last row is a hard constraint rather than a preference. If any other view will read this one, the policy is not available and the predicate has to go in the `WHERE` clause. -**Reach for `EXPIRE ROWS` when the cutoff is a knob.** There is no +The two forms are not exact negations of each other on `NULL`s: `WHERE` keeps a +row only when the predicate is `TRUE`, while `EXPIRE ROWS WHEN` expires a row +only when it is `TRUE`. A `NULL` amount is dropped by `WHERE amount >= 1.5` and +kept by `EXPIRE ROWS WHEN amount < 1.5` — see [NULLs](#nulls). + +The two compose, and on a passthrough view that combination is usually the right +shape: the `WHERE` clause fixes what the view is about, and the `WHEN` policy +fixes how long it keeps what it has. + +```questdb-sql title="A filter for the subject, a policy for the horizon" +CREATE MATERIALIZED VIEW trades_btc_recent AS ( + SELECT * FROM trades WHERE symbol = 'BTC' +) EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 1h; +``` + +### When a deterministic cutoff still belongs in a policy + +One case pulls a deterministic predicate back into `EXPIRE ROWS`: a fixed +threshold you expect to advance by hand. There is no `ALTER MATERIALIZED VIEW ... AS `, so changing a `WHERE` clause means dropping the view and re-creating it, which re-materializes it from the base. Changing a policy is a metadata operation: @@ -120,26 +153,10 @@ base that no longer holds everything the view held. A view whose retention horizon is longer than its base table's cannot afford to be rebuilt, so its cutoff belongs in a policy. -Two things only `EXPIRE ROWS` can express at all, whichever way you lean: - -- **Rules that move with the clock.** A view's defining query rejects - non-deterministic functions, so `WHERE ts > dateadd('d', -7, now())` is not - accepted. `EXPIRE ROWS WHEN ts < dateadd('d', -7, now())` is the supported way - to write a rolling window. -- **Rules that compare rows against each other.** `KEEP LATEST`, - `KEEP N HIGHEST/LOWEST` and window predicates have no `WHERE` equivalent — a - `LATEST ON` or a window function in the defining query makes the view - non-passthrough. - -In practice the two compose, and on a passthrough view that is usually the right -shape: the `WHERE` clause fixes what the view is about, and the policy fixes how -long it keeps what it has. - -```questdb-sql title="A filter for the subject, a policy for the horizon" -CREATE MATERIALIZED VIEW trades_btc_recent AS ( - SELECT * FROM trades WHERE symbol = 'BTC' -) EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 1h; -``` +A rule that compares rows against each other has no `WHERE` equivalent either: +`KEEP LATEST`, `KEEP N HIGHEST/LOWEST` and window predicates cannot be expressed +in the defining query, because a `LATEST ON` or a window function there makes +the view non-passthrough. ## The modes @@ -223,8 +240,12 @@ this small dataset that is effectively instant. ### Per-row predicate: `WHEN` -A per-row predicate expires a row when it evaluates `TRUE`. Here, expire small -trades (`amount < 1.5`): +A per-row predicate expires a row when it evaluates `TRUE`. The example below +uses a deterministic predicate — expire small trades (`amount < 1.5`) — +because it makes the keep-set easy to read off the sample data. In production +that rule belongs in the view's `WHERE` clause (`WHERE amount >= 1.5`), which +keeps those rows out of the view entirely; the rolling window further down is +the case `WHEN` exists for. ```questdb-sql title="Expire rows where amount < 1.5" CREATE MATERIALIZED VIEW trades_sized AS ( @@ -244,8 +265,10 @@ The two `amount = 1.0` rows are expired. `amount = 1.5` is kept (`1.5 < 1.5` is `FALSE`), and any `NULL` amount would be kept too (the comparison is `UNKNOWN`, not `TRUE` — see [NULLs](#nulls)). -A predicate on the designated timestamp gives a **rolling retention window**, -re-evaluated on every read so the visible set rolls forward with the clock: +A predicate on the designated timestamp gives a **rolling retention window** — +the main use for `WHEN`. It is re-evaluated on every read, so the visible set +rolls forward with the clock, and no `WHERE` clause can express it because the +defining query rejects `now()`: ```questdb-sql title="Keep the last 1 day" CREATE MATERIALIZED VIEW trades_recent AS ( @@ -578,7 +601,7 @@ rather than breaking subsequent reads. - [Materialized views](/docs/concepts/materialized-views/) — the view type `EXPIRE ROWS` runs on - [Passthrough views](/docs/concepts/materialized-views/#passthrough-views) — - the non-aggregating view shape `EXPIRE ROWS` is designed for + the non-aggregating views `EXPIRE ROWS` applies to - [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/) — full create syntax, including the `EXPIRE ROWS` clause - [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index f8f3e7eea8..973e2401a9 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -185,9 +185,10 @@ summary of it: - **A narrowed replica** — one symbol, one tenant, one region, or a handful of columns out of a wide table, kept current automatically and queried without the base table's scan cost. -- **A row-level retention target** — a passthrough view is the shape - [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) is built for. Its rows - *are* base rows, so reclaiming one is permanent. +- **A row-level retention target** — attach an + [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy to keep only + some of the view's rows: the latest per key, the top-N per group, or rows + inside a rolling time window. The base table is left alone. Passthrough views refresh incrementally like any other materialized view, and accept the same `REFRESH IMMEDIATE` (the default), `REFRESH MANUAL` and diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index c02337bdb9..66eec973a1 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -55,11 +55,17 @@ reclamation follows in the background. See ## Examples -```questdb-sql title="Per-row predicate, with a tighter cleanup cadence" +```questdb-sql title="Rolling 7-day window, with a tighter cleanup cadence" ALTER MATERIALIZED VIEW trades_mirror - SET EXPIRE ROWS WHEN amount < 1.5 CLEANUP EVERY 30m; + SET EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 30m; ``` +A `WHEN` predicate is the right tool for a cutoff that moves with the clock like +this one. A deterministic predicate such as `amount < 1.5` is accepted too, but +it selects the same rows more cheaply as a `WHERE` clause in the view's defining +query — see +[`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/deep-dive/expire-rows/#where-filter-or-expire-rows). + ```questdb-sql title="Keep the latest row per symbol" ALTER MATERIALIZED VIEW trades_mirror SET EXPIRE ROWS KEEP LATEST PARTITION BY symbol; diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 135e25e1fd..68f9f2e624 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -304,6 +304,13 @@ CREATE MATERIALIZED VIEW trades_latest AS ( ) EXPIRE ROWS KEEP LATEST PARTITION BY symbol; ``` +A `WHEN` predicate is for rules that move with **wall-clock time**, such as a +rolling `ts < dateadd('d', -7, now())` window — the defining query cannot +express those, because it rejects non-deterministic functions. A predicate that +depends only on the row's own values belongs in the query's `WHERE` clause +instead, which keeps those rows out of the view entirely; see +[`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/deep-dive/expire-rows/#where-filter-or-expire-rows). + The clause goes after the query (and after `PARTITION BY` if present): ``` From c83a79ccbc7f58d7558f582c5df298d76a131d33 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Tue, 25 Aug 2026 10:39:03 +0200 Subject: [PATCH 07/25] Add use-case examples for passthrough views The passthrough views section gave two reasons to reach for one, a narrowed replica and a target for EXPIRE ROWS, as bullet points with no SQL under them. A reader who wanted either had nothing to copy. Add a "Use cases" section with three worked examples, each opening with a one-sentence scenario and naming the domain it comes from, so the section covers sensor telemetry as well as capital markets. The examples are a current-reading-per-sensor view using KEEP LATEST, a live options chain using a WHEN predicate, and the largest trades per symbol using KEEP 10 HIGHEST. Every statement was run against a live server before being written down. Two behaviors are recorded here that were not documented anywhere else. First, the cleanup job reclaims disk only when the cutoff names the view's designated timestamp. The options example compares a contract's expiry column against now(), and materialized_views() reports options_live expiry < now() FILTER_ONLY options_ctl ts < dateadd('d', -7, now()) FILTER_AND_RECLAIM for two views over the same base table, so the column is what decides it, not the shape of the predicate. Expired contracts stop showing up in queries straight away either way; only the disk behavior differs. Second, when two rows tie at the Nth place under KEEP N, the newer row survives. An order book example was written first and then dropped. KEEP N HIGHEST ranks over everything the view holds rather than over a recent window, so a view of the ten highest bids per symbol fills up with the highest prices ever seen. With one bid from the previous day at 200.00 and a live book around 100, the stale bid outranked every current one and the live bids were expired out of the view. An order book is state that gets superseded, and a materialized view can only add rows, so no policy can remove an order that was later cancelled. The trades example replaces it because a trade is never retracted, which makes ranking over all history the intended meaning rather than a defect. The closing paragraph of that example now points at KEEP LATEST for values that do get superseded, to save the next reader the same detour. Also drop the list of policy shapes from the second bullet, since the examples below it now demonstrate each one. Co-Authored-By: Claude Opus 5 --- documentation/concepts/materialized-views.md | 97 +++++++++++++++++++- 1 file changed, 95 insertions(+), 2 deletions(-) diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index 973e2401a9..637f3cbe5b 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -187,14 +187,107 @@ summary of it: the base table's scan cost. - **A row-level retention target** — attach an [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy to keep only - some of the view's rows: the latest per key, the top-N per group, or rows - inside a rolling time window. The base table is left alone. + some of the view's rows. The base table is left alone. Passthrough views refresh incrementally like any other materialized view, and accept the same `REFRESH IMMEDIATE` (the default), `REFRESH MANUAL` and `REFRESH EVERY` strategies. `REFRESH PERIOD` is rejected: there are no buckets for a period to align to. +### Use cases + +On its own, a passthrough view is a maintained copy of the base table. Add an +[`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy and it becomes a +maintained *subset*: you describe which rows are worth keeping, and QuestDB +keeps that set current as new data arrives. The three examples below are drawn +from sensor telemetry and from capital markets, and each keeps a different kind +of subset. + +#### IoT: The current reading from every sensor + +A building management platform records temperature and humidity from tens of +thousands of sensors, and its operations screen shows the newest reading from +each one. + +Sensors report at their own pace. Some send a reading every second, others go +quiet for days. Against the base table that screen runs +`LATEST ON ts PARTITION BY sensor_id`, which reads backwards until it has found +a row for even the quietest sensor, and over a long history that is most of the +table. + +A view that keeps only the newest row per sensor answers the same question from +a handful of rows: + +```questdb-sql title="Latest reading per sensor" +CREATE MATERIALIZED VIEW sensor_current AS ( + SELECT * FROM sensor_readings +) EXPIRE ROWS KEEP LATEST PARTITION BY sensor_id; +``` + +`sensor_current` holds one row per `sensor_id` and moves forward on its own as +readings arrive. Superseded rows stop appearing in queries but stay on disk, so +the view keeps growing at the same rate as the base table. Give it a +[TTL](/docs/concepts/ttl/) to cap its size. + +#### Finance: Options that have not expired yet + +A market maker quotes an options chain where contracts expire every Friday, and +the pricing screen must never show a contract that has already expired. + +That rule cannot live in the view's query. The query runs when rows are written +into the view, and it may not call `now()`, so there is no way to say "expiry is +still in the future" in a `WHERE` clause. An `EXPIRE ROWS WHEN` predicate is +evaluated on every read, which is what this case needs: + +```questdb-sql title="Options that have not expired yet" +CREATE MATERIALIZED VIEW options_live AS ( + SELECT * FROM options_quotes +) EXPIRE ROWS WHEN expiry < now(); +``` + +Contracts leave `options_live` as their expiry passes. There is no job to +schedule and nothing to re-create. + +One caveat, about disk rather than about results. QuestDB deletes expired rows +only when it can tell that a row, once expired, can never qualify again. It can +tell that for a cutoff on the view's designated timestamp, which only moves +forward. `expiry` is a different column, so QuestDB takes the safe route: +expired contracts stop appearing in queries straight away, but their rows stay +on disk. Add a [TTL](/docs/concepts/ttl/) if you want that space back, and see +[when expired rows are deleted from disk](/docs/concepts/deep-dive/expire-rows/#monotonicity-and-cleanup-safety). + +#### Finance: The largest trades per symbol + +A surveillance desk watches for block trades and wants the ten biggest prints +for every instrument on hand at all times. + +A trade is a fact that does not change once it has happened, so "the ten biggest +so far" is a set that only gets refined as larger trades arrive. That is what a +top-N policy maintains: + +```questdb-sql title="Ten largest trades per symbol" +CREATE MATERIALIZED VIEW trades_largest AS ( + SELECT * FROM trades +) EXPIRE ROWS KEEP 10 HIGHEST amount PARTITION BY symbol; +``` + +`trades_largest` holds ten rows per symbol however far the base table grows, and +the desk reads it directly instead of ranking the base table on every query. +When an eleventh large trade arrives, the smallest of the ten drops out. Trades +tied on `amount` at the tenth place are separated by the designated timestamp, +with the newer one staying. + +Ten rows per symbol are visible, but the view still stores every base row it has +taken in, because a top-N policy never frees disk. A [TTL](/docs/concepts/ttl/) +is what bounds that, at the price of changing the question the view answers from +"the biggest so far" to "the biggest still retained". See +[combining with TTL](/docs/concepts/deep-dive/expire-rows/#combining-with-ttl). + +The ranking covers everything the view holds rather than a recent window, so +`KEEP N` fits records that stay true once written. For a value that gets +superseded later, such as a resting order that is then cancelled, `KEEP LATEST` +is the mode that tracks the current version. + ### What a passthrough view inherits A passthrough view takes its shape from the base table instead of from a From c578889f7c89bf6f7991cbd555da78b5b407fd7c Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Tue, 25 Aug 2026 11:01:33 +0200 Subject: [PATCH 08/25] Document the NULL EXPIRE ROWS threshold rejection CREATE and ALTER now refuse a scalar WHEN threshold that is constant at definition time and evaluates to NULL, since such a policy expires nothing. Add a Semantics section covering the explicit CAST(NULL AS TIMESTAMP) and the arithmetic spellings that overflow onto the reserved NULL value, and note that a clock-based threshold is evaluated per read and so cannot be checked in advance. Add the error to the SET EXPIRE error table, and a pointer to the new section from the CREATE MATERIALIZED VIEW reference. Also steer the rolling-window example toward dateadd. It and now() - 86400000000 retain the same rows, but only the dateadd form reduces to a timestamp range the scan can use to skip partitions; the arithmetic form is evaluated row by row across the whole view on every read. Both reclaim disk, so the difference is read cost alone. Co-Authored-By: Claude Opus 5 --- .../concepts/deep-dive/expire-rows.md | 42 +++++++++++++++++++ .../query/sql/alter-mat-view-set-expire.md | 1 + documentation/query/sql/create-mat-view.md | 6 +++ 3 files changed, 49 insertions(+) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md index f2c3a1ab0d..cffc08b87e 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -279,6 +279,13 @@ CREATE MATERIALIZED VIEW trades_recent AS ( (With the 2024 sample timestamps above, every row is already older than a day and would be hidden; use recent data to see rows retained.) +Write the offset with `dateadd`, as above, rather than as clock arithmetic. +`WHEN ts < dateadd('d', -1, now())` and `WHEN ts < now() - 86400000000` retain +the same rows, but only the `dateadd` form reduces to a timestamp range that the +scan can use to skip partitions. The arithmetic form is evaluated row by row +across the whole view on every read. Both forms reclaim disk — the cleanup job +proves either monotonic — so the difference is read cost alone. + ### Keep latest per key: `KEEP LATEST` Keep only the most recent row per key — turning the passthrough view into a @@ -440,6 +447,41 @@ Therefore: `KEEP HIGHEST/LOWEST` (no `N`) when every `NULL` must be kept regardless of type. +### A `NULL` threshold is rejected + +A `WHEN` threshold that evaluates to a constant `NULL` expires nothing — +`ts < NULL` is never `TRUE` — so the policy would be inert. QuestDB refuses it at +`CREATE` and `ALTER` time rather than storing a view that silently never +reclaims: + +```questdb-sql title="Rejected: the threshold is NULL" +CREATE MATERIALIZED VIEW trades_recent AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN ts < CAST(NULL AS TIMESTAMP); +-- invalid EXPIRE ROWS predicate: the threshold is NULL, so no row can ever expire +``` + +The check matters most where the `NULL` is not written down. QuestDB stores a +`NULL` `TIMESTAMP`, `LONG` or `INT` as a reserved value at the bottom of the +type's range, and integer arithmetic wraps silently when it overflows, so an +arithmetic threshold can land on that value: + +```questdb-sql title="Also rejected: arithmetic that overflows onto NULL" +-- LONG overflow +CREATE MATERIALIZED VIEW trades_recent AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN ts < 4611686018427387904 * 2; + +-- INT overflow, reached three orders of magnitude sooner +CREATE MATERIALIZED VIEW trades_recent AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN ts < 2147483647 + 1; +``` + +Only thresholds that are constant at definition time are checked this way. One +built from a clock, such as `ts < now() - 3600000000`, is evaluated per read and +cannot be checked in advance. + ### Ties and determinism `KEEP HIGHEST/LOWEST` keeps **all** rows tied at the max/min — deterministic by diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 66eec973a1..503f7754e7 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -113,6 +113,7 @@ GRANT ALTER MATERIALIZED VIEW ON trades_mirror TO user1; | `invalid EXPIRE ROWS KEEP LATEST column: ...` | A `PARTITION BY` key column does not exist | | `EXPIRE ROWS KEEP / window retention cannot be used on a view with a column named '__qdb_re_keep'` | The view exposes a column named like the reserved keep column | | `invalid EXPIRE ROWS predicate: ...` | The predicate does not parse, bind, or type-check against the view's columns | +| `invalid EXPIRE ROWS predicate: the threshold is NULL, so no row can ever expire` | A `WHEN` threshold that is constant at definition time evaluates to `NULL`, e.g. `CAST(NULL AS TIMESTAMP)` or arithmetic that overflows | | `permission denied` | Missing `ALTER MATERIALIZED VIEW` permission (Enterprise) | ## See also diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 68f9f2e624..a5bc445b7a 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -321,6 +321,12 @@ EXPIRE ROWS [ CLEANUP EVERY duration ] ``` +A `WHEN` threshold that is constant at definition time and evaluates to `NULL` +is rejected, since it would expire nothing. That covers the explicit +`ts < CAST(NULL AS TIMESTAMP)` and arithmetic that overflows onto the reserved +`NULL` value, such as `ts < 2147483647 + 1` — see +[A `NULL` threshold is rejected](/docs/concepts/deep-dive/expire-rows/#a-null-threshold-is-rejected). + Expired rows are hidden from queries immediately in every mode. They are reclaimed on disk in the background (`CLEANUP EVERY`, default `1h`) under a monotonic `WHEN` predicate; `KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N` and From 8795cbee2e562dcdb6bc6e6b3a459cc5fc9c1509 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Thu, 27 Aug 2026 10:27:28 +0200 Subject: [PATCH 09/25] docs: correct expire rows behavior --- .../concepts/deep-dive/expire-rows.md | 48 +++++++++++-------- .../query/sql/alter-mat-view-set-expire.md | 5 ++ 2 files changed, 34 insertions(+), 19 deletions(-) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md index cffc08b87e..b95c8539af 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -174,6 +174,12 @@ only when the rule selects it for removal. `KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a window predicate, so the window `WHEN` is the general escape hatch. +The bare `KEEP HIGHEST/LOWEST` form accepts `BYTE`, `SHORT`, `INT`, `LONG`, +`FLOAT`, `DOUBLE`, `DATE`, `TIMESTAMP` and `DECIMAL` columns. The top-N form has +a broader type surface: `KEEP N HIGHEST/LOWEST` ranks with `ORDER BY`, so it +accepts any orderable column type. For example, use `KEEP 1 HIGHEST symbol`, not +`KEEP HIGHEST symbol`, to rank a `SYMBOL` column. + The **Frees disk** column is the difference between hiding a row and deleting it. Every mode hides expired rows from every read, immediately. Only a monotonic `WHEN` predicate also has those rows deleted from disk by the cleanup job: a @@ -252,14 +258,14 @@ CREATE MATERIALIZED VIEW trades_sized AS ( SELECT * FROM trades ) EXPIRE ROWS WHEN amount < 1.5; -SELECT * FROM trades_sized; +SELECT * FROM trades_sized ORDER BY ts; ``` | symbol | price | amount | ts | | ------ | ----- | ------ | --------------------------- | +| ETH | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | | BTC | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | | BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | -| ETH | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | The two `amount = 1.0` rows are expired. `amount = 1.5` is kept (`1.5 < 1.5` is `FALSE`), and any `NULL` amount would be kept too (the comparison is `UNKNOWN`, @@ -279,12 +285,8 @@ CREATE MATERIALIZED VIEW trades_recent AS ( (With the 2024 sample timestamps above, every row is already older than a day and would be hidden; use recent data to see rows retained.) -Write the offset with `dateadd`, as above, rather than as clock arithmetic. `WHEN ts < dateadd('d', -1, now())` and `WHEN ts < now() - 86400000000` retain -the same rows, but only the `dateadd` form reduces to a timestamp range that the -scan can use to skip partitions. The arithmetic form is evaluated row by row -across the whole view on every read. Both forms reclaim disk — the cleanup job -proves either monotonic — so the difference is read cost alone. +the same rows and both reclaim disk — the cleanup job proves either monotonic. ### Keep latest per key: `KEEP LATEST` @@ -296,13 +298,13 @@ CREATE MATERIALIZED VIEW trades_latest AS ( SELECT * FROM trades ) EXPIRE ROWS KEEP LATEST PARTITION BY symbol; -SELECT * FROM trades_latest; +SELECT * FROM trades_latest ORDER BY ts; ``` | symbol | price | amount | ts | | ------ | ----- | ------ | --------------------------- | -| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | | ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | +| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | One row per symbol — the latest by the designated timestamp `ts`. As new trades arrive, the kept row advances automatically. `PARTITION BY` may list multiple key @@ -397,11 +399,19 @@ time-based predicate) reappears on the next read. ### Physical cleanup (best-effort) -A background job reclaims disk for non-active partitions: a fully-expired -partition is removed, and a partially-expired one is compacted down to its -survivors. It runs at the `CLEANUP EVERY` cadence (default `1h`) and is -**best-effort** — the read filter is authoritative, so deferred or skipped -reclamation only affects disk usage, never query results. +A background job reclaims disk for non-active partitions. A fully-expired +partition is removed. Under a rolling clock-based predicate, a partially-expired +partition is compacted down to its survivors only when the expired-row fraction +reaches `cairo.mat.view.row.expiry.cleanup.min.expired.fraction`, which defaults +to `0.5`. This avoids repeatedly rewriting a boundary partition as the cutoff +moves through it. Set the property to `0` to compact on the first expired row, +or to `1` to disable partial-partition compaction; fully-expired partitions are +still removed. The threshold does not delay a fixed, deterministic predicate, +whose expired-row verdicts cannot change with time. + +The job runs at the `CLEANUP EVERY` cadence (default `1h`) and is **best-effort** +— the read filter is authoritative, so deferred or skipped reclamation only +affects disk usage, never query results. The job runs only under a monotonic `WHEN` predicate. It skips `KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N` and window policies entirely: a later refresh @@ -413,11 +423,11 @@ On QuestDB Enterprise, cleanup runs on the **primary only**, but the reclamation still replicates: the compaction commits are ordinary WAL transactions, so replicas reclaim the identical rows by applying them. A read-only replica neither runs the job nor needs to. Disable the job with -`cairo.row.expiry.cleanup.enabled=false` in `server.conf` (reads stay filtered; -only reclamation stops — the setting does not disable `EXPIRE ROWS` itself); the -setting is read at startup, so changing it requires a restart. A failing sweep -retries after one second, doubling the per-view retry gap up to a 10-minute -cap. +`cairo.mat.view.row.expiry.cleanup.enabled=false` in `server.conf` (reads stay +filtered; only reclamation stops — the setting does not disable `EXPIRE ROWS` +itself). Cleanup settings are read at startup, so changing this property or the +minimum expired fraction requires a restart. A failing sweep retries after one +second, doubling the per-view retry gap up to a 10-minute cap. To observe reclamation, compare the physical row count per partition before and after a sweep: diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 503f7754e7..2a775ce2a8 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -34,6 +34,10 @@ ALTER MATERIALIZED VIEW viewName DROP EXPIRE | `KEEP [N] HIGHEST\|LOWEST col` | Keep the rows at the max/min of `col` per group, or the top `N` | | `CLEANUP EVERY` | Background reclamation cadence (e.g. `30m`, `1h`). Defaults to `1h` if omitted | +Without `N`, the keep column must be `BYTE`, `SHORT`, `INT`, `LONG`, `FLOAT`, +`DOUBLE`, `DATE`, `TIMESTAMP` or `DECIMAL`. `KEEP N HIGHEST/LOWEST` ranks with +`ORDER BY` and accepts any orderable column type. + For the full description of each mode and its semantics, see the [Expiring rows](/docs/concepts/deep-dive/expire-rows/) concept page. @@ -111,6 +115,7 @@ GRANT ALTER MATERIALIZED VIEW ON trades_mirror TO user1; | `cannot set an EXPIRE ROWS policy on '...': it is the base of N materialized view(s), which would copy expired rows on refresh` | Other materialized views derive from this view | | `EXPIRE ROWS KEEP LATEST ON must name the designated timestamp ...` | `ON` names a column other than the designated timestamp | | `invalid EXPIRE ROWS KEEP LATEST column: ...` | A `PARTITION BY` key column does not exist | +| `EXPIRE ROWS KEEP HIGHEST/LOWEST requires a BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, DATE, TIMESTAMP or DECIMAL column, but '' is ; use KEEP HIGHEST/LOWEST to rank an orderable column of any type` | The bare `KEEP HIGHEST/LOWEST` form was given an unsupported column type; use a supported numeric/date type or the top-N form | | `EXPIRE ROWS KEEP / window retention cannot be used on a view with a column named '__qdb_re_keep'` | The view exposes a column named like the reserved keep column | | `invalid EXPIRE ROWS predicate: ...` | The predicate does not parse, bind, or type-check against the view's columns | | `invalid EXPIRE ROWS predicate: the threshold is NULL, so no row can ever expire` | A `WHEN` threshold that is constant at definition time evaluates to `NULL`, e.g. `CAST(NULL AS TIMESTAMP)` or arithmetic that overflows | From c66ce5e088626b5d785e80999ef7be357baf677f Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Thu, 27 Aug 2026 10:56:35 +0200 Subject: [PATCH 10/25] docs: address EXPIRE ROWS review feedback --- .../concepts/{deep-dive => }/expire-rows.md | 265 +++++++++--------- documentation/concepts/materialized-views.md | 12 +- documentation/concepts/ttl.md | 2 + documentation/query/functions/meta.md | 43 +-- .../query/sql/alter-mat-view-set-expire.md | 19 +- documentation/query/sql/create-mat-view.md | 16 +- documentation/sidebars.js | 2 +- 7 files changed, 179 insertions(+), 180 deletions(-) rename documentation/concepts/{deep-dive => }/expire-rows.md (94%) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/expire-rows.md similarity index 94% rename from documentation/concepts/deep-dive/expire-rows.md rename to documentation/concepts/expire-rows.md index b95c8539af..1b13e0b159 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/expire-rows.md @@ -1,6 +1,6 @@ --- title: Expiring rows (EXPIRE ROWS) -sidebar_label: Expiring rows +sidebar_label: EXPIRE ROWS description: EXPIRE ROWS is a row-level retention policy for passthrough materialized views. Keep the latest row per key, the top-N per group, or rows matching a @@ -22,14 +22,61 @@ on-disk storage is reclaimed afterwards by a background job under a monotonic [The modes](#the-modes) and [Monotonicity and cleanup safety](#monotonicity-and-cleanup-safety). +## The modes + +Every mode keeps a defined set of rows and expires the rest. A row is expired +only when the rule selects it for removal. + +| Mode | What it keeps | Syntax | Frees disk | +| --------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | --------------------- | +| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when monotonic | +| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | No — read filter only | +| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | +| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | +| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No — read filter only | + +`KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a +window predicate, so the window `WHEN` is the general escape hatch. + +The bare `KEEP HIGHEST/LOWEST` form accepts `BYTE`, `SHORT`, `INT`, `LONG`, +`FLOAT`, `DOUBLE`, `DATE`, `TIMESTAMP` and `DECIMAL` columns. The top-N form has +a broader type surface: `KEEP N HIGHEST/LOWEST` ranks with `ORDER BY`, so it +accepts any orderable column type. For example, use `KEEP 1 HIGHEST symbol`, not +`KEEP HIGHEST symbol`, to rank a `SYMBOL` column. + +For how read filtering and physical reclamation differ between modes, see +[How it works](#how-it-works). You can inspect the behavior selected for a view +through `materialized_views().expire_enforcement`; see +[Inspecting a policy](#inspecting-a-policy). + +The clause is attached to a passthrough `CREATE MATERIALIZED VIEW` (after the +query, and after `PARTITION BY` if present), or set later with +[`ALTER MATERIALIZED VIEW ... SET EXPIRE ROWS`](/docs/query/sql/alter-mat-view-set-expire/): + +``` +EXPIRE ROWS + { WHEN predicate + | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] + | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } + [ CLEANUP EVERY duration ] +``` + +| Element | Meaning | +| ------------------ | ---------------------------------------------------------------------------------------- | +| `predicate` | Any boolean expression over the view's columns. A row expires when it evaluates `TRUE`. | +| `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp. | +| `ON timestampCol` | Optional; if given it must name the view's designated timestamp. | +| `HIGHEST\|LOWEST` | Keep rows at the max / min of `col` per group (`N` omitted), or the top `N` per group. | +| `CLEANUP EVERY` | How often the background reclamation job runs for this view: `` with unit `s`/`m`/`h`/`d`/`w`. Defaults to `1h` if omitted. | + :::note `EXPIRE ROWS` is **materialized-view-only**: `CREATE TABLE ... EXPIRE ROWS` is rejected. It is designed for a **passthrough** (non-aggregating) view — `SELECT * FROM base` with no `SAMPLE BY` / `GROUP BY` — where the view mirrors base rows 1:1 and reclamation is permanent. An aggregating view is **accepted with a -logged advisory**: reads stay filtered, but a later refresh can regenerate -reclaimed rows from base rows that still exist (see +logged advisory** because a later refresh can regenerate reclaimed rows from +base rows that still exist (see [Requirements](#requirements)). For base-table retention use [TTL](/docs/concepts/ttl/) or, on Enterprise, [storage policies](/docs/concepts/storage-policy/). @@ -76,141 +123,15 @@ is effectively a continuously-maintained replica. `EXPIRE ROWS` prunes that replica down to the rows you want to keep — without touching the base table. A **non-passthrough (aggregating) view is accepted with a logged advisory** -rather than rejected: the read filter still hides expired rows, but physical -reclamation only sticks when base-table retention is aligned with the expiry -horizon — a later incremental or full refresh can regenerate a reclaimed row -from base rows that still exist. +rather than rejected: physical reclamation only sticks when base-table retention +is aligned with the expiry horizon — a later incremental or full refresh can +regenerate a reclaimed row from base rows that still exist. A policied view must also stand alone: `CREATE MATERIALIZED VIEW` rejects a defining query that reads a policied view (as its base or in a join), and `ALTER ... SET EXPIRE` is rejected on a view that other materialized views derive from — otherwise those views would copy expired rows on refresh. -## `WHERE` filter or `EXPIRE ROWS`? - -A passthrough view can exclude rows in two places: a `WHERE` clause in its -defining query, or an `EXPIRE ROWS WHEN` predicate. The dividing line is the -clock. - -**Use `EXPIRE ROWS WHEN` for rules that move with wall-clock time.** A rolling -window cannot be written as a `WHERE` clause at all: a view's defining query -rejects non-deterministic functions, so `WHERE ts > dateadd('d', -7, now())` is -not accepted. `EXPIRE ROWS WHEN ts < dateadd('d', -7, now())` is the supported -way to say "keep the last 7 days" — the read filter re-evaluates `now()` on -every read, so the window rolls forward on its own and the cleanup job reclaims -the disk behind it. This is what the `WHEN` form is for. - -**Put a deterministic predicate in the `WHERE` clause.** A predicate that -depends only on the row's own values — `symbol = 'BTC'`, `amount >= 1.5` — -describes the same surviving rows either way, so the two are near-equivalent in -what the view contains, and `WHERE` is the cheaper of the two at every stage. A -row the `WHERE` clause excludes is never copied into the view: - -| | `WHERE` in the query | `EXPIRE ROWS WHEN` | -| --- | --- | --- | -| Storage | Row is never written | Row is written, and occupies disk until a sweep reclaims it | -| Read cost | None | The keep-set filter is applied on every read of the view | -| Write cost | None | The cleanup job rewrites partitions on the `CLEANUP EVERY` cadence | -| After a full refresh | Still excluded | Re-materialized from the base, then hidden and swept again | -| Can be another view's base | Yes | No — a policied view is rejected as a base | - -That last row is a hard constraint rather than a preference. If any other view -will read this one, the policy is not available and the predicate has to go in -the `WHERE` clause. - -The two forms are not exact negations of each other on `NULL`s: `WHERE` keeps a -row only when the predicate is `TRUE`, while `EXPIRE ROWS WHEN` expires a row -only when it is `TRUE`. A `NULL` amount is dropped by `WHERE amount >= 1.5` and -kept by `EXPIRE ROWS WHEN amount < 1.5` — see [NULLs](#nulls). - -The two compose, and on a passthrough view that combination is usually the right -shape: the `WHERE` clause fixes what the view is about, and the `WHEN` policy -fixes how long it keeps what it has. - -```questdb-sql title="A filter for the subject, a policy for the horizon" -CREATE MATERIALIZED VIEW trades_btc_recent AS ( - SELECT * FROM trades WHERE symbol = 'BTC' -) EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 1h; -``` - -### When a deterministic cutoff still belongs in a policy - -One case pulls a deterministic predicate back into `EXPIRE ROWS`: a fixed -threshold you expect to advance by hand. There is no -`ALTER MATERIALIZED VIEW ... AS `, so changing a `WHERE` clause means -dropping the view and re-creating it, which re-materializes it from the base. -Changing a policy is a metadata operation: - -```questdb-sql title="Retuning a retention horizon without a rebuild" -ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-06-01T00:00:00.000000Z'; -ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-07-01T00:00:00.000000Z'; -ALTER MATERIALIZED VIEW trades_recent DROP EXPIRE; -``` - -The rebuild a `WHERE` change forces is not only slow, it can lose data: if the -base table has its own [TTL](/docs/concepts/ttl/), re-creating the view reads a -base that no longer holds everything the view held. A view whose retention -horizon is longer than its base table's cannot afford to be rebuilt, so its -cutoff belongs in a policy. - -A rule that compares rows against each other has no `WHERE` equivalent either: -`KEEP LATEST`, `KEEP N HIGHEST/LOWEST` and window predicates cannot be expressed -in the defining query, because a `LATEST ON` or a window function there makes -the view non-passthrough. - -## The modes - -Every mode keeps a defined set of rows and expires the rest. A row is expired -only when the rule selects it for removal. - -| Mode | What it keeps | Syntax | Frees disk | -| --------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | --------------------- | -| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when monotonic | -| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | No — read filter only | -| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | -| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | -| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No — read filter only | - -`KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a -window predicate, so the window `WHEN` is the general escape hatch. - -The bare `KEEP HIGHEST/LOWEST` form accepts `BYTE`, `SHORT`, `INT`, `LONG`, -`FLOAT`, `DOUBLE`, `DATE`, `TIMESTAMP` and `DECIMAL` columns. The top-N form has -a broader type surface: `KEEP N HIGHEST/LOWEST` ranks with `ORDER BY`, so it -accepts any orderable column type. For example, use `KEEP 1 HIGHEST symbol`, not -`KEEP HIGHEST symbol`, to rank a `SYMBOL` column. - -The **Frees disk** column is the difference between hiding a row and deleting -it. Every mode hides expired rows from every read, immediately. Only a monotonic -`WHEN` predicate also has those rows deleted from disk by the cleanup job: a -mode whose keep-set depends on the other rows in the view keeps its expired rows -on disk until a full refresh rebuilds the view. A `KEEP LATEST` view therefore -holds a full copy of its base table unless the view's own -[TTL](/docs/concepts/ttl/) bounds it — see -[Combining with TTL](#combining-with-ttl). -`materialized_views().expire_enforcement` reports which of the two a given view -gets; see [Inspecting a policy](#inspecting-a-policy). - -The clause is attached to a passthrough `CREATE MATERIALIZED VIEW` (after the -query, and after `PARTITION BY` if present), or set later with -[`ALTER MATERIALIZED VIEW ... SET EXPIRE ROWS`](/docs/query/sql/alter-mat-view-set-expire/): - -``` -EXPIRE ROWS - { WHEN predicate - | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] - | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } - [ CLEANUP EVERY duration ] -``` - -| Element | Meaning | -| ------------------ | ---------------------------------------------------------------------------------------- | -| `predicate` | Any boolean expression over the view's columns. A row expires when it evaluates `TRUE`. | -| `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp. | -| `ON timestampCol` | Optional; if given it must name the view's designated timestamp. | -| `HIGHEST\|LOWEST` | Keep rows at the max / min of `col` per group (`N` omitted), or the top `N` per group. | -| `CLEANUP EVERY` | How often the background reclamation job runs for this view: `` with unit `s`/`m`/`h`/`d`/`w`. Defaults to `1h` if omitted. | - ## Worked examples The following walks through every mode on a small fixed dataset so you can see @@ -374,6 +295,78 @@ express richer rules, for example keeping rows within 5% of the peak (`WHEN price < 0.95 * max(price) OVER (PARTITION BY symbol)`) or a ranked window (`WHEN row_number() OVER (PARTITION BY symbol ORDER BY ts DESC) > 100`). +## `WHERE` filter or `EXPIRE ROWS`? + +A passthrough view can exclude rows in two places: a `WHERE` clause in its +defining query, or an `EXPIRE ROWS WHEN` predicate. The dividing line is the +clock. + +**Use `EXPIRE ROWS WHEN` for rules that move with wall-clock time.** A rolling +window cannot be written as a `WHERE` clause at all: a view's defining query +rejects non-deterministic functions, so `WHERE ts > dateadd('d', -7, now())` is +not accepted. `EXPIRE ROWS WHEN ts < dateadd('d', -7, now())` is the supported +way to say "keep the last 7 days" — the read filter re-evaluates `now()` on +every read, so the window rolls forward on its own and the cleanup job reclaims +the disk behind it. This is what the `WHEN` form is for. + +**Put a deterministic predicate in the `WHERE` clause.** A predicate that +depends only on the row's own values — `symbol = 'BTC'`, `amount >= 1.5` — +describes the same surviving rows either way, so the two are near-equivalent in +what the view contains, and `WHERE` is the cheaper of the two at every stage. A +row the `WHERE` clause excludes is never copied into the view: + +| | `WHERE` in the query | `EXPIRE ROWS WHEN` | +| --- | --- | --- | +| Storage | Row is never written | Row is written, and occupies disk until a sweep reclaims it | +| Read cost | None | The keep-set filter is applied on every read of the view | +| Write cost | None | The cleanup job rewrites partitions on the `CLEANUP EVERY` cadence | +| After a full refresh | Still excluded | Re-materialized from the base, then hidden and swept again | +| Can be another view's base | Yes | No — a policied view is rejected as a base | + +That last row is a hard constraint rather than a preference. If any other view +will read this one, the policy is not available and the predicate has to go in +the `WHERE` clause. + +The two forms are not exact negations of each other on `NULL`s: `WHERE` keeps a +row only when the predicate is `TRUE`, while `EXPIRE ROWS WHEN` expires a row +only when it is `TRUE`. A `NULL` amount is dropped by `WHERE amount >= 1.5` and +kept by `EXPIRE ROWS WHEN amount < 1.5` — see [NULLs](#nulls). + +The two compose, and on a passthrough view that combination is usually the right +shape: the `WHERE` clause fixes what the view is about, and the `WHEN` policy +fixes how long it keeps what it has. + +```questdb-sql title="A filter for the subject, a policy for the horizon" +CREATE MATERIALIZED VIEW trades_btc_recent AS ( + SELECT * FROM trades WHERE symbol = 'BTC' +) EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 1h; +``` + +### When a deterministic cutoff still belongs in a policy + +One case pulls a deterministic predicate back into `EXPIRE ROWS`: a fixed +threshold you expect to advance by hand. There is no +`ALTER MATERIALIZED VIEW ... AS `, so changing a `WHERE` clause means +dropping the view and re-creating it, which re-materializes it from the base. +Changing a policy is a metadata operation: + +```questdb-sql title="Retuning a retention horizon without a rebuild" +ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-06-01T00:00:00.000000Z'; +ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-07-01T00:00:00.000000Z'; +ALTER MATERIALIZED VIEW trades_recent DROP EXPIRE; +``` + +The rebuild a `WHERE` change forces is not only slow, it can lose data: if the +base table has its own [TTL](/docs/concepts/ttl/), re-creating the view reads a +base that no longer holds everything the view held. A view whose retention +horizon is longer than its base table's cannot afford to be rebuilt, so its +cutoff belongs in a policy. + +A rule that compares rows against each other has no `WHERE` equivalent either: +`KEEP LATEST`, `KEEP N HIGHEST/LOWEST` and window predicates cannot be expressed +in the defining query, because a `LATEST ON` or a window function there makes +the view non-passthrough. + ## How it works `EXPIRE ROWS` has two cooperating parts: an authoritative read-time filter and a @@ -632,11 +625,9 @@ rather than breaking subsequent reads. - **`KEEP LATEST [ON ts]`.** The optional `ON ts` is accepted for familiarity but the view's designated timestamp is always used; naming a different column is rejected. -- **Only a monotonic `WHEN` predicate frees disk.** The relative and window - modes hide expired rows on every read but never delete them, and neither does - a `WHEN` predicate that cannot be proven monotonic — see - [monotonicity](#monotonicity-and-cleanup-safety) above. Check a view's verdict - with `materialized_views().expire_enforcement`. +- **Cleanup eligibility.** See + [monotonicity and cleanup safety](#monotonicity-and-cleanup-safety), and check + a view's verdict with `materialized_views().expire_enforcement`. - **Reserved column name.** The window/keep modes compute the keep-set through a synthetic boolean column named `__qdb_re_keep`; a policy is rejected on a view that exposes a column with that name. diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index 637f3cbe5b..aaa8fcc523 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -186,7 +186,7 @@ summary of it: columns out of a wide table, kept current automatically and queried without the base table's scan cost. - **A row-level retention target** — attach an - [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy to keep only + [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy to keep only some of the view's rows. The base table is left alone. Passthrough views refresh incrementally like any other materialized view, and @@ -197,7 +197,7 @@ for a period to align to. ### Use cases On its own, a passthrough view is a maintained copy of the base table. Add an -[`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy and it becomes a +[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy and it becomes a maintained *subset*: you describe which rows are worth keeping, and QuestDB keeps that set current as new data arrives. The three examples below are drawn from sensor telemetry and from capital markets, and each keeps a different kind @@ -254,7 +254,7 @@ tell that for a cutoff on the view's designated timestamp, which only moves forward. `expiry` is a different column, so QuestDB takes the safe route: expired contracts stop appearing in queries straight away, but their rows stay on disk. Add a [TTL](/docs/concepts/ttl/) if you want that space back, and see -[when expired rows are deleted from disk](/docs/concepts/deep-dive/expire-rows/#monotonicity-and-cleanup-safety). +[when expired rows are deleted from disk](/docs/concepts/expire-rows/#monotonicity-and-cleanup-safety). #### Finance: The largest trades per symbol @@ -281,7 +281,7 @@ Ten rows per symbol are visible, but the view still stores every base row it has taken in, because a top-N policy never frees disk. A [TTL](/docs/concepts/ttl/) is what bounds that, at the price of changing the question the view answers from "the biggest so far" to "the biggest still retained". See -[combining with TTL](/docs/concepts/deep-dive/expire-rows/#combining-with-ttl). +[combining with TTL](/docs/concepts/expire-rows/#combining-with-ttl). The ranking covers everything the view holds rather than a recent window, so `KEEP N` fits records that stay true once written. For a value that gets @@ -336,9 +336,9 @@ messages. To keep only some of a passthrough view's rows over time — the latest per key, the top-N per group, or rows matching a predicate — attach an -[`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy. That page also +[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. That page also covers -[when to put a predicate in the view's `WHERE` clause instead](/docs/concepts/deep-dive/expire-rows/#where-filter-or-expire-rows). +[when to put a predicate in the view's `WHERE` clause instead](/docs/concepts/expire-rows/#where-filter-or-expire-rows). ## Creating a materialized view diff --git a/documentation/concepts/ttl.md b/documentation/concepts/ttl.md index 9c1ee9f40f..4a9040b7f2 100644 --- a/documentation/concepts/ttl.md +++ b/documentation/concepts/ttl.md @@ -89,6 +89,8 @@ A view's TTL is independent of its base table's TTL. For full syntax, see [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/#ttl-time-to-live) and [ALTER MATERIALIZED VIEW SET TTL](/docs/query/sql/alter-mat-view-set-ttl/). +For row-level retention on passthrough materialized views, such as keeping the +latest row per key, see [`EXPIRE ROWS`](/docs/concepts/expire-rows/). ## How TTL works diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index 945fbfce75..b489cabf0d 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -288,7 +288,7 @@ Returns a `table` including the following information: merges two adjacent intervals instead of paying for a second commit. `0` means merging is disabled - `expire_clause` - the view's - [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy as written, or + [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy as written, or `NULL` when the view has no policy - `expire_cleanup_every` - how often the cleanup job runs for the policy, or `NULL` @@ -301,7 +301,7 @@ Returns a `table` including the following information: `materialized_views()` on its own returns every column listed above. The example below projects a readable subset over three views: an aggregating one with no retention policy, and two passthrough views with an -[`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy. +[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. ```questdb-sql title="List all materialized views" SELECT view_name, view_status, base_table_name, refresh_base_table_txn, @@ -790,20 +790,20 @@ Returns a `table` with the following columns: ### Basic table information -| Column | Type | Description | -| --------------------- | ------- | ------------------------------------------------------------------ | -| `id` | INT | Internal table ID | -| `table_name` | STRING | Table name | -| `designatedTimestamp` | STRING | Name of the designated timestamp column, or `null` | -| `partitionBy` | STRING | Partition strategy: `NONE`, `HOUR`, `DAY`, `WEEK`, `MONTH`, `YEAR` | -| `walEnabled` | BOOLEAN | Whether WAL (Write-Ahead Log) is enabled | -| `dedup` | BOOLEAN | Whether deduplication is enabled | -| `ttlValue` | INT | TTL (Time-To-Live) value | -| `ttlUnit` | STRING | TTL unit: `HOUR`, `DAY`, `WEEK`, `MONTH`, `YEAR` | -| `matView` | BOOLEAN | Whether this is a materialized view | -| `directoryName` | STRING | Directory name on disk (includes ` (->)` suffix for symlinks) | -| `maxUncommittedRows` | INT | Table's `maxUncommittedRows` setting | -| `o3MaxLag` | LONG | Table's `o3MaxLag` setting in microseconds | +| Column | Type | Description | +| ---------------------- | ------- | ------------------------------------------------------------------ | +| `id` | INT | Internal table ID | +| `table_name` | STRING | Table name | +| `designatedTimestamp` | STRING | Name of the designated timestamp column, or `null` | +| `partitionBy` | STRING | Partition strategy: `NONE`, `HOUR`, `DAY`, `WEEK`, `MONTH`, `YEAR` | +| `walEnabled` | BOOLEAN | Whether WAL (Write-Ahead Log) is enabled | +| `dedup` | BOOLEAN | Whether deduplication is enabled | +| `ttlValue` | INT | TTL (Time-To-Live) value | +| `ttlUnit` | STRING | TTL unit: `HOUR`, `DAY`, `WEEK`, `MONTH`, `YEAR` | +| `matView` | BOOLEAN | Whether this is a materialized view | +| `directoryName` | STRING | Directory name on disk (includes ` (->)` suffix for symlinks) | +| `maxUncommittedRows` | INT | Table's `maxUncommittedRows` setting | +| `o3MaxLag` | LONG | Table's `o3MaxLag` setting in microseconds | :::note @@ -873,6 +873,17 @@ These columns are populated on **replicas only** via replication download tracki On primary instances, these columns will be `0` or `false`. +### Row-expiry policy + +| Column | Type | Description | +| ---------------------- | ------ | -------------------------------------------------------------- | +| `expire_clause` | STRING | Materialized view's `EXPIRE ROWS` policy as written, or `null` | +| `expire_cleanup_every` | STRING | Cleanup cadence for the policy, or `null` | + +Unlike `materialized_views()`, `tables()` does not expose +`expire_enforcement`; use `materialized_views()` to inspect whether a policy is +`FILTER_AND_RECLAIM` or `FILTER_ONLY`. + ### Data precision and limitations These values are approximations, not precise real-time metrics: diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 2a775ce2a8..3303c43bd5 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -6,11 +6,10 @@ description: documentation. --- -Sets, replaces, or removes an [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) +Sets, replaces, or removes an [`EXPIRE ROWS`](/docs/concepts/expire-rows/) row-retention policy on a materialized view (designed for **passthrough** -views — see the concept page). Expired rows are hidden from queries immediately -in every mode, and reclaimed on disk in the background under a monotonic `WHEN` -predicate — the relative and window modes hide rows without freeing disk. +views — see the concept page). For filtering and disk-reclamation behavior, see +[How `EXPIRE ROWS` works](/docs/concepts/expire-rows/#how-it-works). ## Syntax @@ -39,7 +38,7 @@ Without `N`, the keep column must be `BYTE`, `SHORT`, `INT`, `LONG`, `FLOAT`, `ORDER BY` and accepts any orderable column type. For the full description of each mode and its semantics, see the -[Expiring rows](/docs/concepts/deep-dive/expire-rows/) concept page. +[Expiring rows](/docs/concepts/expire-rows/) concept page. ## When to use @@ -53,9 +52,8 @@ For the full description of each mode and its semantics, see the `SET EXPIRE ROWS` validates the new policy against the view's columns first (compiling the predicate / checking the key columns), so an invalid predicate or an unknown column is rejected immediately rather than breaking later reads. Once -set, queries against the view are filtered to the kept rows immediately; physical -reclamation follows in the background. See -[How it works](/docs/concepts/deep-dive/expire-rows/#how-it-works). +set, the policy takes effect without rebuilding the view. See +[How it works](/docs/concepts/expire-rows/#how-it-works). ## Examples @@ -68,7 +66,7 @@ A `WHEN` predicate is the right tool for a cutoff that moves with the clock like this one. A deterministic predicate such as `amount < 1.5` is accepted too, but it selects the same rows more cheaply as a `WHERE` clause in the view's defining query — see -[`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/deep-dive/expire-rows/#where-filter-or-expire-rows). +[`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/expire-rows/#where-filter-or-expire-rows). ```questdb-sql title="Keep the latest row per symbol" ALTER MATERIALIZED VIEW trades_mirror @@ -96,7 +94,6 @@ ALTER MATERIALIZED VIEW trades_mirror DROP EXPIRE; | Passthrough recommended | An aggregating view is accepted with a logged advisory: a later refresh can regenerate reclaimed rows, so align base-table retention with the expiry horizon | | No dependent views | Rejected when other materialized views derive from this view (they would copy expired rows on refresh) | | Validation | The policy is checked against the view's columns before it is applied | -| Immediate effect | Reads are filtered to the kept rows as soon as the policy is set | | Replication | The policy and the reclamation it drives replicate as normal WAL traffic | ## Permissions (Enterprise) @@ -123,7 +120,7 @@ GRANT ALTER MATERIALIZED VIEW ON trades_mirror TO user1; ## See also -- [Expiring rows (EXPIRE ROWS) concept](/docs/concepts/deep-dive/expire-rows/) +- [Expiring rows (EXPIRE ROWS) concept](/docs/concepts/expire-rows/) - [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/) - [Materialized views concept](/docs/concepts/materialized-views/) - [ALTER MATERIALIZED VIEW SET TTL](/docs/query/sql/alter-mat-view-set-ttl/) diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index a5bc445b7a..75bdb8d5f1 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -309,7 +309,7 @@ rolling `ts < dateadd('d', -7, now())` window — the defining query cannot express those, because it rejects non-deterministic functions. A predicate that depends only on the row's own values belongs in the query's `WHERE` clause instead, which keeps those rows out of the view entirely; see -[`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/deep-dive/expire-rows/#where-filter-or-expire-rows). +[`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/expire-rows/#where-filter-or-expire-rows). The clause goes after the query (and after `PARTITION BY` if present): @@ -325,21 +325,19 @@ A `WHEN` threshold that is constant at definition time and evaluates to `NULL` is rejected, since it would expire nothing. That covers the explicit `ts < CAST(NULL AS TIMESTAMP)` and arithmetic that overflows onto the reserved `NULL` value, such as `ts < 2147483647 + 1` — see -[A `NULL` threshold is rejected](/docs/concepts/deep-dive/expire-rows/#a-null-threshold-is-rejected). +[A `NULL` threshold is rejected](/docs/concepts/expire-rows/#a-null-threshold-is-rejected). -Expired rows are hidden from queries immediately in every mode. They are -reclaimed on disk in the background (`CLEANUP EVERY`, default `1h`) under a -monotonic `WHEN` predicate; `KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N` and -window predicates hide rows without freeing disk. Change or remove a policy -with +For filtering and disk-reclamation behavior, see +[How `EXPIRE ROWS` works](/docs/concepts/expire-rows/#how-it-works). Change or +remove a policy with [`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/). A view can carry both `TTL` and `EXPIRE ROWS`. `TTL` comes first in the statement and first in effect: it removes rows from the view, and the `EXPIRE ROWS` policy then applies to the rows that stay — see -[Combining with TTL](/docs/concepts/deep-dive/expire-rows/#combining-with-ttl). +[Combining with TTL](/docs/concepts/expire-rows/#combining-with-ttl). -See the [Expiring rows](/docs/concepts/deep-dive/expire-rows/) concept page for +See the [Expiring rows](/docs/concepts/expire-rows/) concept page for all modes, worked examples, and semantics (NULLs, ties, monotonicity). ## Complete example diff --git a/documentation/sidebars.js b/documentation/sidebars.js index 6cc84b3cc3..23ae94729c 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -627,6 +627,7 @@ module.exports = { "concepts/deduplication", "concepts/delivery-semantics", "concepts/ttl", + "concepts/expire-rows", "concepts/storage-policy", { id: "concepts/cold-storage", @@ -643,7 +644,6 @@ module.exports = { items: [ "concepts/deep-dive/indexes", "concepts/deep-dive/posting-index", - "concepts/deep-dive/expire-rows", "concepts/deep-dive/interval-scan", "concepts/deep-dive/jit-compiler", "concepts/deep-dive/query-tracing", From 6b86c2ab2b29928cbfe19812a9c2023620937bf8 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Thu, 27 Aug 2026 12:11:06 +0200 Subject: [PATCH 11/25] docs: address expire rows review comments --- documentation/changelog.mdx | 1 + documentation/concepts/expire-rows.md | 162 +++++++++--------- documentation/concepts/materialized-views.md | 20 +-- .../query/sql/alter-mat-view-set-expire.md | 7 +- documentation/query/sql/create-mat-view.md | 39 +++-- 5 files changed, 121 insertions(+), 108 deletions(-) diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 65d12a89d5..1977726f06 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -16,6 +16,7 @@ This page tracks significant updates to the QuestDB documentation. ### New +- [Expiring rows (`EXPIRE ROWS`)](/docs/concepts/expire-rows/) and [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) - New row-level retention for materialized views, with creation syntax in [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/), immediate read filtering, and background reclamation for eligible policies - [Cold storage](/docs/concepts/cold-storage/) - Historical partitions move to S3, Google Cloud Storage, Azure Blob Storage, or a filesystem store as Parquet and stay queryable with normal SQL, covering the partition lifecycle, the range-read path, the immutability boundary at `DROP LOCAL`, and the manager and refresher roles - [Operating cold storage](/docs/operations/cold-storage/) - Setup guide and runbook: preparing the object store prefix, configuring the cluster, the two-step manager handoff, remote garbage collection, cold-read metrics, backup interaction, and a troubleshooting table - [Cold storage configuration](/docs/configuration/cold-storage/) - Every `cold.storage.*` setting, covering the store connection, roles, upload, read path, garbage collection, and request timeouts diff --git a/documentation/concepts/expire-rows.md b/documentation/concepts/expire-rows.md index 1b13e0b159..f489e402f2 100644 --- a/documentation/concepts/expire-rows.md +++ b/documentation/concepts/expire-rows.md @@ -4,15 +4,15 @@ sidebar_label: EXPIRE ROWS description: EXPIRE ROWS is a row-level retention policy for passthrough materialized views. Keep the latest row per key, the top-N per group, or rows matching a - predicate — recomputed continuously, with expired rows hidden immediately, and - reclaimed in the background under the modes that allow it. + predicate. Policies are recomputed continuously, with expired rows hidden + immediately and reclaimed in the background under the modes that allow it. --- `EXPIRE ROWS` is a row-level retention policy for [materialized views](/docs/concepts/materialized-views/). Where [TTL](/docs/concepts/ttl/) drops whole partitions once they age out, `EXPIRE -ROWS` decides retention **row by row** — keep the latest row per key, the top-N -per group, rows matching a predicate, and so on — and recomputes the result +ROWS` decides retention **row by row**. It can keep the latest row per key, the +top-N per group, rows matching a predicate, and so on. It recomputes the result continuously as the view refreshes. Expired rows disappear from query results **immediately** in every mode. Their @@ -30,10 +30,10 @@ only when the rule selects it for removal. | Mode | What it keeps | Syntax | Frees disk | | --------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | --------------------- | | Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when monotonic | -| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | No — read filter only | -| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | -| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | -| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No — read filter only | +| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | No (read filter only) | +| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | +| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | +| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No (read filter only) | `KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a window predicate, so the window `WHEN` is the general escape hatch. @@ -72,9 +72,9 @@ EXPIRE ROWS :::note `EXPIRE ROWS` is **materialized-view-only**: `CREATE TABLE ... EXPIRE ROWS` is -rejected. It is designed for a **passthrough** (non-aggregating) view — `SELECT -* FROM base` with no `SAMPLE BY` / `GROUP BY` — where the view mirrors base -rows 1:1 and reclamation is permanent. An aggregating view is **accepted with a +rejected. It is designed for a **passthrough** (non-aggregating) view, where +`SELECT * FROM base` has no `SAMPLE BY` / `GROUP BY`, the view mirrors base rows +1:1, and reclamation is permanent. An aggregating view is **accepted with a logged advisory** because a later refresh can regenerate reclaimed rows from base rows that still exist (see [Requirements](#requirements)). For base-table retention use @@ -88,29 +88,29 @@ base rows that still exist (see Reach for `EXPIRE ROWS` on a passthrough materialized view when you want a continuously-maintained, pruned copy of a base table: -- **Current-state-per-key tables** — keep only the latest row per device, +- **Current-state-per-key tables**: keep only the latest row per device, symbol, or session (`KEEP LATEST`). -- **Per-group extremes or leaderboards** — keep the highest/lowest value per +- **Per-group extremes or leaderboards**: keep the highest/lowest value per group, or the top-N (`KEEP HIGHEST/LOWEST`, `KEEP N`). -- **Rolling row-level windows** — keep rows newer than a moving cutoff such as +- **Rolling row-level windows**: keep rows newer than a moving cutoff such as `now() - 7d`, at finer granularity than TTL's whole-partition drops (`WHEN predicate`). The `WHEN` form earns its keep on predicates that involve **wall-clock time**. A -deterministic predicate — one that depends only on the row's own values — -selects the same rows more cheaply as a `WHERE` clause in the view's defining -query, which never copies the excluded rows into the view at all. See +deterministic predicate depends only on the row's own values and selects the +same rows more cheaply as a `WHERE` clause in the view's defining query, which +never copies the excluded rows into the view at all. See [`WHERE` filter or `EXPIRE ROWS`?](#where-filter-or-expire-rows). Use [TTL](/docs/concepts/ttl/) instead when partition-granularity, age-based -retention on a base table is enough — it is cheaper and has no passthrough-view +retention on a base table is enough. It is cheaper and has no passthrough-view requirement. ## Requirements `EXPIRE ROWS` is designed for a **passthrough materialized view**: -- The view query keeps view rows 1:1 with base rows — a projection over a single +- The view query keeps view rows 1:1 with base rows as a projection over a single table, with or without a `WHERE` filter. See [which queries are passthrough](/docs/concepts/materialized-views/#which-queries-are-passthrough) for the full rules. @@ -120,17 +120,17 @@ requirement. A passthrough view mirrors its base table 1:1 and refreshes incrementally, so it is effectively a continuously-maintained replica. `EXPIRE ROWS` prunes that -replica down to the rows you want to keep — without touching the base table. +replica down to the rows you want to keep without touching the base table. A **non-passthrough (aggregating) view is accepted with a logged advisory** rather than rejected: physical reclamation only sticks when base-table retention -is aligned with the expiry horizon — a later incremental or full refresh can -regenerate a reclaimed row from base rows that still exist. +is aligned with the expiry horizon because a later incremental or full refresh +can regenerate a reclaimed row from base rows that still exist. A policied view must also stand alone: `CREATE MATERIALIZED VIEW` rejects a defining query that reads a policied view (as its base or in a join), and `ALTER ... SET EXPIRE` is rejected on a view that other materialized views -derive from — otherwise those views would copy expired rows on refresh. +derive from because those views would copy expired rows on refresh. ## Worked examples @@ -160,7 +160,7 @@ INSERT INTO trades VALUES A materialized view starts an **asynchronous initial refresh** on creation, so it may briefly return no rows. Check progress with `SELECT view_name, view_status, base_table_txn, refresh_base_table_txn FROM materialized_views();` -— the view is up to date when `refresh_base_table_txn = base_table_txn`. With +The view is up to date when `refresh_base_table_txn = base_table_txn`. With this small dataset that is effectively instant. ::: @@ -168,8 +168,8 @@ this small dataset that is effectively instant. ### Per-row predicate: `WHEN` A per-row predicate expires a row when it evaluates `TRUE`. The example below -uses a deterministic predicate — expire small trades (`amount < 1.5`) — -because it makes the keep-set easy to read off the sample data. In production +uses a deterministic predicate to expire small trades (`amount < 1.5`) because +it makes the keep-set easy to read off the sample data. In production that rule belongs in the view's `WHERE` clause (`WHERE amount >= 1.5`), which keeps those rows out of the view entirely; the rolling window further down is the case `WHEN` exists for. @@ -190,12 +190,12 @@ SELECT * FROM trades_sized ORDER BY ts; The two `amount = 1.0` rows are expired. `amount = 1.5` is kept (`1.5 < 1.5` is `FALSE`), and any `NULL` amount would be kept too (the comparison is `UNKNOWN`, -not `TRUE` — see [NULLs](#nulls)). +not `TRUE`). See [NULLs](#nulls). -A predicate on the designated timestamp gives a **rolling retention window** — -the main use for `WHEN`. It is re-evaluated on every read, so the visible set -rolls forward with the clock, and no `WHERE` clause can express it because the -defining query rejects `now()`: +A predicate on the designated timestamp gives a **rolling retention window**, +which is the main use for `WHEN`. It is re-evaluated on every read, so the +visible set rolls forward with the clock, and no `WHERE` clause can express it +because the defining query rejects `now()`: ```questdb-sql title="Keep the last 1 day" CREATE MATERIALIZED VIEW trades_recent AS ( @@ -207,12 +207,13 @@ CREATE MATERIALIZED VIEW trades_recent AS ( and would be hidden; use recent data to see rows retained.) `WHEN ts < dateadd('d', -1, now())` and `WHEN ts < now() - 86400000000` retain -the same rows and both reclaim disk — the cleanup job proves either monotonic. +the same rows and both reclaim disk because the cleanup job proves either form +monotonic. ### Keep latest per key: `KEEP LATEST` -Keep only the most recent row per key — turning the passthrough view into a -live, current-state-per-symbol table: +Keep only the most recent row per key to turn the passthrough view into a live, +current-state-per-symbol table: ```questdb-sql title="Keep the latest row per symbol" CREATE MATERIALIZED VIEW trades_latest AS ( @@ -227,10 +228,10 @@ SELECT * FROM trades_latest ORDER BY ts; | ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | | BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | -One row per symbol — the latest by the designated timestamp `ts`. As new trades -arrive, the kept row advances automatically. `PARTITION BY` may list multiple key -columns. You may write `KEEP LATEST ON ts PARTITION BY symbol`, but the `ON` -column must be the view's designated timestamp. +The designated timestamp `ts` determines the latest row for each symbol. As new +trades arrive, the kept row advances automatically. `PARTITION BY` may list +multiple key columns. You may write `KEEP LATEST ON ts PARTITION BY symbol`, but +the `ON` column must be the view's designated timestamp. ### Keep extremes per group: `KEEP HIGHEST` / `KEEP LOWEST` @@ -255,7 +256,7 @@ are kept (a `NULL` is never less than the max). ### Keep top-N per group: `KEEP N HIGHEST` / `KEEP N LOWEST` -Keep a per-group leaderboard — the `N` highest (or lowest) rows: +Keep a per-group leaderboard with the `N` highest (or lowest) rows: ```questdb-sql title="Keep the 2 highest-priced trades per symbol" CREATE MATERIALIZED VIEW trades_top2 AS ( @@ -290,8 +291,8 @@ CREATE MATERIALIZED VIEW trades_peak_win AS ( ``` A row expires when its price is below its symbol's maximum, so only the peak per -symbol survives — the same result as `trades_peak` above. From here you can -express richer rules, for example keeping rows within 5% of the peak +symbol survives. This produces the same result as `trades_peak` above. From here +you can express richer rules, for example keeping rows within 5% of the peak (`WHEN price < 0.95 * max(price) OVER (PARTITION BY symbol)`) or a ranked window (`WHEN row_number() OVER (PARTITION BY symbol ORDER BY ts DESC) > 100`). @@ -305,15 +306,15 @@ clock. window cannot be written as a `WHERE` clause at all: a view's defining query rejects non-deterministic functions, so `WHERE ts > dateadd('d', -7, now())` is not accepted. `EXPIRE ROWS WHEN ts < dateadd('d', -7, now())` is the supported -way to say "keep the last 7 days" — the read filter re-evaluates `now()` on +way to say "keep the last 7 days". The read filter re-evaluates `now()` on every read, so the window rolls forward on its own and the cleanup job reclaims the disk behind it. This is what the `WHEN` form is for. **Put a deterministic predicate in the `WHERE` clause.** A predicate that -depends only on the row's own values — `symbol = 'BTC'`, `amount >= 1.5` — -describes the same surviving rows either way, so the two are near-equivalent in -what the view contains, and `WHERE` is the cheaper of the two at every stage. A -row the `WHERE` clause excludes is never copied into the view: +depends only on the row's own values, such as `symbol = 'BTC'` or +`amount >= 1.5`, describes the same surviving rows either way, so the two are +near-equivalent in what the view contains, and `WHERE` is the cheaper of the two +at every stage. A row the `WHERE` clause excludes is never copied into the view: | | `WHERE` in the query | `EXPIRE ROWS WHEN` | | --- | --- | --- | @@ -321,7 +322,7 @@ row the `WHERE` clause excludes is never copied into the view: | Read cost | None | The keep-set filter is applied on every read of the view | | Write cost | None | The cleanup job rewrites partitions on the `CLEANUP EVERY` cadence | | After a full refresh | Still excluded | Re-materialized from the base, then hidden and swept again | -| Can be another view's base | Yes | No — a policied view is rejected as a base | +| Can be another view's base | Yes | No (a policied view is rejected as a base) | That last row is a hard constraint rather than a preference. If any other view will read this one, the policy is not available and the predicate has to go in @@ -330,7 +331,7 @@ the `WHERE` clause. The two forms are not exact negations of each other on `NULL`s: `WHERE` keeps a row only when the predicate is `TRUE`, while `EXPIRE ROWS WHEN` expires a row only when it is `TRUE`. A `NULL` amount is dropped by `WHERE amount >= 1.5` and -kept by `EXPIRE ROWS WHEN amount < 1.5` — see [NULLs](#nulls). +kept by `EXPIRE ROWS WHEN amount < 1.5`. See [NULLs](#nulls). The two compose, and on a passthrough view that combination is usually the right shape: the `WHERE` clause fixes what the view is about, and the `WHEN` policy @@ -375,8 +376,8 @@ best-effort background cleanup. ### Read-time filter (authoritative) Every query against a policied view is transparently rewritten so that only the -kept rows are visible — **immediately, and regardless of whether cleanup has -run**. This is what makes results correct at all times: +kept rows are visible **immediately, regardless of whether cleanup has run**. +This is what makes results correct at all times: - **Per-row `WHEN`** keeps rows where the predicate is not `TRUE`. QuestDB filtering is three-valued, so `FALSE` **and** `NULL` are kept (see @@ -402,8 +403,8 @@ or to `1` to disable partial-partition compaction; fully-expired partitions are still removed. The threshold does not delay a fixed, deterministic predicate, whose expired-row verdicts cannot change with time. -The job runs at the `CLEANUP EVERY` cadence (default `1h`) and is **best-effort** -— the read filter is authoritative, so deferred or skipped reclamation only +The job runs at the `CLEANUP EVERY` cadence (default `1h`) and is **best-effort**. +The read filter is authoritative, so deferred or skipped reclamation only affects disk usage, never query results. The job runs only under a monotonic `WHEN` predicate. It skips `KEEP LATEST`, @@ -417,7 +418,7 @@ still replicates: the compaction commits are ordinary WAL transactions, so replicas reclaim the identical rows by applying them. A read-only replica neither runs the job nor needs to. Disable the job with `cairo.mat.view.row.expiry.cleanup.enabled=false` in `server.conf` (reads stay -filtered; only reclamation stops — the setting does not disable `EXPIRE ROWS` +filtered, but only reclamation stops; the setting does not disable `EXPIRE ROWS` itself). Cleanup settings are read at startup, so changing this property or the minimum expired fraction requires a restart. A failing sweep retries after one second, doubling the per-view retry gap up to a 10-minute cap. @@ -452,8 +453,8 @@ Therefore: ### A `NULL` threshold is rejected -A `WHEN` threshold that evaluates to a constant `NULL` expires nothing — -`ts < NULL` is never `TRUE` — so the policy would be inert. QuestDB refuses it at +A `WHEN` threshold that evaluates to a constant `NULL` expires nothing because +`ts < NULL` is never `TRUE`, so the policy would be inert. QuestDB refuses it at `CREATE` and `ALTER` time rather than storing a view that silently never reclaims: @@ -487,11 +488,11 @@ cannot be checked in advance. ### Ties and determinism -`KEEP HIGHEST/LOWEST` keeps **all** rows tied at the max/min — deterministic by -construction. `KEEP N` makes the order total by appending the designated -timestamp as a tiebreak, so the N-th boundary is deterministic (pair the base -table with [`DEDUP UPSERT KEYS`](/docs/concepts/deduplication/) if `(col, ts)` -is not already unique). +`KEEP HIGHEST/LOWEST` keeps **all** rows tied at the max/min, making the result +deterministic by construction. `KEEP N` makes the order total by appending the +designated timestamp as a tiebreak, so the N-th boundary is deterministic (pair +the base table with [`DEDUP UPSERT KEYS`](/docs/concepts/deduplication/) if +`(col, ts)` is not already unique). ### Combining with TTL @@ -514,7 +515,7 @@ CREATE MATERIALIZED VIEW trades_peak_3d AS ( This view reports the highest price of the **last three days**, so its answer can go **down** as the window moves: when the day holding a symbol's maximum ages out, the next-highest price within the window takes over. That is what the -two clauses ask for together — the view is no longer "the highest price ever", +two clauses ask for together. The view is no longer "the highest price ever"; it is "the highest price still retained". The base table is unaffected; it keeps whatever its own retention settings keep. @@ -530,8 +531,8 @@ expired now must stay expired forever. Two separate things can break that. The relative and window modes (`KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N`, window `WHEN`) decide each row's fate by comparing it against the other rows in the view. A later refresh can remove or replace the row a key currently keeps, -which promotes an older row back into the keep-set — so the cleanup job never -deletes for these modes, whatever their predicate looks like. +which promotes an older row back into the keep-set. For this reason, the cleanup +job never deletes for these modes, whatever their predicate looks like. A scalar `WHEN predicate` judges each row on its own, so it is eligible. It is arbitrary SQL, so the cleanup job reclaims disk only for predicates it can @@ -543,8 +544,9 @@ arbitrary SQL, so the cleanup job reclaims disk only for predicates it can (`ts < now() - 7200000000`), or a fixed-unit look-back `dateadd` on a bare clock (`ts < dateadd('d', -1, now())`, units `s`/`m`/`h`/`d`/`w` and finer). -Anything else **skips cleanup**: calendar units (`dateadd('M', -1, now())` — a -month is a variable amount), look-forward offsets (`dateadd('h', 1, now())`), +Anything else **skips cleanup**: calendar units such as +`dateadd('M', -1, now())` (a month is a variable amount), look-forward offsets +(`dateadd('h', 1, now())`), further clock arithmetic, non-constant offsets, and arbitrary window `WHEN` predicates. A skipped policy stays correct at read time (the filter recomputes on every read), but its disk is not reclaimed until the policy is changed to a @@ -555,9 +557,9 @@ proven shape. A non-monotonic predicate such as `WHEN ts > now()` expires *future* rows that **un-expire** as `now()` advances. The read filter recomputes `now()` on every read and stays correct, and the cleanup job skips such a policy rather than -risk physically deleting a row a later read must show — at the cost of disk -never being reclaimed for it. Write `WHEN` predicates that expire things in the -**past** or against fixed thresholds — never rows that the passage of time will +risk physically deleting a row a later read must show. The tradeoff is that its +disk is never reclaimed. Write `WHEN` predicates that expire things in the +**past** or against fixed thresholds, never rows that the passage of time will later keep. ::: @@ -588,15 +590,15 @@ FROM materialized_views(); `expire_enforcement` is the verdict the cleanup job acts on: -- `FILTER_AND_RECLAIM` — reads hide the expired rows and the job deletes them +- `FILTER_AND_RECLAIM`: reads hide the expired rows and the job deletes them from disk. -- `FILTER_ONLY` — reads hide the expired rows and they stay on disk. Every +- `FILTER_ONLY`: reads hide the expired rows and they stay on disk. Every relative and window policy reports this, as does a `WHEN` predicate that cannot be proven monotonic. ## Changing or removing a policy -Set, change, or drop a policy on an existing passthrough view — see +Set, change, or drop a policy on an existing passthrough view. See [`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/): ```questdb-sql @@ -617,7 +619,7 @@ rather than breaking subsequent reads. keep-set over the whole physical view on every read. `KEEP LATEST` on an [indexed](/docs/concepts/deep-dive/indexes/) symbol key is cheap; the window modes (and non-indexed keep-latest) scan the view. A tighter `CLEANUP EVERY` - keeps the physical residue — and therefore the read cost — small. + keeps the physical residue, and therefore the read cost, small. - **Cleanup defers under continuous refresh.** Reclamation only proceeds when the view is quiescent and fully applied, so a view being refreshed continuously defers reclamation to a quiet sweep. The read filter stays authoritative @@ -641,14 +643,14 @@ rather than breaking subsequent reads. ## Related documentation -- [Materialized views](/docs/concepts/materialized-views/) — the view type +- [Materialized views](/docs/concepts/materialized-views/): the view type `EXPIRE ROWS` runs on -- [Passthrough views](/docs/concepts/materialized-views/#passthrough-views) — - the non-aggregating views `EXPIRE ROWS` applies to -- [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/) — full create +- [Passthrough views](/docs/concepts/materialized-views/#passthrough-views): the + non-aggregating views `EXPIRE ROWS` applies to +- [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/): full create syntax, including the `EXPIRE ROWS` clause -- [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) - — set, change, or drop a policy -- [Time To Live (TTL)](/docs/concepts/ttl/) — partition-level retention by age -- [Storage policy](/docs/concepts/storage-policy/) — graduated partition +- [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/): + set, change, or drop a policy +- [Time To Live (TTL)](/docs/concepts/ttl/): partition-level retention by age +- [Storage policy](/docs/concepts/storage-policy/): graduated partition lifecycle (Enterprise) diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index aaa8fcc523..62c5397c5f 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -182,10 +182,10 @@ CREATE MATERIALIZED VIEW trades_btc AS ( Reach for one when you want a maintained *subset* of a large table rather than a summary of it: -- **A narrowed replica** — one symbol, one tenant, one region, or a handful of +- **A narrowed replica**: one symbol, one tenant, one region, or a handful of columns out of a wide table, kept current automatically and queried without the base table's scan cost. -- **A row-level retention target** — attach an +- **A row-level retention target**: attach an [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy to keep only some of the view's rows. The base table is left alone. @@ -310,21 +310,21 @@ table qualifies, with or without a filter: | Query | | | ----- | --- | | `SELECT * FROM trades` | Passthrough | -| `SELECT timestamp, symbol, price FROM trades` | Passthrough — column subset | -| `SELECT timestamp, symbol AS ticker FROM trades` | Passthrough — aliases are fine | -| `SELECT timestamp, price * amount AS notional FROM trades` | Passthrough — a row-local expression | -| `SELECT * FROM trades WHERE symbol = 'BTC'` | Passthrough — a filter only removes rows | +| `SELECT timestamp, symbol, price FROM trades` | Passthrough: column subset | +| `SELECT timestamp, symbol AS ticker FROM trades` | Passthrough: aliases are fine | +| `SELECT timestamp, price * amount AS notional FROM trades` | Passthrough: a row-local expression | +| `SELECT * FROM trades WHERE symbol = 'BTC'` | Passthrough: a filter only removes rows | | `... SAMPLE BY 1h`, or `GROUP BY` on a timestamp | Aggregating view | | `SELECT DISTINCT ...` | Rejected | | `... LATEST ON timestamp PARTITION BY symbol` | Rejected | | `JOIN`, `UNION` | Rejected | | `row_number() OVER (...)` and other window functions | Rejected | | `LIMIT` | Rejected | -| `ORDER BY` a non-timestamp column | Rejected — the view loses its designated timestamp | +| `ORDER BY` a non-timestamp column | Rejected: the view loses its designated timestamp | Everything in the rejected group produces output that depends on rows *other than* the one being emitted. An incremental refresh sees only the newly-arrived -slice of the base table, so it cannot compute those correctly — a `LIMIT 100` +slice of the base table, so it cannot compute those correctly. A `LIMIT 100` would admit 100 rows per refresh rather than 100 in total, and a `row_number()` would restart within each slice. @@ -334,8 +334,8 @@ SAMPLE BY or GROUP BY timestamp_floor()`: the query looked like it meant to aggregate but named no bucket. `LIMIT` and window functions have their own messages. -To keep only some of a passthrough view's rows over time — the latest per key, -the top-N per group, or rows matching a predicate — attach an +To keep only some of a passthrough view's rows over time, such as the latest per +key, the top-N per group, or rows matching a predicate, attach an [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. That page also covers [when to put a predicate in the view's `WHERE` clause instead](/docs/concepts/expire-rows/#where-filter-or-expire-rows). diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 3303c43bd5..2fed2afa47 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -7,8 +7,9 @@ description: --- Sets, replaces, or removes an [`EXPIRE ROWS`](/docs/concepts/expire-rows/) -row-retention policy on a materialized view (designed for **passthrough** -views — see the concept page). For filtering and disk-reclamation behavior, see +row-retention policy on a materialized view. It is designed for +**passthrough views**; see the concept page. For filtering and disk-reclamation +behavior, see [How `EXPIRE ROWS` works](/docs/concepts/expire-rows/#how-it-works). ## Syntax @@ -65,7 +66,7 @@ ALTER MATERIALIZED VIEW trades_mirror A `WHEN` predicate is the right tool for a cutoff that moves with the clock like this one. A deterministic predicate such as `amount < 1.5` is accepted too, but it selects the same rows more cheaply as a `WHERE` clause in the view's defining -query — see +query. See [`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/expire-rows/#where-filter-or-expire-rows). ```questdb-sql title="Keep the latest row per symbol" diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 75bdb8d5f1..57988083b9 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -18,22 +18,31 @@ CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] viewName [ START timestamp ] [ TIME ZONE timezone ] [ PERIOD ( LENGTH length [ TIME ZONE tz ] [ DELAY delay ] ) ] [ PERIOD ( SAMPLE BY INTERVAL ) ] ] -AS [ ( ] query [ ) ] -[ TIMESTAMP ( columnRef ) ] -[ PARTITION BY ( YEAR | MONTH | WEEK | DAY | HOUR ) - [ TTL n timeUnit ] ] -[ EXPIRE ROWS expirePolicy [ CLEANUP EVERY duration ] ] -[ OWNED BY ownerName ] +AS +{ query +| ( query ) + [ TIMESTAMP ( columnRef ) ] + [ PARTITION BY ( YEAR | MONTH | WEEK | DAY | HOUR ) + [ TTL n timeUnit ] ] + [ EXPIRE ROWS expirePolicy [ CLEANUP EVERY duration ] ] + [ OWNED BY ownerName ] +} ``` +Parentheses around `query` are optional only when the query ends the statement. +They are required when any trailing clause follows the query, including +`TIMESTAMP`, `PARTITION BY`, `TTL`, `EXPIRE ROWS`, or `OWNED BY`. + Where: + - `interval`: Duration like `1m`, `10m`, `1h`, `1d` - `timeUnit`: `HOURS | DAYS | WEEKS | MONTHS | YEARS` - `query`: Either an aggregating query with `SAMPLE BY` or a time-based `GROUP BY`, or a [passthrough](/docs/concepts/materialized-views/#passthrough-views) projection over a single table -- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON ts] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]` — designed for passthrough views (see below) +- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON ts] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]`. + This policy is designed for passthrough views (see below). ## Parameters @@ -287,14 +296,14 @@ The view's TTL is independent of the base table's TTL. See ## EXPIRE ROWS Attach a row-level retention policy with `EXPIRE ROWS`. Unlike `TTL` (which drops -whole partitions by age), `EXPIRE ROWS` keeps a defined set of rows — the latest -per key, the top-N per group, or rows matching a predicate — recomputed -continuously as the view refreshes. +whole partitions by age), `EXPIRE ROWS` keeps a defined set of rows: the latest +per key, the top-N per group, or rows matching a predicate. It recomputes that +set continuously as the view refreshes. `EXPIRE ROWS` is designed for [**passthrough (non-aggregating) views**](/docs/concepts/materialized-views/#passthrough-views) -— a projection over a single table, with no `SAMPLE BY` / `GROUP BY`, whose rows -stay 1:1 with the base. An aggregating view is accepted with a logged advisory +(a projection over a single table with no `SAMPLE BY` / `GROUP BY`, whose rows +stay 1:1 with the base). An aggregating view is accepted with a logged advisory (a later refresh can regenerate reclaimed rows). The defining query must not read another policied view, as its base or in a join: @@ -305,7 +314,7 @@ CREATE MATERIALIZED VIEW trades_latest AS ( ``` A `WHEN` predicate is for rules that move with **wall-clock time**, such as a -rolling `ts < dateadd('d', -7, now())` window — the defining query cannot +rolling `ts < dateadd('d', -7, now())` window. The defining query cannot express those, because it rejects non-deterministic functions. A predicate that depends only on the row's own values belongs in the query's `WHERE` clause instead, which keeps those rows out of the view entirely; see @@ -324,7 +333,7 @@ EXPIRE ROWS A `WHEN` threshold that is constant at definition time and evaluates to `NULL` is rejected, since it would expire nothing. That covers the explicit `ts < CAST(NULL AS TIMESTAMP)` and arithmetic that overflows onto the reserved -`NULL` value, such as `ts < 2147483647 + 1` — see +`NULL` value, such as `ts < 2147483647 + 1`. See [A `NULL` threshold is rejected](/docs/concepts/expire-rows/#a-null-threshold-is-rejected). For filtering and disk-reclamation behavior, see @@ -334,7 +343,7 @@ remove a policy with A view can carry both `TTL` and `EXPIRE ROWS`. `TTL` comes first in the statement and first in effect: it removes rows from the view, and the -`EXPIRE ROWS` policy then applies to the rows that stay — see +`EXPIRE ROWS` policy then applies to the rows that stay. See [Combining with TTL](/docs/concepts/expire-rows/#combining-with-ttl). See the [Expiring rows](/docs/concepts/expire-rows/) concept page for From 92dae9b567027369a73a64218d75d34001ed59b1 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Thu, 27 Aug 2026 12:45:19 +0200 Subject: [PATCH 12/25] docs: refine expire rows reference --- documentation/concepts/expire-rows.md | 19 +++++++++----- .../configuration/materialized-views.md | 25 ++++++++++++++++--- documentation/query/functions/meta.md | 2 +- .../query/sql/alter-mat-view-set-expire.md | 7 +++--- documentation/query/sql/show.md | 6 +++-- 5 files changed, 44 insertions(+), 15 deletions(-) diff --git a/documentation/concepts/expire-rows.md b/documentation/concepts/expire-rows.md index f489e402f2..5d251b0f3b 100644 --- a/documentation/concepts/expire-rows.md +++ b/documentation/concepts/expire-rows.md @@ -42,7 +42,11 @@ The bare `KEEP HIGHEST/LOWEST` form accepts `BYTE`, `SHORT`, `INT`, `LONG`, `FLOAT`, `DOUBLE`, `DATE`, `TIMESTAMP` and `DECIMAL` columns. The top-N form has a broader type surface: `KEEP N HIGHEST/LOWEST` ranks with `ORDER BY`, so it accepts any orderable column type. For example, use `KEEP 1 HIGHEST symbol`, not -`KEEP HIGHEST symbol`, to rank a `SYMBOL` column. +`KEEP HIGHEST symbol`, to rank a `SYMBOL` column. This changes tie and `NULL` +behavior: the top-N form keeps exactly one row per group using the designated +timestamp as a descending tiebreaker. An integer or timestamp `NULL` sorts last +and is expired, while a floating-point `NULL` sorts first. The bare form keeps +every row tied at the extreme and every `NULL`. For how read filtering and physical reclamation differ between modes, see [How it works](#how-it-works). You can inspect the behavior selected for a view @@ -535,14 +539,17 @@ which promotes an older row back into the keep-set. For this reason, the cleanup job never deletes for these modes, whatever their predicate looks like. A scalar `WHEN predicate` judges each row on its own, so it is eligible. It is -arbitrary SQL, so the cleanup job reclaims disk only for predicates it can -**prove** monotonic: +arbitrary SQL. QuestDB recognizes `now()`, `now_ns()`, `sysdate()`, +`systimestamp()` and `systimestamp_ns()` as wall-clock functions, and gives each +the same monotonicity proof. The cleanup job reclaims disk only for predicates +it can **prove** monotonic: - clock-free predicates (`WHEN amount < 1.5`), and - designated-timestamp thresholds of a proven advancing-clock shape: a bare - clock (`ts < now()`), a bare clock minus a non-negative constant - (`ts < now() - 7200000000`), or a fixed-unit look-back `dateadd` on a bare - clock (`ts < dateadd('d', -1, now())`, units `s`/`m`/`h`/`d`/`w` and finer). + clock (for example, `ts < now()`), a bare clock minus a non-negative constant + (for example, `ts < now() - 7200000000`), or a fixed-unit look-back `dateadd` + on a bare clock (for example, `ts < dateadd('d', -1, now())`, with units + `s`/`m`/`h`/`d`/`w` and finer). Anything else **skips cleanup**: calendar units such as `dateadd('M', -1, now())` (a month is a variable amount), look-forward offsets diff --git a/documentation/configuration/materialized-views.md b/documentation/configuration/materialized-views.md index 49f73b12ca..4d22ddc6b2 100644 --- a/documentation/configuration/materialized-views.md +++ b/documentation/configuration/materialized-views.md @@ -3,9 +3,9 @@ title: Materialized views description: Configuration settings for materialized views in QuestDB. --- -These settings control materialized view SQL support and the background refresh -job. Materialized views can use dedicated worker threads or share the server's -common pool. +These settings control materialized view SQL support, background refresh, and +row-expiry cleanup. Materialized views can use dedicated worker threads or share +the server's common pool. ## cairo.mat.view.enabled @@ -22,6 +22,25 @@ Enables or disables SQL support and the refresh job for materialized views. When disabled, SQL executed by the materialized view refresh job always runs single-threaded. +## cairo.mat.view.row.expiry.cleanup.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables the background job that reclaims rows removed by an eligible +[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. Disabling the job does not +disable read filtering, so expired rows remain hidden from query results. + +## cairo.mat.view.row.expiry.cleanup.min.expired.fraction + +- **Default**: `0.5` +- **Reloadable**: no + +Minimum fraction of expired rows required before the background cleanup job +compacts a partially expired partition. Set this property to `0` to compact on +the first expired row, or to `1` to disable partial-partition compaction. Fully +expired partitions are still removed. + ## mat.view.refresh.worker.affinity - **Default**: equal to the CPU core count diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index b489cabf0d..a6d726a146 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -312,7 +312,7 @@ FROM materialized_views(); | view_name | view_status | base_table_name | refresh_base_table_txn | base_table_txn | expire_clause | expire_cleanup_every | expire_enforcement | | --------------- | ----------- | --------------- | ---------------------- | -------------- | ----------------------------------- | -------------------- | ------------------ | | trades_OHLC_15m | valid | trades | 1 | 1 | null | null | null | -| trades_recent | valid | trades | 1 | 1 | timestamp < dateadd('d', -7, now()) | 30m | FILTER_AND_RECLAIM | +| trades_recent | valid | trades | 1 | 1 | ts < dateadd('d', -7, now()) | 30m | FILTER_AND_RECLAIM | | trades_latest | valid | trades | 1 | 1 | KEEP LATEST PARTITION BY symbol | 1h | FILTER_ONLY | ## memory_metrics diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 2fed2afa47..a796e2464d 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -32,7 +32,7 @@ ALTER MATERIALIZED VIEW viewName DROP EXPIRE | `WHEN predicate` | Per-row (or window) predicate; a row expires when it evaluates `TRUE` | | `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp | | `KEEP [N] HIGHEST\|LOWEST col` | Keep the rows at the max/min of `col` per group, or the top `N` | -| `CLEANUP EVERY` | Background reclamation cadence (e.g. `30m`, `1h`). Defaults to `1h` if omitted | +| `CLEANUP EVERY` | Background reclamation cadence as ``, where `unit` is `s`, `m`, `h`, `d`, or `w`. Defaults to `1h` if omitted | Without `N`, the keep column must be `BYTE`, `SHORT`, `INT`, `LONG`, `FLOAT`, `DOUBLE`, `DATE`, `TIMESTAMP` or `DECIMAL`. `KEEP N HIGHEST/LOWEST` ranks with @@ -110,10 +110,11 @@ GRANT ALTER MATERIALIZED VIEW ON trades_mirror TO user1; | Error | Cause | | ----- | ----- | | `materialized view does not exist` | View with the specified name doesn't exist | -| `cannot set an EXPIRE ROWS policy on '...': it is the base of N materialized view(s), which would copy expired rows on refresh` | Other materialized views derive from this view | +| `cannot set an EXPIRE ROWS policy on '...': it is the base of N view(s), including '', which would copy expired rows on refresh` | Other views derive from this view; the message names the first dependent | | `EXPIRE ROWS KEEP LATEST ON must name the designated timestamp ...` | `ON` names a column other than the designated timestamp | -| `invalid EXPIRE ROWS KEEP LATEST column: ...` | A `PARTITION BY` key column does not exist | +| `invalid EXPIRE ROWS KEEP LATEST PARTITION BY column: ...` | A `PARTITION BY` key column does not exist | | `EXPIRE ROWS KEEP HIGHEST/LOWEST requires a BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, DATE, TIMESTAMP or DECIMAL column, but '' is ; use KEEP HIGHEST/LOWEST to rank an orderable column of any type` | The bare `KEEP HIGHEST/LOWEST` form was given an unsupported column type; use a supported numeric/date type or the top-N form | +| `EXPIRE ROWS KEEP HIGHEST/LOWEST requires an orderable column, but '' is ` | The top-N form was given a column type that cannot be ordered | | `EXPIRE ROWS KEEP / window retention cannot be used on a view with a column named '__qdb_re_keep'` | The view exposes a column named like the reserved keep column | | `invalid EXPIRE ROWS predicate: ...` | The predicate does not parse, bind, or type-check against the view's columns | | `invalid EXPIRE ROWS predicate: the threshold is NULL, so no row can ever expire` | A `WHEN` threshold that is constant at definition time evaluates to `NULL`, e.g. `CAST(NULL AS TIMESTAMP)` or arithmetic that overflows | diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index 23e0f298b3..a737d8b809 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -41,7 +41,7 @@ SHOW { COLUMNS FROM tableName - `SHOW CREATE LIVE VIEW` returns a DDL query that allows you to recreate a live view. - `SHOW CREATE MATERIALIZED VIEW` returns a DDL query that allows you to - recreate a materialized view. + recreate a materialized view, including its retention clauses. - `SHOW CREATE TABLE` returns a DDL query that allows you to recreate the table. - `SHOW CREATE VIEW` returns a DDL query that allows you to recreate a view. - `SHOW GROUPS` shows all groups the user belongs or all groups in the system @@ -225,7 +225,9 @@ SHOW CREATE MATERIALIZED VIEW bbo_1s; | CREATE MATERIALIZED VIEW 'bbo_1s' WITH BASE 'market_data' REFRESH IMMEDIATE AS (SELECT timestamp, symbol, last(bids[1][1]) AS bid, last(asks[1][1]) AS ask FROM market_data SAMPLE BY 1s) PARTITION BY DAY; | This returns the `CREATE MATERIALIZED VIEW` statement that would recreate the -materialized view, including its base table, refresh strategy, and partitioning. +materialized view, including its base table, refresh strategy, partitioning, +`TTL`, and `EXPIRE ROWS` policy. When the policy specifies `CLEANUP EVERY`, the +output includes that cadence as well, so the DDL preserves the complete policy. ### SHOW CREATE TABLE From a85ebfde0d8e99c32fb6c7c5935fce9c557825b7 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Thu, 27 Aug 2026 14:04:14 +0200 Subject: [PATCH 13/25] docs: correct expire rows cleanup semantics --- documentation/concepts/expire-rows.md | 74 ++++++++++++++++++--------- 1 file changed, 49 insertions(+), 25 deletions(-) diff --git a/documentation/concepts/expire-rows.md b/documentation/concepts/expire-rows.md index 5d251b0f3b..49dfff4df8 100644 --- a/documentation/concepts/expire-rows.md +++ b/documentation/concepts/expire-rows.md @@ -193,8 +193,8 @@ SELECT * FROM trades_sized ORDER BY ts; | BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | The two `amount = 1.0` rows are expired. `amount = 1.5` is kept (`1.5 < 1.5` is -`FALSE`), and any `NULL` amount would be kept too (the comparison is `UNKNOWN`, -not `TRUE`). See [NULLs](#nulls). +`FALSE`), and any `NULL` amount would be kept too because a comparison against +`NULL` evaluates to `FALSE` in QuestDB. See [NULLs](#nulls). A predicate on the designated timestamp gives a **rolling retention window**, which is the main use for `WHEN`. It is re-evaluated on every read, so the @@ -322,10 +322,10 @@ at every stage. A row the `WHERE` clause excludes is never copied into the view: | | `WHERE` in the query | `EXPIRE ROWS WHEN` | | --- | --- | --- | -| Storage | Row is never written | Row is written, and occupies disk until a sweep reclaims it | +| Storage | Row is never written | Row is written; only a `FILTER_AND_RECLAIM` policy can reclaim it later | | Read cost | None | The keep-set filter is applied on every read of the view | -| Write cost | None | The cleanup job rewrites partitions on the `CLEANUP EVERY` cadence | -| After a full refresh | Still excluded | Re-materialized from the base, then hidden and swept again | +| Write cost | None | Cleanup can rewrite partitions for `FILTER_AND_RECLAIM` policies | +| After a full refresh | Still excluded | Re-materialized from the base, then hidden; eligible policies sweep it again | | Can be another view's base | Yes | No (a policied view is rejected as a base) | That last row is a hard constraint rather than a preference. If any other view @@ -384,8 +384,9 @@ kept rows are visible **immediately, regardless of whether cleanup has run**. This is what makes results correct at all times: - **Per-row `WHEN`** keeps rows where the predicate is not `TRUE`. QuestDB - filtering is three-valued, so `FALSE` **and** `NULL` are kept (see - [NULLs](#nulls)). + comparisons use two-valued boolean semantics, so a comparison against `NULL` + is `FALSE`. Whether the complete predicate keeps or expires a `NULL` row + depends on operators such as `NOT`, `!=`, and `IS NULL` (see [NULLs](#nulls)). - **`KEEP LATEST`** returns the latest row per key using the designated timestamp. - **`KEEP HIGHEST/LOWEST/N` and window `WHEN`** compute the keep-set with a @@ -397,15 +398,21 @@ time-based predicate) reappears on the next read. ### Physical cleanup (best-effort) -A background job reclaims disk for non-active partitions. A fully-expired -partition is removed. Under a rolling clock-based predicate, a partially-expired -partition is compacted down to its survivors only when the expired-row fraction -reaches `cairo.mat.view.row.expiry.cleanup.min.expired.fraction`, which defaults -to `0.5`. This avoids repeatedly rewriting a boundary partition as the cutoff -moves through it. Set the property to `0` to compact on the first expired row, -or to `1` to disable partial-partition compaction; fully-expired partitions are -still removed. The threshold does not delay a fixed, deterministic predicate, -whose expired-row verdicts cannot change with time. +A background job reclaims disk for non-active partitions. It never rewrites the +active logical partition that receives new rows. A young view with only one +partition therefore reclaims no disk yet, even when its policy reports +`FILTER_AND_RECLAIM`. Once data creates a newer active partition, the older one +becomes eligible for cleanup. Read filtering remains effective throughout. + +A fully-expired eligible partition is removed. Under a rolling clock-based +predicate, a partially-expired partition is compacted down to its survivors only +when the expired-row fraction reaches +`cairo.mat.view.row.expiry.cleanup.min.expired.fraction`, which defaults to +`0.5`. This avoids repeatedly rewriting a boundary partition as the cutoff moves +through it. Set the property to `0` to compact on the first expired row, or to +`1` to disable partial-partition compaction; fully-expired partitions are still +removed. The threshold does not delay a fixed, deterministic predicate, whose +expired-row verdicts cannot change with time. The job runs at the `CLEANUP EVERY` cadence (default `1h`) and is **best-effort**. The read filter is authoritative, so deferred or skipped reclamation only @@ -431,9 +438,14 @@ To observe reclamation, compare the physical row count per partition before and after a sweep: ```questdb-sql title="Physical rows still on disk per partition" -SELECT name, numRows FROM table_partitions('trades_latest'); +SELECT name, numRows FROM table_partitions('trades_recent'); ``` +Use a view whose `expire_enforcement` is `FILTER_AND_RECLAIM`, such as +`trades_recent`, for this check. Its active partition remains unchanged after a +sweep. Insert data into a newer partition before expecting the current active +partition to become eligible for reclamation. + Reclamation **defers while a view is being refreshed continuously** and resumes on a quiet sweep. @@ -441,12 +453,20 @@ on a quiet sweep. ### NULLs -The keep-set is computed with three-valued logic, so a `NULL` value is never -*less than* a group maximum (the comparison is `UNKNOWN`, not `TRUE`). -Therefore: - -- **`KEEP HIGHEST/LOWEST` and value-based `WHEN`** predicates **keep** rows whose - value is `NULL`. +QuestDB comparisons use two-valued boolean semantics: a comparison against +`NULL` evaluates to `FALSE`, not `UNKNOWN`, and `EXPIRE ROWS WHEN` expires a row +only when the complete predicate evaluates to `TRUE`. The complete predicate +therefore determines whether a `NULL` row survives: + +- **A direct comparison such as `amount < 1.5`** is `FALSE` for a `NULL` amount, + so the policy keeps the row. +- **`NOT (amount >= 1.5)`** is `TRUE` for a `NULL` amount because the inner + comparison is `FALSE`, so the policy expires the row. Although this predicate + resembles `amount < 1.5`, the two differ for `NULL` values. +- **`amount != 1.5` and `amount IS NULL`** are also `TRUE` for a `NULL` amount, + so both expire the row. +- **`KEEP HIGHEST/LOWEST`** keeps a `NULL` because its comparison against the + group extreme is `FALSE`. - **`KEEP LATEST`** uses the designated timestamp, which is never `NULL`. - **`KEEP N` is the exception.** It ranks rows with `row_number()`, and QuestDB has no `NULLS LAST`, so where a `NULL` lands is **type-dependent**: under @@ -625,8 +645,12 @@ rather than breaking subsequent reads. - **Reads recompute the keep-set.** A relative/window policy computes its keep-set over the whole physical view on every read. `KEEP LATEST` on an [indexed](/docs/concepts/deep-dive/indexes/) symbol key is cheap; the window - modes (and non-indexed keep-latest) scan the view. A tighter `CLEANUP EVERY` - keeps the physical residue, and therefore the read cost, small. + modes (and non-indexed keep-latest) scan the view. +- **Cleanup tuning applies only to reclaiming policies.** For a monotonic scalar + `WHEN` policy that reports `FILTER_AND_RECLAIM`, a tighter `CLEANUP EVERY` + reduces how long expired rows remain in eligible non-active partitions. It has + no reclamation effect on relative or window policies that report + `FILTER_ONLY`. - **Cleanup defers under continuous refresh.** Reclamation only proceeds when the view is quiescent and fully applied, so a view being refreshed continuously defers reclamation to a quiet sweep. The read filter stays authoritative From e5ea2dd41e4312e2ef1af82d05406b9d27f8b550 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Thu, 27 Aug 2026 14:26:22 +0200 Subject: [PATCH 14/25] docs: align expire rows examples and retention guidance --- documentation/concepts/expire-rows.md | 156 +++++++++--------- documentation/operations/data-retention.md | 21 ++- documentation/query/functions/meta.md | 2 +- .../query/sql/alter-mat-view-set-expire.md | 10 +- documentation/query/sql/create-mat-view.md | 13 +- documentation/query/sql/show.md | 4 +- 6 files changed, 116 insertions(+), 90 deletions(-) diff --git a/documentation/concepts/expire-rows.md b/documentation/concepts/expire-rows.md index 49dfff4df8..073f7e8021 100644 --- a/documentation/concepts/expire-rows.md +++ b/documentation/concepts/expire-rows.md @@ -27,13 +27,13 @@ on-disk storage is reclaimed afterwards by a background job under a monotonic Every mode keeps a defined set of rows and expires the rest. A row is expired only when the rule selects it for removal. -| Mode | What it keeps | Syntax | Frees disk | -| --------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | --------------------- | -| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when monotonic | -| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | No (read filter only) | -| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | -| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | -| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No (read filter only) | +| Mode | What it keeps | Syntax | Frees disk | +| --------------------- | --------------------------------------------------- | ------------------------------------------------------------------- | --------------------- | +| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when monotonic | +| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON timestamp] PARTITION BY cols` | No (read filter only) | +| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | +| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | +| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No (read filter only) | `KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a window predicate, so the window `WHEN` is the general escape hatch. @@ -146,17 +146,18 @@ exactly which rows each policy keeps. ```questdb-sql title="Base table and sample data" CREATE TABLE trades ( symbol SYMBOL, + side SYMBOL, price DOUBLE, amount DOUBLE, - ts TIMESTAMP -) TIMESTAMP(ts) PARTITION BY DAY WAL; + timestamp TIMESTAMP +) TIMESTAMP(timestamp) PARTITION BY DAY WAL; INSERT INTO trades VALUES - ('BTC', 100.0, 1.0, '2024-01-01T10:00:00.000000Z'), - ('BTC', 105.0, 2.0, '2024-01-01T11:00:00.000000Z'), - ('BTC', 102.0, 1.5, '2024-01-02T09:00:00.000000Z'), - ('ETH', 50.0, 3.0, '2024-01-01T10:30:00.000000Z'), - ('ETH', 55.0, 1.0, '2024-01-02T08:00:00.000000Z'); + ('BTC', 'buy', 100.0, 1.0, '2024-01-01T10:00:00.000000Z'), + ('BTC', 'sell', 105.0, 2.0, '2024-01-01T11:00:00.000000Z'), + ('BTC', 'buy', 102.0, 1.5, '2024-01-02T09:00:00.000000Z'), + ('ETH', 'sell', 50.0, 3.0, '2024-01-01T10:30:00.000000Z'), + ('ETH', 'buy', 55.0, 1.0, '2024-01-02T08:00:00.000000Z'); ``` :::note @@ -183,14 +184,14 @@ CREATE MATERIALIZED VIEW trades_sized AS ( SELECT * FROM trades ) EXPIRE ROWS WHEN amount < 1.5; -SELECT * FROM trades_sized ORDER BY ts; +SELECT * FROM trades_sized ORDER BY timestamp; ``` -| symbol | price | amount | ts | -| ------ | ----- | ------ | --------------------------- | -| ETH | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | -| BTC | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | -| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | +| symbol | side | price | amount | timestamp | +| ------ | ---- | ----- | ------ | --------------------------- | +| ETH | sell | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | +| BTC | sell | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | +| BTC | buy | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | The two `amount = 1.0` rows are expired. `amount = 1.5` is kept (`1.5 < 1.5` is `FALSE`), and any `NULL` amount would be kept too because a comparison against @@ -204,15 +205,15 @@ because the defining query rejects `now()`: ```questdb-sql title="Keep the last 1 day" CREATE MATERIALIZED VIEW trades_recent AS ( SELECT * FROM trades -) EXPIRE ROWS WHEN ts < dateadd('d', -1, now()); +) EXPIRE ROWS WHEN timestamp < dateadd('d', -1, now()); ``` (With the 2024 sample timestamps above, every row is already older than a day and would be hidden; use recent data to see rows retained.) -`WHEN ts < dateadd('d', -1, now())` and `WHEN ts < now() - 86400000000` retain -the same rows and both reclaim disk because the cleanup job proves either form -monotonic. +`WHEN timestamp < dateadd('d', -1, now())` and +`WHEN timestamp < now() - 86400000000` retain the same rows and both reclaim +disk because the cleanup job proves either form monotonic. ### Keep latest per key: `KEEP LATEST` @@ -224,18 +225,19 @@ CREATE MATERIALIZED VIEW trades_latest AS ( SELECT * FROM trades ) EXPIRE ROWS KEEP LATEST PARTITION BY symbol; -SELECT * FROM trades_latest ORDER BY ts; +SELECT * FROM trades_latest ORDER BY timestamp; ``` -| symbol | price | amount | ts | -| ------ | ----- | ------ | --------------------------- | -| ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | -| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | +| symbol | side | price | amount | timestamp | +| ------ | ---- | ----- | ------ | --------------------------- | +| ETH | buy | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | +| BTC | buy | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | -The designated timestamp `ts` determines the latest row for each symbol. As new -trades arrive, the kept row advances automatically. `PARTITION BY` may list -multiple key columns. You may write `KEEP LATEST ON ts PARTITION BY symbol`, but -the `ON` column must be the view's designated timestamp. +The designated `timestamp` column determines the latest row for each symbol. As +new trades arrive, the kept row advances automatically. `PARTITION BY` may list +multiple key columns. You may write +`KEEP LATEST ON timestamp PARTITION BY symbol`, but the `ON` column must be the +view's designated timestamp. ### Keep extremes per group: `KEEP HIGHEST` / `KEEP LOWEST` @@ -249,10 +251,10 @@ CREATE MATERIALIZED VIEW trades_peak AS ( SELECT * FROM trades_peak; ``` -| symbol | price | amount | ts | -| ------ | ----- | ------ | --------------------------- | -| BTC | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | -| ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | +| symbol | side | price | amount | timestamp | +| ------ | ---- | ----- | ------ | --------------------------- | +| BTC | sell | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | +| ETH | buy | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | `KEEP LOWEST price PARTITION BY symbol` keeps the cheapest instead (BTC `100.0`, ETH `50.0`). All rows **tied** at the extreme are kept, and `NULL`-valued rows @@ -270,12 +272,12 @@ CREATE MATERIALIZED VIEW trades_top2 AS ( SELECT * FROM trades_top2 ORDER BY symbol, price DESC; ``` -| symbol | price | amount | ts | -| ------ | ----- | ------ | --------------------------- | -| BTC | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | -| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | -| ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | -| ETH | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | +| symbol | side | price | amount | timestamp | +| ------ | ---- | ----- | ------ | --------------------------- | +| BTC | sell | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | +| BTC | buy | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | +| ETH | buy | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | +| ETH | sell | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | BTC keeps its two highest (`105`, `102`) and drops `100`; ETH has only two rows, so both survive. Ties are broken by the designated timestamp, so the N-th @@ -298,7 +300,7 @@ A row expires when its price is below its symbol's maximum, so only the peak per symbol survives. This produces the same result as `trades_peak` above. From here you can express richer rules, for example keeping rows within 5% of the peak (`WHEN price < 0.95 * max(price) OVER (PARTITION BY symbol)`) or a ranked window -(`WHEN row_number() OVER (PARTITION BY symbol ORDER BY ts DESC) > 100`). +(`WHEN row_number() OVER (PARTITION BY symbol ORDER BY timestamp DESC) > 100`). ## `WHERE` filter or `EXPIRE ROWS`? @@ -308,11 +310,12 @@ clock. **Use `EXPIRE ROWS WHEN` for rules that move with wall-clock time.** A rolling window cannot be written as a `WHERE` clause at all: a view's defining query -rejects non-deterministic functions, so `WHERE ts > dateadd('d', -7, now())` is -not accepted. `EXPIRE ROWS WHEN ts < dateadd('d', -7, now())` is the supported -way to say "keep the last 7 days". The read filter re-evaluates `now()` on -every read, so the window rolls forward on its own and the cleanup job reclaims -the disk behind it. This is what the `WHEN` form is for. +rejects non-deterministic functions, so +`WHERE timestamp > dateadd('d', -7, now())` is not accepted. +`EXPIRE ROWS WHEN timestamp < dateadd('d', -7, now())` is the supported way to +say "keep the last 7 days". The read filter re-evaluates `now()` on every read, +so the window rolls forward on its own and the cleanup job reclaims the disk +behind it. This is what the `WHEN` form is for. **Put a deterministic predicate in the `WHERE` clause.** A predicate that depends only on the row's own values, such as `symbol = 'BTC'` or @@ -344,7 +347,7 @@ fixes how long it keeps what it has. ```questdb-sql title="A filter for the subject, a policy for the horizon" CREATE MATERIALIZED VIEW trades_btc_recent AS ( SELECT * FROM trades WHERE symbol = 'BTC' -) EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 1h; +) EXPIRE ROWS WHEN timestamp < dateadd('d', -7, now()) CLEANUP EVERY 1h; ``` ### When a deterministic cutoff still belongs in a policy @@ -356,8 +359,10 @@ dropping the view and re-creating it, which re-materializes it from the base. Changing a policy is a metadata operation: ```questdb-sql title="Retuning a retention horizon without a rebuild" -ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-06-01T00:00:00.000000Z'; -ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-07-01T00:00:00.000000Z'; +ALTER MATERIALIZED VIEW trades_recent + SET EXPIRE ROWS WHEN timestamp < '2024-06-01T00:00:00.000000Z'; +ALTER MATERIALIZED VIEW trades_recent + SET EXPIRE ROWS WHEN timestamp < '2024-07-01T00:00:00.000000Z'; ALTER MATERIALIZED VIEW trades_recent DROP EXPIRE; ``` @@ -478,14 +483,14 @@ therefore determines whether a `NULL` row survives: ### A `NULL` threshold is rejected A `WHEN` threshold that evaluates to a constant `NULL` expires nothing because -`ts < NULL` is never `TRUE`, so the policy would be inert. QuestDB refuses it at -`CREATE` and `ALTER` time rather than storing a view that silently never -reclaims: +`timestamp < NULL` is never `TRUE`, so the policy would be inert. QuestDB +refuses it at `CREATE` and `ALTER` time rather than storing a view that silently +never reclaims: ```questdb-sql title="Rejected: the threshold is NULL" CREATE MATERIALIZED VIEW trades_recent AS ( SELECT * FROM trades -) EXPIRE ROWS WHEN ts < CAST(NULL AS TIMESTAMP); +) EXPIRE ROWS WHEN timestamp < CAST(NULL AS TIMESTAMP); -- invalid EXPIRE ROWS predicate: the threshold is NULL, so no row can ever expire ``` @@ -498,17 +503,17 @@ arithmetic threshold can land on that value: -- LONG overflow CREATE MATERIALIZED VIEW trades_recent AS ( SELECT * FROM trades -) EXPIRE ROWS WHEN ts < 4611686018427387904 * 2; +) EXPIRE ROWS WHEN timestamp < 4611686018427387904 * 2; -- INT overflow, reached three orders of magnitude sooner CREATE MATERIALIZED VIEW trades_recent AS ( SELECT * FROM trades -) EXPIRE ROWS WHEN ts < 2147483647 + 1; +) EXPIRE ROWS WHEN timestamp < 2147483647 + 1; ``` Only thresholds that are constant at definition time are checked this way. One -built from a clock, such as `ts < now() - 3600000000`, is evaluated per read and -cannot be checked in advance. +built from a clock, such as `timestamp < now() - 3600000000`, is evaluated per +read and cannot be checked in advance. ### Ties and determinism @@ -516,7 +521,7 @@ cannot be checked in advance. deterministic by construction. `KEEP N` makes the order total by appending the designated timestamp as a tiebreak, so the N-th boundary is deterministic (pair the base table with [`DEDUP UPSERT KEYS`](/docs/concepts/deduplication/) if -`(col, ts)` is not already unique). +`(col, timestamp)` is not already unique). ### Combining with TTL @@ -566,10 +571,11 @@ it can **prove** monotonic: - clock-free predicates (`WHEN amount < 1.5`), and - designated-timestamp thresholds of a proven advancing-clock shape: a bare - clock (for example, `ts < now()`), a bare clock minus a non-negative constant - (for example, `ts < now() - 7200000000`), or a fixed-unit look-back `dateadd` - on a bare clock (for example, `ts < dateadd('d', -1, now())`, with units - `s`/`m`/`h`/`d`/`w` and finer). + clock (for example, `timestamp < now()`), a bare clock minus a non-negative + constant (for example, `timestamp < now() - 7200000000`), or a fixed-unit + look-back `dateadd` on a bare clock (for example, + `timestamp < dateadd('d', -1, now())`, with units `s`/`m`/`h`/`d`/`w` and + finer). Anything else **skips cleanup**: calendar units such as `dateadd('M', -1, now())` (a month is a variable amount), look-forward offsets @@ -581,11 +587,11 @@ proven shape. :::warning -A non-monotonic predicate such as `WHEN ts > now()` expires *future* rows that -**un-expire** as `now()` advances. The read filter recomputes `now()` on every -read and stays correct, and the cleanup job skips such a policy rather than -risk physically deleting a row a later read must show. The tradeoff is that its -disk is never reclaimed. Write `WHEN` predicates that expire things in the +A non-monotonic predicate such as `WHEN timestamp > now()` expires *future* +rows that **un-expire** as `now()` advances. The read filter recomputes `now()` +on every read and stays correct, and the cleanup job skips such a policy rather +than risk physically deleting a row a later read must show. The tradeoff is that +its disk is never reclaimed. Write `WHEN` predicates that expire things in the **past** or against fixed thresholds, never rows that the passage of time will later keep. @@ -593,7 +599,9 @@ later keep. ## Inspecting a policy -`SHOW CREATE MATERIALIZED VIEW` renders the clause as written: +`SHOW CREATE MATERIALIZED VIEW` renders the policy in replayable DDL. It omits +`CLEANUP EVERY` when the cadence is the default `1h` and includes it for a +non-default cadence: ```questdb-sql SHOW CREATE MATERIALIZED VIEW trades_latest; @@ -655,9 +663,9 @@ rather than breaking subsequent reads. view is quiescent and fully applied, so a view being refreshed continuously defers reclamation to a quiet sweep. The read filter stays authoritative meanwhile. -- **`KEEP LATEST [ON ts]`.** The optional `ON ts` is accepted for familiarity but - the view's designated timestamp is always used; naming a different column is - rejected. +- **`KEEP LATEST [ON timestamp]`.** The optional `ON timestamp` is accepted for + familiarity, but the view's designated timestamp is always used; naming a + different column is rejected. - **Cleanup eligibility.** See [monotonicity and cleanup safety](#monotonicity-and-cleanup-safety), and check a view's verdict with `materialized_views().expire_enforcement`. diff --git a/documentation/operations/data-retention.md b/documentation/operations/data-retention.md index cb27786907..b35fe7f969 100644 --- a/documentation/operations/data-retention.md +++ b/documentation/operations/data-retention.md @@ -8,14 +8,21 @@ description: How to employ a data retention strategy to delete old data and save The nature of [time-series data](/blog/what-is-time-series-data/) is that the relevance of information diminishes over time. If stale data is no longer required, users can delete old data from QuestDB to either save disk space or adhere to a data retention policy. This is -achieved in QuestDB by removing data partitions from a table. +achieved by removing or tiering table partitions, or by applying row-level +retention to a materialized view. -QuestDB offers four approaches for data retention: +QuestDB offers five approaches for data retention: - **TTL** _(automatic, open source)_: Use [Time To Live (TTL)](/docs/concepts/ttl/) to automatically drop partitions when data ages beyond a specified threshold. This is the simplest approach and is available in both open source and Enterprise editions. +- **EXPIRE ROWS** _(automatic, materialized views only)_: Use + [`EXPIRE ROWS`](/docs/concepts/expire-rows/) for row-granular retention on a + materialized view, such as keeping the latest row per key, the top-N rows per + group, or expiring rows selected by a predicate. The base table is unaffected. + Some policies hide expired rows without reclaiming their disk space, so check + the policy's enforcement mode when storage reduction matters. - **Storage policy** _(automatic, Enterprise only)_: Use a [storage policy](/docs/concepts/storage-policy/) to automate the partition lifecycle — convert to Parquet locally and drop old data on a schedule. This is @@ -30,8 +37,9 @@ QuestDB offers four approaches for data retention: - **Manual**: Use `DROP PARTITION` commands as described on this page for explicit control over which partitions to remove and when. -This page provides a high-level overview of partitioning with examples to drop -data by date. For more details on partitioning, see the +This page summarizes the available retention approaches, then provides +partitioning examples that drop data by date. For more details on partitioning, +see the [partitioning](/docs/concepts/partitions/) page. ## Manual partition management @@ -109,8 +117,9 @@ syntax. Partitions may be dropped by: chronologically. Depending on the types of queries users are performing on a dataset, it may not be desirable to have gaps caused by dropped partitions. - Unlike TTL, `DROP PARTITION` commands must be triggered manually or via - external scheduling (e.g., cron jobs). For fully automated retention, consider - using [TTL](/docs/concepts/ttl/) instead. + external scheduling (e.g., cron jobs). For automated partition retention, + consider [TTL](/docs/concepts/ttl/) or a + [storage policy](/docs/concepts/storage-policy/) instead. ### Example diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index a6d726a146..b489cabf0d 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -312,7 +312,7 @@ FROM materialized_views(); | view_name | view_status | base_table_name | refresh_base_table_txn | base_table_txn | expire_clause | expire_cleanup_every | expire_enforcement | | --------------- | ----------- | --------------- | ---------------------- | -------------- | ----------------------------------- | -------------------- | ------------------ | | trades_OHLC_15m | valid | trades | 1 | 1 | null | null | null | -| trades_recent | valid | trades | 1 | 1 | ts < dateadd('d', -7, now()) | 30m | FILTER_AND_RECLAIM | +| trades_recent | valid | trades | 1 | 1 | timestamp < dateadd('d', -7, now()) | 30m | FILTER_AND_RECLAIM | | trades_latest | valid | trades | 1 | 1 | KEEP LATEST PARTITION BY symbol | 1h | FILTER_ONLY | ## memory_metrics diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index a796e2464d..56d53c6543 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -58,9 +58,17 @@ set, the policy takes effect without rebuilding the view. See ## Examples +These examples use `trades_mirror`, a passthrough materialized view over a +`trades` table with `symbol`, `side`, `price`, `amount`, and designated +`timestamp` columns: + +```questdb-sql title="Create the passthrough view" +CREATE MATERIALIZED VIEW trades_mirror AS (SELECT * FROM trades); +``` + ```questdb-sql title="Rolling 7-day window, with a tighter cleanup cadence" ALTER MATERIALIZED VIEW trades_mirror - SET EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 30m; + SET EXPIRE ROWS WHEN timestamp < dateadd('d', -7, now()) CLEANUP EVERY 30m; ``` A `WHEN` predicate is the right tool for a cutoff that moves with the clock like diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 57988083b9..efabfc9ed5 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -41,7 +41,7 @@ Where: `GROUP BY`, or a [passthrough](/docs/concepts/materialized-views/#passthrough-views) projection over a single table -- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON ts] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]`. +- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON timestamp] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]`. This policy is designed for passthrough views (see below). ## Parameters @@ -84,10 +84,11 @@ Where: ```questdb-sql title="Base table" CREATE TABLE trades ( - timestamp TIMESTAMP, symbol SYMBOL, + side SYMBOL, price DOUBLE, - amount DOUBLE + amount DOUBLE, + timestamp TIMESTAMP ) TIMESTAMP(timestamp) PARTITION BY DAY; ``` @@ -314,7 +315,7 @@ CREATE MATERIALIZED VIEW trades_latest AS ( ``` A `WHEN` predicate is for rules that move with **wall-clock time**, such as a -rolling `ts < dateadd('d', -7, now())` window. The defining query cannot +rolling `timestamp < dateadd('d', -7, now())` window. The defining query cannot express those, because it rejects non-deterministic functions. A predicate that depends only on the row's own values belongs in the query's `WHERE` clause instead, which keeps those rows out of the view entirely; see @@ -332,8 +333,8 @@ EXPIRE ROWS A `WHEN` threshold that is constant at definition time and evaluates to `NULL` is rejected, since it would expire nothing. That covers the explicit -`ts < CAST(NULL AS TIMESTAMP)` and arithmetic that overflows onto the reserved -`NULL` value, such as `ts < 2147483647 + 1`. See +`timestamp < CAST(NULL AS TIMESTAMP)` and arithmetic that overflows onto the +reserved `NULL` value, such as `timestamp < 2147483647 + 1`. See [A `NULL` threshold is rejected](/docs/concepts/expire-rows/#a-null-threshold-is-rejected). For filtering and disk-reclamation behavior, see diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index a737d8b809..197d1df8b3 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -226,8 +226,8 @@ SHOW CREATE MATERIALIZED VIEW bbo_1s; This returns the `CREATE MATERIALIZED VIEW` statement that would recreate the materialized view, including its base table, refresh strategy, partitioning, -`TTL`, and `EXPIRE ROWS` policy. When the policy specifies `CLEANUP EVERY`, the -output includes that cadence as well, so the DDL preserves the complete policy. +`TTL`, and `EXPIRE ROWS` policy. The output includes `CLEANUP EVERY` only when +the cadence differs from the default `1h`. ### SHOW CREATE TABLE From 00bdfbaf4e26e02caf7f53fb19b86049c07131ba Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Mon, 7 Sep 2026 16:33:11 +0200 Subject: [PATCH 15/25] Document materialized view LIMIT compatibility Explain the LIMIT restriction on query branches that read the base table, including nested queries, in the CREATE MATERIALIZED VIEW reference and documentation changelog. Describe the effect on recreating existing views and replaying SHOW CREATE output. Explain that applying the limit when querying the view changes which rows it stores. Limited subqueries over other tables remain allowed. Validate both files with the MDX parser, check the changelog section link, and verify the diff has no whitespace errors. --- documentation/changelog.mdx | 13 +++++++++++++ documentation/query/sql/create-mat-view.md | 22 ++++++++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 1977726f06..b3f70ce9cf 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -8,6 +8,19 @@ description: Recent updates and improvements to the QuestDB documentation. This page tracks significant updates to the QuestDB documentation. +## Unreleased + +### Compatibility + +- [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/#limit-restrictions-and-existing-definitions) - + The release introducing `EXPIRE ROWS` also rejects `LIMIT` in query branches that + read the base table, including nested queries. Existing views continue + refreshing, but recreating affected definitions, including replaying + `SHOW CREATE MATERIALIZED VIEW` output, requires removing or restructuring the + limit. To cap query results, apply `ORDER BY ... LIMIT ...` when reading the view; + moving the limit changes which rows the view stores. Limited subqueries over + other tables remain allowed. + ## August 2026 ### Releases diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index efabfc9ed5..aa9bee0258 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -71,6 +71,28 @@ Where: | PERIOD requires SAMPLE BY | The `PERIOD` clause only works with `SAMPLE BY` queries | | EVERY minimum | Minimum timer interval is `1m` | +### LIMIT restrictions and existing definitions + +`LIMIT` is not supported in query branches that read the base table, including +nested queries, for both aggregating and passthrough materialized views. An +incremental refresh evaluates the defining query over changed timestamp ranges; +a `LIMIT 100` would cap each refreshed range rather than the view as a whole. +Limited subqueries over other tables remain allowed. + +:::note Upgrading existing definitions + +Existing materialized views whose definitions contain such a `LIMIT` continue +refreshing. However, recreating them after an upgrade from a version that accepted +those definitions requires removing or restructuring the limit. This also applies +when replaying `SHOW CREATE MATERIALIZED VIEW` output during a migration or restore. + +If the limit only controls how many results an application receives, remove it +from the view definition and apply `ORDER BY ... LIMIT ...` when querying the view. +This changes which rows the view stores and does not preserve every original +definition's behavior. + +::: + ## Valid clause combinations | Refresh | DEFERRED | PERIOD | Valid | From 66f35406794275b9b182e39072ddc63a8678e628 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Wed, 9 Sep 2026 15:55:21 +0200 Subject: [PATCH 16/25] Clarify expiry conflicts with dependent views Replace the obsolete SET EXPIRE dependency rejection rule with refresh- time conflict detection for existing materialized and live views. Explain delayed invalidation, completion against earlier snapshots, and why source expiry does not retroactively filter stored dependent rows. Document FULL refresh and live-view recreation as recovery steps after resolving a source conflict. Correct the general FULL-refresh visibility and failure contract: rebuilding starts with a truncate and does not restore previous contents after a later failure. Validate the documentation with a successful Docusaurus production build. --- documentation/concepts/expire-rows.md | 64 ++++++++++++++----- documentation/concepts/live-views.md | 15 ++++- documentation/concepts/materialized-views.md | 12 ++++ .../query/sql/alter-mat-view-set-expire.md | 12 +++- documentation/query/sql/refresh-mat-view.md | 8 ++- 5 files changed, 88 insertions(+), 23 deletions(-) diff --git a/documentation/concepts/expire-rows.md b/documentation/concepts/expire-rows.md index 073f7e8021..2e02322831 100644 --- a/documentation/concepts/expire-rows.md +++ b/documentation/concepts/expire-rows.md @@ -4,8 +4,8 @@ sidebar_label: EXPIRE ROWS description: EXPIRE ROWS is a row-level retention policy for passthrough materialized views. Keep the latest row per key, the top-N per group, or rows matching a - predicate. Policies are recomputed continuously, with expired rows hidden - immediately and reclaimed in the background under the modes that allow it. + predicate. Once a policy takes effect, reads hide expired rows without waiting + for background reclamation under the modes that allow it. --- `EXPIRE ROWS` is a row-level retention policy for @@ -15,8 +15,10 @@ ROWS` decides retention **row by row**. It can keep the latest row per key, the top-N per group, rows matching a predicate, and so on. It recomputes the result continuously as the view refreshes. -Expired rows disappear from query results **immediately** in every mode. Their -on-disk storage is reclaimed afterwards by a background job under a monotonic +Once the policy takes effect, ordinary queries hide expired rows in every mode +**without waiting for physical cleanup**. This does not mean synchronous ALTER +completion or immediate invalidation of dependent views. A background job +reclaims their on-disk storage afterwards under a monotonic `WHEN` predicate; the relative modes (`KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N`) and window predicates hide rows without freeing disk. See [The modes](#the-modes) and @@ -131,10 +133,40 @@ rather than rejected: physical reclamation only sticks when base-table retention is aligned with the expiry horizon because a later incremental or full refresh can regenerate a reclaimed row from base rows that still exist. -A policied view must also stand alone: `CREATE MATERIALIZED VIEW` rejects a -defining query that reads a policied view (as its base or in a join), and -`ALTER ... SET EXPIRE` is rejected on a view that other materialized views -derive from because those views would copy expired rows on refresh. +## Dependent materialized and live views + +`CREATE MATERIALIZED VIEW` and `CREATE LIVE VIEW` reject a defining query that +reads a materialized view with an active `EXPIRE ROWS` policy, including references +through joins and subqueries. + +`SET EXPIRE` is allowed when dependent views already exist: + +```questdb-sql +-- Both views already exist. +ALTER MATERIALIZED VIEW source +SET EXPIRE ROWS WHEN v < 2; + +-- The dependent may remain active until it next refreshes. +``` + +Dependent views detect the conflict when they next refresh and become invalid. +ALTER completion and dependent invalidation are separate events: an idle +dependent may continue reporting active, and a refresh already underway may +finish against its earlier snapshot. A policy on a joined source may remain +undetected until the dependent's declared base triggers work or an operator +requests a refresh. + +Ordinary queries against the source apply its expiry filter once the policy +takes effect. The policy does not retroactively filter rows already stored in +dependent views, and invalidated views keep serving their existing contents. + +`DROP EXPIRE` does not automatically restore an invalidated dependent. After +resolving the source conflict, materialized views require +[FULL refresh](/docs/query/sql/refresh-mat-view/#full); live views require saving +the definition and [recreation](/docs/concepts/live-views/#base-table-lifecycle). +FULL refresh retains its general truncate-first failure behavior; see the linked +reference for visibility during rebuilding and failure recovery. Rebuilding can +only recover source rows that still exist. ## Worked examples @@ -329,11 +361,12 @@ at every stage. A row the `WHERE` clause excludes is never copied into the view: | Read cost | None | The keep-set filter is applied on every read of the view | | Write cost | None | Cleanup can rewrite partitions for `FILTER_AND_RECLAIM` policies | | After a full refresh | Still excluded | Re-materialized from the base, then hidden; eligible policies sweep it again | -| Can be another view's base | Yes | No (a policied view is rejected as a base) | +| Can remain a usable source for other views | Yes | No: CREATE rejects an active policy; existing dependents detect SET on refresh | -That last row is a hard constraint rather than a preference. If any other view -will read this one, the policy is not available and the predicate has to go in -the `WHERE` clause. +If this view must remain a usable source for other views, put the predicate in +the defining query's `WHERE` clause. Existing dependents do not prevent setting +an expiry policy, but they become invalid when a refresh detects it. See +[Dependent materialized and live views](#dependent-materialized-and-live-views). The two forms are not exact negations of each other on `NULL`s: `WHERE` keeps a row only when the predicate is `TRUE`, while `EXPIRE ROWS WHEN` expires a row @@ -384,9 +417,10 @@ best-effort background cleanup. ### Read-time filter (authoritative) -Every query against a policied view is transparently rewritten so that only the -kept rows are visible **immediately, regardless of whether cleanup has run**. -This is what makes results correct at all times: +Once a policy takes effect, ordinary queries against its source view apply a +read filter so that only the kept rows are visible, **regardless of whether +cleanup has run**. Dependent-view refreshes reject an active policy instead of +materializing this time-varying keep-set: - **Per-row `WHEN`** keeps rows where the predicate is not `TRUE`. QuestDB comparisons use two-valued boolean semantics, so a comparison against `NULL` diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md index 008c125ad2..7b26c7df48 100644 --- a/documentation/concepts/live-views.md +++ b/documentation/concepts/live-views.md @@ -239,14 +239,23 @@ base columns its query references: - `DROP PARTITION`, `TRUNCATE`, and base TTL eviction freeze the already-emitted rows and the view continues forward from where it was. +When a materialized-view source gains an +[`EXPIRE ROWS` policy](/docs/concepts/expire-rows/#dependent-materialized-and-live-views), +the live view detects the conflict on its next refresh and becomes invalid. +An idle view may continue reporting active, and a refresh already underway may +finish against its earlier snapshot. Detection stops further refreshes but does +not retroactively remove previously emitted rows. + An invalidated view keeps serving its existing data and reports the reason in [`live_views()`](/docs/query/functions/meta/#live_views). It stops refreshing. -Invalidation is permanent: reversing the schema change does not automatically +Invalidation is permanent: reversing the schema change or removing the source's +expiry policy does not automatically revalidate the view, and `ALTER LIVE VIEW ... RESUME WAL` only recovers a suspended WAL writer. -To recover, inspect `invalidation_reason`, repair the base-table schema, and -save the definition before dropping the view: +To recover, inspect `invalidation_reason`, resolve the source conflict (repair +the base-table schema or remove its expiry policy), and save the definition +before dropping the view: ```questdb-sql SHOW CREATE LIVE VIEW trades_ma; diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index 62c5397c5f..0a81ea6233 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -639,6 +639,18 @@ modified in incompatible ways: - Renaming the base table - `TRUNCATE` or `UPDATE` operations +An active [`EXPIRE ROWS` policy](/docs/concepts/expire-rows/#dependent-materialized-and-live-views) +on a referenced materialized view also causes invalidation when refresh detects +it. `SET EXPIRE` is allowed with existing dependents; invalidation is not +synchronous with ALTER. An idle dependent may retain its status, and an +already-running refresh may finish against its earlier snapshot. The policy does +not retroactively filter rows already stored in the dependent. + +Removing the source policy does not automatically restore an invalidated view. +Resolve the source conflict, then request a +[FULL refresh](/docs/query/sql/refresh-mat-view/#full). FULL refresh deletes the +existing contents before rebuilding; failure does not restore those contents. + Check for invalid views: ```questdb-sql title="Find invalid views" diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 56d53c6543..2f55f62264 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -56,6 +56,15 @@ an unknown column is rejected immediately rather than breaking later reads. Once set, the policy takes effect without rebuilding the view. See [How it works](/docs/concepts/expire-rows/#how-it-works). +`SET EXPIRE` is allowed when materialized or live views already depend on this +view. Those dependents detect an applied policy when they next refresh and +become invalid; ALTER completion and dependent invalidation are separate events. +An idle dependent may remain active, and an already-running refresh may finish +against its earlier snapshot. Existing dependent rows are not retroactively +filtered. Removing the policy does not automatically reverse invalidation. See +[Dependent materialized and live views](/docs/concepts/expire-rows/#dependent-materialized-and-live-views) +for consequences and recovery. + ## Examples These examples use `trades_mirror`, a passthrough materialized view over a @@ -101,7 +110,7 @@ ALTER MATERIALIZED VIEW trades_mirror DROP EXPIRE; | Aspect | Description | | ----------------------- | ---------------------------------------------------------------------------- | | Passthrough recommended | An aggregating view is accepted with a logged advisory: a later refresh can regenerate reclaimed rows, so align base-table retention with the expiry horizon | -| No dependent views | Rejected when other materialized views derive from this view (they would copy expired rows on refresh) | +| Dependent views | SET is allowed; existing materialized and live views detect the conflict and invalidate on refresh, not synchronously with ALTER | | Validation | The policy is checked against the view's columns before it is applied | | Replication | The policy and the reclamation it drives replicate as normal WAL traffic | @@ -118,7 +127,6 @@ GRANT ALTER MATERIALIZED VIEW ON trades_mirror TO user1; | Error | Cause | | ----- | ----- | | `materialized view does not exist` | View with the specified name doesn't exist | -| `cannot set an EXPIRE ROWS policy on '...': it is the base of N view(s), including '', which would copy expired rows on refresh` | Other views derive from this view; the message names the first dependent | | `EXPIRE ROWS KEEP LATEST ON must name the designated timestamp ...` | `ON` names a column other than the designated timestamp | | `invalid EXPIRE ROWS KEEP LATEST PARTITION BY column: ...` | A `PARTITION BY` key column does not exist | | `EXPIRE ROWS KEEP HIGHEST/LOWEST requires a BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, DATE, TIMESTAMP or DECIMAL column, but '' is ; use KEEP HIGHEST/LOWEST to rank an orderable column of any type` | The bare `KEEP HIGHEST/LOWEST` form was given an unsupported column type; use a supported numeric/date type or the top-N form | diff --git a/documentation/query/sql/refresh-mat-view.md b/documentation/query/sql/refresh-mat-view.md index 90f8173eeb..2a58f2e092 100644 --- a/documentation/query/sql/refresh-mat-view.md +++ b/documentation/query/sql/refresh-mat-view.md @@ -45,8 +45,10 @@ REFRESH MATERIALIZED VIEW trades_hourly FULL; - You need to rebuild after dropping partitions from the base table :::warning -Full refresh on large base tables can take significant time. The view remains -queryable during refresh but returns stale data until complete. +FULL refresh deletes existing data before rebuilding and can take significant +time on large base tables. The view remains queryable: queries can observe empty +or partially rebuilt contents as refresh transactions apply. If rebuilding fails, +the refresh does not automatically restore the previous contents. ::: ### INCREMENTAL @@ -151,7 +153,7 @@ REFRESH MATERIALIZED VIEW trades_hourly | Aspect | Description | | ------ | ----------- | | Execution | Asynchronous - command returns immediately | -| View availability | View remains queryable during refresh (returns current data) | +| View availability | View remains queryable; FULL can expose empty or partially rebuilt contents (see [FULL](#full)) | | Concurrency | Only one refresh runs at a time per view | | Cancellation | Use [`CANCEL QUERY`](/docs/query/sql/cancel-query/) to stop a long-running refresh | From 9d4d01cf1c7d516ffc40a309784bff3111fff055 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Mon, 7 Sep 2026 15:55:09 +0200 Subject: [PATCH 17/25] docs: explain restricted access to expiring materialized views --- documentation/concepts/materialized-views.md | 87 ++++++++++++++++++++ documentation/query/sql/compile-view.md | 8 ++ documentation/security/rbac.md | 17 ++++ 3 files changed, 112 insertions(+) diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index 0a81ea6233..924b03b508 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -762,6 +762,93 @@ This happens asynchronously, minimizing write performance impact. ## Enterprise features +### Restricted access with row expiry + +An `EXPIRE ROWS` policy on a passthrough materialized view filters expired rows +from queries before background cleanup removes them. In Enterprise, readers +with column-level SELECT grants also need permission on columns used to enforce +the policy, even when those columns are absent from the query's output. + +#### Direct materialized-view access + +Grant the requested columns and the policy's predicate, partition, and order +columns. For example, on a materialized view `mv` with columns `sym`, `k`, `v`, +`secret`, and designated timestamp `ts`: + +| Expiry policy | Grants needed for `SELECT sym FROM mv` | +| --- | --- | +| `WHEN v < 2.0` | `SELECT ON mv(sym, v)` | +| `WHEN ts < '2025-01-01T00:00:01.000000Z'` | `SELECT ON mv(sym)` | +| `KEEP LATEST PARTITION BY k` | `SELECT ON mv(sym, k)` | +| `KEEP HIGHEST v PARTITION BY k` | `SELECT ON mv(sym, v, k)` | + +Column-level grants implicitly include the designated timestamp. A table-level +SELECT grant covers all columns, including those needed by the policy. + +Review direct grants whenever you enable or change expiry. Granting a policy +column lets the reader query that column explicitly. If it must remain hidden, +use an ordinary SQL view as described below. Different readers can use either +access pattern. + +**COUNT limitation:** `SELECT count() FROM mv` can require SELECT on unrelated +columns as well as policy columns. With sufficient policy-column grants, use an +explicit timestamp projection to count retained rows: + +```questdb-sql +SELECT count() FROM (SELECT ts FROM mv); +``` + +This still requires the policy-column permissions. Direct COUNT also works with +a table-level SELECT grant. + +#### Hide policy columns with an ordinary view + +After configuring expiry and waiting for it to apply, an authorized administrator +can create an ordinary SQL view with only the intended output columns: + +```questdb-sql +CREATE VIEW mv_public AS (SELECT sym FROM mv); +GRANT SELECT ON mv_public TO reader; +``` + +The reader needs the appropriate connection permission, such as `PGWIRE` or +`HTTP`, and SELECT on `mv_public`. They do not need any grant on `mv` or its +policy columns. Both SELECT and COUNT through `mv_public` operate on retained +rows, and its schema exposes only `sym`. Different output column sets can use +separate ordinary views. + +#### Change expiry beneath an existing ordinary view + +Adding expiry, or replacing a policy with one that uses a new hidden column, can +make reads through an existing ordinary view fail with access denied. The view's +saved dependencies must be refreshed by reissuing its complete, unchanged +original definition: + +```questdb-sql +ALTER MATERIALIZED VIEW mv SET EXPIRE ROWS WHEN secret < 20; +SELECT wait_wal_table('mv'); +ALTER VIEW mv_public AS (SELECT sym FROM mv); +``` + +Run these statements in order as an authorized administrator, waiting for each +to complete successfully. The WAL wait ensures that the policy has been applied +before `ALTER VIEW` collects its dependencies; an ALTER acknowledgement alone, +including over the PostgreSQL protocol, does not establish application. + +The `ALTER VIEW` statement preserves existing grants on `mv_public`. Use the +original definition for your view, including any filters and output restrictions. +Readers can receive access-denied errors between policy application and the +view-definition update. Reissuing the definition before policy application does +not pick up the new dependencies. + +Background view compilation and `COMPILE VIEW` do not refresh these dependency +permissions. This procedure also applies to timestamp-only expiry when the +ordinary view predates the policy: implicit timestamp permission on a direct +materialized-view grant does not extend to an ordinary-view-only reader. + +`ALTER MATERIALIZED VIEW mv DROP EXPIRE` requires no ordinary-view repair after +it applies. Rows that have already been physically cleaned up are not restored. + ### Replicated views Replication of the base table is independent of materialized view maintenance. diff --git a/documentation/query/sql/compile-view.md b/documentation/query/sql/compile-view.md index 956501705c..f5f150a166 100644 --- a/documentation/query/sql/compile-view.md +++ b/documentation/query/sql/compile-view.md @@ -99,6 +99,14 @@ Use `COMPILE VIEW` when you want to: 4. **Diagnose issues**: Check why a view is invalid by triggering compilation errors +### Expiry-policy dependency changes + +When an expiry policy adds hidden column dependencies beneath an ordinary view, +`COMPILE VIEW` does not refresh its saved dependency permissions. In Enterprise, +restricted readers can continue to receive access-denied errors even if the view +is valid. Wait for policy application, then reissue the original definition with +`ALTER VIEW`. See the [policy-change procedure](/docs/concepts/materialized-views/#change-expiry-beneath-an-existing-ordinary-view). + ## Errors | Error | Cause | diff --git a/documentation/security/rbac.md b/documentation/security/rbac.md index f905e0b8be..da059f0f44 100644 --- a/documentation/security/rbac.md +++ b/documentation/security/rbac.md @@ -119,6 +119,23 @@ GRANT SELECT ON aapl_trades TO aapl_analyst; The user `aapl_analyst` can only see AAPL trades. They have no access to the underlying `trades` table. +### Materialized views with row expiry + +Column-restricted readers of a materialized view with `EXPIRE ROWS` need SELECT +on the requested columns and the columns used by its policy. Enabling or changing +expiry can therefore require updating direct grants. Those grants also permit +reading the policy columns explicitly. + +To keep policy columns hidden, grant access to an ordinary SQL view exposing only +the intended output. Its readers need no underlying materialized-view grants. +When expiry adds dependencies beneath an existing ordinary view, wait for WAL +application and reissue the original definition with `ALTER VIEW`; reads can be +denied until that update. `COMPILE VIEW` does not repair these dependencies. + +See [restricted access with row expiry](/docs/concepts/materialized-views/#restricted-access-with-row-expiry) +for grant examples, the COUNT limitation and workaround, and the policy-change +procedure. + ## Common scenarios ### Read-only analyst From c0473ca3a10dcfab34c1cb1ee603e99e6eae79eb Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Fri, 18 Sep 2026 16:11:47 +0200 Subject: [PATCH 18/25] Simplify language --- documentation/changelog.mdx | 28 +- documentation/concepts/expire-rows.md | 829 +++++++++--------- documentation/concepts/live-views.md | 27 +- documentation/concepts/materialized-views.md | 314 ++++--- .../configuration/materialized-views.md | 17 +- documentation/operations/data-retention.md | 25 +- documentation/query/functions/meta.md | 37 +- .../query/sql/alter-mat-view-set-expire.md | 69 +- documentation/query/sql/compile-view.md | 11 +- documentation/query/sql/create-mat-view.md | 93 +- documentation/query/sql/refresh-mat-view.md | 10 +- documentation/security/rbac.md | 21 +- 12 files changed, 757 insertions(+), 724 deletions(-) diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 9794107a65..e900505fa4 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -8,22 +8,26 @@ description: Recent updates and improvements to the QuestDB documentation. This page tracks significant updates to the QuestDB documentation. -## Unreleased - ### New -- [Expiring rows (`EXPIRE ROWS`)](/docs/concepts/expire-rows/) and [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) - New row-level retention for materialized views, with creation syntax in [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/), immediate read filtering, and background reclamation for eligible policies - -### Compatibility +- [Expiring rows (`EXPIRE ROWS`)](/docs/concepts/expire-rows/) and [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) - Keep only some rows in a materialized view. Set it when you create the view (see [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/)) or add it later. Queries hide expired rows right away, and a background job frees their disk space for the policies where that is safe. - [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/#limit-restrictions-and-existing-definitions) - - The release introducing `EXPIRE ROWS` also rejects `LIMIT` in query branches that - read the base table, including nested queries. Existing views continue - refreshing, but recreating affected definitions, including replaying - `SHOW CREATE MATERIALIZED VIEW` output, requires removing or restructuring the - limit. To cap query results, apply `ORDER BY ... LIMIT ...` when reading the view; - moving the limit changes which rows the view stores. Limited subqueries over - other tables remain allowed. + You can no longer use `LIMIT` in the part of a materialized view's query that + reads its base table (this includes a subquery on the base table). A `LIMIT` in + a subquery over a *different* table is still allowed. + + Views created before this change keep refreshing. The restriction only matters + when you **recreate** such a view — most often when you run + `SHOW CREATE MATERIALIZED VIEW` and replay its output during an upgrade, + migration, or restore. That statement is now rejected until you remove the + `LIMIT` or rewrite the query without it. + + If the `LIMIT` was only there to return fewer rows to your application, remove + it from the view definition and instead add `ORDER BY ... LIMIT ...` to the + query you run against the view. This is not an exact swap: once the `LIMIT` is + gone from the definition, the view stores all matching rows rather than a + capped set. ## September 2026 diff --git a/documentation/concepts/expire-rows.md b/documentation/concepts/expire-rows.md index 2e02322831..b2ab2ba3e6 100644 --- a/documentation/concepts/expire-rows.md +++ b/documentation/concepts/expire-rows.md @@ -1,62 +1,76 @@ --- title: Expiring rows (EXPIRE ROWS) -sidebar_label: EXPIRE ROWS +sidebar_label: Expiring rows description: - EXPIRE ROWS is a row-level retention policy for passthrough materialized - views. Keep the latest row per key, the top-N per group, or rows matching a - predicate. Once a policy takes effect, reads hide expired rows without waiting - for background reclamation under the modes that allow it. + EXPIRE ROWS is a way to keep only some rows in a passthrough materialized + view. Keep the latest row per key, the top-N per group, or rows that match a + condition. Once a policy is set, queries hide the expired rows right away, and + a background job frees their disk space later when it is safe to do so. --- -`EXPIRE ROWS` is a row-level retention policy for -[materialized views](/docs/concepts/materialized-views/). Where -[TTL](/docs/concepts/ttl/) drops whole partitions once they age out, `EXPIRE -ROWS` decides retention **row by row**. It can keep the latest row per key, the -top-N per group, rows matching a predicate, and so on. It recomputes the result -continuously as the view refreshes. +`EXPIRE ROWS` lets you keep only some of the rows in a +[materialized view](/docs/concepts/materialized-views/), and drop the rest. -Once the policy takes effect, ordinary queries hide expired rows in every mode -**without waiting for physical cleanup**. This does not mean synchronous ALTER -completion or immediate invalidation of dependent views. A background job -reclaims their on-disk storage afterwards under a monotonic -`WHEN` predicate; the relative modes (`KEEP LATEST`, `KEEP HIGHEST/LOWEST`, -`KEEP N`) and window predicates hide rows without freeing disk. See -[The modes](#the-modes) and -[Monotonicity and cleanup safety](#monotonicity-and-cleanup-safety). +[TTL](/docs/concepts/ttl/) works at the partition level: it deletes a whole +partition once it is old enough. `EXPIRE ROWS` works **one row at a time**. You +can keep the latest row for each key, the highest few rows per group, only the +rows that match a condition, and so on. QuestDB keeps that set up to date as the +view refreshes. -## The modes +Once you set a policy, queries against the view stop showing the expired rows +straight away. You do not have to wait for anything to be cleaned up on disk. +Two things to keep in mind: -Every mode keeps a defined set of rows and expires the rest. A row is expired -only when the rule selects it for removal. +- Setting a policy does not instantly invalidate other views that read this one. + That happens later, when those views next refresh. +- A background job deletes the expired rows from disk afterwards, but only for + the policies where that is safe (the plain `WHEN` condition). The other modes + (`KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N`, and window conditions) hide + the rows from queries but leave them on disk. + +See [Modes of expiry](#modes-of-expiry) and +[When expired rows are deleted from disk](#monotonicity-and-cleanup-safety). + +## Modes of expiry + +The modes come in two shapes. A `WHEN` condition describes the rows to +**expire**: a row expires when the condition is `TRUE`, and everything else is +kept. A `KEEP` mode does the opposite: it describes the rows to **keep**, and +everything else is expired. | Mode | What it keeps | Syntax | Frees disk | | --------------------- | --------------------------------------------------- | ------------------------------------------------------------------- | --------------------- | -| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when monotonic | -| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON timestamp] PARTITION BY cols` | No (read filter only) | -| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | -| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | -| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No (read filter only) | - -`KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a -window predicate, so the window `WHEN` is the general escape hatch. - -The bare `KEEP HIGHEST/LOWEST` form accepts `BYTE`, `SHORT`, `INT`, `LONG`, -`FLOAT`, `DOUBLE`, `DATE`, `TIMESTAMP` and `DECIMAL` columns. The top-N form has -a broader type surface: `KEEP N HIGHEST/LOWEST` ranks with `ORDER BY`, so it -accepts any orderable column type. For example, use `KEEP 1 HIGHEST symbol`, not -`KEEP HIGHEST symbol`, to rank a `SYMBOL` column. This changes tie and `NULL` -behavior: the top-N form keeps exactly one row per group using the designated -timestamp as a descending tiebreaker. An integer or timestamp `NULL` sorts last -and is expired, while a floating-point `NULL` sorts first. The bare form keeps -every row tied at the extreme and every `NULL`. - -For how read filtering and physical reclamation differ between modes, see -[How it works](#how-it-works). You can inspect the behavior selected for a view -through `materialized_views().expire_enforcement`; see +| Per-row condition | Rows where the condition is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when it is safe | +| Window condition | Rows where a condition using `OVER (...)` is **not** `TRUE` | `EXPIRE ROWS WHEN predicate OVER (...)` | No (hides only) | +| Keep latest | The latest row per key (the current value per key) | `EXPIRE ROWS KEEP LATEST [ON timestamp] PARTITION BY cols` | No (hides only) | +| Keep highest / lowest | Rows tied at the group's highest / lowest value | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No (hides only) | +| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No (hides only) | + +`KEEP HIGHEST/LOWEST` and `KEEP N` are shortcuts. Under the hood they turn into +a window condition, so a window `WHEN` is the general-purpose form when the +shortcuts do not fit. + +The plain `KEEP HIGHEST/LOWEST` form works on these column types: `BYTE`, +`SHORT`, `INT`, `LONG`, `FLOAT`, `DOUBLE`, `DATE`, `TIMESTAMP` and `DECIMAL`. +The top-N form (`KEEP N HIGHEST/LOWEST`) sorts rows with `ORDER BY`, so it works +on any column type you can sort. For example, to rank a `SYMBOL` column, use +`KEEP 1 HIGHEST symbol`, not `KEEP HIGHEST symbol`. + +The two forms treat ties and `NULL`s differently: + +- The top-N form keeps exactly one row per group. It breaks ties using the + designated timestamp, newest first. An integer or timestamp `NULL` sorts last, + so it is expired. A floating-point `NULL` sorts first, so it is kept. +- The plain form keeps every row tied at the highest (or lowest) value, and + keeps every `NULL`. + +To see how each mode handles hiding versus deleting on disk, see +[How it works](#how-it-works). To check which behavior a view uses, look at the +`expire_enforcement` column of `materialized_views()`; see [Inspecting a policy](#inspecting-a-policy). -The clause is attached to a passthrough `CREATE MATERIALIZED VIEW` (after the -query, and after `PARTITION BY` if present), or set later with +You attach the clause to a passthrough `CREATE MATERIALIZED VIEW` (after the +query, and after `PARTITION BY` if you use it), or add it later with [`ALTER MATERIALIZED VIEW ... SET EXPIRE ROWS`](/docs/query/sql/alter-mat-view-set-expire/): ``` @@ -69,21 +83,23 @@ EXPIRE ROWS | Element | Meaning | | ------------------ | ---------------------------------------------------------------------------------------- | -| `predicate` | Any boolean expression over the view's columns. A row expires when it evaluates `TRUE`. | -| `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp. | -| `ON timestampCol` | Optional; if given it must name the view's designated timestamp. | -| `HIGHEST\|LOWEST` | Keep rows at the max / min of `col` per group (`N` omitted), or the top `N` per group. | -| `CLEANUP EVERY` | How often the background reclamation job runs for this view: `` with unit `s`/`m`/`h`/`d`/`w`. Defaults to `1h` if omitted. | +| `predicate` | Any true/false expression over the view's columns. A row expires when it is `TRUE`. | +| `KEEP LATEST` | Keep the latest row for each `PARTITION BY` key, by the designated timestamp. | +| `ON timestampCol` | Optional. If given, it must be the view's designated timestamp. | +| `HIGHEST\|LOWEST` | Keep the rows at the highest / lowest value of `col` per group (no `N`), or the top `N`. | +| `CLEANUP EVERY` | How often the background cleanup job runs for this view: ``, where the unit is `s`/`m`/`h`/`d`/`w`. Defaults to `1h`. | :::note -`EXPIRE ROWS` is **materialized-view-only**: `CREATE TABLE ... EXPIRE ROWS` is -rejected. It is designed for a **passthrough** (non-aggregating) view, where -`SELECT * FROM base` has no `SAMPLE BY` / `GROUP BY`, the view mirrors base rows -1:1, and reclamation is permanent. An aggregating view is **accepted with a -logged advisory** because a later refresh can regenerate reclaimed rows from -base rows that still exist (see -[Requirements](#requirements)). For base-table retention use +`EXPIRE ROWS` only works on materialized views. `CREATE TABLE ... EXPIRE ROWS` +is rejected. It is built for a **passthrough** view: one that copies base rows +directly (`SELECT * FROM base` with no `SAMPLE BY` / `GROUP BY`), so each view +row matches one base row. On a passthrough view, deleting an expired row is +permanent. + +An aggregating view is **allowed, but only with a warning in the log**, because +a later refresh can rebuild a deleted row from base rows that still exist (see +[Requirements](#requirements)). To trim the base table instead, use [TTL](/docs/concepts/ttl/) or, on Enterprise, [storage policies](/docs/concepts/storage-policy/). @@ -91,86 +107,92 @@ base rows that still exist (see ## When to use EXPIRE ROWS -Reach for `EXPIRE ROWS` on a passthrough materialized view when you want a -continuously-maintained, pruned copy of a base table: +Use `EXPIRE ROWS` on a passthrough materialized view when you want a trimmed, +always-current copy of a base table: -- **Current-state-per-key tables**: keep only the latest row per device, - symbol, or session (`KEEP LATEST`). -- **Per-group extremes or leaderboards**: keep the highest/lowest value per - group, or the top-N (`KEEP HIGHEST/LOWEST`, `KEEP N`). -- **Rolling row-level windows**: keep rows newer than a moving cutoff such as - `now() - 7d`, at finer granularity than TTL's whole-partition drops +- **Current value per key**: keep only the latest row per device, symbol, or + session (`KEEP LATEST`). +- **Highest/lowest per group, or leaderboards**: keep the top value per group, + or the top-N (`KEEP HIGHEST/LOWEST`, `KEEP N`). +- **A rolling time window**: keep rows newer than a moving cutoff such as + `now() - 7d`. This is finer than TTL, which can only drop whole partitions (`WHEN predicate`). -The `WHEN` form earns its keep on predicates that involve **wall-clock time**. A -deterministic predicate depends only on the row's own values and selects the -same rows more cheaply as a `WHERE` clause in the view's defining query, which -never copies the excluded rows into the view at all. See +The `WHEN` form is most useful when the rule depends on the **current time**. If +your rule only looks at a row's own values, put it in the view's `WHERE` clause +instead. That is cheaper, because those rows are never copied into the view in +the first place. See [`WHERE` filter or `EXPIRE ROWS`?](#where-filter-or-expire-rows). -Use [TTL](/docs/concepts/ttl/) instead when partition-granularity, age-based -retention on a base table is enough. It is cheaper and has no passthrough-view -requirement. +If age-based retention on a base table, one whole partition at a time, is all you +need, use [TTL](/docs/concepts/ttl/) instead. It is cheaper and does not require +a passthrough view. ## Requirements -`EXPIRE ROWS` is designed for a **passthrough materialized view**: +`EXPIRE ROWS` is built for a **passthrough materialized view**: -- The view query keeps view rows 1:1 with base rows as a projection over a single - table, with or without a `WHERE` filter. See +- The view's query keeps one view row per base row. It reads a single table and + projects columns from it, with or without a `WHERE` filter. See [which queries are passthrough](/docs/concepts/materialized-views/#which-queries-are-passthrough) for the full rules. -- The view inherits the base table's - [designated timestamp](/docs/concepts/designated-timestamp/), partitioning and - symbol indexes. +- The view takes its + [designated timestamp](/docs/concepts/designated-timestamp/), partitioning, + and symbol indexes from the base table. -A passthrough view mirrors its base table 1:1 and refreshes incrementally, so it -is effectively a continuously-maintained replica. `EXPIRE ROWS` prunes that -replica down to the rows you want to keep without touching the base table. +Because a passthrough view mirrors its base table one-for-one and refreshes as +new data arrives, it is basically a live copy of the base table. `EXPIRE ROWS` +trims that copy down to the rows you want to keep. It never touches the base +table. -A **non-passthrough (aggregating) view is accepted with a logged advisory** -rather than rejected: physical reclamation only sticks when base-table retention -is aligned with the expiry horizon because a later incremental or full refresh -can regenerate a reclaimed row from base rows that still exist. +An **aggregating view is allowed, but you get a warning in the log** instead of +an error. Deleting rows from disk is only reliable if the base table's own +retention lines up with the expiry cutoff, because a later refresh can rebuild a +deleted row from base rows that are still there. ## Dependent materialized and live views -`CREATE MATERIALIZED VIEW` and `CREATE LIVE VIEW` reject a defining query that -reads a materialized view with an active `EXPIRE ROWS` policy, including references -through joins and subqueries. +`CREATE MATERIALIZED VIEW` and `CREATE LIVE VIEW` will not let you create a view +whose query reads a materialized view that has an active `EXPIRE ROWS` policy. +This includes reading it through a join or a subquery. -`SET EXPIRE` is allowed when dependent views already exist: +You *can* add a policy with `SET EXPIRE` even when other views already read this +one: ```questdb-sql -- Both views already exist. ALTER MATERIALIZED VIEW source SET EXPIRE ROWS WHEN v < 2; --- The dependent may remain active until it next refreshes. +-- The dependent view may keep working until it next refreshes. ``` -Dependent views detect the conflict when they next refresh and become invalid. -ALTER completion and dependent invalidation are separate events: an idle -dependent may continue reporting active, and a refresh already underway may -finish against its earlier snapshot. A policy on a joined source may remain -undetected until the dependent's declared base triggers work or an operator -requests a refresh. +A dependent view notices the conflict the next time it refreshes, and then marks +itself invalid. This does not happen at the same moment as the `ALTER`: -Ordinary queries against the source apply its expiry filter once the policy -takes effect. The policy does not retroactively filter rows already stored in -dependent views, and invalidated views keep serving their existing contents. +- A dependent view that is idle may keep reporting itself as active. +- A refresh that was already running may finish using the older data it started + with. +- If the policy is on a table reached through a join, the dependent view may not + notice until something triggers it to do work, or someone asks it to refresh. -`DROP EXPIRE` does not automatically restore an invalidated dependent. After -resolving the source conflict, materialized views require -[FULL refresh](/docs/query/sql/refresh-mat-view/#full); live views require saving -the definition and [recreation](/docs/concepts/live-views/#base-table-lifecycle). -FULL refresh retains its general truncate-first failure behavior; see the linked -reference for visibility during rebuilding and failure recovery. Rebuilding can -only recover source rows that still exist. +Queries against the source view apply the expiry filter as soon as the policy is +set. The policy does not go back and remove rows that dependent views already +stored, and an invalidated view keeps serving whatever it already has. -## Worked examples +`DROP EXPIRE` does not automatically bring an invalidated dependent back to life. +After you clear the conflict on the source, you have to rebuild the dependent: +materialized views need a +[FULL refresh](/docs/query/sql/refresh-mat-view/#full); live views need you to +save the definition and +[recreate them](/docs/concepts/live-views/#base-table-lifecycle). A FULL refresh +still deletes the view's contents before rebuilding (see the linked reference for +what queries see during the rebuild and what happens if it fails). Rebuilding +can only recover rows that still exist in the source. -The following walks through every mode on a small fixed dataset so you can see +## Examples + +These examples run every mode over one small, fixed dataset, so you can see exactly which rows each policy keeps. ### Setup @@ -185,31 +207,32 @@ CREATE TABLE trades ( ) TIMESTAMP(timestamp) PARTITION BY DAY WAL; INSERT INTO trades VALUES - ('BTC', 'buy', 100.0, 1.0, '2024-01-01T10:00:00.000000Z'), - ('BTC', 'sell', 105.0, 2.0, '2024-01-01T11:00:00.000000Z'), - ('BTC', 'buy', 102.0, 1.5, '2024-01-02T09:00:00.000000Z'), - ('ETH', 'sell', 50.0, 3.0, '2024-01-01T10:30:00.000000Z'), - ('ETH', 'buy', 55.0, 1.0, '2024-01-02T08:00:00.000000Z'); + ('BTC', 'buy', 100.0, 1.0, '2026-01-01T10:00:00.000000Z'), + ('BTC', 'sell', 105.0, 2.0, '2026-01-01T11:00:00.000000Z'), + ('BTC', 'buy', 102.0, 1.5, '2026-01-02T09:00:00.000000Z'), + ('ETH', 'sell', 50.0, 3.0, '2026-01-01T10:30:00.000000Z'), + ('ETH', 'buy', 55.0, 1.0, '2026-01-02T08:00:00.000000Z'); ``` :::note -A materialized view starts an **asynchronous initial refresh** on creation, so -it may briefly return no rows. Check progress with +A materialized view starts refreshing in the background as soon as you create +it, so it may return no rows for a moment. Check its progress with: `SELECT view_name, view_status, base_table_txn, refresh_base_table_txn FROM materialized_views();` -The view is up to date when `refresh_base_table_txn = base_table_txn`. With -this small dataset that is effectively instant. +The view is up to date when `refresh_base_table_txn = base_table_txn`. With a +dataset this small, that is basically instant. ::: -### Per-row predicate: `WHEN` +### Per-row condition: `WHEN` + +A per-row condition expires a row when it is `TRUE`. The example below expires +small trades (`amount < 1.5`), just because it makes the kept rows easy to read +off the sample data. -A per-row predicate expires a row when it evaluates `TRUE`. The example below -uses a deterministic predicate to expire small trades (`amount < 1.5`) because -it makes the keep-set easy to read off the sample data. In production -that rule belongs in the view's `WHERE` clause (`WHERE amount >= 1.5`), which -keeps those rows out of the view entirely; the rolling window further down is -the case `WHEN` exists for. +In real use, that rule belongs in the view's `WHERE` clause +(`WHERE amount >= 1.5`), which keeps those rows out of the view entirely. The +rolling window further down is the case `WHEN` is really for. ```questdb-sql title="Expire rows where amount < 1.5" CREATE MATERIALIZED VIEW trades_sized AS ( @@ -221,18 +244,18 @@ SELECT * FROM trades_sized ORDER BY timestamp; | symbol | side | price | amount | timestamp | | ------ | ---- | ----- | ------ | --------------------------- | -| ETH | sell | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | -| BTC | sell | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | -| BTC | buy | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | +| ETH | sell | 50.0 | 3.0 | 2026-01-01T10:30:00.000000Z | +| BTC | sell | 105.0 | 2.0 | 2026-01-01T11:00:00.000000Z | +| BTC | buy | 102.0 | 1.5 | 2026-01-02T09:00:00.000000Z | -The two `amount = 1.0` rows are expired. `amount = 1.5` is kept (`1.5 < 1.5` is -`FALSE`), and any `NULL` amount would be kept too because a comparison against -`NULL` evaluates to `FALSE` in QuestDB. See [NULLs](#nulls). +The two `amount = 1.0` rows are expired. `amount = 1.5` is kept, because +`1.5 < 1.5` is `FALSE`. A row with a `NULL` amount would also be kept, because +comparing anything to `NULL` gives `FALSE` in QuestDB. See [NULLs](#nulls). -A predicate on the designated timestamp gives a **rolling retention window**, -which is the main use for `WHEN`. It is re-evaluated on every read, so the -visible set rolls forward with the clock, and no `WHERE` clause can express it -because the defining query rejects `now()`: +A condition on the designated timestamp gives you a **rolling window**, which is +the main reason to use `WHEN`. QuestDB re-checks it on every read, so the visible +rows move forward with the clock. You cannot do this with a `WHERE` clause, +because the view's query is not allowed to call `now()`: ```questdb-sql title="Keep the last 1 day" CREATE MATERIALIZED VIEW trades_recent AS ( @@ -240,17 +263,18 @@ CREATE MATERIALIZED VIEW trades_recent AS ( ) EXPIRE ROWS WHEN timestamp < dateadd('d', -1, now()); ``` -(With the 2024 sample timestamps above, every row is already older than a day -and would be hidden; use recent data to see rows retained.) +(With the 2026 timestamps above, every row is already more than a day old and +would be hidden. Use recent data to see rows stay.) `WHEN timestamp < dateadd('d', -1, now())` and -`WHEN timestamp < now() - 86400000000` retain the same rows and both reclaim -disk because the cleanup job proves either form monotonic. +`WHEN timestamp < now() - 86400000000` keep the same rows, and both let the +cleanup job free disk, because QuestDB can tell that either form only moves +forward in time. ### Keep latest per key: `KEEP LATEST` -Keep only the most recent row per key to turn the passthrough view into a live, -current-state-per-symbol table: +Keep only the most recent row per key, to turn the view into a live +"current value per symbol" table: ```questdb-sql title="Keep the latest row per symbol" CREATE MATERIALIZED VIEW trades_latest AS ( @@ -262,18 +286,18 @@ SELECT * FROM trades_latest ORDER BY timestamp; | symbol | side | price | amount | timestamp | | ------ | ---- | ----- | ------ | --------------------------- | -| ETH | buy | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | -| BTC | buy | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | +| ETH | buy | 55.0 | 1.0 | 2026-01-02T08:00:00.000000Z | +| BTC | buy | 102.0 | 1.5 | 2026-01-02T09:00:00.000000Z | -The designated `timestamp` column determines the latest row for each symbol. As -new trades arrive, the kept row advances automatically. `PARTITION BY` may list -multiple key columns. You may write -`KEEP LATEST ON timestamp PARTITION BY symbol`, but the `ON` column must be the +The designated `timestamp` column decides which row is the latest for each +symbol. As new trades arrive, the kept row moves forward on its own. You can list +several key columns in `PARTITION BY`. You may also write +`KEEP LATEST ON timestamp PARTITION BY symbol`, but the `ON` column has to be the view's designated timestamp. -### Keep extremes per group: `KEEP HIGHEST` / `KEEP LOWEST` +### Keep highest/lowest per group: `KEEP HIGHEST` / `KEEP LOWEST` -Keep the rows tied at the group maximum (or minimum) of a column: +Keep the rows tied at the highest (or lowest) value of a column, per group: ```questdb-sql title="Keep the highest-priced trade per symbol" CREATE MATERIALIZED VIEW trades_peak AS ( @@ -285,16 +309,16 @@ SELECT * FROM trades_peak; | symbol | side | price | amount | timestamp | | ------ | ---- | ----- | ------ | --------------------------- | -| BTC | sell | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | -| ETH | buy | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | +| BTC | sell | 105.0 | 2.0 | 2026-01-01T11:00:00.000000Z | +| ETH | buy | 55.0 | 1.0 | 2026-01-02T08:00:00.000000Z | `KEEP LOWEST price PARTITION BY symbol` keeps the cheapest instead (BTC `100.0`, -ETH `50.0`). All rows **tied** at the extreme are kept, and `NULL`-valued rows -are kept (a `NULL` is never less than the max). +ETH `50.0`). Every row tied at the highest (or lowest) value is kept, and +`NULL`-valued rows are kept too (a `NULL` is never below the highest value). ### Keep top-N per group: `KEEP N HIGHEST` / `KEEP N LOWEST` -Keep a per-group leaderboard with the `N` highest (or lowest) rows: +Keep a per-group leaderboard: the `N` highest (or lowest) rows. ```questdb-sql title="Keep the 2 highest-priced trades per symbol" CREATE MATERIALIZED VIEW trades_top2 AS ( @@ -306,220 +330,220 @@ SELECT * FROM trades_top2 ORDER BY symbol, price DESC; | symbol | side | price | amount | timestamp | | ------ | ---- | ----- | ------ | --------------------------- | -| BTC | sell | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | -| BTC | buy | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | -| ETH | buy | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | -| ETH | sell | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | +| BTC | sell | 105.0 | 2.0 | 2026-01-01T11:00:00.000000Z | +| BTC | buy | 102.0 | 1.5 | 2026-01-02T09:00:00.000000Z | +| ETH | buy | 55.0 | 1.0 | 2026-01-02T08:00:00.000000Z | +| ETH | sell | 50.0 | 3.0 | 2026-01-01T10:30:00.000000Z | -BTC keeps its two highest (`105`, `102`) and drops `100`; ETH has only two rows, -so both survive. Ties are broken by the designated timestamp, so the N-th -boundary is deterministic. +BTC keeps its two highest (`105`, `102`) and drops `100`. ETH has only two rows, +so both stay. Ties are broken by the designated timestamp, so the cutoff at the +N-th row is always decided the same way. -### Window predicate: the escape hatch +### Window condition: the general-purpose form -`KEEP HIGHEST/LOWEST` and `KEEP N` are shorthand for window predicates. When you -need a rule they do not cover, write the window predicate directly in `WHEN`. -For example, this is exactly what `KEEP HIGHEST price PARTITION BY symbol` -expands to: +`KEEP HIGHEST/LOWEST` and `KEEP N` are just shortcuts for window conditions. When +you need a rule they do not cover, write the window condition directly in `WHEN`. +For example, this is exactly what `KEEP HIGHEST price PARTITION BY symbol` turns +into: -```questdb-sql title="Equivalent to KEEP HIGHEST, written as a window predicate" +```questdb-sql title="The same as KEEP HIGHEST, written as a window condition" CREATE MATERIALIZED VIEW trades_peak_win AS ( SELECT * FROM trades ) EXPIRE ROWS WHEN price < max(price) OVER (PARTITION BY symbol); ``` -A row expires when its price is below its symbol's maximum, so only the peak per -symbol survives. This produces the same result as `trades_peak` above. From here -you can express richer rules, for example keeping rows within 5% of the peak -(`WHEN price < 0.95 * max(price) OVER (PARTITION BY symbol)`) or a ranked window +A row expires when its price is below the highest price for its symbol, so only +the peak per symbol survives. This gives the same result as `trades_peak` above. +From here you can write richer rules, for example keep rows within 5% of the peak +(`WHEN price < 0.95 * max(price) OVER (PARTITION BY symbol)`) or keep the 100 +newest rows per symbol (`WHEN row_number() OVER (PARTITION BY symbol ORDER BY timestamp DESC) > 100`). ## `WHERE` filter or `EXPIRE ROWS`? -A passthrough view can exclude rows in two places: a `WHERE` clause in its -defining query, or an `EXPIRE ROWS WHEN` predicate. The dividing line is the -clock. - -**Use `EXPIRE ROWS WHEN` for rules that move with wall-clock time.** A rolling -window cannot be written as a `WHERE` clause at all: a view's defining query -rejects non-deterministic functions, so -`WHERE timestamp > dateadd('d', -7, now())` is not accepted. -`EXPIRE ROWS WHEN timestamp < dateadd('d', -7, now())` is the supported way to -say "keep the last 7 days". The read filter re-evaluates `now()` on every read, -so the window rolls forward on its own and the cleanup job reclaims the disk -behind it. This is what the `WHEN` form is for. - -**Put a deterministic predicate in the `WHERE` clause.** A predicate that -depends only on the row's own values, such as `symbol = 'BTC'` or -`amount >= 1.5`, describes the same surviving rows either way, so the two are -near-equivalent in what the view contains, and `WHERE` is the cheaper of the two -at every stage. A row the `WHERE` clause excludes is never copied into the view: +A passthrough view can drop rows in two places: a `WHERE` clause in its query, or +an `EXPIRE ROWS WHEN` condition. The deciding factor is whether the rule depends +on the current time. + +**Use `EXPIRE ROWS WHEN` when the rule moves with the clock.** A rolling window +cannot be a `WHERE` clause at all, because a view's query is not allowed to call +non-deterministic functions, so `WHERE timestamp > dateadd('d', -7, now())` is +rejected. `EXPIRE ROWS WHEN timestamp < dateadd('d', -7, now())` is the supported +way to say "keep the last 7 days". The filter re-checks `now()` on every read, so +the window rolls forward by itself, and the cleanup job frees the disk behind it. +This is what `WHEN` is for. + +**Put a rule that only looks at the row itself in the `WHERE` clause.** A +condition like `symbol = 'BTC'` or `amount >= 1.5` picks out the same rows either +way, so the view ends up with almost the same contents. But `WHERE` is cheaper at +every step, because a row the `WHERE` clause drops is never copied into the view: | | `WHERE` in the query | `EXPIRE ROWS WHEN` | | --- | --- | --- | -| Storage | Row is never written | Row is written; only a `FILTER_AND_RECLAIM` policy can reclaim it later | -| Read cost | None | The keep-set filter is applied on every read of the view | -| Write cost | None | Cleanup can rewrite partitions for `FILTER_AND_RECLAIM` policies | -| After a full refresh | Still excluded | Re-materialized from the base, then hidden; eligible policies sweep it again | -| Can remain a usable source for other views | Yes | No: CREATE rejects an active policy; existing dependents detect SET on refresh | - -If this view must remain a usable source for other views, put the predicate in -the defining query's `WHERE` clause. Existing dependents do not prevent setting -an expiry policy, but they become invalid when a refresh detects it. See +| Storage | Row is never written | Row is written; only a `FILTER_AND_RECLAIM` policy can delete it later | +| Read cost | None | The keep rule is applied on every read of the view | +| Write cost | None | Cleanup may rewrite partitions for `FILTER_AND_RECLAIM` policies | +| After a full refresh | Still dropped | Written again from the base, then hidden; an eligible policy sweeps it again | +| Can still be a source for other views | Yes | No: CREATE rejects an active policy; existing dependents notice a SET on refresh | + +If this view has to stay usable as a source for other views, put the rule in the +query's `WHERE` clause. Existing dependents do not stop you from adding an expiry +policy, but they become invalid when a refresh notices it. See [Dependent materialized and live views](#dependent-materialized-and-live-views). -The two forms are not exact negations of each other on `NULL`s: `WHERE` keeps a -row only when the predicate is `TRUE`, while `EXPIRE ROWS WHEN` expires a row -only when it is `TRUE`. A `NULL` amount is dropped by `WHERE amount >= 1.5` and -kept by `EXPIRE ROWS WHEN amount < 1.5`. See [NULLs](#nulls). +The two forms are not exact opposites when it comes to `NULL`s. `WHERE` keeps a +row only when the condition is `TRUE`, while `EXPIRE ROWS WHEN` expires a row +only when the condition is `TRUE`. So a `NULL` amount is dropped by +`WHERE amount >= 1.5` but kept by `EXPIRE ROWS WHEN amount < 1.5`. See +[NULLs](#nulls). -The two compose, and on a passthrough view that combination is usually the right -shape: the `WHERE` clause fixes what the view is about, and the `WHEN` policy -fixes how long it keeps what it has. +You can use both together, and on a passthrough view that is usually the right +setup: the `WHERE` clause decides what the view is about, and the `WHEN` policy +decides how long it keeps what it has. -```questdb-sql title="A filter for the subject, a policy for the horizon" +```questdb-sql title="A filter for the subject, a policy for the time window" CREATE MATERIALIZED VIEW trades_btc_recent AS ( SELECT * FROM trades WHERE symbol = 'BTC' ) EXPIRE ROWS WHEN timestamp < dateadd('d', -7, now()) CLEANUP EVERY 1h; ``` -### When a deterministic cutoff still belongs in a policy +### When a fixed cutoff still belongs in a policy -One case pulls a deterministic predicate back into `EXPIRE ROWS`: a fixed -threshold you expect to advance by hand. There is no -`ALTER MATERIALIZED VIEW ... AS `, so changing a `WHERE` clause means -dropping the view and re-creating it, which re-materializes it from the base. -Changing a policy is a metadata operation: +There is one case where a fixed (non-time-based) cutoff still belongs in +`EXPIRE ROWS`: a threshold you plan to move by hand from time to time. There is +no `ALTER MATERIALIZED VIEW ... AS `, so changing a `WHERE` clause +means dropping the view and creating it again, which rebuilds it from the base. +Changing a policy is just a metadata change: -```questdb-sql title="Retuning a retention horizon without a rebuild" +```questdb-sql title="Moving a cutoff without a rebuild" ALTER MATERIALIZED VIEW trades_recent - SET EXPIRE ROWS WHEN timestamp < '2024-06-01T00:00:00.000000Z'; + SET EXPIRE ROWS WHEN timestamp < '2026-06-01T00:00:00.000000Z'; ALTER MATERIALIZED VIEW trades_recent - SET EXPIRE ROWS WHEN timestamp < '2024-07-01T00:00:00.000000Z'; + SET EXPIRE ROWS WHEN timestamp < '2026-07-01T00:00:00.000000Z'; ALTER MATERIALIZED VIEW trades_recent DROP EXPIRE; ``` -The rebuild a `WHERE` change forces is not only slow, it can lose data: if the -base table has its own [TTL](/docs/concepts/ttl/), re-creating the view reads a -base that no longer holds everything the view held. A view whose retention -horizon is longer than its base table's cannot afford to be rebuilt, so its -cutoff belongs in a policy. +Rebuilding for a `WHERE` change is not just slow, it can lose data. If the base +table has its own [TTL](/docs/concepts/ttl/), rebuilding the view reads a base +that no longer holds everything the view held. A view that keeps data for longer +than its base table cannot afford to be rebuilt, so its cutoff belongs in a +policy. -A rule that compares rows against each other has no `WHERE` equivalent either: -`KEEP LATEST`, `KEEP N HIGHEST/LOWEST` and window predicates cannot be expressed -in the defining query, because a `LATEST ON` or a window function there makes -the view non-passthrough. +A rule that compares rows against each other has no `WHERE` equivalent either. +`KEEP LATEST`, `KEEP N HIGHEST/LOWEST`, and window conditions cannot go in the +query, because a `LATEST ON` or a window function there would stop the view from +being passthrough. ## How it works -`EXPIRE ROWS` has two cooperating parts: an authoritative read-time filter and a -best-effort background cleanup. +`EXPIRE ROWS` has two parts that work together: a read-time filter that always +applies, and a background cleanup that runs when it can. -### Read-time filter (authoritative) +### Read-time filter (always applies) -Once a policy takes effect, ordinary queries against its source view apply a -read filter so that only the kept rows are visible, **regardless of whether -cleanup has run**. Dependent-view refreshes reject an active policy instead of -materializing this time-varying keep-set: +Once a policy is set, queries against the view apply a filter so that only the +kept rows show up, **whether or not cleanup has run yet**. (Refreshes of +dependent views are the exception: they reject an active policy rather than try +to copy this moving set of rows.) -- **Per-row `WHEN`** keeps rows where the predicate is not `TRUE`. QuestDB - comparisons use two-valued boolean semantics, so a comparison against `NULL` - is `FALSE`. Whether the complete predicate keeps or expires a `NULL` row - depends on operators such as `NOT`, `!=`, and `IS NULL` (see [NULLs](#nulls)). -- **`KEEP LATEST`** returns the latest row per key using the designated +- **Per-row `WHEN`** keeps rows where the condition is not `TRUE`. In QuestDB, a + comparison against `NULL` is `FALSE`, so whether a `NULL` row is kept or + expired depends on the operators you use, such as `NOT`, `!=`, and `IS NULL` + (see [NULLs](#nulls)). +- **`KEEP LATEST`** returns the latest row per key, using the designated timestamp. -- **`KEEP HIGHEST/LOWEST/N` and window `WHEN`** compute the keep-set with a +- **`KEEP HIGHEST/LOWEST/N` and window `WHEN`** work out the kept rows with a window function over the whole view. -Because the filter is applied at query time, a freshly-refreshed row that should -be expired is hidden the moment it lands, and a row that should reappear (under a -time-based predicate) reappears on the next read. - -### Physical cleanup (best-effort) - -A background job reclaims disk for non-active partitions. It never rewrites the -active logical partition that receives new rows. A young view with only one -partition therefore reclaims no disk yet, even when its policy reports -`FILTER_AND_RECLAIM`. Once data creates a newer active partition, the older one -becomes eligible for cleanup. Read filtering remains effective throughout. - -A fully-expired eligible partition is removed. Under a rolling clock-based -predicate, a partially-expired partition is compacted down to its survivors only -when the expired-row fraction reaches -`cairo.mat.view.row.expiry.cleanup.min.expired.fraction`, which defaults to -`0.5`. This avoids repeatedly rewriting a boundary partition as the cutoff moves -through it. Set the property to `0` to compact on the first expired row, or to -`1` to disable partial-partition compaction; fully-expired partitions are still -removed. The threshold does not delay a fixed, deterministic predicate, whose -expired-row verdicts cannot change with time. - -The job runs at the `CLEANUP EVERY` cadence (default `1h`) and is **best-effort**. -The read filter is authoritative, so deferred or skipped reclamation only -affects disk usage, never query results. - -The job runs only under a monotonic `WHEN` predicate. It skips `KEEP LATEST`, -`KEEP HIGHEST/LOWEST`, `KEEP N` and window policies entirely: a later refresh -can remove the row those modes currently keep, which promotes an older row back -into the keep-set, and the job cannot reconstruct a row it has already deleted. -Those views accumulate their expired rows on disk. - -On QuestDB Enterprise, cleanup runs on the **primary only**, but the reclamation -still replicates: the compaction commits are ordinary WAL transactions, so -replicas reclaim the identical rows by applying them. A read-only replica neither -runs the job nor needs to. Disable the job with +Because this filter runs at query time, a freshly-refreshed row that should be +expired is hidden the moment it lands, and a row that should come back (under a +time-based rule) shows up again on the next read. + +### Physical cleanup (best effort) + +A background job frees disk for partitions that are no longer being written to. +It never rewrites the active partition, the one currently receiving new rows. So +a young view with only one partition frees no disk yet, even if its policy is +`FILTER_AND_RECLAIM`. Once new data creates a newer active partition, the older +one can be cleaned up. Read filtering still works the whole time. + +When a partition is fully expired and eligible, the job removes it. Under a +rolling time-based rule, a partition that is only partly expired is rewritten +down to just its surviving rows, but only once the share of expired rows reaches +`cairo.mat.view.row.expiry.cleanup.min.expired.fraction` (default `0.5`). This +avoids rewriting the same boundary partition over and over as the cutoff creeps +through it. Set the property to `0` to rewrite as soon as any row expires, or to +`1` to turn off rewriting of partly-expired partitions (fully-expired ones are +still removed). This threshold does not delay a fixed rule, whose expired rows +never change over time. + +The job runs on the `CLEANUP EVERY` schedule (default `1h`) and is best effort. +Because the read filter is what queries rely on, cleanup running late or not at +all only affects disk usage, never query results. + +The job only runs for a `WHEN` condition that QuestDB can prove only ever expires +more rows over time. It skips `KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N`, and +window policies entirely. In those modes, a later refresh can remove the row the +mode currently keeps, which would bring an older row back into the kept set, and +the job cannot bring back a row it already deleted. Those views keep their +expired rows on disk. + +On QuestDB Enterprise, cleanup runs on the **primary only**, but the freed space +still shows up on replicas. The rewrites are ordinary WAL transactions, so +replicas delete the same rows when they apply them. A read-only replica does not +run the job and does not need to. Turn the job off with `cairo.mat.view.row.expiry.cleanup.enabled=false` in `server.conf` (reads stay -filtered, but only reclamation stops; the setting does not disable `EXPIRE ROWS` +filtered; this only stops disk cleanup, it does not turn off `EXPIRE ROWS` itself). Cleanup settings are read at startup, so changing this property or the -minimum expired fraction requires a restart. A failing sweep retries after one -second, doubling the per-view retry gap up to a 10-minute cap. +minimum expired fraction needs a restart. If a cleanup pass fails, it retries +after one second, then doubles the gap for that view each time, up to a 10-minute +cap. -To observe reclamation, compare the physical row count per partition before and -after a sweep: +To watch cleanup happen, compare the number of rows per partition before and +after a pass: -```questdb-sql title="Physical rows still on disk per partition" +```questdb-sql title="Rows still on disk per partition" SELECT name, numRows FROM table_partitions('trades_recent'); ``` Use a view whose `expire_enforcement` is `FILTER_AND_RECLAIM`, such as -`trades_recent`, for this check. Its active partition remains unchanged after a -sweep. Insert data into a newer partition before expecting the current active -partition to become eligible for reclamation. +`trades_recent`, for this check. Its active partition stays the same after a +pass. Insert data into a newer partition before you expect the current active +partition to be cleaned up. -Reclamation **defers while a view is being refreshed continuously** and resumes -on a quiet sweep. +Cleanup **pauses while a view is refreshing continuously** and resumes on a quiet +pass. ## Semantics ### NULLs -QuestDB comparisons use two-valued boolean semantics: a comparison against -`NULL` evaluates to `FALSE`, not `UNKNOWN`, and `EXPIRE ROWS WHEN` expires a row -only when the complete predicate evaluates to `TRUE`. The complete predicate -therefore determines whether a `NULL` row survives: +In QuestDB, a comparison against `NULL` is `FALSE` (not "unknown"), and +`EXPIRE ROWS WHEN` expires a row only when the whole condition is `TRUE`. So the +whole condition decides whether a `NULL` row survives: -- **A direct comparison such as `amount < 1.5`** is `FALSE` for a `NULL` amount, - so the policy keeps the row. -- **`NOT (amount >= 1.5)`** is `TRUE` for a `NULL` amount because the inner - comparison is `FALSE`, so the policy expires the row. Although this predicate - resembles `amount < 1.5`, the two differ for `NULL` values. +- **A plain comparison like `amount < 1.5`** is `FALSE` for a `NULL` amount, so + the row is kept. +- **`NOT (amount >= 1.5)`** is `TRUE` for a `NULL` amount (the inner comparison + is `FALSE`), so the row is expired. Even though this looks like `amount < 1.5`, + the two behave differently for `NULL`. - **`amount != 1.5` and `amount IS NULL`** are also `TRUE` for a `NULL` amount, so both expire the row. -- **`KEEP HIGHEST/LOWEST`** keeps a `NULL` because its comparison against the - group extreme is `FALSE`. +- **`KEEP HIGHEST/LOWEST`** keeps a `NULL`, because its comparison against the + group's extreme is `FALSE`. - **`KEEP LATEST`** uses the designated timestamp, which is never `NULL`. - **`KEEP N` is the exception.** It ranks rows with `row_number()`, and QuestDB - has no `NULLS LAST`, so where a `NULL` lands is **type-dependent**: under - `DESC` a floating-point `NULL` (NaN) sorts first (kept while there is room in + has no `NULLS LAST`, so where a `NULL` lands depends on the column type. Under + `DESC`, a floating-point `NULL` (NaN) sorts first (kept while there is room in `N`), while an integer/timestamp `NULL` sorts last (expired first). Use - `KEEP HIGHEST/LOWEST` (no `N`) when every `NULL` must be kept regardless of + `KEEP HIGHEST/LOWEST` (no `N`) when every `NULL` must be kept, whatever the type. ### A `NULL` threshold is rejected -A `WHEN` threshold that evaluates to a constant `NULL` expires nothing because -`timestamp < NULL` is never `TRUE`, so the policy would be inert. QuestDB -refuses it at `CREATE` and `ALTER` time rather than storing a view that silently -never reclaims: +A `WHEN` threshold that comes out as a constant `NULL` would expire nothing, +because `timestamp < NULL` is never `TRUE`. QuestDB rejects it at `CREATE` and +`ALTER` time rather than store a view that quietly never cleans up: ```questdb-sql title="Rejected: the threshold is NULL" CREATE MATERIALIZED VIEW trades_recent AS ( @@ -528,42 +552,41 @@ CREATE MATERIALIZED VIEW trades_recent AS ( -- invalid EXPIRE ROWS predicate: the threshold is NULL, so no row can ever expire ``` -The check matters most where the `NULL` is not written down. QuestDB stores a -`NULL` `TIMESTAMP`, `LONG` or `INT` as a reserved value at the bottom of the -type's range, and integer arithmetic wraps silently when it overflows, so an -arithmetic threshold can land on that value: +This check matters most when the `NULL` is not obvious. QuestDB stores a `NULL` +`TIMESTAMP`, `LONG`, or `INT` as a special value at the very bottom of the type's +range, and integer math wraps around silently when it overflows, so an arithmetic +threshold can land right on that value: -```questdb-sql title="Also rejected: arithmetic that overflows onto NULL" +```questdb-sql title="Also rejected: math that overflows onto NULL" -- LONG overflow CREATE MATERIALIZED VIEW trades_recent AS ( SELECT * FROM trades ) EXPIRE ROWS WHEN timestamp < 4611686018427387904 * 2; --- INT overflow, reached three orders of magnitude sooner +-- INT overflow, reached a thousand times sooner CREATE MATERIALIZED VIEW trades_recent AS ( SELECT * FROM trades ) EXPIRE ROWS WHEN timestamp < 2147483647 + 1; ``` -Only thresholds that are constant at definition time are checked this way. One -built from a clock, such as `timestamp < now() - 3600000000`, is evaluated per -read and cannot be checked in advance. +Only thresholds that are constant when the view is defined are checked this way. +A threshold built from the clock, such as `timestamp < now() - 3600000000`, is +worked out on every read and cannot be checked ahead of time. ### Ties and determinism -`KEEP HIGHEST/LOWEST` keeps **all** rows tied at the max/min, making the result -deterministic by construction. `KEEP N` makes the order total by appending the -designated timestamp as a tiebreak, so the N-th boundary is deterministic (pair -the base table with [`DEDUP UPSERT KEYS`](/docs/concepts/deduplication/) if -`(col, timestamp)` is not already unique). +`KEEP HIGHEST/LOWEST` keeps **all** rows tied at the highest/lowest value, so the +result is always the same. `KEEP N` breaks ties with the designated timestamp, so +the cutoff at the N-th row is always decided the same way (pair the base table +with [`DEDUP UPSERT KEYS`](/docs/concepts/deduplication/) if `(col, timestamp)` +is not already unique). ### Combining with TTL -A view can carry a [TTL](/docs/concepts/ttl/) and an `EXPIRE ROWS` policy at the -same time, and the order is fixed: **TTL removes rows from the view first, then -the policy applies to the rows that stay.** TTL drops whole partitions from the -view's own storage as they age out, and the keep-set is computed over what -remains. +A view can have both a [TTL](/docs/concepts/ttl/) and an `EXPIRE ROWS` policy at +the same time, and the order is fixed: **TTL removes rows from the view first, +then the policy applies to what is left.** TTL drops whole partitions from the +view as they age out, and the kept set is worked out over whatever remains. ```questdb-sql title="Highest price per symbol, over a 3-day window" CREATE MATERIALIZED VIEW trades_peak_3d AS ( @@ -572,79 +595,75 @@ CREATE MATERIALIZED VIEW trades_peak_3d AS ( EXPIRE ROWS KEEP HIGHEST price PARTITION BY symbol; ``` -`TTL` goes before `EXPIRE ROWS` in the statement, as it does after any +`TTL` goes before `EXPIRE ROWS` in the statement, just as it goes after any `PARTITION BY`. -This view reports the highest price of the **last three days**, so its answer -can go **down** as the window moves: when the day holding a symbol's maximum -ages out, the next-highest price within the window takes over. That is what the -two clauses ask for together. The view is no longer "the highest price ever"; -it is "the highest price still retained". The base table is unaffected; it keeps -whatever its own retention settings keep. +This view reports the highest price of the **last three days**, so its answer can +go **down** as the window moves: when the day holding a symbol's high ages out, +the next-highest price still in the window takes over. That is what the two +clauses ask for together. The view is no longer "the highest price ever"; it is +"the highest price still kept". The base table is not affected; it keeps whatever +its own settings keep. -TTL is also the only control that bounds the size of a `KEEP LATEST`, -`KEEP HIGHEST/LOWEST` or `KEEP N` view, since the cleanup job never reclaims -disk for those modes. +TTL is also the only way to cap the size of a `KEEP LATEST`, `KEEP HIGHEST/LOWEST` +or `KEEP N` view, because the cleanup job never frees disk for those modes. ### Monotonicity and cleanup safety -Physical deletion is only safe when expiry is **monotonic**: a row that is -expired now must stay expired forever. Two separate things can break that. +Deleting rows from disk is only safe when expiry is a one-way street: a row that +is expired now must stay expired forever. Two things can break that. The relative and window modes (`KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N`, window `WHEN`) decide each row's fate by comparing it against the other rows in the view. A later refresh can remove or replace the row a key currently keeps, -which promotes an older row back into the keep-set. For this reason, the cleanup -job never deletes for these modes, whatever their predicate looks like. - -A scalar `WHEN predicate` judges each row on its own, so it is eligible. It is -arbitrary SQL. QuestDB recognizes `now()`, `now_ns()`, `sysdate()`, -`systimestamp()` and `systimestamp_ns()` as wall-clock functions, and gives each -the same monotonicity proof. The cleanup job reclaims disk only for predicates -it can **prove** monotonic: - -- clock-free predicates (`WHEN amount < 1.5`), and -- designated-timestamp thresholds of a proven advancing-clock shape: a bare - clock (for example, `timestamp < now()`), a bare clock minus a non-negative - constant (for example, `timestamp < now() - 7200000000`), or a fixed-unit - look-back `dateadd` on a bare clock (for example, +which brings an older row back into the kept set. Because of this, the cleanup job +never deletes rows for these modes, no matter what their condition looks like. + +A plain `WHEN predicate` judges each row on its own, so it can be eligible. It is +ordinary SQL. QuestDB treats `now()`, `now_ns()`, `sysdate()`, `systimestamp()`, +and `systimestamp_ns()` as clock functions and checks each the same way. The +cleanup job only frees disk for conditions it can **prove** are one-way: + +- conditions that do not use the clock (`WHEN amount < 1.5`), and +- cutoffs on the designated timestamp that only ever move forward: a bare clock + (for example `timestamp < now()`), a bare clock minus a fixed non-negative + amount (for example `timestamp < now() - 7200000000`), or a fixed-unit + look-back with `dateadd` on a bare clock (for example `timestamp < dateadd('d', -1, now())`, with units `s`/`m`/`h`/`d`/`w` and finer). -Anything else **skips cleanup**: calendar units such as -`dateadd('M', -1, now())` (a month is a variable amount), look-forward offsets -(`dateadd('h', 1, now())`), -further clock arithmetic, non-constant offsets, and arbitrary window `WHEN` -predicates. A skipped policy stays correct at read time (the filter recomputes -on every read), but its disk is not reclaimed until the policy is changed to a -proven shape. +Everything else **skips cleanup**: calendar units such as +`dateadd('M', -1, now())` (a month is not a fixed length), look-forward offsets +(`dateadd('h', 1, now())`), further clock math, offsets that are not constant, +and general window `WHEN` conditions. A skipped policy still gives correct query +results (the filter re-runs on every read), but its disk is not freed until you +change the policy to a shape QuestDB can prove. :::warning -A non-monotonic predicate such as `WHEN timestamp > now()` expires *future* -rows that **un-expire** as `now()` advances. The read filter recomputes `now()` -on every read and stays correct, and the cleanup job skips such a policy rather -than risk physically deleting a row a later read must show. The tradeoff is that -its disk is never reclaimed. Write `WHEN` predicates that expire things in the -**past** or against fixed thresholds, never rows that the passage of time will -later keep. +A condition like `WHEN timestamp > now()` expires *future* rows, which +**un-expire** as `now()` moves forward. The read filter re-checks `now()` on +every read and stays correct, and the cleanup job skips this kind of policy +rather than risk deleting a row a later read has to show. The cost is that its +disk is never freed. Write `WHEN` conditions that expire things in the **past**, +or against fixed thresholds, never rows that time will later bring back. ::: ## Inspecting a policy -`SHOW CREATE MATERIALIZED VIEW` renders the policy in replayable DDL. It omits -`CLEANUP EVERY` when the cadence is the default `1h` and includes it for a -non-default cadence: +`SHOW CREATE MATERIALIZED VIEW` shows the policy as DDL you can replay. It leaves +out `CLEANUP EVERY` when the schedule is the default `1h`, and includes it +otherwise: ```questdb-sql SHOW CREATE MATERIALIZED VIEW trades_latest; -- ... EXPIRE ROWS KEEP LATEST PARTITION BY symbol ``` -The [`materialized_views()`](/docs/query/functions/meta/) function exposes the -policy in the `expire_clause`, `expire_cleanup_every` and `expire_enforcement` -columns (all `NULL` when no policy is set): +The [`materialized_views()`](/docs/query/functions/meta/) function shows the +policy in the `expire_clause`, `expire_cleanup_every`, and `expire_enforcement` +columns (all `NULL` when there is no policy): ```questdb-sql title="List EXPIRE ROWS policies" SELECT view_name, expire_clause, expire_cleanup_every, expire_enforcement @@ -657,17 +676,17 @@ FROM materialized_views(); | trades_latest | KEEP LATEST PARTITION BY symbol | 1h | FILTER_ONLY | | trades_top2 | KEEP 2 HIGHEST price ... | 1h | FILTER_ONLY | -`expire_enforcement` is the verdict the cleanup job acts on: +`expire_enforcement` tells you what the cleanup job does: -- `FILTER_AND_RECLAIM`: reads hide the expired rows and the job deletes them - from disk. -- `FILTER_ONLY`: reads hide the expired rows and they stay on disk. Every - relative and window policy reports this, as does a `WHEN` predicate that - cannot be proven monotonic. +- `FILTER_AND_RECLAIM`: reads hide the expired rows, and the job also deletes + them from disk. +- `FILTER_ONLY`: reads hide the expired rows, but they stay on disk. Every + relative and window policy is `FILTER_ONLY`, and so is a `WHEN` condition that + QuestDB cannot prove is one-way. ## Changing or removing a policy -Set, change, or drop a policy on an existing passthrough view. See +You can set, change, or drop a policy on an existing passthrough view. See [`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/): ```questdb-sql @@ -678,41 +697,39 @@ ALTER MATERIALIZED VIEW trades_latest SET EXPIRE ROWS KEEP LATEST PARTITION BY s ALTER MATERIALIZED VIEW trades_latest DROP EXPIRE; ``` -`SET EXPIRE ROWS` validates the new policy against the view's columns before -applying it, so an invalid predicate or an unknown column is rejected up front -rather than breaking subsequent reads. +`SET EXPIRE ROWS` checks the new policy against the view's columns before +applying it, so an invalid condition or an unknown column is rejected up front +instead of breaking later reads. ## Limitations and operational notes -- **Reads recompute the keep-set.** A relative/window policy computes its - keep-set over the whole physical view on every read. `KEEP LATEST` on an +- **Reads recompute the kept set.** A relative or window policy works out its + kept set over the whole view on every read. `KEEP LATEST` on an [indexed](/docs/concepts/deep-dive/indexes/) symbol key is cheap; the window - modes (and non-indexed keep-latest) scan the view. -- **Cleanup tuning applies only to reclaiming policies.** For a monotonic scalar - `WHEN` policy that reports `FILTER_AND_RECLAIM`, a tighter `CLEANUP EVERY` - reduces how long expired rows remain in eligible non-active partitions. It has - no reclamation effect on relative or window policies that report - `FILTER_ONLY`. -- **Cleanup defers under continuous refresh.** Reclamation only proceeds when the - view is quiescent and fully applied, so a view being refreshed continuously - defers reclamation to a quiet sweep. The read filter stays authoritative - meanwhile. + modes (and keep-latest on a non-indexed column) scan the view. +- **Cleanup tuning only affects reclaiming policies.** For a `WHEN` policy that + reports `FILTER_AND_RECLAIM`, a shorter `CLEANUP EVERY` means expired rows are + removed sooner from older partitions. It does nothing for relative or window + policies that report `FILTER_ONLY`. +- **Cleanup pauses under continuous refresh.** Cleanup only runs when the view is + idle and fully up to date, so a view that is refreshing continuously defers + cleanup to a quiet pass. The read filter still works in the meantime. - **`KEEP LATEST [ON timestamp]`.** The optional `ON timestamp` is accepted for - familiarity, but the view's designated timestamp is always used; naming a - different column is rejected. -- **Cleanup eligibility.** See - [monotonicity and cleanup safety](#monotonicity-and-cleanup-safety), and check - a view's verdict with `materialized_views().expire_enforcement`. -- **Reserved column name.** The window/keep modes compute the keep-set through a - synthetic boolean column named `__qdb_re_keep`; a policy is rejected on a view - that exposes a column with that name. -- **No line comments in the clause.** The clause text is stored verbatim and - embedded into generated SQL, so `--` comments are rejected inside an - `EXPIRE ROWS` clause; terminated block comments (`/* ... */`) are fine. -- **Compacting a Parquet partition rewrites it as native storage.** When cleanup - compacts a *partially*-expired partition held in Parquet, the partition - reverts to native QuestDB storage until the Parquet-conversion job re-converts - it. Reclamation correctness is unaffected. + readability, but the view always uses its designated timestamp; naming any + other column is rejected. +- **When cleanup runs.** See + [Monotonicity and cleanup safety](#monotonicity-and-cleanup-safety), and check + a view's behavior with `materialized_views().expire_enforcement`. +- **Reserved column name.** The window/keep modes work through a hidden true/false + column named `__qdb_re_keep`, so a policy is rejected on a view that already has + a column with that name. +- **No line comments in the clause.** The clause text is stored as-is and put into + generated SQL, so `--` comments are rejected inside an `EXPIRE ROWS` clause. + Block comments (`/* ... */`) are fine. +- **Cleaning a Parquet partition rewrites it as native storage.** When cleanup + rewrites a *partly*-expired partition held in Parquet, that partition goes back + to native QuestDB storage until the Parquet-conversion job converts it again. + The row deletion itself is still correct. ## Related documentation @@ -725,5 +742,5 @@ rather than breaking subsequent reads. - [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/): set, change, or drop a policy - [Time To Live (TTL)](/docs/concepts/ttl/): partition-level retention by age -- [Storage policy](/docs/concepts/storage-policy/): graduated partition - lifecycle (Enterprise) +- [Storage policy](/docs/concepts/storage-policy/): staged partition lifecycle + (Enterprise) diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md index 81769c082b..a7a187b767 100644 --- a/documentation/concepts/live-views.md +++ b/documentation/concepts/live-views.md @@ -244,25 +244,24 @@ base columns its query references: - `DROP PARTITION`, `TRUNCATE`, and base TTL eviction freeze the already-emitted rows and the view continues forward from where it was. -When a materialized-view source gains an +If a materialized view this live view reads gets an [`EXPIRE ROWS` policy](/docs/concepts/expire-rows/#dependent-materialized-and-live-views), -the live view detects the conflict on its next refresh and becomes invalid. -An idle view may continue reporting active, and a refresh already underway may -finish against its earlier snapshot. Detection stops further refreshes but does -not retroactively remove previously emitted rows. +the live view notices the conflict on its next refresh and becomes invalid. A +live view that is idle may keep reporting itself as active, and a refresh already +running may finish with the data it started from. Once noticed, the view stops +refreshing, but it does not go back and remove rows it already produced. -An invalidated view keeps serving its existing data and reports the reason in +An invalid view keeps serving the data it already has and reports why in [`live_views()`](/docs/query/functions/meta/#live_views). It stops refreshing. -Invalidation is permanent: reversing the schema change or removing the source's -expiry policy does not automatically -revalidate the view, and `ALTER LIVE VIEW ... RESUME WAL` only recovers a -suspended WAL writer. +This is permanent: undoing the schema change or removing the source's expiry +policy does not make the view valid again, and `ALTER LIVE VIEW ... RESUME WAL` +only restarts a suspended WAL writer. -To recover, inspect `invalidation_reason` and remove the cause first: repair -the base-table schema, drop the source's `EXPIRE ROWS` policy, or raise +To recover, check `invalidation_reason` and fix the cause first: repair the +base-table schema, drop the source's `EXPIRE ROWS` policy, or raise [`cairo.live.view.refresh.memory.limit.bytes`](/docs/configuration/live-views/#cairoliveviewrefreshmemorylimitbytes) -when the reason is a memory limit breach (a recreated view picks up the new -value). Then save the definition before dropping the view: +if the reason is a memory limit breach (a recreated view picks up the new value). +Then save the definition before dropping the view: ```questdb-sql SHOW CREATE LIVE VIEW trades_ma; diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index c5dfe22414..b2781b679f 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -12,9 +12,9 @@ their results at query time, materialized views persist their data to disk, making them particularly efficient for expensive aggregate queries that are run frequently. -Most materialized views aggregate, bucketing base rows with `SAMPLE BY` or a -time-based `GROUP BY`. A view can also be a -[passthrough view](#passthrough-views), which projects base rows one-for-one +Most materialized views summarise data. They group base rows into time buckets +with `SAMPLE BY` or a time-based `GROUP BY`. A view can also be a +[passthrough view](#passthrough-views), which copies base rows one-for-one instead of summarising them. ## What are materialized views for? @@ -121,10 +121,10 @@ reads on a smaller, pre-aggregated dataset. ### Not suited for: data enrichment -Materialized views support JOINs only in an aggregating query. A -[passthrough view](#passthrough-views) keeps raw rows, but it must read a single -table. So neither shape lets you keep raw rows while adding columns from another -table. +Materialized views support JOINs, but only in a query that aggregates. A +[passthrough view](#passthrough-views) keeps raw rows, but it can only read one +table. So neither kind lets you keep raw rows and add columns from another table +at the same time. For example, joining aggregated trades with instrument metadata works: @@ -165,11 +165,10 @@ will support data enrichment use cases. Stay tuned for updates. ## Passthrough views -Not every materialized view aggregates. A **passthrough view** projects base -rows one-for-one instead of bucketing them, so the view is a -continuously-maintained copy of its base table, optionally narrowed to a subset -of columns, a subset of rows, or both. Its query has no `SAMPLE BY` and no -time-based `GROUP BY`: +Not every materialized view summarises data. A **passthrough view** copies base +rows one-for-one instead of grouping them, so the view is a live copy of its base +table. You can narrow it to some columns, some rows, or both. Its query has no +`SAMPLE BY` and no time-based `GROUP BY`: ```questdb-sql title="Passthrough view: a maintained, filtered copy of trades" CREATE MATERIALIZED VIEW trades_btc AS ( @@ -179,29 +178,27 @@ CREATE MATERIALIZED VIEW trades_btc AS ( ); ``` -Reach for one when you want a maintained *subset* of a large table rather than a -summary of it: +Use one when you want a live *slice* of a large table instead of a summary of it: -- **A narrowed replica**: one symbol, one tenant, one region, or a handful of - columns out of a wide table, kept current automatically and queried without - the base table's scan cost. -- **A row-level retention target**: attach an - [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy to keep only - some of the view's rows. The base table is left alone. +- **A narrowed copy**: one symbol, one tenant, one region, or a few columns out + of a wide table, kept up to date automatically and queried without scanning + the base table. +- **A place to apply row-level retention**: attach an + [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy to keep only some of the + view's rows. The base table is not touched. -Passthrough views refresh incrementally like any other materialized view, and -accept the same `REFRESH IMMEDIATE` (the default), `REFRESH MANUAL` and -`REFRESH EVERY` strategies. `REFRESH PERIOD` is rejected: there are no buckets -for a period to align to. +Passthrough views refresh as new data arrives, like any other materialized view, +and accept the same `REFRESH IMMEDIATE` (the default), `REFRESH MANUAL`, and +`REFRESH EVERY` strategies. `REFRESH PERIOD` is rejected, because there are no +time buckets for a period to line up with. ### Use cases -On its own, a passthrough view is a maintained copy of the base table. Add an -[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy and it becomes a -maintained *subset*: you describe which rows are worth keeping, and QuestDB -keeps that set current as new data arrives. The three examples below are drawn -from sensor telemetry and from capital markets, and each keeps a different kind -of subset. +On its own, a passthrough view is a live copy of the base table. Add an +[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy and it becomes a live +*slice*: you say which rows are worth keeping, and QuestDB keeps that set up to +date as new data arrives. The three examples below come from sensor data and +from finance, and each keeps a different kind of slice. #### IoT: The current reading from every sensor @@ -210,13 +207,13 @@ thousands of sensors, and its operations screen shows the newest reading from each one. Sensors report at their own pace. Some send a reading every second, others go -quiet for days. Against the base table that screen runs -`LATEST ON ts PARTITION BY sensor_id`, which reads backwards until it has found -a row for even the quietest sensor, and over a long history that is most of the +quiet for days. Against the base table, that screen runs +`LATEST ON ts PARTITION BY sensor_id`, which reads backwards until it finds a row +for even the quietest sensor. Over a long history, that means reading most of the table. A view that keeps only the newest row per sensor answers the same question from -a handful of rows: +just a handful of rows: ```questdb-sql title="Latest reading per sensor" CREATE MATERIALIZED VIEW sensor_current AS ( @@ -224,20 +221,20 @@ CREATE MATERIALIZED VIEW sensor_current AS ( ) EXPIRE ROWS KEEP LATEST PARTITION BY sensor_id; ``` -`sensor_current` holds one row per `sensor_id` and moves forward on its own as -readings arrive. Superseded rows stop appearing in queries but stay on disk, so -the view keeps growing at the same rate as the base table. Give it a -[TTL](/docs/concepts/ttl/) to cap its size. +`sensor_current` holds one row per `sensor_id` and updates itself as readings +arrive. Older rows stop showing up in queries but stay on disk, so the view keeps +growing at the same rate as the base table. Give it a [TTL](/docs/concepts/ttl/) +to cap its size. #### Finance: Options that have not expired yet A market maker quotes an options chain where contracts expire every Friday, and the pricing screen must never show a contract that has already expired. -That rule cannot live in the view's query. The query runs when rows are written -into the view, and it may not call `now()`, so there is no way to say "expiry is -still in the future" in a `WHERE` clause. An `EXPIRE ROWS WHEN` predicate is -evaluated on every read, which is what this case needs: +That rule cannot go in the view's query. The query runs when rows are written +into the view, and it is not allowed to call `now()`, so there is no way to say +"expiry is still in the future" in a `WHERE` clause. An `EXPIRE ROWS WHEN` +condition is checked on every read, which is exactly what this case needs: ```questdb-sql title="Options that have not expired yet" CREATE MATERIALIZED VIEW options_live AS ( @@ -245,15 +242,15 @@ CREATE MATERIALIZED VIEW options_live AS ( ) EXPIRE ROWS WHEN expiry < now(); ``` -Contracts leave `options_live` as their expiry passes. There is no job to -schedule and nothing to re-create. +Contracts drop out of `options_live` as they expire. There is no job to schedule +and nothing to recreate. -One caveat, about disk rather than about results. QuestDB deletes expired rows -only when it can tell that a row, once expired, can never qualify again. It can -tell that for a cutoff on the view's designated timestamp, which only moves -forward. `expiry` is a different column, so QuestDB takes the safe route: -expired contracts stop appearing in queries straight away, but their rows stay -on disk. Add a [TTL](/docs/concepts/ttl/) if you want that space back, and see +There is one thing to note, about disk space rather than results. QuestDB only +deletes expired rows when it can be sure a row, once expired, can never come +back. It can be sure of that for a cutoff on the view's designated timestamp, +which only moves forward. `expiry` is a different column, so QuestDB plays it +safe: expired contracts disappear from queries right away, but their rows stay on +disk. Add a [TTL](/docs/concepts/ttl/) to get that space back, and see [when expired rows are deleted from disk](/docs/concepts/expire-rows/#monotonicity-and-cleanup-safety). #### Finance: The largest trades per symbol @@ -261,9 +258,9 @@ on disk. Add a [TTL](/docs/concepts/ttl/) if you want that space back, and see A surveillance desk watches for block trades and wants the ten biggest prints for every instrument on hand at all times. -A trade is a fact that does not change once it has happened, so "the ten biggest -so far" is a set that only gets refined as larger trades arrive. That is what a -top-N policy maintains: +A trade does not change once it has happened, so "the ten biggest so far" is a +set that only gets sharper as larger trades arrive. That is what a top-N policy +keeps: ```questdb-sql title="Ten largest trades per symbol" CREATE MATERIALIZED VIEW trades_largest AS ( @@ -271,41 +268,41 @@ CREATE MATERIALIZED VIEW trades_largest AS ( ) EXPIRE ROWS KEEP 10 HIGHEST amount PARTITION BY symbol; ``` -`trades_largest` holds ten rows per symbol however far the base table grows, and -the desk reads it directly instead of ranking the base table on every query. -When an eleventh large trade arrives, the smallest of the ten drops out. Trades -tied on `amount` at the tenth place are separated by the designated timestamp, -with the newer one staying. +`trades_largest` holds ten rows per symbol no matter how large the base table +grows, and the desk reads it directly instead of ranking the base table on every +query. When an eleventh large trade arrives, the smallest of the ten drops out. +If two trades tie on `amount` at the tenth spot, the newer one stays (decided by +the designated timestamp). -Ten rows per symbol are visible, but the view still stores every base row it has -taken in, because a top-N policy never frees disk. A [TTL](/docs/concepts/ttl/) -is what bounds that, at the price of changing the question the view answers from -"the biggest so far" to "the biggest still retained". See +Ten rows per symbol show up in queries, but the view still stores every base row +it has taken in, because a top-N policy never frees disk. A +[TTL](/docs/concepts/ttl/) caps that, but it changes the question the view +answers from "the biggest so far" to "the biggest still kept". See [combining with TTL](/docs/concepts/expire-rows/#combining-with-ttl). -The ranking covers everything the view holds rather than a recent window, so -`KEEP N` fits records that stay true once written. For a value that gets -superseded later, such as a resting order that is then cancelled, `KEEP LATEST` -is the mode that tracks the current version. +The ranking covers everything the view holds, not just a recent window, so +`KEEP N` fits records that stay true once written. For a value that gets replaced +later, such as a resting order that is then cancelled, use `KEEP LATEST` instead, +which tracks the current version. ### What a passthrough view inherits -A passthrough view takes its shape from the base table instead of from a -`SAMPLE BY` clause: +A passthrough view gets its shape from the base table, not from a `SAMPLE BY` +clause: -- **Designated timestamp and partitioning** come from the base table. The - projection has to keep the designated timestamp; a query that drops it is - rejected with `materialized view query is required to have designated - timestamp`. `PARTITION BY` and `TTL` can still be stated explicitly. +- **Designated timestamp and partitioning** come from the base table. The query + has to keep the designated timestamp; a query that drops it is rejected with + `materialized view query is required to have designated timestamp`. You can + still set `PARTITION BY` and `TTL` yourself. - **Symbol indexes are inherited.** A base column declared `SYMBOL INDEX` stays - indexed in the view, under whatever alias the projection gives it, so an - indexed lookup on the view costs what it costs on the base. An aggregating - view never inherits an index, because its rows are not base rows. + indexed in the view, under whatever name the query gives it, so an indexed + lookup on the view costs the same as on the base table. An aggregating view + never inherits an index, because its rows are not base rows. ### Which queries are passthrough -The rule is that view rows stay 1:1 with base rows. A projection over a single -table qualifies, with or without a filter: +The rule is simple: each view row must match one base row. A query that reads a +single table and picks columns from it qualifies, with or without a filter: | Query | | | ----- | --- | @@ -322,23 +319,21 @@ table qualifies, with or without a filter: | `LIMIT` | Rejected | | `ORDER BY` a non-timestamp column | Rejected: the view loses its designated timestamp | -Everything in the rejected group produces output that depends on rows *other -than* the one being emitted. An incremental refresh sees only the newly-arrived -slice of the base table, so it cannot compute those correctly. A `LIMIT 100` -would admit 100 rows per refresh rather than 100 in total, and a `row_number()` -would restart within each slice. +Everything in the rejected group gives a result that depends on rows *other than* +the one being written. Each refresh only sees the rows that just arrived, so it +cannot work those out correctly. A `LIMIT 100` would let in 100 rows per refresh +rather than 100 in total, and a `row_number()` would start over on each batch. -A query that is neither passthrough nor aggregating is rejected at creation -time. Most report `materialized view query requires a sampling interval, use +A query that is neither passthrough nor aggregating is rejected when you create +it. Most give `materialized view query requires a sampling interval, use SAMPLE BY or GROUP BY timestamp_floor()`: the query looked like it meant to -aggregate but named no bucket. `LIMIT` and window functions have their own +summarise, but named no time bucket. `LIMIT` and window functions have their own messages. To keep only some of a passthrough view's rows over time, such as the latest per -key, the top-N per group, or rows matching a predicate, attach an -[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. That page also -covers -[when to put a predicate in the view's `WHERE` clause instead](/docs/concepts/expire-rows/#where-filter-or-expire-rows). +key, the top-N per group, or rows that match a condition, attach an +[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. That page also explains +[when to put a condition in the view's `WHERE` clause instead](/docs/concepts/expire-rows/#where-filter-or-expire-rows). ## Creating a materialized view @@ -398,8 +393,8 @@ interval: ### The query An aggregating materialized view uses a `SAMPLE BY` or time-based `GROUP BY` -query. (The other shape is a [passthrough view](#passthrough-views), which does -not aggregate; the rules below apply to aggregating views.) +query. (The other kind is a [passthrough view](#passthrough-views), which does +not summarise data; the rules below are for aggregating views.) **Supported:** @@ -639,29 +634,30 @@ modified in incompatible ways: - Renaming the base table - `TRUNCATE` or `UPDATE` operations -Two further conditions invalidate a view from the refresh side rather than -from a base-table change: +Two more things can invalidate a view. These come from the refresh side, not from +a base-table change: -- **Out-of-memory failures.** An incremental or scheduled refresh that fails - with an out-of-memory error, including a breach of the +- **Out-of-memory failures.** A refresh that fails with an out-of-memory error, + including going over the [refresh memory limit](/docs/configuration/cairo-engine/#memory-limits), is - retried on a timer. The view is invalidated once the - [deferred retries](/docs/configuration/materialized-views/#cairomatviewrefreshbusyretrylimit) - are exhausted. Before setting that limit, measure what a refresh needs by - running the view's query over one refresh worth of data, as described in + retried on a timer. The view is invalidated once it runs out of + [retries](/docs/configuration/materialized-views/#cairomatviewrefreshbusyretrylimit). + Before setting that limit, measure what a refresh needs by running the view's + query over one refresh worth of data, as described in [Sizing a limit](/docs/configuration/cairo-engine/#sizing-a-limit). -- **An `EXPIRE ROWS` policy on a source view.** A refresh that reads a - materialized view with an active - [`EXPIRE ROWS` policy](/docs/concepts/expire-rows/#dependent-materialized-and-live-views) - invalidates the dependent view as soon as it detects the policy. `SET EXPIRE` - is allowed while dependents exist, and the invalidation is not synchronous - with the `ALTER`: an idle dependent keeps its current status until it next - refreshes, and a refresh already underway finishes against its earlier - snapshot. The policy does not retroactively filter rows already stored in the - dependent, which keeps serving them. - -In every case, invalidation is sticky: reverting the base-table change, freeing -memory, or dropping the source policy does not revalidate the view on its own. +- **An `EXPIRE ROWS` policy on a source view.** If a view reads a materialized + view that has an active + [`EXPIRE ROWS` policy](/docs/concepts/expire-rows/#dependent-materialized-and-live-views), + its next refresh notices the policy and invalidates the view. You are allowed + to run `SET EXPIRE` while dependents exist, and the invalidation does not + happen at the same moment as the `ALTER`: an idle dependent keeps its current + status until it next refreshes, and a refresh already running finishes with the + data it started from. The policy does not go back and remove rows the dependent + already stored, so the dependent keeps serving them. + +In every case, invalidation sticks: undoing the base-table change, freeing +memory, or dropping the source policy does not make the view valid again on its +own. Check for invalid views: @@ -673,13 +669,13 @@ WHERE view_status = 'invalid'; ### Refreshing an invalid view -Restore an invalid view with a full refresh. Remove the cause first, because the +Bring an invalid view back with a full refresh. Fix the cause first, because a full refresh runs the same query under the same conditions: if `invalidation_reason` reports a memory limit breach, raise [`cairo.mat.view.refresh.memory.limit.bytes`](/docs/configuration/cairo-engine/#memory-limits); -if it reports an `EXPIRE ROWS` conflict, drop the policy on the source view. -A full refresh that still hits the conflict fails before touching the view's -contents, but a failure later in the rebuild does not restore them. +if it reports an `EXPIRE ROWS` conflict, drop the policy on the source view. A +full refresh that still hits the conflict fails before it touches the view's +contents, but a failure later in the rebuild leaves the view empty. ```questdb-sql REFRESH MATERIALIZED VIEW view_name FULL; @@ -782,65 +778,65 @@ This happens asynchronously, minimizing write performance impact. ### Restricted access with row expiry -An `EXPIRE ROWS` policy on a passthrough materialized view filters expired rows -from queries before background cleanup removes them. In Enterprise, readers -with column-level SELECT grants also need permission on columns used to enforce -the policy, even when those columns are absent from the query's output. +An `EXPIRE ROWS` policy on a passthrough materialized view hides expired rows +from queries before the background job removes them from disk. In Enterprise, if +a reader has column-level SELECT grants, they also need permission on the columns +the policy uses, even if those columns are not in the query's output. #### Direct materialized-view access -Grant the requested columns and the policy's predicate, partition, and order -columns. For example, on a materialized view `mv` with columns `sym`, `k`, `v`, -`secret`, and designated timestamp `ts`: +Grant the columns the reader asks for, plus the columns the policy uses in its +condition, its `PARTITION BY`, and its ordering. For example, on a materialized +view `mv` with columns `sym`, `k`, `v`, `secret`, and designated timestamp `ts`: | Expiry policy | Grants needed for `SELECT sym FROM mv` | | --- | --- | | `WHEN v < 2.0` | `SELECT ON mv(sym, v)` | -| `WHEN ts < '2025-01-01T00:00:01.000000Z'` | `SELECT ON mv(sym)` | +| `WHEN ts < '2026-01-01T00:00:01.000000Z'` | `SELECT ON mv(sym)` | | `KEEP LATEST PARTITION BY k` | `SELECT ON mv(sym, k)` | | `KEEP HIGHEST v PARTITION BY k` | `SELECT ON mv(sym, v, k)` | -Column-level grants implicitly include the designated timestamp. A table-level -SELECT grant covers all columns, including those needed by the policy. +A column-level grant always includes the designated timestamp. A table-level +SELECT grant covers every column, including the ones the policy needs. -Review direct grants whenever you enable or change expiry. Granting a policy -column lets the reader query that column explicitly. If it must remain hidden, -use an ordinary SQL view as described below. Different readers can use either -access pattern. +Review these grants whenever you turn expiry on or change it. Granting a policy +column also lets the reader query that column directly. If a column must stay +hidden, use an ordinary SQL view instead, as shown below. Different readers can +use whichever approach fits. -**COUNT limitation:** `SELECT count() FROM mv` can require SELECT on unrelated -columns as well as policy columns. With sufficient policy-column grants, use an -explicit timestamp projection to count retained rows: +**COUNT limitation:** `SELECT count() FROM mv` may need SELECT on unrelated +columns as well as policy columns. If you have the policy-column grants, count +the kept rows by selecting the timestamp explicitly: ```questdb-sql SELECT count() FROM (SELECT ts FROM mv); ``` -This still requires the policy-column permissions. Direct COUNT also works with -a table-level SELECT grant. +This still needs the policy-column permissions. A plain `count()` also works if +the reader has a table-level SELECT grant. #### Hide policy columns with an ordinary view -After configuring expiry and waiting for it to apply, an authorized administrator -can create an ordinary SQL view with only the intended output columns: +Once expiry is set up and has taken effect, an administrator can create an +ordinary SQL view that exposes only the columns readers should see: ```questdb-sql CREATE VIEW mv_public AS (SELECT sym FROM mv); GRANT SELECT ON mv_public TO reader; ``` -The reader needs the appropriate connection permission, such as `PGWIRE` or -`HTTP`, and SELECT on `mv_public`. They do not need any grant on `mv` or its -policy columns. Both SELECT and COUNT through `mv_public` operate on retained -rows, and its schema exposes only `sym`. Different output column sets can use -separate ordinary views. +The reader needs a connection permission, such as `PGWIRE` or `HTTP`, and SELECT +on `mv_public`. They need no grant on `mv` or its policy columns. Both SELECT and +`count()` through `mv_public` see only the kept rows, and the view exposes only +`sym`. If you want to expose different sets of columns, create a separate view +for each. #### Change expiry beneath an existing ordinary view Adding expiry, or replacing a policy with one that uses a new hidden column, can -make reads through an existing ordinary view fail with access denied. The view's -saved dependencies must be refreshed by reissuing its complete, unchanged -original definition: +make reads through an existing ordinary view fail with "access denied". You have +to refresh the view's saved dependencies by running its complete, unchanged +original definition again: ```questdb-sql ALTER MATERIALIZED VIEW mv SET EXPIRE ROWS WHEN secret < 20; @@ -848,24 +844,26 @@ SELECT wait_wal_table('mv'); ALTER VIEW mv_public AS (SELECT sym FROM mv); ``` -Run these statements in order as an authorized administrator, waiting for each -to complete successfully. The WAL wait ensures that the policy has been applied -before `ALTER VIEW` collects its dependencies; an ALTER acknowledgement alone, -including over the PostgreSQL protocol, does not establish application. +Run these statements in order as an administrator, and wait for each one to +finish. The WAL wait makes sure the policy has taken effect before `ALTER VIEW` +collects its dependencies. An `ALTER` acknowledgement on its own, including over +the PostgreSQL protocol, does not mean the policy has taken effect yet. -The `ALTER VIEW` statement preserves existing grants on `mv_public`. Use the -original definition for your view, including any filters and output restrictions. -Readers can receive access-denied errors between policy application and the -view-definition update. Reissuing the definition before policy application does +`ALTER VIEW` keeps the existing grants on `mv_public`. Use the view's original +definition, including any filters and column restrictions. Readers may get +"access denied" errors in the gap between the policy taking effect and the +`ALTER VIEW`. Running the definition again *before* the policy takes effect does not pick up the new dependencies. Background view compilation and `COMPILE VIEW` do not refresh these dependency -permissions. This procedure also applies to timestamp-only expiry when the -ordinary view predates the policy: implicit timestamp permission on a direct -materialized-view grant does not extend to an ordinary-view-only reader. - -`ALTER MATERIALIZED VIEW mv DROP EXPIRE` requires no ordinary-view repair after -it applies. Rows that have already been physically cleaned up are not restored. +permissions. This also applies to timestamp-only expiry when the ordinary view +existed before the policy: the automatic timestamp permission on a direct +materialized-view grant does not carry over to a reader who only has the ordinary +view. + +After `ALTER MATERIALIZED VIEW mv DROP EXPIRE` takes effect, you do not need to +repair the ordinary view. Rows that have already been deleted from disk are not +brought back. ### Replicated views diff --git a/documentation/configuration/materialized-views.md b/documentation/configuration/materialized-views.md index e621a6ca7c..46a3193d8b 100644 --- a/documentation/configuration/materialized-views.md +++ b/documentation/configuration/materialized-views.md @@ -6,7 +6,7 @@ description: --- These settings control materialized view SQL support, background refresh, and -row-expiry cleanup. Materialized views can use dedicated worker threads or share +row-expiry cleanup. Materialized views can use their own worker threads or share the server's common pool. To cap the native memory a single refresh may allocate, see @@ -75,19 +75,20 @@ has no effect; deferred out-of-memory retries use this backoff. - **Default**: `true` - **Reloadable**: no -Enables the background job that reclaims rows removed by an eligible -[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. Disabling the job does not -disable read filtering, so expired rows remain hidden from query results. +Turns on the background job that frees disk for rows removed by an eligible +[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. Turning the job off does not +turn off read filtering, so expired rows stay hidden from query results either +way. ## cairo.mat.view.row.expiry.cleanup.min.expired.fraction - **Default**: `0.5` - **Reloadable**: no -Minimum fraction of expired rows required before the background cleanup job -compacts a partially expired partition. Set this property to `0` to compact on -the first expired row, or to `1` to disable partial-partition compaction. Fully -expired partitions are still removed. +How large the share of expired rows in a partition must be before the background +cleanup job rewrites that partly-expired partition to reclaim the space. Set this +to `0` to rewrite as soon as any row expires, or to `1` to turn off rewriting of +partly-expired partitions. Fully expired partitions are still removed. ## mat.view.refresh.worker.affinity diff --git a/documentation/operations/data-retention.md b/documentation/operations/data-retention.md index b35fe7f969..94bfe4c300 100644 --- a/documentation/operations/data-retention.md +++ b/documentation/operations/data-retention.md @@ -5,11 +5,11 @@ description: How to employ a data retention strategy to delete old data and save ## Background -The nature of [time-series data](/blog/what-is-time-series-data/) is that the relevance of information diminishes -over time. If stale data is no longer required, users can delete old data from -QuestDB to either save disk space or adhere to a data retention policy. This is -achieved by removing or tiering table partitions, or by applying row-level -retention to a materialized view. +With [time-series data](/blog/what-is-time-series-data/), older data usually +matters less over time. When you no longer need old data, you can delete it from +QuestDB to save disk space or to meet a data retention policy. You do this by +removing or tiering table partitions, or by applying row-level retention to a +materialized view. QuestDB offers five approaches for data retention: @@ -18,11 +18,11 @@ QuestDB offers five approaches for data retention: data ages beyond a specified threshold. This is the simplest approach and is available in both open source and Enterprise editions. - **EXPIRE ROWS** _(automatic, materialized views only)_: Use - [`EXPIRE ROWS`](/docs/concepts/expire-rows/) for row-granular retention on a - materialized view, such as keeping the latest row per key, the top-N rows per - group, or expiring rows selected by a predicate. The base table is unaffected. - Some policies hide expired rows without reclaiming their disk space, so check - the policy's enforcement mode when storage reduction matters. + [`EXPIRE ROWS`](/docs/concepts/expire-rows/) to keep only some rows in a + materialized view, such as the latest row per key, the top-N rows per group, + or rows matching a condition. The base table is not touched. Some policies hide + expired rows without freeing their disk space, so check the policy's + enforcement mode if you care about reclaiming storage. - **Storage policy** _(automatic, Enterprise only)_: Use a [storage policy](/docs/concepts/storage-policy/) to automate the partition lifecycle — convert to Parquet locally and drop old data on a schedule. This is @@ -37,9 +37,8 @@ QuestDB offers five approaches for data retention: - **Manual**: Use `DROP PARTITION` commands as described on this page for explicit control over which partitions to remove and when. -This page summarizes the available retention approaches, then provides -partitioning examples that drop data by date. For more details on partitioning, -see the +This page lists the retention options, then shows partitioning examples that drop +data by date. For more on partitioning, see the [partitioning](/docs/concepts/partitions/) page. ## Manual partition management diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index 5fe72ed25c..fc0ba3decf 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -312,29 +312,29 @@ Returns a `table` including the following information: - `period_delay` - delay before a completed period is refreshed (`PERIOD (DELAY ...)`) - `period_delay_unit` - how long each period-delay unit is -- `refresh_avg_commit_nanos` - moving average of one refresh commit, in - nanoseconds. Held in memory only, so it resets on restart -- `refresh_avg_scan_sample_nanos` - moving average of one refresh iteration's - base-table scan, in nanoseconds. In memory only -- `refresh_avg_scan_range_ts_units` - moving average of the timestamp range one - refresh iteration covers, in the base table's timestamp unit. In memory only -- `refresh_gap_threshold_ts_units` - timestamp gap below which the refresh job - merges two adjacent intervals instead of paying for a second commit. `0` means - merging is disabled +- `refresh_avg_commit_nanos` - recent average time for one refresh commit, in + nanoseconds. Kept in memory only, so it resets on restart +- `refresh_avg_scan_sample_nanos` - recent average time one refresh spends + scanning the base table, in nanoseconds. In memory only +- `refresh_avg_scan_range_ts_units` - recent average size of the time range one + refresh covers, in the base table's timestamp unit. In memory only +- `refresh_gap_threshold_ts_units` - if two time ranges to refresh are closer + than this, the refresh job handles them together instead of committing twice. + `0` turns this off - `expire_clause` - the view's - [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy as written, or - `NULL` when the view has no policy + [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy as written, or `NULL` if + the view has no policy - `expire_cleanup_every` - how often the cleanup job runs for the policy, or `NULL` -- `expire_enforcement` - `FILTER_AND_RECLAIM` when the cleanup job frees disk - for the policy, `FILTER_ONLY` when reads hide the expired rows but they stay - on disk, `NULL` when the view has no policy +- `expire_enforcement` - `FILTER_AND_RECLAIM` if the cleanup job also frees disk + for the policy, `FILTER_ONLY` if reads hide the expired rows but they stay on + disk, `NULL` if the view has no policy **Examples:** `materialized_views()` on its own returns every column listed above. The example -below projects a readable subset over three views: an aggregating one with no -retention policy, and two passthrough views with an +below picks a readable subset of columns over three views: an aggregating one +with no retention policy, and two passthrough views with an [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. ```questdb-sql title="List all materialized views" @@ -1014,9 +1014,8 @@ On primary instances, these columns will be `0` or `false`. | `expire_clause` | STRING | Materialized view's `EXPIRE ROWS` policy as written, or `null` | | `expire_cleanup_every` | STRING | Cleanup cadence for the policy, or `null` | -Unlike `materialized_views()`, `tables()` does not expose -`expire_enforcement`; use `materialized_views()` to inspect whether a policy is -`FILTER_AND_RECLAIM` or `FILTER_ONLY`. +`tables()` does not include `expire_enforcement`. To check whether a policy is +`FILTER_AND_RECLAIM` or `FILTER_ONLY`, use `materialized_views()` instead. ### Data precision and limitations diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 2f55f62264..4450064276 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -7,10 +7,9 @@ description: --- Sets, replaces, or removes an [`EXPIRE ROWS`](/docs/concepts/expire-rows/) -row-retention policy on a materialized view. It is designed for -**passthrough views**; see the concept page. For filtering and disk-reclamation -behavior, see -[How `EXPIRE ROWS` works](/docs/concepts/expire-rows/#how-it-works). +row-retention policy on a materialized view. It is meant for **passthrough +views**; see the concept page. For how expired rows are filtered and freed from +disk, see [How `EXPIRE ROWS` works](/docs/concepts/expire-rows/#how-it-works). ## Syntax @@ -29,16 +28,16 @@ ALTER MATERIALIZED VIEW viewName DROP EXPIRE | Parameter | Description | | ---------------- | --------------------------------------------------------------------------------- | | `viewName` | Name of the passthrough materialized view to modify | -| `WHEN predicate` | Per-row (or window) predicate; a row expires when it evaluates `TRUE` | +| `WHEN predicate` | A per-row (or window) condition; a row expires when it is `TRUE` | | `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp | -| `KEEP [N] HIGHEST\|LOWEST col` | Keep the rows at the max/min of `col` per group, or the top `N` | -| `CLEANUP EVERY` | Background reclamation cadence as ``, where `unit` is `s`, `m`, `h`, `d`, or `w`. Defaults to `1h` if omitted | +| `KEEP [N] HIGHEST\|LOWEST col` | Keep the rows at the highest/lowest value of `col` per group, or the top `N` | +| `CLEANUP EVERY` | How often the background cleanup job runs, as ``, where `unit` is `s`, `m`, `h`, `d`, or `w`. Defaults to `1h` | Without `N`, the keep column must be `BYTE`, `SHORT`, `INT`, `LONG`, `FLOAT`, -`DOUBLE`, `DATE`, `TIMESTAMP` or `DECIMAL`. `KEEP N HIGHEST/LOWEST` ranks with -`ORDER BY` and accepts any orderable column type. +`DOUBLE`, `DATE`, `TIMESTAMP`, or `DECIMAL`. `KEEP N HIGHEST/LOWEST` sorts rows +with `ORDER BY`, so it accepts any column type you can sort. -For the full description of each mode and its semantics, see the +For a full description of each mode and how it behaves, see the [Expiring rows](/docs/concepts/expire-rows/) concept page. ## When to use @@ -50,20 +49,21 @@ For the full description of each mode and its semantics, see the ## How it works -`SET EXPIRE ROWS` validates the new policy against the view's columns first -(compiling the predicate / checking the key columns), so an invalid predicate or -an unknown column is rejected immediately rather than breaking later reads. Once -set, the policy takes effect without rebuilding the view. See +`SET EXPIRE ROWS` checks the new policy against the view's columns first (it +compiles the condition and checks the key columns), so an invalid condition or an +unknown column is rejected right away instead of breaking later reads. Once set, +the policy takes effect without rebuilding the view. See [How it works](/docs/concepts/expire-rows/#how-it-works). -`SET EXPIRE` is allowed when materialized or live views already depend on this -view. Those dependents detect an applied policy when they next refresh and -become invalid; ALTER completion and dependent invalidation are separate events. -An idle dependent may remain active, and an already-running refresh may finish -against its earlier snapshot. Existing dependent rows are not retroactively -filtered. Removing the policy does not automatically reverse invalidation. See +You can run `SET EXPIRE` even when other materialized or live views already read +this view. Those dependents notice the policy the next time they refresh, and +then become invalid. This does not happen at the same moment as the `ALTER`: an +idle dependent may stay active, and a refresh already running may finish with the +data it started from. Rows the dependents already stored are not removed. +Dropping the policy later does not automatically make an invalid dependent valid +again. See [Dependent materialized and live views](/docs/concepts/expire-rows/#dependent-materialized-and-live-views) -for consequences and recovery. +for the details and how to recover. ## Examples @@ -80,10 +80,9 @@ ALTER MATERIALIZED VIEW trades_mirror SET EXPIRE ROWS WHEN timestamp < dateadd('d', -7, now()) CLEANUP EVERY 30m; ``` -A `WHEN` predicate is the right tool for a cutoff that moves with the clock like -this one. A deterministic predicate such as `amount < 1.5` is accepted too, but -it selects the same rows more cheaply as a `WHERE` clause in the view's defining -query. See +A `WHEN` condition is the right tool for a cutoff that moves with the clock like +this one. A fixed rule such as `amount < 1.5` is also accepted, but it picks out +the same rows more cheaply as a `WHERE` clause in the view's query. See [`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/expire-rows/#where-filter-or-expire-rows). ```questdb-sql title="Keep the latest row per symbol" @@ -101,6 +100,20 @@ ALTER MATERIALIZED VIEW trades_mirror SET EXPIRE ROWS KEEP 2 HIGHEST price PARTITION BY symbol; ``` +For a rule the `KEEP` shortcuts do not cover, write a window condition directly +with `WHEN ... OVER (...)`. This example keeps only the rows within 5% of each +symbol's highest price, and expires the rest: + +```questdb-sql title="Window condition: keep rows within 5% of each symbol's peak" +ALTER MATERIALIZED VIEW trades_mirror + SET EXPIRE ROWS WHEN price < 0.95 * max(price) OVER (PARTITION BY symbol); +``` + +A window condition is always `FILTER_ONLY`: reads hide the expired rows, but the +cleanup job never frees their disk (a later row can change which rows qualify). +See +[When expired rows are deleted from disk](/docs/concepts/expire-rows/#monotonicity-and-cleanup-safety). + ```questdb-sql title="Remove the policy" ALTER MATERIALIZED VIEW trades_mirror DROP EXPIRE; ``` @@ -109,10 +122,10 @@ ALTER MATERIALIZED VIEW trades_mirror DROP EXPIRE; | Aspect | Description | | ----------------------- | ---------------------------------------------------------------------------- | -| Passthrough recommended | An aggregating view is accepted with a logged advisory: a later refresh can regenerate reclaimed rows, so align base-table retention with the expiry horizon | -| Dependent views | SET is allowed; existing materialized and live views detect the conflict and invalidate on refresh, not synchronously with ALTER | +| Passthrough recommended | An aggregating view is allowed, but only with a warning in the log: a later refresh can rebuild deleted rows, so line the base table's retention up with the expiry cutoff | +| Dependent views | SET is allowed; existing materialized and live views notice the conflict and become invalid on their next refresh, not at the same moment as the ALTER | | Validation | The policy is checked against the view's columns before it is applied | -| Replication | The policy and the reclamation it drives replicate as normal WAL traffic | +| Replication | The policy and the row deletions it causes replicate as ordinary WAL traffic | ## Permissions (Enterprise) diff --git a/documentation/query/sql/compile-view.md b/documentation/query/sql/compile-view.md index f5f150a166..998f1bdcb3 100644 --- a/documentation/query/sql/compile-view.md +++ b/documentation/query/sql/compile-view.md @@ -101,11 +101,12 @@ Use `COMPILE VIEW` when you want to: ### Expiry-policy dependency changes -When an expiry policy adds hidden column dependencies beneath an ordinary view, -`COMPILE VIEW` does not refresh its saved dependency permissions. In Enterprise, -restricted readers can continue to receive access-denied errors even if the view -is valid. Wait for policy application, then reissue the original definition with -`ALTER VIEW`. See the [policy-change procedure](/docs/concepts/materialized-views/#change-expiry-beneath-an-existing-ordinary-view). +When an expiry policy adds hidden column dependencies under an ordinary view, +`COMPILE VIEW` does not update its saved dependency permissions. In Enterprise, +restricted readers may keep getting "access denied" errors even though the view +is valid. Wait for the policy to take effect, then run the view's original +definition again with `ALTER VIEW`. See the +[policy-change procedure](/docs/concepts/materialized-views/#change-expiry-beneath-an-existing-ordinary-view). ## Errors diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index aa9bee0258..f028ca0731 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -29,20 +29,20 @@ AS } ``` -Parentheses around `query` are optional only when the query ends the statement. -They are required when any trailing clause follows the query, including -`TIMESTAMP`, `PARTITION BY`, `TTL`, `EXPIRE ROWS`, or `OWNED BY`. +You can leave out the parentheses around `query` only when the query is the last +thing in the statement. They are required if any clause comes after the query, +such as `TIMESTAMP`, `PARTITION BY`, `TTL`, `EXPIRE ROWS`, or `OWNED BY`. Where: - `interval`: Duration like `1m`, `10m`, `1h`, `1d` - `timeUnit`: `HOURS | DAYS | WEEKS | MONTHS | YEARS` -- `query`: Either an aggregating query with `SAMPLE BY` or a time-based - `GROUP BY`, or a - [passthrough](/docs/concepts/materialized-views/#passthrough-views) projection - over a single table +- `query`: Either an aggregating query (with `SAMPLE BY` or a time-based + `GROUP BY`), or a + [passthrough](/docs/concepts/materialized-views/#passthrough-views) query that + copies rows from a single table - `expirePolicy`: `WHEN predicate | KEEP LATEST [ON timestamp] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]`. - This policy is designed for passthrough views (see below). + This policy is meant for passthrough views (see below). ## Parameters @@ -64,7 +64,7 @@ Where: | Rule | Description | | ---- | ----------- | -| Query must aggregate or be passthrough | Either `SAMPLE BY` / `GROUP BY` with a designated timestamp, or a 1:1 [passthrough](/docs/concepts/materialized-views/#passthrough-views) projection over a single table | +| Query must aggregate or be passthrough | Either `SAMPLE BY` / `GROUP BY` with a designated timestamp, or a [passthrough](/docs/concepts/materialized-views/#passthrough-views) query that copies rows one-for-one from a single table | | Default refresh | `IMMEDIATE` (refreshes after each base table transaction) | | WITH BASE required | Must specify when query contains JOINs | | PARTITION BY sizing | Should be larger than or equal to `SAMPLE BY` interval | @@ -73,23 +73,23 @@ Where: ### LIMIT restrictions and existing definitions -`LIMIT` is not supported in query branches that read the base table, including -nested queries, for both aggregating and passthrough materialized views. An -incremental refresh evaluates the defining query over changed timestamp ranges; -a `LIMIT 100` would cap each refreshed range rather than the view as a whole. -Limited subqueries over other tables remain allowed. +You cannot use `LIMIT` in any part of the query that reads the base table, +including nested queries. This applies to both aggregating and passthrough views. +Each refresh runs the query over the rows that just changed, so a `LIMIT 100` +would cap each batch rather than the view as a whole. A `LIMIT` in a subquery over +a *different* table is still allowed. :::note Upgrading existing definitions -Existing materialized views whose definitions contain such a `LIMIT` continue -refreshing. However, recreating them after an upgrade from a version that accepted -those definitions requires removing or restructuring the limit. This also applies -when replaying `SHOW CREATE MATERIALIZED VIEW` output during a migration or restore. +Materialized views that already contain such a `LIMIT` keep refreshing. But if you +recreate one after upgrading from a version that allowed it, you have to remove or +rework the limit first. The same applies when you replay +`SHOW CREATE MATERIALIZED VIEW` output during a migration or restore. -If the limit only controls how many results an application receives, remove it -from the view definition and apply `ORDER BY ... LIMIT ...` when querying the view. -This changes which rows the view stores and does not preserve every original -definition's behavior. +If the limit was only there to cap how many rows your application receives, remove +it from the view and instead use `ORDER BY ... LIMIT ...` when you query the view. +Note that moving the limit changes which rows the view stores, so it does not +behave exactly like the original. ::: @@ -318,17 +318,18 @@ The view's TTL is independent of the base table's TTL. See ## EXPIRE ROWS -Attach a row-level retention policy with `EXPIRE ROWS`. Unlike `TTL` (which drops -whole partitions by age), `EXPIRE ROWS` keeps a defined set of rows: the latest -per key, the top-N per group, or rows matching a predicate. It recomputes that -set continuously as the view refreshes. +Add a row-level retention policy with `EXPIRE ROWS`. Where `TTL` drops whole +partitions by age, `EXPIRE ROWS` keeps a chosen set of rows: the latest per key, +the top-N per group, or rows that match a condition. QuestDB keeps that set up to +date as the view refreshes. -`EXPIRE ROWS` is designed for -[**passthrough (non-aggregating) views**](/docs/concepts/materialized-views/#passthrough-views) -(a projection over a single table with no `SAMPLE BY` / `GROUP BY`, whose rows -stay 1:1 with the base). An aggregating view is accepted with a logged advisory -(a later refresh can regenerate reclaimed rows). The defining query must not -read another policied view, as its base or in a join: +`EXPIRE ROWS` is built for +[**passthrough (non-aggregating) views**](/docs/concepts/materialized-views/#passthrough-views): +a query over a single table with no `SAMPLE BY` / `GROUP BY`, where each view row +matches one base row. An aggregating view is allowed, but only with a warning in +the log, because a later refresh can rebuild rows that were deleted. The query +must not read another view that has a policy, whether as its base table or +through a join: ```questdb-sql title="Passthrough view that keeps the latest row per symbol" CREATE MATERIALIZED VIEW trades_latest AS ( @@ -336,11 +337,11 @@ CREATE MATERIALIZED VIEW trades_latest AS ( ) EXPIRE ROWS KEEP LATEST PARTITION BY symbol; ``` -A `WHEN` predicate is for rules that move with **wall-clock time**, such as a -rolling `timestamp < dateadd('d', -7, now())` window. The defining query cannot -express those, because it rejects non-deterministic functions. A predicate that -depends only on the row's own values belongs in the query's `WHERE` clause -instead, which keeps those rows out of the view entirely; see +A `WHEN` condition is for rules that depend on the **current time**, such as a +rolling `timestamp < dateadd('d', -7, now())` window. The view's query cannot do +this, because it is not allowed to call functions like `now()`. A rule that only +looks at a row's own values belongs in the query's `WHERE` clause instead, which +keeps those rows out of the view completely; see [`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/expire-rows/#where-filter-or-expire-rows). The clause goes after the query (and after `PARTITION BY` if present): @@ -353,24 +354,24 @@ EXPIRE ROWS [ CLEANUP EVERY duration ] ``` -A `WHEN` threshold that is constant at definition time and evaluates to `NULL` -is rejected, since it would expire nothing. That covers the explicit -`timestamp < CAST(NULL AS TIMESTAMP)` and arithmetic that overflows onto the +A `WHEN` threshold that is a fixed value at definition time and comes out as +`NULL` is rejected, because it would expire nothing. This covers the obvious +`timestamp < CAST(NULL AS TIMESTAMP)` and also math that overflows onto the reserved `NULL` value, such as `timestamp < 2147483647 + 1`. See [A `NULL` threshold is rejected](/docs/concepts/expire-rows/#a-null-threshold-is-rejected). -For filtering and disk-reclamation behavior, see +For how expired rows are filtered and freed from disk, see [How `EXPIRE ROWS` works](/docs/concepts/expire-rows/#how-it-works). Change or remove a policy with [`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/). -A view can carry both `TTL` and `EXPIRE ROWS`. `TTL` comes first in the -statement and first in effect: it removes rows from the view, and the -`EXPIRE ROWS` policy then applies to the rows that stay. See +A view can have both `TTL` and `EXPIRE ROWS`. `TTL` comes first in the statement +and first in effect: it removes rows from the view, and the `EXPIRE ROWS` policy +then applies to what is left. See [Combining with TTL](/docs/concepts/expire-rows/#combining-with-ttl). -See the [Expiring rows](/docs/concepts/expire-rows/) concept page for -all modes, worked examples, and semantics (NULLs, ties, monotonicity). +See the [Expiring rows](/docs/concepts/expire-rows/) concept page for all modes, +examples, and details (NULLs, ties, and when rows are deleted from disk). ## Complete example diff --git a/documentation/query/sql/refresh-mat-view.md b/documentation/query/sql/refresh-mat-view.md index 2a58f2e092..5fa070d346 100644 --- a/documentation/query/sql/refresh-mat-view.md +++ b/documentation/query/sql/refresh-mat-view.md @@ -45,10 +45,10 @@ REFRESH MATERIALIZED VIEW trades_hourly FULL; - You need to rebuild after dropping partitions from the base table :::warning -FULL refresh deletes existing data before rebuilding and can take significant -time on large base tables. The view remains queryable: queries can observe empty -or partially rebuilt contents as refresh transactions apply. If rebuilding fails, -the refresh does not automatically restore the previous contents. +A FULL refresh deletes the view's data before rebuilding it, and can take a long +time on large base tables. You can still query the view while it rebuilds, but +queries may see it empty or half-rebuilt as the refresh runs. If the rebuild +fails, the view is not automatically restored to its previous contents. ::: ### INCREMENTAL @@ -153,7 +153,7 @@ REFRESH MATERIALIZED VIEW trades_hourly | Aspect | Description | | ------ | ----------- | | Execution | Asynchronous - command returns immediately | -| View availability | View remains queryable; FULL can expose empty or partially rebuilt contents (see [FULL](#full)) | +| View availability | Still queryable; a FULL refresh may show empty or half-rebuilt contents (see [FULL](#full)) | | Concurrency | Only one refresh runs at a time per view | | Cancellation | Use [`CANCEL QUERY`](/docs/query/sql/cancel-query/) to stop a long-running refresh | diff --git a/documentation/security/rbac.md b/documentation/security/rbac.md index 43d36ba3fc..f18ac75e45 100644 --- a/documentation/security/rbac.md +++ b/documentation/security/rbac.md @@ -120,16 +120,17 @@ underlying `trades` table. ### Materialized views with row expiry -Column-restricted readers of a materialized view with `EXPIRE ROWS` need SELECT -on the requested columns and the columns used by its policy. Enabling or changing -expiry can therefore require updating direct grants. Those grants also permit -reading the policy columns explicitly. - -To keep policy columns hidden, grant access to an ordinary SQL view exposing only -the intended output. Its readers need no underlying materialized-view grants. -When expiry adds dependencies beneath an existing ordinary view, wait for WAL -application and reissue the original definition with `ALTER VIEW`; reads can be -denied until that update. `COMPILE VIEW` does not repair these dependencies. +A reader with column-level access to a materialized view that has `EXPIRE ROWS` +needs SELECT on the columns they query *and* on the columns the policy uses. So +turning expiry on or changing it can mean you have to update those grants. Note +that granting a policy column also lets the reader query that column directly. + +To keep policy columns hidden, give readers access to an ordinary SQL view that +exposes only the columns they should see. Those readers need no grant on the +materialized view itself. If expiry adds new column dependencies under an existing +ordinary view, wait for the change to take effect (WAL applied), then run the +view's original definition again with `ALTER VIEW`; reads may be denied until you +do. `COMPILE VIEW` does not fix these dependencies. See [restricted access with row expiry](/docs/concepts/materialized-views/#restricted-access-with-row-expiry) for grant examples, the COUNT limitation and workaround, and the policy-change From 5eac2ec9646ef9750c4797dfb45eea4902b23fb7 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Fri, 18 Sep 2026 16:15:37 +0200 Subject: [PATCH 19/25] docs: explain query and cleanup costs of row expiry --- documentation/concepts/materialized-views.md | 24 ++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index b2781b679f..81b3cad424 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -597,6 +597,30 @@ WHERE timestamp > dateadd('M', -1, now()); This returns in single-digit milliseconds. The data is pre-aggregated, so no aggregation work is needed at query time. +### Performance with row expiry + +`EXPIRE ROWS` adds work when you query the view: + +- **`WHEN`** checks which rows have expired. A cutoff on the designated timestamp + can let QuestDB skip older partitions. +- **`KEEP LATEST`** finds the newest row for each key. An index on the symbol key + can speed this up. +- **`KEEP HIGHEST/LOWEST`, `KEEP N`, and window conditions** compare rows across + the view. Keeping ten rows per group does not mean QuestDB only reads ten rows + per group. + +The `KEEP` modes and window conditions hide rows without deleting them from disk. +Queries can become slower as the stored data grows. Use TTL to remove old +partitions, but remember that the policy then chooses only from the remaining +rows. + +Some `WHEN` policies also delete expired rows in the background. This cleanup +adds disk activity, especially when it must rewrite a partition to remove only +some rows. `CLEANUP EVERY` sets how often cleanup runs. Queries hide expired rows +even before cleanup runs. + +See [How EXPIRE ROWS works](/docs/concepts/expire-rows/#how-it-works) for details. + ## Managing materialized views ### Listing views From f86b37fdbc8d177a0594cf696fac819112d34409 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Fri, 18 Sep 2026 16:30:41 +0200 Subject: [PATCH 20/25] docs: clarify KEEP LATEST sensor use case and TTL tradeoff --- documentation/concepts/materialized-views.md | 24 ++++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index 81b3cad424..89cde68d1f 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -206,14 +206,8 @@ A building management platform records temperature and humidity from tens of thousands of sensors, and its operations screen shows the newest reading from each one. -Sensors report at their own pace. Some send a reading every second, others go -quiet for days. Against the base table, that screen runs -`LATEST ON ts PARTITION BY sensor_id`, which reads backwards until it finds a row -for even the quietest sensor. Over a long history, that means reading most of the -table. - -A view that keeps only the newest row per sensor answers the same question from -just a handful of rows: +A view can give every application the newest reading per sensor without requiring +each application to write its own `LATEST ON` query: ```questdb-sql title="Latest reading per sensor" CREATE MATERIALIZED VIEW sensor_current AS ( @@ -221,10 +215,16 @@ CREATE MATERIALIZED VIEW sensor_current AS ( ) EXPIRE ROWS KEEP LATEST PARTITION BY sensor_id; ``` -`sensor_current` holds one row per `sensor_id` and updates itself as readings -arrive. Older rows stop showing up in queries but stay on disk, so the view keeps -growing at the same rate as the base table. Give it a [TTL](/docs/concepts/ttl/) -to cap its size. +Applications can query `sensor_current` with a simple `SELECT` and get one row +per `sensor_id`. As the view refreshes, new readings replace older ones in query +results. QuestDB still finds the latest rows at query time, and older readings +remain on disk. This simplifies application queries; it does not by itself make +the lookup faster. + +A [TTL](/docs/concepts/ttl/) can limit disk usage by removing old partitions. +However, if a sensor stops reporting, TTL can remove its last reading too. Use TTL +only if it is acceptable for sensors with no recent readings to disappear from +results. #### Finance: Options that have not expired yet From 98bcfe6acab5fbc8b5e35768d45f22f090dbf678 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Fri, 18 Sep 2026 16:41:54 +0200 Subject: [PATCH 21/25] Improve create mat view page --- documentation/query/sql/create-mat-view.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index f028ca0731..e0f40be1f8 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -373,9 +373,15 @@ then applies to what is left. See See the [Expiring rows](/docs/concepts/expire-rows/) concept page for all modes, examples, and details (NULLs, ties, and when rows are deleted from disk). -## Complete example +## Larger example -Putting it all together: +Create a materialized view that: + +- Checks for updates every 15 minutes (`EVERY 15m`) +- Processes data in 1-hour chunks, waiting 5 minutes for late data (`PERIOD`) +- Aggregates from `trades` table (`WITH BASE trades`) +- Stores hourly averages and volumes (`SAMPLE BY 1h`) +- Keeps 30 days of data (`TTL 30 DAYS`) ```questdb-sql title="Fully specified materialized view" CREATE MATERIALIZED VIEW IF NOT EXISTS trades_hourly_stats @@ -396,13 +402,6 @@ AS ( PARTITION BY DAY TTL 30 DAYS; ``` -This creates a view that: -- Checks for updates every 15 minutes (`EVERY 15m`) -- Processes data in 1-hour chunks, waiting 5 minutes for late data (`PERIOD`) -- Aggregates from `trades` table (`WITH BASE trades`) -- Stores hourly averages and volumes (`SAMPLE BY 1h`) -- Keeps 30 days of data (`TTL 30 DAYS`) - ## Metadata Query view metadata with `materialized_views()`: @@ -490,7 +489,9 @@ GRANT DROP MATERIALIZED VIEW ON trades_hourly TO user1; ## See also - [Materialized views concept](/docs/concepts/materialized-views/) +- [Row expiry concept](/docs/concepts/expire-rows/) - [REFRESH MATERIALIZED VIEW](/docs/query/sql/refresh-mat-view/) - [DROP MATERIALIZED VIEW](/docs/query/sql/drop-mat-view/) - [ALTER MATERIALIZED VIEW SET REFRESH](/docs/query/sql/alter-mat-view-set-refresh/) - [ALTER MATERIALIZED VIEW SET TTL](/docs/query/sql/alter-mat-view-set-ttl/) +- [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) From bea05b4dd0362b6650b37c6b16797e067b7fc035 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Mon, 21 Sep 2026 09:52:59 +0200 Subject: [PATCH 22/25] Clarify performance --- documentation/concepts/expire-rows.md | 28 ++++++++++ documentation/concepts/materialized-views.md | 57 +++++++++++++------- 2 files changed, 67 insertions(+), 18 deletions(-) diff --git a/documentation/concepts/expire-rows.md b/documentation/concepts/expire-rows.md index b2ab2ba3e6..6f146f2bbd 100644 --- a/documentation/concepts/expire-rows.md +++ b/documentation/concepts/expire-rows.md @@ -46,6 +46,24 @@ everything else is expired. | Keep highest / lowest | Rows tied at the group's highest / lowest value | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No (hides only) | | Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No (hides only) | +:::tip Which mode deletes rows from disk? + +Only a per-row `WHEN` policy can delete expired rows from disk. QuestDB does this +when it can tell that an expired row will never be needed again. Deleting the +rows saves disk space and gives later queries fewer rows to scan. + +To check a policy, run: + +```questdb-sql +SELECT view_name, expire_enforcement FROM materialized_views(); +``` + +`FILTER_AND_RECLAIM` means that QuestDB hides expired rows and later deletes them +from disk. `FILTER_ONLY` means that QuestDB hides the rows but leaves them on +disk. `KEEP` modes and window conditions are always `FILTER_ONLY`. + +::: + `KEEP HIGHEST/LOWEST` and `KEEP N` are shortcuts. Under the hood they turn into a window condition, so a window `WHEN` is the general-purpose form when the shortcuts do not fit. @@ -295,6 +313,16 @@ several key columns in `PARTITION BY`. You may also write `KEEP LATEST ON timestamp PARTITION BY symbol`, but the `ON` column has to be the view's designated timestamp. +:::note `KEEP LATEST` does not save the result ahead of time + +`KEEP LATEST` makes application queries simpler, but older rows stay on disk. +Each time you query the view, QuestDB finds the latest row for every key from the +rows the view stores. To reduce the number of rows this lookup must search, use +the +[pre-aggregation approach for speeding up `LATEST ON`](/docs/concepts/materialized-views/#advanced-latest-on-optimization). + +::: + ### Keep highest/lowest per group: `KEEP HIGHEST` / `KEEP LOWEST` Keep the rows tied at the highest (or lowest) value of a column, per group: diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index 89cde68d1f..591ffc9306 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -268,16 +268,15 @@ CREATE MATERIALIZED VIEW trades_largest AS ( ) EXPIRE ROWS KEEP 10 HIGHEST amount PARTITION BY symbol; ``` -`trades_largest` holds ten rows per symbol no matter how large the base table -grows, and the desk reads it directly instead of ranking the base table on every -query. When an eleventh large trade arrives, the smallest of the ten drops out. -If two trades tie on `amount` at the tenth spot, the newer one stays (decided by -the designated timestamp). - -Ten rows per symbol show up in queries, but the view still stores every base row -it has taken in, because a top-N policy never frees disk. A -[TTL](/docs/concepts/ttl/) caps that, but it changes the question the view -answers from "the biggest so far" to "the biggest still kept". See +Applications can query `trades_largest` directly and get ten rows per symbol, +without writing the ranking in every query. When a larger trade arrives, the +smallest result may drop out. If two trades tie on `amount` for the last place, +the newer one stays (decided by the designated timestamp). + +QuestDB still stores every row and works out the ranking each time you query the +view. A [TTL](/docs/concepts/ttl/) can reduce the number of rows stored and +ranked, but it changes the result from "the biggest so far" to "the biggest still +kept". See [combining with TTL](/docs/concepts/expire-rows/#combining-with-ttl). The ranking covers everything the view holds, not just a recent window, so @@ -609,15 +608,28 @@ aggregation work is needed at query time. the view. Keeping ten rows per group does not mean QuestDB only reads ten rows per group. -The `KEEP` modes and window conditions hide rows without deleting them from disk. -Queries can become slower as the stored data grows. Use TTL to remove old -partitions, but remember that the policy then chooses only from the remaining -rows. +Only a per-row `WHEN` policy can delete expired rows from disk. QuestDB does +this when it can tell that an expired row will never be needed again. For +example, cleanup works with a cutoff on the designated timestamp: + +```questdb-sql +EXPIRE ROWS WHEN timestamp < dateadd('d', -7, now()) +``` + +Deleting rows saves disk space and gives later queries fewer rows to scan. The +`expire_enforcement` column returned by `materialized_views()` shows whether +QuestDB will do this: + +- `FILTER_AND_RECLAIM`: hide expired rows, then delete them in the background. +- `FILTER_ONLY`: hide expired rows, but leave them on disk. -Some `WHEN` policies also delete expired rows in the background. This cleanup -adds disk activity, especially when it must rewrite a partition to remove only -some rows. `CLEANUP EVERY` sets how often cleanup runs. Queries hide expired rows -even before cleanup runs. +The `KEEP` modes and window conditions are always `FILTER_ONLY`. Their stored +data keeps growing, which can make queries slower. TTL can remove old partitions, +but the policy then works only with the rows that remain. + +Cleanup also uses disk resources, especially when it rewrites a partition to +remove only some rows. `CLEANUP EVERY` sets how often cleanup runs. Expired rows +are hidden from queries even before cleanup runs. See [How EXPIRE ROWS works](/docs/concepts/expire-rows/#how-it-works) for details. @@ -721,6 +733,15 @@ SELECT * FROM trades LATEST ON timestamp PARTITION BY symbol; This might scan billions of rows to find the latest entry for rarely-updated symbols. +:::note `KEEP LATEST` does not replace this optimization + +[`EXPIRE ROWS KEEP LATEST`](/docs/concepts/expire-rows/#keep-latest-per-key-keep-latest) +lets applications use a simple `SELECT`, but older rows stay on disk. Each query +still finds the latest row for every key from the rows stored in the view. Use +the pre-aggregation below to give that lookup fewer rows to search. + +::: + ### Solution: Pre-aggregate with a materialized view Create a view that stores one row per symbol per day: From d703c8827aa8c3c29ce7ef2cd268c45932092754 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Mon, 21 Sep 2026 13:43:48 +0200 Subject: [PATCH 23/25] Add ON to KEEP HIGHEST/LOWEST syntax --- documentation/concepts/expire-rows.md | 50 +++++++++---------- documentation/concepts/materialized-views.md | 8 +-- documentation/query/functions/meta.md | 18 +++---- .../query/sql/alter-mat-view-set-expire.md | 23 +++++---- documentation/query/sql/create-mat-view.md | 6 +-- 5 files changed, 53 insertions(+), 52 deletions(-) diff --git a/documentation/concepts/expire-rows.md b/documentation/concepts/expire-rows.md index 6f146f2bbd..f4366e15f0 100644 --- a/documentation/concepts/expire-rows.md +++ b/documentation/concepts/expire-rows.md @@ -41,10 +41,10 @@ everything else is expired. | Mode | What it keeps | Syntax | Frees disk | | --------------------- | --------------------------------------------------- | ------------------------------------------------------------------- | --------------------- | | Per-row condition | Rows where the condition is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when it is safe | -| Window condition | Rows where a condition using `OVER (...)` is **not** `TRUE` | `EXPIRE ROWS WHEN predicate OVER (...)` | No (hides only) | -| Keep latest | The latest row per key (the current value per key) | `EXPIRE ROWS KEEP LATEST [ON timestamp] PARTITION BY cols` | No (hides only) | -| Keep highest / lowest | Rows tied at the group's highest / lowest value | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No (hides only) | -| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No (hides only) | +| Window condition | Rows where a condition using `OVER (...)` is **not** `TRUE` | `EXPIRE ROWS WHEN predicate OVER (...)` | No (hides only) | +| Keep latest | The latest row per key (the current value per key) | `EXPIRE ROWS KEEP LATEST [ON timestamp] PARTITION BY cols` | No (hides only) | +| Keep highest / lowest | Rows tied at the group's highest / lowest value | `EXPIRE ROWS KEEP HIGHEST\|LOWEST ON col [PARTITION BY cols]` | No (hides only) | +| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST ON col [PARTITION BY cols]` | No (hides only) | :::tip Which mode deletes rows from disk? @@ -72,7 +72,7 @@ The plain `KEEP HIGHEST/LOWEST` form works on these column types: `BYTE`, `SHORT`, `INT`, `LONG`, `FLOAT`, `DOUBLE`, `DATE`, `TIMESTAMP` and `DECIMAL`. The top-N form (`KEEP N HIGHEST/LOWEST`) sorts rows with `ORDER BY`, so it works on any column type you can sort. For example, to rank a `SYMBOL` column, use -`KEEP 1 HIGHEST symbol`, not `KEEP HIGHEST symbol`. +`KEEP 1 HIGHEST ON symbol`, not `KEEP HIGHEST ON symbol`. The two forms treat ties and `NULL`s differently: @@ -95,16 +95,16 @@ query, and after `PARTITION BY` if you use it), or add it later with EXPIRE ROWS { WHEN predicate | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] - | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } + | KEEP [ N ] ( HIGHEST | LOWEST ) ON col [ PARTITION BY col [, col ...] ] } [ CLEANUP EVERY duration ] ``` -| Element | Meaning | -| ------------------ | ---------------------------------------------------------------------------------------- | -| `predicate` | Any true/false expression over the view's columns. A row expires when it is `TRUE`. | -| `KEEP LATEST` | Keep the latest row for each `PARTITION BY` key, by the designated timestamp. | -| `ON timestampCol` | Optional. If given, it must be the view's designated timestamp. | -| `HIGHEST\|LOWEST` | Keep the rows at the highest / lowest value of `col` per group (no `N`), or the top `N`. | +| Element | Meaning | +| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------- | +| `predicate` | Any true/false expression over the view's columns. A row expires when it is `TRUE`. | +| `KEEP LATEST` | Keep the latest row for each `PARTITION BY` key, by the designated timestamp. | +| `ON timestampCol` | Optional. If given, it must be the view's designated timestamp. | +| `HIGHEST\|LOWEST ON col` | Keep the rows at the highest / lowest value of `col` per group (no `N`), or the top `N`. | | `CLEANUP EVERY` | How often the background cleanup job runs for this view: ``, where the unit is `s`/`m`/`h`/`d`/`w`. Defaults to `1h`. | :::note @@ -297,7 +297,7 @@ Keep only the most recent row per key, to turn the view into a live ```questdb-sql title="Keep the latest row per symbol" CREATE MATERIALIZED VIEW trades_latest AS ( SELECT * FROM trades -) EXPIRE ROWS KEEP LATEST PARTITION BY symbol; +) EXPIRE ROWS KEEP LATEST ON timestamp PARTITION BY symbol; SELECT * FROM trades_latest ORDER BY timestamp; ``` @@ -309,9 +309,9 @@ SELECT * FROM trades_latest ORDER BY timestamp; The designated `timestamp` column decides which row is the latest for each symbol. As new trades arrive, the kept row moves forward on its own. You can list -several key columns in `PARTITION BY`. You may also write -`KEEP LATEST ON timestamp PARTITION BY symbol`, but the `ON` column has to be the -view's designated timestamp. +several key columns in `PARTITION BY`. The `ON timestamp` part is optional - +`KEEP LATEST PARTITION BY symbol` means the same thing - but when you write it, +it has to name the view's designated timestamp. :::note `KEEP LATEST` does not save the result ahead of time @@ -330,7 +330,7 @@ Keep the rows tied at the highest (or lowest) value of a column, per group: ```questdb-sql title="Keep the highest-priced trade per symbol" CREATE MATERIALIZED VIEW trades_peak AS ( SELECT * FROM trades -) EXPIRE ROWS KEEP HIGHEST price PARTITION BY symbol; +) EXPIRE ROWS KEEP HIGHEST ON price PARTITION BY symbol; SELECT * FROM trades_peak; ``` @@ -340,7 +340,7 @@ SELECT * FROM trades_peak; | BTC | sell | 105.0 | 2.0 | 2026-01-01T11:00:00.000000Z | | ETH | buy | 55.0 | 1.0 | 2026-01-02T08:00:00.000000Z | -`KEEP LOWEST price PARTITION BY symbol` keeps the cheapest instead (BTC `100.0`, +`KEEP LOWEST ON price PARTITION BY symbol` keeps the cheapest instead (BTC `100.0`, ETH `50.0`). Every row tied at the highest (or lowest) value is kept, and `NULL`-valued rows are kept too (a `NULL` is never below the highest value). @@ -351,7 +351,7 @@ Keep a per-group leaderboard: the `N` highest (or lowest) rows. ```questdb-sql title="Keep the 2 highest-priced trades per symbol" CREATE MATERIALIZED VIEW trades_top2 AS ( SELECT * FROM trades -) EXPIRE ROWS KEEP 2 HIGHEST price PARTITION BY symbol; +) EXPIRE ROWS KEEP 2 HIGHEST ON price PARTITION BY symbol; SELECT * FROM trades_top2 ORDER BY symbol, price DESC; ``` @@ -371,7 +371,7 @@ N-th row is always decided the same way. `KEEP HIGHEST/LOWEST` and `KEEP N` are just shortcuts for window conditions. When you need a rule they do not cover, write the window condition directly in `WHEN`. -For example, this is exactly what `KEEP HIGHEST price PARTITION BY symbol` turns +For example, this is exactly what `KEEP HIGHEST ON price PARTITION BY symbol` turns into: ```questdb-sql title="The same as KEEP HIGHEST, written as a window condition" @@ -620,7 +620,7 @@ view as they age out, and the kept set is worked out over whatever remains. CREATE MATERIALIZED VIEW trades_peak_3d AS ( SELECT * FROM trades ) PARTITION BY DAY TTL 3 DAYS - EXPIRE ROWS KEEP HIGHEST price PARTITION BY symbol; + EXPIRE ROWS KEEP HIGHEST ON price PARTITION BY symbol; ``` `TTL` goes before `EXPIRE ROWS` in the statement, just as it goes after any @@ -686,7 +686,7 @@ otherwise: ```questdb-sql SHOW CREATE MATERIALIZED VIEW trades_latest; --- ... EXPIRE ROWS KEEP LATEST PARTITION BY symbol +-- ... EXPIRE ROWS KEEP LATEST ON timestamp PARTITION BY symbol ``` The [`materialized_views()`](/docs/query/functions/meta/) function shows the @@ -701,8 +701,8 @@ FROM materialized_views(); | view_name | expire_clause | expire_cleanup_every | expire_enforcement | | ------------- | ------------------------------- | -------------------- | ------------------ | | trades_sized | amount < 1.5 | 1h | FILTER_AND_RECLAIM | -| trades_latest | KEEP LATEST PARTITION BY symbol | 1h | FILTER_ONLY | -| trades_top2 | KEEP 2 HIGHEST price ... | 1h | FILTER_ONLY | +| trades_latest | KEEP LATEST ON timestamp PARTITION BY symbol | 1h | FILTER_ONLY | +| trades_top2 | KEEP 2 HIGHEST ON price ... | 1h | FILTER_ONLY | `expire_enforcement` tells you what the cleanup job does: @@ -719,7 +719,7 @@ You can set, change, or drop a policy on an existing passthrough view. See ```questdb-sql -- set or replace the policy -ALTER MATERIALIZED VIEW trades_latest SET EXPIRE ROWS KEEP LATEST PARTITION BY symbol; +ALTER MATERIALIZED VIEW trades_latest SET EXPIRE ROWS KEEP LATEST ON timestamp PARTITION BY symbol; -- remove it (keeps all rows again) ALTER MATERIALIZED VIEW trades_latest DROP EXPIRE; diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index 591ffc9306..0fdf6eb841 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -212,7 +212,7 @@ each application to write its own `LATEST ON` query: ```questdb-sql title="Latest reading per sensor" CREATE MATERIALIZED VIEW sensor_current AS ( SELECT * FROM sensor_readings -) EXPIRE ROWS KEEP LATEST PARTITION BY sensor_id; +) EXPIRE ROWS KEEP LATEST ON timestamp PARTITION BY sensor_id; ``` Applications can query `sensor_current` with a simple `SELECT` and get one row @@ -265,7 +265,7 @@ keeps: ```questdb-sql title="Ten largest trades per symbol" CREATE MATERIALIZED VIEW trades_largest AS ( SELECT * FROM trades -) EXPIRE ROWS KEEP 10 HIGHEST amount PARTITION BY symbol; +) EXPIRE ROWS KEEP 10 HIGHEST ON amount PARTITION BY symbol; ``` Applications can query `trades_largest` directly and get ten rows per symbol, @@ -838,8 +838,8 @@ view `mv` with columns `sym`, `k`, `v`, `secret`, and designated timestamp `ts`: | --- | --- | | `WHEN v < 2.0` | `SELECT ON mv(sym, v)` | | `WHEN ts < '2026-01-01T00:00:01.000000Z'` | `SELECT ON mv(sym)` | -| `KEEP LATEST PARTITION BY k` | `SELECT ON mv(sym, k)` | -| `KEEP HIGHEST v PARTITION BY k` | `SELECT ON mv(sym, v, k)` | +| `KEEP LATEST ON ts PARTITION BY k` | `SELECT ON mv(sym, k)` | +| `KEEP HIGHEST ON v PARTITION BY k` | `SELECT ON mv(sym, v, k)` | A column-level grant always includes the designated timestamp. A table-level SELECT grant covers every column, including the ones the policy needs. diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index fc0ba3decf..714366210a 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -322,8 +322,8 @@ Returns a `table` including the following information: than this, the refresh job handles them together instead of committing twice. `0` turns this off - `expire_clause` - the view's - [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy as written, or `NULL` if - the view has no policy + [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy in its canonical form, or + `NULL` if the view has no policy - `expire_cleanup_every` - how often the cleanup job runs for the policy, or `NULL` - `expire_enforcement` - `FILTER_AND_RECLAIM` if the cleanup job also frees disk @@ -347,7 +347,7 @@ FROM materialized_views(); | --------------- | ----------- | --------------- | ---------------------- | -------------- | ----------------------------------- | -------------------- | ------------------ | | trades_OHLC_15m | valid | trades | 1 | 1 | null | null | null | | trades_recent | valid | trades | 1 | 1 | timestamp < dateadd('d', -7, now()) | 30m | FILTER_AND_RECLAIM | -| trades_latest | valid | trades | 1 | 1 | KEEP LATEST PARTITION BY symbol | 1h | FILTER_ONLY | +| trades_latest | valid | trades | 1 | 1 | KEEP LATEST ON timestamp PARTITION BY symbol | 1h | FILTER_ONLY | ## memory_metrics @@ -882,7 +882,7 @@ SELECT * FROM table_storage(); | trades | true | DAY | 954 | 1000848308 | 32764798760 | | ethblocks_json | true | DAY | 3328 | 20688364 | 28311960478 | -
+--- Filter tables with WAL enabled. @@ -899,7 +899,7 @@ WHERE walEnabled = true; | trades | 1000850255 | 32764804264 | | ethblocks_json | 20688364 | 28311960478 | -
+--- Show tables partitioned by `HOUR`. @@ -1009,10 +1009,10 @@ On primary instances, these columns will be `0` or `false`. ### Row-expiry policy -| Column | Type | Description | -| ---------------------- | ------ | -------------------------------------------------------------- | -| `expire_clause` | STRING | Materialized view's `EXPIRE ROWS` policy as written, or `null` | -| `expire_cleanup_every` | STRING | Cleanup cadence for the policy, or `null` | +| Column | Type | Description | +| ---------------------- | ------ | -------------------------------------------------------------- | +| `expire_clause` | STRING | Materialized view's `EXPIRE ROWS` policy in its canonical form, or `null` | +| `expire_cleanup_every` | STRING | Cleanup cadence for the policy, or `null` | `tables()` does not include `expire_enforcement`. To check whether a policy is `FILTER_AND_RECLAIM` or `FILTER_ONLY`, use `materialized_views()` instead. diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 4450064276..f3775c4998 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -17,7 +17,7 @@ disk, see [How `EXPIRE ROWS` works](/docs/concepts/expire-rows/#how-it-works). ALTER MATERIALIZED VIEW viewName SET EXPIRE ROWS { WHEN predicate | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] - | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } + | KEEP [ N ] ( HIGHEST | LOWEST ) ON col [ PARTITION BY col [, col ...] ] } [ CLEANUP EVERY duration ] ALTER MATERIALIZED VIEW viewName DROP EXPIRE @@ -25,12 +25,12 @@ ALTER MATERIALIZED VIEW viewName DROP EXPIRE ## Parameters -| Parameter | Description | -| ---------------- | --------------------------------------------------------------------------------- | -| `viewName` | Name of the passthrough materialized view to modify | -| `WHEN predicate` | A per-row (or window) condition; a row expires when it is `TRUE` | -| `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp | -| `KEEP [N] HIGHEST\|LOWEST col` | Keep the rows at the highest/lowest value of `col` per group, or the top `N` | +| Parameter | Description | +| ---------------- | ---------------------------------------------------------------------------------------------------------------------------- | +| `viewName` | Name of the passthrough materialized view to modify | +| `WHEN predicate` | A per-row (or window) condition; a row expires when it is `TRUE` | +| `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp | +| `KEEP [N] HIGHEST\|LOWEST ON col` | Keep the rows at the highest/lowest value of `col` per group, or the top `N` | | `CLEANUP EVERY` | How often the background cleanup job runs, as ``, where `unit` is `s`, `m`, `h`, `d`, or `w`. Defaults to `1h` | Without `N`, the keep column must be `BYTE`, `SHORT`, `INT`, `LONG`, `FLOAT`, @@ -87,17 +87,17 @@ the same rows more cheaply as a `WHERE` clause in the view's query. See ```questdb-sql title="Keep the latest row per symbol" ALTER MATERIALIZED VIEW trades_mirror - SET EXPIRE ROWS KEEP LATEST PARTITION BY symbol; + SET EXPIRE ROWS KEEP LATEST ON timestamp PARTITION BY symbol; ``` ```questdb-sql title="Keep the highest-priced row per symbol" ALTER MATERIALIZED VIEW trades_mirror - SET EXPIRE ROWS KEEP HIGHEST price PARTITION BY symbol; + SET EXPIRE ROWS KEEP HIGHEST ON price PARTITION BY symbol; ``` ```questdb-sql title="Keep the 2 highest-priced rows per symbol" ALTER MATERIALIZED VIEW trades_mirror - SET EXPIRE ROWS KEEP 2 HIGHEST price PARTITION BY symbol; + SET EXPIRE ROWS KEEP 2 HIGHEST ON price PARTITION BY symbol; ``` For a rule the `KEEP` shortcuts do not cover, write a window condition directly @@ -140,9 +140,10 @@ GRANT ALTER MATERIALIZED VIEW ON trades_mirror TO user1; | Error | Cause | | ----- | ----- | | `materialized view does not exist` | View with the specified name doesn't exist | +| `'on' expected` | `KEEP HIGHEST`/`KEEP LOWEST` was not followed by `ON `; the keep column must be introduced by `ON` | | `EXPIRE ROWS KEEP LATEST ON must name the designated timestamp ...` | `ON` names a column other than the designated timestamp | | `invalid EXPIRE ROWS KEEP LATEST PARTITION BY column: ...` | A `PARTITION BY` key column does not exist | -| `EXPIRE ROWS KEEP HIGHEST/LOWEST requires a BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, DATE, TIMESTAMP or DECIMAL column, but '' is ; use KEEP HIGHEST/LOWEST to rank an orderable column of any type` | The bare `KEEP HIGHEST/LOWEST` form was given an unsupported column type; use a supported numeric/date type or the top-N form | +| `EXPIRE ROWS KEEP HIGHEST/LOWEST requires a BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, DATE, TIMESTAMP or DECIMAL column, but '' is ; use KEEP HIGHEST/LOWEST ON to rank an orderable column of any type` | The bare `KEEP HIGHEST/LOWEST` form was given an unsupported column type; use a supported numeric/date type or the top-N form | | `EXPIRE ROWS KEEP HIGHEST/LOWEST requires an orderable column, but '' is ` | The top-N form was given a column type that cannot be ordered | | `EXPIRE ROWS KEEP / window retention cannot be used on a view with a column named '__qdb_re_keep'` | The view exposes a column named like the reserved keep column | | `invalid EXPIRE ROWS predicate: ...` | The predicate does not parse, bind, or type-check against the view's columns | diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index e0f40be1f8..60f24d32e0 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -41,7 +41,7 @@ Where: `GROUP BY`), or a [passthrough](/docs/concepts/materialized-views/#passthrough-views) query that copies rows from a single table -- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON timestamp] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]`. +- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON timestamp] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) ON col [PARTITION BY cols]`. This policy is meant for passthrough views (see below). ## Parameters @@ -334,7 +334,7 @@ through a join: ```questdb-sql title="Passthrough view that keeps the latest row per symbol" CREATE MATERIALIZED VIEW trades_latest AS ( SELECT * FROM trades -) EXPIRE ROWS KEEP LATEST PARTITION BY symbol; +) EXPIRE ROWS KEEP LATEST ON timestamp PARTITION BY symbol; ``` A `WHEN` condition is for rules that depend on the **current time**, such as a @@ -350,7 +350,7 @@ The clause goes after the query (and after `PARTITION BY` if present): EXPIRE ROWS { WHEN predicate | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] - | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } + | KEEP [ N ] ( HIGHEST | LOWEST ) ON col [ PARTITION BY col [, col ...] ] } [ CLEANUP EVERY duration ] ``` From 7e1af771c2143e1c6ef3a3d5f4dbe92a294ff165 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Mon, 28 Sep 2026 12:33:49 +0200 Subject: [PATCH 24/25] Document fixed columns of passthrough views QuestDB now stores a passthrough materialized view's query with its top-level SELECT * expanded into the base table's columns at CREATE, so a column the base table gains later, through ALTER TABLE ... ADD COLUMN or a new ILP field, does not reach the view. The view keeps refreshing its original columns and stays valid. The Passthrough views section of the materialized views concept page gains a "Columns are fixed at creation" subsection. It shows the expanded query that materialized_views() and SHOW CREATE MATERIALIZED VIEW report, and states the limitation in a note: to pick up a new base column, drop and recreate the view. The note covers what recreating costs (base-table TTL may already have removed rows, and the EXPIRE ROWS policy has to be restated), contrasts SQL views, whose SELECT * includes later columns, and links to View invalidation for dropped and renamed base columns. The Rules and defaults table on the CREATE MATERIALIZED VIEW page gains a row that links to the new subsection. Co-Authored-By: Claude Opus 5.5 --- documentation/concepts/materialized-views.md | 39 ++++++++++++++++++++ documentation/query/sql/create-mat-view.md | 1 + 2 files changed, 40 insertions(+) diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index 0fdf6eb841..547f7dd943 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -298,6 +298,45 @@ clause: lookup on the view costs the same as on the base table. An aggregating view never inherits an index, because its rows are not base rows. +### Columns are fixed at creation + +A passthrough view's columns are set when you create it. If the query uses +`SELECT *`, QuestDB expands the `*` into the base table's columns at that moment +and stores the query with that column list. The `view_sql` column of +[`materialized_views()`](/docs/query/functions/meta/#materialized_views) and the +output of `SHOW CREATE MATERIALIZED VIEW` both show the expanded query: + +```questdb-sql title="SELECT * is stored as a column list" +CREATE MATERIALIZED VIEW trades_copy AS (SELECT * FROM trades); + +SELECT view_sql FROM materialized_views() WHERE view_name = 'trades_copy'; +``` + +| view_sql | +| --------------------------------------------------------- | +| SELECT symbol, side, price, amount, timestamp FROM trades | + +A column that the base table gains later, through `ALTER TABLE ... ADD COLUMN` +or a new field in InfluxDB Line Protocol (ILP) ingestion, does not appear in the +view. The view keeps refreshing the columns it was created with and stays valid. + +:::note Limitation: new base-table columns do not reach the view + +To include a column added after the view was created, drop the view and create +it again. The new view fills itself from the rows the base table holds at that +point, so rows that a [TTL](/docs/concepts/ttl/) on the base table has already +removed do not come back. Include any `EXPIRE ROWS` policy in the new `CREATE` +statement, because dropping the view drops its policy too. + +A [SQL view](/docs/concepts/views/) behaves differently: it runs its query each +time you read it, so its `SELECT *` includes columns added later. + +::: + +Dropping or renaming a base-table column that the view reads invalidates the +view, as it does for any materialized view. See +[View invalidation](#view-invalidation). + ### Which queries are passthrough The rule is simple: each view row must match one base row. A query that reads a diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 60f24d32e0..b5e1509b8f 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -65,6 +65,7 @@ Where: | Rule | Description | | ---- | ----------- | | Query must aggregate or be passthrough | Either `SAMPLE BY` / `GROUP BY` with a designated timestamp, or a [passthrough](/docs/concepts/materialized-views/#passthrough-views) query that copies rows one-for-one from a single table | +| Passthrough columns fixed at creation | A passthrough view stores `SELECT *` expanded into the base table's current columns. A column added to the base table later does not reach the view; recreate the view to include it. See [Columns are fixed at creation](/docs/concepts/materialized-views/#columns-are-fixed-at-creation) | | Default refresh | `IMMEDIATE` (refreshes after each base table transaction) | | WITH BASE required | Must specify when query contains JOINs | | PARTITION BY sizing | Should be larger than or equal to `SAMPLE BY` interval | From fbc3b0ace9f0c24afc254c3a43313abfc412b4c0 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Tue, 29 Sep 2026 13:38:54 +0200 Subject: [PATCH 25/25] Note EXPIRE ROWS views on time-frame join pages WINDOW JOIN and HORIZON JOIN accept only a table on their right-hand side. A materialized view with an EXPIRE ROWS policy qualifies only when the policy compares the designated timestamp with a constant, now(), or dateadd() over now(). Any other policy reads the view through a filtered sub-query, and the join fails with an error that names the view. The requirements lists on the WINDOW JOIN and HORIZON JOIN pages now say so in one sentence each, with an example policy that qualifies. Co-Authored-By: Claude Opus 5.5 --- documentation/query/sql/horizon-join.md | 4 ++++ documentation/query/sql/window-join.md | 6 +++++- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/documentation/query/sql/horizon-join.md b/documentation/query/sql/horizon-join.md index df2fef0425..51ef4efadb 100644 --- a/documentation/query/sql/horizon-join.md +++ b/documentation/query/sql/horizon-join.md @@ -354,6 +354,10 @@ Look for these indicators in the plan: right-hand tables must each have a designated timestamp column. - **Right-hand side must be a table**: Each right-hand side of HORIZON JOIN must be a table with an optional filter, more complex subqueries aren't supported. + A materialized view with an [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy + qualifies only when the policy compares the designated timestamp with a + constant, `now()`, or `dateadd()` over `now()`, such as + `WHEN ts < dateadd('d', -30, now())`. - **Left-hand side queries are restricted as well**. On the left hand side, the query that works best is a table with an optional filter. Some other query types are also supported, but they degrade the query plan to single-threaded diff --git a/documentation/query/sql/window-join.md b/documentation/query/sql/window-join.md index 8d57fd9bd2..bb142b04e1 100644 --- a/documentation/query/sql/window-join.md +++ b/documentation/query/sql/window-join.md @@ -116,7 +116,11 @@ static bounds when a fixed window size is sufficient. 1. Both tables must have [designated timestamps](/docs/concepts/designated-timestamp/) and be partitioned -2. The right table must be a direct table reference, not a subquery +2. The right table must be a direct table reference, not a subquery. A + materialized view with an [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy + qualifies only when the policy compares the designated timestamp with a + constant, `now()`, or `dateadd()` over `now()`, such as + `WHEN ts < dateadd('d', -30, now())` 3. Aggregate functions are required - you cannot select non-aggregated columns from the right table 4. Symbol-based join conditions enable "Fast Join" optimization when matching on