Skip to content

test(tui-tests): measure Next, long history and running turns in the perf harness - #442

Open
BunsDev wants to merge 1 commit into
mainfrom
test/417-next-perf
Open

BunsDev wants to merge 1 commit into
mainfrom
test/417-next-perf

Conversation

@BunsDev

@BunsDev BunsDev commented Nov 4, 2026

Copy link
Copy Markdown
Member

Summary

Extends the #425 performance harness to measure Next, long transcripts and running turns, and records the first matched Classic and Next results. It finds that Next on main fails #417's memory gate at 100,000 lines, and that #440 fixes it.

Context

Changes

  • --ui next selects Next. Classic runs set no selector, so they stay comparable with the test(tui-tests): add raw PTY performance baseline harness #425 baseline.
  • --history-lines N resumes a synthetic session. A resume must prove it loaded, because a failed one falls back to an empty session and would look like a very fast long conversation. Next must show seeded text; Classic must print the CLI's load confirmation, because its 80x24 welcome header can leave no transcript rows.
  • --stream runs a turn against a fake claude in the disposable home, then reports input-to-paint during the turn and the gap between paints.
  • Keys are checked to confirm they cannot match seeded text, streamed text or the working status.

Results

No history

Measure Classic, main Next, main Next, #440
Cold usable frame p50 (ms) 69-77 69-81 71-85
Input-to-paint p95 (ms) 0.6-0.9 0.3-0.6 0.4-0.6
Resize-to-paint p95 (ms) 1.3-1.8 0.3-1.5 0.3-1.5
Idle RSS p50 (MiB) 46 46 46
Idle PTY writes per second 58 58-59 58
Input-to-paint during a turn, p95 (ms) 0.6-1.1 0.4-0.9 0.2-0.8
Paint gap during a turn, p95 (ms) 17.5-17.8 17.3-17.8 17.3-17.7

100,000 transcript lines

Measure Classic, main Next, main Next, #440
Cold usable frame p50 (ms) 143-155 82-100 84-198
Input-to-paint p95 (ms) 14.1-17.9 10.9-11.9 0.5-0.8
Resize-to-paint p95 (ms) 79.0-104.4 10.6-12.1 13.3-24.3
Idle RSS p50 (MiB) 165-166 251-253 73-74
Idle PTY writes per second 32-33 36 58
Input-to-paint during a turn, p95 (ms) n/a 15.6-17.7 0.6-0.9
Paint gap during a turn, p95 (ms) n/a 28.0-28.9 17.5-17.6

Each cell is the range across both sizes (80x24, 120x40) and both passes, using each profile's summary from 5 runs. Each 100,000-line Next profile has about 100 key samples per run set and about 575 paint gaps. The two passes ran in opposite order to expose load drift. Classic at 100,000 lines has no turn figures because of #441.

Against #417's gates

Gate Next, main Next, #440
Key to visible p95 under 50 ms Pass Pass
Resize p95 under 100 ms Pass Pass
Active render under 33 ms (paint gap proxy) Pass, 28.9 ms at worst Pass, 17.6 ms at worst
Cold start within 10% of Classic Inconclusive with no history; pass at 100,000 lines Same, see limits
RSS within 15% of Classic, no history Pass Pass
RSS within 15% of Classic, 100,000 lines Fail, about 52% over Pass, about 55% under
No idle redraw loop Fail, about 58 writes per second Fail, about 58 writes per second

What this says about #440

  • Key press at 100,000 lines: p95 falls from about 11 ms to under 1 ms.
  • Key press during a turn: p95 falls from about 17 ms to under 1 ms.
  • Memory at 100,000 lines: falls from about 252 MiB to about 74 MiB.
  • Resize at 100,000 lines gets slower: p95 rises from about 12 ms to as much as 24 ms, because width changes rebuild the kept rows. This is still far inside the gate.

Limits

  • One macOS host under other load (load average 7.8 to 15), so these are not reference-machine results.
  • Release builds from source, not packaged archives. Binaries: main 3f73b63 (sha256 5fd76a57), feat(tui): keep Next transcript rows between frames and add jump to start (#439) #440 6f16a8a (sha256 53f55406). macOS 27.2, Apple M5 Max, rustc 1.99.1.
  • Cold start is too noisy here to settle a 10% limit: no-history medians overlap (Classic 69-77 ms, Next 69-81 ms). feat(tui): keep Next transcript rows between frames and add jump to start (#439) #440's 100,000-line range of 84-198 ms comes from one loaded pass; the earlier matrix had it at 68-72 ms. A quiet-host rerun should settle this.
  • The paint gap is a boundary proxy. Its floor is the 16 ms event-loop poll, and it depends on the fake producer's pace.
  • Resize records the first visible chunk, not a completed frame.

Validation

  • python3 -m unittest discover -s scripts/tui-tests -p 'test_*.py': new tests failed first, then pass.
  • Terminal echo verified off during probes, so key timing measures the engine, not the kernel.
  • Two-pass matrix: 120 runs, with failures only for Classic --stream at 100,000 lines (20 of 20, Classic transcript height wraps past 65,535 rows, hiding the live end of the conversation #441).
  • git diff --check: clean.
  • N/A: cargo fmt/check/clippy/test; no Rust changed.
  • Not run: Linux, Windows.

Refs #417, #439

…perf harness

The #425 harness only drove Classic with an empty session, so it could not
measure #417's gates for Next or the cost #439 is about.

- --ui next selects the Next renderer; Classic runs set no selector, so they
  stay comparable with the recorded baseline.
- --history-lines N resumes a synthetic session. A resume must prove it
  loaded, because a failed one falls back to an empty session and would
  measure as a very fast long conversation.
- --stream runs a turn against a fake claude and reports input-to-paint and
  the gap between paints during it.

Refs #417, #439
@vercel

vercel Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Nov 4, 2026 1:15pm UTC

Request Review

This branch was successfully deployed

1 active deployment
Preview — aa1de02d Deployed Nov 4, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant