Skip to content

Performance: speed up building the incidence tables - #234

Open
danielsclint wants to merge 1 commit into
ActivitySim:masterfrom
danielsclint:incidence-table-bincount
Open

danielsclint wants to merge 1 commit into
ActivitySim:masterfrom
danielsclint:incidence-table-bincount

Conversation

@danielsclint

@danielsclint danielsclint commented Nov 10, 2026 •

Copy link
Copy Markdown

Performance: speed up building the incidence tables

Performance-only refactor. This PR does not change what the incidence tables contain. The incidence table, grouped incidence table and household_groups are identical to before, including dtypes and index; only the way they are computed changes. Example outputs are byte-identical.

Problem

setup_data_structures spends most of its time in two places on large seeds:

  • build_incidence_table runs a groupby over every person for each person control, so household ids are re-factorized once per control. The result is a one-column DataFrame, and assigning it to the incidence table forces a reindex.
  • build_grouped_incidence_table recomputes each household's group_id by merging the household table against the grouped table on every control column plus the seed geography.

Change

  • build_incidence_table factorizes the persons' household ids once and maps them to household rows with get_indexer. For each person control whose incidence is int64 (bool expressions included, after the existing * 1), the household sums come from np.bincount as a Series aligned to the households index.
    • np.bincount accumulates in float64. It is used only when that is exact: when max(|value|) * n_persons < 2**53, every partial sum is below 2**53. The incidence must also be aligned with the persons table.
    • Float expressions, households without persons, persons whose household id is missing, and anything else outside that case keep the existing groupby. Its dtype and NaN rules therefore still apply; for example, a household without persons still gets NaN and the column stays float64.
  • build_grouped_incidence_table takes group_id from hh_grouper.ngroup(). The grouper sorts its keys, so its group numbers follow the row order of the grouped table it produced, which is the same group_id the merge found.

The incidence table, grouped incidence table and household_groups are identical to before: same values, dtypes and index.

Evidence

Tests. tests/test_incidence_table.py restates the previous implementations and compares them with the new functions using assert_frame_equal(check_exact=True), which also checks dtypes and index. The synthetic seeds cover:

  • int, bool, np.where and float person expressions
  • an int expression too large for the float64 sum
  • households without persons
  • NaN household ids
  • persons whose household is not in the households table
  • shuffled persons with a non-default index
  • more than 100 groups

The test also checks that the np.bincount path runs for the int and bool controls where it should, and nowhere else. Full suite: 51 passed (41 existing + 10 new).

Benchmark. python scripts/benchmark_incidence_table.py uses synthetic data only: about 2.4 persons per household and 52 controls, of which 49 are person controls. Time is wall clock for one call. Peak is the tracemalloc peak of a separate call. In the grouped benchmark, both sides include an incidence.copy() of their input.

households persons function old s new s speedup old peak MiB new peak MiB
100,000 239,751 build_incidence_table 0.35 0.11 3.4x 55 48
100,000 239,751 build_grouped_incidence_table 0.17 0.08 2.3x 261 136
1,000,000 2,400,515 build_incidence_table 3.16 0.79 4.0x 511 475
1,000,000 2,400,515 build_grouped_incidence_table 2.04 1.02 2.0x 2516 1308

The script asserts that the old and new tables are identical at each size.

End to end. I ran examples/example_calm and examples/example_test with python -m populationsim -w <copy> -c configs -d data -o output. The old side used a clean master (64449c8) checkout on PYTHONPATH, and I checked that the module resolved there. The new side used this branch. Both examples set GROUP_BY_INCIDENCE_SIGNATURE: True, so both changed functions run.

  • Every output CSV except mem.csv and timing_log.csv is byte-identical: 7 of 7 for each example.
  • Every non-checkpoint pipeline.h5 table is identical under assert_frame_equal(check_exact=True): 37 tables for calm, 35 for test.
  • The setup_data_structures step time comes from mem.csv timestamps:
    • calm: 0.059 s old, 0.048 s new
    • test: 0.033 s old, 0.026 s new
  • RSS growth across the step:
    • calm: +6.7 MB old, +6.2 MB new
    • test: +2.2 MB old, +1.7 MB new
  • Whole-process max RSS for calm ranged from 2.5 to 3.07 GB across repeated runs of both old and new. The peak comes in the summarize step, which this change does not touch, and the example seeds are small.

Not covered locally

  • These results are from macOS with Python 3.12 only (pandas 3.0.3, numpy 2.4.6). Other Python versions, Linux and Windows rely on CI.
  • I did not test the multiprocessing configuration end to end beyond the existing test_steps_mp.
  • Timings are single runs on one machine. Across two benchmark runs, the old grouped timing at 1M households moved from 2.97 s to 2.04 s; the other old timings moved by under 5%.

🤖 Generated with Claude Code

build_incidence_table ran a groupby over every person for each person
control, re-factorizing household ids each time, and assigned the
one-column result as a DataFrame, which forced a reindex. Household ids
are now factorized once and int64 person incidence (which includes
bool expressions, after the existing `* 1`) is summed with np.bincount
into a Series aligned to the households index.

np.bincount accumulates in float64, so it is used only when that is
exact (every partial sum below 2**53) and when the incidence is aligned
with the persons table. Float expressions, households without persons
and persons without a household id keep the groupby, so its dtype and
NaN rules still apply and the table is unchanged.

build_grouped_incidence_table recomputed each household's group_id by
merging on all control columns. hh_grouper.ngroup() gives the same ids
directly, since the grouper sorts its keys in the same order as the
grouped table it produced.

Tests compare both functions with the previous implementations,
restated in the test, on synthetic seeds (integer, bool and float
expressions, households without persons, missing and unknown household
ids, shuffled persons, many groups) with exact frame equality including
dtypes and index. scripts/benchmark_incidence_table.py times old against
new on synthetic data and checks the tables are identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@danielsclint danielsclint changed the title Speed up building the incidence tables Performance: speed up building the incidence tables Nov 10, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant