Repository navigation
Performance: speed up building the incidence tables - #234
Open
danielsclint wants to merge 1 commit into
Open
danielsclint wants to merge 1 commit into
danielsclint wants to merge 1 commit into
Conversation
build_incidence_table ran a groupby over every person for each person control, re-factorizing household ids each time, and assigned the one-column result as a DataFrame, which forced a reindex. Household ids are now factorized once and int64 person incidence (which includes bool expressions, after the existing `* 1`) is summed with np.bincount into a Series aligned to the households index. np.bincount accumulates in float64, so it is used only when that is exact (every partial sum below 2**53) and when the incidence is aligned with the persons table. Float expressions, households without persons and persons without a household id keep the groupby, so its dtype and NaN rules still apply and the table is unchanged. build_grouped_incidence_table recomputed each household's group_id by merging on all control columns. hh_grouper.ngroup() gives the same ids directly, since the grouper sorts its keys in the same order as the grouped table it produced. Tests compare both functions with the previous implementations, restated in the test, on synthetic seeds (integer, bool and float expressions, households without persons, missing and unknown household ids, shuffled persons, many groups) with exact frame equality including dtypes and index. scripts/benchmark_incidence_table.py times old against new on synthetic data and checks the tables are identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Performance: speed up building the incidence tables
Performance-only refactor. This PR does not change what the incidence tables contain. The incidence table, grouped incidence table and
household_groupsare identical to before, including dtypes and index; only the way they are computed changes. Example outputs are byte-identical.Problem
setup_data_structuresspends most of its time in two places on large seeds:build_incidence_tableruns agroupbyover every person for each person control, so household ids are re-factorized once per control. The result is a one-column DataFrame, and assigning it to the incidence table forces a reindex.build_grouped_incidence_tablerecomputes each household'sgroup_idby merging the household table against the grouped table on every control column plus the seed geography.Change
build_incidence_tablefactorizes the persons' household ids once and maps them to household rows withget_indexer. For each person control whose incidence is int64 (bool expressions included, after the existing* 1), the household sums come fromnp.bincountas a Series aligned to the households index.np.bincountaccumulates in float64. It is used only when that is exact: whenmax(|value|) * n_persons < 2**53, every partial sum is below 2**53. The incidence must also be aligned with the persons table.groupby. Its dtype and NaN rules therefore still apply; for example, a household without persons still gets NaN and the column stays float64.build_grouped_incidence_tabletakesgroup_idfromhh_grouper.ngroup(). The grouper sorts its keys, so its group numbers follow the row order of the grouped table it produced, which is the samegroup_idthe merge found.The incidence table, grouped incidence table and
household_groupsare identical to before: same values, dtypes and index.Evidence
Tests.
tests/test_incidence_table.pyrestates the previous implementations and compares them with the new functions usingassert_frame_equal(check_exact=True), which also checks dtypes and index. The synthetic seeds cover:np.whereand float person expressionsThe test also checks that the
np.bincountpath runs for the int and bool controls where it should, and nowhere else. Full suite: 51 passed (41 existing + 10 new).Benchmark.
python scripts/benchmark_incidence_table.pyuses synthetic data only: about 2.4 persons per household and 52 controls, of which 49 are person controls. Time is wall clock for one call. Peak is the tracemalloc peak of a separate call. In the grouped benchmark, both sides include anincidence.copy()of their input.The script asserts that the old and new tables are identical at each size.
End to end. I ran
examples/example_calmandexamples/example_testwithpython -m populationsim -w <copy> -c configs -d data -o output. The old side used a cleanmaster(64449c8) checkout onPYTHONPATH, and I checked that the module resolved there. The new side used this branch. Both examples setGROUP_BY_INCIDENCE_SIGNATURE: True, so both changed functions run.mem.csvandtiming_log.csvis byte-identical: 7 of 7 for each example.pipeline.h5table is identical underassert_frame_equal(check_exact=True): 37 tables for calm, 35 for test.setup_data_structuresstep time comes frommem.csvtimestamps:summarizestep, which this change does not touch, and the example seeds are small.Not covered locally
test_steps_mp.🤖 Generated with Claude Code