Repository navigation
fix(propensity): seed and stratify the early-stopping validation split - #1051
Merged
jeongyoonlee merged 1 commit intoOct 3, 2026
Merged
jeongyoonlee merged 1 commit into
jeongyoonlee merged 1 commit into
Conversation
GradientBoostedPropensityModel drew its early-stopping validation set with no random_state and no stratification. The classifier was seeded, but the split came from the global numpy stream, so the same seed gave different propensity scores on every fit. The split was also not stratified on treatment, so rare-treatment data could leave the validation set with no treated units. Pass the same seed the classifier uses, and stratify on treatment when every class has at least two members. Fall back to a plain split for degenerate treatment vectors so this does not introduce a new error. Adds two regression tests: repeated fits with the same seed agree, and the split receives the seed and treatment stratification.
|
|
jeongyoonlee
self-requested a review
October 3, 2026 19:35
jeongyoonlee
approved these changes
Oct 3, 2026
jeongyoonlee
left a comment
Collaborator
There was a problem hiding this comment.
LGTM. Thanks for your contribution.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1045.
GradientBoostedPropensityModel.fit()drew its early-stopping validation set with norandom_stateand nostratify. The classifier was seeded, but the split came from the global numpy stream, so the same seed gave different propensity scores on each fit (up to ~0.55 apart in the issue's example). The split was also not stratified on treatment, so rare-treatment data could leave the validation set without treated units.Changes:
self.model_kwargs.get("random_state", 42), sorandom_state=Nonestill means random.ValueErrorpath.Validation:
tests/test_propensity.py: repeated fits with the same seed return identical scores, and the split receives the seed and treatment stratification. Both fail onmasterand pass with this change.pytest tests/test_propensity.py: 9/9 pass.pytest tests: 561 passed. 4 failures are in the uplift tree visualization tests (pydotplusvs the installedpyparsing). They fail the same way without this change.black --checkon both changed files: clean.