Skip to content

Rebuild diabetes from NHANES, and decide what to do about its imputed labels #137

Open

Description

@valterschutz

data/diabetes/diabetes.csv is the GDFS repository's copy of Kachuee et al.'s NHANES 1999–2016 preprocessing (load_diabetes in mkachuee/Opportunistic/nhanes.py); nothing in AFABench rebuilds it (docs/reference/datasets/diabetes.md).

Two problems:

  1. Labels are mostly imputed. The loader fills a missing fasting glucose (LBXGLU) with the mean before thresholding it at 100 and 125 mg/dL (preproc_impute), so every participant without a measurement gets the mean's class. 73,573 of the 92,062 instances (80%) are class 1 (prediabetes), which suggests most class 1 labels are imputed. The file does not record which.
  2. Not reproducible or traceable. The loader shuffles with an unseeded permutation and the file has no participant ID (SEQN) and unnamed columns 0–44; Kachuee's preprocessed diabetes.pkl link is dead (Google Drive 404).

Decide whether to keep the dataset as is, drop it, or port the loader into DiabetesDataset.make_prepared_source (from the NHANES XPT files, keeping SEQN order, naming the columns, and dropping participants without a fasting glucose measurement instead of imputing it). Rebuilding would make a new dataset; results on the current one would not carry over.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageMaintainer needs to evaluate this issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions