Skip to content

SQLite import for mimic3 gives mixed column type warning #1237

Open
Open
SQLite import for mimic3 gives mixed column type warning#1237

Description

@armando-fandango
armando-fandango
opened on Jan 25, 2022

Prerequisites

Description

While trying to import mimic3 into SQLite with import.py, I get the following error:

Starting processing DATETIMEEVENTS.csv.gz
mimic-code/mimic-iii/buildmimic/sqlite/import.py:25: DtypeWarning: Columns (13) have mixed types. Specify dtype option on import or set low_memory=False.
  for chunk in pd.read_csv(f, index_col="ROW_ID", chunksize=CHUNKSIZE):
...
Starting processing INPUTEVENTS_CV.csv.gz
/home/armando/projects/mimic-code/mimic-iii/buildmimic/sqlite/import.py:25: DtypeWarning: Columns (20,21) have mixed types. Specify dtype option on import or set low_memory=False.
  for chunk in pd.read_csv(f, index_col="ROW_ID", chunksize=CHUNKSIZE):
...
Starting processing NOTEEVENTS.csv.gz
/home/armando/projects/mimic-code/mimic-iii/buildmimic/sqlite/import.py:25: DtypeWarning: Columns (4,5) have mixed types. Specify dtype option on import or set low_memory=False.
  for chunk in pd.read_csv(f, index_col="ROW_ID", chunksize=CHUNKSIZE):
...
Starting processing CHARTEVENTS.csv.gz
/home/armando/projects/mimic-code/mimic-iii/buildmimic/sqlite/import.py:25: DtypeWarning: Columns (13) have mixed types. Specify dtype option on import or set low_memory=False.
  for chunk in pd.read_csv(f, index_col="ROW_ID", chunksize=CHUNKSIZE):
...

Activity

  1. pshuwei commented on Jun 9, 2023

    @pshuwei
    pshuwei
    on Jun 9, 2023

    Hi, I also am running the import.py code and I ran into the same problem...

    Did you manage to figure it out or find an alternative solution?

  2. alistairewj commented on Jun 9, 2023

    @alistairewj
    alistairewj
    on Jun 9, 2023
    Member

    It's not strictly an error but it may result in an inconsistent data load (I haven't checked). Essentially the load uses pandas as a convenience. pandas tries a low memory load, fails, and reverts to a high memory load. It can be fixed by specifying the known data types for each table in the read_csv call.

  3. armando-fandango commented on Jun 21, 2023

    @armando-fandango
    armando-fandango
    on Jun 21, 2023
    ContributorAuthor

    Since the column types are already known in advance and are not going to change since its a frozen/snapshot dataset, hence would it be good to add the column type to the import script? I can send a pull request if this solution is acceptable.

  4. alistairewj commented on Jun 21, 2023

    @alistairewj
    alistairewj
    on Jun 21, 2023
    Member

    Yes it would for sure, and yes we would love a PR!

  5. Chessing234 commented on Jul 19, 2026

    @Chessing234
    Chessing234
    on Jul 19, 2026
    Contributor

    pinned the mixed columns + low_memory=false in import.py. pr up.

  6. Chessing234 commented on Jul 19, 2026

    @Chessing234
    Chessing234
    on Jul 19, 2026
    Contributor

    also opened #2082 so plain .csv (not only .csv.gz) loads in the iii sqlite import.py.

  7. Chessing234 commented on Jul 23, 2026

    @Chessing234
    Chessing234
    on Jul 23, 2026
    Contributor

    addressed in #2124

  8. Chessing234 commented on Jul 23, 2026

    @Chessing234
    Chessing234
    on Jul 23, 2026
    Contributor

    sqlite import.py pins dtypes + low_memory=False. See #2124.

  9. Chessing234 commented on Jul 23, 2026

    @Chessing234
    Chessing234
    on Jul 23, 2026
    Contributor

    sqlite import dtypes pinned in #2124.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions