Skip to content

large negative increment values in line profiler #195

Description

@nathantsoi

i am using memory profile as follows:

an_instance = AClass()
precision=1
backend='psutil'
with open("mem-profile.txt", 'w') as stream:
  prof = mp.LineProfiler(backend=backend)
  res = prof(an_instance.window)(template)
  mp.show_results(prof, stream=stream, precision=precision)
return res

however the results seem strange, I see large negative increments, much larger than any calls with positive increments:

Line #    Mem usage    Increment
================================
    51    192.2 MiB    192.2 MiB
    52                          
    53                          
    54                          
    55                          
    56                          
    57                          
    58    192.2 MiB      0.0 MiB
    59    736.4 MiB     -8.1 MiB
    60    736.4 MiB     -6.8 MiB
    61    736.4 MiB     -7.4 MiB
    62    736.4 MiB     -8.1 MiB
    63                          
    64    736.4 MiB     -7.7 MiB
    65    736.4 MiB     -8.0 MiB
    66    736.4 MiB     -7.9 MiB
    67    736.4 MiB     -8.1 MiB
    68                          
    69    736.4 MiB     -7.9 MiB
    70    736.4 MiB     -8.0 MiB
    71    736.4 MiB     -8.1 MiB
    72    736.4 MiB     -8.1 MiB
    73                          
    74    736.4 MiB     -8.1 MiB
    75    736.4 MiB     -8.1 MiB
    76    736.4 MiB     -8.1 MiB
    77                          
    78                          
    79                          
    80                          
    81    736.4 MiB    -47.7 MiB
    82    736.4 MiB  -1028.5 MiB
    83    736.4 MiB  -5789.3 MiB
    84    736.4 MiB  -4799.5 MiB
    85                          
    86    736.4 MiB  -4796.0 MiB
    87                          
    88                          
    89    736.4 MiB  -4789.0 MiB
    90                          
    91                          
    92                          
    93                          
    94    736.4 MiB  -4790.4 MiB
    95    599.3 MiB      6.2 MiB
    96    599.3 MiB     85.1 MiB
    97    599.3 MiB     48.6 MiB
    98                          
    99    599.3 MiB      0.0 MiB
   100    599.3 MiB     42.1 MiB
   101    599.3 MiB      0.0 MiB
   102    599.3 MiB      2.3 MiB
   103                          
   104    736.4 MiB  -4782.2 MiB
   105    736.4 MiB  -4767.1 MiB
   106    736.4 MiB  -4795.4 MiB
   107    736.4 MiB  -4797.3 MiB
   108                          
   109    736.4 MiB  -4605.9 MiB
   110                          
   111                          
   112                          
   113    736.4 MiB      0.0 MiB
   114    753.1 MiB   -807.8 MiB
   115    753.1 MiB -22614.2 MiB
   116    753.1 MiB -21805.3 MiB
   117    753.1 MiB -21806.2 MiB
   118    753.1 MiB -21806.2 MiB
   119    753.1 MiB -12706.5 MiB
   120                          
   121    753.1 MiB -12691.2 MiB
   122    753.1 MiB -12707.0 MiB
   123                          
   124    753.1 MiB  -9099.8 MiB
   125    753.1 MiB   -807.8 MiB
   126    753.1 MiB   -807.8 MiB
   127                          
   128    753.1 MiB   -807.8 MiB
   129    753.1 MiB   -807.8 MiB
   130    753.1 MiB   -807.8 MiB
   131    753.1 MiB   -807.8 MiB
   132                          
   133    753.1 MiB   -807.8 MiB
   134                          
   135    753.1 MiB   -807.8 MiB
   136    753.1 MiB   -807.8 MiB
   137                          
   138                          
   139    753.1 MiB   -807.7 MiB
   140    753.1 MiB   -807.7 MiB
   141                          
   142    753.1 MiB   -807.8 MiB
   143                          
   144    752.9 MiB     -0.2 MiB
   145                          
   146                          
   147                          
   148                          
   149                          
   150                          
   151                          
   152    752.9 MiB      0.0 MiB

Activity

  1. alowayed commented on Apr 24, 2018

    @alowayed

    Similar issue here. Totals don't add up and the increments are large and negative.

    Line #    Mem usage    Increment   Line Contents
    ================================================
         5   23.891 MiB   23.891 MiB
         6
         7
         8   23.891 MiB    0.000 MiB
         9   23.891 MiB    0.000 MiB
        10
        11   24.137 MiB    0.246 MiB
        12
        13
        14  112.980 MiB   88.844 MiB
        15
        16  112.980 MiB    0.000 MiB
        17
        18  136.840 MiB -2654.883 MiB
        19
        20  136.840 MiB -2676.617 MiB
    
  2. fabianp commented on Apr 30, 2018

    @fabianp
    Collaborator

    Thanks @nathantsoi and @alowayed for the report. This bug doesn't seem to appear in the standard "test" suite (i.e., typing make test from the root directory). Could you paste a minimal reproducing example for this?

  3. karan10111 commented on Jun 1, 2018

    @karan10111

    Any progress on this?

  4. astrojuanlu commented on Jun 1, 2018

    @astrojuanlu
    Collaborator

    @karan10111 we would need some help in having a short, self-contained, correct example to reproduce this. Do you have a short snippet that we can use for that?

  5. alowayed commented on Jun 1, 2018

    @alowayed

    @fabianp I'll need some time to reduce the actual code. This appeared while an XML tree was being walked through Hadoop.

  6. karan10111 commented on Jun 7, 2018

    @karan10111

    So, most of the times you will see this in loops.
    train, test = next(islice(cv.split(X, y), 2, 3))

    cv is from sklearn.

  7. fabianp commented on Jun 7, 2018

    @fabianp
    Collaborator

    Thanks for the info. That makes sense. When a line is executed multiple times, only the maximum is displayed, which might lead to some confusing information being displayed. PRs or ideas on how to handle this better are welcome.

  8. astrojuanlu commented on Jun 14, 2018

    @astrojuanlu
    Collaborator

    Unless we find other cases where this happens, I guess this is a duplicate of #9.

  9. added a commit that references this issue on Jun 28, 2018
    5fe38da
  10. astrojuanlu commented on Jul 1, 2018

    @astrojuanlu
    Collaborator

    @fabianp proposed a fix for this in #202. @karan10111 @alowayed @nathantsoi could you please try pip install https://github.com/fabianp/memory_profiler/archive/fix_195.zip and rerun your examples?

  11. added a commit that references this issue on Aug 16, 2018
    957ff4e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions