Skip to content

dotnet-gcdump: report verb - #791

Merged
josalem merged 28 commits into
dotnet:masterfrom
mustakimali:dotnet-gcdump-stdout
Apr 21, 2020
Merged

josalem merged 28 commits into
dotnet:masterfrom
mustakimali:dotnet-gcdump-stdout

Conversation

@mustakimali

@mustakimali mustakimali commented Jan 30, 2020 •

Copy link
Copy Markdown
Contributor

Files produced by dotnet-gcdump can not be viewed without Visual Studio or PerfView.
This PR introduces a new verb to generate report in the stdout. This will
allow viewing the results when a Windows PC isn't available.

  • dotnet gcdump report [-t heapstat] <gcdump_filename> - Reads a previously generated gcdump file and produces a heapstat report in the stdout.
  • dotnet gcdump report -p <processId> [-t heapstat] - Generate a report from a running dotnet process.

Report type (-t) is optional as only only one type of reports are produced. Other type of reports may be added on the future (#809 (comment)). I have not idea about how to get those information from the gcdump file at the moment so .I had to keep them outside of the scope of this PR.

Proposal is here: #809

@dnfclas

dnfclas commented Jan 30, 2020 •

Copy link
Copy Markdown

CLA assistant check
All CLA requirements met.

@mustakimali
mustakimali marked this pull request as ready for review January 30, 2020 22:46
@mustakimali
mustakimali force-pushed the dotnet-gcdump-stdout branch 2 times, most recently from bd897fa to 155b9f4 Compare January 30, 2020 23:09

@josalem josalem left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First round of feedback. I really like this! Have you considered adding the ability to read an existing and print this? I believe with this change, all the necessary logic is available.. We would just need to add a dotnet gcdump print file.gcdump (or some other verb) command to the tool.

@mustakimali

Copy link
Copy Markdown
Contributor Author

First round of feedback. I really like this! Have you considered adding the ability to read an existing and print this? I believe with this change, all the necessary logic is available.. We would just need to add a dotnet gcdump print file.gcdump (or some other verb) command to the tool.

That's a great idea, didn't cross my mind. Would you prefer a separate PR for this or this one?

@josalem

josalem commented Jan 31, 2020

Copy link
Copy Markdown
Contributor

Would you prefer a separate PR for this or this one?

I think it's fine to do that in another PR 😃. In preparation for that PR though, I might refactor the printing logic into its own file for reuse in another Command(the S.CommandLine API that equates to verbs).

@mustakimali mustakimali changed the title dotnet-gcdump: Allow writing the data into stdout dotnet-gcdump: Allow reading a .gcdump file and print the data into stdout Feb 1, 2020
@mustakimali
mustakimali force-pushed the dotnet-gcdump-stdout branch 3 times, most recently from 39b8ff7 to c9889cc Compare February 1, 2020 01:44
@mustakimali mustakimali changed the title dotnet-gcdump: Allow reading a .gcdump file and print the data into stdout dotnet-gcdump: Allow reading a .gcdump file and print the data into stdout cross-platform Feb 1, 2020
josalem
josalem previously approved these changes Feb 3, 2020

@josalem josalem left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to try running this locally a few times to try some scenarios out. I'm going to approve it for now, and merge it once I've had a chance to play with it a bit 😄

@josalem

josalem commented Feb 3, 2020

Copy link
Copy Markdown
Contributor

As a heads up, I just chatted with @noahfalk and we want to do a little more thinking to make sure the naming for the verb and option make sense. If you have some extra cycles, could you do another PR that adds the verb and option to the gcdump spec (under docs/design-docs). That will give us a good spot to discuss the appropriate names. The code here looks good, we just need to settle on the names.

@mustakimali

Copy link
Copy Markdown
Contributor Author

Naming is hard! Let's settle this on another PR first. Sure I'll do that. 👍

@mustakimali

mustakimali commented Feb 7, 2020 •

Copy link
Copy Markdown
Contributor Author

As a heads up, I just chatted with @noahfalk and we want to do a little more thinking to make sure the naming for the verb and option make sense. If you have some extra cycles, could you do another PR that adds the verb and option to the gcdump spec (under docs/design-docs). That will give us a good spot to discuss the appropriate names. The code here looks good, we just need to settle on the names.

Thank you, I've had a chance to create the separate PR to include the verb option. Let me know if that works.

#809

@noahfalk
noahfalk requested a review from josalem February 26, 2020 19:11
@noahfalk

Copy link
Copy Markdown
Member

@josalem - I cleared your previous review to make it apparent that the work has been ongoing and I assume an updated review would be necessary once it finishes.

@noahfalk
noahfalk dismissed josalem’s stale review February 26, 2020 19:15

Review is weeks old and work has been ongoing

@mustakimali
mustakimali force-pushed the dotnet-gcdump-stdout branch from 4df026c to 6fb642f Compare March 30, 2020 00:09
@mustakimali
mustakimali requested a review from josalem April 5, 2020 23:27

@josalem josalem left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've noticed there seem to always bee a set of entries in the reports that look like this:

			 -1         0  System.Threading.Tasks.VoidTaskResult  [System.Private.CoreLib.dll]
             -1         0  System.UInt16  [System.Private.CoreLib.dll]
             -1         0  System.UInt32  [System.Private.CoreLib.dll]
             -1         0  System.Char  [System.Private.CoreLib.dll]
             -1         0  System.IntPtr  [System.Private.CoreLib.dll]
             -1         0  System.Diagnostics.Tracing.EventSource  [System.Private.CoreLib.dll]
             -1         0  System.Boolean  [System.Private.CoreLib.dll]
             -1         0  System.IO.TextReader  [System.Private.CoreLib.dll]
             -1         0  System.Text.Encoding  [System.Private.CoreLib.dll]
             -1         0  System.IO.TextWriter  [System.Private.CoreLib.dll]
             -1         0  Entry<System.String,System.Object>  [System.Private.CoreLib.dll]
             -1         0  System.SByte  [System.Private.CoreLib.dll]
             -1         0  System.Byte  [System.Private.CoreLib.dll]
             -1         0  System.Globalization.CalendarId  [System.Private.CoreLib.dll]
             -1         0  Entry<System.String,System.Globalization.CultureInfo>  [System.Private.CoreLib.dll]
             -1         0  Entry<System.String,System.IntPtr>  [System.Private.CoreLib.dll]
             -1         0  bucket  [System.Private.CoreLib.dll]
             -1         0  Entry<System.String,System.Globalization.CultureData>  [System.Private.CoreLib.dll]
             -1         0  Entry<System.RuntimeType,System.RuntimeType>  [System.Private.CoreLib.dll]
             -1         0  System.Reflection.CustomAttributeRecord  [System.Private.CoreLib.dll]
             -1         0  System.UInt64  [System.Private.CoreLib.dll]
             -1         0  System.Text.StringOrCharArray  [System.Console.dll]
             -1         0  System.ConsoleKeyInfo  [System.Console.dll]
             -1         0  Entry<System.Text.StringOrCharArray,System.ConsoleKeyInfo>[]  [System.Private.CoreLib.dll]
             -1         0  Entry<System.Text.StringOrCharArray,System.ConsoleKeyInfo>  [System.Private.CoreLib.dll]

I don't think this is necessarily something wrong with your code, but I think the fact that there are 0 count, -1 size entries is worth investigating. I'll take a look at this offline and see what I can dig up. These might be types that have been created but there aren't any "current" instances in the heap once the gcdump was finished.

Other than that, just a couple nits for this round 😄

mustakimali and others added 4 commits April 13, 2020 15:17
Co-Authored-By: John Salem <josalem@microsoft.com>
Co-Authored-By: John Salem <josalem@microsoft.com>
Co-Authored-By: John Salem <josalem@microsoft.com>
Co-Authored-By: John Salem <josalem@microsoft.com>
@mustakimali

Copy link
Copy Markdown
Contributor Author

I've noticed there seem to always bee a set of entries in the reports that look like this:

			 -1         0  System.Threading.Tasks.VoidTaskResult  [System.Private.CoreLib.dll]
             -1         0  System.UInt16  [System.Private.CoreLib.dll]
             -1         0  System.UInt32  [System.Private.CoreLib.dll]
             -1         0  System.Char  [System.Private.CoreLib.dll]
             -1         0  System.IntPtr  [System.Private.CoreLib.dll]
             -1         0  System.Diagnostics.Tracing.EventSource  [System.Private.CoreLib.dll]
             -1         0  System.Boolean  [System.Private.CoreLib.dll]
             -1         0  System.IO.TextReader  [System.Private.CoreLib.dll]
             -1         0  System.Text.Encoding  [System.Private.CoreLib.dll]
             -1         0  System.IO.TextWriter  [System.Private.CoreLib.dll]
             -1         0  Entry<System.String,System.Object>  [System.Private.CoreLib.dll]
             -1         0  System.SByte  [System.Private.CoreLib.dll]
             -1         0  System.Byte  [System.Private.CoreLib.dll]
             -1         0  System.Globalization.CalendarId  [System.Private.CoreLib.dll]
             -1         0  Entry<System.String,System.Globalization.CultureInfo>  [System.Private.CoreLib.dll]
             -1         0  Entry<System.String,System.IntPtr>  [System.Private.CoreLib.dll]
             -1         0  bucket  [System.Private.CoreLib.dll]
             -1         0  Entry<System.String,System.Globalization.CultureData>  [System.Private.CoreLib.dll]
             -1         0  Entry<System.RuntimeType,System.RuntimeType>  [System.Private.CoreLib.dll]
             -1         0  System.Reflection.CustomAttributeRecord  [System.Private.CoreLib.dll]
             -1         0  System.UInt64  [System.Private.CoreLib.dll]
             -1         0  System.Text.StringOrCharArray  [System.Console.dll]
             -1         0  System.ConsoleKeyInfo  [System.Console.dll]
             -1         0  Entry<System.Text.StringOrCharArray,System.ConsoleKeyInfo>[]  [System.Private.CoreLib.dll]
             -1         0  Entry<System.Text.StringOrCharArray,System.ConsoleKeyInfo>  [System.Private.CoreLib.dll]

I don't think this is necessarily something wrong with your code, but I think the fact that there are 0 count, -1 size entries is worth investigating. I'll take a look at this offline and see what I can dig up. These might be types that have been created but there aren't any "current" instances in the heap once the gcdump was finished.

Other than that, just a couple nits for this round 😄

Should I just ignore those lines from the report?

@josalem

josalem commented Apr 17, 2020

Copy link
Copy Markdown
Contributor

Sorry for the delay! I'm currently investigating what these entries are. They appear to be type events that came across without there being any instances of the types in the GC heap. This is strange, though, since the list of types that get sent during collection should only contain the list of types that the GC saw when it was enumerating the heap. This is something that I think is worth investigating, but shouldn't necessarily block this PR. For now, let's hide these 0 count entries from the report and we can review the results of that.

@josalem

josalem commented Apr 17, 2020

Copy link
Copy Markdown
Contributor

I just ran some simple experiments and found the following: These entries show up in reports made from gcdumps collected using PerfView (our control group in this case), but not in the PerfView UI. This leads me to believe that these are innocuous. I'll dig through the PerfView code at some point in the future to see if there are any notes about what these entries mean. For now, feel free to make the reports ignore 0 count entries 😄

@mustakimali
mustakimali force-pushed the dotnet-gcdump-stdout branch from e464733 to 4f1bf92 Compare April 19, 2020 13:05
@mustakimali

ghost commented Apr 19, 2020

Copy link
Copy Markdown
Contributor Author

These entries show up in reports made from gcdumps collected using PerfView (our control group in this case), but not in the PerfView UI.

Thanks for looking into this 🙇 , so this means Perfview UI had to do the same at some point.
I have updated the PR with the changes (the output is clean now).

ghost left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry this took so long! Thanks for all the effort on this feature! 🚀

CC @noahfalk @sywhang @tommcdon

@josalem
josalem merged commit e6a3e4d into dotnet:master Apr 21, 2020
@mustakimali

ghost commented Apr 21, 2020

Copy link
Copy Markdown
Contributor Author

Sorry this took so long! Thanks for all the effort on this feature! rocket

CC @noahfalk @sywhang @tommcdon

Thank you for helping me along the way with all the useful suggestions 👍

@mustakimali
mustakimali deleted the dotnet-gcdump-stdout branch April 21, 2020 22:57
@colotiline

ghost commented Jul 15, 2020

Copy link
Copy Markdown

Do you have plans when report will be released?

@noahfalk

ghost commented Jul 15, 2020

Copy link
Copy Markdown
Member

@mikem8361 was looking into releasing a new version of our dotnet-* diagnostic tools relatively soon I think? I'll let him follow up with any additional info on timeline if he has it.

If you are eager to get a version before we officially release it syncing and building this repo is hopefully straightforward too. The resulting dotnet-gcdump app is a standard .Net Core console app that can be copied to where you want it.

@mikem8361

ghost commented Jul 15, 2020

Copy link
Copy Markdown
Contributor

I'm hoping to release the official build early next week. If you can't wait you can install the latest build this way:

dotnet tool uninstall -g dotnet-gcdump
dotnet tool install -g dotnet-gcdump --add-source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5/nuget/v3/index.json

@github-actions github-actions Bot locked and limited conversation to collaborators Jan 19, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants