Skip to content

[mono][gc] Objects larger than LOS section size call the LOS heap to grow without bound #97991

Closed
#98154
Closed
[mono][gc] Objects larger than LOS section size call the LOS heap to grow without bound#97991
#98154
Labels
area-GC-mono

Description

@lambdageek
lambdageek
opened on Feb 5, 2024

consider this program:

long bigger_than_los_section = 32 * 1024 * 1024;
long bigger_than_small_obj = 100 * 1024;
long small_obj = 50;

long alloc_size = bigger_than_small_obj;

long cnt = 0;
long[] data = new long[1024*1024*32];
var end = DateTime.Now + TimeSpan.FromMilliseconds(10000);
do
{
    // these are just dummy computations to keep CPU busy
    Interlocked.Increment(ref cnt);
    data = new long[alloc_size];
    data[0] = cnt;
    // force some GC
    if(cnt % 10000 == 0)
    {
        GC.Collect();
        var total = GC.GetTotalMemory(true);
        Console.WriteLine("GC.GetTotalMemory: "+total);
                        }
}
while(DateTime.Now<end);

if alloc_size is small_obj then the LOS size stays constant, as we would expect

$ MONO_LOG_MASK=gc MONO_LOG_LEVEL=debug dotnet run
...
Mono: GC_MAJOR: (user request) time 0.05ms, stw 0.08ms los size: 1024K in use: 24K
...
Mono: GC_MAJOR: (user request) time 0.05ms, stw 0.06ms los size: 1024K in use: 24K

if alloc_size is bigger_than_small_obj then we also have a LOS size that stays constant

$ MONO_LOG_MASK=gc MONO_LOG_LEVEL=debug dotnet run
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.23ms, stw 0.12ms los size: 3072K in use: 1625K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.24ms, stw 0.14ms los size: 3072K in use: 1625K

if alloc_size is bigger_than_los_section then the LOS heap size grows bigger and bigger (despite us not actually placing the new long[alloc_size] in the LOS section (!)

$ MONO_LOG_MASK=gc MONO_LOG_LEVEL=debug dotnet run
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.20ms, stw 0.13ms los size: 525344K in use: 524313K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.19ms, stw 0.13ms los size: 787504K in use: 786457K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.14ms, stw 0.09ms los size: 1049664K in use: 1048601K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.20ms, stw 0.14ms los size: 1573984K in use: 1572889K
...

This showed up in a MT WASM test #97970

Activity

  1. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Feb 5, 2024
  2. ghost added
    area-GC-coreclr
    on Feb 5, 2024
  3. lambdageek
    added
    area-GC-mono
    and removed
    area-GC-coreclr
    untriagedNew issue has not been triaged by the area owner
    on Feb 5, 2024
  5. lambdageek
    added
    untriagedNew issue has not been triaged by the area owner
    on Feb 5, 2024

  6. lambdageek commented on Feb 5, 2024

    @lambdageek
    lambdageek
    on Feb 5, 2024
    MemberAuthor

    It's weird that we keep printing a larger and larger LOS size. sgen_los_free_object has a sgen_los_memory_usage_total -= size; in the case where size > LOS_SECTION_OBJECT_LIMIT

    I thought maybe the problem is in the call to ensure_free_space before the size > LOS_SECTION_OBJECT_LIMIT check in this code

    runtime/src/mono/mono/sgen/sgen-los.c

    Lines 442 to 460 in b94a82d

    sgen_ensure_free_space (size, GENERATION_OLD);
    #ifdef USE_MALLOC
    obj = g_malloc (size + sizeof (LOSObject));
    memset (obj, 0, size + sizeof (LOSObject));
    #else
    if (size > LOS_SECTION_OBJECT_LIMIT) {
    size_t obj_size = size + sizeof (LOSObject);
    int pagesize = mono_pagesize ();
    size_t alloc_size = SGEN_ALIGN_UP_TO (obj_size, pagesize);
    if (sgen_memgov_try_alloc_space (alloc_size, SPACE_LOS)) {
    obj = (LOSObject *)sgen_alloc_os_memory (alloc_size, (SgenAllocFlags)(SGEN_ALLOC_HEAP | SGEN_ALLOC_ACTIVATE), NULL, MONO_MEM_ACCOUNT_SGEN_LOS);
    if (obj) {
    sgen_los_memory_usage_total += alloc_size;
    obj = randomize_los_object_start (obj, obj_size, alloc_size, pagesize);
    }
    }
    } else {
    obj = get_los_section_memory (size + sizeof (LOSObject));

    but I don't see ensure_free_space actually asking the memory governor for more space. So it's something in sgen-los itself, I guess.

  7. lambdageek commented on Feb 6, 2024

    @lambdageek
    lambdageek
    on Feb 6, 2024
    MemberAuthor

    Update With the interpreter too, it stays constant, like with the JIT. with MONO_ENV_OPTIONS=--interp the LOS size to a higher point and then oscillates there:

    $ MONO_ENV_OPTIONS=--interp MONO_LOG_MASK=gc MONO_LOG_LEVEL=debug dotnet run
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.21ms, stw 0.07ms los size: 525344K in use: 524313K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.23ms, stw 0.16ms los size: 787504K in use: 786457K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.23ms, stw 0.17ms los size: 2098304K in use: 2097177K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.23ms, stw 0.16ms los size: 2360464K in use: 2359321K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.24ms, stw 0.18ms los size: 2360464K in use: 2359321K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.22ms, stw 0.16ms los size: 2098304K in use: 2097177K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.24ms, stw 0.17ms los size: 1836144K in use: 1835033K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.25ms, stw 0.18ms los size: 2360464K in use: 2359321K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.24ms, stw 0.17ms los size: 2360464K in use: 2359321K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.23ms, stw 0.16ms los size: 2098304K in use: 2097177K

    With the interpreter it climbs higher and then oscillates around 2098304K.
    This is on osx-arm64 with the .NET 8.0.1 SDK

    I think it's a problem with the test case. This variant runs with constant LOS size:

    using System.Runtime.CompilerServices;

long bigger_than_los_section = 32 * 1024 * 1024;
long bigger_than_small_obj = 100 * 1024;
long small_obj = 50;

long alloc_size = bigger_than_los_section;

long cnt = 0;
long[] data = null;
AssignData(ref data, alloc_size);
var end = DateTime.Now + TimeSpan.FromMilliseconds(10000);
do
{
    // these are just dummy computations to keep CPU busy
    Interlocked.Increment(ref cnt);
    AssignData(ref data, alloc_size);
    data[0] = cnt;
    // force some GC
    if(cnt % 10000 == 0)
    {
        GC.Collect();
        // GC.WaitForPendingFinalizers();
        // GC.Collect();
        // GC.WaitForPendingFinalizers();
        var total = GC.GetTotalMemory(true);
        Console.WriteLine("GC.GetTotalMemory: "+total);
    }
}
while(DateTime.Now<end);


[MethodImpl(MethodImplOptions.NoInlining)]
static void AssignData (ref long[] d, long size, int recurse = 20)
{
    if (recurse >= 0) {
        AssignData(ref d, size, recurse - 1);
    } else {
        d = new long[size];
    }
}
    ...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.22ms, stw 0.14ms los size: 525344K in use: 524313K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.14ms, stw 0.08ms los size: 1049664K in use: 1048601K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.23ms, stw 0.16ms los size: 787504K in use: 786457K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.23ms, stw 0.17ms los size: 787504K in use: 786457K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.23ms, stw 0.17ms los size: 787504K in use: 786457K

    it starts a bit lower, spikes and then reaches a steady state of 787504K

    @pavelsavara @BrzVlad

  8. lambdageek commented on Feb 6, 2024

    @lambdageek
    lambdageek
    on Feb 6, 2024
    MemberAuthor

    I can get the interpreter to hold steady at the same LOS size as the JIT if I chance the recurse dept to be 200 instead of 20

    Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.22ms, stw 0.13ms los size: 787504K in use: 786457K
...
Mono: GC_MAJOR_CONCURRENT_FINISH: (finishing) time 0.20ms, stw 0.12ms los size: 787504K in use: 786457K
  9. ghost added
    in-prThere is an active PR which will close this issue when it is merged
    on Feb 8, 2024
  10. lewing
    closed this as completedin #98154on Feb 9, 2024
  11. ghost removed
    in-prThere is an active PR which will close this issue when it is merged
    on Feb 9, 2024
  12. ghost removed
    untriagedNew issue has not been triaged by the area owner
    on Feb 9, 2024
  13. github-actions
    locked and limited conversation to collaborators on Mar 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-GC-mono

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions