Skip to content

Out of memory error crash in Axis Python OpenGL #2510

Closed

Description

@mbuesch

When I start up the sim/axis_mm.ini example on my Debian Bookworm or Debian Sid machines with the LinuxCNC 2.9 or LinuxCNC master branch, I get a strange out of memory crash in Python OpenGL code.

$:~/develop/git/linuxcnc/scripts$ ./linuxcnc
LINUXCNC - 2.9.0~pre1
Machine configuration directory is '/home/mb/develop/git/linuxcnc/configs/sim/axis'
Machine configuration file is 'axis_mm.ini'
Starting LinuxCNC...
linuxcnc TPMOD=tpmod HOMEMOD=homemod EMCMOT=motmod
Note: Using POSIX realtime
Found file(lib): /home/mb/develop/git/linuxcnc/lib/hallib/core_sim.hal
Found file(lib): /home/mb/develop/git/linuxcnc/lib/hallib/sim_spindle_encoder.hal
Found file(lib): /home/mb/develop/git/linuxcnc/lib/hallib/axis_manualtoolchange.hal
Found file(lib): /home/mb/develop/git/linuxcnc/lib/hallib/simulated_home.hal
note: MAXV     max: 53.340 units/sec 3200.400 units/min
note: LJOG     max: 53.340 units/sec 3200.400 units/min
note: LJOG default: 30.480 units/sec 1828.800 units/min
note: jog_order='XYZ'
note: jog_invert=set()
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/OpenGL/latebind.py", line 43, in __call__
    return self._finalCall( *args, **named )
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mb/develop/git/linuxcnc/bin/axis", line 3964, in <module>
    get_coordinate_font(vars.dro_large_font.get())
  File "/home/mb/develop/git/linuxcnc/bin/axis", line 3842, in get_coordinate_font
    glnav.use_pango_font(coordinate_font, 0, 128)
  File "/home/mb/develop/git/linuxcnc/lib/python/glnav.py", line 65, in use_pango_font
    glBitmap(1, 0, 0, 0, 0, h-d, bytearray([0]*4))
  File "/usr/lib/python3/dist-packages/OpenGL/latebind.py", line 47, in __call__
    return self._finalCall( *args, **named )
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/OpenGL/wrapper.py", line 700, in wrapperCall
    raise err
  File "/usr/lib/python3/dist-packages/OpenGL/wrapper.py", line 693, in wrapperCall
    result = wrappedOperation( *cArguments )
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/OpenGL/platform/baseplatform.py", line 415, in __call__
    return self( *args, **named )
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/OpenGL/error.py", line 230, in glCheckError
    raise self._errorClass(
OpenGL.error.GLError: GLError(
	err = 1285,
	description = b'out of memory',
	baseOperation = glBitmap,
	pyArgs = (
		1,
		0,
		0,
		0,
		0,
		16.0,
		<OpenGL.arrays._buffers.Py_buffer object at 0x7fc4a2a1b5c0>,
	),
	cArgs = (
		1,
		0,
		0,
		0,
		0,
		16.0,
		<OpenGL.arrays._buffers.Py_buffer object at 0x7fc4a2a1b5c0>,
	),
	cArguments = (
		1,
		0,
		0,
		0,
		0,
		16.0,
		<OpenGL.arrays._buffers.Py_buffer object at 0x7fc4a2a1b5c0>,
	)
)
Shutting down and cleaning up LinuxCNC...

Here are the steps I follow to reproduce the issue:

  1. Happens during start of axis sim/axis_mm.ini example.

Information about my hardware and software:

This happens on my older PC with 8G system RAM with a very old PCI-E graphics card from 2009. Not sure how much graphics RAM it has. But it probably only has a couple hundred MB.
But it also happens on my new Ryzen based system with 24G system RAM. I think I set the graphics RAM to 1G.

I doubt that this really is an out of memory problem unless LinuxCNC needs that much graphics memory now.

I even tried to raise memlock limit to 4G on the older PC. Didn't help.

On the old PC I use Debian Bookworm and on the Ryzen I use Debian Sid.

Activity

  1. Sigma1912 commented on May 29, 2023

    @Sigma1912
    Contributor

    looks like the same issue as this:
    #1599

  2. mbuesch commented on May 29, 2023

    @mbuesch
    ContributorAuthor

    Well, I'm not so sure it is. The error text in #1599 talks about a boost error and not about out of memory. Whereas there's no mention of boost in my error message.
    To me this looks unrelated.

  3. Sigma1912 commented on May 29, 2023

    @Sigma1912
    Contributor

    Ah yes, have a look at this one on the forum, seems to fit your error:
    https://forum.linuxcnc.org/38-general-linuxcnc-questions/47910-opengl-out-of-memory-error#261447

    [edit]
    If you go back to the above linked issue you will see that the 'out of memory' error appears later in the discussion. Looking at this update from Seb this issue should be fixed since February though:

    I think this issue talks about a number of problems, but the OpenGL "glBitmap out of memory" problem should be fixed by @jepler's https://github.com/LinuxCNC/linuxcnc/pull/2328. That fix is in 2.9 and master/2.10 now.

  4. mbuesch commented on May 29, 2023

    @mbuesch
    ContributorAuthor

    Yes, I know about these discussions, but this still seems to be a new problem.

    I think this issue talks about a number of problems, but the OpenGL "glBitmap out of memory" problem should be fixed by @jepler's #2328. That fix is in 2.9 and master/2.10 now.

    It doesn't fix it for me. I use latest 2.9 and master (as of the day I opened this Issue).

    What I haven't tried is commenting out random pieces of code, like in all the suggestions.
    But that doesn't really look like a solution to me.
    I think the first step should be to understand what's really going on here.
    From all those posts it seems like everybody is poking in the dark. That's why I opened this Issue.

    If you have any suggestion for me to try, I'm completely open for it. It is 100% reproducible on two completely different machines. Therefore, I can try things out to track this down to the real root cause. But because I don't know anything about OpenGL, I'm a bit lost and need some help here.

  5. rene-dev commented on May 29, 2023

    @rene-dev
    Member

    this is mesa bug: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8904
    more info here: #2446
    replace the 2 calls with glBitmap(1, 1, 0, 0, w, -h+d, bytearray([0]*4)) as a workaround.

  6. mbuesch commented on May 29, 2023

    @mbuesch
    ContributorAuthor

    Thanks for the suggestion. Changing the second parameter from 0 to 1 fixed the problem for me. No more crashes (at least in axis demo mode. Now I need to run my actual setup and see...)

    However, the change from #2446 changes the first parameter to 0 and the last one to None. I also tried that. That does not fix the problem. Therefore merging that PR would not fix it for me.

  7. rene-dev commented on May 29, 2023

    @rene-dev
    Member

    exactly. the PR does it correctly, but due to the bug in mesa that doesnt work.
    afaik 1, 1 breaks it on some other systems, thats why it was changes to 1, 0.
    it only moves the cursor, and should not need memory or a size at all.

  8. mbuesch commented on Jun 2, 2023

    @mbuesch
    ContributorAuthor

    What about catching the exception and trying again with the alternative parameter set? That way we could support buggy and fixed versions.

  9. petterreinholdtsen commented on Jun 17, 2023

    @petterreinholdtsen
    Collaborator

    You mean something like this?

    diff --git a/lib/python/glnav.py b/lib/python/glnav.py
    index 77c6fbf7fc..c60cd5c001 100644
    --- a/lib/python/glnav.py
    +++ b/lib/python/glnav.py
    @@ -2,6 +2,7 @@ import math
     import array, itertools
     import sys
     
    +from OpenGL.error import GLError
     from OpenGL.GL import *
     from OpenGL.GLU import *
     
    @@ -62,8 +63,10 @@ def use_pango_font(font, start, count, will_call_prepost=False):
             context.restore()
             w, h = int(w / Pango.SCALE), int(h / Pango.SCALE)
             glNewList(base+i, GL_COMPILE)
    -        glBitmap(1, 0, 0, 0, 0, h-d, bytearray([0]*4))
    -        #glDrawPixels(0, 0, 0, 0, 0, h-d, '');
    +        try:
    +            glBitmap(0, 0, 0, 0, 0, h-d, bytearray([0]*4))
    +        except OpenGL.error.GLError:
    +            glBitmap(1, 0, 0, 0, 0, h-d, bytearray([0]*4))
             if not will_call_prepost:
                 pango_font_pre()
             if w and h: 
    @@ -72,8 +75,11 @@ def use_pango_font(font, start, count, will_call_prepost=False):
                     glDrawPixels(w, h, GL_LUMINANCE, GL_UNSIGNED_BYTE, a.tobytes())
                 except Exception as e:
                     print("glnav Exception ",e)
    -
    -        glBitmap(1, 0, 0, 0, w, -h+d, bytearray([0]*4))
    +        try:
    +            glBitmap(0, 0, 0, 0, w, -h+d, bytearray([0]*4))
    +        except OpenGL.error.GLError:
    +            glBitmap(1, 0, 0, 0, 0, h-d, bytearray([0]*4))
    +        if not will_call_prepost:
             if not will_call_prepost:
                 pango_font_post()
             glEndList()
    diff --git a/scripts/linuxcnc.in b/scripts/linuxcnc.in
    index 75abe58b78..051c9f357e 100644
    --- a/scripts/linuxcnc.in
    +++ b/scripts/linuxcnc.in
    @@ -22,15 +22,6 @@ if test "xyes" = "x@RUN_IN_PLACE@"; then
         fi
     fi
     
    -# Force pyopengl (python3-opengl) to use its GLX backend, even
    -# if we're running on Wayland where it would normally use EGL.
    -# Similar for the Gtk and Qt backends.
    -if [[ ! -v LINUXCNC_OPENGL_PLATFORM || "${LINUXCNC_OPENGL_PLATFORM}" == "glx" ]]; then
    -    export PYOPENGL_PLATFORM="x11"
    -    export GDK_BACKEND="x11"
    -    export QT_QPA_PLATFORM="xcb"
    -fi
    -
     ################################################################################
     # 0. Values that come from configure
     ################################################################################
    

    Note, the patch is untested and is only intended as a starting point and to document the locations adjusted to fix the OpenGL issue.

  10. mbuesch commented on Jun 17, 2023

    @mbuesch
    ContributorAuthor

    Yes, something like that. Having the correct code for the non-buggy version in the try branch and then having one or more alternatives in the exception branch for buggy versions.

    In my case (Debian Bookworm) I need these variants:

    glBitmap(1, 1, 0, 0, 0, h-d, bytearray([0]*4))
    
    glBitmap(1, 1, 0, 0, w, -h+d, bytearray([0]*4))
    
  11. coredump commented on Sep 12, 2023

    @coredump

    Had the same problem but with qtdragon_hd instead of axis. Using the patch from #2510 (comment) with mbuesch values for Bookworm fixed it, or at least fixed it enough that the interface opens again.

  12. andypugh commented on Sep 13, 2023

    @andypugh
    Collaborator

    Is anyone looking further at Petter's suggestion?

  13. added a commit that references this issue on Sep 14, 2023
  14. added a commit that references this issue on Sep 22, 2023
  15. jethornton commented on Sep 24, 2023

    @jethornton
    Collaborator

    Just tested a fresh pull of master and no open gl errors on Debian 12

  16. mbuesch commented on Sep 27, 2023

    @mbuesch
    ContributorAuthor

    Yes, I can confirm that it works without patching now on Debian Bookworm.
    Thanks for your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions