Repository navigation
JVMHookResourceReaper fails in Quarkus continuous testing for multi module project #5284
Description
When running Quarkus continuous testing in a multi-module project, something like the following is logged when terminating via ctrl+c:
Exception in thread "Thread-368" com.github.dockerjava.api.exception.ConflictException: Status 409: {"message":"a prune operation is already running"}
at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:245)
at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:124)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.PruneCmdExec.execute(PruneCmdExec.java:33)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.PruneCmdExec.execute(PruneCmdExec.java:13)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
at org.testcontainers.shaded.com.github.dockerjava.core.command.PruneCmdImpl.exec(PruneCmdImpl.java:100)
at org.testcontainers.utility.JVMHookResourceReaper.prune(JVMHookResourceReaper.java:53)
at org.testcontainers.utility.JVMHookResourceReaper.lambda$performCleanup$3(JVMHookResourceReaper.java:28)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.testcontainers.utility.JVMHookResourceReaper.performCleanup(JVMHookResourceReaper.java:28)
at java.base/java.lang.Thread.run(Thread.java:833)
Exception in thread "Thread-76" com.github.dockerjava.api.exception.ConflictException: Status 409: {"message":"a prune operation is already running"}
at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:245)
at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:124)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.PruneCmdExec.execute(PruneCmdExec.java:33)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.PruneCmdExec.execute(PruneCmdExec.java:13)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
at org.testcontainers.shaded.com.github.dockerjava.core.command.PruneCmdImpl.exec(PruneCmdImpl.java:100)
at org.testcontainers.utility.JVMHookResourceReaper.prune(JVMHookResourceReaper.java:53)
at org.testcontainers.utility.JVMHookResourceReaper.lambda$performCleanup$3(JVMHookResourceReaper.java:28)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.testcontainers.utility.JVMHookResourceReaper.performCleanup(JVMHookResourceReaper.java:28)
at java.base/java.lang.Thread.run(Thread.java:833)
Exception in thread "Thread-113" com.github.dockerjava.api.exception.ConflictException: Status 409: {"message":"a prune operation is already running"}
at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:245)
at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:124)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.PruneCmdExec.execute(PruneCmdExec.java:33)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.PruneCmdExec.execute(PruneCmdExec.java:13)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
at org.testcontainers.shaded.com.github.dockerjava.core.command.PruneCmdImpl.exec(PruneCmdImpl.java:100)
at org.testcontainers.utility.JVMHookResourceReaper.prune(JVMHookResourceReaper.java:53)
at org.testcontainers.utility.JVMHookResourceReaper.lambda$performCleanup$3(JVMHookResourceReaper.java:28)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.testcontainers.utility.JVMHookResourceReaper.performCleanup(JVMHookResourceReaper.java:28)
at java.base/java.lang.Thread.run(Thread.java:833)
I'm short on time currently, but I know the internals a bit and I suppose this is due to testcontainers (org.testcontainers.utility.ResourceReaper.DEATH_NOTE respectively) being loaded multiple times by mutliple classloaders.
So each shutdown hook might be thinking it's alone in the current JVM, but it isn't, hence all run concurrently.
This of course only happens if TESTCONTAINERS_RYUK_DISABLED is true, but I want TC to be as fast as possible to provide a reusable(!) MariaDB container and I found the RYUK setup to waste a measurable amount of time.
I don't need/want any kind of cleanup by TC.
I see two options:
- provide a env/system property to opt out of the shutdown hook(s)
- and/or possibly don't register any cleanup in case all containers involved are reusable/reused
Activity
famod commentedon Apr 26, 2022
/cc @stuartwdouglas @geoand FYI, please intervene if my assumption is rubbish.
Alternatively, we might want to consider loading TC parent-first in Quarkus, but I have little to no time to work on Quarkus currently.
geoand commentedon Apr 26, 2022
I thought we already loaded TC parent first, but I'm probably misremembering
famod commentedon Apr 27, 2022
@bsideup Looking at #4938, can it be that I don't "need"
TESTCONTAINERS_RYUK_DISABLEDanymore if all my containers are reusable?kiview commentedon Apr 27, 2022
@famod You are right, that is the intention behind #4938, to provide better startup performance in those cases.
famod commentedon Apr 27, 2022
Ok, so this isssue here will only affect people that have non-reusable containers (or just don't have resuse activated?) and disable ryuk via
TESTCONTAINERS_RYUK_DISABLED.Not having reuse enabled is probably very common, but having disabled ryuk is probably rare (?).
In my case, removing
TESTCONTAINERS_RYUK_DISABLEDis the way to go! 👍bsideup commentedon May 7, 2022
I think loading TC parent-first in Quarkus is a great idea and should solve a few other potential issues 👍