Repository navigation
Is it possible to avoid memory leaks using Firestore on Glassfish/Payara ? #237
Description
Describe your environment
- Operating System version: Ubuntu 16.04 or Docker
- Firebase SDK version: 6.6.0
- JVM : Java(TM) SE Runtime Environment (build 1.8.0_191-b12) / Oracle
- Payara Server: 5.184
- Firebase Product: Firestore
Describe the problem
I store few information in Firestore but I can get/update/delete data.
Here is my problem, after undeploying a web application (war) I get these weird errors :
Severe: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@495224dd]) and a value of type [io.grpc.Context] (value [io.grpc.Context@f94c7c7]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Severe: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@41a52511]) and a value of type [io.grpc.netty.shaded.io.netty.util.internal.InternalThreadLocalMap] (value [io.grpc.netty.shaded.io.netty.util.internal.InternalThreadLocalMap@41c177f2]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Severe: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@495224dd]) and a value of type [io.grpc.Context] (value [io.grpc.Context@f94c7c7]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Severe: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@495224dd]) and a value of type [io.grpc.Context] (value [io.grpc.Context@f94c7c7]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Severe: The web application [] created a ThreadLocal with key of type [com.google.api.client.util.escape.Platform$1] (value [com.google.api.client.util.escape.Platform$1@196dc7db]) and a value of type [char[]] (value [[C@306e81ce]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Severe: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@41a52511]) and a value of type [io.grpc.netty.shaded.io.netty.util.internal.InternalThreadLocalMap] (value [io.grpc.netty.shaded.io.netty.util.internal.InternalThreadLocalMap@588ba77f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Severe: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@495224dd]) and a value of type [io.grpc.Context] (value [io.grpc.Context@f94c7c7]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Severe: The web application [] created a ThreadLocal with key of type [io.grpc.protobuf.lite.ProtoLiteUtils$1] (value [io.grpc.protobuf.lite.ProtoLiteUtils$1@12bf357b]) and a value of type [java.lang.ref.WeakReference] (value [java.lang.ref.WeakReference@651b7be8]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Severe: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@41a52511]) and a value of type [io.grpc.netty.shaded.io.netty.util.internal.InternalThreadLocalMap] (value [io.grpc.netty.shaded.io.netty.util.internal.InternalThreadLocalMap@213ff674]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
I'm just using this code :
import com.google.auth.oauth2.GoogleCredentials;
import com.google.cloud.firestore.Firestore;
import com.google.firebase.FirebaseApp;
import com.google.firebase.FirebaseOptions;
// Use a service account
InputStream serviceAccount = new FileInputStream("path/to/serviceAccount.json");
GoogleCredentials credentials = GoogleCredentials.fromStream(serviceAccount);
FirebaseOptions options = new FirebaseOptions.Builder()
.setCredentials(credentials)
.build();
FirebaseApp.initializeApp(options);
Firestore db = FirestoreClient.getFirestore();
Activity
google-oss-bot commentedon Jan 4, 2019
I found a few problems with this issue:
hiranya911 commentedon Jan 4, 2019
Try calling
close()on theFirestoreinstance. In the future this will happen automatically whenFirebaseApp.delete()is called. See #227whyvrafvr commentedon Jan 4, 2019
Hi @hiranya911.
I've already performed a
close()on theFirestoreinstance. Unfortunately, the errors remain.Thinking about a potential issue with
io.grpc, I've implemented InternalThreadLocalMap.destroy(). The errors remain.I've read that
close()was blocking : Firestore javadoc... but the Threads are not well terminated and seem to be interrupted stopping/undeploying the webapp 😩hiranya911 commentedon Jan 4, 2019
Hi @Skonx. It seems the problem is deep in the bowels of the Firestore client implementation. Can you report this at https://github.com/googleapis/google-cloud-java. They are the ones who can do something about this. We'll keep this issue open as well for our own tracking.
hiranya911 commentedon Jan 4, 2019
In RTDB we encountered a similar issue with Netty and fixed it by doing
FastThreadLocal.removeAll(). It's hacky but that's another thing you can try.whyvrafvr commentedon Jan 7, 2019
Thanks @hiranya911. I will test this asap 👍 and keep you post of course.
BTW, could you please provide best practices regarding firebase/firestore closing? I've not found something in the documentation about closing the db or the firebase app itself.
whyvrafvr commentedon Jan 7, 2019
Unfortunately,
FastThreadLocal.removeAll()does not work 😕whyvrafvr commentedon Jan 7, 2019
Hi @hiranya911, I've opened an issue as recommended.
whyvrafvr commentedon Jan 12, 2019
Hi @hiranya911, no response from Google Cloud Java. Could you please push up the opened issue?
It is a real production issue...
hiranya911 commentedon Jan 14, 2019
@wilhuff can you direct this to somebody at the Firestore SDK side?
@Skonx you can also request for free support if this is critical for you: https://firebase.google.com/support/troubleshooter/
whyvrafvr commentedon Jan 15, 2019
Thanks @hiranya911, google-cloud-java guys replied with a forward from the thread I follow... no real improvement yet 😢
whyvrafvr commentedon Jan 15, 2019
I understand this a beta product but the impact on performances and security are reals. We are talking about memory leaks. If I did something wrong following the official documentation I'd like to know it.
7 remaining items
whyvrafvr commentedon Jan 16, 2019
I don't feel confortable activating/destroying firebase or firestore in the
ServletContextListener.It should be performed in the Servlet lifecycle... The exceptions are probably thrown because I close firestore before the FirebaseApp, will test that.
whyvrafvr commentedon Jan 16, 2019
I've removed
fa.delete()fromcontextDestroyed(ServletContextEvent sce)but the exception remains.whyvrafvr commentedon Jan 16, 2019
I'm pushing this code update in a second branch called "listener".
matejsp commentedon Jan 16, 2019
This is not really a memory leak. Well not when running normally in production.
This is a classloader leak, when application is stopping/redeploying, because running threads are still referencing some classes in threadlocal.
Of course it would be nice to get this fixed, but this is not critical issue as the title suggests. I would fix the issue title to be more appropriate.
whyvrafvr commentedon Jan 16, 2019
Hi @matejsp.
I guess you're right but as you mention they are "running threads" and they should not run after the undeployment and by the way if you perform additionnal firestore queries you will get many times this severe log as much as you perform queries. This is where I suspect a real memory leak in fact. The
io.grpcthreads are not closed and thefirestore.close()does not block waiting the threads are cleaned.matejsp commentedon Jan 16, 2019
Can you do a couple of thread dumps when running additional queries?
Also can you try a 10 second sleep after closing firebase. It seems that
https://github.com/grpc/grpc-java/blob/master/core/src/main/java/io/grpc/internal/SharedResourceHolder.java is doing delayed async shutdown after 1 second: see DESTROY_DELAY_SECONDS constant.
wilhuff commentedon Jan 16, 2019
Apologies for the delay in response. I've been doing a bunch of digging on what's going on here.
There are a few interrelated issues here:
Firestore.close()isn't waiting for shutdown: this is a gax-grpc bug, fixed in Shutdown GRPC managed channel properly on close googleapis/gax-java#628, first released in gax-grpc 1.36, which was after your initial reports.The first fix should address the issue where threads are running past the end of your deployment.
Unfortunately, the other two issues are hard to address. Internal discussion indicates that getting Netty to clean up is possible by shutting down the Netty EventLoopGroup but at this point the plumbing does not exist to be able to control this: it appears that gax-java is just using the default EventLoopGroup. You could conceivably implement an alternative
TransportChannelProviderthat does create its own EventLoopGroup and shuts things down as you like. See InstantiatingGrpcChannelProvider.There's currently no mechanism to clean up gRPC thread locals on shutdown. Making this possible requires the addition of an API by the gRPC team.
At this point, considering the scope of what's required to support this kind of redeployment this isn't something the Firestore team can take on. I wish I had a better answer for this, but we have to prioritize and this kind of environment is rare enough that we can't put this ahead of other work we need to do.
My advice is to either increase permgen size and ignore the warnings you're seeing, or shut everything down through public APIs, wait, and then reflectively find and remove the ThreadLocals that are causing warnings.
whyvrafvr commentedon Jan 16, 2019
Thank you for this clear explanation.
I was afraid of that.
wilhuff commentedon Jan 16, 2019
OK. Closing for now. If others are trying to use a similar configuration and are running into this we may reconsider.
In the meantime, both gax-java and grpc-java are open source and those teams might accept feature requests to expose this kind of full shutdown control.
I did find some code that shows how to remove ThreadLocals without direct support on stackoverflow. I can't vouch for this and would not use an approach that enumerates all the thread locals. I would consider doing this for just the ones that you're specifically getting warnings about in your logs. For example, the gRPC thread local is likely related to this ThreadLocal declaration.
Good luck!
ejona86 commentedon Jan 26, 2019
grpc/grpc-java#5290 should fix the Context ThreadLocal.
You can clean up the current thread's Netty ThreadLocal by calling io.netty.util.concurrent.FastThreadLocal.removeAll() or destroy(). While being cleaner than the reflection-based StackOverflow answer, it would be pretty hard to run on all the threads and so isn't really a solution.
wilhuff commentedon May 23, 2019
Again, this isn't a configuration we can support because not all of our dependencies support it. The Firestore SDK itself does not directly leak via ThreadLocals.
The gRPC PR above, was released as part of grpc-java 1.19, so any version later than will have one fewer ThreadLocal to clean up, but Netty and others will still leak this way so you still need manual clean-up.
If this kind of operation is important to you, then you'll need to add code to reflectively clean these up, as described above.
wilhuff commentedon May 23, 2019
This likely comes down to differences in configuration between the two environments. I can't comment one way or another on how to resolve it though.