This repository was archived by the owner on May 6, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on May 6, 2026. It is now read-only.
[GAE python 2.7] query().filter().fetch() sometimes spawns threads preventing webapp2 requests finish within a deadline #336
Description
Certain cloud-ndb queries causes grpc to spawn threads that prevent accomplishing webapp2/django requests as GAE standard env for python 2 joins all request threads.
Requests failes with the following exception
Thread running after request. Creation traceback:
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_lib/versions/1/google/appengine/runtime/runtime.py", line 80, in Run
target(*args, **kw)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_dist/lib/python2.7/threading.py", line 794, in __bootstrap
self.__bootstrap_inner()
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_dist/lib/python2.7/threading.py", line 821, in __bootstrap_inner
self.run()
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_dist/lib/python2.7/threading.py", line 774, in run
self.__target(*self.__args, **self.__kwargs)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_lib/versions/third_party/grpcio-1.0.0/grpc/_channel.py", line 678, in channel_spin
event = state.completion_queues[key].poll()
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_lib/versions/third_party/grpcio-1.0.0/grpc/_plugin_wrapping.py", line 107, in __call__
wrapped_context, AuthMetadataPluginCallback(wrapped_cygrpc_callback))
File "/base/data/home/apps/.../lib/google/auth/transport/grpc.py", line 87, in __call__
future = self._pool.submit(self._get_authorization_headers, context)
File "/base/data/home/apps/.../lib/concurrent/futures/thread.py", line 137, in submit
self._adjust_thread_count()
File "/base/data/home/apps/.../lib/concurrent/futures/thread.py", line 159, in _adjust_thread_count
t.start()
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_dist/lib/python2.7/threading.py", line 749, in start
_start_new_thread(self.__bootstrap, ())
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_lib/versions/1/google/appengine/runtime/runtime.py", line 83, in StartNewThread
return base_start_new_thread(Run, ())
The following shows one example that after re-writing it in a get-by-id manner has no such issue.
Model1.query(Model1.repeatedKeyProp.IN(keys)).fetch()
Shutting down ThreadPoolExecutor from concurrent.future helped to localize the issue caused by additional threads caused by querying data.
Any ideas how to coupe with it? We cannot just move everything to Python 3 env in a one step, both envs are going to co-exist for sometime.
Activity
[-][GAE python 2.7] query().filter().fetch() sometimes spawns threads preventing webapp2 requests to hang over the deadline[/-][+][GAE python 2.7] query().filter().fetch() sometimes spawns threads preventing webapp2 requests finish within a deadline[/+]andrewsg commentedon Feb 14, 2020
Thanks for this report. To clarify, in this case your query was fired and completed, and results delivered, entirely within the request; then, at the end of the request, there was an error? Or is the problem that the query was not completed at all and errored out immediately, even though the request had not completed?
VladNF commentedon Feb 15, 2020
Yes, a query is completed and results are delivered. The things is AppEngine runtime joins all threads spawned within a request, somehow those additional threads after query with IN,AND etc are not finished. I tried to figure out what are these threads and seems like they are from grpc concurrent.future.thread.ThreadPoolExecutor. In our code we do not start any background thread, just using cloud-ndb to handle data (worked fine with AppEngine built-in ndb module before migration to cloud-ndb).
oliverchang commentedon Feb 20, 2020
We also ran into this (google/clusterfuzz#1298 (comment)).
The workaround for us is to manually invoke the GC at the end of the request, which seems to clean up stray threads.
VladNF commentedon Feb 20, 2020
@oliverchang thank you so much, seems to work for us
chrisrossi commentedon Feb 20, 2020
Reopening this so we can investigate whether it makes sense to do something on the NDB side.
@oliverchang mentions this happens with
MultiQuerybut not other calls? Is it worth exploring why?Would it make sense to add a call to
gc.collectsomewhere in NDB?fix: call `gc.collect()` when exiting the NDB runtime context
chrisrossi commentedon Apr 9, 2020
This is fixed in
google.authwith googleapis/google-auth-library-python#467The
gc.collect()work-around should no longer be required.