Repository navigation
Threads created and not destroyed with each client #239
Description
Each time you search on a client, new threads are created that are not destroyed. e.g. if you need to use many clients and run
def test():
auth_string = """****"""
client = google_ads.client.GoogleAdsClient.load_from_string(auth_string)
service = client.get_service("GoogleAdsService")
res = service.search(
"****", "select campaign.resource_name from campaign limit 1"
)
res = list(res)
for i in range(10000):
test()Then run watch -n1 "ps -eLf | wc -l" in another terminal, you can see the number of threads creeping up. After a while you'll reach your machines thread limit then have lots of system wide problems.
If you comment out the last line turning the results generator into a list, the problem disappears, so I think the thread creation is occurring there?
As far as I can tell there isn't a .close() function on the client or service I'm missing?
Reacted by John Walker
Activity
wihl commentedon Mar 7, 2020
What version of the library are you using? We just released v5 yesterday.
dgmp88 commentedon Mar 8, 2020
It was on the previous version, but I just tested on 5 and still see the thread count creeping up. I'm running on Ubuntu 18.04.
BenRKarl commentedon Mar 9, 2020
@dgmp88 If you put a
print(len(threading.enumerate()))call after the call totestyou can see the thread count increase by one after each call. I'd expect the threads to get closed after the generator is finished with the request. I'm going to file an issue with thegRPCteam to see if they can help identify the issue.BenRKarl commentedon Mar 9, 2020
@dgmp88 can you clarify which version of the
grpciomodule you have installed when you see this? I don't see the issue when I use the newest1.27.2dgmp88 commentedon Mar 9, 2020
Thanks for the quick replies!
threading - good point, that's far simpler!
I just checked, I'm also using 1.27.2 and still seeing this.
dgmp88 commentedon Mar 9, 2020
I also just updated everything i could think of (had an older version of
grpc-cpp), but still seeing the issueBenRKarl commentedon Mar 9, 2020
@dgmp88 which version of Python are you using? And are you using a virtualenv?
BenRKarl commentedon Mar 9, 2020
@dgmp88 Can you confirm whether you see this issue if you downgrade google-auth to version 1.7.1? I can consistently repro with 1.7.2 and later, but not with 1.7.1.
dgmp88 commentedon Mar 9, 2020
I'm using python 3.7 with anaconda (pip for installing the google ads api) and was on google-auth 1.11.2 .
Yep, it's google-auth! Good find. Threads don't increase on 1.7.1, and do on 1.7.2 for me as well.
Thanks so much for the speedy diagnosis! I'll downgrade for the time being.
BenRKarl commentedon Mar 9, 2020
@dgmp88 I opened this issue with the Auth team, you can track progress there. Until there's a change on their end the best way forward is to pin
google-authto1.7.1BenRKarl commentedon Mar 30, 2020
Version 1.12.0 of
google-authwas released which fixes this issue, when installinggoogle-ads-pythonit will get downloaded automatically so I'm going to close this issue. If anyone has this problem again make sure to update your dependencies and verify you have a>= 1.12.0version ofgoogle-authinstalled.