Repository navigation
Guideline for background threads handling #239
Description
We are hosting a REST API inside of a lambda as described here. Within an API call, we usually trigger code that executes in the background in order not to slow down the API call. For example, all analytics events are sent to a queue that is flushed once in a while as described here.
As explained in this blog, all the background threads are paused when the API call returns:
But let’s say you have a background process running when the function finishes – what happens to it if the container is reused? In this case, Lambda will actually “freeze” the process and thaw it out the next time you call the function (but only if the container is reused, which isn’t a guarantee). So in the reuse case, your background processes will still be there, but they won’t have been executing while you were away.
It talks about background processes but our experiments shows that the same is true for background threads.
So my question is the following: is there an event that we could hook to in order to wait for the completion of these background threads? For example, in the case of our Analytics queue mentioned above we would call the Flush() method to be sure that the queue is empty.
Ideally, this hook would be called after the response has been sent to API Gateway (thus in the way of being sent to the client) but before the lambda container freezes. In other words, is there a way to be sure that some code gets executed after an API call completes but without adding latency to the API call.
Activity
normj commentedon Mar 6, 2018
There is no way to return data back to the caller of the Lambda function, in this API Gateway, and continue running code in your Lambda function. Can you have your Lambda function that is handling the request coming from API Gateway invoke another Lambda function to do the analytics work and then return back to API Gateway?
mabead commentedon Mar 7, 2018
@normj We could fire another lambda function but that would involve a lot of changes in a code base that is already working perfectly in non-serverless (i.e. docker) mode. It would be "unpleasant" if we have to do lots of change to port this API to a lambda hosting.
To give more context, here are three things that I can remember of that require background processing:
1- send logs to loggly
We use loggly to manage our logs. So whenever we use the ASP.Net logging infrastructure (ex: myLogger.LogInformation("hello")), it simply creates a json object and puts it in a queue. The content of this queue is sent to the loggly REST API whenever the queue is full or it has been idle for too long.
Obviously, it wouldn't make sense to modify all the logging code to invoke a lambda function. But in this case, a good solution would be to stop using our custom in-process forwarding to loggly and simply write our logs to CloudWatch. We could then hook a lambda on CloudWatch to send these logs to loggly. So, there's no a real problem here. Just an improvement of the way we work ;)
2- Kinesis events
Sending events to Kinesis is quite slow. I just looked again at our metrics and 95% of our calls to send events to Kinesis take 110ms or less. So let's say that each route of our API does a call to Kinesis. We don't want to slow down all these routes by 100ms just for that. For this reason, we put the events in a queue and we flush this queue once in a while (in a similar way as what we do for loggly).
We could do this by triggering a lambda but it would require changes in lots of places in the code base. It would also require us to fire a lambda which would also add latency... it's probably negligible.
3- Analytics events
As already mentionned we use Segment to send analytics events. Once again, we could fire a lambda to do this or we could configure to Segment to operate in non-buffered mode but once again, it would either require changes in lots of places or add more latency to our API (because it would include the latency to push data to the Segment servers).
Conclusion
So in the end, we are looking for a solution that allows:
One option that I was thinking of was too add an ASP.NET core middleware similar to the following: Ex:
What do you think of this option? Is it a dead end because this function is called anyway before the data is returned to API Gateway?
normj commentedon Mar 9, 2018
Using the OnComplete is not going to give you want you want. It is called when the final response is created and about to be returned to API Gateway. I know it is not the true meaning of when this is supposed to be called but it is the best we can do in a Lambda context.
I would be careful about maintaining states in queues in a Serverless environment because you can never predict when the compute environment will be replaced. Only keep things in state that you don't care if they disappear or are easy to recreate.
normj commentedon Mar 22, 2018
Closing for lack of activity