Repository navigation
[discussion] Java Async #175
Description
Activity
thanks for bringing this up. A lot of the thinking comes from the Microsoft Graph SDK work (pre-kiota) and was kind of recaped here and there.
Long story short; completable future seemed to be the best compromise at the time, only raising API level to 24 (we rose to 26 because of the DateTime and other things we needed). It's not ideal, but its the one with the broader compatibility and broader support.
API level 26 is now supported by 93% of devices (compared with 91% in late 2020), and 24 96.4%. Over time the gains of investing into a broader compatibility solution will naturally reduce.
As per investing into a better developer experience solution, we should demonstrate a significant improvement before doing so. I wouldn't be surprised if Fibers come up with helper functions to wrap completable future to ease up the migration path.Lastly, we're currently wrapping the call using enqueue which OkHttp will defer to another thread or not depending on whether other requests are being executed, and depending on the dispatcher. That gives full control to the caller (they can implement/swap their dispatcher) rather than implementing our own custom logic.
Thanks a lot for the context Vincent Biret (@baywet) , I think that your answer perfectly recaps the reasoning behind the current decisions/implementation 👍
do you think further discussion is needed or should we close this issue at this time?
Perfectly fine closing this, thanks for taking the time to share the background.
We briefly touched on this point during the community meeting today.
Reopening to gather additional feedback from Emond Papegaaij (@papegaaij) and Xavier Bouclet (@mikrethor) 🙏
For additional context we have one precedent of a language not being async: go. There is no primitive for that in go because usually defer the routines at the highest level.
I'm guessing that if we took away the async API surface, people could always do it similarly to go (in this case wrapping the call in a future or other construct themselves).
Do we have any visibility over a potential async/await set of keywords equivalent in Java being worked on? or the reason why this might have been declined? (hopefully we're not talking about compatibility again)Do we have any visibility over a potential async/await set of keywords equivalent in Java being worked on? or the reason why this might have been declined? (hopefully we're not talking about compatibility again)
In Java land, there is an exciting turnaround with Project Loom and part of it is going to land in Java 21 (going to be released GA in September).
To play well with Loom execution model we should only have blocking calls 🙂It would be interesting to do a quick experiment to see how much we can generalize some utility methods in an external "extension" to wrap the most relevant calls with async.
I don't think you can actually wrap a synchronous call in an async utility. For true async, you need to be async all the way down. I don't know how good the async implementation for OkHttp actually is, but in theory a good http client using Http 2 could multiplex multiple http requests in parallel over a limited set of connections using only a few threads. There is no way to get this if the SDK performs the calls sync.
That being said, very few people actually use async in Java and using it "the right way" is very hard. Many SDKs in Java provide both options. Many Java developers really prefer a sync API. A suggestion would be something like
getandgetAsync. This is very common in many Java SDKs and should be possible in the current request builders.The current Microsoft Graph SDK offers both API surfaces for the last couple of years. However it's built in a different way (using our older generator) and we don't have telemetry over which API surface people are using. Our current plan for the Microsoft Graph Java SDK is to release a preview based of kiota and see what kind of feedback we get.
As per okHttp, we're calling enqueue which in turns calls enqueue on the dispatcher.
As far as I understand kotlin the dispatcher then uses a combination of thread pools and queues to defer the call to a different thread.
This is not "truly" async but between that and the fact we wrap the callback to a Future, it's close enough.These pools are one of the major issues with using async APIs. We've had several major production outages to some of our other applications due to pools getting exhausted and the whole thing deadlocking.
We use the current Graph SDK in Topicus KeyHub and we use the sync API as do other applications in our company AFAIK.
Maybe someone like Bruno Borges (@brunoborges) (on Microsoft side) and Sanne Grinovero (@Sanne) (on RH) are interested in weighting in on this decision before we go GA 🙏
Hi all - I'm over on the Azure SDK side of the Microsoft house. We've been building client libraries since 2018, and have a fairly large amount of experience in this area. We did a lot of user testing and investigations back when we started, and we ended up using Project Reactor for our async story, but we ship sync and async clients side-by-side (in the same library). We have full sync and full async stacks. This allows developers to choose to use sync or async, as their needs dictate. Whilst I have zero evidence of this, my gut feeling is that most developers choose to use our sync stack implementation, for its simplicity and debuggability. I imagine only our largest customers bother with the complexities of async, to eek out the final few percentage points worth of performance.
Obviously when we started the concept of virtual threads was still a long way off, and even then, it feels too low-level of an abstraction to give to developers, as it would be asking them to manage threads (and, realistically, block on them). Similarly, CompletableFuture is good in concept for many use cases, but in some places (in particular, streaming cases), it falls down. This is why we ended up going with Reactor for our async story - it gave a complete story for developers to understand, for all async cases.
Looking ahead to the future, I've done some early thinking about how I would do things today, and it would probably be a custom async API that wasn't based on a third party dependency. This is because Reactor does break more often than we would like, both in terms of implementation and public-facing API. If I had more time back in the 2018 time frame, I would probably have dived more into this, but as it stands I've only got high level thoughts (and a lot of investigations I would like to do).
I should also add, it sounds like you are working on a number of problems that we have already worked through in our azure-core library. I do wonder if there is any benefit to you making use of our libraries, or finding better ways to collaborate, rather than duplicate a lot of effort?
15 remaining items
When having both blocking (sync) and asynchronous capabilities in the same stack, we try really hard to avoid the two needing to interact: every time an asynchronous operation needs to invoke a blocking one it's a headache of integration and design issues, and every time a blocking method needs to invoke an asynchronous one it needs to, at very least, switch the physical carrier thread which comes at a significant cost of performance.
I wouldn't recommend that.
The one reason in other frameworks we exposed "natively asynch" APIs is precisly for these reasons; if you think you'd better be off with a very thin adaptation layer, I'd recommend rather expose only one of them and let the users deal with it explicitly - it would be a better experience as they would be aware and responsible for how exactly the translation layer works.
Vincent Biret (@baywet) I'm suggesting that you can flip the http abstraction layer in sync-mode, so it will do the http call in the caller thread and simply return
CompletableFuture.completedFuture(value). This will make theCompletableFuturenothing more than a wrapper for the actual value, without all the async behavior attached.The main problem I'm having with an async API in an application that's designed to use sync behavior is the need for a separate thread pool to handle the async calls. If too small, you will get significant congestion on these calls (even up to the point of deadlocks), if too big, they will waste resources (even up to the point of OutOfMemoryError). We've seen both in our applications.
Sanne Grinovero (@Sanne) yep I was aware of the kind of issues sync over async creates. And async over sync doesn't provide much value when compared to only sync.
Emond Papegaaij (@papegaaij) I don't think returning a completed future over a sync call would provide much value. And we've already ruled out providing both APIs due to a size constraint/duplicating the pipelines.
The decision is sync XOR async API. And if we maintain async (with completable futures or something else) we should make sure it plays nicely with the HTTP client and its thread pool scheduler (if it uses any).
Another aspect we discussed when looking at this issue if we switch to a sync API surface, we'd probably have to declare the throws of the custom exceptions generated from the description.
If the actual async behavior is being handled in the OK layer (using a thread pool maintained by OK), then does that mean users/community can solve the problem Emond Papegaaij (@papegaaij) has by contributing their own HTTP client layer? I haven't dug into the library the way that Andrea Peruffo (@andreaTP) has, but I know that was discussed. We were interested because we wanted to e.g. provide a Quarkus specific HTTP client implementation.
But if we can do that, then someone could presumably implement the approach Emond Papegaaij (@papegaaij) suggested, by providing an HTTP client that actually does the call synchronously and then just wraps the result in an already-completed future.
Personally, as a consumer of the generated SDKs, I would prefer a synchronous API because that's just easier to use and debug. And for the use-cases I care about, client scalability isn't really much of an issue. But I understand why async is being used now - it's probably only possible to practically adapt async->sync, not the other way around: as mentioned, adapting a sync API to async doesn't really accomplish the point of async for scalability - it needs to be actually non-blocking/async all the way down.
We briefly touched on the subject another couple of times, I think that my mind is settled now, on 2 opposite directions 🙂 :
- from the "theoretical" point of view(threads control etc.) the async implementation is by far preferable
- from the pragmatic point of view, using async code in a sync codebase(which is the majority according to any data I've ever collected), is extremely unpractical and doesn't play well with most of the constructs (exceptions are always wrapped, controlling execution in lambdas is challenging, the "Functional interface" APIs are mostly unusable) and it's likely to force the introduction of breaking changes in public APIs
And I think it is safe to assume that with the introduction of virtual threads in Java 21, async will become more of niche. Virtual threads bring most of the benefits of async without the downsides of unreadable and impossible to debug code.
Go has adopted virtual threads from the start and uses blocking sync APIs everywhere (at least that's what I see in the code I work with). Threads (or goroutines) are cheap and you can spawn as many as you like. There's no problem in having lots of them blocking in a sync call. I wouldn't be surprised if Java is heading the same way when more application servers and frameworks start supporting virtual threads.
Reacted by Eric Wittmann and Maísa RissiHello everyone! Thanks for all the inputs and discussion here! Things like this is what help us to make the best product and better decisions.
After some considerations like the introduction of virtual thread in Java 21 and conversations with Jonathan Giles (@JonathanGiles) Bruno Borges (@brunoborges) and internally on Graph DevX team, we believe that sync only is the best way to go. With that, we can improve debugging capabilities as well as decrease complexity on development.
We are building on top of Jonathan Giles (@JonathanGiles) work that have been working with Java community and got feedback like this https://twitter.com/JonathanGiles/status/1711583034007564484 where most developers believe sync only is also the way to go. This also gives people the possibility to wrap calls in their own preferred async way and gives us the possibility to provide our own wrappers in the future, if we want to.Reacted by Bruno Borges, Vincent Biret, Sébastien Levert, Andrea Peruffo and Clayton WalkerThank you Maísa Rissi (@maisarissi) for going the extra mile here and getting both quantitative data, and experts input in addition to the amazing feedback we got here.
Ramses Sanchez-Hernandez (@ramsessanchez) please go ahead and clean up completable futures from the API surface here, also remove any async suffix we might have + make the relevant generation change (kiota/snippets/raptor).
Reacted by Sébastien LevertReacted by Sébastien Levert
Metadata
Metadata
Type
Projects
- Status
I think it's worth having this conversation even just to keep track of the reasoning behind the decisions.
When I looked at the codebase, at first, I was pretty impressed by the fact that we are using async calls everywhere for Java.
This is a subject pretty much full of nuances in Java land as far as I know it.
Let's try to summarize a few key points:
CompletableFutureAPI is not great and doesn't play well with most of the rest of the standard library, also, using it makes the code much much harder to understand and follow (e.g. we miss anasync/awaitconstruct)CompletableFutureinterface is going to play nicely with all the frameworks adopting this paradigm (Akka, Vert-X etc. etc.)CompletableFutures might not be the ideal solution given the number of options that comes (scheduling on which Thread Pool? Keep separated IO blocking operations from in-process computing etc. etc.)