An simple way to mock an httpClient.GetAsync(..) method for unit tests?

@davidfowl

System.Net.Http has now been uploaded to the repo 😄 🎉 🎈

Whenever I've used this in some service, it works great but makes it hard to unit test => my unit tests don't want to actually ever hit that real end point.

Ages ago, I asked @davidfowl what should we do? I hoping I paraphrase and don't misquote him - but he suggested that I need to fake up a message handler (ie. HttpClientHandler), wire that up, etc.

As such, I ended up making a helper library called HttpClient.Helpers to help me run my unit tests.

So this works ... but it feels very messy and .. complicated. I'm sure I'm not the first person that needs to make sure my unit tests don't do a real call to an external service.

Is there an easier way? Like .. can't we just have an IHttpClient interface and we can inject that into our service?

HttpClient is nothing more than an abstraction layer over another HTTP library. One of its primary purposes is allowing you to replace the behavior (implementation), including but not limited to the ability to create deterministic unit tests.

In other works, HttpClient itself serves as the both the "real" and the "mock" object, and the HttpMessageHandler is what you select to meet the needs of your code.

But if feels like sooo much work is required to setup the message handler when (it feels like) this could be handled with a nice interface? Am I totally misunderstanding the solution?

@PureKrome

@PureKrome - thanks for bringing this up for discussion. Can you please elaborate on what you mean by "so much work is required to setup the message handler"?

One way to unit test HttpClient without hitting the network is -

Create a new Handler class (e.g. FooHandler) that derives from HttpMessageHandler
Implement the SendAsync method as per your requirements - don't hit the network/log the request-response/etc.
Pass an instance of this FooHandler into the constructor of the HttpClient:
var handler = new FooHandler();
var client = new HttpClient(handler);

Your HttpClient object will then use your Handler instead of the inbuilt HttpClientHandler.

Thanks,
Sid

@SidharthNabar

Hi @SidharthNabar - Thank you for reading my issue, I really appreciate it also.

Create a new Handler class

You just answered my question :)

That's a large dance to wiggle too, just to ask my real code to not hit the network.

I even made a the HttpClient.Helpers repo and nuget package .. just to make testing easier! Scenario's like the Happy path or an exception thrown by the network endpoint ...

That's the problem -> can we not do all of this and ... just a mock a method instead?

I'll try and explain via code..

Goal: Download something from the interwebs.

public async Foo GetSomethingFromTheInternetAsync()
{
    ....
    using (var httpClient = new HttpClient())
    {
        html = await httpClient.GetStringAsync("http://www.google.com.au");
    }
    ....
}

Lets look at two examples:

Given an interface (if it existed):

public interface IHttpClient
{
    Task<string> GetStringAsync(string requestUri);    
}

My code can now look like this...

public class SomeService(IHttpClient httpClient = null)
{
    public async Foo GetSomethingFromTheInternetAsync()
    {
        ....
        using (var httpClient = _httpClient ?? new HttpClient()) // <-- CHANGE dotnet/corefx#1
        {
            html = await httpClient.GetStringAsync("http://www.google.com.au");
        }
        ....
    } 
}

and the test class

public async Task GivenAValidEndPoint_GetSomethingFromTheInternetAsync_ReturnsSomeData()
{
    // Create the Mock : FakeItEasy > Moq.
    var httpClient = A.Fake<IHttpClient>();

    // Define what the mock returns.
    A.CallTo(()=>httpClient.GetStringAsync("http://www.google.com.au")).Returns("some html here");

    // Inject the mock.
    var service = new SomeService(httpClient);
    ...
}

Yay! done.

Ok, now with the current way...

create a new Handler class - yes, a class!
Inject the handler into the service

public class SomeService(IHttpClient httpClient = null)
{
    public async Foo GetSomethingFromTheInternetAsync()
    {
        ....
        using (var httpClient = _handler == null 
                                    ? new HttpClient()
                                    : new HttpClient(handler))
        {
            html = await httpClient.GetStringAsync("http://www.google.com.au");
        }
        ....
    } 
}

But the pain is now that I have to make a class. You've now hurt me.

public class FakeHttpMessageHandler : HttpClientHandler
{
 ...
}

And this class starts out pretty simple. Until I when I have multiple GetAsync calls (so i need to provide multiple Handler instances?) -or- multiple httpClients in a single service. Also, do we Dispose of the handler or reuse it if we're doing multiple calls in the same block of logic (which is a ctor option)?

eg.

public async Foo GetSomethingFromTheInternetAsync()
{
    string[] results;
    using (var httpClient = new HttpClient())
    {
        var task1 = httpClient.GetStringAsync("http://www.google.com.au");
        var task2 = httpClient.GetStringAsync("http://www.microsoft.com.au");

        results = Task.WhenAll(task1, task2).Result;
    }
    ....
}

this can be made so much easier with this..

var httpClient = A.Fake<IHttpClient>();
A.CallTo(() = >httpClient.GetStringAsync("http://www.google.com.au"))
    .Returns("gooz was here");
A.CallTo(() = >httpClient.GetStringAsync("http://www.microsoft.com.au"))
    .Returns("ms was here");

clean clean clean :)

Then - there is the next bit : Discoverability

When I first started using MS.Net.Http.HttpClient the api was pretty obvious 👍 Ok - get string and do it asyncly.. simple ...

... but then I hit testing .... and now I'm suppose to learn about HttpClientHandlers? Um why? I feel that this should all be hidden under the covers and I shouldn't have to worry about all this implementation detail. It's too much! You're saying that I should start to look inside the box and learn some of the plumbing ... which hurts 😢

It's all making this more complex than it should be, IMO.

Help me Microsoft - you're my only hope.

I too would love to see an easy and simple way to test various things that use HttpClient. 👍

@ericstj

Thanks for the detailed response and code snippets - that really helps.

First, I notice that you are creating new instances of HttpClient for sending each request - that is not the intended design pattern for HttpClient. Creating one HttpClient instance and reusing it for all your requests helps optimize connection pooling and memory management. Please consider reusing a single instance of HttpClient. Once you do this, you can insert the fake handler into just one instance of HttpClient and you're done.

Second - you are right. The API design of HttpClient does not lend itself to a pure interface-based mechanism for testing. We are considering adding a static method/factory pattern in the future that will allow you to alter the behavior of all HttpClient instances created from that factory or after that method. We haven't yet settled on a design for this yet, though. But the key issue will remain - you will need to define a fake Handler and insert it below the HttpClient object.

@ericstj - thoughts on this?

Thanks,
Sid.

@SidharthNabar

@SidharthNabar why are you/team so hesitant to offering an interface for this class? I was hoping that the whole point of a developer having to learn about handlers and then having to create fake handler classes is enough of a pain point to justify (or at the very least, highlight) the need for an interface?

Yeah, I don't see why an Interface would be a bad thing. It would make testing HttpClient so much easier.

@KrzysztofCwalina

One of the main reasons we avoid interfaces in the framework is that they don't version well. Folks can always create their own if they have a need and that won't get broken when the next version of the framework needs to add a new member. @KrzysztofCwalina or @terrajobst might be able to share more of the history/detail of this design guideline. This particular issue is a near religious debate: I'm too pragmatic to take part in that.

HttpClient has lots of options for unit testability. It's not sealed and most of its members are virtual. It has a base class that can be used to simplify the abstraction as well as a carefully designed extension point in HttpMessageHandler as @sharwell points out. IMO it's a quite well designed API, thanks to @HenrikFrystykNielsen.

👍 an interface

@ericstj

👋 @ericstj Thanks heaps for jumping in 👍 🍰

One of the main reasons we avoid interfaces in the framework is that they don't version well.

Yep - great point.

This particular issue is a near religious debate

yeah ... point well taken on that.

HttpClient has lots of options for unit testability

Oh? I'm struggling on this 😊 hence the reason for this issue 😊

It's not sealed and most of its members are virtual.

Members are virtual? Oh drats, I totally missed that! If they are virtual, then mocking frameworks can mock those members out 👍 and we don't need an interface!

Lets have a looksies at HttpClient.cs ...

public Task<HttpResponseMessage> GetAsync(string requestUri) { .. }
public Task<HttpResponseMessage> GetAsync(Uri requestUri) { .. }

hmm. those aren't virtual ... lets search the file ... er .. no virtual keyword found. So do you mean other members to other related classes are virtual? If so .. then we're back at the issue I'm raising - we have to now look under the hood to see what GetAsync is doing so we then know what to create/wireup/etc...

I guess I'm just not understanding something really basic, here ¯(°_°)/¯ ?

EDIT: maybe those methods can be virtual? I can PR!

SendAsync is virtual and almost every other API layers on top of that. 'Most' was incorrect, my memory serves me wrong there. My impression was that most were effectively virtual since they build on top of a virtual member. Usually we don't make things virtual if they are cascading overloads. There is a more specific overload of SendAsync that is not virtual, that one could be fixed.

Ah! Gotcha 😊 So all those methods end up calling SendAsync .. which does all the heavy lifting. So this still means we have a discoverability issue here .. but lets put that aside (that's opinionated).

How would we mock SendAsync give this basic sample...
Install-Package Microsoft.Net.Http
Install-Package xUnit

public class SomeService
{
    public async Task<string> GetSomeData(string url)
    {
        string result;
        using (var httpClient = new HttpClient())
        {
            result = await httpClient.GetStringAsync(url);
        }

        return result;
    }
}

public class Facts
{
    [Fact]
    public async Task GivenAValidUrl_GetSomeData_ReturnsSomeData()
    {
        // Arrange.
        var service = new SomeService();

        // Act.
        var result = await service.GetSomeData("http://www.google.com");

        // Assert.
        Assert.True(result.Length > 0);
    }
}

One of the main reasons we avoid interfaces in the framework is that they don't version well. Folks can always create their own if they have a need and that won't get broken when the next version of the framework needs to add a new member.

I can totally understand this way of thinking with the full .NET Framework.

However, .NET Core is split into many small packages, each versioned independently. Use semantic versioning, bump the major version when there's a breaking change, and you're done. The only people affected will be the ones explicitly updating their packages to the new major version - knowing there are documented breaking changes.

I'm not advocating that you should break every interface every day: breaking changes should ideally be batched together into a new major version.

I find it sad to cripple APIs for future compatibility reasons. Wasn't one of the goal of .NET Core to iterate faster since you don't have to worry anymore about a subtle .NET update breaking thousands of already installed applications?

My two cents.

@MrJul

@MrJul 👍
Versioning shouldn't be a problem. That's why the version numbers exist :)

@dasjestyr

Heh @dasjestyr - I too think you might have missed a major point of my discussion.

The fact that we (the developer) needs to learn so much about Message Handlers, etc .. to fake the response is my main point about all of this. Not so much about interfaces. Sure, I (personally) prefer interfaces to virtuals r wrappers with respect to testing (and therefore mocking) ... but those are implementation details.

I'm hoping the main gist of this epic thread is to highlight that .. when using HttpClient in an application, it's a PITA to test with it.

The status quo of "go learn the plumbing of HttpClient, which will lead you to HttpMessageHandler, etc" is a poor situation. We don't need to do this for many other libraries, etc.

So I was hoping something can be done to alleviate this PITA.

Yes, the PITA is an opinion - I'll totally admit to that. Some people don't think it's a PITA at all, etc.

The real problem is the way that people look at HttpClient. It's not some random class that should be newed up whenever you think to call the internet.

Agreed! But until fairly recently, this wasn't well known - which might lead to some lack of documentation or teaching or something.

Plus, your dependent classes shouldn't care about the HttpClient, only the data that gets returned by it

Yep - agreed.

which comes from the handler.

a what? huh? Oh -- now u're asking me to open up the hood and learn about the plumbing? ... Refer to above again and again.

Think of it this way: are you ever going to replace the HttpClient implementation with something else?

No.

.. etc ..

Now, I'm not meaning to troll, etc. So please don't think that I'm trying to be a jerk by always replying and repeating the same stuff over and over again.

I've been using my helper library for ages now which is just a glorified way of using a custom message handler. Great! It works and works great. I just think that this could be all exposed a bit nicer ... and that's what I'm hoping this thread really hits home at.

EDIT: formatting.

(I've only just noticed the grammar error in this issue's title and now I can't unsee it)

And that has been my real long game :)

Q.E.D.

(actually - so embarrassing 😊 )

EDIT: part of me doesn't want to edit it .. for nostalgia....

(I've only just noticed the grammar error in this issue's title and now I can't unsee it)

I know! Same!

The fact that we (the developer) needs to learn so much about Message Handlers, etc .. to fake the response is my main point about all of this. Not so much about interfaces. Sure, I (personally) prefer interfaces to virtuals r wrappers with respect to testing (and therefore mocking) ... but those are implementation details.

wut?

Have you even looked at HttpMessageHandler? It's an abstract class that is literally a single method that takes an HttpRequestMessage and returns a HttpResponseMessage -- made to intercept behavior separate of all the low level transport junk which is exactly what you'd want in a unit test. To fake it, just implement it. The message that goes in is the one that you sent HttpClient, and the response that comes out is up to you. For example, if I want to know that my code is dealing with a json body correctly, then just have the response return a json body that you expect. If you want to see if it's handling 404 correctly, then have it return a 404. It doesn't get more basic than that. To use the handler, just send it in as a ctor argument for HttpClient. You don't need to pull any wires out or learn the internal workings of anything.

I'm hoping the main gist of this epic thread is to highlight that .. when using HttpClient in an application, it's a PITA to test with it.

And the main gist of what many people have pointed out is that you're doing it wrong (which is why it's a PITA, and I say this directly but respectfully) and demanding that HttpClient be interfaced for testing purposes is akin to creating features to compensate for bugs instead of addressing the root problem which is, in this case, that you're operating at the wrong level.

I think that HttpClient in the windows namespace actually did separate the concept of handler into filter, but it does the exact same thing. I think that handler/filter in that implementation is actually interfaced though which is kinda what I was suggesting earlier.

wut?
Have you even looked at HttpMessageHandler?

Initially no, because of this:

var payloadData = await _httpClient.GetStringAsync("https://......");

meaning, the exposed methods on HttpClient are really nice and make things really easy :) Yay! Fast, simple, works, WIN!

Ok - so now lets use it in a test ... and now we need to spend time learning what happens underneath .. which leads us to learning about HttpMessageHandler etc.

Again I keep saying this => this extra learning about the plumbing of HttpClient (i.e. HMH, etc) is a PITA. Once you have done that learning .. yes, I then know how to use it etc ... even though I also don't like to keep using it but need to, in order to mock remote calls to 3rd party endpoints. Sure, that's opinionated. We can only agree to disagree. So please - I know about HMH and how to use it.

And the main gist of what many people have pointed out is that you're doing it wrong

Yep - people disagree with me. No probs. Also - people agree me too. Again, no probs.

and demanding that HttpClient be interfaced for testing purposes is akin to creating features to compensate for bugs instead of addressing the root problem which is, in this case, that you're operating at the wrong level.

Ok - I respectfully disagree with you here (which is ok). Again, different opinions.

But srsly. This thread has gone on for so long. I've really moved on by now. I said my piece, some people say YES! some said NO! Nothing has changed and I've just ... accepted the status quo and rolled with everything.

I'm sorry you just don't accept my train of thought. I'm not a bad person and try not sound rude or mean. This was just me expressing how I felt while developing and how I believe others also felt. That's all.

I'm not at all offended. I originally hopped on this thread with the same opinion that the client should be mockable, but then some really good points were made about what HttpClient is so I sat and really thought about it, and then tried to challenge it on my own and eventually I came to the same conclusion which is that HttpClient is the wrong thing to mock and trying to do so is futile exercise that causes far more work than it's worth. Accepting that made life much easier. So I too have moved on. I'm just hoping that others will eventually give themselves a break and take it for what it is.

As an aside:

Initially no, because of this:

var payloadData = await _httpClient.GetStringAsync("https://......");

meaning, the exposed methods on HttpClient are really nice and make things really easy :) Yay! Fast, >simple, works, WIN!

I'd argue that in terms of SOLID, this interface is inappropriate anyway. IMO client, request, response are 3 different responsibilities. I can appreciate the convenience methods on the client, but it promotes tight coupling by combining requests and responses with the client, but that's personal preference. I have some extensions to HttpResponseMessage that accomplish the same thing but keep the responsibility of reading the response with the response message. In my experience, with large projects, "Simple" is never "Simple" and almost always ends in a BBOM. This however, is a completely different discussion :)

Now that we have a new repo specifically for design discussions, please continue the discussion in https://github.com/dotnet/designs/issues/9 or open a new issue in https://github.com/dotnet/designs

Thanks.

Maybe it could be considered below solution for testing purpose only:

public class GeoLocationServiceForTest : GeoLocationService, IGeoLocationService
{
public GeoLocationServiceForTest(something: something, HttpClient httpClient) : base(something)
{
base._httpClient = httpClient;
}
}

@richardszalay

I ended up using @richardszalay's MockHttp.
Thanks, Richard, you saved my day!

Ok, I can live with a no-interfaced HttpClient. But being forced to implement a class inheriting HttpMessageHandler because SendAsync is protected just sucks.

I use NSubstitute, and this doesn't work:

var httpMessageHandler = 
    Substitute.For<HttpMessageHandler>(); // cannot be mocked, not virtual nor interfaced
httpMessageHandler.SendAsync(message, cancellationToken)
    .Return(whatever); // doesn't compile, it's protected
var httpClient = new HttpClient(httpMessageHandler)

Really disappointing.

Navigation Menu

Repository navigation

An simple way to mock an httpClient.GetAsync(..) method for unit tests? #14535

Goal: Download something from the interwebs.

133 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

An simple way to mock an httpClient.GetAsync(..) method for unit tests? #14535

Description

Activity

sharwell commented on May 4, 2015

PureKrome commented on May 4, 2015

SidharthNabar commented on May 5, 2015

PureKrome commented on May 5, 2015

Goal: Download something from the interwebs.

aarondandy commented on May 6, 2015

SidharthNabar commented on May 8, 2015

PureKrome commented on May 8, 2015

luisrudge commented on May 8, 2015

ericstj commented on May 8, 2015

TerribleDev commented on May 8, 2015

PureKrome commented on May 8, 2015

ericstj commented on May 8, 2015

PureKrome commented on May 8, 2015

MrJul commented on May 8, 2015

luisrudge commented on May 8, 2015

133 remaining items

PureKrome commented on Apr 11, 2017

richardszalay commented on Apr 11, 2017

PureKrome commented on Apr 11, 2017

jnm2 commented on Apr 11, 2017

dasjestyr commented on Apr 12, 2017

PureKrome commented on Apr 12, 2017

dasjestyr commented on Apr 13, 2017

dasjestyr commented on Apr 13, 2017

davidsh commented on Jul 10, 2017

AchotStepanian commented on Jul 11, 2017

arialdomartini commented on Aug 23, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions