Repository navigation
Azure App Service HTTP requests to Azure VNet IP Addresses fail after upgrading to .NET 5.0 #44686
Description
I have an ASP.NET Core 3.1 application running in Azure App Service. It makes calls to Elasticsearch running on an Ubuntu Server 16.04 LTS Azure Virtual Machine, connected to the App Service via an Azure VNet. When I upgraded the application to target net5.0, all of my calls to Elasticsearch started failing with the following exception:
System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions. (10.1.0.5:9200)
---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.
Through trial and error, I eliminated Elasticsearch from the equation and reproduced the issue with simple HTTP requests via HttpClient:
public async Task<IActionResult> VNetRequest()
{
string? body;
try
{
using var requestMsg = new HttpRequestMessage(HttpMethod.Get, _testRequest.VNetEndpoint);
using var responseMsg = await _httpClient.SendAsync(requestMsg);
body = await responseMsg.Content.ReadAsStringAsync();
}
catch (Exception ex)
{
body = ex.ToString();
}
return Content(body);
}When targeting netcoreapp3.1, this call will succeed and display some nginx boilerplate text.
When targeting net5.0, this call will fail with the following exception:
System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions. (10.1.0.4:80)
---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Http.HttpConnectionPool.DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
at AppServiceVNetCalls.Controllers.HomeController.VNetRequest() in D:\home\site\repository\src\AppServiceVNetCalls\Controllers\HomeController.cs:line 34
When I run these tests locally, I don't get any failures; everything works fine.
At this point, I'm stuck, and I don't know how where to go next.
Reproduction repository and Azure applications
I have a simple ASP.NET Core application that demonstrates the issue. This repository has two branches, net31 and net5:
net31: This demonstrates thatHttpClientGET requests to an Azure VNet IP address succeed. It also demonstrates that requests to a non-Azure VNet IP address succeed. See: https://appservicevnettest31.azurewebsites.net/net5: This demonstrates thatHttpClientGET requests to an Azure VNet IP address fail. It also demonstrates that requests to a non-Azure VNet IP address succeed. See: https://appservicevnettest50.azurewebsites.net/
The architecture of my Azure reproduction is as follows:
- An Azure App Service hosting the ASP.NET applications. It's an S1 App Service Plan hosted in US West. They are Windows plans.
- An Azure Virtual Machine running Ubuntu Server 20.04 LTS on a Gen2 Standard B1s VM. I installed nginx solely to respond on port 80.
- The VM is connected to the App Service via an Azure VNet. As I am not a systems engineer, setting this up was laborious, and I'm not sure I could properly document it. However, it did require setting up an Azure Gateway VPN to connect the App Service to the VM.
Thank you.
ETA: Please let me know when you no longer need these VMs/App Services so that I can delete them. Thanks.
Activity
wfurt commentedon Nov 14, 2020
Do you have SELinux enabled?
jonsagara commentedon Nov 14, 2020
I don't believe so. I'm using Windows App Service, not Linux.
geroeinimo commentedon Nov 16, 2020
I have the same issue calling a service over vnet, this blocks the upgrade to net5.0 for us.
kwesseling commentedon Nov 16, 2020
We have the same issue, it appears. Discovered a bizarre thing:
Azure App Service on a Windows service plan, running .net50
kwesseling commentedon Nov 16, 2020
We found a solution: our app service was connected to our vnet through the gateway subnet. Disconnecting it and reconnecting the app to our vnet through a new subnet solved the issue.
For a short while.
Disconnecting (but not reconnecting) solves the issue, apparently vnet integration is the problem with the HttpClient in .Net 5.0.
Could someone please explain why .Net Core 3.1 does work and .Net 5.0 doesn't?
antonfirsov commentedon Nov 16, 2020
@jonsagara and others facing this issue, any chance you can also raise a bug report on the Azure Portal feedback page (if doesn't exist yet):
https://feedback.azure.com/forums/223579-azure-portal/category/129061-bugs
At this moment it seems to us that the issue is more likely on their side, and they may be able to root cause it much faster since the investigation needs Azure infrastructural knowledge much more than anything else.
After opening it, can you share the link of the report so we can use it when contacting the Azure team?
jonsagara commentedon Nov 16, 2020
Will do.
jonsagara commentedon Nov 16, 2020
@antonfirsov Here you go: https://feedback.azure.com/forums/223579-azure-portal/suggestions/41965039-azure-app-service-http-requests-to-azure-vnet-ip-a
46 remaining items
EvanMachusakNCQA commentedon Jan 6, 2021
@karelz Yes, sorry, I meant .NET Core 3.1.
The specific situation we encountered is that we are using a single App Service Plan in our dev environment, so we have a .NET Core 3.1 App Service for LOB application "A" and a NodeJS Azure Functions app for LOB application "B" in the same App Service Plan.
When a developer for LOB app "A" tried to upgrade his .NET Core 3.1 app to .NET 5, he encountered the issue described in the ticket. But this also inadvertently broke application "B", because once .NET 5 is activated a single time on an App Service Plan, from that point forward, any service that uses .NET's new networking stack in any capacity and does not give you an explicit .NET tooling choice (e.g., NodeJS Azure Functions) will be broken.
It could be considered a separate issue, but I wanted to bring this information here to underscore the criticality of this problem. It doesn't just break your .NET Core app. It breaks your entire App Service Plan.
JohnYoungers commentedon Jan 6, 2021
It looks like Microsoft has a fix per this comment: https://github.com/MicrosoftDocs/azure-docs/issues/45991#issuecomment-755582909
Adding the setting
WEBSITE_VNET_SUPPORT_DUAL_STACK_SOCKETS=1resolved this problem for our applicationjonsagara commentedon Jan 6, 2021
Confirmed on my test site. I added the setting to the App Service configuration, restarted the site, and the following call no longer throws:
https://appservicevnettest50.azurewebsites.net/Home/VNetRequest
ericsampson commentedon Jan 6, 2021
Now I'm curious, is this a temporary workaround that's planned to be set to true permanently without users having to do anything?
SFinistrosa commentedon Jan 7, 2021
Hi, we are facing the same issue using HC on our web app. Adding the flag did not solve the issue. Any idea?
Thanks
karelz commentedon Jan 7, 2021
@SFinistrosa I would ask Azure App Service why it does not work for you.
@ericsampson also no idea, I would wait for Azure App Service official resolution ...
jraadt commentedon Jan 8, 2021
We continue to have this issue with our App Services. Incredibly frustrating. The flag does not solve the issue. For testing we created a simple ASP.NET 5 web API in App Service and a NodeJS server running on a VM that simply returns "Hello World!". The App Service and VM are in our virtual network.
The ASP.NET web API has two endpoints. One uses HttpClient to call the NodeJS server. The other uses HttpClient as well but forced to be IPv4.
When I deploy the ASP.NET web API to a new App Service (on an App Service Plan V3) the simple HttpClient fails with
An attempt was made to access a socket in a way forbidden by its access permissions.and the HttpClient with forced IPv4 returns "Hello World!" successfully. This behavior is the same regardless if I have setWEBSITE_VNET_SUPPORT_DUAL_STACK_SOCKETS=1.HOWEVER, I just tried to "downgrade" my test App Service Plan to V2 and BOTH HttpClient calls worked, again regardless of the flag. I flipped back and forth between an V2 and V3 app service plans and it always works for V2 and never for V3. Any ideas why?
jraadt commentedon Jan 9, 2021
@karelz - Appreciate your help as we navigate this issue. How would we ask Azure App Service for help? Is that through the portal support features or do they have a GitHub issues type thing? I would like to voice my support that this is important for us to have fixed. And is there any way of tracking Azure App Service resolution on this? Thanks again.
karelz commentedon Jan 11, 2021
@jraadt they don't have GitHub AFAIK. Support portal sounds reasonable (I don't know how that works though).
antonfirsov commentedon Jan 28, 2021
We've got a confirmation from the Azure team regarding the fix:
According to our information, in their next release the switch will be enabled by default, and it will also contain a fix for Hybrid Connections, but we can not provide any ETA or promise on their behalf.
I'm closing this now, since the original issue is fixed, and I believe related problems are better tracked in separate issues.
Thanks again for the detailed reports, and all the amazing contribution we've seen here from the community!
ericsampson commentedon Jan 28, 2021
Thank you very much for closing the circle here @antonfirsov , and to everyone involved in addressing the issue. Thanks to @karelz too for managing this ticket.
Cheers
jonsagara commentedon Jan 28, 2021
I removed the
WEBSITE_VNET_SUPPORT_DUAL_STACK_SOCKETSconfiguration setting, restarted the app, and confirmed that VNet requests still work.I'm going to delete both of the test app services now.
Thank you @antonfirsov and everyone else who helped!
Danthar commentedon Feb 4, 2021
I'd like to note, that we started deploying a netcore5 application to azure, January this month. And the auto detect logic from the Azure team does not work for us.
So to anyone who finds this issue. You might have to toggle the magic setting (WEBSITE_VNET_SUPPORT_DUAL_STACK_SOCKETS) yourself. Or utilize the workaround httpclient mentioned by @antonfirsov earlier in this thread.
Be aware though that if you do use the SocketHttpClient you will run into this problem: #31261 (TLDR, no AI Dependency logging on your http calls)