Skip to content

Duplicate response body when running on Lambda and manipulating response body stream in middleware #1185

Closed

Description

@madmox

Describe the bug

I have a ASP.NET middleware which temporarily replaces the HTTP response body stream with a memory stream so that it can perform work involving seek operations on the response stream before the response is sent to the client. The code works properly when running locally using Kestrel or the integration tests server, but when running on AWS, all response bodies are duplicated (i.e. {"foo":"bar"} becomes {"foo":"bar"}{"foo":"bar"}).

Expected Behavior

The response should be a valid JSON result on all hosting environments (e.g. {"foo":"bar"}).

Current Behavior

The response is invalid when running on AWS Lambda only (duplicated JSON, e.g. {"foo":"bar"}{"foo":"bar"}).

Reproduction Steps

Simplified middleware code looks like this:

public async Task InvokeAsync(HttpContext context)
{
    Stream originalResponseBodyStream = context.Response.Body;
    using var buffer = new MemoryStream();
    context.Response.Body = buffer;

    try
    {
        await this._next(context);
        // Do work involving seek on body stream
    }
    finally
    {
        context.Response.Body.Seek(0, SeekOrigin.Begin);
        await context.Response.Body.CopyToAsync(originalResponseBodyStream);
        context.Response.Body = originalResponseBodyStream;
    }
}

Possible Solution

No response

Additional Information/Context

My ASP.NET application is using the Minimal API hosting pattern. The AWS Lambda hosting mode is set to LambdaEventSource.RestApi.

AWS .NET SDK and/or Package version used

  • Amazon.Lambda.AspNetCoreServer 7.1.0
  • Amazon.Lambda.AspNetCoreServer.Hosting 1.1.0

Targeted .NET Platform

.NET 6

Operating System and version

AmazonLinux

Activity

  1. added
    bugThis issue is a bug.
    needs-triageThis issue or PR still needs to be triaged.
    on May 20, 2022
  2. madmox commented on May 20, 2022

    @madmox
    Author

    Sample repo to reproduce the issue.

  3. ashishdhingra commented on May 20, 2022

    @ashishdhingra
    Contributor

    Hi @madmox,

    Good afternoon.

    Thanks for sharing the sample code. Unfortunately, I'm unable to reproduce the issue. For reproduction,

    • Cloned your code.
    • Used your deployment PowerShell script to deploy the CloudFormation template.
     .\deploy.ps1 -StackName teststack-issue1185 -DeploymentBucketName testbucket-issue1880                    ❮  
    D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\IndexController.cs(13,42): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread. [D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\AWSLambdaTest.csproj]
    
    Build succeeded.
    
    D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\IndexController.cs(13,42): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread. [D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\AWSLambdaTest.csproj]
        1 Warning(s)
        0 Error(s)
    
    Time Elapsed 00:00:06.91
    Amazon Lambda Tools for .NET Core applications (5.3.0)
    Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet
    
    Executing publish command
    ... invoking 'dotnet publish', working folder 'D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\bin\Release\net6.0\publish'
    ... dotnet publish "D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest" --output "D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\bin\Release\net6.0\publish" --configuration "Release" --framework "net6.0" /p:GenerateRuntimeConfigurationFiles=true --runtime linux-x64 --self-contained false
    ... publish: Microsoft (R) Build Engine version 17.1.1+a02f73656 for .NET
    ... publish: Copyright (C) Microsoft Corporation. All rights reserved.
    ... publish:   Determining projects to restore...
    ... publish:   Restored D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\AWSLambdaTest.csproj (in 493 ms).
    ... publish: D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\IndexController.cs(13,42): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread. [D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\AWSLambdaTest.csproj]
    ... publish:   AWSLambdaTest -> D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\bin\Release\net6.0\linux-x64\AWSLambdaTest.dll
    ... publish:   AWSLambdaTest -> D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\bin\Release\net6.0\publish\
    Zipping publish folder D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\bin\Release\net6.0\publish to D:\Lambda_Issue1185\awslambdadotnetissue1185\artifacts\AWSLambdaTest.zip
    Creating directory D:\Lambda_Issue1185\awslambdadotnetissue1185\artifacts
    ... zipping: Amazon.Lambda.APIGatewayEvents.dll
    ... zipping: Amazon.Lambda.ApplicationLoadBalancerEvents.dll
    ... zipping: Amazon.Lambda.AspNetCoreServer.dll
    ... zipping: Amazon.Lambda.AspNetCoreServer.Hosting.dll
    ... zipping: Amazon.Lambda.Core.dll
    ... zipping: Amazon.Lambda.Logging.AspNetCore.dll
    ... zipping: Amazon.Lambda.RuntimeSupport.dll
    ... zipping: Amazon.Lambda.Serialization.SystemTextJson.dll
    ... zipping: appsettings.json
    ... zipping: AWSLambdaTest
    ... zipping: AWSLambdaTest.deps.json
    ... zipping: AWSLambdaTest.dll
    ... zipping: AWSLambdaTest.pdb
    ... zipping: AWSLambdaTest.runtimeconfig.json
    Created publish archive (D:\Lambda_Issue1185\awslambdadotnetissue1185\artifacts\AWSLambdaTest.zip).
    Lambda project successfully packaged: D:\Lambda_Issue1185\awslambdadotnetissue1185\artifacts\AWSLambdaTest.zip
    Amazon Lambda Tools for .NET Core applications (5.3.0)
    Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet
    
    Processing CloudFormation resource ServerlessApi
    Processing CloudFormation resource AspNetCoreFunction
    Initiate packaging of . for resource AspNetCoreFunction
    Uploading to S3. (Bucket: testbucket-issue1880 Key: AWSLambdaTest-637886578304268713.zip)
    ... Progress: 45%
    ... Progress: 91%
    ... Progress: 100%
    Uploading to S3. (Bucket: testbucket-issue1880 Key: teststack-issue1185-cloudformation-637886578309076380.yaml)
    ... Progress: 100%
    Found existing stack: False
    CloudFormation change set created
    ... Waiting for change set to be reviewed
    Created CloudFormation stack teststack-issue1185
    
    Timestamp            Logical Resource Id                      Status
    -------------------- ---------------------------------------- ----------------------------------------
    5/20/2022 3:37 PM    teststack-issue1185                      CREATE_IN_PROGRESS
    5/20/2022 3:37 PM    LambdaExecutionIamRole                   CREATE_IN_PROGRESS
    5/20/2022 3:37 PM    LambdaExecutionIamRole                   CREATE_IN_PROGRESS
    5/20/2022 3:37 PM    LambdaExecutionIamRole                   CREATE_COMPLETE
    5/20/2022 3:37 PM    AspNetCoreFunction                       CREATE_IN_PROGRESS
    5/20/2022 3:37 PM    AspNetCoreFunction                       CREATE_IN_PROGRESS
    5/20/2022 3:37 PM    AspNetCoreFunction                       CREATE_COMPLETE
    5/20/2022 3:37 PM    ServerlessApi                            CREATE_IN_PROGRESS
    5/20/2022 3:37 PM    ServerlessApi                            CREATE_IN_PROGRESS
    5/20/2022 3:37 PM    ServerlessApi                            CREATE_COMPLETE
    5/20/2022 3:37 PM    AspNetCoreFunctionRootResourcePermissionProd CREATE_IN_PROGRESS
    5/20/2022 3:37 PM    AspNetCoreFunctionProxyResourcePermissionProd CREATE_IN_PROGRESS
    5/20/2022 3:37 PM    ServerlessApiDeploymentcfb7a37fc3        CREATE_IN_PROGRESS
    5/20/2022 3:37 PM    AspNetCoreFunctionProxyResourcePermissionProd CREATE_IN_PROGRESS
    5/20/2022 3:37 PM    AspNetCoreFunctionRootResourcePermissionProd CREATE_IN_PROGRESS
    5/20/2022 3:37 PM    ServerlessApiDeploymentcfb7a37fc3        CREATE_IN_PROGRESS
    5/20/2022 3:37 PM    ServerlessApiDeploymentcfb7a37fc3        CREATE_COMPLETE
    5/20/2022 3:37 PM    ServerlessApiProdStage                   CREATE_IN_PROGRESS
    5/20/2022 3:37 PM    ServerlessApiProdStage                   CREATE_IN_PROGRESS
    5/20/2022 3:37 PM    ServerlessApiProdStage                   CREATE_COMPLETE
    5/20/2022 3:38 PM    AspNetCoreFunctionProxyResourcePermissionProd CREATE_COMPLETE
    5/20/2022 3:38 PM    AspNetCoreFunctionRootResourcePermissionProd CREATE_COMPLETE
    5/20/2022 3:38 PM    teststack-issue1185                      CREATE_COMPLETE
    Stack finished updating with status: CREATE_COMPLETE
    
    • Test the changes (this works fine)

    Screen Shot 2022-05-20 at 3 43 39 PM

    Executing dotnet --version gives output 6.0.202.

    Please let me know if I'm missing any step.

    Thanks,
    Ashish

  4. added
    response-requestedWaiting on additional info and feedback. Will move to close soon in 7 days.
    and removed on May 20, 2022
  5. madmox commented on May 20, 2022

    @madmox
    Author

    Oh sorry, I made a last-minute change to my reproduction sample and it seems to be related to the buggy behavior! I updated the repo, just deployed the sample, and could reproduce the issue again.

    It seems the bug has to do with the following lines:

    context.Response.Body.Seek(0, SeekOrigin.Begin);
    await context.Response.Body.CopyToAsync(originalResponseBodyStream);
    context.Response.Body = originalResponseBodyStream;

    If I replace this section with the following, it works as expected:

    buffer.Seek(0, SeekOrigin.Begin);
    await buffer.CopyToAsync(originalResponseBodyStream);
    context.Response.Body = originalResponseBodyStream;

    Although buffer and context.Response.Body are supposed to be the same object! (context.Response.Body = buffer)

  6. removed
    response-requestedWaiting on additional info and feedback. Will move to close soon in 7 days.
    on May 21, 2022
  7. added
    investigatingThis issue is being investigated and/or work is in progress to resolve the issue.
    on Jun 3, 2022
  8. added and removed
    investigatingThis issue is being investigated and/or work is in progress to resolve the issue.
    on Jun 24, 2022
  9. added
    p2This is a standard priority issue
    and removed on Nov 1, 2022
  10. added
    p2This is a standard priority issue
    and removed
    p2This is a standard priority issue
    on Mar 29, 2023
  11. dscpinheiro commented on Oct 20, 2026

    @dscpinheiro
    Contributor

    Fixed in Amazon.Lambda.AspNetCoreServer 10.1.1 (release_2026-05-18). Root cause was the same as #1702: InvokeFeatures.Set<TFeature> did not bump the feature collection revision, so ASP.NET Core kept writing to the original body stream after middleware replaced Response.Body.

    Please reopen if it still reproduces on 10.1.1 or later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions