Skip to content

[C++] Compatibility with newer aws sdk #45304

Closed
Closed
[C++] Compatibility with newer aws sdk#45304
Assignees
pitrou
Labels
Component: C++Component: PackagingComponent: PythonType: enhancement
Milestone
19.0.1

Description

@h-vetinari
h-vetinari
opened on Jan 20, 2025

Describe the enhancement requested

Not sure whether to classify this as a bug or as an enhancement. It feels like a regression, but likely in AWS itself. In conda-forge, we don't build all patch versions of the SDK, because it gets released way too often to migrate ever time. Recently we went from 1.11.458 to 1.11.488, which ran into consistent errors on unix. I didn't look too closely at first (looked just like a flaky failure), but it reproduces 100% of the time. As such, I'm reverting the bump across our maintenance branches again.

Failures look roughly like this

[----------] 28 tests from TestS3FS
[ RUN      ] TestS3FS.GetFileInfoRoot
$SRC_DIR/cpp/src/arrow/filesystem/s3fs_test.cc:519: Failure
Failed
'PopulateTestBucket()' failed with IOError: AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding

[  FAILED  ] TestS3FS.GetFileInfoRoot (43 ms)
[ RUN      ] TestS3FS.GetFileInfoBucket
$SRC_DIR/cpp/src/arrow/filesystem/s3fs_test.cc:519: Failure
Failed
'PopulateTestBucket()' failed with IOError: AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding

[  FAILED  ] TestS3FS.GetFileInfoBucket (52 ms)

all due to malformed chunked encoding

This also shows up in the python tests (these are from arrow v15, which is the last version where we built the python bindings on the same feedstock; since the errors are uniform across versions though, I strongly suspect this to be the case also for main, as well as v19 and down).

=========================== short test summary info ============================
FAILED pyarrow/tests/test_fs.py::test_filesystem_is_functional_after_pickling[builtin_pickle-S3FileSystem] - OSError: When creating key 'a/' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_filesystem_is_functional_after_pickling[cloudpickle-S3FileSystem] - OSError: When creating key 'a/' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_dataset.py::test_write_dataset_s3 - OSError: When creating key 'dataset/' in bucket 'mybucket': AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_create_dir[S3FileSystem] - OSError: When creating key 'test-directory/' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_copy_file[S3FileSystem] - OSError: When uploading part for key 'test-copy-source-file' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during UploadPart operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_move_file[S3FileSystem] - OSError: When uploading part for key 'test-move-source-file' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during UploadPart operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_delete_file[S3FileSystem] - OSError: When uploading part for key 'test-delete-target-file' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during UploadPart operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_filesystem_from_uri_s3 - OSError: When creating key 'foo/' in bucket 'mybucket': AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding

Xref also #45195

Component(s)

C++, Python, Packaging

Activity

  1. h-vetinari
    added
    Type: enhancement
    on Jan 20, 2025
  2. github-actions
    added
    Component: C++
    Component: Python
    Component: Packaging
    on Jan 20, 2025

  3. pitrou commented on Jan 20, 2025

    @pitrou
    pitrou
    on Jan 20, 2025
    Member

    The error message comes from Minio. So this is probably an incompatibility between Minio and the official AWS SDK... :(

  4. pitrou commented on Jan 20, 2025

    @pitrou
    pitrou
    on Jan 20, 2025
    Member

    Ok, I can reproduce, and even with the latest Minio version is still occurs. So I'll probably have to run Wireshark on this...

  5. pitrou commented on Jan 20, 2025

    @pitrou
    pitrou
    on Jan 20, 2025
    Member

    I have it down to this HTTP exchange:

    • Request by AWS SDK:
    PUT /bucket/emptydir/ HTTP/1.1
Host: 127.0.0.1:39491
Accept: */*
amz-sdk-invocation-id: B545DDA1-9E33-4987-B074-2B147AB8BD75
amz-sdk-request: attempt=1
authorization: AWS4-HMAC-SHA256 Credential=minio/20250120/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-type;host;transfer-encoding;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-trailer, Signature=5be6a95be231511f69fdc161843bbe83413ba52540e1607bc9de36ea42a827c5
content-encoding: aws-chunked
content-type: binary/octet-stream
transfer-encoding: chunked
user-agent: aws-sdk-cpp/1.11.488 ua/2.1 api/S3 os/Linux#5.15.0-130-generic lang/c++#C++11 md/aws-crt#0.29.9-dev+d5a3907e md/arch#x86_64 md/GCC#13.3.0
x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
x-amz-date: 20250120T094325Z
x-amz-decoded-content-length: 0
x-amz-trailer: x-amz-checksum-crc64nvme
Expect: 100-continue
    • Response by Minio:
    HTTP/1.1 400 Bad Request
Accept-Ranges: bytes
Content-Length: 331
Content-Type: application/xml
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: a4f5403139a65a8fd51b2aa0caa208291435f782459a282182de2ddf9eec98bf
X-Amz-Request-Id: 181C5D5BF688A7B0
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 11618
X-Ratelimit-Remaining: 11618
X-Xss-Protection: 1; mode=block
Date: Mon, 20 Jan 2025 09:43:25 GMT
Connection: close

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>BadRequest</Code><Message>malformed chunked encoding</Message><Key>emptydir/</Key><BucketName>bucket</BucketName><Resource>/bucket/emptydir/</Resource><RequestId>181C5D5BF688A7B0</RequestId><HostId>a4f5403139a65a8fd51b2aa0caa208291435f782459a282182de2ddf9eec98bf</HostId></Error>

  6. pitrou commented on Jan 20, 2025

    @pitrou
    pitrou
    on Jan 20, 2025
    Member

    Interesting, another PUT request before (to create a bucket) is successful.

    • Request:
    PUT /empty-bucket HTTP/1.1
Host: 127.0.0.1:39491
Accept: */*
amz-sdk-invocation-id: 54812C34-9549-4ECB-B54E-15B98759BCB9
amz-sdk-request: attempt=1
authorization: AWS4-HMAC-SHA256 Credential=minio/20250120/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date, Signature=65a56d02027d24a6b29815814d051b012501bdb545bd32dc051f2f4c7b01ee7a
content-length: 0
content-type: application/xml
user-agent: aws-sdk-cpp/1.11.488 ua/2.1 api/S3 os/Linux#5.15.0-130-generic lang/c++#C++11 md/aws-crt#0.29.9-dev+d5a3907e md/arch#x86_64 md/GCC#13.3.0
x-amz-api-version: 2006-03-01
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20250120T094325Z
    • Response:
    HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 0
Location: /empty-bucket
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: a4f5403139a65a8fd51b2aa0caa208291435f782459a282182de2ddf9eec98bf
X-Amz-Request-Id: 181C5D5BF637BDF2
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 11618
X-Ratelimit-Remaining: 11618
X-Xss-Protection: 1; mode=block
Date: Mon, 20 Jan 2025 09:43:25 GMT

  7. xhochy commented on Jan 20, 2025

    @xhochy
    xhochy
    on Jan 20, 2025
    Member

    Would it help your debugging if I build some of the versions in between on conda-forge?

  8. pitrou commented on Jan 20, 2025

    @pitrou
    pitrou
    on Jan 20, 2025
    Member

    Probably not, thanks for the proposal.

  9. pitrou commented on Jan 20, 2025

    @pitrou
    pitrou
    on Jan 20, 2025
    Member

    It appears there might be an easy workaround for this.

  10. github-actions
    assigned
    pitrou
    on Jan 20, 2025

  11. pitrou commented on Jan 20, 2025

    @pitrou
    pitrou
    on Jan 20, 2025
    Member

    Judging by https://httpwg.org/specs/rfc9112.html#chunked.encoding , this is really a bug in the AWS SDK. When using chunked encoding, there must be an explicit 0-sized chunk at the end, but it appears the AWS SDK client doesn't send one.

  12. h-vetinari commented on Jan 20, 2025

    @h-vetinari
    h-vetinari
    on Jan 20, 2025
    ContributorAuthor

    Judging by https://httpwg.org/specs/rfc9112.html#chunked.encoding , this is really a bug in the AWS SDK. When using chunked encoding, there must be an explicit 0-sized chunk at the end, but it appears the AWS SDK client doesn't send one.

    Thanks for digging into this so quickly! Could you open an issue in https://github.com/aws/aws-sdk-cpp? That would be useful for tracking/fixing this - if you want I can also do it, but I'd only be able to do a pale approximation, given that you understand the subject much better and did all the debugging.

  13. pitrou commented on Jan 20, 2025

    @pitrou
    pitrou
    on Jan 20, 2025
    Member

    I'll open an issue, yes. I think I've found the culprit.

  14. pitrou commented on Jan 20, 2025

    @pitrou
    pitrou
    on Jan 20, 2025
    Member

    Reported upstream at aws/aws-sdk-cpp#3259

  15. pitrou
    added this to the 20.0.0 milestone on Jan 22, 2025

  16. pitrou commented on Jan 22, 2025

    @pitrou
    pitrou
    on Jan 22, 2025
    Member

    Issue resolved by pull request 45310
    #45310

  17. pitrou
    closed this as completedon Jan 22, 2025
  18. amoeba
    modified the milestones: 20.0.0, 19.0.1 on Jan 23, 2025
  19. amoeba
    added 2 commits that reference this issue on Jan 31, 2025

    GH-45304: [C++][S3] Workaround compatibility issue between AWS SDK an…

    a622f10

    GH-45304: [C++][S3] Workaround compatibility issue between AWS SDK an…

    4be13f2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Component: C++Component: PackagingComponent: PythonType: enhancement

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions