Skip to content

--download reports an "incomplete download" with Content-Encoding: gzip and content-length. #423

Open
Open
--download reports an "incomplete download" with Content-Encoding: gzip and content-length.#423
Labels
bugSomething isn't working

Description

@twopir
twopir
opened on Dec 25, 2015

When using HTTPie to download a file which is both gzip'd and has a content-length header sent, the final output report says that the download is incomplete. This is apparently because HTTPie tracks the number of bytes it wrote, not the number of bytes it received.

Here is an example, which reproduces in HTTPie head as of about 10 minutes ago.

$ http --debug --download GET http://torcache.net/torrent/640FE84C613C17F663551D218689A64E8AEBEABE.torrent 
HTTPie 1.0.0-dev
HTTPie data: /Users/pi/.httpie
Requests 2.6.0
Pygments 2.0.2
Python 2.7.10 (default, Oct 23 2015, 18:05:06) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] darwin

>>> requests.request(**{'allow_redirects': True,
 'auth': None,
 'cert': None,
 'data': OrderedDict(),
 'files': DataDict(),
 'headers': {'Accept-Encoding': None, 'User-Agent': 'HTTPie/1.0.0-dev'},
 'method': u'get',
 'params': ParamsDict(),
 'proxies': {},
 'stream': True,
 'timeout': 30,
 'url': u'http://torcache.net/torrent/640FE84C613C17F663551D218689A64E8AEBEABE.torrent',
 'verify': True})

HTTP/1.1 200 OK
Accept-Ranges: bytes
Connection: close
Content-Encoding: gzip
Content-Length: 40082
Content-Type: application/x-bittorrent
Date: Fri, 25 Dec 2015 04:39:03 GMT
ETag: "567c9773-9c92"
Last-Modified: Fri, 25 Dec 2015 01:10:11 GMT
Server: nginx

Downloading 39.14 kB to "640FE84C613C17F663551D218689A64E8AEBEABE.torrent"
 -   0.00 %     0.00 B     0.00 B/s  -:--:-- ETA
http: error: Incomplete download: size=40082; downloaded=40260
Done. 39.32 kB in 0.22557s (174.30 kB/s)

Note that size is the value of Content-length, but downloaded is greater than size because of the gzip encoding.

Activity

  1. jkbrzt
    added
    bugSomething isn't working
    on Dec 25, 2015

  2. fbidu commented on Jan 30, 2016

    @fbidu
    fbidu
    on Jan 30, 2016

    Yeah, I could validate it here... I'll try to fix it

  3. twopir commented on Jan 30, 2016

    @twopir
    twopir
    on Jan 30, 2016
    Author

    I think that fix is still vulnerable to a problem where the gzip content is incompletely downloaded, but still expands to greater than the content-length. I don't have a demonstration handy, though.

  4. fbidu commented on Jan 30, 2016

    @fbidu
    fbidu
    on Jan 30, 2016

    hmm yeah, it could still be trouble... I'll try a better fix

  5. jkbrzt commented on Feb 28, 2016

    @jkbrzt
    jkbrzt
    on Feb 28, 2016
    Member

    https://github.com/jkbrzt/httpie/blob/2b51cb6687df037b94c901d5a3651296fd3cf07c/httpie/downloads.py#L181-L182

  6. giuscri commented on Mar 6, 2016

    @giuscri
    giuscri
    on Mar 6, 2016

    To me, it seems like Response.raw should be used; otherwise HTTPResponse.iter_body will return gzip -d'd chunks.

  7. jkbrzt
    added a commit that references this issue on Mar 9, 2016

    Add `Accept-Encoding: identity` for --download

    25d1e8e

  8. hholst80 commented on Oct 18, 2023

    @hholst80
    hholst80
    on Oct 18, 2023

    Greetings from 2023.
    This is still an issue.

    [I] root@xps /tmp
# http -d 'https://XXX.blob.core.windows.net/CONTAINER/path/to/processed.json?sp=r&st=2023-10-18T11:45:05Z&se=2023-10-18T19:45:05Z&spr=https&sv=2022-11-02&sr=b&sig=PKZTjGYM0wfeU%2BpF%2FMZ93Y2RxDyVHs1xbOeg4of2nuA%3D'
HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Content-Encoding: gzip
Content-Length: 2087
Content-MD5: Q6dkiGs7Jcp6PimVCRH09A==
Content-Type: application/json
Date: Wed, 18 Oct 2023 11:45:21 GMT
ETag: "0x8DB4C84489078F7"
Last-Modified: Thu, 04 May 2023 09:45:23 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-blob-type: BlockBlob
x-ms-creation-time: Thu, 04 May 2023 09:45:23 GMT
x-ms-lease-state: available
x-ms-lease-status: unlocked
x-ms-request-id: ce24f842-101e-006a-72b8-01eea9000000
x-ms-server-encrypted: true
x-ms-version: 2022-11-02

Downloading to processed.json
Done. 4.3 kB in 00:0.10676 (39.9 kB/s)

http: LogLevel.ERROR: Incomplete download: size=2087; downloaded=4264


[I] root@xps /tmp [1]
#

  9. hholst80 commented on Oct 18, 2023

    @hholst80
    hholst80
    on Oct 18, 2023

    To me, it seems like Response.raw should be used; otherwise HTTPResponse.iter_body will return gzip -d'd chunks.

    I think the output should be the decompressed output as it is now. It is just the warning that should go away. If the raw download matches what the http header said, all is good.

    Make raw an opt-in for those cases when you really do not want the raw content encoding to be downloaded as-is. I think this issue should focus on getting rid of the error message by measuring the right thing.

  10. Ousret commented on May 23, 2024

    @Ousret
    Ousret
    on May 23, 2024 · Hidden as spam
  11. jkbrzt
    added a commit that references this issue on Jun 14, 2024

    Decoded downloads: refactor, add decoded size notice, tests, etc.

    540d3f3
  12. o1x3
    added a commit that references this issue on Feb 5, 2026

    Fix false "Incomplete download" error with Content-Encoding

    85de63b
  13. rodionsteshenko
    added a commit that references this issue on Feb 18, 2026

    Fix httpie#1642: Don't flag download as interrupted when Content-Enco…

    e47f365
  14. pfurpass
    added a commit that references this issue on Mar 13, 2026

    Fix false 'Incomplete download' error when Content-Encoding is set

    9e45790
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions