how to turn off automatic decompression when download gzipped file #509

Activity

1. zezha-msft commented on Oct 10, 2018

   

   Ze Qian Zhang (zezha-msft)

   on Oct 10, 2018

   Contributor

   More actions

   Hi imeya, thanks for reaching out!

   I've seen a similar issue in the past, I'll investigate and let you know. Thanks!
2. Ze Qian Zhang (zezha-msft)

   self-assigned this

   on Oct 10, 2018
3. Ze Qian Zhang (zezha-msft)

   added

   investigating

   on Oct 10, 2018
4. imeya commented on Oct 22, 2018

   

   imeya

   on Oct 22, 2018

   Author

   More actions

   Ze Qian Zhang (@zezha-msft) , any feedback?
5. zezha-msft commented on Oct 22, 2018

   

   Ze Qian Zhang (zezha-msft)

   on Oct 22, 2018

   Contributor

   More actions

   Hi imeya, sorry for the delay.

   Could you please confirm with me what was the Content-type and Content-encoding of your blob?
6. imeya commented on Oct 23, 2018

   

   imeya

   on Oct 23, 2018

   Author

   More actions

   how can I check that in azure portal -> blob storage? I have checked the blob property from portal, but no such property you mentioned.
7. zezha-msft commented on Oct 23, 2018

   

   Ze Qian Zhang (zezha-msft)

   on Oct 23, 2018

   Contributor

   More actions

   Hi imeya, you could do a get blob properties call with any SDK. You could also use the Storage Explorer either on the portal or on the desktop; do a right click on your blob and then select properties; finally scroll to the bottom to see the Content-type and Content-encoding .
8. zezha-msft commented on Oct 25, 2018

   

   Ze Qian Zhang (zezha-msft)

   on Oct 25, 2018

   Contributor

   More actions

   Hi imeya, I did some digging and here's an update:

   When the Content-Encoding header is set to gzip (by the user who uploaded the blob, there is no default value), the requests module would decode it automatically, which is the expected behavior.

   If you have control over the data, I think the easiest way to fix this problem is to not set the Content-Encoding in the blobs' properties. If you cannot do that, then it's a bit more work to prevent the body from being decoded.

   One way to do that is to replace the _http_client that the BlockBlobService uses to perform HTTP requests. Currently, it reads the body by calling response.content which decodes the body. Here is an explanation on how to read the raw data instead.

   Here is a an example:

   from azure.storage.common._http.httpclient import (
       _HTTPClient,
       HTTPResponse,
   )
   from azure.storage.common._serialization import _get_data_bytes_or_stream_only
   import requests
   
   
   class ExampleRawBodyReadingClient(_HTTPClient):
       def perform_request(self, request):
           '''
           Sends an HTTPRequest to Azure Storage and returns an HTTPResponse. If
           the response code indicates an error, raise an HTTPError.
   
           :param HTTPRequest request:
               The request to serialize and send.
           :return: An HTTPResponse containing the parsed HTTP response.
           :rtype: :class:`~azure.storage.common._http.HTTPResponse`
           '''
           # Verify the body is in bytes or either a file-like/stream object
           if request.body:
               request.body = _get_data_bytes_or_stream_only('request.body', request.body)
   
           # Construct the URI
           uri = self.protocol.lower() + '://' + request.host + request.path
   
           # Send the request
           response = self.session.request(request.method,
                                           uri,
                                           params=request.query,
                                           headers=request.headers,
                                           data=request.body or None,
                                           timeout=self.timeout,
                                           proxies=self.proxies,
                                           stream=True)
   
           # Parse the response
           status = int(response.status_code)
           response_headers = {}
           for key, name in response.headers.items():
               # Preserve the case of metadata
               if key.lower().startswith('x-ms-meta-'):
                   response_headers[key] = name
               else:
                   response_headers[key.lower()] = name
   
           wrap = HTTPResponse(status, response.reason, response_headers, response.raw.read())
           response.close()
   
           return wrap

   The important part to note is that I added stream=True to the call self.session.request, and also used response.raw.read() instead of response.content when creating a HTTPResponse.

   I tested it with the following code:

   def run():
       bs = BlockBlobService(account_name="", account_key="")
       bs._httpclient = ExampleRawBodyReadingClient(session=requests.session(), protocol="https", timeout=2000)
       bs.get_blob_to_path(container_name="test_container", blob_name="test_blob",
                           file_path="./test_file")

   I verified that the file downloaded this way has the right content, by computing its MD5 and comparing against the gzipped file. Please let me know if you have any question.
9. Ze Qian Zhang (zezha-msft)

   closed this as completedon Oct 25, 2018
10. Shige Takeda (sfc-gh-stakeda)

    added a commit that references this issue on Dec 9, 2019

    SNOW-116535: fix GET command for Azure for workaround Azure/azure-sto…

    ...

    583fa06
11. Ankit Bhatnagar (ankit-bhatnagar167)

    added a commit that references this issue on Dec 12, 2019

    SNOW-116535: fix GET command for Azure for workaround Azure/azure-sto…

    ...

    ba691f8
12. Adam Ling (sfc-gh-aling)

    added a commit that references this issue on Oct 17, 2022

    SNOW-116535: fix GET command for Azure for workaround Azure/azure-sto…

    ...

    2419eea