Change default encoding to utf-8 in `normalize_header_key` and `normalize_header_value` functions

[SonderZhong]

Description

I have encountered decoding errors with some requests that use ASCII encoding. Changing the default encoding to UTF-8 resolves these errors. I propose updating the normalize_header_key and normalize_header_value functions in _utils.py to use UTF-8 as the default encoding.

Steps to Reproduce

1. Call normalize_header_key or normalize_header_value with a non-ASCII string and no encoding specified.
2. Observe the decoding failure with ASCII encoding.
3. Change the encoding to UTF-8 and observe that the error is resolved.

Example Code

header_key_unicode = "内容类型"
normalized_key_unicode = normalize_header_key(header_key_unicode, lower=True)
# This raises a UnicodeEncodeError with ASCII encoding.

normalized_key_unicode_utf8 = normalize_header_key(header_key_unicode, lower=True, encoding="utf-8")
print(normalized_key_unicode_utf8)  # Works correctly with UTF-8 encoding.

Proposed Solution

Modify the _utils.py file to use UTF-8 as the default encoding:

def normalize_header_key(
    value: str | bytes,
    lower: bool,
    encoding: str | None = None,
) -> bytes:
    """
    Coerce str/bytes into a strictly byte-wise HTTP header key.
    """
    if isinstance(value, bytes):
        bytes_value = value
    else:
        bytes_value = value.encode(encoding or "utf-8")

    return bytes_value.lower() if lower else bytes_value

def normalize_header_value(value: str | bytes, encoding: str | None = None) -> bytes:
    """
    Coerce str/bytes into a strictly byte-wise HTTP header value.
    """
    if isinstance(value, bytes):
        return value
    return value.encode(encoding or "utf-8")

Rationale

Using UTF-8 as the default encoding ensures that the functions can handle a wider range of input values without raising an error. UTF-8 encoding is capable of encoding a larger set of characters compared to ASCII.

[iamjatinyadav]

hey guys, can I work on this?

[SonderZhong]

hey guys, can I work on this?
Thank you for your interest in this issue! I am currently working on a Pull Request to address this problem. If you have any suggestions or would like to review the changes, your input would be greatly appreciated.

[lovelydinosaur]

Heya, thanks for the consideration... I think this may be valid, tho could you re-frame it so that you're describing the issue against public API. For example describe this using httpx.Headers(...). We can then unpick exactly what should and should not be valid.

Edit: Okay, I see PR #3241 now. We don't want to support utf-8 in header keys which have a constrained set of allowed characters. I do think it's a sensible default for header values tho.

[SonderZhong]

嘿，谢谢你的考虑......我认为这可能是有效的，您能否重新构建它，以便您描述针对公共 API 的问题。例如，使用 来描述这一点。然后，我们可以准确地选择哪些内容应该有效，哪些内容不应该有效。httpx.Headers(...)

编辑：好的，我现在看到 PR #3241。我们不希望在具有一组受约束的允许字符的标头键中支持 utf-8。我确实认为这是标头值的明智默认值。

In a request, if the headers contain non-ASCII characters and the encoding is not specified, an error will be raised. For example:

from httpx import Headers

x = Headers(
    {
        "Referer": "https://www.google.com/search?q=テスト",
    }
)
print(x)

In the example code, the Referer field contains the Japanese word テスト, which will raise an error and interrupt the request.

The same issue can occur in responses. If the server's response does not specify the encoding, httpx will use ASCII encoding to decode テスト, resulting in an error.

Therefore, I suggest changing the default encoding from ASCII to UTF-8 to better handle such cases.

[Cycloctane]

@ZM25XC UTF-8 is not accepted in headers. You can only use ISO-8859-1 in HTTP headers according to rfc.

If you want to put utf8 characters in the url and pass it to referer, you should url encode all utf8 characters first. In fact, this is how this url actually is.

from httpx import Headers
from urllib.parse import quote

x = Headers(
    {
        "Referer": f"https://www.google.com/search?q={quote('テスト')}",
    }
)
print(x)
# Headers({'referer': 'https://www.google.com/search?q=%E3%83%86%E3%82%B9%E3%83%88'})

[andrewtlw]

hello @lovelydinosaur, if not utf-8 in the headers, would it make sense to default to iso8859-1 instead? I am also facing the same issue as the OP

[lovelydinosaur]

I am also facing the same issue as the OP

Could you be more specific?

[andrewtlw]

Could you be more specific?

Sure.

from httpx import Response

x = Response(status_code=200, headers={"x-ratelimit-reset-requests": "59.999µs"})
print(x.headers)

Results the below error.

Traceback (most recent call last):
  File "/Users/user/development/httpx/demo.py", line 10, in <module>
    x = Response(status_code=200, headers={"x-ratelimit-reset-requests": "59.999µs"})
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/development/httpx/httpx/_models.py", line 532, in __init__
    self.headers = Headers(headers)
                   ^^^^^^^^^^^^^^^^
  File "/Users/user/development/httpx/httpx/_models.py", line 156, in __init__
    bytes_value = _normalize_header_value(v, encoding)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/development/httpx/httpx/_models.py", line 82, in _normalize_header_value
    return value.encode(encoding or "ascii")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'ascii' codec can't encode character '\xb5' in position 6: ordinal not in range(128)

I suppose this is because self.headers = Headers(headers) is called during __init__ and the default_encoding is not passed?

For header values, perhaps it should default to iso-8859-1 or utf-8.

[lovelydinosaur]

The behaviour for header encoding/decoding outside of the ascii range isn't defined, so the safest approach is to keep to strict ascii, with URL quoting if required. (Eg. #3238 (comment))

[SonderZhong]

The behaviour for header encoding/decoding outside of the ascii range isn't defined, so the safest approach is to keep to strict ascii, with URL quoting if required. (Eg. #3238 (comment))
I understand this is for security reasons, but some developers' responses have headers that are returned in an irregular format. When we use httpx to make requests, we can't manually encode them. Could you add a parameter to allow us to specify the parsing encoding?

[SonderZhong]

When using httpx to actively send requests, we can manually encode and handle the headers, but we cannot pre-encode and handle the hesders of the request response. Most errors occur when parsing the headers of the response

[lovelydinosaur]

we can manually encode and handle the headers, but we cannot pre-encode and handle the hesders of the request response.

Correct yep.

Using iso-8859-1 would be robust to servers returning headers including obs-text byte ranges %x80-FF. RFC-9110 indicates that clients should allow these byte ranges and treat them as "opaque data" ie. no defined encoding.

As a useful data point, do you have an example URL that's returning headers with these values?

[andrewtlw]

I don't have a public URL that returns headers with these values, but the example I cited above is based on headers in the OpenAI API spec, where it is possible that some responses have non-ascii characters in header values "x-ratelimit-reset-requests": "90µs"