Repository navigation
Ensure JSON representation is compact. #3363
Description
Activity
This bug have a huge impact on the openai package (1.52.0).
Openai package uses httpx to make it requests, and it turns out that when we make call in French, the GPT models receives text in the wrong encoding. The encoding should be UTF-8 !For example :
- input text in French : "salut ça va ? très bien ou pas bien ?"
- GPT receives in payload : "salut \u00e7a va ? tr\u00e8s bien ou pas bien ?"
This could have huge impact on text understanding by LLMs, and also could generate more costs for intesive llm users.
Here's a comparison in the official openai tokenizer to prove the point :
UTF-8 correct encoding ==> 11 tokens for GPT-4o
ASCII encoding ==> 21 tokens !!!
Please have this issue fixed.
For those who are looking for a workaround, you can override the method that causes the issue in the entry point of your app (main.py for example).
The patch would be :
import httpx._content from httpx import ByteStream from json import dumps as json_dumps def custom_httpx_encode_json(json: Any) -> tuple[dict[str, str], ByteStream]: # disable ascii for json_dumps body = json_dumps(json, ensure_ascii=False).encode("utf-8") # <------------------------------------------------ content_length = str(len(body)) content_type = "application/json" headers = {"Content-Length": content_length, "Content-Type": content_type} return headers, ByteStream(body) # fix encoding utf-8 bug httpx._content.encode_json = custom_httpx_encode_json
Openai package uses httpx to make it requests, and it turns out that when we make call in French, the GPT models receives text in the wrong encoding.
We're compliant with the JSON spec here, and parsers will be decoding text escape sequences into the correct unicode text. Tho you are correct that a more compact format would be sensible here. Would you like to submit a pull request resolving this?
The following related issues could all be resolved together...
- We should use
ensure_ascii=Falsefor more compact text representations. - We should use
separators = (',', ':')for more compact list and object representations. - We should use
allow_nan=Falseto disallow invalidInfinityandNaNrepresentations.
Useful review points are the
UNICODE_JSON,COMPACT_JSONandSTRICT_JSONsettings in REST framework where we previously worked through these design issues.- We should use
Openai package uses httpx to make it requests, and it turns out that when we make call in French, the GPT models receives text in the wrong encoding.
We're compliant with the JSON spec here, and parsers will be decoding text escape sequences into the correct unicode text. Tho you are correct that a more compact format would be sensible here. Would you like to submit a pull request resolving this?
The following related issues could all be resolved together...
- We should use
ensure_ascii=Falsefor more compact text representations. - We should use
separators = (',', ':')for more compact list and object representations. - We should use
allow_nan=Falseto disallow invalidInfinityandNaNrepresentations.
Useful review points are the
UNICODE_JSON,COMPACT_JSONandSTRICT_JSONsettings in REST framework where we previously worked through these design issues.Thank you for your reply.
I will submit a pull request to resolve the issue.
RegardsReacted by Chris- We should use
- changed the title
[-]httpx forces converting json content to ascii[/-][+]Ensure JSON representation is compact.[/+]on Oct 25, 2024 - added a commit that references this issue
on Oct 26, 2024 - added a commit that references this issue
on Oct 28, 2024 Thanks @BERRADA-Omar
Reacted by BERRADA-Omar- added a commit that references this issue
on Feb 20, 2025 Hi, I’d like to work on this issue.
I plan to add configurable JSON serialization options that allow users to control Unicode encoding (e.g. ensure_ascii), while maintaining full backward compatibility with the existing behavior.
I’ll submit a PR once ready. Thanks!
Discussed in #3204
Originally posted by alleml May 17, 2024
httpx.Request blindly dumps provided json to ascii characters by invoking json.dumps without any parameters. Currently there is no way to provide
ensure_ascii=Falseargument to the Request instance (or even better to the client).It leads to the following scenario:
when I need properly encoded unicode bytes:
b"cze\xc5\x9b\xc4\x87"