Unable to serialize Retry error with msgpack #481

Activity

1. einarwar commented on Oct 17, 2024

   

   einarwar

   on Oct 17, 2024

   Last edited by einarwar

   More actions

   Related to #309. I am experiencing the same issue myself, and haven't found a way of serializing exceptions. I am using orjson, but like msgpack it cannot serialize exceptions either.

   One solution could be to convert the exception to a string and then return that one, e.g

   import sys
   import traceback
   
   try:
       ... # Your code here
   except MyException:
       exception_string = "".join(traceback.format_exception(*sys.exc_info()))
       return exception_string

   But this is not a really good solution as you would need to manually process this string in the caller, and it is a really non-standard way of handling exceptions
2. einarwar commented on Oct 17, 2024

   

   einarwar

   on Oct 17, 2024

   More actions

   I found out my ArqRedis had set decode_responses=True, which was the reason i needed orjson as a custom serializer in the first place. By setting decode_responses=False (the default), the default serializer (pickle) can serialize both exceptions as well as "normal" return values
3. SimonDahrs commented on Feb 5, 2025

   

   SimonDahrs

   on Feb 5, 2025

   More actions

   Found a workaround when using msgpack: you can define your custom serialization function that checks if the 'r' field (result) in the dict passed to it is of type Exception. If it is, you can override that field with the str repr of the exception so that msgpack can serialize it. This is how I do it:

   def job_msgpack_serialize(obj: dict[str, Any]) -> bytes:
       if "r" in obj and isinstance(obj["r"], Exception):
           obj["r"] = repr(obj["r"])  # make sure exceptions are serializable by msgpack
       return msgpack.packb(obj)