Repository navigation
Exception serialization improvement #309
Description
When job failed, the result becomes an exception, which causes problems on serialization without pickle.
async def job_failed(exc: BaseException) -> None:
self.jobs_failed += 1
result_data_ = serialize_result(
function=function_name,
args=args,
kwargs=kwargs,
job_try=job_try,
enqueue_time_ms=enqueue_time_ms,
success=False,
result=exc,
start_ms=start_ms,
finished_ms=timestamp_ms(),
ref=f'{job_id}:{function_name}',
serializer=self.job_serializer,
queue_name=self.queue_name,
)
await asyncio.shield(self.finish_failed_job(job_id, result_data_))
Activity
iamlikeme commentedon Nov 19, 2022
Since pickle is the standard serializer, you are apparently using a custom serializer. So it is up to you how to serialize exceptions.
Perhaps I misunderstand your question?
kigawas commentedon Nov 24, 2022
How can I serialize exceptions to JSON when the job failed?
iamlikeme commentedon Dec 11, 2022
How to serialize an exception to JSON is a very generic question outside the scope of arq. StackOverflow has some suggestion.
heshiming commentedon Aug 9, 2023
I think what the OP meant to ask was how to serialize the exception when
job.result()is supposed to reraise it but can't due to serialization.I'm on version 0.25.0. When
job.result()is called on ajobusing themsgpackdeserializer, it immediately fails witharq.jobs.SerializationError: unable to serialize result. From the code, deserialization happens before reraising. And sincemsgpackcannot serialize an exception,info.resultwill not be anException.As a serializer,
msgpackcannot deal with the case where the job failed due to an exception.SimonDahrs commentedon Feb 5, 2025
Found a workaround when using msgpack: you can define your custom serialization function that checks if the 'r' field (result) in the dict passed to it is of type Exception. If it is, you can override that field with the str repr of the exception so that msgpack can serialize it. This is how I do it: