Repository navigation
💥 Enforce payload and memo size limits - #3149
jmaeagle99 wants to merge 8 commits into
Conversation
2e2f296 to
28a65bd
Compare
28a65bd to
c9ab565
Compare
| RespondActivityTaskFailedRequest request = pendingPayloadLimitReport.getAndSet(null); | ||
| if (request == null) { | ||
| return; | ||
| } |
There was a problem hiding this comment.
Instead of having a thread grab the payload limit report and try sending it with retries, I think a cleaner design would be: save the report in HeartbeatContextImpl while we still have the lock, and then ask all subsequent threads who send a heartbeats to send the error report instead. That mirrors what we already have for activity results, and we wouldn't suffer the headache of extra concurrency.
There was a problem hiding this comment.
We try to fail the activity as soon as the heartbeat is rejected, match what was done for Core based SDKs and the server's own behavior for oversized heartbeat details. This allows the server to retry immediately even if the activity is slow to observe the cancellation. State has been consolidated into one object as described in #3149
What was changed
Workers now enforce the namespace's payload and memo size limits before sending, and every client warns about oversized payloads.
Validation at the gRPC layer. A new interceptor in
temporal-serviceclientchecks every WorkflowService request on the final message, after user interceptors have run. Fields over a warning threshold are logged with[TMPRL1103]and still sent. When a call carries error limits and a field exceeds them, the call fails locally withINVALID_ARGUMENTand is never sent or retried.Worker enforcement. Workers read the namespace's blob and memo error limits from the
DescribeNamespacecall they already make at startup. They attach the limits only to their own completion and heartbeat calls, so other client calls only warn. Each worker turns a rejected call into a retryable failure instead of sending it:PAYLOADS_TOO_LARGE.ApplicationFailureof typePayloadsTooLarge.Each case records a
PayloadsTooLargefailure reason on the existing task failure metrics.Configuration.
WorkflowServiceStubsOptions.Builder.setPayloadLimitssets the warning thresholds: 512 KiB per payload and 2 KiB per memo by default, with 0 disabling a warning. It is experimental.WorkerOptions.Builder.setDisablePayloadErrorLimitturns off worker enforcement.Why?
The server already rejects oversized payloads, but harshly: it terminates the workflow, or fails the activity non-retryably. Failing the task retryably in the worker lets a corrected workflow or activity be redeployed and recover. This also brings Java in line with the core-based SDKs and Go, which already enforce these limits.
Checklist