Skip to content

fix(workflow): accept work items larger than gRPC's 4MB default - #1337

Merged
CasperGN merged 1 commit into
dapr:mainfrom
JoshVanL:fix/workflow-worker-max-receive-size
Oct 22, 2026
Merged

CasperGN merged 1 commit into
dapr:mainfrom
JoshVanL:fix/workflow-worker-max-receive-size

Conversation

@JoshVanL

Copy link
Copy Markdown
Contributor

The workflow worker's gRPC channel kept gRPC's default 4MB receive limit unless the app set max_grpc_message_length or DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES. The sidecar sends work items up to its own max body size, which apps raise for large payloads, so a work item over 4MB (for example a fan-out of five 1MB activity results) failed the stream with RESOURCE_EXHAUSTED. The worker reconnected, the sidecar redelivered the same work item, and it failed again, so every workflow on the worker stalled.

The workflow worker's gRPC channel kept gRPC's default 4MB receive
limit unless the app set max_grpc_message_length or
DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES. The sidecar sends work items
up to its own max body size, which apps raise for large payloads, so a
work item over 4MB (for example a fan-out of five 1MB activity
results) failed the stream with RESOURCE_EXHAUSTED. The worker
reconnected, the sidecar redelivered the same work item, and it failed
again, so every workflow on the worker stalled.

Signed-off-by: joshvanl <me@joshvanl.dev>
Copilot AI balanced review requested due to automatic review settings October 22, 2026 11:52
@JoshVanL
JoshVanL requested review from a team as code owners October 22, 2026 11:52
@codecov

codecov Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.33%. Comparing base (fddc30f) to head (cf70540).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1337      +/-   ##
==========================================
+ Coverage   87.27%   87.33%   +0.06%     
==========================================
  Files         163      163              
  Lines       13069    13069              
==========================================
+ Hits        11406    11414       +8     
+ Misses       1663     1655       -8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused change preserves explicit limits and includes targeted regression coverage, with no unresolved findings.

No open issues 🎉

What changed in this PR

This PR removes the workflow worker’s default 4 MiB receive limit so large work items do not cause repeated stream failures and stall workflows.

Changes:

  • Allows unlimited receive size by default while preserving explicitly configured limits.
  • Adds regression tests for oversized work-item processing and explicit-limit precedence.
  • Updates runtime documentation to describe the new default behavior.
File Description
tests/​ext/​workflow/​durabletask/​test_worker_large_work_item.py Tests large work-item delivery and explicit receive-limit preservation.
dapr/​ext/​workflow/​workflow_runtime.py Documents the default send and receive behavior.
dapr/​ext/​workflow/​_durabletask/​worker.py Sets an unlimited default receive size before applying caller options.

Review effort: Balanced

@CasperGN CasperGN self-assigned this Oct 22, 2026
@CasperGN
CasperGN added this pull request to the merge queue Oct 22, 2026
Merged via the queue into dapr:main with commit ba2b297 Oct 22, 2026
28 checks passed
dapr-bot added a commit that referenced this pull request Oct 22, 2026
… (#1339)

The workflow worker's gRPC channel kept gRPC's default 4MB receive
limit unless the app set max_grpc_message_length or
DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES. The sidecar sends work items
up to its own max body size, which apps raise for large payloads, so a
work item over 4MB (for example a fan-out of five 1MB activity
results) failed the stream with RESOURCE_EXHAUSTED. The worker
reconnected, the sidecar redelivered the same work item, and it failed
again, so every workflow on the worker stalled.


(cherry picked from commit ba2b297)

Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: dapr-bot <dapr-bot@users.noreply.github.com>
Co-authored-by: Josh van Leeuwen <me@joshvanl.dev>
@CasperGN CasperGN removed their assignment Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants