Skip to content

[Rust SDK] Add the Cargo workspace, Gradle integration and model crate - #40508

Open
prodriguezdefino wants to merge 1 commit into
apache:masterfrom
prodriguezdefino:rust/01-workspace-model
Open

prodriguezdefino wants to merge 1 commit into
apache:masterfrom
prodriguezdefino:rust/01-workspace-model

Conversation

@prodriguezdefino

@prodriguezdefino prodriguezdefino commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First change in the series that adds an Apache Beam Rust SDK (#21089).

This change adds:

  • The Cargo workspace at sdks/rust (rustfmt and clippy configuration, workspace lints, dev profiles) with a single member.
  • apache-beam-model: protobuf and gRPC bindings generated at build time from the model protos in this repository (model/pipeline, model/fn-execution, model/job-management).
  • Gradle integration (:sdks:rust), so the SDK builds like the rest of the repository: build, check (rustfmt + clippy -D warnings), test, fmt and clean. sdks/rust/build.gradle.kts applies the org.apache.beam.rust plugin of buildSrc (package org.apache.beam.gradle.rust); the tasks call cargo, which handles incremental builds. Gradle passes the Beam version from gradle.properties to cargo, no Beam version is hard-coded in the Rust sources.
  • Release tooling: release/src/main/scripts/set_version.sh bumps the Cargo manifests through sdks/rust/scripts/set_version.sh.
  • local-env-setup.sh installs Rust and protoc.
  • README.md and docs/development.md.

@github-actions github-actions Bot added the build label Oct 11, 2026
@codecov

codecov Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.57%. Comparing base (84a02fe) to head (f15d27c).
⚠️ Report is 154 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #40508      +/-   ##
============================================
+ Coverage     58.54%   58.57%   +0.03%     
  Complexity     2276     2276              
============================================
  Files          1139     1133       -6     
  Lines        169887   169980      +93     
  Branches       1482     1482              
============================================
+ Hits          99452    99574     +122     
+ Misses        67886    67857      -29     
  Partials       2549     2549              
Flag Coverage Δ
python 80.56% <ø> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@CH-Abhinav

Copy link
Copy Markdown
Contributor

Hey @prodriguezdefino, I would like to help this PR as much as possible since I didn't know Beam is getting a Rust SDK. I am new to beam and I am excited to see Rust based work here.

I tested this branch locally on Windows with rustc 1.98 and protoc 36.0 — cargo check built cleanly and all proto/gRPC definitions generated without issues.

One small note: building on Windows requires protoc in PATH (or setting the PROTOC environment variable). Would it be helpful to add a quick mention in docs/development.md regarding installing protoc on Windows (e.g., via winget install Google.Protobuf or manual binary download)?

Happy to help test upcoming PRs on Windows or contribute to the downstream crates as the SDK progresses.

@prodriguezdefino
prodriguezdefino force-pushed the rust/01-workspace-model branch 2 times, most recently from 50d94ea to fbfacb2 Compare October 17, 2026 00:33
@prodriguezdefino

Copy link
Copy Markdown
Contributor Author

Thanks for testing this out on Windows @CH-Abhinav, great to hear cargo check and proto generation worked cleanly!

In Beam, host dependencies are normally set up via local-env-setup.sh or by using the Dockerized development environment via start-build-env.sh at the root of the repository, rather than listing per-OS installation steps in each SDK's docs.

Once the RFC document (it's on the dev@ distro if you're curious about it) gets reviewed and hopefully, eventually approved, I planned to start working on having the initial PRs land. This draft is more about making sure checks are in place for once/if things get in motion.

It would be great to have your help making sure things work smoothly on Windows!

@prodriguezdefino

Copy link
Copy Markdown
Contributor Author

/gemini review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR cleanly scaffolds the new Apache Beam Rust SDK workspace (sdks/rust), Gradle integration (:sdks:rust), release version tooling, and the apache-beam-model crate for compiling portability Protobuf and gRPC definitions. The structure is well-organized and follows Beam's repository conventions.

🔍 General Feedback

  • Environment Setup (local-env-setup.sh): While protoc was added to the macOS setup path, it is currently missing from the Linux setup path (dev-support/docker/pkglist does not include protobuf-compiler).
  • Release Script (release/src/main/scripts/set_version.sh): The git add 'sdks/rust/*Cargo.toml' pathspec only matches single-depth subdirectories in Git, which misses crate manifests nested deeper such as sdks/rust/beam/model/Cargo.toml.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

Comment thread local-env-setup.sh Resolved
Comment thread release/src/main/scripts/set_version.sh Resolved
Comment thread sdks/rust/README.md Resolved
@prodriguezdefino

Copy link
Copy Markdown
Contributor Author

/gemini review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR cleanly sets up the initial Cargo workspace, Gradle build integration (:sdks:rust), version-bumping scripts, environment setup, and the apache-beam-model crate with generated Protobuf/gRPC bindings. All feedback from the previous review (adding protobuf-compiler to dev-support/docker/pkglist for Linux setups and resolving the broken link in README.md) has been addressed.

🔍 General Feedback

  • The Gradle-to-Cargo task delegation (build, check, test, fmt, clean) and version propagation from gradle.properties integrate smoothly with existing Beam build and release conventions.
  • Protobuf compilation in beam/model/build.rs properly registers cargo:rerun-if-changed directives for all model proto inputs.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@prodriguezdefino
prodriguezdefino force-pushed the rust/01-workspace-model branch 3 times, most recently from 9a8d7b9 to a6c9738 Compare October 22, 2026 05:30
@github-actions github-actions Bot added the size/xl XL: >=1000 lines changed label Oct 24, 2026
@CH-Abhinav CH-Abhinav mentioned this pull request Oct 24, 2026
@CH-Abhinav

Copy link
Copy Markdown
Contributor

/gemini review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR cleanly bootstraps the Apache Beam Rust SDK workspace (:sdks:rust), providing the initial apache-beam-model crate with prost/tonic protobuf and gRPC code generation alongside Gradle task integration and version management scripts. The workspace structure, crate re-exports, and developer documentation are well-organized and align well with existing multi-language SDK conventions in the repository.

🔍 General Feedback

  • The apache-beam-model crate setup in build.rs and src/lib.rs neatly encapsulates the deep protobuf package hierarchy behind a private org module while exposing flat pipeline, fn_execution, job_management, and expansion modules.
  • A couple of minor edge cases in sdks/rust/scripts/set_version.sh and sdks/rust/gradle/cargo.gradle around dependency attribute ordering and empty Gradle property handling are worth tightening.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

Comment thread sdks/rust/scripts/set_version.sh Resolved
Comment thread sdks/rust/gradle/cargo.gradle Resolved
@prodriguezdefino

Copy link
Copy Markdown
Contributor Author

/gemini review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR introduces the initial foundation for the Apache Beam Rust SDK under sdks/rust, including Cargo workspace setup (beam, beam-proto, beam-worker), tonic/prost protobuf generation for the Beam Runner/Fn/Job APIs, Gradle integration, and release version scripts.

🔍 General Feedback

  • The overall structure of the Rust workspace and beam-proto build script is clean and idiomatic.
  • There are a couple of issues in the Gradle task wiring (test lifecycle task not wired to cargoTest) and the shell script logic (local-env-setup.sh operator precedence bug on Linux, and set_version.sh failure when Cargo.lock is gitignored/absent).

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

💬 Inline Findings

  • local-env-setup.sh (line 71): 🟠 High: In Bash, && has higher precedence than ||. In command -v rustup && rustup update || curl ..., if rustup is already installed (command -v rustup succeeds) and rustup update fails (e.g., due to a transient network issue or toolchain error), the || branch will execute and re-run the rustup.rs installer script. Additionally, when rustup is freshly installed via curl ... | sh -s -- -y, ~/.cargo/env is not sourced in the current shell session, so subsequent commands in the same session won't find cargo or rustup on PATH (and on macOS at line 159, rustup-init -y is interactive unless invoked or flags are passed appropriately, or rustup update is run).
    if command -v rustup >/dev/null 2>&1; then rustup update; else curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y; fi
  • sdks/rust/build.gradle.kts (line 68): 🟠 High: org.apache.beam.module only applies the Gradle base plugin by default (unless a language nature is applied), which defines check and build, but not a test task. Here, tasks.register("test") registers an empty lifecycle task that does not depend on cargoTest. Running ./gradlew :sdks:rust:test will therefore succeed without running any Rust tests.
tasks.register("test") { dependsOn(cargoTest) }
  • sdks/rust/scripts/set_version.sh (line 53): 🟠 High: sdks/rust/Cargo.lock is ignored in .gitignore and not tracked in the repository. When release/src/main/scripts/set_version.sh invokes this script on a clean checkout where cargo has not been run (or if cargo is not installed on the machine running the version bump script), sdks/rust/Cargo.lock will not exist and sed -i ... "$SCRIPT_DIR/../Cargo.lock" will fail under set -e (and git add sdks/rust/Cargo.lock in release/src/main/scripts/set_version.sh will also fail due to .gitignore). Guard the Cargo.lock update with an existence check or only update Cargo.toml files.
if [[ -f "$SCRIPT_DIR/../Cargo.lock" ]]; then sed -i -e '/name = "apache-beam/,/version =/{s/version = .*/version = "'"$TARGET_VERSION"'"/;}' "$SCRIPT_DIR/../Cargo.lock"; fi
  • sdks/rust/beam-proto/build.rs (line 57): 🟡 Medium: Only emitting cargo:rerun-if-changed for the parent directory (proto_root) only watches the directory entry itself, not modifications to .proto files inside nested subdirectories (org/apache/beam/model/...). Iterating over proto_files and emitting cargo:rerun-if-changed={proto_file} ensures Cargo recompiles the crate whenever any .proto definition is modified.
    for proto in &proto_files { println!("cargo:rerun-if-changed={}", proto.display()); }
  • sdks/rust/Cargo.toml (line 31): 🟢 Low: The repository URL points to the personal fork https://github.com/robertwb/beam instead of the canonical https://github.com/apache/beam repository.
repository = "https://github.com/apache/beam"

@prodriguezdefino

prodriguezdefino commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor Author

These findings don't match this revision (f9338b3), so they look like they came from a different version of the code or were hallucinated:

  • local-env-setup.sh has no command -v rustup && … || curl chain. It checks type -P rustc in an if block.
  • There is no sdks/rust/build.gradle.kts and no cargoTest. test is itself an Exec task that runs cargo test (gradle/cargo.gradle).
  • sdks/rust/Cargo.lock is tracked. It is only excluded from RAT, not gitignored.
  • beam/model/build.rs already emits rerun-if-changed for each .proto file.
  • repository is already https://github.com/apache/beam.

/gemini review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR cleanly bootstraps the Apache Beam Rust SDK workspace (sdks/rust), Gradle task integration (:sdks:rust), release version tooling, and the apache-beam-model crate with prost/tonic Protobuf and gRPC code generation. All issues raised in earlier review rounds have been addressed.

🔍 General Feedback

  • The Cargo workspace configuration, lint rules, and apache-beam-model protobuf build script (build.rs) are well-structured and idiomatic.
  • The Gradle integration (sdks/rust/gradle/cargo.gradle and versions.gradle) and version-bumping script (sdks/rust/scripts/set_version.sh) align cleanly with existing Apache Beam build and release conventions.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@CH-Abhinav

Copy link
Copy Markdown
Contributor

/gemini review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR cleanly scaffolds the new Apache Beam Rust SDK Cargo workspace (sdks/rust), adds the apache-beam-model crate for Protobuf/gRPC model bindings, and integrates it with Gradle and release scripts. Several minor issues were identified around git pathspec globbing in set_version.sh for nested crates, PATH availability after installing rustup in local-env-setup.sh, and documentation/configuration details.

🔍 General Feedback

  • The Cargo workspace configuration, lint rules, and tonic-prost build setup for the Beam portability model are well-structured and idiomatic.
  • In release/src/main/scripts/set_version.sh, the pathspec 'sdks/rust/*Cargo.toml' relies on shell/git behavior that may not match nested manifests (or fails if unexpanded depending on git configuration); recursive matching should be used.
  • sdks/rust/gradle/versions.gradle references beam/core/build.rs, which is not yet included in this PR.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

Comment thread release/src/main/scripts/set_version.sh Resolved
Comment thread local-env-setup.sh Resolved
Comment thread sdks/rust/Cargo.toml Resolved
Comment thread sdks/rust/gradle/versions.gradle Resolved
@prodriguezdefino
prodriguezdefino force-pushed the rust/01-workspace-model branch 3 times, most recently from 381e369 to c9c4e3e Compare October 27, 2026 23:52
@prodriguezdefino
prodriguezdefino force-pushed the rust/01-workspace-model branch 5 times, most recently from 136705a to 3eac799 Compare November 12, 2026 18:48
@CH-Abhinav

Copy link
Copy Markdown
Contributor

/gemini review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR introduces the initial foundation for the Apache Beam Rust SDK (:sdks:rust), including the Cargo workspace configuration, Gradle lifecycle integration, release version-bumping scripts, and the apache-beam-model crate for building Beam portability protobuf/gRPC definitions. The overall structure, workspace lint setup, and incremental Gradle task design are clean and well-documented.

🔍 General Feedback

  • Gradle buildWorkspace vs clippy tasks (sdks/rust/gradle/cargo.gradle): buildWorkspace.doLast references an undefined beamCargoEnv property (which will cause a MissingPropertyException at runtime) and runs cargo clippy --workspace --tests --benches during build rather than during clippy/check.
  • Release script staging (release/src/main/scripts/set_version.sh): The git add 'sdks/rust/*Cargo.toml' pathspec does not match nested crate manifests under sdks/rust/beam/*/Cargo.toml when git literal pathspecs or deeper directory trees are involved.
  • awk portability (sdks/rust/scripts/set_version.sh): The awk invocation uses sub(/\nversion = .../) which fails to match on macOS/BSD awk where \n inside a regex literal is not treated as a newline.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

Comment thread sdks/rust/gradle/cargo.gradle Resolved
Comment thread release/src/main/scripts/set_version.sh Resolved
Comment thread sdks/rust/scripts/set_version.sh Resolved
Comment thread sdks/rust/scripts/set_version.sh Resolved
Comment thread sdks/rust/clippy.toml Resolved
@prodriguezdefino
prodriguezdefino marked this pull request as ready for review November 16, 2026 21:08
@prodriguezdefino prodriguezdefino changed the title WIP [Rust SDK] Add the Cargo workspace, Gradle integration and model crate [Rust SDK] Add the Cargo workspace, Gradle integration and model crate Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @Amar3tto for label build.
Backup reviewer: Abacn (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@kennknowles kennknowles left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with two pieces of feedback on gradle setup

Comment thread sdks/rust/build.gradle Outdated
// Cargo does the real work; these tasks give maintainers the same commands as the rest
// of the repository (`./gradlew :sdks:rust:build`, `:check`, `:test`).

plugins { id 'org.apache.beam.module' }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you can, use build.gradle.kts from the outset. Adding the types reduces annoying stuff in the gradle builds. With agents it may or may not matter, but eliminating possible hallucinations and adding an automated check is still probably good. Beam started down this path before LLMs and never finished. If you actually already have a bunch of non-Kotlin gradle stuff it is fine, but FYI that we want it to be Kotlin.

Comment thread sdks/rust/build.gradle Outdated

description = "Apache Beam :: SDKs :: Rust"

apply from: 'gradle/versions.gradle'

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is OK but it is better if you can make it a proper plugin in the buildSrc magic subdirectory. There are examples there and an agent should be able to port it in a oneshot (or tell me why that cannot work)

First change in the series that adds the Apache Beam Rust SDK.

This change adds:
- The Cargo workspace at sdks/rust (rustfmt and clippy configuration,
  workspace lints, dev profiles) with a single member.
- apache-beam-model: protobuf and gRPC bindings generated at build time
  from the model protos in this repository (model/pipeline,
  model/fn-execution, model/job-management).
- Gradle integration (:sdks:rust), so the SDK builds like the rest of
  the repository: build, check (rustfmt + clippy -D warnings), test, fmt
  and clean. build.gradle.kts applies the org.apache.beam.rust plugin of
  buildSrc (package org.apache.beam.gradle.rust); the tasks call cargo,
  which handles incremental builds. Gradle passes the Beam version from
  gradle.properties to cargo; no Beam version is hard-coded in the Rust
  sources.
- Release tooling: release/src/main/scripts/set_version.sh bumps the
  Cargo manifests through sdks/rust/scripts/set_version.sh.
- local-env-setup.sh installs Rust and protoc (protobuf-compiler in
  dev-support/docker/pkglist on Linux and in the Docker build env).
- README.md and docs/development.md.
@prodriguezdefino

prodriguezdefino commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the review @kennknowles!

I think I can follow the approach of having a proper module for this and grow the needed types there. I made the change in the PR to have sdks/rust/build.gradle.kts applying a new plugin in buildSrc, and the tasks stay the same.

One question for the PRs that would follow. They add more functions to the Gradle build:

  • SDK container images
  • Prism and Dataflow launch tasks
  • ValidatesRunner and cross-language test suites
  • I/O integration tests (GCP, Kafka, Iceberg)
  • Quality tasks (coverage, mutation tests, complexity) and profiling
  • Benchmarks (harness, Nexmark, I/O)

Which layout would be preferred for these?

  • All of them in the buildSrc plugin.
  • The shared part in buildSrc (cargo tasks, lint, tests, containers). The specific parts, such as benchmarks and cross-language suites, go in subprojects with their own build.gradle.kts.

I would personally go with the second option, because it keeps buildSrc small.

JIC, the plugin is in Groovy with @CompileStatic, like the other buildSrc plugins. Tell me if you want it in Kotlin.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants