Repository navigation
[Rust SDK] Add the Cargo workspace, Gradle integration and model crate - #40508
prodriguezdefino wants to merge 1 commit into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #40508 +/- ##
============================================
+ Coverage 58.54% 58.57% +0.03%
Complexity 2276 2276
============================================
Files 1139 1133 -6
Lines 169887 169980 +93
Branches 1482 1482
============================================
+ Hits 99452 99574 +122
+ Misses 67886 67857 -29
Partials 2549 2549
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Hey @prodriguezdefino, I would like to help this PR as much as possible since I didn't know Beam is getting a Rust SDK. I am new to beam and I am excited to see Rust based work here. I tested this branch locally on Windows with One small note: building on Windows requires Happy to help test upcoming PRs on Windows or contribute to the downstream crates as the SDK progresses. |
50d94ea to
fbfacb2
Compare
|
Thanks for testing this out on Windows @CH-Abhinav, great to hear In Beam, host dependencies are normally set up via Once the RFC document (it's on the dev@ distro if you're curious about it) gets reviewed and hopefully, eventually approved, I planned to start working on having the initial PRs land. This draft is more about making sure checks are in place for once/if things get in motion. It would be great to have your help making sure things work smoothly on Windows! |
|
/gemini review |
There was a problem hiding this comment.
📋 Review Summary
This PR cleanly scaffolds the new Apache Beam Rust SDK workspace (sdks/rust), Gradle integration (:sdks:rust), release version tooling, and the apache-beam-model crate for compiling portability Protobuf and gRPC definitions. The structure is well-organized and follows Beam's repository conventions.
🔍 General Feedback
- Environment Setup (
local-env-setup.sh): Whileprotocwas added to the macOS setup path, it is currently missing from the Linux setup path (dev-support/docker/pkglistdoes not includeprotobuf-compiler). - Release Script (
release/src/main/scripts/set_version.sh): Thegit add 'sdks/rust/*Cargo.toml'pathspec only matches single-depth subdirectories in Git, which misses crate manifests nested deeper such assdks/rust/beam/model/Cargo.toml.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
fbfacb2 to
0443786
Compare
|
/gemini review |
There was a problem hiding this comment.
📋 Review Summary
This PR cleanly sets up the initial Cargo workspace, Gradle build integration (:sdks:rust), version-bumping scripts, environment setup, and the apache-beam-model crate with generated Protobuf/gRPC bindings. All feedback from the previous review (adding protobuf-compiler to dev-support/docker/pkglist for Linux setups and resolving the broken link in README.md) has been addressed.
🔍 General Feedback
- The Gradle-to-Cargo task delegation (
build,check,test,fmt,clean) and version propagation fromgradle.propertiesintegrate smoothly with existing Beam build and release conventions. - Protobuf compilation in
beam/model/build.rsproperly registerscargo:rerun-if-changeddirectives for all model proto inputs.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
9a8d7b9 to
a6c9738
Compare
|
/gemini review |
There was a problem hiding this comment.
📋 Review Summary
This PR cleanly bootstraps the Apache Beam Rust SDK workspace (:sdks:rust), providing the initial apache-beam-model crate with prost/tonic protobuf and gRPC code generation alongside Gradle task integration and version management scripts. The workspace structure, crate re-exports, and developer documentation are well-organized and align well with existing multi-language SDK conventions in the repository.
🔍 General Feedback
- The
apache-beam-modelcrate setup inbuild.rsandsrc/lib.rsneatly encapsulates the deep protobuf package hierarchy behind a privateorgmodule while exposing flatpipeline,fn_execution,job_management, andexpansionmodules. - A couple of minor edge cases in
sdks/rust/scripts/set_version.shandsdks/rust/gradle/cargo.gradlearound dependency attribute ordering and empty Gradle property handling are worth tightening.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
a6c9738 to
f9338b3
Compare
|
/gemini review |
There was a problem hiding this comment.
📋 Review Summary
This PR introduces the initial foundation for the Apache Beam Rust SDK under sdks/rust, including Cargo workspace setup (beam, beam-proto, beam-worker), tonic/prost protobuf generation for the Beam Runner/Fn/Job APIs, Gradle integration, and release version scripts.
🔍 General Feedback
- The overall structure of the Rust workspace and
beam-protobuild script is clean and idiomatic. - There are a couple of issues in the Gradle task wiring (
testlifecycle task not wired tocargoTest) and the shell script logic (local-env-setup.shoperator precedence bug on Linux, andset_version.shfailure whenCargo.lockis gitignored/absent).
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
💬 Inline Findings
local-env-setup.sh(line 71): 🟠 High: In Bash,&&has higher precedence than||. Incommand -v rustup && rustup update || curl ..., ifrustupis already installed (command -v rustupsucceeds) andrustup updatefails (e.g., due to a transient network issue or toolchain error), the||branch will execute and re-run therustup.rsinstaller script. Additionally, whenrustupis freshly installed viacurl ... | sh -s -- -y,~/.cargo/envis not sourced in the current shell session, so subsequent commands in the same session won't findcargoorrustuponPATH(and on macOS at line 159,rustup-init -yis interactive unless invoked or flags are passed appropriately, orrustup updateis run).
if command -v rustup >/dev/null 2>&1; then rustup update; else curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y; fi
sdks/rust/build.gradle.kts(line 68): 🟠 High:org.apache.beam.moduleonly applies the Gradlebaseplugin by default (unless a language nature is applied), which definescheckandbuild, but not atesttask. Here,tasks.register("test")registers an empty lifecycle task that does not depend oncargoTest. Running./gradlew :sdks:rust:testwill therefore succeed without running any Rust tests.
tasks.register("test") { dependsOn(cargoTest) }
sdks/rust/scripts/set_version.sh(line 53): 🟠 High:sdks/rust/Cargo.lockis ignored in.gitignoreand not tracked in the repository. Whenrelease/src/main/scripts/set_version.shinvokes this script on a clean checkout wherecargohas not been run (or ifcargois not installed on the machine running the version bump script),sdks/rust/Cargo.lockwill not exist andsed -i ... "$SCRIPT_DIR/../Cargo.lock"will fail underset -e(andgit add sdks/rust/Cargo.lockinrelease/src/main/scripts/set_version.shwill also fail due to.gitignore). Guard theCargo.lockupdate with an existence check or only updateCargo.tomlfiles.
if [[ -f "$SCRIPT_DIR/../Cargo.lock" ]]; then sed -i -e '/name = "apache-beam/,/version =/{s/version = .*/version = "'"$TARGET_VERSION"'"/;}' "$SCRIPT_DIR/../Cargo.lock"; fi
sdks/rust/beam-proto/build.rs(line 57): 🟡 Medium: Only emittingcargo:rerun-if-changedfor the parent directory (proto_root) only watches the directory entry itself, not modifications to.protofiles inside nested subdirectories (org/apache/beam/model/...). Iterating overproto_filesand emittingcargo:rerun-if-changed={proto_file}ensures Cargo recompiles the crate whenever any.protodefinition is modified.
for proto in &proto_files { println!("cargo:rerun-if-changed={}", proto.display()); }
sdks/rust/Cargo.toml(line 31): 🟢 Low: The repository URL points to the personal forkhttps://github.com/robertwb/beaminstead of the canonicalhttps://github.com/apache/beamrepository.
repository = "https://github.com/apache/beam"
|
These findings don't match this revision (f9338b3), so they look like they came from a different version of the code or were hallucinated:
/gemini review |
There was a problem hiding this comment.
📋 Review Summary
This PR cleanly bootstraps the Apache Beam Rust SDK workspace (sdks/rust), Gradle task integration (:sdks:rust), release version tooling, and the apache-beam-model crate with prost/tonic Protobuf and gRPC code generation. All issues raised in earlier review rounds have been addressed.
🔍 General Feedback
- The Cargo workspace configuration, lint rules, and
apache-beam-modelprotobuf build script (build.rs) are well-structured and idiomatic. - The Gradle integration (
sdks/rust/gradle/cargo.gradleandversions.gradle) and version-bumping script (sdks/rust/scripts/set_version.sh) align cleanly with existing Apache Beam build and release conventions.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
|
/gemini review |
There was a problem hiding this comment.
📋 Review Summary
This PR cleanly scaffolds the new Apache Beam Rust SDK Cargo workspace (sdks/rust), adds the apache-beam-model crate for Protobuf/gRPC model bindings, and integrates it with Gradle and release scripts. Several minor issues were identified around git pathspec globbing in set_version.sh for nested crates, PATH availability after installing rustup in local-env-setup.sh, and documentation/configuration details.
🔍 General Feedback
- The Cargo workspace configuration, lint rules, and
tonic-prostbuild setup for the Beam portability model are well-structured and idiomatic. - In
release/src/main/scripts/set_version.sh, the pathspec'sdks/rust/*Cargo.toml'relies on shell/git behavior that may not match nested manifests (or fails if unexpanded depending on git configuration); recursive matching should be used. sdks/rust/gradle/versions.gradlereferencesbeam/core/build.rs, which is not yet included in this PR.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
381e369 to
c9c4e3e
Compare
136705a to
3eac799
Compare
|
/gemini review |
There was a problem hiding this comment.
📋 Review Summary
This PR introduces the initial foundation for the Apache Beam Rust SDK (:sdks:rust), including the Cargo workspace configuration, Gradle lifecycle integration, release version-bumping scripts, and the apache-beam-model crate for building Beam portability protobuf/gRPC definitions. The overall structure, workspace lint setup, and incremental Gradle task design are clean and well-documented.
🔍 General Feedback
- Gradle
buildWorkspacevsclippytasks (sdks/rust/gradle/cargo.gradle):buildWorkspace.doLastreferences an undefinedbeamCargoEnvproperty (which will cause aMissingPropertyExceptionat runtime) and runscargo clippy --workspace --tests --benchesduringbuildrather than duringclippy/check. - Release script staging (
release/src/main/scripts/set_version.sh): Thegit add 'sdks/rust/*Cargo.toml'pathspec does not match nested crate manifests undersdks/rust/beam/*/Cargo.tomlwhen git literal pathspecs or deeper directory trees are involved. awkportability (sdks/rust/scripts/set_version.sh): Theawkinvocation usessub(/\nversion = .../)which fails to match on macOS/BSDawkwhere\ninside a regex literal is not treated as a newline.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
3eac799 to
f15d27c
Compare
|
Assigning reviewers: R: @Amar3tto for label build. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
kennknowles
left a comment
There was a problem hiding this comment.
LGTM with two pieces of feedback on gradle setup
| // Cargo does the real work; these tasks give maintainers the same commands as the rest | ||
| // of the repository (`./gradlew :sdks:rust:build`, `:check`, `:test`). | ||
|
|
||
| plugins { id 'org.apache.beam.module' } |
There was a problem hiding this comment.
If you can, use build.gradle.kts from the outset. Adding the types reduces annoying stuff in the gradle builds. With agents it may or may not matter, but eliminating possible hallucinations and adding an automated check is still probably good. Beam started down this path before LLMs and never finished. If you actually already have a bunch of non-Kotlin gradle stuff it is fine, but FYI that we want it to be Kotlin.
|
|
||
| description = "Apache Beam :: SDKs :: Rust" | ||
|
|
||
| apply from: 'gradle/versions.gradle' |
There was a problem hiding this comment.
This is OK but it is better if you can make it a proper plugin in the buildSrc magic subdirectory. There are examples there and an agent should be able to port it in a oneshot (or tell me why that cannot work)
First change in the series that adds the Apache Beam Rust SDK. This change adds: - The Cargo workspace at sdks/rust (rustfmt and clippy configuration, workspace lints, dev profiles) with a single member. - apache-beam-model: protobuf and gRPC bindings generated at build time from the model protos in this repository (model/pipeline, model/fn-execution, model/job-management). - Gradle integration (:sdks:rust), so the SDK builds like the rest of the repository: build, check (rustfmt + clippy -D warnings), test, fmt and clean. build.gradle.kts applies the org.apache.beam.rust plugin of buildSrc (package org.apache.beam.gradle.rust); the tasks call cargo, which handles incremental builds. Gradle passes the Beam version from gradle.properties to cargo; no Beam version is hard-coded in the Rust sources. - Release tooling: release/src/main/scripts/set_version.sh bumps the Cargo manifests through sdks/rust/scripts/set_version.sh. - local-env-setup.sh installs Rust and protoc (protobuf-compiler in dev-support/docker/pkglist on Linux and in the Docker build env). - README.md and docs/development.md.
f15d27c to
52d06b7
Compare
|
Thanks for the review @kennknowles! I think I can follow the approach of having a proper module for this and grow the needed types there. I made the change in the PR to have One question for the PRs that would follow. They add more functions to the Gradle build:
Which layout would be preferred for these?
I would personally go with the second option, because it keeps buildSrc small. JIC, the plugin is in Groovy with @CompileStatic, like the other buildSrc plugins. Tell me if you want it in Kotlin. |
First change in the series that adds an Apache Beam Rust SDK (#21089).
This change adds:
apache-beam-model: protobuf and gRPC bindings generated at build time from the model protos in this repository (model/pipeline,model/fn-execution,model/job-management).:sdks:rust), so the SDK builds like the rest of the repository: build, check (rustfmt + clippy -D warnings), test, fmt and clean.sdks/rust/build.gradle.ktsapplies theorg.apache.beam.rust pluginofbuildSrc(packageorg.apache.beam.gradle.rust); the tasks call cargo, which handles incremental builds. Gradle passes the Beam version fromgradle.propertiesto cargo, no Beam version is hard-coded in the Rust sources.release/src/main/scripts/set_version.shbumps the Cargo manifests throughsdks/rust/scripts/set_version.sh.local-env-setup.shinstalls Rust and protoc.README.mdanddocs/development.md.