Repository navigation
Fail instead of silently falling behind when a Lang-SDK vendored Dag schema drifts - #74091
Conversation
0a3913f to
1b8d138
Compare
…schema drifts from airflow-core A core Dag-serialization schema change shipped with the Go SDK's vendored copy left behind, uncaught by any hook: the sync hooks for Go and Java were manual only, and the one check that runs on every commit watched the vendored copies, not the Python source, so it never saw the schema change at all. The same drift already caused a two-release-old bug in the Go SDK's generated models (apache#73954). Go's and Java's sync hooks now run on every commit and are triggered by the Python source too, the way the TypeScript SDK's already was, refreshing the vendored copy and failing when they had to. Java's Gradle task keeps the same task name and stays silent for generateDagDsl's own dependency, so an ordinary in-repo build still refreshes a schema someone is mid-edit on without breaking; only the prek hook's invocation passes the flag that makes it fail.
1b8d138 to
4db6674
Compare
jason810496
left a comment
There was a problem hiding this comment.
After discussed with @pierrejeambrun, we decided to couple the Core schema bump and the SDK schema bump until 3.4 is out.
To prevent the critical drift between them before the first SDK release out.
Ideally, we should introduce the periodical CI kind of approach, I think we might able to make the job itself commit a sentinel file that will fail the specific CI job so that will be way more visible comparing to just sending a notification to that #internal-ci-cd-channel. But still decopule the core and SDK development, and will auto-route to the correct stakeholder to solve the issue
|
Lets try this while we don't have the other 'periodic' check and see if this help / is too annoying for development. We can always revert if it's too big of a constraint for core schema updates to have to update sdks too. |
sync-go-sdk-schemas stopped being manual in apache#74091, so the doc no longer describes what a contributor meets. It still says nothing announces a stale vendored copy, which is the problem that PR fixed, and its two refresh commands pass --hook-stage manual, which no longer selects the hook.
sync-go-sdk-schemas stopped being manual in apache#74091, so the doc no longer describes what a contributor meets. It still says nothing announces a stale vendored copy, which is the problem that PR fixed, and its two refresh commands pass --hook-stage manual, which no longer selects the hook.
Why
A core Dag-serialization schema change shipped recently with the Go SDK's vendored copy left behind, and nothing caught it: the Go and Java sync hooks were manual-only, and the check that runs on every commit (
check-go-sdk-generated-drift) watches the vendored copies, not the Python source, so it never saw the schema change at all. The same class of drift already caused a two-release-old bug in the Go SDK's generated models (#73954).What
sync-go-sdk-schemasandsync-java-sdk-dag-schemanow run on every commit (droppingstages: ['manual']), and are both triggered by the Python schema source itself, not just the vendored copy — matching howsync-ts-sdk-dag-schemaalready worked. Each refreshes the vendored copy and fails when it had to, so the refresh gets reviewed and committed rather than shipping unnoticed.check-go-sdk-generated-drift's trigger now also includes the Python schema sources, so it fires in the same commit as the sync hook above and catches generated Go code that still needs regenerating from the freshly-synced copy.SyncDagSchemaTask) gains afailOnRefreshflag, off by default.generateDagDsl's existing dependency keeps the flag off, so an ordinary in-repo build still refreshes a schema a developer is actively editing without breaking. The prek hook now calls a second task instance (checkDagSchemaSync) with the flag on.How I verified it
Ran each hook directly with a deliberately corrupted vendored copy and confirmed it refreshes the file and fails:
prek run sync-java-sdk-dag-schema --files airflow-core/src/airflow/serialization/schema.jsonprek run sync-go-sdk-schemas --files airflow-core/src/airflow/serialization/schema.jsonprek run check-go-sdk-generated-drift --files airflow-core/src/airflow/serialization/schema.jsonAlso confirmed
./java-sdk/gradlew :sdk:generateDagDslstill refreshes silently and succeeds (no regression to the normal in-repo build), while:sdk:checkDagSchemaSyncfails on the same drift.No newsfragment: CI/build tooling only, no user-facing behavior change.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.7) following the guidelines