Skip to content

Fail instead of silently falling behind when a Lang-SDK vendored Dag schema drifts - #74091

Merged
pierrejeambrun merged 1 commit into
apache:mainfrom
astronomer:detect-lang-sdk-schema-drift
Oct 5, 2026
Merged

pierrejeambrun merged 1 commit into
apache:mainfrom
astronomer:detect-lang-sdk-schema-drift

Conversation

@pierrejeambrun

Copy link
Copy Markdown
Member

Why

A core Dag-serialization schema change shipped recently with the Go SDK's vendored copy left behind, and nothing caught it: the Go and Java sync hooks were manual-only, and the check that runs on every commit (check-go-sdk-generated-drift) watches the vendored copies, not the Python source, so it never saw the schema change at all. The same class of drift already caused a two-release-old bug in the Go SDK's generated models (#73954).

What

  • sync-go-sdk-schemas and sync-java-sdk-dag-schema now run on every commit (dropping stages: ['manual']), and are both triggered by the Python schema source itself, not just the vendored copy — matching how sync-ts-sdk-dag-schema already worked. Each refreshes the vendored copy and fails when it had to, so the refresh gets reviewed and committed rather than shipping unnoticed.
  • check-go-sdk-generated-drift's trigger now also includes the Python schema sources, so it fires in the same commit as the sync hook above and catches generated Go code that still needs regenerating from the freshly-synced copy.
  • Java's underlying Gradle task (SyncDagSchemaTask) gains a failOnRefresh flag, off by default. generateDagDsl's existing dependency keeps the flag off, so an ordinary in-repo build still refreshes a schema a developer is actively editing without breaking. The prek hook now calls a second task instance (checkDagSchemaSync) with the flag on.

How I verified it

Ran each hook directly with a deliberately corrupted vendored copy and confirmed it refreshes the file and fails:

  • prek run sync-java-sdk-dag-schema --files airflow-core/src/airflow/serialization/schema.json
  • prek run sync-go-sdk-schemas --files airflow-core/src/airflow/serialization/schema.json
  • prek run check-go-sdk-generated-drift --files airflow-core/src/airflow/serialization/schema.json

Also confirmed ./java-sdk/gradlew :sdk:generateDagDsl still refreshes silently and succeeds (no regression to the normal in-repo build), while :sdk:checkDagSchemaSync fails on the same drift.

No newsfragment: CI/build tooling only, no user-facing behavior change.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

…schema drifts from airflow-core

A core Dag-serialization schema change shipped with the Go SDK's vendored copy
left behind, uncaught by any hook: the sync hooks for Go and Java were manual
only, and the one check that runs on every commit watched the vendored copies,
not the Python source, so it never saw the schema change at all. The same
drift already caused a two-release-old bug in the Go SDK's generated models
(apache#73954).

Go's and Java's sync hooks now run on every commit and are triggered by the
Python source too, the way the TypeScript SDK's already was, refreshing the
vendored copy and failing when they had to. Java's Gradle task keeps the same
task name and stays silent for generateDagDsl's own dependency, so an
ordinary in-repo build still refreshes a schema someone is mid-edit on
without breaking; only the prek hook's invocation passes the flag that makes
it fail.

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After discussed with @pierrejeambrun, we decided to couple the Core schema bump and the SDK schema bump until 3.4 is out.

To prevent the critical drift between them before the first SDK release out.

Ideally, we should introduce the periodical CI kind of approach, I think we might able to make the job itself commit a sentinel file that will fail the specific CI job so that will be way more visible comparing to just sending a notification to that #internal-ci-cd-channel. But still decopule the core and SDK development, and will auto-route to the correct stakeholder to solve the issue

@pierrejeambrun

pierrejeambrun commented Oct 5, 2026 •

Copy link
Copy Markdown
Member Author

Lets try this while we don't have the other 'periodic' check and see if this help / is too annoying for development.

We can always revert if it's too big of a constraint for core schema updates to have to update sdks too.

@pierrejeambrun
pierrejeambrun merged commit b191cc4 into apache:main Oct 5, 2026
101 checks passed
@pierrejeambrun
pierrejeambrun deleted the detect-lang-sdk-schema-drift branch October 5, 2026 09:05
henry3260 added a commit to henry3260/airflow that referenced this pull request Oct 31, 2026
sync-go-sdk-schemas stopped being manual in apache#74091, so the doc no longer
describes what a contributor meets. It still says nothing announces a stale
vendored copy, which is the problem that PR fixed, and its two refresh
commands pass --hook-stage manual, which no longer selects the hook.
henry3260 added a commit to henry3260/airflow that referenced this pull request Oct 31, 2026
sync-go-sdk-schemas stopped being manual in apache#74091, so the doc no longer
describes what a contributor meets. It still says nothing announces a stale
vendored copy, which is the problem that PR fixed, and its two refresh
commands pass --hook-stage manual, which no longer selects the hook.
henry3260 added a commit that referenced this pull request Oct 31, 2026
…76144)

sync-go-sdk-schemas stopped being manual in #74091, so the doc no longer
describes what a contributor meets. It still says nothing announces a stale
vendored copy, which is the problem that PR fixed, and its two refresh
commands pass --hook-stage manual, which no longer selects the hook.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants