Skip to content

Link dbt Cloud runs to the triggering Airflow task via OpenLineage - #70131

Merged
mobuchowski merged 2 commits into
mainfrom
dbt-cloud-cause
Sep 15, 2026
Merged

mobuchowski merged 2 commits into
mainfrom
dbt-cloud-cause

Conversation

@mobuchowski

Copy link
Copy Markdown
Contributor

dbt Cloud does not natively propagate the OpenLineage parent run of the Airflow task that triggered a job, so a triggered run has no stored link back to that task. Any tool that reads dbt Cloud runs on their own therefore sees them as isolated jobs, with no way to attach them to the Airflow run that started them. This adds an openlineage_inject_parent_job_info option to DbtCloudRunJobOperator. When enabled, it writes the task's OpenLineage parent and root run identifiers, as JSON, into the triggered run's cause field, so a consumer reading dbt Cloud runs can parse them and rebuild the ParentRunFacet that links the run to its Airflow task.

It is conceptually the same thing we've added in #44477 #64513 or #64807 but using different, dbt cloud specific mechanism.

The option defaults to the existing openlineage.spark_inject_parent_job_info config value, matching the convention already used by the Glue and EMR operators. Because dbt Cloud limits cause to 255 characters, the payload is trimmed to fit: the parent identifiers are always kept, the root block is dropped if the two together are too long, and injection is skipped if even the parent alone does not fit. When injection is enabled the cause is replaced entirely with these identifiers rather than the human-readable trigger reason, since the correlation is the point of the option.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 4.8) following the guidelines

@kacpermuda kacpermuda left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left two comments, lgtm overall

@mobuchowski
mobuchowski force-pushed the dbt-cloud-cause branch 2 times, most recently from b025b7c to bb141e2 Compare July 28, 2026 08:29
mobuchowski and others added 2 commits September 11, 2026 09:34
Signed-off-by: Maciej Obuchowski <maciej.obuchowski@datadoghq.com>
…dbt.py

Co-authored-by: Kacper Muda <mudakacper@gmail.com>
@mobuchowski
mobuchowski merged commit 0038a0d into main Sep 15, 2026
80 checks passed
@mobuchowski
mobuchowski deleted the dbt-cloud-cause branch September 15, 2026 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants