Skip to content

Metric dag.dag_id.task_id.queued_duration missing #63503

Closed

Description

@melicheradam

Apache Airflow version

3.1.8

If "Other Airflow 3 version" selected, which one?

No response

What happened?

The metric dag.dag_id.task_id.queued_duration is not being emitted after Airflow 3 upgrade. There are multiple issues like this where metrics are reported missing (we have also noticed ti.finish, ti.start missing, but that has been fixed in 3.1.8).

What you think should happen instead?

All of the metrics that are described in the documentation https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#metric-descriptions should be emitted.

How to reproduce

Run airflow, run task, observe metrics

Operating System

Linux

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. added 2 commits that reference this issue on Mar 13, 2026
    32a76f0
    c47731a
  2. added 3 commits that reference this issue on Mar 23, 2026
    d3658b4
    d6b7fc1
    0752d81
  3. added 6 commits that reference this issue on Apr 5, 2026
    da40da0
    b8b18e8
    1df08fa
    e05790c
    2084a6c
    77b6a47
  4. 17 remaining items

  5. eladkal commented on May 26, 2026

    @eladkal
    Contributor

    @seanmuth you previously fixed some of the missing metrics. Will you have time to look into this one?

  6. wilmerdooley commented on May 28, 2026

    @wilmerdooley
    Contributor

    I had a look at this one. The root cause is that queued_duration is emitted by TaskInstance.emit_state_change_metric(RUNNING), which in Airflow 2 ran when the task transitioned to RUNNING. In Airflow 3 that transition moved to the execution API (the ti_run route), so the call site in taskinstance.py no longer fires and the metric is never emitted.

    Its sibling scheduled_duration still works because it is emitted on the QUEUED transition, which is still triggered server side from the scheduler (scheduler_job_runner).

    The change that keeps the two symmetric is to emit task.queued_duration server side in the ti_run handler when a task goes from QUEUED to RUNNING, computing now - queued_dttm. The metric template already maps task.queued_duration to the legacy dag.<dag_id>.<task_id>.queued_duration name, so both the tagged and statsd forms are covered.

    @seanmuth are you planning to take this one? If not, I am happy to open a PR with a test.

  7. mahesh-github commented on May 28, 2026

    @mahesh-github

    Can I take this one pls

  8. added a commit that references this issue on Jun 1, 2026
    7e7bd6b
  9. added a commit that references this issue on Jun 16, 2026
    4a2a311
  10. added 2 commits that reference this issue on Jul 30, 2026
    948cbcd
    0f3fb97
  11. added a commit that references this issue on Sep 7, 2026
    aa9383e
  12. added a commit that references this issue on Oct 14, 2026
    21c858e
  13. added 2 commits that reference this issue on Oct 18, 2026
    88d425b
    ec22ea6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions