Skip to content

Backfill task is stuck in "No status" state #58554

Closed

Description

@aru-trackunit

Apache Airflow version

3.1.3

If "Other Airflow 2/3 version" selected, which one?

No response

What happened?

Backfill task is stuck in "No status" state.

There is around 100 DAGs running on different schedule. Usually around 30 tasks constantly running.

When triggering a backfill for 2025-11-17, I successfully pass the form step, the dag run is successfully created and is in running status, however the task itself does not start, and is state in database is null.

I restarted all the airflow kubernetes pods, removed a dag data using delete dag and readded to airflow and after the restart task still does not progress.

I have checked parallelism which is set to 40 and max tasks runnning were around 30, there are 93 open slots in the pool.

I have waited for 30 minutes for this task to be picked up but nothing happened.

dag record:

dag_id is_paused is_stale last_parsed_time last_parse_duration last_expired fileloc relative_fileloc bundle_name bundle_version owners dag_display_name description timetable_summary timetable_description asset_expression deadline max_active_tasks max_active_runs max_consecutive_failed_dag_runs has_task_concurrency_limits has_import_errors next_dagrun next_dagrun_data_interval_start next_dagrun_data_interval_end next_dagrun_create_after
ble-monitoring FALSE FALSE 2025-11-21 10:45:56.027054+00 0.14697423899997375 /opt/airflow/dags/dashboards/dag_ble_monitoring.py dashboards/dag_ble_monitoring.py dags-folder team_iot_data BLE Monitoring Dashboard on PowerBi 30 0 * * * At 00:30 null null 16 1 0 FALSE FALSE 2025-11-21 00:30:00+00 2025-11-21 00:30:00+00 2025-11-22 00:30:00+00 2025-11-22 00:30:00+00

dag_run record:

id dag_id queued_at logical_date start_date end_date state run_id creating_job_id run_type triggered_by triggering_user_name conf data_interval_start data_interval_end run_after last_scheduling_decision log_template_id updated_at clear_number backfill_id bundle_version scheduled_by_job_id context_carrier span_status created_dag_version_id
8980 ble-monitoring 2025-11-21 10:31:50.877612+00 2025-11-11 00:30:00+00 2025-11-21 10:31:51.01211+00 NULL running backfill__2025-11-12T00:30:00+00:00 null backfill BACKFILL okta_00ud5ffefmodlaFL2357 {} 2025-11-11 00:30:00+00 2025-11-12 00:30:00+00 2025-11-12 00:30:00+00 null 1 2025-11-21 10:31:51.012965+00 0 9 null null {"__var": {}, "__type": "dict"} not_started 019aa5d3-ae8f-7d68-9ef4-b97916d1470b

task_run record:

task_id state pool queue start_date end_date try_number
ble_monitoring NULL default_pool default NULL NULL 0

backfill record:

id dag_id from_date to_date dag_run_conf is_paused reprocess_behavior max_active_runs created_at completed_at updated_at triggering_user_name
9 ble-monitoring 2025-11-11 00:30:00+00 2025-11-11 00:30:00+00 {} FALSE completed 1 2025-11-21 10:31:50.861649+00 NULL 2025-11-21 10:31:50.861653+00 okta_00ud5ffefmodlaFL2357

backfill_dag_run record:

id backfill_id dag_run_id exception_reason logical_date sort_ordinal
9 9 8980 NULL 2025-11-11 00:30:00+00 1

I wonder if I am the first one to see this - Do you think might be the case that database state is corrupted?

My god feeling is that scheduler is not catching up properly, do you know what metrics should observe?

Image

What you think should happen instead?

I would expect backfill task runs to have executed and completed or failed, within 5-10 minutes.

How to reproduce

  1. Click on Backfill button in a Dag
  2. Pick any date that fits the and Run Backfill
Image

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon          9.17.0
apache-airflow-providers-cncf-kubernetes 10.10.0
apache-airflow-providers-common-compat   1.8.0
apache-airflow-providers-common-io       1.6.4
apache-airflow-providers-common-sql      1.29.0
apache-airflow-providers-databricks      7.7.5
apache-airflow-providers-fab             3.0.2
apache-airflow-providers-github          2.9.4
apache-airflow-providers-hashicorp       4.3.4
apache-airflow-providers-http            5.5.0
apache-airflow-providers-microsoft-mssql 4.3.3
apache-airflow-providers-mysql           6.3.5
apache-airflow-providers-postgres        6.4.1
apache-airflow-providers-sftp            5.4.2
apache-airflow-providers-slack           9.4.0
apache-airflow-providers-smtp            2.3.1
apache-airflow-providers-ssh             4.1.6
apache-airflow-providers-standard        1.9.1

Deployment

Official Apache Airflow Helm Chart

Deployment details

Example dag:

import datetime as dt
from collections import defaultdict

from airflow_addons.operators.databricks import DatabricksSubmitRunOperator # our internal wrapper of DatabricksSubmitRunOperator

from airflow.sdk import DAG, Variable

DAG_ID = "ble-monitoring"

default_args = {
    "owner": "team_iot_data",
    "depends_on_past": False,
    "start_date": dt.datetime(2025, 11, 1),
}

ble_monitoring_task = {
    "notebook_path": "ble_monitoring/ble_monitoring_data_refresh",
    "base_parameters": {
        "COMPUTE_FOR_DATE": "{{ data_interval_start.strftime('%Y-%m-%d') }}"
    }
}

with DAG(
        dag_id=DAG_ID,
        description="BLE Monitoring Dashboard on PowerBi",
        default_args=default_args,
        schedule="30 0 * * *",
        catchup=False,
        max_active_runs=1,
        tags={"dashboards"},
        doc_md=__doc__,
) as dag:

    DatabricksSubmitRunOperator(
        task_id="ble_monitoring",
        new_cluster={},
        notebook_task=ble_monitoring_task,
        timeout_seconds=3600,  # 1 hours
        polling_period_seconds=30,
        retries=0,
        databricks_retry_limit=10,
        databricks_retry_delay=10,
        do_xcom_push=True,
        git_url="https://github.com/repo/",
    )

env vars:

AIRFLOW_API_SERVER_PORT_8080_TCP_PROTO=tcp
AIRFLOW_STATSD_PORT_9102_TCP_PROTO=tcp
AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVALS=True
AIRFLOW__WEBSERVER__SHOW_TRIGGER_FORM_IF_NO_PARAMS=True
AIRFLOW_API_SERVER_PORT_8080_TCP=tcp://172.20.254.149:8080
AIRFLOW_API_SERVER_PORT=tcp://172.20.254.149:8080
AIRFLOW_STATSD_SERVICE_PORT_STATSD_INGEST=9125
AIRFLOW_USER_HOME_DIR=/home/airflow
AIRFLOW__CORE__DAGS_FOLDER=/opt/airflow/dags
AIRFLOW_PGBOUNCER_PORT_6543_TCP_PORT=6543

AIRFLOW__CORE__PARALLELISM=40
AIRFLOW_PGBOUNCER_PORT_9127_TCP_PORT=9127
AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW_AUTH_ROLES_MAPPING={"app_airflow_prod_admin": [ "Admin" ], "app_airflow_prod_user": [ "User" ], "app_airflow_prod_viewer": [ "Viewer" ]}
AIRFLOW_API_SERVER_PORT_8080_TCP_ADDR=172.20.254.149
AIRFLOW_VERSION=3.1.3
AIRFLOW__CORE__LOAD_EXAMPLES=false
AIRFLOW_STATSD_PORT_9125_UDP_ADDR=172.20.167.106
AIRFLOW_URL_REDIRECT_URI=https://redirect-url/auth/oauth-authorized/okta
AIRFLOW_API_SERVER_SERVICE_PORT=8080
AIRFLOW_STATSD_PORT_9102_TCP=tcp://172.20.167.106:9102
AIRFLOW_PGBOUNCER_SERVICE_HOST=172.20.108.237
AIRFLOW_HOME=/opt/airflow
AIRFLOW_STATSD_SERVICE_PORT=9125
AIRFLOW_USE_UV=false
AIRFLOW_PGBOUNCER_PORT_6543_TCP_ADDR=172.20.108.237
AIRFLOW_PGBOUNCER_PORT_9127_TCP_ADDR=172.20.108.237
AIRFLOW_PIP_VERSION=25.3

AIRFLOW_STATSD_PORT_9102_TCP_ADDR=172.20.167.106
AIRFLOW_STATSD_PORT_9125_UDP_PROTO=udp
AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__CORE__PARALLELISM=40
AIRFLOW_PGBOUNCER_PORT=tcp://172.20.108.237:6543
AIRFLOW_PGBOUNCER_SERVICE_PORT_PGB_METRICS=9127
AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW_VAR_ENV_NAME=prod
AIRFLOW_UV_VERSION=0.9.9
AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__PYTHONASYNCIODEBUG=1
AIRFLOW_PGBOUNCER_SERVICE_PORT_PGBOUNCER=6543
AIRFLOW_STATSD_PORT_9125_UDP_PORT=9125
AIRFLOW_PGBOUNCER_PORT_6543_TCP_PROTO=tcp
AIRFLOW_PGBOUNCER_PORT_9127_TCP=tcp://172.20.108.237:9127

AIRFLOW_STATSD_PORT_9102_TCP_PORT=9102
AIRFLOW_STATSD_SERVICE_PORT_STATSD_SCRAPE=9102
AIRFLOW_VAR_ENV_NAME=prod
AIRFLOW_STATSD_SERVICE_HOST=172.20.167.106
AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW_URL_REDIRECT_URI=https://redirect-url/auth/oauth-authorized/okta

AIRFLOW_API_SERVER_SERVICE_HOST=172.20.254.149
AIRFLOW_IMAGE_TYPE=prod
AIRFLOW_PYTHON_VERSION=3.12.12
AIRFLOW_INSTALLATION_METHOD=apache-airflow
AIRFLOW_API_SERVER_SERVICE_PORT_AIRFLOW_UI=8080
AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVALS=True
AIRFLOW_API_SERVER_PORT_8080_TCP_PORT=8080
AIRFLOW_PGBOUNCER_PORT_6543_TCP=tcp://172.20.108.237:6543
AIRFLOW_PGBOUNCER_PORT_9127_TCP_PROTO=tcp
AIRFLOW__CORE__TEST_CONNECTION=Enabled
AIRFLOW_UID=50000
AIRFLOW_STATSD_PORT=udp://172.20.167.106:9125
AIRFLOW_PGBOUNCER_SERVICE_PORT=6543
AIRFLOW_STATSD_PORT_9125_UDP=udp://172.20.167.106:9125
AIRFLOW_AUTH_ROLES_MAPPING={"app_airflow_prod_admin": [ "Admin" ], "app_airflow_prod_user": [ "User" ], "app_airflow_prod_viewer": [ "Viewer" ]}

airflow.cfg

[api]
enable_proxy_fix = True
log_config = 

[celery]
flower_url_prefix = 
worker_concurrency = 16

[celery_kubernetes_executor]
kubernetes_queue = kubernetes

[core]
auth_manager = airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
colored_console_log = False
dags_folder = /opt/airflow/dags
default_task_retries = 2
execution_api_server_url = http://airflow-api-server:8080/execution/
executor = KubernetesExecutor
load_examples = False
min_serialized_dag_fetch_interval = 300
min_serialized_dag_update_interval = 300
remote_logging = True

[dag_processor]
min_file_process_interval = 300
parsing_processes = 1
print_stats_interval = 300
refresh_interval = 300
stale_bundle_cleanup_min_versions = 2

[elasticsearch]
json_format = True
log_id_template = {dag_id}_{task_id}_{execution_date}_{try_number}

[elasticsearch_configs]
max_retries = 3
retry_timeout = True
timeout = 30

[email]
default_email_on_failure = False
default_email_on_retry = False

[fab]
enable_proxy_fix = True

[kerberos]
ccache = /var/kerberos-ccache/cache
keytab = /etc/airflow.keytab
principal = airflow@FOO.COM
reinit_frequency = 3600

[kubernetes]
airflow_configmap = airflow-config
airflow_local_settings_configmap = airflow-config
multi_namespace_mode = False
namespace = airflow
pod_template_file = /opt/airflow/pod_templates/pod_template_file.yaml
worker_container_repository = docker-hub/analytics-airflow
worker_container_tag = 0.0.342

[kubernetes_executor]
airflow_configmap = airflow-config
airflow_local_settings_configmap = airflow-config
delete_worker_pods_on_failure = True
multi_namespace_mode = False
namespace = airflow
pod_template_file = /opt/airflow/pod_templates/pod_template_file.yaml
worker_container_repository = docker-hub/analytics-airflow
worker_container_tag = 0.0.342

[logging]
colored_console_log = False
logging_level = INFO
remote_log_conn_id = tu-s3-logs
remote_logging = True

[metrics]
statsd_host = airflow-statsd
statsd_on = True
statsd_port = 9125
statsd_prefix = airflow

[scheduler]
run_duration = 41460
standalone_dag_processor = True
statsd_host = airflow-statsd
statsd_on = True
statsd_port = 9125
statsd_prefix = airflow

Anything else?

If I missed something please point that I will update the docs
Could you please highlight areas that I could start investigation?

Maybe worth noting is that we run our airflow with AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVALS=True flag.

Thank you for your contribution I see a lot of good work has been put in Airflow 3!

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. aru-trackunit commented on Nov 21, 2025

    @aru-trackunit
    ContributorAuthor

    Edit:

    Backfill eventually picked up the job after waiting around an hour. Is it a normal behavior?
    Could you please suggest some solution that I could speed up processing of backfills?

    Also another unexpected behavior that after the job succeeded it's not marked as completed for 50 minutes already:
    Time of taking screenshot 2025-11-21 14:08

    Image
  2. tirkarthi commented on Nov 22, 2025

    @tirkarthi
    Contributor

    When the dagrun is running with task instance in None state can you please click on task instance's details tab to see if there are pending dependencies?

  3. aru-trackunit commented on Nov 22, 2025

    @aru-trackunit
    ContributorAuthor

    Hi tirkarthi thanks for a reply, have a look on the attached task instance's details:

    Image
  4. tirkarthi commented on Nov 23, 2025

    @tirkarthi
    Contributor

    My guess is that you might be hitting one of the below since by default scheduler examines 20 dagruns by default and with 30 concurrent tasks it might lead to this dagrun not picked up. There are tips to fine tune the scheduler and I was referring to max_dagruns_per_loop_to_schedule with 20 as default value. You can also enable debug log or run scheduler with -vvv that should give you more logs to see when the task and dagrun is being picked up for queuing.

    #49508
    #45636

    https://airflow.apache.org/docs/apache-airflow/2.5.1/administration-and-deployment/scheduler.html#how-to-approach-scheduler-s-fine-tuning

  5. LomotHo commented on Dec 2, 2025

    @LomotHo

    same issue , got "No status" task when use backfill

  6. aru-trackunit commented on Dec 2, 2025

    @aru-trackunit
    ContributorAuthor

    thank you @tirkarthi max_dagruns_per_loop_to_schedule solved our issue.

  7. zh-jn commented on Sep 4, 2026

    @zh-jn

    Adding a root-cause data point for future readers landing here. On Airflow 3.2.2 the same symptom (backfill dag_run state=running, all task instances state=NULL, last_scheduling_decision=NULL forever) reproduces deterministically whenever running
    non-backfill dag_runs ≥ max_dagruns_per_loop_to_schedule. Raising the config does mask it, but the underlying cause is a sort-direction bug in get_running_dag_runs_to_examine.

    Root cause — airflow-core/src/airflow/models/dagrun.py:619:

    .order_by(
      nulls_first(cast("ColumnElement[Any]", BackfillDagRun.sort_ordinal), session=session),   # ← reversed
      nulls_first(cast("ColumnElement[Any]", cls.last_scheduling_decision), session=session),
      cls.run_after,
    )

    sort_ordinal lives on the LEFT-JOINed backfill_dag_run row and is NULL for every non-backfill dag_run. nulls_first therefore sorts non-backfill runs before backfill runs, exactly the opposite of the field's apparent intent. With
    .limit(DEFAULT_DAGRUNS_TO_EXAMINE) (default 20) and any healthy multi-DAG scheduled workload, backfill dag_runs never fit into the window and are silently skipped — no log, no scheduled_by_job_id, no last_scheduling_decision.

    Reproducer (read-only, in the scheduler pod):

    # Production query — backfill dag_run absent
    from airflow.models.dagrun import DagRun
    from airflow.utils.session import create_session
    with create_session() as s:
      runs = list(DagRun.get_running_dag_runs_to_examine(session=s))
      print(len(runs), sum(1 for r in runs if r.backfill_id is not None))
    
    # Same query with nullslast(sort_ordinal) — backfill dag_runs jump to the top
    from sqlalchemy import select, false, func, nullslast
    from airflow.models.backfill import BackfillDagRun
    from airflow.models.dag import DagModel
    from airflow.utils.state import DagRunState
    with create_session() as s:
      q = (select(DagRun)
    	   .where(DagRun.state == DagRunState.RUNNING)
    	   .join(DagModel, DagModel.dag_id == DagRun.dag_id)
    	   .join(BackfillDagRun, BackfillDagRun.dag_run_id == DagRun.id, isouter=True)
    	   .where(DagModel.is_paused == false(), DagModel.is_stale == false())
    	   .order_by(nullslast(BackfillDagRun.sort_ordinal), DagRun.run_after)
    	   .where(DagRun.run_after <= func.now())
    	   .limit(DagRun.DEFAULT_DAGRUNS_TO_EXAMINE))
      runs = list(s.scalars(q).unique())
      print(len(runs), sum(1 for r in runs if r.backfill_id is not None))

    On the same session we saw production return 20 runs / 0 backfill, and nullslast return 16 runs / 2 backfill (both stuck ones, top of the list).

    One-line fix — swap nulls_first → nulls_last on the sort_ordinal column only (keep nulls_first on last_scheduling_decision, that one is intentional):

    - nulls_first(cast("ColumnElement[Any]", BackfillDagRun.sort_ordinal), session=session),
    + nulls_last (cast("ColumnElement[Any]", BackfillDagRun.sort_ordinal), session=session),

    Because this issue is closed I'll open a fresh one referencing here + #49508 + #69658 so the fix has a place to land. Wanted to leave the diagnosis in this thread since it's the top search hit for the symptom.

  8. seanmuth commented on Sep 8, 2026

    @seanmuth
    Contributor

    @zh-jn:

    sort_ordinal lives on the LEFT-JOINed backfill_dag_run row and is NULL for every non-backfill dag_run. nulls_first therefore sorts non-backfill runs before backfill runs, exactly the opposite of the field's apparent intent

    AFAIK sending the backfills to the bottom of the sorted list is intentional, designed so that a large backfill job, say 100 or 365 runs, doesn't starve currently-scheduled runs. The idea being that the backfills are less time sensitive than currently scheduled dags.

    Agreed that the default of just 20 runs per loop to schedule creates unscheduleable TI's in backfill runs on even moderately "busy" Airflow deployments, and should be looked at to get to a better UX here.

  9. seanmuth commented on Sep 8, 2026

    @seanmuth
    Contributor

    Actually @zh-jn just had a discussion with @dstandish and we do consider the sorting of running state backfill runs to the very bottom to be a bug, and not-quite intended behavior. The idea was definitely deprioritization, but not permanent starvation.

    Our proposed fix is to simply remove that nulls_last sort on the backfill. If we're already to the stage where the backfill DR is in Running we should run the TIs, using last_scheduling_decision as the batching/sort key.

    PR to follow shortly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:backfillSpecifically for backfill relatedarea:corekind:bugThis is a clearly a bugneeds-triagelabel for new issues that we didn't triage yet

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions