Repository navigation
Description
Apache Airflow version
3.1.3
If "Other Airflow 2/3 version" selected, which one?
No response
What happened?
Backfill task is stuck in "No status" state.
There is around 100 DAGs running on different schedule. Usually around 30 tasks constantly running.
When triggering a backfill for 2025-11-17, I successfully pass the form step, the dag run is successfully created and is in running status, however the task itself does not start, and is state in database is null.
I restarted all the airflow kubernetes pods, removed a dag data using delete dag and readded to airflow and after the restart task still does not progress.
I have checked parallelism which is set to 40 and max tasks runnning were around 30, there are 93 open slots in the pool.
I have waited for 30 minutes for this task to be picked up but nothing happened.
dag record:
| dag_id | is_paused | is_stale | last_parsed_time | last_parse_duration | last_expired | fileloc | relative_fileloc | bundle_name | bundle_version | owners | dag_display_name | description | timetable_summary | timetable_description | asset_expression | deadline | max_active_tasks | max_active_runs | max_consecutive_failed_dag_runs | has_task_concurrency_limits | has_import_errors | next_dagrun | next_dagrun_data_interval_start | next_dagrun_data_interval_end | next_dagrun_create_after |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ble-monitoring | FALSE | FALSE | 2025-11-21 10:45:56.027054+00 | 0.14697423899997375 | /opt/airflow/dags/dashboards/dag_ble_monitoring.py | dashboards/dag_ble_monitoring.py | dags-folder | team_iot_data | BLE Monitoring Dashboard on PowerBi | 30 0 * * * | At 00:30 | null | null | 16 | 1 | 0 | FALSE | FALSE | 2025-11-21 00:30:00+00 | 2025-11-21 00:30:00+00 | 2025-11-22 00:30:00+00 | 2025-11-22 00:30:00+00 |
dag_run record:
| id | dag_id | queued_at | logical_date | start_date | end_date | state | run_id | creating_job_id | run_type | triggered_by | triggering_user_name | conf | data_interval_start | data_interval_end | run_after | last_scheduling_decision | log_template_id | updated_at | clear_number | backfill_id | bundle_version | scheduled_by_job_id | context_carrier | span_status | created_dag_version_id |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 8980 | ble-monitoring | 2025-11-21 10:31:50.877612+00 | 2025-11-11 00:30:00+00 | 2025-11-21 10:31:51.01211+00 | NULL | running | backfill__2025-11-12T00:30:00+00:00 | null | backfill | BACKFILL | okta_00ud5ffefmodlaFL2357 | {} | 2025-11-11 00:30:00+00 | 2025-11-12 00:30:00+00 | 2025-11-12 00:30:00+00 | null | 1 | 2025-11-21 10:31:51.012965+00 | 0 | 9 | null | null | {"__var": {}, "__type": "dict"} | not_started | 019aa5d3-ae8f-7d68-9ef4-b97916d1470b |
task_run record:
| task_id | state | pool | queue | start_date | end_date | try_number |
|---|---|---|---|---|---|---|
| ble_monitoring | NULL | default_pool | default | NULL | NULL | 0 |
backfill record:
| id | dag_id | from_date | to_date | dag_run_conf | is_paused | reprocess_behavior | max_active_runs | created_at | completed_at | updated_at | triggering_user_name |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 9 | ble-monitoring | 2025-11-11 00:30:00+00 | 2025-11-11 00:30:00+00 | {} | FALSE | completed | 1 | 2025-11-21 10:31:50.861649+00 | NULL | 2025-11-21 10:31:50.861653+00 | okta_00ud5ffefmodlaFL2357 |
backfill_dag_run record:
| id | backfill_id | dag_run_id | exception_reason | logical_date | sort_ordinal |
|---|---|---|---|---|---|
| 9 | 9 | 8980 | NULL | 2025-11-11 00:30:00+00 | 1 |
I wonder if I am the first one to see this - Do you think might be the case that database state is corrupted?
My god feeling is that scheduler is not catching up properly, do you know what metrics should observe?
What you think should happen instead?
I would expect backfill task runs to have executed and completed or failed, within 5-10 minutes.
How to reproduce
- Click on Backfill button in a Dag
- Pick any date that fits the and Run Backfill
Operating System
Debian GNU/Linux 12 (bookworm)
Versions of Apache Airflow Providers
apache-airflow-providers-amazon 9.17.0
apache-airflow-providers-cncf-kubernetes 10.10.0
apache-airflow-providers-common-compat 1.8.0
apache-airflow-providers-common-io 1.6.4
apache-airflow-providers-common-sql 1.29.0
apache-airflow-providers-databricks 7.7.5
apache-airflow-providers-fab 3.0.2
apache-airflow-providers-github 2.9.4
apache-airflow-providers-hashicorp 4.3.4
apache-airflow-providers-http 5.5.0
apache-airflow-providers-microsoft-mssql 4.3.3
apache-airflow-providers-mysql 6.3.5
apache-airflow-providers-postgres 6.4.1
apache-airflow-providers-sftp 5.4.2
apache-airflow-providers-slack 9.4.0
apache-airflow-providers-smtp 2.3.1
apache-airflow-providers-ssh 4.1.6
apache-airflow-providers-standard 1.9.1
Deployment
Official Apache Airflow Helm Chart
Deployment details
Example dag:
import datetime as dt
from collections import defaultdict
from airflow_addons.operators.databricks import DatabricksSubmitRunOperator # our internal wrapper of DatabricksSubmitRunOperator
from airflow.sdk import DAG, Variable
DAG_ID = "ble-monitoring"
default_args = {
"owner": "team_iot_data",
"depends_on_past": False,
"start_date": dt.datetime(2025, 11, 1),
}
ble_monitoring_task = {
"notebook_path": "ble_monitoring/ble_monitoring_data_refresh",
"base_parameters": {
"COMPUTE_FOR_DATE": "{{ data_interval_start.strftime('%Y-%m-%d') }}"
}
}
with DAG(
dag_id=DAG_ID,
description="BLE Monitoring Dashboard on PowerBi",
default_args=default_args,
schedule="30 0 * * *",
catchup=False,
max_active_runs=1,
tags={"dashboards"},
doc_md=__doc__,
) as dag:
DatabricksSubmitRunOperator(
task_id="ble_monitoring",
new_cluster={},
notebook_task=ble_monitoring_task,
timeout_seconds=3600, # 1 hours
polling_period_seconds=30,
retries=0,
databricks_retry_limit=10,
databricks_retry_delay=10,
do_xcom_push=True,
git_url="https://github.com/repo/",
)
env vars:
AIRFLOW_API_SERVER_PORT_8080_TCP_PROTO=tcp
AIRFLOW_STATSD_PORT_9102_TCP_PROTO=tcp
AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVALS=True
AIRFLOW__WEBSERVER__SHOW_TRIGGER_FORM_IF_NO_PARAMS=True
AIRFLOW_API_SERVER_PORT_8080_TCP=tcp://172.20.254.149:8080
AIRFLOW_API_SERVER_PORT=tcp://172.20.254.149:8080
AIRFLOW_STATSD_SERVICE_PORT_STATSD_INGEST=9125
AIRFLOW_USER_HOME_DIR=/home/airflow
AIRFLOW__CORE__DAGS_FOLDER=/opt/airflow/dags
AIRFLOW_PGBOUNCER_PORT_6543_TCP_PORT=6543
AIRFLOW__CORE__PARALLELISM=40
AIRFLOW_PGBOUNCER_PORT_9127_TCP_PORT=9127
AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW_AUTH_ROLES_MAPPING={"app_airflow_prod_admin": [ "Admin" ], "app_airflow_prod_user": [ "User" ], "app_airflow_prod_viewer": [ "Viewer" ]}
AIRFLOW_API_SERVER_PORT_8080_TCP_ADDR=172.20.254.149
AIRFLOW_VERSION=3.1.3
AIRFLOW__CORE__LOAD_EXAMPLES=false
AIRFLOW_STATSD_PORT_9125_UDP_ADDR=172.20.167.106
AIRFLOW_URL_REDIRECT_URI=https://redirect-url/auth/oauth-authorized/okta
AIRFLOW_API_SERVER_SERVICE_PORT=8080
AIRFLOW_STATSD_PORT_9102_TCP=tcp://172.20.167.106:9102
AIRFLOW_PGBOUNCER_SERVICE_HOST=172.20.108.237
AIRFLOW_HOME=/opt/airflow
AIRFLOW_STATSD_SERVICE_PORT=9125
AIRFLOW_USE_UV=false
AIRFLOW_PGBOUNCER_PORT_6543_TCP_ADDR=172.20.108.237
AIRFLOW_PGBOUNCER_PORT_9127_TCP_ADDR=172.20.108.237
AIRFLOW_PIP_VERSION=25.3
AIRFLOW_STATSD_PORT_9102_TCP_ADDR=172.20.167.106
AIRFLOW_STATSD_PORT_9125_UDP_PROTO=udp
AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__CORE__PARALLELISM=40
AIRFLOW_PGBOUNCER_PORT=tcp://172.20.108.237:6543
AIRFLOW_PGBOUNCER_SERVICE_PORT_PGB_METRICS=9127
AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW_VAR_ENV_NAME=prod
AIRFLOW_UV_VERSION=0.9.9
AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__PYTHONASYNCIODEBUG=1
AIRFLOW_PGBOUNCER_SERVICE_PORT_PGBOUNCER=6543
AIRFLOW_STATSD_PORT_9125_UDP_PORT=9125
AIRFLOW_PGBOUNCER_PORT_6543_TCP_PROTO=tcp
AIRFLOW_PGBOUNCER_PORT_9127_TCP=tcp://172.20.108.237:9127
AIRFLOW_STATSD_PORT_9102_TCP_PORT=9102
AIRFLOW_STATSD_SERVICE_PORT_STATSD_SCRAPE=9102
AIRFLOW_VAR_ENV_NAME=prod
AIRFLOW_STATSD_SERVICE_HOST=172.20.167.106
AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW_URL_REDIRECT_URI=https://redirect-url/auth/oauth-authorized/okta
AIRFLOW_API_SERVER_SERVICE_HOST=172.20.254.149
AIRFLOW_IMAGE_TYPE=prod
AIRFLOW_PYTHON_VERSION=3.12.12
AIRFLOW_INSTALLATION_METHOD=apache-airflow
AIRFLOW_API_SERVER_SERVICE_PORT_AIRFLOW_UI=8080
AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVALS=True
AIRFLOW_API_SERVER_PORT_8080_TCP_PORT=8080
AIRFLOW_PGBOUNCER_PORT_6543_TCP=tcp://172.20.108.237:6543
AIRFLOW_PGBOUNCER_PORT_9127_TCP_PROTO=tcp
AIRFLOW__CORE__TEST_CONNECTION=Enabled
AIRFLOW_UID=50000
AIRFLOW_STATSD_PORT=udp://172.20.167.106:9125
AIRFLOW_PGBOUNCER_SERVICE_PORT=6543
AIRFLOW_STATSD_PORT_9125_UDP=udp://172.20.167.106:9125
AIRFLOW_AUTH_ROLES_MAPPING={"app_airflow_prod_admin": [ "Admin" ], "app_airflow_prod_user": [ "User" ], "app_airflow_prod_viewer": [ "Viewer" ]}
airflow.cfg
[api]
enable_proxy_fix = True
log_config =
[celery]
flower_url_prefix =
worker_concurrency = 16
[celery_kubernetes_executor]
kubernetes_queue = kubernetes
[core]
auth_manager = airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
colored_console_log = False
dags_folder = /opt/airflow/dags
default_task_retries = 2
execution_api_server_url = http://airflow-api-server:8080/execution/
executor = KubernetesExecutor
load_examples = False
min_serialized_dag_fetch_interval = 300
min_serialized_dag_update_interval = 300
remote_logging = True
[dag_processor]
min_file_process_interval = 300
parsing_processes = 1
print_stats_interval = 300
refresh_interval = 300
stale_bundle_cleanup_min_versions = 2
[elasticsearch]
json_format = True
log_id_template = {dag_id}_{task_id}_{execution_date}_{try_number}
[elasticsearch_configs]
max_retries = 3
retry_timeout = True
timeout = 30
[email]
default_email_on_failure = False
default_email_on_retry = False
[fab]
enable_proxy_fix = True
[kerberos]
ccache = /var/kerberos-ccache/cache
keytab = /etc/airflow.keytab
principal = airflow@FOO.COM
reinit_frequency = 3600
[kubernetes]
airflow_configmap = airflow-config
airflow_local_settings_configmap = airflow-config
multi_namespace_mode = False
namespace = airflow
pod_template_file = /opt/airflow/pod_templates/pod_template_file.yaml
worker_container_repository = docker-hub/analytics-airflow
worker_container_tag = 0.0.342
[kubernetes_executor]
airflow_configmap = airflow-config
airflow_local_settings_configmap = airflow-config
delete_worker_pods_on_failure = True
multi_namespace_mode = False
namespace = airflow
pod_template_file = /opt/airflow/pod_templates/pod_template_file.yaml
worker_container_repository = docker-hub/analytics-airflow
worker_container_tag = 0.0.342
[logging]
colored_console_log = False
logging_level = INFO
remote_log_conn_id = tu-s3-logs
remote_logging = True
[metrics]
statsd_host = airflow-statsd
statsd_on = True
statsd_port = 9125
statsd_prefix = airflow
[scheduler]
run_duration = 41460
standalone_dag_processor = True
statsd_host = airflow-statsd
statsd_on = True
statsd_port = 9125
statsd_prefix = airflow
Anything else?
If I missed something please point that I will update the docs
Could you please highlight areas that I could start investigation?
Maybe worth noting is that we run our airflow with AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVALS=True flag.
Thank you for your contribution I see a lot of good work has been put in Airflow 3!
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
aru-trackunit commentedon Nov 21, 2025
Edit:
Backfill eventually picked up the job after waiting around an hour. Is it a normal behavior?
Could you please suggest some solution that I could speed up processing of backfills?
Also another unexpected behavior that after the job succeeded it's not marked as completed for 50 minutes already:
Time of taking screenshot
2025-11-21 14:08tirkarthi commentedon Nov 22, 2025
When the dagrun is running with task instance in None state can you please click on task instance's details tab to see if there are pending dependencies?
aru-trackunit commentedon Nov 22, 2025
Hi tirkarthi thanks for a reply, have a look on the attached task instance's details:
tirkarthi commentedon Nov 23, 2025
My guess is that you might be hitting one of the below since by default scheduler examines 20 dagruns by default and with 30 concurrent tasks it might lead to this dagrun not picked up. There are tips to fine tune the scheduler and I was referring to max_dagruns_per_loop_to_schedule with 20 as default value. You can also enable debug log or run scheduler with
-vvvthat should give you more logs to see when the task and dagrun is being picked up for queuing.#49508
#45636
https://airflow.apache.org/docs/apache-airflow/2.5.1/administration-and-deployment/scheduler.html#how-to-approach-scheduler-s-fine-tuning
LomotHo commentedon Dec 2, 2025
same issue , got "No status" task when use backfill
aru-trackunit commentedon Dec 2, 2025
thank you @tirkarthi
max_dagruns_per_loop_to_schedulesolved our issue.zh-jn commentedon Sep 4, 2026
Adding a root-cause data point for future readers landing here. On Airflow 3.2.2 the same symptom (backfill dag_run
state=running, all task instancesstate=NULL,last_scheduling_decision=NULLforever) reproduces deterministically whenever runningnon-backfill dag_runs ≥
max_dagruns_per_loop_to_schedule. Raising the config does mask it, but the underlying cause is a sort-direction bug inget_running_dag_runs_to_examine.Root cause —
airflow-core/src/airflow/models/dagrun.py:619:sort_ordinallives on the LEFT-JOINedbackfill_dag_runrow and isNULLfor every non-backfill dag_run.nulls_firsttherefore sorts non-backfill runs before backfill runs, exactly the opposite of the field's apparent intent. With.limit(DEFAULT_DAGRUNS_TO_EXAMINE)(default 20) and any healthy multi-DAG scheduled workload, backfill dag_runs never fit into the window and are silently skipped — no log, noscheduled_by_job_id, nolast_scheduling_decision.Reproducer (read-only, in the scheduler pod):
On the same session we saw production return 20 runs / 0 backfill, and
nullslastreturn 16 runs / 2 backfill (both stuck ones, top of the list).One-line fix — swap
nulls_first→nulls_laston thesort_ordinalcolumn only (keepnulls_firstonlast_scheduling_decision, that one is intentional):Because this issue is closed I'll open a fresh one referencing here + #49508 + #69658 so the fix has a place to land. Wanted to leave the diagnosis in this thread since it's the top search hit for the symptom.
seanmuth commentedon Sep 8, 2026
@zh-jn:
AFAIK sending the backfills to the bottom of the sorted list is intentional, designed so that a large backfill job, say 100 or 365 runs, doesn't starve currently-scheduled runs. The idea being that the backfills are less time sensitive than currently scheduled dags.
Agreed that the default of just 20 runs per loop to schedule creates unscheduleable TI's in backfill runs on even moderately "busy" Airflow deployments, and should be looked at to get to a better UX here.
seanmuth commentedon Sep 8, 2026
Actually @zh-jn just had a discussion with @dstandish and we do consider the sorting of running state backfill runs to the very bottom to be a bug, and not-quite intended behavior. The idea was definitely deprioritization, but not permanent starvation.
Our proposed fix is to simply remove that nulls_last sort on the backfill. If we're already to the stage where the backfill DR is in
Runningwe should run the TIs, using last_scheduling_decision as the batching/sort key.PR to follow shortly