manipatnam opened a new issue, #55007:
URL: https://github.com/apache/airflow/issues/55007
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.12.0
### Apache Airflow version
Airflow 2.10.2
### Operating System
debian
### Deployment
Astronomer
### Deployment details
_No response_
### What happened
Created a simple sleep glue job and triggered using the GlueJob Operator in
deferrable mode and it is failing when deferrable=True
error:
```
[2025-08-28, 02:54:18 UTC] {taskinstance.py:288} INFO - Pausing task as
DEFERRED. dag_id=trigger_glue_job_dag, task_id=trigger_glue_job_deferred,
run_id=manual__2025-08-28T02:52:33.959631+00:00,
execution_date=20250828T025233, start_date=20250828T025416
[2025-08-28, 02:54:18 UTC] {taskinstance.py:340} ▼ Post task execution logs
[2025-08-28, 02:54:18 UTC] {local_task_job_runner.py:260} INFO - Task exited
with return code 100 (task deferral)
[2025-08-28, 02:54:18 UTC] {local_task_job_runner.py:245} ▲▲▲ Log group end
[2025-08-28, 02:55:56 UTC] {base.py:84} INFO - Retrieving connection
'aws_default'
[2025-08-28, 02:55:56 UTC] {connection_wrapper.py:337} INFO - AWS Connection
(conn_id='aws_default', conn_type='aws') credentials retrieved from extra.
[2025-08-28, 02:55:56 UTC] {connection_wrapper.py:344} INFO - AWS Connection
(conn_id='aws_default', conn_type='aws') session token retrieved from extra,
please note you are responsible for renewing these.
[2025-08-28, 02:55:58 UTC] {local_task_job_runner.py:123} ▶ Pre task
execution logs
[2025-08-28, 02:55:58 UTC] {baseoperator.py:1787} ERROR - Trigger failed:
Traceback (most recent call last):
File
"/usr/local/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py",
line 558, in cleanup_finished_triggers
result = details["task"].result()
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py",
line 630, in run_trigger
async for event in trigger.run():
File
"/usr/local/lib/python3.12/site-packages/airflow/providers/amazon/aws/triggers/base.py",
line 145, in run
async with await hook.get_async_conn() as client:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiobotocore/session.py",
line 25, in __aenter__
self._client = await self._coro
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiobotocore/session.py",
line 215, in _create_client
client = await client_creator.create_client(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiobotocore/client.py", line
82, in create_client
client_args = self._get_client_args(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiobotocore/client.py", line
291, in _get_client_args
return args_creator.get_client_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiobotocore/args.py", line
99, in get_client_args
ruleset_resolver = self._build_endpoint_resolver(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiobotocore/args.py", line
165, in _build_endpoint_resolver
resolver_builtins = self.compute_endpoint_resolver_builtin_defaults(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: ClientArgsCreator.compute_endpoint_resolver_builtin_defaults()
missing 2 required positional arguments: 'credentials' and
'account_id_endpoint_mode'
[2025-08-28, 02:55:58 UTC] {taskinstance.py:3310} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/usr/local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line
767, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line
733, in _execute_callable
return ExecutionCallableRunner(
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/airflow/utils/operator_helpers.py",
line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line
1788, in resume_execution
raise TaskDeferralError(next_kwargs.get("error", "Unknown"))
airflow.exceptions.TaskDeferralError: Trigger failure
```
### What you think should happen instead
It should be able to successfully fetch the status of the job even when
deferrable=True
### How to reproduce
Create this simple dag
```from airflow import DAG
from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
from datetime import datetime
from airflow.utils.trigger_rule import TriggerRule
default_args = {
'owner': 'airflow',
'retries': 0,
}
with DAG(
dag_id='trigger_glue_job_dag',
default_args=default_args,
schedule_interval=None, # or use a cron expression
start_date=datetime(2023, 1, 1),
catchup=False,
tags=['aws', 'glue'],
) as dag:
trigger_glue_job = GlueJobOperator(
task_id='trigger_glue_job',
job_name='sleep', # The name of your Glue job
region_name='ap-southeast-2', # Update as needed
# iam_role_name='your-glue-service-role', # Optional if configured
in Glue
job_desc='Triggered from Airflow DAG',
trigger_rule=TriggerRule.ALL_DONE,
wait_for_completion=True,
# verbose=True,
# deferrable= True
)
trigger_glue_job_verbose = GlueJobOperator(
task_id='trigger_my_glue_job_verbose',
job_name='sleep', # The name of your Glue job
region_name='ap-southeast-2', # Update as needed
# iam_role_name='your-glue-service-role', # Optional if configured
in Glue
job_desc='Triggered from Airflow DAG',
trigger_rule=TriggerRule.ALL_DONE,
wait_for_completion=True,
verbose=True
# deferrable= True
)
trigger_glue_job_deferred = GlueJobOperator(
task_id='trigger_glue_job_deferred',
job_name='sleep', # The name of your Glue job
region_name='ap-southeast-2', # Update as needed
# iam_role_name='your-glue-service-role', # Optional if configured
in Glue
job_desc='Triggered from Airflow DAG',
trigger_rule=TriggerRule.ALL_DONE,
deferrable= True
)
trigger_glue_job_deferred_verbose = GlueJobOperator(
task_id='trigger_glue_job_deferred_verbose',
job_name='sleep', # The name of your Glue job
region_name='ap-southeast-2', # Update as needed
# iam_role_name='your-glue-service-role', # Optional if configured
in Glue
job_desc='Triggered from Airflow DAG',
trigger_rule=TriggerRule.ALL_DONE,
verbose=True,
deferrable= True
)
trigger_glue_job >> trigger_glue_job_verbose >>
trigger_glue_job_deferred >> trigger_glue_job_deferred_verbose```
### Anything else
its works fine when wait_for_completion= True and verbose=true but fails
when deferrable = True
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
