pankajkoti commented on code in PR #31071:
URL: https://github.com/apache/airflow/pull/31071#discussion_r1185458643
##########
tests/system/providers/amazon/aws/example_emr.py:
##########
@@ -164,10 +164,14 @@ def get_step_id(step_ids: list):
task_id="add_steps",
job_flow_id=create_job_flow.output,
steps=SPARK_STEPS,
- wait_for_completion=True,
execution_role_arn=execution_role_arn,
)
# [END howto_operator_emr_add_steps]
+ add_steps.wait_for_completion = True
+ # On rare occasion (1 in 50ish?) this system test times out. Extending
+ # the delay and max_attempts to attempt to mitigate the flaky test.
+ add_steps.waiter_delay = 60
+ add_steps.waiter_max_attempts = 90
Review Comment:
What if we just set `waiter_max_attempts` to the desired value? I mean, if
the current `waiter_delay` is `30` seconds, and we're desiring` 90 attempts*60
seconds`, then we could achieve the same with `180 attempts * 30 (current
waiter_delay)`
Like that, we do not have to wait for (60 - current waiter_delay) seconds
for the last attempt. I feel it's a small optimisation, but could still save us
some seconds waiting for the CI run to finish :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
