dominikhei commented on PR #51692: URL: https://github.com/apache/airflow/pull/51692#issuecomment-3061220517
> I see. So just to confirm: the core of the issue here is that, while `EcsRegisterTaskDefinitionOperator` registers a task definition, `EcsRunTaskOperator` actually spins up an instance based on that task definition, but it doesn't necessarily wait for its associated containers to be in a terminal state before moving on, which can cause an empty list of containers to be seen. The `RunTaskOperator` executes a task definition, yes. Our problem is: the operator does not require a container name to be specified. However when you want to ship logs from cloud watch to airflow and set `awslogs_group` and `awslogs_stream_prefix` a container name is required, as the prefix always contains a container name. The current solution was to simply get the container name from the return of `ECS.Client.run_task(**kwargs)`. In most cases this works perfectly fine. But in some edge cases, as the ECS endpoints are eventually consistent and the task is in an intermediary state, the container name might not be part of the response. If `awslogs_group` and `awslogs_stream_prefix` are not set, getting the container name is now skipped as it is not required. If they are set and no container name is specified we use `ECS.Client.describe_task(**kwargs)` to get the container name, with some delay to account for the intermediary states. The issue is related to the change of the cloud watch log stream prefixes introduced a while ago. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
