devgonvarun commented on issue #53926:
URL: https://github.com/apache/airflow/issues/53926#issuecomment-3183307665
@Owen-CH-Leung thanks for sharing your screenshot. I investigated into this
and found the following:
1. The reason why we see different elasticsearch_id values in our postgres
is because i am running airflow in kubernetes using the latest helm chart
v1.18.0 and i observed that in this helm chart the default value of the
elasticsearch log_id_template config is set to
`{dag_id}_{task_id}_{execution_date}_{try_number}` instead of
`{dag_id}-{task_id}-{run_id}-{map_index}-{try_number}`. Because of this my the
elasticsearch_id was set to `{dag_id}_{task_id}_{execution_date}_{try_number}`
in postgres.
2. Opensearch task handler uses this elasticsearch_id thereby ignoring its
own log_id_template value.
3. Once I override the log_id_template of the elasticsearch config to
`{dag_id}-{task_id}-{run_id}-{map_index}-{try_number}` in my values.yaml then
opensearch's log_id works correctly.
So to summarise the problem is that opensearch's task handler ignores its
own log_id_template config and uses the log_id_template of elasticsearch
config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
