[
https://issues.apache.org/jira/browse/AIRFLOW-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aizhamal Nurmamat kyzy updated AIRFLOW-3879:
--------------------------------------------
Labels: kubernetes (was: )
Component/s: (was: kubernetes)
operators
Moving to operators component, and labeling with kubernetes as part of the
component refactor.
> When a pod created by KubernetesPodOperator is evicted the task fails
> ---------------------------------------------------------------------
>
> Key: AIRFLOW-3879
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3879
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
> Affects Versions: 1.10.0
> Reporter: Jane
> Priority: Major
> Labels: kubernetes
>
> When a pod is evicted it's status does not match the expected status, causing
> the task to error due to a Type error.
>
> At a minimum the status of the pod should be reported in the error.
> {code:java}
> [2019-02-07 18:56:19,058] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique [2019-02-07 18:56:19,055] {models.py:1736} ERROR -
> 'NoneType' object is not iterable
> [2019-02-07 18:56:19,058] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique Traceback (most recent call last):
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File "/usr/local/lib/airflow/airflow/models.py",
> line 1633, in _run_raw_task
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique result = task_copy.execute(context=context)
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File
> "/home/airflow/gcs/dags/workflow/operators/airflow_operators.py", line 112,
> in execute
> [2019-02-07 18:56:19,059] {models.py:1764} INFO - Marking task as FAILED.
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique return super().execute(context)
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File
> "/usr/local/lib/airflow/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 115, in execute
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique get_logs=self.get_logs)
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File
> "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line 81,
> in run_pod
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique return self._monitor_pod(pod, get_logs)
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File
> "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line 98,
> in _monitor_pod
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique while self.base_container_is_running(pod):
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File
> "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line
> 127, in base_container_is_running
> [2019-02-07 18:56:19,060] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique event.status.container_statuses)), None)
> [2019-02-07 18:56:19,060] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique TypeError: 'NoneType' object is not iterable
> [2019-02-07 18:56:19,060] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique [2019-02-07 18:56:19,059] {models.py:1764} INFO -
> Marking task as FAILED.
> [2019-02-07 18:56:20,512] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique Traceback (most recent call last):
> [2019-02-07 18:56:20,512] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File "/usr/local/bin/airflow", line 6, in <module>
> [2019-02-07 18:56:20,512] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique exec(compile(open(__file__).read(), __file__,
> 'exec'))
> [2019-02-07 18:56:20,512] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File "/usr/local/lib/airflow/airflow/bin/airflow",
> line 32, in <module>
> [2019-02-07 18:56:20,512] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique args.func(args)
> [2019-02-07 18:56:20,512] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File "/usr/local/lib/airflow/airflow/utils/cli.py",
> line 74, in wrapper
> [2019-02-07 18:56:20,512] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique return f(*args, **kwargs)
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File "/usr/local/lib/airflow/airflow/bin/cli.py",
> line 496, in run
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique _run(args, dag, ti)
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File "/usr/local/lib/airflow/airflow/bin/cli.py",
> line 402, in _run
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique pool=args.pool,
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File "/usr/local/lib/airflow/airflow/utils/db.py",
> line 74, in wrapper
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique return func(*args, **kwargs)
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File "/usr/local/lib/airflow/airflow/models.py",
> line 1633, in _run_raw_task
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique result = task_copy.execute(context=context)
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File
> "/home/airflow/gcs/dags/workflow/operators/airflow_operators.py", line 112,
> in execute
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique return super().execute(context)
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File
> "/usr/local/lib/airflow/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 115, in execute
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique get_logs=self.get_logs)
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File
> "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line 81,
> in run_pod
> [2019-02-07 18:56:20,514] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique return self._monitor_pod(pod, get_logs)
> [2019-02-07 18:56:20,514] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File
> "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line 98,
> in _monitor_pod
> [2019-02-07 18:56:20,514] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique while self.base_container_is_running(pod):
> [2019-02-07 18:56:20,514] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique File
> "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line
> 127, in base_container_is_running
> [2019-02-07 18:56:20,514] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique event.status.container_statuses)), None)
> [2019-02-07 18:56:20,514] {base_task_runner.py:107} INFO - Job 10690: Subtask
> data-processing-unique TypeError: 'NoneType' object is not iterable
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
