[
https://issues.apache.org/jira/browse/HADOOP-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125413#comment-17125413
]
Dyno commented on HADOOP-17063:
-------------------------------
log in the spark executor,
Jun 3, 2020 @ 22:57:23.032
2020-06-03 22:57:23,032 INFO impl.MetricsSystemImpl: s3a-file-system metrics
system shutdown complete.
Jun 3, 2020 @ 22:57:23.0322020-06-03 22:57:23,032 INFO impl.MetricsSystemImpl:
Stopping s3a-file-system metrics system...105
Jun 3, 2020 @ 22:57:23.0322020-06-03 22:57:23,032 INFO impl.MetricsSystemImpl:
s3a-file-system metrics system stopped.
Jun 3, 2020 @ 22:57:22.9732020-06-03 22:57:22,973 INFO
util.ShutdownHookManager: Deleting directory
/var/data/spark-ff208630-1fc7-48bc-93a1-6bdf94921c64/spark-eb4613f4-a41a-4985-845b-34b58ae95c50
Jun 3, 2020 @ 22:57:22.9722020-06-03 22:57:22,972 INFO
util.ShutdownHookManager: Shutdown hook called
Jun 3, 2020 @ 22:57:22.964
2020-06-03 22:57:22,964 INFO storage.DiskBlockManager: Shutdown hook called
Jun 3, 2020 @ 22:57:22.960
2020-06-03 22:57:22,960 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED
SIGNAL TERM <-- kill to make it move forward
Jun 3, 2020 @ 21:52:40.5502020-06-03 21:52:40,549 INFO executor.Executor:
Finished task 2927.0 in stage 12.0 (TID 31771). 4696 bytes result sent to driver
Jun 3, 2020 @ 21:52:40.5412020-06-03 21:52:40,541 INFO
output.FileOutputCommitter: Saved output of task
'attempt_20200603213232_0012_m_002927_31771' to s3a://com....
Jun 3, 2020 @ 21:52:40.5412020-06-03 21:52:40,541 INFO
mapred.SparkHadoopMapRedUtil: attempt_20200603213232_0012_m_002927_31771:
Committed
Jun 3, 2020 @ 21:52:34.9722020-06-03 21:52:34,971 INFO executor.Executor:
Finished task 2922.0 in stage 12.0 (TID 31766). 4696 bytes result sent to driver
Jun 3, 2020 @ 21:52:34.9632020-06-03 21:52:34,962 INFO
output.FileOutputCommitter: Saved out ...
> S3ABlockOutputStream.putObject looks stuck and never timeout
> ------------------------------------------------------------
>
> Key: HADOOP-17063
> URL: https://issues.apache.org/jira/browse/HADOOP-17063
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 3.2.1
> Environment: hadoop 3.2.1
> spark 2.4.4
>
> Reporter: Dyno
> Priority: Major
>
> {code}
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:523)
> com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82)
>
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.putObject(S3ABlockOutputStream.java:446)
>
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:365)
>
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> org.apache.parquet.hadoop.util.HadoopPositionOutputStream.close(HadoopPositionOutputStream.java:64)
> org.apache.parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:685)
> org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:122)
>
> org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:165)
>
> org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:42)
>
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:57)
>
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:74)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
>
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
> org.apache.spark.scheduler.Task.run(Task.scala:123)
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {code}
>
> we are using spark 2.4.4 with hadoop 3.2.1 on kubernetes/spark-operator,
> sometimes we see this hang with the stacktrace above. it looks like the
> putObject never return, we have to kill the executor to make the job move
> forward.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
