Tianqi Wan created SPARK-50595:
----------------------------------
Summary: Temp file storing state doesn't renamed to delta file,
but was read by next task
Key: SPARK-50595
URL: https://issues.apache.org/jira/browse/SPARK-50595
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 3.2.0
Environment: spark on yarn
Reporter: Tianqi Wan
Structured streaming job failed due to delta file does not exist
!image-2024-12-17-10-35-18-053.png!
we checked the access log for the missing file, and found that the file was
never created. And we found that the correspond tmp file exists whose file name
is
[.506.delta.936315d9-b58b-4f18-b3ed-b413cd92646f.TID3368893.tmp|https://www.cosmos09.osdinfra.net/cosmos/searchDM/raw/AnaheimRawLogs/Anaheim/RealTimeCheckpoints/Execution0/20241215_application_1733980441401_12313/state/0/1721/.506.delta.936315d9-b58b-4f18-b3ed-b413cd92646f.TID3368893.tmp?property=info]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
