vladislavkeda3 opened a new issue, #12654:
URL: https://github.com/apache/iceberg/issues/12654
### Apache Iceberg version
1.7.1
### Query engine
Flink
### Please describe the bug 🐞
2025-03-26 14:56:16,346 WARN org.apache.iceberg.util.Tasks
[] - Retrying task after failure: Failed to delete file:
s3a://cids-data/eq/bdvpf_cdc/metadata/153039-e3dd4fbe-6180-43e1-b1ae-c4a4e5a5a6de.metadata.json
org.apache.iceberg.exceptions.RuntimeIOException: Failed to delete file:
s3a://cids-data/eq/bdvpf_cdc/metadata/153039-e3dd4fbe-6180-43e1-b1ae-c4a4e5a5a6de.metadata.json
at
org.apache.iceberg.hadoop.HadoopFileIO.deleteFile(HadoopFileIO.java:107)
~[job-parser-with-dependencies.jar:?]
at
org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
[job-parser-with-dependencies.jar:?]
at org.apache.iceberg.util.Tasks$Builder$1.run(Tasks.java:315)
[job-parser-with-dependencies.jar:?]
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
[?:?]
at java.base/java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) [?:?]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
s3a://cids-data/eq/bdvpf_cdc/metadata/153039-e3dd4fbe-6180-43e1-b1ae-c4a4e5a5a6de.metadata.json:
com.amazonaws.SdkClientException: Unable to unmarshall response (Trying to
access closed classloader. Please check if you store classloaders directly or
indirectly in static fields. If the stacktrace suggests that the leak occurs in
a third party library and cannot be fixed immediately, you can disable this
check with the configuration 'classloader.check-leaked-classloader'.). Response
Code: 200, Response Text: OK: Unable to unmarshall response (Trying to access
closed classloader. Please check if you store classloaders directly or
indirectly in static fields. If the stacktrace suggests that the leak occurs in
a third party library and cannot be fixed immediately, you can disable this
check with the configuration 'classloader.check-leaked-classloader'.). Response
Code: 200, Response Text: OK
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:217)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:174)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3796)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3652)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteWithoutCloseCheck(S3AFileSystem.java:3254)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:3228)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.iceberg.hadoop.HadoopFileIO.deleteFile(HadoopFileIO.java:105)
~[job-parser-with-dependencies.jar:?]
... 7 more
Caused by: com.amazonaws.SdkClientException: Unable to unmarshall response
(Trying to access closed classloader. Please check if you store classloaders
directly or indirectly in static fields. If the stacktrace suggests that the
leak occurs in a third party library and cannot be fixed immediately, you can
disable this check with the configuration
'classloader.check-leaked-classloader'.). Response Code: 200, Response Text: OK
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1851)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1509)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1416)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1183)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:838)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:805)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:779)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:735)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:717)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:581)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5588)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5535)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.services.s3.AmazonS3Client.listObjectsV2(AmazonS3Client.java:1007)
~[job-parser-with-dependencies.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listObjects$12(S3AFileSystem.java:2715)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
~[job-parser-with-dependencies.jar:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
~[job-parser-with-dependencies.jar:?]
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
~[hadoop-aws-3.3.6.jar:?]
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:2706)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3768)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3652)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteWithoutCloseCheck(S3AFileSystem.java:3254)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:3228)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.iceberg.hadoop.HadoopFileIO.deleteFile(HadoopFileIO.java:105)
~[job-parser-with-dependencies.jar:?]
... 7 more
Caused by: java.lang.IllegalStateException: Trying to access closed
classloader. Please check if you store classloaders directly or indirectly in
static fields. If the stacktrace suggests that the leak occurs in a third party
library and cannot be fixed immediately, you can disable this check with the
configuration 'classloader.check-leaked-classloader'.
at
org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:189)
~[flink-dist-1.20.1.jar:1.20.1]
at
org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.getResources(FlinkUserCodeClassLoaders.java:224)
~[flink-dist-1.20.1.jar:1.20.1]
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(Unknown
Source) ~[?:?]
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown
Source) ~[?:?]
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown
Source) ~[?:?]
at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source) ~[?:?]
at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source) ~[?:?]
at
java.xml/org.xml.sax.helpers.XMLReaderFactory.lambda$findServiceProvider$0(Unknown
Source) ~[?:?]
at java.base/java.security.AccessController.doPrivileged(Unknown
Source) ~[?:?]
at
java.xml/org.xml.sax.helpers.XMLReaderFactory.findServiceProvider(Unknown
Source) ~[?:?]
at
java.xml/org.xml.sax.helpers.XMLReaderFactory.createXMLReader(Unknown Source)
~[?:?]
at
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.<init>(XmlResponsesSaxParser.java:136)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsV2Unmarshaller.unmarshall(Unmarshallers.java:147)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsV2Unmarshaller.unmarshall(Unmarshallers.java:137)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:44)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:30)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1827)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1509)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1416)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1183)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:838)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:805)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:779)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:735)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:717)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:581)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5588)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5535)
~[job-parser-with-dependencies.jar:?]
at
com.amazonaws.services.s3.AmazonS3Client.listObjectsV2(AmazonS3Client.java:1007)
~[job-parser-with-dependencies.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listObjects$12(S3AFileSystem.java:2715)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
~[job-parser-with-dependencies.jar:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
~[job-parser-with-dependencies.jar:?]
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
~[hadoop-aws-3.3.6.jar:?]
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:2706)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3768)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3652)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteWithoutCloseCheck(S3AFileSystem.java:3254)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:3228)
~[hadoop-aws-3.3.6.jar:?]
at
org.apache.iceberg.hadoop.HadoopFileIO.deleteFile(HadoopFileIO.java:105)
~[job-parser-with-dependencies.jar:?]
... 7 more
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
