ochanism commented on issue #12220:
URL: https://github.com/apache/iceberg/issues/12220#issuecomment-2652510589
@pvary Oh, thanks for the information! That's very helpful to understand the
current status.
Here are my issue details.
- error stack trace
```
server error: ErrorResponse(code=500, type=RejectedExecutionException,
message=Task java.util.concurrent.FutureTask@3d5cba52[Not completed, task =
org.apache.iceberg.aws.s3.S3FileIO$$Lambda/0x00007face4ddb8c0@58a93454]
rejected from java.util.concurrent.ThreadPoolExecutor@6eeffebd[Terminated, pool
size = 0, active threads = 0, queued tasks = 0, completed tasks = 88])
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.FutureTask@3d5cba52[Not completed, task =
org.apache.iceberg.aws.s3.S3FileIO$$Lambda/0x00007face4ddb8c0@58a93454]
rejected from java.util.concurrent.ThreadPoolExecutor@6eeffebd[Terminated, pool
size = 0, active threads = 0, queued tasks = 0, completed tasks = 88]
at
java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2081)
at
java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:841)
at
java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1376)
at
java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
at
java.base/java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:790)
at org.apache.iceberg.aws.s3.S3FileIO.deleteFiles(S3FileIO.java:219)
at
org.apache.iceberg.CatalogUtil.deleteRemovedMetadataFiles(CatalogUtil.java:548)
at
org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:126)
at
org.apache.iceberg.rest.CatalogHandlers.lambda$commit$7(CatalogHandlers.java:431)
at
org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
at
org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at
org.apache.iceberg.rest.CatalogHandlers.commit(CatalogHandlers.java:407)
at
org.apache.iceberg.rest.CatalogHandlers.updateTable(CatalogHandlers.java:348)
at
org.apache.iceberg.rest.RESTCatalogAdapter.handleRequest(RESTCatalogAdapter.java:413)
at
org.apache.iceberg.rest.PuffinCatalogAdapter.handleRequest(PuffinCatalogAdapter.java:35)
at
org.apache.iceberg.rest.PuffinCatalogAdapter.execute(PuffinCatalogAdapter.java:90)
at
org.apache.iceberg.rest.RESTCatalogServlet.execute(RESTCatalogServlet.java:100)
at
org.apache.iceberg.rest.RESTCatalogServlet.doPost(RESTCatalogServlet.java:78)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at
com.giffing.bucket4j.spring.boot.starter.filter.servlet.ServletRequestFilter.doFilterInternal(ServletRequestFilter.java:66)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at
org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108)
at
org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231)
at
org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:479)
at
org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:340)
at
org.springframework.security.web.ObservationFilterChainDecorator.lambda$wrapSecured$0(ObservationFilterChainDecorator.java:82)
at
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:128)
at
org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:101)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)
at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at
com.naver.alaska.puffin.security.CatalogAuthenticationFilter.doFilter(CatalogAuthenticationFilter.java:71)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
at
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at
org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
at
org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at
org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82)
at
org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at
org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at
org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at
org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$0(ObservationFilterChainDecorator.java:323)
at
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:224)
at
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)
at
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)
at
org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)
at
org.springframework.web.servlet.handler.HandlerMappingIntrospector.lambda$createCacheFilter$3(HandlerMappingIntrospector.java:243)
at
org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)
at
org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74)
at
org.springframework.security.config.annotation.web.configuration.WebMvcSecurityConfiguration$CompositeFilterChainProxy.doFilter(WebMvcSecurityConfiguration.java:238)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:362)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:278)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at
org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at
com.naver.alaska.puffin.filter.UrlRewriteFilter.doFilterInternal(UrlRewriteFilter.java:30)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at
com.naver.alaska.puffin.filter.TraceFilter.doFilterInternal(TraceFilter.java:34)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at
org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:114)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at
org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:731)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:397)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:905)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at java.base/java.lang.VirtualThread.run(VirtualThread.java:329)
```
The active request tried to commit some changes and delete the oldest
metadata according to the `TableProperties.METADATA_PREVIOUS_VERSIONS_MAX`.
But an executor service for the metadata deletion was
`ThreadPools.newExitingWorkerPool("iceberg-s3fileio-delete",
s3FileIOProperties.deleteThreads())`, so the executor service already was
terminated.
With this executor service, requests under a commit cannot be completed
gracefully.
What do you think about this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
