pingtimeout opened a new issue, #3836: URL: https://github.com/apache/polaris/issues/3836
### Describe the bug While I was testing the https://github.com/apache/polaris-tools/pull/173/ code, I ran into lot of errors like the one below during the write workload: ``` 2026-02-19 11:29:43,998 ERROR [org.apa.pol.ser.exc.IcebergExceptionMapper] [a4ef00c5-6eba-4c69-8e57-f35c13548212_0000000000000000676,POLARIS] [,,,] (executor-thread-6) Unhandled exception returning INTERNAL_SERVER_ERROR: java.lang.IllegalStateException: grants_version_going_backward: entity={} grantsVersion={}, [name=catalog_administrator_C_0;id=660631469561579369;parentId=1137621942110885014;entityVersion=1;type=CATALOG_ROLE;subType=NULL_SUBTYPE;internalProperties={}, 136] at com.google.common.base.Preconditions.checkState(Preconditions.java:853) at org.apache.polaris.core.PolarisDefaultDiagServiceImpl.check(PolarisDefaultDiagServiceImpl.java:126) at org.apache.polaris.core.persistence.ResolvedPolarisEntity.<init>(ResolvedPolarisEntity.java:66) at org.apache.polaris.core.persistence.cache.InMemoryEntityCache.getAndRefreshIfNeeded(InMemoryEntityCache.java:343) at org.apache.polaris.core.persistence.resolver.Resolver.bulkValidate(Resolver.java:576) at org.apache.polaris.core.persistence.resolver.Resolver.runResolvePass(Resolver.java:440) at org.apache.polaris.core.persistence.resolver.Resolver.resolveAll(Resolver.java:250) at org.apache.polaris.core.persistence.resolver.PolarisResolutionManifest.resolveAll(PolarisResolutionManifest.java:135) at org.apache.polaris.service.admin.PolarisAdminService.authorizeGrantOnNamespaceOperationOrThrow(PolarisAdminService.java:459) at org.apache.polaris.service.admin.PolarisAdminService.grantPrivilegeOnNamespaceToRole(PolarisAdminService.java:1690) at org.apache.polaris.service.admin.PolarisAdminService_ClientProxy.grantPrivilegeOnNamespaceToRole(Unknown Source) at org.apache.polaris.service.admin.PolarisServiceImpl.addGrantToCatalogRole(PolarisServiceImpl.java:641) at org.apache.polaris.service.admin.PolarisServiceImpl_Subclass.addGrantToCatalogRole$$superforward(Unknown Source) at org.apache.polaris.service.admin.PolarisCatalogsEventServiceDelegator_BLUoP2dn7JG2IbxCE7Nu_CXmT7s_Delegate_Subclass.addGrantToCatalogRole(Unknown Source) at org.apache.polaris.service.admin.PolarisCatalogsEventServiceDelegator.addGrantToCatalogRole(PolarisCatalogsEventServiceDelegator.java:199) at org.apache.polaris.service.admin.PolarisServiceImpl_Subclass.addGrantToCatalogRole(Unknown Source) at org.apache.polaris.service.admin.PolarisServiceImpl_ClientProxy.addGrantToCatalogRole(Unknown Source) at org.apache.polaris.service.admin.api.PolarisCatalogsApi.addGrantToCatalogRole(PolarisCatalogsApi.java:105) at org.apache.polaris.service.admin.api.PolarisCatalogsApi_Subclass.addGrantToCatalogRole$$superforward(Unknown Source) at org.apache.polaris.service.admin.api.PolarisCatalogsApi_Subclass$$function$$1.apply(Unknown Source) at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73) at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97) at io.quarkus.hibernate.validator.runtime.interceptor.AbstractMethodValidationInterceptor.validateMethodInvocation(AbstractMethodValidationInterceptor.java:71) at io.quarkus.hibernate.validator.runtime.jaxrs.ResteasyReactiveEndPointValidationInterceptor.validateMethodInvocation(ResteasyReactiveEndPointValidationInterceptor.java:21) at io.quarkus.hibernate.validator.runtime.jaxrs.ResteasyReactiveEndPointValidationInterceptor_Bean.intercept(Unknown Source) at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42) at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70) at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97) at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$8(FaultToleranceInterceptor.java:364) at io.smallrye.faulttolerance.core.Future.from(Future.java:85) at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$9(FaultToleranceInterceptor.java:364) at io.smallrye.faulttolerance.core.FaultToleranceContext.call(FaultToleranceContext.java:20) at io.smallrye.faulttolerance.core.Invocation.apply(Invocation.java:29) at io.smallrye.faulttolerance.core.metrics.MetricsCollector.apply(MetricsCollector.java:98) at io.smallrye.faulttolerance.FaultToleranceInterceptor.syncFlow(FaultToleranceInterceptor.java:367) at io.smallrye.faulttolerance.FaultToleranceInterceptor.intercept(FaultToleranceInterceptor.java:205) at io.smallrye.faulttolerance.FaultToleranceInterceptor_Bean.intercept(Unknown Source) at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42) at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70) at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97) at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor.timedMethod(MicrometerTimedInterceptor.java:79) at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor_Bean.intercept(Unknown Source) at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42) at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70) at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97) at io.quarkus.security.runtime.interceptor.SecurityHandler.handle(SecurityHandler.java:27) at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor.intercept(RolesAllowedInterceptor.java:29) at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor_Bean.intercept(Unknown Source) at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42) at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70) at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62) at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor.intercept(StandardSecurityCheckInterceptor.java:44) at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor_RolesAllowedInterceptor_Bean.intercept(Unknown Source) at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42) at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30) at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27) at org.apache.polaris.service.admin.api.PolarisCatalogsApi_Subclass.addGrantToCatalogRole(Unknown Source) at org.apache.polaris.service.admin.api.PolarisCatalogsApi$quarkusrestinvoker$addGrantToCatalogRole_56882127262eb1301ad4a14f5e3fb19c9e966db3.invoke(Unknown Source) at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29) at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:183) at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147) at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:645) at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651) at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630) at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589) at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11) at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:1583) ``` This caused multiple HTTP 500 errors on the client side. Attached is the full Polaris log file. [polaris.log.gz](https://github.com/user-attachments/files/25419316/polaris.log.gz) ### To Reproduce 1. Checkout the branch from https://github.com/apache/polaris-tools/pull/173/ (until it is merged). 2. Create an `application.conf` file as given below, adjust the Polaris base URL accordingly 3. Run `make create-dataset-simulation` from the `benchmarks/` directory ``` http { base-url = "http://localhost:8181"; } auth { client-id = "root" client-secret = "s3cr3t" } rbac { enabled = true num-principals = 5 catalog-role-names = ["catalog_reader", "catalog_contributor"] principal-role-names = ["data_engineer", "data_scientist"] } dataset.tree { num-catalogs = 5 namespace-width = 10 namespace-depth = 5 tables-per-namespace = 10 views-per-namespace = 10 columns-per-table = 10 columns-per-view = 10 default-base-location = "file:///tmp/polaris" namespace-properties = 10 table-properties = 10 view-properties = 10 } workload { create-tree-dataset { table-concurrency = 10 view-concurrency = 10 } } ``` ### Actual Behavior _No response_ ### Expected Behavior _No response_ ### Additional context _No response_ ### System information _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
