estebanz01 commented on issue #12547: URL: https://github.com/apache/pinot/issues/12547#issuecomment-1980904683
here's the full log of a fresh minion pod: <details> <summary>minion pod log</summary> ```java SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/pinot/lib/pinot-all-1.0.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-environment/pinot-azure/pinot-azure-1.0.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-file-system/pinot-s3/pinot-s3-1.0.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-input-format/pinot-clp-log/pinot-clp-log-1.0.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-input-format/pinot-orc/pinot-orc-1.0.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-input-format/pinot-parquet/pinot-parquet-1.0.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-metrics/pinot-dropwizard/pinot-dropwizard-1.0.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-metrics/pinot-yammer/pinot-yammer-1.0.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-stream-ingestion/pinot-pulsar/pinot-pulsar-1.0.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. ERROR StatusLogger Reconfiguration failed: No configuration found for 'Default' at 'null' in 'null' WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/opt/pinot/lib/pinot-all-1.0.0-jar-with-dependencies.jar) to method java.lang.Object.finalize() WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Mar 06, 2024 1:24:34 PM org.glassfish.grizzly.http.server.NetworkListener start INFO: Started listener bound to [0.0.0.0:9514] Mar 06, 2024 1:24:34 PM org.glassfish.grizzly.http.server.HttpServer start INFO: [HttpServer] Started. 13:25:18.815 [TaskStateModelFactory-task_thread-0] ERROR org.apache.pinot.minion.taskfactory.TaskFactoryRegistry - Caught exception while executing task: Task_RealtimeToOfflineSegmentsTask_ddf2fd57-ed8f-4ee8-8c04-1e21137ed566_1709731500049_0 org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 1 attempts at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.common.utils.fetcher.HttpSegmentFetcher.fetchSegmentToLocal(HttpSegmentFetcher.java:62) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocalInternal(SegmentFetcherFactory.java:158) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:152) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocalInternal(SegmentFetcherFactory.java:202) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(SegmentFetcherFactory.java:190) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:201) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:77) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.runInternal(TaskFactoryRegistry.java:157) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.run(TaskFactoryRegistry.java:118) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.helix.task.TaskRunner.run(TaskRunner.java:75) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] ``` </details> and the pod description: <details> <summary>Minion pod description</summary> ```bash Name: pinot-minion-0 Namespace: pinot Priority: 0 Service Account: pinot Node: <internal-DNS-name>/<internal-IP> Start Time: Wed, 06 Mar 2024 08:23:42 -0500 Labels: app=pinot app.kubernetes.io/managed-by=Helm app.kubernetes.io/version=0.2.7 component=minion controller-revision-hash=pinot-minion-84bbfbc6f4 helm.sh/chart=pinot-0.2.7 heritage=Helm release=pinot statefulset.kubernetes.io/pod-name=pinot-minion-0 Annotations: kubectl.kubernetes.io/restartedAt: 2024-03-06T15:16:59+00:00 kubernetes.io/psp: eks.privileged Status: Running IP: <internal-IP> IPs: IP: <internal-IP> Controlled By: StatefulSet/pinot-minion Containers: minion: Container ID: containerd://e2cfae774017937ba2aa4f217d5f84a20809e4961c8920a82165bed4e290d2bf Image: apachepinot/pinot:release-1.0.0 Image ID: docker.io/apachepinot/pinot@sha256:ef93c03cb223a30e2a0eb75452dfb2db1eab05271a59e2913845bff9814556bc Port: 9514/TCP Host Port: 0/TCP Args: StartMinion -clusterName pinot -zkAddress pinot-zookeeper:2181 -configFileName /var/pinot/minion/config/pinot-minion.conf State: Running Started: Wed, 06 Mar 2024 08:23:49 -0500 Ready: True Restart Count: 0 Limits: cpu: 400m memory: 1Gi Requests: cpu: 200m memory: 512Mi Liveness: http-get http://:9514/health delay=60s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:9514/health delay=60s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: s3-deep-storage-user Secret Optional: false Environment: JAVA_OPTS: -XX:ActiveProcessorCount=2 -XX:MaxRAMPercentage=70.0 -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-minion.log -Dlog4j2.configurationFile=/opt/pinot/etc/config/pinot-minion-log4j2.xml -Dplugins.dir=/opt/pinot/plugins LOG4J_CONSOLE_LEVEL: info Mounts: /var/pinot/minion/config from config (rw) /var/pinot/minion/data from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xhbhh (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-pinot-minion-0 ReadOnly: false config: Type: ConfigMap (a volume populated by a ConfigMap) Name: pinot-minion-config Optional: false kube-api-access-xhbhh: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 15m default-scheduler Successfully assigned pinot/pinot-minion-0 to <internal-dns-name> Normal Pulled 15m kubelet Container image "apachepinot/pinot:release-1.0.0" already present on machine Normal Created 15m kubelet Created container minion Normal Started 15m kubelet Started container minion ``` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
