sgowda97 opened a new issue, #672:
URL: https://github.com/apache/solr-operator/issues/672
Hi,
I am running solr-operator 0.8.0, solr 9.4.0 on GKE. Here is my YAML file
`
apiVersion: solr.apache.org/v1beta1
kind: SolrCloud
metadata:
name: dev
spec:
solrAddressability:
commonServicePort: 80
external:
domainName: HOST
method: Ingress
nodePortOverride: 80
useExternalAddress: false
podPort: 8983
customSolrKubeOptions:
podOptions:
resources:
limits:
memory: 512Mi
cpu: 500m
requests:
cpu: 500m
memory: 256Mi
dataStorage:
persistent:
pvcTemplate:
spec:
resources:
requests:
storage: 10Gi
reclaimPolicy: Delete
replicas: 2
solrImage:
repository: solr
tag: "9.4.0"
solrJavaMem: -Xms256M -Xmx256M
updateStrategy:
method: StatefulSet
zookeeperRef:
provided:
chroot: /dev
image:
pullPolicy: IfNotPresent
repository: pravega/zookeeper
tag: 0.2.8
persistence:
reclaimPolicy: Delete
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
replicas: 1
zookeeperPodPolicy:
resources:
limits:
memory: 256Mi
cpu: 500m
requests:
cpu: 500m
memory: 128Mi
solrSecurity:
authenticationType: Basic
`
The pods keep crashing. To inspect, I logged one of the pod. This is the
error I am getting;
`
$ ksolr logs pod/dev-solrcloud-1 -f
Defaulted container "solrcloud-node" out of: solrcloud-node, cp-solr-xml
(init), setup-zk (init)
Starting Solr
Java 17 detected. Enabled workaround for SOLR-16463
[0.002s][warning][pagesize] UseLargePages disabled, no large pages
configured and available on the system.
CompileCommand: exclude
com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
WARNING: A command line option has enabled the Security Manager
WARNING: The Security Manager is deprecated and will be removed in a future
release
2024-01-03 07:27:01.682 INFO (main) [] o.e.j.s.Server jetty-10.0.17; built:
2023-10-09T18:22:21.150Z; git: af15f12297adf5c5083e1f2f8f4c5974438bca25; jvm
17.0.9+9
2024-01-03 07:27:04.075 WARN (main) [] o.e.j.u.DeprecationWarning Using
@Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2024-01-03 07:27:04.279 INFO (main) [] o.a.s.s.CoreContainerProvider Using
logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2024-01-03 07:27:04.369 INFO (main) [] o.a.s.s.CoreContainerProvider ___
_ Welcome to Apache Solr™ version 9.4.0
2024-01-03 07:27:04.370 INFO (main) [] o.a.s.s.CoreContainerProvider / __|
___| |_ _ Starting in cloud mode on port 8983
2024-01-03 07:27:04.370 INFO (main) [] o.a.s.s.CoreContainerProvider \__ \/
_ \ | '_| Install dir: /opt/solr-9.4.0
2024-01-03 07:27:04.370 INFO (main) [] o.a.s.s.CoreContainerProvider
|___/\___/_|_| Start time: 2024-01-03T07:27:04.370560543Z
2024-01-03 07:27:04.376 INFO (main) [] o.a.s.s.CoreContainerProvider Solr
started with "-XX:+CrashOnOutOfMemoryError" that will crash on any
OutOfMemoryError exception. The cause of the OOME will be logged in the crash
file at the following path: /var/solr/logs/jvm_crash_13.log
2024-01-03 07:27:04.376 INFO (main) [] o.a.s.s.CoreContainerProvider Log
level override, property solr.log.level=INFO
2024-01-03 07:27:04.570 INFO (main) [] o.a.s.s.CoreContainerProvider Solr
Home: /var/solr/data (source: system property: solr.solr.home)
2024-01-03 07:27:04.687 WARN (main) [] o.a.s.c.c.SolrZkClient Using default
ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty
list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2024-01-03 07:27:04.967 INFO (main) [] o.a.s.c.c.ConnectionManager Waiting
up to 30000ms for client to connect to ZooKeeper
2024-01-03 07:27:05.067 INFO (zkConnectionManagerCallback-2-thread-1) []
o.a.s.c.c.ConnectionManager zkClient has connected
2024-01-03 07:27:05.069 INFO (main) [] o.a.s.c.c.ConnectionManager Client
is connected to ZooKeeper
2024-01-03 07:27:05.069 WARN (main) [] o.a.s.c.c.SolrZkClient Using default
ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE'
ACLs to Zookeeper nodes
2024-01-03 07:27:05.188 INFO (main) [] o.a.s.c.NodeConfig Loading solr.xml
from SolrHome (not found in ZooKeeper)
2024-01-03 07:27:05.189 INFO (main) [] o.a.s.c.SolrXmlConfig Loading
solr.xml from /var/solr/data/solr.xml
2024-01-03 07:27:05.387 INFO (main) [] o.a.s.c.SolrResourceLoader Added 1
libs to classloader, from paths: [/opt/solr-9.4.0/lib]
2024-01-03 07:27:10.179 INFO (main) [] o.a.s.u.t.SimplePropagator Always-on
trace id generation enabled.
2024-01-03 07:27:11.977 WARN (main) [] o.a.s.u.StartupLoggingUtils Jetty
request logging enabled. Will retain logs for last 3 days. See chapter
"Configuring Logging" in reference guide for how to configure.
2024-01-03 07:27:11.977 INFO (main) [] o.a.s.c.ZkContainer Zookeeper
client=dev-solrcloud-zookeeper-0.dev-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181/dev
2024-01-03 07:27:12.072 WARN (main) [] o.a.s.c.c.SolrZkClient Using default
ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty
list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2024-01-03 07:27:12.075 INFO (main) [] o.a.s.c.c.ConnectionManager Waiting
up to 15000ms for client to connect to ZooKeeper
2024-01-03 07:27:12.087 INFO (zkConnectionManagerCallback-12-thread-1) []
o.a.s.c.c.ConnectionManager zkClient has connected
2024-01-03 07:27:12.088 INFO (main) [] o.a.s.c.c.ConnectionManager Client
is connected to ZooKeeper
2024-01-03 07:27:12.088 WARN (main) [] o.a.s.c.c.SolrZkClient Using default
ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE'
ACLs to Zookeeper nodes
2024-01-03 07:27:12.275 INFO (main) []
o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater
with useDistributedStateUpdate=false. Solr will be using Overseer based cluster
state updates.
2024-01-03 07:27:12.288 INFO (main) [] o.a.s.c.c.ConnectionManager Waiting
up to 15000ms for client to connect to ZooKeeper
2024-01-03 07:27:12.366 INFO (zkConnectionManagerCallback-14-thread-1) []
o.a.s.c.c.ConnectionManager zkClient has connected
2024-01-03 07:27:12.366 INFO (main) [] o.a.s.c.c.ConnectionManager Client
is connected to ZooKeeper
2024-01-03 07:27:12.778 WARN (main) [] o.a.s.c.ZkController Contents of
zookeeper /security.json are world-readable; consider setting up ACLs as
described in
https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2024-01-03 07:27:12.969 INFO (main) []
o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater
with useDistributedStateUpdate=false. Solr will be using Overseer based cluster
state updates.
2024-01-03 07:27:13.067 INFO (main) [] o.a.s.c.OverseerElectionContext I am
going to be the leader dev-solrcloud-1.solr:80_solr
2024-01-03 07:27:13.080 INFO (main) [] o.a.s.c.Overseer Overseer
(id=72063280979050514-dev-solrcloud-1.solr:80_solr-n_0000000000) starting
2024-01-03 07:27:13.585 INFO
(OverseerStateUpdate-72063280979050514-dev-solrcloud-1.solr:80_solr-n_0000000000)
[] o.a.s.c.Overseer Starting to work on the main queue :
dev-solrcloud-1.solr:80_solr
2024-01-03 07:27:13.668 INFO (main) [] o.a.s.c.ZkController Register node
as live in ZooKeeper:/live_nodes/dev-solrcloud-1.solr:80_solr
2024-01-03 07:27:13.770 INFO (zkCallback-13-thread-1) []
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2024-01-03 07:27:14.079 ERROR (main) [] o.a.s.s.CoreContainerProvider Could
not start Solr. Check solr/home property and the logs =>
java.lang.NullPointerException: Cannot invoke "java.util.Map.get(Object)"
because the return value of
"org.apache.solr.handler.admin.SecurityConfHandler$SecurityConfig.getData()" is
null
at
org.apache.solr.core.CoreContainer.reloadSecurityProperties(CoreContainer.java:1186)
java.lang.NullPointerException: Cannot invoke "java.util.Map.get(Object)"
because the return value of
"org.apache.solr.handler.admin.SecurityConfHandler$SecurityConfig.getData()" is
null
at
org.apache.solr.core.CoreContainer.reloadSecurityProperties(CoreContainer.java:1186)
~[?:?]
at
org.apache.solr.core.CoreContainer.loadInternal(CoreContainer.java:856) ~[?:?]
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:765)
~[?:?]
at
org.apache.solr.servlet.CoreContainerProvider.createCoreContainer(CoreContainerProvider.java:427)
~[?:?]
at
org.apache.solr.servlet.CoreContainerProvider.init(CoreContainerProvider.java:246)
~[?:?]
at
org.apache.solr.servlet.CoreContainerProvider.contextInitialized(CoreContainerProvider.java:116)
~[?:?]
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1049)
~[jetty-server-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:624)
~[jetty-servlet-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:984)
~[jetty-server-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:740)
~[jetty-servlet-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392)
~[jetty-servlet-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1304)
~[jetty-webapp-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:901)
~[jetty-server-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306)
~[jetty-servlet-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:532)
~[jetty-webapp-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89)
~[jetty-server-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89)
~[jetty-server-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89)
~[jetty-server-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89)
~[jetty-server-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89)
~[jetty-server-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:221)
~[jetty-server-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
~[jetty-util-10.0.17.jar:10.0.17]
at org.eclipse.jetty.server.Server.start(Server.java:470)
~[jetty-server-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89)
~[jetty-server-10.0.17.jar:10.0.17]
at org.eclipse.jetty.server.Server.doStart(Server.java:415)
~[jetty-server-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
~[jetty-util-10.0.17.jar:10.0.17]
at
org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1919)
~[jetty-xml-10.0.17.jar:10.0.17]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.eclipse.jetty.start.Main.invokeMain(Main.java:229)
~[start.jar:10.0.17]
at org.eclipse.jetty.start.Main.start(Main.java:528)
~[start.jar:10.0.17]
at org.eclipse.jetty.start.Main.main(Main.java:76) ~[start.jar:10.0.17]
2024-01-03 07:27:14.171 ERROR (main) [] o.a.s.s.CoreContainerProvider Error
processing the request. CoreContainer is either not initialized or shutting
down.
2024-01-03 07:27:14.171 ERROR (main) [] o.a.s.s.SolrDispatchFilter Could not
start Dispatch Filter. => javax.servlet.UnavailableException: Error processing
the request. CoreContainer is either not initialized or shutting down.
at
org.apache.solr.servlet.CoreContainerProvider.waitForCoreContainer(CoreContainerProvider.java:156)
`
This does not happen if I `kubectl apply -f yaml.yaml` without the basic
auth property. One more thing I noticed is, the error miraculously goes away if
I just `kubectl delete pod/dev-solrcloud-1` since it gets re-created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
