This seems unrelated and more like a user error somewhere. Can you just follow the steps, without any security settings i.e. not even uploading security.json and see if you still see this? Sorry, but I don't have access to the code right now, I try and look at this later tonight.
On Fri, Nov 20, 2015 at 3:07 PM, Oakley, Craig (NIH/NLM/NCBI) [C] < [email protected]> wrote: > Thank you for opening SOLR-8326 > > As a side note, in the procedure you listed, even before adding the > collection-admin-edit authorization, I'm already hitting trouble: stopping > and restarting a node results in the following > > INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4 > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; > Publishing state of core solr8326_shard2_replica1 as recovering, leader is > http://{IP-address-redacted}:8983/solr/solr8326_shard2_replica2/ and I am > http://{IP-address-redacted}:7574/solr/solr8326_shard2_replica1/ > INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4 > x:solr8326_shard2_replica1] org.apache.solr.cloud.ZkController; publishing > state=recovering > INFO - 2015-11-20 22:48:41.278; [c:solr8326 s:shard1 r:core_node3 > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; > Publishing state of core solr8326_shard1_replica1 as recovering, leader is > http://{IP-address-redacted}:8983/solr/solr8326_shard1_replica2/ and I am > http://{IP-address-redacted}:7574/solr/solr8326_shard1_replica1/ > INFO - 2015-11-20 22:48:41.280; [c:solr8326 s:shard1 r:core_node3 > x:solr8326_shard1_replica1] org.apache.solr.cloud.ZkController; publishing > state=recovering > INFO - 2015-11-20 22:48:41.282; [c:solr8326 s:shard2 r:core_node4 > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Sending > prep recovery command to http://{IP-address-redacted}:8983/solr; > WaitForState: > action=PREPRECOVERY&core=solr8326_shard2_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node4&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true > INFO - 2015-11-20 22:48:41.289; [ ] > org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change: > WatchedEvent state:SyncConnected type:NodeDataChanged > path:/collections/solr8326/state.json for collection solr8326 has occurred > - updating... (live nodes size: 2) > INFO - 2015-11-20 22:48:41.290; [c:solr8326 s:shard1 r:core_node3 > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Sending > prep recovery command to http://{IP-address-redacted}:8983/solr; > WaitForState: > action=PREPRECOVERY&core=solr8326_shard1_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true > INFO - 2015-11-20 22:48:41.291; [ ] > org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326 to > ver 25 > ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4 > x:solr8326_shard2_replica1] org.apache.solr.common.SolrException; Error > while trying to recover.:java.util.concurrent.ExecutionException: > org.apache.http.ParseException: Invalid content type: > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227) > Caused by: org.apache.http.ParseException: Invalid content type: > at org.apache.http.entity.ContentType.parse(ContentType.java:273) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512) > at > org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270) > at > org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > INFO - 2015-11-20 22:48:41.298; [ ] > org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change: > WatchedEvent state:SyncConnected type:NodeDataChanged > path:/collections/solr8326/state.json for collection solr8326 has occurred > - updating... (live nodes size: 2) > ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4 > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; > Recovery failed - trying again... (4) > INFO - 2015-11-20 22:48:41.300; [c:solr8326 s:shard2 r:core_node4 > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait > 32.0 seconds before trying to recover again (5) > ERROR - 2015-11-20 22:48:41.300; [c:solr8326 s:shard1 r:core_node3 > x:solr8326_shard1_replica1] org.apache.solr.common.SolrException; Error > while trying to recover.:java.util.concurrent.ExecutionException: > org.apache.http.ParseException: Invalid content type: > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227) > Caused by: org.apache.http.ParseException: Invalid content type: > at org.apache.http.entity.ContentType.parse(ContentType.java:273) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512) > at > org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270) > at > org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > ERROR - 2015-11-20 22:48:41.318; [c:solr8326 s:shard1 r:core_node3 > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; > Recovery failed - trying again... (4) > INFO - 2015-11-20 22:48:41.318; [ ] > org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326 to > ver 26 > INFO - 2015-11-20 22:48:41.319; [c:solr8326 s:shard1 r:core_node3 > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait > 32.0 seconds before trying to recover again (5) > > > I would not be surprised if this were to be some unrelated issue (the > symptoms are quite different) > > > > Thanks again > > > -----Original Message----- > From: Anshum Gupta [mailto:[email protected]] > Sent: Friday, November 20, 2015 1:31 PM > To: [email protected] > Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA > > Collections API were available before November of 2014, if that is when you > took the class. However, it was only with Solr 5.0 (released in Feb 2015) > that the only supported mechanism to create a collection was restricted to > Collections API. > > Here are the list of steps that you'd need to run to see that things are > fine for you without the read permission: > * Untar and setup Solr, don't start it yet > * Start clean zookeeper > * Put the security.json in zk, without anything other than a security-edit > permission. Find the content of the file below. Upload it using your own zk > client or through the solr script: > > solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 > -cmd putfile /security.json ~/security.json > > security.json: > > {"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= > > Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":{"solr":["admin"]},"permissions":[{"name":"security-edit","role":"admin"}]}} > > * Start solr: > > solr-5.3.1/bin/solr start -e cloud -z localhost:2181 > > You would need to key in a few things e.g. #nodes and ports, leave them at > the default values of 2 nodes and 8983/7574, unless you want to run Solr on > a different port. Then let it create a default collection to just make sure > that everything works fine. > > * Add the collection-admin-edit command: > > curl --user solr:SolrRocks > http://localhost:8983/solr/admin/authorization > -H 'Content-type:application/json' -d '{"set-permission" : > {"name":"collection-admin-edit", "role":"admin"}}' > > At this point, everything should be working fine. Restarting the nodes > should also work fine. You can try 2 things at this point: > 1. Create a new collection with 1 shard and 1 replica and then try adding a > replica, here's how: > > curl --user solr:SolrRocks > > http://localhost:8983/solr/admin/collections?action=CREATE&name=testcollection&collection.configName=gettingstarted&numShards=1 > > > curl --user solr:SolrRocks > > http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=testcollection&shard=shard1 > > This should work fine. > > 2. After this, try restarting the solr cluster. Here's how you can do so, > assuming you didn't change any of the defaults and you are running zk on > localhost:2181. If not, just change those values below: > > bin/solr stop -all > > After this, check that Solr was actually stopped. I'd also suggest you tail > the logs on both nodes when they are coming up to see any errors, if any. > The logs would be here: example/cloud/node1/logs/solr.log > and example/cloud/node2/logs/solr.log > > > bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z localhost:2181 > > bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181 > > If you get to this checkpoint fine, try adding a read permission. > Add a permission: > > curl --user solr:SolrRocks > http://localhost:8983/solr/admin/authorization > -H 'Content-type:application/json' -d '{"set-permission" : {"name":"read", > "role":"read"}}' > > Add a user: > > curl --user solr:SolrRocks > http://localhost:8983/solr/admin/authentication > -H 'Content-type:application/json' -d '{"set-user" : > {"solrread":"solrRocks"}}' > > Assign a role to the user: > >curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization > -H 'Content-type:application/json' -d '{"set-user-role" : > {"solrread":["read"]}}' > > After this, you should start having issues with ADDREPLICA. > Also, as you would at this point have a collection with a shard that has a > replication factor > 1 (remember the ADDREPLICA we did earlier), you would > have issues when you restart the cluster again using the steps I mentioned > above. > > > Can you confirm this? I guess I'll just use this text to create a new JIRA > now. > > > On Fri, Nov 20, 2015 at 10:04 AM, Oakley, Craig (NIH/NLM/NCBI) [C] < > [email protected]> wrote: > > > Thank you again for the reply. > > > > Below is the Email I was about to send prior to your reply a moment ago: > > shall I try again without "read" in the security.json? > > > > > > > > The Collections API method was not discussed in the "Unleashed" class at > > the conference in DC in 2014 (probably because it was not yet available), > > so I was using the method I knew. > > > > I have now tried again using admin/collections?action=CREATE (using > > different port numbers to avoid confusion from the failed previous > > attempts: the previously created nodes had been shutdown and their > > core.properties files renamed so as not to be discovered), but the > results > > are the same: > > INFO - 2015-11-20 16:56:25.283; [c:xmpl3 s:shard1 r:core_node2 > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Starting > > Replication Recovery. > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2 > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Begin > > buffering updates. > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2 > > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Starting to > > buffer updates. FSUpdateLog{state=ACTIVE, tlog=null} > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2 > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; > Attempting > > to replicate from http:// > > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1/. > > ERROR - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2 > > x:xmpl3_shard1_replica2] org.apache.solr.common.SolrException; Error > while > > trying to > > > recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: > > Error from server at http:// > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1: > > Expected mime type application/octet-stream but got text/html. <html> > > <head> > > <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/> > > <title>Error 401 Unauthorized request, Response code: 401</title> > > </head> > > <body><h2>HTTP ERROR 401</h2> > > <p>Problem accessing /solr/xmpl3_shard1_replica1/update. Reason: > > <pre> Unauthorized request, Response code: > > 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/> > > > > </body> > > </html> > > > > at > > > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528) > > at > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234) > > at > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226) > > at > > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135) > > at > > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152) > > at > > > org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207) > > at > > > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147) > > at > > > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437) > > at > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227) > > > > INFO - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2 > > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping > > buffered updates FSUpdateLog{state=BUFFERING, tlog=null} > > ERROR - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2 > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Recovery > > failed - trying again... (2) > > INFO - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2 > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Wait 8.0 > > seconds before trying to recover again (3) > > > > > > Below is a list of the steps I took. > > > > ./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr/xmpl3 > > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd putfile /security.json > > ~/solr/security151119a.json > > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd upconfig -confdir > > ../../solr/configsets/basic_configs/conf -confname xmpl3 > > cd ../../../bin/ > > ./solr -c -p 4695 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/ > -z > > localhost:4545/solr/xmpl3 -s > > ~dbman/solr/straight531outofbox/solr-5.3.1/example/solr > > ./solr -c -p 4685 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/ > -z > > localhost:4545/solr/xmpl3 -s > > ~dbman/solr/straight531outofbox/solr-5.3.1/server/solr > > curl -u solr:SolrRocks ' > > > http://nosqltest11:4685/solr/admin/collections?action=CREATE&name=xmpl3&numShards=1&replicationFactor=1&createNodeSet={IP-address-redacted}:4685_solr > > ' > > curl -u solr:SolrRocks ' > > > http://nosqltest11:4685/solr/admin/collections?action=ADDREPLICA&collection=xmpl3&shard=shard1&node={IP-address-redacted}:4695_solr&wt=json&indent=true > > ' > > > > > > > > > > Can you provide a list of steps to take in an out-of-the-box directory > > tree whereby ADDREPLICA _will_ work with security.json already in place? > > > > > > > > > > -----Original Message----- > > From: Anshum Gupta [mailto:[email protected]] > > Sent: Thursday, November 19, 2015 3:44 PM > > To: [email protected] > > Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA > > > > I'll try out what you did later in the day, as soon as I get time but why > > exactly are you creating cores manually? Seems like you manually create a > > core and the try to add a replica. Can you try using the Collections API > to > > create a collection? > > > > Starting Solr 5.0, the only supported way to create a new collection is > via > > the Collections API. Creating a core would lead to a collection creation > > but that's not really supported. It was just something that was done when > > there were no Collections API. > > > > > > On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] < > > [email protected]> wrote: > > > > > I tried again with the following security.json, but the results were > the > > > same: > > > > > > { > > > "authentication":{ > > > "class":"solr.BasicAuthPlugin", > > > "credentials":{ > > > "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= > > > Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=", > > > "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE= > > > 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="}, > > > "":{"v":9}}, > > > "authorization":{ > > > "class":"solr.RuleBasedAuthorizationPlugin", > > > "user-role":{ > > > "solr":[ > > > "admin", > > > "read", > > > "xmpladmin", > > > "xmplgen", > > > "xmplsel"], > > > "solruser":[ > > > "read", > > > "xmplgen", > > > "xmplsel"]}, > > > "permissions":[ > > > { > > > "name":"security-edit", > > > "role":"admin"}, > > > { > > > "name":"xmpl_admin", > > > "collection":"xmpl", > > > "path":"/admin/*", > > > "role":"xmpladmin"}, > > > { > > > "name":"xmpl_sel", > > > "collection":"xmpl", > > > "path":"/select/*", > > > "role":null}, > > > { > > > "name":"all-admin", > > > "collection":null, > > > "path":"/*", > > > "role":"xmplgen"}, > > > { > > > "name":"all-core-handlers", > > > "path":"/*", > > > "role":"xmplgen"}], > > > "":{"v":42}}} > > > > > > -----Original Message----- > > > From: Oakley, Craig (NIH/NLM/NCBI) [C] > > > Sent: Thursday, November 19, 2015 1:46 PM > > > To: '[email protected]' <[email protected]> > > > Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA > > > > > > I note that the thread called "Security Problems" (most recent post by > > > Nobel Paul) seems like it may help with much of what I'm trying to do. > I > > > will see to what extent that may help. > > > > > > > > > > > -- > > Anshum Gupta > > > > > > -- > Anshum Gupta > -- Anshum Gupta
