Re: Could not find collection when upgrading to Solr 7
On 2/24/2018 8:05 AM, Marvin Bredal Lillehaug wrote: This is for doing local development and running tests. So starting with embedded zookeeper in SolrCloud mode is intentional. The idea is to just run the script setting up the folder structure for the cores and everything is up and running, ready for indexing and querying locally. The applications using Solr also use Zookeeper for configuration and coordination of their own stuff. There are no persmission issues. I have put the scripts here https://github.com/computerlove/start-solr The file config specifies what version to use. When running run-dev-solr.sh that version is downloaded, solrhome is copied to the right place and the instance is started. I grabbed the git repo and proceeded to fiddle with it. As you said, it works when the solr version that is downloaded is 6.6.2, but fails with 7.2.1. Based on what gets logged and what I know about how Solr does initialization, I'm betting the bootstrap feature is broken. I can't tell what's wrong, but I think that's probably where the problem is. But as I already said, I strongly recommend that you don't use that feature. At all. The cores that are in the git repo are empty -- there's no index. If the source cores are empty when you run the script "for real", then I have an alternate idea for you to try: Instead of copying the cores and bootstrapping a non-cloud install into a cloud install: Remove the bootstrap option. Create the collections using "bin/solr create", with -replicationFactor 1 -shards 1, and the -d option pointing at the config you want uploaded for the collection. You could even go with a higher shard count if you want. Increasing replicationFactor is probably possible, but as all this would be running on a single server, there's no point. Thanks, Shawn
Re: Could not find collection when upgrading to Solr 7
Thank your for your reply. I will take a look at both the code for bootstrapping, and look at what you are suggesting to determine what to do. On Sun, Feb 25, 2018 at 5:17 PM, Shawn Heisey wrote: > On 2/24/2018 8:05 AM, Marvin Bredal Lillehaug wrote: > >> This is for doing local development and running tests. >> So starting with embedded zookeeper in SolrCloud mode is intentional. The >> idea is to just run the script setting up the folder structure for the >> cores and everything is up and running, ready for indexing and querying >> locally. >> The applications using Solr also use Zookeeper for configuration and >> coordination of their own stuff. >> >> There are no persmission issues. >> >> I have put the scripts here https://github.com/computerlove/start-solr >> >> The file config specifies what version to use. When running >> run-dev-solr.sh >> that version is downloaded, solrhome is copied to the right place and the >> instance is started. >> > > I grabbed the git repo and proceeded to fiddle with it. > > As you said, it works when the solr version that is downloaded is 6.6.2, > but fails with 7.2.1. Based on what gets logged and what I know about how > Solr does initialization, I'm betting the bootstrap feature is broken. I > can't tell what's wrong, but I think that's probably where the problem is. > > But as I already said, I strongly recommend that you don't use that > feature. At all. > > The cores that are in the git repo are empty -- there's no index. If the > source cores are empty when you run the script "for real", then I have an > alternate idea for you to try: > > Instead of copying the cores and bootstrapping a non-cloud install into a > cloud install: Remove the bootstrap option. Create the collections using > "bin/solr create", with -replicationFactor 1 -shards 1, and the -d option > pointing at the config you want uploaded for the collection. You could > even go with a higher shard count if you want. Increasing > replicationFactor is probably possible, but as all this would be running on > a single server, there's no point. > > Thanks, > Shawn > > -- med vennlig hilsen, Marvin B. Lillehaug
security authentication API via solrj?
Hi, I was wondering if 7.2.1 solrj had native support for the security/authentication endpoint? I couldn't find anything in the docs about it, but maybe someone has some experience with it? Note: This is about adding/deleting users on a solr instance using solrj, not authenticating (that is well documented). Thanks, Peter
Solr 7. Why UpdateRequestProcessorChain does not take any effect?
Hello,
Could you please help, for some reason my custom updateRequestProcessorChain
does not take any effect.
I am using Solr v.7.2.1 with two nodes.
I have the following schema:
In my solrconfig.xml I added:
I enabled TRACE logging for UpdateRequestProcessorChain,
CloneFieldUpdateProcessorFactory and DistributedUpdateProcessorFactory (but
still cannot see this in the logs).
After I add a new document like {"text_en":"test"}, the field
"text_en_exact" is never populated with this value by
CloneFieldUpdateProcessorFactory.
What am I doing wrong?
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr 7. Why UpdateRequestProcessorChain does not take any effect?
Also in the logs I can see that my custom "mychain" processor is registered on both nodes: -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr 7. Why UpdateRequestProcessorChain does not take any effect?
On 2/25/2018 2:10 PM, FiMko wrote:
Could you please help, for some reason my custom updateRequestProcessorChain
does not take any effect.
I am using Solr v.7.2.1 with two nodes.
I have the following schema:
In my solrconfig.xml I added:
I enabled TRACE logging for UpdateRequestProcessorChain,
CloneFieldUpdateProcessorFactory and DistributedUpdateProcessorFactory (but
still cannot see this in the logs).
After I add a new document like {"text_en":"test"}, the field
"text_en_exact" is never populated with this value by
CloneFieldUpdateProcessorFactory.
The things you referenced did not make it to the mailing list. What I
have quoted above is exactly what made it to the mailing list.
By going to the Nabble forum (which is a mirror of the list) I was able
to see the two things you referenced. Nabble does not always send
everything that its users include to the list.
The logs you mentioned in your second message aren't there, not even on
Nabble.
The information you included as your schema is not useful. If we end up
needing that information, and I'm not sure we will, then we need the
actual schema contents, not information from the Schema tab in the admin UI.
The information from solrconfig.xml is just an update chain definition.
Unless you *activate* that update chain, it will not take effect. You
can only have one update chain active, so if you're currently using the
"add-unknown-fields" chain commonly found in examples, that will no
longer be running once you activate this new one.
Why are you using an update chain to do what the schema can do natively
with copyField?
Thanks,
Shawn
Re: security authentication API via solrj?
On 2/25/2018 1:28 PM, Peter Sturge wrote: I was wondering if 7.2.1 solrj had native support for the security/authentication endpoint? I couldn't find anything in the docs about it, but maybe someone has some experience with it? Note: This is about adding/deleting users on a solr instance using solrj, not authenticating (that is well documented). At first I was looking for how to use authentication with SolrJ. I came up with this: Looks like this is not available when using the sugar objects like SolrQuery. To use authentication, it seems you have to create the request objects yourself. https://lucene.apache.org/solr/guide/7_2/basic-authentication-plugin.html#using-basic-auth-with-solrj Then I noticed you were talking about the actual security endpoint -- adding users. I have been looking over the objects available in SolrJ, and I do not see anything useful. It looks like you might need a new request object class, implemented similar to DirectXMLRequest, but using JSON and not XML. It might be possible to make it an implicitly defined class rather than creating a whole class file. A proper sugar class for handling the security endpoint should be created. I would do it, but I'm not sure how. Thanks, Shawn
Re: Indexing timeout issues with SolrCloud 7.1
DBQ is something of a heavyweight action. Basically in order to preserve ordering it has to lock out updates while it executes since all docs (which may live on all shards) have to be deleted before subsequent adds of one of the affected docs is processed. In order to do that, things need to be locked. Delete-by-id OTOH, can use the normal optimistic locking to insure proper ordering. So if object_id is your , this may be much more robust if you delete-by-id Best, Erick On Sat, Feb 24, 2018 at 1:37 AM, Deepak Goel wrote: > From the error list, i can see multiple errors: > > 1. Failure to recover replica > 2. Peer sync error > 3. Failure to download file > > On 24 Feb 2018 03:10, "Tom Peters" wrote: > > I included the last 25 lines from the logs from each of the five nodes > during that time period. > > I _think_ I'm running into issues with bulking up deleteByQuery. Quick > background: we have objects in our system that may have multiple > availability windows. So when we index an object, will store it as separate > documents each with their own begins and expires date. At index time we > don't know if the all of the windows are still valid or not, so we remove > all of them with a deleteByQuery (e.g. deleteByQuery=object_id:12345) and > then index one or more documents. > > I ran an isolated test a number of times where I indexed 1500 documents in > this manner (deletes then index). In Solr 3.4, it takes about 15s to > complete. In Solr 7.1, it's taking about 5m. If I remove the deleteByQuery, > the indexing times are nearly identical. > > When run in normal production mode where we have lots of processes indexing > at once (~20 or so), it starts to cause lots of issues (which you see > below). > > > Please let me know if anything I mentioned is unclear. Thanks! > > > > > solr2-a: > 2018-02-23 04:09:36.551 ERROR (updateExecutor-2-thread-2672- > processing-http:solr2-b:8080//solr//mycollection_shard1_replica_n1 > x:mycollection_shard1_replica_n6 r:core_node9 n:solr2-a.vam.be.cmh. > mycollection.com:8080_solr s:shard1 c:mycollection) [c:mycollection > s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u. > ErrorReportingConcurrentUpdateSolrClient error > 2018-02-23 04:09:36.551 ERROR (updateExecutor-2-thread-2692- > processing-http:solr2-d:8080//solr//mycollection_shard1_replica_n11 > x:mycollection_shard1_replica_n6 r:core_node9 n:solr2-a.vam.be.cmh. > mycollection.com:8080_solr s:shard1 c:mycollection) [c:mycollection > s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u. > ErrorReportingConcurrentUpdateSolrClient error > 2018-02-23 04:09:36.551 ERROR (updateExecutor-2-thread-2711- > processing-http:solr2-e:8080//solr//mycollection_shard1_replica_n4 > x:mycollection_shard1_replica_n6 r:core_node9 n:solr2-a.vam.be.cmh. > mycollection.com:8080_solr s:shard1 c:mycollection) [c:mycollection > s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u. > ErrorReportingConcurrentUpdateSolrClient error > 2018-02-23 04:09:36.552 ERROR (qtp1595212853-32739) [c:mycollection > s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] > o.a.s.u.p.DistributedUpdateProcessor > Setting up to try to start recovery on replica http://solr2-b:8080/solr/ > mycollection_shard1_replica_n1/ > 2018-02-23 04:09:36.552 ERROR (qtp1595212853-32739) [c:mycollection > s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] > o.a.s.u.p.DistributedUpdateProcessor > Setting up to try to start recovery on replica http://solr2-d:8080/solr/ > mycollection_shard1_replica_n11/ > 2018-02-23 04:09:36.552 ERROR (qtp1595212853-32739) [c:mycollection > s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] > o.a.s.u.p.DistributedUpdateProcessor > Setting up to try to start recovery on replica http://solr2-e:8080/solr/ > mycollection_shard1_replica_n4/ > 2018-02-23 04:09:38.217 ERROR (updateExecutor-2-thread-2712- > processing-http:solr2-e:8080//solr//mycollection_shard1_replica_n4 > x:mycollection_shard1_replica_n6 r:core_node9 n:solr2-a.vam.be.cmh. > mycollection.com:8080_solr s:shard1 c:mycollection) [c:mycollection > s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u. > ErrorReportingConcurrentUpdateSolrClient error > 2018-02-23 04:09:38.217 ERROR (updateExecutor-2-thread-2726- > processing-http:solr2-d:8080//solr//mycollection_shard1_replica_n11 > x:mycollection_shard1_replica_n6 r:core_node9 n:solr2-a.vam.be.cmh. > mycollection.com:8080_solr s:shard1 c:mycollection) [c:mycollection > s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u. > ErrorReportingConcurrentUpdateSolrClient error > 2018-02-23 04:09:38.217 ERROR (updateExecutor-2-thread-2727- > processing-http:solr2-b:8080//solr//mycollection_shard1_replica_n1 > x:mycollection_shard1_replica_n6 r:core_node9 n:solr2-a.vam.be.cmh. > mycollection.com:8080_solr s:shard1 c:mycollection) [c:mycollection > s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u. > ErrorReportingConcurrentUpdateSolrClient erro
Re: Could not find collection when upgrading to Solr 7
Bootstrapping is not broken but the default in Solr 7 is to set the cluster property legacyCloud=false which means that bootstrapped cores cannot register new collections automatically anymore. If you want to rely on the old behavior then you need to set the cluster property legacyCloud=true. This was also mentioned in the upgrade section for Solr 7 in CHANGES.txt On Mon, Feb 26, 2018 at 1:28 AM, Marvin Bredal Lillehaug < marvin.lilleh...@gmail.com> wrote: > Thank your for your reply. > I will take a look at both the code for bootstrapping, and look at what you > are suggesting to determine what to do. > > On Sun, Feb 25, 2018 at 5:17 PM, Shawn Heisey > wrote: > > > On 2/24/2018 8:05 AM, Marvin Bredal Lillehaug wrote: > > > >> This is for doing local development and running tests. > >> So starting with embedded zookeeper in SolrCloud mode is intentional. > The > >> idea is to just run the script setting up the folder structure for the > >> cores and everything is up and running, ready for indexing and querying > >> locally. > >> The applications using Solr also use Zookeeper for configuration and > >> coordination of their own stuff. > >> > >> There are no persmission issues. > >> > >> I have put the scripts here https://github.com/computerlove/start-solr > >> > >> The file config specifies what version to use. When running > >> run-dev-solr.sh > >> that version is downloaded, solrhome is copied to the right place and > the > >> instance is started. > >> > > > > I grabbed the git repo and proceeded to fiddle with it. > > > > As you said, it works when the solr version that is downloaded is 6.6.2, > > but fails with 7.2.1. Based on what gets logged and what I know about > how > > Solr does initialization, I'm betting the bootstrap feature is broken. I > > can't tell what's wrong, but I think that's probably where the problem > is. > > > > But as I already said, I strongly recommend that you don't use that > > feature. At all. > > > > The cores that are in the git repo are empty -- there's no index. If the > > source cores are empty when you run the script "for real", then I have an > > alternate idea for you to try: > > > > Instead of copying the cores and bootstrapping a non-cloud install into a > > cloud install: Remove the bootstrap option. Create the collections using > > "bin/solr create", with -replicationFactor 1 -shards 1, and the -d option > > pointing at the config you want uploaded for the collection. You could > > even go with a higher shard count if you want. Increasing > > replicationFactor is probably possible, but as all this would be running > on > > a single server, there's no point. > > > > Thanks, > > Shawn > > > > > > > -- > med vennlig hilsen, > Marvin B. Lillehaug > -- Regards, Shalin Shekhar Mangar.
Re: Could not find collection when upgrading to Solr 7
Ah, didn't get that legacyCloud was related to bootstrap Have a new script working without bootstrapping. Thank you for your input! On Mon, Feb 26, 2018 at 7:36 AM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > Bootstrapping is not broken but the default in Solr 7 is to set the cluster > property legacyCloud=false which means that bootstrapped cores cannot > register new collections automatically anymore. If you want to rely on the > old behavior then you need to set the cluster property legacyCloud=true. > This was also mentioned in the upgrade section for Solr 7 in CHANGES.txt > > On Mon, Feb 26, 2018 at 1:28 AM, Marvin Bredal Lillehaug < > marvin.lilleh...@gmail.com> wrote: > > > Thank your for your reply. > > I will take a look at both the code for bootstrapping, and look at what > you > > are suggesting to determine what to do. > > > > On Sun, Feb 25, 2018 at 5:17 PM, Shawn Heisey > > wrote: > > > > > On 2/24/2018 8:05 AM, Marvin Bredal Lillehaug wrote: > > > > > >> This is for doing local development and running tests. > > >> So starting with embedded zookeeper in SolrCloud mode is intentional. > > The > > >> idea is to just run the script setting up the folder structure for the > > >> cores and everything is up and running, ready for indexing and > querying > > >> locally. > > >> The applications using Solr also use Zookeeper for configuration and > > >> coordination of their own stuff. > > >> > > >> There are no persmission issues. > > >> > > >> I have put the scripts here https://github.com/ > computerlove/start-solr > > >> > > >> The file config specifies what version to use. When running > > >> run-dev-solr.sh > > >> that version is downloaded, solrhome is copied to the right place and > > the > > >> instance is started. > > >> > > > > > > I grabbed the git repo and proceeded to fiddle with it. > > > > > > As you said, it works when the solr version that is downloaded is > 6.6.2, > > > but fails with 7.2.1. Based on what gets logged and what I know about > > how > > > Solr does initialization, I'm betting the bootstrap feature is > broken. I > > > can't tell what's wrong, but I think that's probably where the problem > > is. > > > > > > But as I already said, I strongly recommend that you don't use that > > > feature. At all. > > > > > > The cores that are in the git repo are empty -- there's no index. If > the > > > source cores are empty when you run the script "for real", then I have > an > > > alternate idea for you to try: > > > > > > Instead of copying the cores and bootstrapping a non-cloud install > into a > > > cloud install: Remove the bootstrap option. Create the collections > using > > > "bin/solr create", with -replicationFactor 1 -shards 1, and the -d > option > > > pointing at the config you want uploaded for the collection. You could > > > even go with a higher shard count if you want. Increasing > > > replicationFactor is probably possible, but as all this would be > running > > on > > > a single server, there's no point. > > > > > > Thanks, > > > Shawn > > > > > > > > > > > > -- > > med vennlig hilsen, > > Marvin B. Lillehaug > > > > > > -- > Regards, > Shalin Shekhar Mangar. > -- med vennlig hilsen, Marvin B. Lillehaug
