Delay in replication between cloud servers
We are using solr cloud with 1 shard and replication factor as 3. We are noticing that the time for data to become available across all replicas from the leader is very high. The data rate is not very high, is there anyway to control this. In master-slave setup with give a replication time. Regards,Rohit
RE: Delay in replication between cloud servers
Further we have noticed that the delay increase a couple of hours after restart. Details related to sorlconfig.xml are given below, 15000 25000 false 1000 Regards,Rohit > From: cooltec...@outlook.com > To: solr-user@lucene.apache.org > Subject: Delay in replication between cloud servers > Date: Tue, 16 Feb 2016 20:20:04 +0530 > > We are using solr cloud with 1 shard and replication factor as 3. We are > noticing that the time for data to become available across all replicas from > the leader is very high. > The data rate is not very high, is there anyway to control this. In > master-slave setup with give a replication time. > Regards,Rohit >
RE: Delay in replication between cloud servers
Hi, Can someone point in the right direction to check this. Regards,Rohit > From: cooltec...@outlook.com > To: solr-user@lucene.apache.org > Subject: RE: Delay in replication between cloud servers > Date: Tue, 16 Feb 2016 22:16:23 +0530 > > Further we have noticed that the delay increase a couple of hours after > restart. Details related to sorlconfig.xml are given below, > >15000 >25000 >false > > > > 1000 >Regards,Rohit > > > From: cooltec...@outlook.com > > To: solr-user@lucene.apache.org > > Subject: Delay in replication between cloud servers > > Date: Tue, 16 Feb 2016 20:20:04 +0530 > > > > We are using solr cloud with 1 shard and replication factor as 3. We are > > noticing that the time for data to become available across all replicas > > from the leader is very high. > > The data rate is not very high, is there anyway to control this. In > > master-slave setup with give a replication time. > > Regards,Rohit > > >
Solr Cloud leader propagates slow to other nodes
Hi, We are using solr 4.8.1 cloud and have a single shard core and 3 replicas in total. We are noticing the following issues, 1) Master propagates slow to to the other nodes/replicas.2) This increase over a period of time, i.e the data propagation time increase as the time from which leader is elected increases? What could be causing this, our commit settings are given below. 12000 25000 false 1000 Regards,Ayush
RE: Solr Cloud leader propagates slow to other nodes
Any suggestion here would be highly appreciated, this is hampering our production performance and we are not able to pin point any reason. > From: cooltec...@outlook.com > To: solr-user@lucene.apache.org > Subject: Solr Cloud leader propagates slow to other nodes > Date: Fri, 4 Mar 2016 18:16:07 +0530 > > Hi, > We are using solr 4.8.1 cloud and have a single shard core and 3 replicas in > total. We are noticing the following issues, > 1) Master propagates slow to to the other nodes/replicas.2) This increase > over a period of time, i.e the data propagation time increase as the time > from which leader is elected increases? > What could be causing this, our commit settings are given below. > >12000 >25000 >false > > > > > 1000 > > Regards,Ayush > > > > > > > > > >
Search in URL
Hi, We have several tweets stored in our system which are of the following nature, https://twitter.com/SharFairies/status/448992742257881088 I am searching for keywords like starcinema or abs-cbn.com, but these results are not returned, how can I search in this setup. My field type is defined as below,
Fails to index if unique field has special characters
Hi, We are migrating from Solr 4.6 standalone to Solr 4.7 cloud version, while reindexing the document we are getting the following error. This is happening when the unique key has special character, this was not noticed in version 4.6 standalone mode, so we are not sure if this is a version problem or a cloud issue. Example of the unique key is given below, http://www.mynews.in/Blog/smrity!!**)))!miami_dolphins_vs_dallas_cowboys_live_stream_on_line_nfl_football_free_video_broadcast_B142707.html Exception Stack Trace ERROR - 2014-04-10 10:51:44.361; org.apache.solr.common.SolrException; java.lang.ArrayIndexOutOfBoundsException: 2 at org.apache.solr.common.cloud.CompositeIdRouter$KeyParser.getHash(CompositeIdRouter.java:296) at org.apache.solr.common.cloud.CompositeIdRouter.sliceHash(CompositeIdRouter.java:58) at org.apache.solr.common.cloud.HashBasedRouter.getTargetSlice(HashBasedRouter.java:33) at org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:218) at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:550) at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100) at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:247) at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1916) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:780) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:427) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:217) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandle Thanks,Ayush
RE: Fails to index if unique field has special characters
Thanks, that was helpful.
Regards,Rohit
> Date: Thu, 10 Apr 2014 08:44:36 -0700
> From: iori...@yahoo.com
> Subject: Re: Fails to index if unique field has special characters
> To: solr-user@lucene.apache.org
>
> Hi Ayush,
>
> I thinks this
>
> ""IBM!12345". The exclamation mark ('!') is critical here, as it
> distinguishes the prefix used to determine which shard to direct the document
> to."
>
> https://cwiki.apache.org/confluence/display/solr/Shards+and+Indexing+Data+in+SolrCloud
>
>
>
>
> On Thursday, April 10, 2014 2:35 PM, Cool Techi
> wrote:
> Hi,
> We are migrating from Solr 4.6 standalone to Solr 4.7 cloud version, while
> reindexing the document we are getting the following error. This is happening
> when the unique key has special character, this was not noticed in version
> 4.6 standalone mode, so we are not sure if this is a version problem or a
> cloud issue. Example of the unique key is given below,
> http://www.mynews.in/Blog/smrity!!**)))!miami_dolphins_vs_dallas_cowboys_live_stream_on_line_nfl_football_free_video_broadcast_B142707.html
> Exception Stack Trace
> ERROR - 2014-04-10 10:51:44.361; org.apache.solr.common.SolrException;
> java.lang.ArrayIndexOutOfBoundsException: 2 at
> org.apache.solr.common.cloud.CompositeIdRouter$KeyParser.getHash(CompositeIdRouter.java:296)
>at
> org.apache.solr.common.cloud.CompositeIdRouter.sliceHash(CompositeIdRouter.java:58)
>at
> org.apache.solr.common.cloud.HashBasedRouter.getTargetSlice(HashBasedRouter.java:33)
>at
> org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:218)
>at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:550)
>at
> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
>at
> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:247)
>at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174)
> at
>
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
>at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>at org.apache.solr.core.SolrCore.execute(SolrCore.java:1916) at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:780)
>at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:427)
>at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:217)
>at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
>at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
>at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at org.eclipse.jetty.server.session.SessionHandle
>
> Thanks,Ayush
RE: Inconsistent response from Cloud Query
We have noticed Solr returns in-consistent results during replica recovery and not all replicas are in the same state, so when your query goes to a replica which might be recovering or still copying the index then the counts may differ. regards,Ayush > Date: Tue, 6 May 2014 16:14:36 +0530 > Subject: Inconsistent response from Cloud Query > From: clearmido...@gmail.com > To: solr-user@lucene.apache.org > > Hi All, > > I have setup cloud-4.6.2 with default configuration on single machine with > 2 shards and 2 replication through > https://cwiki.apache.org/confluence/display/solr/Getting+Started+with+SolrCloud > > Cloud was up and running and I indexed the example data xml to it, it went > fine. > Now when I am querying with *distrib=true* it is giving inconsistent > result, sometimes it gives 4 Result Response and sometimes 8(Actual Number) > > Has anyone been through the situation. Looking for positive and quick > response. > > Thanks!
Storing tweets For WC2014
Hi, We have a requirement from one of our customers to provide search and analytics on the upcoming Soccer World cup, given the sheer volume of tweet's that would be generated at such an event I cannot imagine what would be required to store this in solr. It would be great if there can be some pointer's on the scale or hardware required, number of shards that should be created etc. Some requirement, All the tweets should be searchable (approximately 100million tweets/date * 60 Days of event). All fields on tweets should be searchable/facet on numeric and date fields. Facets would be run on TwitterId's (unique users), tweet created on date, Location, Sentiment (some fields which we generate) If anyone has attempted anything like this it would be helpful. Regards,Rohit
Inconsistant result's on solr cloud 4.8
Hi, We are using solr cloud with solr version 4.8, we have 2 shard/2 replica servers in Solr Cloud. During two consecutive request to the solr cloud, the "total results" number varies. 1) As per my understanding this can happen when the leader and the replica have inconsistant number of results. 2) This inconsistant number of docs between leader and replica can happen only when replica is recovering. Should a request be sent to a node which is recovering. Since this is happening on our live setup, we tend to question how much can we rely on solr. What could be causing this and what's the fix. Regards
SolrCloud Issues
Hi, We have moved to SolrCloud (version 4.8) and are facing several issues in this setup compared to the master/slave setup we have had for a while now, 1) We have a 2 shard set-up with one replica each, we notice that most of the time the replica's are in recovering status. What could be causing this? 2) In-consistant results count, especially when one of the nodes is recovering, I have asked another question earlier on this. To our understanding the recovering node, doesn't returns any results, what else can cause this? 3) Solr node goes down very frequently, there is no OOM or otherwise in the logs, but a node keeps going down. Also, at times we have noticed that tomcat stops responding. Since there are so many parts to solr cloud, checking what's causing the issue is difficult in a quick time, so if anyone else has faced this, it would be very helpful. Since this is happening on our UAT environment, we need to get a fix soon. Regards,Ayush
RE: SolrCloud Issues
1) The Zookeepers are on the same node as Solr, should we move them out? What would be the basic config's of machine just running Zookeepers? 2) The servers are pretty big, 2 X Quad Core processors 64 bit 96Gb of RAM 500GB SSD drive on which solr resides The index side between the two shards is 250GB, how can we detect performance problems if any. Also, what keep the node in recovering mode for most of the time. We index about 2K documents/ minute. What would be the ideal configuration for such load and increasing index size. Regards,Ayush > From: dan.mur...@buy4now.com > To: solr-user@lucene.apache.org > Subject: RE: SolrCloud Issues > Date: Thu, 17 Jul 2014 13:27:17 + > > Have you deployed ZooKeeper on servers other than the Solr nodes? > If you have them on the Solr nodes, then you may be getting elections when > under pressure. > > -Original Message- > From: Shawn Heisey [mailto:s...@elyograg.org] > Sent: 17 July 2014 14:25 > To: solr-user@lucene.apache.org > Subject: Re: SolrCloud Issues > > On 7/17/2014 6:57 AM, Cool Techi wrote: > > We have moved to SolrCloud (version 4.8) and are facing several issues > > in this setup compared to the master/slave setup we have had for a > > while now, > > 1) We have a 2 shard set-up with one replica each, we notice that most of > > the time the replica's are in recovering status. What could be causing this? > > 2) In-consistant results count, especially when one of the nodes is > > recovering, I have asked another question earlier on this. To our > > understanding the recovering node, doesn't returns any results, what else > > can cause this? > > 3) Solr node goes down very frequently, there is no OOM or otherwise in the > > logs, but a node keeps going down. Also, at times we have noticed that > > tomcat stops responding. Since there are so many parts to solr cloud, > > checking what's causing the issue is difficult in a quick time, so if > > anyone else has faced this, it would be very helpful. > > I don't have anything specific for you, but if you are having any kind of > performance issues at all, it can lead to bizarre SolrCloud behavior. > > The basic zookeeper client timeout defaults to 15 seconds. This is a very > long timeout, but if anything is happening that makes any part of SolrCloud > wait longer than 15 seconds, SolrCloud will think there's a problem that > needs recovery. > > Here's a summary of common performance problems and some possible solutions: > > http://wiki.apache.org/solr/SolrPerformanceProblems > > Thanks, > Shawn >
Solr Result Tagging
Hi, My search queries to solr are of the following nature, (A OR B OR C) OR (X AND Y AND Z) OR ((ABC AND DEF) - XYZ) What I am trying to achieve is when I fire the query the results returned should be able to tagged with which part or the OR resulted in the result. In case all three parts above are applicable then the result should indicate the same. I tried group.query feature, but doesn't seem like it works on solr cloud. Thanks,Ayush
Faceting within groups
Hi,
I am not sure if faceting with groups is supported, the documents do seem to
suggest it works, but cant seem to get the intended results.
("Amazon Cloud" OR ("IBM Cloud")sourceIdsentimenttruetrue
Also, if it work's does solr cloud support it.
Regards,Ayush
RE: Faceting within groups
Any help here?
> From: cooltec...@outlook.com
> To: solr-user@lucene.apache.org
> Subject: Faceting within groups
> Date: Sat, 7 Dec 2013 14:00:20 +0530
>
> Hi,
> I am not sure if faceting with groups is supported, the documents do seem to
> suggest it works, but cant seem to get the intended results.
> ("Amazon Cloud" OR ("IBM Cloud") name="group.field">sourceIdsentiment name="group">truetrue
> Also, if it work's does solr cloud support it.
> Regards,Ayush
Unknow Query exception
I am seeing the following exception in solr logs after regular intervals, we
are not firing any query to have such error, what could be causing this?
ERROR - 2013-12-14 05:35:52.722; org.apache.solr.common.SolrException;
org.apache.solr.common.SolrException: org.apache.solr.search.SyntaxError:
Cannot parse 'docKey:()': Encountered " ")" ") "" at line 1, column 8.
Was expecting one of:
...
"+" ...
"-" ...
...
"(" ...
"*" ...
...
...
...
...
...
"[" ...
"{" ...
...
...
...
"*" ...
at
org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:147)
Solr replicating at 5 mb/sec
Hi, I am running solr replication between two machine which are connected by a 1 gb network speed. The best speed I am getting for replication is 5mb/sec, how can this be increased. The replication keeps failing and this is the first time replication of an index over 300Gb in size. We are using solr verson 4.1 on master and solr 4.3 on slave. Regards,Ayush
Solr 4.3 Master/Slave Issues
We have set up solr 4.3 with master/setup and are facing a couple of issues, Index locking, the index on slave hangs at time and when we restart the core the core get's locked up. I have checked the logs and there are no OOM error or anything else other than the error given below,Caused by: org.apache.solr.common.SolrException: Index locked for write for core XXX at org.apache.solr.core.SolrCore.(SolrCore.java:821) at org.apache.solr.core.SolrCore.(SolrCore.java:618) at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984) at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597) at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ... 1 more Caused by: org.apache.lucene.store.LockObtainFailedException: Index locked for write for core XXat org.apache.solr.core.SolrCore.initIndex(SolrCore.java:484) at org.apache.solr.core.SolrCore.(SolrCore.java:745) ... 13 moreWhen we fire backup on Slave, the whole core hangs after a while and also replication stops. This was not happening with our earlier setup of solr3.6 Regards, Ayush
RE: Solr 4.3 Master/Slave Issues
1) That was my initial suspicion, but when I run ps -aux | grep "java", but there it doesn't show any other program running. I kill the process and start again and it locks. 2) When we fire backup on Slave, the whole core hangs after a while and also replication stops. This was not happening with our earlier setup of solr3.6 Regards, > Date: Fri, 5 Jul 2013 16:49:24 +0530 > Subject: Re: Solr 4.3 Master/Slave Issues > From: shalinman...@gmail.com > To: solr-user@lucene.apache.org > > This can mean multiple things: > 1. You had killed a solr process earlier which left the lock file in place > 2. You have more than one Solr core pointing to the same data directory > 3. A solr process is already running and you are trying to start > another one with the same config. > > On Fri, Jul 5, 2013 at 2:34 PM, Cool Techi wrote: > > We have set up solr 4.3 with master/setup and are facing a couple of issues, > > > > Index locking, the index on slave hangs at time and when we restart the > > core the core get's locked up. I have checked the logs and there are no OOM > > error or anything else other than the error given below,Caused by: > > org.apache.solr.common.SolrException: Index locked for write for core XXX > > at org.apache.solr.core.SolrCore.(SolrCore.java:821) > > at org.apache.solr.core.SolrCore.(SolrCore.java:618) > > at > > org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949) > > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984) > > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597) > > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > ... 1 more > > Caused by: org.apache.lucene.store.LockObtainFailedException: Index locked > > for write for core XXat > > org.apache.solr.core.SolrCore.initIndex(SolrCore.java:484) > > at org.apache.solr.core.SolrCore.(SolrCore.java:745) > > ... 13 moreWhen we fire backup on Slave, the whole core hangs after a > > while and also replication stops. This was not happening with our earlier > > setup of solr3.6 > > > > > > > > Regards, > > Ayush > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Regards, > Shalin Shekhar Mangar.
RE: Solr 4.3 Master/Slave Issues
The normal tomcat shutdown doesn't stop the server and take a long time, so i do issue a kill -9 command. Any other suggestion to do this without the locking. I would initiate a backup again and send the logs. regards, Ayush > Date: Fri, 5 Jul 2013 19:40:12 +0530 > Subject: Re: Solr 4.3 Master/Slave Issues > From: shalinman...@gmail.com > To: solr-user@lucene.apache.org > > On Fri, Jul 5, 2013 at 6:14 PM, Cool Techi wrote: > > > > 1) That was my initial suspicion, but when I run ps -aux | grep "java", but > > there it doesn't show any other program running. I kill the process and > > start again and it locks. > > How are you killing the process? A SIGKILL will leave a lock file. > > > > > 2) When we fire backup on Slave, the whole core hangs after a while and > > also replication stops. This was not happening with our earlier setup of > > solr3.6 > > > > Just to confirm, are you using the backup command of replication > handler? Can you post a thread dump of the process to pastebin or > http://apaste.info/ ? > > > Regards, > > > > > >> Date: Fri, 5 Jul 2013 16:49:24 +0530 > >> Subject: Re: Solr 4.3 Master/Slave Issues > >> From: shalinman...@gmail.com > >> To: solr-user@lucene.apache.org > >> > >> This can mean multiple things: > >> 1. You had killed a solr process earlier which left the lock file in place > >> 2. You have more than one Solr core pointing to the same data directory > >> 3. A solr process is already running and you are trying to start > >> another one with the same config. > >> > >> On Fri, Jul 5, 2013 at 2:34 PM, Cool Techi wrote: > >> > We have set up solr 4.3 with master/setup and are facing a couple of > >> > issues, > >> > > >> > Index locking, the index on slave hangs at time and when we restart the > >> > core the core get's locked up. I have checked the logs and there are no > >> > OOM error or anything else other than the error given below,Caused by: > >> > org.apache.solr.common.SolrException: Index locked for write for core XXX > >> > at org.apache.solr.core.SolrCore.(SolrCore.java:821) > >> > at org.apache.solr.core.SolrCore.(SolrCore.java:618) > >> > at > >> > org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949) > >> > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984) > >> > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597) > >> > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592) > >> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > >> > at > >> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > >> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > >> > at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >> > at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >> > ... 1 more > >> > Caused by: org.apache.lucene.store.LockObtainFailedException: Index > >> > locked for write for core XXat > >> > org.apache.solr.core.SolrCore.initIndex(SolrCore.java:484) > >> > at org.apache.solr.core.SolrCore.(SolrCore.java:745) > >> > ... 13 moreWhen we fire backup on Slave, the whole core hangs after > >> > a while and also replication stops. This was not happening with our > >> > earlier setup of solr3.6 > >> > > >> > > >> > > >> > Regards, > >> > Ayush > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > >> > >> > >> -- > >> Regards, > >> Shalin Shekhar Mangar. > > > > > > -- > Regards, > Shalin Shekhar Mangar.
Backup stops replication
Hi, We migrated from solr 3.6 to solr 4.3, when we fire a backup command (/replication?command=backup&location=/disk4/backups ) on the salve the slave stops replicating OR starts full replication from the master. This was not the behavior in the earlier version or solr, I have check the logs but dont see anything strange. Correct me if I am wrong but isn't replication handler supposed to be async. I want the backup to be async how can this be achieved. Regards, Ayush
RE: Solr 4.3 Master/Slave Issues
thanks Erick. My tomcat setup is running on Ubuntu and has nothing else deployed other than the Solr war. My suspicion is it takes a long time to de-allocate the memory it has reserved for itself, but will get dump to find better. regards, Ayush > Date: Sat, 6 Jul 2013 07:55:57 -0400 > Subject: Re: Solr 4.3 Master/Slave Issues > From: erickerick...@gmail.com > To: solr-user@lucene.apache.org > > kill -9 is evil, you aren't certain what the state is afterwards, > so the presence of the lock file is not surprising.. > > solrconfig.xml has a commented-out entry > > false > > I haven't personally used it, but it looks like it might > help if you insist on kill -9. > > Personally I'd try to figure out why your setup takes > so long to shut down, I know lots of Tomcat installations > that seem to function so this might point to something > else in your setup that needs fixing. > > Best > Erick > > > On Fri, Jul 5, 2013 at 11:54 AM, Cool Techi wrote: > > > The normal tomcat shutdown doesn't stop the server and take a long time, > > so i do issue a kill -9 command. Any other suggestion to do this without > > the locking. > > > > I would initiate a backup again and send the logs. > > > > regards, > > Ayush > > > > > Date: Fri, 5 Jul 2013 19:40:12 +0530 > > > Subject: Re: Solr 4.3 Master/Slave Issues > > > From: shalinman...@gmail.com > > > To: solr-user@lucene.apache.org > > > > > > On Fri, Jul 5, 2013 at 6:14 PM, Cool Techi > > wrote: > > > > > > > > 1) That was my initial suspicion, but when I run ps -aux | grep > > "java", but there it doesn't show any other program running. I kill the > > process and start again and it locks. > > > > > > How are you killing the process? A SIGKILL will leave a lock file. > > > > > > > > > > > 2) When we fire backup on Slave, the whole core hangs after a while > > and also replication stops. This was not happening with our earlier setup > > of solr3.6 > > > > > > > > > > Just to confirm, are you using the backup command of replication > > > handler? Can you post a thread dump of the process to pastebin or > > > http://apaste.info/ ? > > > > > > > Regards, > > > > > > > > > > > >> Date: Fri, 5 Jul 2013 16:49:24 +0530 > > > >> Subject: Re: Solr 4.3 Master/Slave Issues > > > >> From: shalinman...@gmail.com > > > >> To: solr-user@lucene.apache.org > > > >> > > > >> This can mean multiple things: > > > >> 1. You had killed a solr process earlier which left the lock file in > > place > > > >> 2. You have more than one Solr core pointing to the same data > > directory > > > >> 3. A solr process is already running and you are trying to start > > > >> another one with the same config. > > > >> > > > >> On Fri, Jul 5, 2013 at 2:34 PM, Cool Techi > > wrote: > > > >> > We have set up solr 4.3 with master/setup and are facing a couple > > of issues, > > > >> > > > > >> > Index locking, the index on slave hangs at time and when we restart > > the core the core get's locked up. I have checked the logs and there are no > > OOM error or anything else other than the error given below,Caused by: > > org.apache.solr.common.SolrException: Index locked for write for core XXX > > > >> > at org.apache.solr.core.SolrCore.(SolrCore.java:821) > > > >> > at org.apache.solr.core.SolrCore.(SolrCore.java:618) > > > >> > at > > org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949) > > > >> > at > > org.apache.solr.core.CoreContainer.create(CoreContainer.java:984) > > > >> > at > > org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597) > > > >> > at > > org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592) > > > >> > at > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > > >> > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > &
solr cloud index corruption
Hi, We are frequently getting issues of index corruption on the cloud, this used to not happen in our master slave setup with solr 3.6. I have tried to check the logs, but don't see an exact reason. I have run the index checker and it recovers, but I am not able to understand as to why this is happening. Any pointers would help. regards, rohit
Autowarming last 15 days data
Hi, We have a solr master slave set up with close to 30 million records. Our index changes/updates very frequently and replication is set up at 60 seconds delay. Now every time replication completes, the new searches take a time. How can this be improved? I have come across that warming would help this scenario, I our case we cannot warm some queries, but most of the users use the last 15 days data only. So would it be possible to auto warm only last 15 days data? Regards, Ayush
RE: Autowarming last 15 days data
Would it make sense if we open a newSearcher with the last 15 days documents? since these is the documents which are mostly used by the users. Also, how could i do this if this is possible? > Date: Wed, 31 Jul 2013 08:54:47 -0600 > From: s...@elyograg.org > To: solr-user@lucene.apache.org > Subject: Re: Autowarming last 15 days data > > On 7/31/2013 7:30 AM, Cool Techi wrote: > > We have a solr master slave set up with close to 30 million records. Our > > index changes/updates very frequently and replication is set up at 60 > > seconds delay. > > > > Now every time replication completes, the new searches take a time. How can > > this be improved? I have come across that warming would help this scenario, > > I our case we cannot warm some queries, but most of the users use the last > > 15 days data only. > > > > So would it be possible to auto warm only last 15 days data? > > Autowarming is generally done automatically when a new searcher is > opened, according to the cache config. It will take the most recent N > queries in the cache (according to the autowarmCount) and re-execute > those queries against the index to populate the cache. The document > cache cannot be warmed directly, but when the query result cache is > warmed, that will also populate the document cache. > > Because you have a potentially very frequent interval for opening new > searchers (possibly replicating every 60 seconds), you will want to > avoid large autowarmCount values. If your autowarming ends up taking > too long, the system will try to open a new searcher while the previous > one is being warmed, which can lead to problems. I have found that the > filterCache is particularly slow to warm. > > Thanks, > Shawn >
RE: Autowarming last 15 days data
Thanks Shwan, get the point. Regards, Ayush > Date: Wed, 31 Jul 2013 09:46:49 -0600 > From: s...@elyograg.org > To: solr-user@lucene.apache.org > Subject: Re: Autowarming last 15 days data > > On 7/31/2013 9:21 AM, Cool Techi wrote: > > Would it make sense if we open a newSearcher with the last 15 days > > documents? since these is the documents which are mostly used by the users. > > Also, how could i do this if this is possible? > > When you open a searcher, it's for the entire index. You may want to go > distributed and keep the newest 15 days of data in a separate index from > the rest. For my own index, I use this hot/cold shard setup. I have a > nightly process that indexes data that needs to be moved into the cold > shards and deletes it from the hot shard. > > http://wiki.apache.org/solr/DistributedSearch > > SolrCloud is the future of distributed search, but it does not have > built-in support for a hot/cold shard setup. You'd need to manage that > yourself with manual sharding. A custom sharding plugin to automate > indexing would likely be very very involved, it would probably be easier > to manage it outside of SolrCloud. > > Thanks, > Shawn >
Document Missing from Share in Solr cloud
Hi, We noticed a strange behavior in our solr cloud setup, we are using solr4.2 with 1:3 replication setting. We noticed that some of the documents were showing up in search sometimes and not at other, the reason being the document was not present in all the shards. We have restarted zookeeper and also entire cloud, but these documents are not being replicated in all the shards for some reason and hence inconsistent search results. Regards, Ayush
RE: Document Missing from Share in Solr cloud
That's what I am trying to say, the document is not replicated across all the replicas for a specific shard, hence the query show different results on every refresh. > Date: Tue, 16 Apr 2013 11:34:18 -0600 > Subject: Re: Document Missing from Share in Solr cloud > From: thelabd...@gmail.com > To: solr-user@lucene.apache.org > > If you are using the default doc router for indexing in SolrCloud, then a > document only exists in a single shard but can be replicated in that shard > to any number of replicas. > > Can you clarify your question as it sounds like you're saying that the > document is not replicated across all the replicas for a specific shard? If > so, that's definitely a problem ... > > > On Tue, Apr 16, 2013 at 11:22 AM, Cool Techi wrote: > > > Hi, > > > > We noticed a strange behavior in our solr cloud setup, we are using > > solr4.2 with 1:3 replication setting. We noticed that some of the > > documents were showing up in search sometimes and not at other, the reason > > being the document was not present in all the shards. > > > > We have restarted zookeeper and also entire cloud, but these documents are > > not being replicated in all the shards for some reason and hence > > inconsistent search results. > > > > Regards, > > Ayush > >
RE: Document Missing from Share in Solr cloud
Field type is string and this has happened for multiple docs over the past week. Regards, Ayush > Date: Tue, 16 Apr 2013 14:06:40 -0600 > Subject: Re: Document Missing from Share in Solr cloud > From: thelabd...@gmail.com > To: solr-user@lucene.apache.org > > btw ... what is the field type of your unique ID field? > > > On Tue, Apr 16, 2013 at 12:34 PM, Timothy Potter wrote: > > > Ok, that makes more sense and is definitely cause for concern. Do you have > > a sense for whether this is ongoing or happened a few times unexpectedly in > > the past? If ongoing, then will probably be easier to track down the root > > cause. > > > > > > On Tue, Apr 16, 2013 at 12:08 PM, Cool Techi wrote: > > > >> That's what I am trying to say, the document is not replicated across all > >> the replicas for a specific shard, hence the query show different results > >> on every refresh. > >> > >> > >> > >> > Date: Tue, 16 Apr 2013 11:34:18 -0600 > >> > Subject: Re: Document Missing from Share in Solr cloud > >> > From: thelabd...@gmail.com > >> > To: solr-user@lucene.apache.org > >> > > >> > If you are using the default doc router for indexing in SolrCloud, then > >> a > >> > document only exists in a single shard but can be replicated in that > >> shard > >> > to any number of replicas. > >> > > >> > Can you clarify your question as it sounds like you're saying that the > >> > document is not replicated across all the replicas for a specific > >> shard? If > >> > so, that's definitely a problem ... > >> > > >> > > >> > On Tue, Apr 16, 2013 at 11:22 AM, Cool Techi > >> wrote: > >> > > >> > > Hi, > >> > > > >> > > We noticed a strange behavior in our solr cloud setup, we are using > >> > > solr4.2 with 1:3 replication setting. We noticed that some of the > >> > > documents were showing up in search sometimes and not at other, the > >> reason > >> > > being the document was not present in all the shards. > >> > > > >> > > We have restarted zookeeper and also entire cloud, but these > >> documents are > >> > > not being replicated in all the shards for some reason and hence > >> > > inconsistent search results. > >> > > > >> > > Regards, > >> > > Ayush > >> > > > >> > >> > > > >
RE: Document Missing from Share in Solr cloud
Shouldnt the number of docs across shards be same, I can see a difference Shard 1 Last Modified:about 2 hours agoNum Docs:26236135Max Doc:26592164Deleted Docs:356029Version:6672183Segment Count:34Shard1 Replica Last Modified: about 2 hours agoNum Docs:26236135Max Doc:26594887Deleted Docs:358752Version:6678209Segment Count: 27 > From: cooltec...@outlook.com > To: solr-user@lucene.apache.org > Subject: RE: Document Missing from Share in Solr cloud > Date: Wed, 17 Apr 2013 13:28:16 +0530 > > Field type is string and this has happened for multiple docs over the past > week. > > Regards, > Ayush > > > Date: Tue, 16 Apr 2013 14:06:40 -0600 > > Subject: Re: Document Missing from Share in Solr cloud > > From: thelabd...@gmail.com > > To: solr-user@lucene.apache.org > > > > btw ... what is the field type of your unique ID field? > > > > > > On Tue, Apr 16, 2013 at 12:34 PM, Timothy Potter > > wrote: > > > > > Ok, that makes more sense and is definitely cause for concern. Do you have > > > a sense for whether this is ongoing or happened a few times unexpectedly > > > in > > > the past? If ongoing, then will probably be easier to track down the root > > > cause. > > > > > > > > > On Tue, Apr 16, 2013 at 12:08 PM, Cool Techi > > > wrote: > > > > > >> That's what I am trying to say, the document is not replicated across all > > >> the replicas for a specific shard, hence the query show different results > > >> on every refresh. > > >> > > >> > > >> > > >> > Date: Tue, 16 Apr 2013 11:34:18 -0600 > > >> > Subject: Re: Document Missing from Share in Solr cloud > > >> > From: thelabd...@gmail.com > > >> > To: solr-user@lucene.apache.org > > >> > > > >> > If you are using the default doc router for indexing in SolrCloud, then > > >> a > > >> > document only exists in a single shard but can be replicated in that > > >> shard > > >> > to any number of replicas. > > >> > > > >> > Can you clarify your question as it sounds like you're saying that the > > >> > document is not replicated across all the replicas for a specific > > >> shard? If > > >> > so, that's definitely a problem ... > > >> > > > >> > > > >> > On Tue, Apr 16, 2013 at 11:22 AM, Cool Techi > > >> wrote: > > >> > > > >> > > Hi, > > >> > > > > >> > > We noticed a strange behavior in our solr cloud setup, we are using > > >> > > solr4.2 with 1:3 replication setting. We noticed that some of the > > >> > > documents were showing up in search sometimes and not at other, the > > >> reason > > >> > > being the document was not present in all the shards. > > >> > > > > >> > > We have restarted zookeeper and also entire cloud, but these > > >> documents are > > >> > > not being replicated in all the shards for some reason and hence > > >> > > inconsistent search results. > > >> > > > > >> > > Regards, > > >> > > Ayush > > >> > > > > >> > > >> > > > > > > >
RE: Document Missing from Share in Solr cloud
Sorry, made a copy paste mistake. The numbers are different. My cloud has two shards with each shard having 1 replica. One of the shards and replica have the same number of docs, while in the other shard there is a mismatch. Regards, Ayush > From: u...@odoko.co.uk > To: solr-user@lucene.apache.org > Subject: Re: Document Missing from Share in Solr cloud > Date: Wed, 17 Apr 2013 09:48:03 +0100 > > Well, your numdocs *is* the same. Your maxdocs isn't, which sounds right > to me. > > maxdocs is the number of documents, including deleted ones. Given > deleted docs are purged by background merges, it makes sense that each > index is deciding differently when to do those merges. But the number of > undeleted docs is the same which is a good thing. > > Do queries against each replica for a shard, with distrib=false, and see > whether the results are the same. > > Upayavira > > On Wed, Apr 17, 2013, at 09:14 AM, Cool Techi wrote: > > Shouldnt the number of docs across shards be same, I can see a difference > > > > Shard 1 > > Last Modified:about 2 hours agoNum Docs:26236135Max Doc:26592164Deleted > > Docs:356029Version:6672183Segment Count:34Shard1 Replica > > > > Last Modified: about 2 hours agoNum Docs:26236135Max Doc:26594887Deleted > > Docs:358752Version:6678209Segment Count: 27 > > > > > From: cooltec...@outlook.com > > > To: solr-user@lucene.apache.org > > > Subject: RE: Document Missing from Share in Solr cloud > > > Date: Wed, 17 Apr 2013 13:28:16 +0530 > > > > > > Field type is string and this has happened for multiple docs over the > > > past week. > > > > > > Regards, > > > Ayush > > > > > > > Date: Tue, 16 Apr 2013 14:06:40 -0600 > > > > Subject: Re: Document Missing from Share in Solr cloud > > > > From: thelabd...@gmail.com > > > > To: solr-user@lucene.apache.org > > > > > > > > btw ... what is the field type of your unique ID field? > > > > > > > > > > > > On Tue, Apr 16, 2013 at 12:34 PM, Timothy Potter > > > > wrote: > > > > > > > > > Ok, that makes more sense and is definitely cause for concern. Do you > > > > > have > > > > > a sense for whether this is ongoing or happened a few times > > > > > unexpectedly in > > > > > the past? If ongoing, then will probably be easier to track down the > > > > > root > > > > > cause. > > > > > > > > > > > > > > > On Tue, Apr 16, 2013 at 12:08 PM, Cool Techi > > > > > wrote: > > > > > > > > > >> That's what I am trying to say, the document is not replicated > > > > >> across all > > > > >> the replicas for a specific shard, hence the query show different > > > > >> results > > > > >> on every refresh. > > > > >> > > > > >> > > > > >> > > > > >> > Date: Tue, 16 Apr 2013 11:34:18 -0600 > > > > >> > Subject: Re: Document Missing from Share in Solr cloud > > > > >> > From: thelabd...@gmail.com > > > > >> > To: solr-user@lucene.apache.org > > > > >> > > > > > >> > If you are using the default doc router for indexing in SolrCloud, > > > > >> > then > > > > >> a > > > > >> > document only exists in a single shard but can be replicated in > > > > >> > that > > > > >> shard > > > > >> > to any number of replicas. > > > > >> > > > > > >> > Can you clarify your question as it sounds like you're saying that > > > > >> > the > > > > >> > document is not replicated across all the replicas for a specific > > > > >> shard? If > > > > >> > so, that's definitely a problem ... > > > > >> > > > > > >> > > > > > >> > On Tue, Apr 16, 2013 at 11:22 AM, Cool Techi > > > > >> > > > > > >> wrote: > > > > >> > > > > > >> > > Hi, > > > > >> > > > > > > >> > > We noticed a strange behavior in our solr cloud setup, we are > > > > >> > > using > > > > >> > > solr4.2 with 1:3 replication setting. We noticed that some of > > > > >> > > the > > > > >> > > documents were showing up in search sometimes and not at other, > > > > >> > > the > > > > >> reason > > > > >> > > being the document was not present in all the shards. > > > > >> > > > > > > >> > > We have restarted zookeeper and also entire cloud, but these > > > > >> documents are > > > > >> > > not being replicated in all the shards for some reason and hence > > > > >> > > inconsistent search results. > > > > >> > > > > > > >> > > Regards, > > > > >> > > Ayush > > > > >> > > > > > > >> > > > > >> > > > > > > > > > > > > > > >
Solr Cloud OR Load Balancing
We are currently using Solr 3.6 with 1:1 replication on our production servers, but now out data has increased a lot and were looking to migrate to something more scalable. We have been experimenting with Solr Cloud over the past 2 months and noticed servers issues, most of them have been fixed in the version releases but some am not sure of, like Docs missing in replica's of a shard. Giving this what would the recommendation be, should we use solr4.2 with replication and load balancing using HA proxy so that we dont run into issue OR do we think Solr Cloud is good for live usage. Regards, Ayush
Solr Hangs on startup
Hi,
We are using solr4.3 cloud setup, but for some reason solr fails to startup, I
see the following in the log file, post this there are no logs.,
org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@17586ed7 main
15293 [coreLoadExecutor-3-thread-1] INFO org.apache.solr.core.SolrCore ?
[cmn_shard1_replica1] Registered new searcher Searcher@17586ed7
main{StandardDirectoryReader(segments_zpt:1650158 _6nln(4.3):C2734441/433094
_6rvw(4.3):C3395530/166626 _6vdg(4.3):C4039667/172929 _6z52(4.3):C4137543/2279
_770z(4.3):C5879498/97346 _71rz(4.3):C4168660/440273 _74hn(4.3):C5900928/134106
_7a5j(4.3):C5892645/269769 _7bcs(4.3):C5502048/2562430
_7gnp(4.3):C2310243/676016 _7klj(4.3):C3753172/184399 _7nxn(4.3):C558455
A thread dump show the following,
"coreLoadExecutor-3-thread-1" prio=10 tid=0x40b44800 nid=0x27ad
runnable [0x7fbbff2d6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcher.pread0(Native Method)
at sun.nio.ch.FileDispatcher.pread(FileDispatcher.java:31)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:199)
at sun.nio.ch.IOUtil.read(IOUtil.java:175)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:612)
at
org.apache.solr.update.ChannelFastInputStream.readWrappedStream(TransactionLog.java:752)
at
org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:89)
at
org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:61)
at
org.apache.solr.update.TransactionLog$ReverseReader.next(TransactionLog.java:702)
at
org.apache.solr.update.UpdateLog$RecentUpdates.update(UpdateLog.java:925)
at
org.apache.solr.update.UpdateLog$RecentUpdates.access$000(UpdateLog.java:863)
at
org.apache.solr.update.UpdateLog.getRecentUpdates(UpdateLog.java:1014)
at org.apache.solr.update.UpdateLog.init(UpdateLog.java:253)
at org.apache.solr.update.UpdateHandler.initLog(UpdateHandler.java:82)
at org.apache.solr.update.UpdateHandler.(UpdateHandler.java:137)
at org.apache.solr.update.UpdateHandler.(UpdateHandler.java:123)
at
org.apache.solr.update.DirectUpdateHandler2.(DirectUpdateHandler2.java:95)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:525)
at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:596)
at org.apache.solr.core.SolrCore.(SolrCore.java:805)
at org.apache.solr.core.SolrCore.(SolrCore.java:618)
at
org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
What could be causing this?
regards,
Ayush
Single Tomcat Multiple Shards
Hey Guys, We are experimenting with solr cloud, this is what we want to set up as, 2 Machines each having have 8 master shards, so total of 16 shards. The assumption is we want to store approximately 4-5 TB data over a period of 1 year of so. Replication factor of 1 which are again distributed across 3-4 machines. Initially we want to start with 8 shards in a single tomcat and single machine, but I cannot find a way of having multiple shards in a single SOLR_HOME and single Tomcat. Can this be achieved? Regards,Ayush
Reloading config to zookeeper
When we make changes to our config files, how do we reload the files into zookeeper. Also, I understand that we would need to reload the collection, would we need to do this at a per shard level or just at the cloud level. Regards, Ayush
RE: Reloading config to zookeeper
Thanks, but why do we need to specify the -solrhome? I am using the following command to load new config, java -classpath .:/Users/solr-cli-lib/* org.apache.solr.cloud.ZkCLI -cmd upconfig -zkhost localhost:2181,localhost:2182,localhost:2183,localhost:2184,localhost:2185 -confdir /Users/config-files -confname myconf So basically reloading is just uploading the configs back again? Regard,s Ayush > Date: Thu, 22 Nov 2012 19:32:27 +0100 > Subject: Re: Reloading config to zookeeper > From: mrzewu...@gmail.com > To: solr-user@lucene.apache.org > > Hi, > > I'm using "cloud-scripts/zkcli.sh" script for reloading configuration, for > example: > $ ./cloud-scripts/zkcli.sh -cmd upconfig -confdir -solrhome > -confname -z > > Then I'm reloading collection on each node in cloud, but maybe someone > knows better solution. > Regards. > > On 22 November 2012 19:23, Cool Techi wrote: > > > When we make changes to our config files, how do we reload the files into > > zookeeper. > > > > Also, I understand that we would need to reload the collection, would we > > need to do this at a per shard level or just at the cloud level. > > > > Regards, > > Ayush > > > >
Solr Cloud 4.0 Production Ready?
Hi, We have been using solr 3.5 in our production for sometime now and facing the problems faced by a large solr index. We wanted to migrate to Solr Cloud and have started some experimentation. But in the mean time also following the user forum and seem to be noticing a lot of bugs which were raised post the release and will be fixed in 4.1. Should we wait for 4.1 release for production or we can go ahead with the current release? Regards, Ayush
Searching for Solr Stop Words
On of my solr fields is configured in the following manned,
This works in cases where i don't want stemming, but now there is
another use case which is causing a problem, people are beginning to
seach for the following combinations,
The Ivy : In this case results with just ivy is being returned,
when the expected result would be with The. I understand that this is
because of the stop word but is the way to achieve this. For example if
they search for "the ivy" within quotes than this should work.(Mom & Me) OR
("mom and me"): In this case also & is
dropped or results including both mom and me in some part of the
statement is returned.
I am ok if only new data behaves in the right way but wouldnt be able
to reindex. Also, would changing the schema.xml file trigger a full
replication?
Regards,
Ayush
Solr commit taking too long
Hi, We have an index of approximately 400GB in size, indexing 5000 documents was taking 20 seconds. But lately, the indexing is taking very long, committing the same amount of document is taking 5-20 mins. On checking the logs I can see that their a frequent merges happening, which I am guessing is the reason for this, how can this be improved. My configurations are given below, false 30 64 regards, Ayush
Not condition not working for Korean search
Hi,
I am no Korean expert and am finding it difficult to fix this, my client is
searching for the following query, but the NOT condition doesn't seem to be
working.
(stnostem:((옵티머스 OR "엘지 스마트폰") AND NOT ("옵티머스 프라임" OR 프라임)))
the search result (xml attached ) return result with the not condition
keywords? How can this be fixed.
Regards,
Ayush
Hourly Faceting
Hi, I want to facet results on an hourly basis, the following query gives me an hourly breakdown, but with the date part. I want just the hour part across the days. Is there any other way of doing this, 2013-02-01T00:00:00Z-330MINUTES true twitterId:191343557 createdOnGMTDate +1HOUR 2013-02-08T23:59:59Z-330MINUTES 0 Result 0 0 0 0 0 0 Desired Result 0 0 0 0 0 0 Regards, Ayush
Not condition not working for Korean search
Any help here, still not able to understand what might be causing this. I have
tried all combinations below, but to no avail,
(stnostem:((옵티머스 OR "엘지 스마트폰") AND NOT stnostem:("옵티머스 프라임" OR 프라임)))
(stnostem:((옵티머스 OR "엘지 스마트폰") AND - stnostem:("옵티머스 프라임" OR 프라임)))
What could be wrong here.
From: cooltec...@outlook.com
To: solr-user@lucene.apache.org
Subject: Not condition not working for Korean search
Date: Thu, 7 Feb 2013 19:46:13 +0530
Hi,
I am no Korean expert and am finding it difficult to fix this, my client is
searching for the following query, but the NOT condition doesn't seem to be
working.
(stnostem:((옵티머스 OR "엘지 스마트폰") AND NOT ("옵티머스 프라임" OR 프라임)))
the search result (xml attached ) return result with the not condition
keywords? How can this be fixed.
Regards,
Ayush
RE: Not condition not working for Korean search
idf(docFreq=5632, maxDocs=290963725)
0.034640186 = queryNorm
7.407695 = (MATCH) fieldWeight(stnostem:옵티머스 in 2006207), product of:
1.0 = tf(termFreq(stnostem:옵티머스)=1)
11.852312 = idf(docFreq=5632, maxDocs=290963725)
0.625 = fieldNorm(field=stnostem, doc=2006207)
0.5 = coord(1/2)
1.520675 = (MATCH) sum of:
1.520675 = (MATCH) product of:
3.04135 = (MATCH) sum of:
3.04135 = (MATCH) weight(stnostem:옵티머스 in 1338974), product of:
0.4105663 = queryWeight(stnostem:옵티머스), product of:
11.852312 = idf(docFreq=5632, maxDocs=290963725)
0.034640186 = queryNorm
7.407695 = (MATCH) fieldWeight(stnostem:옵티머스 in 1338974), product of:
1.0 = tf(termFreq(stnostem:옵티머스)=1)
11.852312 = idf(docFreq=5632, maxDocs=290963725)
0.625 = fieldNorm(field=stnostem, doc=1338974)
0.5 = coord(1/2)
> From: j...@basetechnology.com
> To: solr-user@lucene.apache.org
> Subject: Re: Not condition not working for Korean search
> Date: Mon, 11 Feb 2013 00:37:03 -0500
>
> Show us the parsed query that you get with &debugQuery=true
>
> Actually this may be the "pure negative" bug where you need to change:
>
> A AND -B
>
> into
>
> A AND (*:* -B)
>
> -- Jack Krupansky
>
> -Original Message-
> From: Cool Techi
> Sent: Monday, February 11, 2013 12:30 AM
> To: solr-user@lucene.apache.org
> Subject: Not condition not working for Korean search
>
> Any help here, still not able to understand what might be causing this. I
> have tried all combinations below, but to no avail,
>
> (stnostem:((옵티머스 OR "엘지 스마트폰") AND NOT stnostem:("옵티머스 프라임"
> OR 프라임)))
>
> (stnostem:((옵티머스 OR "엘지 스마트폰") AND - stnostem:("옵티머스 프라임"
> OR 프라임)))
>
> What could be wrong here.
>
>
>
> From: cooltec...@outlook.com
> To: solr-user@lucene.apache.org
> Subject: Not condition not working for Korean search
> Date: Thu, 7 Feb 2013 19:46:13 +0530
>
>
>
>
> Hi,
>
> I am no Korean expert and am finding it difficult to fix this, my client is
> searching for the following query, but the NOT condition doesn't seem to be
> working.
>
> (stnostem:((옵티머스 OR "엘지 스마트폰") AND NOT ("옵티머스 프라임" OR
> 프라임)))
>
> the search result (xml attached ) return result with the not condition
> keywords? How can this be fixed.
>
> Regards,
> Ayush
>
>
SEVERE RecoveryStrategy Recovery failed - trying again... (9)
I am seeing the following error in my Admin console and the core/ cloud status is taking forever to load. SEVERERecoveryStrategyRecovery failed - trying again... (9) What causes this and how can I recover from this mode? Regards, Rohit
RE: SEVERE RecoveryStrategy Recovery failed - trying again... (9)
There is not error I can see in the logs, my shards are divided over three
machines, the cloud runs fine when I don't bring up one of the nodes, the
moment I start that particular note, the cloud stops responding,
Feb 19, 2013 5:22:22 AM
org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener
newSearcher
INFO: Loading spell index for spellchecker: default
Feb 19, 2013 5:22:22 AM
org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener
newSearcher
INFO: Loading spell index for spellchecker: wordbreak
Feb 19, 2013 5:22:22 AM org.apache.solr.core.SolrCore registerSearcher
INFO: [cmn] Registered new searcher Searcher@3b47788d
main{StandardDirectoryReader(segments_1dvf:1488121 _2acm(4.1):C13967428/87404
_62w6(4.1):C259989/31792 _8ehw(4.1):C405062/57136 _8um4(4.1):C228434/26526
_a0i1(4.1):C171825/43653 _bgu3(4.1):C315311/30246 _ao6h(4.1):C176468/44702
_b7uu(4.1):C97823/27124 _bjzb(4.1):C77280/8476 _bra3(4.1):C142681/21340
_bzpo(4.1):C198058/23506 _c0jh(4.1):C18201/8171 _c307(4.1):C37984/5305
_c2e0(4.1):C22300/9788 _c1o6(4.1):C23523/8630 _c3hl(4.1):C12034/2871
_c3kw(4.1):C5821/971 _c3l6(4.1):C1106 _c3lh(4.1):C707/1 _c3lu(4.1):C509/2
_c3mf(4.1):C482/1 _c3m5(4.1):C374/2 _c3mc(4.1):C164/2 _c3mh(4.1):C64/3
_c3mi(4.1):C49 _c3mj(4.1):C25 _c3mk(4.1):C12)}
Feb 19, 2013 5:22:22 AM org.apache.solr.cloud.ZkController publish
INFO: publishing core=cmn state=down
Feb 19, 2013 5:22:22 AM org.apache.solr.cloud.ZkController publish
INFO: numShards not found on descriptor - reading it from system property
Feb 19, 2013 5:22:22 AM org.apache.solr.core.CoreContainer registerCore
INFO: registering core: cmn
Feb 19, 2013 5:22:22 AM org.apache.solr.cloud.ZkController register
INFO: Register replica - core:cmn address:http://10.0.0.205:8080/solr
collection:cmn shard:shard2
Feb 19, 2013 5:22:22 AM org.apache.solr.client.solrj.impl.HttpClientUtil
createClient
INFO: Creating new http client,
config:maxConnections=1&maxConnectionsPerHost=20&connTimeout=3&socketTimeout=3&retry=false
Regards,
Ayush
> Subject: Re: SEVERERecoveryStrategyRecovery failed - trying again...
> (9)
> From: markrmil...@gmail.com
> Date: Mon, 18 Feb 2013 10:21:53 -0500
> To: solr-user@lucene.apache.org
>
> We need to see more of your logs to determine why - there should be some
> exceptions logged.
>
> - Mark
>
> On Feb 18, 2013, at 9:47 AM, Cool Techi wrote:
>
> > I am seeing the following error in my Admin console and the core/ cloud
> > status is taking forever to load.
> >
> > SEVERERecoveryStrategyRecovery failed - trying again... (9)
> >
> > What causes this and how can I recover from this mode?
> >
> > Regards,
> > Rohit
> >
> >
>
Solr on https
Hi, We are trying to run solr on https, these are few of the issues or problems that are coming up. Just wanted to understand if anyone else is facing these problems, we have some shards running on https, but in shards parameter in solr we don't specify the protocol, how can we achieve thisWill replication work on https Will commit and other functions working normally? Regards,atpug
Solr Sharding Or Solr Cloud
Hi, We have reached a state where a single solr server is not proving to be enough and are exploring options of sharding or solr cloud. Just wanted to check the pros and cons of each one of them if there are any. What would be the best size of the shards or Regards, R
New data not indexing, after data deletion.
Hi, We have a huge index size, running to around 350Gb+, we yesterday deleted some data on it. The deletion completed successfully, but post the deletion new data was not being indexed. We restarted the server, but neither the tomcat logs or Solr logs are giving any indication of why this might be happening. All we keep getting is a timeout exception, but at the same time all the queries are running fine and returning normal results. Has anyone faced a similar issue earlier or if anyone can point to how we debug this? Regards
Architecture Question
Hi, I am not sure if this is the right forum for this question, but it would be great if I could be pointed in the right direction. We have been using a combination of MySql and Solr for all our company full text and query needs. But as our customers have grow so has the amount of data and MySql is just not proving to be a right option for storing/querying. I have been looking at Solr Cloud and it looks really impressive, but and not sure if we should give away our storage system. So, I have been exploring DataStax but a commercial option is out of question. So we were thinking of using hbase to store the data and at the same time index the data into Solr cloud, but for many reasons this design doesn't seem convincing (Also seen basic of Lilly). 1) Would it be recommended to just user Solr cloud with multiple replication or hbase-solr seems like good option 2) How much strain would be to keep both Solr Shard and Hbase node on the same machine 3) if there a calculation on what kind of machine configuration would I need to store 500-1000 million records. Most of these with be social data (Twitter/facebook/blogs etc) and how many shards. Regards, Ayush
RE: Architecture Question
Hi Otis, Thanks for your reply, just wanted to check what NoSql structure would be best suited to store data and use the least amount of memory, since for most of my work Solr would be sufficient and I want to store data just in case we want to reindex and as a backup. Regards, Ayush > Date: Fri, 16 Nov 2012 15:47:40 -0500 > Subject: Re: Architecture Question > From: otis.gospodne...@gmail.com > To: solr-user@lucene.apache.org > > Hello, > > > > > I am not sure if this is the right forum for this question, but it would > > be great if I could be pointed in the right direction. We have been using a > > combination of MySql and Solr for all our company full text and query > > needs. But as our customers have grow so has the amount of data and MySql > > is just not proving to be a right option for storing/querying. > > > > I have been looking at Solr Cloud and it looks really impressive, but and > > not sure if we should give away our storage system. So, I have been > > exploring DataStax but a commercial option is out of question. So we were > > thinking of using hbase to store the data and at the same time index the > > data into Solr cloud, but for many reasons this design doesn't seem > > convincing (Also seen basic of Lilly). > > > > 1) Would it be recommended to just user Solr cloud with multiple > > replication or hbase-solr seems like good option > > > > If you trust SolrCloud with replication and keep all your fields stored > then you could live without an external DB. At this point I personally > would still want an external DB. Whether HBase is the right DB for the job > I can't tell because I don't know anything about your data, volume, access > patterns, etc. I can tell you that HBase does scale well - we have tables > with many billions of rows stored in it for instance. > > > > 2) How much strain would be to keep both Solr Shard and Hbase node on the > > same machine > > > > HBase loves memory. So does Solr. They both dislike disk IO (who > doesn't!). Solr can use a lot of CPU for indexing/searching, depending on > the volume. HBase RegionServers can use a lot of CPU if you run MapReuce > on data in HBase. > > > > 3) if there a calculation on what kind of machine configuration would I > > need to store 500-1000 million records. Most of these with be social data > > (Twitter/facebook/blogs etc) and how many shards. > > > > No recipe here, unfortunately. You'd have to experiment and test, do load > and performance testing, etc. If you need help with Solr + HBase, we > happen to have a lot of experience with both and have even used them > together for some of our clients. > > Otis > -- > Performance Monitoring - http://sematext.com/spm/index.html > Search Analytics - http://sematext.com/search-analytics/index.html
Question about Solr Cloud
Hi, I have just started working with Solr cloud and have a few questions related to the same, 1) In the start script we provide the the following, what's the purpose of providing this. -Dbootstrap_confdir=./solr/collection1/conf Since we don't yet have a config in zookeeper, this parameter causes the local configuration directory ./solr/conf to be uploaded as the "myconf" config. The name "myconf" is taken from the "collection.configName" param below. -Dcollection.configName=myconf sets the config to use for the new collection. Omitting this param will cause the config name to default to "configuration1" 2) When we make any changes into the config/schema do we need to copy it to all the shards running in the cloud manually?3) If we want to start with 10 shards on 2 machines, anticipating the future growth, do all these shards needs to run on separate jetty instances4) Is there any advantage of running solr on jetty then Tomcat? Thanks,Ayush
