Solr performance for Instance updates
Hi, We want to update the index based on TIB listener, whenever database changes happens we want to update my index instantly this may happen very frequently for number of records. Could anyone please tell me how would be the performance for these scenarios? Question related linguistic support How is linguistic support for Solr and is it compatibility to add third party packages(mass Boston, Cambridge,..etc for linguistic support) Any help would appreciate!! Thanks & Regards, Mahendra
Re: Solr performance for Instance updates
Hi, Instantly I want to update each doc(based on db changes) and commit, I hope for every commit it takes more time.I don't want to post some bulk docs and commit. How can be the performance for this scenario... Also every time if I am going to update the docs the index size is going to increase, the old doc is physically not going to delete. Any Idea on this.. Thanks & Regards, Mahendra --- On Sat, 9/27/08, Otis Gospodnetic <[EMAIL PROTECTED]> wrote: From: Otis Gospodnetic <[EMAIL PROTECTED]> Subject: Re: Solr performance for Instance updates To: solr-user@lucene.apache.org Date: Saturday, September 27, 2008, 2:19 AM Hi, - Original Message ---- > From: mahendra mahendra <[EMAIL PROTECTED]> > To: solr-user@lucene.apache.org > Sent: Friday, September 26, 2008 3:52:57 PM > Subject: Solr performance for Instance updates > > Hi, > > We want to update the index based on TIB listener, whenever database changes > happens we want to update my index instantly this may happen very frequently for > number of records. > > Could anyone please tell me how would be the performance for these scenarios? It depends on the complexity of your analyzers, but I've seen indexing at over 500 docs/second on modest hardware. > Question related linguistic support > How is linguistic support for Solr and is it compatibility to add third party > packages(mass Boston, Cambridge,..etc for linguistic support) Could you please be more specific about what type of linguistic support you are after? Are you after NE extraction? If so, no such thing built into Solr, but it can certainly be built and integrated with Solr. ;) Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
Changing dataDir without restatrting server
Is there any way to change the index location with out restarting the server? Requirement We want to share the index location by multiple search nodes. As part of my project maintanance we want to shift the current index location to other location and delete the current index location for every week. Is this possible to change the index location without restarting the server. The index location will be reading from some properties file.. Example We have two index locations called IndexA, IndexB. Presume currently my configuration is pointing to IndexA, after a week we are going to have fresh data in indexB and point the current index location to IndexB and remove the IndexA. The same thing will happen in reverse case for other week. Any help would be appreciate!! Thanks & Regards, Mahendra
Index partitioning
Hi, I want to partition my index based on category information. Also, while indexing I want to store particular category data to corresponding index partition. In the same way I need to search for category information on corresponding partition.. I found some information on wiki link http://wiki.apache.org/solr/IndexPartitioning. But it couldn't help much in implementation. Could some one give example with how I can configure, index data into particular partitions and search on particular partition. Would appreciate for any help!! Thanks & Regards, Mahendra
Performance issue
Hi, I have configured solr in two different servers in different machines. One server I am using for indexing and another server using for searching data. Both servers are pointing to the common data directory (shared drive). When indexer server posting the data to shared drive (data directory), search server is not displaying the data which is posted newly from indexer server unless restarting the search server or reloading search server cores. It’s having unnecessary load on the search server. Is there anyway I can get the results without restarting the server or reloading cores from search machine. Would appreciate for any help!! Thanks & Regards, Mahendra
Exact match search problem
Hi, I have indexed my data as "custom123, customer, custom" for the "UserName" field. I need to search the records for exact match, when I am trying to search with UserName:"customer" I am finding the records where UserName is custom123 and custom. As per my understanding solr splits the AlphaNumeric words into sub words custom123 => "custom","123" As per above the above logic when I search for UserName:"customer", it shouldn't display the custom123 and custom. Could you please tell me why it is behaving like that or how I can search for exact match. I am using the following declaration for text field in schema file Thanks in advance!! - Mahendra
wildcard search issue
Hi, The case sensitive wild-card search is not working for TextField type. I have tried searching for UserName:cust*, it gave the results, but UserName:Cust* didn't give any results. How can I make it work.. I have defined my TextField in following way. Any help would appreciate!! Thanks & Regards, Mahendra
exceeded limit of maxWarmingSearchers
Hi, I have scheduled Incremental indexing to run for every 2 min. Some times due to more number of records the first instance of the incremental couldn't complete before second instance start. This is causing the below error. org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Is there any configuration parameter to increase maxWarmingSearchers. Any help would appriciate !! Thanks & Regards, Mahendra
Re: exceeded limit of maxWarmingSearchers
Hi Shalin, Which auto-warming I have to remove to make commits faster. The below configuration have in solrconfig.xml. Also, what is maxWarmingSearchers value. Thanks for your help!! Thanks & Regards, Mahendra --- On Mon, 2/23/09, Shalin Shekhar Mangar wrote: From: Shalin Shekhar Mangar Subject: Re: exceeded limit of maxWarmingSearchers To: solr-user@lucene.apache.org Date: Monday, February 23, 2009, 10:35 AM On Mon, Feb 23, 2009 at 10:23 AM, mahendra mahendra < mahendra_featu...@yahoo.com> wrote: > Hi, > > I have scheduled Incremental indexing to run for every 2 min. Some times > due to more number of records the first instance of the incremental couldn't > complete before second instance start. This is causing the below error. > > org.apache.solr.common.SolrException: Error opening new searcher. exceeded > limit of maxWarmingSearchers=2, try again later. > Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try > again later. > > Is there any configuration parameter to increase maxWarmingSearchers. > Any help would appriciate !! Increasing maxWarmingSearchers will make things even slower. The right problem to solve is to increase the time between commits or to remove auto-warming to make commits faster. -- Regards, Shalin Shekhar Mangar.
Indexing issue
Hi, I have two cores in different machines which are referring to the same data directory. I have implemented this mechanism to have fault tolerance in place, if any of the machines are down the fault tolerance take care to index data from other machine. Since two cores are referring to the same data directory some times reindexing getting failed and it is showing the below error. HTTP Status 500 - java.io.FileNotFoundException: \\SolrShare\CollectionSet2\English\Auction\Auction0\index\_c.fdt (The system cannot find the file specified) java.lang.RuntimeException: java.io.FileNotFoundException: \\SolrShare\CollectionSet2\English\Auction\Auction0\index\_c.fdt (The system cannot find the file specified) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:960) at org.apache.solr.core.SolrCore.(SolrCore.java:470) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:323) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:107) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at I have changed lockType to simple and none, but still no luck… Could you please correct me if I am doing wrong? Thanks in advance!! Regards, Mahendra
Commit is taking very long time
Hello,
I am experiencing strange problems while doing commit. I am doing indexing for
every 10 min to update index with data base values. commit is taking 7 to 10
min approximately and my indexing is failing due to null pointer exception. If
first thread is not completed in 10 min the second thread will be starting to
index data.
I changed wait=false for the listener from solrconfig.xml file. It stopped
getting Null pointer exception but the commit is taking 7 to 10 min. I have
approximately 70 to 90 kb of data every time.
solr/bin/snapshooter
.
false
arg1 arg2
MYVAR=val1
I kept all default parameter values in solrconfig.xml except the ramBuffersize
to 512.
Could you please tell me how can I overcome these problems, also some times I
see "INFO: Failed to unregister mbean: partitioned because it was not registered
Mar 13, 2009 11:49:16 AM org.apache.solr.core.JmxMonitoredMap unregister" in my
log files.
Log file
ar 13, 2009 1:28:40 PM org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction1-0] webapp=/solr path=/update
params={wt=javabin&waitFlush=true&commit=true&waitSearcher=true&version=2.2}
status=0 QTime=247232
Mar 13, 2009 1:30:32 PM org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: {add=[79827482, 79845504, 79850902, 79850913, 79850697, 79850833,
79850901, 79798207, ...(93 more)]} 0 62578
Mar 13, 2009 1:30:32 PM org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction1-0] webapp=/solr path=/update
params={wt=javabin&version=2.2} status=0 QTime=62578
Mar 13, 2009 1:30:32 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=false,waitFlush=true,waitSearcher=true)
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher
INFO: Opening searc...@1ba5edf main
Mar 13, 2009 1:34:38 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming searc...@1ba5edf main from searc...@81f25 main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for searc...@1ba5edf main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming searc...@1ba5edf main from searc...@81f25 main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=63,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for searc...@1ba5edf main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=94,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming searc...@1ba5edf main from searc...@81f25 main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for searc...@1ba5edf main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener sending requests to searc...@1ba5edf main
Mar 13, 2009 1:34:38 PM org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction1-0] webapp=null path=null params={rows=10&start=0&q=solr}
hits=0 status=0 QTime=0
Mar 13, 2009 1:34:38 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
Mar 13, 2009 1:34:38 PM org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction1-0] webapp=null path=null
params={rows=10&start=0&q=rocks} hits=223 status=0 QTime=0
Mar 13, 2009 1:34:38 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
Mar 13, 2009 1:34:38 PM org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction1-0] webapp=null path=null
params={q=static+newSearcher+warming+query+from+solrconfig.xml} hits=4297
status=0 QTime=0
Mar 13, 2009 1:34:38 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
Mar 13, 2009 1:34:38 PM org.apache.solr.core.SolrCore registerSearcher
INFO: [EnglishAuction1-0] Registered new searche
Re: Commit is taking very long time
Hi,
Sorry in delaying to mail!
My application is in prod and quite frequently getting NullPointerException.
Initially I thought this is happening because of memory issue, so I reduced
mergeFactor to 5 and reduced number of document per commit to 2000. After these
changes for some time it stopped getting NullPointerException.
In one of my prod box again started getting NullPointerException. Its taking
more time at commit and throwing an exception. Its trying to commit only 15
records when it is throwing exception. I deleted the index directory manually
and reindexed same data, it just completed in few seconds.
How I can overcome these problems. Thanks in advance!!
Log
INFO: [EnglishAuction2-0] webapp=/solr path=/admin/ping
params={wt=javabin&version=2.2} hits=0 status=0 QTime=47
19-Mar-2009 18:55:03 org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction2-0] webapp=/solr path=/admin/ping
params={wt=javabin&version=2.2} status=0 QTime=47
19-Mar-2009 18:57:06 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[27812087, 27812088, 27812089, 27812090, 27812091, 27812079,
27812080, 27812081, ...(15 more)]} 0 122952
19-Mar-2009 18:57:06 org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction2-0] webapp=/solr path=/update
params={wt=javabin&version=2.2} status=0 QTime=122952
19-Mar-2009 18:59:03 org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction2-0] webapp=/solr path=/admin/ping
params={wt=javabin&version=2.2} hits=0 status=0 QTime=16
19-Mar-2009 18:59:03 org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction2-0] webapp=/solr path=/admin/ping
params={wt=javabin&version=2.2} status=0 QTime=16
19-Mar-2009 18:59:03 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[27812087, 27812088, 27812089, 27812090, 27812091, 27812079,
27812080, 27812081, ...(15 more)]} 0 141
19-Mar-2009 18:59:03 org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction2-0] webapp=/solr path=/update
params={wt=javabin&version=2.2} status=0 QTime=141
19-Mar-2009 18:59:03 org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=false,waitFlush=false,waitSearcher=false)
StackTrace
java.lang.NullPointerException
at
com.fm.search.incrementalindex.service.AuctionCollectionServiceImpl.indexData(AuctionCollectionServiceImpl.java:251)
at
com.fm.search.incrementalindex.service.AuctionCollectionServiceImpl.process(AuctionCollectionServiceImpl.java:135)
at
com.fm.search.job.SearchIndexingJob.executeInternal(SearchIndexingJob.java:68)
at
org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)
Thanks,
Mahendra
--- On Sat, 3/14/09, Yonik Seeley wrote:
From: Yonik Seeley
Subject: Re: Commit is taking very long time
To: solr-user@lucene.apache.org
Date: Saturday, March 14, 2009, 6:59 AM
>From your logs, it looks like the time is spent in closing of the index.
There may be some pending deletes buffered, but they shouldn't take too
long.
There could also be a merge triggered... but this would only happen
sometimes, not every time you commit.
One more relatively recent change in Lucene is to sync the index files
for safety.
Are you perhaps running on Linux with the ext3 filesystem?
Not sure what's causing the null pointer exception... do you have a stack
trace?
-Yonik
http://www.lucidimagination.com
On Fri, Mar 13, 2009 at 9:05 PM, mahendra mahendra
wrote:
> Hello,
>
> I am experiencing strange problems while doing commit. I am doing indexing
for every 10 min to update index with data base values. commit is taking 7 to 10
min approximately and my indexing is failing due to null pointer exception. If
first thread is not completed in 10 min the second thread will be starting to
index data.
> I changed wait=false for the listener from solrconfig.xml file. It stopped
getting Null pointer exception but the commit is taking 7 to 10 min. I have
approximately 70 to 90 kb of data every time.
>
> solr/bin/snapshooter
> .
> false
> arg1
arg2
>
MYVAR=val1
>
> I kept all default parameter values in solrconfig.xml except the
ramBuffersize to 512.
> Could you please tell me how can I overcome these problems, also some
times I see "INFO: Failed to unregister mbean: partitioned because it was
not registered
> Mar 13, 2009 11:49:16 AM org.apache.solr.core.JmxMonitoredMap
unregister" in my log files.
>
> Log file
>
> ar 13, 2009 1:28:40 PM org.apache.solr.core.SolrCore execute
> INFO: [EnglishAuction1-0] webapp=/solr path=/update
params={wt=javabin&waitFlush=true&commit=true&waitSearcher=true&version=2.2}
status=0 QTime=247232
> Mar 13, 2009 1:30:32 PM
org.apache.solr.update.processor.LogUpdateProcessor finish
> INFO: {a
