disabled omitNorms in index but still see default value
Hi We have a huge index with more than 50 million documents. In the beginning we disabled norms for some fields by setting omitNorms=true. Recently we decided to add norms to few other fields and we removed omitNorms=true from schema. I read in solr forum that if one of the document in any segment has omitNorms=true, during the next merge its copied to all documents. I am confused by this design. what is the purpose behind this behavior ? Looks like we need to wipe the index and start fresh, if we have to enable omitNorms for a field. Thanks Srini -- View this message in context: http://lucene.472066.n3.nabble.com/disabled-omitNorms-in-index-but-still-see-default-value-tp4061724.html Sent from the Solr - User mailing list archive at Nabble.com.
ids request to shard with star query are slow
ids request to shard with star query are slow I have a distributed solr environment and I am investigating all the request where the shard took significant amount of time. One common pattern i saw was all the ids request with q=*:* and ids= took around 2-3sec. i picked some shard request q=xyz and ids= and all of them took only few milliseconds. I copied the params and manually sent the same request to that particular shard and again it took around 2.5 sec. But when i removed the query (q=*:*) parameter and sent the same set of params to the same shard i got the response back in 10 or millisecond. in both cases the response had the document i am looking for. took 2-3 sec - q=*:*& qt=search& ids=123& isShard=true took 20ms - qt=search& ids=123& isShard=true In my understanding ids param is used to get the stored field in a distributed search. Why does the query parameter (q=) matter here ? Thanks Srini -- View this message in context: http://lucene.472066.n3.nabble.com/ids-request-to-shard-with-star-query-are-slow-tp4047395.html Sent from the Solr - User mailing list archive at Nabble.com.
copyField after analyzer
Hi, I want to copy/append different fields to one field, while applying a different analyzer for each field. Lets assume i have specified different analyzers/filters for each of the above source fields cat, name, manu, features and includes. I read online that copyField copies before the analysis is done. Is it possible to copyField after applying the analyzer to all the source string ? Just to give you one use case, i would like to remove duplicates in some of the source fields like manna, includes, etc and i don't want to do it for all source fields. Thanks, Srini -- View this message in context: http://lucene.472066.n3.nabble.com/copyField-after-analyzer-tp3900337p3900337.html Sent from the Solr - User mailing list archive at Nabble.com.
dynamic analyzer based on condition
Hi, I want to pick different analyzers for the same field for different languages. I can determine the language from a different field. I would have different fieldTypes defined in my schema.xml such as text_en, text_de, text_fr, etc where i specify which analyzer and filter to use during indexing and query time. but i would like to define the field dynamically. for e.g if lang=="en" else if lang=="de" ... Can I achieve this somehow ? If this approach cannot be done then i can just create one field for every language. Thanks Srini -- View this message in context: http://lucene.472066.n3.nabble.com/dynamic-analyzer-based-on-condition-tp3909345p3909345.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: dynamic analyzer based on condition
Hi Erick, Thanks a lot for your reply. I have around 10-15 searchable text fields (and 5-6 languages). If I create one per language will that increase the memory occupied by my index. Even though only one field will have a value at a time, will there be a case the empty fields in the index will occupy some memory ? will that happen if i enable field caching ? Thanks Srini -- View this message in context: http://lucene.472066.n3.nabble.com/dynamic-analyzer-based-on-condition-tp3909345p3912605.html Sent from the Solr - User mailing list archive at Nabble.com.
QueryElevationComponent and distributed search
Hi, I am using solr 3.6. I saw in Solr wiki that QueryElevationComponent is not supported for distributed search. https://issues.apache.org/jira/browse/SOLR-2949 When I checked the above ticket, it looks like its fixed in Solr 4.0. Does anyone have any idea when a stable version of solr 4.0 will be released (approx time frame). If not, are these changes independent to other solr 4.0 changes that i can just copy this patch to my setup for now? I would like to use solr 3.6 because i would like to use a stable version in production. Thanks Srini -- View this message in context: http://lucene.472066.n3.nabble.com/QueryElevationComponent-and-distributed-search-tp3936998p3936998.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: QueryElevationComponent and distributed search
Can anyone tell me whether support for distributed search is added to QueryElevationComponent in solr 4.0 ? I looked at QueryElevationComponent code in trunk and it doesnt seem to have any code to support distributed search. Am I missing something here ? -- View this message in context: http://lucene.472066.n3.nabble.com/QueryElevationComponent-and-distributed-search-tp3936998p3939627.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: QueryElevationComponent and distributed search
Hi Mark Thanks a lot for your response. Yeah, I was looking for distributed methods. I would like to know the reason why it didn't work in Solr 3.6. When i try (i only have solar 3.6) Note: the same query below works when i replace elevate with select, by which i am assuming my setup is correct. http://localhost:8080/solr/shard1/elevate/?q=dress&debugQuery=true&shards=localhost:8080/solr/shard1,localhost:8080/solr/shard2 I get the error Problem accessing /solr/shard1/elevate/. Reason: Index: 1, Size: 0 java.lang.IndexOutOfBoundsException: Index: 1, Size: 0 at java.util.ArrayList.RangeCheck(ArrayList.java:547) at java.util.ArrayList.get(ArrayList.java:322) at org.apache.solr.common.util.NamedList.getVal(NamedList.java:137) at org.apache.solr.handler.component.ShardFieldSortedHitQueue$ShardComparator.sortVal(ShardDoc.java:221) at org.apache.solr.handler.component.ShardFieldSortedHitQueue$2.compare(ShardDoc.java:259) at org.apache.solr.handler.component.ShardFieldSortedHitQueue.lessThan(ShardDoc.java:160) at org.apache.solr.handler.component.ShardFieldSortedHitQueue.lessThan(ShardDoc.java:101) at org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:223) at org.apache.lucene.util.PriorityQueue.add(PriorityQueue.java:132) at org.apache.lucene.util.PriorityQueue.insertWithOverflow(PriorityQueue.java:148) at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:786) at org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:587) at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:566) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:283) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:351) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:890) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:944) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:680) It would be great if you can give me an overview of how its fixed in solr 4.0. Is it with the modifications to the ElevationComparatorSource class ? Thanks Srini -- View this message in context: http://lucene.472066.n3.nabble.com/QueryElevationComponent-and-distributed-search-tp3936998p3939953.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: QueryElevationComponent and distributed search
Can anyone help me out in understand the fix to QueryElevationComponent (in Solr 4.0) to make it work for distributed search. -- View this message in context: http://lucene.472066.n3.nabble.com/QueryElevationComponent-and-distributed-search-tp3936998p3942221.html Sent from the Solr - User mailing list archive at Nabble.com.
why no uppercase filter in solr
I am wondering why solr doesnt have an uppercase filter. I want the analyzed output to be in upper case to be compatible with legacy data. Will there be any problem if i create my own uppercase filter and use it ? -- View this message in context: http://lucene.472066.n3.nabble.com/why-no-uppercase-filter-in-solr-tp3984758.html Sent from the Solr - User mailing list archive at Nabble.com.
pass custom parameters from client to solr
Hi I would like to pass a bunch of custom parameters to turn on/off or set thresholds for various custom implementations we have in our solr. What would be the best way to pass these custom parameters to solr server from the client ? The client is a Java based application and hence we are planning to use solrj. Thanks Srini -- View this message in context: http://lucene.472066.n3.nabble.com/pass-custom-parameters-from-client-to-solr-tp3987511.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: pass custom parameters from client to solr
Thanks for your reply, that was helpful. Is there any restriction on how many parameters i can pass ? also can i pass some data (as we pass the document to be indexed, when we call the update url) along with the parameters which the solr server will use for search. -- View this message in context: http://lucene.472066.n3.nabble.com/pass-custom-parameters-from-client-to-solr-tp3987511p3987625.html Sent from the Solr - User mailing list archive at Nabble.com.
random results at specific slots
Hi, I would like to return results sorted by score (desc), but i would like to insert random results into some predefined slots (lets say 10, 14 and 18). The reason I want to do that is I boost click-through rate based features significantly and i want to give a chance to documents which doesnt have enough click through rate data. This would help the results stay fresh. I looked into solr code and it looks like i need a custom QueryComponent where once the top results are ordered, i can insert some random results at my predefined slots and then return. I am wondering whether there is any other way I can achieve the same? Thanks Srini -- View this message in context: http://lucene.472066.n3.nabble.com/random-results-at-specific-slots-tp3987719.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: random results at specific slots
Other option I could think of is to write a custom component which implements handleResponses, where i can pick random documents from across shards and insert it into the ResponseBuilder's resultIds ? I would place this component at the end (or after QueryCOmponent). will that work ? is there a better solution ? -- View this message in context: http://lucene.472066.n3.nabble.com/random-results-at-specific-slots-tp3987719p3987725.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: pass custom parameters from client to solr
What would be a good place to read the custom solr params I passed from the client to solr ? I saw that all the params passed to solr is available in rb.req. I have a business requirement to collapse or combine some properties together based on some conditions. Currently I have a custom component (added it as the first component in solrconfig), which reads the custom params from rb.req.getParams() and remove it from req and put it into context. I feel that probably custom component is not the best place and there could be a better place to do it. Does anyone have any suggestions ? -- View this message in context: http://lucene.472066.n3.nabble.com/pass-custom-parameters-from-client-to-solr-tp3987511p3988066.html Sent from the Solr - User mailing list archive at Nabble.com.
writing unit test for a search component which works only with distributed search
I have written a custom search component, but the code only supports distributed search. Since we dont use non-distributed search and the search works differently for non-distributed case, we decided to concentrate only on distributed search. I am trying to write unit test for my custom component. I could see that BaseDistributedSearchTestCase's query method compares the result of a single sharded control and multisharded test. I cannot use this method as my component would work only for multisharded search. Ideally I would like to use something like SolrTestCaseJ4's assertQ method, where i can use an xpath expression to validate the results. Does SolrTestCaseJ4 already support distributed search or do I need to customize it ? Or is there any other way to write unit test case that would work for distributed-only cases ? Thanks for your help! Srini -- View this message in context: http://lucene.472066.n3.nabble.com/writing-unit-test-for-a-search-component-which-works-only-with-distributed-search-tp3991795.html Sent from the Solr - User mailing list archive at Nabble.com.
many open index searchers listed in stats page
In the solr admin stats page I see multiple index searchers open. My
understanding is there will be two searchers open during replication and
only one otherwise. We have a multicore setup with 8 cores in the server
(each core with its own index). And we have master-slave replication setup.
Our replication poll interval is every 5 mins. Here is the information from
stats page for one of the cores
name:Searcher@56231e00 main
class: org.apache.solr.search.SolrIndexSearcher
version: 1.0
description: index searcher
stats: searcherName : Searcher@56231e00 main
caching : true
numDocs : 936408
maxDoc : 936592
reader :
SolrIndexReader{this=511a90e2,r=ReadOnlyDirectoryReader@511a90e2,refCnt=1,segments=16}
readerDir :
org.apache.lucene.store.NIOFSDirectory@/cacheDir/lucene/ns02/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@35896b6
indexVersion : 1343339555090
openedAt : Thu Aug 09 17:40:01 PDT 2012
registeredAt : Thu Aug 09 17:40:14 PDT 2012
warmupTime : 3600
name:core
class: ns02
version: 1.0
description: SolrCore
stats: coreName : ns02
startTime : Thu Aug 09 17:24:21 PDT 2012
refCount : 2
aliases : [ns02]
name:searcher
class: org.apache.solr.search.SolrIndexSearcher
version: 1.0
description: index searcher
stats: searcherName : Searcher@56231e00 main
caching : true
numDocs : 936408
maxDoc : 936592
reader :
SolrIndexReader{this=511a90e2,r=ReadOnlyDirectoryReader@511a90e2,refCnt=1,segments=16}
readerDir :
org.apache.lucene.store.NIOFSDirectory@/cacheDir/lucene/ns02/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@35896b6
indexVersion : 1343339555090
openedAt : Thu Aug 09 17:40:01 PDT 2012
registeredAt : Thu Aug 09 17:40:14 PDT 2012
warmupTime : 3600
name:Searcher@43234720 main
class: org.apache.solr.search.SolrIndexSearcher
version: 1.0
description: index searcher
stats: searcherName : Searcher@43234720 main
caching : true
numDocs : 836907
maxDoc : 836908
reader :
SolrIndexReader{this=143ed7c9,r=ReadOnlyDirectoryReader@143ed7c9,refCnt=1,segments=22}
readerDir :
org.apache.lucene.store.NIOFSDirectory@/cacheDir/lucene/ns02/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@35896b6
indexVersion : 1343339555061
openedAt : Thu Aug 09 17:24:21 PDT 2012
registeredAt : Thu Aug 09 17:24:39 PDT 2012
warmupTime : 0
name:Searcher@6fb8e6b9 main
class: org.apache.solr.search.SolrIndexSearcher
version: 1.0
description: index searcher
stats: searcherName : Searcher@6fb8e6b9 main
caching : true
numDocs : 916408
maxDoc : 916592
reader :
SolrIndexReader{this=4732ae19,r=ReadOnlyDirectoryReader@4732ae19,refCnt=1,segments=23}
readerDir :
org.apache.lucene.store.NIOFSDirectory@/cacheDir/lucene/ns02/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@35896b6
indexVersion : 1343339555084
openedAt : Thu Aug 09 17:25:12 PDT 2012
registeredAt : Thu Aug 09 17:25:27 PDT 2012
warmupTime : 5162
Note the refCount=1 for searchers. Also the num of segments and indexVersion
is different in each of the index searchers listed. Does it mean that there
is a leak somewhere and the index searcher doesnt close ?
--
View this message in context:
http://lucene.472066.n3.nabble.com/many-open-index-searchers-listed-in-stats-page-tp4000384.html
Sent from the Solr - User mailing list archive at Nabble.com.
NoHttpResponseException: The server failed to respond
We have a distributed solr setup with 8 servers and 8 cores on each server in production. We see this error multiple times in our solr servers. we are using solr 3.6.1. Has anyone seen this error before and have you resolved it ? 2012-09-04 02:16:40,995 [http-nio-8080-exec-7] ERROR org.apache.solr.core.SolrCore - org.apache.solr.common.SolrException at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:275) at com.nextag.search.solr.handler.ProductSearchHandler.handleRequestBody(ProductSearchHandler.java:269) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1001) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.solr.client.solrj.SolrServerException: org.apache.commons.httpclient.NoHttpResponseException: The server solr2-vip.servername.com failed to respond at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:475) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:249) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:129) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:103) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) ... 3 more Caused by: org.apache.commons.httpclient.NoHttpResponseException: The server solr2-vip.servername.com failed to respond at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1976) at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:419) Here is the shardHandlerFactory setting in our config 5000 3 We checked that Full GC is not running so frequently and the server is not too busy. -- View this message in context: http://lucene.472066.n3.nabble.com/NoHttpResponseException-The-server-failed-to-respond-tp4006017.html Sent from the Solr - User mailing list archive at Nabble.com.
