Sorting on solr results

HI All, Please provide me your idea for below problem. I required to sort product on webshop price with position. e.g. If we have three product (A, B ,C) needs to sort Price asc and position asc. ID Price Position A 10 3 B 10 2 C 20 5 Result should be so

Re: Questions about commits and OOE

On Wed, Dec 4, 2013 at 6:36 PM, OSMAN Metin wrote: > During this massive update, we have sometimes a peak of active threads > exceeding the limit of 8192 process authorized for the user running the > tomcat and zookeeper process. > When this happens, every hardCommit is failing with an "OutOfMemor

Re: facet.method=fcs vs facet.method=fc on solr slaves

Hello Patrick, Replication flushes UnInvertedField cache that impacts fc, but doesn't harm Lucene's FieldCache which is for fcs. You can check how much time in millis is spend on UnInvertedField cache regeneration in INFO logs like "UnInverted multi-valued field ,time=### ..." On Thu, Dec 5,

Re: SOLR 4 not utilizing multi CPU cores

Hi, I did moreless the same but didn't get that behaviour...could you give us more details Best, Gazza On 5 Dec 2013 06:54, "Salman Akram" wrote: > Hi, > > We recently upgraded to SOLR 4.6 from SOLR 1.4.1. Overall the performance > went down for large phrase queries. On some analysis we have see

SOLR 4 not utilizing multi CPU cores

Hi, We recently upgraded to SOLR 4.6 from SOLR 1.4.1. Overall the performance went down for large phrase queries. On some analysis we have seen that 1.4.1 utilized multiple cpu cores for such queries but SOLR 4.6 is only utilizing single cpu core. Any idea on what could be the reason? Note: We ar

Re: SOLR Master-Slave Repeater with Load balancer

Erick is right, you have been put in a terrible position. You need to get agreement, in writing, that it is OK for search to go down when one server is out of service. This might be for scheduled maintenance or even a config update. When one server is down, search is down, period. This requirem

Re: SOLR Master-Slave Repeater with Load balancer

bq: but we have limitation on the number of servers that we can use due to budget concerns (limit is 2) really, really, really push back to your project managers on this. So what you need 3 machines for a ZooKeeper quorum? The needs of ZK are quite light, they don't need a powerful machine. Your

Re: a core for every user, lots of users... are there issues

I don't know of anyone who's tried and failed to combine transient cores and SolrCloud. I also don't know of anyone who's tried and succeeded. I'm saying that the transient core stuff has been thoroughly tested in non-cloud mode. And people have been working with it for a couple of releases now. I

Re: how to increase each index file size

Hi Erick Thanks for your reply. Regards 2013/12/4 Erick Erickson > Why do you want to do this? Are you seeing performance problems? > If not, I'd just ignore this problem, premature optimization and all that. > > If you _really_ want to do this, your segments files are closed every > time

Prioritize search returns by URL path?

We have a Telligent based community with Solr as the search engine. We want to prioritize search returns from within the community by the type of content: Wiki articles as most relevant, then blog posts, then Verified answer and Suggested answer forum posts, then remaining forum posts. We have also

Re: Inconsistent numFound in SC when querying core directly

Keep in mind, there have been a *lot* of bug fixes since 4.3.1. - Mark On Dec 4, 2013, at 7:07 PM, Tim Vaillancourt wrote: > Hey all, > > Now that I am getting correct results with "distrib=false", I've identified > that 1 of my nodes has just 1/3rd of the total data set and totally explains

Re: Inconsistent numFound in SC when querying core directly

Hey all, Now that I am getting correct results with "distrib=false", I've identified that 1 of my nodes has just 1/3rd of the total data set and totally explains the flapping in results. The fix for this is obvious (rebuild replica) but the cause is less obvious. There is definately more tha

Re: SOLR Master-Slave Repeater with Load balancer

Hi Erick, Thanks a lot for your explanation. We initially considered Solr Cloud but we have limitation on the number of servers that we can use due to budget concerns (limit is 2) Solr Cloud requires minimum 3. I have tried out the solution you suggested and so far its going well and we are not doi

Re: Inconsistent numFound in SC when querying core directly

Chris, this is extremely helpful and it's silly I didn't think of this sooner! Thanks a lot, this makes the situation make much more sense. I will gather some proper data with your suggestion and get back to the thread shortly. Thanks!! Tim On 04/12/13 02:57 PM, Chris Hostetter wrote: : :

Re: Inconsistent numFound in SC when querying core directly

: : I may be incorrect here, but I assumed when querying a single core of a : SolrCloud collection, the SolrCloud routing is bypassed and I am talking : directly to a plain/non-SolrCloud core. No ... every query received from a client by solr is handled by a single core -- if that core knows it'

Re: Inconsistent numFound in SC when querying core directly

Thanks Markus, I'm not sure if I'm encountering the same issue. This JIRA mentions 10s of docs difference, I'm seeing differences in the multi-millions of docs, and even more strangely it very predictably flaps between a 123M value and an 87M value, a 30M+ doc difference. Secondly, I'm not c

RE: Inconsistent numFound in SC when querying core directly

https://issues.apache.org/jira/browse/SOLR-4260 Join the club Tim! Can you upgrade to trunk or incorporate the latest patches of related issues? You can fix it by trashing the bad node's data, although without multiple clusters it may be difficult to decide which node is bad. We use the latest

Re: Inconsistent numFound in SC when querying core directly

To add two more pieces of data: 1) This occurs with real, conditional queries as well (eg: "q=key:timvaillancourt"), not just the "q=*:*" I provided in my email. 2) I've noticed when I bring a node of the SolrCloud down it is remaining "state: active" in my /clusterstate.json - something is rea

Re: Shouldn't fuzzy version of a solr query always return a super set of its not-fuzzy equivalent

I'm using snowball stemmer and, you are correct, swimming has been stored as swim. Should I wrap snowball filter in a multiterm analyzer? Thanks On Dec 4, 2013 2:02 PM, "Jack Krupansky" wrote: > Ah... although the lower case filtering does get applied properly in a > "multiterm" analysis scena

Inconsistent numFound in SC when querying core directly

Hey guys, I'm looking into a strange issue on an unhealthy 4.3.1 SolrCloud with 3-node external Zookeeper and 1 collection (2 shards, 2 replicas). Currently we are noticing inconsistent results from the SolrCloud when performing the same simple /select query many times to our collection. Alm

Re: a core for every user, lots of users... are there issues

Oh my... when you say "I don't know anyone who's combined the two." do you mean that those that have tried have failed or that no one has gotten around to trying? It sounds like you are saying you have some specific knowledge that right now these wont work, otherwise you wouldnt say "committers wil

Re: Shouldn't fuzzy version of a solr query always return a super set of its not-fuzzy equivalent

Ah... although the lower case filtering does get applied properly in a "multiterm" analysis scenario, stemming does not. What stemmer are you using? I suspect that "swimming" normally becomes "swim". Compare the debug output of the two queries. -- Jack Krupansky -Original Message- Fr

Re: a core for every user, lots of users... are there issues

Hank: I should add that lots of cores and SolrCloud aren't guaranteed to play nice together. I think some of the committers will be addressing this sometime soon. I'm not saying that this will certainly fail, OTOH I don't know anyone who's combined the two. Erick On Wed, Dec 4, 2013 at 3:18 PM

Re: Solr Stalls on Bulk indexing, no logs or errors

Wait, crashes? Or just stops accepting updates? At any rate, this should be fixed in 4.6. If you can dump a stack trace, we can identify whether this is the same issue quickly. jstack is popular. If you're still having queries served, it's probably not your commit settings, try searching the JIRA

Re: starting up solr automatically

thanks greg I got it starting but the collection file is not avail. I will use the script that you gave the url for and the env settings. Thanks On Wed, Dec 4, 2013 at 4:26 PM, Greg Walters wrote: > I almost forgot, you'll need a file to setup the environment a bit too: > > ** > JAVA_HOME=/usr/

Re: starting up solr automatically

I almost forgot, you'll need a file to setup the environment a bit too: ** JAVA_HOME=/usr/java/default JAVA_OPTIONS="-Xmx15g \ -Xms15g \ -XX:+PrintGCApplicationStoppedTime \ -XX:+PrintGCDateStamps \ -XX:+PrintGCDetails \ -XX:+UseConcMarkSweepGC \ -XX:+UseParNewGC \ -XX:+UseTLAB \ -XX:+CMSParallelR

Re: Querying for results

Follow-up: Would anyone very familiar with DIH be willing to jump on a side thread with me and my developer to help troubleshoot some issues we're having? Please little r me at: robert [at] mavenbridge.com. Thanks! On Wed, Dec 4, 2013 at 1:14 PM, Rob Veliz wrote: > Hello, > > I am running S

Re: starting up solr automatically

I found the instructions and scripts on that page to be unclear and/or not work. Here's the script I've been using for solr 4.5.1: https://gist.github.com/gregwalters/7795791 Do note that you'll have to change a couple of paths to get things working correctly. Thanks, Greg On Dec 4, 2013, at 3

starting up solr automatically

Hey all, I'm pretty new to solr. I'm installing it on an amazon linux (rpm based) ec2 instance and have it running. I even have nutch feeding it pages from a crawl. I'm very happy about that. I want solr to start on a reboot and am following the instructions at http://wiki.apache.org/solr/SolrJ

Querying for results

Hello, I am running Solr from Magento and using DIH to import/index data from 1 other source (external). I am trying to query for results...two questions: 1. The query I'm using runs against "fulltext_1_en" which is a specific shard created by the Magento deployment in solrconfig.xml. Should I

Re: Solr Stalls on Bulk indexing, no logs or errors

Yes I can continue to query after this importer goes down and whilst it running. The bulk commit is done via a JSON handler in php. There is 121,000 records that need to go into the index. So this is done in 5000 chunked mySQL retrieve calls and parsing to the data as required. workflo

Re: a core for every user, lots of users... are there issues

Super helpful. Thanks. On Wed, Dec 4, 2013 at 2:53 PM, Shawn Heisey wrote: > On 12/4/2013 12:34 PM, hank williams wrote: > >> Ok one more simple question. We just upgraded to 4.6 from 4.2. In 4.2 we >> were *trying* to use the rest API function "create" to create cores >> without >> having to m

facet.method=fcs vs facet.method=fc on solr slaves

Is there any advantage on a Solr slave to receive queries using facet.method=fcs instead of the default of facet.method=fc? Most of the segment files are unchanged between replication events - but I wasn't sure if replication would cause the unchanged segment field caches to be lost anyway. -- Pat

Re: Tika not extracting content from ODT / ODS (open document / libreoffice) in Solr 4.2.1

Hello everybody, First of all, sorry about my bad english. Giving updates on this bug, i maybe have found a solution for it. I would like to have opinions on this solution. I have found out that tika, when reading .odt files, would return more than one document. The first one for content.xml, w

Re: Solr Stalls on Bulk indexing, no logs or errors

There's a known issue with SolrCloud with multiple shards, but you haven't told us whether you're using that. The test for whether you're running in to that is whether you can continue to _query_, just not update. But you need to tell us more about our setup. In particular hour commit settings (ha

Re: a core for every user, lots of users... are there issues

On 12/4/2013 12:34 PM, hank williams wrote: Ok one more simple question. We just upgraded to 4.6 from 4.2. In 4.2 we were *trying* to use the rest API function "create" to create cores without having to manually mess with files on the server. Is this what "create" was supposed to do? If so it was

Re: a core for every user, lots of users... are there issues

Ok one more simple question. We just upgraded to 4.6 from 4.2. In 4.2 we were *trying* to use the rest API function "create" to create cores without having to manually mess with files on the server. Is this what "create" was supposed to do? If so it was borken or we werent using it right. In any ca

Solr Stalls on Bulk indexing, no logs or errors

I am finding with a bulk index using SOLR 4.3 on Tomcat, that when I reach 69578 records the server stops adding anything more. I've tried reducing the data sent to the bare minimum of fields and using ASC and DESC data to see if it could be a field issue. Is there anything I could look at for t

Re: Shouldn't fuzzy version of a solr query always return a super set of its not-fuzzy equivalent

Debug shows that all terms are lowercased properly. Thanks On Dec 4, 2013 3:18 AM, "Erik Hatcher" wrote: > Chances are you're not getting those fuzzy terms analyzed as you'd like. > See debug (&debug=true) output to be sure. Most likely the fuzzy terms > are not being lowercased. See > http:/

Re: Setting routerField/shardKey on specific collection?

Many thanks Timothy, I tried this today but ran into issues getting the new collection to persist (so that I could search for the parameter). It's good to have this confirmed as a viable approach though, and I'll persevere with this tomorrow. If I figure it out I'll reply with the details. Th

RE: Setting routerField/shardKey on specific collection?

Hi Daniel, I'm not sure how this would apply to an existing collection (in your case collection1). Try using the collections API to create a new collection and pass the router.field parameter. Grep'ing over the code, the parameter is named: router.field (not routerField or routeField). Cheers,

Re: Questions about commits and OOE

I'd second the use of jstack to check your threads. Each request (be it a search or update) will generate a request handler thread on the Solr side (unless you've set the limits in the HttpShardHandlerFactory (solr.xml for solr-wide faults and/or under the requestHandler in SolrConfig.xml), we set

Re: json update moves doc to end

: Well, both have a score of -Infinity. So they're "equal" and : the tiebreaker is the internal Lucene doc ID. : : Now this is not helpful since the question now is where : -Infinity comes from, this looks suspicious: : -Infinity = (MATCH) FunctionQuery(log(int(clicks))), product of: : -Infi

Re: SolrCloud FunctionQuery inconsistency

: There is no default value for ptime. It is generated by users. thank you, that rules out my previous wild guess. : I was trying query with a function query({!boost b=dateDeboost(ptime)} : channelid:0082 && title:abc), which leads differents results from the same : shard(using the param: shards

Re: Programmatically upload configuration into ZooKeeper

On 12/4/2013 9:23 AM, Artem Karpenko wrote: > so it's SolrZkClient indeed. I've tried it out and it seems to do just > the job I need. Thank you! > > On a related note - is there a similar way to create/reload > core/collection, using maybe CloudSolrServer or smth. inside it? Didn't > found any me

RE: json update moves doc to end

I changed my boost-function log(clickrate)^8 to div(clciks,displays)^8 and it works now. I get the following output from debug 0.0022668892 = (MATCH) FunctionQuery(div(const(2),const(5))), product of: 0.4 = div(const(2),const(5)) 8.0 = boost 7.0840283E-4 = queryNorm Am i undestanding

Setting routerField/shardKey on specific collection?

Hi, I'm using Solr 4.6 and trying to specify a router.field (shard key) on a specific collection so that all documents with the same value in the specified field end up in the same collection. However, I can't find an example of how to do this via the solr.xml? I see in this ticket https://i

Re: Programmatically upload configuration into ZooKeeper

Hello Greg, so it's SolrZkClient indeed. I've tried it out and it seems to do just the job I need. Thank you! On a related note - is there a similar way to create/reload core/collection, using maybe CloudSolrServer or smth. inside it? Didn't found any methods that could do the thing. Regar

RE: Questions about commits and OOE

Hi Metin, I think removing the softCommit=true parameter on the client side will definitely help as NRT wasn't designed to re-open searchers after every document. Try every 1 second (or even every few seconds), I doubt your users will notice. To get an idea of what threads are running in your J

Re: Solr Performance Issue

On 12/4/2013 6:31 AM, kumar wrote: > I am having almost 5 to 6 crores of indexed documents in solr. And when i am > going to change anything in the configuration file solr server is going > down. If you mean crore and not core, then you are talking about 50 to 60 million documents. That's a lot.

Re: Programmatically upload configuration into ZooKeeper

Hi Artem, This question (or one very like it) has been asked on this list before so there's some prior art you could modify to suit your needs. Taken from Timothy Potter : ** public static void updateClusterstateJsonInZk(CloudSolrServer cloudSolrServer, CommandLine cli) throws Exception {

Re: Using Payloads as a Coefficient For Score At a Custom QParser That extends ExtendedDismaxQParser

Sounds great Furkan, Do you have the permission to donate this code? I would be great if you could create a JIra ticket. Thanks, Joel On Tue, Dec 3, 2013 at 3:26 PM, Furkan KAMACI wrote: > I've implemented what I want. I can add payload score into the document > score. I've modified ExtendedDi

Questions about commits and OOE

Hi all, let me first explain our situation : We have - two virtual servers with each : 4x SolR 4.4.0 on Tomcat 6 (+ with mod_cluster 1.2.0), each JVM has -Xms2048m -Xmx2048m -XX:MaxPermSize=384m 1x Zookeeper 3.4.5 (Only one of the two Zookeeper is active.) CentOS 6.4 Sun JDK 1.6.0-31 16

Programmatically upload configuration into ZooKeeper

What is the best way to upload Solr configuration files into ZooKeeper programmatically, i.e. - from within Java code? I know that there are cloud-scripts for this, but in the end they should use some Java client library, don't they? This question raised because we use special configuration sys

Re: Solr Doubts

bq: id This isn't correct, there's no "required" param for . Just remove the entire node AND make the field definition required="false". I.e. you should have something like: set required="false" there. To increase memory, you just specify -Xmx when you start, something like: java -Xmx2G -Xms2G

Re: json update moves doc to end

Well, both have a score of -Infinity. So they're "equal" and the tiebreaker is the internal Lucene doc ID. Now this is not helpful since the question now is where -Infinity comes from, this looks suspicious: -Infinity = (MATCH) FunctionQuery(log(int(clicks))), product of: -Infinity = log(int(

Re: Solr Performance Issue

You need to give us more of the exception trace, the real cause is often buried down the stack with some text like "Caused by..." But at a glance your cache sizes and autowarm counts are far higher than they should be. Try reducing particularly the autowarm count down to, say, 16 or so. It's actua

Re: how to increase each index file size

Why do you want to do this? Are you seeing performance problems? If not, I'd just ignore this problem, premature optimization and all that. If you _really_ want to do this, your segments files are closed every time you to a commit, opensearcher=true|false doesn't matter. BUT, the longer these are

Re: post filtering for boolean filter queries

OK, so cache=false and cost=100 should do it, see: http://searchhub.org/2012/02/22/custom-security-filtering-in-solr/ Best, Erick On Wed, Dec 4, 2013 at 5:56 AM, Dmitry Kan wrote: > Thanks Yonik. > > For our use case, we would like to skip caching only one particular filter > cache, yet apply

Solr Performance Issue

I am having almost 5 to 6 crores of indexed documents in solr. And when i am going to change anything in the configuration file solr server is going down. As a new user to solr i can't able to find the exact reason for going server down. I am using cache's in the following way : and i am not

Re: Faceting Query in Solr

The standard way of handling this kind of thing is with filter queries. For multi-select, you have to put in some javascript or something to make an OR clause when they check the boxes. So your query looks like fq=categoryID:(1 OR 2 OR 3) rather than fq=categoryID:1&fq=categoryID:2&fq=categoryID:3

Re: solrinitialisationerrors: Error during shutdown of writer.

The crux is: "java.lang.NoClassDefFoundError:" Usually this means your classpath is wrong and the JVM can't find the jars. Or you have multiple jars from different versions in your classpath. It's pretty tedious to track down, but that's where I'd start. In your log, you'll see a bunch of lines

Re: Deleting and committing inside a SearchComponent

I agree with Upayavira. This seems architecturally questionable. In your example, the crux of the matter is "Only differ by one field". Figuring that out is going to be expensive, are you burdening searches with this kind of logic? Why not create a custom update processor that does this and use s

Re: [Solr Wiki] Your wiki account data

Sure. Unfortunately we had a problem a while ago with spam bots creating pages so had to lock it down. Done, you should be able to edit the Solr Wiki. Erick On Wed, Dec 4, 2013 at 8:06 AM, Mehdi Burgy wrote: > Hello, > > We've recently launched a job search engine using Solr, and would like t

Fwd: [Solr Wiki] Your wiki account data

Hello, We've recently launched a job search engine using Solr, and would like to add it here: https://wiki.apache.org/solr/PublicServers Would it be possible to allow me be part of the publishing group? Thank you for your help Kind Regards, Mehdi Burgy New Job Search Engine: www.jobreez.com -

Solr Doubts

Hai Team, I am new to Solr. I am trying to index 7GB CSV file. My questions: 1.How to index without using uniquekey ? I tried with id I got --> Document is missing mandatory uniqueKey field: id I am using query to update csv : localhost:9050/solr-4.5.1/collection1/update/csv?stream.fil

RE: json update moves doc to end

Hi Erick Here are the last 2 results from a search and i am not understanding why the last one with the boost editorschoice^200 isn't at the top. By the way can i also give a substantial boost to results that contain the hole search-request and not just 3 or 4 letters (tokens)? -Infinity = (MATC

Re: Using the flexible query parser in Solr instead of classic

Hi Jack Kurpansky, Hi folks We could recreate the edismax QueryParser from classic to flexible. But is this a need for someone else? In long text: ExtendedDismaxQParser uses ExtendedSolrQueryParser. ExtendedSolrQueryParser is derived from SolrQueryParser. So it is based on org.apache.solr.parse

Solr cuts highlighted sentences

Hi guys, when searching for a phrase I get results and would like to show a highlighting. The highlightings beeing shown begin somewhere in the sentence, beginning with a coma or something else. I'd like to get highlightings beginning with a stence. How to manage this. I've tried so many things fo

Re: What type of Solr plugin do I need for filtering results?

Thanks a lot for both of your answers! The QParserPlugin is probably what I meant, but join queries also look interesting and like the could maybe solve my use case, too, without any custom code. However, since this would make it impossible (I think) to have a score for the results but I do want

Re: Shouldn't fuzzy version of a solr query always return a super set of its not-fuzzy equivalent

Chances are you're not getting those fuzzy terms analyzed as you'd like. See debug (&debug=true) output to be sure. Most likely the fuzzy terms are not being lowercased. See http://wiki.apache.org/solr/MultitermQueryAnalysis for more details (this applies to fuzzy, not just wildcard) terms to

Re: post filtering for boolean filter queries

Thanks Yonik. For our use case, we would like to skip caching only one particular filter cache, yet apply a high cost for it to make sure it executes last of all filter queries. So this means, the rest of the fqs will execute and cache as usual. On Tue, Dec 3, 2013 at 9:58 PM, Yonik Seeley w

RE: SolrCloud FunctionQuery inconsistency

Hi Raju, Collection is a concept in solrcloud, and core is in standalone mode. So you can create multiple cores in solr standalone mode, not collections. -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-FunctionQuery-inconsistency-tp4104346p4104888.html Sent from th

Re: Automatically build spellcheck dictionary on replicas

Ok, thanks for pointing that out! 2013/12/3 Kydryavtsev Andrey > Yep, sorry, it doesn't work for file-based dictionaries: > > > In particular, you still need to index the dictionary file once by > issuing a search with &spellcheck.build=true on the end of the URL; if you > system doesn't update

Solr Suggester ranked by boost

I want to implement a Solr Suggester (http://wiki.apache.org/solr/Suggester) that ranks suggestions by document boost factor. As I understand the documentation, the following config should work: Solrconfig.xml: ... true 7 true

Shouldn't fuzzy version of a solr query always return a super set of its not-fuzzy equivalent

I'm using the following query to do a fuzzy search on Solr 4.5.1 and am getting empty result. qt=standard&q=+(field1|en_CA|:Swimming~2 field1|en|:Swimming~2) +(field1|en_CA|:Goggle~1 field1|en|:Goggle~1) +(+startDate:[* TO 2013-12-04T00:23:00Z] -endDate:[* TO 2013-12-04T00:23:00Z])&start=0&rows=10

how to increase each index file size

Hi I'm using the SolrCloud integreted with HDFS,I found there are lots of small size files. So,I'd like to increase the index file size while doing DIH full-import. Any suggestion to achieve this goal. Regards.

Faceting Query in Solr

Hi, I indexed data into solr by using 5 categories. Each category is differentiated by categoryId. Now i have a situation that i need to show the results based on facets. Ex: []-category1 []-category2 []-category3 []-category4 []-category5 If the user checks the category1 it has to show the re

solrinitialisationerrors: Error during shutdown of writer.

I dont why all of a sudden I started getting this errror : this is the sreenshot: I thought there might be some problem with tomcat,so I uninstalled it ,but i still get the same error. I have no idea why is this happening,initially i