Re: Sorting performance
Here are the memory parameters I'm using now(Tomcat): -Xms2024m -Xmx2024m With those values, the second query is way faster. Only the first one is very slow. Thanks for the tip. However, I'm wondering if will be enough and I will not hit the same issues when I will have many users searching at the same time: I will do a stress test to check this. Thanks Christophe christophe wrote: It is slow each time I run it. (I test it from the Solr admin console or from a JAVA program using the Http client). I do not get the OOM each time. Thx Christophe Otis Gospodnetic wrote: Is the sorted query slow only the first time or every time you run it? You got an OOM? What -Xmx value are you using? Try increasing it. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: christophe <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Friday, October 17, 2008 1:28:52 PM Subject: Sorting performance Hi, I'm doing some tests with Solr1.3 I have loaded around 7M documents, each with a few stored and indexed fields. This query: text:sometext returns the results, sorted by score in a few milliseconds. (I display 10 out of 8747 matched documents) This one: text:sometext;id desc takes something like 60s or more to return the data (when it doesn't fails with an out of memory error). (id is a string type). I have tried to display only id, same results. Any ideas ? I'm sure I'm doing something wrong. My schema is based on the sample, with the following fields: /> multiValued="true" /> default="NOW" multiValued="false"/> Thanks Christophe
solr 1.3 multi language?
Hi everybody, I would like you to help me a bit about managing this multi-language part, actually an example would be excellent. So I did multi index in one core and I would like you to let me know what you think about the way that I've managed that, is there more parameter that I don't know, some help and an example would be great full. Thanks a lot, I need to manage this language : French (FR) English (EN) German (DE) Spanish (ES) Russian (RU) Portuguese (Brazilian) (PT) Polish (PO) Dutch (NL) Greek (GR) Japanese (JA) Turkish (TR) My schema looks like that : ... .. My data-config: video.title, IF(video.language = 'fr', video.title, NULL) as title_fr, IF(video.language = 'en', video.title, NULL) as title_en, IF(video.language = 'de', video.title, NULL) as title_de, IF(video.language = 'es', video.title, NULL) as title_es, IF(video.language = 'nl', video.title, NULL) as title_nl, IF(video.language = 'el', video.title, NULL) as title_el, IF(video.language = 'ja', video.title, NULL) as title_ja, IF(video.language = 'it', video.title, NULL) as title_it, video.description, IF(video.language = 'fr', video.description, NULL) as description_fr, IF(video.language = 'en', video.description, NULL) as description_en, IF(video.language = 'de', video.description, NULL) as description_de, IF(video.language = 'es', video.description, NULL) as description_es, IF(video.language = 'nl', video.description, NULL) as description_nl, IF(video.language = 'el', video.description, NULL) as description_el, IF(video.language = 'ja', video.description, NULL) as description_ja, IF(video.language = 'it', video.description, NULL) as description_it, Thanks for your time, -- View this message in context: http://www.nabble.com/solr-1.3-multi-language--tp20047636p20047636.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Tree Faceting Component
Jeremy, Great troubleshooting! You were spot on. I've posted a new patch that fixes the issue. Erik On Oct 16, 2008, at 9:53 PM, Jeremy Hinegardner wrote: After a bit more investigating, it appears that any facet tree where the first item is numerical or boolean or some non-textual type does not produce any secondary facets. This includes sint, sfloat, boolean and such. For instance, on the sample index: facet.tree=sku,cat => works facet.tree=cat,sku => works facet.tree=manu_exact,cat => works facet.tree=cat,manu_exact => works facet.tree=popularity,inStock => fails facet.tree=inStock,popularity => fails facet.tree=manu_exact,weight => works facet.tree=weight,manu_exact => fails I'm not very familiar with the Solr / Lucene Java API, so this is slow going here. Maybe I'm barking up the wrong tree, but is the TermQuery for the secondary SimpleFacet messing up some how? I tried to dig into the code, but was unsuccessful. It appears to me that the searcher never returns a docSet for any TermQuery where the field being searched has a type that is non-textual. As a final test, I changed the schema and made the inStock field a 'text' field instead of 'boolean'. When I did that, and reindexed the sample data then the tree facet would work correctly as either facet.tree=cat,inStock or facet.tree=inStock,cat. Whereas before it would only work in the former. enjoy, -jeremy On Thu, Oct 16, 2008 at 10:55:49AM -0600, Jeremy Hinegardner wrote: Erik, After some more experiments, I can get it to perform incorrectly using the sample solr data. The example query from SOLR-792 ticket: http://localhost:8983/solr/select?q=*:*&rows=0&facet=on&facet.field=cat&facet.tree=cat,inStock&wt=json&indent=on Make a few altertions to the query: 1) swap the tree order - all tree facets are 0 http://localhost:8983/solr/select?q=*:*&rows=0&facet=on&facet.field=cat&facet.tree=inStock,cat&wt=json&indent=on 2) swap tree order and change facet.field to be the primary( inStock ) http://localhost:8983/solr/select?q=*:*&rows=0&facet=on&facet.field=inStock&facet.tree=inStock,cat&wt=json&indent=on Also, can tree faceting work distributed? enjoy, -jeremy On Wed, Oct 15, 2008 at 05:41:21PM -0700, Erik Hatcher wrote: Jeremy, What's the full request you're making to Solr? Do you get values when you facet normally on date_id and type? &facet.field=date_id&facet.field=type Erik p.s. this e-mail is not on the list (on a hotel net connection blocking outgoing mail) - feel free to reply to this back on the list though. On Oct 15, 2008, at 5:29 PM, Jeremy Hinegardner wrote: Hi all, I'm testing out using the Tree Faceting Component (SOLR-792) on top of Solr 1.3. It looks like it would do exactly what I want, but something is not working correctly with my schema. When I use the example schema, it works just fine, but I swap out the example schema's and example index and then put in my index and and schema, tree facet does not work. Both of the fields I want to facet can be faceted individually, but when I say facet.tree=date_id,type then all of the values are 0. Does anyone have any ideas on where I should start looking ? enjoy, -jeremy -- = = = = = === Jeremy Hinegardner [EMAIL PROTECTED] -- = = = = Jeremy Hinegardner [EMAIL PROTECTED] -- = = == Jeremy Hinegardner [EMAIL PROTECTED]
Re: Tree Faceting Component
Erik, Thanks, its working great. Next is to make it distributed. I was thinking of working on this, is the FacetCompoent a good model to work from to make the TreeFacet distributed? I should probably join solr-dev for that conversation I assume :-). -jeremy On Thu, Oct 16, 2008 at 11:12:45PM -0700, Erik Hatcher wrote: > Jeremy, > > Great troubleshooting! You were spot on. > > I've posted a new patch that fixes the issue. > > Erik > > > On Oct 16, 2008, at 9:53 PM, Jeremy Hinegardner wrote: > >> After a bit more investigating, it appears that any facet tree where the >> first >> item is numerical or boolean or some non-textual type does not produce any >> secondary facets. This includes sint, sfloat, boolean and such. >> >> For instance, on the sample index: >> >> facet.tree=sku,cat => works >> facet.tree=cat,sku => works >> facet.tree=manu_exact,cat => works >> facet.tree=cat,manu_exact => works >> facet.tree=popularity,inStock => fails >> facet.tree=inStock,popularity => fails >> facet.tree=manu_exact,weight => works >> facet.tree=weight,manu_exact => fails >> >> I'm not very familiar with the Solr / Lucene Java API, so this is slow >> going >> here. Maybe I'm barking up the wrong tree, but is the TermQuery for the >> secondary SimpleFacet messing up some how? I tried to dig into the code, >> but >> was unsuccessful. >> >> It appears to me that the searcher never returns a docSet for any >> TermQuery >> where the field being searched has a type that is non-textual. >> >> As a final test, I changed the schema and made the inStock field a 'text' >> field >> instead of 'boolean'. When I did that, and reindexed the sample data then >> the >> tree facet would work correctly as either facet.tree=cat,inStock or >> facet.tree=inStock,cat. Whereas before it would only work in the former. >> >> enjoy, >> >> -jeremy >> >> On Thu, Oct 16, 2008 at 10:55:49AM -0600, Jeremy Hinegardner wrote: >>> Erik, >>> >>> After some more experiments, I can get it to perform incorrectly using >>> the >>> sample solr data. >>> >>> The example query from SOLR-792 ticket: >>> >>> http://localhost:8983/solr/select?q=*:*&rows=0&facet=on&facet.field=cat&facet.tree=cat,inStock&wt=json&indent=on >>> >>> Make a few altertions to the query: >>> >>> 1) swap the tree order - all tree facets are 0 >>> >>> http://localhost:8983/solr/select?q=*:*&rows=0&facet=on&facet.field=cat&facet.tree=inStock,cat&wt=json&indent=on >>> >>> 2) swap tree order and change facet.field to be the primary( inStock ) >>> >>> http://localhost:8983/solr/select?q=*:*&rows=0&facet=on&facet.field=inStock&facet.tree=inStock,cat&wt=json&indent=on >>> >>> Also, can tree faceting work distributed? >>> >>> enjoy, >>> >>> -jeremy >>> >>> On Wed, Oct 15, 2008 at 05:41:21PM -0700, Erik Hatcher wrote: Jeremy, What's the full request you're making to Solr? Do you get values when you facet normally on date_id and type? &facet.field=date_id&facet.field=type Erik p.s. this e-mail is not on the list (on a hotel net connection blocking outgoing mail) - feel free to reply to this back on the list though. On Oct 15, 2008, at 5:29 PM, Jeremy Hinegardner wrote: > Hi all, > > I'm testing out using the Tree Faceting Component (SOLR-792) on top of > Solr 1.3. > > It looks like it would do exactly what I want, but something is not > working > correctly with my schema. When I use the example schema, it works just > fine, > but I swap out the example schema's and example index and then put in > my > index > and and schema, tree facet does not work. > > Both of the fields I want to facet can be faceted individually, but > when I > say > facet.tree=date_id,type then all of the values are 0. > > Does anyone have any ideas on where I should start looking ? > > enjoy, > > -jeremy > > -- > > Jeremy Hinegardner [EMAIL PROTECTED] >>> >>> -- >>> >>> Jeremy Hinegardner [EMAIL PROTECTED] >>> >> >> -- >> >> Jeremy Hinegardner [EMAIL PROTECTED] > -- Jeremy Hinegardner [EMAIL PROTECTED]
Re: Sorting performance
You need to setup a warming query that sorts so that the initial long query is done behind the scenes. Users first query will then be fast. Solrconfig. - Mark On Oct 18, 2008, at 1:34 AM, christophe <[EMAIL PROTECTED]> wrote: Here are the memory parameters I'm using now(Tomcat): -Xms2024m - Xmx2024m With those values, the second query is way faster. Only the first one is very slow. Thanks for the tip. However, I'm wondering if will be enough and I will not hit the same issues when I will have many users searching at the same time: I will do a stress test to check this. Thanks Christophe christophe wrote: It is slow each time I run it. (I test it from the Solr admin console or from a JAVA program using the Http client). I do not get the OOM each time. Thx Christophe Otis Gospodnetic wrote: Is the sorted query slow only the first time or every time you run it? You got an OOM? What -Xmx value are you using? Try increasing it. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: christophe <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Friday, October 17, 2008 1:28:52 PM Subject: Sorting performance Hi, I'm doing some tests with Solr1.3 I have loaded around 7M documents, each with a few stored and indexed fields. This query: text:sometext returns the results, sorted by score in a few milliseconds. (I display 10 out of 8747 matched documents) This one: text:sometext;id desc takes something like 60s or more to return the data (when it doesn't fails with an out of memory error). (id is a string type). I have tried to display only id, same results. Any ideas ? I'm sure I'm doing something wrong. My schema is based on the sample, with the following fields: /> multiValued="true" /> default="NOW" multiValued="false"/> Thanks Christophe
