Re: optimization advice?

String ;-) - we only allow sorting on string fields. On Fri, Mar 27, 2009 at 9:21 PM, Otis Gospodnetic wrote: > > Steve, > > A field named "name" sounds like a free text field.  What is its type, string > or text?  Fields you sort by should not be tokenized and should be indexed.   > I have a hu

Re: solr date parsing issue

On Sat, Mar 28, 2009 at 8:17 AM, Suryasnat Das wrote: > Hi, > > I am implementing a project using SOLR in which we need to do a search > based > on date range. I am passing the date in SOLR date format. During formation > of the SOLR query i am encoding the date string using UTF-8 encoding. After

Re: optimization advice?

Steve, A field named "name" sounds like a free text field. What is its type, string or text? Fields you sort by should not be tokenized and should be indexed. I have a hunch your name field is tokenized. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Me

Re: solr date parsing issue

Mr. Das, Can you provide a little more details here? Helpful information would be: - The query string you're using - The fieldtype you're using for indexing the value in question. - The exact error message you're getting from Solr. Suryasnat Das wrote: > > Hi, > > I am implementing a proj

Re: optimization advice?

We sort by default on "name", which varies quite a bit (we're never going to make sorting by field go away). The thing is solr has been pretty amazing across 1 million records. Now that we've doubled the size of the dataset things are definitely slower in a nonlinear way...I'm wondering what facto

More Robust Search Timeouts (to Kill Zombie Queries)?

I've noticed that some of my queries take so long (5 min+) that by the time they return, there is no longer any plausible use for the search results. I've started calling these zombie queries because, well, they should be dead, but they just won't die. Instead, they stick around, wasting my Solr bo

solr date parsing issue

Hi, I am implementing a project using SOLR in which we need to do a search based on date range. I am passing the date in SOLR date format. During formation of the SOLR query i am encoding the date string using UTF-8 encoding. After forming the whole query string i am posting the search request to

Re: OOM at MultiSegmentReader.norms

That's a tiny heap. Part of it is used for indexing, too. And the fact that your heap is so small shows you are not really making use of that nice ramBufferSizeMB setting. :) Also, use omitNorms="true" for fields that don't need norms (if their types don't already do that). Otis -- Sematext

Re: How to optimize Index Process?

Hi, Answers inlined. -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > We have a distributed Solr system (2-3 boxes with each running 2 > instances of Solr and each Solr instance can write to multiple cores). Is this really optimal? How many CPU co

Re: optimization advice?

OK, we are a step closer. Sorting makes things slower. What field(s) do you sort on, what are their types, and if there is a date in there, are the dates very granular, and if they are, do you really need them to be that precise? Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nu

OOM at MultiSegmentReader.norms

Hi, I've index of size 50G (around 100 million documents) and growing - around 2000 records (1 rec = 500 byes) are being written every second continuously. If I make any search on this index I get OOM. I'm using default cache settings (512,512,256) in the solrconfig.xml. The search is using the

using multisearcher

Hi everybody, I'm interested in using Solr to search multiple indexes at once. We currently use our own search application which uses lucene's multisearcher. Has anyone attempted to or successfully replaced SolrIndexSearcher with some kind of multisearcher? I have looked at the Distributed

Apachecon 2009 Europe

Hi all, you came back with a head full of impressions from Apachecon Europe. Thanks a lot for the great Speeches and the inspiring personal talks. I strongly believe that solr will have great future. Olivier -- Olivier Dobberkau d.k.d Internet Service GmbH fon: +49 (0)69 - 43 05 61-70 fax: +

Re: Encoding problem

On Sat, Mar 28, 2009 at 12:51 AM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > > I see that you are specifying the topologyname's value in the query itself. > It might be a bug in DataImportHandler because it reads the data-config as a > string from an InputStream. If your default plat

RE: large index vs multicore

Thanks for the reply. Yes in most of the usecase the data would be from both the indices. It's like a parent child relation. The usecase requires the data from the child be displayed along with parent product information. Thanks, Kalyan Manepalli -Original Message- From: Ryan McKinley

How to optimize Index Process?

Hi, We have a distributed Solr system (2-3 boxes with each running 2 instances of Solr and each Solr instance can write to multiple cores). Our use case is high index volume - we can get up to 100 million records (1 record = 500 bytes) per day, but very low query traffic (only administrators may

Re: Solr date parsing issue

Hello, yes, I use both: I have a multicore architecture, multiple indexes but I have been able to manage a common schema. Giovanni On 3/27/09, Shalin Shekhar Mangar wrote: > > On Sat, Mar 28, 2009 at 12:46 AM, Giovanni De Stefano < > giovanni.destef...@gmail.com> wrote: > > > Hello, > > > > th

Re: Question about Solr memory usage.

On Sat, Mar 28, 2009 at 12:13 AM, Jim Adams wrote: > I'm running an old version of Solr -- it's 1.2, and I'm about to upgrade to > 1.3. But I have a question about Solr 1.2 memory usage. > > I am occasionally seeing out of memory errors in my Solr log. > > Doesn't Solr release memory after a doc

Re: Solr date parsing issue

On Sat, Mar 28, 2009 at 12:46 AM, Giovanni De Stefano < giovanni.destef...@gmail.com> wrote: > Hello, > > the problem is that I use both Solrj and DIH but I would like to perform > such a change only in 1 place. > > Is there any way to do it? Otherwise I will stick with the other > approach... >

Re: Encoding problem

On Fri, Mar 27, 2009 at 8:41 PM, Rui Pereira wrote: > I'm having problems with encoding in responses from search queries. The > encoding problem only occurs in the topologyname field, if a instancename > has accents it is returned correctly. In all my configurations I have > UTF-8. > > > > >

Re: Solr date parsing issue

Hello, the problem is that I use both Solrj and DIH but I would like to perform such a change only in 1 place. Is there any way to do it? Otherwise I will stick with the other approach... Cheers, Giovanni On 3/27/09, Shalin Shekhar Mangar wrote: > > On Fri, Mar 27, 2009 at 8:17 PM, Giovanni D

Re: Clarifying use of

On Fri, Mar 27, 2009 at 8:00 PM, fergus mcmenemie wrote: > Hello, > > Due to limitations with the way my content is organised and DIH I have > to add “-imgCaption:[* TO *]” to some of my queries. I discovered the > name=”appends” functionality tucked away inside solconfig.xml. This > looks a very

Re: Solr date parsing issue

On Fri, Mar 27, 2009 at 8:17 PM, Giovanni De Stefano < giovanni.destef...@gmail.com> wrote: > Hello, > > I am having a problem indexing a date field. > > In my schema the date field is defined the standard way: > > omitNorms="true"/> > > I know the Solr format is 1995-12-31T23:59:59Z, but the dat

Re: use extrernal index for spellcheck component

On Sat, Mar 28, 2009 at 12:16 AM, Marc Sturlese wrote: > > Hey there, > I have a doubt with spellcheck component... > If I tell the spellcheck component to load the dictionary from a field of > my > solr main index there's no problem but... Does someone know how to tell the > spellcheck component

use extrernal index for spellcheck component

Hey there, I have a doubt with spellcheck component... If I tell the spellcheck component to load the dictionary from a field of my solr main index there's no problem but... Does someone know how to tell the spellcheck component to load the dictionary from a filed of an external index? What I do i

Question about Solr memory usage.

I'm running an old version of Solr -- it's 1.2, and I'm about to upgrade to 1.3. But I have a question about Solr 1.2 memory usage. I am occasionally seeing out of memory errors in my Solr log. Doesn't Solr release memory after a document has been indexed ? I would not think it is right for th

Test

Sorry, I am having trouble sending a message to this Distribution list. This is a test.

Re: optimization advice?

> Steve, > > Maybe you can tell us about: sure > - your hardware 2.5GB RAM, pretty modern virtual servers > - query rate Let's say a few queries per second max... < 4 And in general the challenge is to get latency on any given query down to something very low - we don't have to worry about a

Re: Faceting question

Thanks Yonik. If it is using enum method then it should also be caching the facet query for every indexed value for the facet fields. 1) Do I need to add filterCache and hashDocSet entry to the solrconfig.xml for this caching to happen.? I did not find any noticeable difference in query time

Re: Best way to unit test solr integration

So in the "building block" story you talked about, that sounds like an integration (functional? user acceptance?) test.. And I would treat Solr the same way you treat your database that you are storing model objects in. If in your tests you bring up a fresh version of the db, populate it

RE: Solr Search Error

Hi Otis, Thanks for the recommendation. Will try with latest nightly build.. I did couple of full data import and got this error at few times while searching.. Thanks. Karthik -Original Message- From: Otis Go

Re: Best way to unit test solr integration

Joe, Have a look at Solr's own unit test, I believe they have pieces of what you need - the ability to start a Solr instance, index docs, run a query, and test if the results contain what you expect to see in them. You can get to Solr's unit test by checking out Solr from svn, or by browising

Re: Solr Search Error

Hi Karthik, First thing I'd do is get the latest Solr nightly build. If that doesn't fix thing, I'd grab the latest Lucene nightly build and use it to replace Lucene jars that are in your version of Solr. If that doesn't work I'd email the ML with a bit more info about the type of search that c

RE: Best way to unit test solr integration

Thanks for the tips, I like the suggestion of testing the document and query generation without having solr involved. That seems like a more bite-sized unit; I think I'll do that. However, here's the test case that I'm considering where I'd like to have a live solr instance: During an exercis

Re: Best way to unit test solr integration

So my first thought is that "unit test + solr integration" is an oxymoron. In the sense that unit test implies the smallest functional unit, and solr integration implies multiple units working together. It sounds like you have two different tasks. the code that generate queies, you can te

Re: Encoding problem

Hi, I had the same problem with DATAIMPORTHandler : i have a utf-8 mysql DATABASE but it's seems that DIH import data in LATIN... So i just use Transformer to (re)encode my strings in UTF-8. Rui Pereira-2 wrote: > > I'm having problems with encoding in responses from search queries. The > encod

Best way to unit test solr integration

Hello, On our project, we have quite a bit of code used to generate Solr queries, and I need to create some unit tests to ensure that these continue to work. In addition, I need to generate some unit tests that will test indexing and retrieval of certain documents, based on our current schema

Solr Search Error

Hi All, I am intermittently getting this Exception when I do the search. What could be the reason?. Caused by: org.apache.solr.common.SolrException: 11938 java.lang.ArrayIndexOutOfBoundsException: 11938 at org.apache.lucene.search.TermScorer.score(TermScorer.java:74)

Re: Faceting question

It would be the enum method... Solr 1.3 doesn't have the "fc" method for multi-valued fields... that's a 1.4 feature. -Yonik http://www.lucidimagination.com On Fri, Mar 27, 2009 at 10:44 AM, rayandev wrote: > > I am using the faceting feature and it works, I get back the facet counts, > but I ne

Encoding problem

I'm having problems with encoding in responses from search queries. The encoding problem only occurs in the topologyname field, if a instancename has accents it is returned correctly. In all my configurations I have UTF-8. ... As

Solr date parsing issue

Hello, I am having a problem indexing a date field. In my schema the date field is defined the standard way: I know the Solr format is 1995-12-31T23:59:59Z, but the dates coming from my "sources" are in the format 2009-04-10T02:02:55+0200 How can I make the conversion? Do I have to extend Da

Faceting question

I am using the faceting feature and it works, I get back the facet counts, but I need to know which facet.method(enum or fc) is used. Is there a way to turn on the debug info for faceting. Here's my setup Solr 1.3 EmbededSolrServer SolrJ Facet fields are indexed as multivalued solr.StrField T

Clarifying use of

Hello, Due to limitations with the way my content is organised and DIH I have to add “-imgCaption:[* TO *]” to some of my queries. I discovered the name=”appends” functionality tucked away inside solconfig.xml. This looks a very useful feature, and I created a new requestHandler to deal with my pr

Re: Solrj exception posting XML docs

Hello all, the null pointer exception was caused by a wrong XML... Basically my doc was something like this: ... ... but it had to be wrapped with a as follow: ... A more useful message would have been nice to have because I had to look at the source code to understand that the

Re: Incorrect sort with with function query in query parameters

Hi Otis, Any documents marked deleted in this index are just the result of updates to those documents. There are no purely deleted documents. Furthermore, the field that I am ordering by in my function query remains untouched over the updates. I've read in other posts that the logic used by th

Re: Search transparently with Solr with multiple cores, different indexes, common response type

Hello Hoss, Steve, thank you very much for your feedbacks, they have been very helpful making me feel more confident now about this architecture. In fact I decided to go for a single shared schema, but keeping multiple indexes (multicore) because those two indexes are very different: one is huge

Solrj exception posting XML docs

Hello all, I am currently using Solr 1.3 and its Solrj. I am trying to post XML docs directly through Solrj but I get the following exception: 13:12:09,119 ERROR [STDERR] Mar 27, 2009 1:12:09 PM org.apache.solr.common.SolrException log SEVERE: java.lang.NullPointerException at org.apache.solr.h

Re: Initial query performance poor after update / delete

Hi Tom, > Thanks Otis. After some further testing - I've noticed that initial searches > are only slow if I include the qt=geo parameter. Searches without this > parameter appear to show no slow down whatsoever after updates - so I'm > wondering if the problem is actually a localsolr one. > >

Re: Initial query performance poor after update / delete

Thanks Otis. After some further testing - I've noticed that initial searches are only slow if I include the qt=geo parameter. Searches without this parameter appear to show no slow down whatsoever after updates - so I'm wondering if the problem is actually a localsolr one. Can you tell me where I

Re: optimization advice?

Steve, Maybe you can tell us about: - your hardware - query rate - document cache and query cache settings - your current response times - any pain points, any slow query patterns - etc. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: Ste

Re: Incorrect sort with with function query in query parameters

Asif, Could it have something to do with the deleted documents in your unoptimized index? There documents are only marked as deleted. When you run optimize you really remove them completely. It could be that they are getting counted by something and that messes up the scoring/order. Otis

Re: "Unable to move index file" error during replication

the latest nightly should do fine On Fri, Mar 27, 2009 at 1:59 PM, sunnyfr wrote: > > Sorry but which one shoud I take?? > where exactly ? > > > Noble Paul നോബിള്‍  नोब्ळ् wrote: >> >> this fix is there in the trunk , >> you may not need to apply the patch >> >> On Fri, Mar 27, 2009 at 6:02 AM, s

Re: "Unable to move index file" error during replication

Sorry but which one shoud I take?? where exactly ? Noble Paul നോബിള്‍ नोब्ळ् wrote: > > this fix is there in the trunk , > you may not need to apply the patch > > On Fri, Mar 27, 2009 at 6:02 AM, sunnyfr wrote: >> >> Hi, >> >> It doesn't seem to work for me, I changed as well this part belo