Re: Synonyms impacting the performance

2008-11-11 Thread Ryan McKinley
also check the timing in debugQuery=true... I suspect most of the time should be spent in: process: On Nov 11, 2008, at 12:33 PM, Manepalli, Kalyan wrote: Hi Otis, I tested by taking out the newly added synonyms data and the query time was back to normal ~125ms. I will verify the debugQue

Re: Synonyms impacting the performance

2008-11-11 Thread Otis Gospodnetic
Yeah. Though, 20 seconds still sounds like crazy, not something that I'd expect from that not terribly complex and demanding query. It's hard to tell where exactly the bottleneck is without looking at the server and a few other things. Otis -- Sematext -- http://sematext.com/ -- Lucene - Sol

Re: Boost Query effect with Standard Request Handler

2008-11-11 Thread Otis Gospodnetic
Hi, It's hard to tell what you are replying to since you removed the original email (sounds like somebody suggested using bq). I think it might help if you send the actual URL you are using. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch F

Re: Query Performance while updating the index

2008-11-11 Thread Otis Gospodnetic
I'll assume you are using the rsync-based replication, not SOLR-561. In that case, bwlimit is your friend. You'll have to modify the scripts and make use of this: --bwlimit=KBPS This option allows you to specify a maximum transfer rate in kilobytes per second. This option i

Re: Solr Core Size limit

2008-11-11 Thread Otis Gospodnetic
Right. With Distributed Search you are limited to # of shards * Integer.MAX_VALUE. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch From: Ryan McKinley <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Tuesday, November 11, 2008 9:38

Re: DataImportHandler not indexing all the records

2008-11-11 Thread Noble Paul നോബിള്‍ नोब्ळ्
why is the id field multivalued? is there a uniqueKey in the schema ? Are you sure there are no duplicates? look at the status host:post/dataimport gives you the status it can give you some clue --Noble On Wed, Nov 12, 2008 at 4:53 AM, Giri <[EMAIL PROTECTED]> wrote: > Hi, > > I have about ~ 2

Re: simple filter query solr processing

2008-11-11 Thread Ryan McKinley
tried removing the plusses i am inserting but now shows too many results &fq=+i_subjects:Film+i_subjects:+media+i_subjects:+mass+communication fq is a multi-valued field, try calling it like: &fq=i_subjects:Film&fq=i_subjects:mass communication&fq=... ryan

Re: Getting Status 400 - org.apache.lucene.queryParser.ParseException on submitting a search query having ^ symbol through Web Application

2008-11-11 Thread Ryan McKinley
In the application I m applying URLEncoding on the search string thus the entire search string gets converted into : http://localhost:8080/apache-solr-1.3.0/core51043/select/? q=Sigma+Survey+for+Police+Officers%26field%3DIndex_Type_s %3AproductIndex%26field%3DproductType_s%3Aproduct%26f

Re: DataImportHandler - composite multivalued fields for merging db rows?

2008-11-11 Thread Noble Paul നോബിള്‍ नोब्ळ्
Any component can use the VariableResolver . If you write a custom transformer that too can use it On Wed, Nov 12, 2008 at 4:40 AM, Mauricio Aristizabal <[EMAIL PROTECTED]> wrote: > Thanks Noble, that worked beautifully. > > What wasn't initially apparent to me was that a transformer can use the s

Re: Solr Core Size limit

2008-11-11 Thread Ryan McKinley
On Nov 11, 2008, at 8:03 PM, Yonik Seeley wrote: On Tue, Nov 11, 2008 at 6:59 PM, Matthew Runo <[EMAIL PROTECTED]> wrote: What happens when we use another in this case? I was under the assumption that if we say styleId then our doc IDs will be our styleIds. Is there a secondary ID that'

Re: Synonyms impacting the performance

2008-11-11 Thread Ryan McKinley
if performance is a problem, you can try adding the synonyms at index time... this should give you similar results without the runtime results. The obvious disadvantage is that you need to have the synonyms at index time... On Nov 11, 2008, at 2:37 PM, Manepalli, Kalyan wrote: Hi Otis,

Query Performance while updating teh index

2008-11-11 Thread oleg_gnatovskiy
Hello. We have an index with 15 million documents working on a distributed environment, with an index distribution setup. While an index on a slave server is being updated, query response times become extremely slow (upwards of 5 seconds). Is there any way to decrease the hit query response times

Re: MoreLikeThis

2008-11-11 Thread Ryan McKinley
did you try debugQuery=true? (i don't know what it does off hand... perhaps nothing, but it may add something for MLT) On Nov 11, 2008, at 6:56 PM, Jeff Newburn wrote: Thanks Otis! I understand what the parameters do to help modify for the most part. What I am trying to understand is how

Re: Boost Query effect with Standard Request Handler

2008-11-11 Thread CameronL
So the bq parameter works with the standard request handler (I don't have to specify defType=dismax)?? I've been running a few tests and it doesn't seem to be picking up the bq. I've boosted one of the values very high (like 1000) compared to the other fields and it doesn't seem to be affecting

Re: Solr Core Size limit

2008-11-11 Thread Yonik Seeley
On Tue, Nov 11, 2008 at 6:59 PM, Matthew Runo <[EMAIL PROTECTED]> wrote: > What happens when we use another in this case? I was under the > assumption that if we say styleId then our doc IDs > will be our styleIds. > > Is there a secondary ID that's kept internal to Solr/Lucene in this case? Ther

Re: Solr Core Size limit

2008-11-11 Thread Matthew Runo
What happens when we use another in this case? I was under the assumption that if we say styleId then our doc IDs will be our styleIds. Is there a secondary ID that's kept internal to Solr/Lucene in this case? Thanks for your time! Matthew Runo Software Engineer, Zappos.com [EMAIL PROTE

Re: MoreLikeThis

2008-11-11 Thread Jeff Newburn
Thanks Otis! I understand what the parameters do to help modify for the most part. What I am trying to understand is how the related items are related. Certain search results return thousands of documents that are related but since I have 6 fields involved in the search I have no idea why. What

DataImportHandler not indexing all the records

2008-11-11 Thread Giri
Hi, I have about ~ 2 million records in a mySQL database table (about 9 fields from a single table), and I am trying to load it to the solr using DataImportHandler using the command=full-import option. it only indexed about 615360 records out of 2 millions. here is my db-data-config.xml

Re: DataImportHandler - composite multivalued fields for merging db rows?

2008-11-11 Thread Mauricio Aristizabal
Thanks Noble, that worked beautifully. What wasn't initially apparent to me was that a transformer can use the same notation as in the query attribute (e.g. ${parententityname.fieldname} ) to access fields from parent entities, which allows them to merge data from multiple related rows, not just d

Re: MoreLikeThis

2008-11-11 Thread Otis Gospodnetic
Jeff, have you tried various mlt.min/max* params listed on http://wiki.apache.org/solr/MoreLikeThis? Those are the knobs that will let you control quality of MLT results. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch From: Jeff Newburn <[

MoreLikeThis

2008-11-11 Thread Jeff Newburn
Hi All, Anybody know of how to get some useful debugging information for morelikethis search component in SOLR 1.4? I am trying to make the results more relevant to each product. Unfortunately, I cannot seem to find a way to get information that would be useful to find out what is matching. I h

Re: Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory

2008-11-11 Thread Yonik Seeley
On Tue, Nov 11, 2008 at 3:35 PM, Burton-West, Tom <[EMAIL PROTECTED]> wrote: > Hi Yonik, > > Thanks for the NIO suggestion. We are using Linux, but our indexes are > NFS mounted. That may be your biggest performance problem right there - try on a local index and see what the difference is. > I t

Re: Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory

2008-11-11 Thread Otis Gospodnetic
Tom, have you tried copying your index to the local disk to see if that resolves the locking problem? Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch From: "Burton-West, Tom" <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Cc: "Farber, P

RE: Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory

2008-11-11 Thread Burton-West, Tom
Hi Yonik, Thanks for the NIO suggestion. We are using Linux, but our indexes are NFS mounted. I thought I saw something about problems with NIO and NFS, but am fuzzy on the details. These results are with Solr 1.2 and I'm wondering if even without the NIO change, upgrading to Solr 1.3 might hel

RE: Synonyms impacting the performance

2008-11-11 Thread Manepalli, Kalyan
Hi Otis, Since I have expand="true" in SynonymFilterFactory, the DebugQuery shows the query is expanded with all the synonyms. Eg; Without synonym the query is: Review:amaz | name:amaz | description:amaz With Synonym the query is: (Review:amaz Review:fabul Review:improbable Review:incred

Re: Going insane with requestHandler definition.

2008-11-11 Thread banished phantom
Thank you very much for your response (and sorry for bothering and NOT paying attention to the warning at the top of the URL). Regards, Lucas. 2008/11/11 Shalin Shekhar Mangar <[EMAIL PROTECTED]> > Hi Lucas, > > The new replication is targeted for Solr 1.4 (the next major release) which > is yet

Going insane with requestHandler definition.

2008-11-11 Thread banished phantom
Hello everyone ! I'm new in the Solr list. I'va been using Solr 1.2 for a while and also the replication through rsync. As the new Solr was released I tried the new replication method, so I read : http://wiki.apache.org/solr/SolrReplication. There it says : "Replication feature is implemented as a

Re: Going insane with requestHandler definition.

2008-11-11 Thread Shalin Shekhar Mangar
Hi Lucas, The new replication is targeted for Solr 1.4 (the next major release) which is yet to happen. For 1.3, you still need to use the rsync based replication. On Wed, Nov 12, 2008 at 12:08 AM, banished phantom <[EMAIL PROTECTED]>wrote: > Hello everyone ! I'm new in the Solr list. I'va been

Re: Going insane with requestHandler definition.

2008-11-11 Thread Jeff Newburn
Lucas, Did you upgrade to version 1.4 of solr? The replication is very newly implemented and not available until very recently. -Jeff On 11/11/08 10:38 AM, "banished phantom" <[EMAIL PROTECTED]> wrote: > Hello everyone ! I'm new in the Solr list. I'va been using Solr 1.2 for a > while and als

Re: Going insane with requestHandler definition.

2008-11-11 Thread Erik Hatcher
Note that the Java replication feature is Solr 1.4 and above. You'll need to try a nightly or trunk build to get to this feature for now. Erik On Nov 11, 2008, at 1:38 PM, banished phantom wrote: Hello everyone ! I'm new in the Solr list. I'va been using Solr 1.2 for a while and al

Re: Solr Core Size limit

2008-11-11 Thread Otis Gospodnetic
Doc ID gaps are zapped during segment merges and index optimization. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch From: Norberto Meijome <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Monday, November 10, 2008 6:45:01 PM Subject:

RE: Synonyms impacting the performance

2008-11-11 Thread Manepalli, Kalyan
Hi Otis, I tested by taking out the newly added synonyms data and the query time was back to normal ~125ms. I will verify the debugQuery and update you with the results Thanks, Kalyan Manepalli -Original Message- From: Otis Gospodnetic [mailto:[EMAIL PROTECTED] Sent: Tuesday, November 11,

Re: Problem with SolrJ and SolrIndexSearcher

2008-11-11 Thread Kraus, Ralf | pixelhouse GmbH
Otis Gospodnetic schrieb: Hi Ralf, Your subject mentions SolrJ, but I don't see any SolrJ code in your email. It looks as if you are messing with Solr's own code. thx for you help... Greets

Re: Problem with SolrJ and SolrIndexSearcher

2008-11-11 Thread Otis Gospodnetic
Hi Ralf, Your subject mentions SolrJ, but I don't see any SolrJ code in your email. It looks as if you are messing with Solr's own code. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch From: "Kraus, Ralf | pixelhouse GmbH" <[EMAIL PROTECTED]>

Re: Synonyms impacting the performance

2008-11-11 Thread Otis Gospodnetic
Hi, That doesn't sound normal, no. Do you know what your query looks like after synonym expansion? (you can use debugQuery=true or peek at the logs) Is that really the only thing that changed? In other words, if you comment out the SynonymFactory in solrconfig.xml and restart Solr do things

Synonyms impacting the performance

2008-11-11 Thread Manepalli, Kalyan
Hi all, I recently implemented query time synonyms in my application and I am seeing drastic performance degradation. The synonyms file is counts around 1000 words. The average querytime without synonyms is around 125 ms and with synonyms it jumps to 20 secs. Am I missing something

Re: SpellChecker Component

2008-11-11 Thread Jeff Newburn
Ok. I have managed to get the search component added (You rock Grant). I am having some interesting issues now with the suggestions. We sell shoes online so I am trying to get it to spellcheck for brand name. When I search konverse with spelling on it returns converse correctly however when I s

Problem with SolrJ and SolrIndexSearcher

2008-11-11 Thread Kraus, Ralf | pixelhouse GmbH
Hi, I want to use a SolrIndexSearcher for some special searches in my app... I startup my Solr with two cores in it (core_de & core_uk). But when I try this then my Solr Server generates a complete new cory instead of using the existing one... After 5-6 searches I run out of memory :-( Examp

Re: SpellChecker Component

2008-11-11 Thread Grant Ingersoll
Hi Jeff, A SearchComponent allows you to connect functionality with any Request Handler, allowing you to inline spelling requests (or other things like MoreLikeThis) with your queries, saving you from having to make an extra request. I walk through a lot of this in my article on Solr 1.3

deduplication & dataimporthandler

2008-11-11 Thread Marc Sturlese
Hey there, Is there any way to use dataimporthandler with deduplication together just doing xml configuration? I have read that deduplication (http://wiki.apache.org/solr/Deduplication) is meant to be used with the handler named /update (wich uses solr.XmlUpdateRequestHandler class). If there's

Query two fields OR other AND

2008-11-11 Thread subasta
Hello, I have problem with two fields. In query I am use q.op=AND but I would like use in two first fields OR. Example: iTitle:(+keyword 1 +keyword2) OR iDescription:(+keyword 1 +keyword2)+iUser:"12345"+iType:"1" This query not work :( Please help. Sorry for my english. -- View this message i

Re: Using DataImportHandler with mysql database

2008-11-11 Thread ant lu
2008-11-11 16:47:36 org.apache.solr.handler.dataimport.SolrWriter upload : Error creating document : SolrInputDocumnt[{}] org.apache.solr.common.SolrException: Document [null] missing required field: id at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:289) at org.ap

Re: Using DataImportHandler with mysql database

2008-11-11 Thread Marc Sturlese
That worked! I was writing in a bad way the > It seems like your data-config does not have any tag. The > following is the correct structure: > > > > > > > > On Tue, Nov 11, 2008 at 12:31 AM, Marc Sturlese > <[EMAIL PROTECTED]>wrote: > >> >> Hey there, >> I am trying to use the