use pooled connections with DataImportHandler

2008-07-29 Thread Jeremy Hinegardner
Hi all, I'm using the DataImportHandler, and its working great. What I would like to do is configure the to use a pooled conncation from our Servlet container (Jetty) instead of having the DataImportHandler instantiate and hold a new connection for each entity in our dataimport-config.xml (which

Re: use pooled connections with DataImportHandler

2008-07-29 Thread Noble Paul നോബിള്‍ नोब्ळ्
Using pooled connection may be of little use here .Because the same connection is used throughout the operation that is one connection/per entity. If I use a pooled connection I still need to get the same no:of connections. That is why we did not have that configuration If you really need it you ca

RE: solr synonyms behaviour

2008-07-29 Thread Laurent Gilles
Hi Swarag, Indeed, we were faced with a problem with what we called Hiearchy synonym search, I think it is a little different of what you are looking for, but who know, maybe it could lead you to find a solution for you problem too. So here was our need: Let's say we have this hierarchy of words

Re: nested data structure definition

2008-07-29 Thread Norberto Meijome
On Mon, 28 Jul 2008 23:37:22 -0700 (PDT) matt connolly <[EMAIL PROTECTED]> wrote: > In my site, I have a document, which may have multiple comments. > > For each comment, I would like to know several pieces of information, like: > text, author, and date. Matt, please read the documentation abou

dismax and empty query

2008-07-29 Thread matt connolly
I'm having trouble setting up a dismax handler. I'm trying something really simple, like this: explicit 0.1 title^1.5 tags^1.0 body^0.5 *,score When I analyse a query, I get this (example) in the response: +DisjunctionMaxQuery((title:chair^1.5 | body:chair

[Announcement] New version of solr javascript library available for review

2008-07-29 Thread Matthias Epheser
hi, A new version of my Google Summe of Code project "Solr javascript library" is available at http://solrstuff.org/svn/solrjs/trunk/ . I also just updated the wiki page at http://wiki.apache.org/solr/SolrJS. An online demo with reasonably testdata will be provided soon. The library now sup

Re: dismax and empty query

2008-07-29 Thread Erik Hatcher
I believe your query is not matching regardless of that dangling empty () clause. The dismax query parser adds in several SHOULD (meaning effectively OR'd) clauses and I've seen (and reported even, I believe) this empty clause appear also with no problem. The default query operator is not

RE: Out of memory on Solr sorting

2008-07-29 Thread Daniel Alheiros
Hi Sundar. Well it would be good if you could do some profiling on your Solr app. I've done it during the indexing process so I could figure out what was going on in the OutOfMemoryErrors I was getting. But you won't definitelly need to have as much memory as your whole index size. I have a 3.5 m

faceting and unique ids

2008-07-29 Thread dudes dudes
Hello all, Please have a look at this url : http://localhost:8983/solr/select?q=bob+david&facet=true&facet.field=firstname&facet.field=lastname&facet.field=company&facet.mincount=1 I'm searching for David and Bob and I would like their relevant information such as first name, last name and com

Re: Out of memory on Solr sorting

2008-07-29 Thread david w
Hi, Daniel I got the same probem like Sundar. Is that possible to tell me what profiling tool you are using? thx a lot. /David On Tue, Jul 29, 2008 at 8:19 PM, Daniel Alheiros <[EMAIL PROTECTED]>wrote: > Hi Sundar. > > Well it would be good if you could do some profiling on your Solr app.

Re: use pooled connections with DataImportHandler

2008-07-29 Thread Jeremy Hinegardner
Thanks for the info. I don't really need it, I was just pondering some options. I noticed, anecdotally, in my logs that it didn't appear to be doing database queries concurrently. It just walked the entity list. That led to the conclusion that if it is doing things serially, it could reuse a con

Re: use pooled connections with DataImportHandler

2008-07-29 Thread Noble Paul നോബിള്‍ नोब्ळ्
Thanks a lot. This is what keeps us going :) On Tue, Jul 29, 2008 at 8:42 PM, Jeremy Hinegardner <[EMAIL PROTECTED]> wrote: > Thanks for the info. I don't really need it, I was just pondering some > options. > > I noticed, anecdotally, in my logs that it didn't appear to be doing > database queri

Re: Multiple Update servers

2008-07-29 Thread Matthew Runo
As far as I know only one machine can write to an index at a time. More than that and I got corrupted indexes. Thanks! Matthew Runo Software Developer Zappos.com 702.943.7833 On Jul 28, 2008, at 11:25 AM, Rakesh Godhani wrote: Hi, we are currently evaluating Solr and have been browsing the

termVectors and faceting

2008-07-29 Thread wojtekpia
Does setting termVectors to true affect faceting speed on a field? I changed a field definition from: to: And I see a significant performance improvement (~6x faster). MyFacetField has ~25,000 unique values. Does it make sense that this change caused the improvement? I made several other cha

Re: termVectors and faceting

2008-07-29 Thread Yonik Seeley
No, faceting does not use term vectors. -Yonik On Tue, Jul 29, 2008 at 1:40 PM, wojtekpia <[EMAIL PROTECTED]> wrote: > > Does setting termVectors to true affect faceting speed on a field? I changed > a field definition from: > > termVectors="true"/> > > to: > > > > And I see a significant perfo

RE: Out of memory on Solr sorting

2008-07-29 Thread Lance Norskog
A sneaky source of OutOfMemory errors is the permanent generation. If you add this: -XX:PermSize=64m -XX:MaxPermSize=96m You will increase the size of the permanent generation. We found this helped. Also note that when you undeploy a war file, the old deployment has permanent storage that

Re: Multiple Update servers

2008-07-29 Thread Rakesh Godhani
Thanks for the input, much appreciated. -Rakesh On 7/29/08 12:18 PM, "Matthew Runo" <[EMAIL PROTECTED]> wrote: > As far as I know only one machine can write to an index at a time. > More than that and I got corrupted indexes. > > Thanks! > > Matthew Runo > Software Developer > Zappos.com > 70

Re: big discrepancy between elapsedtime and qtime although enableLazyFieldLoading= true

2008-07-29 Thread Mike Klaas
On 28-Jul-08, at 11:16 PM, Britske wrote: That sounds interesting. Let me explain my situation, which may be a variant of what you are proposing. My documents contain more than 10.000 fields, but these fields are divided like: 1. about 20 general purpose fields, of which more than 1 can b

Re: Multiple Update servers

2008-07-29 Thread Alexander Ramos Jardim
You could implement a script that woiuld control which master server is indexing and put them behind something like a NAT. I use that that control my master redundancy. 2008/7/29 Rakesh Godhani <[EMAIL PROTECTED]> > Thanks for the input, much appreciated. > -Rakesh > > > > On 7/29/08 12:18 PM, "

Re: Multiple Update servers

2008-07-29 Thread Rakesh Godhani
After Matthew's comment I was thinking about putting them both behind a load balancer, with the LB directing all traffic to one until it fails and then kick over to the other one. In your architectures I'm guessing the masters share the same physical index, but do the slaves share the same index a

Using Solr for Info Retreval not so much Search...

2008-07-29 Thread Jim Murphy
I need to store 100 million documents in our Solr instance and be able to retrieve them with simple term queries - keyword matches. I'm NOT implementing a search application where documents are scored and ranked...they either match the keywords or not. Also, I have an external ranking system tha

Re: Using Solr for Info Retreval not so much Search...

2008-07-29 Thread Yonik Seeley
A function query might fit your needs... you could move some or all of your external ranking system into Solr. -Yonik On Tue, Jul 29, 2008 at 7:08 PM, Jim Murphy <[EMAIL PROTECTED]> wrote: > > I need to store 100 million documents in our Solr instance and be able to > retrieve them with simple te

Re: Using Solr for Info Retreval not so much Search...

2008-07-29 Thread Jim Murphy
I take it I can add my own functions that would take care of calling out to my external ranking system? Looking for docs on that... Jim Yonik Seeley wrote: > > A function query might fit your needs... you could move some or all of > your external ranking system into Solr. > > -Yonik > > On

Re: Using Solr for Info Retreval not so much Search...

2008-07-29 Thread Yonik Seeley
Calling out will be an order of magnitude (or two) slower compared to moving the rankings into Solr, but it is doable. See ValueSource (it's used by FunctionQuery). -Yonik On Tue, Jul 29, 2008 at 8:23 PM, Jim Murphy <[EMAIL PROTECTED]> wrote: > > I take it I can add my own functions that would t

Re: Using Solr for Info Retreval not so much Search...

2008-07-29 Thread Jim Murphy
If figured that it would be - but the rankings are dynamically calculated. I'd like to limit the number of calculations performed for this very reason. Still not sure if this approach will be better than naivly filtering docs after the query has happened. Reading about ValueSource thanks... J

Re: Using Solr for Info Retreval not so much Search...

2008-07-29 Thread Walter Underwood
You might be able to split the ranking into a common score and a dynamic score. Return the results nearly the right order, then do a minimal reordering after. If you plan to move a result by a maximum of five positions, then you could fetch 15 results to show 10 results. That is far, far cheaper th

Quick shards question

2008-07-29 Thread swarag
I'm currently looking through the source, but just wanted to verify how shards work. If a request is made to: http://localhost:8983/solr/select?shards=localhost:8983/solr,localhost:7574/solr Does the 8983 instance of Solr make an http request to both 7574 AND 8983 to search? Or does it know tha

Re: Quick shards question

2008-07-29 Thread Yonik Seeley
On Tue, Jul 29, 2008 at 9:35 PM, swarag <[EMAIL PROTECTED]> wrote: > I'm currently looking through the source, but just wanted to verify how > shards work. If a request is made to: > > http://localhost:8983/solr/select?shards=localhost:8983/solr,localhost:7574/solr > > Does the 8983 instance of So

Re: Using Solr for Info Retreval not so much Search...

2008-07-29 Thread Jim Murphy
Thanks Walter, My requirements are this: 1. Query the index for entries matching keyword. 2. remove any entries that are below a threshold score from the external system I'm looking at building a custom field type similar to ExternalFileField that can dole out a ValueSource that calls my exter