Re: Authenticated Indexing Not working

2009-04-25 Thread Allahbaksh Asadullah
HI Otis, I am using HTTPClient for authentication. When I use the server with Authentication for searching it works fine. But when I use it for indexing it throws error. Regards, Allahbaksh On 4/25/09, Otis Gospodnetic wrote: > > My guess is you could provide the credentials to the underlying Htt

Re: Solr-1.4 indexing slower ?

2009-04-25 Thread Marcus Herou
Strange, now I am reindexing a lot of items and have 1000 docs/sec again... This is really, really nice, sorry for bothering... /M On Sat, Apr 25, 2009 at 3:39 PM, Otis Gospodnetic < otis_gospodne...@yahoo.com> wrote: > > Yes, versions of Lucene have changed, but should only be faster. A simpl

Re: Date faceting - howto improve performance

2009-04-25 Thread Marcus Herou
Hmm looking in the code for the IndexMerger in Solr (org.apache.solr.update.DirectUpdateHandler(2) See that the IndexWriter.addIndexesNoOptimize(dirs) is used (union of indexes) ? And the test class org.apache.solr.client.solrj.MergeIndexesExampleTestBase suggests: add doc A to index1 with id=AAA

Re: Date faceting - howto improve performance

2009-04-25 Thread Marcus Herou
Oh and the indexing strategy is just a stupid random across the shards. What I asked about was a "Best Practice" of achieving most MB/sec indexing. I feel that the java-api should be less efficient than something more raw like curl or so but that is just my hunch. /M On Sat, Apr 25, 2009 at 5:1

Re: Date faceting - howto improve performance

2009-04-25 Thread Marcus Herou
Guys! Thanks for these insights, I think we will head for Lucene level merging strategy (two or more indexes). When merging I guess the second index need to have the same doc ids somehow. This is an internal id in Lucene, not that easy to get hold of right ? So you are saying the the solr: Extern

Get the field value that caused the result

2009-04-25 Thread Wouter Samaey
Hi, I've been following this mailinglist for some time now, but don't think this question was here recently. If it was, sorry for repeating it :-) I'm looking into a way to determine the value of a field that caused the result to be returned. For example, we're searching through blog posts by ind

Re: Date faceting - howto improve performance

2009-04-25 Thread Otis Gospodnetic
I should emphasize that the PR trick I mentioned is something you'd do at the Lucene level, outside Solr, and then you'd just slip the modified index back into Solr. Of, if you like the bleeding edge, perhaps you can make use of Ning Li's Solr index merging functionality (patch in JIRA). Otis

Re: Authenticated Indexing Not working

2009-04-25 Thread Otis Gospodnetic
My guess is you could provide the credentials to the underlying HttpClient (used by SolrJ), and let it do the authentication. I don't have the API handy, sorry. But this may slow things down and I have to wonder if you really really need authentication there or whether using HTTP authenticatio

Re: Date faceting - howto improve performance

2009-04-25 Thread Otis Gospodnetic
Yes, you could simply round the date, no need for a non-date type field. Yes, you can add a field after the fact by making use of ParallelReader and merging (I don't recall the details, search the ML for ParallelReader and Andrzej), I remember he once provided the working recipe. Otis -- Semat

Re: Solr-1.4 indexing slower ?

2009-04-25 Thread Otis Gospodnetic
Yes, versions of Lucene have changed, but should only be faster. A simple way to see what's happening is to get the thread dump (e.g. through Solr admin pages) to see what the JVM is doing when things slow down. Do a few dumps and see. Perhaps the avg indexing rate is slower due to larger in

Authenticated Indexing Not working

2009-04-25 Thread Allahbaksh Asadullah
Hi, I have configured basic authentication in Solr using web.xml. It is working fine when I search using SolrJ. But when I try to index with Authentication enabled using SolrJ it is throwing exception. Is secured indexing is not enabled? How I am suppose to use secured indexing. Regards, Allahbak

RE: Date faceting - howto improve performance

2009-04-25 Thread Smiley, David W.
Hi Marcus. You must supply dates in the format that you are doing now -- ISO-8601 with the Z to indicate there is no time-zone offset occurring. To reduce cardinality to the day level instead of to the second that you are currently performing, the date you supply can include DateMathParser ope

Solr-1.4 indexing slower ?

2009-04-25 Thread Marcus Herou
Hi. We upgraded to solr-trunk (1.4-dev) for a few weeks ago and I've notices that the performance really went down. Not sure if I can blame solr 100% though so take these comments for what they might be (bullshit) However: For a few months ago I know we had indexing speed of about 100 docs/sec per

Date faceting - howto improve performance

2009-04-25 Thread Marcus Herou
Hi. One of our faceting use-cases: We are creating trend graphs of how many blog posts that contains a certain term and groups it by day/week/year etc. with the nice DateMathParser functions. The performance degrades really fast and consumes a lot of memory which forces OOM from time to time We t