Parallelizing warmup queries

2013-11-16 Thread Otis Gospodnetic
Hi, It looks like warmup queries execute sequentially. Considering servers have N CPU cores these days, would it make sense to make them (optionally) run in parallel? This should help with cases where warmup queries are CPU bound by letting Solr use more than 1 thread and thus more than 1 CPU co

Re: Parallelizing warmup queries

2013-11-16 Thread Saar Carmi
Otis, This raises a newbie question - How would one know what query is 1-CPU bounded and what is multi-threaded? Saar On Sat, Nov 16, 2013 at 10:28 AM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > Hi, > > It looks like warmup queries execute sequentially. > > Considering servers hav

Re: Parallelizing warmup queries

2013-11-16 Thread Otis Gospodnetic
Hi, You'd use something like SPM for Solr (http://sematext.com/spm/solr-performance-monitoring/) and correlate (long) warmup times with CPU usage on an N-core system and when you see the CPU go up during warmup, but not quite "all the way" then, I believe, you should be able to say "Hm, the CPUs a

Re: Very long warmup query vs. frequent soft commit with new searcher

2013-11-16 Thread Shawn Heisey
On 11/15/2013 11:59 PM, Otis Gospodnetic wrote: > What happens when one has a *single* vry long *warming* running > query that takes, say, 10 minutes, and a soft commit that opens a new > searcher happening every 1 minute? > > Could one run into a situation where each soft commit triggers the

Re: Adding DocValues after or in the middle of indexing

2013-11-16 Thread Shawn Heisey
On 11/16/2013 12:39 AM, Otis Gospodnetic wrote: > Can one introduce DocValues (by adding them to the schema.xml) post facto? > If that is done, do newly added documents end up using DocValues, > while the old ones remain without DocValues? Short answer: You need to reindex. Long answer: That wou

Re: distributed search is significantly slower than direct search

2013-11-16 Thread Michael Sokolov
Did you say what the memory profile of your machine is? How much memory, and how large are the shards? This is just a random guess, but it might be that if you are memory-constrained, there is a lot of thrashing caused by paging (swapping?) in and out the sharded indexes while a single index c

Invalid use of SingleClientConnManager: connection still allocated

2013-11-16 Thread jimi.hullegard
Hi, We have a problem with solr in our test environment in our new project. The stack trace can be seen below. The thing is that this only effects the search that is performed by the CMS itself. Our own custom searches still works fine. Anyone know what could cause this error? A restart doesn't

Why EmbeddedSolrServer with a RAMDirectoryFactory Needs a Data Directory?

2013-11-16 Thread Furkan KAMACI
I am trying test frameworks of Solr and I've used AbstractSolrTestCase at my custom test codes. I do not test my Solr instance, I just test my custom application's interaction with Solr. I think that using an EmbeddedSolrServer with a RAMDirectoryFactory is a fast (everything happens at memory) an

Re: An UpdateHandler to run following a MySql DataImport

2013-11-16 Thread Erick Erickson
Yep, that should work, never crossed my mind but it sure makes sense! Good sleuthing! Erick On Fri, Nov 15, 2013 at 9:15 AM, Dileepa Jayakody wrote: > I found out that you can configure any requestHandler to run a > requestProcessor chain. > So in my /dataimport requestHandler I just called m

Calling Shutdown Method of EmbeddedSolrServer with a RAMDirectoryFactory

2013-11-16 Thread Furkan KAMACI
I use EmbeddedSolrServer with a RAMDirectoryFactory at my test classes. Should I call shutdown() method at @AfterClass method? What is the usage for shutdown() methods of SolrServer's (especially in my case)?