> On Mar 24, 2018, at 5:21 PM, Deepak Goel wrote:
>
> My first test was to test with static queries. Does Solr scale-up as we
> increase the load of same query?
>
> The second test would be to check with 'Different Queries'.
>
> And then finally check with 80% similar queries and 20% different
Some observations:
*#* The CPU Load on idxa1 never crosses above 91% mark mostly even if you
increase the load (by increasing the number of threads). This is similar to
my environment (I can never cross 90% on Linux even if I increase the load.
For Windows I can never cross 65% for some reason)
*
On 3/25/2018 7:15 AM, Deepak Goel wrote:
$ Why is the 'qps' not increasing with increase in threads? (If I
understand the qps parameter right?)
Likely because I sent all these queries to a single copy of the index.
We only have two copies of the index in production, plus a third copy on
a de
Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
deic...@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
On Sun, Mar 25, 2018 at 2:24 PM, Shawn Heisey wrote:
> On 3/25/2018 1:45 AM, Shawn Heise
On 3/25/2018 1:45 AM, Shawn Heisey wrote:
I have written a little test program that can pound the system harder,
need a little more time to gather what I learned with it.
Here's the code and three results with different threadcounts:
https://gist.github.com/elyograg/abedf4ae28467059e46781f7d4
On 3/24/2018 10:42 PM, Deepak Goel wrote:
I believe you ran this query with a 1 user load. Or was it a multi-user
load test? If it was multi-user load test, how many users did you test for?
And what were the utilisations and tps?
It was late Saturday night when I did that. There's almost no lo
On 25 Mar 2018 6:49 am, "Shawn Heisey" wrote:
On 3/24/2018 6:21 PM, Deepak Goel wrote:
> Do you have any documented proof of the same (1 to 5ms)? Or is it an
> educated guess
>
Just now, I did a test. I did a "*:*" query (all docs), the QTime was 194
milliseconds, numFound was 188635489. Then
On 3/24/2018 6:21 PM, Deepak Goel wrote:
Do you have any documented proof of the same (1 to 5ms)? Or is it an
educated guess
Just now, I did a test. I did a "*:*" query (all docs), the QTime was
194 milliseconds, numFound was 188635489. Then I did the exact same
query again. QTime dropped
Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
deic...@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
On Sun, Mar 25, 2018 at 4:00 AM, Shawn Heisey wrote:
> On 3/24/2018 1:25 PM, Deepak Goel
On 3/24/2018 1:25 PM, Deepak Goel wrote:
Please check the section *Questions from ‘Around the World’* in the
following doc for answers to your questions:
*https://docs.google.com/document/d/1ZwyveG-Zjy7tbsvh9xjMug4bnIMRqKnNax3jh4GJlzM/edit?usp=sharing
The document says that 80 percent of the t
Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
deic...@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
On Sat, Mar 24, 2018 at 6:03 AM, Rick Leir wrote:
>
>
> Deep,
> What is the test so I ca
Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
deic...@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
On Sat, Mar 24, 2018 at 5:16 AM, Shawn Heisey wrote:
> On 3/23/2018 11:31 AM, Deepak Goe
Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
deic...@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
On Sat, Mar 24, 2018 at 6:18 AM, Shawn Heisey wrote:
> On 3/23/2018 1:13 PM, Deepak Goel
On 3/23/2018 1:13 PM, Deepak Goel wrote:
> Yes I am now creating a client object only once. On Linux it has superb
> results (performance improves by around two times). However on Windows it
> has no improvement
>
> *SoftwareThroughput (/sec)Response Time (msec)Utilization (%CPU)UnTuned
> (Windows)
Deep,
What is the test so I can try it.
75 or 90 ms .. is that the JVM startup time?
Cheers -- Rick
>>
>>
>I have stated the numbers which I found during my test. The best way to
>verify them is for someone else to run the same test. Otherwise I don't
>see
>how we can verify the results
--
S
On 3/23/2018 11:31 AM, Deepak Goel wrote:
> Do you have any specific questions about the benchmark setup?
How many docs are in the Solr index? How much disk space does it
consume? How much total memory is in the machine? How much memory is
allocated to Java heaps? Is there any other software r
Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
deic...@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
On Fri, Mar 23, 2018 at 11:38 PM, Shawn Heisey wrote:
> On 3/23/2018 11:21 AM, Deepak Go
On 3/23/2018 11:21 AM, Deepak Goel wrote:
>> I tried the above suggestion. The throughput and utilisation remain the
>> same (they dont increase even if I increase the load). The response time
>> comes down.
>>
Are you still creating a new client object for every query? Changing
how the client ob
Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
deic...@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
On Tue, Mar 20, 2018 at 3:32 AM, Shawn Heisey wrote:
> On 3/16/2018 4:24 PM, Deepak Goel
Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
deic...@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
On Thu, Mar 22, 2018 at 1:25 AM, Deepak Goel wrote:
>
>
>
>
> Deepak
> "Please stop crue
Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
deic...@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
On Mon, Mar 19, 2018 at 2:40 AM, Walter Underwood
wrote:
> > On Mar 17, 2018, at 3:23 AM
Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
deic...@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
On Sat, Mar 17, 2018 at 2:56 AM, Shawn Heisey wrote:
> On 3/16/2018 2:21 PM, Deepak Goel
On 3/16/2018 4:24 PM, Deepak Goel wrote:
> It is taking less than 100ms to create a HttpSolrClient Object
"Less than 100ms" is vague. Let's say by that you mean it takes at
least 50 milliseconds. This is a lot slower than I expected it to be,
but if you've measured it, I'll accept that.
If ever
> On Mar 17, 2018, at 3:23 AM, Deepak Goel wrote:
>
> Sorry for being rude. But the ' results ' please, not the ' road to the
> results '
We have 15 different search collections, all different sizes and all with
different kinds of queries. Here are the two major ones.
22 million docs
32 server
Looks like I've opened up a very interesting can of worms
Thank you to all that are posting to this thread, I'm learning a lot...
The way I see it now... a Single Solr instance on this machine, seems like
the most intelligent choice.
And then as upgrade path, adding in-expensive machines. Thi
On 17 Mar 2018 05:19, "Walter Underwood" wrote:
> On Mar 16, 2018, at 3:26 PM, Deepak Goel wrote:
>
> Can you please post results of your test?
>
> Please tell us the tps at 25%, 50%, 75%, 100% of your CPU resource
I could, but it probably would not be useful for your documents or your
queries
> On Mar 16, 2018, at 3:26 PM, Deepak Goel wrote:
>
> Can you please post results of your test?
>
> Please tell us the tps at 25%, 50%, 75%, 100% of your CPU resource
I could, but it probably would not be useful for your documents or your queries.
We have 22 million homework problems. Our que
On Sat, Mar 17, 2018 at 3:11 AM, Walter Underwood
wrote:
> > On Mar 16, 2018, at 1:21 PM, Deepak Goel wrote:
> >
> > However a single client object with thousands of queries coming in would
> > surely become a bottleneck. I can test this scenario too.
>
> No it isn’t. The single client object is
On Sat, Mar 17, 2018 at 2:56 AM, Shawn Heisey wrote:
> On 3/16/2018 2:21 PM, Deepak Goel wrote:
> > I wanted to test how many max connections can Solr handle concurrently.
> > Also I would have to implement an 'connection pooling' of the
> client-object
> > connections rather than a single connec
> On Mar 16, 2018, at 1:21 PM, Deepak Goel wrote:
>
> However a single client object with thousands of queries coming in would
> surely become a bottleneck. I can test this scenario too.
No it isn’t. The single client object is thread-safe and manages a pool of
connections.
Your benchmark is p
On 3/16/2018 2:21 PM, Deepak Goel wrote:
> I wanted to test how many max connections can Solr handle concurrently.
> Also I would have to implement an 'connection pooling' of the client-object
> connections rather than a single connection thread
>
> However a single client object with thousands of
iginal Message-
From: Deepak Goel [mailto:deic...@gmail.com]
Sent: Friday, March 16, 2018 4:22 PM
To: solr-user@lucene.apache.org
Subject: Re: Some performance questions
On Sat, Mar 17, 2018 at 1:06 AM, Shawn Heisey wrote:
> On 3/16/2018 7:38 AM, Deepak Goel wrote:
> > I did a performanc
On Sat, Mar 17, 2018 at 1:06 AM, Shawn Heisey wrote:
> On 3/16/2018 7:38 AM, Deepak Goel wrote:
> > I did a performance study of Solr a while back. And I found that it does
> > not scale beyond a particular point on a single machine (could be due to
> > the way its coded). Hence multiple instance
On 3/16/2018 7:38 AM, Deepak Goel wrote:
> I did a performance study of Solr a while back. And I found that it does
> not scale beyond a particular point on a single machine (could be due to
> the way its coded). Hence multiple instances might make sense.
>
> https://docs.google.com/document/d/1kUq
> That benchmark is on Windows, so not interesting for most of us.
I guess I must have missed this in the author's question. Did he describe
his OS?
Also other applications scale well on Windows. Why would Solr be different?
The Solr page does not say about any performance limits on windows
(shou
> On Mar 16, 2018, at 6:26 AM, Deepak Goel wrote:
>
> I would try multiple Solr instances rather a single Solr instance (it
> definitely will give a performance boost)
> I would avoid multiple Solr instances on single machine. I can use all 36
cores on our servers with one Solr process.
Is your l
On Mar 16, 2018, at 6:38 AM, Deepak Goel wrote:
>
> I did a performance study of Solr a while back. And I found that it does
> not scale beyond a particular point on a single machine (could be due to
> the way its coded). Hence multiple instances might make sense.
>
> https://docs.google.com/doc
> On Mar 16, 2018, at 6:26 AM, Deepak Goel wrote:
>
> I would try multiple Solr instances rather a single Solr instance (it
> definitely will give a performance boost)
I would avoid multiple Solr instances on single machine. I can use all 36 cores
on our servers with one Solr process.
wunder
On Fri, Mar 16, 2018 at 6:03 PM, Shawn Heisey wrote:
> On 3/15/2018 6:34 AM, BlackIce wrote:
>
>> However the main app that will be
>> running is more or less a single threated app which takes advantage when
>> run under several instances, ie: parallelism, so I thought, since I'm at
>> it
>> I ma
>I think there is no benefit in having multiple Solr instances on a single
>server, unless the heap memory required by the JVM is too big.
Deepak***
I would try multiple Solr instances rather a single Solr instance (it
definitely will give a performance boost)
Deepak***
>And remember that t
On 3/15/2018 6:34 AM, BlackIce wrote:
However the main app that will be
running is more or less a single threated app which takes advantage when
run under several instances, ie: parallelism, so I thought, since I'm at it
I may give solr a few instances as well
Solr is a fully threaded app, capa
*Single Solr Instance VS Multiple Solr instances on Single Server
*
I think there is no benefit in having multiple Solr instances on a single
server, unless the heap memory required by the JVM is too big.
And remember that this has relatively to do with the index size ( inverted
index is memory ma
Please see inline...
Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
deic...@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
On Thu, Mar 15, 2018 at 6:04 PM, BlackIce wrote:
> Shawn:
> well
Shawn:
well the idea was to utilize system resources more efficiently.. this is
not due so much to Solr, as I sayd I don't know that much about Solr,
except Shema.xml and Solarconfig.xml - However the main app that will be
running is more or less a single threated app which takes advantage when
run
On 3/14/2018 5:49 AM, BlackIce wrote:
I was just thinking Do I really need separate VM's in order to run
multiple Solr instances? Doesn't it suffice to have each instance in its
own user account?
You can run multiple instances all under the same account on one
machine. But for a single ma
Have you measured the overhead of VM anytime? Or have you read it somewhere?
On 14 Mar 2018 18:10, "BlackIce" wrote:
> but it should be possible, without the overhead of VM's
>
> On Wed, Mar 14, 2018 at 1:30 PM, Deepak Goel wrote:
>
> > The OS resources would be shared in that case
> >
> > On 1
but it should be possible, without the overhead of VM's
On Wed, Mar 14, 2018 at 1:30 PM, Deepak Goel wrote:
> The OS resources would be shared in that case
>
> On 14 Mar 2018 17:19, "BlackIce" wrote:
>
> > I was just thinking Do I really need separate VM's in order to run
> > multiple Solr
The OS resources would be shared in that case
On 14 Mar 2018 17:19, "BlackIce" wrote:
> I was just thinking Do I really need separate VM's in order to run
> multiple Solr instances? Doesn't it suffice to have each instance in its
> own user account?
>
> Greetz
>
> On Mon, Mar 12, 2018 at 7:4
I was just thinking Do I really need separate VM's in order to run
multiple Solr instances? Doesn't it suffice to have each instance in its
own user account?
Greetz
On Mon, Mar 12, 2018 at 7:41 PM, BlackIce wrote:
> I don't have any production logs and this all sounds to complicated.
>
I don't have any production logs and this all sounds to complicated.
So, I'll just trow the system together in a way it makes the most sense for
now.. collect some logs and then do some testing further down the road. For
now just get the sucker up and running.
Thanks all
On Mon, Mar 12, 2018
I am not sure if I understand your question
*"How do I test this?"*
You have to run test (benchmark test) of transactions (queries) which are
most representative of your system (requirement).
You can use a performance testing tool like JMeter (along with PerfMon
configured for utilisation metrics
So Im thinking following scenarios :
Single instance with drives in raid 0, raid 10 and raid 5.
And then having 3 Vms and 4 Solr instances each with its own HD.
How do I test this?
Greetz
On Mar 12, 2018 1:16 PM, "BlackIce" wrote:
> OK, so we're gone nowhere, since I've already lost lots of
Benchmark with production logs. Replay them at a constant request rate. Measure
the response time and look at the median and 90th or 95th percentile. Do not
use the average response time, because that will be thrown off by outliers.
It is best to run a few thousand warming queries before startin
OK, so we're gone nowhere, since I've already lost lots of time... A few
days more or less won't make a difference I'd be willing to benchmark
if some tells me how to.
Greetz
On Mar 12, 2018 7:17 AM, "Deepak Goel" wrote:
> Now you are mixing your original question about performance with
On 3/12/2018 3:22 AM, Deepak Goel wrote:
A single OS and JVM does not scale linearly for higher loads. If you have
seperate OS and Java, the load is distributed across multiple instances
(with each instance only requiered to support a smaller load and hence
would scale nicely)
I had found this f
Now you are mixing your original question about performance with reliability
On 12 Mar 2018 02:29, "BlackIce" wrote:
> Second to this wouldn't 4 Solr instances each with its own HD be fault
> tolerant? vs. one solr instance with 4 HD's in RAID 0? Plus to his comes
> the storage capacity, I need
We need benchmarks or data to support the claim.
A single OS and JVM does not scale linearly for higher loads. If you have
seperate OS and Java, the load is distributed across multiple instances
(with each instance only requiered to support a smaller load and hence
would scale nicely)
I had found
On 3/11/2018 7:39 PM, Deepak Goel wrote:
I doubt this. It would be great if someone can subtantiate this with hard
facts
This seems to be in response to my claim that virtualization always has
overhead. I don't see how this statement can be at all controversial.
Virtualization isn't free,
On 12 Mar 2018 05:51, "Shawn Heisey" wrote:
On 3/11/2018 11:35 AM, BlackIce wrote:
> I have some questions regarding performance.
>
> Lets says I have a dual CPU with a total of 8 cores and 24 GB RAM for my
> Solr and some other stuff.
>
> Would it be more beneficial to only run 1 instance of So
On 3/11/2018 11:35 AM, BlackIce wrote:
I have some questions regarding performance.
Lets says I have a dual CPU with a total of 8 cores and 24 GB RAM for my
Solr and some other stuff.
Would it be more beneficial to only run 1 instance of Solr with the
collection stored on 4 HD's in RAID 0?? Or.
Second to this wouldn't 4 Solr instances each with its own HD be fault
tolerant? vs. one solr instance with 4 HD's in RAID 0? Plus to his comes
the storage capacity, I need the capacity of those 4 drives... the more I
read.. the more questions
On Sun, Mar 11, 2018 at 9:43 PM, BlackIce wrote:
Thnx for the pointers.
I haven't given much thought to Solr, asides shemal.xml and solrconfig.xml
and I'm just diving into a bit more deeper stuff!
Greetz
RRK
On Sun, Mar 11, 2018 at 8:58 PM, Deepak Goel wrote:
> To rephrase your Question
>
> "Does Solr do well with Scale-up or Scale-out?"
>
To rephrase your Question
"Does Solr do well with Scale-up or Scale-out?"
Are there any Performance Benchmarks for the same out there supporting the
claim?
On 11 Mar 2018 23:05, "BlackIce" wrote:
> Hi,
>
> I have some questions regarding performance.
>
> Lets says I have a dual CPU with a tota
Hi,
I have some questions regarding performance.
Lets says I have a dual CPU with a total of 8 cores and 24 GB RAM for my
Solr and some other stuff.
Would it be more beneficial to only run 1 instance of Solr with the
collection stored on 4 HD's in RAID 0?? Or Have several Virtual
Machines ea
Hi,
We use Solr like a search engine / document store / database. We are
currently optimizing a test environment and would welcome any relevant
suggestions.
I've taken a lot of time researching this mailing list and found a lot of
relevant information.
Here's our current setup :
SolrCloud 6.
On Aug 30, 2007, at 11:37 PM, Walter Underwood wrote:
Sorry dude, I'm pining for Python and coding in Java. --wunder
No need to be sorry well, unless you do really enjoy forced
whitespace and __init__ ugliness. :)
Why don't we tie in scripting engine support into Solr via BSF or the
Only if you think the rest of Solr would be better written in JRuby too!
> -Original Message-
> From: Erik Hatcher [mailto:[EMAIL PROTECTED]
> Sent: 31 August 2007 02:57
> To: solr-user@lucene.apache.org
> Subject: Re: performance questions
>
>
> On Aug 30
Sorry dude, I'm pining for Python and coding in Java. --wunder
On 8/30/07 6:57 PM, "Erik Hatcher" <[EMAIL PROTECTED]> wrote:
>
> On Aug 30, 2007, at 6:31 PM, Mike Klaas wrote:
>> Another reason why people use stored procs is to prevent multiple
>> round-trips in a multi-stage query operation. T
On Aug 30, 2007, at 6:31 PM, Mike Klaas wrote:
Another reason why people use stored procs is to prevent multiple
round-trips in a multi-stage query operation. This is exactly what
complex RequestHandlers do (and the equivalent to a custom stored
proc would be writing your own handler).
A
On 30-Aug-07, at 3:18 PM, Chris Hostetter wrote:
2. Someone asked me if SOLR utilizes anything like a "stored
procedure" to make queries faster. Does SOLR support anything
such as this?
it's kind of an apples vs orange-juice comparison, ut typcailly
when people talk about DB stored pr
2. Someone asked me if SOLR utilizes anything like a "stored procedure"
to make queries faster. Does SOLR support anything such as this?
it's kind of an apples vs orange-juice comparison, ut typcailly when
people talk about DB stored procedures being faster then raw SQL they are
refering to
On 30-Aug-07, at 9:51 AM, Andrew Nagy wrote:
Here are a few SOLR performance questions:
1. I have noticed with 500,000+ records that my facets run quite
fast regarding my dataset when there is a large number of matches,
but on a small result set (say 10 - 50) the facet queries become
Here are a few SOLR performance questions:
1. I have noticed with 500,000+ records that my facets run quite fast regarding
my dataset when there is a large number of matches, but on a small result set
(say 10 - 50) the facet queries become very slow. Any suggestions as to how to
improve this
73 matches
Mail list logo
