Wittenberg, Lucas wrote:
> As suggested I switched to using DocValues and SortedDocValues.
> Now QTime is down to an average of 1100, which is much, much better
> but still far from the 30 I had with SOLR 4.
> I suppose it is due to the block-oriented compression you mentioned.
I apologize for be
d'origine-
De : Wittenberg, Lucas
Envoyé : lundi 26 août 2019 18:01
À : solr-user@lucene.apache.org
Objet : SOLR 7+ / Lucene 7+ and performance issues with DelegatingCollector and
PostFilter
Hello all,
Here is the situation I am facing.
I am migrating from SOLR 4 to SOLR 7. SOLR 4 is runni
> I don't know the precedence rules for stored vs. dovValues in Solr
DocValues are used if (and only if) all the fields being returned have
docValues=“true” _and_ are single-valued, or if you’ve explicitly
set useDocValuesAsStored.
single-valued docValues are they only situation where the respon
On Tue, 2019-08-27 at 09:05 +, Wittenberg, Lucas wrote:
> But the "customid" field is already set as docValues="true" actually.
> Well, I guess so as it is a type="string" which by default has
> docValues="true".
>
> required="true" multiValued="false" />
> docValues="true" />
Yeah, it's a
On Mon, 2019-08-26 at 16:01 +, Wittenberg, Lucas wrote:
> @Override
> public void collect(int docNumber) throws IOException {
> if (null != this.reader &&
> isValid(this.reader.document(docNumber).get("customid")))
> {
> super.collec
multiValued="false" />
> docValues="true" />
>
>
> -Message d'origine-
> De : Wittenberg, Lucas
> Envoyé : lundi 26 août 2019 18:01
> À : solr-user@lucene.apache.org
> Objet : SOLR 7+ / Lucene 7+ and performance issues with DelegatingC
voyé : lundi 26 août 2019 18:01
À : solr-user@lucene.apache.org
Objet : SOLR 7+ / Lucene 7+ and performance issues with DelegatingCollector and
PostFilter
Hello all,
Here is the situation I am facing.
I am migrating from SOLR 4 to SOLR 7. SOLR 4 is running on Tomcat 8, SOLR 7
runs with built in Jetty 9.
The la
Is “customid” a docValues=true field? I suspect not, in which case
I think this is the problem (but do be warned, I don’t spend much time in
Lucene code).
this.reader.document(docNumber).get("customid”)
document(docNumber)
goes out to do a disk read I think. If it were docValues=true, it could b
Hello all,
Here is the situation I am facing.
I am migrating from SOLR 4 to SOLR 7. SOLR 4 is running on Tomcat 8, SOLR 7
runs with built in Jetty 9.
The largest core contains about 1,800,000 documents (about 3 GB).
The migration went through smoothly. But something's bothering me.
I have a Pos
and I would
>> like it to work for all documents.
>>
>> Best regards
>>
>> Martin
>>
>>
>> Internal - KMD A/S
>>
>> -Original Message-
>> From: Jörn Franke
>> Sent: 21. marts 2019 17:11
>> To: solr-user@lucen
Message-
> From: Jörn Franke
> Sent: 21. marts 2019 17:11
> To: solr-user@lucene.apache.org
> Subject: Re: highlighter, stored documents and performance
>
> I don’t think so - to highlight any possible query you need the full document.
>
> You could optimize it by
Subject: Re: highlighter, stored documents and performance
I don’t think so - to highlight any possible query you need the full document.
You could optimize it by only storing a subset of the document and highlight
only in this subset.
Alternatively you can store a summary and show only the summary
I don’t think so - to highlight any possible query you need the full document.
You could optimize it by only storing a subset of the document and highlight
only in this subset.
Alternatively you can store a summary and show only the summary without
highlighting.
> Am 21.03.2019 um 17:05 schri
Hi,
I am wondering how performance highlighting in Solr performs when the number of
documents get large?
Right now we have about 1 TB of data in all sorts of file types and I was
wondering how storing these documents within Solr (for highlighting purpose)
will affect performance?
Is it possib
has been out for about 6 months now. (Sep-2017 to Mar-2018)
> > We are planning some major upgrades from 6.2 and 6.4 versions of Solr
> and I
> > wanted to see how is Solr 7 looking in terms of stability and
> performance.
> > (Have seen http://lucene.apache.org/solr/news.htm
018, at 8:55 PM, S G wrote:
>
> Hi,
>
> Solr 7 has been out for about 6 months now. (Sep-2017 to Mar-2018)
> We are planning some major upgrades from 6.2 and 6.4 versions of Solr and I
> wanted to see how is Solr 7 looking in terms of stability and performance.
> (Have seen
Hi,
Solr 7 has been out for about 6 months now. (Sep-2017 to Mar-2018)
We are planning some major upgrades from 6.2 and 6.4 versions of Solr and I
wanted to see how is Solr 7 looking in terms of stability and performance.
(Have seen http://lucene.apache.org/solr/news.html but some real users
gt; To: solr-user@lucene.apache.org
> Subject: Tracing Solr Query Execution and Performance
>
>
> Hi there
>
> I have a few very expensive queries (atleast thats what the QTime tells
> me) that is causing high CPU problems on a few nodes. Is there a way where
> I can "tr
esday, February 26, 2014 5:25 PM
To: solr-user@lucene.apache.org
Subject: Tracing Solr Query Execution and Performance
Hi there
I have a few very expensive queries (atleast thats what the QTime tells
me) that is causing high CPU problems on a few nodes. Is there a way where
I can "trace
Hi there
I have a few very expensive queries (atleast thats what the QTime tells
me) that is causing high CPU problems on a few nodes. Is there a way where
I can "trace" or do an "explain" on the solr query to see where it spends
more time? More like profiling on a per sub query basis?
I have t
Hi
I have an index consisting of a double value that can range between certain
values and an associated tag. I am trying to find all the docs which match
a certain tag (or combination of tags) and a certain range.
I'm trying to use the TermRangeTermsEnum from the Flex API as part of a
custom pars
Hi All,
I'm using the Admin UI dataimport page to load some documents into my
index. There's a rows parameter that you can leave blank (to load all
documents).
When I change it to the maximum number of documents, the performance drops
by a factor of 10.
For example, I have 1627 root entities. If
Thanks Shawn and Aditya. Really appreciate your help. Based on your advice
and reading the SolrPerformance article Shawn linked me to, I ended up
getting Intel Dual Core (2 Core) i3 3220 3.3Ghz with 36GB RAM with 2 x
125GB SSD drives for 227$ per month. It's still expensive for me but I got
it anyw
Hi
It totally depends upon your affordability. If you could afford go for
bigger RAM, SSD drive and 64 Bit OS.
Benchmark your application, with certain set of docs, how much RAM it
takes, Indexing time, Search time etc. Increase the document count and
perform benchmarking tasks again. This will p
On 7/17/2013 9:35 AM, Ayman Plaha wrote:
> In my solrconfig.xml I've got these caching config by default which I
don't
> think I will need. Since my index is updated with new documents every 3
> minutes caching anything would be pointless. Am I on the right ?
>
>
That depends on how many queri
Wow! Thanks Shawn. That's great info and helped and thanks for the
SolrPerformance article link, great article, helped a lot :)
I can't use Cloud hosting now since they charge on basis of the memory used
and it will be too expensive and like you said RAM and SSD is what I need
for SOLR performance
On 7/17/2013 1:22 AM, Ayman Plaha wrote:
>*will this effect the query performance of the client website if the
>index grew to 10 million records ? I mean while the commit is happening
>does that *effect the performance of queries* and how will this effect
>the queries if the index g
Thanks Aditya, can I also please get some advice on hosting.
- What *hosting specs* should I get ? How much RAM ? Considering my
- client application is very simple that just register users to database
and queries SOLR and displays SOLR results.
- simple batch program adds the 1000 OR
Hi
It will not affect the performance. We are doing this regularly. If you do
optimize and search then there may be some impact.
Regards
Aditya
www.findbestopensource.com
On Wed, Jul 17, 2013 at 12:52 PM, Ayman Plaha wrote:
> Hey Guys,
>
> I've finally finished my Spring Java application th
Hey Guys,
I've finally finished my Spring Java application that uses SOLR for
searches and just had performance related question about SOLR. I'm indexing
exactly 1000 *OR* 2000 records every second. Every record having 13 fields
including 'id'. Majority of the fields are solr.StrField (no filters)
: Our number one problem: Doing a commit from loading records, which can
: happen throughout the day, makes all queries stop for 5-7 seconds.
: This is a showstopper for deployment.
Best guess: your queries rely on th FieldCache in some way (either sorting
or faceting) and you aren't doing an
Hello Andy,
On Tue, Nov 13, 2012 at 1:26 PM, Andy Lester wrote:
> We're getting close to deploying our Solr search solution, and we're doing
> performance testing, and we've run into some questions and concerns.
>
> Our number one problem: Doing a commit from loading records, which can
> happen
Andy,
Solr is supposed to serve requests by old searcher for a while. If the
pause lasts few seconds you can take a thread dump and see clear what it
waits for.
Just a guess: if you have many threads configured in servlet container pool
and push high load then warming can significantly impact your
We're getting close to deploying our Solr search solution, and we're doing
performance testing, and we've run into some questions and concerns.
Our number one problem: Doing a commit from loading records, which can happen
throughout the day, makes all queries stop for 5-7 seconds. This is a
sh
We're getting close to deploying our Solr search solution, and we're doing
performance testing, and we've run into some questions and concerns.
Our number one problem: Doing a commit from loading records, which can happen
throughout the day, makes all queries stop for 5-7 seconds. This is a
sh
Thanks for the tips! will check out those links and see what i can find!
On Wed, Aug 29, 2012 at 9:44 AM, Alexey Serba wrote:
> > Any tips on load testing solr? Ideally we would like caching to not
> effect
> > the result as much as possible.
>
> 1. Siege tool
> This is probably the simplest opt
> Any tips on load testing solr? Ideally we would like caching to not effect
> the result as much as possible.
1. Siege tool
This is probably the simplest option. You can generate urls.txt file
and pass it to the tool. You should also capture server performance
(CPU, memory, qps, etc) using tools
-user@lucene.apache.org
Betreff: Sharing and performance testing question.
Good morning all.
We are working on a project where we will have somewhere north of 10 solr
instances running in different data centers around the world. Each instance
will have the same schemes, but different data.
We
1> Yes. Note that the distributed tf/idf is an issue, although it's changing.
That is, if your documents are statistically very different across
shards, the scores aren't really comparable. This is changing, but
I don't think it's committed yet.
2> Well, you're mixing apples and orang
hola,
1) When distributing search across several Shards, is the merged result
reflects the overall ranking, cross-shards?
I'm talking about stuff like "document frequency".
I guess it does, otherwise distributed search wouldn't have overhead.
talking about overhead,
2) is there a known ratio of t
message in context:
http://lucene.472066.n3.nabble.com/maxMergeDocs-and-performance-tuning-tp1162695p1183064.html
Sent from the Solr - User mailing list archive at Nabble.com.
/maxMergeDocs-and-performance-tuning-tp1162695p1168480.html
Sent from the Solr - User mailing list archive at Nabble.com.
the wiki.
Thanks!
Andrew.
--
View this message in context:
http://lucene.472066.n3.nabble.com/maxMergeDocs-and-performance-tuning-tp1162695p1162695.html
Sent from the Solr - User mailing list archive at Nabble.com.
-Full-Import-DB-and-Performance-tp861068p864297.html
Sent from the Solr - User mailing list archive at Nabble.com.
.
Colin.
> -Original Message-
> From: stockii [mailto:st...@shopgate.com]
> Sent: Tuesday, June 01, 2010 12:31 PM
> To: solr-user@lucene.apache.org
> Subject: RE: DIH, Full-Import, DB and Performance.
>
>
> thx for the reply =)
>
>
> i try out selectMet
thx for the reply =)
i try out selectMethod="cursor" but the load of the server is going bigger
and bigger during a import =(
selectMethod="cursor" only solve the problem with the locking ? right ?
--
View this message in context:
http://lucene.472066.n3.nabble.com/DI
particular situation.
Colin.
> -Original Message-
> From: stockii [mailto:st...@shopgate.com]
> Sent: Tuesday, June 01, 2010 7:44 AM
> To: solr-user@lucene.apache.org
> Subject: Re: DIH, Full-Import, DB and Performance.
>
>
> do you think that the option
>
> responseB
round
200.000 doc's.
why is the load so much
--
View this message in context:
http://lucene.472066.n3.nabble.com/DIH-Full-Import-DB-and-Performance-tp861068p861262.html
Sent from the Solr - User mailing list archive at Nabble.com.
ndler's
dataSource configuration.
--
View this message in context:
http://lucene.472066.n3.nabble.com/DIH-Full-Import-DB-and-Performance-tp861068p861134.html
Sent from the Solr - User mailing list archive at Nabble.com.
tter ? can the dih use other options ? make it
sense to use anoter batchSize = "-1"
--
View this message in context:
http://lucene.472066.n3.nabble.com/DIH-Full-Import-DB-and-Performance-tp861068p861068.html
Sent from the Solr - User mailing list archive at Nabble.com.
: I am using Solr 1.4 with large query strings with 20+ terms and faceting on
: a single multi-valued field in a 1 million record system. I am using Solr to
: categorize text, that why the query strings are big.
:
: The performance get's worse the more search terms are used. Is there any
can y
I can improve performance? I've tried several shingling but it had no
effect and tried everything in here
http://wiki.apache.org/solr/SolrPerformanceFactors
Is there anything else I can try? Will sharding help?
--
View this message in context:
http://old.nabble.com/Large-Query-Strings-and-p
21, 2008 10:59 AM
To: solr-user@lucene.apache.org
Subject: Re: shards and performance
2008/8/21 Otis Gospodnetic <[EMAIL PROTECTED]>
> Uh uh. 6 instances per node all pointing to the same index?
> Yes, this can increase performance, but only because it essentially
> gives
olr-user@lucene.apache.org
> > Sent: Wednesday, August 20, 2008 9:49:04 AM
> > Subject: Re: shards and performance
> >
> > Another thing to consider on your sharding is the access rate you want to
> > guarantee.
> >
> > In the project I am working, I need to g
gust 20, 2008 9:49:04 AM
> Subject: Re: shards and performance
>
> Another thing to consider on your sharding is the access rate you want to
> guarantee.
>
> In the project I am working, I need to guarantee at least 200hits/second
> with various facets in all queries.
>
2008/8/20 Ian Connor <[EMAIL PROTECTED]>
> So, because the OS is doing the caching in RAM. It means I could have
> 6 jetty servers per machine all pointing to the same data. Once the
> index is built, I can load up some more servers on different ports and
> it will boost performance.
>
> That does
So, because the OS is doing the caching in RAM. It means I could have
6 jetty servers per machine all pointing to the same data. Once the
index is built, I can load up some more servers on different ports and
it will boost performance.
That does sound promising - thanks for the tip. What made you
Another thing to consider on your sharding is the access rate you want to
guarantee.
In the project I am working, I need to guarantee at least 200hits/second
with various facets in all queries.
I am not using sharding, but I have 6 Solr instances per cluster node, and I
have 3 nodes, to a total o
I have based my machines on bare bones servers (I call them ghetto
servers). I essentially have motherboards in a rack sitting on
catering trays (heat resistance is key).
http://web.mac.com/iconnor/iWeb/Site/ghetto-servers.html
Motherboards: GIGABYTE GA-G33M-S2L (these are small mATX with 4 RAM
s
As long as Solr/Lucene makes smart use from memory (and they from my
experiences), it is really easy to calculate how long a huge query/update
will take when you know how much the smaller ones will take. Just keep in
mind that the resource consumption of memory and disk space is almost always
propo
On 19-Aug-08, at 12:58 PM, Phillip Farber wrote:
So you experience differs from Mike's. Obviously it's an important
decision as to whether to buy more machines. Can you (or Mike)
weigh in on what factors led to your different take on local shards
vs. shards distributed across machines?
Thanks, Ian, for the considered reply. See below.
Ian Connor wrote:
I have not seen any boost by having an index split into shards on the
same machine. However, when you split it into smaller shards on
different machines (cpu/ram/hdd), the performance boost worth it.
So you experience differs
I have not seen any boost by having an index split into shards on the
same machine. However, when you split it into smaller shards on
different machines (cpu/ram/hdd), the performance boost worth it.
At least for building the index, the number of shards really does
help. To index Medline (1.6e7 do
On 19-Aug-08, at 10:18 AM, Phillip Farber wrote:
I'm trying to understand how splitting a monolithic index into
shards improves query response time. Please tell me if I'm on the
right track here. Were does the increase in performance come
from? Is it that in-memory arrays are smaller
I'm trying to understand how splitting a monolithic index into shards
improves query response time. Please tell me if I'm on the right track
here. Were does the increase in performance come from? Is it that
in-memory arrays are smaller when the index is partitioned into shards?
Or is it
65 matches
Mail list logo
