/sematext.com/
>
>
>
> > On 24 Jun 2020, at 16:38, Odysci wrote:
> >
> > Hi,
> >
> > I have a Solrcloud configuration with 2 nodes and 2 shards/2 replicas.
> > I configure the sizes of the solr caches on solrconfig.xml, which I
> > believe apply to
I configure the sizes of the solr caches on solrconfig.xml, which I
> believe apply to nodes.
>
> But when I look at the caches in the Solr UI, they are shown per core
> (e.g., shard1_replica_N1). Are the cache sizes defined in the
> solrconfig.xml the total size (adding up the caches
Hi,
I have a Solrcloud configuration with 2 nodes and 2 shards/2 replicas.
I configure the sizes of the solr caches on solrconfig.xml, which I
believe apply to nodes.
But when I look at the caches in the Solr UI, they are shown per core
(e.g., shard1_replica_N1). Are the cache sizes defined in
"The reason I asked about the Cache sizes is I had read that
configuring the Cache sizes of Solr does not provide you enough
benefits"
Where is "somewhere"? Because this is simply wrong as a blanket statement.
filterCache can have tremendous impact on query performance, depending
on the how m
Hi Shawn,
Thanks for the reply, it is useful. The reason I asked about the Cache
sizes is I had read that configuring the Cache sizes of Solr does not
provide you enough benefits, instead it is better to provide a lot of
memory space to the Solr outside the JVM heap.
Is it true that in general the
On 5/11/2017 4:58 PM, Suresh Pendap wrote:
> This question might have been asked on the solr user mailing list earlier.
> Solr has four different types of Cache DocumentCache, QueryResultCache,
> FieldValueCache and FilterQueryCache
> I would like to know which of these Caches are off heap cache?
Hi,
This question might have been asked on the solr user mailing list earlier.
Solr has four different types of Cache DocumentCache, QueryResultCache,
FieldValueCache and FilterQueryCache.
Are these Caches memory mapped or they reside in the JVM heap?
Which Caches have the maximum impact on the qu
Hi,
This question might have been asked on the solr user mailing list earlier. Solr
has four different types of Cache DocumentCache, QueryResultCache,
FieldValueCache and FilterQueryCache
I would like to know which of these Caches are off heap cache? Which Caches
have the maximum impact on the
Hello Upayavira,
It's a long month ago! I just described this approach in
http://blog.griddynamics.com/2015/08/scoring-join-party-in-solr-53.html
Coming back to our discussion I think I miss {!func} which turn fieldname
into function query.
On Fri, Jul 24, 2015 at 3:41 PM, Upayavira wrote:
> Mik
I think it's intended for
{!join fromIndex=other from=other_key to=key score=max}my_boost_value_field
thus it runs functional query, which matches all docs at "other" core with
field value 'my_boost_value_field' as a score. Then, this score is passed
through join query for other.other_key=key. Do
Mikhail,
I've tried this out, but to be honest I can't work out what the score=
parameter is supposed to add.
I assume that if I do {!join fromIndex=other from=other_key to=key
score=max}somefield:(abc dev)
It will calculate the score for each document that has the same "key"
value, and include
Hi Erick,
You are right that I could actually be asking for a stored field. That's
an exceptionally good point, and yes, would suck. Better would be to
retrieve a docValue from document. I'll look into that.
Upayavira
On Fri, Jul 10, 2015, at 06:28 PM, Erick Erickson wrote:
> Upayavira:
>
> bq:
Upayavira:
bq: retrieve the value of field Y for that doc
If this is fetching the stored field it's going to be horrible as
it'll probably read/decompress a 16K block each time. Yccck. If
you can read the value from a DocValues field (or, indeed, any indexed
field which would only really work
Mikhail,
Thanks for pointing this out.
I'd say that ticket is in distinct need of some examples or use-cases.
It is extremely hard to work out what "scoring" actually means. What is
used to score what?
It'd be great to see some examples and some explanations as to what
effect those examples have
I've heard that people use https://issues.apache.org/jira/browse/SOLR-6234
for such purpose - adding scores from fast moving core to the bigger slow
moving one
On Fri, Jul 10, 2015 at 4:54 PM, Upayavira wrote:
> All,
>
> I have knocked up what I think could be a really cool function query -
> it
All,
I have knocked up what I think could be a really cool function query -
it allows you to retrieve a value from another core (much like a pseudo
join) and use that value during scoring (much like an
ExternalFileField).
Examples:
* Selective boosting of documents based upon a category based va
nt-of-memory-used-by-different-solr-caches
> <http://stackoverflow.com/questions/26909948/how-to-determine-amount-of-memory-used-by-different-solr-caches>
>
> Haven't received a response, so I am hoping to get the answer here. This is
> the question
>
> Solr wiki ht
groups?gid=6713853
On 25 November 2014 at 11:07, sumitj25 wrote:
> Hi,
>
> I posted a question on stackoverflow regarding this
> http://stackoverflow.com/questions/26909948/how-to-determine-amount-of-memory-used-by-different-solr-caches
> <http://stackoverflow.com/questions/2690994
Hi,
I posted a question on stackoverflow regarding this
http://stackoverflow.com/questions/26909948/how-to-determine-amount-of-memory-used-by-different-solr-caches
<http://stackoverflow.com/questions/26909948/how-to-determine-amount-of-memory-used-by-different-solr-caches>
Haven't
ct decision on Query 2 ? The one thing I am NOT very sure
about is whether its appropriate / justifiable in my Use Case to have the
same query parameter "CompanyName" in both the "q" and "fq".
Also, need to mention that I fell into the trap of setting extremely huge
ca
: FilterCache:
...
: So if a query contains two fq params, it will create two separate entries
: for each of these fq params. The value of each entry is the list of ids of
: all documents across the index that match the corresponding fq param. Each
: entry is independent of any other entry
Otis
Performance Monitoring for Solr / ElasticSearch / HBase -
http://sematext.com/spm
>
> From: Rahul R
>To: solr-user@lucene.apache.org
>Sent: Tuesday, May 15, 2012 3:20 PM
>Subject: Solr Caches
>
>Hello,
>I am trying to understand how I ca
Hello,
I am trying to understand how I can size the caches for my solr powered
application. Some details on the index and application :
Solr Version : 1.3
JDK : 1.5.0_14 32 bit
OS : Solaris 10
App Server : Weblogic 10 MP1
Number of documents : 1 million
Total number of fields : 1000 (750 strings, 2
: Although I don't have statistics to back my claim, I suspect that the really
: nasty filters don't have as high a hitcount as the ones that are more simple.
: Typically the really nasty filters are used when an employee logs into the
: site. Employees have access to a lot more than customers do
On 11/10/2011 11:55 AM, Shawn Heisey wrote:
Do Solr's LRU caches pay attention to hitcount when deciding which
entries to age out and use for autowarming, or is it purely based on
the last time that entry was touched? Is it a reasonable idea to come
up with an algorithm that uses hitcount alon
Do Solr's LRU caches pay attention to hitcount when deciding which
entries to age out and use for autowarming, or is it purely based on the
last time that entry was touched? Is it a reasonable idea to come up
with an algorithm that uses hitcount along with entry age, ideally with
a configurabl
e that was expensive to
> compute, or recompute it using hte new Searcher. ... but none of the
> default cache regenerators for the stock solr caches work this way)
>
>
> :
> :
> :
> : Thanks,
> : - Savvas
> :
>
> -Hoss
but none of the
default cache regenerators for the stock solr caches work this way)
:
:
:
: Thanks,
: - Savvas
:
-Hoss
Hello,
I am going through the wiki page related to cache configuration
http://wiki.apache.org/solr/SolrCaching and I have a question regarding the
general cache architecture and implementation:
In my understanding, the Current Index Searcher uses a cache instance and
when a New Index Searcher
ight solution :-)
Cheers,
Martin
> A request that needs more than a minute isn't the standard, even when I
> consider all the other postings about response-performance...
>
> Regards
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Use-terracotta-big
y-for-solr-caches-tp2328257p2330652.html
Sent from the Solr - User mailing list archive at Nabble.com.
Hi,
as the biggest parts of our jvm heap are used by solr caches I asked myself
if it wouldn't make sense to run solr caches backed by terracotta's
bigmemory (http://www.terracotta.org/bigmemory).
The goal is to reduce the time needed for full / stop-the-world GC cycles,
as with our 8G
t; searching.
>
> When i perform an update. the search-instance dont get the new documents.
> when i start a commit on searcher he found it. how can i say the searcher
> that he alwas look not only the "old" index. automatic refresh ? XD
> --
> View this message in context
only the "old" index. automatic refresh ? XD
--
View this message in context:
http://lucene.472066.n3.nabble.com/Tuning-Solr-caches-with-high-commit-rates-NRT-tp1461275p2005738.html
Sent from the Solr - User mailing list archive at Nabble.com.
Many thanks, Peter K. for posting up on the wiki - great!
Yes, fc = field cache. Field Collapsing is something very nice indeed,
but is entirely different.
As Erik mentions in the wiki post, using per-segment faceting can be a
huge boon to performance. It does require the latest Solr trunk build
(10/11/16 8:36), Jonathan Rochkind wrote:
In Solr 1.4, facet.method=enum DOES work on multi-valued fields, I'm pretty
certain.
Correct, and I didn't say that facet.method=enum doesn't work for multiValued/tokenized field in my
previous mail.
I think Koji's explanation is based on before So
Koji Sekiguchi wrote:
Usually, you do not need to set facet.method because Solr
automatically uses most appropriate facet method for
each field type:
boolean: TermEnum
multiValued/tokenized: UnInvertedField
other than those above: FieldCache
As I understand it, in Solr 1.4, (and I may NOT un
(10/11/16 6:43), Dennis Gearon wrote:
fc='field collapsing'?
fc of facet.method=fc stands for Lucene's FieldCache.
enum of facet.method=enum stands for Lucene's TermEnum.
Usually, you do not need to set facet.method because Solr
automatically uses most appropriate facet method for
each field t
Mon, November 15, 2010 1:37:00 PM
Subject: Re: Tuning Solr caches with high commit rates (NRT)
Hi Jonathan,
I am too using fc because it simply was faster. Not sure if this can be applied
in general.
I will add this info to the wiki.
Regards,
Peter.
Awesome. I'm not sure his point 1 abo
you do not have to make them yourself.
from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
EARTH has a Right To Life,
otherwise we all die.
- Original Message
From: Peter Karich
To: solr-user@lucene.apache.org
Sent: Mon, November 15, 2010 1:37:00 PM
Subject: Re: Tuning
ecurity/?p=4501&tag=nl.e036'
EARTH has a Right To Life,
otherwise we all die.
- Original Message
From: Peter Karich
To: solr-user@lucene.apache.org
Sent: Mon, November 15, 2010 1:37:00 PM
Subject: Re: Tuning Solr caches with high commit rates (NRT)
Hi Jonathan,
I am too using
Hi Jonathan,
I am too using fc because it simply was faster. Not sure if this can be
applied in general.
I will add this info to the wiki.
Regards,
Peter.
Awesome. I'm not sure his point 1 about facet.method=enum is still
valid in Solr 1.4+. The "fc" facet.method was changed significantly
Awesome. I'm not sure his point 1 about facet.method=enum is still valid
in Solr 1.4+. The "fc" facet.method was changed significantly in 1.4,
and generally no longer takes a lot of memory -- for facets with "many"
unique values, method fc in fact should take less than enum, I think?
Peter Ka
Just in case someone is interested:
I put the emails of Peter Sturge with some minor edits in the wiki:
http://wiki.apache.org/solr/NearRealtimeSearchTuning
I found myself search the thread again and again ;-)
Feel free to add and edit content!
Regards,
Peter.
Hi Erik,
I thought this woul
Hi,
why do you need to change the lockType? Does a readonly instance need
locks at all?
thanks,
Anders.
On Tue, 14 Sep 2010 15:00:54 +0200, Peter Karich wrote:
> Peter Sturge,
>
> this was a nice hint, thanks again! If you are here in Germany anytime I
> can invite you to a beer or an apfels
We are having some memory and GC issues. I'm trying to get a handle on the
contribution of the Solr caches. Is there a way to estimate the amount of
memory used by the documentCache and the queryResultCache?
I assume if we know the average size of our stored fields we can just multiply
> One strategy that I like, but haven't found in discussion lists is
> auto-limiting cache size/warming based on available resources (similar
> to the way file system caches use free memory). This would allow
> caches to adjust to their memory environment as indexes grow.
I've written such a cache
t; From: Erick Erickson
>> Subject: Re: Tuning Solr caches with high commit rates (NRT)
>> To: solr-user@lucene.apache.org
>> Date: Friday, September 17, 2010, 1:05 PM
>> Near Real Time...
>>
>> Erick
>>
>> On Fri, Sep 17, 2010 at 12:55 PM, Dennis Gearon wrote
Does Solr use Lucene NRT?
--- On Fri, 9/17/10, Erick Erickson wrote:
> From: Erick Erickson
> Subject: Re: Tuning Solr caches with high commit rates (NRT)
> To: solr-user@lucene.apache.org
> Date: Friday, September 17, 2010, 1:05 PM
> Near Real Time...
>
> Erick
>
&
rick Erickson
> Subject: Re: Tuning Solr caches with high commit rates (NRT)
> To: solr-user@lucene.apache.org
> Date: Friday, September 17, 2010, 10:05 AM
> Near Real Time...
>
> Erick
>
> On Fri, Sep 17, 2010 at 12:55 PM, Dennis Gearon wrote:
>
> > BTW, what i
> Laugh at http://www.yert.com/film.php
>
>
> --- On Fri, 9/17/10, Peter Sturge wrote:
>
> > From: Peter Sturge
> > Subject: Re: Tuning Solr caches with high commit rates (NRT)
> > To: solr-user@lucene.apache.org
> > Date: Friday, September 17, 2010, 2:18 AM
> >
BTW, what is NRT?
Dennis Gearon
Signature Warning
EARTH has a Right To Life,
otherwise we all die.
Read 'Hot, Flat, and Crowded'
Laugh at http://www.yert.com/film.php
--- On Fri, 9/17/10, Peter Sturge wrote:
> From: Peter Sturge
> Subject: Re: Tuning
Hi,
It's great to see such a fantastic response to this thread - NRT is
alive and well!
I'm hoping to collate this information and add it to the wiki when I
get a few free cycles (thanks Erik for the heads up).
In the meantime, I thought I'd add a few tidbits of additional
information that might
Peter Sturge,
this was a nice hint, thanks again! If you are here in Germany anytime I
can invite you to a beer or an apfelschorle ! :-)
I only needed to change the lockType to none in the solrconfig.xml,
disable the replication and set the data dir to the master data dir!
Regards,
Peter Karich.
Hi Peter,
this scenario would be really great for us - I didn't know that this is
possible and works, so: thanks!
At the moment we are doing similar with replicating to the readonly
instance but
the replication is somewhat lengthy and resource-intensive at this
datavolume ;-)
Regards,
Peter.
> 1
ject: Re: Tuning Solr caches with high commit rates (NRT)
> To: solr-user@lucene.apache.org
> Date: Monday, September 13, 2010, 1:33 AM
> On Mon, Sep 13, 2010 at 8:02 AM,
> Dennis Gearon
> wrote:
> > BTW, what is a segment?
>
> On the Lucene level an index is compo
- On Sun, 9/12/10, Jason Rutherglen wrote:
>
>> From: Jason Rutherglen
>> Subject: Re: Tuning Solr caches with high commit rates (NRT)
>> To: solr-user@lucene.apache.org
>> Date: Sunday, September 12, 2010, 7:52 PM
>> Yeah there's no patch... I think
>&
therglen wrote:
>
>> From: Jason Rutherglen
>> Subject: Re: Tuning Solr caches with high commit rates (NRT)
>> To: solr-user@lucene.apache.org
>> Date: Sunday, September 12, 2010, 7:52 PM
>> Yeah there's no patch... I think
>> Yonik can write it. :-) Ya
Hi Erik,
I thought this would be good for the wiki, but I've not submitted to
the wiki before, so I thought I'd put this info out there first, then
add it if it was deemed useful.
If you could let me know the procedure for submitting, it probably
would be worth getting it into the wiki (couldn't d
1. You can run multiple Solr instances in separate JVMs, with both
having their solr.xml configured to use the same index folder.
You need to be careful that one and only one of these instances will
ever update the index at a time. The best way to ensure this is to use
one for writing only,
and the
The balanced segment merging is a really cool idea. I'll definetely
have a look at this, thanks!
One thing I forgot to mention in the original post is we use a
mergeFactor of 25. Somewhat on the high side, so that incoming commits
aren't trying to merge new data into large segments.
25 is a good b
9/12/10, Jason Rutherglen wrote:
> From: Jason Rutherglen
> Subject: Re: Tuning Solr caches with high commit rates (NRT)
> To: solr-user@lucene.apache.org
> Date: Sunday, September 12, 2010, 7:52 PM
> Yeah there's no patch... I think
> Yonik can write it. :-) Yah... The
&
Yeah there's no patch... I think Yonik can write it. :-) Yah... The
Lucene version shouldn't matter. The distributed faceting
theoretically can easily be applied to multiple segments, however the
way it's written for me is a challenge to untangle and apply
successfully to a working patch. Also I
Thanks, Peter. This is really great info.
One setting I've found to be very useful for the problem of overlapping
onDeskSearchers is to reduce the value of maxWarmingSearchers in
solrconfig.xml. I've reduced this to 1, so if a slave is already busy doing
pre-warming, it won't try to also pre-
Bravo!
Other tricks: here is a policy for deciding when to merge segments that
attempts to balance merging with performance. It was contributed by
LinkedIn- they also run index&search in the same instance (not Solr, a
different Lucene app).
lucene/contrib/misc/src/java/org/apache/lucene/inde
Hi Jason,
I've tried some limited testing with the 4.x trunk using fcs, and I
must say, I really like the idea of per-segment faceting.
I was hoping to see it in 3.x, but I don't see this option in the
branch_3x trunk. Is your SOLR-1606 patch referred to in SOLR-1617 the
one to use with 3.1?
There
Peter,
Are you using per-segment faceting, eg, SOLR-1617? That could help
your situation.
On Sun, Sep 12, 2010 at 12:26 PM, Peter Sturge wrote:
> Hi,
>
> Below are some notes regarding Solr cache tuning that should prove
> useful for anyone who uses Solr with frequent commits (e.g. <5min).
>
>
Peter,
thanks a lot for your in-depth explanations!
Your findings will be definitely helpful for my next performance
improvement tests :-)
Two questions:
1. How would I do that:
> or a local read-only instance that reads the same core as the indexing
> instance (for the latter, you'll need som
Peter Sturge
> Subject: Tuning Solr caches with high commit rates (NRT)
> To: solr-user@lucene.apache.org
> Date: Sunday, September 12, 2010, 9:26 AM
> Hi,
>
> Below are some notes regarding Solr cache tuning that
> should prove
> useful for anyone who uses Sol
Peter:
This kind of information is extremely useful to document, thanks! Do you
have the time/energy to put it up on the Wiki? Anyone can edit it by
creating
a logon. If you don't, would it be OK if someone else did it (with
attribution,
of course)? I guess that by bringing it up I'm volunteering
Hi,
Below are some notes regarding Solr cache tuning that should prove
useful for anyone who uses Solr with frequent commits (e.g. <5min).
Environment:
Solr 1.4.1 or branch_3x trunk.
Note the 4.x trunk has lots of neat new features, so the notes here
are likely less relevant to the 4.x environmen
: Is it possible to use solr caches such as query cache , filter cache
: and document cache from external caching system like memcached as it
: has several advantages such as centralized caching system and reducing the
: pause time of JVM 's garbage collection as we can assign
Hi,
Is it possible to use solr caches such as query cache , filter cache
and document cache from external caching system like memcached as it
has several advantages such as centralized caching system and reducing the
pause time of JVM 's garbage collection as we can assign
On Mon, Apr 5, 2010 at 9:10 PM, Chris Hostetter
wrote:
>
>
> : > ... the reusing the FieldCache seems like hte only thing that would be
> : > advantageous in that case
> :
> : And FieldCache entries are currently reused when there have only been
> : deletions on a segment (since Solr 1.4).
>
> But
: > ... the reusing the FieldCache seems like hte only thing that would be
: > advantageous in that case
:
: And FieldCache entries are currently reused when there have only been
: deletions on a segment (since Solr 1.4).
But that's kind of orthoginal to (what i think) Lance's point was: that
On Mon, Apr 5, 2010 at 9:04 PM, Chris Hostetter
wrote:
> ... the reusing the FieldCache seems like hte only thing that would be
> advantageous in that case
And FieldCache entries are currently reused when there have only been
deletions on a segment (since Solr 1.4).
-Yonik
http://www.lucidimagin
: We had exactly this problem in a consumer app; we had a small but
: continuously growing list of obscene documents in the index, and did
: not want to display these. So, we had a filter query with all of the
: obscene words, and used this with every query.
that doesn't seem like it would really
: times. Is there any way to have the index keep its caches when the only thing
: that happens is deletions, then invalidate them when it's time to actually add
: data? It would have to be something I can dynamically change when switching
: between deletions and the daily import.
The problem is
In a word: "no".
What you can do instead of deleting them is to add them to a growing
list of "don't search for these documents". This could be listed in a
filter query.
We had exactly this problem in a consumer app; we had a small but
continuously growing list of obscene documents in the index,
I asked this question on Friday evening (US timezone), but nobody's
responded. Could be just that it's Easter weekend, but my question was
a little convoluted, so I'll re-ask it in a simpler way.
If all I'm doing in an index update is deleting documents, it seems that
it should be possible to
My index has a number of shards that are nearly static, each with about
7 million documents. By nearly static, I mean that the only changes
that normally happen to them are document deletions, done with the xml
update handler. The process that does these deletions runs once every
two minutes,
the difference between a commit and an add? Is there a good rule
> of thumb for how many docs one can add in one batch, and how many
> uncommitted docs are too many?
>
> 2) At what stage during updates do the solr caches get cleared out? My
> searches are facet heavy and require h
uring updates do the solr caches get cleared out? My
searches are facet heavy and require heavy usage of the solr caches to run
quickly - I'd like to minimize the frequency at which the caches are cleared
out.
--
View this message in context:
http://old.nabble.com/Commits-vs.-Adds-and-solr-c
83 matches
Mail list logo
