> >
> > > > > > > > If hard commit max time is 300 sec then commit happens every
> > 300
> > > > sec
> > > > > > on
> > > > > > > > tlog leader. And new segments pop up on th
rick,
> do
> > you
> > > > > mean
> > > > > > > that on all other tlog replicas(not leaders) commit occurs
> every
> > > > poll?
> > > > > > > воскресенье, 09 декабря 2018г., 19:21 +03:00 от Erick Ericks
ng interval is half the commit
> > > > interval
> > > > > > >
> > > > > > >This has always bothered me a little bit, I wonder at the
> utility
> > > of a
> > > > > > >config param. We already have old-style re
searcher(it
may be called soft commit).
--
Vadim
> -Original Message-
> From: Edward Ribeiro [mailto:edward.ribe...@gmail.com]
> Sent: Thursday, December 13, 2018 8:27 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Soft commit and new replica types
>
> Hi Tomás,
>
; > > > >On Sun, Dec 9, 2018 at 1:48 AM Vadim Ivanov
> > > > < vadim.iva...@spb.ntk-intourist.ru> wrote:
> > > > >
> > > > > Thanks, Edward, for clues.
> > > > > What bothers me is newSearcher s
gt; > < vadim.iva...@spb.ntk-intourist.ru> wrote:
> > > >
> > > > Thanks, Edward, for clues.
> > > > What bothers me is newSearcher start, warming, cache clear... all
> that
> > > CPU consuming stuff in my heavy-indexing scenario.
&g
more than every 5 min on every replica.
> > > To have more or less the same effect with TLOG - PULL collection,
> > > I suppose, I have to have : 30
> > > (yes, I understand that newSearchers start asynchronously on leader and
> > replicas)
> > &
effect with TLOG - PULL collection,
> > I suppose, I have to have : 30
> > (yes, I understand that newSearchers start asynchronously on leader and
> replicas)
> > Am I right?
> > --
> > Vadim
> >
> >
> >> -Original Message-
> >&g
[mailto:edward.ribe...@gmail.com]
>> Sent: Sunday, December 09, 2018 12:42 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Soft commit and new replica types
>>
>> Some insights in the new replica types below:
>>
>> On Sat, December 8, 2018 08:42, Vadi
--Original Message-
> > From: Edward Ribeiro [mailto:edward.ribe...@gmail.com]
> > Sent: Sunday, December 09, 2018 12:42 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Soft commit and new replica types
> >
> > Some insights in the new replica types below:
>
018 12:42 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Soft commit and new replica types
>
> Some insights in the new replica types below:
>
> On Sat, December 8, 2018 08:42, Vadim Ivanov <
> vadim.iva...@spb.ntk-intourist.ru wrote:
>
> >
> > From
Some insights in the new replica types below:
On Sat, December 8, 2018 08:42, Vadim Ivanov <
vadim.iva...@spb.ntk-intourist.ru wrote:
>
> From Ref guide we have:
> " NRT is the only type of replica that supports soft-commits..."
> "If TLOG replica does become a leader, it will behave the same as
Hi Erick,
Thanks for the response.
First thing we not indexing on Slave. And we are not re-indexing/optimizing
entire the core in Master node.
The only warning which I see in the log is "Unable clean the unused index
directory so starting full copy".
That one i can understand and I don't ha
My first guess is that you're indexing to the slave nodes.
Second guess is that you're re-indexing your entire corpus on the master node.
Third guess is that you're optimizing on the master node (don't do this)
What does the slave's log say is the reason? If all the segments on
the master have c
First, if you specify commit it's doing a hard commit with
openSearcher=true by default so the softCommit isn't necessary here.
I'd do one or the other, as it's possible that Solr is stopping at the
first one.
bq: when i do the hardcommit manually . then its shows the result on website.
I don't k
On 12/19/2016 7:12 PM, Lasitha Wattaladeniya wrote:
> *Requirement *is, we are showing a list of entries on a page. For each
> user there's a read / unread flag. The data for listing is fetched
> from solr. And you can see the entry was previously read or not. So
> when a user views an entry by cli
Hi,
so, the app already has a database connection because it updates the
READ flag when the user clicks an entry, right? If you only need the
flag for display purposes, it sounds like it would make sense to also
fetch it directly from the database when displaying the listing. Of
course if you
You probably need a database instead of a search engine.
What requirement makes you want to do this with a search engine?
wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Dec 19, 2016, at 6:34 PM, Lasitha Wattaladeniya wrote:
>
> Hi Hendrik,
>
> T
Hi Hendrik,
Thanks for your input. Previously I was using the hard commit
(SolrClient.commit()) but then I got some error when there are concurrent
real time index requests from my app. The error was "Exceeded limit of
maxWarmingSearchers=2, try again later", then i changed the code to use
only s
Hi Shawn,
Thanks for your well detailed explanation. Now I understand, I won't be
able to achieve the 100ms softcommit timeout with my hardware setup.
However let's say someone has a requirement as below (quoted from my
previous mail)
*Requirement *is, we are showing a list of entries on a page.
Hi,
the SolrJ API has this method: SolrClient.commit(String collection,
boolean waitFlush, boolean waitSearcher, boolean softCommit).
My assumption so far was that when you set waitSearcher to true that the
method call only returns once a search would find the new data, which
sounds what you w
On 12/18/2016 7:09 PM, Lasitha Wattaladeniya wrote:
> @eric : thanks for the lengthy reply. So let's say I increase the
> autosoftcommit time out to may be 100 ms. In that case do I have to
> wait much that time from client side before calling search ?. What's
> the correct way of achieving this?
I didn't look much onto REALTIME GET handler. Thanks for mentioning
that. I'm checking it now
On 19 Dec 2016 10:09, "Lasitha Wattaladeniya" wrote:
> Hi all,
>
> Thanks for your replies,
>
> @dorian : the requirement is, we are showing a list of entries on a page.
> For each user there's a re
Hi all,
Thanks for your replies,
@dorian : the requirement is, we are showing a list of entries on a page.
For each user there's a read / unread flag. The data for listing is
fetched from solr. And you can see the entry was previously read or not. So
when a user views an entry by clicking. We
1 ms autocommit is far too frequent. And it's not
helping you anyway.
There is some lag between when a commit happens
and when the docs are really available. The sequence is:
1> commit (soft or hard-with-opensearcher=true doesn't matter).
2> a new searcher is opened and autowarming starts
3> until
There's a very high probability that you're using the wrong tool for the
job if you need 1ms softCommit time. Especially when you always need it (ex
there are apps where you need commit-after-insert very rarely).
So explain what you're using it for ?
On Sun, Dec 18, 2016 at 3:38 PM, Lasitha Watta
Hi Furkan,
Thanks for the links. I had read the first one but not the second one. I
did read it after you sent. So in my current solrconfig.xml settings below
are the configurations,
${solr.autoSoftCommit.maxTime:1}
15000
false
The problem i'm facing is, just after adding the do
Hi Lasitha,
First of all, did you check these:
https://cwiki.apache.org/confluence/display/solr/Near+Real+Time+Searching
https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
after that, if you cannot adjust your configuration you can give more
Hi furkan,
Thanks for your reply, it is generally a query heavy system. We are using
realtime indexing for editing the available data
Regards,
Lasitha
Lasitha Wattaladeniya
Software Engineer
Mobile : +6593896893
Blog : techreadme.blogspot.com
On Sun, Dec 18, 2016 at 8:12 PM, Furkan KAMACI
wro
Hi Lasitha,
What is your indexing / querying requirements. Do you have an index
heavy/light - query heavy/light system?
Kind Regards,
Furkan KAMACI
On Sun, Dec 18, 2016 at 11:35 AM, Lasitha Wattaladeniya
wrote:
> Hello devs,
>
> I'm here with another problem i'm facing. I'm trying to do a com
Got it. Thanks.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Soft-commit-from-curl-tp4302288p4302615.html
Sent from the Solr - User mailing list archive at Nabble.com.
The best way is to look at your Solr logs. When you see the commit
message, you'll see things like
"start
commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}"
that ought to work, as should something like:
curl blah blah/update?sof
some environments."
Thanks & Regards,
Bhaumik Joshi
From: billnb...@gmail.com
Sent: Monday, April 11, 2016 7:07 AM
To: solr-user@lucene.apache.org
Subject: Re: Soft commit does not affecting query performance
Why do you think it would ?
Bill Bell
Se
Why do you think it would ?
Bill Bell
Sent from mobile
> On Apr 11, 2016, at 7:48 AM, Bhaumik Joshi wrote:
>
> Hi All,
>
> We are doing query performance test with different soft commit intervals. In
> the test with 1sec of soft commit interval and 1min of soft commit interval
> we didn't n
In particular please give us additional details about your search use case .
If the master is not searched, do you mean you have a master/slave
architecture ?
In the case, how replication is managed ?
If you are replicating old style, you are going to be able to see only what
is in the disk at the
Dear Midas,
Hi,
AFAIK, currently Solr uses virtual memory for storing memory maps.
Therefore using 36GB from 48GB of ram for Java heap is not recommended. As
a rule of thumb do not access more than 25% of your total memory to Solr
JVM in usual situations.
About your main question, setting softcommi
In a word "yes". The Solr servers are independently keeping their own
timers and one could trip on replica X while an update was in
transmission from the leader say. Or any one of a zillion other timing
conditions. In fact, this is why the indexes will have different
segments on replicas in a slice
Thanks for closing this off. it'd have been a pretty serious
thing if soft commits weren't working.
Erick
On Wed, Nov 26, 2014 at 12:58 PM, Andreas Hubold
wrote:
> Thank you, Shawn and Erick!
>
> With your hint about the re-used searcher I was able to find my error. I
> must wait for the newly o
Thank you, Shawn and Erick!
With your hint about the re-used searcher I was able to find my error. I
must wait for the newly opened searcher when calling the commit method:
solrServer.commit(false, true /*waitSearcher*/, true /*softCommit*/);
instead of
solrServer.commit(false, false, true);
As Shawn says, deletes should be
visible after a soft commit.
Let's see the code though. If you re-use a searcher that
you had open before the commit, it'll still see the old
snapshot of the index including the deleted documents.
Or if you do open a new searcher and any autowarming
hasn't complete
On 11/26/2014 8:18 AM, Andreas Hubold wrote:
> But I'm still not totally sure. Does a soft commit also make deleted
> documents invisible?
>
> In a test with an EmbeddedSolrServer I triggered a soft commit and was
> still able to find a deleted document afterwards. Is this as expected?
All change
Tim,
my suggestion was very concise, sorry for that. But not at all "rude" or
anything. Instead, tried to help you.
Dmitry
On Wed, Oct 9, 2013 at 9:28 PM, Tim Vaillancourt wrote:
> Apologies all. I think the suggestion that I was replying "to get noticed"
> is what erked me, otherwise I would
Apologies all. I think the suggestion that I was replying "to get noticed"
is what erked me, otherwise I would have moved on. I'll follow this advice.
Cheers,
Tim
On 9 October 2013 05:20, Erick Erickson wrote:
> Tim:
>
> I think you're mis-interpreting. By replying to a post with the subject:
Tim:
I think you're mis-interpreting. By replying to a post with the subject:
{soft}Commit and cache flushing
but going in a different direction, it's easy for people to think "I'm
not interested in that
thread, I'll ignore it", thereby missing the fact that you're asking a
somewhat different
qu
I have a genuine question with substance here. If anything this
nonconstructive, rude response was "to get noticed". Thanks for
contributing to the discussion.
Tim
On 8 October 2013 05:31, Dmitry Kan wrote:
> Tim,
> I suggest you open a new thread and not reply to this one to get noticed.
> Dm
Tim,
I suggest you open a new thread and not reply to this one to get noticed.
Dmitry
On Mon, Oct 7, 2013 at 9:44 PM, Tim Vaillancourt wrote:
> Is there a way to make autoCommit only commit if there are pending changes,
> ie: if there are 0 adds pending commit, don't autoCommit (open-a-searcher
bq: If so, using soft commit without calling hard commit could cause OOM
no. Aside from anything you have configured for auto(hard) commit, the
ramBufferSizeMB in solrconfig.xml will flush the in-memory structures out
to the segments when the size reaches this limit. It won't _close_ the
current
Is there a way to make autoCommit only commit if there are pending changes,
ie: if there are 0 adds pending commit, don't autoCommit (open-a-searcher
and wipe the caches)?
Cheers,
Tim
On 2 October 2013 00:52, Dmitry Kan wrote:
> right. We've got the autoHard commit configured only atm. The so
Out of Memory Exception is well known as OOM.
Guido.
On 07/10/13 14:11, adfel70 wrote:
Sorry, by "OOE" I meant Out of memory exception...
--
View this message in context:
http://lucene.472066.n3.nabble.com/Soft-commit-and-flush-tp4091726p4093902.html
Sent from the Solr - User mailing list a
Sorry, by "OOE" I meant Out of memory exception...
--
View this message in context:
http://lucene.472066.n3.nabble.com/Soft-commit-and-flush-tp4091726p4093902.html
Sent from the Solr - User mailing list archive at Nabble.com.
bq: Does the NRTCachingDirectoryFactory relevant for both types of commit, or
just for hard commit
Don't know the code deeply, but NRT==Near Real Time == Soft commit I'd guess.
bq: If soft commit does not flush...
soft commit flushes the transaction log. On restart if the content of
the tlog isn
I understand the bottom line that soft commits are about visibility, hard
commits are about durability. I am just trying to gain better understanding
what happens under the hood...
2 more related questions you made me think of:
1. Does the NRTCachingDirectoryFactory relevant for both types of commi
right. We've got the autoHard commit configured only atm. The soft-commits
are controlled on the client. It was just easier to implement the first
version of our internal commit policy that will commit to all solr
instances at once. This is where we have noticed the reported behavior.
On Wed, Oct
if there are no modifications to an index and a softCommit or hardCommit
issued, then solr flushes the cache.
Indeed. The easiest way to work around this is by disabling auto commits
and only commit when you have to.
Thanks a lot Shawn for an exhaustive reply!
Regards,
Dmitry
On Tue, Oct 1, 2013 at 5:37 PM, Shawn Heisey wrote:
> On 10/1/2013 2:48 AM, Dmitry Kan wrote:
> > This is a minor thing, perhaps, but thought to ask / share:
> >
> > if there are no modifications to an index and a softCommit or hardCo
On 10/1/2013 2:48 AM, Dmitry Kan wrote:
> This is a minor thing, perhaps, but thought to ask / share:
>
> if there are no modifications to an index and a softCommit or hardCommit
> issued, then solr flushes the cache.
Any time you do a commit that opens a new Searcher object
(openSearcher=true, w
Why do you care? Curiosity or are you trying to find a
behavior you can count on?
because "soft commits are about visibility, hard commits are
about durability". Meaning you can't count on a soft commit
writing anything to disk at all. I suspect in your tests the soft
commit had nothing to do with
On 9/24/2013 5:51 AM, adfel70 wrote:
My conclusion is that soft commit always flushes the data, but because of
the implementation of NRTCachingDirectoryFactory, the data will be written
to the disk when its getting too big.
The NRTCachingDirectoryFactory (which creates NRTCachingDirectory
inst
Hi,
I believe data is not fsynched to disk until a hard commit (and even
then disks can lie to you and tell you data is safe even though it's
still in disk cache waiting to really be written to the medium) ,
which is why you can lose it between hard commits. Soft commits just
make newly added doc
Thanks Shawn and Mark! That was very helpful.
-Niran
>
> From: Shawn Heisey
>To: solr-user@lucene.apache.org
>Sent: Monday, April 22, 2013 5:30 PM
>Subject: Re: Soft Commit and Document Cache
>
>
>On 4/22/2013 4:16 PM, Niran Fajemisi
On 4/22/2013 4:16 PM, Niran Fajemisin wrote:
A quick (and hopefully simply) question: Does the document cache (or any of the
other caches for that matter), get invalidated after a soft commit has been
performed?
All Solr caches are invalidated when you issue a commit with
openSearcher set to
Yup - all of the top level caches are. It's a trade off - don't NRT more than
you need to.
- Mark
On Apr 22, 2013, at 6:16 PM, Niran Fajemisin wrote:
> Hi all,
>
> A quick (and hopefully simply) question: Does the document cache (or any of
> the other caches for that matter), get invalidated
What is your evidence that it doesn't work
when you specify it in solrconfig.xml? You
haven't provided enough information about
what you've tried to give us much to go on.
It might help to review:
http://wiki.apache.org/solr/UsingMailingLists
Best
Erick
On Tue, Jan 3, 2012 at 8:17 AM, ramires w
Address the points I brought up or don't reply with funny name calling.
Below are two key points reiterated and re-articulated is an easy to answer way:
* Multi-select faceting is per-segment (true or false)
* Filters are cached per-segment (true or false)
On Tue, Jan 3, 2012 at 2:16 PM, Yonik
On Tue, Jan 3, 2012 at 5:03 PM, Jason Rutherglen
wrote:
> Yikes. I'd love to see a test showing that un-inverted field cache
> (which is for ALL segments as a single unit) can be used efficiently
> with NRT / soft commit.
Please stop being a troll.
Solr as multiple faceting methods - only one us
The main point is, Solr unlike for example Elastic Search and other
Lucene based systems does NOT cache filters or facets per-segment.
This is a fundamental design flaw.
On Tue, Jan 3, 2012 at 1:50 PM, Yonik Seeley wrote:
> On Tue, Jan 3, 2012 at 4:36 PM, Erik Hatcher wrote:
>> As I understand
> multi-select faceting
Yikes. I'd love to see a test showing that un-inverted field cache
(which is for ALL segments as a single unit) can be used efficiently
with NRT / soft commit.
On Tue, Jan 3, 2012 at 1:50 PM, Yonik Seeley wrote:
> On Tue, Jan 3, 2012 at 4:36 PM, Erik Hatcher wrote:
>> A
On Tue, Jan 3, 2012 at 4:36 PM, Erik Hatcher wrote:
> As I understand it, the document and filter caches add value *intra* request
> such that it keeps additional work (like fetching stored fields from disk
> more than once) from occurring.
Yep. Highlighting, multi-select faceting, and distrib
As I understand it, the document and filter caches add value *intra* request
such that it keeps additional work (like fetching stored fields from disk more
than once) from occurring.
Erik
On Jan 3, 2012, at 16:26 , Jason Rutherglen wrote:
> *Laugh*
>
> I stand by what Mark said:
>
>
*Laugh*
I stand by what Mark said:
"Right - in most NRT cases (very frequent soft commits), the cache should
probably be disabled."
On Mon, Jan 2, 2012 at 7:45 PM, Yonik Seeley wrote:
> On Mon, Jan 2, 2012 at 9:58 PM, Jason Rutherglen
> wrote:
>>> It still normally makes sense to have the cach
On Mon, Jan 2, 2012 at 9:58 PM, Jason Rutherglen
wrote:
>> It still normally makes sense to have the caches enabled (esp filter and
>> document caches).
>
> In the NRT case that statement is completely incorrect
*shrug*
To each their own. I stand my my statement.
-Yonik
http://www.lucidimagin
> It still normally makes sense to have the caches enabled (esp filter and
> document caches).
In the NRT case that statement is completely incorrect
On Mon, Jan 2, 2012 at 5:37 PM, Yonik Seeley wrote:
> On Mon, Jan 2, 2012 at 1:28 PM, Mark Miller wrote:
>> Right - in most NRT cases (very freq
On Mon, Jan 2, 2012 at 1:28 PM, Mark Miller wrote:
> Right - in most NRT cases (very frequent soft commits), the cache should
> probably be disabled.
Did you mean autowarm should be disabled (as it already is in the
example config)?
It still normally makes sense to have the caches enabled (esp fi
Right - in most NRT cases (very frequent soft commits), the cache should
probably be disabled.
2012/1/2 Tomás Fernández Löbbe
> Yes, soft commit currently clears Solr's caches.
>
> On Mon, Jan 2, 2012 at 12:01 PM, ramires wrote:
>
> > hi
> >
> > After soft-commit with below command all cache ar
Yes, soft commit currently clears Solr's caches.
On Mon, Jan 2, 2012 at 12:01 PM, ramires wrote:
> hi
>
> After soft-commit with below command all cache are cleared. Is it normal?
>
> curl http://localhost:8984/solr/update -H "Content-Type: text/xml"
> --data-binary ' waitSearcher="false"/>'
>
>
75 matches
Mail list logo
