Re: Replication in soft commit

2020-09-03 Thread Emir Arnautović
raining - http://sematext.com/ >> >> >> >>> On 3 Sep 2020, at 08:38, Tushar Arora wrote: >>> >>> Hi, >>> I want to ask if the soft commit works in replication. >>> One of our use cases deals with indexing the data every second on a

Re: Replication in soft commit

2020-09-03 Thread Tushar Arora
tp://sematext.com/ > > > > > On 3 Sep 2020, at 08:38, Tushar Arora wrote: > > > > Hi, > > I want to ask if the soft commit works in replication. > > One of our use cases deals with indexing the data every second on a > master > > server. And then it h

Re: Replication in soft commit

2020-09-03 Thread Emir Arnautović
avoided. HTH, Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 3 Sep 2020, at 08:38, Tushar Arora wrote: > > Hi, > I want to ask if the soft commit works in replication. > One of our u

Replication in soft commit

2020-09-02 Thread Tushar Arora
Hi, I want to ask if the soft commit works in replication. One of our use cases deals with indexing the data every second on a master server. And then it has to replicate to slaves. So if we use soft commit, then does the data replicate immediately to the slave server or after the hard commit

Re: Slow soft-commit

2019-06-05 Thread Erick Erickson
onds, no >> explicit soft-commits but documents added with commitWhitin=5000 or 1000 >> depending on the use case. No warm-up queries, caches set to zero. >> >> I enabled infostream and debug logging. Here is a little test case where I >> stopped any othe

Re: Slow soft-commit

2019-06-04 Thread Erick Erickson
onds, no >> explicit soft-commits but documents added with commitWhitin=5000 or 1000 >> depending on the use case. No warm-up queries, caches set to zero. >> >> I enabled infostream and debug logging. Here is a little test case where I >> stopped any othe

Re: Slow soft-commit

2019-06-04 Thread André Widhani
m-up queries, caches set to zero. > > I enabled infostream and debug logging. Here is a little test case where I > stopped any other requests to Solr and just manually added a single > document and then posted a soft commit request. > > 2019-05-22 17:19:42.160 INFO (qtp267

Slow soft-commit

2019-05-22 Thread André Widhani
, caches set to zero. I enabled infostream and debug logging. Here is a little test case where I stopped any other requests to Solr and just manually added a single document and then posted a soft commit request. 2019-05-22 17:19:42.160 INFO (qtp26728049-20) o.a.s.u.DirectUpdateHandler2 start commit

Re: Soft commit and new replica types

2018-12-14 Thread Tomás Fernández Löbbe
t seeing reloads. > > > > Ah, good. > > > > > > > > I am trying to understand the interactions > > > > between hard commit, soft commit, transaction log update with a TLOG > > > > cluster for both leader and follower replicas.

Re: Soft commit and new replica types

2018-12-14 Thread Edward Ribeiro
n Fri, Dec 14, 2018 at 3:08 AM Tomás Fernández Löbbe wrote: > > > > > > No, I am not seeing reloads. > > Ah, good. > > > > > I am trying to understand the interactions > > > between hard commit, soft commit, transaction log update with a TLOG >

Re: Soft commit and new replica types

2018-12-13 Thread Tomás Fernández Löbbe
> > > > No, I am not seeing reloads. Ah, good. > > I am trying to understand the interactions > > between hard commit, soft commit, transaction log update with a TLOG > > cluster for both leader and follower replicas. For example, after getting > > new s

RE: Soft commit and new replica types

2018-12-13 Thread Vadim Ivanov
bq. , after getting new segments from the leader the follower replica will still apply the hard/soft commit? As was described in one of the videos below, follower tlog replica look for max docid in received new segments and purge its transaction log of older records. Than it starts new

Re: Soft commit and new replica types

2018-12-13 Thread Edward Ribeiro
Hi Tomás, No, I am not seeing reloads. I am trying to understand the interactions between hard commit, soft commit, transaction log update with a TLOG cluster for both leader and follower replicas. For example, after getting new segments from the leader the follower replica will still apply the

Re: Soft commit and new replica types

2018-12-10 Thread Tomás Fernández Löbbe
gt; > < vadim.iva...@spb.ntk-intourist.ru> wrote: > > > > > > > > Thanks, Edward, for clues. > > > > What bothers me is newSearcher start, warming, cache clear... all > that > > > CPU consuming stuff in my heavy-indexing scenario. &g

Re: Soft commit and new replica types

2018-12-10 Thread Erick Erickson
more than every 5 min on every replica. > > > To have more or less the same effect with TLOG - PULL collection, > > > I suppose, I have to have : 30 > > > (yes, I understand that newSearchers start asynchronously on leader and > > replicas) > > &

Re: Soft commit and new replica types

2018-12-09 Thread Edward Ribeiro
effect with TLOG - PULL collection, > > I suppose, I have to have : 30 > > (yes, I understand that newSearchers start asynchronously on leader and > replicas) > > Am I right? > > -- > > Vadim > > > > > >> -Original Message- > >&g

Re: Soft commit and new replica types

2018-12-09 Thread vadim . ivanov
[mailto:edward.ribe...@gmail.com] >> Sent: Sunday, December 09, 2018 12:42 AM >> To: solr-user@lucene.apache.org >> Subject: Re: Soft commit and new replica types >> >> Some insights in the new replica types below: >> >> On Sat, December 8, 2018 08:42, Vadi

Re: Soft commit and new replica types

2018-12-09 Thread Erick Erickson
--Original Message- > > From: Edward Ribeiro [mailto:edward.ribe...@gmail.com] > > Sent: Sunday, December 09, 2018 12:42 AM > > To: solr-user@lucene.apache.org > > Subject: Re: Soft commit and new replica types > > > > Some insights in the new replica types below: >

RE: Soft commit and new replica types

2018-12-09 Thread Vadim Ivanov
018 12:42 AM > To: solr-user@lucene.apache.org > Subject: Re: Soft commit and new replica types > > Some insights in the new replica types below: > > On Sat, December 8, 2018 08:42, Vadim Ivanov < > vadim.iva...@spb.ntk-intourist.ru wrote: > > > > > From

Re: Soft commit and new replica types

2018-12-08 Thread Edward Ribeiro
periodically poll the leader for changes in index segments' files and download those segments from the leader. If hard commit max time is defined in solrconfig.xml the polling interval of each replica will be half that value. Or else if the soft commit max time is defined then the replicas wil

Soft commit and new replica types

2018-12-08 Thread Vadim Ivanov
Before 7.x all replicas in SolrCloud were NRT type. And following rules were applicable: https://stackoverflow.com/questions/45998804/when-should-we-apply-hard-commit-and-soft-commit-in-solr and https://lucene.apache.org/solr/guide/7_5/updatehandlers-in-solrconfig.html#commit-and-softcommit But

Re: Soft commit impact on replication

2018-06-17 Thread Adarsh_infor
nd I don't have issue with that as its normal behaviour but most of the time. But most of the time it’s just triggers full-copy without any details in the log. And recently in one of the nodes i enabled soft-commit in master nodes and monitored the corresponding slave node, what i observed is

Re: Soft commit impact on replication

2018-06-15 Thread Erick Erickson
My first guess is that you're indexing to the slave nodes. Second guess is that you're re-indexing your entire corpus on the master node. Third guess is that you're optimizing on the master node (don't do this) What does the slave's log say is the reason? If all the segments on the master have c

Soft commit impact on replication

2018-06-15 Thread Adarsh Hd
Hi All, Current am using SOLR 5.2.1 on Linux machine. I have cluster of 5 nodes with master and salve configuration, which gives 5 master nodes and 5slave node. We have enabled only hard commit on master nodes and both soft & hard commit on the slave nodes since the search will happen on slave

Re: Soft commit uploading datas cant search on website

2017-08-11 Thread Erick Erickson
ods_v6.6] o.a.s.c.S.Request [goods_v6.6] webapp=/solr > path=/select > params={q=keyword:"*solrtest*"&start=0&fq=storePrice:[0+TO+9]&fq=goodsStatus:0&sort=goodsClick+desc&rows=12&wt=javabin&version=2} > hits=0 status=0 QTime=2 > > > any co

Soft commit uploading datas cant search on website

2017-08-11 Thread Abdul Ekfhy
mp;sort=goodsClick+desc&rows=12&wt=javabin&version=2} hits=0 status=0 QTime=2 any configurations i need to add more to solrconfig.xml ?? -- View this message in context: http://lucene.472066.n3.nabble.com/Soft-commit-uploading-datas-cant-search-on-website-tp4350186.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Soft commit and reading data just after the commit

2016-12-20 Thread Shawn Heisey
On 12/19/2016 7:12 PM, Lasitha Wattaladeniya wrote: > *Requirement *is, we are showing a list of entries on a page. For each > user there's a read / unread flag. The data for listing is fetched > from solr. And you can see the entry was previously read or not. So > when a user views an entry by cli

Re: Soft commit and reading data just after the commit

2016-12-19 Thread Ere Maijala
Hi, so, the app already has a database connection because it updates the READ flag when the user clicks an entry, right? If you only need the flag for display purposes, it sounds like it would make sense to also fetch it directly from the database when displaying the listing. Of course if you

Re: Soft commit and reading data just after the commit

2016-12-19 Thread Walter Underwood
+6593896893 >>>>>> Blog : techreadme.blogspot.com >>>>>> >>>>>> On Sun, Dec 18, 2016 at 8:46 PM, Furkan KAMACI >>>>>> >>>>>> wrote: >>>>>> >>>>>> Hi Lasitha, >>>>>>&g

Re: Soft commit and reading data just after the commit

2016-12-19 Thread Lasitha Wattaladeniya
; >>>>>> https://cwiki.apache.org/confluence/display/solr/Near+ >>>>>> >>>>> Real+Time+Searching >>>>> >>>>>> https://lucidworks.com/blog/2013/08/23/understanding- >>>>>> transaction-logs-softc

Re: Soft commit and reading data just after the commit

2016-12-19 Thread Lasitha Wattaladeniya
Hi Shawn, Thanks for your well detailed explanation. Now I understand, I won't be able to achieve the 100ms softcommit timeout with my hardware setup. However let's say someone has a requirement as below (quoted from my previous mail) *Requirement *is, we are showing a list of entries on a page.

Re: Soft commit and reading data just after the commit

2016-12-19 Thread Hendrik Haddorp
16 at 11:35 AM, Lasitha Wattaladeniya < watt...@gmail.com> wrote: Hello devs, I'm here with another problem i'm facing. I'm trying to do a commit (soft commit) through solrj and just after the commit, retrieve the data from solr (requirement is to get updated data list).

Re: Soft commit and reading data just after the commit

2016-12-19 Thread Shawn Heisey
On 12/18/2016 7:09 PM, Lasitha Wattaladeniya wrote: > @eric : thanks for the lengthy reply. So let's say I increase the > autosoftcommit time out to may be 100 ms. In that case do I have to > wait much that time from client side before calling search ?. What's > the correct way of achieving this?

Re: Soft commit and reading data just after the commit

2016-12-18 Thread Lasitha Wattaladeniya
tcommit-and-commit-in-sorlcloud/ >> >> > >> >> > after that, if you cannot adjust your configuration you can give more >> >> > information and we can find a solution. >> >> > >> >> > Kind Regards, >> >> > Furkan KA

Re: Soft commit and reading data just after the commit

2016-12-18 Thread Lasitha Wattaladeniya
ttaladeniya < > >> watt...@gmail.com> > >> > wrote: > >> > > >> >> Hi furkan, > >> >> > >> >> Thanks for your reply, it is generally a query heavy system. We are > >> using > >> >> real

Re: Soft commit and reading data just after the commit

2016-12-18 Thread Erick Erickson
We are >> using >> >> realtime indexing for editing the available data >> >> >> >> Regards, >> >> Lasitha >> >> >> >> Lasitha Wattaladeniya >> >> Software Engineer >> >> >> >> Mobile

Re: Soft commit and reading data just after the commit

2016-12-18 Thread Dorian Hoxha
t.com > >> > >> On Sun, Dec 18, 2016 at 8:12 PM, Furkan KAMACI > >> wrote: > >> > >>> Hi Lasitha, > >>> > >>> What is your indexing / querying requirements. Do you have an index > >>> heavy/light - query heavy/light sys

Re: Soft commit and reading data just after the commit

2016-12-18 Thread Lasitha Wattaladeniya
t;> >>> Hi Lasitha, >>> >>> What is your indexing / querying requirements. Do you have an index >>> heavy/light - query heavy/light system? >>> >>> Kind Regards, >>> Furkan KAMACI >>> >>> On Sun, Dec 18, 2016 at 11:35 AM, La

Re: Soft commit and reading data just after the commit

2016-12-18 Thread Furkan KAMACI
exing / querying requirements. Do you have an index >> heavy/light - query heavy/light system? >> >> Kind Regards, >> Furkan KAMACI >> >> On Sun, Dec 18, 2016 at 11:35 AM, Lasitha Wattaladeniya < >> watt...@gmail.com> >> wrote: >> >

Re: Soft commit and reading data just after the commit

2016-12-18 Thread Lasitha Wattaladeniya
> > > > I'm here with another problem i'm facing. I'm trying to do a commit (soft > > commit) through solrj and just after the commit, retrieve the data from > > solr (requirement is to get updated data list). > > > > I'm using soft commit ins

Re: Soft commit and reading data just after the commit

2016-12-18 Thread Furkan KAMACI
I'm trying to do a commit (soft > commit) through solrj and just after the commit, retrieve the data from > solr (requirement is to get updated data list). > > I'm using soft commit instead of the hard commit, is previously I got an > error "Exceeded limit of maxWarmingSe

Soft commit and reading data just after the commit

2016-12-18 Thread Lasitha Wattaladeniya
Hello devs, I'm here with another problem i'm facing. I'm trying to do a commit (soft commit) through solrj and just after the commit, retrieve the data from solr (requirement is to get updated data list). I'm using soft commit instead of the hard commit, is previously I go

Re: Soft commit from curl

2016-10-22 Thread mimino
Got it. Thanks. -- View this message in context: http://lucene.472066.n3.nabble.com/Soft-commit-from-curl-tp4302288p4302615.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Soft commit from curl

2016-10-21 Thread Erick Erickson
lah blah/update?softCommit=true&commit=true Best, Erick On Thu, Oct 20, 2016 at 12:23 PM, Michal Danilák wrote: > Does the following command issue soft commit or hard commit? > > curl http://localhost:8984/solr/update?softCommit=true -H "Content-Type: > text/xml" --data-b

Soft commit from curl

2016-10-20 Thread Michal Danilák
Does the following command issue soft commit or hard commit? curl http://localhost:8984/solr/update?softCommit=true -H "Content-Type: text/xml" --data-binary '' How to find out which commit was triggered? Can I get it somewhere in logs? Thanks.

Re: Soft commit does not affecting query performance

2016-04-13 Thread Bhaumik Joshi
some environments." Thanks & Regards, Bhaumik Joshi From: billnb...@gmail.com Sent: Monday, April 11, 2016 7:07 AM To: solr-user@lucene.apache.org Subject: Re: Soft commit does not affecting query performance Why do you think it would ? Bill Bell Se

Re: Soft commit does not affecting query performance

2016-04-11 Thread billnbell
Why do you think it would ? Bill Bell Sent from mobile > On Apr 11, 2016, at 7:48 AM, Bhaumik Joshi wrote: > > Hi All, > > We are doing query performance test with different soft commit intervals. In > the test with 1sec of soft commit interval and 1min of soft commit int

Soft commit does not affecting query performance

2016-04-11 Thread Bhaumik Joshi
Hi All, We are doing query performance test with different soft commit intervals. In the test with 1sec of soft commit interval and 1min of soft commit interval we didn't notice any improvement in query timings. We did test with SolrMeter (Standalone java tool for stress tests with

Re: Does soft commit re-opens searchers in disk?

2016-01-04 Thread Daniel Collins
If you have already done a soft commit and that opened a new searcher, then the document will be visible from that point on. The results returned by that searcher cannot be changed by the hard commit (whatever that is doing under the hood, the segment that has that document in must still be

Re: Does soft commit re-opens searchers in disk?

2016-01-04 Thread Emir Arnautovic
Hi Gili, Visibility is related to searcher - if you reopen searcher it will be visible. If hard commit happens without reopening searcher, documents will not be visible till next soft commit happens. You can find more details about commits on https://lucidworks.com/blog/2013/08/23

Does soft commit re-opens searchers in disk?

2016-01-04 Thread Gili Nachum
Hello, When a new document is added, it becomes visible after a soft commit, during which it is written to a Lucene RAMDirectory (in heap). Then after a hard commit, the RAMDirectory is removed from memory and the docs are written to the index on disk. What happens if I hard commit (write to disk

Re: Soft commit and hard commit

2015-11-30 Thread Alessandro Benedetti
the moment of the replication, which means only the hard committed segments . Is this your case ? Do you need soft commit at all ? >From Erick's guide : Heavy (bulk) indexing > The assumption here is that you’re interested in getting lots of data to > the index as quickly as possi

Re: Soft commit and hard commit

2015-11-30 Thread Ali Nazemian
-and-commit-in-sorlcloud/ Best regards. On Mon, Nov 30, 2015 at 9:48 AM, Midas A wrote: > Machine configuration > > RAM: 48 GB > CPU: 8 core > JVM : 36 GB > > We are updating 70 , 000 docs / hr . what should be our soft commit and > hard commit time to get b

Soft commit and hard commit

2015-11-29 Thread Midas A
Machine configuration RAM: 48 GB CPU: 8 core JVM : 36 GB We are updating 70 , 000 docs / hr . what should be our soft commit and hard commit time to get best results. Current configuration : 6 false 60 There are no read on master server.

Solrcloud (4.10) reports the end of soft commit before all shard replicas finished committing

2015-10-22 Thread vsolakhian
We have a strange behavior of our Sorlcloud related code after upgrading from from Solr 4.4 to Solr 4.10 (as part of upgrading from Cloudera CDH 4.6 to Cloudera CDH 5.4.5). We have a Solrcloud collection with three replicas of one shard. Our code does batch indexing, then submits a soft commit

Re: soft commit through leader

2015-05-19 Thread Erick Erickson
he first commit, i.e. probably within a few milliseconds of each other. This might still be an issue, but the gap isn't that wide. Solr promises _eventual_ consistency If you need to control this, if you issue a soft commit from a client (URL, SolrJ client, curl, etc) then it _is_ distributed

soft commit through leader

2015-05-19 Thread Gopal Jee
hi wanted to know, when we do soft commit through configuration in solrconfig.xml, will different replicas commit at different point of time depending upon when the replica started...or will leader send commit to all replicas at same time as per commit interval set in solrconfig. thanks gopal

Re: Solr tlog and soft commit

2015-03-17 Thread Shalin Shekhar Mangar
message in context: > http://lucene.472066.n3.nabble.com/Solr-tlog-and-soft-commit-tp4193105p4193559.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- Regards, Shalin Shekhar Mangar.

Re: Solr tlog and soft commit

2015-03-17 Thread vidit.asthana
someone explain in what way tlogs gets involved while sending a atomic update request? -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-tlog-and-soft-commit-tp4193105p4193559.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr tlog and soft commit

2015-03-16 Thread vidit.asthana
Can someone please reply to these questions? Thanks in advance. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-tlog-and-soft-commit-tp4193105p4193311.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr tlog and soft commit

2015-03-15 Thread vidit.asthana
nabble.com/Solr-tlog-and-soft-commit-tp4193105p4193129.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr tlog and soft commit

2015-03-15 Thread Erick Erickson
archable not durable. > 4. As soon as I issue a soft commit, the documents inside the latest segment > becomes searchable, but they are still not merged with main index. Hence > they are still not durable. > 5. At this point if a node goes down, the docs will still needs to be > re

Re: Solr tlog and soft commit

2015-03-15 Thread vidit.asthana
hough the index directory size will grow, but the docs are neither searchable not durable. 4. As soon as I issue a soft commit, the documents inside the latest segment becomes searchable, but they are still not merged with main index. Hence they are still not durable. 5. At this point if a node goes

Re: Solr tlog and soft commit

2015-03-15 Thread Yonik Seeley
ing though... if you do a few adds and then do a soft commit, a new small segment will be created and flushed to the Directory, but not fsync'd. But by default, the directory we use is NRTCachingDirectory which caches small segments in memory, so those small segments won't even get writt

Re: Solr tlog and soft commit

2015-03-15 Thread Erick Erickson
date requests are handled and what exactly happens at file system > level. > > 1. So lets say I send an update request, and I don't issue any type of > commit(neither hard nor soft), so will the document ever touch index > directory? From the blog, I understand that it gets written t

Re: Solr tlog and soft commit

2015-03-15 Thread vidit.asthana
touch index directory? From the blog, I understand that it gets written to tlog directory. 2. Now if I issue a soft commit, then what will happen inside the index directory? 3. By the time I don't issue a soft commit, where will that document reside(completely in memory)? -- View this message

Re: Solr tlog and soft commit

2015-03-15 Thread Yonik Seeley
to do with some data being cached in memory on soft commits but always being flushed to disk on hard commits. -Yonik On Sun, Mar 15, 2015 at 9:05 AM, vidit.asthana wrote: > I want to know what all thing gets written to index from tlog directory > whenever a soft commit is issued. > > I

Solr tlog and soft commit

2015-03-15 Thread vidit.asthana
I want to know what all thing gets written to index from tlog directory whenever a soft commit is issued. I have a test SolrCloud setup and I can see that even if I disable the hardcommit, and if I only issue soft commits, then also index directory keeps increasing little by little, so I am

Re: soft commit and deletions

2014-11-26 Thread Erick Erickson
1/26/2014 05:35 PM: >> >> As Shawn says, deletes should be >> visible after a soft commit. >> >> Let's see the code though. If you re-use a searcher that >> you had open before the commit, it'll still see the old >> snapshot of the index including the delet

Re: soft commit and deletions

2014-11-26 Thread Andreas Hubold
); Thanks, Andreas Erick Erickson wrote on 11/26/2014 05:35 PM: As Shawn says, deletes should be visible after a soft commit. Let's see the code though. If you re-use a searcher that you had open before the commit, it'll still see the old snapshot of the index including the deleted doc

Re: soft commit and deletions

2014-11-26 Thread Erick Erickson
As Shawn says, deletes should be visible after a soft commit. Let's see the code though. If you re-use a searcher that you had open before the commit, it'll still see the old snapshot of the index including the deleted documents. Or if you do open a new searcher and any autowarm

Re: soft commit and deletions

2014-11-26 Thread Shawn Heisey
On 11/26/2014 8:18 AM, Andreas Hubold wrote: > But I'm still not totally sure. Does a soft commit also make deleted > documents invisible? > > In a test with an EmbeddedSolrServer I triggered a soft commit and was > still able to find a deleted document afterwards. Is th

soft commit and deletions

2014-11-26 Thread Andreas Hubold
Hi, I've read about soft commits in Erick Erickson's excellent blog article [1]: > The thing to understand most about soft commits are that they will make documents visible But I'm still not totally sure. Does a soft commit also make deleted documents invisible?

Re: Does soft commit block on autowarming?

2014-09-24 Thread Yonik Seeley
On Wed, Sep 24, 2014 at 6:56 PM, Bruce Johnson wrote: > Is it reliably true that once a soft commit request returns, > any subsequent queries will hit a new (and autowarmed) searcher? Yes. The default for commit and softCommit commands is waitSearcher=true, which will not return until

Does soft commit block on autowarming?

2014-09-24 Thread Bruce Johnson
I currently have an algorithm that needs to know whether query results are fresh up to a known point in time, and I'm using an explicit soft commit request to act as a latch point. I record the time T just before I issue a soft commit request, and when it returns, I assume that query re

Re: Delete by query with soft commit

2014-04-12 Thread Furkan KAMACI
Hi Jess; Could you check here first: http://search-lucene.com/m/QTPaSxpsW/Commit+Within+and+%252Fupdate%252Fextract+handler&subj=Re+Commit+Within+and+update+extract+handler and then here: http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Thanks; Fu

Delete by query with soft commit

2014-04-08 Thread youknow...@heroicefforts.net
It appears that UpdateResponse.setCommitWithin is not honored when executing a delete query against SolrCloud (SolrJ 4.6). However, setting the hard commit parameter functions as expected. Is this a known bug? Thanks, -Jess

Re: bulk indexing - EofExceptions and big latencies after soft-commit

2014-03-18 Thread adfel70
lf. It > sounds like a setting for some other piece of software, perhaps a > client, load balancer, or servlet container. > > Thanks, > Shawn -- View this message in context: http://lucene.472066.n3.nabble.com/bulk-indexing-EofExceptions-and-big-latencies-after-soft-commit-tp4124574p4125214.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: bulk indexing - EofExceptions and big latencies after soft-commit

2014-03-17 Thread Shawn Heisey
On 3/17/2014 7:07 AM, adfel70 wrote: > we currently have arround 200gb in a server. > I'm aware of the RAM issue, but it somehow doesnt seems related. > I would expect search latency problems. not strange eofexceptions. > > regarding the http.timeout - I didn't change anything concerning this. > D

Re: bulk indexing - EofExceptions and big latencies after soft-commit

2014-03-17 Thread adfel70
tes. > > How much index data does each server have on it? This would be the sum > total of the index directories of all your cores. > >> I recently started seeing the following problems while indexing - every >> 10 >> minutes ( and I assume that this is the 10

Re: bulk indexing - EofExceptions and big latencies after soft-commit

2014-03-16 Thread Shawn Heisey
ng - every 10 > minutes ( and I assume that this is the 10minutes soft-commit cycles) I get > the following errors: > 1. EofExcpetion from jetty in HttpOutput.write send from SolrDispatchFilter > 2. queries to all cores start getting high latencies (more the 10 seconds) EofException errors

bulk indexing - EofExceptions and big latencies after soft-commit

2014-03-16 Thread adfel70
s and hardCommit openSearcher=false to 15 minutes. I recently started seeing the following problems while indexing - every 10 minutes ( and I assume that this is the 10minutes soft-commit cycles) I get the following errors: 1. EofExcpetion from jetty in HttpOutput.write send from SolrDispatchFil

Re: Very long warmup query vs. frequent soft commit with new searcher

2013-11-16 Thread Shawn Heisey
On 11/15/2013 11:59 PM, Otis Gospodnetic wrote: > What happens when one has a *single* vry long *warming* running > query that takes, say, 10 minutes, and a soft commit that opens a new > searcher happening every 1 minute? > > Could one run into a situation where each soft comm

Very long warmup query vs. frequent soft commit with new searcher

2013-11-15 Thread Otis Gospodnetic
Hi, What happens when one has a *single* vry long *warming* running query that takes, say, 10 minutes, and a soft commit that opens a new searcher happening every 1 minute? Could one run into a situation where each soft commit triggers the same long warming query, thus queueing them one

Re: Optimal interval for soft commit

2013-10-28 Thread Erick Erickson
To reply to your original question, when you soft commit the top-level caches are thrown away. I.e. the filterCache, documentResultCache, all the ones in solrconfig.xml. And if you have a high autowarm count on them, you wind up doing a lot of work for no gain. Say your soft commit interval is 1

Re: Optimal interval for soft commit

2013-10-28 Thread Mugoma Joseph O.
might also be a solution. > > > > - > Thanks, > Michael > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Optimal-interval-for-soft-commit-tp4098016p4098022.html > Sent from the Solr - User mailing list archive at Nabble.com. >

Re: Optimal interval for soft commit

2013-10-28 Thread michael.boom
.472066.n3.nabble.com/Optimal-interval-for-soft-commit-tp4098016p4098022.html Sent from the Solr - User mailing list archive at Nabble.com.

Optimal interval for soft commit

2013-10-28 Thread Mugoma Joseph O.
Hello, We have solr index with about 1m docs. Every day we add 5,000 to 8,000 docs. We have defined 15 sec interval for soft commit. But for the impatient user 15 secs looks like eternity. The wiki http://wiki.apache.org/solr/NearRealtimeSearch advises on 1s soft commit interval but warns &qu

Re: {soft}Commit and cache flusing

2013-10-10 Thread Dmitry Kan
rked me, otherwise I would have moved on. I'll follow this advice. > > Cheers, > > Tim > > > On 9 October 2013 05:20, Erick Erickson wrote: > > > Tim: > > > > I think you're mis-interpreting. By replying to a post with the subject: > > >

Re: {soft}Commit and cache flusing

2013-10-09 Thread Tim Vaillancourt
ng to a post with the subject: > > {soft}Commit and cache flushing > > but going in a different direction, it's easy for people to think "I'm > not interested in that > thread, I'll ignore it", thereby missing the fact that you're asking a > some

Re: {soft}Commit and cache flusing

2013-10-09 Thread Erick Erickson
Tim: I think you're mis-interpreting. By replying to a post with the subject: {soft}Commit and cache flushing but going in a different direction, it's easy for people to think "I'm not interested in that thread, I'll ignore it", thereby missing the fact that you&

Re: {soft}Commit and cache flusing

2013-10-08 Thread Tim Vaillancourt
I have a genuine question with substance here. If anything this nonconstructive, rude response was "to get noticed". Thanks for contributing to the discussion. Tim On 8 October 2013 05:31, Dmitry Kan wrote: > Tim, > I suggest you open a new thread and not reply to this one to get noticed. > Dm

Re: {soft}Commit and cache flusing

2013-10-08 Thread Dmitry Kan
Tim, I suggest you open a new thread and not reply to this one to get noticed. Dmitry On Mon, Oct 7, 2013 at 9:44 PM, Tim Vaillancourt wrote: > Is there a way to make autoCommit only commit if there are pending changes, > ie: if there are 0 adds pending commit, don't autoCommit (open-a-searcher

Re: Soft commit and flush

2013-10-07 Thread Erick Erickson
bq: If so, using soft commit without calling hard commit could cause OOM no. Aside from anything you have configured for auto(hard) commit, the ramBufferSizeMB in solrconfig.xml will flush the in-memory structures out to the segments when the size reaches this limit. It won't _close_ the cu

Re: {soft}Commit and cache flusing

2013-10-07 Thread Tim Vaillancourt
Is there a way to make autoCommit only commit if there are pending changes, ie: if there are 0 adds pending commit, don't autoCommit (open-a-searcher and wipe the caches)? Cheers, Tim On 2 October 2013 00:52, Dmitry Kan wrote: > right. We've got the autoHard commit configured only atm. The so

Re: Soft commit and flush

2013-10-07 Thread Guido Medina
Out of Memory Exception is well known as OOM. Guido. On 07/10/13 14:11, adfel70 wrote: Sorry, by "OOE" I meant Out of memory exception... -- View this message in context: http://lucene.472066.n3.nabble.com/Soft-commit-and-flush-tp4091726p4093902.html Sent from the Solr - User ma

Re: Soft commit and flush

2013-10-07 Thread adfel70
Sorry, by "OOE" I meant Out of memory exception... -- View this message in context: http://lucene.472066.n3.nabble.com/Soft-commit-and-flush-tp4091726p4093902.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Soft commit and flush

2013-10-07 Thread Erick Erickson
bq: Does the NRTCachingDirectoryFactory relevant for both types of commit, or just for hard commit Don't know the code deeply, but NRT==Near Real Time == Soft commit I'd guess. bq: If soft commit does not flush... soft commit flushes the transaction log. On restart if the content o

Re: Soft commit and flush

2013-10-06 Thread adfel70
commit, or just for hard commit? 2. If soft commit does not flush - all data exists in RAM until we call hard commit? If so, using soft commit without calling hard commit could cause OOE ... ? -- View this message in context: http://lucene.472066.n3.nabble.com/Soft-commit-and-flush-tp4091726p4093834

Re: {soft}Commit and cache flusing

2013-10-02 Thread Dmitry Kan
right. We've got the autoHard commit configured only atm. The soft-commits are controlled on the client. It was just easier to implement the first version of our internal commit policy that will commit to all solr instances at once. This is where we have noticed the reported behavior. On Wed, Oct

  1   2   >