A couple of notes: TLOG replicas will have the same issue. When I said that leaders forwarded to followers, what that's really about is that the follower guarantees that the docs have been written to the TLOG. So if you change your model to use TLOG replicas, don't expect a change.
PULL replicas, OTOH, only pull down changed segments, but you're replace the raw document forwarding with segment copying so it's not clear to me how that would change the number of messages flying around. bq. All our updates are single documents. We need to track the availability of online tutors, so we don’t batch them. I'm inferring that this means the updates aren't all that frequent and if you waited for, say, 100 changes you might wait a long time. FYI, here is the result of some experimentation I did for the difference between various batch sizes: https://lucidworks.com/2015/10/05/really-batch-updates-solr-2/, may not apply if your tutors come and go slowly of course. Best, Erick On Tue, Aug 21, 2018 at 8:02 AM, Walter Underwood <wun...@wunderwood.org> wrote: > Thanks, that is exactly what I was curious about. > > All our updates are single documents. We need to track the availability of > online > tutors, so we don’t batch them. > > Right now, we have a replication factor of 36 (way too many), so that means > each > update means 3 x 35 internal communications. Basically, a 100X update > amplification > for our cluster. > > We’ll be reducing the cluster to four hosts as soon as we get out of the > current > blackout on prod changes. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > >> On Aug 20, 2018, at 10:05 PM, Erick Erickson <erickerick...@gmail.com> wrote: >> >> Walter: >> >> Each update is roughly >> >> request goes to leader (may be forwarded) >> >> leader sends the update to _each_ replica. depending on how many docs >> you're sending per update request this may be more than one request. >> IIRC there was some JIRA a while ago where the forwarding wasn't all >> that efficient, but that's going from (shaky) memoryh. >> >> each follower acks back to the leader >> >> leader acks back to the client. >> >> So perhaps you're seeing the individual forwards to followers? Your >> logs should show update requests with FROMLEADER for these >> sub-requests (updates and queries). Does that help? >> >> Erick >> >> >> >> On Mon, Aug 20, 2018 at 8:03 PM, Walter Underwood <wun...@wunderwood.org> >> wrote: >>> I’m comparing request counts from New Relic, which is reporting 16 krpm >>> aggregate >>> requests across the cluster, and the AWS load balancer is reporting 1 krpm. >>> Or it might >>> be 1k requests per 5 minutes because CloudWatch is like that. >>> >>> This is a 36 node cluster, not sharded. We are going to shrink it, but I’d >>> like to understand it. >>> >>> wunder >>> Walter Underwood >>> wun...@wunderwood.org >>> http://observer.wunderwood.org/ (my blog) >>> >>>> On Aug 20, 2018, at 7:02 PM, Shalin Shekhar Mangar >>>> <shalinman...@gmail.com> wrote: >>>> >>>> There are a single persistent HTTP connection open from the leader to each >>>> replica in the shard. All updates coming to the leader are expanded (for >>>> atomic updates) and streamed over that single connection. When using >>>> in-place docvalues updates, there is a possibility of the replica making a >>>> request to the leader if updates has been re-ordered and the replica does >>>> not have enough context to process the update. >>>> >>>> Can you quantify the "tons of internal traffic"? Are you seeing higher >>>> number of open connections as well? >>>> >>>> On Fri, Aug 17, 2018 at 11:17 PM Walter Underwood <wun...@wunderwood.org> >>>> wrote: >>>> >>>>> How many messages are sent back and forth between a leader and replica >>>>> with NRT? >>>>> >>>>> We have a collection that gets frequent updates and we are seeing a ton of >>>>> internal >>>>> cluster traffic. >>>>> >>>>> wunder >>>>> Walter Underwood >>>>> wun...@wunderwood.org >>>>> http://observer.wunderwood.org/ (my blog) >>>>> >>>>> >>>> >>>> -- >>>> Regards, >>>> Shalin Shekhar Mangar. >>> >
