[ 
https://issues.apache.org/jira/browse/LUCENE-10448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507884#comment-17507884
 ] 

Vigya Sharma commented on LUCENE-10448:
---------------------------------------

[~jpountz] - we could handle the edge case of when a large byte buffer arrives 
for writes in {{{}writeBytes(){}}}. I made a small change ( 
[PR#738|https://github.com/apache/lucene/pull/738] ) to write it in chunks and 
check for pauses in between..

 

[~kkewwei] - When you do - {{rateLimiter.pause(bytesSinceLastPause, 
System.nanoTime() - writeStartingTime);}} - I think you end up excluding 
pauseTime from your write rate calculations (since writeStartingTime begins 
after we are out of the pause). We need to include the pause time, otherwise 
the write rate will always be high. After all, it's only via these pauses that 
we are able to control it.

I suggest you reset your writeStartTime, right before you call 
{{currentMinPauseCheckBytes = rateLimiter.getMinPauseCheckBytes();}}. Then, we 
would've included pause time in the writeTimeElapsed, and if rate is still 
higher than threshold, we'll know that rateLimiter did not give us an adequate 
pause. From code, I don't see this happening unless writeBytes is called.

> MergeRateLimiter doesn't always limit instant rate.
> ---------------------------------------------------
>
>                 Key: LUCENE-10448
>                 URL: https://issues.apache.org/jira/browse/LUCENE-10448
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/other
>    Affects Versions: 8.11.1
>            Reporter: kkewwei
>            Priority: Major
>          Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> We can see the code in *MergeRateLimiter*:
> {code:java}
> private long maybePause(long bytes, long curNS) throws 
> MergePolicy.MergeAbortedException {
>    
>     double rate = mbPerSec; 
>     double secondsToPause = (bytes / 1024. / 1024.) / rate;
>     long targetNS = lastNS + (long) (1000000000 * secondsToPause);
>     long curPauseNS = targetNS - curNS;
>     // We don't bother with thread pausing if the pause is smaller than 2 
> msec.
>     if (curPauseNS <= MIN_PAUSE_NS) {
>       // Set to curNS, not targetNS, to enforce the instant rate, not
>       // the "averaged over all history" rate:
>       lastNS = curNS;
>       return -1;
>     }
>    ......
>   }
> {code}
> If a Segment is been merged, *maybePause* is called in 7:00, lastNS=7:00, 
> then the *maybePause* is called in 7:05 again,  so the value of 
> *targetNS=lastNS + (long) (1000000000 * secondsToPause)* must be smaller than 
> *curNS*, no matter how big the bytes is, we will return -1 and ignore to 
> pause. 
> I count the total times(callTimes) calling *maybePause* and ignored pause 
> times(ignorePauseTimes) and detail ignored bytes(detailBytes):
> {code:java}
> [2022-03-02T15:16:51,972][DEBUG][o.e.i.e.I.EngineMergeScheduler] [node1] 
> [index1][21] merge segment [_4h] done: took [26.8s], [123.6 MB], [61,219 
> docs], [0s stopped], [24.4s throttled], [242.5 MB written], [11.2 MB/sec 
> throttle], [callTimes=857], [ignorePauseTimes=25],  [detailBytes(mb) = 
> [0.28899956, 0.28140354, 0.28015518, 0.27990818, 0.2801447, 0.27991104, 
> 0.27990723, 0.27990913, 0.2799101, 0.28010082, 0.2799921, 0.2799673, 
> 0.28144264, 0.27991295, 0.27990818, 0.27993107, 0.2799387, 0.27998447, 
> 0.28002167, 0.27992058, 0.27998066, 0.28098202, 0.28125, 0.28125, 0.28125]]
> {code}
> There are 857 times calling *maybePause*, including 25 times which is ignored 
> to pause, we can see that the ignored detail bytes (such as 0.28125mb) are 
> not small.
> As long as the interval between two *maybePause* calls is relatively long, 
> the pause action that should be executed will not be executed.
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

Reply via email to