[jira] [Comment Edited] (LUCENE-10448) MergeRateLimiter doesn't always limit instant rate.
[
https://issues.apache.org/jira/browse/LUCENE-10448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509380#comment-17509380
]
Vigya Sharma edited comment on LUCENE-10448 at 3/20/22, 6:59 AM:
-
{color:#172b4d}Thanks for making the changes and confirming, [~kkewwei] !{color}
Re: reordering the lines, I think it is better to check and pause before
actually writing instead of after. Sometimes, the write may happen after a long
time gap. In such cases, MergeRateLimiter does not pause, as the write is
already arriving after a long pause (gap). {_}If we write first and checkRate
later{_}, then we will end up not pausing even for the next subsequent write..
Just my understanding from working on this part of code, and I might have
missed something. Like you, I'm also curious to learn about the design choices
here, from more seasoned Lucene developers.
was (Author: vigyas):
{color:#172b4d}Thanks for making the changes and confirming, [~kkewwei] !{color}
Re: reordering the lines, I think it is better to check and pause before
actually writing instead of after. Sometimes, the write may happen after a long
time gap. In such cases, MergeRateLimiter does not pause, as the write is
already arriving after a long pause (gap). {_}If we write first and checkRate
later{_}, then we will end up not pausing even for the next subsequent write..
Just my understanding from working on this part of code, and I might have
missed something. Like you, I'm also curious to learn about the design choices
here, from more seasoned Lucene developers.
> MergeRateLimiter doesn't always limit instant rate.
> ---
>
> Key: LUCENE-10448
> URL: https://issues.apache.org/jira/browse/LUCENE-10448
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/other
>Affects Versions: 8.11.1
>Reporter: kkewwei
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> We can see the code in *MergeRateLimiter*:
> {code:java}
> private long maybePause(long bytes, long curNS) throws
> MergePolicy.MergeAbortedException {
>
> double rate = mbPerSec;
> double secondsToPause = (bytes / 1024. / 1024.) / rate;
> long targetNS = lastNS + (long) (10 * secondsToPause);
> long curPauseNS = targetNS - curNS;
> // We don't bother with thread pausing if the pause is smaller than 2
> msec.
> if (curPauseNS <= MIN_PAUSE_NS) {
> // Set to curNS, not targetNS, to enforce the instant rate, not
> // the "averaged over all history" rate:
> lastNS = curNS;
> return -1;
> }
>..
> }
> {code}
> If a Segment is been merged, *maybePause* is called in 7:00, lastNS=7:00,
> then the *maybePause* is called in 7:05 again, so the value of
> *targetNS=lastNS + (long) (10 * secondsToPause)* must be smaller than
> *curNS*, no matter how big the bytes is, we will return -1 and ignore to
> pause.
> I count the total times(callTimes) calling *maybePause* and ignored pause
> times(ignorePauseTimes) and detail ignored bytes(detailBytes):
> {code:java}
> [2022-03-02T15:16:51,972][DEBUG][o.e.i.e.I.EngineMergeScheduler] [node1]
> [index1][21] merge segment [_4h] done: took [26.8s], [123.6 MB], [61,219
> docs], [0s stopped], [24.4s throttled], [242.5 MB written], [11.2 MB/sec
> throttle], [callTimes=857], [ignorePauseTimes=25], [detailBytes(mb) =
> [0.28899956, 0.28140354, 0.28015518, 0.27990818, 0.2801447, 0.27991104,
> 0.27990723, 0.27990913, 0.2799101, 0.28010082, 0.2799921, 0.2799673,
> 0.28144264, 0.27991295, 0.27990818, 0.27993107, 0.2799387, 0.27998447,
> 0.28002167, 0.27992058, 0.27998066, 0.28098202, 0.28125, 0.28125, 0.28125]]
> {code}
> There are 857 times calling *maybePause*, including 25 times which is ignored
> to pause, we can see that the ignored detail bytes (such as 0.28125mb) are
> not small.
> As long as the interval between two *maybePause* calls is relatively long,
> the pause action that should be executed will not be executed.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta commented on pull request #755: Add note for smoke tester --tmp-dir option in rc announcing
mocobeta commented on pull request #755: URL: https://github.com/apache/lucene/pull/755#issuecomment-1073452732 @janhoy Would you take a look at this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10470) Unable to Tessellate polygon
[
https://issues.apache.org/jira/browse/LUCENE-10470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509596#comment-17509596
]
Yixun Xu commented on LUCENE-10470:
---
Looks like it's due to the changes to the \{{filterPoints}} logic in [this
PR]([https://github.com/apache/lucene-solr/pull/771/files#diff-d56e36a4e0e7183471b2982cc1127c678f2a414280a21064b3da1e293cead562R920).]
I printed the linked list of vertices before the call to
[earcutLinkedList|[https://github.com/apache/lucene/blob/a4b30b4cf4c0c9b4de0c27893ea2350af498b1c0/lucene/core/src/java/org/apache/lucene/geo/Tessellator.java#L134],]
and the list contains some sets of 3 or more colinear vertices where the
middle vertex was not removed by \{{filterPoints}} due to the check on
\{{isNextEdgeFromPolygon}}. If I filter those middle vertices out then the
tessellation can finish successfully.
I think the colinear vertices were probably introduced when \{{eliminateHoles}}
added edges to connect the holes with the outer polygon, and some added edges
happened to be colinear with an existing edge on the outer polygon (you can see
from the visualization that some vertices on the holes share the same latitude
as vertices on the outer polygon).
I'm not sure what's the best way to fix it yet, will need to look a bit more
into how the isNextEdgeFromPolygon flag is being set and used.
> Unable to Tessellate polygon
>
>
> Key: LUCENE-10470
> URL: https://issues.apache.org/jira/browse/LUCENE-10470
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Affects Versions: 9.0
>Reporter: Yixun Xu
>Priority: Major
> Attachments: image-2022-03-16-18-12-43-411.png
>
>
> I have a polygon that causes {{Tessellator.tessellate}} to throw an "Unable
> to Tessellate shape" error. I tried several versions of Lucene, and the issue
> does not happen with Lucene 8.2.0, but seems to happen with all Lucene
> versions >=8.3.0, including the latest main branch.
> I created a branch that reproduces the issue:
> [https://github.com/apache/lucene/compare/main...yixunx:yx/reproduce-tessellator-error?expand=1]
> This is the polygon rendered on geojson.io:
> !image-2022-03-16-18-12-43-411.png|width=379,height=234!
> Is this a bug in the Tesselator logic, or is there anything wrong with this
> polygon that maybe wasn't caught by Lucene 8.2.0?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-10470) Unable to Tessellate polygon
[
https://issues.apache.org/jira/browse/LUCENE-10470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509596#comment-17509596
]
Yixun Xu edited comment on LUCENE-10470 at 3/21/22, 4:28 AM:
-
Looks like it's due to the changes to the {{filterPoints}} logic in [this
PR|https://github.com/apache/lucene-solr/pull/771/files#diff-d56e36a4e0e7183471b2982cc1127c678f2a414280a21064b3da1e293cead562R920].
I printed the linked list of vertices before the call to
[earcutLinkedList|#L134],] and the list contains some sets of 3 or more
colinear vertices where the middle vertex was not removed by {{filterPoints}}
due to the check on {{{}isNextEdgeFromPolygon{}}}. If I filter those middle
vertices out then the tessellation can finish successfully.
I think the colinear vertices were probably introduced when {{eliminateHoles}}
added edges to connect the holes with the outer polygon, and some added edges
happened to be colinear with an existing edge on the outer polygon (you can see
from the visualization that some vertices on the holes share the same latitude
as vertices on the outer polygon).
I'm not sure what's the best way to fix it yet, will need to look a bit more
into how the isNextEdgeFromPolygon flag is being set and used.
was (Author: JIRAUSER286707):
Looks like it's due to the changes to the \{{filterPoints}} logic in [this
PR]([https://github.com/apache/lucene-solr/pull/771/files#diff-d56e36a4e0e7183471b2982cc1127c678f2a414280a21064b3da1e293cead562R920).]
I printed the linked list of vertices before the call to
[earcutLinkedList|[https://github.com/apache/lucene/blob/a4b30b4cf4c0c9b4de0c27893ea2350af498b1c0/lucene/core/src/java/org/apache/lucene/geo/Tessellator.java#L134],]
and the list contains some sets of 3 or more colinear vertices where the
middle vertex was not removed by \{{filterPoints}} due to the check on
\{{isNextEdgeFromPolygon}}. If I filter those middle vertices out then the
tessellation can finish successfully.
I think the colinear vertices were probably introduced when \{{eliminateHoles}}
added edges to connect the holes with the outer polygon, and some added edges
happened to be colinear with an existing edge on the outer polygon (you can see
from the visualization that some vertices on the holes share the same latitude
as vertices on the outer polygon).
I'm not sure what's the best way to fix it yet, will need to look a bit more
into how the isNextEdgeFromPolygon flag is being set and used.
> Unable to Tessellate polygon
>
>
> Key: LUCENE-10470
> URL: https://issues.apache.org/jira/browse/LUCENE-10470
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Affects Versions: 9.0
>Reporter: Yixun Xu
>Priority: Major
> Attachments: image-2022-03-16-18-12-43-411.png
>
>
> I have a polygon that causes {{Tessellator.tessellate}} to throw an "Unable
> to Tessellate shape" error. I tried several versions of Lucene, and the issue
> does not happen with Lucene 8.2.0, but seems to happen with all Lucene
> versions >=8.3.0, including the latest main branch.
> I created a branch that reproduces the issue:
> [https://github.com/apache/lucene/compare/main...yixunx:yx/reproduce-tessellator-error?expand=1]
> This is the polygon rendered on geojson.io:
> !image-2022-03-16-18-12-43-411.png|width=379,height=234!
> Is this a bug in the Tesselator logic, or is there anything wrong with this
> polygon that maybe wasn't caught by Lucene 8.2.0?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-10470) Unable to Tessellate polygon
[
https://issues.apache.org/jira/browse/LUCENE-10470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509596#comment-17509596
]
Yixun Xu edited comment on LUCENE-10470 at 3/21/22, 4:28 AM:
-
Looks like it's due to the changes to the {{filterPoints}} logic in [this
PR|https://github.com/apache/lucene-solr/pull/771/files#diff-d56e36a4e0e7183471b2982cc1127c678f2a414280a21064b3da1e293cead562R920].
I printed the linked list of vertices before the call to
[earcutLinkedList|#L134], and the list contains some sets of 3 or more colinear
vertices where the middle vertex was not removed by {{filterPoints}} due to the
check on {{{}isNextEdgeFromPolygon{}}}. If I filter those middle vertices out
then the tessellation can finish successfully.
I think the colinear vertices were probably introduced when {{eliminateHoles}}
added edges to connect the holes with the outer polygon, and some added edges
happened to be colinear with an existing edge on the outer polygon (you can see
from the visualization that some vertices on the holes share the same latitude
as vertices on the outer polygon).
I'm not sure what's the best way to fix it yet, will need to look a bit more
into how the isNextEdgeFromPolygon flag is being set and used.
was (Author: JIRAUSER286707):
Looks like it's due to the changes to the {{filterPoints}} logic in [this
PR|https://github.com/apache/lucene-solr/pull/771/files#diff-d56e36a4e0e7183471b2982cc1127c678f2a414280a21064b3da1e293cead562R920].
I printed the linked list of vertices before the call to
[earcutLinkedList|#L134],] and the list contains some sets of 3 or more
colinear vertices where the middle vertex was not removed by {{filterPoints}}
due to the check on {{{}isNextEdgeFromPolygon{}}}. If I filter those middle
vertices out then the tessellation can finish successfully.
I think the colinear vertices were probably introduced when {{eliminateHoles}}
added edges to connect the holes with the outer polygon, and some added edges
happened to be colinear with an existing edge on the outer polygon (you can see
from the visualization that some vertices on the holes share the same latitude
as vertices on the outer polygon).
I'm not sure what's the best way to fix it yet, will need to look a bit more
into how the isNextEdgeFromPolygon flag is being set and used.
> Unable to Tessellate polygon
>
>
> Key: LUCENE-10470
> URL: https://issues.apache.org/jira/browse/LUCENE-10470
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Affects Versions: 9.0
>Reporter: Yixun Xu
>Priority: Major
> Attachments: image-2022-03-16-18-12-43-411.png
>
>
> I have a polygon that causes {{Tessellator.tessellate}} to throw an "Unable
> to Tessellate shape" error. I tried several versions of Lucene, and the issue
> does not happen with Lucene 8.2.0, but seems to happen with all Lucene
> versions >=8.3.0, including the latest main branch.
> I created a branch that reproduces the issue:
> [https://github.com/apache/lucene/compare/main...yixunx:yx/reproduce-tessellator-error?expand=1]
> This is the polygon rendered on geojson.io:
> !image-2022-03-16-18-12-43-411.png|width=379,height=234!
> Is this a bug in the Tesselator logic, or is there anything wrong with this
> polygon that maybe wasn't caught by Lucene 8.2.0?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-10470) Unable to Tessellate polygon
[
https://issues.apache.org/jira/browse/LUCENE-10470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509596#comment-17509596
]
Yixun Xu edited comment on LUCENE-10470 at 3/21/22, 4:30 AM:
-
Looks like it's due to the changes to the {{filterPoints}} logic in [this
PR|https://github.com/apache/lucene-solr/pull/771/files#diff-d56e36a4e0e7183471b2982cc1127c678f2a414280a21064b3da1e293cead562R920].
I printed the linked list of vertices before the call to
[earcutLinkedList|#L134], and the list contains some sets of 3 or more colinear
vertices where the middle vertex was not removed by {{filterPoints}} due to the
check on {{{}isNextEdgeFromPolygon{}}}. If I filter those middle vertices out
then the tessellation can finish successfully.
I think the colinear vertices were probably introduced when {{eliminateHoles}}
added edges to connect the holes with the outer polygon, and some added edges
happened to be colinear with an existing edge on the outer polygon (you can see
from the visualization that some vertices on the holes share the same latitude
as vertices on the outer polygon).
I'm not sure what's the best way to fix it yet, will need to look a bit more
into how the isNextEdgeFromPolygon flag is being set and used.
The printed lists of vertices, from the latest Lucene and Lucene 8.2.0:
[^vertices-latest-lucene.txt]
[^vertices-lucene-820.txt]
was (Author: JIRAUSER286707):
Looks like it's due to the changes to the {{filterPoints}} logic in [this
PR|https://github.com/apache/lucene-solr/pull/771/files#diff-d56e36a4e0e7183471b2982cc1127c678f2a414280a21064b3da1e293cead562R920].
I printed the linked list of vertices before the call to
[earcutLinkedList|#L134], and the list contains some sets of 3 or more colinear
vertices where the middle vertex was not removed by {{filterPoints}} due to the
check on {{{}isNextEdgeFromPolygon{}}}. If I filter those middle vertices out
then the tessellation can finish successfully.
I think the colinear vertices were probably introduced when {{eliminateHoles}}
added edges to connect the holes with the outer polygon, and some added edges
happened to be colinear with an existing edge on the outer polygon (you can see
from the visualization that some vertices on the holes share the same latitude
as vertices on the outer polygon).
I'm not sure what's the best way to fix it yet, will need to look a bit more
into how the isNextEdgeFromPolygon flag is being set and used.
> Unable to Tessellate polygon
>
>
> Key: LUCENE-10470
> URL: https://issues.apache.org/jira/browse/LUCENE-10470
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Affects Versions: 9.0
>Reporter: Yixun Xu
>Priority: Major
> Attachments: image-2022-03-16-18-12-43-411.png,
> vertices-latest-lucene.txt, vertices-lucene-820.txt
>
>
> I have a polygon that causes {{Tessellator.tessellate}} to throw an "Unable
> to Tessellate shape" error. I tried several versions of Lucene, and the issue
> does not happen with Lucene 8.2.0, but seems to happen with all Lucene
> versions >=8.3.0, including the latest main branch.
> I created a branch that reproduces the issue:
> [https://github.com/apache/lucene/compare/main...yixunx:yx/reproduce-tessellator-error?expand=1]
> This is the polygon rendered on geojson.io:
> !image-2022-03-16-18-12-43-411.png|width=379,height=234!
> Is this a bug in the Tesselator logic, or is there anything wrong with this
> polygon that maybe wasn't caught by Lucene 8.2.0?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10448) MergeRateLimiter doesn't always limit instant rate.
[
https://issues.apache.org/jira/browse/LUCENE-10448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509637#comment-17509637
]
kkewwei commented on LUCENE-10448:
--
{quote}
I think it is better to check and pause before actually writing instead of
after
{quote}
In such cases, the code should be:
{code:java}
@Override
public void writeBytes(byte[] b, int offset, int length) throws IOException {
checkRate();
bytesSinceLastPause += length;
delegate.writeBytes(b, offset, length);
}
{code}
instead of current way, maybe it's a minor change and not worth change it.
> MergeRateLimiter doesn't always limit instant rate.
> ---
>
> Key: LUCENE-10448
> URL: https://issues.apache.org/jira/browse/LUCENE-10448
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/other
>Affects Versions: 8.11.1
>Reporter: kkewwei
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> We can see the code in *MergeRateLimiter*:
> {code:java}
> private long maybePause(long bytes, long curNS) throws
> MergePolicy.MergeAbortedException {
>
> double rate = mbPerSec;
> double secondsToPause = (bytes / 1024. / 1024.) / rate;
> long targetNS = lastNS + (long) (10 * secondsToPause);
> long curPauseNS = targetNS - curNS;
> // We don't bother with thread pausing if the pause is smaller than 2
> msec.
> if (curPauseNS <= MIN_PAUSE_NS) {
> // Set to curNS, not targetNS, to enforce the instant rate, not
> // the "averaged over all history" rate:
> lastNS = curNS;
> return -1;
> }
>..
> }
> {code}
> If a Segment is been merged, *maybePause* is called in 7:00, lastNS=7:00,
> then the *maybePause* is called in 7:05 again, so the value of
> *targetNS=lastNS + (long) (10 * secondsToPause)* must be smaller than
> *curNS*, no matter how big the bytes is, we will return -1 and ignore to
> pause.
> I count the total times(callTimes) calling *maybePause* and ignored pause
> times(ignorePauseTimes) and detail ignored bytes(detailBytes):
> {code:java}
> [2022-03-02T15:16:51,972][DEBUG][o.e.i.e.I.EngineMergeScheduler] [node1]
> [index1][21] merge segment [_4h] done: took [26.8s], [123.6 MB], [61,219
> docs], [0s stopped], [24.4s throttled], [242.5 MB written], [11.2 MB/sec
> throttle], [callTimes=857], [ignorePauseTimes=25], [detailBytes(mb) =
> [0.28899956, 0.28140354, 0.28015518, 0.27990818, 0.2801447, 0.27991104,
> 0.27990723, 0.27990913, 0.2799101, 0.28010082, 0.2799921, 0.2799673,
> 0.28144264, 0.27991295, 0.27990818, 0.27993107, 0.2799387, 0.27998447,
> 0.28002167, 0.27992058, 0.27998066, 0.28098202, 0.28125, 0.28125, 0.28125]]
> {code}
> There are 857 times calling *maybePause*, including 25 times which is ignored
> to pause, we can see that the ignored detail bytes (such as 0.28125mb) are
> not small.
> As long as the interval between two *maybePause* calls is relatively long,
> the pause action that should be executed will not be executed.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
