[jira] [Commented] (LUCENE-10184) Move "how to contribute" documentation to the git repo
[ https://issues.apache.org/jira/browse/LUCENE-10184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540366#comment-17540366 ] ASF subversion and git services commented on LUCENE-10184: -- Commit 10a43d916e7d282cf98f35305c411fc9060004ab in lucene's branch refs/heads/main from Tomoko Uchida [ https://gitbox.apache.org/repos/asf?p=lucene.git;h=10a43d916e7 ] LUCENE-10184: avoid smoke tester failure > Move "how to contribute" documentation to the git repo > -- > > Key: LUCENE-10184 > URL: https://issues.apache.org/jira/browse/LUCENE-10184 > Project: Lucene - Core > Issue Type: Task > Components: general/website >Reporter: Tomoko Uchida >Priority: Trivial > Time Spent: 2h 40m > Remaining Estimate: 0h > > This "HowToContrribute" guide in Confluence is well organized and helpful > (especially for new devs who don't belong to organizations having rich > practice to contribute to Lucene). > [https://cwiki.apache.org/confluence/display/lucene/HowToContribute] > I think it would be great if we have this in the top-level folder as > "CONTRIBUTING.md" (the file name and place is a common practice on GitHub), > so that it is more reachable from outside developers. > https://github.com/github/docs/blob/main/CONTRIBUTING.md > By having this as Markdown format, it would also be possible to integrate the > guide into our general documentation with flexmark. > [https://github.com/apache/lucene/blob/main/gradle/documentation/markdown.gradle] > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on a diff in pull request #912: Initial rewrite of MMapDirectory for JDK-19 preview Panama APIs (>= JDK-19-ea+23)
uschindler commented on code in PR #912:
URL: https://github.com/apache/lucene/pull/912#discussion_r878662352
##
lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java:
##
@@ -232,55 +224,62 @@ public IndexInput openInput(String name, IOContext
context) throws IOException {
ensureOpen();
ensureCanRead(name);
Path path = directory.resolve(name);
-try (FileChannel c = FileChannel.open(path, StandardOpenOption.READ)) {
- final String resourceDescription = "MMapIndexInput(path=\"" +
path.toString() + "\")";
- final boolean useUnmap = getUseUnmap();
- return ByteBufferIndexInput.newInstance(
- resourceDescription,
- map(resourceDescription, c, 0, c.size()),
- c.size(),
- chunkSizePower,
- new ByteBufferGuard(resourceDescription, useUnmap ? CLEANER : null));
+final String resourceDescription = "MemorySegmentIndexInput(path=\"" +
path.toString() + "\")";
+
+// Work around for JDK-8259028: we need to unwrap our test-only file
system layers
+path = Unwrappable.unwrapAll(path);
+
+boolean success = false;
+final MemorySession session = MemorySession.openShared();
+try (var fc = FileChannel.open(path)) {
+ final long fileSize = fc.size();
+ final MemorySegment[] segments = map(session, resourceDescription, fc,
fileSize);
+ final IndexInput in =
+ MemorySegmentIndexInput.newInstance(
+ resourceDescription, session, segments, fileSize,
chunkSizePower);
+ success = true;
+ return in;
+} finally {
+ if (success == false) {
+session.close();
+ }
}
}
- /** Maps a file into a set of buffers */
- final ByteBuffer[] map(String resourceDescription, FileChannel fc, long
offset, long length)
+ /** Maps a file into a set of segments */
+ final MemorySegment[] map(
+ MemorySession session, String resourceDescription, FileChannel fc, long
length)
throws IOException {
if ((length >>> chunkSizePower) >= Integer.MAX_VALUE)
- throw new IllegalArgumentException(
- "RandomAccessFile too big for chunk size: " + resourceDescription);
+ throw new IllegalArgumentException("File too big for chunk size: " +
resourceDescription);
final long chunkSize = 1L << chunkSizePower;
-// we always allocate one more buffer, the last one may be a 0 byte one
-final int nrBuffers = (int) (length >>> chunkSizePower) + 1;
+// we always allocate one more segments, the last one may be a 0 byte one
+final int nrSegments = (int) (length >>> chunkSizePower) + 1;
-ByteBuffer[] buffers = new ByteBuffer[nrBuffers];
+final MemorySegment[] segments = new MemorySegment[nrSegments];
-long bufferStart = 0L;
-for (int bufNr = 0; bufNr < nrBuffers; bufNr++) {
- int bufSize =
- (int) ((length > (bufferStart + chunkSize)) ? chunkSize : (length -
bufferStart));
- MappedByteBuffer buffer;
+long startOffset = 0L;
+for (int segNr = 0; segNr < nrSegments; segNr++) {
+ long segSize = (length > (startOffset + chunkSize)) ? chunkSize :
(length - startOffset);
Review Comment:
I just rrwrote the old code. Yes could be final. We could also use `final
var` which looks like bullshit :-)
Previously it was `int` because it was about bytebufers, which are limited
to 2 Gigabytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta commented on a diff in pull request #912: Initial rewrite of MMapDirectory for JDK-19 preview Panama APIs (>= JDK-19-ea+23)
mocobeta commented on code in PR #912:
URL: https://github.com/apache/lucene/pull/912#discussion_r878663653
##
lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java:
##
@@ -232,55 +224,62 @@ public IndexInput openInput(String name, IOContext
context) throws IOException {
ensureOpen();
ensureCanRead(name);
Path path = directory.resolve(name);
-try (FileChannel c = FileChannel.open(path, StandardOpenOption.READ)) {
- final String resourceDescription = "MMapIndexInput(path=\"" +
path.toString() + "\")";
- final boolean useUnmap = getUseUnmap();
- return ByteBufferIndexInput.newInstance(
- resourceDescription,
- map(resourceDescription, c, 0, c.size()),
- c.size(),
- chunkSizePower,
- new ByteBufferGuard(resourceDescription, useUnmap ? CLEANER : null));
+final String resourceDescription = "MemorySegmentIndexInput(path=\"" +
path.toString() + "\")";
+
+// Work around for JDK-8259028: we need to unwrap our test-only file
system layers
+path = Unwrappable.unwrapAll(path);
+
+boolean success = false;
+final MemorySession session = MemorySession.openShared();
+try (var fc = FileChannel.open(path)) {
+ final long fileSize = fc.size();
+ final MemorySegment[] segments = map(session, resourceDescription, fc,
fileSize);
+ final IndexInput in =
+ MemorySegmentIndexInput.newInstance(
+ resourceDescription, session, segments, fileSize,
chunkSizePower);
+ success = true;
+ return in;
+} finally {
+ if (success == false) {
+session.close();
+ }
}
}
- /** Maps a file into a set of buffers */
- final ByteBuffer[] map(String resourceDescription, FileChannel fc, long
offset, long length)
+ /** Maps a file into a set of segments */
+ final MemorySegment[] map(
+ MemorySession session, String resourceDescription, FileChannel fc, long
length)
throws IOException {
if ((length >>> chunkSizePower) >= Integer.MAX_VALUE)
- throw new IllegalArgumentException(
- "RandomAccessFile too big for chunk size: " + resourceDescription);
+ throw new IllegalArgumentException("File too big for chunk size: " +
resourceDescription);
final long chunkSize = 1L << chunkSizePower;
-// we always allocate one more buffer, the last one may be a 0 byte one
-final int nrBuffers = (int) (length >>> chunkSizePower) + 1;
+// we always allocate one more segments, the last one may be a 0 byte one
+final int nrSegments = (int) (length >>> chunkSizePower) + 1;
-ByteBuffer[] buffers = new ByteBuffer[nrBuffers];
+final MemorySegment[] segments = new MemorySegment[nrSegments];
-long bufferStart = 0L;
-for (int bufNr = 0; bufNr < nrBuffers; bufNr++) {
- int bufSize =
- (int) ((length > (bufferStart + chunkSize)) ? chunkSize : (length -
bufferStart));
- MappedByteBuffer buffer;
+long startOffset = 0L;
+for (int segNr = 0; segNr < nrSegments; segNr++) {
+ long segSize = (length > (startOffset + chunkSize)) ? chunkSize :
(length - startOffset);
Review Comment:
thanks, I surely missed the description.
> ByteBuffer API is limited to 32bit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta commented on a diff in pull request #912: Initial rewrite of MMapDirectory for JDK-19 preview Panama APIs (>= JDK-19-ea+23)
mocobeta commented on code in PR #912:
URL: https://github.com/apache/lucene/pull/912#discussion_r87874
##
lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java:
##
@@ -232,55 +224,62 @@ public IndexInput openInput(String name, IOContext
context) throws IOException {
ensureOpen();
ensureCanRead(name);
Path path = directory.resolve(name);
-try (FileChannel c = FileChannel.open(path, StandardOpenOption.READ)) {
- final String resourceDescription = "MMapIndexInput(path=\"" +
path.toString() + "\")";
- final boolean useUnmap = getUseUnmap();
- return ByteBufferIndexInput.newInstance(
- resourceDescription,
- map(resourceDescription, c, 0, c.size()),
- c.size(),
- chunkSizePower,
- new ByteBufferGuard(resourceDescription, useUnmap ? CLEANER : null));
+final String resourceDescription = "MemorySegmentIndexInput(path=\"" +
path.toString() + "\")";
+
+// Work around for JDK-8259028: we need to unwrap our test-only file
system layers
+path = Unwrappable.unwrapAll(path);
+
+boolean success = false;
+final MemorySession session = MemorySession.openShared();
+try (var fc = FileChannel.open(path)) {
+ final long fileSize = fc.size();
+ final MemorySegment[] segments = map(session, resourceDescription, fc,
fileSize);
+ final IndexInput in =
+ MemorySegmentIndexInput.newInstance(
+ resourceDescription, session, segments, fileSize,
chunkSizePower);
+ success = true;
+ return in;
+} finally {
+ if (success == false) {
+session.close();
+ }
}
}
- /** Maps a file into a set of buffers */
- final ByteBuffer[] map(String resourceDescription, FileChannel fc, long
offset, long length)
+ /** Maps a file into a set of segments */
+ final MemorySegment[] map(
+ MemorySession session, String resourceDescription, FileChannel fc, long
length)
throws IOException {
if ((length >>> chunkSizePower) >= Integer.MAX_VALUE)
- throw new IllegalArgumentException(
- "RandomAccessFile too big for chunk size: " + resourceDescription);
+ throw new IllegalArgumentException("File too big for chunk size: " +
resourceDescription);
final long chunkSize = 1L << chunkSizePower;
-// we always allocate one more buffer, the last one may be a 0 byte one
-final int nrBuffers = (int) (length >>> chunkSizePower) + 1;
+// we always allocate one more segments, the last one may be a 0 byte one
+final int nrSegments = (int) (length >>> chunkSizePower) + 1;
-ByteBuffer[] buffers = new ByteBuffer[nrBuffers];
+final MemorySegment[] segments = new MemorySegment[nrSegments];
-long bufferStart = 0L;
-for (int bufNr = 0; bufNr < nrBuffers; bufNr++) {
- int bufSize =
- (int) ((length > (bufferStart + chunkSize)) ? chunkSize : (length -
bufferStart));
- MappedByteBuffer buffer;
+long startOffset = 0L;
+for (int segNr = 0; segNr < nrSegments; segNr++) {
+ long segSize = (length > (startOffset + chunkSize)) ? chunkSize :
(length - startOffset);
Review Comment:
I'm resolving this - it's a local variable in the loop with a smaller scope,
I just noticed that it looks that other parts of this patch try to use final as
far as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] vigyasharma commented on a diff in pull request #892: LUCENE-10581: Optimize stored fields bulk merges on the first segment
vigyasharma commented on code in PR #892:
URL: https://github.com/apache/lucene/pull/892#discussion_r878666817
##
lucene/core/src/java/org/apache/lucene/codecs/lucene90/compressing/Lucene90CompressingStoredFieldsWriter.java:
##
@@ -553,14 +554,20 @@ private void copyChunks(
final long toPointer =
toDocID == sub.maxDoc ? reader.getMaxPointer() :
index.getStartPointer(toDocID);
if (fromPointer < toPointer) {
- if (numBufferedDocs > 0) {
-flush(true);
- }
final IndexInput rawDocs = reader.getFieldsStream();
rawDocs.seek(fromPointer);
do {
final int base = rawDocs.readVInt();
final int code = rawDocs.readVInt();
+final boolean dirtyChunk = (code & 2) != 0;
+if (copyDirtyChunks) {
+ if (numBufferedDocs > 0) {
+flush(true);
+ }
+} else if (dirtyChunk || numBufferedDocs > 0) {
+ // Don't copy a dirty chunk or force a flush, which would create a
dirty chunk
+ break;
+}
Review Comment:
Is it that if buffered documents are present, it would create a flush, which
would create a dirty chunk? And so, in a `CLEAN_BULK`, we need to break out of
the loop and copy docs one at a time?
How do you feel about rewriting this if-else as something like:
```java
if (sub.mergeStrategy == MergeStrategy.CLEAN_BULK
&& (dirtyChunk || numBufferedDocs > 0)) {
break;
}
// and then proceed with regular chunk copy code...
...
if (numBufferedDocs > 0) {
flush(true);
}
...
```
Essentially, if we wanted a `CLEAN_BULK`, we break out at the first
possibility of dirty chunks - i.e. when we actually see a dirty chunk, or if
there are buffered docs present.
Did I miss any conditions here? If not, do you think this would make it
easier on the eyes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta commented on a diff in pull request #912: Initial rewrite of MMapDirectory for JDK-19 preview Panama APIs (>= JDK-19-ea+23)
mocobeta commented on code in PR #912:
URL: https://github.com/apache/lucene/pull/912#discussion_r87874
##
lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java:
##
@@ -232,55 +224,62 @@ public IndexInput openInput(String name, IOContext
context) throws IOException {
ensureOpen();
ensureCanRead(name);
Path path = directory.resolve(name);
-try (FileChannel c = FileChannel.open(path, StandardOpenOption.READ)) {
- final String resourceDescription = "MMapIndexInput(path=\"" +
path.toString() + "\")";
- final boolean useUnmap = getUseUnmap();
- return ByteBufferIndexInput.newInstance(
- resourceDescription,
- map(resourceDescription, c, 0, c.size()),
- c.size(),
- chunkSizePower,
- new ByteBufferGuard(resourceDescription, useUnmap ? CLEANER : null));
+final String resourceDescription = "MemorySegmentIndexInput(path=\"" +
path.toString() + "\")";
+
+// Work around for JDK-8259028: we need to unwrap our test-only file
system layers
+path = Unwrappable.unwrapAll(path);
+
+boolean success = false;
+final MemorySession session = MemorySession.openShared();
+try (var fc = FileChannel.open(path)) {
+ final long fileSize = fc.size();
+ final MemorySegment[] segments = map(session, resourceDescription, fc,
fileSize);
+ final IndexInput in =
+ MemorySegmentIndexInput.newInstance(
+ resourceDescription, session, segments, fileSize,
chunkSizePower);
+ success = true;
+ return in;
+} finally {
+ if (success == false) {
+session.close();
+ }
}
}
- /** Maps a file into a set of buffers */
- final ByteBuffer[] map(String resourceDescription, FileChannel fc, long
offset, long length)
+ /** Maps a file into a set of segments */
+ final MemorySegment[] map(
+ MemorySession session, String resourceDescription, FileChannel fc, long
length)
throws IOException {
if ((length >>> chunkSizePower) >= Integer.MAX_VALUE)
- throw new IllegalArgumentException(
- "RandomAccessFile too big for chunk size: " + resourceDescription);
+ throw new IllegalArgumentException("File too big for chunk size: " +
resourceDescription);
final long chunkSize = 1L << chunkSizePower;
-// we always allocate one more buffer, the last one may be a 0 byte one
-final int nrBuffers = (int) (length >>> chunkSizePower) + 1;
+// we always allocate one more segments, the last one may be a 0 byte one
+final int nrSegments = (int) (length >>> chunkSizePower) + 1;
-ByteBuffer[] buffers = new ByteBuffer[nrBuffers];
+final MemorySegment[] segments = new MemorySegment[nrSegments];
-long bufferStart = 0L;
-for (int bufNr = 0; bufNr < nrBuffers; bufNr++) {
- int bufSize =
- (int) ((length > (bufferStart + chunkSize)) ? chunkSize : (length -
bufferStart));
- MappedByteBuffer buffer;
+long startOffset = 0L;
+for (int segNr = 0; segNr < nrSegments; segNr++) {
+ long segSize = (length > (startOffset + chunkSize)) ? chunkSize :
(length - startOffset);
Review Comment:
I'm resolving this - it's a local variable in the loop with a smaller scope,
I just noticed that it looks like that other parts of this patch try to use
final as far as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] msokolov commented on pull request #913: Lucene 10577
msokolov commented on PR #913: URL: https://github.com/apache/lucene/pull/913#issuecomment-1133605270 > It's not a part of the mandatory test suite but you can run it headlessly with a virtual display (if you are on Linux). Please see: Thanks, I guess it runs as part of `check` tasks? Anyway after installing xvfb I get a pass; thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] msokolov commented on a diff in pull request #913: Lucene 10577
msokolov commented on code in PR #913:
URL: https://github.com/apache/lucene/pull/913#discussion_r878686983
##
lucene/core/src/java/org/apache/lucene/util/VectorUtil.java:
##
@@ -213,4 +213,21 @@ public static void add(float[] u, float[] v) {
u[i] += v[i];
}
}
+
+ public static float dotProduct(BytesRef a, int aOffset, BytesRef b, int
bOffset, int len) {
+// fixme -- move to codec? What if later we want to access the bytes some
other way?
+int total = 0;
+for (int i = 0; i < len; i++) {
+ total += a.bytes[aOffset++] * b.bytes[bOffset++];
Review Comment:
> i wonder if we can coerce autovectorization of this loop somehow too.
That would be cool. Continuing down the Vector track, I ran some JMH
benchmarks. I don't want to get too deep into it here, but it does seem that
`Vector.castShape()` followed by `reduceLanesToLong` produces the
correct result, is quite a bit faster than the equivalent operation over
`FloatVector`, and is nearly as fast as adding up and accumulating Bytes (which
produces the wrong result).`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] msokolov commented on a diff in pull request #913: Lucene 10577
msokolov commented on code in PR #913:
URL: https://github.com/apache/lucene/pull/913#discussion_r878686983
##
lucene/core/src/java/org/apache/lucene/util/VectorUtil.java:
##
@@ -213,4 +213,21 @@ public static void add(float[] u, float[] v) {
u[i] += v[i];
}
}
+
+ public static float dotProduct(BytesRef a, int aOffset, BytesRef b, int
bOffset, int len) {
+// fixme -- move to codec? What if later we want to access the bytes some
other way?
+int total = 0;
+for (int i = 0; i < len; i++) {
+ total += a.bytes[aOffset++] * b.bytes[bOffset++];
Review Comment:
> i wonder if we can coerce autovectorization of this loop somehow too.
That would be cool. Continuing down the Vector track, I ran some JMH
benchmarks. I don't want to get too deep into it here, but it does seem that
`Vector.castShape()` followed by `reduceLanesToLong` produces the
correct result, is quite a bit faster than the equivalent operation over
`FloatVector`, and is nearly as fast as adding up and accumulating Bytes (which
produces the wrong result).`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta commented on pull request #917: LUCENE-10531: Disable distribution test (gui test) on windows.
mocobeta commented on PR #917: URL: https://github.com/apache/lucene/pull/917#issuecomment-1133646096 I would merge this now not to fail the CI in unrelated pull requests; we could re-enable it on windows vm in the future if the GH actions runner is improved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta merged pull request #917: LUCENE-10531: Disable distribution test (gui test) on windows.
mocobeta merged PR #917: URL: https://github.com/apache/lucene/pull/917 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10531) Mark testLukeCanBeLaunched @Nightly test and make a dedicated Github CI workflow for it
[
https://issues.apache.org/jira/browse/LUCENE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540436#comment-17540436
]
ASF subversion and git services commented on LUCENE-10531:
--
Commit 59b6d41bd10dcf80959cd9d40cd042287b3801ab in lucene's branch
refs/heads/main from Tomoko Uchida
[ https://gitbox.apache.org/repos/asf?p=lucene.git;h=59b6d41bd10 ]
LUCENE-10531: Disable distribution test (gui test) on windows. (#917)
> Mark testLukeCanBeLaunched @Nightly test and make a dedicated Github CI
> workflow for it
> ---
>
> Key: LUCENE-10531
> URL: https://issues.apache.org/jira/browse/LUCENE-10531
> Project: Lucene - Core
> Issue Type: Task
> Components: general/test
>Reporter: Tomoko Uchida
>Priority: Minor
> Fix For: 10.0 (main)
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> We are going to allow running the test on Xvfb (a virtual display that speaks
> X protocol) in [LUCENE-10528], this tweak is available only on Linux.
> I'm just guessing but it could confuse or bother also Mac and Windows users
> (we can't know what window manager developers are using); it may be better to
> make it opt-in by marking it as slow tests.
> Instead, I think we can enable a dedicated Github actions workflow for the
> distribution test that is triggered only when the related files are changed.
> Besides Linux, we could run it both on Mac and Windows which most users run
> the app on - it'd be slow, but if we limit the scope of the test I suppose it
> works functionally just fine (I'm running actions workflows on mac and
> windows elsewhere).
> To make it "slow test", we could add the same {{@Slow}} annotation as the
> {{test-framework}} to the distribution tests, for consistency.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10531) Mark testLukeCanBeLaunched @Nightly test and make a dedicated Github CI workflow for it
[
https://issues.apache.org/jira/browse/LUCENE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540437#comment-17540437
]
ASF subversion and git services commented on LUCENE-10531:
--
Commit ca977de41859fa6a4d9037e79cee4f01382e83f3 in lucene's branch
refs/heads/branch_9x from Tomoko Uchida
[ https://gitbox.apache.org/repos/asf?p=lucene.git;h=ca977de4185 ]
LUCENE-10531: Disable distribution test (gui test) on windows. (#917)
> Mark testLukeCanBeLaunched @Nightly test and make a dedicated Github CI
> workflow for it
> ---
>
> Key: LUCENE-10531
> URL: https://issues.apache.org/jira/browse/LUCENE-10531
> Project: Lucene - Core
> Issue Type: Task
> Components: general/test
>Reporter: Tomoko Uchida
>Priority: Minor
> Fix For: 10.0 (main)
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> We are going to allow running the test on Xvfb (a virtual display that speaks
> X protocol) in [LUCENE-10528], this tweak is available only on Linux.
> I'm just guessing but it could confuse or bother also Mac and Windows users
> (we can't know what window manager developers are using); it may be better to
> make it opt-in by marking it as slow tests.
> Instead, I think we can enable a dedicated Github actions workflow for the
> distribution test that is triggered only when the related files are changed.
> Besides Linux, we could run it both on Mac and Windows which most users run
> the app on - it'd be slow, but if we limit the scope of the test I suppose it
> works functionally just fine (I'm running actions workflows on mac and
> windows elsewhere).
> To make it "slow test", we could add the same {{@Slow}} annotation as the
> {{test-framework}} to the distribution tests, for consistency.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #917: LUCENE-10531: Disable distribution test (gui test) on windows.
dweiss commented on PR #917: URL: https://github.com/apache/lucene/pull/917#issuecomment-1133651444 I'm not sure what's causing this. It looks strange. I'll play around and report if I find out anything. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta commented on pull request #917: LUCENE-10531: Disable distribution test (gui test) on windows.
mocobeta commented on PR #917: URL: https://github.com/apache/lucene/pull/917#issuecomment-1133656519 Thanks. I muted it for now since I thought we shouldn't experiment in the upstream repository, but I'll also look at it when I have spare time. It looks like it can be easily reproducible but repeated job runs would be needed for debugging (it fails once in five times?)... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta commented on pull request #917: LUCENE-10531: Disable distribution test (gui test) on windows.
mocobeta commented on PR #917: URL: https://github.com/apache/lucene/pull/917#issuecomment-1133669396 Just a quick note: I noticed that it looks like some virtual display/monitor is enabled on the Windows Server VM in https://github.com/mocobeta/lucene/pull/2. I'm not sure the Windows VM in Policeman Jenkins Server has a virtual display though, I wonder if this could be a difference between GitHub Actions and the Jenkins server. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #917: LUCENE-10531: Disable distribution test (gui test) on windows.
dweiss commented on PR #917: URL: https://github.com/apache/lucene/pull/917#issuecomment-1133749382 I've been trying to reproduce this with more debugging info but can't, in spite of several dozen attempts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #917: LUCENE-10531: Disable distribution test (gui test) on windows.
dweiss commented on PR #917: URL: https://github.com/apache/lucene/pull/917#issuecomment-1133749390 https://github.com/dweiss/lucene/actions/runs/2363545679 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mdmarshmallow commented on a diff in pull request #841: LUCENE-10274: Add hyperrectangle faceting capabilities
mdmarshmallow commented on code in PR #841:
URL: https://github.com/apache/lucene/pull/841#discussion_r878767917
##
lucene/facet/src/java/org/apache/lucene/facet/hyperrectangle/HyperRectangle.java:
##
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.hyperrectangle;
+
+/** Holds the name and the number of dims for a HyperRectangle */
Review Comment:
Made this comment more accurate
##
lucene/facet/src/java/org/apache/lucene/facet/hyperrectangle/HyperRectangleFacetCounts.java:
##
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.hyperrectangle;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import org.apache.lucene.document.LongPoint;
+import org.apache.lucene.facet.FacetResult;
+import org.apache.lucene.facet.Facets;
+import org.apache.lucene.facet.FacetsCollector;
+import org.apache.lucene.facet.LabelAndValue;
+import org.apache.lucene.index.BinaryDocValues;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.search.DocIdSetIterator;
+
+/** Get counts given a list of HyperRectangles (which must be of the same
type) */
+public class HyperRectangleFacetCounts extends Facets {
+ /** Hypper rectangles passed to constructor. */
+ protected final HyperRectangle[] hyperRectangles;
+
+ /** Counts, initialized in subclass. */
+ protected final int[] counts;
+
+ /** Our field name. */
+ protected final String field;
+
+ /** Number of dimensions for field */
+ protected final int dims;
+
+ /** Total number of hits. */
+ protected int totCount;
+
+ /**
+ * Create HyperRectangleFacetCounts using this
+ *
+ * @param field Field name
+ * @param hits Hits to facet on
+ * @param hyperRectangles List of hyper rectangle facets
+ * @throws IOException If there is a problem reading the field
+ */
+ public HyperRectangleFacetCounts(
+ String field, FacetsCollector hits, HyperRectangle... hyperRectangles)
throws IOException {
+assert hyperRectangles.length > 0 : "Hyper rectangle ranges cannot be
empty";
Review Comment:
I think that these should just throw `IllegalArgumentExceptions`, I changed
this to a conditional and included a `null` check.
##
lucene/facet/src/java/org/apache/lucene/facet/hyperrectangle/HyperRectangleFacetCounts.java:
##
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.hyperrectangle;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import org.apache.l
[GitHub] [lucene] dsmiley merged pull request #902: LUCENE-8519 MultiDocValues.getNormValues should not call getMergedFieldInfos
dsmiley merged PR #902: URL: https://github.com/apache/lucene/pull/902 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8519) MultiDocValues.getNormValues should not call getMergedFieldInfos
[
https://issues.apache.org/jira/browse/LUCENE-8519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540521#comment-17540521
]
ASF subversion and git services commented on LUCENE-8519:
-
Commit da3be3ac4201352e0162d0534d993dba1e088485 in lucene's branch
refs/heads/branch_9x from Rushabh Shah
[ https://gitbox.apache.org/repos/asf?p=lucene.git;h=da3be3ac420 ]
LUCENE-8519: MultiDocValues.getNormValues should not call getMergedFieldInfos
(#902)
Co-authored-by: Rushabh Shah
> MultiDocValues.getNormValues should not call getMergedFieldInfos
>
>
> Key: LUCENE-8519
> URL: https://issues.apache.org/jira/browse/LUCENE-8519
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: David Smiley
>Priority: Minor
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> {{MultiDocValues.getNormValues}} should not call {{getMergedFieldInfos}}
> because it's a needless expense. getNormValues simply wants to know if each
> LeafReader that has this field has norms as well; that's all.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta commented on pull request #917: LUCENE-10531: Disable distribution test (gui test) on windows.
mocobeta commented on PR #917: URL: https://github.com/apache/lucene/pull/917#issuecomment-1133810346 Repeating the test in the same machine didn't reproduce the failure for me too. I tried setting up 20 virtual machines and one of them failed. https://github.com/mocobeta/lucene/pull/2 It looks like there is something wrong with the machine setup (?), I have no idea of how to debug it yet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8519) MultiDocValues.getNormValues should not call getMergedFieldInfos
[
https://issues.apache.org/jira/browse/LUCENE-8519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540523#comment-17540523
]
ASF subversion and git services commented on LUCENE-8519:
-
Commit d17c6056d8caada6db6c1c4f280f54960e058ee2 in lucene's branch
refs/heads/main from Rushabh Shah
[ https://gitbox.apache.org/repos/asf?p=lucene.git;h=d17c6056d8c ]
LUCENE-8519: MultiDocValues.getNormValues should not call getMergedFieldInfos
(#902)
Co-authored-by: Rushabh Shah
> MultiDocValues.getNormValues should not call getMergedFieldInfos
>
>
> Key: LUCENE-8519
> URL: https://issues.apache.org/jira/browse/LUCENE-8519
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: David Smiley
>Priority: Minor
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> {{MultiDocValues.getNormValues}} should not call {{getMergedFieldInfos}}
> because it's a needless expense. getNormValues simply wants to know if each
> LeafReader that has this field has norms as well; that's all.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dsmiley commented on pull request #898: LUCENE-8519 MultiDocValues.getNormValues should not call getMergedFieldInfos
dsmiley commented on PR #898: URL: https://github.com/apache/lucene/pull/898#issuecomment-1133815329 I ended up cherry picking #902 so I'll close this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dsmiley closed pull request #898: LUCENE-8519 MultiDocValues.getNormValues should not call getMergedFieldInfos
dsmiley closed pull request #898: LUCENE-8519 MultiDocValues.getNormValues should not call getMergedFieldInfos URL: https://github.com/apache/lucene/pull/898 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-8519) MultiDocValues.getNormValues should not call getMergedFieldInfos
[
https://issues.apache.org/jira/browse/LUCENE-8519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley resolved LUCENE-8519.
--
Fix Version/s: 9.3
Resolution: Fixed
Thanks for contributing Rushabh!
> MultiDocValues.getNormValues should not call getMergedFieldInfos
>
>
> Key: LUCENE-8519
> URL: https://issues.apache.org/jira/browse/LUCENE-8519
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: David Smiley
>Priority: Minor
> Fix For: 9.3
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> {{MultiDocValues.getNormValues}} should not call {{getMergedFieldInfos}}
> because it's a needless expense. getNormValues simply wants to know if each
> LeafReader that has this field has norms as well; that's all.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3373) waitForMerges deadlocks if background merge fails
[
https://issues.apache.org/jira/browse/LUCENE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540526#comment-17540526
]
Vigya Sharma commented on LUCENE-3373:
--
Sorry you ran into this Thomas. I tried beasting some concurrent IndexWriter
tests but haven't been able to repro this yet.
{quote}i would suggest updating the MergeThread to catch all exceptions and
allow processing the next merge. right now, any merge failure results in a
ThreadDeath, which seems rather nasty. should probably just catch the exception
and log a index trace message
{quote}
We already seem to have a {{catch (Throwable exc)}} that wraps all the logic in
the thread {{run()}} method
([code|https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L720]).
{{waitForMerges()}} waits for both pendingMerges and runningMerges to get
empty. The {{OneMerge}} object gets removed from {{pendingMerges}} before
{{getNextMerge()}} returns. Similarly, it is removed from runningMerges when
{{onMergeFinished()}} is called, which is done in the finally clause of the try
wrapping run() code.
It's been a while since this root cause and fix in this issue were suggested.
From code, I don't see why a background thread dying in
ConcurrentMergeScheduler would lead to an endless stall in waitForMerges.
Probably, the issue in run() has been subsequently addressed.
But since you ran into the problem still, there is probably some underlying
deadlock case still lurking in IndexWriter shutdown.
(It is possible that
[LUCENE-10583|https://issues.apache.org/jira/browse/LUCENE-10583] is an
independent issue. I'll mention it here to keep the two linked, since
LUCENE-10583 mentions the problem you've seen recently).
> waitForMerges deadlocks if background merge fails
> -
>
> Key: LUCENE-3373
> URL: https://issues.apache.org/jira/browse/LUCENE-3373
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Affects Versions: 3.0.3
>Reporter: Tim Smith
>Priority: Major
>
> waitForMerges can deadlock if a merge fails for ConcurrentMergeScheduler
> this is because the merge thread will die, but pending merges are still
> available
> normally, the merge thread will pick up the next merge once it finishes the
> previous merge, but in the event of a merge exception, the pending work is
> not resumed, but waitForMerges won't complete until all pending work is
> complete
> i worked around this by overriding doMerge() like so:
> {code}
> protected final void doMerge(MergePolicy.OneMerge merge) throws IOException
> {
> try {
> super.doMerge(merge);
> } catch (Throwable exc) {
> // Just logging the exception and not rethrowing
> // insert logging code here
> }
> }
> {code}
> Here's the rough steps i used to reproduce this issue:
> override doMerge like so
> {code}
> protected final void doMerge(MergePolicy.OneMerge merge) throws IOException
> {
> try {Thread.sleep(500L);} catch (InterruptedException e) { }
> super.doMerge(merge);
> throw new IOException("fail");
> }
> {code}
> then, if you do the following:
> loop 50 times:
> addDocument // any doc
> commit
> waitForMerges // This will deadlock sometimes
> SOLR-2017 may be related to this (stack trace for deadlock looked related)
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3373) waitForMerges deadlocks if background merge fails
[
https://issues.apache.org/jira/browse/LUCENE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540531#comment-17540531
]
Vigya Sharma commented on LUCENE-3373:
--
I have a hypothesis...
When we IndexWriter.close(), we invoke
[mergeScheduler.merge|https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java#L2707],
giving it one last change to process all pending merges. Within the conc.
merge scheduler, there is rate limiting logic, that [stalls
threads|https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L539-L541]
to adjust for merge I/O rate. It seems like we do not want to stall a
{{mergeThread}}, and so, if a mergeThread calls {{maybeStall()}}, we
[break|https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L540]
out of the while loop that is assigning threads to pending merges and draining
the pendingMerges queue.
Hence, if the IW.shutdown call was to get picked up by a merge thread, it would
break out of the loop before scheduling all pending merges. The outer shutdown
call will expect all merges to have been scheduled (or get scheduled), and keep
waiting for pendingMerges and runningMerges queues to get drained. Since the
whole thing is also
[gated|https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java#L1315]
to allow only one thread to do this, we miss our only chance to schedule
threads, which leads to an endless wait.
All of this hinges on a MergeThread getting picked for mergeScheduler.merge. Or
somehow, the thread that gets picked, exiting without scheduling all of them.
Is that a possibility? I'm not sure how a merge thread could get picked for
IW.close(). Any ideas for a concurrent test that would make this scenario more
probable to hit?
> waitForMerges deadlocks if background merge fails
> -
>
> Key: LUCENE-3373
> URL: https://issues.apache.org/jira/browse/LUCENE-3373
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Affects Versions: 3.0.3
>Reporter: Tim Smith
>Priority: Major
>
> waitForMerges can deadlock if a merge fails for ConcurrentMergeScheduler
> this is because the merge thread will die, but pending merges are still
> available
> normally, the merge thread will pick up the next merge once it finishes the
> previous merge, but in the event of a merge exception, the pending work is
> not resumed, but waitForMerges won't complete until all pending work is
> complete
> i worked around this by overriding doMerge() like so:
> {code}
> protected final void doMerge(MergePolicy.OneMerge merge) throws IOException
> {
> try {
> super.doMerge(merge);
> } catch (Throwable exc) {
> // Just logging the exception and not rethrowing
> // insert logging code here
> }
> }
> {code}
> Here's the rough steps i used to reproduce this issue:
> override doMerge like so
> {code}
> protected final void doMerge(MergePolicy.OneMerge merge) throws IOException
> {
> try {Thread.sleep(500L);} catch (InterruptedException e) { }
> super.doMerge(merge);
> throw new IOException("fail");
> }
> {code}
> then, if you do the following:
> loop 50 times:
> addDocument // any doc
> commit
> waitForMerges // This will deadlock sometimes
> SOLR-2017 may be related to this (stack trace for deadlock looked related)
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10583) Deadlock with MMapDirectory while waitForMerges
[
https://issues.apache.org/jira/browse/LUCENE-10583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540532#comment-17540532
]
Vigya Sharma commented on LUCENE-10583:
---
Added some comments to
[LUCENE-3373|https://issues.apache.org/jira/browse/LUCENE-3373], where a
similar issue was encountered, and there is some existing discussion.
> Deadlock with MMapDirectory while waitForMerges
> ---
>
> Key: LUCENE-10583
> URL: https://issues.apache.org/jira/browse/LUCENE-10583
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Affects Versions: 8.11.1
> Environment: Java 17
> OS: Windows 2016
>Reporter: Thomas Hoffmann
>Priority: Critical
>
> Hello,
> a deadlock situation happened in our application. We are using MMapDirectory
> on Windows 2016 and got the following stacktrace:
> {code:java}
> "https-openssl-nio-443-exec-30" #166 daemon prio=5 os_prio=0 cpu=78703.13ms
> "https-openssl-nio-443-exec-30" #166 daemon prio=5 os_prio=0 cpu=78703.13ms
> elapsed=81248.18s tid=0x2860af10 nid=0x237c in Object.wait()
> [0x413fc000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(java.base@17.0.2/Native Method)
> - waiting on
> at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4983)
> - locked <0x0006ef1fc020> (a org.apache.lucene.index.IndexWriter)
> at
> org.apache.lucene.index.IndexWriter.waitForMerges(IndexWriter.java:2697)
> - locked <0x0006ef1fc020> (a org.apache.lucene.index.IndexWriter)
> at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1236)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1278)
> at
> com.speed4trade.ebs.module.search.SearchService.updateSearchIndex(SearchService.java:1723)
> - locked <0x0006d5c00208> (a org.apache.lucene.store.MMapDirectory)
> at
> com.speed4trade.ebs.module.businessrelations.ticket.TicketChangedListener.postUpdate(TicketChangedListener.java:142)
> ...{code}
> All threads were waiting to lock <0x0006d5c00208> which got never
> released.
> A lucene thread was also blocked, I dont know if this is relevant:
> {code:java}
> "Lucene Merge Thread #0" #18466 daemon prio=5 os_prio=0 cpu=15.63ms
> elapsed=3499.07s tid=0x459453e0 nid=0x1f8 waiting for monitor entry
> [0x5da9e000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.lucene.store.FSDirectory.deletePendingFiles(FSDirectory.java:346)
> - waiting to lock <0x0006d5c00208> (a
> org.apache.lucene.store.MMapDirectory)
> at
> org.apache.lucene.store.FSDirectory.maybeDeletePendingFiles(FSDirectory.java:363)
> at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:248)
> at
> org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$1.createOutput(ConcurrentMergeScheduler.java:289)
> at
> org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.(CompressingStoredFieldsWriter.java:121)
> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:130)
> at
> org.apache.lucene.codecs.lucene87.Lucene87StoredFieldsFormat.fieldsWriter(Lucene87StoredFieldsFormat.java:141)
> at
> org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:227)
> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:105)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4757)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4361)
> at
> org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:5920)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684){code}
> If looks like the merge operation never finished and released the lock.
> Is there any option to prevent this deadlock or how to investigate it further?
> A load-test didn't show this problem unfortunately.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta commented on pull request #917: LUCENE-10531: Disable distribution test (gui test) on windows.
mocobeta commented on PR #917: URL: https://github.com/apache/lucene/pull/917#issuecomment-1133823694 I tested several timeouts. - 60 seconds: occasionally fails - 120 seconds: occasionally fails - 600 seconds: didn't see any failure in several dozens of runs It looks like increasing timeout effectively solves the problem to me :) And there seem to be no substantial increases in the workflow's total execution time when I increased the timeout to a large value. The system clock in the VM may be sometimes messed up? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] shaie commented on a diff in pull request #841: LUCENE-10274: Add hyperrectangle faceting capabilities
shaie commented on code in PR #841:
URL: https://github.com/apache/lucene/pull/841#discussion_r878796006
##
lucene/core/src/java/org/apache/lucene/document/DoublePointDocValuesField.java:
##
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.document;
Review Comment:
I am not opposed to this change, but I find it a bit strange that we add a
"general" Point DV support without any tests that exercise it, and the only
usage of it is in the Facet module. Do we see a use case in the future for
other DV usage? Like Sorting?
Anyway I'm fine either way, just wanted to comment here that since it's
`@lucene.experimental` we could have also left it in the facet package and then
move here if a more general use case came up.
##
lucene/facet/src/test/org/apache/lucene/facet/hyperrectangle/TestHyperRectangleFacetCounts.java:
##
@@ -0,0 +1,374 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.hyperrectangle;
+
+import java.io.IOException;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.DoublePointDocValuesField;
+import org.apache.lucene.document.LongPointDocValuesField;
+import org.apache.lucene.facet.FacetResult;
+import org.apache.lucene.facet.FacetTestCase;
+import org.apache.lucene.facet.Facets;
+import org.apache.lucene.facet.FacetsCollector;
+import org.apache.lucene.facet.FacetsCollectorManager;
+import org.apache.lucene.facet.LabelAndValue;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.tests.index.RandomIndexWriter;
+
+public class TestHyperRectangleFacetCounts extends FacetTestCase {
+
+ public void testBasicLong() throws Exception {
+Directory d = newDirectory();
+RandomIndexWriter w = new RandomIndexWriter(random(), d);
+
+for (long l = 0; l < 100; l++) {
+ Document doc = new Document();
+ LongPointDocValuesField field = new LongPointDocValuesField("field", l,
l + 1L, l + 2L);
+ doc.add(field);
+ w.addDocument(doc);
+}
+
+// Also add point with Long.MAX_VALUE
+Document doc = new Document();
+LongPointDocValuesField field =
+new LongPointDocValuesField(
+"field", Long.MAX_VALUE - 2L, Long.MAX_VALUE - 1L, Long.MAX_VALUE);
+doc.add(field);
+w.addDocument(doc);
+
+IndexReader r = w.getReader();
+w.close();
+
+IndexSearcher s = newSearcher(r);
+FacetsCollector fc = s.search(new MatchAllDocsQuery(), new
FacetsCollectorManager());
+
+Facets facets =
+new HyperRectangleFacetCounts(
+"field",
+fc,
+new LongHyperRectangle(
+"less than (10, 11, 12)",
+new HyperRectangle.LongRangePair(0L, true, 10L, false),
+new HyperRectangle.LongRangePair(0L, true, 11L, false),
+new HyperRectangle.LongRangePair(0L, true, 12L, false)),
+new LongHyperRectangle(
+"less than or equal to (10, 11, 12)",
+new HyperRectangle.LongRangePair(0L, true, 10L, true),
+new HyperRectangle.LongRangePair(0L, true, 11L, true),
+new HyperRectangle.LongRangePair(0L, true, 12L, true)),
+new LongHyperRectangle(
+"over (90
[jira] [Created] (LUCENE-10586) Minor refactoring in Lucene90BlockTreeTermsReader local variables: metaIn, indexMetaIn, termsMetaIn
Tomoko Uchida created LUCENE-10586:
--
Summary: Minor refactoring in Lucene90BlockTreeTermsReader local
variables: metaIn, indexMetaIn, termsMetaIn
Key: LUCENE-10586
URL: https://issues.apache.org/jira/browse/LUCENE-10586
Project: Lucene - Core
Issue Type: Improvement
Reporter: Tomoko Uchida
Those three local variables refer to the same {{IndexInput}} object (no clone()
is called).
{code}
indexMetaIn = termsMetaIn = metaIn;
{code}
I'm not sure but maybe there are some historical reasons. I wonder if it would
be better to have only one reference for the underlying {{IndexInput}} object
to make it a little easy to follow the code.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta opened a new pull request, #918: Lucene-10586: Minor cleanup for local variables in BlockTreeTermsReader
mocobeta opened a new pull request, #918: URL: https://github.com/apache/lucene/pull/918 See https://issues.apache.org/jira/browse/LUCENE-10586 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #917: LUCENE-10531: Disable distribution test (gui test) on windows.
dweiss commented on PR #917: URL: https://github.com/apache/lucene/pull/917#issuecomment-1133832324 It is impossible and insane - something is wrong. I don't believe forking a simple process requires 600 seconds to complete, while running gradle tasks completes orders of magnitude faster. Thank you for testing though. Can you merge the changes from [script-testing-windows](https://github.com/dweiss/lucene/tree/script-testing-windows) branch and rerun your stress test? It emits more logging, let's see what's happening there. Separately from the above, those Windows VMs do tend to hang occasionally - I've seen this not just on Lucene but also on other projects. The logs are simply truncated then, no reason or cause can be determined. They're just flaky, occasionally. But in this case I think it looks too predictably stalling in one place to be caused by mere flakiness. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
