[jira] [Created] (LUCENE-9316) Incorporate all :precommit tasks into :check
Dawid Weiss created LUCENE-9316: --- Summary: Incorporate all :precommit tasks into :check Key: LUCENE-9316 URL: https://issues.apache.org/jira/browse/LUCENE-9316 Project: Lucene - Core Issue Type: Sub-task Reporter: Dawid Weiss Assignee: Dawid Weiss -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9316) Incorporate all :precommit tasks into :check
[
https://issues.apache.org/jira/browse/LUCENE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081215#comment-17081215
]
Dawid Weiss commented on LUCENE-9316:
-
Something is wrong but I can't quite pinpoint the source of the problem. When I
modify precommit.gradle with this patch:
{code}
diff --git a/gradle/validation/precommit.gradle
b/gradle/validation/precommit.gradle
index 94ed800cb28..79347646454 100644
--- a/gradle/validation/precommit.gradle
+++ b/gradle/validation/precommit.gradle
@@ -44,5 +44,9 @@ configure(rootProject) {
]}
}
}
+
+ // Each validation task should be attached to check but make sure
+ // precommit() as a whole is a dependency on rootProject.check
+ check.dependsOn precommit
}
{code}
and then run:
{code}
./gradlew clean check -x test -x javadoc --console=plain
-Pvalidation.git.failOnModified=false
{code}
everything works fine.
But when the default javadoc is allowed there are strange file-not-found
problems; how are these related I've no idea.
{code}
./gradlew clean check -x test --console=plain
-Pvalidation.git.failOnModified=false
{code}
results in (the exact problem may vary depending on task ordering, it's not
constant):
{code}
> Task :lucene:facet:javadoc
FAILURE: Build failed with an exception.
* Where:
Script '/home/dweiss/work-ssd/lucene-solr/gradle/render-javadoc.gradle' line:
136
* What went wrong:
Execution failed for task ':lucene:demo:renderJavadoc'.
> Javadoc generation failed for :lucene:demo,
Options file at:
/home/dweiss/work-ssd/lucene-solr/lucene/demo/build/tmp/renderJavadoc/javadoc-options.txt
Command output at:
/home/dweiss/work-ssd/lucene-solr/lucene/demo/build/tmp/renderJavadoc/javadoc-output.txt
{code}
The strange thing is that this file is at the location above (exists).
> Incorporate all :precommit tasks into :check
>
>
> Key: LUCENE-9316
> URL: https://issues.apache.org/jira/browse/LUCENE-9316
> Project: Lucene - Core
> Issue Type: Sub-task
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9316) Incorporate all :precommit tasks into :check
[
https://issues.apache.org/jira/browse/LUCENE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081218#comment-17081218
]
Dawid Weiss commented on LUCENE-9316:
-
Not only the file is there but it contains valid data. And when I re-run
javadoc with the options file it finishes just fine. The log from gradle
executions shows this:
{code}
...
> Task :lucene:facet:renderJavadoc
> Task :lucene:backward-codecs:renderJavadoc
...
> Task :lucene:demo:renderJavadoc FAILED
Loading source files for package org.apache.lucene...
Constructing Javadoc information...
javadoc: error - Error reading file:
/home/dweiss/work-ssd/lucene-solr/lucene/facet/build/docs/javadoc/element-list
javadoc: error - Error reading file:
/home/dweiss/work-ssd/lucene-solr/lucene/expressions/build/docs/javadoc/element-list
Standard Doclet version 11.0.4
Building tree for all the packages and classes...
Generating
/home/dweiss/work-ssd/lucene-solr/lucene/demo/build/docs/javadoc/org/apache/lucene/demo/IndexFiles.html...
Generating
/home/dweiss/work-ssd/lucene-solr/lucene/demo/build/docs/javadoc/org/apache/lucene/demo/SearchFiles.html...
...
Note: Custom tags that were not seen: @lucene.internal, @lucene.experimental,
@lucene.spi
2 errors
{code}
How is this even possible? [~tomoko] do you have any idea? :)
> Incorporate all :precommit tasks into :check
>
>
> Key: LUCENE-9316
> URL: https://issues.apache.org/jira/browse/LUCENE-9316
> Project: Lucene - Core
> Issue Type: Sub-task
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] s1monw merged pull request #1397: LUCENE-9304: Refactor DWPTPool to pool DWPT directly
s1monw merged pull request #1397: LUCENE-9304: Refactor DWPTPool to pool DWPT directly URL: https://github.com/apache/lucene-solr/pull/1397 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9304) Clean up DWPTPool
[ https://issues.apache.org/jira/browse/LUCENE-9304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081230#comment-17081230 ] ASF subversion and git services commented on LUCENE-9304: - Commit 2602269f3ec856f55a08a0c5a32a1364fd9c25ea in lucene-solr's branch refs/heads/master from Simon Willnauer [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2602269 ] LUCENE-9304: Refactor DWPTPool to pool DWPT directly (#1397) This change removes the ThreadState indirection from DWPTPool and pools DWPT directly. The tracking information and locking semantics are mostly moved to DWPT directly and the pool semantics have changed slightly such that DWPT need to be checked-out in the pool once they need to be flushed or aborted. This automatically grows and shrinks the number of DWPT in the system when number of threads grow or shrink. Access of pooled DWPTs is more straight forward and doesn't require ordinal. Instead consumers can just iterate over the elements in the pool. This allowed for removal of indirections in DWPTFlushControl like BlockedFlush, the removal of DWPTPool setter and getter in IndexWriterConfig and the addition of stronger assertions in DWPT and DW. > Clean up DWPTPool > -- > > Key: LUCENE-9304 > URL: https://issues.apache.org/jira/browse/LUCENE-9304 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: master (9.0), 8.6 >Reporter: Simon Willnauer >Priority: Major > Time Spent: 9h 10m > Remaining Estimate: 0h > > DWPTPool currently uses an indirection called ThreadState to hold DWPT > instances. This class holds several information that belongs in other places, > inherits from ReentrantLock and has a mutable nature. Instead we could pool > the DWPT directly and remove other indirections inside DWPTFlushControl if > we move some of the ThreadState properties to DWPT directly. The threadpool > also has a problem that is grows it's ThreadStates to the number of > concurrently indexing threads but never shrinks it if they are reduced. With > pooling DWPT directly this limitation could be removed. > In summary, this component has seen quite some refactoring and requires some > cleanups and docs changes in order to stay the test of time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9304) Clean up DWPTPool
[ https://issues.apache.org/jira/browse/LUCENE-9304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081235#comment-17081235 ] ASF subversion and git services commented on LUCENE-9304: - Commit 83b34d47900febbd8b5e337a1277b5696c536f78 in lucene-solr's branch refs/heads/branch_8x from Simon Willnauer [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=83b34d4 ] LUCENE-9304: Refactor DWPTPool to pool DWPT directly (#1397) This change removes the ThreadState indirection from DWPTPool and pools DWPT directly. The tracking information and locking semantics are mostly moved to DWPT directly and the pool semantics have changed slightly such that DWPT need to be checked-out in the pool once they need to be flushed or aborted. This automatically grows and shrinks the number of DWPT in the system when number of threads grow or shrink. Access of pooled DWPTs is more straight forward and doesn't require ordinal. Instead consumers can just iterate over the elements in the pool. This allowed for removal of indirections in DWPTFlushControl like BlockedFlush, the removal of DWPTPool setter and getter in IndexWriterConfig and the addition of stronger assertions in DWPT and DW. > Clean up DWPTPool > -- > > Key: LUCENE-9304 > URL: https://issues.apache.org/jira/browse/LUCENE-9304 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: master (9.0), 8.6 >Reporter: Simon Willnauer >Assignee: Simon Willnauer >Priority: Major > Fix For: master (9.0), 8.6 > > Time Spent: 9h 10m > Remaining Estimate: 0h > > DWPTPool currently uses an indirection called ThreadState to hold DWPT > instances. This class holds several information that belongs in other places, > inherits from ReentrantLock and has a mutable nature. Instead we could pool > the DWPT directly and remove other indirections inside DWPTFlushControl if > we move some of the ThreadState properties to DWPT directly. The threadpool > also has a problem that is grows it's ThreadStates to the number of > concurrently indexing threads but never shrinks it if they are reduced. With > pooling DWPT directly this limitation could be removed. > In summary, this component has seen quite some refactoring and requires some > cleanups and docs changes in order to stay the test of time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9304) Clean up DWPTPool
[ https://issues.apache.org/jira/browse/LUCENE-9304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer resolved LUCENE-9304. - Fix Version/s: 8.6 master (9.0) Lucene Fields: New,Patch Available (was: New) Assignee: Simon Willnauer Resolution: Fixed > Clean up DWPTPool > -- > > Key: LUCENE-9304 > URL: https://issues.apache.org/jira/browse/LUCENE-9304 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: master (9.0), 8.6 >Reporter: Simon Willnauer >Assignee: Simon Willnauer >Priority: Major > Fix For: master (9.0), 8.6 > > Time Spent: 9h 10m > Remaining Estimate: 0h > > DWPTPool currently uses an indirection called ThreadState to hold DWPT > instances. This class holds several information that belongs in other places, > inherits from ReentrantLock and has a mutable nature. Instead we could pool > the DWPT directly and remove other indirections inside DWPTFlushControl if > we move some of the ThreadState properties to DWPT directly. The threadpool > also has a problem that is grows it's ThreadStates to the number of > concurrently indexing threads but never shrinks it if they are reduced. With > pooling DWPT directly this limitation could be removed. > In summary, this component has seen quite some refactoring and requires some > cleanups and docs changes in order to stay the test of time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-9314) Lucene monitor module uses ByteBuffersDirectory rather than MemoryIndex when matching a single document
[ https://issues.apache.org/jira/browse/LUCENE-9314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081244#comment-17081244 ] Pierre-Luc Perron edited comment on LUCENE-9314 at 4/11/20, 11:40 AM: -- Thanks for the feedback. I'll do it over the week end. was (Author: pl.perron): Thank for the feedback. I'll do it over the week end. > Lucene monitor module uses ByteBuffersDirectory rather than MemoryIndex when > matching a single document > --- > > Key: LUCENE-9314 > URL: https://issues.apache.org/jira/browse/LUCENE-9314 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/other >Affects Versions: 8.2, 8.3, 8.4, 8.5 >Reporter: Pierre-Luc Perron >Priority: Minor > Labels: easyfix, performance, pull-request-available > Attachments: LUCENE-9314.patch, LUCENE-9314.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Lucene monitor function, [match single > document|https://github.com/apache/lucene-solr/blob/e376582e25d02f85b415eb1c0456f3fdc800fc84/lucene/monitor/src/java/org/apache/lucene/monitor/Monitor.java#L276], > wraps a single document into a array of documents. Hence, it always calls > the function, [match many > documents|https://github.com/apache/lucene-solr/blob/e376582e25d02f85b415eb1c0456f3fdc800fc84/lucene/monitor/src/java/org/apache/lucene/monitor/Monitor.java#L256], > which builds a > [MultiDocumentBatch|https://github.com/apache/lucene-solr/blob/71d335ff688982cef10a648c914623c81ced/lucene/monitor/src/java/org/apache/lucene/monitor/DocumentBatch.java#L56] > rather than a > [SingletonDocumentBatch|https://github.com/apache/lucene-solr/blob/71d335ff688982cef10a648c914623c81ced/lucene/monitor/src/java/org/apache/lucene/monitor/DocumentBatch.java#L46]. > The former uses ByteBuffersDirectory while later uses MemoryIndex. > As per documentation, > [MemoryIndex|https://lucene.apache.org/core/8_2_0/memory/index.html] is a > high-performance single-document main memory Apache Lucene fulltext search > index. Hence, Lucene monitor should use it when matching a single document. > The patch routes [match single > document|https://github.com/apache/lucene-solr/blob/e376582e25d02f85b415eb1c0456f3fdc800fc84/lucene/monitor/src/java/org/apache/lucene/monitor/Monitor.java#L276] > to a SingletonDocumentBatch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9314) Lucene monitor module uses ByteBuffersDirectory rather than MemoryIndex when matching a single document
[ https://issues.apache.org/jira/browse/LUCENE-9314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081244#comment-17081244 ] Pierre-Luc Perron commented on LUCENE-9314: --- Thank for the feedback. I'll do it over the week end. > Lucene monitor module uses ByteBuffersDirectory rather than MemoryIndex when > matching a single document > --- > > Key: LUCENE-9314 > URL: https://issues.apache.org/jira/browse/LUCENE-9314 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/other >Affects Versions: 8.2, 8.3, 8.4, 8.5 >Reporter: Pierre-Luc Perron >Priority: Minor > Labels: easyfix, performance, pull-request-available > Attachments: LUCENE-9314.patch, LUCENE-9314.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Lucene monitor function, [match single > document|https://github.com/apache/lucene-solr/blob/e376582e25d02f85b415eb1c0456f3fdc800fc84/lucene/monitor/src/java/org/apache/lucene/monitor/Monitor.java#L276], > wraps a single document into a array of documents. Hence, it always calls > the function, [match many > documents|https://github.com/apache/lucene-solr/blob/e376582e25d02f85b415eb1c0456f3fdc800fc84/lucene/monitor/src/java/org/apache/lucene/monitor/Monitor.java#L256], > which builds a > [MultiDocumentBatch|https://github.com/apache/lucene-solr/blob/71d335ff688982cef10a648c914623c81ced/lucene/monitor/src/java/org/apache/lucene/monitor/DocumentBatch.java#L56] > rather than a > [SingletonDocumentBatch|https://github.com/apache/lucene-solr/blob/71d335ff688982cef10a648c914623c81ced/lucene/monitor/src/java/org/apache/lucene/monitor/DocumentBatch.java#L46]. > The former uses ByteBuffersDirectory while later uses MemoryIndex. > As per documentation, > [MemoryIndex|https://lucene.apache.org/core/8_2_0/memory/index.html] is a > high-performance single-document main memory Apache Lucene fulltext search > index. Hence, Lucene monitor should use it when matching a single document. > The patch routes [match single > document|https://github.com/apache/lucene-solr/blob/e376582e25d02f85b415eb1c0456f3fdc800fc84/lucene/monitor/src/java/org/apache/lucene/monitor/Monitor.java#L276] > to a SingletonDocumentBatch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-9909) Nuke one of DefaultSolrThreadFactory and SolrjNamedThreadFactory
[ https://issues.apache.org/jira/browse/SOLR-9909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081253#comment-17081253 ] Shalin Shekhar Mangar commented on SOLR-9909: - Well, we can deprecate SolrjNamedThreadFactory and add a SolrNamedThreadFactory in 8x. The former can be deleted on master. > Nuke one of DefaultSolrThreadFactory and SolrjNamedThreadFactory > > > Key: SOLR-9909 > URL: https://issues.apache.org/jira/browse/SOLR-9909 > Project: Solr > Issue Type: Task >Reporter: Shalin Shekhar Mangar >Priority: Trivial > Fix For: 6.7, 7.0 > > Attachments: SOLR-9909-01.patch, SOLR-9909-02.patch, > SOLR-9909-03.patch, SOLR-9909.patch > > > DefaultSolrThreadFactory and SolrjNamedThreadFactory have exactly the same > code. Let's remove one of them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9316) Incorporate all :precommit tasks into :check
[ https://issues.apache.org/jira/browse/LUCENE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081254#comment-17081254 ] Tomoko Uchida commented on LUCENE-9316: --- I cannot reproduce the error... but "javadoc" and "renderJavadoc" tasks share same output folder (_project_/build/docs), that is problematic when they are executed in parallel ? > Incorporate all :precommit tasks into :check > > > Key: LUCENE-9316 > URL: https://issues.apache.org/jira/browse/LUCENE-9316 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-9909) Nuke one of DefaultSolrThreadFactory and SolrjNamedThreadFactory
[ https://issues.apache.org/jira/browse/SOLR-9909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-9909: Attachment: SOLR-9909.patch > Nuke one of DefaultSolrThreadFactory and SolrjNamedThreadFactory > > > Key: SOLR-9909 > URL: https://issues.apache.org/jira/browse/SOLR-9909 > Project: Solr > Issue Type: Task >Reporter: Shalin Shekhar Mangar >Priority: Trivial > Fix For: 6.7, 7.0 > > Attachments: SOLR-9909-01.patch, SOLR-9909-02.patch, > SOLR-9909-03.patch, SOLR-9909.patch, SOLR-9909.patch > > > DefaultSolrThreadFactory and SolrjNamedThreadFactory have exactly the same > code. Let's remove one of them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-9909) Nuke one of DefaultSolrThreadFactory and SolrjNamedThreadFactory
[ https://issues.apache.org/jira/browse/SOLR-9909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081256#comment-17081256 ] Shalin Shekhar Mangar commented on SOLR-9909: - Updated patch which deprecates SolrjNamedThreadFactory and adds a SolrNamedThreadFactory. Once this patch is applied on master and branch_8x, I will follow up with a commit on master to delete the deprecated SolrjNamedThreadFactory. > Nuke one of DefaultSolrThreadFactory and SolrjNamedThreadFactory > > > Key: SOLR-9909 > URL: https://issues.apache.org/jira/browse/SOLR-9909 > Project: Solr > Issue Type: Task >Reporter: Shalin Shekhar Mangar >Priority: Trivial > Fix For: 6.7, 7.0 > > Attachments: SOLR-9909-01.patch, SOLR-9909-02.patch, > SOLR-9909-03.patch, SOLR-9909.patch, SOLR-9909.patch > > > DefaultSolrThreadFactory and SolrjNamedThreadFactory have exactly the same > code. Let's remove one of them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9316) Incorporate all :precommit tasks into :check
[ https://issues.apache.org/jira/browse/LUCENE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081261#comment-17081261 ] Dawid Weiss commented on LUCENE-9316: - How many cores do you have on your machine, Tomoko? It happens fairly when I apply the patch above and with check command (minus tests). > Incorporate all :precommit tasks into :check > > > Key: LUCENE-9316 > URL: https://issues.apache.org/jira/browse/LUCENE-9316 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9316) Incorporate all :precommit tasks into :check
[
https://issues.apache.org/jira/browse/LUCENE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081265#comment-17081265
]
Tomoko Uchida commented on LUCENE-9316:
---
Sorry, I've run it on wrong branch. I also see similar errors (though the
message is Japanese).
{code}
* What went wrong:
Execution failed for task ':lucene:core:renderJavadoc'.
> java.io.FileNotFoundException:
> /mnt/hdd/repo/lucene-solr/lucene/core/build/tmp/renderJavadoc/javadoc-options.txt
> (そのようなファイルやディレクトリはありません)
{code}
> Incorporate all :precommit tasks into :check
>
>
> Key: LUCENE-9316
> URL: https://issues.apache.org/jira/browse/LUCENE-9316
> Project: Lucene - Core
> Issue Type: Sub-task
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9316) Incorporate all :precommit tasks into :check
[
https://issues.apache.org/jira/browse/LUCENE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081286#comment-17081286
]
Tomoko Uchida commented on LUCENE-9316:
---
Please ignore my previous comment, the error isn't related to parallel
execution of javadoc and renderJavadoc, but it seems something wrong with
"renderJavadoc" task.
This works for me.
{code:java}
$ ./gradlew clean
BUILD SUCCESSFUL in 1s
$ ./gradlew :lucene:core:renderJavadoc
{code}
But this does not work. (I have no idea about it for now, will look at it.)
{code:java}
$ ./gradlew clean :lucene:core:renderJavadoc
FAILURE: Build failed with an exception.
* Where:
Script '/mnt/hdd/repo/lucene-solr/gradle/render-javadoc.gradle' line: 104
* What went wrong:
Execution failed for task ':lucene:core:renderJavadoc'.
> java.io.FileNotFoundException:
> /mnt/hdd/repo/lucene-solr/lucene/core/build/tmp/renderJavadoc/javadoc-options.txt
> (そのようなファイルやディレクトリはありません)
{code}
> Incorporate all :precommit tasks into :check
>
>
> Key: LUCENE-9316
> URL: https://issues.apache.org/jira/browse/LUCENE-9316
> Project: Lucene - Core
> Issue Type: Sub-task
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-9316) Incorporate all :precommit tasks into :check
[
https://issues.apache.org/jira/browse/LUCENE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081286#comment-17081286
]
Tomoko Uchida edited comment on LUCENE-9316 at 4/11/20, 1:34 PM:
-
Please ignore my previous comment, the error isn't related to parallel
execution of javadoc and renderJavadoc, but it seems something wrong with
"renderJavadoc" task.
This works for me.
{code:java}
$ ./gradlew clean
BUILD SUCCESSFUL in 1s
$ ./gradlew :lucene:core:renderJavadoc
{code}
But this does not work. (I have no idea about it for now, will look at it.)
{code:java}
$ ./gradlew clean :lucene:core:renderJavadoc
FAILURE: Build failed with an exception.
* Where:
Script '/mnt/hdd/repo/lucene-solr/gradle/render-javadoc.gradle' line: 104
* What went wrong:
Execution failed for task ':lucene:core:renderJavadoc'.
> java.io.FileNotFoundException:
> /mnt/hdd/repo/lucene-solr/lucene/core/build/tmp/renderJavadoc/javadoc-options.txt
> (そのようなファイルやディレクトリはありません)
{code}
Seems like "clean" and "renderJavadoc" are executed in parallel here ?
was (Author: tomoko uchida):
Please ignore my previous comment, the error isn't related to parallel
execution of javadoc and renderJavadoc, but it seems something wrong with
"renderJavadoc" task.
This works for me.
{code:java}
$ ./gradlew clean
BUILD SUCCESSFUL in 1s
$ ./gradlew :lucene:core:renderJavadoc
{code}
But this does not work. (I have no idea about it for now, will look at it.)
{code:java}
$ ./gradlew clean :lucene:core:renderJavadoc
FAILURE: Build failed with an exception.
* Where:
Script '/mnt/hdd/repo/lucene-solr/gradle/render-javadoc.gradle' line: 104
* What went wrong:
Execution failed for task ':lucene:core:renderJavadoc'.
> java.io.FileNotFoundException:
> /mnt/hdd/repo/lucene-solr/lucene/core/build/tmp/renderJavadoc/javadoc-options.txt
> (そのようなファイルやディレクトリはありません)
{code}
> Incorporate all :precommit tasks into :check
>
>
> Key: LUCENE-9316
> URL: https://issues.apache.org/jira/browse/LUCENE-9316
> Project: Lucene - Core
> Issue Type: Sub-task
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9305) Sometimes ConcurrentModificationException occurs in QueryNodeImpl.set
[
https://issues.apache.org/jira/browse/LUCENE-9305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Khludnev resolved LUCENE-9305.
--
Resolution: Abandoned
> Sometimes ConcurrentModificationException occurs in QueryNodeImpl.set
> -
>
> Key: LUCENE-9305
> URL: https://issues.apache.org/jira/browse/LUCENE-9305
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/queryparser
>Affects Versions: 7.6
> Environment: Happens both on Windows and Linux, any sort of hardware
> (we have about 30 installations in productions and it happens on each)
>Reporter: Ivan Fedenkov
>Priority: Major
>
> Hello.
> Sometimes a ConcurrentModificationException occures in my application with
> such a stacktrace:
> {noformat}
> java.util.ConcurrentModificationException: null
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
> at java.util.ArrayList$Itr.next(ArrayList.java:859)
> at
> org.apache.lucene.queryparser.flexible.core.nodes.QueryNodeImpl.set(QueryNodeImpl.java:104)
> at
> org.apache.lucene.queryparser.flexible.core.processors.QueryNodeProcessorImpl.processChildren(QueryNodeProcessorImpl.java:137)
> at
> org.apache.lucene.queryparser.flexible.core.processors.QueryNodeProcessorImpl.processIteration(QueryNodeProcessorImpl.java:96)
> at
> org.apache.lucene.queryparser.flexible.core.processors.QueryNodeProcessorImpl.processChildren(QueryNodeProcessorImpl.java:124)
> at
> org.apache.lucene.queryparser.flexible.core.processors.QueryNodeProcessorImpl.processIteration(QueryNodeProcessorImpl.java:96)
> at
> org.apache.lucene.queryparser.flexible.core.processors.QueryNodeProcessorImpl.processChildren(QueryNodeProcessorImpl.java:124)
> at
> org.apache.lucene.queryparser.flexible.core.processors.QueryNodeProcessorImpl.processIteration(QueryNodeProcessorImpl.java:96)
> at
> org.apache.lucene.queryparser.flexible.core.processors.QueryNodeProcessorImpl.processChildren(QueryNodeProcessorImpl.java:124)
> at
> org.apache.lucene.queryparser.flexible.core.processors.QueryNodeProcessorImpl.processIteration(QueryNodeProcessorImpl.java:96)
> at
> org.apache.lucene.queryparser.flexible.core.processors.QueryNodeProcessorImpl.processChildren(QueryNodeProcessorImpl.java:124)
> at
> org.apache.lucene.queryparser.flexible.core.processors.QueryNodeProcessorImpl.processIteration(QueryNodeProcessorImpl.java:96)
> at
> org.apache.lucene.queryparser.flexible.core.processors.QueryNodeProcessorImpl.process(QueryNodeProcessorImpl.java:89)
> at
> org.apache.lucene.queryparser.flexible.core.processors.QueryNodeProcessorPipeline.process(QueryNodeProcessorPipeline.java:89)
> at
> org.apache.lucene.queryparser.flexible.core.QueryParserHelper.parse(QueryParserHelper.java:250)
> at
> org.apache.lucene.queryparser.flexible.standard.StandardQueryParser.parse(StandardQueryParser.java:158)
> {noformat}
> It happens very rare, I would say in 0.01% of all query parsing.
> I tried to reproduce it with concurrent tests, by have no success.
> My Lucene version is 7.6.0
> QueryNodeImpl.set(QueryNodeImpl.java:104) looks like this on my version
> {noformat}
> // reset parent value
> for (QueryNode child : children) {
> child.removeFromParent();
> }
> {noformat}
> Could I get any help, please?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9316) Incorporate all :precommit tasks into :check
[
https://issues.apache.org/jira/browse/LUCENE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081323#comment-17081323
]
Tomoko Uchida commented on LUCENE-9316:
---
[~dweiss] I was able to reproduce the same error you shared here. I had to
execute "clean" on separate command to do so due to the problem I mentioned
above.
{code:java}
$ ./gradlew clean
$ ./gradlew check -x test --console=plain -Pvalidation.git.failOnModified=false
> Task :lucene:facet:javadoc
FAILURE: Build completed with 3 failures.
1: Task failed with an exception.
---
* Where:
Script '/mnt/hdd/repo/lucene-solr/gradle/render-javadoc.gradle' line: 137
* What went wrong:
Execution failed for task ':lucene:classification:renderJavadoc'.
> Javadoc generation failed for :lucene:classification,
Options file at:
/mnt/hdd/repo/lucene-solr/lucene/classification/build/tmp/renderJavadoc/javadoc-options.txt
Command output at:
/mnt/hdd/repo/lucene-solr/lucene/classification/build/tmp/renderJavadoc/javadoc-output.txt
..
$ cat
/mnt/hdd/repo/lucene-solr/lucene/classification/build/tmp/renderJavadoc/javadoc-output.txt
Loading source files for package org.apache.lucene...
Constructing Javadoc information...
javadoc: error - Error reading file:
/mnt/hdd/repo/lucene-solr/lucene/grouping/build/docs/javadoc/element-list
{code}
I'm not quite sure that if this is caused by parallel execution, but it may be
possible that while "javadoc" task holds a write lock for the "element-list"
file "renderJavadoc" try to read the same file at the same time.
> Incorporate all :precommit tasks into :check
>
>
> Key: LUCENE-9316
> URL: https://issues.apache.org/jira/browse/LUCENE-9316
> Project: Lucene - Core
> Issue Type: Sub-task
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9316) Incorporate all :precommit tasks into :check
[
https://issues.apache.org/jira/browse/LUCENE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081364#comment-17081364
]
Tomoko Uchida commented on LUCENE-9316:
---
Update: *precommit fails even when I don't apply the patch on
"precommit.gradle" here.*
{code}
$ ./gradlew precommit -Pvalidation.git.failOnModified=false
FAILURE: Build failed with an exception.
$ cat
/mnt/hdd/repo/lucene-solr/lucene/benchmark/build/tmp/renderJavadoc/javadoc-output.txt
Loading source files for package org.apache.lucene...
Constructing Javadoc information...
javadoc: error - Error reading file:
/mnt/hdd/repo/lucene-solr/lucene/highlighter/build/docs/javadoc/element-list
{code}
There should be something wrong with renderJavadoc task (when it called as a
dependency from precommit) but I've not yet figured out the cause...
> Incorporate all :precommit tasks into :check
>
>
> Key: LUCENE-9316
> URL: https://issues.apache.org/jira/browse/LUCENE-9316
> Project: Lucene - Core
> Issue Type: Sub-task
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Issue Comment Deleted] (LUCENE-9316) Incorporate all :precommit tasks into :check
[
https://issues.apache.org/jira/browse/LUCENE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomoko Uchida updated LUCENE-9316:
--
Comment: was deleted
(was: Update: *precommit fails even when I don't apply the patch on
"precommit.gradle" here.*
{code}
$ ./gradlew precommit -Pvalidation.git.failOnModified=false
FAILURE: Build failed with an exception.
$ cat
/mnt/hdd/repo/lucene-solr/lucene/benchmark/build/tmp/renderJavadoc/javadoc-output.txt
Loading source files for package org.apache.lucene...
Constructing Javadoc information...
javadoc: error - Error reading file:
/mnt/hdd/repo/lucene-solr/lucene/highlighter/build/docs/javadoc/element-list
{code}
There should be something wrong with renderJavadoc task (when it called as a
dependency from precommit) but I've not yet figured out the cause...
)
> Incorporate all :precommit tasks into :check
>
>
> Key: LUCENE-9316
> URL: https://issues.apache.org/jira/browse/LUCENE-9316
> Project: Lucene - Core
> Issue Type: Sub-task
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9316) Incorporate all :precommit tasks into :check
[
https://issues.apache.org/jira/browse/LUCENE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081376#comment-17081376
]
Tomoko Uchida commented on LUCENE-9316:
---
Yes it's surely caused by parallel execution of javadoc and renderJavadoc. When
I intentionally added "javadoc" as a dependency for "renderJavadoc" by this
patch to execute them sequentially, the check task succeeded without {{-x
javadoc}} option for me.
{code}
$ git diff
diff --git a/gradle/render-javadoc.gradle b/gradle/render-javadoc.gradle
index bd2278919ac..a3e5d4cd2c5 100644
--- a/gradle/render-javadoc.gradle
+++ b/gradle/render-javadoc.gradle
@@ -28,6 +28,8 @@ allprojects {
description "Generates Javadoc API documentation for the main source
code. This directly invokes javadoc tool."
group "documentation"
+ dependsOn "javadoc"
+
ext {
linksource = false
linkJUnit = false
diff --git a/gradle/validation/precommit.gradle
b/gradle/validation/precommit.gradle
index 94ed800cb28..8bfff2ae43a 100644
--- a/gradle/validation/precommit.gradle
+++ b/gradle/validation/precommit.gradle
@@ -44,5 +44,7 @@ configure(rootProject) {
]}
}
}
+
+ check.dependsOn precommit
}
{code}
> Incorporate all :precommit tasks into :check
>
>
> Key: LUCENE-9316
> URL: https://issues.apache.org/jira/browse/LUCENE-9316
> Project: Lucene - Core
> Issue Type: Sub-task
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7788) fail precommit on unparameterised log.trace messages
[
https://issues.apache.org/jira/browse/LUCENE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081387#comment-17081387
]
Erick Erickson commented on LUCENE-7788:
[~cpoerschke] [~mbraun688] [~jpountz] [~dsmiley] [~varun] Spamming all of
you from several JIRAs since you've all commented. Trying to centralize here.
I looked at trying to use the current validate source patterns, but it operates
on regexes. I don't think regexes can do what we want here. And even if one
could be created, it'd be horrible to maintain.
Here's what I've got. I wrote a new groovy script we could include as a new
Gradle task (part of "check"?) that
1> flags all logger declarations that are not exactly "log". If we fix that,
then we can check for log.info() etc. blindly
2> Collects all log statements (including multi-line), strips all the quoted
text out and examines the remainder for "+" or function calls and dumps the
line out if found. This would be a failure going forward.
3> does _NOT_ do <2> if the log line has //ok or // ok etc. (case insensitive).
Or maybe we want "//function calls cheap" or "//calls cheap". Hmm, thinking
about it "//calls cheap" at least gives a hint whereas "//ok" gives no clue so
I'll propose that as the tag.
<3> addresses David's comments elsewhere that we should leave it to the
programmer's discretion if a log message that does a function call is "cheap
enough". If we don't adopt some convention like this, then this stuff will
creep back.
So far, there are 4,325 lines flagged, and the run takes less than 3 seconds so
I think it's reasonable to add to check. NOTE: I intend to add this to the
Gradle build only. I'll clean up what I find for 8x, and if some things creep
back in over the next few months, I don't mind if we catch them later.
So first I want people's opinions on
* whether adding a tag to verified log messages that add function calls is the
way to go here. It's kind of clumsy, but it takes care of both littering the
code with a ton of "if (log.isDebugEnabled){}" clauses and allowing expensive,
unchecked log messages to creep back in the code. Ok, it litters the code with
comments I concede
* Whether adding this to check is reasonable
If there's consensus, I'll start moving this forward. WARNING: this will mean
there are a _LOT_ of code changes, I'll try to do them a couple dozen files at
a time. Once it's clean, I'll add the Gradle task if that's the consensus. I'm
sure I'll find things that aren't currently caught, but before going there I
want to be sure the work won't be wasted.
Let me know what you all think.
> fail precommit on unparameterised log.trace messages
>
>
> Key: LUCENE-7788
> URL: https://issues.apache.org/jira/browse/LUCENE-7788
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: LUCENE-7788.patch, LUCENE-7788.patch
>
>
> SOLR-10415 would be removing existing unparameterised log.trace messages use
> and once that is in place then this ticket's one-line change would be for
> 'ant precommit' to reject any future unparameterised log.trace message use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9309) IW#addIndices(CodecReader) might delete files concurrently to IW#rollback
[
https://issues.apache.org/jira/browse/LUCENE-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081393#comment-17081393
]
ASF subversion and git services commented on LUCENE-9309:
-
Commit 8c1f9815dbb01f0ac0b74b6fb0f4b07015105ab9 in lucene-solr's branch
refs/heads/master from Simon Willnauer
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=8c1f981 ]
LUCENE-9309: ensure stopMerges is set under IW lock
> IW#addIndices(CodecReader) might delete files concurrently to IW#rollback
> -
>
> Key: LUCENE-9309
> URL: https://issues.apache.org/jira/browse/LUCENE-9309
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Simon Willnauer
>Assignee: Simon Willnauer
>Priority: Major
> Fix For: master (9.0), 8.6
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> During work on LUCENE-9304 [~mikemccand] ran into a failure:
> {noformat}
> org.apache.lucene.index.TestAddIndexes > test suite's output saved to
> /home/mike/src/simon/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.index.TestAddIndexes.txt,
> copied below:
>> java.nio.file.NoSuchFileException:
> _gt_Lucene85FieldsIndex-doc_ids_6u.tmp
>> at
> __randomizedtesting.SeedInfo.seed([4760FA81FBD4B2CE:A147156E5F7BD9B0]:0)
>> at
> org.apache.lucene.store.ByteBuffersDirectory.deleteFile(ByteBuffersDirectory.java:148)
>> at
> org.apache.lucene.store.MockDirectoryWrapper.deleteFile(MockDirectoryWrapper.java:607)
>> at
> org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
>> at
> org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696)
>> at
> org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690)
>> at
> org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:449)
>> at
> org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2334)
>> at
> org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2275)
>> at
> org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2268)
>> at
> org.apache.lucene.index.TestAddIndexes.testAddIndexesWithRollback(TestAddIndexes.java:974)
> 2> NOTE: reproduce with: ant test -Dtestcase=TestAddIndexes
> -Dtests.method=testAddIndexesWithRollback -Dtests.seed=4760FA81FBD4B2CE
> -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=fr-GP -Dtests.t\
> imezone=Asia/Tbilisi -Dtests.asserts=true -Dtests.file.encoding=UTF-8
> 2> NOTE: test params are: codec=Asserting(Lucene84):
> {c=PostingsFormat(name=LuceneFixedGap),
> id=PostingsFormat(name=LuceneFixedGap),
> f1=PostingsFormat(name=LuceneFixedGap), f2=BlockTreeOrds(blocksize=128)\
> , version=BlockTreeOrds(blocksize=128), content=FST50},
> docValues:{dv=DocValuesFormat(name=Lucene80),
> soft_delete=DocValuesFormat(name=Lucene80),
> doc=DocValuesFormat(name=Lucene80), id=DocValuesFormat(name=\
> Asserting), content=DocValuesFormat(name=Asserting),
> doc2d=DocValuesFormat(name=Lucene80)}, maxPointsInLeafNode=982,
> maxMBSortInHeap=5.837219998050092,
> sim=Asserting(org.apache.lucene.search.similarities.As\
> sertingSimilarity@6ce38471), locale=fr-GP, timezone=Asia/Tbilisi
> {noformat}
> While this unfortunately doesn't reproduce it's likely a bug that exists for
> quite some time but never showed up until LUCENE-9147 which uses a temporary
> output. That's fine but with IW#addIndices(CodecReader...) not registering
> the merge it does in the IW we never wait for the merge to finish while
> rollback and if that merge finishes concurrently it will also remove these
> .tmp files.
> There are many ways to fix this and I can work on a patch, but hey do we
> really need to be able to add indices while we index and do that on an open
> and live IW or can it be a tool on top of it?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9309) IW#addIndices(CodecReader) might delete files concurrently to IW#rollback
[
https://issues.apache.org/jira/browse/LUCENE-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081394#comment-17081394
]
ASF subversion and git services commented on LUCENE-9309:
-
Commit c54885fb99c64d2f520b5a4a4bc10440c88317be in lucene-solr's branch
refs/heads/branch_8x from Simon Willnauer
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c54885f ]
LUCENE-9309: ensure stopMerges is set under IW lock
> IW#addIndices(CodecReader) might delete files concurrently to IW#rollback
> -
>
> Key: LUCENE-9309
> URL: https://issues.apache.org/jira/browse/LUCENE-9309
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Simon Willnauer
>Assignee: Simon Willnauer
>Priority: Major
> Fix For: master (9.0), 8.6
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> During work on LUCENE-9304 [~mikemccand] ran into a failure:
> {noformat}
> org.apache.lucene.index.TestAddIndexes > test suite's output saved to
> /home/mike/src/simon/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.index.TestAddIndexes.txt,
> copied below:
>> java.nio.file.NoSuchFileException:
> _gt_Lucene85FieldsIndex-doc_ids_6u.tmp
>> at
> __randomizedtesting.SeedInfo.seed([4760FA81FBD4B2CE:A147156E5F7BD9B0]:0)
>> at
> org.apache.lucene.store.ByteBuffersDirectory.deleteFile(ByteBuffersDirectory.java:148)
>> at
> org.apache.lucene.store.MockDirectoryWrapper.deleteFile(MockDirectoryWrapper.java:607)
>> at
> org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
>> at
> org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696)
>> at
> org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690)
>> at
> org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:449)
>> at
> org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2334)
>> at
> org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2275)
>> at
> org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2268)
>> at
> org.apache.lucene.index.TestAddIndexes.testAddIndexesWithRollback(TestAddIndexes.java:974)
> 2> NOTE: reproduce with: ant test -Dtestcase=TestAddIndexes
> -Dtests.method=testAddIndexesWithRollback -Dtests.seed=4760FA81FBD4B2CE
> -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=fr-GP -Dtests.t\
> imezone=Asia/Tbilisi -Dtests.asserts=true -Dtests.file.encoding=UTF-8
> 2> NOTE: test params are: codec=Asserting(Lucene84):
> {c=PostingsFormat(name=LuceneFixedGap),
> id=PostingsFormat(name=LuceneFixedGap),
> f1=PostingsFormat(name=LuceneFixedGap), f2=BlockTreeOrds(blocksize=128)\
> , version=BlockTreeOrds(blocksize=128), content=FST50},
> docValues:{dv=DocValuesFormat(name=Lucene80),
> soft_delete=DocValuesFormat(name=Lucene80),
> doc=DocValuesFormat(name=Lucene80), id=DocValuesFormat(name=\
> Asserting), content=DocValuesFormat(name=Asserting),
> doc2d=DocValuesFormat(name=Lucene80)}, maxPointsInLeafNode=982,
> maxMBSortInHeap=5.837219998050092,
> sim=Asserting(org.apache.lucene.search.similarities.As\
> sertingSimilarity@6ce38471), locale=fr-GP, timezone=Asia/Tbilisi
> {noformat}
> While this unfortunately doesn't reproduce it's likely a bug that exists for
> quite some time but never showed up until LUCENE-9147 which uses a temporary
> output. That's fine but with IW#addIndices(CodecReader...) not registering
> the merge it does in the IW we never wait for the merge to finish while
> rollback and if that merge finishes concurrently it will also remove these
> .tmp files.
> There are many ways to fix this and I can work on a patch, but hey do we
> really need to be able to add indices while we index and do that on an open
> and live IW or can it be a tool on top of it?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9309) IW#addIndices(CodecReader) might delete files concurrently to IW#rollback
[
https://issues.apache.org/jira/browse/LUCENE-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081395#comment-17081395
]
Simon Willnauer commented on LUCENE-9309:
-
the push above was to fix a test failure we have on elastic CI:
{noformat}
15:10:24[junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging
15:10:24[junit4] 1>
org.apache.lucene.index.MergePolicy$MergeAbortedException: merge is aborted:
_2u(9.0.0):c199/18:[diagnostics={source=flush, os.arch=amd64,
java.runtime.version=11.0.2+7, os.version=4.15.0-1060-gcp, os=Linux,
java.vendor=Oracle Corporation, timestamp=1586617824513, java.version=11.0.2,
java.vm.version=11.0.2+7,
lucene.version=9.0.0}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=1
_2x(9.0.0):c425:[diagnostics={source=flush, os.arch=amd64,
java.runtime.version=11.0.2+7, os.version=4.15.0-1060-gcp, os=Linux,
java.vendor=Oracle Corporation, timestamp=1586617824590, java.version=11.0.2,
java.vm.version=11.0.2+7,
lucene.version=9.0.0}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.registerMerge(IndexWriter.java:4125)
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:2203)
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2160)
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5142)
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.maybeProcessEvents(IndexWriter.java:5132)
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1360)
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1647)
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1289)
15:10:24[junit4] 1> at
org.apache.lucene.index.TestIndexWriterMerging$1.run(TestIndexWriterMerging.java:416)
15:10:24[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestIndexWriterMerging -Dtests.method=testNoWaitClose
-Dtests.seed=1D4119EDE2022625 -Dtests.slow=true -Dtests.badapples=true
-Dtests.locale=sv-AX -Dtests.timezone=Europe/Busingen -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
15:10:24[junit4] ERROR 0.90s J2 | TestIndexWriterMerging.testNoWaitClose
<<<
15:10:24[junit4]> Throwable #1:
org.apache.lucene.index.MergePolicy$MergeAbortedException: merge is aborted:
_2u(9.0.0):c199/18:[diagnostics={source=flush, os.arch=amd64,
java.runtime.version=11.0.2+7, os.version=4.15.0-1060-gcp, os=Linux,
java.vendor=Oracle Corporation, timestamp=1586617824513, java.version=11.0.2,
java.vm.version=11.0.2+7,
lucene.version=9.0.0}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=1
_2x(9.0.0):c425:[diagnostics={source=flush, os.arch=amd64,
java.runtime.version=11.0.2+7, os.version=4.15.0-1060-gcp, os=Linux,
java.vendor=Oracle Corporation, timestamp=1586617824590, java.version=11.0.2,
java.vm.version=11.0.2+7,
lucene.version=9.0.0}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
15:10:24[junit4]> at
__randomizedtesting.SeedInfo.seed([1D4119EDE2022625:3AB0B83C3114F6D1]:0)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.registerMerge(IndexWriter.java:4125)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:2203)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2160)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5142)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.maybeProcessEvents(IndexWriter.java:5132)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1360)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1647)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1289)
15:10:24[junit4]> at
org.apache.lucene.index.TestIndexWriterMerging$1.run(TestIndexWriterMerging.java:416)
15:10:24[junit4] 2> NOTE: test params are: codec=Asserting(Lucene84),
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@285b2588),
locale=sv-AX, timezone=Europe/Busingen
15:10:24[junit4] 2> NOTE: Linux 4.15.0-1060-gcp amd64/Oracle Corporation
11.0.2 (64-bit)/cpus=32,threads=1,free=387385984,total=536870912
15:10:24[junit4] 2> NOTE: All tests run in this JVM:
[TestNRTReaderCleanup, TestMutableValues, TestDistributionSPL,
TestBinaryDocument, TestRollback, TestXYMultiPolygonShapeQueries,
TestTwoPhaseCommitTool, TestDocID
[jira] [Commented] (SOLR-9909) Nuke one of DefaultSolrThreadFactory and SolrjNamedThreadFactory
[
https://issues.apache.org/jira/browse/SOLR-9909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081529#comment-17081529
]
Lucene/Solr QA commented on SOLR-9909:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 24 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 0m 2s{color} | {color:green} Release audit (RAT) rat-sources
passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 0m 3s{color} | {color:green} Release audit (RAT) rat-sources
passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 0m 3s{color} | {color:green} Release audit (RAT) rat-sources
passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 0m 3s{color} | {color:green} Release audit (RAT) rat-sources
passed {color} |
| {color:red}-1{color} | {color:red} Release audit (RAT) {color} | {color:red}
0m 4s{color} | {color:red} Release audit (RAT) rat-sources failed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 0m 3s{color} | {color:green} Release audit (RAT) rat-sources
passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 0m 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 0m 2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
3s{color} | {color:green} tools in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
52s{color} | {color:green} analytics in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} prometheus-exporter in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 45m 13s{color}
| {color:red} core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
32s{color} | {color:green} solrj in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} test-framework in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | solr.search.TestRecovery |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-9909 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12999628/SOLR-9909.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene1-us-west 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 8c1f9815dbb |
| ant | version: Apache Ant(TM) version 1.10.5 compiled on March 28 2019 |
| Default Java | LTS |
| Release audit (RAT) |
https://builds.apache.org/job/PreCommit-SOLR-Build/734/artifact/out/patch-rat-sources-solr_solrj.txt
|
| unit |
https://builds.apache.org/job/PreCommit-SOLR-Build/734/artifact/out/patch-unit-solr_core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-SOLR-Build/734/testReport/ |
| modules | C: . lucene/tools solr/contrib/analytics
solr/contrib/prometheus-exporter solr/core solr/solrj solr/test-framework U: . |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/734/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Nuke one of DefaultSolrThreadFactory and SolrjNamedThreadFactory
>
>
> Key: SOLR-9909
> URL: https://issues.
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1351: LUCENE-9280: Collectors to skip noncompetitive documents
mayya-sharipova commented on a change in pull request #1351: LUCENE-9280:
Collectors to skip noncompetitive documents
URL: https://github.com/apache/lucene-solr/pull/1351#discussion_r407107952
##
File path: lucene/core/src/java/org/apache/lucene/search/TopFieldCollector.java
##
@@ -302,20 +312,27 @@ private TopFieldCollector(FieldValueHitQueue pq,
int numHits,
this.numHits = numHits;
this.hitsThresholdChecker = hitsThresholdChecker;
this.numComparators = pq.getComparators().length;
-FieldComparator fieldComparator = pq.getComparators()[0];
+FieldComparator firstComparator = pq.getComparators()[0];
int reverseMul = pq.reverseMul[0];
-if
(fieldComparator.getClass().equals(FieldComparator.RelevanceComparator.class)
+if
(firstComparator.getClass().equals(FieldComparator.RelevanceComparator.class)
&& reverseMul == 1 // if the natural sort is preserved (sort by
descending relevance)
&& hitsThresholdChecker.getHitsThreshold() != Integer.MAX_VALUE) {
- firstComparator = (FieldComparator.RelevanceComparator) fieldComparator;
+ relevanceComparator = (FieldComparator.RelevanceComparator)
firstComparator;
scoreMode = ScoreMode.TOP_SCORES;
canSetMinScore = true;
} else {
- firstComparator = null;
+ relevanceComparator = null;
scoreMode = needsScores ? ScoreMode.COMPLETE :
ScoreMode.COMPLETE_NO_SCORES;
canSetMinScore = false;
}
this.minScoreAcc = minScoreAcc;
+
+if ((firstComparator instanceof FilteringFieldComparator) &&
(hitsThresholdChecker.getHitsThreshold() != Integer.MAX_VALUE)) {
Review comment:
@romseygeek Addressed in 719882e
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1351: LUCENE-9280: Collectors to skip noncompetitive documents
mayya-sharipova commented on a change in pull request #1351: LUCENE-9280:
Collectors to skip noncompetitive documents
URL: https://github.com/apache/lucene-solr/pull/1351#discussion_r407108032
##
File path: lucene/core/src/java/org/apache/lucene/search/ScoreMode.java
##
@@ -51,10 +59,51 @@ public boolean needsScores() {
public boolean needsScores() {
return true;
}
+@Override
+public boolean isExhaustive() {
+ return false;
+}
+ },
+
+ /**
+ * ScoreMode for top field collectors that can provide their own iterators,
+ * to optionally allow to skip for non-competitive docs
+ */
+ TOP_DOCS {
+@Override
+public boolean needsScores() {
+ return false;
+}
+@Override
+public boolean isExhaustive() {
+ return false;
+}
+ },
+
+ /**
+ * ScoreMode for top field collectors that can provide their own iterators,
+ * to optionally allow to skip for non-competitive docs.
+ * This mode is used when there is a secondary sort by _score.
+ */
+ TOP_DOCS_WITH_SCORES {
+@Override
+public boolean needsScores() {
+ return true;
+}
+@Override
+public boolean isExhaustive() {
+ return false;
+}
};
/**
* Whether this {@link ScoreMode} needs to compute scores.
*/
public abstract boolean needsScores();
+
+ /**
+ * Returns {@code true} if for this {@link ScoreMode} it is necessary to
process all documents,
+ * or {@code false} if is enough to go through top documents only.
+ */
+ public abstract boolean isExhaustive();
Review comment:
@romseygeek Thanks, a very good suggestion, this indeed made the code more
clear and less verbose. Addressed in 719882e
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1351: LUCENE-9280: Collectors to skip noncompetitive documents
mayya-sharipova commented on a change in pull request #1351: LUCENE-9280:
Collectors to skip noncompetitive documents
URL: https://github.com/apache/lucene-solr/pull/1351#discussion_r407108607
##
File path:
lucene/core/src/test/org/apache/lucene/search/TestSortOptimization.java
##
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.search;
+
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.FloatDocValuesField;
+import org.apache.lucene.document.LongPoint;
+import org.apache.lucene.document.FloatPoint;
+import org.apache.lucene.document.NumericDocValuesField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.util.LuceneTestCase;
+
+import java.io.IOException;
+
+public class TestSortOptimization extends LuceneTestCase {
+
+ public void testLongSortOptimization() throws IOException {
+final Directory dir = newDirectory();
+final IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig());
+final int numDocs = atLeast(1);
+for (int i = 0; i < numDocs; ++i) {
+ final Document doc = new Document();
+ doc.add(new NumericDocValuesField("my_field", i));
+ doc.add(new LongPoint("my_field", i));
+ writer.addDocument(doc);
+}
+final IndexReader reader = DirectoryReader.open(writer);
+IndexSearcher searcher = new IndexSearcher(reader);
+final SortField sortField = new SortField("my_field", SortField.Type.LONG);
+sortField.allowFilterNonCompetitveDocs();
Review comment:
@romseygeek Thanks for the suggestion. It is difficult to wrap `SortField`
into another class that extends `SortField`, as `SortField` is a concrete class
with several constructors.
Instead, I am setting `sort.fields[0].allowFilterNonCompetitveDocs()` when
we create a `TopFieldCollector` with a set hitsTreshold.
Partially addressed in 719882e
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1351: LUCENE-9280: Collectors to skip noncompetitive documents
mayya-sharipova commented on a change in pull request #1351: LUCENE-9280:
Collectors to skip noncompetitive documents
URL: https://github.com/apache/lucene-solr/pull/1351#discussion_r407108607
##
File path:
lucene/core/src/test/org/apache/lucene/search/TestSortOptimization.java
##
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.search;
+
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.FloatDocValuesField;
+import org.apache.lucene.document.LongPoint;
+import org.apache.lucene.document.FloatPoint;
+import org.apache.lucene.document.NumericDocValuesField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.util.LuceneTestCase;
+
+import java.io.IOException;
+
+public class TestSortOptimization extends LuceneTestCase {
+
+ public void testLongSortOptimization() throws IOException {
+final Directory dir = newDirectory();
+final IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig());
+final int numDocs = atLeast(1);
+for (int i = 0; i < numDocs; ++i) {
+ final Document doc = new Document();
+ doc.add(new NumericDocValuesField("my_field", i));
+ doc.add(new LongPoint("my_field", i));
+ writer.addDocument(doc);
+}
+final IndexReader reader = DirectoryReader.open(writer);
+IndexSearcher searcher = new IndexSearcher(reader);
+final SortField sortField = new SortField("my_field", SortField.Type.LONG);
+sortField.allowFilterNonCompetitveDocs();
Review comment:
@romseygeek Thanks for the suggestion. It makes sense.
It is difficult to wrap `SortField` into another class that extends
`SortField`, as `SortField` is a concrete class with several constructors.
Instead, I am setting `sort.fields[0].allowFilterNonCompetitveDocs()` when
we create a `TopFieldCollector` with a set hitsTreshold. What do you think of
this implementation?
Partially addressed in 719882e
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14291) OldAnalyticsRequestConverter should support fields names with dots
[
https://issues.apache.org/jira/browse/SOLR-14291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Khludnev updated SOLR-14291:
Attachment: SOLR-14291.patch
Status: Patch Available (was: Patch Available)
Attaching brackets around all param as non capturing groups. Also, there's a
slight test coverage for the case we encounter.
[~houston] is it good enough to commit?
> OldAnalyticsRequestConverter should support fields names with dots
> --
>
> Key: SOLR-14291
> URL: https://issues.apache.org/jira/browse/SOLR-14291
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search, SearchComponents - other
>Reporter: Anatolii Siuniaev
>Priority: Trivial
> Attachments: SOLR-14291.patch, SOLR-14291.patch, SOLR-14291.patch
>
>
> If you send a query with range facets using old olap-style syntax (see pdf
> [here|https://issues.apache.org/jira/browse/SOLR-5302]),
> OldAnalyticsRequestConverter just silently (no exception thrown) omits
> parameters like
> {code:java}
> olap..rangefacet..start
> {code}
> in case if __ has dots inside (for instance field name is
> _Project.Value_). And thus no range facets are returned in response.
> Probably the same happens in case of field faceting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (LUCENE-7788) fail precommit on unparameterised log.trace messages
[ https://issues.apache.org/jira/browse/LUCENE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson reassigned LUCENE-7788: -- Assignee: Erick Erickson (was: Christine Poerschke) > fail precommit on unparameterised log.trace messages > > > Key: LUCENE-7788 > URL: https://issues.apache.org/jira/browse/LUCENE-7788 > Project: Lucene - Core > Issue Type: Task >Reporter: Christine Poerschke >Assignee: Erick Erickson >Priority: Minor > Attachments: LUCENE-7788.patch, LUCENE-7788.patch > > > SOLR-10415 would be removing existing unparameterised log.trace messages use > and once that is in place then this ticket's one-line change would be for > 'ant precommit' to reject any future unparameterised log.trace message use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] ErickErickson opened a new pull request #1425: LUCENE-7788
ErickErickson opened a new pull request #1425: LUCENE-7788 URL: https://github.com/apache/lucene-solr/pull/1425 Sample of what cleaning up the logging calls looks like to see if we want to pursue this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7788) fail precommit on unparameterised log.trace messages
[ https://issues.apache.org/jira/browse/LUCENE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081600#comment-17081600 ] Erick Erickson commented on LUCENE-7788: I just attached a PR, you may have to expand the issue links above, but here it is so it's easy to see. [https://github.com/apache/lucene-solr/pull/1425] It's a bit ugly frankly. Constructs like: ((currentI % sliceCount) + 1) are flagged as function calls due to the parens and the + sign. Even in cases where you surround the call with an "if (log.isInfoEnabled)", you still have to add the //verified ok (I changed the verbiage, I think I'll change it even further to just //verified). Anyway, I'm done pending consensus, I wanted to see what it looked like to change a directory and chose *solr/test-framework/src/java/org/apache/solr/cloud* arbitrarily. Gives us something concrete to talk about. > fail precommit on unparameterised log.trace messages > > > Key: LUCENE-7788 > URL: https://issues.apache.org/jira/browse/LUCENE-7788 > Project: Lucene - Core > Issue Type: Task >Reporter: Christine Poerschke >Assignee: Erick Erickson >Priority: Minor > Attachments: LUCENE-7788.patch, LUCENE-7788.patch > > Time Spent: 10m > Remaining Estimate: 0h > > SOLR-10415 would be removing existing unparameterised log.trace messages use > and once that is in place then this ticket's one-line change would be for > 'ant precommit' to reject any future unparameterised log.trace message use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9317) Resolve package name conflicts for StandardAnalyzer to allow Java module system support
David Ryan created LUCENE-9317:
--
Summary: Resolve package name conflicts for StandardAnalyzer to
allow Java module system support
Key: LUCENE-9317
URL: https://issues.apache.org/jira/browse/LUCENE-9317
Project: Lucene - Core
Issue Type: Improvement
Components: core/other
Affects Versions: master (9.0)
Reporter: David Ryan
To allow Lucene to be modularised there are a few preparatory tasks to be
completed prior to this being possible. The Java module system requires that
jars do not use the same package name in different jars. The lucene-core and
lucene-analyzers-common both share the package
org.apache.lucene.analysis.standard.
Possible resolutions to this issue are discussed by Uwe on the mailing list
here:
[http://mail-archives.apache.org/mod_mbox/lucene-dev/202004.mbox/%3CCAM21Rt8FHOq_JeUSELhsQJH0uN0eKBgduBQX4fQKxbs49TLqzA%40mail.gmail.com%3E]
{quote}About StandardAnalyzer: Unfortunately I aggressively complained a while
back when Mike McCandless wanted to move standard analyzer out of the analysis
package into core (“for convenience”). This was a bad step, and IMHO we should
revert that or completely rename the packages and everything. The problem here
is: As the analysis services are only part of lucene-analyzers, we had to leave
the factory classes there, but move the implementation classes in core. The
package has to be the same. The only way around that is to move the analysis
factory framework also to core (I would not be against that). This would
include all factory base classes and the service loading stuff. Then we can
move standard analyzer and some of the filters/tokenizers including their
factories to core an that problem would be solved.
{quote}
There are two options here, either move factory framework into core or revert
StandardAnalyzer back to lucene-analyzers. In the email, the solution lands on
reverting back as per the task list:
{quote}Add some preparatory issues to cleanup class hierarchy: Move Analysis
SPI to core / remove StandardAnalyzer and related classes out of core back to
anaysis
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9318) Fix Codec API to not rely on package-private classes as part of changes to support java module system
David Ryan created LUCENE-9318:
--
Summary: Fix Codec API to not rely on package-private classes as
part of changes to support java module system
Key: LUCENE-9318
URL: https://issues.apache.org/jira/browse/LUCENE-9318
Project: Lucene - Core
Issue Type: Improvement
Components: core/other
Affects Versions: master (9.0)
Reporter: David Ryan
To allow Lucene to be modularised with Java module system there are a few
preparatory tasks to be completed prior to this being possible. These are
detailed by Uwe on the mailing list here:
[http://mail-archives.apache.org/mod_mbox/lucene-dev/202004.mbox/%3c0a5e01d60ff2$563f9c80$02bed580$@thetaphi.de%3e]
This task is:
{quote}Fix Codec API to not rely on package-private classes, so we can have a
completely public API with abstract classes for codecs, so stuff in
backwards-codecs does not need to have access to package private stuff in core.
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9319) Clean up Sandbox project by retiring/delete functionality or move it to Lucene core
David Ryan created LUCENE-9319:
--
Summary: Clean up Sandbox project by retiring/delete functionality
or move it to Lucene core
Key: LUCENE-9319
URL: https://issues.apache.org/jira/browse/LUCENE-9319
Project: Lucene - Core
Issue Type: Improvement
Components: core/other
Affects Versions: master (9.0)
Reporter: David Ryan
To allow Lucene to be modularised with Java module system there are a few
preparatory tasks to be completed prior to this being possible. These are
detailed by Uwe on the mailing list here:
[http://mail-archives.apache.org/mod_mbox/lucene-dev/202004.mbox/%3c0a5e01d60ff2$563f9c80$02bed580$@thetaphi.de%3e]
The lucene-sandbox currently shares package names with lucene-core which is not
allowed in the Java module system. There are two ways to deal with this.
Either prefix all packages with "sandbox" or retire the lucene-sandbox all
together. As per the email:
{quote}Cleanup sandbox to prefix all classes there with “sandbox” package and
where needed remove package-private access. If it’s needed for internal access,
WTF: Just move the stuff to core! We have a new version 9.0, so either
retire/delete Sandbox stuff or make it part of Lucene core.
{quote}
The suggested way forward is to move sandbox code to core.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9320) Enable support the Java module system by adding module-info to projects
David Ryan created LUCENE-9320:
--
Summary: Enable support the Java module system by adding
module-info to projects
Key: LUCENE-9320
URL: https://issues.apache.org/jira/browse/LUCENE-9320
Project: Lucene - Core
Issue Type: Improvement
Components: core/other
Affects Versions: master (9.0)
Reporter: David Ryan
To allow Lucene to be modularised with Java module system there are a number of
tasks to be completed prior to this being possible. These are detailed by Uwe
on the mailing list here:
[http://mail-archives.apache.org/mod_mbox/lucene-dev/202004.mbox/%3c0a5e01d60ff2$563f9c80$02bed580$@thetaphi.de%3e]
The preparatory tasks are detailed in:
# LUCENE-9317
# LUCENE-9318
# LUCENE-9319
Once these are complete and the move to Gradle is completed the following steps
are required to make Lucene modules:
{quote}As we are on Java 11 already, add module-info.java everywhere
Fix gradle build to create and test modules (Latest Gradle needed)
Migrate all META-INF/services/* to module-info.java (before doing this,
figure out of the META-INF files must stays for non-module usage, or if Java is
clever enough to also look into module descriptor to find services). We may
need all services at both locations (for module or classpath usage; we need a
build helper to check that it’s in-line)
{quote}
As an interim step it is also possible to add Automatic-module-name to
libraries to stop Java from using an invalid name. This can be done in Gradle
by adding:
{quote}ext.moduleName = "com.acme.mylibrary"
jar {
inputs.property("moduleName", moduleName)
manifest {
attributes 'Automatic-Module-Name': moduleName
}
}{{}}
{quote}
This has the advantage that if you're not ready to add module-info.java to
projects you can reserve the module name. Users of the library using modules
will not be required to update for the Java automatic name based on the jar
file name (e.g. lucene.core ) to the new name (e.g. org.apache.lucene.core )
h1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9201) Port documentation-lint task to Gradle build
[ https://issues.apache.org/jira/browse/LUCENE-9201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomoko Uchida updated LUCENE-9201: -- Attachment: task_deps1.png task_deps2.png Status: Open (was: Open) > Port documentation-lint task to Gradle build > > > Key: LUCENE-9201 > URL: https://issues.apache.org/jira/browse/LUCENE-9201 > Project: Lucene - Core > Issue Type: Sub-task >Affects Versions: master (9.0) >Reporter: Tomoko Uchida >Assignee: Tomoko Uchida >Priority: Major > Attachments: LUCENE-9201-ecj-2.patch, LUCENE-9201-ecj.patch, > LUCENE-9201-missing-docs.patch, LUCENE-9201.patch, javadocGRADLE.png, > javadocHTML4.png, javadocHTML5.png, task_deps1.png, task_deps2.png > > Time Spent: 4.5h > Remaining Estimate: 0h > > Ant build's "documentation-lint" target consists of those two sub targets. > * "-ecj-javadoc-lint" (Javadoc linting by ECJ) > * "-documentation-lint"(Missing javadocs / broken links check by python > scripts) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-9909) Nuke one of DefaultSolrThreadFactory and SolrjNamedThreadFactory
[ https://issues.apache.org/jira/browse/SOLR-9909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081655#comment-17081655 ] David Smiley commented on SOLR-9909: Cool; thanks. > Nuke one of DefaultSolrThreadFactory and SolrjNamedThreadFactory > > > Key: SOLR-9909 > URL: https://issues.apache.org/jira/browse/SOLR-9909 > Project: Solr > Issue Type: Task >Reporter: Shalin Shekhar Mangar >Priority: Trivial > Fix For: 6.7, 7.0 > > Attachments: SOLR-9909-01.patch, SOLR-9909-02.patch, > SOLR-9909-03.patch, SOLR-9909.patch, SOLR-9909.patch > > > DefaultSolrThreadFactory and SolrjNamedThreadFactory have exactly the same > code. Let's remove one of them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9201) Port documentation-lint task to Gradle build
[ https://issues.apache.org/jira/browse/LUCENE-9201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081660#comment-17081660 ] Tomoko Uchida commented on LUCENE-9201: --- Before proceeding this issue, I'd like to reorganize task dependencies around javadoc and all other Lucene/Solr documents. It can look like this (here, "checkBrokenLinks" and "documentation" tasks are still missing and "documentation" could be further split into sub-tasks) : !task_deps1.png! Or, this would be also possible: !task_deps2.png! Eather way, "javadoc" task do nothing itself but works as a placeholder for "renderJavadoc". I feel like that it'd be better that we first get the "documentation" task done and return here to finish "checkBrokenLinks" (and close this issue). Porting "documentation" task (or its equivalent) seems not to be listed in the to-do list on LUCENE-9077. A dedicated issue would be needed because it requires some extra work to decipher Ant code and port it to gradle (I just looked through the scripts and soon lost track of where i was). [~dweiss] can I open a new sub-task on LUCENE-9077, let's say "Port documentation task to gradle" ? > Port documentation-lint task to Gradle build > > > Key: LUCENE-9201 > URL: https://issues.apache.org/jira/browse/LUCENE-9201 > Project: Lucene - Core > Issue Type: Sub-task >Affects Versions: master (9.0) >Reporter: Tomoko Uchida >Assignee: Tomoko Uchida >Priority: Major > Attachments: LUCENE-9201-ecj-2.patch, LUCENE-9201-ecj.patch, > LUCENE-9201-missing-docs.patch, LUCENE-9201.patch, javadocGRADLE.png, > javadocHTML4.png, javadocHTML5.png, task_deps1.png, task_deps2.png > > Time Spent: 4.5h > Remaining Estimate: 0h > > Ant build's "documentation-lint" target consists of those two sub targets. > * "-ecj-javadoc-lint" (Javadoc linting by ECJ) > * "-documentation-lint"(Missing javadocs / broken links check by python > scripts) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14291) OldAnalyticsRequestConverter should support fields names with dots
[
https://issues.apache.org/jira/browse/SOLR-14291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081666#comment-17081666
]
Lucene/Solr QA commented on SOLR-14291:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 0m 53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 0m 53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 0m 53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
54s{color} | {color:green} analytics in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 8m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-14291 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12999668/SOLR-14291.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene2-us-west.apache.org 4.4.0-170-generic #199-Ubuntu SMP
Thu Nov 14 01:45:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 8c1f981 |
| ant | version: Apache Ant(TM) version 1.9.6 compiled on July 20 2018 |
| Default Java | LTS |
| Test Results |
https://builds.apache.org/job/PreCommit-SOLR-Build/735/testReport/ |
| modules | C: solr/contrib/analytics U: solr/contrib/analytics |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/735/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> OldAnalyticsRequestConverter should support fields names with dots
> --
>
> Key: SOLR-14291
> URL: https://issues.apache.org/jira/browse/SOLR-14291
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search, SearchComponents - other
>Reporter: Anatolii Siuniaev
>Priority: Trivial
> Attachments: SOLR-14291.patch, SOLR-14291.patch, SOLR-14291.patch
>
>
> If you send a query with range facets using old olap-style syntax (see pdf
> [here|https://issues.apache.org/jira/browse/SOLR-5302]),
> OldAnalyticsRequestConverter just silently (no exception thrown) omits
> parameters like
> {code:java}
> olap..rangefacet..start
> {code}
> in case if __ has dots inside (for instance field name is
> _Project.Value_). And thus no range facets are returned in response.
> Probably the same happens in case of field faceting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
