[GitHub] [lucene-solr] dweiss commented on pull request #1830: LUCENE-9506: Gradle: split validateSourcePatterns into per-project an…
dweiss commented on pull request #1830: URL: https://github.com/apache/lucene-solr/pull/1830#issuecomment-688661171 I'm committing it then, ok Uwe? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] iverase merged pull request #1697: LUCENE-9292: BKDWriter refactor: Group point configuration in its own class
iverase merged pull request #1697: URL: https://github.com/apache/lucene-solr/pull/1697 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9292) BKDWriter refactor: Group point configuration in its own class
[
https://issues.apache.org/jira/browse/LUCENE-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17191991#comment-17191991
]
ASF subversion and git services commented on LUCENE-9292:
-
Commit 59b17366ff45d958810b1f8e4950eebd93f1b20d in lucene-solr's branch
refs/heads/master from Ignacio Vera
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=59b1736 ]
LUCENE-9292: Refactor BKD point configuration into its own class (#1697)
> BKDWriter refactor: Group point configuration in its own class
> ---
>
> Key: LUCENE-9292
> URL: https://issues.apache.org/jira/browse/LUCENE-9292
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Ignacio Vera
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> In order to build a BKD tree, we need to provide to the {{BKDWriter}} the
> point configuration we are indexing. That is the {{number of dimensions}},
> the {{number of indexed dimension}}, the {{number of bytes per dimensions}}
> and the max {{number of points per leaf}}. From this information, we
> actually derive some important parameters like the {{number of bytes per
> point}}
>
> The idea of this change is to group all this information into its own class
> so we can share this information more easily with other objects.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1830: LUCENE-9506: Gradle: split validateSourcePatterns into per-project an…
dweiss commented on a change in pull request #1830:
URL: https://github.com/apache/lucene-solr/pull/1830#discussion_r484695767
##
File path: gradle/validation/validate-source-patterns.gradle
##
@@ -29,50 +33,127 @@ buildscript {
}
}
-configure(rootProject) {
- task("validateSourcePatterns", type: ValidateSourcePatternsTask) { task ->
+def extensions = [
+'adoc',
+'bat',
+'cmd',
+'css',
+'g4',
+'gradle',
+'groovy',
+'html',
+'java',
+'jflex',
+'jj',
+'js',
+'json',
+'md',
+'mdtext',
+'pl',
+'policy',
+'properties',
+'py',
+'sh',
+'template',
+'vm',
+'xml',
+'xsl',
+]
+
+// Create source validation task local for each project's files.
+subprojects {
+ task validateSourcePatterns(type: ValidateSourcePatternsTask) { task ->
group = 'Verification'
description = 'Validate Source Patterns'
// This task has no proper outputs.
setupDummyOutputs(task)
-sourceFiles = project.fileTree(project.rootDir) {
- [
-'java', 'jflex', 'py', 'pl', 'g4', 'jj', 'html', 'js',
-'css', 'xml', 'xsl', 'vm', 'sh', 'cmd', 'bat', 'policy',
-'properties', 'mdtext', 'groovy', 'gradle',
-'template', 'adoc', 'json',
- ].each{
-include "lucene/**/*.${it}"
-include "solr/**/*.${it}"
-include "dev-tools/**/*.${it}"
-include "gradle/**/*.${it}"
-include "*.${it}"
+sourceFiles = fileTree(projectDir) {
+ extensions.each{
+include "**/*.${it}"
}
- // TODO: For now we don't scan txt / md files, so we
- // check licenses in top-level folders separately:
- include '*.txt'
- include '*/*.txt'
- include '*.md'
- include '*/*.md'
- // excludes:
+
+ // Don't go into child projects (scanned separately).
+ childProjects.keySet().each{
+exclude "${it}/**"
+ }
+
+ // default excludes.
+ exclude 'build/**'
+}
+ }
+
+ // Add source validation to per-project checks as well.
+ check.dependsOn validateSourcePatterns
+}
+
+configure(project(':lucene:benchmark')) {
+ project.tasks.withType(ValidateSourcePatternsTask) {
+sourceFiles.exclude 'temp/**'
+sourceFiles.exclude 'work/**'
+ }
+}
+
+configure(project(':solr:core')) {
+ project.tasks.withType(ValidateSourcePatternsTask) {
+sourceFiles.exclude 'src/**/CheckLoggingConfiguration.java'
+sourceFiles.exclude 'src/test/org/apache/hadoop/**'
+ }
+}
+
+configure(rootProject) {
+ task validateRootAndOtherFiles(type: ValidateSourcePatternsTask) { task ->
+// This task has no proper outputs.
+setupDummyOutputs(task)
+
+sourceFiles = fileTree(projectDir) {
+ extensions.each{
+include "**/*.${it}"
+ }
+
+ // We do not scan for *.txt by default (broken files in subprojects),
+ // but in root we can do this).
+ include '**/*.txt'
+
+ // Don't go into child projects (scanned separately).
+ childProjects.keySet().each{
+exclude "${it}/**"
+ }
+
+ // default excludes.
exclude '**/build/**'
- exclude '**/dist/**'
exclude 'dev-tools/missing-doclet/src/**/*.java' // <-- TODO: remove
once we allow "var" on master
- exclude 'lucene/benchmark/work/**'
- exclude 'lucene/benchmark/temp/**'
- exclude '**/CheckLoggingConfiguration.java'
- exclude 'solr/core/src/test/org/apache/hadoop/**'
- exclude '**/validate-source-patterns.gradle' // ourselves :-)
+
+ // ourselves :-)
+ exclude 'gradle/validation/validate-source-patterns.gradle'
}
}
+
+ task validateSourcePatterns() {
+group = 'Verification'
+description = 'Validate Source Patterns'
+
+// Make it depend on all so-named tasks from subprojects.
Review comment:
Yes, it does work. Like I said - a task without a project qualifier is
in fact an invocation of any such task in any subproject. So:
```
> gradlew -p lucene/analysis validateSourcePatterns --console=plain
...
> Task :lucene:analysis:validateSourcePatterns UP-TO-DATE
> Task :lucene:analysis:phonetic:validateSourcePatterns UP-TO-DATE
> Task :lucene:analysis:opennlp:validateSourcePatterns UP-TO-DATE
> Task :lucene:analysis:stempel:validateSourcePatterns UP-TO-DATE
> Task :lucene:analysis:morfologik:validateSourcePatterns UP-TO-DATE
> Task :lucene:analysis:smartcn:validateSourcePatterns UP-TO-DATE
> Task :lucene:analysis:nori:validateSourcePatterns UP-TO-DATE
> Task :lucene:analysis:kuromoji:validateSourcePatterns UP-TO-DATE
> Task :lucene:analysis:icu:validateSourcePatterns UP-TO-DATE
> Task :lucene:analysis:common:validateSourcePatterns UP-TO-DATE
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For
[GitHub] [lucene-solr] uschindler commented on pull request #1830: LUCENE-9506: Gradle: split validateSourcePatterns into per-project an…
uschindler commented on pull request #1830: URL: https://github.com/apache/lucene-solr/pull/1830#issuecomment-688665608 I don't think that's the case. Running the whole thing a second time is quite fast. I remember that all those filters in Maven and Ant are based on the same code pattern (DirectoryScanner from Ant or Maven's Plexus Utils), which has two scanning modes, GM fast and slow. The slow one is dumb but also collects information about files filtered out. At least for includes those are intelligent. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1830: LUCENE-9506: Gradle: split validateSourcePatterns into per-project an…
uschindler commented on pull request #1830: URL: https://github.com/apache/lucene-solr/pull/1830#issuecomment-68890 Ok merge it. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] uschindler edited a comment on pull request #1830: LUCENE-9506: Gradle: split validateSourcePatterns into per-project an…
uschindler edited a comment on pull request #1830: URL: https://github.com/apache/lucene-solr/pull/1830#issuecomment-688665608 I don't think that's the case. Running the whole thing a second time is quite fast. I remember that all those filters in Maven and Ant are based on the same code pattern (DirectoryScanner from Ant or Maven's Plexus Utils), which has two scanning modes, fast and slow. The slow one is dumb but also collects information about files filtered out. At least for includes those are intelligent. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on pull request #1830: LUCENE-9506: Gradle: split validateSourcePatterns into per-project an…
dweiss commented on pull request #1830: URL: https://github.com/apache/lucene-solr/pull/1830#issuecomment-688666744 > I don't think that's the case. Running the whole thing a second time is quite fast. Because it's cached in the daemon? No way of telling without actually digging at gradle's code (which I've done before and it's not so scary). Let's leave it. If it's annoying then we'll think of a workaround. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1830: LUCENE-9506: Gradle: split validateSourcePatterns into per-project an…
uschindler commented on pull request #1830: URL: https://github.com/apache/lucene-solr/pull/1830#issuecomment-688668964 Of course, running without daemon. Configure time is same. Execution is done very fast. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on a change in pull request #1830: LUCENE-9506: Gradle: split validateSourcePatterns into per-project an…
uschindler commented on a change in pull request #1830:
URL: https://github.com/apache/lucene-solr/pull/1830#discussion_r484709609
##
File path: gradle/validation/validate-source-patterns.gradle
##
@@ -181,8 +252,13 @@ class ValidateSourcePatternsTask extends DefaultTask {
}
}
+ProgressLogger progress = progressLoggerFactory.newOperation(this.class)
+progress.start(this.name, this.name)
+
sourceFiles.each{ f ->
- logger.debug('Scanning source file: {}', f);
+ progress.progress("Scanning ${f.name}")
+ logger.info('Scanning source file: {}', f);
Review comment:
I just wanted to add this to this dicsussion:
https://github.com/gradle/gradle/issues/1010
I am working on that!
(I also figured out that we miss our lockfactory integration test in
`:lucene:core`, in ANT it's still there (many JVMs hammering a single directory
with lockfactories). We need a task in gradle for this, too. I will open
another issue)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1830: LUCENE-9506: Gradle: split validateSourcePatterns into per-project an…
uschindler commented on pull request #1830: URL: https://github.com/apache/lucene-solr/pull/1830#issuecomment-688686424 I checked source code: Actually Gradle does not use the Ant/Plexus DirectoryScanner, it has a new Java 7 based one. This one handles inclusions and exclusions in the same way and skips exclusions. If it is a directory then it passes `SKIP_SUBTREE`: https://github.com/gradle/gradle/blob/aeea780a34d61620ded70b2ebe9e68dd9bc82158/subprojects/file-collections/src/main/java/org/gradle/api/internal/file/collections/DefaultDirectoryWalker.java#L88-L99 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14840) Convert Overseer Google doc to asciidoc for addition to Git repo
[
https://issues.apache.org/jira/browse/SOLR-14840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192016#comment-17192016
]
Ilan Ginzburg commented on SOLR-14840:
--
Thanks [~ctargett] this is awesome! I didn't expect the rendering to be so easy
to read.
I scrolled through the whole doc (on a phone) and everything looks (more than)
fine.
Only issue I noticed is in the bugs section
https://github.com/apache/lucene-solr/blob/jira/solr-14840/solr/dev-docs/overseer/overseer.adoc#potential-identified-issues
the bookmark links are not actual links (at least when using a phone).
> Convert Overseer Google doc to asciidoc for addition to Git repo
>
>
> Key: SOLR-14840
> URL: https://issues.apache.org/jira/browse/SOLR-14840
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: Cassandra Targett
>Priority: Minor
>
> Back in April [~ilan] shared via the mailing list a Google doc he had written
> which provided extensive documentation of the Overseer. It's an impressive
> document, and one we should preserve in the repo for other developers to use.
> I recently offered to convert the Google doc to Asciidoc format to add it to
> the {{solr/dev-docs}} directory, so this issue is mostly so I can make a
> branch for Ilan (and others) to review.
> Mailing list thread:
> https://lists.apache.org/thread.html/rf41ae1356bd2e4e68b3c3d835c5e368493ae0ee14a024402c88c795e%40%3Cdev.lucene.apache.org%3E
> Original Google doc:
> https://docs.google.com/document/d/1KTHq3noZBVUQ7QNuBGEhujZ_duwTVpAsvN3Nz5anQUY/edit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1830: LUCENE-9506: Gradle: split validateSourcePatterns into per-project an…
uschindler commented on pull request #1830: URL: https://github.com/apache/lucene-solr/pull/1830#issuecomment-688688433 I just pushed one more small change, ready to go! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] iverase opened a new pull request #1840: [Backport] LUCENE-9292: Refactor BKD point configuration into its own class (#1697)
iverase opened a new pull request #1840: URL: https://github.com/apache/lucene-solr/pull/1840 backport of #1697 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss merged pull request #1830: LUCENE-9506: Gradle: split validateSourcePatterns into per-project an…
dweiss merged pull request #1830: URL: https://github.com/apache/lucene-solr/pull/1830 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9506) Gradle: split validateSourcePatterns into per-project and root-specific tasks (allow parallelism)
[ https://issues.apache.org/jira/browse/LUCENE-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192025#comment-17192025 ] ASF subversion and git services commented on LUCENE-9506: - Commit b988d557d75d92f4378e98804261f6733f04378a in lucene-solr's branch refs/heads/master from Dawid Weiss [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=b988d55 ] LUCENE-9506: Gradle: split validateSourcePatterns into per-project an… (#1830) * LUCENE-9506: Gradle: split validateSourcePatterns into per-project and root-specific tasks (allow parallelism) Co-authored-by: Uwe Schindler > Gradle: split validateSourcePatterns into per-project and root-specific tasks > (allow parallelism) > - > > Key: LUCENE-9506 > URL: https://issues.apache.org/jira/browse/LUCENE-9506 > Project: Lucene - Core > Issue Type: Task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Major > Time Spent: 8h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9506) Gradle: split validateSourcePatterns into per-project and root-specific tasks (allow parallelism)
[ https://issues.apache.org/jira/browse/LUCENE-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192026#comment-17192026 ] ASF subversion and git services commented on LUCENE-9506: - Commit b988d557d75d92f4378e98804261f6733f04378a in lucene-solr's branch refs/heads/master from Dawid Weiss [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=b988d55 ] LUCENE-9506: Gradle: split validateSourcePatterns into per-project an… (#1830) * LUCENE-9506: Gradle: split validateSourcePatterns into per-project and root-specific tasks (allow parallelism) Co-authored-by: Uwe Schindler > Gradle: split validateSourcePatterns into per-project and root-specific tasks > (allow parallelism) > - > > Key: LUCENE-9506 > URL: https://issues.apache.org/jira/browse/LUCENE-9506 > Project: Lucene - Core > Issue Type: Task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Major > Time Spent: 8h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on pull request #1820: LUCENE-9464: Add high(er)-level hit highlighter example that demonstrates and uses low-level components
dweiss commented on pull request #1820: URL: https://github.com/apache/lucene-solr/pull/1820#issuecomment-688700244 Ping, @romseygeek ? Let me know if you don't have time to take a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9292) BKDWriter refactor: Group point configuration in its own class
[
https://issues.apache.org/jira/browse/LUCENE-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192031#comment-17192031
]
ASF subversion and git services commented on LUCENE-9292:
-
Commit fcb7e13f63576fb1941a183710f4d8b44718c237 in lucene-solr's branch
refs/heads/branch_8x from Ignacio Vera
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=fcb7e13 ]
LUCENE-9292: Refactor BKD point configuration into its own class (#1697) (#1840)
> BKDWriter refactor: Group point configuration in its own class
> ---
>
> Key: LUCENE-9292
> URL: https://issues.apache.org/jira/browse/LUCENE-9292
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Ignacio Vera
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> In order to build a BKD tree, we need to provide to the {{BKDWriter}} the
> point configuration we are indexing. That is the {{number of dimensions}},
> the {{number of indexed dimension}}, the {{number of bytes per dimensions}}
> and the max {{number of points per leaf}}. From this information, we
> actually derive some important parameters like the {{number of bytes per
> point}}
>
> The idea of this change is to group all this information into its own class
> so we can share this information more easily with other objects.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] iverase merged pull request #1840: [Backport] LUCENE-9292: Refactor BKD point configuration into its own class (#1697)
iverase merged pull request #1840: URL: https://github.com/apache/lucene-solr/pull/1840 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1830: LUCENE-9506: Gradle: split validateSourcePatterns into per-project an…
uschindler commented on pull request #1830: URL: https://github.com/apache/lucene-solr/pull/1830#issuecomment-688708102 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9292) BKDWriter refactor: Group point configuration in its own class
[
https://issues.apache.org/jira/browse/LUCENE-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ignacio Vera resolved LUCENE-9292.
--
Fix Version/s: 8.7
Assignee: Ignacio Vera
Resolution: Fixed
> BKDWriter refactor: Group point configuration in its own class
> ---
>
> Key: LUCENE-9292
> URL: https://issues.apache.org/jira/browse/LUCENE-9292
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Ignacio Vera
>Assignee: Ignacio Vera
>Priority: Major
> Fix For: 8.7
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> In order to build a BKD tree, we need to provide to the {{BKDWriter}} the
> point configuration we are indexing. That is the {{number of dimensions}},
> the {{number of indexed dimension}}, the {{number of bytes per dimensions}}
> and the max {{number of points per leaf}}. From this information, we
> actually derive some important parameters like the {{number of bytes per
> point}}
>
> The idea of this change is to group all this information into its own class
> so we can share this information more easily with other objects.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9512) Add test-lock-factory to Gradle build
Uwe Schindler created LUCENE-9512:
-
Summary: Add test-lock-factory to Gradle build
Key: LUCENE-9512
URL: https://issues.apache.org/jira/browse/LUCENE-9512
Project: Lucene - Core
Issue Type: Task
Components: core/store
Reporter: Uwe Schindler
Assignee: Uwe Schindler
When porting to Gradle, the following task was midded to be ported to Gradle:
https://github.com/apache/lucene-solr/blob/branch_8x/lucene/core/build.xml#L148-L234
This is somehow an integration test. It's not part of the test suite, as the
code is part of main distribution and is a client-server implementation that
has one coordinator to handle other JVMs to hammer a directories' lock factory.
It may be included into the test suite, but as it spawns multiple JVMs (thats
essential for it to work), I see this as a separate thing.
I would like to implement that snippet of ANT code in Gradle and attach it to
{{:lucene:core}}'s {{test}} task. If we have an integration test framework at
some point we can make a real {{integTest}} out of it, but for now a simple
Groovy script is fine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9512) Add test-lock-factory to Gradle build
[
https://issues.apache.org/jira/browse/LUCENE-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe Schindler updated LUCENE-9512:
--
Description:
When porting to Gradle, the following task was missed to be ported to Gradle:
https://github.com/apache/lucene-solr/blob/branch_8x/lucene/core/build.xml#L148-L234
This is somehow an integration test. It's not part of the test suite, as the
code is part of main distribution and is a client-server implementation that
has one coordinator to handle other JVMs to hammer a directories' lock factory.
It may be included into the test suite, but as it spawns multiple JVMs (thats
essential for it to work), I see this as a separate thing.
I would like to implement that snippet of ANT code in Gradle and attach it to
{{:lucene:core}}'s {{test}} task. If we have an integration test framework at
some point we can make a real {{integTest}} out of it, but for now a simple
Groovy script is fine.
was:
When porting to Gradle, the following task was midded to be ported to Gradle:
https://github.com/apache/lucene-solr/blob/branch_8x/lucene/core/build.xml#L148-L234
This is somehow an integration test. It's not part of the test suite, as the
code is part of main distribution and is a client-server implementation that
has one coordinator to handle other JVMs to hammer a directories' lock factory.
It may be included into the test suite, but as it spawns multiple JVMs (thats
essential for it to work), I see this as a separate thing.
I would like to implement that snippet of ANT code in Gradle and attach it to
{{:lucene:core}}'s {{test}} task. If we have an integration test framework at
some point we can make a real {{integTest}} out of it, but for now a simple
Groovy script is fine.
> Add test-lock-factory to Gradle build
> -
>
> Key: LUCENE-9512
> URL: https://issues.apache.org/jira/browse/LUCENE-9512
> Project: Lucene - Core
> Issue Type: Task
> Components: core/store
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
>
> When porting to Gradle, the following task was missed to be ported to Gradle:
> https://github.com/apache/lucene-solr/blob/branch_8x/lucene/core/build.xml#L148-L234
> This is somehow an integration test. It's not part of the test suite, as the
> code is part of main distribution and is a client-server implementation that
> has one coordinator to handle other JVMs to hammer a directories' lock
> factory.
> It may be included into the test suite, but as it spawns multiple JVMs (thats
> essential for it to work), I see this as a separate thing.
> I would like to implement that snippet of ANT code in Gradle and attach it to
> {{:lucene:core}}'s {{test}} task. If we have an integration test framework at
> some point we can make a real {{integTest}} out of it, but for now a simple
> Groovy script is fine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] atris opened a new pull request #1841: SOLR 13528: Move Rate Limiter Configuration To Be A Plugin
atris opened a new pull request #1841: URL: https://github.com/apache/lucene-solr/pull/1841 This commit refactors rate limiters configuration to now be modelled as a plugin and move its configuration from web.xml to a dedicated section in solr.xml This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14843) Define strongly-typed cluster configuration API
Andrzej Bialecki created SOLR-14843:
---
Summary: Define strongly-typed cluster configuration API
Key: SOLR-14843
URL: https://issues.apache.org/jira/browse/SOLR-14843
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Andrzej Bialecki
Current cluster-level configuration uses a hodgepodge of traditional Solr
config sources (solr.xml, system properties) and the new somewhat arbitrary
config files kept in ZK ({{/clusterprops.json, /security.json, /packages.json,
/autoscaling.json}} etc...). There's no uniform strongly-typed API to access
and manage these configs - currently each config source has its own CRUD, often
relying on direct access to Zookeeper. There's also no uniform method for
monitoring changes to these config sources.
This issue proposes a uniform config API facade with the following
characteristics:
* Using a single hierarchical (or at least key-based) facade for accessing any
global config.
* Using strongly-typed sub-system configs instead of opaque Map-s: components
would no longer deal with JSON parsing/writing, instead they would use properly
annotated Java objects for config CRUD. Config objects would include versioning
information (eg. lastModified timestamp).
* Isolating access to the underlying config persistence layer: components
would no longer directly interact with Zookeeper or files. Most likely the
default implementation would continue using different ZK files per-subsystem in
order to limit the complexity of file formats and to reduce the cost of
notifications for unmodified parts of the configs.
* Providing uniform way to register listeners for monitoring changes in
specific configs: components would no longer need to interact with ZK watches,
they would instead be notified about modified configs that they are interested
in.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14843) Define strongly-typed cluster configuration API
[
https://issues.apache.org/jira/browse/SOLR-14843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrzej Bialecki updated SOLR-14843:
Fix Version/s: master (9.0)
> Define strongly-typed cluster configuration API
> ---
>
> Key: SOLR-14843
> URL: https://issues.apache.org/jira/browse/SOLR-14843
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Andrzej Bialecki
>Priority: Major
> Labels: clean-api
> Fix For: master (9.0)
>
>
> Current cluster-level configuration uses a hodgepodge of traditional Solr
> config sources (solr.xml, system properties) and the new somewhat arbitrary
> config files kept in ZK ({{/clusterprops.json, /security.json,
> /packages.json, /autoscaling.json}} etc...). There's no uniform
> strongly-typed API to access and manage these configs - currently each config
> source has its own CRUD, often relying on direct access to Zookeeper. There's
> also no uniform method for monitoring changes to these config sources.
> This issue proposes a uniform config API facade with the following
> characteristics:
> * Using a single hierarchical (or at least key-based) facade for accessing
> any global config.
> * Using strongly-typed sub-system configs instead of opaque Map-s:
> components would no longer deal with JSON parsing/writing, instead they would
> use properly annotated Java objects for config CRUD. Config objects would
> include versioning information (eg. lastModified timestamp).
> * Isolating access to the underlying config persistence layer: components
> would no longer directly interact with Zookeeper or files. Most likely the
> default implementation would continue using different ZK files per-subsystem
> in order to limit the complexity of file formats and to reduce the cost of
> notifications for unmodified parts of the configs.
> * Providing uniform way to register listeners for monitoring changes in
> specific configs: components would no longer need to interact with ZK
> watches, they would instead be notified about modified configs that they are
> interested in.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14843) Define strongly-typed cluster configuration API
[
https://issues.apache.org/jira/browse/SOLR-14843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192099#comment-17192099
]
Ishan Chattopadhyaya commented on SOLR-14843:
-
This warrants a SIP, imho.
> Define strongly-typed cluster configuration API
> ---
>
> Key: SOLR-14843
> URL: https://issues.apache.org/jira/browse/SOLR-14843
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Andrzej Bialecki
>Priority: Major
> Labels: clean-api
> Fix For: master (9.0)
>
>
> Current cluster-level configuration uses a hodgepodge of traditional Solr
> config sources (solr.xml, system properties) and the new somewhat arbitrary
> config files kept in ZK ({{/clusterprops.json, /security.json,
> /packages.json, /autoscaling.json}} etc...). There's no uniform
> strongly-typed API to access and manage these configs - currently each config
> source has its own CRUD, often relying on direct access to Zookeeper. There's
> also no uniform method for monitoring changes to these config sources.
> This issue proposes a uniform config API facade with the following
> characteristics:
> * Using a single hierarchical (or at least key-based) facade for accessing
> any global config.
> * Using strongly-typed sub-system configs instead of opaque Map-s:
> components would no longer deal with JSON parsing/writing, instead they would
> use properly annotated Java objects for config CRUD. Config objects would
> include versioning information (eg. lastModified timestamp).
> * Isolating access to the underlying config persistence layer: components
> would no longer directly interact with Zookeeper or files. Most likely the
> default implementation would continue using different ZK files per-subsystem
> in order to limit the complexity of file formats and to reduce the cost of
> notifications for unmodified parts of the configs.
> * Providing uniform way to register listeners for monitoring changes in
> specific configs: components would no longer need to interact with ZK
> watches, they would instead be notified about modified configs that they are
> interested in.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14843) Define strongly-typed cluster configuration API
[
https://issues.apache.org/jira/browse/SOLR-14843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192104#comment-17192104
]
Ilan Ginzburg commented on SOLR-14843:
--
I would suggest this new system allows setting default configuration at dev
time or pre deploy time (before ZK is even started) that are reflected in
deployed clusters. Otherwise defaults will live in Java code which is less
convenient to change.
Also would be useful to provide standardized "override using system properties"
as is possible for configuration values defined in {{solr.xml}}.
Finally, cases in which code and configuration need to be deployed atomically
can't be ignored. Old code1 needs config1, new code2 version needs config2. How
do we mange this? Here's a few options (non exhaustive list):
* Updating to code2 while still keeping config1 (assumes code2 knows how to
read and act upon config1) then updating the config to config2 when no more
copies of code1 are running,
* Creating some indirection to have both config1 and config2 present at the
same time, with code1 reading config1 and code2 reading config2,
* Coding such breaking changes (between code1 and code2) as new different code
using totally new config (similar to previous item but implemented by
considering code1 and code2 as totally different rather than two versions of
the same thing).
> Define strongly-typed cluster configuration API
> ---
>
> Key: SOLR-14843
> URL: https://issues.apache.org/jira/browse/SOLR-14843
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Andrzej Bialecki
>Priority: Major
> Labels: clean-api
> Fix For: master (9.0)
>
>
> Current cluster-level configuration uses a hodgepodge of traditional Solr
> config sources (solr.xml, system properties) and the new somewhat arbitrary
> config files kept in ZK ({{/clusterprops.json, /security.json,
> /packages.json, /autoscaling.json}} etc...). There's no uniform
> strongly-typed API to access and manage these configs - currently each config
> source has its own CRUD, often relying on direct access to Zookeeper. There's
> also no uniform method for monitoring changes to these config sources.
> This issue proposes a uniform config API facade with the following
> characteristics:
> * Using a single hierarchical (or at least key-based) facade for accessing
> any global config.
> * Using strongly-typed sub-system configs instead of opaque Map-s:
> components would no longer deal with JSON parsing/writing, instead they would
> use properly annotated Java objects for config CRUD. Config objects would
> include versioning information (eg. lastModified timestamp).
> * Isolating access to the underlying config persistence layer: components
> would no longer directly interact with Zookeeper or files. Most likely the
> default implementation would continue using different ZK files per-subsystem
> in order to limit the complexity of file formats and to reduce the cost of
> notifications for unmodified parts of the configs.
> * Providing uniform way to register listeners for monitoring changes in
> specific configs: components would no longer need to interact with ZK
> watches, they would instead be notified about modified configs that they are
> interested in.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14821) docValuesTermsFilter should support single-valued docValues fields
[ https://issues.apache.org/jira/browse/SOLR-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gerlowski resolved SOLR-14821. Fix Version/s: 8.7 master (9.0) Resolution: Fixed Thanks again Anatolli. This will be in starting in 8.7 (or 9.0 if there's no 8.7) > docValuesTermsFilter should support single-valued docValues fields > -- > > Key: SOLR-14821 > URL: https://issues.apache.org/jira/browse/SOLR-14821 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: query parsers >Reporter: Anatolii Siuniaev >Assignee: Jason Gerlowski >Priority: Minor > Fix For: master (9.0), 8.7 > > Attachments: SOLR-14821.patch > > > SOLR-13890 introduced a post-filter implementation for docValuesTermsFilter > in TermsQParserPlugin. But now it supports only multi-valued docValues > fields (i.e. SORTED_SET type DocValues) > It doesn't work for single-valued docValues fields (i.e. SORTED type > DocValues), though it should. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9512) Add test-lock-factory to Gradle build
[
https://issues.apache.org/jira/browse/LUCENE-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192152#comment-17192152
]
Dawid Weiss commented on LUCENE-9512:
-
And why this can't be a regular test forking subprocesses, Uwe? You could pass
the required stuff from gradle, but the test itself could just run as a regular
JUnit test (in-process)?
> Add test-lock-factory to Gradle build
> -
>
> Key: LUCENE-9512
> URL: https://issues.apache.org/jira/browse/LUCENE-9512
> Project: Lucene - Core
> Issue Type: Task
> Components: core/store
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
>
> When porting to Gradle, the following task was missed to be ported to Gradle:
> https://github.com/apache/lucene-solr/blob/branch_8x/lucene/core/build.xml#L148-L234
> This is somehow an integration test. It's not part of the test suite, as the
> code is part of main distribution and is a client-server implementation that
> has one coordinator to handle other JVMs to hammer a directories' lock
> factory.
> It may be included into the test suite, but as it spawns multiple JVMs (thats
> essential for it to work), I see this as a separate thing.
> I would like to implement that snippet of ANT code in Gradle and attach it to
> {{:lucene:core}}'s {{test}} task. If we have an integration test framework at
> some point we can make a real {{integTest}} out of it, but for now a simple
> Groovy script is fine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9512) Add test-lock-factory to Gradle build
[
https://issues.apache.org/jira/browse/LUCENE-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192155#comment-17192155
]
Uwe Schindler commented on LUCENE-9512:
---
I checked the code. IMHO, it's easier to create a simple Unit Test out of it,
which starts the LockVerifyServer in the test process and then spawns to
clients.
Much easier than the current Ant code. The VerifyServer/Client is still inside
the main Lucene JAR, but the test is just using its APIs.
Once this is done, no separate task is needed anymore! I'll work on this.
> Add test-lock-factory to Gradle build
> -
>
> Key: LUCENE-9512
> URL: https://issues.apache.org/jira/browse/LUCENE-9512
> Project: Lucene - Core
> Issue Type: Task
> Components: core/store
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
>
> When porting to Gradle, the following task was missed to be ported to Gradle:
> https://github.com/apache/lucene-solr/blob/branch_8x/lucene/core/build.xml#L148-L234
> This is somehow an integration test. It's not part of the test suite, as the
> code is part of main distribution and is a client-server implementation that
> has one coordinator to handle other JVMs to hammer a directories' lock
> factory.
> It may be included into the test suite, but as it spawns multiple JVMs (thats
> essential for it to work), I see this as a separate thing.
> I would like to implement that snippet of ANT code in Gradle and attach it to
> {{:lucene:core}}'s {{test}} task. If we have an integration test framework at
> some point we can make a real {{integTest}} out of it, but for now a simple
> Groovy script is fine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9512) Add test-lock-factory to Gradle build
[
https://issues.apache.org/jira/browse/LUCENE-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192156#comment-17192156
]
Uwe Schindler commented on LUCENE-9512:
---
Hah same idea same time. I'll work on it.
> Add test-lock-factory to Gradle build
> -
>
> Key: LUCENE-9512
> URL: https://issues.apache.org/jira/browse/LUCENE-9512
> Project: Lucene - Core
> Issue Type: Task
> Components: core/store
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
>
> When porting to Gradle, the following task was missed to be ported to Gradle:
> https://github.com/apache/lucene-solr/blob/branch_8x/lucene/core/build.xml#L148-L234
> This is somehow an integration test. It's not part of the test suite, as the
> code is part of main distribution and is a client-server implementation that
> has one coordinator to handle other JVMs to hammer a directories' lock
> factory.
> It may be included into the test suite, but as it spawns multiple JVMs (thats
> essential for it to work), I see this as a separate thing.
> I would like to implement that snippet of ANT code in Gradle and attach it to
> {{:lucene:core}}'s {{test}} task. If we have an integration test framework at
> some point we can make a real {{integTest}} out of it, but for now a simple
> Groovy script is fine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9512) Add test-lock-factory to Gradle build
[
https://issues.apache.org/jira/browse/LUCENE-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192158#comment-17192158
]
Dawid Weiss commented on LUCENE-9512:
-
Ping me when you have it for review, I'll take a look.
> Add test-lock-factory to Gradle build
> -
>
> Key: LUCENE-9512
> URL: https://issues.apache.org/jira/browse/LUCENE-9512
> Project: Lucene - Core
> Issue Type: Task
> Components: core/store
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
>
> When porting to Gradle, the following task was missed to be ported to Gradle:
> https://github.com/apache/lucene-solr/blob/branch_8x/lucene/core/build.xml#L148-L234
> This is somehow an integration test. It's not part of the test suite, as the
> code is part of main distribution and is a client-server implementation that
> has one coordinator to handle other JVMs to hammer a directories' lock
> factory.
> It may be included into the test suite, but as it spawns multiple JVMs (thats
> essential for it to work), I see this as a separate thing.
> I would like to implement that snippet of ANT code in Gradle and attach it to
> {{:lucene:core}}'s {{test}} task. If we have an integration test framework at
> some point we can make a real {{integTest}} out of it, but for now a simple
> Groovy script is fine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] romseygeek commented on a change in pull request #1820: LUCENE-9464: Add high(er)-level hit highlighter example that demonstrates and uses low-level components
romseygeek commented on a change in pull request #1820:
URL: https://github.com/apache/lucene-solr/pull/1820#discussion_r484891054
##
File path:
lucene/highlighter/src/test/org/apache/lucene/search/matchhighlight/TestMatchHighlighter.java
##
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.search.matchhighlight;
+
+import com.carrotsearch.randomizedtesting.RandomizedTest;
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.TokenStream;
+import org.apache.lucene.analysis.Tokenizer;
+import org.apache.lucene.analysis.core.WhitespaceTokenizer;
+import org.apache.lucene.analysis.miscellaneous.PerFieldAnalyzerWrapper;
+import org.apache.lucene.analysis.synonym.SynonymGraphFilter;
+import org.apache.lucene.analysis.synonym.SynonymMap;
+import org.apache.lucene.document.Field;
+import org.apache.lucene.document.FieldType;
+import org.apache.lucene.document.TextField;
+import org.apache.lucene.index.IndexOptions;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.queries.intervals.IntervalQuery;
+import org.apache.lucene.queries.intervals.Intervals;
+import org.apache.lucene.search.BooleanClause;
+import org.apache.lucene.search.BooleanQuery;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.search.PhraseQuery;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.TermQuery;
+import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.util.CharsRef;
+import org.apache.lucene.util.LuceneTestCase;
+import org.hamcrest.Matchers;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TestMatchHighlighter extends LuceneTestCase {
+ private static final String FLD_ID = "id";
+ private static final String FLD_TEXT1 = "text1";
+ private static final String FLD_TEXT2 = "text2";
+
+ private FieldType TYPE_TEXT_POSITIONS_OFFSETS;
+ private FieldType TYPE_TEXT_POSITIONS;
+
+ private PerFieldAnalyzerWrapper analyzer;
+
+ @Before
+ public void setup() throws IOException {
+TYPE_TEXT_POSITIONS = TextField.TYPE_STORED;
+
+TYPE_TEXT_POSITIONS_OFFSETS = new FieldType(TextField.TYPE_STORED);
+
TYPE_TEXT_POSITIONS_OFFSETS.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
+TYPE_TEXT_POSITIONS_OFFSETS.freeze();
+
+Map fieldAnalyzers = new HashMap<>();
+
+// Create an analyzer with some synonyms, just to showcase them.
+SynonymMap synonymMap = buildSynonymMap(new String[][]{
+{"moon\ushine", "firewater"},
+{"firewater", "moon\ushine"},
+});
+
+// Make a non-empty offset gap so that break iterator doesn't go haywire
on multivalues
+// glued together.
+final int offsetGap = RandomizedTest.randomIntBetween(1, 2);
+final int positionGap = RandomizedTest.randomFrom(new int[]{0, 1, 100});
+Analyzer synonymsAnalyzer =
+new AnalyzerWithGaps(offsetGap, positionGap, new Analyzer() {
+ @Override
+ protected TokenStreamComponents createComponents(String fieldName) {
+Tokenizer tokenizer = new WhitespaceTokenizer();
+TokenStream tokenStream = new SynonymGraphFilter(tokenizer,
synonymMap, true);
+return new TokenStreamComponents(tokenizer, tokenStream);
+ }
+});
+
+fieldAnalyzers.put(FLD_TEXT1, synonymsAnalyzer);
+fieldAnalyzers.put(FLD_TEXT2, synonymsAnalyzer);
+
+analyzer = new PerFieldAnalyzerWrapper(new MissingAnalyzer(),
fieldAnalyzers);
+ }
+
+ static SynonymMap buildSynonymMap(String[][] synonyms) throws IOException {
+SynonymMap.Builder builder = new SynonymMap.Builder();
+for (String
[jira] [Commented] (SOLR-14768) Error 500 on PDF extraction: java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
[
https://issues.apache.org/jira/browse/SOLR-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192193#comment-17192193
]
Colvin Cowie commented on SOLR-14768:
-
[~dsmiley] thanks, I've built a new solr-core locally with the change from your
PR and it resolves the problems as expected.
> Error 500 on PDF extraction: java.lang.NoClassDefFoundError:
> org/eclipse/jetty/server/MultiParts
>
>
> Key: SOLR-14768
> URL: https://issues.apache.org/jira/browse/SOLR-14768
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: contrib - Solr Cell (Tika extraction)
>Affects Versions: 8.6, 8.6.1
>Reporter: Markus Kalkbrenner
>Assignee: David Smiley
>Priority: Major
> Attachments: Solr v8.6.x fails with multipart MIME in commands.eml,
> Solr v8.6.x fails with multipart MIME in commands.eml, Solr v8.6.x fails with
> multipart MIME in commands.eml
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> See [https://www.mail-archive.com/solr-user@lucene.apache.org/msg152182.html]
> The integration tests of the solarium PHP library and the integration tests
> of the Search API Solr Drupal module both fail on PDF extraction if executed
> on Solr 8.6.
> They still work on Solr 8.5.1 an earlier versions.
> {quote}2020-08-20 12:30:35.279 INFO (qtp855700733-19) [ x:5f3e6ce2810ef]
> o.a.s.u.p.LogUpdateProcessorFactory [5f3e6ce2810ef] webapp=/solr
> path=/update/extract
> params=\{json.nl=flat&commitWithin=0&omitHeader=false&resource.name=testpdf.pdf&literal.id=extract-test&commit=true&extractOnly=false&uprefix=attr_&wt=json}{add=[extract-test
> (1675547519474466816)],commit=} 0 957
> solr8_1 | 2020-08-20 12:30:35.280 WARN (qtp855700733-19) [ ]
> o.e.j.s.HttpChannel /solr/5f3e6ce2810ef/update/extract =>
> java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
> solr8_1 | at
> org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
> solr8_1 | java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
> solr8_1 | at
> org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
> ~[?:?]
> solr8_1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:443)
> ~[?:?]
> solr8_1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> ~[?:?]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> ~[jetty-security-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> ~[jetty-server-9.4.27.v2020
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1820: LUCENE-9464: Add high(er)-level hit highlighter example that demonstrates and uses low-level components
dweiss commented on a change in pull request #1820:
URL: https://github.com/apache/lucene-solr/pull/1820#discussion_r484900156
##
File path:
lucene/highlighter/src/test/org/apache/lucene/search/matchhighlight/TestMatchHighlighter.java
##
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.search.matchhighlight;
+
+import com.carrotsearch.randomizedtesting.RandomizedTest;
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.TokenStream;
+import org.apache.lucene.analysis.Tokenizer;
+import org.apache.lucene.analysis.core.WhitespaceTokenizer;
+import org.apache.lucene.analysis.miscellaneous.PerFieldAnalyzerWrapper;
+import org.apache.lucene.analysis.synonym.SynonymGraphFilter;
+import org.apache.lucene.analysis.synonym.SynonymMap;
+import org.apache.lucene.document.Field;
+import org.apache.lucene.document.FieldType;
+import org.apache.lucene.document.TextField;
+import org.apache.lucene.index.IndexOptions;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.queries.intervals.IntervalQuery;
+import org.apache.lucene.queries.intervals.Intervals;
+import org.apache.lucene.search.BooleanClause;
+import org.apache.lucene.search.BooleanQuery;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.search.PhraseQuery;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.TermQuery;
+import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.util.CharsRef;
+import org.apache.lucene.util.LuceneTestCase;
+import org.hamcrest.Matchers;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TestMatchHighlighter extends LuceneTestCase {
+ private static final String FLD_ID = "id";
+ private static final String FLD_TEXT1 = "text1";
+ private static final String FLD_TEXT2 = "text2";
+
+ private FieldType TYPE_TEXT_POSITIONS_OFFSETS;
+ private FieldType TYPE_TEXT_POSITIONS;
+
+ private PerFieldAnalyzerWrapper analyzer;
+
+ @Before
+ public void setup() throws IOException {
+TYPE_TEXT_POSITIONS = TextField.TYPE_STORED;
+
+TYPE_TEXT_POSITIONS_OFFSETS = new FieldType(TextField.TYPE_STORED);
+
TYPE_TEXT_POSITIONS_OFFSETS.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
+TYPE_TEXT_POSITIONS_OFFSETS.freeze();
+
+Map fieldAnalyzers = new HashMap<>();
+
+// Create an analyzer with some synonyms, just to showcase them.
+SynonymMap synonymMap = buildSynonymMap(new String[][]{
+{"moon\ushine", "firewater"},
+{"firewater", "moon\ushine"},
+});
+
+// Make a non-empty offset gap so that break iterator doesn't go haywire
on multivalues
+// glued together.
+final int offsetGap = RandomizedTest.randomIntBetween(1, 2);
+final int positionGap = RandomizedTest.randomFrom(new int[]{0, 1, 100});
+Analyzer synonymsAnalyzer =
+new AnalyzerWithGaps(offsetGap, positionGap, new Analyzer() {
+ @Override
+ protected TokenStreamComponents createComponents(String fieldName) {
+Tokenizer tokenizer = new WhitespaceTokenizer();
+TokenStream tokenStream = new SynonymGraphFilter(tokenizer,
synonymMap, true);
+return new TokenStreamComponents(tokenizer, tokenStream);
+ }
+});
+
+fieldAnalyzers.put(FLD_TEXT1, synonymsAnalyzer);
+fieldAnalyzers.put(FLD_TEXT2, synonymsAnalyzer);
+
+analyzer = new PerFieldAnalyzerWrapper(new MissingAnalyzer(),
fieldAnalyzers);
+ }
+
+ static SynonymMap buildSynonymMap(String[][] synonyms) throws IOException {
+SynonymMap.Builder builder = new SynonymMap.Builder();
+for (String[] p
[jira] [Created] (SOLR-14844) Upgrade Jetty to 9.4.30+
Cassandra Targett created SOLR-14844: Summary: Upgrade Jetty to 9.4.30+ Key: SOLR-14844 URL: https://issues.apache.org/jira/browse/SOLR-14844 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Affects Versions: 8.6 Reporter: Cassandra Targett A CVE was found in Jetty 9.4.27-9.4.29 that has some security scanning tools raising red flags (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17638). Here's the Jetty issue: https://bugs.eclipse.org/bugs/show_bug.cgi?id=564984. It's fixed in 9.4.30+, so we should upgrade to that for 8.7 It has a simple mitigation (raise Jetty's responseHeaderSize to higher than requestHeaderSize), but I don't know how Solr uses Jetty well enough to a) know if this problem is even exploitable in Solr, or b) if the workaround suggested is even possible in Solr. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request #1839: LUCENE-9511: Include StoredFieldsWriter in DWPT accounting
mikemccand commented on a change in pull request #1839:
URL: https://github.com/apache/lucene-solr/pull/1839#discussion_r484909064
##
File path:
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThread.java
##
@@ -351,7 +354,7 @@ FlushedSegment flush(DocumentsWriter.FlushNotifications
flushNotifications) thro
flushState.liveDocs.clear(deleteDocIDs[i]);
}
flushState.delCountOnFlush = numDeletedDocIds;
- bytesUsed.addAndGet(-(deleteDocIDs.length * Integer.SIZE));
+ bytesUsed.addAndGet(-(deleteDocIDs.length * Integer.BYTES));
Review comment:
Whoa, this was a latent (over-counting) bug? Maybe `grep` for other
places we are possibly incorrectly using `Integer.SIZE`!
Though, I think this array is typically tiny (usually 0 length), since it
holds docs that hit non-aborting exception during indexing?
##
File path:
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThread.java
##
@@ -292,7 +295,7 @@ private void deleteLastDocs(int docCount) {
for (int docId = from; docId < to; docId++) {
deleteDocIDs[numDeletedDocIds++] = docId;
}
-bytesUsed.addAndGet((deleteDocIDs.length - size) * Integer.SIZE);
+bytesUsed.addAndGet((deleteDocIDs.length - size) * Integer.BYTES);
Review comment:
Also a latent over-counting bug?
##
File path:
lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java
##
@@ -52,7 +52,7 @@
/** Default general purpose indexing chain, which handles
* indexing all types of fields. */
final class DefaultIndexingChain extends DocConsumer {
- final Counter bytesUsed;
+ final Counter bytesUsed = Counter.newCounter();
Review comment:
So we are breaking out a separate `Counter` for `DefaultIndexingChain`,
from `DWPT`, when previously they shared the same counter?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14840) Convert Overseer Google doc to asciidoc for addition to Git repo
[
https://issues.apache.org/jira/browse/SOLR-14840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192237#comment-17192237
]
Cassandra Targett commented on SOLR-14840:
--
bq. Only issue I noticed is in the bugs section
https://github.com/apache/lucene-solr/blob/jira/solr-14840/solr/dev-docs/overseer/overseer.adoc#potential-identified-issues
the bookmark links are not actual links (at least when using a phone).
Oh, good catch, I totally missed those. I'll fix them.
> Convert Overseer Google doc to asciidoc for addition to Git repo
>
>
> Key: SOLR-14840
> URL: https://issues.apache.org/jira/browse/SOLR-14840
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: Cassandra Targett
>Priority: Minor
>
> Back in April [~ilan] shared via the mailing list a Google doc he had written
> which provided extensive documentation of the Overseer. It's an impressive
> document, and one we should preserve in the repo for other developers to use.
> I recently offered to convert the Google doc to Asciidoc format to add it to
> the {{solr/dev-docs}} directory, so this issue is mostly so I can make a
> branch for Ilan (and others) to review.
> Mailing list thread:
> https://lists.apache.org/thread.html/rf41ae1356bd2e4e68b3c3d835c5e368493ae0ee14a024402c88c795e%40%3Cdev.lucene.apache.org%3E
> Original Google doc:
> https://docs.google.com/document/d/1KTHq3noZBVUQ7QNuBGEhujZ_duwTVpAsvN3Nz5anQUY/edit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1725: LUCENE-9449 Skip docs with _doc sort and "after"
mayya-sharipova commented on a change in pull request #1725:
URL: https://github.com/apache/lucene-solr/pull/1725#discussion_r484953402
##
File path:
lucene/core/src/test/org/apache/lucene/search/TestFieldSortOptimizationSkipping.java
##
@@ -290,5 +299,114 @@ public void testFloatSortOptimization() throws
IOException {
dir.close();
}
+ public void testDocSortOptimizationWithAfter() throws IOException {
+final Directory dir = newDirectory();
+final IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig());
+final int numDocs = atLeast(1500);
+for (int i = 0; i < numDocs; ++i) {
+ final Document doc = new Document();
+ writer.addDocument(doc);
+ if ((i > 0) && (i % 500 == 0)) {
+writer.commit();
+ }
+}
+final IndexReader reader = DirectoryReader.open(writer);
+IndexSearcher searcher = new IndexSearcher(reader);
+final int numHits = 3;
+final int totalHitsThreshold = 3;
+final int searchAfter = 1400;
+
+// sort by _doc with search after should trigger optimization
+{
+ final Sort sort = new Sort(FIELD_DOC);
+ FieldDoc after = new FieldDoc(searchAfter, Float.NaN, new
Integer[]{searchAfter});
+ final TopFieldCollector collector = TopFieldCollector.create(sort,
numHits, after, totalHitsThreshold);
+ searcher.search(new MatchAllDocsQuery(), collector);
+ TopDocs topDocs = collector.topDocs();
+ assertEquals(topDocs.scoreDocs.length, numHits);
+ for (int i = 0; i < numHits; i++) {
+int expectedDocID = searchAfter + 1 + i;
+assertEquals(expectedDocID, topDocs.scoreDocs[i].doc);
+ }
+ assertTrue(collector.isEarlyTerminated());
+ // check that very few hits were collected, and most hits before
searchAfter were skipped
+ assertTrue(topDocs.totalHits.value < (numDocs - searchAfter));
+}
+
+// sort by _doc + _score with search after should trigger optimization
+{
+ final Sort sort = new Sort(FIELD_DOC, FIELD_SCORE);
+ FieldDoc after = new FieldDoc(searchAfter, Float.NaN, new
Object[]{searchAfter, 1.0f});
+ final TopFieldCollector collector = TopFieldCollector.create(sort,
numHits, after, totalHitsThreshold);
+ searcher.search(new MatchAllDocsQuery(), collector);
+ TopDocs topDocs = collector.topDocs();
+ assertEquals(topDocs.scoreDocs.length, numHits);
+ for (int i = 0; i < numHits; i++) {
+int expectedDocID = searchAfter + 1 + i;
+assertEquals(expectedDocID, topDocs.scoreDocs[i].doc);
+ }
+ assertTrue(collector.isEarlyTerminated());
+ // assert that very few hits were collected, and most hits before
searchAfter were skipped
+ assertTrue(topDocs.totalHits.value < (numDocs - searchAfter));
+}
+
+// sort by _doc desc should not trigger optimization
+{
+ final Sort sort = new Sort(new SortField(null, SortField.Type.DOC,
true));
+ FieldDoc after = new FieldDoc(searchAfter, Float.NaN, new
Integer[]{searchAfter});
+ final TopFieldCollector collector = TopFieldCollector.create(sort,
numHits, after, totalHitsThreshold);
+ searcher.search(new MatchAllDocsQuery(), collector);
+ TopDocs topDocs = collector.topDocs();
+ for (int i = 0; i < numHits; i++) {
+int expectedDocID = searchAfter - 1 - i;
+assertEquals(expectedDocID, topDocs.scoreDocs[i].doc);
+ }
+ assertEquals(topDocs.scoreDocs.length, numHits);
+ // assert that many hits were collected including all hits before
searchAfter
+ assertTrue(topDocs.totalHits.value > searchAfter);
+
+}
+
+writer.close();
+reader.close();
+dir.close();
+ }
+
+
+ public void testDocSortOptimization() throws IOException {
+final Directory dir = newDirectory();
+final IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig());
+final int numDocs = atLeast(1500);
Review comment:
Addressed in 493d9eb
##
File path:
lucene/core/src/test/org/apache/lucene/search/TestFieldSortOptimizationSkipping.java
##
@@ -290,5 +299,114 @@ public void testFloatSortOptimization() throws
IOException {
dir.close();
}
+ public void testDocSortOptimizationWithAfter() throws IOException {
+final Directory dir = newDirectory();
+final IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig());
+final int numDocs = atLeast(1500);
Review comment:
Addressed in 493d9eb
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apac
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1725: LUCENE-9449 Skip docs with _doc sort and "after"
mayya-sharipova commented on a change in pull request #1725:
URL: https://github.com/apache/lucene-solr/pull/1725#discussion_r484953808
##
File path:
lucene/core/src/test/org/apache/lucene/search/TestFieldSortOptimizationSkipping.java
##
@@ -290,5 +299,114 @@ public void testFloatSortOptimization() throws
IOException {
dir.close();
}
+ public void testDocSortOptimizationWithAfter() throws IOException {
+final Directory dir = newDirectory();
+final IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig());
+final int numDocs = atLeast(1500);
+for (int i = 0; i < numDocs; ++i) {
+ final Document doc = new Document();
+ writer.addDocument(doc);
+ if ((i > 0) && (i % 500 == 0)) {
+writer.commit();
+ }
+}
+final IndexReader reader = DirectoryReader.open(writer);
+IndexSearcher searcher = new IndexSearcher(reader);
+final int numHits = 3;
+final int totalHitsThreshold = 3;
+final int searchAfter = 1400;
+
+// sort by _doc with search after should trigger optimization
Review comment:
Addressed in 493d9eb
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14845) Backup failing with solr 7.7.2 java.io.IOException: Interrupted system call
Jeff created SOLR-14845:
---
Summary: Backup failing with solr 7.7.2 java.io.IOException:
Interrupted system call
Key: SOLR-14845
URL: https://issues.apache.org/jira/browse/SOLR-14845
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: Backup/Restore
Affects Versions: 7.2.2
Reporter: Jeff
I have a 12 node solrcloud cluster with 48 shards.
800GB on each node.
7.3 million docs and around 98 GB per shard.
When I issue the backup command it runs for several hours and produces most of
the backup but fails on some shards.
Command issued
curl -XPOST
'http://xx.xxx.xxx.xxx:8983/solr/admin/collections?action=BACKUP&name=prod1&collection=PROD&location=/mnt/prodstorage/backup&async=13&wt=xml'
"Response":"TaskId: 127375391376590965 webapp=null path=/admin/cores
params={core=PROD_shard8_1_replica_n156&async=127375391376590965&qt=/admin/cores&name=shard8_1&action=BACKUPCORE&location=file:///mnt/prodstorage/backup/prod1&wt=javabin&version=2}
status=0 QTime=0"},
"127375391376904569":{
"responseHeader":{
"status":0,
"QTime":0},
"STATUS":"failed",
"Response":"Failed to backup core=PROD_shard19_1_replica_n263 because
java.io.IOException: Interrupted system call"},
"status":{
"state":"failed", "msg":"found [12] in failed tasks"}}
Can I lengthen the timeout? Manaully backup?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9504) DocumentsWriterDeleteQueue.forceApplyGlobalSlice has needless locking
[
https://issues.apache.org/jira/browse/LUCENE-9504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob resolved LUCENE-9504.
---
Fix Version/s: master (9.0)
Resolution: Fixed
> DocumentsWriterDeleteQueue.forceApplyGlobalSlice has needless locking
> -
>
> Key: LUCENE-9504
> URL: https://issues.apache.org/jira/browse/LUCENE-9504
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Minor
> Fix For: master (9.0)
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{DocumentsWriterDeleteQueue.forceApplyGlobalSlice}} is only called from
> {{getBufferedUpdateTermsSize}} so we end up doing double the locking. It's
> not a huge problem because re-entrant locks are cheap, but we can probably
> unwrap the methods a little bit here.
> This may impact HotSpot inlining, I haven't tried measuring that yet. If that
> is a concern, please let me know.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9504) DocumentsWriterDeleteQueue.forceApplyGlobalSlice has needless locking
[
https://issues.apache.org/jira/browse/LUCENE-9504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192268#comment-17192268
]
ASF subversion and git services commented on LUCENE-9504:
-
Commit 4c5c8c4ead74c92b6ab0ec55bca709d36ad152d9 in lucene-solr's branch
refs/heads/master from Mike Drob
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=4c5c8c4 ]
LUCENE-9504 Remove extra lock in DocumentsWriterDeleteQueue (#1826)
> DocumentsWriterDeleteQueue.forceApplyGlobalSlice has needless locking
> -
>
> Key: LUCENE-9504
> URL: https://issues.apache.org/jira/browse/LUCENE-9504
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{DocumentsWriterDeleteQueue.forceApplyGlobalSlice}} is only called from
> {{getBufferedUpdateTermsSize}} so we end up doing double the locking. It's
> not a huge problem because re-entrant locks are cheap, but we can probably
> unwrap the methods a little bit here.
> This may impact HotSpot inlining, I haven't tried measuring that yet. If that
> is a concern, please let me know.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob merged pull request #1826: LUCENE-9504 Remove extra lock in DocumentsWriterDeleteQueue
madrob merged pull request #1826: URL: https://github.com/apache/lucene-solr/pull/1826 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob commented on pull request #1732: Clean up many small fixes
madrob commented on pull request #1732: URL: https://github.com/apache/lucene-solr/pull/1732#issuecomment-688947909 @vthacker this is also some error-prone warnings, so may overlap with what you're looking at. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob merged pull request #1469: SOLR-14770 Avoid reregistering JVM Guage as well
madrob merged pull request #1469: URL: https://github.com/apache/lucene-solr/pull/1469 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14770) Multiple CoreContainers register same JVM gauge in metrics
[ https://issues.apache.org/jira/browse/SOLR-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob resolved SOLR-14770. -- Fix Version/s: master (9.0) Resolution: Fixed > Multiple CoreContainers register same JVM gauge in metrics > -- > > Key: SOLR-14770 > URL: https://issues.apache.org/jira/browse/SOLR-14770 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: master (9.0) > > > Follow on to SOLR-14274, we fixed registering the metric sets, but didn't > change registering the metrics gauges at all. There's a bit more performance > that we can squeeze out of this stone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14770) Multiple CoreContainers register same JVM gauge in metrics
[ https://issues.apache.org/jira/browse/SOLR-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192272#comment-17192272 ] ASF subversion and git services commented on SOLR-14770: Commit 984466f31bcb9dfcb1781b053eb621cb9c86b9a3 in lucene-solr's branch refs/heads/master from Mike Drob [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=984466f ] SOLR-14770 Avoid reregistering JVM Guage as well (#1469) > Multiple CoreContainers register same JVM gauge in metrics > -- > > Key: SOLR-14770 > URL: https://issues.apache.org/jira/browse/SOLR-14770 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: master (9.0) > > Time Spent: 10m > Remaining Estimate: 0h > > Follow on to SOLR-14274, we fixed registering the metric sets, but didn't > change registering the metrics gauges at all. There's a bit more performance > that we can squeeze out of this stone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14844) Upgrade Jetty to 9.4.30+
[ https://issues.apache.org/jira/browse/SOLR-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson reassigned SOLR-14844: - Assignee: Erick Erickson > Upgrade Jetty to 9.4.30+ > > > Key: SOLR-14844 > URL: https://issues.apache.org/jira/browse/SOLR-14844 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 8.6 >Reporter: Cassandra Targett >Assignee: Erick Erickson >Priority: Major > > A CVE was found in Jetty 9.4.27-9.4.29 that has some security scanning tools > raising red flags > (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17638). > Here's the Jetty issue: https://bugs.eclipse.org/bugs/show_bug.cgi?id=564984. > It's fixed in 9.4.30+, so we should upgrade to that for 8.7 > It has a simple mitigation (raise Jetty's responseHeaderSize to higher than > requestHeaderSize), but I don't know how Solr uses Jetty well enough to a) > know if this problem is even exploitable in Solr, or b) if the workaround > suggested is even possible in Solr. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14840) Convert Overseer Google doc to asciidoc for addition to Git repo
[
https://issues.apache.org/jira/browse/SOLR-14840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192275#comment-17192275
]
Cassandra Targett commented on SOLR-14840:
--
Pushed fixes to the bookmarks. There was 1 item in the list of potential issues
which I removed: #6: {{It is possible to acquire conflicting locks in
collection processing.}} - this is in the section that was crossed out so I
removed it from the list of issues.
> Convert Overseer Google doc to asciidoc for addition to Git repo
>
>
> Key: SOLR-14840
> URL: https://issues.apache.org/jira/browse/SOLR-14840
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: Cassandra Targett
>Priority: Minor
>
> Back in April [~ilan] shared via the mailing list a Google doc he had written
> which provided extensive documentation of the Overseer. It's an impressive
> document, and one we should preserve in the repo for other developers to use.
> I recently offered to convert the Google doc to Asciidoc format to add it to
> the {{solr/dev-docs}} directory, so this issue is mostly so I can make a
> branch for Ilan (and others) to review.
> Mailing list thread:
> https://lists.apache.org/thread.html/rf41ae1356bd2e4e68b3c3d835c5e368493ae0ee14a024402c88c795e%40%3Cdev.lucene.apache.org%3E
> Original Google doc:
> https://docs.google.com/document/d/1KTHq3noZBVUQ7QNuBGEhujZ_duwTVpAsvN3Nz5anQUY/edit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob merged pull request #1732: Clean up many small fixes
madrob merged pull request #1732: URL: https://github.com/apache/lucene-solr/pull/1732 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14844) Upgrade Jetty to 9.4.31
[ https://issues.apache.org/jira/browse/SOLR-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson updated SOLR-14844: -- Summary: Upgrade Jetty to 9.4.31 (was: Upgrade Jetty to 9.4.30+) > Upgrade Jetty to 9.4.31 > --- > > Key: SOLR-14844 > URL: https://issues.apache.org/jira/browse/SOLR-14844 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 8.6 >Reporter: Cassandra Targett >Assignee: Erick Erickson >Priority: Major > > A CVE was found in Jetty 9.4.27-9.4.29 that has some security scanning tools > raising red flags > (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17638). > Here's the Jetty issue: https://bugs.eclipse.org/bugs/show_bug.cgi?id=564984. > It's fixed in 9.4.30+, so we should upgrade to that for 8.7 > It has a simple mitigation (raise Jetty's responseHeaderSize to higher than > requestHeaderSize), but I don't know how Solr uses Jetty well enough to a) > know if this problem is even exploitable in Solr, or b) if the workaround > suggested is even possible in Solr. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] rmuir commented on pull request #1837: LUCENE-7882: First idea of using Java 15 hidden anonymous classes for Lucene expressions
rmuir commented on pull request #1837: URL: https://github.com/apache/lucene-solr/pull/1837#issuecomment-688956220 Nice improvement here. I think we should not let the stacktrace stuff be a blocker for this optimization. Out of box, the user should never even see exceptions anyway: the expression stuff all works on doubles. So all those math functions just return NaN/Inf for special cases because that's how java Math functions want to work. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14840) Convert Overseer Google doc to asciidoc for addition to Git repo
[
https://issues.apache.org/jira/browse/SOLR-14840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192285#comment-17192285
]
Ilan Ginzburg commented on SOLR-14840:
--
I think we're good now. I fixed a few occurrences of bold text not displaying
correctly.
> Convert Overseer Google doc to asciidoc for addition to Git repo
>
>
> Key: SOLR-14840
> URL: https://issues.apache.org/jira/browse/SOLR-14840
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: Cassandra Targett
>Priority: Minor
>
> Back in April [~ilan] shared via the mailing list a Google doc he had written
> which provided extensive documentation of the Overseer. It's an impressive
> document, and one we should preserve in the repo for other developers to use.
> I recently offered to convert the Google doc to Asciidoc format to add it to
> the {{solr/dev-docs}} directory, so this issue is mostly so I can make a
> branch for Ilan (and others) to review.
> Mailing list thread:
> https://lists.apache.org/thread.html/rf41ae1356bd2e4e68b3c3d835c5e368493ae0ee14a024402c88c795e%40%3Cdev.lucene.apache.org%3E
> Original Google doc:
> https://docs.google.com/document/d/1KTHq3noZBVUQ7QNuBGEhujZ_duwTVpAsvN3Nz5anQUY/edit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on a change in pull request #1839: LUCENE-9511: Include StoredFieldsWriter in DWPT accounting
s1monw commented on a change in pull request #1839: URL: https://github.com/apache/lucene-solr/pull/1839#discussion_r485030437 ## File path: lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThread.java ## @@ -351,7 +354,7 @@ FlushedSegment flush(DocumentsWriter.FlushNotifications flushNotifications) thro flushState.liveDocs.clear(deleteDocIDs[i]); } flushState.delCountOnFlush = numDeletedDocIds; - bytesUsed.addAndGet(-(deleteDocIDs.length * Integer.SIZE)); + bytesUsed.addAndGet(-(deleteDocIDs.length * Integer.BYTES)); Review comment: yeah I checked all instanced in our codebase after I found this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on a change in pull request #1839: LUCENE-9511: Include StoredFieldsWriter in DWPT accounting
s1monw commented on a change in pull request #1839:
URL: https://github.com/apache/lucene-solr/pull/1839#discussion_r485030591
##
File path:
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThread.java
##
@@ -292,7 +295,7 @@ private void deleteLastDocs(int docCount) {
for (int docId = from; docId < to; docId++) {
deleteDocIDs[numDeletedDocIds++] = docId;
}
-bytesUsed.addAndGet((deleteDocIDs.length - size) * Integer.SIZE);
+bytesUsed.addAndGet((deleteDocIDs.length - size) * Integer.BYTES);
Review comment:
yeah same thing
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on a change in pull request #1839: LUCENE-9511: Include StoredFieldsWriter in DWPT accounting
s1monw commented on a change in pull request #1839:
URL: https://github.com/apache/lucene-solr/pull/1839#discussion_r485031017
##
File path:
lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java
##
@@ -52,7 +52,7 @@
/** Default general purpose indexing chain, which handles
* indexing all types of fields. */
final class DefaultIndexingChain extends DocConsumer {
- final Counter bytesUsed;
+ final Counter bytesUsed = Counter.newCounter();
Review comment:
exactly that's the main change here otherwise we need to do back and
forth accounting inside the StoredFieldsWriter
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on pull request #1839: LUCENE-9511: Include StoredFieldsWriter in DWPT accounting
s1monw commented on pull request #1839: URL: https://github.com/apache/lucene-solr/pull/1839#issuecomment-688977953 > Change looks great! Was there a real user use case that triggered this? Or a randomized test failure? I'm curious if the amount of RAM StoredFieldsWriter uses is sometimes substantial ... This was triggered by a test failure that hit and OOM due to huge amount of flushing (maxBufferedDocs=3) and blocked flushes. The SFW uses about 650kb memory per DWPT instance and that might add quickly if flushing can't keep up with indexing. I don't think it will have a big impact on performance but these settings are valid and should never cause OOM. Now that it's accounted for IW stall protection will kick in and safes us from going OOM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14768) Error 500 on PDF extraction: java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
[
https://issues.apache.org/jira/browse/SOLR-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192293#comment-17192293
]
Joe Doupnik commented on SOLR-14768:
Colvin and David,
That's good news, but looking from a distance it is not yet convincing. What
would help us is using regular external agents which feed files into Solr and
do core create/removal as well. That is a final assembly test done in a manner
understandable by others.
I have volunteered my Linux based search service (PHP based) for this, found on
[https://netlab1.net/] as the Solr/Lucene Search Service. That takes less than
an hour to build up completely. As its command line crawler does the work of
interest here it easily blends into the needed final assembly and test
operation. It's log file tells the tale.
> Error 500 on PDF extraction: java.lang.NoClassDefFoundError:
> org/eclipse/jetty/server/MultiParts
>
>
> Key: SOLR-14768
> URL: https://issues.apache.org/jira/browse/SOLR-14768
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: contrib - Solr Cell (Tika extraction)
>Affects Versions: 8.6, 8.6.1
>Reporter: Markus Kalkbrenner
>Assignee: David Smiley
>Priority: Major
> Attachments: Solr v8.6.x fails with multipart MIME in commands.eml,
> Solr v8.6.x fails with multipart MIME in commands.eml, Solr v8.6.x fails with
> multipart MIME in commands.eml
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> See [https://www.mail-archive.com/solr-user@lucene.apache.org/msg152182.html]
> The integration tests of the solarium PHP library and the integration tests
> of the Search API Solr Drupal module both fail on PDF extraction if executed
> on Solr 8.6.
> They still work on Solr 8.5.1 an earlier versions.
> {quote}2020-08-20 12:30:35.279 INFO (qtp855700733-19) [ x:5f3e6ce2810ef]
> o.a.s.u.p.LogUpdateProcessorFactory [5f3e6ce2810ef] webapp=/solr
> path=/update/extract
> params=\{json.nl=flat&commitWithin=0&omitHeader=false&resource.name=testpdf.pdf&literal.id=extract-test&commit=true&extractOnly=false&uprefix=attr_&wt=json}{add=[extract-test
> (1675547519474466816)],commit=} 0 957
> solr8_1 | 2020-08-20 12:30:35.280 WARN (qtp855700733-19) [ ]
> o.e.j.s.HttpChannel /solr/5f3e6ce2810ef/update/extract =>
> java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
> solr8_1 | at
> org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
> solr8_1 | java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
> solr8_1 | at
> org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
> ~[?:?]
> solr8_1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:443)
> ~[?:?]
> solr8_1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> ~[?:?]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> ~[jetty-security-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.j
[jira] [Commented] (LUCENE-9511) Include StoredFieldsWriter in DWPT accounting
[
https://issues.apache.org/jira/browse/LUCENE-9511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192299#comment-17192299
]
ASF subversion and git services commented on LUCENE-9511:
-
Commit 98e55f0ea824ea0f39b6cc66f9d40b2532a85466 in lucene-solr's branch
refs/heads/master from Simon Willnauer
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=98e55f0 ]
LUCENE-9511: Include StoredFieldsWriter in DWPT accounting (#1839)
StoredFieldsWriter might consume some heap space memory that
can have a significant impact on decisions made in the IW if
writers should be stalled or DWPTs should be flushed if memory
settings are small in IWC and flushes are frequent. This change adds
RAM accounting to the StoredFieldsWriter since it's part of the
DWPT lifecycle and not just present during flush.
> Include StoredFieldsWriter in DWPT accounting
> -
>
> Key: LUCENE-9511
> URL: https://issues.apache.org/jira/browse/LUCENE-9511
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Simon Willnauer
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> StoredFieldsWriter might consume some heap space memory that can have a
> significant impact on decisions made in the IW if writers should be stalled
> or DWPTs should be flushed if memory settings are small in IWC and flushes
> are frequent. We should add some accounting to the StoredFieldsWriter since
> it's part of the DWPT lifecycle and not just present during flush.
> Our nightly builds ran into some OOMs due to the large chunk size used in the
> CompressedStoredFieldsFormat. The reason are very frequent flushes due to
> small maxBufferedDocs which causes 300+ DWPTs to be blocked for flush causing
> ultimately an OOM exception.
> {noformat}
>
> NOTE: reproduce with: ant test -Dtestcase=TestIndexingSequenceNumbers
> -Dtests.method=testStressConcurrentCommit -Dtests.seed=A04943A98C8E2954
> -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true
> -Dtests.locale=vo-001 -Dtests.timezone=Africa/Ouagadougou
> -Dtests.asserts=true -Dtests.file.encoding=UTF8*06:06:15*[junit4] ERROR
> 107s J3 | TestIndexingSequenceNumbers.testStressConcurrentCommit
> <<<*06:06:15*[junit4]> Throwable #1:
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is
> closed*06:06:15*[junit4]>at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:876)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:890)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3727)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit(TestIndexingSequenceNumbers.java:228)*06:06:15*
> [junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)*06:06:15*[junit4]>at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)*06:06:15*
> [junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*06:06:15*
> [junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)*06:06:15*
> [junit4]>at
> java.base/java.lang.Thread.run(Thread.java:834)*06:06:15*[junit4]>
> Caused by: java.lang.OutOfMemoryError: Java heap space*06:06:15*[junit4]
> > at
> __randomizedtesting.SeedInfo.seed([A04943A98C8E2954]:0)*06:06:15*[junit4]
>> at
> org.apache.lucene.store.GrowableByteArrayDataOutput.(GrowableByteArrayDataOutput.java:46)*06:06:15*
> [junit4]> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.(CompressingStoredFieldsWriter.java:111)*06:06:15*
> [junit4]> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:130)*06:06:15*
> [junit4]> at
> org.apache.lucene.codecs.lucene87.Lucene87StoredFieldsFormat.fieldsWriter(Lucene87StoredFieldsFormat.java:141)*06:06:15*
> [junit4]>at
> org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:48)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:39)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:46)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:426)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.DefaultIndexingChai
[GitHub] [lucene-solr] s1monw merged pull request #1839: LUCENE-9511: Include StoredFieldsWriter in DWPT accounting
s1monw merged pull request #1839: URL: https://github.com/apache/lucene-solr/pull/1839 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jtibshirani commented on a change in pull request #1834: Make sure to test normal scorers with asserting wrappers.
jtibshirani commented on a change in pull request #1834:
URL: https://github.com/apache/lucene-solr/pull/1834#discussion_r485055929
##
File path:
lucene/test-framework/src/java/org/apache/lucene/search/AssertingWeight.java
##
@@ -85,11 +85,18 @@ public long cost() {
@Override
public BulkScorer bulkScorer(LeafReaderContext context) throws IOException {
-BulkScorer inScorer = in.bulkScorer(context);
+BulkScorer inScorer;
+// We explicitly test both the delegate's bulk scorer, and also the normal
scorer.
+// This ensures that normal scorers are sometimes tested with an asserting
wrapper.
+if (random.nextBoolean()) {
+ inScorer = in.bulkScorer(context);
+} else {
+ inScorer = super.bulkScorer(context);
+}
Review comment:
That makes sense to me, I can reduce the frequency with which we take
the `Scorer` path.
I didn't know about the practice of testing both with a top-level query and
a filter. To me this change is still valuable, it ensures we check the scorer/
iterator contract sometimes in a clear + solid way.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9511) Include StoredFieldsWriter in DWPT accounting
[
https://issues.apache.org/jira/browse/LUCENE-9511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192312#comment-17192312
]
ASF subversion and git services commented on LUCENE-9511:
-
Commit 72448a9f77c9b6bd9458ddcb0322d2dd5b3c890c in lucene-solr's branch
refs/heads/branch_8x from Simon Willnauer
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=72448a9 ]
LUCENE-9511: Include StoredFieldsWriter in DWPT accounting (#1839)
StoredFieldsWriter might consume some heap space memory that
can have a significant impact on decisions made in the IW if
writers should be stalled or DWPTs should be flushed if memory
settings are small in IWC and flushes are frequent. This change adds
RAM accounting to the StoredFieldsWriter since it's part of the
DWPT lifecycle and not just present during flush.
> Include StoredFieldsWriter in DWPT accounting
> -
>
> Key: LUCENE-9511
> URL: https://issues.apache.org/jira/browse/LUCENE-9511
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Simon Willnauer
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> StoredFieldsWriter might consume some heap space memory that can have a
> significant impact on decisions made in the IW if writers should be stalled
> or DWPTs should be flushed if memory settings are small in IWC and flushes
> are frequent. We should add some accounting to the StoredFieldsWriter since
> it's part of the DWPT lifecycle and not just present during flush.
> Our nightly builds ran into some OOMs due to the large chunk size used in the
> CompressedStoredFieldsFormat. The reason are very frequent flushes due to
> small maxBufferedDocs which causes 300+ DWPTs to be blocked for flush causing
> ultimately an OOM exception.
> {noformat}
>
> NOTE: reproduce with: ant test -Dtestcase=TestIndexingSequenceNumbers
> -Dtests.method=testStressConcurrentCommit -Dtests.seed=A04943A98C8E2954
> -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true
> -Dtests.locale=vo-001 -Dtests.timezone=Africa/Ouagadougou
> -Dtests.asserts=true -Dtests.file.encoding=UTF8*06:06:15*[junit4] ERROR
> 107s J3 | TestIndexingSequenceNumbers.testStressConcurrentCommit
> <<<*06:06:15*[junit4]> Throwable #1:
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is
> closed*06:06:15*[junit4]>at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:876)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:890)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3727)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit(TestIndexingSequenceNumbers.java:228)*06:06:15*
> [junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)*06:06:15*[junit4]>at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)*06:06:15*
> [junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*06:06:15*
> [junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)*06:06:15*
> [junit4]>at
> java.base/java.lang.Thread.run(Thread.java:834)*06:06:15*[junit4]>
> Caused by: java.lang.OutOfMemoryError: Java heap space*06:06:15*[junit4]
> > at
> __randomizedtesting.SeedInfo.seed([A04943A98C8E2954]:0)*06:06:15*[junit4]
>> at
> org.apache.lucene.store.GrowableByteArrayDataOutput.(GrowableByteArrayDataOutput.java:46)*06:06:15*
> [junit4]> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.(CompressingStoredFieldsWriter.java:111)*06:06:15*
> [junit4]> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:130)*06:06:15*
> [junit4]> at
> org.apache.lucene.codecs.lucene87.Lucene87StoredFieldsFormat.fieldsWriter(Lucene87StoredFieldsFormat.java:141)*06:06:15*
> [junit4]>at
> org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:48)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:39)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:46)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:426)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.DefaultInde
[jira] [Commented] (SOLR-14151) Make schema components load from packages
[
https://issues.apache.org/jira/browse/SOLR-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192315#comment-17192315
]
Tomas Eduardo Fernandez Lobbe commented on SOLR-14151:
--
Sorry to be late at your latest comments Noble (It was a long weekend in the US
and I was out). Regarding [the async reload of
SolrCore|https://github.com/apache/lucene-solr/pull/1815#discussion_r483241419],
I think you should just remove that method. It's just dead and untested code
now.
> Make schema components load from packages
> -
>
> Key: SOLR-14151
> URL: https://issues.apache.org/jira/browse/SOLR-14151
> Project: Solr
> Issue Type: Sub-task
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Labels: packagemanager
> Fix For: 8.7
>
> Time Spent: 12h 40m
> Remaining Estimate: 0h
>
> Example:
> {code:xml}
>
>
>
>generateNumberParts="0" catenateWords="0"
> catenateNumbers="0" catenateAll="0"/>
>
>
>
>
> {code}
> * When a package is updated, the entire {{IndexSchema}} object is refreshed,
> but the SolrCore object is not reloaded
> * Any component can be prefixed with the package name
> * The semantics of loading plugins remain the same as that of the components
> in {{solrconfig.xml}}
> * Plugins can be registered using schema API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14723) Rremove the class atribute for the caches in the _default/example configsets
[
https://issues.apache.org/jira/browse/SOLR-14723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192342#comment-17192342
]
Tomas Eduardo Fernandez Lobbe commented on SOLR-14723:
--
bq. Should I do my changes on the master branch in my forked repo and raise a
PR against the master in main repo?
While this is usually the way to go, this Jira issue in particular is about
removing the {{class}} attribute from the 8.x configsets (you won't find the
attribute in master anymore). So you should work on your fork's
{{branch_8x}}(or create a branch our of {{branch_8x}}) and then create a PR
against {{branch_8x}} of the main Apache GH repo.
> Rremove the class atribute for the caches in the _default/example configsets
>
>
> Key: SOLR-14723
> URL: https://issues.apache.org/jira/browse/SOLR-14723
> Project: Solr
> Issue Type: Improvement
>Reporter: Tomas Eduardo Fernandez Lobbe
>Priority: Trivial
> Labels: newdev
>
> In 8.x branch, the default and example configsets include LRUCache and
> FastLRUCache as the implementation for the various caches.
> {code:xml}
> size="512"
> initialSize="512"
> autowarmCount="0"/>
> {code}
> Those implementations are deprecated. We should remove the attribute
> completely and let the default take over (as it's in master), this should
> make upgrading configsets easier:
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] uschindler opened a new pull request #1842: LUCENE-9512: Move LockFactory stress test to be a unit/integration test
uschindler opened a new pull request #1842: URL: https://github.com/apache/lucene-solr/pull/1842 This PR moves the lock factory stress test to be a :lucene:core unit/integration test. In verbose mode the output of clients is inherited, in standard test mode its written to files (to allow debugging). Unfortunately, as the server uses TCP/IP socket to comunicate with the childs, I have to add ephemeral to lucene's policy file (@rmuir looks ok?). If this gets backported to 8.x, we can get rid of the rather complicated Ant Target, too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9512) Add test-lock-factory to Gradle build
[
https://issues.apache.org/jira/browse/LUCENE-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192346#comment-17192346
]
Uwe Schindler commented on LUCENE-9512:
---
Hi Dawid, I created a PR: https://github.com/apache/lucene-solr/pull/1842
> Add test-lock-factory to Gradle build
> -
>
> Key: LUCENE-9512
> URL: https://issues.apache.org/jira/browse/LUCENE-9512
> Project: Lucene - Core
> Issue Type: Task
> Components: core/store
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When porting to Gradle, the following task was missed to be ported to Gradle:
> https://github.com/apache/lucene-solr/blob/branch_8x/lucene/core/build.xml#L148-L234
> This is somehow an integration test. It's not part of the test suite, as the
> code is part of main distribution and is a client-server implementation that
> has one coordinator to handle other JVMs to hammer a directories' lock
> factory.
> It may be included into the test suite, but as it spawns multiple JVMs (thats
> essential for it to work), I see this as a separate thing.
> I would like to implement that snippet of ANT code in Gradle and attach it to
> {{:lucene:core}}'s {{test}} task. If we have an integration test framework at
> some point we can make a real {{integTest}} out of it, but for now a simple
> Groovy script is fine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9512) Add test-lock-factory to Gradle build
[
https://issues.apache.org/jira/browse/LUCENE-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192348#comment-17192348
]
Uwe Schindler commented on LUCENE-9512:
---
output looks like that (verbose mode):
{noformat}
$ gradlew :lucene:core:test -Dtests.verbose=true --tests
TestLockFactoriesMultiJVM
[...]
> Task :lucene:core:test
1> filesystem:
ExtrasFS(HandleLimitFS(LeakFS(ShuffleFS(DisableFsyncFS(VerboseFS(sun.nio.fs.WindowsFileSystemProvider@79123fdb))
1> FS 0 [2020-09-08T17:17:38.542260800Z;
SUITE-TestLockFactoriesMultiJVM-seed#[36EDDD406FA33CBE]-worker]:
createDirectory: ..\tests-tmp (FAILED:
java.nio.file.FileAlreadyExistsException: C:\Users\Uwe
Schindler\Projects\lucene\trunk-lusolr1\lucene\core\build\tmp\tests-tmp)
1> Loaded codecs: [Lucene87, Asserting, CheapBastard,
DeflateWithPresetCompressingStoredFieldsData, FastCompressingStoredFieldsData,
FastDecompressionCompressingStoredFieldsData,
HighCompressionCompressingStoredFieldsData,
LZ4WithPresetCompressingStoredFieldsData, DummyCompressingStoredFieldsData,
SimpleText]
1> Loaded postingsFormats: [Lucene84, MockRandom, RAMOnly, LuceneFixedGap,
LuceneVarGapFixedInterval, LuceneVarGapDocFreqInterval,
TestBloomFilteredLucenePostings, Asserting, UniformSplitRot13,
STUniformSplitRot13, BlockTreeOrds, BloomFilter, Direct, FST50, UniformSplit,
SharedTermsUniformSplit]
1> FS 0 [2020-09-08T17:17:38.730612800Z; main]: createDirectory:
..\tests-tmp\lucene.store.TestLockFactoriesMultiJVM_36EDDD406FA33CBE-001
1> FS 0 [2020-09-08T17:17:38.732606300Z; main]: createDirectory:
..\tests-tmp\lucene.store.TestLockFactoriesMultiJVM_36EDDD406FA33CBE-001\NativeFSLockFactory-001
1> Listening on /127.0.0.1:49744...
Connecting to server /127.0.0.1:49744 and registering as client 1...Connecting
to server /127.0.0.1:49744 and registering as client 0...
1> All clients started, fire gun...
0.0% done.
0.0% done.
1> Server terminated.
Finished 500 tries.
Finished 500 tries.
1> FS 0 [2020-09-08T17:17:41.080680200Z; main]: createDirectory:
..\tests-tmp\lucene.store.TestLockFactoriesMultiJVM_36EDDD406FA33CBE-001\SimpleFSLockFactory-001
1> Listening on /127.0.0.1:49747...
Connecting to server /127.0.0.1:49747 and registering as client 0...
Connecting to server /127.0.0.1:49747 and registering as client 1...
1> All clients started, fire gun...
0.0% done.
0.0% done.
Finished 500 tries.
1> Server terminated.
Finished 500 tries.
1> FS 0 [2020-09-08T17:17:42.774322100Z;
SUITE-TestLockFactoriesMultiJVM-seed#[36EDDD406FA33CBE]-worker]: delete:
..\tests-tmp\lucene.store.TestLockFactoriesMultiJVM_36EDDD406FA33CBE-001\SimpleFSLockFactory-001
1> FS 0 [2020-09-08T17:17:42.775321100Z;
SUITE-TestLockFactoriesMultiJVM-seed#[36EDDD406FA33CBE]-worker]: delete:
..\tests-tmp\lucene.store.TestLockFactoriesMultiJVM_36EDDD406FA33CBE-001\NativeFSLockFactory-001\test.lock
1> FS 0 [2020-09-08T17:17:42.775321100Z;
SUITE-TestLockFactoriesMultiJVM-seed#[36EDDD406FA33CBE]-worker]: delete:
..\tests-tmp\lucene.store.TestLockFactoriesMultiJVM_36EDDD406FA33CBE-001\NativeFSLockFactory-001
1> FS 0 [2020-09-08T17:17:42.776319Z;
SUITE-TestLockFactoriesMultiJVM-seed#[36EDDD406FA33CBE]-worker]: delete:
..\tests-tmp\lucene.store.TestLockFactoriesMultiJVM_36EDDD406FA33CBE-001
2> NOTE: test params are:
codec=LZ4WithPresetCompressingStoredFieldsData(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=BEST_SPEED,
chunkSize=15163, maxDocsPerChunk=752, blockShift=22),
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=BEST_SPEED,
chunkSize=15163, blockSize=22)),
sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=ar-MR,
timezone=America/Port-au-Prince
2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 11.0.2
(64-bit)/cpus=8,threads=1,free=221295104,total=268435456
2> NOTE: All tests run in this JVM: [TestLockFactoriesMultiJVM]
:lucene:core:test (SUCCESS): 2 test(s)
The slowest tests (exceeding 500 ms) during this run:
2.36s TestLockFactoriesMultiJVM.testNativeFSLockFactory (:lucene:core)
1.70s TestLockFactoriesMultiJVM.testSimpleFSLockFactory (:lucene:core)
{noformat}
> Add test-lock-factory to Gradle build
> -
>
> Key: LUCENE-9512
> URL: https://issues.apache.org/jira/browse/LUCENE-9512
> Project: Lucene - Core
> Issue Type: Task
> Components: core/store
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When porting to Gradle, the following task was missed to be ported to Gradle:
> https://github.com/apache/lucene-solr/blob/branch_8x/lucene/core/build.xml#L148-L234
> This is somehow an integration test. It's not part of the test suite, as the
> code is part of main distribution and is a client-server implementation t
[GitHub] [lucene-solr] murblanc commented on pull request #1684: SOLR-14613: strongly typed placement plugin interface and implementation
murblanc commented on pull request #1684: URL: https://github.com/apache/lucene-solr/pull/1684#issuecomment-689041781 I am closing this PR now. Will create a new one with the up to date state of the code as resulting from the discussions here, but things became way too slow to still be usable in this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] murblanc closed pull request #1684: SOLR-14613: strongly typed placement plugin interface and implementation
murblanc closed pull request #1684: URL: https://github.com/apache/lucene-solr/pull/1684 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] murblanc commented on pull request #1684: SOLR-14613: strongly typed placement plugin interface and implementation
murblanc commented on pull request #1684: URL: https://github.com/apache/lucene-solr/pull/1684#issuecomment-689042430 And please DO NOT DELETE the murblanc:SOLR-14613-gumi branch, I need it around for a while. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9449) Skip non-competitive documents when sort by _doc with search after
[ https://issues.apache.org/jira/browse/LUCENE-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192379#comment-17192379 ] ASF subversion and git services commented on LUCENE-9449: - Commit 99220677fe409bab5c4206dc31f186d0c4335280 in lucene-solr's branch refs/heads/master from Mayya Sharipova [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=9922067 ] LUCENE-9449 Skip docs with _doc sort and "after" (#1725) - Enhance DocComparator to provide an iterator over competitive documents when searching with "after". This iterator can quickly position on the desired "after" document skipping all documents and segments before "after". - Redesign numeric comparators to provide skipping functionality by default. Relates to LUCENE-9280 > Skip non-competitive documents when sort by _doc with search after > -- > > Key: LUCENE-9449 > URL: https://issues.apache.org/jira/browse/LUCENE-9449 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Mayya Sharipova >Priority: Minor > Time Spent: 4h 40m > Remaining Estimate: 0h > > Enhance DocComparator to provide an iterator over competitive documents when > search ing with "after" FieldDoc. > This iterator can quickly position on the desired "after" document, and skip > all documents or even segments that contain documents before "after" > This is especially efficient when "after" is high. > > Related to LUCENE-9280 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova merged pull request #1725: LUCENE-9449 Skip docs with _doc sort and "after"
mayya-sharipova merged pull request #1725: URL: https://github.com/apache/lucene-solr/pull/1725 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9280) Add ability to skip non-competitive documents on field sort
[ https://issues.apache.org/jira/browse/LUCENE-9280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192380#comment-17192380 ] ASF subversion and git services commented on LUCENE-9280: - Commit 99220677fe409bab5c4206dc31f186d0c4335280 in lucene-solr's branch refs/heads/master from Mayya Sharipova [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=9922067 ] LUCENE-9449 Skip docs with _doc sort and "after" (#1725) - Enhance DocComparator to provide an iterator over competitive documents when searching with "after". This iterator can quickly position on the desired "after" document skipping all documents and segments before "after". - Redesign numeric comparators to provide skipping functionality by default. Relates to LUCENE-9280 > Add ability to skip non-competitive documents on field sort > > > Key: LUCENE-9280 > URL: https://issues.apache.org/jira/browse/LUCENE-9280 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Mayya Sharipova >Priority: Minor > Fix For: master (9.0) > > Time Spent: 18h 20m > Remaining Estimate: 0h > > Today collectors, once they collect enough docs, can instruct scorers to > update their iterators to skip non-competitive documents. This is applicable > only for a case when we need top docs by _score. > It would be nice to also have an ability to skip non-competitive docs when we > need top docs sorted by other fields different from _score. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14846) Avoid use of Optional.ofNullable.orElse in the same line
Mike Drob created SOLR-14846:
Summary: Avoid use of Optional.ofNullable.orElse in the same line
Key: SOLR-14846
URL: https://issues.apache.org/jira/browse/SOLR-14846
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Mike Drob
Assignee: Mike Drob
Optional was meant to be used in public APIs to convey to the caller that they
really need to null-check, not as an internal class to avoid doing our own null
checks. There are better methods in {{Objects}} for that if we want them.
See also Brian Goetz at JavaOne - https://youtu.be/MFzlgAuanU0?t=2719
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob opened a new pull request #1843: SOLR-14846 Remove Optional.ofNullable.orElse pattern
madrob opened a new pull request #1843: URL: https://github.com/apache/lucene-solr/pull/1843 https://issues.apache.org/jira/browse/SOLR-14846 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14846) Avoid use of Optional.ofNullable.orElse in the same line
[
https://issues.apache.org/jira/browse/SOLR-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192385#comment-17192385
]
Mike Drob commented on SOLR-14846:
--
He also discourages using Optional as an Object field, so we could clean that
up in a few places too, but that would change the API maybe? What do folks
think - it looks like mostly around Backup and Restore operations.
> Avoid use of Optional.ofNullable.orElse in the same line
>
>
> Key: SOLR-14846
> URL: https://issues.apache.org/jira/browse/SOLR-14846
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Optional was meant to be used in public APIs to convey to the caller that
> they really need to null-check, not as an internal class to avoid doing our
> own null checks. There are better methods in {{Objects}} for that if we want
> them.
> See also Brian Goetz at JavaOne - https://youtu.be/MFzlgAuanU0?t=2719
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14846) Avoid use of Optional.ofNullable.orElse in the same line
[
https://issues.apache.org/jira/browse/SOLR-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192386#comment-17192386
]
Mike Drob commented on SOLR-14846:
--
cc: [~varun] - you contributed some of the Backup stuff, WDYT?
> Avoid use of Optional.ofNullable.orElse in the same line
>
>
> Key: SOLR-14846
> URL: https://issues.apache.org/jira/browse/SOLR-14846
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Optional was meant to be used in public APIs to convey to the caller that
> they really need to null-check, not as an internal class to avoid doing our
> own null checks. There are better methods in {{Objects}} for that if we want
> them.
> See also Brian Goetz at JavaOne - https://youtu.be/MFzlgAuanU0?t=2719
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1842: LUCENE-9512: Move LockFactory stress test to be a unit/integration test
dweiss commented on a change in pull request #1842:
URL: https://github.com/apache/lucene-solr/pull/1842#discussion_r485116034
##
File path: gradle/testing/randomization/policies/tests.policy
##
@@ -64,6 +64,9 @@ grant {
permission java.lang.RuntimePermission "getClassLoader";
permission java.lang.RuntimePermission "setContextClassLoader";
+ // TestLockFactoriesMultiJVM opens a random port on 127.0.0.1 (port 0 =
ephemeral port range):
Review comment:
This property wouldn't be needed if you forked the "server: side as well
as clients... :)
##
File path:
lucene/core/src/test/org/apache/lucene/store/TestLockFactoriesMultiJVM.java
##
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.store;
+
+import java.lang.ProcessBuilder.Redirect;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.NamedThreadFactory;
+import org.apache.lucene.util.SuppressForbidden;
+
+public class TestLockFactoriesMultiJVM extends LuceneTestCase {
+
+ @SuppressForbidden(reason = "ProcessBuilder only allows to redirect to
java.io.File")
+ private static final ProcessBuilder applyRedirection(ProcessBuilder pb, int
client, Path dir) {
+if (LuceneTestCase.VERBOSE) {
+ return pb.inheritIO();
+} else {
+ return pb
+.redirectError(dir.resolve("err-" + client + ".txt").toFile())
+.redirectOutput(dir.resolve("out-" + client + ".txt").toFile())
+.redirectInput(Redirect.INHERIT);
+}
+ }
+
+ private void runImpl(Class impl) throws Exception {
+// make sure we are in clean state:
+LockVerifyServer.PORT.set(-1);
+
+final int clients = 2;
+final String host = "127.0.0.1";
+final int delay = 1;
+final int rounds = (LuceneTestCase.TEST_NIGHTLY ? 3 : 500) *
LuceneTestCase.RANDOM_MULTIPLIER;
+
+final Path dir = LuceneTestCase.createTempDir(impl.getSimpleName());
+
+// create the LockVerifyServer in a separate thread
+final ExecutorService pool = Executors.newSingleThreadExecutor(new
NamedThreadFactory("lockfactory-tester-"));
+try {
+ pool.submit(() -> {
+LockVerifyServer.main(host, Integer.toString(clients));
+return (Void) null;
+ });
+
+ // wait for it to boot up
+ int port;
+ while ((port = LockVerifyServer.PORT.get()) == -1) {
+Thread.sleep(100L);
+ }
+
+ // spawn clients as separate Java processes
+ final List processes = new ArrayList<>();
+ for (int i = 0; i < clients; i++) {
+processes.add(applyRedirection(new ProcessBuilder(
+Paths.get(System.getProperty("java.home"), "bin",
"java").toString(),
+"-cp",
+System.getProperty("java.class.path"),
+LockStressTest.class.getName(),
+Integer.toString(i),
+host,
+Integer.toString(port),
+impl.getName(),
+dir.toString(),
+Integer.toString(delay),
+Integer.toString(rounds)
+ ), i, dir).start());
+ }
+
+ // wait for all processes to exit
+ int exited = 0;
+ while (exited < clients) {
+for (Process p : processes) {
+ if (p.waitFor(1, TimeUnit.SECONDS)) {
Review comment:
1 second may not be enough on heavy loaded machines?
##
File path:
lucene/core/src/test/org/apache/lucene/store/TestLockFactoriesMultiJVM.java
##
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/
[jira] [Commented] (LUCENE-9511) Include StoredFieldsWriter in DWPT accounting
[
https://issues.apache.org/jira/browse/LUCENE-9511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192398#comment-17192398
]
Michael McCandless commented on LUCENE-9511:
{quote}The reason are very frequent flushes due to small maxBufferedDocs which
causes 300+ DWPTs to be blocked for flush causing ultimately an OOM exception.
{quote}
Wild :) Go randomized testing!
I guess this would mean applications using more threads to index would have
seen RAM consumed but not tracked properly by {{IndexWriter}} until this fix.
> Include StoredFieldsWriter in DWPT accounting
> -
>
> Key: LUCENE-9511
> URL: https://issues.apache.org/jira/browse/LUCENE-9511
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Simon Willnauer
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> StoredFieldsWriter might consume some heap space memory that can have a
> significant impact on decisions made in the IW if writers should be stalled
> or DWPTs should be flushed if memory settings are small in IWC and flushes
> are frequent. We should add some accounting to the StoredFieldsWriter since
> it's part of the DWPT lifecycle and not just present during flush.
> Our nightly builds ran into some OOMs due to the large chunk size used in the
> CompressedStoredFieldsFormat. The reason are very frequent flushes due to
> small maxBufferedDocs which causes 300+ DWPTs to be blocked for flush causing
> ultimately an OOM exception.
> {noformat}
>
> NOTE: reproduce with: ant test -Dtestcase=TestIndexingSequenceNumbers
> -Dtests.method=testStressConcurrentCommit -Dtests.seed=A04943A98C8E2954
> -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true
> -Dtests.locale=vo-001 -Dtests.timezone=Africa/Ouagadougou
> -Dtests.asserts=true -Dtests.file.encoding=UTF8*06:06:15*[junit4] ERROR
> 107s J3 | TestIndexingSequenceNumbers.testStressConcurrentCommit
> <<<*06:06:15*[junit4]> Throwable #1:
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is
> closed*06:06:15*[junit4]>at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:876)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:890)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3727)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit(TestIndexingSequenceNumbers.java:228)*06:06:15*
> [junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)*06:06:15*[junit4]>at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)*06:06:15*
> [junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*06:06:15*
> [junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)*06:06:15*
> [junit4]>at
> java.base/java.lang.Thread.run(Thread.java:834)*06:06:15*[junit4]>
> Caused by: java.lang.OutOfMemoryError: Java heap space*06:06:15*[junit4]
> > at
> __randomizedtesting.SeedInfo.seed([A04943A98C8E2954]:0)*06:06:15*[junit4]
>> at
> org.apache.lucene.store.GrowableByteArrayDataOutput.(GrowableByteArrayDataOutput.java:46)*06:06:15*
> [junit4]> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.(CompressingStoredFieldsWriter.java:111)*06:06:15*
> [junit4]> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:130)*06:06:15*
> [junit4]> at
> org.apache.lucene.codecs.lucene87.Lucene87StoredFieldsFormat.fieldsWriter(Lucene87StoredFieldsFormat.java:141)*06:06:15*
> [junit4]>at
> org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:48)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:39)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:46)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:426)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:462)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:233)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.DocumentsWriter.updateDocum
[GitHub] [lucene-solr] mikemccand commented on pull request #1837: LUCENE-7882: First idea of using Java 15 hidden anonymous classes for Lucene expressions
mikemccand commented on pull request #1837: URL: https://github.com/apache/lucene-solr/pull/1837#issuecomment-689070197 Whoa, thanks @uschindler -- this looks awesome. > @mikemccand can you test this with JDK 15 (release candidate) and your test. You should not see any locks anymore, speed should be higher, and the created anonymous classes should be unloaded very fast. We (Amazon customer facing product search team) make heavy use of Lucene's (powerful!) expressions module -- I will try to figure out a way to test this (we are on JDK 11 now for production). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] rmuir commented on a change in pull request #1842: LUCENE-9512: Move LockFactory stress test to be a unit/integration test
rmuir commented on a change in pull request #1842:
URL: https://github.com/apache/lucene-solr/pull/1842#discussion_r485136795
##
File path:
lucene/core/src/test/org/apache/lucene/store/TestLockFactoriesMultiJVM.java
##
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.store;
+
+import java.lang.ProcessBuilder.Redirect;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.NamedThreadFactory;
+import org.apache.lucene.util.SuppressForbidden;
+
+public class TestLockFactoriesMultiJVM extends LuceneTestCase {
+
+ @SuppressForbidden(reason = "ProcessBuilder only allows to redirect to
java.io.File")
+ private static final ProcessBuilder applyRedirection(ProcessBuilder pb, int
client, Path dir) {
+if (LuceneTestCase.VERBOSE) {
+ return pb.inheritIO();
+} else {
+ return pb
+.redirectError(dir.resolve("err-" + client + ".txt").toFile())
+.redirectOutput(dir.resolve("out-" + client + ".txt").toFile())
+.redirectInput(Redirect.INHERIT);
+}
+ }
+
+ private void runImpl(Class impl) throws Exception {
+// make sure we are in clean state:
+LockVerifyServer.PORT.set(-1);
+
+final int clients = 2;
+final String host = "127.0.0.1";
+final int delay = 1;
+final int rounds = (LuceneTestCase.TEST_NIGHTLY ? 3 : 500) *
LuceneTestCase.RANDOM_MULTIPLIER;
+
+final Path dir = LuceneTestCase.createTempDir(impl.getSimpleName());
+
+// create the LockVerifyServer in a separate thread
+final ExecutorService pool = Executors.newSingleThreadExecutor(new
NamedThreadFactory("lockfactory-tester-"));
+try {
+ pool.submit(() -> {
+LockVerifyServer.main(host, Integer.toString(clients));
+return (Void) null;
+ });
+
+ // wait for it to boot up
+ int port;
+ while ((port = LockVerifyServer.PORT.get()) == -1) {
+Thread.sleep(100L);
+ }
+
+ // spawn clients as separate Java processes
+ final List processes = new ArrayList<>();
+ for (int i = 0; i < clients; i++) {
+processes.add(applyRedirection(new ProcessBuilder(
+Paths.get(System.getProperty("java.home"), "bin",
"java").toString(),
+"-cp",
+System.getProperty("java.class.path"),
+LockStressTest.class.getName(),
+Integer.toString(i),
+host,
+Integer.toString(port),
+impl.getName(),
+dir.toString(),
+Integer.toString(delay),
+Integer.toString(rounds)
+ ), i, dir).start());
+ }
+
+ // wait for all processes to exit
+ int exited = 0;
+ while (exited < clients) {
+for (Process p : processes) {
+ if (p.waitFor(1, TimeUnit.SECONDS)) {
Review comment:
this is inside a loop though: so it will effectively wait forever?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] rmuir commented on a change in pull request #1842: LUCENE-9512: Move LockFactory stress test to be a unit/integration test
rmuir commented on a change in pull request #1842:
URL: https://github.com/apache/lucene-solr/pull/1842#discussion_r485138469
##
File path:
lucene/core/src/test/org/apache/lucene/store/TestLockFactoriesMultiJVM.java
##
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.store;
+
+import java.lang.ProcessBuilder.Redirect;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.NamedThreadFactory;
+import org.apache.lucene.util.SuppressForbidden;
+
+public class TestLockFactoriesMultiJVM extends LuceneTestCase {
Review comment:
FYI I ran it locally on a 2-core laptop. The tests are a bit slow but
not the slowest in lucene-core.
The slowest tests (exceeding 500 ms) during this run:
2.76s TestIndexWriter.testMaxCompletedSequenceNumber (:lucene:core)
1.96s TestNRTCachingDirectory.testNRTAndCommit (:lucene:core)
1.69s TestIndexWriter.testThreadInterruptDeadlock (:lucene:core)
1.60s TestShardSearching.testSimple (:lucene:core)
1.55s TestStandardAnalyzer.testLargePartiallyMatchingToken (:lucene:core)
1.46s TestTieredMergePolicy.testSimulateAppendOnly (:lucene:core)
**1.32s TestLockFactoriesMultiJVM.testNativeFSLockFactory (:lucene:core)**
**1.23s TestLockFactoriesMultiJVM.testSimpleFSLockFactory (:lucene:core)**
1.17s TestLevenshteinAutomata.testLev2 (:lucene:core)
1.15s TestIndexSorting.testConcurrentUpdates (:lucene:core)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1842: LUCENE-9512: Move LockFactory stress test to be a unit/integration test
dweiss commented on a change in pull request #1842:
URL: https://github.com/apache/lucene-solr/pull/1842#discussion_r485152152
##
File path:
lucene/core/src/test/org/apache/lucene/store/TestLockFactoriesMultiJVM.java
##
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.store;
+
+import java.lang.ProcessBuilder.Redirect;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.NamedThreadFactory;
+import org.apache.lucene.util.SuppressForbidden;
+
+public class TestLockFactoriesMultiJVM extends LuceneTestCase {
+
+ @SuppressForbidden(reason = "ProcessBuilder only allows to redirect to
java.io.File")
+ private static final ProcessBuilder applyRedirection(ProcessBuilder pb, int
client, Path dir) {
+if (LuceneTestCase.VERBOSE) {
+ return pb.inheritIO();
+} else {
+ return pb
+.redirectError(dir.resolve("err-" + client + ".txt").toFile())
+.redirectOutput(dir.resolve("out-" + client + ".txt").toFile())
+.redirectInput(Redirect.INHERIT);
+}
+ }
+
+ private void runImpl(Class impl) throws Exception {
+// make sure we are in clean state:
+LockVerifyServer.PORT.set(-1);
+
+final int clients = 2;
+final String host = "127.0.0.1";
+final int delay = 1;
+final int rounds = (LuceneTestCase.TEST_NIGHTLY ? 3 : 500) *
LuceneTestCase.RANDOM_MULTIPLIER;
+
+final Path dir = LuceneTestCase.createTempDir(impl.getSimpleName());
+
+// create the LockVerifyServer in a separate thread
+final ExecutorService pool = Executors.newSingleThreadExecutor(new
NamedThreadFactory("lockfactory-tester-"));
+try {
+ pool.submit(() -> {
+LockVerifyServer.main(host, Integer.toString(clients));
+return (Void) null;
+ });
+
+ // wait for it to boot up
+ int port;
+ while ((port = LockVerifyServer.PORT.get()) == -1) {
+Thread.sleep(100L);
+ }
+
+ // spawn clients as separate Java processes
+ final List processes = new ArrayList<>();
+ for (int i = 0; i < clients; i++) {
+processes.add(applyRedirection(new ProcessBuilder(
+Paths.get(System.getProperty("java.home"), "bin",
"java").toString(),
+"-cp",
+System.getProperty("java.class.path"),
+LockStressTest.class.getName(),
+Integer.toString(i),
+host,
+Integer.toString(port),
+impl.getName(),
+dir.toString(),
+Integer.toString(delay),
+Integer.toString(rounds)
+ ), i, dir).start());
+ }
+
+ // wait for all processes to exit
+ int exited = 0;
+ while (exited < clients) {
+for (Process p : processes) {
+ if (p.waitFor(1, TimeUnit.SECONDS)) {
Review comment:
Ah, correct. A completable future wait-for would be more elegant here I
think but fine as it is too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14846) Avoid use of Optional.ofNullable.orElse in the same line
[
https://issues.apache.org/jira/browse/SOLR-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192418#comment-17192418
]
Varun Thacker commented on SOLR-14846:
--
Thanks for linking the video! I think the changes makes sense!
> Avoid use of Optional.ofNullable.orElse in the same line
>
>
> Key: SOLR-14846
> URL: https://issues.apache.org/jira/browse/SOLR-14846
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Optional was meant to be used in public APIs to convey to the caller that
> they really need to null-check, not as an internal class to avoid doing our
> own null checks. There are better methods in {{Objects}} for that if we want
> them.
> See also Brian Goetz at JavaOne - https://youtu.be/MFzlgAuanU0?t=2719
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] vthacker commented on a change in pull request #1843: SOLR-14846 Remove Optional.ofNullable.orElse pattern
vthacker commented on a change in pull request #1843:
URL: https://github.com/apache/lucene-solr/pull/1843#discussion_r485154167
##
File path:
solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpClientBuilderFactory.java
##
@@ -36,7 +35,7 @@
*by configured (optional).
* @return the {@linkplain SolrHttpClientBuilder}
*/
- public SolrHttpClientBuilder
getHttpClientBuilder(Optional builder);
+ public SolrHttpClientBuilder getHttpClientBuilder(SolrHttpClientBuilder
builder);
Review comment:
I'm wondering if this is a change we want to do in 9.0 only?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14844) Upgrade Jetty to 9.4.31
[ https://issues.apache.org/jira/browse/SOLR-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192420#comment-17192420 ] Erick Erickson commented on SOLR-14844: --- I'm getting a 100% reproducible error on 8x that I won't be able to get to for a day or two. The seed doesn't seem to matter. Doesn't happen without the jetty upgrade. 4873 INFO (TEST-BasicHttpSolrClientTest.testCompression-seed#[56B06FED8C90466]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testCompression NOTE: reproduce with: ant test -Dtestcase=BasicHttpSolrClientTest -Dtests.method=testCompression -Dtests.seed=56B06FED8C90466 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ca -Dtests.timezone=Pacific/Apia -Dtests.asserts=true -Dtests.file.encoding=UTF-8 org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: https://127.0.0.1:53586/solr/debug/foo at __randomizedtesting.SeedInfo.seed([56B06FED8C90466:FE5E2098D217F435]:0) at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:695) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1003) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1018) at org.apache.solr.client.solrj.impl.BasicHttpSolrClientTest.testCompression(BasicHttpSolrClientTest.java:491) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com
[jira] [Created] (SOLR-14847) Solr packaging needs to produce server tgz for release
Mike Drob created SOLR-14847: Summary: Solr packaging needs to produce server tgz for release Key: SOLR-14847 URL: https://issues.apache.org/jira/browse/SOLR-14847 Project: Solr Issue Type: Task Security Level: Public (Default Security Level. Issues are Public) Reporter: Mike Drob We currently have a solr/packaging assemble target that creates a directory of all the server contents, but we don't have any targets that would generate a tgz like for releases. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob opened a new pull request #1844: SOLR-14847 Create Solr Server TGZ
madrob opened a new pull request #1844: URL: https://github.com/apache/lucene-solr/pull/1844 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14843) Define strongly-typed cluster configuration API
[
https://issues.apache.org/jira/browse/SOLR-14843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192425#comment-17192425
]
David Smiley commented on SOLR-14843:
-
Yeah a SIP. Lets look at this in a light-weight way... copy-paste to a
Confluence doc, and start a dev list discussion pointing to it, perhaps with
the same text in the email as well. The dev list discussion enhances
visibility about SIP-worthy matters. The SIP part I'm more dubious on is
voting, but it could be very lazy-consensus -- at some point say what you plan
to do and just start doing it unless you get a veto. CC [~janhoy]
[~ilan] Code/config evolution is a general issue that is not specific to this
proposal. I've seen the first option you list (code2 reads config1) used most
often. It's basically the writer of the code in question to deal with this,
not the platform. It's not clear to me how this proposal would be modified to
handle this concern in the platform itself... and I'm a little skeptical about
wether it should.
> Define strongly-typed cluster configuration API
> ---
>
> Key: SOLR-14843
> URL: https://issues.apache.org/jira/browse/SOLR-14843
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Andrzej Bialecki
>Priority: Major
> Labels: clean-api
> Fix For: master (9.0)
>
>
> Current cluster-level configuration uses a hodgepodge of traditional Solr
> config sources (solr.xml, system properties) and the new somewhat arbitrary
> config files kept in ZK ({{/clusterprops.json, /security.json,
> /packages.json, /autoscaling.json}} etc...). There's no uniform
> strongly-typed API to access and manage these configs - currently each config
> source has its own CRUD, often relying on direct access to Zookeeper. There's
> also no uniform method for monitoring changes to these config sources.
> This issue proposes a uniform config API facade with the following
> characteristics:
> * Using a single hierarchical (or at least key-based) facade for accessing
> any global config.
> * Using strongly-typed sub-system configs instead of opaque Map-s:
> components would no longer deal with JSON parsing/writing, instead they would
> use properly annotated Java objects for config CRUD. Config objects would
> include versioning information (eg. lastModified timestamp).
> * Isolating access to the underlying config persistence layer: components
> would no longer directly interact with Zookeeper or files. Most likely the
> default implementation would continue using different ZK files per-subsystem
> in order to limit the complexity of file formats and to reduce the cost of
> notifications for unmodified parts of the configs.
> * Providing uniform way to register listeners for monitoring changes in
> specific configs: components would no longer need to interact with ZK
> watches, they would instead be notified about modified configs that they are
> interested in.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1844: SOLR-14847 Create Solr Server TGZ
dweiss commented on a change in pull request #1844:
URL: https://github.com/apache/lucene-solr/pull/1844#discussion_r485162521
##
File path: solr/packaging/build.gradle
##
@@ -109,6 +109,21 @@ task toDir(type: Sync) {
}
}
+task toTgz(type: Tar) {
Review comment:
It is likely that this will require explicit file permissions to work
properly on Windows machines. Don't know if tar task will accept a workaround
like this one -
https://github.com/carrot2/carrot2/blob/master/distribution/build.gradle#L94-L98
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on a change in pull request #1842: LUCENE-9512: Move LockFactory stress test to be a unit/integration test
uschindler commented on a change in pull request #1842:
URL: https://github.com/apache/lucene-solr/pull/1842#discussion_r485169448
##
File path:
lucene/core/src/test/org/apache/lucene/store/TestLockFactoriesMultiJVM.java
##
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.store;
+
+import java.lang.ProcessBuilder.Redirect;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.NamedThreadFactory;
+import org.apache.lucene.util.SuppressForbidden;
+
+public class TestLockFactoriesMultiJVM extends LuceneTestCase {
+
+ @SuppressForbidden(reason = "ProcessBuilder only allows to redirect to
java.io.File")
+ private static final ProcessBuilder applyRedirection(ProcessBuilder pb, int
client, Path dir) {
+if (LuceneTestCase.VERBOSE) {
+ return pb.inheritIO();
+} else {
+ return pb
+.redirectError(dir.resolve("err-" + client + ".txt").toFile())
+.redirectOutput(dir.resolve("out-" + client + ".txt").toFile())
+.redirectInput(Redirect.INHERIT);
+}
+ }
+
+ private void runImpl(Class impl) throws Exception {
+// make sure we are in clean state:
+LockVerifyServer.PORT.set(-1);
+
+final int clients = 2;
+final String host = "127.0.0.1";
+final int delay = 1;
+final int rounds = (LuceneTestCase.TEST_NIGHTLY ? 3 : 500) *
LuceneTestCase.RANDOM_MULTIPLIER;
+
+final Path dir = LuceneTestCase.createTempDir(impl.getSimpleName());
+
+// create the LockVerifyServer in a separate thread
+final ExecutorService pool = Executors.newSingleThreadExecutor(new
NamedThreadFactory("lockfactory-tester-"));
+try {
+ pool.submit(() -> {
+LockVerifyServer.main(host, Integer.toString(clients));
+return (Void) null;
+ });
+
+ // wait for it to boot up
+ int port;
+ while ((port = LockVerifyServer.PORT.get()) == -1) {
+Thread.sleep(100L);
+ }
+
+ // spawn clients as separate Java processes
+ final List processes = new ArrayList<>();
+ for (int i = 0; i < clients; i++) {
+processes.add(applyRedirection(new ProcessBuilder(
+Paths.get(System.getProperty("java.home"), "bin",
"java").toString(),
Review comment:
i tested 32M, coming in next push
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on a change in pull request #1842: LUCENE-9512: Move LockFactory stress test to be a unit/integration test
uschindler commented on a change in pull request #1842:
URL: https://github.com/apache/lucene-solr/pull/1842#discussion_r485169213
##
File path: gradle/testing/randomization/policies/tests.policy
##
@@ -64,6 +64,9 @@ grant {
permission java.lang.RuntimePermission "getClassLoader";
permission java.lang.RuntimePermission "setContextClassLoader";
+ // TestLockFactoriesMultiJVM opens a random port on 127.0.0.1 (port 0 =
ephemeral port range):
Review comment:
This does not work, as there needs to be a communication, so the test
known when to start the clients. This was not different in Ant (the main
process was running fork=false, the clients fork=true)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on a change in pull request #1842: LUCENE-9512: Move LockFactory stress test to be a unit/integration test
uschindler commented on a change in pull request #1842:
URL: https://github.com/apache/lucene-solr/pull/1842#discussion_r485170130
##
File path:
lucene/core/src/test/org/apache/lucene/store/TestLockFactoriesMultiJVM.java
##
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.store;
+
+import java.lang.ProcessBuilder.Redirect;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.NamedThreadFactory;
+import org.apache.lucene.util.SuppressForbidden;
+
+public class TestLockFactoriesMultiJVM extends LuceneTestCase {
Review comment:
It will be sligtly faster soon, but can be tuned. But marking it as
`@Slow` is a bad idea because this is a very important test! Especially to
collect information from users, if it really works everywhere.
To really do extensive testing do `-Dtests.nightly=true`, then it runs about
2 minutes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on a change in pull request #1842: LUCENE-9512: Move LockFactory stress test to be a unit/integration test
uschindler commented on a change in pull request #1842:
URL: https://github.com/apache/lucene-solr/pull/1842#discussion_r485170867
##
File path:
lucene/core/src/test/org/apache/lucene/store/TestLockFactoriesMultiJVM.java
##
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.store;
+
+import java.lang.ProcessBuilder.Redirect;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.NamedThreadFactory;
+import org.apache.lucene.util.SuppressForbidden;
+
+public class TestLockFactoriesMultiJVM extends LuceneTestCase {
+
+ @SuppressForbidden(reason = "ProcessBuilder only allows to redirect to
java.io.File")
+ private static final ProcessBuilder applyRedirection(ProcessBuilder pb, int
client, Path dir) {
+if (LuceneTestCase.VERBOSE) {
+ return pb.inheritIO();
+} else {
+ return pb
+.redirectError(dir.resolve("err-" + client + ".txt").toFile())
+.redirectOutput(dir.resolve("out-" + client + ".txt").toFile())
+.redirectInput(Redirect.INHERIT);
+}
+ }
+
+ private void runImpl(Class impl) throws Exception {
+// make sure we are in clean state:
+LockVerifyServer.PORT.set(-1);
+
+final int clients = 2;
+final String host = "127.0.0.1";
+final int delay = 1;
+final int rounds = (LuceneTestCase.TEST_NIGHTLY ? 3 : 500) *
LuceneTestCase.RANDOM_MULTIPLIER;
+
+final Path dir = LuceneTestCase.createTempDir(impl.getSimpleName());
+
+// create the LockVerifyServer in a separate thread
+final ExecutorService pool = Executors.newSingleThreadExecutor(new
NamedThreadFactory("lockfactory-tester-"));
+try {
+ pool.submit(() -> {
+LockVerifyServer.main(host, Integer.toString(clients));
+return (Void) null;
+ });
+
+ // wait for it to boot up
+ int port;
+ while ((port = LockVerifyServer.PORT.get()) == -1) {
+Thread.sleep(100L);
+ }
+
+ // spawn clients as separate Java processes
+ final List processes = new ArrayList<>();
+ for (int i = 0; i < clients; i++) {
+processes.add(applyRedirection(new ProcessBuilder(
+Paths.get(System.getProperty("java.home"), "bin",
"java").toString(),
+"-cp",
+System.getProperty("java.class.path"),
+LockStressTest.class.getName(),
+Integer.toString(i),
+host,
+Integer.toString(port),
+impl.getName(),
+dir.toString(),
+Integer.toString(delay),
+Integer.toString(rounds)
+ ), i, dir).start());
+ }
+
+ // wait for all processes to exit
+ int exited = 0;
+ while (exited < clients) {
+for (Process p : processes) {
Review comment:
I copied this code from another test doing similar stuff (crushing JVM).
I will work change it a bit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1842: LUCENE-9512: Move LockFactory stress test to be a unit/integration test
dweiss commented on a change in pull request #1842:
URL: https://github.com/apache/lucene-solr/pull/1842#discussion_r485175148
##
File path:
lucene/core/src/test/org/apache/lucene/store/TestLockFactoriesMultiJVM.java
##
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.store;
+
+import java.lang.ProcessBuilder.Redirect;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.NamedThreadFactory;
+import org.apache.lucene.util.SuppressForbidden;
+
+public class TestLockFactoriesMultiJVM extends LuceneTestCase {
+
+ @SuppressForbidden(reason = "ProcessBuilder only allows to redirect to
java.io.File")
+ private static final ProcessBuilder applyRedirection(ProcessBuilder pb, int
client, Path dir) {
+if (LuceneTestCase.VERBOSE) {
+ return pb.inheritIO();
+} else {
+ return pb
+.redirectError(dir.resolve("err-" + client + ".txt").toFile())
+.redirectOutput(dir.resolve("out-" + client + ".txt").toFile())
+.redirectInput(Redirect.INHERIT);
+}
+ }
+
+ private void runImpl(Class impl) throws Exception {
+// make sure we are in clean state:
+LockVerifyServer.PORT.set(-1);
+
+final int clients = 2;
+final String host = "127.0.0.1";
+final int delay = 1;
+final int rounds = (LuceneTestCase.TEST_NIGHTLY ? 3 : 500) *
LuceneTestCase.RANDOM_MULTIPLIER;
+
+final Path dir = LuceneTestCase.createTempDir(impl.getSimpleName());
+
+// create the LockVerifyServer in a separate thread
+final ExecutorService pool = Executors.newSingleThreadExecutor(new
NamedThreadFactory("lockfactory-tester-"));
+try {
+ pool.submit(() -> {
+LockVerifyServer.main(host, Integer.toString(clients));
+return (Void) null;
+ });
+
+ // wait for it to boot up
+ int port;
+ while ((port = LockVerifyServer.PORT.get()) == -1) {
+Thread.sleep(100L);
+ }
+
+ // spawn clients as separate Java processes
+ final List processes = new ArrayList<>();
+ for (int i = 0; i < clients; i++) {
+processes.add(applyRedirection(new ProcessBuilder(
+Paths.get(System.getProperty("java.home"), "bin",
"java").toString(),
+"-cp",
+System.getProperty("java.class.path"),
+LockStressTest.class.getName(),
+Integer.toString(i),
+host,
+Integer.toString(port),
+impl.getName(),
+dir.toString(),
+Integer.toString(delay),
+Integer.toString(rounds)
+ ), i, dir).start());
+ }
+
+ // wait for all processes to exit
+ int exited = 0;
+ while (exited < clients) {
+for (Process p : processes) {
Review comment:
onExit is from Java 9 (so wasn't available). I vaguely remember
destroyForcibly didn't work as expected if it forked processes of its own (only
killed the forked process, not its entire hierarchy) -- we have wrappers that
take care of that (because we use process forking extensively in integration
tests). From Java 11 on you can enumerate the hierarchy and kill it manually.
https://github.com/carrotsearch/procfork/blob/master/src/main/java/com/carrotsearch/procfork/ForkedProcess.java#L92-L93
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on a change in pull request #1842: LUCENE-9512: Move LockFactory stress test to be a unit/integration test
uschindler commented on a change in pull request #1842:
URL: https://github.com/apache/lucene-solr/pull/1842#discussion_r485169213
##
File path: gradle/testing/randomization/policies/tests.policy
##
@@ -64,6 +64,9 @@ grant {
permission java.lang.RuntimePermission "getClassLoader";
permission java.lang.RuntimePermission "setContextClassLoader";
+ // TestLockFactoriesMultiJVM opens a random port on 127.0.0.1 (port 0 =
ephemeral port range):
Review comment:
This does not work, as there needs to be a communication, so the test
knows when to start the clients. This was not different in Ant (the main
process was running fork=false, the clients fork=true)
I don't want to start any other processes. I think @rmuir is fine with this
addition of SocketPermission
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1842: LUCENE-9512: Move LockFactory stress test to be a unit/integration test
uschindler commented on pull request #1842: URL: https://github.com/apache/lucene-solr/pull/1842#issuecomment-689117698 I pushed some fixes: - Removed the AtomicInteger for the port number. As the server is running in test, we just call the server directly and it uses a Consumer(InetAddr) as callback. - Because of the callback, we can start the child processes without using a thread(pool). - Cleanup of processes improved. Keep in mind: The waitFor of 15s is is now way too much. Once the server ended and we run into the cleanup, all processes dies already, because the server shuts down when all clients dicsonnected. The cleanup is only needed if anything goes wrong on startup (like client can't connect to server due to network error on lo interface). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
