[jira] [Commented] (LUCENE-10307) Add sanity checks for the distribution
[ https://issues.apache.org/jira/browse/LUCENE-10307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464349#comment-17464349 ] Dawid Weiss commented on LUCENE-10307: -- We want the binary distribution to include just the required subset of dependencies to run Luke, not all binary dependencies required by all Lucene modules. This was discussed in another issue, can't remember where exactly. So whatever Luke needs to run - it should be included, for anything extra, people will probably use Maven/ Gradle/ whatever and POMs provide the necessary dependency graph. > Add sanity checks for the distribution > -- > > Key: LUCENE-10307 > URL: https://issues.apache.org/jira/browse/LUCENE-10307 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Priority: Major > > Re-add the tests from the jms branch validating that module names are > constant, all modules are named, that service layer is consistent, etc. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10338) Scan for tests only by convention file name pattern (Test*)
[ https://issues.apache.org/jira/browse/LUCENE-10338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464379#comment-17464379 ] Uwe Schindler commented on LUCENE-10338: Yes. You can merge the test detection from my branch about Panama. I copied those over from ANT. > Scan for tests only by convention file name pattern (Test*) > --- > > Key: LUCENE-10338 > URL: https://issues.apache.org/jira/browse/LUCENE-10338 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Trivial > > Gradle is smart by trying to figure out which class is a JUnit test and which > isn't. This works only to certain limits - as Uwe pointed out, when the > runtime JDK is newer than gradle, the bytecode cannot be parsed and the > detection fails. Tests under the module system are also affected by this > "scanning". > Since we enforce test class names anyway, I think we can scan for tests by > file name rather than using this gradle feature. > Patch coming. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10334) Introduce a BlockReader based on ForUtil and use it for NumericDocValues
[
https://issues.apache.org/jira/browse/LUCENE-10334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464380#comment-17464380
]
Feng Guo commented on LUCENE-10334:
---
Hi all! Since all existing luceneutil tasks look good, I wonder if we need to
add some more tasks or try this approach in Amazon's product search engine
benchmark (like what we did in
https://issues.apache.org/jira/browse/LUCENE-10033) to justify this change? I'm
willing to do any work to futher test this if any. Or if we think existing
luceneUtil tasks are enough to justify this, I've fixed CI issues and the PR is
probably ready for a reivew now :)
n this PR, I only replaced the {{DirectReader}} used in
{{NumericDocValues#longValue}} with {{BlockReader}} but i suspect this could
probably be used in some other places (e.g. {{DirectMonotonicReader}}, stored
fields, even in BKD https://issues.apache.org/jira/browse/LUCENE-10315). I'll
justify those changes in follow ups.
> Introduce a BlockReader based on ForUtil and use it for NumericDocValues
>
>
> Key: LUCENE-10334
> URL: https://issues.apache.org/jira/browse/LUCENE-10334
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Reporter: Feng Guo
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Previous talk is here: https://github.com/apache/lucene/pull/557
> This is trying to add a new BlockReader based on ForUtil to replace the
> DirectReader we are using for NumericDocvalues
> *Benchmark based on wiki10m*
> {code:java}
> TaskQPS baseline StdDevQPS
> my_modified_version StdDevPct diff p-value
>OrNotHighHigh 694.17 (8.2%) 685.83
> (7.0%) -1.2% ( -15% - 15%) 0.618
> Respell 75.15 (2.7%) 74.32
> (2.0%) -1.1% ( -5% -3%) 0.146
> Prefix3 220.11 (5.1%) 217.78
> (5.8%) -1.1% ( -11% - 10%) 0.541
> Wildcard 129.75 (3.7%) 128.63
> (2.5%) -0.9% ( -6% -5%) 0.383
> LowSpanNear 68.54 (2.1%) 68.00
> (2.4%) -0.8% ( -5% -3%) 0.269
> OrNotHighMed 732.90 (6.8%) 727.49
> (5.3%) -0.7% ( -12% - 12%) 0.703
> BrowseRandomLabelTaxoFacets11879.03 (8.6%)11799.33
> (5.5%) -0.7% ( -13% - 14%) 0.769
> HighSloppyPhrase6.87 (2.9%)6.83
> (2.3%) -0.6% ( -5% -4%) 0.496
> OrHighNotMed 827.54 (9.2%) 822.94
> (8.0%) -0.6% ( -16% - 18%) 0.838
> MedSpanNear 18.92 (5.7%) 18.82
> (5.6%) -0.5% ( -11% - 11%) 0.759
> OrHighMedDayTaxoFacets 10.27 (4.0%) 10.21
> (4.3%) -0.5% ( -8% -8%) 0.676
> PKLookup 207.98 (4.0%) 206.85
> (2.7%) -0.5% ( -7% -6%) 0.621
> LowIntervalsOrdered 159.17 (2.3%) 158.32
> (2.2%) -0.5% ( -4% -3%) 0.445
> HighSpanNear6.32 (4.2%)6.28
> (4.1%) -0.5% ( -8% -8%) 0.691
> MedIntervalsOrdered 85.31 (3.2%) 84.88
> (2.9%) -0.5% ( -6% -5%) 0.607
> HighTerm 1170.55 (5.8%) 1164.79
> (3.9%) -0.5% ( -9% -9%) 0.753
> LowSloppyPhrase 14.54 (3.1%) 14.48
> (2.9%) -0.4% ( -6% -5%) 0.651
> HighPhrase 112.81 (4.4%) 112.39
> (4.1%) -0.4% ( -8% -8%) 0.781
> OrNotHighLow 858.02 (5.9%) 854.99
> (4.8%) -0.4% ( -10% - 10%) 0.835
> HighIntervalsOrdered 25.08 (2.8%) 25.00
> (2.6%) -0.3% ( -5% -5%) 0.701
>MedPhrase 27.20 (2.1%) 27.11
> (2.9%) -0.3% ( -5% -4%) 0.689
> MedTermDayTaxoFacets 81.55 (2.3%) 81.35
> (2.9%) -0.3% ( -5% -5%) 0.762
> IntNRQ 63.36 (2.0%) 63.21
> (2.5%) -0.2% ( -4% -4%) 0.740
> Fuzzy2 73.24 (5.5%) 73.10
> (6.2%) -0.2% ( -11% - 12%) 0.916
> AndHighMedDayTaxoFacets 76.08 (3.5%) 75.98
> (3.4%) -0.1% ( -6% -7%) 0.905
> AndHighHigh 62.20 (2.0%) 62.18
> (2.4%) -0.0% ( -4% -4%) 0.954
>BrowseMonthTaxoFacets11993.48 (6.7%)11989.53
> (4.8%)
[jira] [Comment Edited] (LUCENE-10334) Introduce a BlockReader based on ForUtil and use it for NumericDocValues
[
https://issues.apache.org/jira/browse/LUCENE-10334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464380#comment-17464380
]
Feng Guo edited comment on LUCENE-10334 at 12/23/21, 9:14 AM:
--
Hi all! Since all existing luceneutil tasks look good, I wonder if we need to
add some more tasks or try this approach in Amazon's product search engine
benchmark (like what we did in
https://issues.apache.org/jira/browse/LUCENE-10033) to justify this change? I'm
willing to do any work to futher test this if any. Or if we think existing
luceneUtil tasks are enough to justify this, I've fixed CI issues and the PR is
probably ready for a reivew now :)
in this PR, I only replaced the {{DirectReader}} used in
{{NumericDocValues#longValue}} with {{BlockReader}} but i suspect this could
probably be used in some other places (e.g. {{DirectMonotonicReader}}, stored
fields, even in BKD https://issues.apache.org/jira/browse/LUCENE-10315). I'll
justify those changes in follow ups.
was (Author: gf2121):
Hi all! Since all existing luceneutil tasks look good, I wonder if we need to
add some more tasks or try this approach in Amazon's product search engine
benchmark (like what we did in
https://issues.apache.org/jira/browse/LUCENE-10033) to justify this change? I'm
willing to do any work to futher test this if any. Or if we think existing
luceneUtil tasks are enough to justify this, I've fixed CI issues and the PR is
probably ready for a reivew now :)
n this PR, I only replaced the {{DirectReader}} used in
{{NumericDocValues#longValue}} with {{BlockReader}} but i suspect this could
probably be used in some other places (e.g. {{DirectMonotonicReader}}, stored
fields, even in BKD https://issues.apache.org/jira/browse/LUCENE-10315). I'll
justify those changes in follow ups.
> Introduce a BlockReader based on ForUtil and use it for NumericDocValues
>
>
> Key: LUCENE-10334
> URL: https://issues.apache.org/jira/browse/LUCENE-10334
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Reporter: Feng Guo
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Previous talk is here: https://github.com/apache/lucene/pull/557
> This is trying to add a new BlockReader based on ForUtil to replace the
> DirectReader we are using for NumericDocvalues
> *Benchmark based on wiki10m*
> {code:java}
> TaskQPS baseline StdDevQPS
> my_modified_version StdDevPct diff p-value
>OrNotHighHigh 694.17 (8.2%) 685.83
> (7.0%) -1.2% ( -15% - 15%) 0.618
> Respell 75.15 (2.7%) 74.32
> (2.0%) -1.1% ( -5% -3%) 0.146
> Prefix3 220.11 (5.1%) 217.78
> (5.8%) -1.1% ( -11% - 10%) 0.541
> Wildcard 129.75 (3.7%) 128.63
> (2.5%) -0.9% ( -6% -5%) 0.383
> LowSpanNear 68.54 (2.1%) 68.00
> (2.4%) -0.8% ( -5% -3%) 0.269
> OrNotHighMed 732.90 (6.8%) 727.49
> (5.3%) -0.7% ( -12% - 12%) 0.703
> BrowseRandomLabelTaxoFacets11879.03 (8.6%)11799.33
> (5.5%) -0.7% ( -13% - 14%) 0.769
> HighSloppyPhrase6.87 (2.9%)6.83
> (2.3%) -0.6% ( -5% -4%) 0.496
> OrHighNotMed 827.54 (9.2%) 822.94
> (8.0%) -0.6% ( -16% - 18%) 0.838
> MedSpanNear 18.92 (5.7%) 18.82
> (5.6%) -0.5% ( -11% - 11%) 0.759
> OrHighMedDayTaxoFacets 10.27 (4.0%) 10.21
> (4.3%) -0.5% ( -8% -8%) 0.676
> PKLookup 207.98 (4.0%) 206.85
> (2.7%) -0.5% ( -7% -6%) 0.621
> LowIntervalsOrdered 159.17 (2.3%) 158.32
> (2.2%) -0.5% ( -4% -3%) 0.445
> HighSpanNear6.32 (4.2%)6.28
> (4.1%) -0.5% ( -8% -8%) 0.691
> MedIntervalsOrdered 85.31 (3.2%) 84.88
> (2.9%) -0.5% ( -6% -5%) 0.607
> HighTerm 1170.55 (5.8%) 1164.79
> (3.9%) -0.5% ( -9% -9%) 0.753
> LowSloppyPhrase 14.54 (3.1%) 14.48
> (2.9%) -0.4% ( -6% -5%) 0.651
> HighPhrase 112.81 (4.4%) 112.39
> (4.1%) -0.4% ( -8% -8%) 0.781
> OrNotHighLow 858.02 (5.9%) 854.99
> (4.8%) -0.4% ( -10% - 10%) 0.835
> HighIntervalsOr
[jira] [Comment Edited] (LUCENE-10334) Introduce a BlockReader based on ForUtil and use it for NumericDocValues
[
https://issues.apache.org/jira/browse/LUCENE-10334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464380#comment-17464380
]
Feng Guo edited comment on LUCENE-10334 at 12/23/21, 9:15 AM:
--
Hi all! Since all existing luceneutil tasks look good, I wonder if we need to
add some more tasks or try this approach in Amazon's product search engine
benchmark (like what we did in
https://issues.apache.org/jira/browse/LUCENE-10033) to justify this change? I'm
willing to do any work to futher test this if any. Or if we think existing
luceneUtil tasks are enough to justify this, I've fixed CI issues and the PR is
probably ready for a reivew now :)
In this PR, I only replaced the {{DirectReader}} used in
{{NumericDocValues#longValue}} with {{BlockReader}} but i suspect this could
be used in some other places (e.g. {{DirectMonotonicReader}}, stored fields,
even in BKD https://issues.apache.org/jira/browse/LUCENE-10315). I'll justify
those changes in follow ups.
was (Author: gf2121):
Hi all! Since all existing luceneutil tasks look good, I wonder if we need to
add some more tasks or try this approach in Amazon's product search engine
benchmark (like what we did in
https://issues.apache.org/jira/browse/LUCENE-10033) to justify this change? I'm
willing to do any work to futher test this if any. Or if we think existing
luceneUtil tasks are enough to justify this, I've fixed CI issues and the PR is
probably ready for a reivew now :)
in this PR, I only replaced the {{DirectReader}} used in
{{NumericDocValues#longValue}} with {{BlockReader}} but i suspect this could
probably be used in some other places (e.g. {{DirectMonotonicReader}}, stored
fields, even in BKD https://issues.apache.org/jira/browse/LUCENE-10315). I'll
justify those changes in follow ups.
> Introduce a BlockReader based on ForUtil and use it for NumericDocValues
>
>
> Key: LUCENE-10334
> URL: https://issues.apache.org/jira/browse/LUCENE-10334
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Reporter: Feng Guo
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Previous talk is here: https://github.com/apache/lucene/pull/557
> This is trying to add a new BlockReader based on ForUtil to replace the
> DirectReader we are using for NumericDocvalues
> *Benchmark based on wiki10m*
> {code:java}
> TaskQPS baseline StdDevQPS
> my_modified_version StdDevPct diff p-value
>OrNotHighHigh 694.17 (8.2%) 685.83
> (7.0%) -1.2% ( -15% - 15%) 0.618
> Respell 75.15 (2.7%) 74.32
> (2.0%) -1.1% ( -5% -3%) 0.146
> Prefix3 220.11 (5.1%) 217.78
> (5.8%) -1.1% ( -11% - 10%) 0.541
> Wildcard 129.75 (3.7%) 128.63
> (2.5%) -0.9% ( -6% -5%) 0.383
> LowSpanNear 68.54 (2.1%) 68.00
> (2.4%) -0.8% ( -5% -3%) 0.269
> OrNotHighMed 732.90 (6.8%) 727.49
> (5.3%) -0.7% ( -12% - 12%) 0.703
> BrowseRandomLabelTaxoFacets11879.03 (8.6%)11799.33
> (5.5%) -0.7% ( -13% - 14%) 0.769
> HighSloppyPhrase6.87 (2.9%)6.83
> (2.3%) -0.6% ( -5% -4%) 0.496
> OrHighNotMed 827.54 (9.2%) 822.94
> (8.0%) -0.6% ( -16% - 18%) 0.838
> MedSpanNear 18.92 (5.7%) 18.82
> (5.6%) -0.5% ( -11% - 11%) 0.759
> OrHighMedDayTaxoFacets 10.27 (4.0%) 10.21
> (4.3%) -0.5% ( -8% -8%) 0.676
> PKLookup 207.98 (4.0%) 206.85
> (2.7%) -0.5% ( -7% -6%) 0.621
> LowIntervalsOrdered 159.17 (2.3%) 158.32
> (2.2%) -0.5% ( -4% -3%) 0.445
> HighSpanNear6.32 (4.2%)6.28
> (4.1%) -0.5% ( -8% -8%) 0.691
> MedIntervalsOrdered 85.31 (3.2%) 84.88
> (2.9%) -0.5% ( -6% -5%) 0.607
> HighTerm 1170.55 (5.8%) 1164.79
> (3.9%) -0.5% ( -9% -9%) 0.753
> LowSloppyPhrase 14.54 (3.1%) 14.48
> (2.9%) -0.4% ( -6% -5%) 0.651
> HighPhrase 112.81 (4.4%) 112.39
> (4.1%) -0.4% ( -8% -8%) 0.781
> OrNotHighLow 858.02 (5.9%) 854.99
> (4.8%) -0.4% ( -10% - 10%) 0.835
> HighIntervalsOrdered
[GitHub] [lucene] mocobeta opened a new pull request #566: LUCENE-10301: Place test-framework into separated modules folder.
mocobeta opened a new pull request #566: URL: https://github.com/apache/lucene/pull/566 test-framework module has to be placed in a separate folder in the binary distribution in order to run the Luke application. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on a change in pull request #565: LUCENE-10338: Scan for tests only by convention file name pattern
uschindler commented on a change in pull request #565:
URL: https://github.com/apache/lucene/pull/565#discussion_r774436595
##
File path: gradle/testing/defaults-tests.gradle
##
@@ -161,6 +161,14 @@ allprojects {
showStandardStreams false
}
+ // Disable automatic test class detection, rely on class names only.
This is needed for testing
+ // against JDKs where the bytecode is unparseable by Gradle, for example.
+ // We require all tests to start with Test*, this simplifies include
patterns greatly.
+ scanForTestClasses = false
+ include '**/Test*'
Review comment:
This no longer adds files ending on Test. At Ant times we had this, did
we rename all tests?
##
File path: gradle/testing/defaults-tests.gradle
##
@@ -161,6 +161,14 @@ allprojects {
showStandardStreams false
}
+ // Disable automatic test class detection, rely on class names only.
This is needed for testing
+ // against JDKs where the bytecode is unparseable by Gradle, for example.
+ // We require all tests to start with Test*, this simplifies include
patterns greatly.
+ scanForTestClasses = false
+ include '**/Test*'
Review comment:
I would add ".class" to not add resources files.
##
File path:
lucene/queryparser/src/test/org/apache/lucene/queryparser/surround/query/BooleanQueryTestFacade.java
##
@@ -28,7 +28,7 @@
import org.apache.lucene.search.SimpleCollector;
import org.junit.Assert;
-public class TestBooleanQuery {
+public class BooleanQueryTestFacade {
Review comment:
I don't think this rename is needed because it is on the exclusion list
above (the dollar one).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #566: LUCENE-10301: Place test-framework into separated modules folder.
uschindler commented on pull request #566: URL: https://github.com/apache/lucene/pull/566#issuecomment-1000168184 +1 I was already wondering why it was moved to the modules folder because although it is a real module now, it should not leak into production code. You're right: spi classes are picked up automatically. That's how they should behave (easy plugin mechanism). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] gf2121 commented on a change in pull request #562: LUCENE-10334: Introduce a BlockReader based on ForUtil and use it for NumericDocValues
gf2121 commented on a change in pull request #562:
URL: https://github.com/apache/lucene/pull/562#discussion_r774440982
##
File path: lucene/core/src/java/org/apache/lucene/util/packed/ForUtil.java
##
@@ -0,0 +1,3311 @@
+// This file has been automatically generated, DO NOT EDIT
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.util.packed;
+
+import java.io.IOException;
+import org.apache.lucene.store.DataInput;
+import org.apache.lucene.store.DataOutput;
+import org.apache.lucene.util.MathUtil;
+
+// Inspired from https://fulmicoton.com/posts/bitpacking/
+// Encodes multiple integers in a long to get SIMD-like speedups.
+// If bitsPerValue <= 8 then we pack 8 ints per long
+// else if bitsPerValue <= 16 we pack 4 ints per long
+// else we pack 2 ints per long
+final class ForUtil {
+
+ static final int BLOCK_SIZE = 128;
+ static final int BLOCK_SIZE_LOG2 = MathUtil.log(BLOCK_SIZE, 2);
+ private static final int BLOCK_SIZE_DIV_2 = BLOCK_SIZE >> 1;
+ private static final int BLOCK_SIZE_DIV_4 = BLOCK_SIZE >> 2;
+ static final int BLOCK_SIZE_DIV_8 = BLOCK_SIZE >> 3;
+ private static final int BLOCK_SIZE_DIV_64 = BLOCK_SIZE >> 6;
+ private static final int BLOCK_SIZE_DIV_8_MUL_1 = BLOCK_SIZE_DIV_8;
+ private static final int BLOCK_SIZE_DIV_8_MUL_2 = BLOCK_SIZE_DIV_8 * 2;
+ private static final int BLOCK_SIZE_DIV_8_MUL_3 = BLOCK_SIZE_DIV_8 * 3;
+ private static final int BLOCK_SIZE_DIV_8_MUL_4 = BLOCK_SIZE_DIV_8 * 4;
+ private static final int BLOCK_SIZE_DIV_8_MUL_5 = BLOCK_SIZE_DIV_8 * 5;
+ private static final int BLOCK_SIZE_DIV_8_MUL_6 = BLOCK_SIZE_DIV_8 * 6;
+ private static final int BLOCK_SIZE_DIV_8_MUL_7 = BLOCK_SIZE_DIV_8 * 7;
+ private static final int BLOCK_SIZE_LOG2_MIN_3 = BLOCK_SIZE_LOG2 - 3;
+
+ private static long expandMask32(long mask32) {
+return mask32 | (mask32 << 32);
+ }
+
+ private static long expandMask16(long mask16) {
+return expandMask32(mask16 | (mask16 << 16));
+ }
+
+ private static long expandMask8(long mask8) {
+return expandMask16(mask8 | (mask8 << 8));
+ }
+
+ private static long mask32(int bitsPerValue) {
+return expandMask32((1L << bitsPerValue) - 1);
+ }
+
+ private static long mask64(int bitsPerValue) {
+return (1L << bitsPerValue) - 1;
+ }
+
+ private static long mask16(int bitsPerValue) {
+return expandMask16((1L << bitsPerValue) - 1);
+ }
+
+ private static long mask8(int bitsPerValue) {
+return expandMask8((1L << bitsPerValue) - 1);
+ }
+
+ private static void expand8(long[] arr) {
+for (int i = 0; i < BLOCK_SIZE_DIV_8; ++i) {
+ long l = arr[i];
+ arr[i] = (l >>> 56) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_1 + i] = (l >>> 48) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_2 + i] = (l >>> 40) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_3 + i] = (l >>> 32) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_4 + i] = (l >>> 24) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_5 + i] = (l >>> 16) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_6 + i] = (l >>> 8) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_7 + i] = l & 0xFFL;
+}
+ }
+
+ private static void collapse8(long[] arr) {
+for (int i = 0; i < BLOCK_SIZE_DIV_8; ++i) {
+ arr[i] =
+ (arr[i] << 56)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_1 + i] << 48)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_2 + i] << 40)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_3 + i] << 32)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_4 + i] << 24)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_5 + i] << 16)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_6 + i] << 8)
+ | arr[BLOCK_SIZE_DIV_8_MUL_7 + i];
+}
+ }
+
+ private static void expand16(long[] arr) {
+for (int i = 0; i < BLOCK_SIZE_DIV_4; ++i) {
+ long l = arr[i];
+ arr[i] = (l >>> 48) & 0xL;
+ arr[BLOCK_SIZE_DIV_8_MUL_2 + i] = (l >>> 32) & 0xL;
+ arr[BLOCK_SIZE_DIV_8_MUL_4 + i] = (l >>> 16) & 0xL;
+ arr[BLOCK_SIZE_DIV_8_MUL_6 + i] = l & 0xL;
+}
+ }
+
+ private static void collapse16(long[] arr) {
+for (int i = 0; i < BLOCK_SIZE_DIV_4; ++i) {
+ arr[i] =
+ (arr[i] << 48)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_2 + i] << 32)
+ | (arr[BLOCK
[jira] [Commented] (LUCENE-10301) The test-framework as a module (or a separate test-framework-module)
[ https://issues.apache.org/jira/browse/LUCENE-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464394#comment-17464394 ] Tomoko Uchida commented on LUCENE-10301: The test-framework module has several service providers (codecs) for testing. When it is placed in the module path, lucene-core tries to consume the services when starting a Java application on module mode; this prevents from starting the Luke app on the binary distribution since it cannot be resolved on that (we won't re-distribute dependencies of test-framework, junit, etc.) I opened a PR to place test-framework module into a separate modules folder. It is actually just a partial revert on the previous commit. https://github.com/apache/lucene/pull/566 > The test-framework as a module (or a separate test-framework-module) > > > Key: LUCENE-10301 > URL: https://issues.apache.org/jira/browse/LUCENE-10301 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Major > Fix For: 9.1 > > Time Spent: 7h 40m > Remaining Estimate: 0h > > The test framework has split packages. It's a follow-up to introducing > modules but eventually the modular test subprojects will need something like > the test framework too. > I'm not sure whether we should start a new subproject for this or try to > refactor the test framework, but it's a follow-up once the modules themselves > are working and testable, I think. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta commented on pull request #566: LUCENE-10301: Place test-framework into separated modules folder.
mocobeta commented on pull request #566: URL: https://github.com/apache/lucene/pull/566#issuecomment-1000170764 @uschindler Thanks for your quick reply. @dweiss Could you take a look at this, please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10335) IOUtils.getDecodingReader(Class, String) is broken with modules
[ https://issues.apache.org/jira/browse/LUCENE-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464452#comment-17464452 ] Uwe Schindler commented on LUCENE-10335: Hi Dawid, thanks for the patch. Actually the Supplier isn't needed (if we call the method before we get the reader), but then we have 2 methods with same signature. I was thinking the whole night about it and I will do the following: - Apply your patch for deprecation, but I will make the deprecated method throw Exception when used in module mode. We can't remove it because third party analyzer packages may use it. - Rewrite all calling code to use try-with-resources (as it allows NULL) - For the NULL Exception I will add a wrapper method like Objects.requireNonNull() but that throws an IOException. I will document it as "workaround" for Class#getResourceAsStream() I will open PR soon. > IOUtils.getDecodingReader(Class, String) is broken with modules > -- > > Key: LUCENE-10335 > URL: https://issues.apache.org/jira/browse/LUCENE-10335 > Project: Lucene - Core > Issue Type: Bug >Reporter: Dawid Weiss >Priority: Major > Attachments: LUCENE-10335-1.patch, LUCENE-10335.patch > > > This method calls clazz.getResourceAsStream() but in a modular application > the method won't see any of the resources in clazz's module, causing an NPE. > We should deprecate or even remove this method entirely, leaving only > getDecodingReader(InputStream) and opening the resource on the caller's side. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] sonatype-lift[bot] commented on a change in pull request #562: LUCENE-10334: Introduce a BlockReader based on ForUtil and use it for NumericDocValues
sonatype-lift[bot] commented on a change in pull request #562:
URL: https://github.com/apache/lucene/pull/562#discussion_r774484808
##
File path: lucene/core/src/java/org/apache/lucene/util/packed/ForUtil.java
##
@@ -0,0 +1,3329 @@
+// This file has been automatically generated, DO NOT EDIT
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.util.packed;
+
+import java.io.IOException;
+import org.apache.lucene.store.DataInput;
+import org.apache.lucene.store.DataOutput;
+import org.apache.lucene.util.MathUtil;
+
+// Inspired from https://fulmicoton.com/posts/bitpacking/
+// Encodes multiple integers in a long to get SIMD-like speedups.
+// If bitsPerValue <= 8 then we pack 8 ints per long
+// else if bitsPerValue <= 16 we pack 4 ints per long
+// else we pack 2 ints per long
+final class ForUtil {
+
+ static final int BLOCK_SIZE = 128;
+ static final int BLOCK_SIZE_LOG2 = MathUtil.log(BLOCK_SIZE, 2);
+ private static final int BLOCK_SIZE_DIV_2 = BLOCK_SIZE >> 1;
+ private static final int BLOCK_SIZE_DIV_4 = BLOCK_SIZE >> 2;
+ static final int BLOCK_SIZE_DIV_8 = BLOCK_SIZE >> 3;
+ private static final int BLOCK_SIZE_DIV_64 = BLOCK_SIZE >> 6;
+ private static final int BLOCK_SIZE_DIV_8_MUL_1 = BLOCK_SIZE_DIV_8;
+ private static final int BLOCK_SIZE_DIV_8_MUL_2 = BLOCK_SIZE_DIV_8 * 2;
+ private static final int BLOCK_SIZE_DIV_8_MUL_3 = BLOCK_SIZE_DIV_8 * 3;
+ private static final int BLOCK_SIZE_DIV_8_MUL_4 = BLOCK_SIZE_DIV_8 * 4;
+ private static final int BLOCK_SIZE_DIV_8_MUL_5 = BLOCK_SIZE_DIV_8 * 5;
+ private static final int BLOCK_SIZE_DIV_8_MUL_6 = BLOCK_SIZE_DIV_8 * 6;
+ private static final int BLOCK_SIZE_DIV_8_MUL_7 = BLOCK_SIZE_DIV_8 * 7;
+ private static final int BLOCK_SIZE_LOG2_MIN_3 = BLOCK_SIZE_LOG2 - 3;
+
+ private static long expandMask32(long mask32) {
+return mask32 | (mask32 << 32);
+ }
+
+ private static long expandMask16(long mask16) {
+return expandMask32(mask16 | (mask16 << 16));
+ }
+
+ private static long expandMask8(long mask8) {
+return expandMask16(mask8 | (mask8 << 8));
+ }
+
+ private static long mask32(int bitsPerValue) {
+return expandMask32((1L << bitsPerValue) - 1);
+ }
+
+ private static long mask64(int bitsPerValue) {
+return (1L << bitsPerValue) - 1;
+ }
+
+ private static long mask16(int bitsPerValue) {
+return expandMask16((1L << bitsPerValue) - 1);
+ }
+
+ private static long mask8(int bitsPerValue) {
+return expandMask8((1L << bitsPerValue) - 1);
+ }
+
+ private static void expand8(long[] arr) {
+for (int i = 0; i < BLOCK_SIZE_DIV_8; ++i) {
+ long l = arr[i];
+ arr[i] = (l >>> 56) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_1 + i] = (l >>> 48) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_2 + i] = (l >>> 40) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_3 + i] = (l >>> 32) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_4 + i] = (l >>> 24) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_5 + i] = (l >>> 16) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_6 + i] = (l >>> 8) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_7 + i] = l & 0xFFL;
+}
+ }
+
+ private static void expand8To32(long[] arr) {
+for (int i = 0; i < BLOCK_SIZE_DIV_8; ++i) {
+ long l = arr[i];
+ arr[i] = (l >>> 24) & 0x00FF00FFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_1 + i] = (l >>> 16) & 0x00FF00FFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_2 + i] = (l >>> 8) & 0x00FF00FFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_3 + i] = l & 0x00FF00FFL;
+}
+ }
+
+ private static void collapse8(long[] arr) {
+for (int i = 0; i < BLOCK_SIZE_DIV_8; ++i) {
+ arr[i] =
+ (arr[i] << 56)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_1 + i] << 48)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_2 + i] << 40)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_3 + i] << 32)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_4 + i] << 24)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_5 + i] << 16)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_6 + i] << 8)
+ | arr[BLOCK_SIZE_DIV_8_MUL_7 + i];
+}
+ }
+
+ private static void expand16(long[] arr) {
+for (int i = 0; i < BLOCK_SIZE_DIV_4; ++i) {
+ long l = arr[i];
+ arr[i] = (l >>> 48) & 0xL;
+
[GitHub] [lucene] uschindler opened a new pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler opened a new pull request #567: URL: https://github.com/apache/lucene/pull/567 See: https://issues.apache.org/jira/browse/LUCENE-10335 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000244112 I have not yet checked Luke's image loading, as I touched this while reviewing all calls to getResource()/getResourceAsStream(). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10335) IOUtils.getDecodingReader(Class, String) is broken with modules
[ https://issues.apache.org/jira/browse/LUCENE-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464494#comment-17464494 ] Dawid Weiss commented on LUCENE-10335: -- +1. Thanks Uwe. > IOUtils.getDecodingReader(Class, String) is broken with modules > -- > > Key: LUCENE-10335 > URL: https://issues.apache.org/jira/browse/LUCENE-10335 > Project: Lucene - Core > Issue Type: Bug >Reporter: Dawid Weiss >Priority: Major > Attachments: LUCENE-10335-1.patch, LUCENE-10335.patch > > Time Spent: 20m > Remaining Estimate: 0h > > This method calls clazz.getResourceAsStream() but in a modular application > the method won't see any of the resources in clazz's module, causing an NPE. > We should deprecate or even remove this method entirely, leaving only > getDecodingReader(InputStream) and opening the resource on the caller's side. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on a change in pull request #566: LUCENE-10301: Place test-framework into separated modules folder.
dweiss commented on a change in pull request #566:
URL: https://github.com/apache/lucene/pull/566#discussion_r774515076
##
File path:
lucene/distribution.tests/src/test/org/apache/lucene/distribution/TestModularLayer.java
##
@@ -101,7 +109,7 @@ public static void checkModulePathProvided() {
+ thirdPartyModulesPath.toAbsolutePath());
}
-coreModulesFinder = ModuleFinder.of(modulesPath);
+coreModulesFinder = ModuleFinder.of(modulesPath, testModulesPath);
Review comment:
Did "coreModules" include the test framework, previously?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on a change in pull request #565: LUCENE-10338: Scan for tests only by convention file name pattern
dweiss commented on a change in pull request #565:
URL: https://github.com/apache/lucene/pull/565#discussion_r774515953
##
File path: gradle/testing/defaults-tests.gradle
##
@@ -161,6 +161,14 @@ allprojects {
showStandardStreams false
}
+ // Disable automatic test class detection, rely on class names only.
This is needed for testing
+ // against JDKs where the bytecode is unparseable by Gradle, for example.
+ // We require all tests to start with Test*, this simplifies include
patterns greatly.
+ scanForTestClasses = false
+ include '**/Test*'
Review comment:
We have a test convention to enforce this? Anything subclassing
LuceneTestCase should start with Test*.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on a change in pull request #565: LUCENE-10338: Scan for tests only by convention file name pattern
dweiss commented on a change in pull request #565:
URL: https://github.com/apache/lucene/pull/565#discussion_r774516325
##
File path:
lucene/queryparser/src/test/org/apache/lucene/queryparser/surround/query/BooleanQueryTestFacade.java
##
@@ -28,7 +28,7 @@
import org.apache.lucene.search.SimpleCollector;
import org.junit.Assert;
-public class TestBooleanQuery {
+public class BooleanQueryTestFacade {
Review comment:
It is needed - this class is public and starts with Test, yet it's not a
test. I know because if fails on my branch otherwise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on a change in pull request #565: LUCENE-10338: Scan for tests only by convention file name pattern
dweiss commented on a change in pull request #565:
URL: https://github.com/apache/lucene/pull/565#discussion_r774517300
##
File path: gradle/testing/defaults-tests.gradle
##
@@ -161,6 +161,14 @@ allprojects {
showStandardStreams false
}
+ // Disable automatic test class detection, rely on class names only.
This is needed for testing
+ // against JDKs where the bytecode is unparseable by Gradle, for example.
+ // We require all tests to start with Test*, this simplifies include
patterns greatly.
+ scanForTestClasses = false
+ include '**/Test*'
Review comment:
I added a commit that restricts this to *.class. This isn't strictly
necessary because classes and resources end up on separate paths (which is
another problem elsewhere, eh) but fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on a change in pull request #565: LUCENE-10338: Scan for tests only by convention file name pattern
dweiss commented on a change in pull request #565:
URL: https://github.com/apache/lucene/pull/565#discussion_r774519046
##
File path: gradle/testing/defaults-tests.gradle
##
@@ -161,6 +161,14 @@ allprojects {
showStandardStreams false
}
+ // Disable automatic test class detection, rely on class names only.
This is needed for testing
+ // against JDKs where the bytecode is unparseable by Gradle, for example.
+ // We require all tests to start with Test*, this simplifies include
patterns greatly.
+ scanForTestClasses = false
+ include '**/Test*'
Review comment:
Ok, I see it now:
```
new VerifyTestClassNamingConvention(
"org.apache.lucene",
Pattern.compile("(.+\\.)(Test)([^.]+)")))
```
so it in fact allows non-leading Test... Darn. This does complicate the
exclusion patterns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000252195 I fixed Luke. I will now check for other places that do getClassLoader().getResource... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on a change in pull request #565: LUCENE-10338: Scan for tests only by convention file name pattern
dweiss commented on a change in pull request #565:
URL: https://github.com/apache/lucene/pull/565#discussion_r774520790
##
File path: gradle/testing/defaults-tests.gradle
##
@@ -161,6 +161,14 @@ allprojects {
showStandardStreams false
}
+ // Disable automatic test class detection, rely on class names only.
This is needed for testing
+ // against JDKs where the bytecode is unparseable by Gradle, for example.
+ // We require all tests to start with Test*, this simplifies include
patterns greatly.
+ scanForTestClasses = false
+ include '**/Test*'
Review comment:
The above pattern should really be changed to be an alternative of *Test
| Test* - like you have on the panama branch, I'll do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on a change in pull request #565: LUCENE-10338: Scan for tests only by convention file name pattern
dweiss commented on a change in pull request #565:
URL: https://github.com/apache/lucene/pull/565#discussion_r774520790
##
File path: gradle/testing/defaults-tests.gradle
##
@@ -161,6 +161,14 @@ allprojects {
showStandardStreams false
}
+ // Disable automatic test class detection, rely on class names only.
This is needed for testing
+ // against JDKs where the bytecode is unparseable by Gradle, for example.
+ // We require all tests to start with Test*, this simplifies include
patterns greatly.
+ scanForTestClasses = false
+ include '**/Test*'
Review comment:
The above pattern should really be changed to be an alternative of
```
.+Test | Test*.
```
like you have on the panama branch, I'll do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on a change in pull request #565: LUCENE-10338: Scan for tests only by convention file name pattern
dweiss commented on a change in pull request #565:
URL: https://github.com/apache/lucene/pull/565#discussion_r774522117
##
File path: gradle/testing/defaults-tests.gradle
##
@@ -161,6 +161,14 @@ allprojects {
showStandardStreams false
}
+ // Disable automatic test class detection, rely on class names only.
This is needed for testing
+ // against JDKs where the bytecode is unparseable by Gradle, for example.
+ // We require all tests to start with Test*, this simplifies include
patterns greatly.
+ scanForTestClasses = false
+ include '**/Test*'
Review comment:
Em, sorry - I read that regexp wrong - it is in fact enforcing the
leading Test (dot must be prior to Test). So Test* is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000255235 We have some more in Benchmark module and tests of Spatial module. We may need to fix those and move resources to exported packages (not a root folder called "data/"). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10335) IOUtils.getDecodingReader(Class, String) is broken with modules
[ https://issues.apache.org/jira/browse/LUCENE-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464500#comment-17464500 ] Uwe Schindler commented on LUCENE-10335: Hi [~dweiss], I did this (and more) in https://github.com/apache/lucene/pull/567 Thanks for opening this issue. > IOUtils.getDecodingReader(Class, String) is broken with modules > -- > > Key: LUCENE-10335 > URL: https://issues.apache.org/jira/browse/LUCENE-10335 > Project: Lucene - Core > Issue Type: Bug >Reporter: Dawid Weiss >Priority: Major > Attachments: LUCENE-10335-1.patch, LUCENE-10335.patch > > Time Spent: 40m > Remaining Estimate: 0h > > This method calls clazz.getResourceAsStream() but in a modular application > the method won't see any of the resources in clazz's module, causing an NPE. > We should deprecate or even remove this method entirely, leaving only > getDecodingReader(InputStream) and opening the resource on the caller's side. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #565: LUCENE-10338: Scan for tests only by convention file name pattern
dweiss commented on pull request #565: URL: https://github.com/apache/lucene/pull/565#issuecomment-1000257251 I've removed TestBase* exclusion - anything starting with Test* must in fact be a test. Those two classes I had to rename - I'm not sure how they work for you on panama branch, they should be failing... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta commented on a change in pull request #566: LUCENE-10301: Place test-framework into separated modules folder.
mocobeta commented on a change in pull request #566:
URL: https://github.com/apache/lucene/pull/566#discussion_r774524778
##
File path:
lucene/distribution.tests/src/test/org/apache/lucene/distribution/TestModularLayer.java
##
@@ -101,7 +109,7 @@ public static void checkModulePathProvided() {
+ thirdPartyModulesPath.toAbsolutePath());
}
-coreModulesFinder = ModuleFinder.of(modulesPath);
+coreModulesFinder = ModuleFinder.of(modulesPath, testModulesPath);
Review comment:
No - I think test-framework firstly appeared in the distribution test at
https://github.com/apache/lucene/commit/a94fbb79ac6e60c1ebfa76b24cab23fabcb60bbf.
There is no previous code to be reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] gf2121 commented on a change in pull request #562: LUCENE-10334: Introduce a BlockReader based on ForUtil and use it for NumericDocValues
gf2121 commented on a change in pull request #562:
URL: https://github.com/apache/lucene/pull/562#discussion_r774526889
##
File path: lucene/core/src/java/org/apache/lucene/util/packed/ForUtil.java
##
@@ -0,0 +1,3311 @@
+// This file has been automatically generated, DO NOT EDIT
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.util.packed;
+
+import java.io.IOException;
+import org.apache.lucene.store.DataInput;
+import org.apache.lucene.store.DataOutput;
+import org.apache.lucene.util.MathUtil;
+
+// Inspired from https://fulmicoton.com/posts/bitpacking/
+// Encodes multiple integers in a long to get SIMD-like speedups.
+// If bitsPerValue <= 8 then we pack 8 ints per long
+// else if bitsPerValue <= 16 we pack 4 ints per long
+// else we pack 2 ints per long
+final class ForUtil {
+
+ static final int BLOCK_SIZE = 128;
+ static final int BLOCK_SIZE_LOG2 = MathUtil.log(BLOCK_SIZE, 2);
+ private static final int BLOCK_SIZE_DIV_2 = BLOCK_SIZE >> 1;
+ private static final int BLOCK_SIZE_DIV_4 = BLOCK_SIZE >> 2;
+ static final int BLOCK_SIZE_DIV_8 = BLOCK_SIZE >> 3;
+ private static final int BLOCK_SIZE_DIV_64 = BLOCK_SIZE >> 6;
+ private static final int BLOCK_SIZE_DIV_8_MUL_1 = BLOCK_SIZE_DIV_8;
+ private static final int BLOCK_SIZE_DIV_8_MUL_2 = BLOCK_SIZE_DIV_8 * 2;
+ private static final int BLOCK_SIZE_DIV_8_MUL_3 = BLOCK_SIZE_DIV_8 * 3;
+ private static final int BLOCK_SIZE_DIV_8_MUL_4 = BLOCK_SIZE_DIV_8 * 4;
+ private static final int BLOCK_SIZE_DIV_8_MUL_5 = BLOCK_SIZE_DIV_8 * 5;
+ private static final int BLOCK_SIZE_DIV_8_MUL_6 = BLOCK_SIZE_DIV_8 * 6;
+ private static final int BLOCK_SIZE_DIV_8_MUL_7 = BLOCK_SIZE_DIV_8 * 7;
+ private static final int BLOCK_SIZE_LOG2_MIN_3 = BLOCK_SIZE_LOG2 - 3;
+
+ private static long expandMask32(long mask32) {
+return mask32 | (mask32 << 32);
+ }
+
+ private static long expandMask16(long mask16) {
+return expandMask32(mask16 | (mask16 << 16));
+ }
+
+ private static long expandMask8(long mask8) {
+return expandMask16(mask8 | (mask8 << 8));
+ }
+
+ private static long mask32(int bitsPerValue) {
+return expandMask32((1L << bitsPerValue) - 1);
+ }
+
+ private static long mask64(int bitsPerValue) {
+return (1L << bitsPerValue) - 1;
+ }
+
+ private static long mask16(int bitsPerValue) {
+return expandMask16((1L << bitsPerValue) - 1);
+ }
+
+ private static long mask8(int bitsPerValue) {
+return expandMask8((1L << bitsPerValue) - 1);
+ }
+
+ private static void expand8(long[] arr) {
+for (int i = 0; i < BLOCK_SIZE_DIV_8; ++i) {
+ long l = arr[i];
+ arr[i] = (l >>> 56) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_1 + i] = (l >>> 48) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_2 + i] = (l >>> 40) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_3 + i] = (l >>> 32) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_4 + i] = (l >>> 24) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_5 + i] = (l >>> 16) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_6 + i] = (l >>> 8) & 0xFFL;
+ arr[BLOCK_SIZE_DIV_8_MUL_7 + i] = l & 0xFFL;
+}
+ }
+
+ private static void collapse8(long[] arr) {
+for (int i = 0; i < BLOCK_SIZE_DIV_8; ++i) {
+ arr[i] =
+ (arr[i] << 56)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_1 + i] << 48)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_2 + i] << 40)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_3 + i] << 32)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_4 + i] << 24)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_5 + i] << 16)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_6 + i] << 8)
+ | arr[BLOCK_SIZE_DIV_8_MUL_7 + i];
+}
+ }
+
+ private static void expand16(long[] arr) {
+for (int i = 0; i < BLOCK_SIZE_DIV_4; ++i) {
+ long l = arr[i];
+ arr[i] = (l >>> 48) & 0xL;
+ arr[BLOCK_SIZE_DIV_8_MUL_2 + i] = (l >>> 32) & 0xL;
+ arr[BLOCK_SIZE_DIV_8_MUL_4 + i] = (l >>> 16) & 0xL;
+ arr[BLOCK_SIZE_DIV_8_MUL_6 + i] = l & 0xL;
+}
+ }
+
+ private static void collapse16(long[] arr) {
+for (int i = 0; i < BLOCK_SIZE_DIV_4; ++i) {
+ arr[i] =
+ (arr[i] << 48)
+ | (arr[BLOCK_SIZE_DIV_8_MUL_2 + i] << 32)
+ | (arr[BLOCK
[GitHub] [lucene] dweiss commented on a change in pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
dweiss commented on a change in pull request #567:
URL: https://github.com/apache/lucene/pull/567#discussion_r774530196
##
File path:
lucene/test-framework/src/java/org/apache/lucene/tests/util/LuceneTestCase.java
##
@@ -2060,18 +2061,14 @@ protected Path getDataPath(String name) throws
IOException {
try {
return Paths.get(
IOUtils.requireResourceNonNull(this.getClass().getResource(name),
name).toURI());
-} catch (Exception e) {
- throw new IOException("Cannot find resource: " + name, e);
+} catch (URISyntaxException e) {
Review comment:
This can actually be a non-filesystem URI in the future (if the test
framework is jarred)...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on a change in pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
dweiss commented on a change in pull request #567:
URL: https://github.com/apache/lucene/pull/567#discussion_r774530775
##
File path:
lucene/test-framework/src/java/org/apache/lucene/tests/util/LuceneTestCase.java
##
@@ -2060,18 +2061,14 @@ protected Path getDataPath(String name) throws
IOException {
try {
return Paths.get(
IOUtils.requireResourceNonNull(this.getClass().getResource(name),
name).toURI());
-} catch (Exception e) {
- throw new IOException("Cannot find resource: " + name, e);
+} catch (URISyntaxException e) {
Review comment:
Sorry - not just the test framework - test classes too. This is related
to running tests as a module. We can leave it until the problem surfaces though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta merged pull request #566: LUCENE-10301: Place test-framework into separated modules folder.
mocobeta merged pull request #566: URL: https://github.com/apache/lucene/pull/566 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10301) The test-framework as a module (or a separate test-framework-module)
[ https://issues.apache.org/jira/browse/LUCENE-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464514#comment-17464514 ] ASF subversion and git services commented on LUCENE-10301: -- Commit 54ebf3c73b72bafc414f834a15b8a3e81fd75ad8 in lucene's branch refs/heads/main from Tomoko Uchida [ https://gitbox.apache.org/repos/asf?p=lucene.git;h=54ebf3c ] LUCENE-10301: Place test-framework into separated modules folder. (#566) > The test-framework as a module (or a separate test-framework-module) > > > Key: LUCENE-10301 > URL: https://issues.apache.org/jira/browse/LUCENE-10301 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Major > Fix For: 9.1 > > Time Spent: 8h 20m > Remaining Estimate: 0h > > The test framework has split packages. It's a follow-up to introducing > modules but eventually the modular test subprojects will need something like > the test framework too. > I'm not sure whether we should start a new subproject for this or try to > refactor the test framework, but it's a follow-up once the modules themselves > are working and testable, I think. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-10334) Introduce a BlockReader based on ForUtil and use it for NumericDocValues
[
https://issues.apache.org/jira/browse/LUCENE-10334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463911#comment-17463911
]
Feng Guo edited comment on LUCENE-10334 at 12/23/21, 12:22 PM:
---
Thanks [~gsmiller] ! Yes I do thought i would get some regression in
sparse-hits tasks and the result suprised me too. Maybe we should thank to the
powerful {{{}ForUtil{}}}? :)
By reading codes in LUCENE-10033, i suspect there are two reasons that could
probably lead to a more obvious regression than this approach:
1. 10033 approach computes bpv for each small block and need to read the
pointer from a {{DirectMonotonicReader}} before seeking. While this approach is
using a global bpv and pointers can be computed by {{{}offset + blockBytes *
block{}}}. This could be faster. A global bpv can lead larger index size but i
think it acceptable since it's what we used to do.
2. 10033 approach decode offset/gcd/delta for each block, some of them could be
auto-vectorized but still a bit heavier. This approach is trying to make the
decoding of blocks as simple as possible and jobs like gcd decoding is only
done for hit docs.
I'm not really sure these are major reasons but should make the benchmark
result a bit more explainable.
was (Author: gf2121):
Thanks [~gsmiller] ! Yes I do thought i would get some regression in
sparse-hits tasks and the result suprised me too. Maybe we should thank to the
powerful {{{}ForUtil{}}}? :)
By reading codes in LUCENE-10033, i suspect there are two reasons that could
probably lead to a more obvious regression than this approach:
1. 10033 approach computes bpv for each small block and need to read the
pointer from a {{DirectMonotonicReader}} before seeking. While this approach is
using a global bpv and pointers can be computed by {{{}offset + blockBytes *
block{}}}. This could be faster. A global bpv can lead larger index size but i
think it acceptable since it's what we used to do.
2. 10033 approach decode offset/gcd/delta for each block, some of them could be
auto-vectorized but still a bit heavier. This approach is trying to make the
decoding of blocks as simple as possible and jobs like gcd decoding is only
done for hit docs.
I'm not really sure these are major reasons but should make the benchmark
result a bit more explainable. By the way, here is my localrun script. I post
it here in case there is something wrong with it. (I personally added
('sortedset:RandomLabel', "RandomLabel") because luceneutil can not run without
this, but i'm not sure this is correct since the readme did not mention it)
{code:python}
if __name__ == '__main__':
sourceData = competition.sourceData()
comp = competition.Competition()
facets = (('taxonomy:Date', 'Date'),('sortedset:Month',
'Month'),('sortedset:DayOfYear', 'DayOfYear'),('sortedset:RandomLabel',
"RandomLabel"))
index = comp.newIndex('lucene_baseline', sourceData, facets=facets,
indexSort='dayOfYearNumericDV:long')
candidate_index = comp.newIndex('lucene_candidate', sourceData,
facets=facets, indexSort='dayOfYearNumericDV:long')
#Warning -- Do not break the order of arguments
#TODO -- Fix the following by using argparser
if len(sys.argv) > 3 and sys.argv[3] == '-concurrentSearches':
concurrentSearches = True
else:
concurrentSearches = False
# create a competitor named baseline with sources in the ../trunk folder
comp.competitor('baseline', 'lucene_baseline',
index = index, concurrentSearches = concurrentSearches)
comp.competitor('my_modified_version', 'lucene_candidate',
index = candidate_index, concurrentSearches =
concurrentSearches)
# start the benchmark - this can take long depending on your index and
machines
comp.benchmark("baseline_vs_patch")
{code}
> Introduce a BlockReader based on ForUtil and use it for NumericDocValues
>
>
> Key: LUCENE-10334
> URL: https://issues.apache.org/jira/browse/LUCENE-10334
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Reporter: Feng Guo
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Previous talk is here: https://github.com/apache/lucene/pull/557
> This is trying to add a new BlockReader based on ForUtil to replace the
> DirectReader we are using for NumericDocvalues
> *Benchmark based on wiki10m*
> {code:java}
> TaskQPS baseline StdDevQPS
> my_modified_version StdDevPct diff p-value
>OrNotHighHigh 694.17 (8.2%) 685.83
> (7.0%) -1.2% ( -15% - 15%) 0.618
> Respell 75.15 (2.7%) 74.32
> (2.0%) -1.1% ( -5% -3%) 0.146
>
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000279111 I found another problematic one: loadStopwordSet in StopAnalyzerBase. This does not work from inside nor/kuromoji and others. I will deprecated in same way. I would like to make this method only visible from the analyzers-common module, but I am not sure how. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10334) Introduce a BlockReader based on ForUtil and use it for NumericDocValues
[
https://issues.apache.org/jira/browse/LUCENE-10334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464531#comment-17464531
]
Robert Muir commented on LUCENE-10334:
--
I think you took a good approach: small steps first, to make some progress.
Sorry I haven't looked at the PR yet, it is holidays here so I've just been
busy.
> Introduce a BlockReader based on ForUtil and use it for NumericDocValues
>
>
> Key: LUCENE-10334
> URL: https://issues.apache.org/jira/browse/LUCENE-10334
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Reporter: Feng Guo
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Previous talk is here: https://github.com/apache/lucene/pull/557
> This is trying to add a new BlockReader based on ForUtil to replace the
> DirectReader we are using for NumericDocvalues
> *Benchmark based on wiki10m*
> {code:java}
> TaskQPS baseline StdDevQPS
> my_modified_version StdDevPct diff p-value
>OrNotHighHigh 694.17 (8.2%) 685.83
> (7.0%) -1.2% ( -15% - 15%) 0.618
> Respell 75.15 (2.7%) 74.32
> (2.0%) -1.1% ( -5% -3%) 0.146
> Prefix3 220.11 (5.1%) 217.78
> (5.8%) -1.1% ( -11% - 10%) 0.541
> Wildcard 129.75 (3.7%) 128.63
> (2.5%) -0.9% ( -6% -5%) 0.383
> LowSpanNear 68.54 (2.1%) 68.00
> (2.4%) -0.8% ( -5% -3%) 0.269
> OrNotHighMed 732.90 (6.8%) 727.49
> (5.3%) -0.7% ( -12% - 12%) 0.703
> BrowseRandomLabelTaxoFacets11879.03 (8.6%)11799.33
> (5.5%) -0.7% ( -13% - 14%) 0.769
> HighSloppyPhrase6.87 (2.9%)6.83
> (2.3%) -0.6% ( -5% -4%) 0.496
> OrHighNotMed 827.54 (9.2%) 822.94
> (8.0%) -0.6% ( -16% - 18%) 0.838
> MedSpanNear 18.92 (5.7%) 18.82
> (5.6%) -0.5% ( -11% - 11%) 0.759
> OrHighMedDayTaxoFacets 10.27 (4.0%) 10.21
> (4.3%) -0.5% ( -8% -8%) 0.676
> PKLookup 207.98 (4.0%) 206.85
> (2.7%) -0.5% ( -7% -6%) 0.621
> LowIntervalsOrdered 159.17 (2.3%) 158.32
> (2.2%) -0.5% ( -4% -3%) 0.445
> HighSpanNear6.32 (4.2%)6.28
> (4.1%) -0.5% ( -8% -8%) 0.691
> MedIntervalsOrdered 85.31 (3.2%) 84.88
> (2.9%) -0.5% ( -6% -5%) 0.607
> HighTerm 1170.55 (5.8%) 1164.79
> (3.9%) -0.5% ( -9% -9%) 0.753
> LowSloppyPhrase 14.54 (3.1%) 14.48
> (2.9%) -0.4% ( -6% -5%) 0.651
> HighPhrase 112.81 (4.4%) 112.39
> (4.1%) -0.4% ( -8% -8%) 0.781
> OrNotHighLow 858.02 (5.9%) 854.99
> (4.8%) -0.4% ( -10% - 10%) 0.835
> HighIntervalsOrdered 25.08 (2.8%) 25.00
> (2.6%) -0.3% ( -5% -5%) 0.701
>MedPhrase 27.20 (2.1%) 27.11
> (2.9%) -0.3% ( -5% -4%) 0.689
> MedTermDayTaxoFacets 81.55 (2.3%) 81.35
> (2.9%) -0.3% ( -5% -5%) 0.762
> IntNRQ 63.36 (2.0%) 63.21
> (2.5%) -0.2% ( -4% -4%) 0.740
> Fuzzy2 73.24 (5.5%) 73.10
> (6.2%) -0.2% ( -11% - 12%) 0.916
> AndHighMedDayTaxoFacets 76.08 (3.5%) 75.98
> (3.4%) -0.1% ( -6% -7%) 0.905
> AndHighHigh 62.20 (2.0%) 62.18
> (2.4%) -0.0% ( -4% -4%) 0.954
>BrowseMonthTaxoFacets11993.48 (6.7%)11989.53
> (4.8%) -0.0% ( -10% - 12%) 0.986
> OrHighNotLow 732.82 (7.2%) 732.80
> (6.2%) -0.0% ( -12% - 14%) 0.999
> Fuzzy1 46.43 (5.3%) 46.45
> (6.0%)0.0% ( -10% - 11%) 0.989
> LowTerm 1608.25 (6.0%) 1608.84
> (4.9%)0.0% ( -10% - 11%) 0.983
>OrHighMed 75.90 (2.3%) 75.93
> (1.8%)0.0% ( -3% -4%) 0.939
>LowPhrase 273.81 (2.9%) 274.04
> (3.3%)0.1% ( -5% -6%) 0.932
>
[GitHub] [lucene] uschindler commented on a change in pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on a change in pull request #567:
URL: https://github.com/apache/lucene/pull/567#discussion_r774546832
##
File path:
lucene/test-framework/src/java/org/apache/lucene/tests/util/LuceneTestCase.java
##
@@ -2060,18 +2061,14 @@ protected Path getDataPath(String name) throws
IOException {
try {
return Paths.get(
IOUtils.requireResourceNonNull(this.getClass().getResource(name),
name).toURI());
-} catch (Exception e) {
- throw new IOException("Cannot find resource: " + name, e);
+} catch (URISyntaxException e) {
Review comment:
Yes, this one may break. So we may need to hack around by using a
ModuleClassLoader directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on a change in pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on a change in pull request #567:
URL: https://github.com/apache/lucene/pull/567#discussion_r774547308
##
File path:
lucene/test-framework/src/java/org/apache/lucene/tests/util/LuceneTestCase.java
##
@@ -2060,18 +2061,14 @@ protected Path getDataPath(String name) throws
IOException {
try {
return Paths.get(
IOUtils.requireResourceNonNull(this.getClass().getResource(name),
name).toURI());
-} catch (Exception e) {
- throw new IOException("Cannot find resource: " + name, e);
+} catch (URISyntaxException e) {
Review comment:
I am 100% sure we can't modify those to run as modules, because this
method is often used to open an index or similar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
rmuir commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000280027 How can we have failing tests for these problems? We need a test-driven approach to fixing this stuff. `gradlew check` is totally happy for me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000280448 > How can we have failing tests for these problems? We need a test-driven approach to fixing this stuff. `gradlew check` is totally happy for me. This is preparatory. We have a branch where it does not work yet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler edited a comment on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler edited a comment on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000280448 > How can we have failing tests for these problems? We need a test-driven approach to fixing this stuff. `gradlew check` is totally happy for me. This is preparatory. We have a branch where it does not work because of this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler edited a comment on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler edited a comment on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000280448 > How can we have failing tests for these problems? We need a test-driven approach to fixing this stuff. `gradlew check` is totally happy for me. This is preparatory. We have a branch where it does not work because of this. See the issue for explanation. The problem is that getResourceAsStream() and getResource() are caller-sensitive and check who calls it. If its a classpath app, it works as the checks are gnored. With module system the method returns null if you try to load a resource in another module. I will add a test to the distribution tests to show this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler edited a comment on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler edited a comment on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000280448 > How can we have failing tests for these problems? We need a test-driven approach to fixing this stuff. `gradlew check` is totally happy for me. This is preparatory. We have a branch where it does not work because of this. See the issue for explanation. The problem is that getResourceAsStream() and getResource() are caller-sensitive and check who calls it. If its a classpath app, it works as the checks are gnored. With module system the method returns null if you try to load a resource in another module. I will add a test to the distribution tests to show this. The basic problem is: analyzer module calls into core module and core module tries to load resource from analyzers module. So the indirection analyzer -> core -> analyzer must go away -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler edited a comment on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler edited a comment on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000280448 > How can we have failing tests for these problems? We need a test-driven approach to fixing this stuff. `gradlew check` is totally happy for me. This is preparatory. We have a branch where it does not work because of this. See the issue for explanation. The problem is that getResourceAsStream() and getResource() are caller-sensitive and check who calls it. If its a classpath app, it works as the checks are ignored. With module system the method returns null if you try to load a resource in another module. I will add a test to the distribution tests to show this. The basic problem is: analyzer module calls into core module and core module tries to load resource from analyzers module. So the indirection analyzer -> core -> analyzer must go away -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000283219 To explain: We need to go away with a public static helper class that calls `Class#getResourceAsStream()` on behalf of somebody else. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
rmuir commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000283590 I understand why the method doesnt work for modules: I just don't care. I am only asking about failing tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000284297 > I understand why the method doesnt work for modules: I just don't care. > > I am only asking about failing tests. For now I can only make sure I don't break the status as is. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000283961 @rmuir we will have a test-driven approach soon after xmas when Dawid finishes the conversion to module system. But all work that can be done before should be done before, as it will be a messy huge patch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
rmuir commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000285508 Also, are we sure this stuff isn't bugs in the JDK? Seems like a design flaw in modules. I have the feeling this is going to break the analysis module left and right. I am concerned about things like analyzer factories, stopwordananalyzerbase, the list goes on and on. I cannot "see" the scope of the problem without failing tests LOL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
rmuir commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000285949 > The problem is that getResourceAsStream() and getResource() are caller-sensitive and check who calls it. That's openjdk internal bullshit and not in the public java docs. we need to push back on openjdk here, this is a bug! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10334) Introduce a BlockReader based on ForUtil and use it for NumericDocValues
[
https://issues.apache.org/jira/browse/LUCENE-10334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464539#comment-17464539
]
Feng Guo commented on LUCENE-10334:
---
Thanks [~rcmuir] for reply! No hurry here, feel free to ignore this and have a
nice holiday :)
> Introduce a BlockReader based on ForUtil and use it for NumericDocValues
>
>
> Key: LUCENE-10334
> URL: https://issues.apache.org/jira/browse/LUCENE-10334
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Reporter: Feng Guo
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Previous talk is here: https://github.com/apache/lucene/pull/557
> This is trying to add a new BlockReader based on ForUtil to replace the
> DirectReader we are using for NumericDocvalues
> *Benchmark based on wiki10m*
> {code:java}
> TaskQPS baseline StdDevQPS
> my_modified_version StdDevPct diff p-value
>OrNotHighHigh 694.17 (8.2%) 685.83
> (7.0%) -1.2% ( -15% - 15%) 0.618
> Respell 75.15 (2.7%) 74.32
> (2.0%) -1.1% ( -5% -3%) 0.146
> Prefix3 220.11 (5.1%) 217.78
> (5.8%) -1.1% ( -11% - 10%) 0.541
> Wildcard 129.75 (3.7%) 128.63
> (2.5%) -0.9% ( -6% -5%) 0.383
> LowSpanNear 68.54 (2.1%) 68.00
> (2.4%) -0.8% ( -5% -3%) 0.269
> OrNotHighMed 732.90 (6.8%) 727.49
> (5.3%) -0.7% ( -12% - 12%) 0.703
> BrowseRandomLabelTaxoFacets11879.03 (8.6%)11799.33
> (5.5%) -0.7% ( -13% - 14%) 0.769
> HighSloppyPhrase6.87 (2.9%)6.83
> (2.3%) -0.6% ( -5% -4%) 0.496
> OrHighNotMed 827.54 (9.2%) 822.94
> (8.0%) -0.6% ( -16% - 18%) 0.838
> MedSpanNear 18.92 (5.7%) 18.82
> (5.6%) -0.5% ( -11% - 11%) 0.759
> OrHighMedDayTaxoFacets 10.27 (4.0%) 10.21
> (4.3%) -0.5% ( -8% -8%) 0.676
> PKLookup 207.98 (4.0%) 206.85
> (2.7%) -0.5% ( -7% -6%) 0.621
> LowIntervalsOrdered 159.17 (2.3%) 158.32
> (2.2%) -0.5% ( -4% -3%) 0.445
> HighSpanNear6.32 (4.2%)6.28
> (4.1%) -0.5% ( -8% -8%) 0.691
> MedIntervalsOrdered 85.31 (3.2%) 84.88
> (2.9%) -0.5% ( -6% -5%) 0.607
> HighTerm 1170.55 (5.8%) 1164.79
> (3.9%) -0.5% ( -9% -9%) 0.753
> LowSloppyPhrase 14.54 (3.1%) 14.48
> (2.9%) -0.4% ( -6% -5%) 0.651
> HighPhrase 112.81 (4.4%) 112.39
> (4.1%) -0.4% ( -8% -8%) 0.781
> OrNotHighLow 858.02 (5.9%) 854.99
> (4.8%) -0.4% ( -10% - 10%) 0.835
> HighIntervalsOrdered 25.08 (2.8%) 25.00
> (2.6%) -0.3% ( -5% -5%) 0.701
>MedPhrase 27.20 (2.1%) 27.11
> (2.9%) -0.3% ( -5% -4%) 0.689
> MedTermDayTaxoFacets 81.55 (2.3%) 81.35
> (2.9%) -0.3% ( -5% -5%) 0.762
> IntNRQ 63.36 (2.0%) 63.21
> (2.5%) -0.2% ( -4% -4%) 0.740
> Fuzzy2 73.24 (5.5%) 73.10
> (6.2%) -0.2% ( -11% - 12%) 0.916
> AndHighMedDayTaxoFacets 76.08 (3.5%) 75.98
> (3.4%) -0.1% ( -6% -7%) 0.905
> AndHighHigh 62.20 (2.0%) 62.18
> (2.4%) -0.0% ( -4% -4%) 0.954
>BrowseMonthTaxoFacets11993.48 (6.7%)11989.53
> (4.8%) -0.0% ( -10% - 12%) 0.986
> OrHighNotLow 732.82 (7.2%) 732.80
> (6.2%) -0.0% ( -12% - 14%) 0.999
> Fuzzy1 46.43 (5.3%) 46.45
> (6.0%)0.0% ( -10% - 11%) 0.989
> LowTerm 1608.25 (6.0%) 1608.84
> (4.9%)0.0% ( -10% - 11%) 0.983
>OrHighMed 75.90 (2.3%) 75.93
> (1.8%)0.0% ( -3% -4%) 0.939
>LowPhrase 273.81 (2.9%) 274.04
> (3.3%)0.1% ( -5% -6%) 0.932
> AndHighLow 717.24 (6.1%) 718.17
> (3.3%)
[GitHub] [lucene] rmuir commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
rmuir commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000286691 Sorry, I have to say I am -1 at the moment to "gutting" all the analysis code, without tests, before first even trying to report these bugs to openjdk (it is THEIR FAULT). We should get it fixed upstream. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000288771 > Also, are we sure this stuff isn't bugs in the JDK? Seems like a design flaw in modules. I was in the discussion before Java 9 came out. It was a mess and I argued several times, because you were not even able to oad class files anymore as resources (now they have an exception). getResourceAsStream() was changed several times to work around some problem with legacy apps. At the end we came out with this and it is clearly documented in the JLS and the API: - a Class instance located in unnamed modules (classpath) can be used to load any resource (it's the backwards compatibility layer) - a Class instance for a class located in a named module can only load resoures that the caller of getResourceAsStream can see (exported packages). This won't be problematic for analyzers module unless we use static utility methods. In the analyzers module it is not a problem anymore, only a few issues are there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler edited a comment on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler edited a comment on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000288771 > Also, are we sure this stuff isn't bugs in the JDK? Seems like a design flaw in modules. I was in the discussion before Java 9 came out. It was a mess and I argued several times, because you were not even able to load class files anymore as resources (now they have an exception for that specific use case in ClassLoader). `Class#getResourceAsStream()` (and getResource, too) was changed several times to work around some problems with legacy apps. At the end we came out with this and it is clearly documented in the JLS and the API and you won't change it anymore: - a Class instance located in unnamed modules (classpath) can be used to load any resource (it's the backwards compatibility layer) - a Class instance for a class located in a named module can only load resoures that the caller of getResourceAsStream can see (exported packages). This won't be problematic for analyzers module unless we use static utility methods. In the analyzers module it is not a problem anymore, only a few issues are there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
rmuir commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000289938 Where is it clearly documented? I don't see any such documentation: BUG. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000290825 Robert, please let me finish this, otherwise I will trash down everything and stop working anymore on module system. If you want modules (and looks like you want) let us do our work and be calm. Thanks. I will work more and commit this a bit later. These changes are not risky at all and IMHO I fixed a lot of other stuff around. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
rmuir commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000292034 My problem is that the justification for fixing this stuff is that, "this code is buggy, it calls a CallerSensitive method". CallerSensitive isn't even publicly documented. This is OpenJDK internal stuff. OpenJDK BUGS! Uwe, I don't meant to discourage you from working on modules. It isn't a personal attack. I am angry about the OpenJDK behavior here and honestly believe this stuff is a bug. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000292571 (https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Class.html#getResourceAsStream(java.lang.String)): > Finds a resource with a given name. > If this class is in a named Module then this method will attempt to find the resource in the module. This is done by delegating to the module's class loader findResource(String,String) method, invoking it with the module name and the absolute name of the resource. Resources in named modules are subject to the rules for encapsulation specified in the Module getResourceAsStream method and so this method returns null when the resource is a non-".class" resource in a package that is not open to the caller's module. > > Otherwise, if this class is not in a named module then the rules for searching resources associated with a given class are implemented by the defining class loader of the class. This method delegates to this object's class loader. If this object was loaded by the bootstrap class loader, the method delegates to ClassLoader.getSystemResourceAsStream(java.lang.String). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler edited a comment on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler edited a comment on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000292571 (https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Class.html#getResourceAsStream(java.lang.String)): > Finds a resource with a given name. > If this class is in a named Module then this method will attempt to find the resource in the module. This is done by delegating to the module's class loader findResource(String,String) method, invoking it with the module name and the absolute name of the resource. Resources in named modules are subject to the rules for encapsulation specified in the Module getResourceAsStream method and so this method *returns null when the resource is a non-".class" resource in a package that is not open to the caller's module.* > > Otherwise, if this class is not in a named module then the rules for searching resources associated with a given class are implemented by the defining class loader of the class. This method delegates to this object's class loader. If this object was loaded by the bootstrap class loader, the method delegates to ClassLoader.getSystemResourceAsStream(java.lang.String). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta commented on pull request #550: LUCENE-8930: script testing in the distribution
mocobeta commented on pull request #550: URL: https://github.com/apache/lucene/pull/550#issuecomment-1000293345 Could I merge the main into this branch to fetch https://github.com/apache/lucene/commit/54ebf3c73b72bafc414f834a15b8a3e81fd75ad8 just to see how Github Actions behave? There is a conflict and I'll be able to resolve it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler edited a comment on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler edited a comment on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000292571 Hi, see the bold text and especially the italic one: (https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Class.html#getResourceAsStream(java.lang.String)): > Finds a resource with a given name. > > If this class is in a named Module then this method will attempt to find the resource in the module. This is done by delegating to the module's class loader findResource(String,String) method, invoking it with the module name and the absolute name of the resource. Resources in named modules are subject to the rules for encapsulation specified in the Module getResourceAsStream method and so this method **returns null when the resource is a non-".class" resource in a package that is not open to the _caller's_ module.** > > Otherwise, if this class is not in a named module then the rules for searching resources associated with a given class are implemented by the defining class loader of the class. This method delegates to this object's class loader. If this object was loaded by the bootstrap class loader, the method delegates to ClassLoader.getSystemResourceAsStream(java.lang.String). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
rmuir commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000295132 Thanks Uwe. I suppose "caller" there is enough to justify their shitty CallerSensitive shenanigans. This really sucks though, they screwed up. Please proceed, sorry for noise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000295957 > Thanks Uwe. I suppose "caller" there is enough to justify their shitty CallerSensitive shenanigans. This really sucks though, they screwed up. I was angry about that 4 or 5 years ago already. By the way the exception for "class" files was my work at that time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000300518 I am thinking about a small solution to still use `StopAnalyzerBase#loadStopwordSet(...,Class,...)` and not duplicate code everywhere like in current patch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
rmuir commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000306918 > I am thinking about a small solution to still use `StopAnalyzerBase#loadStopwordSet(...,Class,...)` and not duplicate code everywhere like in current patch. But how will we fix analyzer factories (ClasspathResourceLoader) ? This seems like the biggest issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
rmuir commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000312182 Maybe, we should just go back to hacky code to unzip jar file, open file on filesystem. after all, "resources" are practically always there on the filesystem anyway. We can just open them in a different way and stop using Class.getResourceAsStream! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000312397 > > I am thinking about a small solution to still use `StopAnalyzerBase#loadStopwordSet(...,Class,...)` and not duplicate code everywhere like in current patch. > > But how will we fix analyzer factories (ClasspathResourceLoader) ? This seems like the biggest issue. The class will work well in ClassPath mode, of course. But we should deprecate the constructor that takes a class. The one taking a ClassLoader can be initialized by an app like a modularized Elasticsearch or Solr using a ClassLoader retrieved from the module. We did this inside Luke already. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler edited a comment on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler edited a comment on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000312397 > > I am thinking about a small solution to still use `StopAnalyzerBase#loadStopwordSet(...,Class,...)` and not duplicate code everywhere like in current patch. > > But how will we fix analyzer factories (ClasspathResourceLoader) ? This seems like the biggest issue. The class will work well in ClassPath mode, of course (the name is alread "Classpath*"). But we should deprecate the constructor that takes a class and/or add a ModuleResourceLoader taing a set of java modules. But that's a separate issue. The one taking a ClassLoader can be initialized by an app like a modularized Elasticsearch or Solr using a ClassLoader retrieved from the module. We did this inside Luke already. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #550: LUCENE-8930: script testing in the distribution
dweiss commented on pull request #550: URL: https://github.com/apache/lucene/pull/550#issuecomment-1000317547 Please, feel free to do anything you like, @mocobeta ! Like I said, I just left it as an example. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler edited a comment on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler edited a comment on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000312397 > > I am thinking about a small solution to still use `StopAnalyzerBase#loadStopwordSet(...,Class,...)` and not duplicate code everywhere like in current patch. > > But how will we fix analyzer factories (ClasspathResourceLoader) ? This seems like the biggest issue. The class will work well in ClassPath mode, of course (the name is alread "Classpath*"). But we should deprecate the constructor that takes a class and/or add a ModuleResourceLoader taking a set of java modules. But that's a separate issue. The ctor taking a ClassLoader can be initialized by an app like a modularized Elasticsearch or Solr using a ClassLoader retrieved from the module. We did this inside Luke already. But still the package needs to open all its resource folders. This will be a pain. Nevertheless for Lucene itsself it is not a problem, because we know our module graph. It is just bad that we can't provide those simple "utility" methods to the outside. I also checked the Module system classlaoders and the Module class, all are caller sensitive. IMHO, it's a pain that so much reflection specific stuff is now caller sensitive (since Java 9), this makes utility methods impossible to expose. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000319417 StackOverflow is full of this shit, just search for it: https://www.google.com/search?q=getresourceasstream+java+11 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler edited a comment on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler edited a comment on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000312397 > > I am thinking about a small solution to still use `StopAnalyzerBase#loadStopwordSet(...,Class,...)` and not duplicate code everywhere like in current patch. > > But how will we fix analyzer factories (ClasspathResourceLoader) ? This seems like the biggest issue. The class will work well in ClassPath mode, of course (the name is alread "Classpath*"). But we should deprecate the constructor that takes a class and/or add a ModuleResourceLoader taking a set of java modules. But that's a separate issue. The ctor taking a ClassLoader can be initialized by an app like a modularized Elasticsearch or Solr using a ClassLoader retrieved from the module. We did this inside Luke already. But still the package needs to open all its resource folders. This will be a pain. Nevertheless for Lucene itsself it is not a problem, because we know our module graph. It is just bad that we can't provide those simple "utility" methods to the outside. I also checked the Module system classlaoders and the Module class, all are caller sensitive. IMHO, it's a pain that so much reflection specific stuff is now caller sensitive (since Java 9), this makes utility methods impossible to expose. It is not only getResource, also a lot of other shit like classloader access and method visibility. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
dweiss commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000320538 I know everyone is busy, so keeping it short. @rmuir - I do have tests that fail because of this. These tests are on a branch where I'm trying to fix classpath vs. module-path issues - the compilation and tests currently use classpath only; once you make them run with true modules, things start to break. It is very messy conceptually - the conventions gradle has predate the module system and they're not always compatible. Every time I think I've solved something, two other things emerge. It is difficult. I am making progress though. > we will have a test-driven approach soon after xmas when Dawid finishes the conversion to module system Thank you for your faith in me, Uwe. I hold on to your word. ;) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler edited a comment on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler edited a comment on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000312397 > > I am thinking about a small solution to still use `StopAnalyzerBase#loadStopwordSet(...,Class,...)` and not duplicate code everywhere like in current patch. > > But how will we fix analyzer factories (ClasspathResourceLoader) ? This seems like the biggest issue. The class will work well in ClassPath mode, of course (the name is alread "Classpath*"). But we should deprecate the constructor that takes a class and/or add a ModuleResourceLoader taking a set of java modules. But that's a separate issue. The ctor taking a ClassLoader can be initialized by an app like a modularized Elasticsearch or Solr using a ClassLoader retrieved from the module. We did this inside Luke already. But still the package needs to open all its resource folders. This will be a pain. Nevertheless for Lucene itsself it is not a problem, because we know our module graph. It is just bad that we can't provide those simple "utility" methods to the outside. I also checked the Module system classlaoders and the Module class, all are caller sensitive. IMHO, it's a pain that so much reflection specific stuff is now caller sensitive (since Java 9), this makes utility methods impossible to expose. It is not only getResource, also a lot of other shit like classloader access and also Class.forName() is now caller sensitive! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
dweiss commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000321303 The code Uwe provided in the patch is fixing a true bug - it's just masked by the tests running in classpath only. In other words: if the tests pass after applying this patch, nothing changes for the classpath mode... and things will improve on the module-system layer side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
uschindler commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000328063 I am giving up for today, have to think about it a bit more. @dweiss you can take my current PR for further testing. From checking the code it looks like we need a ModuleResourceLoader class as complement to ClasspathResourceLoader. But I will open a separate PR about this when Dawid is done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10301) The test-framework as a module (or a separate test-framework-module)
[ https://issues.apache.org/jira/browse/LUCENE-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464587#comment-17464587 ] ASF subversion and git services commented on LUCENE-10301: -- Commit cc894d90d87da3a068cb8959abbf0fd78fe89aa2 in lucene's branch refs/heads/branch_9x from Tomoko Uchida [ https://gitbox.apache.org/repos/asf?p=lucene.git;h=cc894d9 ] LUCENE-10301: Place test-framework into separated modules folder. (#566) > The test-framework as a module (or a separate test-framework-module) > > > Key: LUCENE-10301 > URL: https://issues.apache.org/jira/browse/LUCENE-10301 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Major > Fix For: 9.1 > > Time Spent: 8h 20m > Remaining Estimate: 0h > > The test framework has split packages. It's a follow-up to introducing > modules but eventually the modular test subprojects will need something like > the test framework too. > I'm not sure whether we should start a new subproject for this or try to > refactor the test framework, but it's a follow-up once the modules themselves > are working and testable, I think. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
rmuir commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000337322 > IMHO, it's a pain that so much reflection specific stuff is now caller sensitive (since Java 9), this makes utility methods impossible to expose. It is not only getResource, also a lot of other shit like classloader access and also Class.forName() is now caller sensitive! Yes, I just did a quick `git grep` and immediately start finding bugs. first hit is `RSLPStemmerBase.parse()`. The class is public/abstract, so the code should be fixed. We can't reduce visibility of the class, it is needed by both Portuguese and Galician (separate packages). So we have to fix signatures etc. No tests will fail, but its a lurking "API bug", if someone tries to use it from the outside. I am trying to think on some ideas to statically identify this stuff. Maybe we just need to use forbidden apis. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
dweiss commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000338704 Thanks Uwe. I will integrate this patch into my working branch with gradle changes. I don't think I can fix everything related to module vs. classpath but I think it's close to being reasonably useful. Merry Christmas! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
rmuir commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000339877 > @rmuir - I do have tests that fail because of this. These tests are on a branch where I'm trying to fix classpath vs. module-path issues - the compilation and tests currently use classpath only; once you make them run with true modules, things start to break. It is very messy conceptually - the conventions gradle has predate the module system and they're not always compatible. Every time I think I've solved something, two other things emerge. It is difficult. I am making progress though. Thanks Dawid. I didn't mean to complain, I was just confused at the sudden issues and wondering about the larger strategy to fix stuff. At this point I'm not even sure tests will be enough, due to the fact lucene is an API (see my example above about RSLP, first one i looked into). So I'm trying to think of other ways to make the build fail when our APIs do this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10334) Introduce a BlockReader based on ForUtil and use it for NumericDocValues
[
https://issues.apache.org/jira/browse/LUCENE-10334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464619#comment-17464619
]
Feng Guo commented on LUCENE-10334:
---
I'm so sorry to say that there is something wrong with my benchmark: The
localrun script is still using the facets described in luceneutil
[readme|https://github.com/mikemccand/luceneutil/blob/master/README.md], like
this:
{code:python}
facets = (('taxonomy:Date', 'Date'),('sortedset:Month',
'Month'),('sortedset:DayOfYear', 'DayOfYear'))
index = comp.newIndex('lucene_baseline', sourceData, facets=facets,
indexSort='dayOfYearNumericDV:long')
{code}
And i got the result mentioned above with this facets. But when i'm cloning a
new luceneutil and rerun the setup.py, it becomes:
{code:python}
index = comp.newIndex('lucene_baseline', sourceData,
facets = (('taxonomy:Date', 'Date'),
('taxonomy:Month', 'Month'),
('taxonomy:DayOfYear', 'DayOfYear'),
('sortedset:Month', 'Month'),
('sortedset:DayOfYear', 'DayOfYear'),
('taxonomy:RandomLabel', 'RandomLabel'),
('sortedset:RandomLabel', 'RandomLabel')))
{code}
And the result is totally different with this:
{code:java}
TaskQPS baseline StdDevQPS my_modified_version
StdDevPct diff p-value
BrowseDayOfYearTaxoFacets 13.65 (8.9%) 10.49
(2.6%) -23.2% ( -31% - -12%) 0.000
BrowseMonthTaxoFacets 13.54 (14.6%) 10.89
(2.9%) -19.6% ( -32% - -2%) 0.000
BrowseDateTaxoFacets 13.50 (8.8%) 11.11
(3.7%) -17.7% ( -27% - -5%) 0.000
BrowseRandomLabelTaxoFacets 11.78 (7.0%)9.94
(5.1%) -15.6% ( -25% - -3%) 0.000
MedTermDayTaxoFacets 47.49 (2.4%) 41.45
(3.4%) -12.7% ( -18% - -7%) 0.000
AndHighMedDayTaxoFacets 130.24 (2.7%) 119.48
(3.9%) -8.3% ( -14% - -1%) 0.000
AndHighHighDayTaxoFacets 28.80 (2.8%) 27.09
(3.1%) -5.9% ( -11% -0%) 0.000
OrHighMedDayTaxoFacets9.68 (2.7%)9.35
(2.8%) -3.4% ( -8% -2%) 0.000
HighTermDayOfYearSort 139.73 (9.6%) 135.74
(10.2%) -2.9% ( -20% - 18%) 0.361
TermDTSort 151.46 (9.0%) 147.40
(7.7%) -2.7% ( -17% - 15%) 0.311
Fuzzy2 35.22 (6.3%) 34.38
(5.9%) -2.4% ( -13% - 10%) 0.213
MedSloppyPhrase 78.99 (6.7%) 77.21
(7.1%) -2.3% ( -15% - 12%) 0.300
LowTerm 1636.38 (6.4%) 1600.26
(9.6%) -2.2% ( -17% - 14%) 0.392
LowPhrase 252.68 (3.8%) 247.11
(6.5%) -2.2% ( -12% -8%) 0.189
Respell 61.23 (2.3%) 59.89
(5.0%) -2.2% ( -9% -5%) 0.078
AndHighHigh 56.54 (2.6%) 55.43
(4.3%) -2.0% ( -8% -5%) 0.084
MedSpanNear 99.37 (2.4%) 97.44
(5.2%) -1.9% ( -9% -5%) 0.128
HighSloppyPhrase 28.58 (5.4%) 28.05
(5.4%) -1.8% ( -11% -9%) 0.280
PKLookup 198.95 (3.0%) 195.34
(4.8%) -1.8% ( -9% -6%) 0.148
AndHighMed 116.50 (3.3%) 114.65
(4.5%) -1.6% ( -9% -6%) 0.204
Fuzzy1 75.07 (6.4%) 73.99
(8.1%) -1.4% ( -14% - 13%) 0.532
HighSpanNear 10.73 (2.8%) 10.58
(3.9%) -1.4% ( -7% -5%) 0.180
LowSpanNear 43.92 (2.4%) 43.30
(3.4%) -1.4% ( -6% -4%) 0.128
LowSloppyPhrase 14.70 (4.4%) 14.50
(4.2%) -1.3% ( -9% -7%) 0.329
HighTermMonthSort 148.80 (8.3%) 146.84
(8.1%) -1.3% ( -16% - 16%) 0.612
OrHighMed 103.00 (3.2%) 101.67
(5.1%) -1.3% ( -9% -7%) 0.341
MedIntervalsOrdered5.44 (2.5%)5.37
(2.2%) -1.3% ( -5% -3%) 0.092
OrHighNotHigh 648.74 (6.7%) 640.81
(8.8%) -1.2% ( -15% - 15%) 0.621
MedPhrase 80.35 (2.7%) 79.38
(4.8%) -1.2% ( -8% -6%) 0.327
HighTerm 1384.91 (6.8%) 1369.27
(8.8%) -1.1% ( -15% - 15%) 0.650
IntNRQ 127.36 (2.8%)
[jira] [Comment Edited] (LUCENE-10334) Introduce a BlockReader based on ForUtil and use it for NumericDocValues
[
https://issues.apache.org/jira/browse/LUCENE-10334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464619#comment-17464619
]
Feng Guo edited comment on LUCENE-10334 at 12/23/21, 2:34 PM:
--
I'm so sorry to say that there is something wrong with my benchmark: The
localrun script is still using the facets described in luceneutil
[readme|https://github.com/mikemccand/luceneutil/blob/master/README.md], like
this:
{code:python}
facets = (('taxonomy:Date', 'Date'),('sortedset:Month',
'Month'),('sortedset:DayOfYear', 'DayOfYear'))
index = comp.newIndex('lucene_baseline', sourceData, facets=facets,
indexSort='dayOfYearNumericDV:long')
{code}
And i got the result mentioned above with this facets. But when i'm cloning a
new luceneutil and rerun the setup.py, it becomes:
{code:python}
index = comp.newIndex('lucene_baseline', sourceData,
facets = (('taxonomy:Date', 'Date'),
('taxonomy:Month', 'Month'),
('taxonomy:DayOfYear', 'DayOfYear'),
('sortedset:Month', 'Month'),
('sortedset:DayOfYear', 'DayOfYear'),
('taxonomy:RandomLabel', 'RandomLabel'),
('sortedset:RandomLabel', 'RandomLabel')))
{code}
And the result is totally different with this:
{code:java}
TaskQPS baseline StdDevQPS my_modified_version
StdDevPct diff p-value
BrowseDayOfYearTaxoFacets 13.65 (8.9%) 10.49
(2.6%) -23.2% ( -31% - -12%) 0.000
BrowseMonthTaxoFacets 13.54 (14.6%) 10.89
(2.9%) -19.6% ( -32% - -2%) 0.000
BrowseDateTaxoFacets 13.50 (8.8%) 11.11
(3.7%) -17.7% ( -27% - -5%) 0.000
BrowseRandomLabelTaxoFacets 11.78 (7.0%)9.94
(5.1%) -15.6% ( -25% - -3%) 0.000
MedTermDayTaxoFacets 47.49 (2.4%) 41.45
(3.4%) -12.7% ( -18% - -7%) 0.000
AndHighMedDayTaxoFacets 130.24 (2.7%) 119.48
(3.9%) -8.3% ( -14% - -1%) 0.000
AndHighHighDayTaxoFacets 28.80 (2.8%) 27.09
(3.1%) -5.9% ( -11% -0%) 0.000
OrHighMedDayTaxoFacets9.68 (2.7%)9.35
(2.8%) -3.4% ( -8% -2%) 0.000
HighTermDayOfYearSort 139.73 (9.6%) 135.74
(10.2%) -2.9% ( -20% - 18%) 0.361
TermDTSort 151.46 (9.0%) 147.40
(7.7%) -2.7% ( -17% - 15%) 0.311
Fuzzy2 35.22 (6.3%) 34.38
(5.9%) -2.4% ( -13% - 10%) 0.213
MedSloppyPhrase 78.99 (6.7%) 77.21
(7.1%) -2.3% ( -15% - 12%) 0.300
LowTerm 1636.38 (6.4%) 1600.26
(9.6%) -2.2% ( -17% - 14%) 0.392
LowPhrase 252.68 (3.8%) 247.11
(6.5%) -2.2% ( -12% -8%) 0.189
Respell 61.23 (2.3%) 59.89
(5.0%) -2.2% ( -9% -5%) 0.078
AndHighHigh 56.54 (2.6%) 55.43
(4.3%) -2.0% ( -8% -5%) 0.084
MedSpanNear 99.37 (2.4%) 97.44
(5.2%) -1.9% ( -9% -5%) 0.128
HighSloppyPhrase 28.58 (5.4%) 28.05
(5.4%) -1.8% ( -11% -9%) 0.280
PKLookup 198.95 (3.0%) 195.34
(4.8%) -1.8% ( -9% -6%) 0.148
AndHighMed 116.50 (3.3%) 114.65
(4.5%) -1.6% ( -9% -6%) 0.204
Fuzzy1 75.07 (6.4%) 73.99
(8.1%) -1.4% ( -14% - 13%) 0.532
HighSpanNear 10.73 (2.8%) 10.58
(3.9%) -1.4% ( -7% -5%) 0.180
LowSpanNear 43.92 (2.4%) 43.30
(3.4%) -1.4% ( -6% -4%) 0.128
LowSloppyPhrase 14.70 (4.4%) 14.50
(4.2%) -1.3% ( -9% -7%) 0.329
HighTermMonthSort 148.80 (8.3%) 146.84
(8.1%) -1.3% ( -16% - 16%) 0.612
OrHighMed 103.00 (3.2%) 101.67
(5.1%) -1.3% ( -9% -7%) 0.341
MedIntervalsOrdered5.44 (2.5%)5.37
(2.2%) -1.3% ( -5% -3%) 0.092
OrHighNotHigh 648.74 (6.7%) 640.81
(8.8%) -1.2% ( -15% - 15%) 0.621
MedPhrase 80.35 (2.7%) 79.38
(4.8%) -1.2% ( -8% -6%) 0.327
HighTerm 1384.91 (6.8%) 1369.27
(8.8%) -1.1% ( -15% - 15%) 0.650
[GitHub] [lucene] rmuir commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
rmuir commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000346720 > The code Uwe provided in the patch is fixing a true bug - it's just masked by the tests running in classpath only. In other words: if the tests pass after applying this patch, nothing changes for the classpath mode... and things will improve on the module-system layer side. Sure, I get that. But If i revert his commit or otherwise introduce regression, tests keep passing :) That's my point. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-10334) Introduce a BlockReader based on ForUtil and use it for NumericDocValues
[
https://issues.apache.org/jira/browse/LUCENE-10334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464619#comment-17464619
]
Feng Guo edited comment on LUCENE-10334 at 12/23/21, 2:37 PM:
--
I'm so sorry to tell that there is something wrong with my benchmark: The
localrun script was still using the facets described in luceneutil
[readme|https://github.com/mikemccand/luceneutil/blob/master/README.md], like
this:
{code:python}
facets = (('taxonomy:Date', 'Date'),('sortedset:Month',
'Month'),('sortedset:DayOfYear', 'DayOfYear'))
index = comp.newIndex('lucene_baseline', sourceData, facets=facets,
indexSort='dayOfYearNumericDV:long')
{code}
And i got the result mentioned above with this facets.
But when i'm cloning a new luceneutil and rerun the setup.py, it becomes:
{code:python}
index = comp.newIndex('lucene_baseline', sourceData,
facets = (('taxonomy:Date', 'Date'),
('taxonomy:Month', 'Month'),
('taxonomy:DayOfYear', 'DayOfYear'),
('sortedset:Month', 'Month'),
('sortedset:DayOfYear', 'DayOfYear'),
('taxonomy:RandomLabel', 'RandomLabel'),
('sortedset:RandomLabel', 'RandomLabel')))
{code}
And the result is totally different with this:
{code:java}
TaskQPS baseline StdDevQPS my_modified_version
StdDevPct diff p-value
BrowseDayOfYearTaxoFacets 13.65 (8.9%) 10.49
(2.6%) -23.2% ( -31% - -12%) 0.000
BrowseMonthTaxoFacets 13.54 (14.6%) 10.89
(2.9%) -19.6% ( -32% - -2%) 0.000
BrowseDateTaxoFacets 13.50 (8.8%) 11.11
(3.7%) -17.7% ( -27% - -5%) 0.000
BrowseRandomLabelTaxoFacets 11.78 (7.0%)9.94
(5.1%) -15.6% ( -25% - -3%) 0.000
MedTermDayTaxoFacets 47.49 (2.4%) 41.45
(3.4%) -12.7% ( -18% - -7%) 0.000
AndHighMedDayTaxoFacets 130.24 (2.7%) 119.48
(3.9%) -8.3% ( -14% - -1%) 0.000
AndHighHighDayTaxoFacets 28.80 (2.8%) 27.09
(3.1%) -5.9% ( -11% -0%) 0.000
OrHighMedDayTaxoFacets9.68 (2.7%)9.35
(2.8%) -3.4% ( -8% -2%) 0.000
HighTermDayOfYearSort 139.73 (9.6%) 135.74
(10.2%) -2.9% ( -20% - 18%) 0.361
TermDTSort 151.46 (9.0%) 147.40
(7.7%) -2.7% ( -17% - 15%) 0.311
Fuzzy2 35.22 (6.3%) 34.38
(5.9%) -2.4% ( -13% - 10%) 0.213
MedSloppyPhrase 78.99 (6.7%) 77.21
(7.1%) -2.3% ( -15% - 12%) 0.300
LowTerm 1636.38 (6.4%) 1600.26
(9.6%) -2.2% ( -17% - 14%) 0.392
LowPhrase 252.68 (3.8%) 247.11
(6.5%) -2.2% ( -12% -8%) 0.189
Respell 61.23 (2.3%) 59.89
(5.0%) -2.2% ( -9% -5%) 0.078
AndHighHigh 56.54 (2.6%) 55.43
(4.3%) -2.0% ( -8% -5%) 0.084
MedSpanNear 99.37 (2.4%) 97.44
(5.2%) -1.9% ( -9% -5%) 0.128
HighSloppyPhrase 28.58 (5.4%) 28.05
(5.4%) -1.8% ( -11% -9%) 0.280
PKLookup 198.95 (3.0%) 195.34
(4.8%) -1.8% ( -9% -6%) 0.148
AndHighMed 116.50 (3.3%) 114.65
(4.5%) -1.6% ( -9% -6%) 0.204
Fuzzy1 75.07 (6.4%) 73.99
(8.1%) -1.4% ( -14% - 13%) 0.532
HighSpanNear 10.73 (2.8%) 10.58
(3.9%) -1.4% ( -7% -5%) 0.180
LowSpanNear 43.92 (2.4%) 43.30
(3.4%) -1.4% ( -6% -4%) 0.128
LowSloppyPhrase 14.70 (4.4%) 14.50
(4.2%) -1.3% ( -9% -7%) 0.329
HighTermMonthSort 148.80 (8.3%) 146.84
(8.1%) -1.3% ( -16% - 16%) 0.612
OrHighMed 103.00 (3.2%) 101.67
(5.1%) -1.3% ( -9% -7%) 0.341
MedIntervalsOrdered5.44 (2.5%)5.37
(2.2%) -1.3% ( -5% -3%) 0.092
OrHighNotHigh 648.74 (6.7%) 640.81
(8.8%) -1.2% ( -15% - 15%) 0.621
MedPhrase 80.35 (2.7%) 79.38
(4.8%) -1.2% ( -8% -6%) 0.327
HighTerm 1384.91 (6.8%) 1369.27
(8.8%) -1.1% ( -15% - 15%) 0.650
[jira] [Comment Edited] (LUCENE-10334) Introduce a BlockReader based on ForUtil and use it for NumericDocValues
[
https://issues.apache.org/jira/browse/LUCENE-10334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464619#comment-17464619
]
Feng Guo edited comment on LUCENE-10334 at 12/23/21, 2:37 PM:
--
I'm so sorry to tell that there is something wrong with my benchmark: The
localrun script is still using the facets described in luceneutil
[readme|https://github.com/mikemccand/luceneutil/blob/master/README.md], like
this:
{code:python}
facets = (('taxonomy:Date', 'Date'),('sortedset:Month',
'Month'),('sortedset:DayOfYear', 'DayOfYear'))
index = comp.newIndex('lucene_baseline', sourceData, facets=facets,
indexSort='dayOfYearNumericDV:long')
{code}
And i got the result mentioned above with this facets. But when i'm cloning a
new luceneutil and rerun the setup.py, it becomes:
{code:python}
index = comp.newIndex('lucene_baseline', sourceData,
facets = (('taxonomy:Date', 'Date'),
('taxonomy:Month', 'Month'),
('taxonomy:DayOfYear', 'DayOfYear'),
('sortedset:Month', 'Month'),
('sortedset:DayOfYear', 'DayOfYear'),
('taxonomy:RandomLabel', 'RandomLabel'),
('sortedset:RandomLabel', 'RandomLabel')))
{code}
And the result is totally different with this:
{code:java}
TaskQPS baseline StdDevQPS my_modified_version
StdDevPct diff p-value
BrowseDayOfYearTaxoFacets 13.65 (8.9%) 10.49
(2.6%) -23.2% ( -31% - -12%) 0.000
BrowseMonthTaxoFacets 13.54 (14.6%) 10.89
(2.9%) -19.6% ( -32% - -2%) 0.000
BrowseDateTaxoFacets 13.50 (8.8%) 11.11
(3.7%) -17.7% ( -27% - -5%) 0.000
BrowseRandomLabelTaxoFacets 11.78 (7.0%)9.94
(5.1%) -15.6% ( -25% - -3%) 0.000
MedTermDayTaxoFacets 47.49 (2.4%) 41.45
(3.4%) -12.7% ( -18% - -7%) 0.000
AndHighMedDayTaxoFacets 130.24 (2.7%) 119.48
(3.9%) -8.3% ( -14% - -1%) 0.000
AndHighHighDayTaxoFacets 28.80 (2.8%) 27.09
(3.1%) -5.9% ( -11% -0%) 0.000
OrHighMedDayTaxoFacets9.68 (2.7%)9.35
(2.8%) -3.4% ( -8% -2%) 0.000
HighTermDayOfYearSort 139.73 (9.6%) 135.74
(10.2%) -2.9% ( -20% - 18%) 0.361
TermDTSort 151.46 (9.0%) 147.40
(7.7%) -2.7% ( -17% - 15%) 0.311
Fuzzy2 35.22 (6.3%) 34.38
(5.9%) -2.4% ( -13% - 10%) 0.213
MedSloppyPhrase 78.99 (6.7%) 77.21
(7.1%) -2.3% ( -15% - 12%) 0.300
LowTerm 1636.38 (6.4%) 1600.26
(9.6%) -2.2% ( -17% - 14%) 0.392
LowPhrase 252.68 (3.8%) 247.11
(6.5%) -2.2% ( -12% -8%) 0.189
Respell 61.23 (2.3%) 59.89
(5.0%) -2.2% ( -9% -5%) 0.078
AndHighHigh 56.54 (2.6%) 55.43
(4.3%) -2.0% ( -8% -5%) 0.084
MedSpanNear 99.37 (2.4%) 97.44
(5.2%) -1.9% ( -9% -5%) 0.128
HighSloppyPhrase 28.58 (5.4%) 28.05
(5.4%) -1.8% ( -11% -9%) 0.280
PKLookup 198.95 (3.0%) 195.34
(4.8%) -1.8% ( -9% -6%) 0.148
AndHighMed 116.50 (3.3%) 114.65
(4.5%) -1.6% ( -9% -6%) 0.204
Fuzzy1 75.07 (6.4%) 73.99
(8.1%) -1.4% ( -14% - 13%) 0.532
HighSpanNear 10.73 (2.8%) 10.58
(3.9%) -1.4% ( -7% -5%) 0.180
LowSpanNear 43.92 (2.4%) 43.30
(3.4%) -1.4% ( -6% -4%) 0.128
LowSloppyPhrase 14.70 (4.4%) 14.50
(4.2%) -1.3% ( -9% -7%) 0.329
HighTermMonthSort 148.80 (8.3%) 146.84
(8.1%) -1.3% ( -16% - 16%) 0.612
OrHighMed 103.00 (3.2%) 101.67
(5.1%) -1.3% ( -9% -7%) 0.341
MedIntervalsOrdered5.44 (2.5%)5.37
(2.2%) -1.3% ( -5% -3%) 0.092
OrHighNotHigh 648.74 (6.7%) 640.81
(8.8%) -1.2% ( -15% - 15%) 0.621
MedPhrase 80.35 (2.7%) 79.38
(4.8%) -1.2% ( -8% -6%) 0.327
HighTerm 1384.91 (6.8%) 1369.27
(8.8%) -1.1% ( -15% - 15%) 0.650
[GitHub] [lucene] gf2121 edited a comment on pull request #557: LUCENE-10333: Speed up BinaryDocValues with a batch reading on LongValues
gf2121 edited a comment on pull request #557: URL: https://github.com/apache/lucene/pull/557#issuecomment-999516790 Hi @rmuir @jpountz , Thanks a lot for all talking about this! ~~I think we **probably** find out a better way there:~~ There is something wrong with this benchmark and i deleted it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] gf2121 edited a comment on pull request #557: LUCENE-10333: Speed up BinaryDocValues with a batch reading on LongValues
gf2121 edited a comment on pull request #557: URL: https://github.com/apache/lucene/pull/557#issuecomment-999516790 Hi @rmuir @jpountz , Thanks a lot for all talking about this! ~~I think we **probably** find out a better way there:~~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
dweiss commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000364229 bq. But If i revert his commit or otherwise introduce regression, tests keep passing :) That's my point. Yeah, I know. It's a vicious cycle though - I can't really progress refactoring the build to use modules if I am blocked by failures; I'm never sure if it's it's a regression I introduced or something else. I just filed an issue for this one because I knew it was a real problem. We don't have to merge it yet to main - I'll merge this from Uwe's branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-10334) Introduce a BlockReader based on ForUtil and use it for NumericDocValues
[
https://issues.apache.org/jira/browse/LUCENE-10334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464619#comment-17464619
]
Feng Guo edited comment on LUCENE-10334 at 12/23/21, 4:11 PM:
--
I'm so sorry to tell that there is something wrong with my benchmark: The
localrun script was still using the facets described in luceneutil
[readme|https://github.com/mikemccand/luceneutil/blob/master/README.md], like
this:
{code:python}
facets = (('taxonomy:Date', 'Date'),('sortedset:Month',
'Month'),('sortedset:DayOfYear', 'DayOfYear'))
index = comp.newIndex('lucene_baseline', sourceData, facets=facets,
indexSort='dayOfYearNumericDV:long')
{code}
And i got the result mentioned above with this facets.
But when i'm cloning a new luceneutil and rerun the setup.py, it becomes:
{code:python}
index = comp.newIndex('lucene_baseline', sourceData,
facets = (('taxonomy:Date', 'Date'),
('taxonomy:Month', 'Month'),
('taxonomy:DayOfYear', 'DayOfYear'),
('sortedset:Month', 'Month'),
('sortedset:DayOfYear', 'DayOfYear'),
('taxonomy:RandomLabel', 'RandomLabel'),
('sortedset:RandomLabel', 'RandomLabel')))
{code}
And the result is totally different with this:
{code:java}
TaskQPS baseline StdDevQPS my_modified_version
StdDevPct diff p-value
BrowseDayOfYearTaxoFacets 13.65 (8.9%) 10.49
(2.6%) -23.2% ( -31% - -12%) 0.000
BrowseMonthTaxoFacets 13.54 (14.6%) 10.89
(2.9%) -19.6% ( -32% - -2%) 0.000
BrowseDateTaxoFacets 13.50 (8.8%) 11.11
(3.7%) -17.7% ( -27% - -5%) 0.000
BrowseRandomLabelTaxoFacets 11.78 (7.0%)9.94
(5.1%) -15.6% ( -25% - -3%) 0.000
MedTermDayTaxoFacets 47.49 (2.4%) 41.45
(3.4%) -12.7% ( -18% - -7%) 0.000
AndHighMedDayTaxoFacets 130.24 (2.7%) 119.48
(3.9%) -8.3% ( -14% - -1%) 0.000
AndHighHighDayTaxoFacets 28.80 (2.8%) 27.09
(3.1%) -5.9% ( -11% -0%) 0.000
OrHighMedDayTaxoFacets9.68 (2.7%)9.35
(2.8%) -3.4% ( -8% -2%) 0.000
HighTermDayOfYearSort 139.73 (9.6%) 135.74
(10.2%) -2.9% ( -20% - 18%) 0.361
TermDTSort 151.46 (9.0%) 147.40
(7.7%) -2.7% ( -17% - 15%) 0.311
Fuzzy2 35.22 (6.3%) 34.38
(5.9%) -2.4% ( -13% - 10%) 0.213
MedSloppyPhrase 78.99 (6.7%) 77.21
(7.1%) -2.3% ( -15% - 12%) 0.300
LowTerm 1636.38 (6.4%) 1600.26
(9.6%) -2.2% ( -17% - 14%) 0.392
LowPhrase 252.68 (3.8%) 247.11
(6.5%) -2.2% ( -12% -8%) 0.189
Respell 61.23 (2.3%) 59.89
(5.0%) -2.2% ( -9% -5%) 0.078
AndHighHigh 56.54 (2.6%) 55.43
(4.3%) -2.0% ( -8% -5%) 0.084
MedSpanNear 99.37 (2.4%) 97.44
(5.2%) -1.9% ( -9% -5%) 0.128
HighSloppyPhrase 28.58 (5.4%) 28.05
(5.4%) -1.8% ( -11% -9%) 0.280
PKLookup 198.95 (3.0%) 195.34
(4.8%) -1.8% ( -9% -6%) 0.148
AndHighMed 116.50 (3.3%) 114.65
(4.5%) -1.6% ( -9% -6%) 0.204
Fuzzy1 75.07 (6.4%) 73.99
(8.1%) -1.4% ( -14% - 13%) 0.532
HighSpanNear 10.73 (2.8%) 10.58
(3.9%) -1.4% ( -7% -5%) 0.180
LowSpanNear 43.92 (2.4%) 43.30
(3.4%) -1.4% ( -6% -4%) 0.128
LowSloppyPhrase 14.70 (4.4%) 14.50
(4.2%) -1.3% ( -9% -7%) 0.329
HighTermMonthSort 148.80 (8.3%) 146.84
(8.1%) -1.3% ( -16% - 16%) 0.612
OrHighMed 103.00 (3.2%) 101.67
(5.1%) -1.3% ( -9% -7%) 0.341
MedIntervalsOrdered5.44 (2.5%)5.37
(2.2%) -1.3% ( -5% -3%) 0.092
OrHighNotHigh 648.74 (6.7%) 640.81
(8.8%) -1.2% ( -15% - 15%) 0.621
MedPhrase 80.35 (2.7%) 79.38
(4.8%) -1.2% ( -8% -6%) 0.327
HighTerm 1384.91 (6.8%) 1369.27
(8.8%) -1.1% ( -15% - 15%) 0.650
[GitHub] [lucene] uschindler commented on a change in pull request #565: LUCENE-10338: Scan for tests only by convention file name pattern
uschindler commented on a change in pull request #565:
URL: https://github.com/apache/lucene/pull/565#discussion_r774723713
##
File path:
lucene/queryparser/src/test/org/apache/lucene/queryparser/surround/query/BooleanQueryTestFacade.java
##
@@ -28,7 +28,7 @@
import org.apache.lucene.search.SimpleCollector;
import org.junit.Assert;
-public class TestBooleanQuery {
+public class BooleanQueryTestFacade {
Review comment:
Ah sorry, this one wasn't an inner class, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #565: LUCENE-10338: Scan for tests only by convention file name pattern
uschindler commented on pull request #565: URL: https://github.com/apache/lucene/pull/565#issuecomment-1000464970 > I've removed TestBase* exclusion - anything starting with Test* must in fact be a test. Those two classes I had to rename - I'm not sure how they work for you on panama branch, they should be failing... I think it is because I did not disable automatic detection, and only core is java 18 class files. So it filters and at same time Gradle filtered out the remaining ones that you renamed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
rmuir commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000482712 Well, it does seem this particular method signature "can't work", unless any code that uses this helper explicitly opens up their module to lucene-core... and that's too confusing. As far as what other stuff is broken by modules, that's my bigger concern: how can we test this? what else is broke? In this example it is literally just a couple words of javadocs that they slid into this method with jdk9 and now we have to deprecate APIs because of it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #567: LUCENE-10335: Deprecate IOUtils.getDecodingReader(Class, String) for removal, as broken with modules
dweiss commented on pull request #567: URL: https://github.com/apache/lucene/pull/567#issuecomment-1000484883 > how can we test this? This is a problem because the code and tests can effectively run in many combinations. I don't have a clear picture of how this can be all reasonably arranged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9650) Errorprone on master/gradle no longer works with JDK-16
[
https://issues.apache.org/jira/browse/LUCENE-9650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464782#comment-17464782
]
Mike Drob commented on LUCENE-9650:
---
We incidentally opened up some internal java modules for spotless plugin
already in SOLR-15602, I think it would be reasonable to revisit this and
enable error-prone compiler again.
> Errorprone on master/gradle no longer works with JDK-16
> ---
>
> Key: LUCENE-9650
> URL: https://issues.apache.org/jira/browse/LUCENE-9650
> Project: Lucene - Core
> Issue Type: Bug
> Components: general/build
>Affects Versions: 9.0
>Reporter: Uwe Schindler
>Assignee: Dawid Weiss
>Priority: Major
> Fix For: 9.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> JDK-16 no longer allows access to internal classes of any module by default.
> It looks like errorprone tries to access some internals from the Java
> compiler. This now fails with Exception. You have to fully open the module or
> pass `--illegal-access=allow`.
> We have 3 options:
> - install an update of errorprone
> - disable error-prone if we detect a runtimeJdk with version >=16
> - run javac as a separate forked task (i think we do already) and pass
> `--illegal-access=allow` or open the internal compile module.
> Error message:
> {noformat}
> > Task :solr:solr-ref-guide:compileJava FAILED
> Exception in thread "main" java.lang.IllegalAccessError: class
> com.google.errorprone.ErrorProneJavacPlugin (in unnamed module @0x887af79)
> cannot access class com.sun.tools.javac.api.BasicJavacTask (in module
> jdk.compiler) because module jdk.compiler does not export
> com.sun.tools.javac.api to unnamed module @0x887af79
> at
> com.google.errorprone.ErrorProneJavacPlugin.init(ErrorProneJavacPlugin.java:38)
> at
> jdk.compiler/com.sun.tools.javac.api.BasicJavacTask.initPlugin(BasicJavacTask.java:254)
> at
> jdk.compiler/com.sun.tools.javac.api.BasicJavacTask.initPlugins(BasicJavacTask.java:228)
> at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:292)
> at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:176)
> at jdk.compiler/com.sun.tools.javac.Main.compile(Main.java:64)
> at jdk.compiler/com.sun.tools.javac.Main.main(Main.java:50)
> {noformat}
> Last failed build:
> https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/29129/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-9650) Errorprone on master/gradle no longer works with JDK-16
[
https://issues.apache.org/jira/browse/LUCENE-9650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464782#comment-17464782
]
Mike Drob edited comment on LUCENE-9650 at 12/23/21, 8:25 PM:
--
We incidentally opened up some internal java modules for spotless plugin
already in LUCENE-9990, I think it would be reasonable to revisit this and
enable error-prone compiler again.
was (Author: mdrob):
We incidentally opened up some internal java modules for spotless plugin
already in SOLR-15602, I think it would be reasonable to revisit this and
enable error-prone compiler again.
> Errorprone on master/gradle no longer works with JDK-16
> ---
>
> Key: LUCENE-9650
> URL: https://issues.apache.org/jira/browse/LUCENE-9650
> Project: Lucene - Core
> Issue Type: Bug
> Components: general/build
>Affects Versions: 9.0
>Reporter: Uwe Schindler
>Assignee: Dawid Weiss
>Priority: Major
> Fix For: 9.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> JDK-16 no longer allows access to internal classes of any module by default.
> It looks like errorprone tries to access some internals from the Java
> compiler. This now fails with Exception. You have to fully open the module or
> pass `--illegal-access=allow`.
> We have 3 options:
> - install an update of errorprone
> - disable error-prone if we detect a runtimeJdk with version >=16
> - run javac as a separate forked task (i think we do already) and pass
> `--illegal-access=allow` or open the internal compile module.
> Error message:
> {noformat}
> > Task :solr:solr-ref-guide:compileJava FAILED
> Exception in thread "main" java.lang.IllegalAccessError: class
> com.google.errorprone.ErrorProneJavacPlugin (in unnamed module @0x887af79)
> cannot access class com.sun.tools.javac.api.BasicJavacTask (in module
> jdk.compiler) because module jdk.compiler does not export
> com.sun.tools.javac.api to unnamed module @0x887af79
> at
> com.google.errorprone.ErrorProneJavacPlugin.init(ErrorProneJavacPlugin.java:38)
> at
> jdk.compiler/com.sun.tools.javac.api.BasicJavacTask.initPlugin(BasicJavacTask.java:254)
> at
> jdk.compiler/com.sun.tools.javac.api.BasicJavacTask.initPlugins(BasicJavacTask.java:228)
> at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:292)
> at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:176)
> at jdk.compiler/com.sun.tools.javac.Main.compile(Main.java:64)
> at jdk.compiler/com.sun.tools.javac.Main.main(Main.java:50)
> {noformat}
> Last failed build:
> https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/29129/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-9650) Errorprone on master/gradle no longer works with JDK-16
[
https://issues.apache.org/jira/browse/LUCENE-9650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464782#comment-17464782
]
Mike Drob edited comment on LUCENE-9650 at 12/23/21, 8:27 PM:
--
We incidentally opened up some internal java modules for spotless plugin
already in LUCENE-10066, I think it would be reasonable to revisit this and
enable error-prone compiler again.
was (Author: mdrob):
We incidentally opened up some internal java modules for spotless plugin
already in LUCENE-9990, I think it would be reasonable to revisit this and
enable error-prone compiler again.
> Errorprone on master/gradle no longer works with JDK-16
> ---
>
> Key: LUCENE-9650
> URL: https://issues.apache.org/jira/browse/LUCENE-9650
> Project: Lucene - Core
> Issue Type: Bug
> Components: general/build
>Affects Versions: 9.0
>Reporter: Uwe Schindler
>Assignee: Dawid Weiss
>Priority: Major
> Fix For: 9.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> JDK-16 no longer allows access to internal classes of any module by default.
> It looks like errorprone tries to access some internals from the Java
> compiler. This now fails with Exception. You have to fully open the module or
> pass `--illegal-access=allow`.
> We have 3 options:
> - install an update of errorprone
> - disable error-prone if we detect a runtimeJdk with version >=16
> - run javac as a separate forked task (i think we do already) and pass
> `--illegal-access=allow` or open the internal compile module.
> Error message:
> {noformat}
> > Task :solr:solr-ref-guide:compileJava FAILED
> Exception in thread "main" java.lang.IllegalAccessError: class
> com.google.errorprone.ErrorProneJavacPlugin (in unnamed module @0x887af79)
> cannot access class com.sun.tools.javac.api.BasicJavacTask (in module
> jdk.compiler) because module jdk.compiler does not export
> com.sun.tools.javac.api to unnamed module @0x887af79
> at
> com.google.errorprone.ErrorProneJavacPlugin.init(ErrorProneJavacPlugin.java:38)
> at
> jdk.compiler/com.sun.tools.javac.api.BasicJavacTask.initPlugin(BasicJavacTask.java:254)
> at
> jdk.compiler/com.sun.tools.javac.api.BasicJavacTask.initPlugins(BasicJavacTask.java:228)
> at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:292)
> at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:176)
> at jdk.compiler/com.sun.tools.javac.Main.compile(Main.java:64)
> at jdk.compiler/com.sun.tools.javac.Main.main(Main.java:50)
> {noformat}
> Last failed build:
> https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/29129/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
