Re: [PR] Use growNoCopy for SortingStoredFieldsConsumer#NO_COMPRESSION [lucene]

2023-11-05 Thread via GitHub
gf2121 merged PR #12733: URL: https://github.com/apache/lucene/pull/12733 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apac

[PR] Enable executing using NFA in RegexpQuery [lucene]

2023-11-05 Thread via GitHub
zhaih opened a new pull request, #12767: URL: https://github.com/apache/lucene/pull/12767 ### Description As title, added a new flag in `RegexpQuery`'s ctor, not sure whether there's a better way, maybe using a static method instead? Like `RegexpQuery.createWithDFA`/`RegexpQu

Re: [PR] speedup arm int functions? [lucene]

2023-11-05 Thread via GitHub
rmuir commented on PR #12743: URL: https://github.com/apache/lucene/pull/12743#issuecomment-1794112273 When looking at `SDOT` to do this, i was able to accomplish it with another vector API, just as basis for comparison: https://godbolt.org/z/9cv5WaGaT Even if you lower `-marc

Re: [PR] Skip docs with Docvalues in NumericLeafComparator [lucene]

2023-11-05 Thread via GitHub
LuXugang commented on PR #12405: URL: https://github.com/apache/lucene/pull/12405#issuecomment-1793958700 Sure thing @jpountz , I would work on this in the next few days. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Remove usage of deprecated java.util.Locale constructor [lucene]

2023-11-05 Thread via GitHub
rmuir commented on PR #12761: URL: https://github.com/apache/lucene/pull/12761#issuecomment-1793871445 > Looks ok to me. > > Did you check all possible benchmark config/ALG files (not all of them are tested) that the locales in them are correctly for usage as language tag? ```

Re: [PR] disable vectors (and don't warn to add incubator module) for jvmci/graal [lucene]

2023-11-05 Thread via GitHub
uschindler commented on code in PR #12766: URL: https://github.com/apache/lucene/pull/12766#discussion_r1382647154 ## lucene/core/src/java/org/apache/lucene/util/Constants.java: ## @@ -66,6 +66,10 @@ private Constants() {} // can't construct /** True iff the Java VM is based

Re: [PR] disable vectors (and don't warn to add incubator module) for jvmci/graal [lucene]

2023-11-05 Thread via GitHub
rmuir commented on PR #12766: URL: https://github.com/apache/lucene/pull/12766#issuecomment-1793846009 I don't want to detect every possible option that could slow thing down, instead ordinary configurations. Look at all the stuff being "sold" on graalvm.org: lower resource usage, faster st

Re: [PR] disable vectors (and don't warn to add incubator module) for jvmci/graal [lucene]

2023-11-05 Thread via GitHub
uschindler commented on code in PR #12766: URL: https://github.com/apache/lucene/pull/12766#discussion_r1382646072 ## lucene/core/src/java/org/apache/lucene/util/Constants.java: ## @@ -66,6 +66,10 @@ private Constants() {} // can't construct /** True iff the Java VM is based

[PR] disable vectors (and don't warn to add incubator module) for jvmci/graal [lucene]

2023-11-05 Thread via GitHub
rmuir opened a new pull request, #12766: URL: https://github.com/apache/lucene/pull/12766 Another performance trap. I see use of this stuff a lot in the wild, lots of users/apps doing native image stuff, but we don't want to use vector api here, we should definitely not be encouraging the u

Re: [I] Reproducible failure in TestIndexWriter.testHasUncommittedChanges [lucene]

2023-11-05 Thread via GitHub
dweiss commented on issue #12763: URL: https://github.com/apache/lucene/issues/12763#issuecomment-1793810448 Here's another seed where it's failing (from a PR): ``` gradlew :lucene:core:test --tests "org.apache.lucene.index.TestIndexWriter.testHasUncommittedChanges" -Ptests.jvms=2 -Pt

Re: [PR] Limit vectorization API to Hotspot VMs (and rename some constants and fix Javadocs) [lucene]

2023-11-05 Thread via GitHub
uschindler merged PR #12765: URL: https://github.com/apache/lucene/pull/12765 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.

Re: [PR] Detect J9 and disable vectorization completely (it is not supported there) [lucene]

2023-11-05 Thread via GitHub
rmuir commented on PR #12764: URL: https://github.com/apache/lucene/pull/12764#issuecomment-1793808077 I like Uwe's PR better here, closing this one -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Detect J9 and disable vectorization completely (it is not supported there) [lucene]

2023-11-05 Thread via GitHub
rmuir closed pull request #12764: Detect J9 and disable vectorization completely (it is not supported there) URL: https://github.com/apache/lucene/pull/12764 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] Limit vectorization API to Hotspot VMs (and rename some constants and fix Javadocs) [lucene]

2023-11-05 Thread via GitHub
uschindler commented on code in PR #12765: URL: https://github.com/apache/lucene/pull/12765#discussion_r1382621462 ## lucene/core/src/java/org/apache/lucene/internal/vectorization/VectorizationProvider.java: ## @@ -111,6 +111,12 @@ static VectorizationProvider lookup(boolean tes

Re: [PR] TestIndexWriterOnVMError.testUnknownError times out (fixes potential IW deadlock on tragic exceptions). [lucene]

2023-11-05 Thread via GitHub
dweiss commented on code in PR #12751: URL: https://github.com/apache/lucene/pull/12751#discussion_r1382620003 ## lucene/core/src/java/org/apache/lucene/index/IndexWriter.java: ## @@ -2560,10 +2560,15 @@ private void rollbackInternalNoCommit() throws IOException {

Re: [PR] Detect J9 and disable vectorization completely (it is not supported there) [lucene]

2023-11-05 Thread via GitHub
uschindler commented on PR #12764: URL: https://github.com/apache/lucene/pull/12764#issuecomment-1793804533 I have another PR already developed: #12765 It does not use sysprops and detects hotspot by the algorithms we already use. Should I close this one? -- This is an automated m

Re: [PR] LUCENE-10195: Improve Gradle build speed [lucene]

2023-11-05 Thread via GitHub
dweiss commented on PR #414: URL: https://github.com/apache/lucene/pull/414#issuecomment-1793803122 > we are both named Dawid, we are both from Poznan, we both bike, we are both in our 40-ties. You are located at Bóżnicza Street (or at least your wife is), What are the odds, eh? Fai

[PR] Detect J9 and disable vectorization completely (it is not supported there) [lucene]

2023-11-05 Thread via GitHub
rmuir opened a new pull request, #12764: URL: https://github.com/apache/lucene/pull/12764 J9 VM doesn't seem to actually implement the vector api, so it falls back to hundreds-of-times-slower-pure-java impl. Currently: ``` VectorUtilBenchmark.floatCosineScalar 1024 thrpt 15

Re: [I] Use max BPV encoding in postings if doc buffer size less than ForUtil.BLOCK_SIZE [lucene]

2023-11-05 Thread via GitHub
easyice commented on issue #12717: URL: https://github.com/apache/lucene/issues/12717#issuecomment-1793763101 I reproduced this using low cardinality fields, for instance we let the posing size be 100, write 10 million docs then force merge to single segment, use `TermInSetQuery` with 512 t

Re: [PR] Specialize arc store for continuous label in FST [lucene]

2023-11-05 Thread via GitHub
easyice commented on code in PR #12748: URL: https://github.com/apache/lucene/pull/12748#discussion_r1382560887 ## lucene/core/src/java/org/apache/lucene/util/fst/FST.java: ## @@ -96,6 +96,13 @@ public enum INPUT_TYPE { */ static final byte ARCS_FOR_DIRECT_ADDRESSING = 1

[I] Reproducible failure in TestIndexWriter.testHasUncommittedChanges [lucene]

2023-11-05 Thread via GitHub
easyice opened a new issue, #12763: URL: https://github.com/apache/lucene/issues/12763 ### Description The failure looks related to PR: https://github.com/apache/lucene/pull/12549 ``` > java.lang.AssertionError > at __randomizedtesting.SeedInfo.seed([35A9341

Re: [PR] Refactor access to VM options and move some VM options to oal.util.Constants [lucene]

2023-11-05 Thread via GitHub
uschindler commented on PR #12754: URL: https://github.com/apache/lucene/pull/12754#issuecomment-1793691267 I fixed a regression with OpenJ9 in https://github.com/apache/lucene/commit/5358b7251ec264f78a8b3c5250f5082b4756f6ca. -- This is an automated message from the Apache Git Service. To

Re: [PR] Speed up vectorutil float scalar methods, unroll properly, use fma where possible [lucene]

2023-11-05 Thread via GitHub
uschindler commented on PR #12737: URL: https://github.com/apache/lucene/pull/12737#issuecomment-1793681356 Thanks for the hard benchmarking work! 🍻 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[PR] Hide the internal data structure of HeapPointWriter [lucene]

2023-11-05 Thread via GitHub
iverase opened a new pull request, #12762: URL: https://github.com/apache/lucene/pull/12762 HeapPointWriter uses a a byte array to hold points on heap. This array is access directly from BKD radix selector to sort points in place. In addition the HeapPointReader uses the array to read the p