Re: [I] Concurrency bug `DocumentsWriterPerThreadPool.getAndLock()` uncovered by OpenJ9 test failures? [lucene]

2023-12-20 Thread via GitHub
singh264 commented on issue #12916: URL: https://github.com/apache/lucene/issues/12916#issuecomment-1865016211 > If any of you manages to reproduce, I'm interested in the command that you used I was able to reproduce the failure in `TestIndexWriterThreadsToSegments.testSegmentCountO

Re: [I] Concurrency bug `DocumentsWriterPerThreadPool.getAndLock()` uncovered by OpenJ9 test failures? [lucene]

2023-12-20 Thread via GitHub
uschindler commented on issue #12916: URL: https://github.com/apache/lucene/issues/12916#issuecomment-1865001473 I made test fail on my AMD Ryzen 3700 (the Policeman Jenkins Sever): ```sh $ ./gradlew -p lucene/core -Dtests.seed=F7B4CD7A5624D5EC beast --tests TestIndexWriterThreadsT

Re: [I] Concurrency bug `DocumentsWriterPerThreadPool.getAndLock()` uncovered by OpenJ9 test failures? [lucene]

2023-12-20 Thread via GitHub
uschindler commented on issue #12916: URL: https://github.com/apache/lucene/issues/12916#issuecomment-1864923856 Thanks. Maybe the OpenJ9 people can help how they reproduced. You can use `gradlew beast` instead of `gradlew test` to run the forked copies inside gradle. More looing in

Re: [I] Concurrency bug `DocumentsWriterPerThreadPool.getAndLock()` uncovered by OpenJ9 test failures? [lucene]

2023-12-20 Thread via GitHub
jpountz commented on issue #12916: URL: https://github.com/apache/lucene/issues/12916#issuecomment-1864917501 Thanks. I'm trying to reproduce failures locally with the following command, without luck so far with JDK 17 and JDK 21. I'll dig more tomorrow. If any of you manages to reproduce,

Re: [I] Concurrency bug `DocumentsWriterPerThreadPool.getAndLock()` uncovered by OpenJ9 test failures? [lucene]

2023-12-20 Thread via GitHub
uschindler commented on issue #12916: URL: https://github.com/apache/lucene/issues/12916#issuecomment-1864883276 See this comment: https://github.com/eclipse-openj9/openj9/issues/18400#issuecomment-1834577142 -- This is an automated message from the Apache Git Service. To respond to the m

Re: [I] Concurrency bug `DocumentsWriterPerThreadPool.getAndLock()` uncovered by OpenJ9 test failures? [lucene]

2023-12-20 Thread via GitHub
jpountz commented on issue #12916: URL: https://github.com/apache/lucene/issues/12916#issuecomment-1864882629 Thanks I had missed that, I'll look more into it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [I] Concurrency bug `DocumentsWriterPerThreadPool.getAndLock()` uncovered by OpenJ9 test failures? [lucene]

2023-12-20 Thread via GitHub
uschindler commented on issue #12916: URL: https://github.com/apache/lucene/issues/12916#issuecomment-1864881002 They made statistics in the linked issue. Hotspot also fails. So they rejected it as openj9 issue. -- This is an automated message from the Apache Git Service. To respond to th

Re: [I] Concurrency bug `DocumentsWriterPerThreadPool.getAndLock()` uncovered by OpenJ9 test failures? [lucene]

2023-12-20 Thread via GitHub
uschindler commented on issue #12916: URL: https://github.com/apache/lucene/issues/12916#issuecomment-1864879720 This is a real bug and not one of openj9. You can reproduce this bug with enough tries on hotspot, too. -- This is an automated message from the Apache Git Service. To respond

Re: [I] Concurrency bug `DocumentsWriterPerThreadPool.getAndLock()` uncovered by OpenJ9 test failures? [lucene]

2023-12-20 Thread via GitHub
jpountz commented on issue #12916: URL: https://github.com/apache/lucene/issues/12916#issuecomment-1864843912 Does someone understand if adding synchronization is fixing a real bug of it it just helps hide a J9 bug? This method is subject to contention and #12199 was about avoiding locking

Re: [PR] Remove remaining sources of contention on indexing. [lucene]

2023-12-20 Thread via GitHub
jpountz closed pull request #12205: Remove remaining sources of contention on indexing. URL: https://github.com/apache/lucene/pull/12205 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Remove remaining sources of contention on indexing. [lucene]

2023-12-20 Thread via GitHub
jpountz commented on PR #12205: URL: https://github.com/apache/lucene/pull/12205#issuecomment-1864768401 This test failure is sneaky, I extracted some bits from this PR into #12958. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Reduce frequencies buffer size when they are not needed [lucene]

2023-12-20 Thread via GitHub
jpountz commented on PR #12954: URL: https://github.com/apache/lucene/pull/12954#issuecomment-1864765767 > so maybe we can consider an other approach: try to avoid the for-loop in reset() if the instance can be reused +1 this sounds like a good idea! -- This is an automated message

Re: [PR] Reduce frequencies buffer size when they are not needed [lucene]

2023-12-20 Thread via GitHub
easyice commented on PR #12954: URL: https://github.com/apache/lucene/pull/12954#issuecomment-1864749851 I took several hours to confirm the results, the benchmark shows it became faster, this exceeded my expectation, we think the speedup is due to remove the loop that initializes the `

Re: [PR] Make FSTCompiler.compile() to only return the FSTMetadata [lucene]

2023-12-20 Thread via GitHub
dungba88 commented on code in PR #12831: URL: https://github.com/apache/lucene/pull/12831#discussion_r1421299753 ## lucene/analysis/common/src/java/org/apache/lucene/analysis/charfilter/NormalizeCharMap.java: ## @@ -111,7 +111,7 @@ public NormalizeCharMap build() { for

Re: [PR] Remove remaining sources of contention on indexing. [lucene]

2023-12-20 Thread via GitHub
jpountz commented on PR #12205: URL: https://github.com/apache/lucene/pull/12205#issuecomment-1864651149 The above failure is a bit scary, I'll try to split this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Replace usage of deprecated size() with length() in ByteBuffersDataInput [lucene]

2023-12-20 Thread via GitHub
dungba88 commented on PR #12948: URL: https://github.com/apache/lucene/pull/12948#issuecomment-1864650439 I think we can remove the `size()` method in 10.0? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Reproducible test failure with Terms#intersect on the default codec [lucene]

2023-12-20 Thread via GitHub
jpountz commented on issue #12957: URL: https://github.com/apache/lucene/issues/12957#issuecomment-1864480721 I just pushed the change, thanks @mikemccand for putting me on the right track. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] Reproducible test failure with Terms#intersect on the default codec [lucene]

2023-12-20 Thread via GitHub
jpountz closed issue #12957: Reproducible test failure with Terms#intersect on the default codec URL: https://github.com/apache/lucene/issues/12957 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [I] Reproducible test failure with Terms#intersect on the default codec [lucene]

2023-12-20 Thread via GitHub
mikemccand commented on issue #12957: URL: https://github.com/apache/lucene/issues/12957#issuecomment-1864450920 OK the `DirectPostingsFormat` failure is also happy with this fix. +1 to merge. Thanks @jpountz! -- This is an automated message from the Apache Git Service. To respond to th

Re: [I] Reproducible test failure with Terms#intersect on the default codec [lucene]

2023-12-20 Thread via GitHub
mikemccand commented on issue #12957: URL: https://github.com/apache/lucene/issues/12957#issuecomment-1864448812 > > Terms.intersect(Automaton a, BytesRef startTerm) requires that startTerm is accepted by the incoming automaton, yet the way CheckIndex is calling it can clearly violate that.

Re: [PR] Improve Javadoc for DocValuesConsumer [lucene]

2023-12-20 Thread via GitHub
jpountz merged PR #12952: URL: https://github.com/apache/lucene/pull/12952 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apa

Re: [I] Reproducible test failure with Terms#intersect on the default codec [lucene]

2023-12-20 Thread via GitHub
jpountz commented on issue #12957: URL: https://github.com/apache/lucene/issues/12957#issuecomment-1864407118 Oh I see, I created binary automata, but the API implicitly treats automata as UTF32 automata, so you need to tell it explicitly that it's a binary automaton. And something like tha

Re: [I] Reproducible test failure with Terms#intersect on the default codec [lucene]

2023-12-20 Thread via GitHub
jpountz commented on issue #12957: URL: https://github.com/apache/lucene/issues/12957#issuecomment-1864395031 > Terms.intersect(Automaton a, BytesRef startTerm) requires that startTerm is accepted by the incoming automaton, yet the way CheckIndex is calling it can clearly violate that.

Re: [I] Reproducible test failure with Terms#intersect on the default codec [lucene]

2023-12-20 Thread via GitHub
mikemccand commented on issue #12957: URL: https://github.com/apache/lucene/issues/12957#issuecomment-1864386831 I'll try to fix `CheckIndex` so that it only uses `startTerm` that is accepted by the automaton. -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] Reproducible test failure with Terms#intersect on the default codec [lucene]

2023-12-20 Thread via GitHub
mikemccand commented on issue #12957: URL: https://github.com/apache/lucene/issues/12957#issuecomment-1864385415 OK I think the issue here may be that `Terms.intersect(Automaton a, BytesRef startTerm)` requires that `startTerm` is accepted by the incoming automaton, yet the way `CheckIndex`

Re: [PR] An improved check for ignoring the c2-crash test if running on a client compiler. [lucene]

2023-12-20 Thread via GitHub
ChrisHegarty commented on PR #12953: URL: https://github.com/apache/lucene/pull/12953#issuecomment-1864374811 I'm way too slow here, sorry. Belated LGTM. And thanks for following up on this. -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[I] Reproducible test failure with Terms#intersect on the default codec [lucene]

2023-12-20 Thread via GitHub
jpountz opened a new issue, #12957: URL: https://github.com/apache/lucene/issues/12957 ### Description The new CheckIndex checks are causing some test failures with the default codec, which are reproducible and look like real bugs? I started looking but I'm not familiar enough with B