[jira] [Created] (LUCENE-10531) Mark testLukeCanBeLaunched @Slow test and make a dedicated Github CI workflow for it

2022-04-23 Thread Tomoko Uchida (Jira) 
Tomoko Uchida created LUCENE-10531:
--

 Summary: Mark testLukeCanBeLaunched @Slow test and make a 
dedicated Github CI workflow for it
 Key: LUCENE-10531
 URL: https://issues.apache.org/jira/browse/LUCENE-10531
 Project: Lucene - Core
  Issue Type: Task
  Components: general/test
Reporter: Tomoko Uchida


We are going to allow running the test on Xvfb (a virtual display that speaks X 
protocol), this tweak is available only on Linux in [LUCENE-10528].

I'm just guessing but it could confuse or bother also Mac and Windows users (we 
can't know what window manager developers are using); it may be better to make 
it opt-in by marking it as slow tests. 
Instead, I suppose we can enable a dedicated Github actions workflow for the 
distribution test that is kicked only when the related files are changed. 
Besides Linux, we could run it both on Mac and Windows which most users run the 
app on - it'd be slow, but if we limit the scope of the test I suppose it works 
functionally just fine (I run actions workflows on mac and windows elsewhere).



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Updated] (LUCENE-10531) Mark testLukeCanBeLaunched @Slow test and make a dedicated Github CI workflow for it

2022-04-23 Thread Tomoko Uchida (Jira) 


 [ 
https://issues.apache.org/jira/browse/LUCENE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tomoko Uchida updated LUCENE-10531:
---
Description: 
We are going to allow running the test on Xvfb (a virtual display that speaks X 
protocol), this tweak is available only on Linux in [LUCENE-10528].

I'm just guessing but it could confuse or bother also Mac and Windows users (we 
can't know what window manager developers are using); it may be better to make 
it opt-in by marking it as slow tests. 
Instead, I think we can enable a dedicated Github actions workflow for the 
distribution test that is triggered only when the related files are changed. 
Besides Linux, we could run it both on Mac and Windows which most users run the 
app on - it'd be slow, but if we limit the scope of the test I suppose it works 
functionally just fine (I'm running actions workflows on mac and windows 
elsewhere).

To make it "slow test", we could add the same {{@Slow}} annotation as the 
{{test-framework}} to the distribution tests, for consistency.

  was:
We are going to allow running the test on Xvfb (a virtual display that speaks X 
protocol), this tweak is available only on Linux in [LUCENE-10528].

I'm just guessing but it could confuse or bother also Mac and Windows users (we 
can't know what window manager developers are using); it may be better to make 
it opt-in by marking it as slow tests. 
Instead, I suppose we can enable a dedicated Github actions workflow for the 
distribution test that is kicked only when the related files are changed. 
Besides Linux, we could run it both on Mac and Windows which most users run the 
app on - it'd be slow, but if we limit the scope of the test I suppose it works 
functionally just fine (I run actions workflows on mac and windows elsewhere).


> Mark testLukeCanBeLaunched @Slow test and make a dedicated Github CI workflow 
> for it
> 
>
> Key: LUCENE-10531
> URL: https://issues.apache.org/jira/browse/LUCENE-10531
> Project: Lucene - Core
>  Issue Type: Task
>  Components: general/test
>Reporter: Tomoko Uchida
>Priority: Minor
>
> We are going to allow running the test on Xvfb (a virtual display that speaks 
> X protocol), this tweak is available only on Linux in [LUCENE-10528].
> I'm just guessing but it could confuse or bother also Mac and Windows users 
> (we can't know what window manager developers are using); it may be better to 
> make it opt-in by marking it as slow tests. 
> Instead, I think we can enable a dedicated Github actions workflow for the 
> distribution test that is triggered only when the related files are changed. 
> Besides Linux, we could run it both on Mac and Windows which most users run 
> the app on - it'd be slow, but if we limit the scope of the test I suppose it 
> works functionally just fine (I'm running actions workflows on mac and 
> windows elsewhere).
> To make it "slow test", we could add the same {{@Slow}} annotation as the 
> {{test-framework}} to the distribution tests, for consistency.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Commented] (LUCENE-10531) Mark testLukeCanBeLaunched @Slow test and make a dedicated Github CI workflow for it

2022-04-23 Thread Tomoko Uchida (Jira) 


[ 
https://issues.apache.org/jira/browse/LUCENE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526796#comment-17526796
 ] 

Tomoko Uchida commented on LUCENE-10531:


I have little time for this right now, I'll take a look sometime soon.

> Mark testLukeCanBeLaunched @Slow test and make a dedicated Github CI workflow 
> for it
> 
>
> Key: LUCENE-10531
> URL: https://issues.apache.org/jira/browse/LUCENE-10531
> Project: Lucene - Core
>  Issue Type: Task
>  Components: general/test
>Reporter: Tomoko Uchida
>Priority: Minor
>
> We are going to allow running the test on Xvfb (a virtual display that speaks 
> X protocol), this tweak is available only on Linux in [LUCENE-10528].
> I'm just guessing but it could confuse or bother also Mac and Windows users 
> (we can't know what window manager developers are using); it may be better to 
> make it opt-in by marking it as slow tests. 
> Instead, I think we can enable a dedicated Github actions workflow for the 
> distribution test that is triggered only when the related files are changed. 
> Besides Linux, we could run it both on Mac and Windows which most users run 
> the app on - it'd be slow, but if we limit the scope of the test I suppose it 
> works functionally just fine (I'm running actions workflows on mac and 
> windows elsewhere).
> To make it "slow test", we could add the same {{@Slow}} annotation as the 
> {{test-framework}} to the distribution tests, for consistency.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Commented] (LUCENE-10528) TestScripts.testLukeCanBeLaunched creates X Window when running the tests

2022-04-23 Thread Tomoko Uchida (Jira) 


[ 
https://issues.apache.org/jira/browse/LUCENE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526799#comment-17526799
 ] 

Tomoko Uchida commented on LUCENE-10528:


I opened [LUCENE-10531] to describe what is in my mind. I still don't have a 
patch and have little time right now but I'll start to work on it sometime soon.

> TestScripts.testLukeCanBeLaunched creates X Window when running the tests
> -
>
> Key: LUCENE-10528
> URL: https://issues.apache.org/jira/browse/LUCENE-10528
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>Priority: Major
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> When running the tests, this one causes my entire desktop to "flicker" when 
> it creates some kind of X-Window very quickly and then destroys it. I use 
> tiling window manager, so whole desktop gets rearranged for a split second, 
> and I'd rather it not happen :)
> I first tried adding -Djava.awt.headless=true to both org.gradle.jvmargs and 
> tests.jvmargs in my .gradle/gradle.properties. doesn't work, as the test 
> doesnt use these when launching luke.
> I next tried hacking the test by adding this to the ProcessBuilderThingy, but 
> it didn't help either:
> {noformat}
> .envvar("LAUNCH_OPTS", "-Djava.awt.headless=true")
> {noformat}
> One way I can work around it, is to unset {{DISPLAY}} env var so that it 
> won't create this window. test still passes:
> {noformat}
> $ unset DISPLAY
> $ ./gradlew :lucene:distribution.tests:test
> ... (no window gets created)
> {noformat}
> So maybe as a workaround, we can just not pass DISPLAY environment variable 
> through to this test?



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Updated] (LUCENE-10531) Mark testLukeCanBeLaunched @Slow test and make a dedicated Github CI workflow for it

2022-04-23 Thread Tomoko Uchida (Jira) 


 [ 
https://issues.apache.org/jira/browse/LUCENE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tomoko Uchida updated LUCENE-10531:
---
Description: 
We are going to allow running the test on Xvfb (a virtual display that speaks X 
protocol) in [LUCENE-10528], this tweak is available only on Linux.

I'm just guessing but it could confuse or bother also Mac and Windows users (we 
can't know what window manager developers are using); it may be better to make 
it opt-in by marking it as slow tests. 
Instead, I think we can enable a dedicated Github actions workflow for the 
distribution test that is triggered only when the related files are changed. 
Besides Linux, we could run it both on Mac and Windows which most users run the 
app on - it'd be slow, but if we limit the scope of the test I suppose it works 
functionally just fine (I'm running actions workflows on mac and windows 
elsewhere).

To make it "slow test", we could add the same {{@Slow}} annotation as the 
{{test-framework}} to the distribution tests, for consistency.

  was:
We are going to allow running the test on Xvfb (a virtual display that speaks X 
protocol), this tweak is available only on Linux in [LUCENE-10528].

I'm just guessing but it could confuse or bother also Mac and Windows users (we 
can't know what window manager developers are using); it may be better to make 
it opt-in by marking it as slow tests. 
Instead, I think we can enable a dedicated Github actions workflow for the 
distribution test that is triggered only when the related files are changed. 
Besides Linux, we could run it both on Mac and Windows which most users run the 
app on - it'd be slow, but if we limit the scope of the test I suppose it works 
functionally just fine (I'm running actions workflows on mac and windows 
elsewhere).

To make it "slow test", we could add the same {{@Slow}} annotation as the 
{{test-framework}} to the distribution tests, for consistency.


> Mark testLukeCanBeLaunched @Slow test and make a dedicated Github CI workflow 
> for it
> 
>
> Key: LUCENE-10531
> URL: https://issues.apache.org/jira/browse/LUCENE-10531
> Project: Lucene - Core
>  Issue Type: Task
>  Components: general/test
>Reporter: Tomoko Uchida
>Priority: Minor
>
> We are going to allow running the test on Xvfb (a virtual display that speaks 
> X protocol) in [LUCENE-10528], this tweak is available only on Linux.
> I'm just guessing but it could confuse or bother also Mac and Windows users 
> (we can't know what window manager developers are using); it may be better to 
> make it opt-in by marking it as slow tests. 
> Instead, I think we can enable a dedicated Github actions workflow for the 
> distribution test that is triggered only when the related files are changed. 
> Besides Linux, we could run it both on Mac and Windows which most users run 
> the app on - it'd be slow, but if we limit the scope of the test I suppose it 
> works functionally just fine (I'm running actions workflows on mac and 
> windows elsewhere).
> To make it "slow test", we could add the same {{@Slow}} annotation as the 
> {{test-framework}} to the distribution tests, for consistency.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[GitHub] [lucene] LuXugang commented on pull request #792: LUCENE-10502: Use IndexedDISI to store docIds and DirectMonotonicWriter/Reader to handle ordToDoc

2022-04-23 Thread GitBox 


LuXugang commented on PR #792:
URL: https://github.com/apache/lucene/pull/792#issuecomment-1107447562

   vector source:
   - 3 dimensions
   - 667179 vectors within 1M documents in one segment
   - do `KnnVectorQuery`
   
   
   
   | NumberOfDocumentsToFind | baseline（search）ms | candidate（search）ms |
   | :-: | :--: | :---: |
   |   10|  58  |  60   |
   |  1000   |  63  |  67   |
   |  1  |  86  |  94   |
   |  2  | 113  |  118  |
   |  5  | 181  |  190  |
   
   | FORMAT | baseline（indexSize） | candidate（indexSize） |
   | :: | :---: | :: |
   |  vec   | 7.6M  |  7.8M  |
   |  vem   | 2.7M  |  174K  |
   |  vex   |  46M  |  46M   |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[GitHub] [lucene] LuXugang commented on a diff in pull request #792: LUCENE-10502: Use IndexedDISI to store docIds and DirectMonotonicWriter/Reader to handle ordToDoc

2022-04-23 Thread GitBox 


LuXugang commented on code in PR #792:
URL: https://github.com/apache/lucene/pull/792#discussion_r856877670


##
lucene/core/src/java/org/apache/lucene/codecs/lucene91/Lucene91HnswVectorsWriter.java:
##
@@ -207,15 +210,41 @@ private void writeMeta(
 // write docIDs
 int count = docsWithField.cardinality();
 meta.writeInt(count);
-if (count == maxDoc) {
-  meta.writeByte((byte) -1); // dense marker, each document has a vector 
value
+if (count == 0) {
+  meta.writeLong(-2); // docsWithFieldOffset
+  meta.writeLong(0L); // docsWithFieldLength
+  meta.writeShort((short) -1); // jumpTableEntryCount
+  meta.writeByte((byte) -1); // denseRankPower
+} else if (count == maxDoc) {
+  meta.writeLong(-1); // docsWithFieldOffset
+  meta.writeLong(0L); // docsWithFieldLength
+  meta.writeShort((short) -1); // jumpTableEntryCount
+  meta.writeByte((byte) -1); // denseRankPower
 } else {
-  meta.writeByte((byte) 0); // sparse marker, some documents don't have 
vector values
-  DocIdSetIterator iter = docsWithField.iterator();
-  for (int doc = iter.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc 
= iter.nextDoc()) {
-meta.writeInt(doc);
-  }
+  long offset = vectorData.getFilePointer();
+  meta.writeLong(offset); // docsWithFieldOffset
+  final short jumpTableEntryCount =
+  IndexedDISI.writeBitSet(
+  docsWithField.iterator(), vectorData, 
IndexedDISI.DEFAULT_DENSE_RANK_POWER);
+  meta.writeLong(vectorData.getFilePointer() - offset); // 
docsWithFieldLength
+  meta.writeShort(jumpTableEntryCount);
+  meta.writeByte(IndexedDISI.DEFAULT_DENSE_RANK_POWER);
+}
+
+// write ordToDoc mapping

Review Comment:
   Thanks @mayya-sharipova ,  it is indeed that there's no need to store 
ordToMap in dense case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[GitHub] [lucene] gf2121 merged pull request #797: LUCENE-10315: Speed up DocIdsWriter by ForUtil

2022-04-23 Thread GitBox 


gf2121 merged PR #797:
URL: https://github.com/apache/lucene/pull/797


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Commented] (LUCENE-10315) Speed up BKD leaf block ids codec by a 512 ints ForUtil

2022-04-23 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/LUCENE-10315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526829#comment-17526829
 ] 

ASF subversion and git services commented on LUCENE-10315:
--

Commit 35ca2d79f73c6dfaf5e648fe241f7e0b37084a90 in lucene's branch 
refs/heads/main from gf2121
[ https://gitbox.apache.org/repos/asf?p=lucene.git;h=35ca2d79f73 ]

LUCENE-10315: Speed up DocIdsWriter by ForUtil (#797)



> Speed up BKD leaf block ids codec by a 512 ints ForUtil
> ---
>
> Key: LUCENE-10315
> URL: https://issues.apache.org/jira/browse/LUCENE-10315
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Feng Guo
>Assignee: Feng Guo
>Priority: Major
> Attachments: addall.svg, cpu_profile_baseline.html, 
> cpu_profile_path.html
>
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> Elasticsearch (which based on lucene) can automatically infers types for 
> users with its dynamic mapping feature. When users index some low cardinality 
> fields, such as gender / age / status... they often use some numbers to 
> represent the values, while ES will infer these fields as {{{}long{}}}, and 
> ES uses BKD as the index of {{long}} fields. When the data volume grows, 
> building the result set of low-cardinality fields will make the CPU usage and 
> load very high.
> This is a flame graph we obtained from the production environment:
> [^addall.svg]
> It can be seen that almost all CPU is used in addAll. When we reindex 
> {{long}} to {{{}keyword{}}}, the cluster load and search latency are greatly 
> reduced ( We spent weeks of time to reindex all indices... ). I know that ES 
> recommended to use {{keyword}} for term/terms query and {{long}} for range 
> query in the document, but there are always some users who didn't realize 
> this and keep their habit of using sql database, or dynamic mapping 
> automatically selects the type for them. All in all, users won't realize that 
> there would be such a big difference in performance between {{long}} and 
> {{keyword}} fields in low cardinality fields. So from my point of view it 
> will make sense if we can make BKD works better for the low/medium 
> cardinality fields.
> As far as i can see, for low cardinality fields, there are two advantages of 
> {{keyword}} over {{{}long{}}}:
> 1. {{ForUtil}} used in {{keyword}} postings is much more efficient than BKD's 
> delta VInt, because its batch reading (readLongs) and SIMD decode.
> 2. When the query term count is less than 16, {{TermsInSetQuery}} can lazily 
> materialize of its result set, and when another small result clause 
> intersects with this low cardinality condition, the low cardinality field can 
> avoid reading all docIds into memory.
> This ISSUE is targeting to solve the first point. The basic idea is trying to 
> use a 512 ints {{ForUtil}} for BKD ids codec. I benchmarked this optimization 
> by mocking some random {{LongPoint}} and querying them with 
> {{PointInSetQuery}}.
> *Benchmark Result*
> |doc count|field cardinality|query point|baseline QPS|candidate QPS|diff 
> percentage|
> |1|32|1|51.44|148.26|188.22%|
> |1|32|2|26.8|101.88|280.15%|
> |1|32|4|14.04|53.52|281.20%|
> |1|32|8|7.04|28.54|305.40%|
> |1|32|16|3.54|14.61|312.71%|
> |1|128|1|110.56|350.26|216.81%|
> |1|128|8|16.6|89.81|441.02%|
> |1|128|16|8.45|48.07|468.88%|
> |1|128|32|4.2|25.35|503.57%|
> |1|128|64|2.13|13.02|511.27%|
> |1|1024|1|536.19|843.88|57.38%|
> |1|1024|8|109.71|251.89|129.60%|
> |1|1024|32|33.24|104.11|213.21%|
> |1|1024|128|8.87|30.47|243.52%|
> |1|1024|512|2.24|8.3|270.54%|
> |1|8192|1|.33|5000|50.00%|
> |1|8192|32|139.47|214.59|53.86%|
> |1|8192|128|54.59|109.23|100.09%|
> |1|8192|512|15.61|36.15|131.58%|
> |1|8192|2048|4.11|11.14|171.05%|
> |1|1048576|1|2597.4|3030.3|16.67%|
> |1|1048576|32|314.96|371.75|18.03%|
> |1|1048576|128|99.7|116.28|16.63%|
> |1|1048576|512|30.5|37.15|21.80%|
> |1|1048576|2048|10.38|12.3|18.50%|
> |1|8388608|1|2564.1|3174.6|23.81%|
> |1|8388608|32|196.27|238.95|21.75%|
> |1|8388608|128|55.36|68.03|22.89%|
> |1|8388608|512|15.58|19.24|23.49%|
> |1|8388608|2048|4.56|5.71|25.22%|
> The indices size is reduced for low cardinality fields and flat for high 
> cardinality fields.
> {code:java}
> 113Mindex_1_doc_32_cardinality_baseline
> 114Mindex_1_doc_32_cardinality_candidate
> 140Mindex_1_doc_128_cardinality_baseline
> 133Mindex_1_doc_128_cardinality_candidate
> 193Mindex_1_doc_1024_cardinality_baseline
> 174Mindex_1_doc_1024

[GitHub] [lucene] gf2121 opened a new pull request, #831: LUCENE-10315: Speed up DocIdsWriter by ForUtil (#797)

2022-04-23 Thread GitBox 


gf2121 opened a new pull request, #831:
URL: https://github.com/apache/lucene/pull/831

   I'll merge if nightly benchmark works well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Commented] (LUCENE-10528) TestScripts.testLukeCanBeLaunched creates X Window when running the tests

2022-04-23 Thread Robert Muir (Jira) 


[ 
https://issues.apache.org/jira/browse/LUCENE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526842#comment-17526842
 ] 

Robert Muir commented on LUCENE-10528:
--

I'm against marking a test slow when it isn't slow. also, i run all the slow 
tests, i ignore the annotation and i am against the concept of it. we should 
keep our tests fast and just bail out like solr.

please, use a correct annotation. if the test isn't slow then don't mark it slow


> TestScripts.testLukeCanBeLaunched creates X Window when running the tests
> -
>
> Key: LUCENE-10528
> URL: https://issues.apache.org/jira/browse/LUCENE-10528
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>Priority: Major
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> When running the tests, this one causes my entire desktop to "flicker" when 
> it creates some kind of X-Window very quickly and then destroys it. I use 
> tiling window manager, so whole desktop gets rearranged for a split second, 
> and I'd rather it not happen :)
> I first tried adding -Djava.awt.headless=true to both org.gradle.jvmargs and 
> tests.jvmargs in my .gradle/gradle.properties. doesn't work, as the test 
> doesnt use these when launching luke.
> I next tried hacking the test by adding this to the ProcessBuilderThingy, but 
> it didn't help either:
> {noformat}
> .envvar("LAUNCH_OPTS", "-Djava.awt.headless=true")
> {noformat}
> One way I can work around it, is to unset {{DISPLAY}} env var so that it 
> won't create this window. test still passes:
> {noformat}
> $ unset DISPLAY
> $ ./gradlew :lucene:distribution.tests:test
> ... (no window gets created)
> {noformat}
> So maybe as a workaround, we can just not pass DISPLAY environment variable 
> through to this test?



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-10528) TestScripts.testLukeCanBeLaunched creates X Window when running the tests

2022-04-23 Thread Robert Muir (Jira) 


[ 
https://issues.apache.org/jira/browse/LUCENE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526842#comment-17526842
 ] 

Robert Muir edited comment on LUCENE-10528 at 4/23/22 11:57 AM:


I'm against marking a test slow when it isn't slow. also, i run all the slow 
tests, i ignore the annotation and i am against the concept of it. we should 
keep our tests fast and not just bail out like solr.

please, use a correct annotation. if the test isn't slow then don't mark it slow



was (Author: rcmuir):
I'm against marking a test slow when it isn't slow. also, i run all the slow 
tests, i ignore the annotation and i am against the concept of it. we should 
keep our tests fast and just bail out like solr.

please, use a correct annotation. if the test isn't slow then don't mark it slow


> TestScripts.testLukeCanBeLaunched creates X Window when running the tests
> -
>
> Key: LUCENE-10528
> URL: https://issues.apache.org/jira/browse/LUCENE-10528
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>Priority: Major
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> When running the tests, this one causes my entire desktop to "flicker" when 
> it creates some kind of X-Window very quickly and then destroys it. I use 
> tiling window manager, so whole desktop gets rearranged for a split second, 
> and I'd rather it not happen :)
> I first tried adding -Djava.awt.headless=true to both org.gradle.jvmargs and 
> tests.jvmargs in my .gradle/gradle.properties. doesn't work, as the test 
> doesnt use these when launching luke.
> I next tried hacking the test by adding this to the ProcessBuilderThingy, but 
> it didn't help either:
> {noformat}
> .envvar("LAUNCH_OPTS", "-Djava.awt.headless=true")
> {noformat}
> One way I can work around it, is to unset {{DISPLAY}} env var so that it 
> won't create this window. test still passes:
> {noformat}
> $ unset DISPLAY
> $ ./gradlew :lucene:distribution.tests:test
> ... (no window gets created)
> {noformat}
> So maybe as a workaround, we can just not pass DISPLAY environment variable 
> through to this test?



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[GitHub] [lucene] rmuir commented on pull request #828: LUCENE-10528: use Xvfb in test to avoid messing up user's desktop

2022-04-23 Thread GitBox 


rmuir commented on PR #828:
URL: https://github.com/apache/lucene/pull/828#issuecomment-1107459835

   I'm against making the test slow. i'm gonna merge this PR to separate this 
from such craziness.
   
   i just want the tests not messing around with my desktop.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[GitHub] [lucene] rmuir merged pull request #828: LUCENE-10528: use Xvfb in test to avoid messing up user's desktop

2022-04-23 Thread GitBox 


rmuir merged PR #828:
URL: https://github.com/apache/lucene/pull/828


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Commented] (LUCENE-10528) TestScripts.testLukeCanBeLaunched creates X Window when running the tests

2022-04-23 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/LUCENE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526843#comment-17526843
 ] 

ASF subversion and git services commented on LUCENE-10528:
--

Commit 1089b482fc9d4b4202ca27e096723d0060446b51 in lucene's branch 
refs/heads/main from Robert Muir
[ https://gitbox.apache.org/repos/asf?p=lucene.git;h=1089b482fc9 ]

LUCENE-10528: use Xvfb in test to avoid messing up user's desktop (#828)

Co-authored-by: Tomoko Uchida 

> TestScripts.testLukeCanBeLaunched creates X Window when running the tests
> -
>
> Key: LUCENE-10528
> URL: https://issues.apache.org/jira/browse/LUCENE-10528
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>Priority: Major
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> When running the tests, this one causes my entire desktop to "flicker" when 
> it creates some kind of X-Window very quickly and then destroys it. I use 
> tiling window manager, so whole desktop gets rearranged for a split second, 
> and I'd rather it not happen :)
> I first tried adding -Djava.awt.headless=true to both org.gradle.jvmargs and 
> tests.jvmargs in my .gradle/gradle.properties. doesn't work, as the test 
> doesnt use these when launching luke.
> I next tried hacking the test by adding this to the ProcessBuilderThingy, but 
> it didn't help either:
> {noformat}
> .envvar("LAUNCH_OPTS", "-Djava.awt.headless=true")
> {noformat}
> One way I can work around it, is to unset {{DISPLAY}} env var so that it 
> won't create this window. test still passes:
> {noformat}
> $ unset DISPLAY
> $ ./gradlew :lucene:distribution.tests:test
> ... (no window gets created)
> {noformat}
> So maybe as a workaround, we can just not pass DISPLAY environment variable 
> through to this test?



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Commented] (LUCENE-10528) TestScripts.testLukeCanBeLaunched creates X Window when running the tests

2022-04-23 Thread Robert Muir (Jira) 


[ 
https://issues.apache.org/jira/browse/LUCENE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526844#comment-17526844
 ] 

Robert Muir commented on LUCENE-10528:
--

I merged this PR so that i can cleanly -1 any @Slow usage in our codebase. we 
don't need such garbage.

> TestScripts.testLukeCanBeLaunched creates X Window when running the tests
> -
>
> Key: LUCENE-10528
> URL: https://issues.apache.org/jira/browse/LUCENE-10528
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>Priority: Major
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> When running the tests, this one causes my entire desktop to "flicker" when 
> it creates some kind of X-Window very quickly and then destroys it. I use 
> tiling window manager, so whole desktop gets rearranged for a split second, 
> and I'd rather it not happen :)
> I first tried adding -Djava.awt.headless=true to both org.gradle.jvmargs and 
> tests.jvmargs in my .gradle/gradle.properties. doesn't work, as the test 
> doesnt use these when launching luke.
> I next tried hacking the test by adding this to the ProcessBuilderThingy, but 
> it didn't help either:
> {noformat}
> .envvar("LAUNCH_OPTS", "-Djava.awt.headless=true")
> {noformat}
> One way I can work around it, is to unset {{DISPLAY}} env var so that it 
> won't create this window. test still passes:
> {noformat}
> $ unset DISPLAY
> $ ./gradlew :lucene:distribution.tests:test
> ... (no window gets created)
> {noformat}
> So maybe as a workaround, we can just not pass DISPLAY environment variable 
> through to this test?



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-10528) TestScripts.testLukeCanBeLaunched creates X Window when running the tests

2022-04-23 Thread Robert Muir (Jira) 


 [ 
https://issues.apache.org/jira/browse/LUCENE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir resolved LUCENE-10528.
--
Fix Version/s: 9.2
   Resolution: Fixed

> TestScripts.testLukeCanBeLaunched creates X Window when running the tests
> -
>
> Key: LUCENE-10528
> URL: https://issues.apache.org/jira/browse/LUCENE-10528
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>Priority: Major
> Fix For: 9.2
>
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> When running the tests, this one causes my entire desktop to "flicker" when 
> it creates some kind of X-Window very quickly and then destroys it. I use 
> tiling window manager, so whole desktop gets rearranged for a split second, 
> and I'd rather it not happen :)
> I first tried adding -Djava.awt.headless=true to both org.gradle.jvmargs and 
> tests.jvmargs in my .gradle/gradle.properties. doesn't work, as the test 
> doesnt use these when launching luke.
> I next tried hacking the test by adding this to the ProcessBuilderThingy, but 
> it didn't help either:
> {noformat}
> .envvar("LAUNCH_OPTS", "-Djava.awt.headless=true")
> {noformat}
> One way I can work around it, is to unset {{DISPLAY}} env var so that it 
> won't create this window. test still passes:
> {noformat}
> $ unset DISPLAY
> $ ./gradlew :lucene:distribution.tests:test
> ... (no window gets created)
> {noformat}
> So maybe as a workaround, we can just not pass DISPLAY environment variable 
> through to this test?



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Commented] (LUCENE-10528) TestScripts.testLukeCanBeLaunched creates X Window when running the tests

2022-04-23 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/LUCENE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526845#comment-17526845
 ] 

ASF subversion and git services commented on LUCENE-10528:
--

Commit b80658ab05530ebda2a42eb05e88e6ba37414511 in lucene's branch 
refs/heads/branch_9x from Robert Muir
[ https://gitbox.apache.org/repos/asf?p=lucene.git;h=b80658ab055 ]

LUCENE-10528: use Xvfb in test to avoid messing up user's desktop (#828)

Co-authored-by: Tomoko Uchida 

> TestScripts.testLukeCanBeLaunched creates X Window when running the tests
> -
>
> Key: LUCENE-10528
> URL: https://issues.apache.org/jira/browse/LUCENE-10528
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>Priority: Major
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> When running the tests, this one causes my entire desktop to "flicker" when 
> it creates some kind of X-Window very quickly and then destroys it. I use 
> tiling window manager, so whole desktop gets rearranged for a split second, 
> and I'd rather it not happen :)
> I first tried adding -Djava.awt.headless=true to both org.gradle.jvmargs and 
> tests.jvmargs in my .gradle/gradle.properties. doesn't work, as the test 
> doesnt use these when launching luke.
> I next tried hacking the test by adding this to the ProcessBuilderThingy, but 
> it didn't help either:
> {noformat}
> .envvar("LAUNCH_OPTS", "-Djava.awt.headless=true")
> {noformat}
> One way I can work around it, is to unset {{DISPLAY}} env var so that it 
> won't create this window. test still passes:
> {noformat}
> $ unset DISPLAY
> $ ./gradlew :lucene:distribution.tests:test
> ... (no window gets created)
> {noformat}
> So maybe as a workaround, we can just not pass DISPLAY environment variable 
> through to this test?



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Created] (LUCENE-10532) Remove @Slow annotation

2022-04-23 Thread Robert Muir (Jira) 
Robert Muir created LUCENE-10532:


 Summary: Remove @Slow annotation
 Key: LUCENE-10532
 URL: https://issues.apache.org/jira/browse/LUCENE-10532
 Project: Lucene - Core
  Issue Type: Task
Reporter: Robert Muir


This annotation is useless, people have gotten so lazy about using it, that now 
there are proposals to mark tests that are not actually slow, with the @Slow 
annotation.

Let's remove the annotation. I can't imagine a situation where we mark a test 
@Slow and i don't veto it. we can keep tests clean.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Commented] (LUCENE-10528) TestScripts.testLukeCanBeLaunched creates X Window when running the tests

2022-04-23 Thread Robert Muir (Jira) 


[ 
https://issues.apache.org/jira/browse/LUCENE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526846#comment-17526846
 ] 

Robert Muir commented on LUCENE-10528:
--

I opened followup: LUCENE-10532

> TestScripts.testLukeCanBeLaunched creates X Window when running the tests
> -
>
> Key: LUCENE-10528
> URL: https://issues.apache.org/jira/browse/LUCENE-10528
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>Priority: Major
> Fix For: 9.2
>
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> When running the tests, this one causes my entire desktop to "flicker" when 
> it creates some kind of X-Window very quickly and then destroys it. I use 
> tiling window manager, so whole desktop gets rearranged for a split second, 
> and I'd rather it not happen :)
> I first tried adding -Djava.awt.headless=true to both org.gradle.jvmargs and 
> tests.jvmargs in my .gradle/gradle.properties. doesn't work, as the test 
> doesnt use these when launching luke.
> I next tried hacking the test by adding this to the ProcessBuilderThingy, but 
> it didn't help either:
> {noformat}
> .envvar("LAUNCH_OPTS", "-Djava.awt.headless=true")
> {noformat}
> One way I can work around it, is to unset {{DISPLAY}} env var so that it 
> won't create this window. test still passes:
> {noformat}
> $ unset DISPLAY
> $ ./gradlew :lucene:distribution.tests:test
> ... (no window gets created)
> {noformat}
> So maybe as a workaround, we can just not pass DISPLAY environment variable 
> through to this test?



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[GitHub] [lucene] rmuir opened a new pull request, #832: LUCENE-10523: remove @Slow annotation

2022-04-23 Thread GitBox 


rmuir opened a new pull request, #832:
URL: https://github.com/apache/lucene/pull/832

   This annotation is such an antipattern, it is so harmful to our codebase.
   * gives the impression that "slow tests are ok"
   * it is like "support for slow tests"
   * allows certain privileged ppl to skip slow tests (those that know the 
gradle magic)
   * probably same people are the ones writing slow tests, not having to live 
with consequences
   * causes differences across runs of different devs for no good reason
   
   we don't need this annotation. we can work together to keep the tests fast. 
if tests are slow we can all share the pain until we fix the test to be fast.
   
   this thing is a real antipattern, i think the idea came from solr, but well, 
look at their tests. let's not be like them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Commented] (LUCENE-10528) TestScripts.testLukeCanBeLaunched creates X Window when running the tests

2022-04-23 Thread Tomoko Uchida (Jira) 


[ 
https://issues.apache.org/jira/browse/LUCENE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526849#comment-17526849
 ] 

Tomoko Uchida commented on LUCENE-10528:


bq. I'm against marking a test slow when it isn't slow.

True - it's not slow, apologies for the misuse of the term. And the name of the 
annotation hid the essential point - I didn't want to confuse anyone's console 
regardless of OS or display manager he/she uses in daily development.

> TestScripts.testLukeCanBeLaunched creates X Window when running the tests
> -
>
> Key: LUCENE-10528
> URL: https://issues.apache.org/jira/browse/LUCENE-10528
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>Priority: Major
> Fix For: 9.2
>
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> When running the tests, this one causes my entire desktop to "flicker" when 
> it creates some kind of X-Window very quickly and then destroys it. I use 
> tiling window manager, so whole desktop gets rearranged for a split second, 
> and I'd rather it not happen :)
> I first tried adding -Djava.awt.headless=true to both org.gradle.jvmargs and 
> tests.jvmargs in my .gradle/gradle.properties. doesn't work, as the test 
> doesnt use these when launching luke.
> I next tried hacking the test by adding this to the ProcessBuilderThingy, but 
> it didn't help either:
> {noformat}
> .envvar("LAUNCH_OPTS", "-Djava.awt.headless=true")
> {noformat}
> One way I can work around it, is to unset {{DISPLAY}} env var so that it 
> won't create this window. test still passes:
> {noformat}
> $ unset DISPLAY
> $ ./gradlew :lucene:distribution.tests:test
> ... (no window gets created)
> {noformat}
> So maybe as a workaround, we can just not pass DISPLAY environment variable 
> through to this test?



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[GitHub] [lucene] xiaoshi2013 commented on pull request #816: LUCENE-10519: ThreadLocal.remove under G1GC takes 100% CPU

2022-04-23 Thread GitBox 


xiaoshi2013 commented on PR #816:
URL: https://github.com/apache/lucene/pull/816#issuecomment-1107515602

   Great


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Commented] (LUCENE-10532) Remove @Slow annotation

2022-04-23 Thread Tomoko Uchida (Jira) 


[ 
https://issues.apache.org/jira/browse/LUCENE-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526873#comment-17526873
 ] 

Tomoko Uchida commented on LUCENE-10532:


Hi, I'm not familiar with the origins of {{@Slow}} annotation and have not even 
used it, but perhaps some concrete figures might be useful for reference? I 
measured the test execution time (wall clock time) several times with 
combinations of "Ptests.slow" and "--max-workers" on a physical machine.
|| ||tests.slow=false||tests.slow=true||
|--max-workers=2|3m 58s|5m 17s|
|--max-workers=4|2m 31s|3m 7s|
|--max-workers=6|2m 25s|2m 22s|
|--max-workers=8|2m 6s|2m 8s|

4 to 8 workers would be reasonable choices on modern commodity machines to me 
(in my case, I'm using Fedora OS on Core i7-8700 with 6 cores and Gradle 
chooses 6 workers by default for me), then it looks there is no critical 
difference between "tests.slow=false" and "tests.slow=true" in terms of wall 
clock time, thanks to concurrent execution.

>From my viewpoint and use-case, +1 to remove {{@Slow}} and keep unit tests 
>sane.

> Remove @Slow annotation
> ---
>
> Key: LUCENE-10532
> URL: https://issues.apache.org/jira/browse/LUCENE-10532
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>Priority: Major
>
> This annotation is useless, people have gotten so lazy about using it, that 
> now there are proposals to mark tests that are not actually slow, with the 
> @Slow annotation.
> Let's remove the annotation. I can't imagine a situation where we mark a test 
> @Slow and i don't veto it. we can keep tests clean.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[GitHub] [lucene] boicehuang commented on pull request #816: LUCENE-10519: ThreadLocal.remove under G1GC takes 100% CPU

2022-04-23 Thread GitBox 


boicehuang commented on PR #816:
URL: https://github.com/apache/lucene/pull/816#issuecomment-1107689248

   > `./gradlew check` still fails. looks like the new test catches 
interruptedexception and doesn't use the exception. in our codebase 
compiler/linter will fail on this. please annotate the variable with 
`@SuppressWarnings("unused")`
   
   @rmuir  fixed. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[GitHub] [lucene] spike-liu commented on pull request #663: Lucene-10188: Give SortedSetDocValues a docValueCount()?

2022-04-23 Thread GitBox 


spike-liu commented on PR #663:
URL: https://github.com/apache/lucene/pull/663#issuecomment-1107703092

   > This change looks good to me, except why did we need to resurrect 
`DirectDocValuesFormat`? Was it an accident to include that in this PR maybe?
   
   Thanks for you review, Michael. "DirectDocValueFormat' was my mistake and 
removed from commits.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org