[jira] [Commented] (LUCENE-9347) Update to forbiddenapis 3.0
[ https://issues.apache.org/jira/browse/LUCENE-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093137#comment-17093137 ] Dawid Weiss commented on LUCENE-9347: - This is a polishing/ refinement thing, Uwe. Much like moving some of the current code/ scripts to plugins or precompiled classes. My first and foremost priority would be to get all aspects of the build working so that they can be verified and functional, remove ant bits, then clean up and optimize the remaining gradle issues. > Update to forbiddenapis 3.0 > --- > > Key: LUCENE-9347 > URL: https://issues.apache.org/jira/browse/LUCENE-9347 > Project: Lucene - Core > Issue Type: Task > Components: general/build >Reporter: Uwe Schindler >Assignee: Uwe Schindler >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Forbiddenapis 3.0 was released a minute ago: > - Support for Java 14 > - Gradle Task Configuration Avoidance support > See: > https://github.com/policeman-tools/forbidden-apis/wiki/Changes#version-30-released-2020-04-27 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] s1monw opened a new pull request #1460: Fix visibility on member variables in IndexWriter and friends
s1monw opened a new pull request #1460: URL: https://github.com/apache/lucene-solr/pull/1460 Today it looks like wild wild west inside IndexWriter and some of it's associated classes. This change makes sure all non-final members have private visibility, methods that are not used outside of IW today are made private unless they have been public. This change also removes some unused or unnecessary members where possible and deleted some dead code from previous refactoring. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9328) SortingGroupHead to reuse DocValues
[ https://issues.apache.org/jira/browse/LUCENE-9328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093214#comment-17093214 ] Alan Woodward commented on LUCENE-9328: --- Would you be able to open this as a github pull request? It's much easier to review than a patch. We need to deal with SortedNumeric and SortedSet iterators, because they also have the notion of a within-document iterator that will need to be independent between all the different consumers. I'm also a bit worried about one consumer being able to advance the underlying values iterator too far. Can we make the ReusingReader control advancing the iterators? > SortingGroupHead to reuse DocValues > --- > > Key: LUCENE-9328 > URL: https://issues.apache.org/jira/browse/LUCENE-9328 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/grouping >Reporter: Mikhail Khludnev >Assignee: Mikhail Khludnev >Priority: Minor > Attachments: LUCENE-9328.patch, LUCENE-9328.patch, LUCENE-9328.patch > > > That's why > https://issues.apache.org/jira/browse/LUCENE-7701?focusedCommentId=17084365&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17084365 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9314) Lucene monitor module uses ByteBuffersDirectory rather than MemoryIndex when matching a single document
[ https://issues.apache.org/jira/browse/LUCENE-9314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Woodward resolved LUCENE-9314. --- Fix Version/s: 8.6 Assignee: Alan Woodward Resolution: Fixed Thanks [~pl.perron]! > Lucene monitor module uses ByteBuffersDirectory rather than MemoryIndex when > matching a single document > --- > > Key: LUCENE-9314 > URL: https://issues.apache.org/jira/browse/LUCENE-9314 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/other >Affects Versions: 8.2, 8.3, 8.4, 8.5 >Reporter: Pierre-Luc Perron >Assignee: Alan Woodward >Priority: Minor > Labels: easyfix, performance, pull-request-available > Fix For: 8.6 > > Attachments: LUCENE-9314.patch, LUCENE-9314.patch, LUCENE-9314.patch, > LUCENE-9314.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Lucene monitor function, [match single > document|https://github.com/apache/lucene-solr/blob/e376582e25d02f85b415eb1c0456f3fdc800fc84/lucene/monitor/src/java/org/apache/lucene/monitor/Monitor.java#L276], > wraps a single document into a array of documents. Hence, it always calls > the function, [match many > documents|https://github.com/apache/lucene-solr/blob/e376582e25d02f85b415eb1c0456f3fdc800fc84/lucene/monitor/src/java/org/apache/lucene/monitor/Monitor.java#L256], > which builds a > [MultiDocumentBatch|https://github.com/apache/lucene-solr/blob/71d335ff688982cef10a648c914623c81ced/lucene/monitor/src/java/org/apache/lucene/monitor/DocumentBatch.java#L56] > rather than a > [SingletonDocumentBatch|https://github.com/apache/lucene-solr/blob/71d335ff688982cef10a648c914623c81ced/lucene/monitor/src/java/org/apache/lucene/monitor/DocumentBatch.java#L46]. > The former uses ByteBuffersDirectory while later uses MemoryIndex. > As per documentation, > [MemoryIndex|https://lucene.apache.org/core/8_2_0/memory/index.html] is a > high-performance single-document main memory Apache Lucene fulltext search > index. Hence, Lucene monitor should use it when matching a single document. > The patch routes [match single > document|https://github.com/apache/lucene-solr/blob/e376582e25d02f85b415eb1c0456f3fdc800fc84/lucene/monitor/src/java/org/apache/lucene/monitor/Monitor.java#L276] > to a SingletonDocumentBatch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9314) Lucene monitor module uses ByteBuffersDirectory rather than MemoryIndex when matching a single document
[ https://issues.apache.org/jira/browse/LUCENE-9314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093274#comment-17093274 ] ASF subversion and git services commented on LUCENE-9314: - Commit 633ae90d7d930c34846cb7dd9be3126d3cffaa34 in lucene-solr's branch refs/heads/branch_8x from Alan Woodward [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=633ae90 ] LUCENE-9314: Use SingletonDocumentBatch in monitor when we only have a single document > Lucene monitor module uses ByteBuffersDirectory rather than MemoryIndex when > matching a single document > --- > > Key: LUCENE-9314 > URL: https://issues.apache.org/jira/browse/LUCENE-9314 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/other >Affects Versions: 8.2, 8.3, 8.4, 8.5 >Reporter: Pierre-Luc Perron >Priority: Minor > Labels: easyfix, performance, pull-request-available > Attachments: LUCENE-9314.patch, LUCENE-9314.patch, LUCENE-9314.patch, > LUCENE-9314.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Lucene monitor function, [match single > document|https://github.com/apache/lucene-solr/blob/e376582e25d02f85b415eb1c0456f3fdc800fc84/lucene/monitor/src/java/org/apache/lucene/monitor/Monitor.java#L276], > wraps a single document into a array of documents. Hence, it always calls > the function, [match many > documents|https://github.com/apache/lucene-solr/blob/e376582e25d02f85b415eb1c0456f3fdc800fc84/lucene/monitor/src/java/org/apache/lucene/monitor/Monitor.java#L256], > which builds a > [MultiDocumentBatch|https://github.com/apache/lucene-solr/blob/71d335ff688982cef10a648c914623c81ced/lucene/monitor/src/java/org/apache/lucene/monitor/DocumentBatch.java#L56] > rather than a > [SingletonDocumentBatch|https://github.com/apache/lucene-solr/blob/71d335ff688982cef10a648c914623c81ced/lucene/monitor/src/java/org/apache/lucene/monitor/DocumentBatch.java#L46]. > The former uses ByteBuffersDirectory while later uses MemoryIndex. > As per documentation, > [MemoryIndex|https://lucene.apache.org/core/8_2_0/memory/index.html] is a > high-performance single-document main memory Apache Lucene fulltext search > index. Hence, Lucene monitor should use it when matching a single document. > The patch routes [match single > document|https://github.com/apache/lucene-solr/blob/e376582e25d02f85b415eb1c0456f3fdc800fc84/lucene/monitor/src/java/org/apache/lucene/monitor/Monitor.java#L276] > to a SingletonDocumentBatch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9314) Lucene monitor module uses ByteBuffersDirectory rather than MemoryIndex when matching a single document
[ https://issues.apache.org/jira/browse/LUCENE-9314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093275#comment-17093275 ] ASF subversion and git services commented on LUCENE-9314: - Commit 5d5b7e14d41e00671fc938ad0db7714d56bd455e in lucene-solr's branch refs/heads/master from Alan Woodward [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5d5b7e1 ] LUCENE-9314: Use SingletonDocumentBatch in monitor when we only have a single document > Lucene monitor module uses ByteBuffersDirectory rather than MemoryIndex when > matching a single document > --- > > Key: LUCENE-9314 > URL: https://issues.apache.org/jira/browse/LUCENE-9314 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/other >Affects Versions: 8.2, 8.3, 8.4, 8.5 >Reporter: Pierre-Luc Perron >Priority: Minor > Labels: easyfix, performance, pull-request-available > Attachments: LUCENE-9314.patch, LUCENE-9314.patch, LUCENE-9314.patch, > LUCENE-9314.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Lucene monitor function, [match single > document|https://github.com/apache/lucene-solr/blob/e376582e25d02f85b415eb1c0456f3fdc800fc84/lucene/monitor/src/java/org/apache/lucene/monitor/Monitor.java#L276], > wraps a single document into a array of documents. Hence, it always calls > the function, [match many > documents|https://github.com/apache/lucene-solr/blob/e376582e25d02f85b415eb1c0456f3fdc800fc84/lucene/monitor/src/java/org/apache/lucene/monitor/Monitor.java#L256], > which builds a > [MultiDocumentBatch|https://github.com/apache/lucene-solr/blob/71d335ff688982cef10a648c914623c81ced/lucene/monitor/src/java/org/apache/lucene/monitor/DocumentBatch.java#L56] > rather than a > [SingletonDocumentBatch|https://github.com/apache/lucene-solr/blob/71d335ff688982cef10a648c914623c81ced/lucene/monitor/src/java/org/apache/lucene/monitor/DocumentBatch.java#L46]. > The former uses ByteBuffersDirectory while later uses MemoryIndex. > As per documentation, > [MemoryIndex|https://lucene.apache.org/core/8_2_0/memory/index.html] is a > high-performance single-document main memory Apache Lucene fulltext search > index. Hence, Lucene monitor should use it when matching a single document. > The patch routes [match single > document|https://github.com/apache/lucene-solr/blob/e376582e25d02f85b415eb1c0456f3fdc800fc84/lucene/monitor/src/java/org/apache/lucene/monitor/Monitor.java#L276] > to a SingletonDocumentBatch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9347) Update to forbiddenapis 3.0
[ https://issues.apache.org/jira/browse/LUCENE-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093279#comment-17093279 ] ASF subversion and git services commented on LUCENE-9347: - Commit 64eed9a1a62223a23d6b3b5c78b0335b06e5d6fc in lucene-solr's branch refs/heads/master from Uwe Schindler [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=64eed9a ] LUCENE-9347: Add support for forbiddenapis 3.0 (#1459) LUCENE-9347: Add support for forbiddenapis 3.0 > Update to forbiddenapis 3.0 > --- > > Key: LUCENE-9347 > URL: https://issues.apache.org/jira/browse/LUCENE-9347 > Project: Lucene - Core > Issue Type: Task > Components: general/build >Reporter: Uwe Schindler >Assignee: Uwe Schindler >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Forbiddenapis 3.0 was released a minute ago: > - Support for Java 14 > - Gradle Task Configuration Avoidance support > See: > https://github.com/policeman-tools/forbidden-apis/wiki/Changes#version-30-released-2020-04-27 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9347) Update to forbiddenapis 3.0
[ https://issues.apache.org/jira/browse/LUCENE-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093280#comment-17093280 ] ASF subversion and git services commented on LUCENE-9347: - Commit 64eed9a1a62223a23d6b3b5c78b0335b06e5d6fc in lucene-solr's branch refs/heads/master from Uwe Schindler [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=64eed9a ] LUCENE-9347: Add support for forbiddenapis 3.0 (#1459) LUCENE-9347: Add support for forbiddenapis 3.0 > Update to forbiddenapis 3.0 > --- > > Key: LUCENE-9347 > URL: https://issues.apache.org/jira/browse/LUCENE-9347 > Project: Lucene - Core > Issue Type: Task > Components: general/build >Reporter: Uwe Schindler >Assignee: Uwe Schindler >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Forbiddenapis 3.0 was released a minute ago: > - Support for Java 14 > - Gradle Task Configuration Avoidance support > See: > https://github.com/policeman-tools/forbidden-apis/wiki/Changes#version-30-released-2020-04-27 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] plperron commented on pull request #1423: LUCENE-9314: Match a document with MemoryIndex on LuceneMonitor
plperron commented on pull request #1423: URL: https://github.com/apache/lucene-solr/pull/1423#issuecomment-619873280 The commit will be included in Lucene 9.0.0 https://gitbox.apache.org/repos/asf?p=lucene-solr.git;a=blob;f=lucene/CHANGES.txt;h=9754bbeee45529a2503e6b227d25b44dc5cb88b1;hb=5d5b7e1#l213 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9347) Update to forbiddenapis 3.0
[ https://issues.apache.org/jira/browse/LUCENE-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093294#comment-17093294 ] ASF subversion and git services commented on LUCENE-9347: - Commit 19b5a76000da62796481c6c4a949ca7aeb5bf4c9 in lucene-solr's branch refs/heads/branch_8x from Uwe Schindler [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=19b5a76 ] LUCENE-9347: Add support for forbiddenapis 3.0 (#1459) LUCENE-9347: Add support for forbiddenapis 3.0 # Conflicts: # build.gradle > Update to forbiddenapis 3.0 > --- > > Key: LUCENE-9347 > URL: https://issues.apache.org/jira/browse/LUCENE-9347 > Project: Lucene - Core > Issue Type: Task > Components: general/build >Reporter: Uwe Schindler >Assignee: Uwe Schindler >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Forbiddenapis 3.0 was released a minute ago: > - Support for Java 14 > - Gradle Task Configuration Avoidance support > See: > https://github.com/policeman-tools/forbidden-apis/wiki/Changes#version-30-released-2020-04-27 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-12697) pure DocValues support for FieldValueFeature
[ https://issues.apache.org/jira/browse/SOLR-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093292#comment-17093292 ] Lucene/Solr QA commented on SOLR-12697: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Release audit (RAT) {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Check forbidden APIs {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Validate source patterns {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 9s{color} | {color:green} ltr in the patch passed. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 4m 55s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | SOLR-12697 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12969051/SOLR-12697.patch | | Optional Tests | compile javac unit ratsources checkforbiddenapis validatesourcepatterns | | uname | Linux lucene1-us-west 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | ant | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh | | git revision | master / 64eed9a1a62 | | ant | version: Apache Ant(TM) version 1.10.5 compiled on March 28 2019 | | Default Java | LTS | | Test Results | https://builds.apache.org/job/PreCommit-SOLR-Build/745/testReport/ | | modules | C: solr/contrib/ltr U: solr/contrib/ltr | | Console output | https://builds.apache.org/job/PreCommit-SOLR-Build/745/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > pure DocValues support for FieldValueFeature > > > Key: SOLR-12697 > URL: https://issues.apache.org/jira/browse/SOLR-12697 > Project: Solr > Issue Type: Sub-task > Components: contrib - LTR >Reporter: Stanislav Livotov >Priority: Major > Attachments: SOLR-12697.patch, SOLR-12697.patch, SOLR-12697.patch, > SOLR-12697.patch, SOLR-12697.patch > > > [~slivotov] wrote in SOLR-12688: > bq. ... FieldValueFeature doesn't support pure DocValues fields (Stored > false). Please also note that for fields which are both stored and DocValues > it is working not optimal because it is extracting just one field from the > stored document. DocValues are obviously faster for such usecases. ... > (Please see SOLR-12688 description for overall context and analysis results.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9347) Update to forbiddenapis 3.0
[ https://issues.apache.org/jira/browse/LUCENE-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093295#comment-17093295 ] ASF subversion and git services commented on LUCENE-9347: - Commit 19b5a76000da62796481c6c4a949ca7aeb5bf4c9 in lucene-solr's branch refs/heads/branch_8x from Uwe Schindler [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=19b5a76 ] LUCENE-9347: Add support for forbiddenapis 3.0 (#1459) LUCENE-9347: Add support for forbiddenapis 3.0 # Conflicts: # build.gradle > Update to forbiddenapis 3.0 > --- > > Key: LUCENE-9347 > URL: https://issues.apache.org/jira/browse/LUCENE-9347 > Project: Lucene - Core > Issue Type: Task > Components: general/build >Reporter: Uwe Schindler >Assignee: Uwe Schindler >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Forbiddenapis 3.0 was released a minute ago: > - Support for Java 14 > - Gradle Task Configuration Avoidance support > See: > https://github.com/policeman-tools/forbidden-apis/wiki/Changes#version-30-released-2020-04-27 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9347) Update to forbiddenapis 3.0
[ https://issues.apache.org/jira/browse/LUCENE-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093322#comment-17093322 ] Uwe Schindler commented on LUCENE-9347: --- Thanks [~dweiss], I just mentioned this because I was looking how forbiddenapis changes affect Lucene in the new gradle build. My main use case when developing was just: "get it working ey"! So the parallel approach in Lucene was just convenient to me for integration testing. BTW, for my quick test setup, I just commented out the {{plugin}} and put some {{buildscript}} hack with {{mavenLocal()}} before. About the task avoidance: I have no idea how this magic affects you at all. I think for standard projects there's nothing at all. So it's more some selling point by the gradle guys. In short: It won't change anything. For forbiddenapis it was also only some reason to move major version to 3.0, nothing more! My personal opinion as a Gradle plugin developer: Gradle is a pain - sorry - I could live without it! > Update to forbiddenapis 3.0 > --- > > Key: LUCENE-9347 > URL: https://issues.apache.org/jira/browse/LUCENE-9347 > Project: Lucene - Core > Issue Type: Task > Components: general/build >Reporter: Uwe Schindler >Assignee: Uwe Schindler >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Forbiddenapis 3.0 was released a minute ago: > - Support for Java 14 > - Gradle Task Configuration Avoidance support > See: > https://github.com/policeman-tools/forbidden-apis/wiki/Changes#version-30-released-2020-04-27 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13942) /api/cluster/zk/* to fetch raw ZK data
[ https://issues.apache.org/jira/browse/SOLR-13942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093327#comment-17093327 ] ASF subversion and git services commented on SOLR-13942: Commit ea46596cb3d6bb1dcef79bf352b87114a881766b in lucene-solr's branch refs/heads/master from Noble Paul [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=ea46596 ] SOLR-13942: /api/cluster/zk/* to fetch raw ZK data * SOLR-13942: /api/cluster/zk/* to fetch raw ZK data > /api/cluster/zk/* to fetch raw ZK data > -- > > Key: SOLR-13942 > URL: https://issues.apache.org/jira/browse/SOLR-13942 > Project: Solr > Issue Type: New Feature > Components: v2 API >Reporter: Noble Paul >Assignee: Noble Paul >Priority: Minor > Time Spent: 8h 40m > Remaining Estimate: 0h > > example > download the {{state.json}} of > {code} > GET http://localhost:8983/api/cluster/zk/collections/gettingstarted/state.json > {code} > get a list of all children under {{/live_nodes}} > {code} > GET http://localhost:8983/api/cluster/zk/live_nodes > {code} > If the requested path is a node with children show the list of child nodes > and their meta data -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13942) /api/cluster/zk/* to fetch raw ZK data
[ https://issues.apache.org/jira/browse/SOLR-13942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093328#comment-17093328 ] ASF subversion and git services commented on SOLR-13942: Commit ea46596cb3d6bb1dcef79bf352b87114a881766b in lucene-solr's branch refs/heads/master from Noble Paul [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=ea46596 ] SOLR-13942: /api/cluster/zk/* to fetch raw ZK data * SOLR-13942: /api/cluster/zk/* to fetch raw ZK data > /api/cluster/zk/* to fetch raw ZK data > -- > > Key: SOLR-13942 > URL: https://issues.apache.org/jira/browse/SOLR-13942 > Project: Solr > Issue Type: New Feature > Components: v2 API >Reporter: Noble Paul >Assignee: Noble Paul >Priority: Minor > Time Spent: 8h 40m > Remaining Estimate: 0h > > example > download the {{state.json}} of > {code} > GET http://localhost:8983/api/cluster/zk/collections/gettingstarted/state.json > {code} > get a list of all children under {{/live_nodes}} > {code} > GET http://localhost:8983/api/cluster/zk/live_nodes > {code} > If the requested path is a node with children show the list of child nodes > and their meta data -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9017) GroupingSearch does not return all the groups when using SortedSetDocValuesField
[ https://issues.apache.org/jira/browse/LUCENE-9017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093329#comment-17093329 ] Alan Woodward commented on LUCENE-9017: --- SortedSetFieldSource by default chooses the minimum value for each document. I don't think there's a way of grouping by all possible values at the moment, the various collectors all assume one grouping value per document. > GroupingSearch does not return all the groups when using > SortedSetDocValuesField > > > Key: LUCENE-9017 > URL: https://issues.apache.org/jira/browse/LUCENE-9017 > Project: Lucene - Core > Issue Type: Bug > Components: core/search >Affects Versions: 8.2 >Reporter: Juraj Kucera >Priority: Major > > I managed to retrieve the groups using the *SortedSetDocValuesField* in > *GroupingSearch* by initializing the groupsearch with *SortedSetFieldSource* > The problem is when a document has multiple values in the field > "SortedSetDocValuesField" than not the grouping query does not return all > the groups. > Let me demonstrate it in my example > // indexing, the first object has the category "one" and the second object > has category "two" and "three" > Document doc = new Document(); > doc.add(new FacetField("Author", "Bob")); > doc.add(new SortedSetDocValuesField("category", new BytesRef("one"))); > indexWriter.addDocument(config.build(taxoWriter, doc)); > doc = new Document(); > doc.add(new FacetField("Author", "Lisa")); > doc.add(new SortedSetDocValuesField("category", new BytesRef("two"))); > doc.add(new SortedSetDocValuesField("category", new BytesRef("three"))); > indexWriter.addDocument(config.build(taxoWriter, doc)); > // initializing the grouping search > ValueSource vs = new SortedSetFieldSource(groupField); > groupingSearch = new GroupingSearch(vs, new HashMap<>()); > // performing the group search > TopGroups groups = groupingSearch.search(searcher, new MatchAllDocsQuery(), > 0, 100); > It returns 2 groups only and I would expect 3 groups ("one", "two" and > "three") > *Is it a bug or am I using the API in a wrong way?* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13942) /api/cluster/zk/* to fetch raw ZK data
[ https://issues.apache.org/jira/browse/SOLR-13942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093332#comment-17093332 ] ASF subversion and git services commented on SOLR-13942: Commit b25eabe6a9c1169410598e7d960423360dde3b91 in lucene-solr's branch refs/heads/master from Noble Paul [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=b25eabe ] SOLR-13942: A read API at /api/cluster/zk/* to fetch raw ZK data and view contents of a ZK direcory > /api/cluster/zk/* to fetch raw ZK data > -- > > Key: SOLR-13942 > URL: https://issues.apache.org/jira/browse/SOLR-13942 > Project: Solr > Issue Type: New Feature > Components: v2 API >Reporter: Noble Paul >Assignee: Noble Paul >Priority: Minor > Time Spent: 8h 40m > Remaining Estimate: 0h > > example > download the {{state.json}} of > {code} > GET http://localhost:8983/api/cluster/zk/collections/gettingstarted/state.json > {code} > get a list of all children under {{/live_nodes}} > {code} > GET http://localhost:8983/api/cluster/zk/live_nodes > {code} > If the requested path is a node with children show the list of child nodes > and their meta data -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13942) /api/cluster/zk/* to fetch raw ZK data
[ https://issues.apache.org/jira/browse/SOLR-13942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093334#comment-17093334 ] ASF subversion and git services commented on SOLR-13942: Commit f5c0f5c3e4a14ac0880c37a208912947cb6f14fb in lucene-solr's branch refs/heads/branch_8x from Noble Paul [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=f5c0f5c ] SOLR-13942: A read API at /api/cluster/zk/* to fetch raw ZK data and view contents of a ZK direcory > /api/cluster/zk/* to fetch raw ZK data > -- > > Key: SOLR-13942 > URL: https://issues.apache.org/jira/browse/SOLR-13942 > Project: Solr > Issue Type: New Feature > Components: v2 API >Reporter: Noble Paul >Assignee: Noble Paul >Priority: Minor > Time Spent: 8h 40m > Remaining Estimate: 0h > > example > download the {{state.json}} of > {code} > GET http://localhost:8983/api/cluster/zk/collections/gettingstarted/state.json > {code} > get a list of all children under {{/live_nodes}} > {code} > GET http://localhost:8983/api/cluster/zk/live_nodes > {code} > If the requested path is a node with children show the list of child nodes > and their meta data -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13942) /api/cluster/zk/* to fetch raw ZK data
[ https://issues.apache.org/jira/browse/SOLR-13942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul resolved SOLR-13942. --- Fix Version/s: 8.6 Resolution: Fixed > /api/cluster/zk/* to fetch raw ZK data > -- > > Key: SOLR-13942 > URL: https://issues.apache.org/jira/browse/SOLR-13942 > Project: Solr > Issue Type: New Feature > Components: v2 API >Reporter: Noble Paul >Assignee: Noble Paul >Priority: Minor > Fix For: 8.6 > > Time Spent: 8h 40m > Remaining Estimate: 0h > > example > download the {{state.json}} of > {code} > GET http://localhost:8983/api/cluster/zk/collections/gettingstarted/state.json > {code} > get a list of all children under {{/live_nodes}} > {code} > GET http://localhost:8983/api/cluster/zk/live_nodes > {code} > If the requested path is a node with children show the list of child nodes > and their meta data -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9347) Update to forbiddenapis 3.0
[ https://issues.apache.org/jira/browse/LUCENE-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093339#comment-17093339 ] Dawid Weiss commented on LUCENE-9347: - I agree Gradle is a pain sometimes. But so is ant and maven. I haven't found a build tool I'd be fully happy with yet (including IDE support, etc.). They all have some pros and cons. > Update to forbiddenapis 3.0 > --- > > Key: LUCENE-9347 > URL: https://issues.apache.org/jira/browse/LUCENE-9347 > Project: Lucene - Core > Issue Type: Task > Components: general/build >Reporter: Uwe Schindler >Assignee: Uwe Schindler >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Forbiddenapis 3.0 was released a minute ago: > - Support for Java 14 > - Gradle Task Configuration Avoidance support > See: > https://github.com/policeman-tools/forbidden-apis/wiki/Changes#version-30-released-2020-04-27 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9255) ValueSource Has Generic Typing Issues
[ https://issues.apache.org/jira/browse/LUCENE-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093362#comment-17093362 ] Alan Woodward commented on LUCENE-9255: --- The only place outside of the `queries` module we use ValueSource in lucene itself is in `grouping`, and that can I think be moved to Solr fairly simply with a bit of test refactoring. I started work on a range grouper using DoubleValuesSource a while back, LUCENE-7889, and we could resuscitate that as well. > ValueSource Has Generic Typing Issues > - > > Key: LUCENE-9255 > URL: https://issues.apache.org/jira/browse/LUCENE-9255 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > ValueSource uses a bunch of weakly typed members which raises compiler > issues. We need to fix this in ValueSource and all of its subclasses. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9033) Update Release docs an scripts with new site instructions
[ https://issues.apache.org/jira/browse/LUCENE-9033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093430#comment-17093430 ] Jan Høydahl commented on LUCENE-9033: - Update the PR with a rough change, just partly tested > Update Release docs an scripts with new site instructions > - > > Key: LUCENE-9033 > URL: https://issues.apache.org/jira/browse/LUCENE-9033 > Project: Lucene - Core > Issue Type: Sub-task > Components: general/tools >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > *releaseWizard.py:* [PR#1324|https://github.com/apache/lucene-solr/pull/1324] > Janhoy has started on this, but will likely not finish before the 8.5 release > *[ReleaseTODO|https://cwiki.apache.org/confluence/display/LUCENE/ReleaseTodo] > page:* I suggest we deprecate this page if folks are happy with > releaseWizard, which should encapsulate all steps and details, and can also > generate an HTML TODO document per release. > *publish-solr-ref-guide.sh:* > [PR#1326|https://github.com/apache/lucene-solr/pull/1326] This script can be > deleted, not in use since we do not publish PDF anymore > *(/) solr-ref-gudie/src/meta-docs/publish.adoc:* Done > > There may be other places affected, such as other WIKI pages? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7788) fail precommit on unparameterised log messages and examine for wasted work/objects
[ https://issues.apache.org/jira/browse/LUCENE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093499#comment-17093499 ] ASF subversion and git services commented on LUCENE-7788: - Commit ff4363675e3f0873cdb874d12740749449e91d9f in lucene-solr's branch refs/heads/master from Erick Erickson [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=ff43636 ] LUCENE-7788: fail precommit on unparameterised log messages and examine for wasted work/objects > fail precommit on unparameterised log messages and examine for wasted > work/objects > -- > > Key: LUCENE-7788 > URL: https://issues.apache.org/jira/browse/LUCENE-7788 > Project: Lucene - Core > Issue Type: Task >Reporter: Christine Poerschke >Assignee: Erick Erickson >Priority: Minor > Attachments: LUCENE-7788.patch, LUCENE-7788.patch, gradle_only.patch, > gradle_only.patch > > Time Spent: 50m > Remaining Estimate: 0h > > SOLR-10415 would be removing existing unparameterised log.trace messages use > and once that is in place then this ticket's one-line change would be for > 'ant precommit' to reject any future unparameterised log.trace message use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7788) fail precommit on unparameterised log messages and examine for wasted work/objects
[ https://issues.apache.org/jira/browse/LUCENE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093508#comment-17093508 ] ASF subversion and git services commented on LUCENE-7788: - Commit 8ef4b332114609bcd24bb8ea094f614a966b0ece in lucene-solr's branch refs/heads/branch_8x from Erick Erickson [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=8ef4b33 ] LUCENE-7788: fail precommit on unparameterised log messages and examine for wasted work/objects > fail precommit on unparameterised log messages and examine for wasted > work/objects > -- > > Key: LUCENE-7788 > URL: https://issues.apache.org/jira/browse/LUCENE-7788 > Project: Lucene - Core > Issue Type: Task >Reporter: Christine Poerschke >Assignee: Erick Erickson >Priority: Minor > Attachments: LUCENE-7788.patch, LUCENE-7788.patch, gradle_only.patch, > gradle_only.patch > > Time Spent: 50m > Remaining Estimate: 0h > > SOLR-10415 would be removing existing unparameterised log.trace messages use > and once that is in place then this ticket's one-line change would be for > 'ant precommit' to reject any future unparameterised log.trace message use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-12798) Structural changes in SolrJ since version 7.0.0 have effectively disabled multipart post
[ https://issues.apache.org/jira/browse/SOLR-12798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093511#comment-17093511 ] Christian Beikov commented on SOLR-12798: - I can confirm that the workaround of adding an additional named empty content stream forces multipart transmission which is a workaround for this. Using SolrJ 8.4.1 here. This is pretty tough to analyze. On the server side I only get a warning although the connection is closed by the server when encountering this. 2020-04-27 12:19:07.958 WARN (qtp1305486145-15) [ ] o.e.j.h.HttpParser URI is too large >8192 On the client side, after a few retries, I only get an exception saying that the connection was closed. > Structural changes in SolrJ since version 7.0.0 have effectively disabled > multipart post > > > Key: SOLR-12798 > URL: https://issues.apache.org/jira/browse/SOLR-12798 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Affects Versions: 7.4 >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Attachments: HOT Balloon Trip_Ultra HD.jpg, > SOLR-12798-approach.patch, SOLR-12798-reproducer.patch, > SOLR-12798-workaround.patch, SOLR-12798.patch, SOLR-12798.patch, > SOLR-12798.patch, no params in url.png, solr-update-request.txt > > > Project ManifoldCF uses SolrJ to post documents to Solr. When upgrading from > SolrJ 7.0.x to SolrJ 7.4, we encountered significant structural changes to > SolrJ's HttpSolrClient class that seemingly disable any use of multipart > post. This is critical because ManifoldCF's documents often contain metadata > in excess of 4K that therefore cannot be stuffed into a URL. > The changes in question seem to have been performed by Paul Noble on > 10/31/2017, with the introduction of the RequestWriter mechanism. Basically, > if a request has a RequestWriter, it is used exclusively to write the > request, and that overrides the stream mechanism completely. I haven't > chased it back to a specific ticket. > ManifoldCF's usage of SolrJ involves the creation of > ContentStreamUpdateRequests for all posts meant for Solr Cell, and the > creation of UpdateRequests for posts not meant for Solr Cell (as well as for > delete and commit requests). For our release cycle that is taking place > right now, we're shipping a modified version of HttpSolrClient that ignores > the RequestWriter when dealing with ContentStreamUpdateRequests. We > apparently cannot use multipart for all requests because on the Solr side we > get "pfountz Should not get here!" errors on the Solr side when we do, which > generate HTTP error code 500 responses. That should not happen either, in my > opinion. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9033) Update Release docs an scripts with new site instructions
[ https://issues.apache.org/jira/browse/LUCENE-9033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093588#comment-17093588 ] Jan Høydahl commented on LUCENE-9033: - Please review the changes. Steps added to update the new website * Update editorconfig.py * Add draft website news for solr/lucene (not TLP) * Use those drafts as input for the release email templates * Better editor default for macOS (TextEdit) I know it is hard to test this without doing a release, but it is possible to only test submenu 1 - 7 "Update the website", and simulate the steps, just answering "n" for the git push commands... You can also select the menu "Generate asciidoc guide" and inspect the generated commands manually to see if you spot something wrong. > Update Release docs an scripts with new site instructions > - > > Key: LUCENE-9033 > URL: https://issues.apache.org/jira/browse/LUCENE-9033 > Project: Lucene - Core > Issue Type: Sub-task > Components: general/tools >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > *releaseWizard.py:* [PR#1324|https://github.com/apache/lucene-solr/pull/1324] > Janhoy has started on this, but will likely not finish before the 8.5 release > *[ReleaseTODO|https://cwiki.apache.org/confluence/display/LUCENE/ReleaseTodo] > page:* I suggest we deprecate this page if folks are happy with > releaseWizard, which should encapsulate all steps and details, and can also > generate an HTML TODO document per release. > *publish-solr-ref-guide.sh:* > [PR#1326|https://github.com/apache/lucene-solr/pull/1326] This script can be > deleted, not in use since we do not publish PDF anymore > *(/) solr-ref-gudie/src/meta-docs/publish.adoc:* Done > > There may be other places affected, such as other WIKI pages? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14432) SOLR Dataimport hanlder going to idle after some time
[ https://issues.apache.org/jira/browse/SOLR-14432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093660#comment-17093660 ] Ravi kumar commented on SOLR-14432: --- Hi, Can someone helpout on this issue > SOLR Dataimport hanlder going to idle after some time > - > > Key: SOLR-14432 > URL: https://issues.apache.org/jira/browse/SOLR-14432 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - DataImportHandler >Affects Versions: 8.5.1 > Environment: Windows >Reporter: Ravi kumar >Priority: Major > Labels: dataimportHandler, solr > Attachments: solr.log > > > I configured data import handler to process bulk PDF documents. after > process 21000 documents. Process going to idle and not processing all the > documents. > When i see the log observed below things (attached log for your reference). > Please let me know is there anyway that i can ignore this issue or any > setting do i need to update. > Error: > 2020-04-23 18:39:55.749 INFO (qtp215219944-24) [ x:DMS] o.a.s.c.S.Request > [DMS] webapp=/solr path=/dataimport > params=\{indent=on&wt=json&command=status&_=1587664092295} status=0 QTime=0 > 2020-04-23 18:39:55.972 WARN (Thread-14) [ ] o.a.p.p.COSParser > T{color:#de350b}he end of the stream is out of range, using workaround to > read the stream, stream start position: 4748210, length: 2007324, expected > end position: 6755534{color} > 2020-04-23 18:39:55.976 WARN (Thread-14) [ ] o.a.p.p.COSParser Removed null > object COSObject\{50, 0} from pages dictionary > 2020-04-23 18:39:55.976 WARN (Thread-14) [ ] o.a.p.p.COSParser Removed null > object COSObject\{60, 0} from pages dictionary > 2020-04-23 18:39:55.997 {color:#de350b}ERROR (Thread-14) [ ] > o.a.p.c.o.s.SetGraphicsStateParameters name for 'gs' operator not found in > resources: /R7{color} > > {color:#172b4d}{color:#de350b}Regards,{color}{color} > {color:#172b4d}{color:#de350b}Ravi kumar{color}{color} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9348) Rework grouping tests to make it simpler to add new GroupSelector implementations
Alan Woodward created LUCENE-9348: - Summary: Rework grouping tests to make it simpler to add new GroupSelector implementations Key: LUCENE-9348 URL: https://issues.apache.org/jira/browse/LUCENE-9348 Project: Lucene - Core Issue Type: Improvement Reporter: Alan Woodward Assignee: Alan Woodward The grouping module tests currently all try and test both grouping by term and grouping by ValueSource. They are quite difficult to follow, however, and it is not at all easy to add tests for a new grouping type. We should refactor things into an abstract base class that can then be extended by tests for each specific grouping type. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] romseygeek opened a new pull request #1461: LUCENE-9348: Add a base grouping test for use with different GroupSelector implementations
romseygeek opened a new pull request #1461: URL: https://github.com/apache/lucene-solr/pull/1461 The grouping module tests currently all try and test both grouping by term and grouping by ValueSource. They are quite difficult to follow, however, and it is not at all easy to add tests for a new grouping type. This commit adds a new BaseGroupSelectorTestCase class which can be extended to test particular GroupSelector implementations, and adds tests for TermGroupSelector and ValueSourceGroupSelector. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] romseygeek commented on pull request #1461: LUCENE-9348: Add a base grouping test for use with different GroupSelector implementations
romseygeek commented on pull request #1461: URL: https://github.com/apache/lucene-solr/pull/1461#issuecomment-620093446 cc @martijnvg @dsmiley We should be able to remove `AllGroupHeadsCollectorTest`, `AllGroupsCollectorTest` and `GroupingSearchTest` once this is merged, and possibly `TestGrouping` as well. It should also make it much easier to add groupers based on Double/LongValuesSource, with the aim of moving the `ValueSourceGroupSelector` into Solr and deprecating ValueSource within lucene. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14428) FuzzyQuery has severe memory usage in 8.5
[ https://issues.apache.org/jira/browse/SOLR-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colvin Cowie updated SOLR-14428: Attachment: SOLR-14428-WeakReferences.patch > FuzzyQuery has severe memory usage in 8.5 > - > > Key: SOLR-14428 > URL: https://issues.apache.org/jira/browse/SOLR-14428 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 8.5, 8.5.1 >Reporter: Colvin Cowie >Assignee: Andrzej Bialecki >Priority: Major > Attachments: FuzzyHammer.java, SOLR-14428-WeakReferences.patch, > image-2020-04-23-09-18-06-070.png, image-2020-04-24-20-09-31-179.png, > screenshot-2.png, screenshot-3.png, screenshot-4.png > > Time Spent: 10m > Remaining Estimate: 0h > > I sent this to the mailing list > I'm moving from 8.3.1 to 8.5.1, and started getting Out Of Memory Errors > while running our normal tests. After profiling it was clear that the > majority of the heap was allocated through FuzzyQuery. > LUCENE-9068 moved construction of the automata from the FuzzyTermsEnum to the > FuzzyQuery's constructor. > I created a little test ( [^FuzzyHammer.java] ) that fires off fuzzy queries > from random UUID strings for 5 minutes > {code} > FIELD_NAME + ":" + UUID.randomUUID().toString().replace("-", "") + "~2" > {code} > When running against a vanilla Solr 8.31 and 8.4.1 there is no problem, while > the memory usage has increased drastically on 8.5.0 and 8.5.1. > Comparison of heap usage while running the attached test against Solr 8.3.1 > and 8.5.1 with a single (empty) shard and 4GB heap: > !image-2020-04-23-09-18-06-070.png! > And with 4 shards on 8.4.1 and 8.5.0: > !screenshot-2.png! > I'm guessing that the memory might be being leaked if the FuzzyQuery objects > are referenced from the cache, while the FuzzyTermsEnum would not have been. > Query Result Cache on 8.5.1: > !screenshot-3.png! > ~316mb in the cache > QRC on 8.3.1 > !screenshot-4.png! > <1mb > With an empty cache, running this query > _field_s:e41848af85d24ac197c71db6888e17bc~2_ results in the following memory > allocation > {noformat} > 8.3.1: CACHE.searcher.queryResultCache.ramBytesUsed: 1520 > 8.5.1: CACHE.searcher.queryResultCache.ramBytesUsed:648855 > {noformat} > ~1 gives 98253 and ~0 gives 6339 on 8.5.1. 8.3.1 is constant at 1520 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14428) FuzzyQuery has severe memory usage in 8.5
[ https://issues.apache.org/jira/browse/SOLR-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093772#comment-17093772 ] Colvin Cowie commented on SOLR-14428: - That's fair. I think that having a distinction between the (Query) key used for cache lookups and the Query object used for actual querying is definitely a _good thing_. But I don't know enough about the internals of it all, so happy to hear what other people think. The one thing I have tried doing just now is using a {{SoftReference}} and a {{WeakReference}} on the {{automata}} so that it can be GCed and rebuilt if it's gone. With SoftReferences the OOMEs stop, but the heap usage stays higher, and with WeakReferences the heap usage looks pretty much the same as in 8.3.1. I'm not sure though whether it provides any real improvement on how things were before the changes made in LUCENE-9068, when the automata was just built on demand. Plus I have never actually used SoftReference or WeakReference before, so can't claim to truely know the ins and outs of them. [^SOLR-14428-WeakReferences.patch] > FuzzyQuery has severe memory usage in 8.5 > - > > Key: SOLR-14428 > URL: https://issues.apache.org/jira/browse/SOLR-14428 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 8.5, 8.5.1 >Reporter: Colvin Cowie >Assignee: Andrzej Bialecki >Priority: Major > Attachments: FuzzyHammer.java, SOLR-14428-WeakReferences.patch, > image-2020-04-23-09-18-06-070.png, image-2020-04-24-20-09-31-179.png, > screenshot-2.png, screenshot-3.png, screenshot-4.png > > Time Spent: 10m > Remaining Estimate: 0h > > I sent this to the mailing list > I'm moving from 8.3.1 to 8.5.1, and started getting Out Of Memory Errors > while running our normal tests. After profiling it was clear that the > majority of the heap was allocated through FuzzyQuery. > LUCENE-9068 moved construction of the automata from the FuzzyTermsEnum to the > FuzzyQuery's constructor. > I created a little test ( [^FuzzyHammer.java] ) that fires off fuzzy queries > from random UUID strings for 5 minutes > {code} > FIELD_NAME + ":" + UUID.randomUUID().toString().replace("-", "") + "~2" > {code} > When running against a vanilla Solr 8.31 and 8.4.1 there is no problem, while > the memory usage has increased drastically on 8.5.0 and 8.5.1. > Comparison of heap usage while running the attached test against Solr 8.3.1 > and 8.5.1 with a single (empty) shard and 4GB heap: > !image-2020-04-23-09-18-06-070.png! > And with 4 shards on 8.4.1 and 8.5.0: > !screenshot-2.png! > I'm guessing that the memory might be being leaked if the FuzzyQuery objects > are referenced from the cache, while the FuzzyTermsEnum would not have been. > Query Result Cache on 8.5.1: > !screenshot-3.png! > ~316mb in the cache > QRC on 8.3.1 > !screenshot-4.png! > <1mb > With an empty cache, running this query > _field_s:e41848af85d24ac197c71db6888e17bc~2_ results in the following memory > allocation > {noformat} > 8.3.1: CACHE.searcher.queryResultCache.ramBytesUsed: 1520 > 8.5.1: CACHE.searcher.queryResultCache.ramBytesUsed:648855 > {noformat} > ~1 gives 98253 and ~0 gives 6339 on 8.5.1. 8.3.1 is constant at 1520 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14428) FuzzyQuery has severe memory usage in 8.5
[ https://issues.apache.org/jira/browse/SOLR-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093864#comment-17093864 ] Mike Drob commented on SOLR-14428: -- That's a pretty good approach, I'm glad you thought of it! The structure of your changes and the reference usage looks correct to me. I probably would leave termLength as final and just worry about the CompiledAutomaton array. I think SoftReferences might be better - the problem with WeakReferences will be that the Automata get rebuilt on every pass through a query visitor. [~romseygeek] WDYT? > FuzzyQuery has severe memory usage in 8.5 > - > > Key: SOLR-14428 > URL: https://issues.apache.org/jira/browse/SOLR-14428 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 8.5, 8.5.1 >Reporter: Colvin Cowie >Assignee: Andrzej Bialecki >Priority: Major > Attachments: FuzzyHammer.java, SOLR-14428-WeakReferences.patch, > image-2020-04-23-09-18-06-070.png, image-2020-04-24-20-09-31-179.png, > screenshot-2.png, screenshot-3.png, screenshot-4.png > > Time Spent: 10m > Remaining Estimate: 0h > > I sent this to the mailing list > I'm moving from 8.3.1 to 8.5.1, and started getting Out Of Memory Errors > while running our normal tests. After profiling it was clear that the > majority of the heap was allocated through FuzzyQuery. > LUCENE-9068 moved construction of the automata from the FuzzyTermsEnum to the > FuzzyQuery's constructor. > I created a little test ( [^FuzzyHammer.java] ) that fires off fuzzy queries > from random UUID strings for 5 minutes > {code} > FIELD_NAME + ":" + UUID.randomUUID().toString().replace("-", "") + "~2" > {code} > When running against a vanilla Solr 8.31 and 8.4.1 there is no problem, while > the memory usage has increased drastically on 8.5.0 and 8.5.1. > Comparison of heap usage while running the attached test against Solr 8.3.1 > and 8.5.1 with a single (empty) shard and 4GB heap: > !image-2020-04-23-09-18-06-070.png! > And with 4 shards on 8.4.1 and 8.5.0: > !screenshot-2.png! > I'm guessing that the memory might be being leaked if the FuzzyQuery objects > are referenced from the cache, while the FuzzyTermsEnum would not have been. > Query Result Cache on 8.5.1: > !screenshot-3.png! > ~316mb in the cache > QRC on 8.3.1 > !screenshot-4.png! > <1mb > With an empty cache, running this query > _field_s:e41848af85d24ac197c71db6888e17bc~2_ results in the following memory > allocation > {noformat} > 8.3.1: CACHE.searcher.queryResultCache.ramBytesUsed: 1520 > 8.5.1: CACHE.searcher.queryResultCache.ramBytesUsed:648855 > {noformat} > ~1 gives 98253 and ~0 gives 6339 on 8.5.1. 8.3.1 is constant at 1520 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14439) Upgrade to Tika 1.24.1
Tim Allison created SOLR-14439: -- Summary: Upgrade to Tika 1.24.1 Key: SOLR-14439 URL: https://issues.apache.org/jira/browse/SOLR-14439 Project: Solr Issue Type: Task Security Level: Public (Default Security Level. Issues are Public) Components: contrib - DataImportHandler Reporter: Tim Allison Assignee: Tim Allison We recently released 1.24.1 with several fixes for DoS vulnerabilities we found via fuzzing: CVE-2020-9489 https://seclists.org/oss-sec/2020/q2/69 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mkhludnev opened a new pull request #1462: LUCENE-9328: open group.sort docvalues once per segment.
mkhludnev opened a new pull request #1462: URL: https://github.com/apache/lucene-solr/pull/1462 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9328) SortingGroupHead to reuse DocValues
[ https://issues.apache.org/jira/browse/LUCENE-9328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093929#comment-17093929 ] Mikhail Khludnev commented on LUCENE-9328: -- [~romseygeek], you are welcome [https://github.com/apache/lucene-solr/pull/1462/] At the patch all type docvalues pooled already. I'm not sure I understand what you mean in _control advancing the iterators_. Now there's no control. It's not so scary as it looked at the first glance. Collector scores docs in-order (o'rlly?), every doc just kicks a certain GroupHead, which pull a value(s) from it. And that's it. I don't even think they ever call dv.advanceExact() for the current dv.docID. I think I'll add a test covering grouping/sorting over sortedset dvs. > SortingGroupHead to reuse DocValues > --- > > Key: LUCENE-9328 > URL: https://issues.apache.org/jira/browse/LUCENE-9328 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/grouping >Reporter: Mikhail Khludnev >Assignee: Mikhail Khludnev >Priority: Minor > Attachments: LUCENE-9328.patch, LUCENE-9328.patch, LUCENE-9328.patch > > Time Spent: 10m > Remaining Estimate: 0h > > That's why > https://issues.apache.org/jira/browse/LUCENE-7701?focusedCommentId=17084365&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17084365 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14173) Ref Guide Redesign
[ https://issues.apache.org/jira/browse/SOLR-14173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093944#comment-17093944 ] Cassandra Targett commented on SOLR-14173: -- I've worked on a few things here and there with this but besides fiddling with tiny things I don't have much left to do. There's been a lack of feedback that makes me nervous, but I'm ready to move on to the next phase (reworking the top-level topics). I'm going to commit this tomorrow (Tuesday 28 April) unless there are any last-minute concerns that I can address. > Ref Guide Redesign > -- > > Key: SOLR-14173 > URL: https://issues.apache.org/jira/browse/SOLR-14173 > Project: Solr > Issue Type: Improvement > Components: documentation >Reporter: Cassandra Targett >Assignee: Cassandra Targett >Priority: Major > Attachments: SOLR-14173.patch, blue-left-nav.png, gray-left-nav.png > > > The current design of the Ref Guide was essentially copied from a > Jekyll-based documentation theme > (https://idratherbewriting.com/documentation-theme-jekyll/), which had a > couple important benefits for that time: > * It was well-documented and since I had little experience with Jekyll and > its Liquid templates and since I was the one doing it, I wanted to make it as > easy on myself as possible > * It was designed for documentation specifically so took care of all the > things like inter-page navigation, etc. > * It helped us get from Confluence to our current system quickly > It had some drawbacks, though: > * It wasted a lot of space on the page > * The theme was built for Markdown files, so did not take advantage of the > features of the {{jekyll-asciidoc}} plugin we use (the in-page TOC being one > big example - the plugin could create it at build time, but the theme > included JS to do it as the page loads, so we use the JS) > * It had a lot of JS and overlapping CSS files. While it used Bootstrap it > used a customized CSS on top of it for theming that made modifications > complex (it was hard to figure out how exactly a change would behave) > * With all the stuff I'd changed in my bumbling way just to get things to > work back then, I broke a lot of the stuff Bootstrap is supposed to give us > in terms of responsiveness and making the Guide usable even on smaller screen > sizes. > After upgrading the Asciidoctor components in SOLR-12786 and stopping the PDF > (SOLR-13782), I wanted to try to set us up for a more flexible system. We > need it for things like Joel's work on the visual guide for streaming > expressions (SOLR-13105), and in order to implement other ideas we might have > on how to present information in the future. > I view this issue as a phase 1 of an overall redesign that I've already > started in a local branch. I'll explain in a comment the changes I've already > made, and will use this issue to create and push a branch where we can > discuss in more detail. > Phase 1 here will be under-the-hood CSS/JS changes + overall page layout > changes. > Phase 2 (issue TBD) will be a wholesale re-organization of all the pages of > the Guide. > Phase 3 (issue TBD) will explore moving us from Jekyll to another static site > generator that is better suited for our content format, file types, and build > conventions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14173) Ref Guide Redesign
[ https://issues.apache.org/jira/browse/SOLR-14173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093956#comment-17093956 ] David Smiley commented on SOLR-14173: - You could share on the user-list to get more input. > Ref Guide Redesign > -- > > Key: SOLR-14173 > URL: https://issues.apache.org/jira/browse/SOLR-14173 > Project: Solr > Issue Type: Improvement > Components: documentation >Reporter: Cassandra Targett >Assignee: Cassandra Targett >Priority: Major > Attachments: SOLR-14173.patch, blue-left-nav.png, gray-left-nav.png > > > The current design of the Ref Guide was essentially copied from a > Jekyll-based documentation theme > (https://idratherbewriting.com/documentation-theme-jekyll/), which had a > couple important benefits for that time: > * It was well-documented and since I had little experience with Jekyll and > its Liquid templates and since I was the one doing it, I wanted to make it as > easy on myself as possible > * It was designed for documentation specifically so took care of all the > things like inter-page navigation, etc. > * It helped us get from Confluence to our current system quickly > It had some drawbacks, though: > * It wasted a lot of space on the page > * The theme was built for Markdown files, so did not take advantage of the > features of the {{jekyll-asciidoc}} plugin we use (the in-page TOC being one > big example - the plugin could create it at build time, but the theme > included JS to do it as the page loads, so we use the JS) > * It had a lot of JS and overlapping CSS files. While it used Bootstrap it > used a customized CSS on top of it for theming that made modifications > complex (it was hard to figure out how exactly a change would behave) > * With all the stuff I'd changed in my bumbling way just to get things to > work back then, I broke a lot of the stuff Bootstrap is supposed to give us > in terms of responsiveness and making the Guide usable even on smaller screen > sizes. > After upgrading the Asciidoctor components in SOLR-12786 and stopping the PDF > (SOLR-13782), I wanted to try to set us up for a more flexible system. We > need it for things like Joel's work on the visual guide for streaming > expressions (SOLR-13105), and in order to implement other ideas we might have > on how to present information in the future. > I view this issue as a phase 1 of an overall redesign that I've already > started in a local branch. I'll explain in a comment the changes I've already > made, and will use this issue to create and push a branch where we can > discuss in more detail. > Phase 1 here will be under-the-hood CSS/JS changes + overall page layout > changes. > Phase 2 (issue TBD) will be a wholesale re-organization of all the pages of > the Guide. > Phase 3 (issue TBD) will explore moving us from Jekyll to another static site > generator that is better suited for our content format, file types, and build > conventions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9328) SortingGroupHead to reuse DocValues
[ https://issues.apache.org/jira/browse/LUCENE-9328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093959#comment-17093959 ] Alan Woodward commented on LUCENE-9328: --- Thanks [~mkhl], I'll have a proper look tomorrow. The GroupHeads don't call advanceExact(), but they use FieldComparators which do; and some of them do some very funky stuff with their iterators, eg see ToParentBlockJoinSortField#getXXXComparator > SortingGroupHead to reuse DocValues > --- > > Key: LUCENE-9328 > URL: https://issues.apache.org/jira/browse/LUCENE-9328 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/grouping >Reporter: Mikhail Khludnev >Assignee: Mikhail Khludnev >Priority: Minor > Attachments: LUCENE-9328.patch, LUCENE-9328.patch, LUCENE-9328.patch > > Time Spent: 10m > Remaining Estimate: 0h > > That's why > https://issues.apache.org/jira/browse/LUCENE-7701?focusedCommentId=17084365&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17084365 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14440) Provide Certificate Authentication Plugin
Mike Drob created SOLR-14440: Summary: Provide Certificate Authentication Plugin Key: SOLR-14440 URL: https://issues.apache.org/jira/browse/SOLR-14440 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: security Reporter: Mike Drob As described in [https://issues.apache.org/jira/browse/SOLR-4407?focusedCommentId=14308429&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14308429|http://example.com] on SOLR-4407, while we support Client SSL certificates we do not have a way to use them with authentication and authorization in an end-to-end fashion. Specifically, we don't have an easy (or any?) way to load the certificate subject via a user principal into the AuthorizationContext. The work in SOLR-10814 would also be good here, since the subject can have much more than just the CN, for example it can have locations and organizational units. {{C=US, ST=California, L=San Francisco, O=Wikimedia Foundation, Inc., CN=*.wikipedia.org}} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14440) Provide Certificate Authentication Plugin
[ https://issues.apache.org/jira/browse/SOLR-14440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated SOLR-14440: - Description: As described in [this comment|https://issues.apache.org/jira/browse/SOLR-4407?focusedCommentId=14308429&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14308429] on SOLR-4407, while we support Client SSL certificates we do not have a way to use them with authentication and authorization in an end-to-end fashion. Specifically, we don't have an easy (or any?) way to load the certificate subject via a user principal into the AuthorizationContext. The work in SOLR-10814 would also be good here, since the subject can have much more than just the CN, for example it can have locations and organizational units. {{C=US, ST=California, L=San Francisco, O=Wikimedia Foundation, Inc., CN=*.wikipedia.org}} was: As described in [https://issues.apache.org/jira/browse/SOLR-4407?focusedCommentId=14308429&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14308429|http://example.com] on SOLR-4407, while we support Client SSL certificates we do not have a way to use them with authentication and authorization in an end-to-end fashion. Specifically, we don't have an easy (or any?) way to load the certificate subject via a user principal into the AuthorizationContext. The work in SOLR-10814 would also be good here, since the subject can have much more than just the CN, for example it can have locations and organizational units. {{C=US, ST=California, L=San Francisco, O=Wikimedia Foundation, Inc., CN=*.wikipedia.org}} > Provide Certificate Authentication Plugin > - > > Key: SOLR-14440 > URL: https://issues.apache.org/jira/browse/SOLR-14440 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: security >Reporter: Mike Drob >Priority: Major > > As described in [this > comment|https://issues.apache.org/jira/browse/SOLR-4407?focusedCommentId=14308429&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14308429] > on SOLR-4407, while we support Client SSL certificates we do not have a way > to use them with authentication and authorization in an end-to-end fashion. > Specifically, we don't have an easy (or any?) way to load the certificate > subject via a user principal into the AuthorizationContext. > The work in SOLR-10814 would also be good here, since the subject can have > much more than just the CN, for example it can have locations and > organizational units. {{C=US, ST=California, L=San Francisco, O=Wikimedia > Foundation, Inc., CN=*.wikipedia.org}} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14428) FuzzyQuery has severe memory usage in 8.5
[ https://issues.apache.org/jira/browse/SOLR-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093979#comment-17093979 ] David Smiley commented on SOLR-14428: - BTW This issue should probably be a Lucene JIRA issue but let's see where we go with this thread further. This is a tricky issue with no easy solutions. I think I lean towards a SoftReference providing the best trade-off. Query objects ought to be light-weight in general, so in that respect, the change to FuzzyQuery is disappointing. Maybe the choice of FuzzyQuery computing the automaton up-front should be an option? It could be a boolean; that's the simplest thing. Or imagine a Function supplied in the constructor that is potentially memo-izable. Someone could even plug in a cache. This is all a bit complicated and the query parser is in charge of the choice. No parsers have an option for this hypothetical choice yet. Or maybe we just accept that sometimes, queries can be big, and thus Lucene users like Solr just have to deal with it. If a Query is beyond the size of some threshold, maybe we don't cache it by default unless the user explicitly chooses to. That's a nice generic solution. The QueryResultKey could cache by the original string input instead of the Query object, thus it wouldn't be affected. This would short-circuit query parsing and be a performance benefit as well? > FuzzyQuery has severe memory usage in 8.5 > - > > Key: SOLR-14428 > URL: https://issues.apache.org/jira/browse/SOLR-14428 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 8.5, 8.5.1 >Reporter: Colvin Cowie >Assignee: Andrzej Bialecki >Priority: Major > Attachments: FuzzyHammer.java, SOLR-14428-WeakReferences.patch, > image-2020-04-23-09-18-06-070.png, image-2020-04-24-20-09-31-179.png, > screenshot-2.png, screenshot-3.png, screenshot-4.png > > Time Spent: 10m > Remaining Estimate: 0h > > I sent this to the mailing list > I'm moving from 8.3.1 to 8.5.1, and started getting Out Of Memory Errors > while running our normal tests. After profiling it was clear that the > majority of the heap was allocated through FuzzyQuery. > LUCENE-9068 moved construction of the automata from the FuzzyTermsEnum to the > FuzzyQuery's constructor. > I created a little test ( [^FuzzyHammer.java] ) that fires off fuzzy queries > from random UUID strings for 5 minutes > {code} > FIELD_NAME + ":" + UUID.randomUUID().toString().replace("-", "") + "~2" > {code} > When running against a vanilla Solr 8.31 and 8.4.1 there is no problem, while > the memory usage has increased drastically on 8.5.0 and 8.5.1. > Comparison of heap usage while running the attached test against Solr 8.3.1 > and 8.5.1 with a single (empty) shard and 4GB heap: > !image-2020-04-23-09-18-06-070.png! > And with 4 shards on 8.4.1 and 8.5.0: > !screenshot-2.png! > I'm guessing that the memory might be being leaked if the FuzzyQuery objects > are referenced from the cache, while the FuzzyTermsEnum would not have been. > Query Result Cache on 8.5.1: > !screenshot-3.png! > ~316mb in the cache > QRC on 8.3.1 > !screenshot-4.png! > <1mb > With an empty cache, running this query > _field_s:e41848af85d24ac197c71db6888e17bc~2_ results in the following memory > allocation > {noformat} > 8.3.1: CACHE.searcher.queryResultCache.ramBytesUsed: 1520 > 8.5.1: CACHE.searcher.queryResultCache.ramBytesUsed:648855 > {noformat} > ~1 gives 98253 and ~0 gives 6339 on 8.5.1. 8.3.1 is constant at 1520 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob opened a new pull request #1463: SOLR-14440 Cert Auth plugin
madrob opened a new pull request #1463: URL: https://github.com/apache/lucene-solr/pull/1463 https://issues.apache.org/jira/browse/SOLR-14440 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14440) Provide Certificate Authentication Plugin
[ https://issues.apache.org/jira/browse/SOLR-14440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093981#comment-17093981 ] Mike Drob commented on SOLR-14440: -- I've created a PR that doesn't have many frills. If the approach looks amenable to folks, I'll also write up some docs for the ref guide next. > Provide Certificate Authentication Plugin > - > > Key: SOLR-14440 > URL: https://issues.apache.org/jira/browse/SOLR-14440 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: security >Reporter: Mike Drob >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > As described in [this > comment|https://issues.apache.org/jira/browse/SOLR-4407?focusedCommentId=14308429&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14308429] > on SOLR-4407, while we support Client SSL certificates we do not have a way > to use them with authentication and authorization in an end-to-end fashion. > Specifically, we don't have an easy (or any?) way to load the certificate > subject via a user principal into the AuthorizationContext. > The work in SOLR-10814 would also be good here, since the subject can have > much more than just the CN, for example it can have locations and > organizational units. {{C=US, ST=California, L=San Francisco, O=Wikimedia > Foundation, Inc., CN=*.wikipedia.org}} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14440) Provide Certificate Authentication Plugin
[ https://issues.apache.org/jira/browse/SOLR-14440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093992#comment-17093992 ] Mike Drob commented on SOLR-14440: -- I _think_ we don't need to do much in the way of validation of the certificate here, because we already have those settings in the Jetty container. Everything like validating the cert chain up until a trusted source and validating the peer name/IP address listed in the cert should be taken care of before we even see the certificate? > Provide Certificate Authentication Plugin > - > > Key: SOLR-14440 > URL: https://issues.apache.org/jira/browse/SOLR-14440 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: security >Reporter: Mike Drob >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > As described in [this > comment|https://issues.apache.org/jira/browse/SOLR-4407?focusedCommentId=14308429&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14308429] > on SOLR-4407, while we support Client SSL certificates we do not have a way > to use them with authentication and authorization in an end-to-end fashion. > Specifically, we don't have an easy (or any?) way to load the certificate > subject via a user principal into the AuthorizationContext. > The work in SOLR-10814 would also be good here, since the subject can have > much more than just the CN, for example it can have locations and > organizational units. {{C=US, ST=California, L=San Francisco, O=Wikimedia > Foundation, Inc., CN=*.wikipedia.org}} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13289) Support for BlockMax WAND
[ https://issues.apache.org/jira/browse/SOLR-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094032#comment-17094032 ] Tomas Eduardo Fernandez Lobbe commented on SOLR-13289: -- The code in [the PR|https://github.com/apache/lucene-solr/pull/1456] is taking shape, please take a look at it if you are interested in the change. Specially, please take a look at the API. I decided to include in the response the {{hitCountRelation}}, similar to what Lucene responds (which can be {{EQUAL_TO}} or {{GREATER_OR_EQUAL_TO}}). I initially thought about just including a boolean to say if the hit count was exact or an approximation, but I decided to include the relation: 1) To be more precise 2) To be open to possible changes in Lucene, which could give more values to this relation I had to ignore a Javabin "forward compatibilty" test, since the binaries change and break the test, however, I believe the changes are compatible ("forward" and "backward"). I left the current default of minExactHits to be MAX_INT (count all) for now. I plan to have a new issue to have that discussion separately. > Support for BlockMax WAND > - > > Key: SOLR-13289 > URL: https://issues.apache.org/jira/browse/SOLR-13289 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Attachments: SOLR-13289.patch, SOLR-13289.patch > > Time Spent: 50m > Remaining Estimate: 0h > > LUCENE-8135 introduced BlockMax WAND as a major speed improvement. Need to > expose this via Solr. When enabled, the numFound returned will not be exact. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-13289) Support for BlockMax WAND
[ https://issues.apache.org/jira/browse/SOLR-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomas Eduardo Fernandez Lobbe reassigned SOLR-13289: Assignee: Tomas Eduardo Fernandez Lobbe (was: Ishan Chattopadhyaya) > Support for BlockMax WAND > - > > Key: SOLR-13289 > URL: https://issues.apache.org/jira/browse/SOLR-13289 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Tomas Eduardo Fernandez Lobbe >Priority: Major > Attachments: SOLR-13289.patch, SOLR-13289.patch > > Time Spent: 50m > Remaining Estimate: 0h > > LUCENE-8135 introduced BlockMax WAND as a major speed improvement. Need to > expose this via Solr. When enabled, the numFound returned will not be exact. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7788) fail precommit on unparameterised log messages and examine for wasted work/objects
[ https://issues.apache.org/jira/browse/LUCENE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094054#comment-17094054 ] ASF subversion and git services commented on LUCENE-7788: - Commit 960610a615a2893062b1df3ab1595d8fef6bcdb4 in lucene-solr's branch refs/heads/master from Erick Erickson [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=960610a ] LUCENE-7788: fail precommit on unparameterised log messages and examine for wasted work/objects > fail precommit on unparameterised log messages and examine for wasted > work/objects > -- > > Key: LUCENE-7788 > URL: https://issues.apache.org/jira/browse/LUCENE-7788 > Project: Lucene - Core > Issue Type: Task >Reporter: Christine Poerschke >Assignee: Erick Erickson >Priority: Minor > Attachments: LUCENE-7788.patch, LUCENE-7788.patch, gradle_only.patch, > gradle_only.patch > > Time Spent: 50m > Remaining Estimate: 0h > > SOLR-10415 would be removing existing unparameterised log.trace messages use > and once that is in place then this ticket's one-line change would be for > 'ant precommit' to reject any future unparameterised log.trace message use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7788) fail precommit on unparameterised log messages and examine for wasted work/objects
[ https://issues.apache.org/jira/browse/LUCENE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094057#comment-17094057 ] ASF subversion and git services commented on LUCENE-7788: - Commit 91999e134680da4d184f4129cba8ffd23ee43443 in lucene-solr's branch refs/heads/branch_8x from Erick Erickson [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=91999e1 ] LUCENE-7788: fail precommit on unparameterised log messages and examine for wasted work/objects > fail precommit on unparameterised log messages and examine for wasted > work/objects > -- > > Key: LUCENE-7788 > URL: https://issues.apache.org/jira/browse/LUCENE-7788 > Project: Lucene - Core > Issue Type: Task >Reporter: Christine Poerschke >Assignee: Erick Erickson >Priority: Minor > Attachments: LUCENE-7788.patch, LUCENE-7788.patch, gradle_only.patch, > gradle_only.patch > > Time Spent: 50m > Remaining Estimate: 0h > > SOLR-10415 would be removing existing unparameterised log.trace messages use > and once that is in place then this ticket's one-line change would be for > 'ant precommit' to reject any future unparameterised log.trace message use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-12642) SolrCmdDistributor should send updates in batch when use Http2SolrClient?
[ https://issues.apache.org/jira/browse/SOLR-12642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094072#comment-17094072 ] Cao Manh Dat commented on SOLR-12642: - Above numbers are not correct. Basically above benchmark was performed on a buggy Jetty version. During indexing, leader failed to send updates to replicas quite frequently without networking issue, therefore replicas were put into recovery mode. Replicas did not do indexing much so that result the gain in CPU time and garbage generated. On Leader's side the CPU time is 50% more since it need to stream index files back to replicas during recovery. > SolrCmdDistributor should send updates in batch when use Http2SolrClient? > - > > Key: SOLR-12642 > URL: https://issues.apache.org/jira/browse/SOLR-12642 > Project: Solr > Issue Type: Improvement >Reporter: Cao Manh Dat >Assignee: Cao Manh Dat >Priority: Major > Fix For: 8.0 > > Attachments: http2-branch.log, master-branch.log > > > In the past, batch updates are sent in a single stream from the leader, the > replica will create a single thread to parse all the updates. For the > simplicity of {{SOLR-12605}}, the leader is now sending individual updates to > replicas, therefore they are now parsing updates in different threads which > increase the usage of memory and CPU. > In the past, this is an unacceptable approach, because, for every update, we > must create different connections to replicas. But with the support of > HTTP/2, all updates will be sent in a single connection from leader to a > replica. Therefore the cost is not as high as it used to be. > On the other hand, sending individual updates will improve the indexing > performance and better error-handling for failures of a single update in a > batch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on pull request #1463: SOLR-14440 Cert Auth plugin
noblepaul commented on pull request #1463: URL: https://github.com/apache/lucene-solr/pull/1463#issuecomment-620333656 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14433) Improve default metrics collected by SolrShardReporter
[ https://issues.apache.org/jira/browse/SOLR-14433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094127#comment-17094127 ] ASF subversion and git services commented on SOLR-14433: Commit daeaffa9d7617031940b2e81b783a7f101c21a45 in lucene-solr's branch refs/heads/master from David Smiley [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=daeaffa ] SOLR-14433: Improve SolrShardReporter default metrics list (#1453) Now includes TLOG and UPDATE./update. These were small bugs to begin with but from user perspective this is an incremental improvement. > Improve default metrics collected by SolrShardReporter > -- > > Key: SOLR-14433 > URL: https://issues.apache.org/jira/browse/SOLR-14433 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: metrics >Reporter: David Smiley >Assignee: David Smiley >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > SolrShardReporter's default metric filters have two problems: > * "Tlog.\*" should be "TLOG.\*" (a bug) > * "UPDATE > ./update/.\*requests" should be "UPDATE > ./update.\*requests" (notice removal of one '/') > Today, the first was fixed and tagged to the issue that incorrectly made this > change – SOLR-12690. What remains is the other. > CC [~ab] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14433) Improve default metrics collected by SolrShardReporter
[ https://issues.apache.org/jira/browse/SOLR-14433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094130#comment-17094130 ] ASF subversion and git services commented on SOLR-14433: Commit 489278b87ab361a26860e4f7649106444255eccf in lucene-solr's branch refs/heads/branch_8x from David Smiley [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=489278b ] SOLR-14433: Improve SolrShardReporter default metrics list (#1453) Now includes TLOG and UPDATE./update. These were small bugs to begin with but from user perspective this is an incremental improvement. (cherry picked from commit daeaffa9d7617031940b2e81b783a7f101c21a45) > Improve default metrics collected by SolrShardReporter > -- > > Key: SOLR-14433 > URL: https://issues.apache.org/jira/browse/SOLR-14433 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: metrics >Reporter: David Smiley >Assignee: David Smiley >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > SolrShardReporter's default metric filters have two problems: > * "Tlog.\*" should be "TLOG.\*" (a bug) > * "UPDATE > ./update/.\*requests" should be "UPDATE > ./update.\*requests" (notice removal of one '/') > Today, the first was fixed and tagged to the issue that incorrectly made this > change – SOLR-12690. What remains is the other. > CC [~ab] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14433) Improve default metrics collected by SolrShardReporter
[ https://issues.apache.org/jira/browse/SOLR-14433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley resolved SOLR-14433. - Fix Version/s: 8.6 Resolution: Fixed > Improve default metrics collected by SolrShardReporter > -- > > Key: SOLR-14433 > URL: https://issues.apache.org/jira/browse/SOLR-14433 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: metrics >Reporter: David Smiley >Assignee: David Smiley >Priority: Minor > Fix For: 8.6 > > Time Spent: 20m > Remaining Estimate: 0h > > SolrShardReporter's default metric filters have two problems: > * "Tlog.\*" should be "TLOG.\*" (a bug) > * "UPDATE > ./update/.\*requests" should be "UPDATE > ./update.\*requests" (notice removal of one '/') > Today, the first was fixed and tagged to the issue that incorrectly made this > change – SOLR-12690. What remains is the other. > CC [~ab] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org