[GitHub] [lucene-solr] jpountz opened a new pull request #1882: LUCENE-9529: Track dirtiness of stored fields via a number of docs, not chunks.
jpountz opened a new pull request #1882: URL: https://github.com/apache/lucene-solr/pull/1882 The problem of tracking dirtiness via numbers of chunks is that larger chunks make stored fields readers more likely to be considered dirty, so I'm trying to work around it by tracking numbers of docs instead. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9529) Larger stored fields block sizes mean we're more likely to disable optimized bulk merging
[ https://issues.apache.org/jira/browse/LUCENE-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197465#comment-17197465 ] Adrien Grand commented on LUCENE-9529: -- I played with this idea. On a synthetic benchmark that indexes 1M docs with only one stored field per doc of about 2kB and maxBufferedDocs=1000, indexing time went from 13.5 secs to 12 secs. > Larger stored fields block sizes mean we're more likely to disable optimized > bulk merging > - > > Key: LUCENE-9529 > URL: https://issues.apache.org/jira/browse/LUCENE-9529 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Whenever possible when merging stored fields, Lucene tries to copy the > compressed data instead of decompressing the source segment to then > re-compressing in the destination segment. A problem with this approach is > that if some blocks are incomplete (typically the last block of a segment) > then it remains incomplete in the destination segment too, and if we do it > for too long we end up with a bad compression ratio. So Lucene keeps track of > these incomplete blocks, and makes sure to keep a ratio of incomplete blocks > below 1%. > But as we increased the block size, it has become more likely to have a high > ratio of incomplete blocks. E.g. if you have a segment with 1MB of stored > fields, with 16kB blocks like before, you have 63 complete blocks and 1 > incomplete block, or 1.6%. But now with ~512kB blocks, you have one complete > block and 1 incomplete block, ie. 50%. > I'm not sure how to fix it or even whether it should be fixed but wanted to > open an issue to track this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14824) Simplify set of Ref Guide build parameters
[
https://issues.apache.org/jira/browse/SOLR-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197481#comment-17197481
]
Uwe Schindler commented on SOLR-14824:
--
bq. (also, FWIW, trying to call toURI on project(':lucene').buildDir.toPath()
or project(':lucene').buildDir.toPath().resolve('documentation') kept giving me
a weird groovy error i couldn't make heads of tails of:
It's toUri(), my fault:
https://docs.oracle.com/javase/7/docs/api/java/nio/file/Path.html#toUri()
> Simplify set of Ref Guide build parameters
> --
>
> Key: SOLR-14824
> URL: https://issues.apache.org/jira/browse/SOLR-14824
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Build, documentation
>Reporter: Cassandra Targett
>Assignee: Chris M. Hostetter
>Priority: Minor
> Attachments: SOLR-14824.patch, SOLR-14824.patch
>
>
> While trying to solve LUCENE-9495, I thought it might be a good idea to try
> to simplify the set of variables and properties used during the Ref Guide
> build.
> There are 3 areas to work on:
> 1. Remove the "barebones-html" build. With Gradle the build is self-contained
> and {{gradlew check}} and {{gradle precommit}} could just build the full docs
> and check them.
> 2. Remove some properties that only existed for a hypothetical need related
> to the now-removed PDF.
> 3. Change remaining properties to be defined directly in build.gradle instead
> of relying on ant properties functionality.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14824) Simplify set of Ref Guide build parameters
[
https://issues.apache.org/jira/browse/SOLR-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197481#comment-17197481
]
Uwe Schindler edited comment on SOLR-14824 at 9/17/20, 8:13 AM:
bq. (also, FWIW, trying to call toURI on project(':lucene').buildDir.toPath()
or project(':lucene').buildDir.toPath().resolve('documentation') kept giving me
a weird groovy error i couldn't make heads of tails of:
It's toUri(), my fault:
[https://docs.oracle.com/javase/7/docs/api/java/nio/file/Path.html#toUri()]
The Gradle error messages during compilation are often a bit crazy, not sure
why. Often it shows a followup error instead of the original error.
was (Author: thetaphi):
bq. (also, FWIW, trying to call toURI on project(':lucene').buildDir.toPath()
or project(':lucene').buildDir.toPath().resolve('documentation') kept giving me
a weird groovy error i couldn't make heads of tails of:
It's toUri(), my fault:
https://docs.oracle.com/javase/7/docs/api/java/nio/file/Path.html#toUri()
> Simplify set of Ref Guide build parameters
> --
>
> Key: SOLR-14824
> URL: https://issues.apache.org/jira/browse/SOLR-14824
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Build, documentation
>Reporter: Cassandra Targett
>Assignee: Chris M. Hostetter
>Priority: Minor
> Attachments: SOLR-14824.patch, SOLR-14824.patch
>
>
> While trying to solve LUCENE-9495, I thought it might be a good idea to try
> to simplify the set of variables and properties used during the Ref Guide
> build.
> There are 3 areas to work on:
> 1. Remove the "barebones-html" build. With Gradle the build is self-contained
> and {{gradlew check}} and {{gradle precommit}} could just build the full docs
> and check them.
> 2. Remove some properties that only existed for a hypothetical need related
> to the now-removed PDF.
> 3. Change remaining properties to be defined directly in build.gradle instead
> of relying on ant properties functionality.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9445) Expose new case insensitive RegExpQuery support in QueryParser
[ https://issues.apache.org/jira/browse/LUCENE-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197483#comment-17197483 ] Mark Harwood commented on LUCENE-9445: -- Sure, Dawid. I'll need to make some changes to throw errors anyway if we agree stricter parsing is the way forward. > Expose new case insensitive RegExpQuery support in QueryParser > -- > > Key: LUCENE-9445 > URL: https://issues.apache.org/jira/browse/LUCENE-9445 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser >Reporter: Mark Harwood >Assignee: Mark Harwood >Priority: Minor > Time Spent: 1.5h > Remaining Estimate: 0h > > LUCENE-9386 added a case insensitive matching option to RegExpQuery. > This proposal is to extend the QueryParser syntax to allow for an optional > `i` (case Insensitive) flag to appear on the end of regular expressions e.g. > /Foo/i > > This is regex syntax supported by a number of programming languages. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9530) Clean up javacc generation script (gradle)
Dawid Weiss created LUCENE-9530: --- Summary: Clean up javacc generation script (gradle) Key: LUCENE-9530 URL: https://issues.apache.org/jira/browse/LUCENE-9530 Project: Lucene - Core Issue Type: Sub-task Reporter: Dawid Weiss Assignee: Dawid Weiss -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss opened a new pull request #1883: LUCENE-9530: cleaned up javacc gradle generation scripts.
dweiss opened a new pull request #1883: URL: https://github.com/apache/lucene-solr/pull/1883 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9445) Expose new case insensitive RegExpQuery support in QueryParser
[ https://issues.apache.org/jira/browse/LUCENE-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197496#comment-17197496 ] Dawid Weiss commented on LUCENE-9445: - Thanks Mark. I think most query parsers offering "advanced" features should be stricter - they should throw errors when they detect a syntax problem. See my e-mail to the mailing list about the current behavior of "/foo/~2"... > Expose new case insensitive RegExpQuery support in QueryParser > -- > > Key: LUCENE-9445 > URL: https://issues.apache.org/jira/browse/LUCENE-9445 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser >Reporter: Mark Harwood >Assignee: Mark Harwood >Priority: Minor > Time Spent: 1.5h > Remaining Estimate: 0h > > LUCENE-9386 added a case insensitive matching option to RegExpQuery. > This proposal is to extend the QueryParser syntax to allow for an optional > `i` (case Insensitive) flag to appear on the end of regular expressions e.g. > /Foo/i > > This is regex syntax supported by a number of programming languages. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss merged pull request #1883: LUCENE-9530: cleaned up javacc gradle generation scripts.
dweiss merged pull request #1883: URL: https://github.com/apache/lucene-solr/pull/1883 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9530) Clean up javacc generation script (gradle)
[ https://issues.apache.org/jira/browse/LUCENE-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Weiss resolved LUCENE-9530. - Fix Version/s: master (9.0) Resolution: Fixed > Clean up javacc generation script (gradle) > -- > > Key: LUCENE-9530 > URL: https://issues.apache.org/jira/browse/LUCENE-9530 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Trivial > Fix For: master (9.0) > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9527) Upgrade javacc to 7.0.5
[ https://issues.apache.org/jira/browse/LUCENE-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Weiss updated LUCENE-9527: Summary: Upgrade javacc to 7.0.5 (was: Upgrade javacc and regeneration tasks) > Upgrade javacc to 7.0.5 > --- > > Key: LUCENE-9527 > URL: https://issues.apache.org/jira/browse/LUCENE-9527 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9530) Clean up javacc generation script (gradle)
[ https://issues.apache.org/jira/browse/LUCENE-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197518#comment-17197518 ] ASF subversion and git services commented on LUCENE-9530: - Commit 4f344cb0d4087171f59cc3c22265237bbeec50a4 in lucene-solr's branch refs/heads/master from Dawid Weiss [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=4f344cb ] LUCENE-9530: cleaned up javacc gradle generation scripts. (#1883) * LUCENE-9530: cleaned up gradle javacc generation/ tweaks script so that it's consistent across runs. Removed ant remnants. > Clean up javacc generation script (gradle) > -- > > Key: LUCENE-9530 > URL: https://issues.apache.org/jira/browse/LUCENE-9530 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Trivial > Fix For: master (9.0) > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9530) Clean up javacc generation script (gradle)
[ https://issues.apache.org/jira/browse/LUCENE-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197517#comment-17197517 ] ASF subversion and git services commented on LUCENE-9530: - Commit 4f344cb0d4087171f59cc3c22265237bbeec50a4 in lucene-solr's branch refs/heads/master from Dawid Weiss [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=4f344cb ] LUCENE-9530: cleaned up javacc gradle generation scripts. (#1883) * LUCENE-9530: cleaned up gradle javacc generation/ tweaks script so that it's consistent across runs. Removed ant remnants. > Clean up javacc generation script (gradle) > -- > > Key: LUCENE-9530 > URL: https://issues.apache.org/jira/browse/LUCENE-9530 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Trivial > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9445) Expose new case insensitive RegExpQuery support in QueryParser
[ https://issues.apache.org/jira/browse/LUCENE-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197522#comment-17197522 ] Mark Harwood commented on LUCENE-9445: -- Yikes. There look to be a number of silent failures currently and some extra strictness is welcome > Expose new case insensitive RegExpQuery support in QueryParser > -- > > Key: LUCENE-9445 > URL: https://issues.apache.org/jira/browse/LUCENE-9445 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser >Reporter: Mark Harwood >Assignee: Mark Harwood >Priority: Minor > Time Spent: 1.5h > Remaining Estimate: 0h > > LUCENE-9386 added a case insensitive matching option to RegExpQuery. > This proposal is to extend the QueryParser syntax to allow for an optional > `i` (case Insensitive) flag to appear on the end of regular expressions e.g. > /Foo/i > > This is regex syntax supported by a number of programming languages. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14656) Deprecate current autoscaling framework, remove from master
[ https://issues.apache.org/jira/browse/SOLR-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197527#comment-17197527 ] Andrzej Bialecki commented on SOLR-14656: - [~ichattopadhyaya] this issue should be a Blocker for 8.7 to add deprecation notices in code and in the Ref Guide. > Deprecate current autoscaling framework, remove from master > --- > > Key: SOLR-14656 > URL: https://issues.apache.org/jira/browse/SOLR-14656 > Project: Solr > Issue Type: Improvement >Reporter: Ishan Chattopadhyaya >Priority: Major > Attachments: Screenshot from 2020-07-18 07-49-01.png > > Time Spent: 3h > Remaining Estimate: 0h > > The autoscaling framework is being re-designed in SOLR-14613 (SIP: > https://cwiki.apache.org/confluence/display/SOLR/SIP-8+Autoscaling+policy+engine+V2). > The current autoscaling framework is very inefficient, improperly designed > and too bloated and doesn't receive the level of support we aspire to provide > for all components that we ship. > This issue is to deprecate current autoscaling framework in 8x, so we can > focus on the new autoscaling framework afresh. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7882) Maybe expression compiler should cache recently compiled expressions?
[
https://issues.apache.org/jira/browse/LUCENE-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197533#comment-17197533
]
Uwe Schindler commented on LUCENE-7882:
---
Hi [~zhai7631],
bq. Thank you for making this fantastic PR!
Thanks! :-)
bq. We've tested this PR by compiling with JDK11 and running with JDK15
(because of some reason it's not easy to compile our service with JDK15
directly).
That's perfectly fine. There's no reason to complie anything with JDK 15
(neither Lucene nor your code), because the defineHiddenClass() is used
dynamically using MethodHandles if the expressions compiler detects this API to
be available. So it's perfectly valid to create and compile with JDK 11, as
long as you benchmark with 15.
bq. But because of the reason I mentioned above, it seems that we don't have
enough expressions compilation now to observe the difference with or without
the PR.
That's fine. I was hoping that maybe you or [~mikemccand] have some isolated
benchmark that creates many expressions with hundreds of threads in parallel :-)
I made this PR more for educational purposes (I just wanted to try out the new
API and figure out if it helps us). [~rcmuir] also wanted to do some testing
with the old code he used back in 2013 when we checked the "separate
classloader per expression" code (which is also mentioned in the JEP about
defineHiddenClass).
I have not yet committed that PR because of the small problem: the stack frame
of the expression gets lost, as it's hidden by the JVM. In Robert's and my
opinion, it's not so important to have the stackframe of
CompiledExpression#evaluate() as long as users don't produce stack overflows or
anything like that in their expression. If all functions referred to in the
expression don't throw any exception, the stack frage is really not needed.
The only open question is: if an expression really fails, we no longer have the
expression source code in the stack trace, too.
> Maybe expression compiler should cache recently compiled expressions?
> -
>
> Key: LUCENE-7882
> URL: https://issues.apache.org/jira/browse/LUCENE-7882
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/expressions
>Reporter: Michael McCandless
>Assignee: Uwe Schindler
>Priority: Major
> Attachments: demo.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> I've been running search performance tests using a simple expression
> ({{_score + ln(1000+unit_sales)}}) for sorting and hit this odd bottleneck:
> {noformat}
> "pool-1-thread-30" #70 prio=5 os_prio=0 tid=0x7eea7000a000 nid=0x1ea8a
> waiting for monitor entry [0x7eea867dd000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.lucene.expressions.js.JavascriptCompiler$CompiledExpression.evaluate(_score
> + ln(1000+unit_sales))
> at
> org.apache.lucene.expressions.ExpressionFunctionValues.doubleValue(ExpressionFunctionValues.java:49)
> at
> com.amazon.lucene.OrderedVELeafCollector.collectInternal(OrderedVELeafCollector.java:123)
> at
> com.amazon.lucene.OrderedVELeafCollector.collect(OrderedVELeafCollector.java:108)
> at
> org.apache.lucene.search.MultiCollectorManager$Collectors$LeafCollectors.collect(MultiCollectorManager.java:102)
> at
> org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:241)
> at
> org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:184)
> at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:658)
> at org.apache.lucene.search.IndexSearcher$5.call(IndexSearcher.java:600)
> at org.apache.lucene.search.IndexSearcher$5.call(IndexSearcher.java:597)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I couldn't see any {{synchronized}} in the sources here, so I'm not sure
> which object monitor it's blocked on.
> I was accidentally compiling a new expression for every query, and that
> bottleneck would cause overall QPS to slow down drastically (~4X slower after
> ~1 hour of redline tests), as if the JVM is getting slower and slower to
> evaluate each expression the more expressions I had compiled.
> I tested JDK 9-ea and it also kept slowing down over time as the performance
> test ran.
> Maybe we should put a small cache in front of the expressions compiler to
> make it less trappy? Or maybe we can get to the root cause of
[jira] [Commented] (LUCENE-7882) Maybe expression compiler should cache recently compiled expressions?
[
https://issues.apache.org/jira/browse/LUCENE-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197539#comment-17197539
]
Uwe Schindler commented on LUCENE-7882:
---
bq. The only open question is: if an expression really fails, we no longer have
the expression source code in the stack trace, too.
I am reeferring to the first stack frame here {{at
org.apache.lucene.expressions.js.JavascriptCompiler$CompiledExpression.evaluate(_score
+ ln(1000+unit_sales))}}. The nice source code of the expression won't be seen
with hiddenClasses enabled.
{noformat}
"pool-1-thread-30" #70 prio=5 os_prio=0 tid=0x7eea7000a000 nid=0x1ea8a
waiting for monitor entry [0x7eea867dd000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.lucene.expressions.js.JavascriptCompiler$CompiledExpression.evaluate(_score
+ ln(1000+unit_sales))
at
org.apache.lucene.expressions.ExpressionFunctionValues.doubleValue(ExpressionFunctionValues.java:49)
{noformat}
> Maybe expression compiler should cache recently compiled expressions?
> -
>
> Key: LUCENE-7882
> URL: https://issues.apache.org/jira/browse/LUCENE-7882
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/expressions
>Reporter: Michael McCandless
>Assignee: Uwe Schindler
>Priority: Major
> Attachments: demo.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> I've been running search performance tests using a simple expression
> ({{_score + ln(1000+unit_sales)}}) for sorting and hit this odd bottleneck:
> {noformat}
> "pool-1-thread-30" #70 prio=5 os_prio=0 tid=0x7eea7000a000 nid=0x1ea8a
> waiting for monitor entry [0x7eea867dd000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.lucene.expressions.js.JavascriptCompiler$CompiledExpression.evaluate(_score
> + ln(1000+unit_sales))
> at
> org.apache.lucene.expressions.ExpressionFunctionValues.doubleValue(ExpressionFunctionValues.java:49)
> at
> com.amazon.lucene.OrderedVELeafCollector.collectInternal(OrderedVELeafCollector.java:123)
> at
> com.amazon.lucene.OrderedVELeafCollector.collect(OrderedVELeafCollector.java:108)
> at
> org.apache.lucene.search.MultiCollectorManager$Collectors$LeafCollectors.collect(MultiCollectorManager.java:102)
> at
> org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:241)
> at
> org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:184)
> at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:658)
> at org.apache.lucene.search.IndexSearcher$5.call(IndexSearcher.java:600)
> at org.apache.lucene.search.IndexSearcher$5.call(IndexSearcher.java:597)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I couldn't see any {{synchronized}} in the sources here, so I'm not sure
> which object monitor it's blocked on.
> I was accidentally compiling a new expression for every query, and that
> bottleneck would cause overall QPS to slow down drastically (~4X slower after
> ~1 hour of redline tests), as if the JVM is getting slower and slower to
> evaluate each expression the more expressions I had compiled.
> I tested JDK 9-ea and it also kept slowing down over time as the performance
> test ran.
> Maybe we should put a small cache in front of the expressions compiler to
> make it less trappy? Or maybe we can get to the root cause of why the JVM
> slows down more and more, the more expressions you compile?
> I won't have time to work on this in the near future so if anyone else feels
> the itch, please scratch it!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14151) Make schema components load from packages
[
https://issues.apache.org/jira/browse/SOLR-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197543#comment-17197543
]
Noble Paul commented on SOLR-14151:
---
[~worleydl] I have committed it to master
> Make schema components load from packages
> -
>
> Key: SOLR-14151
> URL: https://issues.apache.org/jira/browse/SOLR-14151
> Project: Solr
> Issue Type: Sub-task
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Labels: packagemanager
> Fix For: 8.7
>
> Time Spent: 12h 50m
> Remaining Estimate: 0h
>
> Example:
> {code:xml}
>
>
>
>generateNumberParts="0" catenateWords="0"
> catenateNumbers="0" catenateAll="0"/>
>
>
>
>
> {code}
> * When a package is updated, the entire {{IndexSchema}} object is refreshed,
> but the SolrCore object is not reloaded
> * Any component can be prefixed with the package name
> * The semantics of loading plugins remain the same as that of the components
> in {{solrconfig.xml}}
> * Plugins can be registered using schema API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14151) Make schema components load from packages
[
https://issues.apache.org/jira/browse/SOLR-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197542#comment-17197542
]
ASF subversion and git services commented on SOLR-14151:
Commit 515608a087f81c13ad22efba72e48fc64d9d5ae7 in lucene-solr's branch
refs/heads/master from noblepaul
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=515608a ]
SOLR-14151: fixed the classloading issue
> Make schema components load from packages
> -
>
> Key: SOLR-14151
> URL: https://issues.apache.org/jira/browse/SOLR-14151
> Project: Solr
> Issue Type: Sub-task
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Labels: packagemanager
> Fix For: 8.7
>
> Time Spent: 12h 50m
> Remaining Estimate: 0h
>
> Example:
> {code:xml}
>
>
>
>generateNumberParts="0" catenateWords="0"
> catenateNumbers="0" catenateAll="0"/>
>
>
>
>
> {code}
> * When a package is updated, the entire {{IndexSchema}} object is refreshed,
> but the SolrCore object is not reloaded
> * Any component can be prefixed with the package name
> * The semantics of loading plugins remain the same as that of the components
> in {{solrconfig.xml}}
> * Plugins can be registered using schema API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14824) Simplify set of Ref Guide build parameters
[
https://issues.apache.org/jira/browse/SOLR-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197570#comment-17197570
]
Uwe Schindler commented on SOLR-14824:
--
Hi Hoss,
bq. Hmmm i don't really feel comfortable trying to tackle those changes
here/now – let's pursue this idea more in SOLR-14870 – because AFAICT those
productDir/buildDir variables return absolute file paths
For this issue I was mainly looking at the {{solrRootPath}} property that's
obviously not a relative URL path, which highly depends on structure of build
system (what happens if the CWD is different when executing the scripts in
later Gradle versions?). The series of ".." makes it completely unclear how it
is used. For the URLs to javadocs folders, I agree that's more the SOLR-14870
issue.
bq. but i'm assuming we could relativize those paths again the ref-guides own
buildDir
of course, instead of calling toUri(), you can do {{Path#relativize()}}, using
the own projects buildDir / output dir in combination with the code shown
above. As you need this as URI path later, don't forget to convert
operating-system specific slashes to forward slashes. See example here:
https://github.com/apache/lucene-solr/blob/master/gradle/documentation/render-javadoc.gradle#L500
{{def relative = pathFrom.relativize(pathTo).toString().replace(File.separator,
'/')}}
Uwe
> Simplify set of Ref Guide build parameters
> --
>
> Key: SOLR-14824
> URL: https://issues.apache.org/jira/browse/SOLR-14824
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Build, documentation
>Reporter: Cassandra Targett
>Assignee: Chris M. Hostetter
>Priority: Minor
> Attachments: SOLR-14824.patch, SOLR-14824.patch
>
>
> While trying to solve LUCENE-9495, I thought it might be a good idea to try
> to simplify the set of variables and properties used during the Ref Guide
> build.
> There are 3 areas to work on:
> 1. Remove the "barebones-html" build. With Gradle the build is self-contained
> and {{gradlew check}} and {{gradle precommit}} could just build the full docs
> and check them.
> 2. Remove some properties that only existed for a hypothetical need related
> to the now-removed PDF.
> 3. Change remaining properties to be defined directly in build.gradle instead
> of relying on ant properties functionality.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14824) Simplify set of Ref Guide build parameters
[
https://issues.apache.org/jira/browse/SOLR-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197570#comment-17197570
]
Uwe Schindler edited comment on SOLR-14824 at 9/17/20, 9:46 AM:
Hi Hoss,
bq. Hmmm i don't really feel comfortable trying to tackle those changes
here/now – let's pursue this idea more in SOLR-14870 – because AFAICT those
productDir/buildDir variables return absolute file paths
For this issue I was mainly looking at the {{solrRootPath}} property that's
obviously not a relative URL path, which highly depends on structure of build
system (what happens if the CWD is different when executing the scripts in
later Gradle versions?). The series of ".." makes it completely unclear how it
is used. For the URLs to javadocs folders, I agree that's more the SOLR-14870
issue.
bq. but i'm assuming we could relativize those paths again the ref-guides own
buildDir
of course, instead of calling toUri(), you can do {{Path#relativize()}}, using
the own projects buildDir / output dir in combination with the code shown
above. As you need this as URI path later, don't forget to convert
operating-system specific slashes to forward slashes. See example here:
https://github.com/apache/lucene-solr/blob/master/gradle/documentation/render-javadoc.gradle#L500
{{def relative = pathFrom.relativize(pathTo).toString().replace(File.separator,
'/')}}
Also make sure that both directories EXIST before calling relativize, otherwise
the result is undefined (operating system dependent). So before calculating
relative path make sure to call mkdir.
Uwe
was (Author: thetaphi):
Hi Hoss,
bq. Hmmm i don't really feel comfortable trying to tackle those changes
here/now – let's pursue this idea more in SOLR-14870 – because AFAICT those
productDir/buildDir variables return absolute file paths
For this issue I was mainly looking at the {{solrRootPath}} property that's
obviously not a relative URL path, which highly depends on structure of build
system (what happens if the CWD is different when executing the scripts in
later Gradle versions?). The series of ".." makes it completely unclear how it
is used. For the URLs to javadocs folders, I agree that's more the SOLR-14870
issue.
bq. but i'm assuming we could relativize those paths again the ref-guides own
buildDir
of course, instead of calling toUri(), you can do {{Path#relativize()}}, using
the own projects buildDir / output dir in combination with the code shown
above. As you need this as URI path later, don't forget to convert
operating-system specific slashes to forward slashes. See example here:
https://github.com/apache/lucene-solr/blob/master/gradle/documentation/render-javadoc.gradle#L500
{{def relative = pathFrom.relativize(pathTo).toString().replace(File.separator,
'/')}}
Uwe
> Simplify set of Ref Guide build parameters
> --
>
> Key: SOLR-14824
> URL: https://issues.apache.org/jira/browse/SOLR-14824
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Build, documentation
>Reporter: Cassandra Targett
>Assignee: Chris M. Hostetter
>Priority: Minor
> Attachments: SOLR-14824.patch, SOLR-14824.patch
>
>
> While trying to solve LUCENE-9495, I thought it might be a good idea to try
> to simplify the set of variables and properties used during the Ref Guide
> build.
> There are 3 areas to work on:
> 1. Remove the "barebones-html" build. With Gradle the build is self-contained
> and {{gradlew check}} and {{gradle precommit}} could just build the full docs
> and check them.
> 2. Remove some properties that only existed for a hypothetical need related
> to the now-removed PDF.
> 3. Change remaining properties to be defined directly in build.gradle instead
> of relying on ant properties functionality.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14873) The SOLR 8 AND JDK Crashed by update document
Leo created SOLR-14873: -- Summary: The SOLR 8 AND JDK Crashed by update document Key: SOLR-14873 URL: https://issues.apache.org/jira/browse/SOLR-14873 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: update, UpdateRequestProcessors Affects Versions: 8.6.2, 8.5.2 Environment: OS: Ubuntu 20.04, CentOS 7 CPU: 24 Core Dell server Mem: 64G HD: SSD Docker: 19.03 docker-compose: 1.26.2 Reporter: Leo Attachments: hs_err_pid13.log We tried to upgrade the current search engine to 8.6, but the system crashed while testing indexing documents.This happens every time a large file is indexing, the system crashes when the document is more than 5M. We deploy SOLR based on cloud mode and run it based on Docker. Each instance allocates 8G of memory. This problem also exists in SOLR 8.5. I have uploaded the report generated by the system to the attachment. Please help me fix it. Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14824) Simplify set of Ref Guide build parameters
[
https://issues.apache.org/jira/browse/SOLR-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197570#comment-17197570
]
Uwe Schindler edited comment on SOLR-14824 at 9/17/20, 9:49 AM:
Hi Hoss,
bq. Hmmm i don't really feel comfortable trying to tackle those changes
here/now – let's pursue this idea more in SOLR-14870 – because AFAICT those
productDir/buildDir variables return absolute file paths
For this issue I was mainly looking at the {{solrRootPath}} property that's
obviously not a relative URL path, which highly depends on structure of build
system (what happens if the CWD is different when executing the scripts in
later Gradle versions?). The series of ".." makes it completely unclear how it
is used. For the URLs to javadocs folders, I agree that's more the SOLR-14870
issue.
bq. but i'm assuming we could relativize those paths again the ref-guides own
buildDir
of course, instead of calling toUri(), you can do {{Path#relativize()}}, using
the own projects buildDir / output dir in combination with the code shown
above. As you need this as URI path later, don't forget to convert
operating-system specific slashes to forward slashes. See example here:
https://github.com/apache/lucene-solr/blob/master/gradle/documentation/render-javadoc.gradle#L500
{{def relative = pathFrom.relativize(pathTo).toString().replace(File.separator,
'/')}}
Also make sure that both directories EXIST before calling relativize, otherwise
the result is undefined (operating system dependent). So before calculating
relative path make sure to call mkdir. So calculating of relative paths must be
done not during gradle configuration phase (where no folders exist yet), but
during the execution of task. So possibly do this in the task's {{doFirst()}},
before executing the asciidoctor/... scripts.
Uwe
was (Author: thetaphi):
Hi Hoss,
bq. Hmmm i don't really feel comfortable trying to tackle those changes
here/now – let's pursue this idea more in SOLR-14870 – because AFAICT those
productDir/buildDir variables return absolute file paths
For this issue I was mainly looking at the {{solrRootPath}} property that's
obviously not a relative URL path, which highly depends on structure of build
system (what happens if the CWD is different when executing the scripts in
later Gradle versions?). The series of ".." makes it completely unclear how it
is used. For the URLs to javadocs folders, I agree that's more the SOLR-14870
issue.
bq. but i'm assuming we could relativize those paths again the ref-guides own
buildDir
of course, instead of calling toUri(), you can do {{Path#relativize()}}, using
the own projects buildDir / output dir in combination with the code shown
above. As you need this as URI path later, don't forget to convert
operating-system specific slashes to forward slashes. See example here:
https://github.com/apache/lucene-solr/blob/master/gradle/documentation/render-javadoc.gradle#L500
{{def relative = pathFrom.relativize(pathTo).toString().replace(File.separator,
'/')}}
Also make sure that both directories EXIST before calling relativize, otherwise
the result is undefined (operating system dependent). So before calculating
relative path make sure to call mkdir.
Uwe
> Simplify set of Ref Guide build parameters
> --
>
> Key: SOLR-14824
> URL: https://issues.apache.org/jira/browse/SOLR-14824
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Build, documentation
>Reporter: Cassandra Targett
>Assignee: Chris M. Hostetter
>Priority: Minor
> Attachments: SOLR-14824.patch, SOLR-14824.patch
>
>
> While trying to solve LUCENE-9495, I thought it might be a good idea to try
> to simplify the set of variables and properties used during the Ref Guide
> build.
> There are 3 areas to work on:
> 1. Remove the "barebones-html" build. With Gradle the build is self-contained
> and {{gradlew check}} and {{gradle precommit}} could just build the full docs
> and check them.
> 2. Remove some properties that only existed for a hypothetical need related
> to the now-removed PDF.
> 3. Change remaining properties to be defined directly in build.gradle instead
> of relying on ant properties functionality.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14873) The SOLR 8 AND JDK Crashed by update document
[ https://issues.apache.org/jira/browse/SOLR-14873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leo updated SOLR-14873: --- Environment: Server 1: * OS: CentOS 7 * CPU: 24 core * Mem: 64G Workstation: * OS: Ubuntu 20.04 (vmware) * Mem: 32G(guest 16G) * CPU: 2 Core Docker: 19.03 docker-compose: 1.26.2 was: OS: Ubuntu 20.04, CentOS 7 CPU: 24 Core Dell server Mem: 64G HD: SSD Docker: 19.03 docker-compose: 1.26.2 > The SOLR 8 AND JDK Crashed by update document > - > > Key: SOLR-14873 > URL: https://issues.apache.org/jira/browse/SOLR-14873 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: update, UpdateRequestProcessors >Affects Versions: 8.5.2, 8.6.2 > Environment: Server 1: > * OS: CentOS 7 > * CPU: 24 core > * Mem: 64G > Workstation: > * OS: Ubuntu 20.04 (vmware) > * Mem: 32G(guest 16G) > * CPU: 2 Core > Docker: 19.03 > docker-compose: 1.26.2 > >Reporter: Leo >Priority: Blocker > Attachments: hs_err_pid13.log > > > We tried to upgrade the current search engine to 8.6, but the system crashed > while testing indexing documents.This happens every time a large file is > indexing, the system crashes when the document is more than 5M. > We deploy SOLR based on cloud mode and run it based on Docker. Each instance > allocates 8G of memory. > This problem also exists in SOLR 8.5. > I have uploaded the report generated by the system to the attachment. > Please help me fix it. Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9527) Upgrade javacc to 7.0.4
[ https://issues.apache.org/jira/browse/LUCENE-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197600#comment-17197600 ] Dawid Weiss commented on LUCENE-9527: - Reverting to 7.0.4. Any javacc between 7.0.5 and 7.0.9 is broken. See this issue: https://github.com/javacc/javacc/issues/183 > Upgrade javacc to 7.0.4 > --- > > Key: LUCENE-9527 > URL: https://issues.apache.org/jira/browse/LUCENE-9527 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9527) Upgrade javacc to 7.0.4
[ https://issues.apache.org/jira/browse/LUCENE-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Weiss updated LUCENE-9527: Summary: Upgrade javacc to 7.0.4 (was: Upgrade javacc to 7.0.5) > Upgrade javacc to 7.0.4 > --- > > Key: LUCENE-9527 > URL: https://issues.apache.org/jira/browse/LUCENE-9527 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss opened a new pull request #1884: LUCENE-9527: upgrade javacc to 7.0.4
dweiss opened a new pull request #1884: URL: https://github.com/apache/lucene-solr/pull/1884 # Description Please provide a short description of the changes you're making with this pull request. # Solution Please provide a short description of the approach taken to implement your solution. # Tests Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem. # Checklist Please review the following and check all that apply: - [ ] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability. - [ ] I have created a Jira issue and added the issue ID to my pull request title. - [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended) - [ ] I have developed this patch against the `master` branch. - [ ] I have run `./gradlew check`. - [ ] I have added tests for my changes. - [ ] I have added documentation for the [Ref Guide](https://github.com/apache/lucene-solr/tree/master/solr/solr-ref-guide) (for Solr changes only). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] murblanc opened a new pull request #1885: SOLR-14613: use set-placement-plugin for both setting and unsetting plugin config
murblanc opened a new pull request #1885: URL: https://github.com/apache/lucene-solr/pull/1885 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss merged pull request #1884: LUCENE-9527: upgrade javacc to 7.0.4
dweiss merged pull request #1884: URL: https://github.com/apache/lucene-solr/pull/1884 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9527) Upgrade javacc to 7.0.4
[ https://issues.apache.org/jira/browse/LUCENE-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Weiss resolved LUCENE-9527. - Fix Version/s: master (9.0) Assignee: Dawid Weiss Resolution: Fixed > Upgrade javacc to 7.0.4 > --- > > Key: LUCENE-9527 > URL: https://issues.apache.org/jira/browse/LUCENE-9527 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Minor > Fix For: master (9.0) > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9527) Upgrade javacc to 7.0.4
[ https://issues.apache.org/jira/browse/LUCENE-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197635#comment-17197635 ] ASF subversion and git services commented on LUCENE-9527: - Commit 6c9d7adf7985e4b895a80e561f1a3bb5820126a8 in lucene-solr's branch refs/heads/master from Dawid Weiss [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=6c9d7ad ] LUCENE-9527: upgrade javacc to 7.0.4 (#1884) > Upgrade javacc to 7.0.4 > --- > > Key: LUCENE-9527 > URL: https://issues.apache.org/jira/browse/LUCENE-9527 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Minor > Fix For: master (9.0) > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14656) Deprecate current autoscaling framework, remove from master
[ https://issues.apache.org/jira/browse/SOLR-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-14656: Priority: Blocker (was: Major) > Deprecate current autoscaling framework, remove from master > --- > > Key: SOLR-14656 > URL: https://issues.apache.org/jira/browse/SOLR-14656 > Project: Solr > Issue Type: Improvement >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Attachments: Screenshot from 2020-07-18 07-49-01.png > > Time Spent: 3h > Remaining Estimate: 0h > > The autoscaling framework is being re-designed in SOLR-14613 (SIP: > https://cwiki.apache.org/confluence/display/SOLR/SIP-8+Autoscaling+policy+engine+V2). > The current autoscaling framework is very inefficient, improperly designed > and too bloated and doesn't receive the level of support we aspire to provide > for all components that we ship. > This issue is to deprecate current autoscaling framework in 8x, so we can > focus on the new autoscaling framework afresh. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9531) Consolidate duplicated generated classes CharStream and FastCharStream
[ https://issues.apache.org/jira/browse/LUCENE-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Weiss updated LUCENE-9531: Parent: LUCENE-9526 Issue Type: Sub-task (was: Improvement) > Consolidate duplicated generated classes CharStream and FastCharStream > -- > > Key: LUCENE-9531 > URL: https://issues.apache.org/jira/browse/LUCENE-9531 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14656) Deprecate current autoscaling framework, remove from master
[ https://issues.apache.org/jira/browse/SOLR-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197638#comment-17197638 ] Ishan Chattopadhyaya commented on SOLR-14656: - Thanks [~ab], would you like to add the notices? > Deprecate current autoscaling framework, remove from master > --- > > Key: SOLR-14656 > URL: https://issues.apache.org/jira/browse/SOLR-14656 > Project: Solr > Issue Type: Improvement >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Fix For: 8.7 > > Attachments: Screenshot from 2020-07-18 07-49-01.png > > Time Spent: 3h > Remaining Estimate: 0h > > The autoscaling framework is being re-designed in SOLR-14613 (SIP: > https://cwiki.apache.org/confluence/display/SOLR/SIP-8+Autoscaling+policy+engine+V2). > The current autoscaling framework is very inefficient, improperly designed > and too bloated and doesn't receive the level of support we aspire to provide > for all components that we ship. > This issue is to deprecate current autoscaling framework in 8x, so we can > focus on the new autoscaling framework afresh. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9531) Consolidate duplicated generated classes CharStream and FastCharStream
Dawid Weiss created LUCENE-9531: --- Summary: Consolidate duplicated generated classes CharStream and FastCharStream Key: LUCENE-9531 URL: https://issues.apache.org/jira/browse/LUCENE-9531 Project: Lucene - Core Issue Type: Improvement Reporter: Dawid Weiss Assignee: Dawid Weiss -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14656) Deprecate current autoscaling framework, remove from master
[ https://issues.apache.org/jira/browse/SOLR-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-14656: Fix Version/s: 8.7 > Deprecate current autoscaling framework, remove from master > --- > > Key: SOLR-14656 > URL: https://issues.apache.org/jira/browse/SOLR-14656 > Project: Solr > Issue Type: Improvement >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Fix For: 8.7 > > Attachments: Screenshot from 2020-07-18 07-49-01.png > > Time Spent: 3h > Remaining Estimate: 0h > > The autoscaling framework is being re-designed in SOLR-14613 (SIP: > https://cwiki.apache.org/confluence/display/SOLR/SIP-8+Autoscaling+policy+engine+V2). > The current autoscaling framework is very inefficient, improperly designed > and too bloated and doesn't receive the level of support we aspire to provide > for all components that we ship. > This issue is to deprecate current autoscaling framework in 8x, so we can > focus on the new autoscaling framework afresh. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9531) Consolidate duplicated generated classes CharStream and FastCharStream
[ https://issues.apache.org/jira/browse/LUCENE-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197639#comment-17197639 ] Dawid Weiss commented on LUCENE-9531: - There are four nearly-exact copies of CharStream (javacc-generated interface) and FastCharStream (Lucene's implementation). This duplication is unnecessary. We can just point at the right implementation/ interface during generation. > Consolidate duplicated generated classes CharStream and FastCharStream > -- > > Key: LUCENE-9531 > URL: https://issues.apache.org/jira/browse/LUCENE-9531 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14656) Deprecate current autoscaling framework, remove from master
[ https://issues.apache.org/jira/browse/SOLR-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya reassigned SOLR-14656: --- Assignee: Ishan Chattopadhyaya > Deprecate current autoscaling framework, remove from master > --- > > Key: SOLR-14656 > URL: https://issues.apache.org/jira/browse/SOLR-14656 > Project: Solr > Issue Type: Improvement >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Attachments: Screenshot from 2020-07-18 07-49-01.png > > Time Spent: 3h > Remaining Estimate: 0h > > The autoscaling framework is being re-designed in SOLR-14613 (SIP: > https://cwiki.apache.org/confluence/display/SOLR/SIP-8+Autoscaling+policy+engine+V2). > The current autoscaling framework is very inefficient, improperly designed > and too bloated and doesn't receive the level of support we aspire to provide > for all components that we ship. > This issue is to deprecate current autoscaling framework in 8x, so we can > focus on the new autoscaling framework afresh. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] rmuir commented on pull request #1882: LUCENE-9529: Track dirtiness of stored fields via a number of docs, not chunks.
rmuir commented on pull request #1882: URL: https://github.com/apache/lucene-solr/pull/1882#issuecomment-694177068 we may want to apply new logic to the CompressingTermVectorsWriter as well for consistency? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on pull request #1882: LUCENE-9529: Track dirtiness of stored fields via a number of docs, not chunks.
jpountz commented on pull request #1882: URL: https://github.com/apache/lucene-solr/pull/1882#issuecomment-694177685 Agreed, I just pushed a change to take care of term vectors as well that I had locally. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss opened a new pull request #1886: LUCENE-9531: Consolidate duplicated generated classes CharStream and FastCharStream
dweiss opened a new pull request #1886: URL: https://github.com/apache/lucene-solr/pull/1886 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-6152) Pre-populating values into search parameters on the query page of solr admin
[ https://issues.apache.org/jira/browse/SOLR-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Furrer updated SOLR-6152: --- Attachment: SOLR-6152.patch > Pre-populating values into search parameters on the query page of solr admin > > > Key: SOLR-6152 > URL: https://issues.apache.org/jira/browse/SOLR-6152 > Project: Solr > Issue Type: Improvement > Components: Admin UI >Affects Versions: 4.3.1 >Reporter: Dmitry Kan >Assignee: Jan Høydahl >Priority: Major > Attachments: SOLR-6152.patch, SOLR-6152.patch, SOLR-6152.patch, > SOLR-6152.patch, SOLR-6152.patch, copy_url_to_clipboard.png, > copy_url_to_clipboard_v2.png, > prefilling_and_extending_the_multivalue_parameter_fq.png, > prepoluate_query_parameters_query_page.bmp > > > In some use cases, it is highly desirable to be able to pre-populate the > query page of solr admin with specific values. > In particular use case of mine, the solr admin user must pass a date range > value without which the query would fail. > It isn't easy to remember the value format for non-solr experts, so I would > like to have a way of hooking that value "example" into the query page. > See the screenshot attached, where I have inserted the fq parameter with date > range into the Raw Query Parameters. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-6152) Pre-populating values into search parameters on the query page of solr admin
[ https://issues.apache.org/jira/browse/SOLR-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197644#comment-17197644 ] Jakob Furrer commented on SOLR-6152: New patch. The latest observations by [~janhoy] have all been addressed. > Pre-populating values into search parameters on the query page of solr admin > > > Key: SOLR-6152 > URL: https://issues.apache.org/jira/browse/SOLR-6152 > Project: Solr > Issue Type: Improvement > Components: Admin UI >Affects Versions: 4.3.1 >Reporter: Dmitry Kan >Assignee: Jan Høydahl >Priority: Major > Attachments: SOLR-6152.patch, SOLR-6152.patch, SOLR-6152.patch, > SOLR-6152.patch, SOLR-6152.patch, copy_url_to_clipboard.png, > copy_url_to_clipboard_v2.png, > prefilling_and_extending_the_multivalue_parameter_fq.png, > prepoluate_query_parameters_query_page.bmp > > > In some use cases, it is highly desirable to be able to pre-populate the > query page of solr admin with specific values. > In particular use case of mine, the solr admin user must pass a date range > value without which the query would fail. > It isn't easy to remember the value format for non-solr experts, so I would > like to have a way of hooking that value "example" into the query page. > See the screenshot attached, where I have inserted the fq parameter with date > range into the Raw Query Parameters. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14874) Add API support to add/edit issuers for JWT Auth
Jan Høydahl created SOLR-14874: -- Summary: Add API support to add/edit issuers for JWT Auth Key: SOLR-14874 URL: https://issues.apache.org/jira/browse/SOLR-14874 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: Authentication Reporter: Jan Høydahl In https://lucene.apache.org/solr/guide/8_4/jwt-authentication-plugin.html#editing-jwt-authentication-plugin-configuration we document that: bq. There is currently no support for adding multiple token issuers though REST API This issue will take care of adding such support. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9531) Consolidate duplicated generated classes CharStream and FastCharStream
[ https://issues.apache.org/jira/browse/LUCENE-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197648#comment-17197648 ] Dawid Weiss commented on LUCENE-9531: - I think it's ready. This does affect downstream API consumers but porting should be trivial (changed import) and the benefits of having one implementation to worry about seem to outweigh the cons. > Consolidate duplicated generated classes CharStream and FastCharStream > -- > > Key: LUCENE-9531 > URL: https://issues.apache.org/jira/browse/LUCENE-9531 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Trivial > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14873) The SOLR 8 AND JDK Crashed by update document
[ https://issues.apache.org/jira/browse/SOLR-14873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson resolved SOLR-14873. --- Resolution: Won't Fix Please raise questions like this on the user's list, we try to reserve JIRAs for known bugs/enhancements rather than usage questions. The JIRA system is not a support portal. See: http://lucene.apache.org/solr/community.html#mailing-lists-irc there are links to both Lucene and Solr mailing lists there. A _lot_ more people will see your question on that list and may be able to help more quickly. If it's determined that this really is a code issue or enhancement to Lucene or Solr and not a configuration/usage problem, we can raise a new JIRA or reopen this one. This is _highly_ likely to be a corrupt JVM and/or operating system and/or configuration issue. So first I'd try this on a stand-alone Solr, then an isolated Docker etc. > The SOLR 8 AND JDK Crashed by update document > - > > Key: SOLR-14873 > URL: https://issues.apache.org/jira/browse/SOLR-14873 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: update, UpdateRequestProcessors >Affects Versions: 8.5.2, 8.6.2 > Environment: Server 1: > * OS: CentOS 7 > * CPU: 24 core > * Mem: 64G > Workstation: > * OS: Ubuntu 20.04 (vmware) > * Mem: 32G(guest 16G) > * CPU: 2 Core > Docker: 19.03 > docker-compose: 1.26.2 > >Reporter: Leo >Priority: Blocker > Attachments: hs_err_pid13.log > > > We tried to upgrade the current search engine to 8.6, but the system crashed > while testing indexing documents.This happens every time a large file is > indexing, the system crashes when the document is more than 5M. > We deploy SOLR based on cloud mode and run it based on Docker. Each instance > allocates 8G of memory. > This problem also exists in SOLR 8.5. > I have uploaded the report generated by the system to the attachment. > Please help me fix it. Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (LUCENE-9528) Clean up obsolete and commented-out cruft from StandardSyntaxParser.jj
[ https://issues.apache.org/jira/browse/LUCENE-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Weiss reassigned LUCENE-9528: --- Assignee: Dawid Weiss > Clean up obsolete and commented-out cruft from StandardSyntaxParser.jj > -- > > Key: LUCENE-9528 > URL: https://issues.apache.org/jira/browse/LUCENE-9528 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > The indentation in that file is crazy. So are micro-optimizations which make > reading the syntax parser much more difficult than it actually is. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9528) Clean up obsolete and commented-out cruft from StandardSyntaxParser.jj
[ https://issues.apache.org/jira/browse/LUCENE-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197650#comment-17197650 ] Erick Erickson commented on LUCENE-9528: Theoretically I suppose some kind of versioning makes sense. OTOH, how many problems have we really had because it's not versioned? My guess is that it's not really worth the effort at this point, but I could be persuaded either way. > Clean up obsolete and commented-out cruft from StandardSyntaxParser.jj > -- > > Key: LUCENE-9528 > URL: https://issues.apache.org/jira/browse/LUCENE-9528 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > The indentation in that file is crazy. So are micro-optimizations which make > reading the syntax parser much more difficult than it actually is. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14792) Remove VelocityResponseWriter from Solr 9
[ https://issues.apache.org/jira/browse/SOLR-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197659#comment-17197659 ] ASF subversion and git services commented on SOLR-14792: Commit 2364a7aded9f04137eeb8520d69ffe97f7caef66 in lucene-solr's branch refs/heads/master from Erik Hatcher [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2364a7a ] SOLR-14792: Remove VelocityResponseWriter > Remove VelocityResponseWriter from Solr 9 > - > > Key: SOLR-14792 > URL: https://issues.apache.org/jira/browse/SOLR-14792 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 9 >Reporter: Erik Hatcher >Priority: Blocker > Time Spent: 1h > Remaining Estimate: 0h > > VelocityResponseWriter was deprecated in SOLR-14065. It can now be removed > from 9's code branch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] murblanc merged pull request #1885: SOLR-14613: use set-placement-plugin for both setting and unsetting plugin config
murblanc merged pull request #1885: URL: https://github.com/apache/lucene-solr/pull/1885 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14613) Provide a clean API for pluggable replica assignment implementations
[
https://issues.apache.org/jira/browse/SOLR-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197672#comment-17197672
]
ASF subversion and git services commented on SOLR-14613:
Commit dbba48b3e56be541572f170a9fae95d039236c1a in lucene-solr's branch
refs/heads/master from Ilan Ginzburg
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=dbba48b ]
SOLR-14613: use set-placement-plugin for both setting and unsetting plugin
config
> Provide a clean API for pluggable replica assignment implementations
>
>
> Key: SOLR-14613
> URL: https://issues.apache.org/jira/browse/SOLR-14613
> Project: Solr
> Issue Type: Improvement
> Components: AutoScaling
>Reporter: Andrzej Bialecki
>Assignee: Ilan Ginzburg
>Priority: Major
> Time Spent: 41h
> Remaining Estimate: 0h
>
> As described in SIP-8 the current autoscaling Policy implementation has
> several limitations that make it difficult to use for very large clusters and
> very large collections. SIP-8 also mentions the possible migration path by
> providing alternative implementations of the placement strategies that are
> less complex but more efficient in these very large environments.
> We should review the existing APIs that the current autoscaling engine uses
> ({{SolrCloudManager}} , {{AssignStrategy}} , {{Suggester}} and related
> interfaces) to see if they provide a sufficient and minimal API for plugging
> in alternative autoscaling placement strategies, and if necessary refactor
> the existing APIs.
> Since these APIs are internal it should be possible to do this without
> breaking back-compat.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] silvestrelosada commented on a change in pull request #1419: SOLR-14397: Vector Search in Solr
silvestrelosada commented on a change in pull request #1419:
URL: https://github.com/apache/lucene-solr/pull/1419#discussion_r490229291
##
File path: solr/core/src/java/org/apache/solr/schema/DenseVectorField.java
##
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.schema;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.FloatBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import org.apache.lucene.document.StoredField;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.queries.function.ValueSource;
+import
org.apache.lucene.queries.function.valuesource.vectors.FloatVectorFieldSource;
+import
org.apache.lucene.queries.function.valuesource.vectors.FloatVectorFieldSource.Encoding;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.BytesRefBuilder;
+import org.apache.solr.common.params.MapSolrParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.TextResponseWriter;
+import org.apache.solr.search.QParser;
+import org.apache.solr.uninverting.UninvertingReader.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static
org.apache.lucene.queries.function.valuesource.vectors.FloatVectorFieldSource.rawStringToVector;
+import static
org.apache.lucene.queries.function.valuesource.vectors.FloatVectorFieldSource.vectorToRawString;
+
+public class DenseVectorField extends FieldType {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String DIMENSIONS_PARAM = "dimensions";
+ public static final int DEFAULT_DIMENSIONS = -1; //Don't enforce.
+ public static int dimensions; //-1 = don't validate
+ public static final String ENCODING = "encoding";
Review comment:
I think it shouldnt be static, if you declare several fields on schema,
the same encoder will be applied to all of them
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13071) Add JWT Auth support in bin/solr
[
https://issues.apache.org/jira/browse/SOLR-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Høydahl updated SOLR-13071:
---
Description:
Once SOLR-12121 gets in, we should add support to {{bin/solr}} start scripts so
they can authenticate with Solr using a JWT token. A preferred way would
perhaps be through {{solr.in.sh}} and add new
{noformat}
SOLR_AUTH_TYPE=token
SOLR_AUTHENTICATION_OPTS=-DjwtToken=
{noformat}
A disadvantage with this method is that the user needs to know how to obtain
the token, and the token needs to be long-lived. A more sophisticated way would
be a {{bin/solr auth login}} command that opens a browser window with the IDP
login screen and saves the short-lived access token and optionally refresh
token, in the file system.
was:
Once SOLR-12121 gets in, we should add support to {{bin/solr}} start scripts so
they can authenticate with Solr using a JWT token. A preferred way would
perhaps be through {{solr.in.sh}} and add new
{noformat}
SOLR_AUTH_TYPE=token
SOLR_AUTHENTICATION_OPTS=-DjwtToken={noformat}
> Add JWT Auth support in bin/solr
>
>
> Key: SOLR-13071
> URL: https://issues.apache.org/jira/browse/SOLR-13071
> Project: Solr
> Issue Type: Improvement
> Components: scripts and tools
>Reporter: Jan Høydahl
>Priority: Major
>
> Once SOLR-12121 gets in, we should add support to {{bin/solr}} start scripts
> so they can authenticate with Solr using a JWT token. A preferred way would
> perhaps be through {{solr.in.sh}} and add new
> {noformat}
> SOLR_AUTH_TYPE=token
> SOLR_AUTHENTICATION_OPTS=-DjwtToken=
> {noformat}
> A disadvantage with this method is that the user needs to know how to obtain
> the token, and the token needs to be long-lived. A more sophisticated way
> would be a {{bin/solr auth login}} command that opens a browser window with
> the IDP login screen and saves the short-lived access token and optionally
> refresh token, in the file system.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14875) Make SolrEventListeners load from packages
Noble Paul created SOLR-14875: - Summary: Make SolrEventListeners load from packages Key: SOLR-14875 URL: https://issues.apache.org/jira/browse/SOLR-14875 Project: Solr Issue Type: Sub-task Components: packages Reporter: Noble Paul Assignee: Noble Paul -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14875) Make SolrEventListeners load from packages
[ https://issues.apache.org/jira/browse/SOLR-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-14875: -- Description: This is a hot reloaded plugin (does not result in a core reload) > Make SolrEventListeners load from packages > -- > > Key: SOLR-14875 > URL: https://issues.apache.org/jira/browse/SOLR-14875 > Project: Solr > Issue Type: Sub-task > Components: packages >Reporter: Noble Paul >Assignee: Noble Paul >Priority: Major > > This is a hot reloaded plugin (does not result in a core reload) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul opened a new pull request #1887: SOLR-14875: Make SolrEventListeners load from packages
noblepaul opened a new pull request #1887: URL: https://github.com/apache/lucene-solr/pull/1887 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz opened a new pull request #1888: Further tune Lucene87StoredFieldsFormat for small documents.
jpountz opened a new pull request #1888: URL: https://github.com/apache/lucene-solr/pull/1888 The increase of the maximum number of chunks per doc done in previous issues was mostly random. I'd like to provide users with a similar trade-off with what the old versions of BEST_SPEED and BEST_COMPRESSION used to do. So since BEST_SPEED used to compress at most 128 docs at once, I think we should roughly make it 128*10 now. I made it 1024 to account for the fact that there is a preset dict as well that need decompressing. And similarly BEST_COMPRESSION used to allow 4x more docs than BEST_SPEED, so I made it 4096. With such larger numbers of docs per chunk, the decoding of metadata became a bottleneck for stored field access so I made it a bit faster by doing bulk decoding of the packed longs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14151) Make schema components load from packages
[
https://issues.apache.org/jira/browse/SOLR-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197712#comment-17197712
]
Daniel Worley commented on SOLR-14151:
--
[~noble.paul] Just tested and looks good! Will this make it into 8.7 or is that
to be determined? Thanks!
> Make schema components load from packages
> -
>
> Key: SOLR-14151
> URL: https://issues.apache.org/jira/browse/SOLR-14151
> Project: Solr
> Issue Type: Sub-task
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Labels: packagemanager
> Fix For: 8.7
>
> Time Spent: 12h 50m
> Remaining Estimate: 0h
>
> Example:
> {code:xml}
>
>
>
>generateNumberParts="0" catenateWords="0"
> catenateNumbers="0" catenateAll="0"/>
>
>
>
>
> {code}
> * When a package is updated, the entire {{IndexSchema}} object is refreshed,
> but the SolrCore object is not reloaded
> * Any component can be prefixed with the package name
> * The semantics of loading plugins remain the same as that of the components
> in {{solrconfig.xml}}
> * Plugins can be registered using schema API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14151) Make schema components load from packages
[
https://issues.apache.org/jira/browse/SOLR-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197714#comment-17197714
]
Noble Paul commented on SOLR-14151:
---
This will go into 8.7
> Make schema components load from packages
> -
>
> Key: SOLR-14151
> URL: https://issues.apache.org/jira/browse/SOLR-14151
> Project: Solr
> Issue Type: Sub-task
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Labels: packagemanager
> Fix For: 8.7
>
> Time Spent: 12h 50m
> Remaining Estimate: 0h
>
> Example:
> {code:xml}
>
>
>
>generateNumberParts="0" catenateWords="0"
> catenateNumbers="0" catenateAll="0"/>
>
>
>
>
> {code}
> * When a package is updated, the entire {{IndexSchema}} object is refreshed,
> but the SolrCore object is not reloaded
> * Any component can be prefixed with the package name
> * The semantics of loading plugins remain the same as that of the components
> in {{solrconfig.xml}}
> * Plugins can be registered using schema API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] ErickErickson commented on pull request #1881: Upgrade zookeeper version to 3.6.2 to use recent version of netty
ErickErickson commented on pull request #1881: URL: https://github.com/apache/lucene-solr/pull/1881#issuecomment-694260257 Does this have an associated Solr JIRA? I couldn't find one... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] murblanc commented on a change in pull request #1758: SOLR-14749: Provide a clean API for cluster-level event processing, Initial draft.
murblanc commented on a change in pull request #1758:
URL: https://github.com/apache/lucene-solr/pull/1758#discussion_r490279251
##
File path:
solr/core/src/java/org/apache/solr/cluster/events/CollectionsAddedEvent.java
##
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cluster.events;
+
+import java.util.Collection;
+
+/**
+ * Event generated when some collections have been added.
+ */
+public interface CollectionsAddedEvent extends ClusterEvent {
+
+ @Override
+ default EventType getType() {
+return EventType.COLLECTIONS_ADDED;
+ }
+
+ Collection getCollectionNames();
Review comment:
Shall we move this to an iterator model?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] murblanc commented on a change in pull request #1758: SOLR-14749: Provide a clean API for cluster-level event processing, Initial draft.
murblanc commented on a change in pull request #1758:
URL: https://github.com/apache/lucene-solr/pull/1758#discussion_r490284824
##
File path:
solr/core/src/java/org/apache/solr/cluster/events/ClusterEventListener.java
##
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cluster.events;
+
+import java.util.Set;
+
+/**
+ * Components that want to be notified of cluster-wide events should use this.
+ *
+ * XXX should this work only for ClusterSingleton-s? some types of events may
be
+ * XXX difficult (or pointless) to propagate to every node.
+ */
+public interface ClusterEventListener {
+
+ /**
+ * The types of events that this listener can process.
+ */
+ Set getEventTypes();
Review comment:
Why have event types listened to as a listener attribute rather than as
a parameter passed upon registration?
Even types part of the listener make it harder to reuse listeners. For
example a listener that logs the event and that can be registered wherever
needed (this model would force subclassing it each time).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] murblanc commented on a change in pull request #1758: SOLR-14749: Provide a clean API for cluster-level event processing, Initial draft.
murblanc commented on a change in pull request #1758:
URL: https://github.com/apache/lucene-solr/pull/1758#discussion_r490285588
##
File path:
solr/core/src/java/org/apache/solr/cluster/events/ClusterEventListener.java
##
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cluster.events;
+
+import java.util.Set;
+
+/**
+ * Components that want to be notified of cluster-wide events should use this.
+ *
+ * XXX should this work only for ClusterSingleton-s? some types of events may
be
+ * XXX difficult (or pointless) to propagate to every node.
+ */
+public interface ClusterEventListener {
+
+ /**
+ * The types of events that this listener can process.
+ */
+ Set getEventTypes();
Review comment:
Similarly, it might be useful to unregister a listener on some event
types but not all...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] murblanc commented on a change in pull request #1758: SOLR-14749: Provide a clean API for cluster-level event processing, Initial draft.
murblanc commented on a change in pull request #1758:
URL: https://github.com/apache/lucene-solr/pull/1758#discussion_r490288384
##
File path:
solr/core/src/java/org/apache/solr/cluster/events/ClusterEventProducer.java
##
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cluster.events;
+
+import org.apache.solr.cloud.ClusterSingleton;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Component that produces {@link ClusterEvent} instances.
+ */
+public interface ClusterEventProducer extends ClusterSingleton {
+
+ String PLUGIN_NAME = "clusterEventProducer";
+
+ /**
+ * Returns a modifiable map of event types and listeners to process events
+ * of a given type.
+ */
+ Map> getEventListeners();
Review comment:
What about using an `EnumMap` here instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] murblanc commented on a change in pull request #1758: SOLR-14749: Provide a clean API for cluster-level event processing, Initial draft.
murblanc commented on a change in pull request #1758:
URL: https://github.com/apache/lucene-solr/pull/1758#discussion_r490294923
##
File path:
solr/core/src/java/org/apache/solr/cluster/events/ClusterPropertiesChangedEvent.java
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cluster.events;
+
+import java.util.Map;
+
+/**
+ * Event generated when {@link
org.apache.solr.common.cloud.ZkStateReader#CLUSTER_PROPS} is modified.
+ */
+public interface ClusterPropertiesChangedEvent extends ClusterEvent {
+
+ @Override
+ default EventType getType() {
+return EventType.CLUSTER_PROPERTIES_CHANGED;
+ }
+
+ Map getOldClusterProperties();
Review comment:
Is it really the role of the event to provide the old vs new properties?
I'd prefer the event to provide the change only, let the subscriber fetch the
new state if it wants to assemble the before/after state (and possibly the
listener is keeping its own copy of what the old state is, and that old state
is not necessarily equal to the old state as known by the change event
implementation).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14656) Deprecate current autoscaling framework, remove from master
[ https://issues.apache.org/jira/browse/SOLR-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya reassigned SOLR-14656: --- Assignee: (was: Ishan Chattopadhyaya) > Deprecate current autoscaling framework, remove from master > --- > > Key: SOLR-14656 > URL: https://issues.apache.org/jira/browse/SOLR-14656 > Project: Solr > Issue Type: Improvement >Reporter: Ishan Chattopadhyaya >Priority: Blocker > Fix For: 8.7 > > Attachments: Screenshot from 2020-07-18 07-49-01.png > > Time Spent: 3h > Remaining Estimate: 0h > > The autoscaling framework is being re-designed in SOLR-14613 (SIP: > https://cwiki.apache.org/confluence/display/SOLR/SIP-8+Autoscaling+policy+engine+V2). > The current autoscaling framework is very inefficient, improperly designed > and too bloated and doesn't receive the level of support we aspire to provide > for all components that we ship. > This issue is to deprecate current autoscaling framework in 8x, so we can > focus on the new autoscaling framework afresh. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] murblanc commented on pull request #1758: SOLR-14749: Provide a clean API for cluster-level event processing, Initial draft.
murblanc commented on pull request #1758: URL: https://github.com/apache/lucene-solr/pull/1758#issuecomment-694280735 Would love to see as part of this PR an example of an event listener doing something (remotely) useful, to get a better feel of the whole process. That sample code could be removed later when "real" code doing event listening is introduced. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1856: LUCENE-9449 Skip docs with _doc sort and "after" (#1725)
mayya-sharipova commented on a change in pull request #1856: URL: https://github.com/apache/lucene-solr/pull/1856#discussion_r490355700 ## File path: lucene/CHANGES.txt ## @@ -59,6 +65,12 @@ Optimizations * LUCENE-9373: FunctionMatchQuery now accepts a "matchCost" optimization hint. (Maxim Glazkov, David Smiley) +* LUCENE-9449: Enhance DocComparator to provide an iterator over competitive + documents when searching with "after". This iterator can quickly position + on the desired "after" document skipping all documents and segments before + "after". Also redesign numeric comparators to provide skipping functionality + by default. (Mayya Sharipova, Jim Ferenczi) Review comment: Addressed in the last commit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9449) Skip non-competitive documents when sort by _doc with search after
[ https://issues.apache.org/jira/browse/LUCENE-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197773#comment-17197773 ] ASF subversion and git services commented on LUCENE-9449: - Commit 754216821001772a0890d7245765e2ffb3f917f2 in lucene-solr's branch refs/heads/branch_8x from Mayya Sharipova [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=7542168 ] LUCENE-9449 Skip docs with _doc sort and "after" (#1725) (#1856) - Enhance DocComparator to provide an iterator over competitive documents when searching with "after". This iterator can quickly position on the desired "after" document skipping all documents and segments before "after". - Redesign numeric comparators to move to separate package. Backport for #LUCENE-9449 > Skip non-competitive documents when sort by _doc with search after > -- > > Key: LUCENE-9449 > URL: https://issues.apache.org/jira/browse/LUCENE-9449 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Mayya Sharipova >Priority: Minor > Time Spent: 5h 20m > Remaining Estimate: 0h > > Enhance DocComparator to provide an iterator over competitive documents when > search ing with "after" FieldDoc. > This iterator can quickly position on the desired "after" document, and skip > all documents or even segments that contain documents before "after" > This is especially efficient when "after" is high. > > Related to LUCENE-9280 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9449) Skip non-competitive documents when sort by _doc with search after
[ https://issues.apache.org/jira/browse/LUCENE-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197774#comment-17197774 ] ASF subversion and git services commented on LUCENE-9449: - Commit 754216821001772a0890d7245765e2ffb3f917f2 in lucene-solr's branch refs/heads/branch_8x from Mayya Sharipova [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=7542168 ] LUCENE-9449 Skip docs with _doc sort and "after" (#1725) (#1856) - Enhance DocComparator to provide an iterator over competitive documents when searching with "after". This iterator can quickly position on the desired "after" document skipping all documents and segments before "after". - Redesign numeric comparators to move to separate package. Backport for #LUCENE-9449 > Skip non-competitive documents when sort by _doc with search after > -- > > Key: LUCENE-9449 > URL: https://issues.apache.org/jira/browse/LUCENE-9449 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Mayya Sharipova >Priority: Minor > Time Spent: 5h 20m > Remaining Estimate: 0h > > Enhance DocComparator to provide an iterator over competitive documents when > search ing with "after" FieldDoc. > This iterator can quickly position on the desired "after" document, and skip > all documents or even segments that contain documents before "after" > This is especially efficient when "after" is high. > > Related to LUCENE-9280 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova merged pull request #1856: LUCENE-9449 Skip docs with _doc sort and "after" (#1725)
mayya-sharipova merged pull request #1856: URL: https://github.com/apache/lucene-solr/pull/1856 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] rmuir commented on pull request #1888: Further tune Lucene87StoredFieldsFormat for small documents.
rmuir commented on pull request #1888: URL: https://github.com/apache/lucene-solr/pull/1888#issuecomment-694337503 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] rmuir commented on pull request #1882: LUCENE-9529: Track dirtiness of stored fields via a number of docs, not chunks.
rmuir commented on pull request #1882: URL: https://github.com/apache/lucene-solr/pull/1882#issuecomment-694339576 +1 We can always try to improve this thing later in subsequent issues. This still keeps a hard safety net on the number of dirty chunks, so I can't see it introducing any new worst-case behavior This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14859) [* TO *] queries on DateRange fields miss results
[
https://issues.apache.org/jira/browse/SOLR-14859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gerlowski updated SOLR-14859:
---
Attachment: SOLR-14859.patch
Status: Open (was: Open)
Thanks for helping lay out an approach here guys. I've attached a patch which
tries to follow that. No tests yet, but I'll add those once we agree where we
want to log vs throw errors. Some notes:
* The added logic lives in AbstractSpatialPrefixTreeFieldType so that it fires
for all field types which use PrefixTreeStrategy, not just DateRangeField.
* The patch uses {{setArgs}} to check the fieldType declaration. There we log
warnings on any explicit fieldType options that contradict the hardcoded
FieldType, and we set property defaults that match the FieldType.
* The patch uses {{checkSchemaField}} to check the field declaration. Unlike
{{setArgs}} which lets us modify the fieldType-properties, {{checkSchemaField}}
doesn't have a great way to modify the SchemaField (afaict), so I had to settle
for throwing SolrException's there.
* I don't love the inconsistency between the two bullet points above - it seems
wrong to warn-and-correct bad {{}} settings but throw exceptions
when those same bad settings exist on the {{}}. Should we throw errors
on both cases? Or if we'd prefer to warn-and-correct on both cases, is there a
way to modify the SchemaField to make corrections from within checkSchemaField?
* There's a couple NOCOMMIT comments where I'd appreciate feedback from any
reviewers.
Pending any feedback I'll aim to make the fieldType and field checking
consistent by having them both throw SolrExceptions on any explicit settings
that contradict the hardcoded FieldType. I'm happy to standardize the other way
too though if anyone has suggestions on how to modify the SchemaField from
within {{checkSchemaFields}}
> [* TO *] queries on DateRange fields miss results
> -
>
> Key: SOLR-14859
> URL: https://issues.apache.org/jira/browse/SOLR-14859
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: 8.5
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-14859.patch, SOLR-14859.patch, query-debug.png,
> reproduce.sh, schema.png
>
>
> "exists" queries ({{[* TO *]}}) on DateRange fields return 0 results
> regardless of docs in the index with values in that field.
> The issue appears to be that the query is converted into a
> {{NormsFieldExistsQuery}}, even though DateRangeField uses omitNorms=true by
> default. Probably introduced by SOLR-11746's changes to these optimizable
> range queries.
> I've attached a script to reproduce the issue (tested on Solr 8.6.2) and
> screenshots showing showing schema and query-parsing info for the
> reproduction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz merged pull request #1888: Further tune Lucene87StoredFieldsFormat for small documents.
jpountz merged pull request #1888: URL: https://github.com/apache/lucene-solr/pull/1888 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz merged pull request #1882: LUCENE-9529: Track dirtiness of stored fields via a number of docs, not chunks.
jpountz merged pull request #1882: URL: https://github.com/apache/lucene-solr/pull/1882 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9529) Larger stored fields block sizes mean we're more likely to disable optimized bulk merging
[ https://issues.apache.org/jira/browse/LUCENE-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197816#comment-17197816 ] ASF subversion and git services commented on LUCENE-9529: - Commit 33f728007853ee53c71f259206734bfd9155593d in lucene-solr's branch refs/heads/master from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=33f7280 ] LUCENE-9529: Track dirtiness of stored fields via a number of docs, not chunks. (#1882) The problem of tracking dirtiness via numbers of chunks is that larger chunks make stored fields readers more likely to be considered dirty, so I'm trying to work around it by tracking numbers of docs instead. > Larger stored fields block sizes mean we're more likely to disable optimized > bulk merging > - > > Key: LUCENE-9529 > URL: https://issues.apache.org/jira/browse/LUCENE-9529 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Whenever possible when merging stored fields, Lucene tries to copy the > compressed data instead of decompressing the source segment to then > re-compressing in the destination segment. A problem with this approach is > that if some blocks are incomplete (typically the last block of a segment) > then it remains incomplete in the destination segment too, and if we do it > for too long we end up with a bad compression ratio. So Lucene keeps track of > these incomplete blocks, and makes sure to keep a ratio of incomplete blocks > below 1%. > But as we increased the block size, it has become more likely to have a high > ratio of incomplete blocks. E.g. if you have a segment with 1MB of stored > fields, with 16kB blocks like before, you have 63 complete blocks and 1 > incomplete block, or 1.6%. But now with ~512kB blocks, you have one complete > block and 1 incomplete block, ie. 50%. > I'm not sure how to fix it or even whether it should be fixed but wanted to > open an issue to track this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9529) Larger stored fields block sizes mean we're more likely to disable optimized bulk merging
[ https://issues.apache.org/jira/browse/LUCENE-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197820#comment-17197820 ] ASF subversion and git services commented on LUCENE-9529: - Commit 33f728007853ee53c71f259206734bfd9155593d in lucene-solr's branch refs/heads/master from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=33f7280 ] LUCENE-9529: Track dirtiness of stored fields via a number of docs, not chunks. (#1882) The problem of tracking dirtiness via numbers of chunks is that larger chunks make stored fields readers more likely to be considered dirty, so I'm trying to work around it by tracking numbers of docs instead. > Larger stored fields block sizes mean we're more likely to disable optimized > bulk merging > - > > Key: LUCENE-9529 > URL: https://issues.apache.org/jira/browse/LUCENE-9529 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Whenever possible when merging stored fields, Lucene tries to copy the > compressed data instead of decompressing the source segment to then > re-compressing in the destination segment. A problem with this approach is > that if some blocks are incomplete (typically the last block of a segment) > then it remains incomplete in the destination segment too, and if we do it > for too long we end up with a bad compression ratio. So Lucene keeps track of > these incomplete blocks, and makes sure to keep a ratio of incomplete blocks > below 1%. > But as we increased the block size, it has become more likely to have a high > ratio of incomplete blocks. E.g. if you have a segment with 1MB of stored > fields, with 16kB blocks like before, you have 63 complete blocks and 1 > incomplete block, or 1.6%. But now with ~512kB blocks, you have one complete > block and 1 incomplete block, ie. 50%. > I'm not sure how to fix it or even whether it should be fixed but wanted to > open an issue to track this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9529) Larger stored fields block sizes mean we're more likely to disable optimized bulk merging
[ https://issues.apache.org/jira/browse/LUCENE-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197829#comment-17197829 ] ASF subversion and git services commented on LUCENE-9529: - Commit 33f728007853ee53c71f259206734bfd9155593d in lucene-solr's branch refs/heads/master from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=33f7280 ] LUCENE-9529: Track dirtiness of stored fields via a number of docs, not chunks. (#1882) The problem of tracking dirtiness via numbers of chunks is that larger chunks make stored fields readers more likely to be considered dirty, so I'm trying to work around it by tracking numbers of docs instead. > Larger stored fields block sizes mean we're more likely to disable optimized > bulk merging > - > > Key: LUCENE-9529 > URL: https://issues.apache.org/jira/browse/LUCENE-9529 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Whenever possible when merging stored fields, Lucene tries to copy the > compressed data instead of decompressing the source segment to then > re-compressing in the destination segment. A problem with this approach is > that if some blocks are incomplete (typically the last block of a segment) > then it remains incomplete in the destination segment too, and if we do it > for too long we end up with a bad compression ratio. So Lucene keeps track of > these incomplete blocks, and makes sure to keep a ratio of incomplete blocks > below 1%. > But as we increased the block size, it has become more likely to have a high > ratio of incomplete blocks. E.g. if you have a segment with 1MB of stored > fields, with 16kB blocks like before, you have 63 complete blocks and 1 > incomplete block, or 1.6%. But now with ~512kB blocks, you have one complete > block and 1 incomplete block, ie. 50%. > I'm not sure how to fix it or even whether it should be fixed but wanted to > open an issue to track this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9529) Larger stored fields block sizes mean we're more likely to disable optimized bulk merging
[ https://issues.apache.org/jira/browse/LUCENE-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197831#comment-17197831 ] ASF subversion and git services commented on LUCENE-9529: - Commit 33f728007853ee53c71f259206734bfd9155593d in lucene-solr's branch refs/heads/master from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=33f7280 ] LUCENE-9529: Track dirtiness of stored fields via a number of docs, not chunks. (#1882) The problem of tracking dirtiness via numbers of chunks is that larger chunks make stored fields readers more likely to be considered dirty, so I'm trying to work around it by tracking numbers of docs instead. > Larger stored fields block sizes mean we're more likely to disable optimized > bulk merging > - > > Key: LUCENE-9529 > URL: https://issues.apache.org/jira/browse/LUCENE-9529 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Whenever possible when merging stored fields, Lucene tries to copy the > compressed data instead of decompressing the source segment to then > re-compressing in the destination segment. A problem with this approach is > that if some blocks are incomplete (typically the last block of a segment) > then it remains incomplete in the destination segment too, and if we do it > for too long we end up with a bad compression ratio. So Lucene keeps track of > these incomplete blocks, and makes sure to keep a ratio of incomplete blocks > below 1%. > But as we increased the block size, it has become more likely to have a high > ratio of incomplete blocks. E.g. if you have a segment with 1MB of stored > fields, with 16kB blocks like before, you have 63 complete blocks and 1 > incomplete block, or 1.6%. But now with ~512kB blocks, you have one complete > block and 1 incomplete block, ie. 50%. > I'm not sure how to fix it or even whether it should be fixed but wanted to > open an issue to track this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9529) Larger stored fields block sizes mean we're more likely to disable optimized bulk merging
[ https://issues.apache.org/jira/browse/LUCENE-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197843#comment-17197843 ] ASF subversion and git services commented on LUCENE-9529: - Commit 33f728007853ee53c71f259206734bfd9155593d in lucene-solr's branch refs/heads/master from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=33f7280 ] LUCENE-9529: Track dirtiness of stored fields via a number of docs, not chunks. (#1882) The problem of tracking dirtiness via numbers of chunks is that larger chunks make stored fields readers more likely to be considered dirty, so I'm trying to work around it by tracking numbers of docs instead. > Larger stored fields block sizes mean we're more likely to disable optimized > bulk merging > - > > Key: LUCENE-9529 > URL: https://issues.apache.org/jira/browse/LUCENE-9529 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Whenever possible when merging stored fields, Lucene tries to copy the > compressed data instead of decompressing the source segment to then > re-compressing in the destination segment. A problem with this approach is > that if some blocks are incomplete (typically the last block of a segment) > then it remains incomplete in the destination segment too, and if we do it > for too long we end up with a bad compression ratio. So Lucene keeps track of > these incomplete blocks, and makes sure to keep a ratio of incomplete blocks > below 1%. > But as we increased the block size, it has become more likely to have a high > ratio of incomplete blocks. E.g. if you have a segment with 1MB of stored > fields, with 16kB blocks like before, you have 63 complete blocks and 1 > incomplete block, or 1.6%. But now with ~512kB blocks, you have one complete > block and 1 incomplete block, ie. 50%. > I'm not sure how to fix it or even whether it should be fixed but wanted to > open an issue to track this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9529) Larger stored fields block sizes mean we're more likely to disable optimized bulk merging
[ https://issues.apache.org/jira/browse/LUCENE-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197844#comment-17197844 ] ASF subversion and git services commented on LUCENE-9529: - Commit 830bd186a8d72ce6cc96f2856c269ef02e98d3c5 in lucene-solr's branch refs/heads/branch_8x from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=830bd18 ] LUCENE-9529: Track dirtiness of stored fields via a number of docs, not chunks. (#1882) The problem of tracking dirtiness via numbers of chunks is that larger chunks make stored fields readers more likely to be considered dirty, so I'm trying to work around it by tracking numbers of docs instead. > Larger stored fields block sizes mean we're more likely to disable optimized > bulk merging > - > > Key: LUCENE-9529 > URL: https://issues.apache.org/jira/browse/LUCENE-9529 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Whenever possible when merging stored fields, Lucene tries to copy the > compressed data instead of decompressing the source segment to then > re-compressing in the destination segment. A problem with this approach is > that if some blocks are incomplete (typically the last block of a segment) > then it remains incomplete in the destination segment too, and if we do it > for too long we end up with a bad compression ratio. So Lucene keeps track of > these incomplete blocks, and makes sure to keep a ratio of incomplete blocks > below 1%. > But as we increased the block size, it has become more likely to have a high > ratio of incomplete blocks. E.g. if you have a segment with 1MB of stored > fields, with 16kB blocks like before, you have 63 complete blocks and 1 > incomplete block, or 1.6%. But now with ~512kB blocks, you have one complete > block and 1 incomplete block, ie. 50%. > I'm not sure how to fix it or even whether it should be fixed but wanted to > open an issue to track this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #1877: SOLR-13181: param macro expansion could throw
dsmiley commented on a change in pull request #1877:
URL: https://github.com/apache/lucene-solr/pull/1877#discussion_r490430100
##
File path: solr/core/src/java/org/apache/solr/request/macro/MacroExpander.java
##
@@ -188,7 +189,7 @@ else if (failOnMissingParams) {
} catch (SyntaxError syntaxError) {
// append the part we would have skipped
-sb.append( val.substring(matchedStart, start) );
+sb.append(val, matchedStart, start);
Review comment:
IntelliJ pointed this out to me :-) It didn't know about the
val.substring(start) (implied val.length()) so I did those manually. Yes, it'd
be nice if everywhere we could do this in one go. Feel free to file an issue
to do so if you have time. IntelliJ can make quick work of that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #1877: SOLR-13181: param macro expansion could throw
dsmiley commented on a change in pull request #1877:
URL: https://github.com/apache/lucene-solr/pull/1877#discussion_r490438268
##
File path: solr/core/src/java/org/apache/solr/request/macro/MacroExpander.java
##
@@ -154,14 +155,14 @@ else if (idx < 0) {
}
if (matchedStart > 0) {
-sb.append(val.substring(start, matchedStart));
+sb.append(val, start, matchedStart);
}
// update "start" to be at the end of ${...}
- start = rbrace + 1;
+ idx = start = rbrace + 1;
- // String inbetween = val.substring(idx, rbrace);
- StrParser parser = new StrParser(val, idx, rbrace);
+ // String in-between = val.substring(idx, rbrace);
Review comment:
```suggestion
// String in-between braces
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14859) [* TO *] queries on DateRange fields miss results
[
https://issues.apache.org/jira/browse/SOLR-14859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197877#comment-17197877
]
Chris M. Hostetter commented on SOLR-14859:
---
{noformat}
+// NOCOMMIT - I'm only setting here the things that are directly specified
in PrefixTreeStrategy.FIELD_TYPE, but
+// there are other values there I'm not checking (docValues, etc.).
Should I check _everything_ here?
{noformat}
I think no matter what we do, it's going to be crufty just because abstraction
/ separation of concerns that exist, and how there's not a very clear
delineation between the "args" that get handled by the 'lucene.FieldType'
(inside the Strategy) vs the args that solr/AbstractSpatialFieldType handles
(multiValued, stored, etc...). ... i just didn't realize how crufty until i saw
your patch and really thought about it : )
But maybe instead of the list of "if" statements, a simpler/cleaner way to
approach this might be...
* declare a static Map of solrArgName -> "hardcoded default that can't be
changed" based on what is in {{PrefixTreeStrategy.FIELD_TYPE}}, ignoring the
things like "stored" that AbstractSpatialFieldType handles directly. something
like...
{code:java}
private static final Map hardcodedDefaults = new HashMap<>();
static {
final lucene.FieldType ft = PrefixTreeStrategy.FIELD_TYPE;
final IndexOptions opts = ft.indexOptions();
hardcodedDefaults.put("omitNorms", ft.omitNorms().toString());
hardcodedDefaults.put("termVectors", ft.storeTermVectors());
hardcodedDefaults.put("omitTermFreqAndPositions",opts.equals(DOCS));
// etc...
}
{code}
* in setArgs, loop over the entries in 'hardcodedDefaults' WARNing if the key
is found in 'args
** NOTE: i'm (now) suggesting that we log a warning even if the arg is set to
the same value as the hardcoded default – because we can WARN "FieldType {}
does not allow {} to be specified in schema, hardcoded behavior is {}={}" so
people who just so happen to be specifying the "default" of foo=bar in their
schema aren't completely suprised when they try foo=yak and discover setting
'foo' has never actaully been supported.
* then "putAll" of hardcodedDefaults to args.
{quote}The patch uses checkSchemaField to check the field declaration. Unlike
setArgs which lets us modify the fieldType-properties, checkSchemaField doesn't
have a great way to modify the SchemaField (afaict), so I had to settle for
throwing SolrException's there.
{quote}
{quote}I don't love the inconsistency between the two bullet points above - it
seems wrong to warn-and-correct bad settings but throw exceptions
when those same bad settings exist on the . Should we throw errors on
both cases? Or if we'd prefer to warn-and-correct on both cases, is there a way
to modify the SchemaField to make corrections from within checkSchemaField?
{quote}
Hmmm, yeah ... SchemaFields are really ment to be immutable, so we'd have to
add something like 'setProperties(int)' to SchemaField to be able to allow
checkSchemaField to modify it but i don't relaly like the that idea.
Or we could change the api of checkSchemaField so that can return the 'final'
properties the SchemaField should use?
... i hate that idea less; but changing the SchemaField / FieldType apis like
that kind of feels like a "disproportionate" response to the problem at hand.
*Odds are, most people aren't going out of their way to set weird options on
these fieldType/field declarations – the main problem is really that the
"defaults" solr "assumes" are wrong at the Lucene level. Fixing those
SchemaField properties so things like the query parser work properly is what's
really important. warning/failing if the user specifies something that's going
to be ignored anyway is secondary.*
So i think you're probably right, and we should probably throw an exception in
both cases
Odds are most people will never see those failure "on upgrade" because they
probably haven't mucked with the field/fieldType settings for these FieldTypes.
and if they do we're doing them a favor by preventing weird ass erors like the
one that prompted this Jira
In which case, going back to my suggestion above: we should probably WARN if
any of the "hardcodedDefault" attributes are specified _even using the same
value as the one that's "hardcoded"_ but FAIL if the value is specified *AND*
differs from the hardcoded default ... right?
> [* TO *] queries on DateRange fields miss results
> -
>
> Key: SOLR-14859
> URL: https://issues.apache.org/jira/browse/SOLR-14859
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: 8.5
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>
[jira] [Resolved] (LUCENE-9529) Larger stored fields block sizes mean we're more likely to disable optimized bulk merging
[ https://issues.apache.org/jira/browse/LUCENE-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand resolved LUCENE-9529. -- Fix Version/s: 8.7 Resolution: Fixed > Larger stored fields block sizes mean we're more likely to disable optimized > bulk merging > - > > Key: LUCENE-9529 > URL: https://issues.apache.org/jira/browse/LUCENE-9529 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Fix For: 8.7 > > Time Spent: 50m > Remaining Estimate: 0h > > Whenever possible when merging stored fields, Lucene tries to copy the > compressed data instead of decompressing the source segment to then > re-compressing in the destination segment. A problem with this approach is > that if some blocks are incomplete (typically the last block of a segment) > then it remains incomplete in the destination segment too, and if we do it > for too long we end up with a bad compression ratio. So Lucene keeps track of > these incomplete blocks, and makes sure to keep a ratio of incomplete blocks > below 1%. > But as we increased the block size, it has become more likely to have a high > ratio of incomplete blocks. E.g. if you have a segment with 1MB of stored > fields, with 16kB blocks like before, you have 63 complete blocks and 1 > incomplete block, or 1.6%. But now with ~512kB blocks, you have one complete > block and 1 incomplete block, ie. 50%. > I'm not sure how to fix it or even whether it should be fixed but wanted to > open an issue to track this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14824) Simplify set of Ref Guide build parameters
[
https://issues.apache.org/jira/browse/SOLR-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197900#comment-17197900
]
Cassandra Targett commented on SOLR-14824:
--
Sorry for delay looking at the patch...I'm a little distracted this week.
I think the changes to {{build.gradle}} that remove the asciidocAttrs section
are going to cause some unexpected consequences. There are a couple parameters
defined there that aren't in {{_config.yml.template}}, specifically:
{code}
-'attribute-missing' : 'warn',
-'icons' : 'font',
-'icon-set' : 'fa',
-'figure-caption!' : '',
-'idprefix' : '',
-'idseparator' : '-',
{code}
I couldn't get the patch to apply so can't check these but they're worth a look.
> Simplify set of Ref Guide build parameters
> --
>
> Key: SOLR-14824
> URL: https://issues.apache.org/jira/browse/SOLR-14824
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Build, documentation
>Reporter: Cassandra Targett
>Assignee: Chris M. Hostetter
>Priority: Minor
> Attachments: SOLR-14824.patch, SOLR-14824.patch
>
>
> While trying to solve LUCENE-9495, I thought it might be a good idea to try
> to simplify the set of variables and properties used during the Ref Guide
> build.
> There are 3 areas to work on:
> 1. Remove the "barebones-html" build. With Gradle the build is self-contained
> and {{gradlew check}} and {{gradle precommit}} could just build the full docs
> and check them.
> 2. Remove some properties that only existed for a hypothetical need related
> to the now-removed PDF.
> 3. Change remaining properties to be defined directly in build.gradle instead
> of relying on ant properties functionality.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-12891) Injection Dangers in Streaming Expressions
[ https://issues.apache.org/jira/browse/SOLR-12891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197901#comment-17197901 ] David Smiley commented on SOLR-12891: - I appreciate the attention to detail on identifying this problem, and the peer review here, but respectfully, IMO the committed solution of modifying MacroExpander to check for one special parameter is a total hack. It feels weird/yucky that user input is subject to parameter substitution -- "q" param or whatever else you concoct into other parameters. I'm not sure if the worst that could happen is merely lots of confusion (e.g. why does my attempt to search for code snippets not work when this syntax is used), or wether it might have sinister effects in some systems. I'd instead rather see macro expansion simply not work on parameters sent into the request, except maybe an allow-list that would include the "json" parameter and maybe one or two others, at most. I could also imagine a local-param for a query to explicitly opt-in. Parameters specified in configuration (request handler or params.json) would always be eligible for macro-expansion. WDYT? I realize I'm proposing a breaking change that would only be acceptable for a major release (e.g. 9.0). > Injection Dangers in Streaming Expressions > -- > > Key: SOLR-12891 > URL: https://issues.apache.org/jira/browse/SOLR-12891 > Project: Solr > Issue Type: Bug > Components: streaming expressions >Affects Versions: 7.5, 8.0 >Reporter: Gus Heck >Priority: Minor > Labels: security > Fix For: 8.1, master (9.0) > > Attachments: SOLR-12891.patch, SOLR-12891.patch, SOLR-12891.patch, > SOLR-12891.patch, SOLR12819example.java > > > I just spent some time fiddling with streaming expressions for fun, reading > Erick Erickson's blog > ([https://lucidworks.com/2017/12/06/streaming-expressions-in-solrj/)] and the > example given in the ref guide > ([https://lucene.apache.org/solr/guide/7_5/streaming-expressions.html#streaming-requests-and-responses)] > and it occurred to me that we are recommending string concatenation into an > expression language with the power to harm the server, or other network > services visible from the server. I'm starting this Jira as a security issue > to avoid creating a public impression of insecurity, feel free to undo that > if I have guessed wrong. I haven't developed an exploit example, but it would > go something like this: > # Some portion of an expression is built including user supplied data using > the techniques we're recommending in the ref guide > # Malicious user constructs input data that breaks out of the expression > (SOLR-10894 is relevant here), probably somewhere inside a let() expression > where one could simply define an additional variable taking the value of a > malicious expression... > # update() expression is executed to add/overwrite data, jdbc() makes a JDBC > connection to a database visible to the server, or the malicious expression > executes some very expensive expression for DOS effect. > Technically this is of course the fault of the end user who allowed unchecked > input into programmatic execution, but when I think about how to check the > input I realize that the only way to be sure is to construct for myself a > notion of exactly how the parser behaves and then determine what needs to be > escaped. To do this I need to dig into the expression parser code... > How to escape input is also already unclear as shown by SOLR-10894 > There's another important wrinkle that would easily be missed by someone > trying to construct their own escaping/protection system relating to > parameter substitution as discussed here: SOLR-8458 > I think the solution to this is that SolrJ API should be enhanced to provide > an escaping utility at a minimum and possibly a "prepared expression" similar > to SQL prepared statements and call this issue to attention in the ref guide > once these tools are available... > Additionally, templating features might be a useful addition to help folks > manage large expressions and facilitate re-use of patterns... such templating > should also have this issue in mind when/if they are added. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #1877: SOLR-13181: param macro expansion could throw
dsmiley commented on a change in pull request #1877:
URL: https://github.com/apache/lucene-solr/pull/1877#discussion_r490480292
##
File path: solr/core/src/java/org/apache/solr/request/macro/MacroExpander.java
##
@@ -136,14 +136,15 @@ private String _expand(String val) {
}
else if (idx < 0) {
if (sb == null) return val;
-sb.append(val.substring(start));
+sb.append(val, start, val.length());
return sb.toString();
}
// found unescaped "${"
+ final int matchedStart = idx;
idx += macroStart.length();
Review comment:
Can now remove idx manipulation until later when idx & start are changed
in one line
```suggestion
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9530) Clean up javacc generation script (gradle)
[ https://issues.apache.org/jira/browse/LUCENE-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197934#comment-17197934 ] Dawid Weiss commented on LUCENE-9530: - [~erickerickson] you may want to check out the javacc.gradle now - I think I did a fairly good job at moving from ant regexps to more, ahem, groovy approach. :) > Clean up javacc generation script (gradle) > -- > > Key: LUCENE-9530 > URL: https://issues.apache.org/jira/browse/LUCENE-9530 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Trivial > Fix For: master (9.0) > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14824) Simplify set of Ref Guide build parameters
[
https://issues.apache.org/jira/browse/SOLR-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris M. Hostetter updated SOLR-14824:
--
Attachment: SOLR-14824.patch
Status: Open (was: Open)
{quote}This looks weird (and comment is inconsistent with code).
{quote}
Ah, yeah – that comment was (obtusely) trying to explain that for the "X.Y" ref
guide we always link to the javadocs of the "X.Y.0" release. I've beefed up the
comments on the version & path variables to make it more clear their purpose &
"style" of values each one contains.
{quote}If this draft status is related to version then it could be
automatically detected (version.contains("SNAPSHOT")).
{quote}
That wouldn't work because by default 'version' (in the top level build.gradle
file) always contains the SNAPSHOT suffix by default (unless 'version.release'
or 'version.suffix' is overriden on the command line.)
The usecase here is that on "branch_9_7" regardless of whether 'baseVersion' is
9.7.0, or 9.7.1, etc... (making the default "version" 9.7.0-SNAPSHOT,
9.7.1-SNAPSHOT, etc..), we want to be able to "build" the "9.7" ref-guide, and
by default the builds should be "\-DRAFT" but optionally a committer should be
able to indicate that it's "non-DRAFT" so they can (re-)publish it to the site.
I suppose instead of using {{\-PsolrGuideDraft=false}} we could document that
the way to build an "non-DRAFT" copy of the guide is to use
{{\-Dversion.suffix=''}} ? ... but the approach in the patch seems easier for
the actual usecase/workflow/process of building & publishing the ref-guide.
(If/When we get to the point of wanting to build & publishing the ref-guide
automatically on every release, then it might make sense to drive the 'DRAFT"
behavior based on wether 'version' has a suffix on it – but let's cross that
bridge when we come to it)
{quote}For this issue I was mainly looking at the solrRootPath property that's
obviously not a relative URL ...
{quote}
Oh, sure – yeah. I fixed that to be based on {{project(':solr')}}
{quote}I think the changes to build.gradle that remove the asciidocAttrs
section are going to cause some unexpected consequences. There are a couple
parameters defined there that aren't in _config.yml.template, specifically:
{quote}
Those attributes have never affected the jekyll build: they've only ever been
used when gradle/ant ran asciidoctor directly for the bare-bones (and PDF)
builds. The ant build.xml file actually had a comment pointing out this
distinction, and I'm pretty sure the reason for that choice was because of how
jekyll uses asciidoctor slightly differently (for things like TOC), and they
were only needed by PDF renderer?
"attribute-missing" and "icons" are actually hard coded in
{{_config.yml.template}} (with those same values) but only in the "asciidoctor
> attributes" section, not in the "solr-attributes" YAML reference, so they
can't mistakenly be used as variables inside of content or in liquid
templates...
{noformat}
attributes:
<<: *solr-attributes-ref
attribute-missing: "warn"
icons: "font"
source-highlighter: "rouge"
rouge-theme: "thankful-eyes"
stem:
{noformat}
...if you think we should add the other 4 we can do that, but we should take
care and review the output carefully since they may (probably?) change the
look/feel of the content from what we currently get on master/8x since they
haven't been used in the jekyll builds up to now.
{quote}I couldn't get the patch to apply so can't check these but they're worth
a look.
{quote}
there were conflicts introduced by the removal of velocity ... the patching i'm
attaching now should apply clean.
> Simplify set of Ref Guide build parameters
> --
>
> Key: SOLR-14824
> URL: https://issues.apache.org/jira/browse/SOLR-14824
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Build, documentation
>Reporter: Cassandra Targett
>Assignee: Chris M. Hostetter
>Priority: Minor
> Attachments: SOLR-14824.patch, SOLR-14824.patch, SOLR-14824.patch
>
>
> While trying to solve LUCENE-9495, I thought it might be a good idea to try
> to simplify the set of variables and properties used during the Ref Guide
> build.
> There are 3 areas to work on:
> 1. Remove the "barebones-html" build. With Gradle the build is self-contained
> and {{gradlew check}} and {{gradle precommit}} could just build the full docs
> and check them.
> 2. Remove some properties that only existed for a hypothetical need related
> to the now-removed PDF.
> 3. Change remaining properties to be defined directly in build.gradle instead
> of relying on ant properties functionality.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---
[jira] [Commented] (SOLR-14871) Use Annotations for v2 APIs in/cluster path
[ https://issues.apache.org/jira/browse/SOLR-14871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197949#comment-17197949 ] ASF subversion and git services commented on SOLR-14871: Commit 5bc7fb286182eeca4e4d2c9ff9dfb98f6a399125 in lucene-solr's branch refs/heads/SOLR-14866 from noblepaul [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5bc7fb2 ] SOLR-14871: remove unused test > Use Annotations for v2 APIs in/cluster path > --- > > Key: SOLR-14871 > URL: https://issues.apache.org/jira/browse/SOLR-14871 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Noble Paul >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > We have custom json specs for each of these APIs. With the annotation > framework , it can be made simple and readable and we can eliminate a lot of > code -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9527) Upgrade javacc to 7.0.4
[ https://issues.apache.org/jira/browse/LUCENE-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197954#comment-17197954 ] ASF subversion and git services commented on LUCENE-9527: - Commit 6c9d7adf7985e4b895a80e561f1a3bb5820126a8 in lucene-solr's branch refs/heads/SOLR-14866 from Dawid Weiss [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=6c9d7ad ] LUCENE-9527: upgrade javacc to 7.0.4 (#1884) > Upgrade javacc to 7.0.4 > --- > > Key: LUCENE-9527 > URL: https://issues.apache.org/jira/browse/LUCENE-9527 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Minor > Fix For: master (9.0) > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14151) Make schema components load from packages
[
https://issues.apache.org/jira/browse/SOLR-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197950#comment-17197950
]
ASF subversion and git services commented on SOLR-14151:
Commit cbb1659640cd51be8b403eda8399c527af1c848e in lucene-solr's branch
refs/heads/SOLR-14866 from noblepaul
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=cbb1659 ]
Revert "Revert "SOLR-14151: Bug fixes (#1815)""
This reverts commit 27a14fe48139019a4c09ba072751e093fc5cb5f1.
Undoing accidental commit
> Make schema components load from packages
> -
>
> Key: SOLR-14151
> URL: https://issues.apache.org/jira/browse/SOLR-14151
> Project: Solr
> Issue Type: Sub-task
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Labels: packagemanager
> Fix For: 8.7
>
> Time Spent: 12h 50m
> Remaining Estimate: 0h
>
> Example:
> {code:xml}
>
>
>
>generateNumberParts="0" catenateWords="0"
> catenateNumbers="0" catenateAll="0"/>
>
>
>
>
> {code}
> * When a package is updated, the entire {{IndexSchema}} object is refreshed,
> but the SolrCore object is not reloaded
> * Any component can be prefixed with the package name
> * The semantics of loading plugins remain the same as that of the components
> in {{solrconfig.xml}}
> * Plugins can be registered using schema API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9530) Clean up javacc generation script (gradle)
[ https://issues.apache.org/jira/browse/LUCENE-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197952#comment-17197952 ] ASF subversion and git services commented on LUCENE-9530: - Commit 4f344cb0d4087171f59cc3c22265237bbeec50a4 in lucene-solr's branch refs/heads/SOLR-14866 from Dawid Weiss [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=4f344cb ] LUCENE-9530: cleaned up javacc gradle generation scripts. (#1883) * LUCENE-9530: cleaned up gradle javacc generation/ tweaks script so that it's consistent across runs. Removed ant remnants. > Clean up javacc generation script (gradle) > -- > > Key: LUCENE-9530 > URL: https://issues.apache.org/jira/browse/LUCENE-9530 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Trivial > Fix For: master (9.0) > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14792) Remove VelocityResponseWriter from Solr 9
[ https://issues.apache.org/jira/browse/SOLR-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197955#comment-17197955 ] ASF subversion and git services commented on SOLR-14792: Commit 2364a7aded9f04137eeb8520d69ffe97f7caef66 in lucene-solr's branch refs/heads/SOLR-14866 from Erik Hatcher [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2364a7a ] SOLR-14792: Remove VelocityResponseWriter > Remove VelocityResponseWriter from Solr 9 > - > > Key: SOLR-14792 > URL: https://issues.apache.org/jira/browse/SOLR-14792 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 9 >Reporter: Erik Hatcher >Priority: Blocker > Time Spent: 1h > Remaining Estimate: 0h > > VelocityResponseWriter was deprecated in SOLR-14065. It can now be removed > from 9's code branch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14613) Provide a clean API for pluggable replica assignment implementations
[
https://issues.apache.org/jira/browse/SOLR-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197956#comment-17197956
]
ASF subversion and git services commented on SOLR-14613:
Commit dbba48b3e56be541572f170a9fae95d039236c1a in lucene-solr's branch
refs/heads/SOLR-14866 from Ilan Ginzburg
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=dbba48b ]
SOLR-14613: use set-placement-plugin for both setting and unsetting plugin
config
> Provide a clean API for pluggable replica assignment implementations
>
>
> Key: SOLR-14613
> URL: https://issues.apache.org/jira/browse/SOLR-14613
> Project: Solr
> Issue Type: Improvement
> Components: AutoScaling
>Reporter: Andrzej Bialecki
>Assignee: Ilan Ginzburg
>Priority: Major
> Time Spent: 41h
> Remaining Estimate: 0h
>
> As described in SIP-8 the current autoscaling Policy implementation has
> several limitations that make it difficult to use for very large clusters and
> very large collections. SIP-8 also mentions the possible migration path by
> providing alternative implementations of the placement strategies that are
> less complex but more efficient in these very large environments.
> We should review the existing APIs that the current autoscaling engine uses
> ({{SolrCloudManager}} , {{AssignStrategy}} , {{Suggester}} and related
> interfaces) to see if they provide a sufficient and minimal API for plugging
> in alternative autoscaling placement strategies, and if necessary refactor
> the existing APIs.
> Since these APIs are internal it should be possible to do this without
> breaking back-compat.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14151) Make schema components load from packages
[
https://issues.apache.org/jira/browse/SOLR-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197953#comment-17197953
]
ASF subversion and git services commented on SOLR-14151:
Commit 515608a087f81c13ad22efba72e48fc64d9d5ae7 in lucene-solr's branch
refs/heads/SOLR-14866 from noblepaul
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=515608a ]
SOLR-14151: fixed the classloading issue
> Make schema components load from packages
> -
>
> Key: SOLR-14151
> URL: https://issues.apache.org/jira/browse/SOLR-14151
> Project: Solr
> Issue Type: Sub-task
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Labels: packagemanager
> Fix For: 8.7
>
> Time Spent: 12h 50m
> Remaining Estimate: 0h
>
> Example:
> {code:xml}
>
>
>
>generateNumberParts="0" catenateWords="0"
> catenateNumbers="0" catenateAll="0"/>
>
>
>
>
> {code}
> * When a package is updated, the entire {{IndexSchema}} object is refreshed,
> but the SolrCore object is not reloaded
> * Any component can be prefixed with the package name
> * The semantics of loading plugins remain the same as that of the components
> in {{solrconfig.xml}}
> * Plugins can be registered using schema API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9529) Larger stored fields block sizes mean we're more likely to disable optimized bulk merging
[ https://issues.apache.org/jira/browse/LUCENE-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197958#comment-17197958 ] ASF subversion and git services commented on LUCENE-9529: - Commit 33f728007853ee53c71f259206734bfd9155593d in lucene-solr's branch refs/heads/SOLR-14866 from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=33f7280 ] LUCENE-9529: Track dirtiness of stored fields via a number of docs, not chunks. (#1882) The problem of tracking dirtiness via numbers of chunks is that larger chunks make stored fields readers more likely to be considered dirty, so I'm trying to work around it by tracking numbers of docs instead. > Larger stored fields block sizes mean we're more likely to disable optimized > bulk merging > - > > Key: LUCENE-9529 > URL: https://issues.apache.org/jira/browse/LUCENE-9529 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Fix For: 8.7 > > Time Spent: 50m > Remaining Estimate: 0h > > Whenever possible when merging stored fields, Lucene tries to copy the > compressed data instead of decompressing the source segment to then > re-compressing in the destination segment. A problem with this approach is > that if some blocks are incomplete (typically the last block of a segment) > then it remains incomplete in the destination segment too, and if we do it > for too long we end up with a bad compression ratio. So Lucene keeps track of > these incomplete blocks, and makes sure to keep a ratio of incomplete blocks > below 1%. > But as we increased the block size, it has become more likely to have a high > ratio of incomplete blocks. E.g. if you have a segment with 1MB of stored > fields, with 16kB blocks like before, you have 63 complete blocks and 1 > incomplete block, or 1.6%. But now with ~512kB blocks, you have one complete > block and 1 incomplete block, ie. 50%. > I'm not sure how to fix it or even whether it should be fixed but wanted to > open an issue to track this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9530) Clean up javacc generation script (gradle)
[ https://issues.apache.org/jira/browse/LUCENE-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197951#comment-17197951 ] ASF subversion and git services commented on LUCENE-9530: - Commit 4f344cb0d4087171f59cc3c22265237bbeec50a4 in lucene-solr's branch refs/heads/SOLR-14866 from Dawid Weiss [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=4f344cb ] LUCENE-9530: cleaned up javacc gradle generation scripts. (#1883) * LUCENE-9530: cleaned up gradle javacc generation/ tweaks script so that it's consistent across runs. Removed ant remnants. > Clean up javacc generation script (gradle) > -- > > Key: LUCENE-9530 > URL: https://issues.apache.org/jira/browse/LUCENE-9530 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Trivial > Fix For: master (9.0) > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
