[jira] [Updated] (SOLR-14237) Add authenticated user principal in Solr admin UI
[ https://issues.apache.org/jira/browse/SOLR-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-14237: Attachment: SOLR-14237-2.patch Screenshot from 2020-04-28 12-48-45.png Status: Open (was: Open) Here's a final patch. Added a "security" panel on the UI's dashboard that shows the authentication and authorization plugins being used, and the current user and his/her roles. !Screenshot from 2020-04-28 12-48-45.png! > Add authenticated user principal in Solr admin UI > - > > Key: SOLR-14237 > URL: https://issues.apache.org/jira/browse/SOLR-14237 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Reporter: mosh >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: AdminUI, kerberos, security > Attachments: SOLR-14237-2.patch, SOLR-14237.patch, Screenshot from > 2020-04-28 12-48-45.png > > > When user is logged in to Solr's admin UI using Kerberos, no authentication > info is displayed. > It would be very useful to see the logged in user principal. This could be > specially crucial when SSO is being used and user not always aware that Solr > is even configured with authentication mechanism. > +Info should include:+ > 1. user principal > 2. mapped role (in case authorization plugin is also configured) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14237) Add authenticated user principal in Solr admin UI
[ https://issues.apache.org/jira/browse/SOLR-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094236#comment-17094236 ] Ishan Chattopadhyaya edited comment on SOLR-14237 at 4/28/20, 7:25 AM: --- Here's a final patch. Added a "security" panel on the UI's dashboard that shows the authentication and authorization plugins being used, and the current user and his/her roles. !Screenshot from 2020-04-28 12-48-45.png|width=600! was (Author: ichattopadhyaya): Here's a final patch. Added a "security" panel on the UI's dashboard that shows the authentication and authorization plugins being used, and the current user and his/her roles. !Screenshot from 2020-04-28 12-48-45.png! > Add authenticated user principal in Solr admin UI > - > > Key: SOLR-14237 > URL: https://issues.apache.org/jira/browse/SOLR-14237 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Reporter: mosh >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: AdminUI, kerberos, security > Attachments: SOLR-14237-2.patch, SOLR-14237.patch, Screenshot from > 2020-04-28 12-48-45.png > > > When user is logged in to Solr's admin UI using Kerberos, no authentication > info is displayed. > It would be very useful to see the logged in user principal. This could be > specially crucial when SSO is being used and user not always aware that Solr > is even configured with authentication mechanism. > +Info should include:+ > 1. user principal > 2. mapped role (in case authorization plugin is also configured) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14237) Add authenticated user principal in Solr admin UI
[ https://issues.apache.org/jira/browse/SOLR-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-14237: Attachment: Screenshot from 2020-04-28 12-48-45.png SOLR-14237-2.patch Status: Open (was: Open) > Add authenticated user principal in Solr admin UI > - > > Key: SOLR-14237 > URL: https://issues.apache.org/jira/browse/SOLR-14237 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Reporter: mosh >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: AdminUI, kerberos, security > Attachments: SOLR-14237-2.patch, SOLR-14237-2.patch, > SOLR-14237.patch, Screenshot from 2020-04-28 12-48-45.png > > > When user is logged in to Solr's admin UI using Kerberos, no authentication > info is displayed. > It would be very useful to see the logged in user principal. This could be > specially crucial when SSO is being used and user not always aware that Solr > is even configured with authentication mechanism. > +Info should include:+ > 1. user principal > 2. mapped role (in case authorization plugin is also configured) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14237) Add authenticated user principal in Solr admin UI
[ https://issues.apache.org/jira/browse/SOLR-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-14237: Attachment: (was: Screenshot from 2020-04-28 12-48-45.png) > Add authenticated user principal in Solr admin UI > - > > Key: SOLR-14237 > URL: https://issues.apache.org/jira/browse/SOLR-14237 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Reporter: mosh >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: AdminUI, kerberos, security > Attachments: SOLR-14237-2.patch, SOLR-14237-2.patch, > SOLR-14237.patch, Screenshot from 2020-04-28 12-48-45.png > > > When user is logged in to Solr's admin UI using Kerberos, no authentication > info is displayed. > It would be very useful to see the logged in user principal. This could be > specially crucial when SSO is being used and user not always aware that Solr > is even configured with authentication mechanism. > +Info should include:+ > 1. user principal > 2. mapped role (in case authorization plugin is also configured) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14237) Add authenticated user principal in Solr admin UI
[ https://issues.apache.org/jira/browse/SOLR-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-14237: Attachment: (was: SOLR-14237-2.patch) > Add authenticated user principal in Solr admin UI > - > > Key: SOLR-14237 > URL: https://issues.apache.org/jira/browse/SOLR-14237 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Reporter: mosh >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: AdminUI, kerberos, security > Attachments: SOLR-14237-2.patch, SOLR-14237.patch, Screenshot from > 2020-04-28 12-48-45.png > > > When user is logged in to Solr's admin UI using Kerberos, no authentication > info is displayed. > It would be very useful to see the logged in user principal. This could be > specially crucial when SSO is being used and user not always aware that Solr > is even configured with authentication mechanism. > +Info should include:+ > 1. user principal > 2. mapped role (in case authorization plugin is also configured) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14237) Add authenticated user principal in Solr admin UI
[ https://issues.apache.org/jira/browse/SOLR-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094244#comment-17094244 ] Noble Paul commented on SOLR-14237: --- The changes to the Authorization plugin is fine. good to go in > Add authenticated user principal in Solr admin UI > - > > Key: SOLR-14237 > URL: https://issues.apache.org/jira/browse/SOLR-14237 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Reporter: mosh >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: AdminUI, kerberos, security > Attachments: SOLR-14237-2.patch, SOLR-14237.patch, Screenshot from > 2020-04-28 12-48-45.png > > > When user is logged in to Solr's admin UI using Kerberos, no authentication > info is displayed. > It would be very useful to see the logged in user principal. This could be > specially crucial when SSO is being used and user not always aware that Solr > is even configured with authentication mechanism. > +Info should include:+ > 1. user principal > 2. mapped role (in case authorization plugin is also configured) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14237) Add authenticated user principal in Solr admin UI
[ https://issues.apache.org/jira/browse/SOLR-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094252#comment-17094252 ] Ishan Chattopadhyaya commented on SOLR-14237: - Thanks [~noble.paul]. I'll commit this soon, if no one has any objections. (As an improvement in a separate issue, we can add a warning in that panel if authc/authz is not configured, with a link to the ref guide). > Add authenticated user principal in Solr admin UI > - > > Key: SOLR-14237 > URL: https://issues.apache.org/jira/browse/SOLR-14237 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Reporter: mosh >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: AdminUI, kerberos, security > Attachments: SOLR-14237-2.patch, SOLR-14237.patch, Screenshot from > 2020-04-28 12-48-45.png > > > When user is logged in to Solr's admin UI using Kerberos, no authentication > info is displayed. > It would be very useful to see the logged in user principal. This could be > specially crucial when SSO is being used and user not always aware that Solr > is even configured with authentication mechanism. > +Info should include:+ > 1. user principal > 2. mapped role (in case authorization plugin is also configured) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14237) Add authenticated user principal in Solr admin UI
[ https://issues.apache.org/jira/browse/SOLR-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094254#comment-17094254 ] Ishan Chattopadhyaya commented on SOLR-14237: - [~moshebla] please review. > Add authenticated user principal in Solr admin UI > - > > Key: SOLR-14237 > URL: https://issues.apache.org/jira/browse/SOLR-14237 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Reporter: mosh >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: AdminUI, kerberos, security > Attachments: SOLR-14237-2.patch, SOLR-14237.patch, Screenshot from > 2020-04-28 12-48-45.png > > > When user is logged in to Solr's admin UI using Kerberos, no authentication > info is displayed. > It would be very useful to see the logged in user principal. This could be > specially crucial when SSO is being used and user not always aware that Solr > is even configured with authentication mechanism. > +Info should include:+ > 1. user principal > 2. mapped role (in case authorization plugin is also configured) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14237) Add authenticated user principal in Solr admin UI
[ https://issues.apache.org/jira/browse/SOLR-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Høydahl reassigned SOLR-14237: -- Assignee: Ishan Chattopadhyaya (was: Jan Høydahl) > Add authenticated user principal in Solr admin UI > - > > Key: SOLR-14237 > URL: https://issues.apache.org/jira/browse/SOLR-14237 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Reporter: mosh >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: AdminUI, kerberos, security > Attachments: SOLR-14237-2.patch, SOLR-14237.patch, Screenshot from > 2020-04-28 12-48-45.png > > > When user is logged in to Solr's admin UI using Kerberos, no authentication > info is displayed. > It would be very useful to see the logged in user principal. This could be > specially crucial when SSO is being used and user not always aware that Solr > is even configured with authentication mechanism. > +Info should include:+ > 1. user principal > 2. mapped role (in case authorization plugin is also configured) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14237) Add authenticated user principal in Solr admin UI
[ https://issues.apache.org/jira/browse/SOLR-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094288#comment-17094288 ] Jan Høydahl commented on SOLR-14237: Nice! Perhaps this can be a starting point for more fine grained role handling in the UI as well, in a followup issue. Imagine if we could somehow map the users roles to a list of permissions. Then the UI app could show/hide/disable various parts of the UI depending on what permissions the user has. Now you are thrown back to the login screen if you get a 401/403 which is not bad, but we could probably do better. > Add authenticated user principal in Solr admin UI > - > > Key: SOLR-14237 > URL: https://issues.apache.org/jira/browse/SOLR-14237 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Reporter: mosh >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: AdminUI, kerberos, security > Attachments: SOLR-14237-2.patch, SOLR-14237.patch, Screenshot from > 2020-04-28 12-48-45.png > > > When user is logged in to Solr's admin UI using Kerberos, no authentication > info is displayed. > It would be very useful to see the logged in user principal. This could be > specially crucial when SSO is being used and user not always aware that Solr > is even configured with authentication mechanism. > +Info should include:+ > 1. user principal > 2. mapped role (in case authorization plugin is also configured) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14237) Add authenticated user principal in Solr admin UI
[ https://issues.apache.org/jira/browse/SOLR-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Høydahl reassigned SOLR-14237: -- Assignee: Jan Høydahl (was: Ishan Chattopadhyaya) > Add authenticated user principal in Solr admin UI > - > > Key: SOLR-14237 > URL: https://issues.apache.org/jira/browse/SOLR-14237 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Reporter: mosh >Assignee: Jan Høydahl >Priority: Major > Labels: AdminUI, kerberos, security > Attachments: SOLR-14237-2.patch, SOLR-14237.patch, Screenshot from > 2020-04-28 12-48-45.png > > > When user is logged in to Solr's admin UI using Kerberos, no authentication > info is displayed. > It would be very useful to see the logged in user principal. This could be > specially crucial when SSO is being used and user not always aware that Solr > is even configured with authentication mechanism. > +Info should include:+ > 1. user principal > 2. mapped role (in case authorization plugin is also configured) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14237) Add authenticated user principal in Solr admin UI
[
https://issues.apache.org/jira/browse/SOLR-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094296#comment-17094296
]
Ishan Chattopadhyaya commented on SOLR-14237:
-
{quote}Imagine if we could somehow map the users roles to a list of
permissions. Then the UI app could show/hide/disable various parts of the UI
depending on what permissions the user has.
{quote}
That is a great idea!
> Add authenticated user principal in Solr admin UI
> -
>
> Key: SOLR-14237
> URL: https://issues.apache.org/jira/browse/SOLR-14237
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Admin UI
>Reporter: mosh
>Assignee: Ishan Chattopadhyaya
>Priority: Major
> Labels: AdminUI, kerberos, security
> Attachments: SOLR-14237-2.patch, SOLR-14237.patch, Screenshot from
> 2020-04-28 12-48-45.png
>
>
> When user is logged in to Solr's admin UI using Kerberos, no authentication
> info is displayed.
> It would be very useful to see the logged in user principal. This could be
> specially crucial when SSO is being used and user not always aware that Solr
> is even configured with authentication mechanism.
> +Info should include:+
> 1. user principal
> 2. mapped role (in case authorization plugin is also configured)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14428) FuzzyQuery has severe memory usage in 8.5
[
https://issues.apache.org/jira/browse/SOLR-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094304#comment-17094304
]
Alan Woodward commented on SOLR-14428:
--
There's some prior art on this one:
https://issues.apache.org/jira/browse/LUCENE-8855
> Or maybe we just accept that sometimes, queries can be big
I think we have to go with this, really - for example, TermInSetQuery can get
very big. FuzzyQuery is probably particularly prone to this because it builds
multiple Levenshtein Automata, which are sizeable in and of themselves. We've
generally dealt with this in the past by saying that large queries shouldn't be
cacheable, but I think that's the wrong trade-off in this situation - as a
heavy query that can take a lot of resources to compute, FuzzyQuery is a really
good candidate for caching. I like the idea of having a separate cache key for
large queries; we can have a default implementation on Query that just returns
`this`, and then heavy queries can have a specialised class that they return.
> FuzzyQuery has severe memory usage in 8.5
> -
>
> Key: SOLR-14428
> URL: https://issues.apache.org/jira/browse/SOLR-14428
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.5, 8.5.1
>Reporter: Colvin Cowie
>Assignee: Andrzej Bialecki
>Priority: Major
> Attachments: FuzzyHammer.java, SOLR-14428-WeakReferences.patch,
> image-2020-04-23-09-18-06-070.png, image-2020-04-24-20-09-31-179.png,
> screenshot-2.png, screenshot-3.png, screenshot-4.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I sent this to the mailing list
> I'm moving from 8.3.1 to 8.5.1, and started getting Out Of Memory Errors
> while running our normal tests. After profiling it was clear that the
> majority of the heap was allocated through FuzzyQuery.
> LUCENE-9068 moved construction of the automata from the FuzzyTermsEnum to the
> FuzzyQuery's constructor.
> I created a little test ( [^FuzzyHammer.java] ) that fires off fuzzy queries
> from random UUID strings for 5 minutes
> {code}
> FIELD_NAME + ":" + UUID.randomUUID().toString().replace("-", "") + "~2"
> {code}
> When running against a vanilla Solr 8.31 and 8.4.1 there is no problem, while
> the memory usage has increased drastically on 8.5.0 and 8.5.1.
> Comparison of heap usage while running the attached test against Solr 8.3.1
> and 8.5.1 with a single (empty) shard and 4GB heap:
> !image-2020-04-23-09-18-06-070.png!
> And with 4 shards on 8.4.1 and 8.5.0:
> !screenshot-2.png!
> I'm guessing that the memory might be being leaked if the FuzzyQuery objects
> are referenced from the cache, while the FuzzyTermsEnum would not have been.
> Query Result Cache on 8.5.1:
> !screenshot-3.png!
> ~316mb in the cache
> QRC on 8.3.1
> !screenshot-4.png!
> <1mb
> With an empty cache, running this query
> _field_s:e41848af85d24ac197c71db6888e17bc~2_ results in the following memory
> allocation
> {noformat}
> 8.3.1: CACHE.searcher.queryResultCache.ramBytesUsed: 1520
> 8.5.1: CACHE.searcher.queryResultCache.ramBytesUsed:648855
> {noformat}
> ~1 gives 98253 and ~0 gives 6339 on 8.5.1. 8.3.1 is constant at 1520
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14428) FuzzyQuery has severe memory usage in 8.5
[
https://issues.apache.org/jira/browse/SOLR-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094309#comment-17094309
]
Colvin Cowie commented on SOLR-14428:
-
{quote}The QueryResultKey could cache by the original string input instead of
the Query object, thus it wouldn't be affected. This would short-circuit query
parsing and be a performance benefit as well?
{quote}
On this point specifically, I think the downside to that would be that any
variations of query strings which can be rewritten to Query objects that are
equal() wouldn't get hits that they currently do, e.g. 'A or B' and 'B or A'
could create equal() Queries. And if the behaviour of the query parser is
modified by params - rather than local params - then the cached results may be
wrong if they're only keyed by the query string, right?
Or do you mean only use the String in the case of excessivly large Query
objects? In which case the first point is a reasonable compromise. I'm not sure
about the second though.
{quote}BTW This issue should probably be a Lucene JIRA issue but let's see
where we go with this thread further.
{quote}
I don't think I can change it now myself. But maybe there should be separate
issue for doing some nice and generic to deal with caching (large) Query
objects which I imagine might be a broader effort, and this/another for
resolving the immediate issue with FuzzyQuery?
> FuzzyQuery has severe memory usage in 8.5
> -
>
> Key: SOLR-14428
> URL: https://issues.apache.org/jira/browse/SOLR-14428
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.5, 8.5.1
>Reporter: Colvin Cowie
>Assignee: Andrzej Bialecki
>Priority: Major
> Attachments: FuzzyHammer.java, SOLR-14428-WeakReferences.patch,
> image-2020-04-23-09-18-06-070.png, image-2020-04-24-20-09-31-179.png,
> screenshot-2.png, screenshot-3.png, screenshot-4.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I sent this to the mailing list
> I'm moving from 8.3.1 to 8.5.1, and started getting Out Of Memory Errors
> while running our normal tests. After profiling it was clear that the
> majority of the heap was allocated through FuzzyQuery.
> LUCENE-9068 moved construction of the automata from the FuzzyTermsEnum to the
> FuzzyQuery's constructor.
> I created a little test ( [^FuzzyHammer.java] ) that fires off fuzzy queries
> from random UUID strings for 5 minutes
> {code}
> FIELD_NAME + ":" + UUID.randomUUID().toString().replace("-", "") + "~2"
> {code}
> When running against a vanilla Solr 8.31 and 8.4.1 there is no problem, while
> the memory usage has increased drastically on 8.5.0 and 8.5.1.
> Comparison of heap usage while running the attached test against Solr 8.3.1
> and 8.5.1 with a single (empty) shard and 4GB heap:
> !image-2020-04-23-09-18-06-070.png!
> And with 4 shards on 8.4.1 and 8.5.0:
> !screenshot-2.png!
> I'm guessing that the memory might be being leaked if the FuzzyQuery objects
> are referenced from the cache, while the FuzzyTermsEnum would not have been.
> Query Result Cache on 8.5.1:
> !screenshot-3.png!
> ~316mb in the cache
> QRC on 8.3.1
> !screenshot-4.png!
> <1mb
> With an empty cache, running this query
> _field_s:e41848af85d24ac197c71db6888e17bc~2_ results in the following memory
> allocation
> {noformat}
> 8.3.1: CACHE.searcher.queryResultCache.ramBytesUsed: 1520
> 8.5.1: CACHE.searcher.queryResultCache.ramBytesUsed:648855
> {noformat}
> ~1 gives 98253 and ~0 gives 6339 on 8.5.1. 8.3.1 is constant at 1520
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8811) Add maximum clause count check to IndexSearcher rather than BooleanQuery
[ https://issues.apache.org/jira/browse/LUCENE-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094321#comment-17094321 ] Alan Woodward commented on LUCENE-8811: --- Hi [~zabetak], the idea is to try and prevent runaway query execution; TermInSetQuery is implemented so that it can handle a very large number of terms efficiently, while the equivalent boolean disjunction would be very slow and use lots of memory. So we primarily want to run this check on BooleanQuery, but we also want to handle nested booleans, booleans wrapped in other queries, etc, in which case a QueryVisitor makes most sense. > I see that TermInSetQuery#visit for example already iterates through all > terms so if we don't need then we are just wasting CPU cycles without a very > good reason. This should definitely be fixed! I think we probably want to change TermInSetQuery#visit to call consumeMatchingTerms() with a lazy runAutomaton implementation; the num clauses check visitor also needs to count consumeMatchingTerms calls. > Add maximum clause count check to IndexSearcher rather than BooleanQuery > > > Key: LUCENE-8811 > URL: https://issues.apache.org/jira/browse/LUCENE-8811 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Assignee: Alan Woodward >Priority: Minor > Fix For: master (9.0) > > Attachments: LUCENE-8811.patch, LUCENE-8811.patch, LUCENE-8811.patch, > LUCENE-8811.patch, LUCENE-8811.patch, LUCENE-8811.patch > > > Currently we only check whether boolean queries have too many clauses. > However there are other ways that queries may have too many clauses, for > instance if you have boolean queries that have themselves inner boolean > queries. > Could we use the new Query visitor API to move this check from BooleanQuery > to IndexSearcher in order to make this check more consistent across queries? > See for instance LUCENE-8810 where a rewrite rule caused the maximum clause > count to be hit even though the total number of leaf queries remained the > same. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14409) Existing violations allow bypassing policy rules when adding new replicas
[
https://issues.apache.org/jira/browse/SOLR-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrzej Bialecki updated SOLR-14409:
Priority: Blocker (was: Critical)
> Existing violations allow bypassing policy rules when adding new replicas
> -
>
> Key: SOLR-14409
> URL: https://issues.apache.org/jira/browse/SOLR-14409
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
>Affects Versions: master (9.0), 8.5, 8.6
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Blocker
> Attachments: SOLR-14409.patch
>
>
> Steps to reproduce:
> * start with an empty cluster policy.
> * create a collection with as many replicas as there are nodes.
> * add one more replica to any node. Now this node has two replicas, all
> other nodes have one.
> * define the following cluster policy:
> {code:java}
> { 'set-cluster-policy': [ {'replica': '<2', 'shard': '#ANY', 'node': '#ANY',
> 'strict': true} ] } {code}
> This automatically creates a violation because of the existing layout.
> * try adding one more replica. This should fail because no node satisfies
> the rules (there must be at most 1 replica per node). However, the command
> succeeds and adds replica to the node that already has 2 replicas, which
> clearly violates the policy and makes matters even worse.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14409) Existing violations allow bypassing policy rules when adding new replicas
[
https://issues.apache.org/jira/browse/SOLR-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094356#comment-17094356
]
Andrzej Bialecki commented on SOLR-14409:
-
Escalating to Blocker - doesn't make sense to release 8.6 without a fix for
this issue.
> Existing violations allow bypassing policy rules when adding new replicas
> -
>
> Key: SOLR-14409
> URL: https://issues.apache.org/jira/browse/SOLR-14409
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
>Affects Versions: master (9.0), 8.5, 8.6
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Blocker
> Attachments: SOLR-14409.patch
>
>
> Steps to reproduce:
> * start with an empty cluster policy.
> * create a collection with as many replicas as there are nodes.
> * add one more replica to any node. Now this node has two replicas, all
> other nodes have one.
> * define the following cluster policy:
> {code:java}
> { 'set-cluster-policy': [ {'replica': '<2', 'shard': '#ANY', 'node': '#ANY',
> 'strict': true} ] } {code}
> This automatically creates a violation because of the existing layout.
> * try adding one more replica. This should fail because no node satisfies
> the rules (there must be at most 1 replica per node). However, the command
> succeeds and adds replica to the node that already has 2 replicas, which
> clearly violates the policy and makes matters even worse.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] iverase commented on pull request #1324: LUCENE-9033 Update ReleaseWizard for new website instructions
iverase commented on pull request #1324: URL: https://github.com/apache/lucene-solr/pull/1324#issuecomment-620513754 Thanks @janhoy, I will review it in the next few days. Thanks you so much! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on pull request #1424: Enable shared store via system property only
janhoy commented on pull request #1424: URL: https://github.com/apache/lucene-solr/pull/1424#issuecomment-620517628 Please link this PR with SOLR-13101 by prefixing the title. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on pull request #807: Remove solr.jetty.https.port when SSL is not used
janhoy commented on pull request #807: URL: https://github.com/apache/lucene-solr/pull/807#issuecomment-620518940 @upendrasoft can you please file a JIRA issue for this and prefix the title of the PR with the JIRA number? Also add a line to CHANGES.txt. Also see the rest of the checklist above. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14428) FuzzyQuery has severe memory usage in 8.5
[
https://issues.apache.org/jira/browse/SOLR-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094401#comment-17094401
]
Adrien Grand commented on SOLR-14428:
-
We currently store automata on the Query because it's more convenient given the
state of the MultiTermQuery API, but they really belong to a Weight? If we
fixed it then we wouldn't have memory issues with the cache anymore. I don't
think we need to give queries the ability to produce cache keys. Queries are
abstractions for an information need, which is typically something that users
would type in a search box. If users can express an information need in a few
characters, then there is no reason why the Query representation would take
much more memory?
> FuzzyQuery has severe memory usage in 8.5
> -
>
> Key: SOLR-14428
> URL: https://issues.apache.org/jira/browse/SOLR-14428
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.5, 8.5.1
>Reporter: Colvin Cowie
>Assignee: Andrzej Bialecki
>Priority: Major
> Attachments: FuzzyHammer.java, SOLR-14428-WeakReferences.patch,
> image-2020-04-23-09-18-06-070.png, image-2020-04-24-20-09-31-179.png,
> screenshot-2.png, screenshot-3.png, screenshot-4.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I sent this to the mailing list
> I'm moving from 8.3.1 to 8.5.1, and started getting Out Of Memory Errors
> while running our normal tests. After profiling it was clear that the
> majority of the heap was allocated through FuzzyQuery.
> LUCENE-9068 moved construction of the automata from the FuzzyTermsEnum to the
> FuzzyQuery's constructor.
> I created a little test ( [^FuzzyHammer.java] ) that fires off fuzzy queries
> from random UUID strings for 5 minutes
> {code}
> FIELD_NAME + ":" + UUID.randomUUID().toString().replace("-", "") + "~2"
> {code}
> When running against a vanilla Solr 8.31 and 8.4.1 there is no problem, while
> the memory usage has increased drastically on 8.5.0 and 8.5.1.
> Comparison of heap usage while running the attached test against Solr 8.3.1
> and 8.5.1 with a single (empty) shard and 4GB heap:
> !image-2020-04-23-09-18-06-070.png!
> And with 4 shards on 8.4.1 and 8.5.0:
> !screenshot-2.png!
> I'm guessing that the memory might be being leaked if the FuzzyQuery objects
> are referenced from the cache, while the FuzzyTermsEnum would not have been.
> Query Result Cache on 8.5.1:
> !screenshot-3.png!
> ~316mb in the cache
> QRC on 8.3.1
> !screenshot-4.png!
> <1mb
> With an empty cache, running this query
> _field_s:e41848af85d24ac197c71db6888e17bc~2_ results in the following memory
> allocation
> {noformat}
> 8.3.1: CACHE.searcher.queryResultCache.ramBytesUsed: 1520
> 8.5.1: CACHE.searcher.queryResultCache.ramBytesUsed:648855
> {noformat}
> ~1 gives 98253 and ~0 gives 6339 on 8.5.1. 8.3.1 is constant at 1520
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14428) FuzzyQuery has severe memory usage in 8.5
[
https://issues.apache.org/jira/browse/SOLR-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094411#comment-17094411
]
Alan Woodward commented on SOLR-14428:
--
I think we can try building the automata in `getTermsEnum()`, and adding some
laziness in `visit()` to prevent it being built unnecessarily. I'll open a PR.
> FuzzyQuery has severe memory usage in 8.5
> -
>
> Key: SOLR-14428
> URL: https://issues.apache.org/jira/browse/SOLR-14428
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.5, 8.5.1
>Reporter: Colvin Cowie
>Assignee: Andrzej Bialecki
>Priority: Major
> Attachments: FuzzyHammer.java, SOLR-14428-WeakReferences.patch,
> image-2020-04-23-09-18-06-070.png, image-2020-04-24-20-09-31-179.png,
> screenshot-2.png, screenshot-3.png, screenshot-4.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I sent this to the mailing list
> I'm moving from 8.3.1 to 8.5.1, and started getting Out Of Memory Errors
> while running our normal tests. After profiling it was clear that the
> majority of the heap was allocated through FuzzyQuery.
> LUCENE-9068 moved construction of the automata from the FuzzyTermsEnum to the
> FuzzyQuery's constructor.
> I created a little test ( [^FuzzyHammer.java] ) that fires off fuzzy queries
> from random UUID strings for 5 minutes
> {code}
> FIELD_NAME + ":" + UUID.randomUUID().toString().replace("-", "") + "~2"
> {code}
> When running against a vanilla Solr 8.31 and 8.4.1 there is no problem, while
> the memory usage has increased drastically on 8.5.0 and 8.5.1.
> Comparison of heap usage while running the attached test against Solr 8.3.1
> and 8.5.1 with a single (empty) shard and 4GB heap:
> !image-2020-04-23-09-18-06-070.png!
> And with 4 shards on 8.4.1 and 8.5.0:
> !screenshot-2.png!
> I'm guessing that the memory might be being leaked if the FuzzyQuery objects
> are referenced from the cache, while the FuzzyTermsEnum would not have been.
> Query Result Cache on 8.5.1:
> !screenshot-3.png!
> ~316mb in the cache
> QRC on 8.3.1
> !screenshot-4.png!
> <1mb
> With an empty cache, running this query
> _field_s:e41848af85d24ac197c71db6888e17bc~2_ results in the following memory
> allocation
> {noformat}
> 8.3.1: CACHE.searcher.queryResultCache.ramBytesUsed: 1520
> 8.5.1: CACHE.searcher.queryResultCache.ramBytesUsed:648855
> {noformat}
> ~1 gives 98253 and ~0 gives 6339 on 8.5.1. 8.3.1 is constant at 1520
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9089) FST.Builder with fluent-style constructor
[ https://issues.apache.org/jira/browse/LUCENE-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomoko Uchida updated LUCENE-9089: -- Attachment: fix-fst-package-summary.patch > FST.Builder with fluent-style constructor > - > > Key: LUCENE-9089 > URL: https://issues.apache.org/jira/browse/LUCENE-9089 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Bruno Roustant >Assignee: Bruno Roustant >Priority: Minor > Fix For: master (9.0) > > Attachments: fix-fst-package-summary.patch > > Time Spent: 2.5h > Remaining Estimate: 0h > > A first step in a try to make the FST code easier to read and evolve. This > step is just about the FST Builder constructor. > By making it fluent, the many calls to it are simplified and it becomes easy > to spot the intent and special param tuning. > No functional change. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9089) FST.Builder with fluent-style constructor
[ https://issues.apache.org/jira/browse/LUCENE-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094469#comment-17094469 ] Tomoko Uchida commented on LUCENE-9089: --- Hi, I happened to notice that the package summary for o.a.l.u.fst has been a bit obsoleted by a few changes including this issue (and this is the latest one). Updated example code on [^fix-fst-package-summary.patch] works for me. Is it okay if I commit it to the master as a follow-up here, say "LUCENE-9089: update FST usage example", without opening an issue / adding changes entry for that? > FST.Builder with fluent-style constructor > - > > Key: LUCENE-9089 > URL: https://issues.apache.org/jira/browse/LUCENE-9089 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Bruno Roustant >Assignee: Bruno Roustant >Priority: Minor > Fix For: master (9.0) > > Attachments: fix-fst-package-summary.patch > > Time Spent: 2.5h > Remaining Estimate: 0h > > A first step in a try to make the FST code easier to read and evolve. This > step is just about the FST Builder constructor. > By making it fluent, the many calls to it are simplified and it becomes easy > to spot the intent and special param tuning. > No functional change. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14173) Ref Guide Redesign
[
https://issues.apache.org/jira/browse/SOLR-14173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094477#comment-17094477
]
Cassandra Targett commented on SOLR-14173:
--
bq. You could share on the user-list to get more input.
I thought of doing that, but where the draft lives is in my personal Apache
space and I worried about some possibly retaining that as some new place to
access the Ref Guide. It happened before when we were transitioning from
Confluence to the current approach.
I also wasn't interested in re-opening the "Why don't we have search"
discussion and getting 10 strongly held opinions about that instead of what I'd
actually like to hear opinions about.
I thought about pushing a DRAFT for 8.6, but that would mean it's in
branch_8_x, but if it's not in master yet it's not in the branch either (unless
I make those branches locally).
> Ref Guide Redesign
> --
>
> Key: SOLR-14173
> URL: https://issues.apache.org/jira/browse/SOLR-14173
> Project: Solr
> Issue Type: Improvement
> Components: documentation
>Reporter: Cassandra Targett
>Assignee: Cassandra Targett
>Priority: Major
> Attachments: SOLR-14173.patch, blue-left-nav.png, gray-left-nav.png
>
>
> The current design of the Ref Guide was essentially copied from a
> Jekyll-based documentation theme
> (https://idratherbewriting.com/documentation-theme-jekyll/), which had a
> couple important benefits for that time:
> * It was well-documented and since I had little experience with Jekyll and
> its Liquid templates and since I was the one doing it, I wanted to make it as
> easy on myself as possible
> * It was designed for documentation specifically so took care of all the
> things like inter-page navigation, etc.
> * It helped us get from Confluence to our current system quickly
> It had some drawbacks, though:
> * It wasted a lot of space on the page
> * The theme was built for Markdown files, so did not take advantage of the
> features of the {{jekyll-asciidoc}} plugin we use (the in-page TOC being one
> big example - the plugin could create it at build time, but the theme
> included JS to do it as the page loads, so we use the JS)
> * It had a lot of JS and overlapping CSS files. While it used Bootstrap it
> used a customized CSS on top of it for theming that made modifications
> complex (it was hard to figure out how exactly a change would behave)
> * With all the stuff I'd changed in my bumbling way just to get things to
> work back then, I broke a lot of the stuff Bootstrap is supposed to give us
> in terms of responsiveness and making the Guide usable even on smaller screen
> sizes.
> After upgrading the Asciidoctor components in SOLR-12786 and stopping the PDF
> (SOLR-13782), I wanted to try to set us up for a more flexible system. We
> need it for things like Joel's work on the visual guide for streaming
> expressions (SOLR-13105), and in order to implement other ideas we might have
> on how to present information in the future.
> I view this issue as a phase 1 of an overall redesign that I've already
> started in a local branch. I'll explain in a comment the changes I've already
> made, and will use this issue to create and push a branch where we can
> discuss in more detail.
> Phase 1 here will be under-the-hood CSS/JS changes + overall page layout
> changes.
> Phase 2 (issue TBD) will be a wholesale re-organization of all the pages of
> the Guide.
> Phase 3 (issue TBD) will explore moving us from Jekyll to another static site
> generator that is better suited for our content format, file types, and build
> conventions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14173) Ref Guide Redesign
[
https://issues.apache.org/jira/browse/SOLR-14173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094477#comment-17094477
]
Cassandra Targett edited comment on SOLR-14173 at 4/28/20, 12:58 PM:
-
bq. You could share on the user-list to get more input.
I thought of doing that, but where the draft lives is in my personal Apache
space and I worried about some possibly retaining that as some new place to
access the Ref Guide. It happened before when we were transitioning from
Confluence to the current approach.
I also wasn't interested in re-opening the "Why don't we have search"
discussion and getting 10 strongly held opinions about that instead of what I'd
actually like to hear opinions about.
I thought about pushing a DRAFT for 8.6 to our site, but that would mean it's
in branch_8_x, but if it's not in master yet it's not in the branch either
(unless I make those branches locally).
was (Author: ctargett):
bq. You could share on the user-list to get more input.
I thought of doing that, but where the draft lives is in my personal Apache
space and I worried about some possibly retaining that as some new place to
access the Ref Guide. It happened before when we were transitioning from
Confluence to the current approach.
I also wasn't interested in re-opening the "Why don't we have search"
discussion and getting 10 strongly held opinions about that instead of what I'd
actually like to hear opinions about.
I thought about pushing a DRAFT for 8.6, but that would mean it's in
branch_8_x, but if it's not in master yet it's not in the branch either (unless
I make those branches locally).
> Ref Guide Redesign
> --
>
> Key: SOLR-14173
> URL: https://issues.apache.org/jira/browse/SOLR-14173
> Project: Solr
> Issue Type: Improvement
> Components: documentation
>Reporter: Cassandra Targett
>Assignee: Cassandra Targett
>Priority: Major
> Attachments: SOLR-14173.patch, blue-left-nav.png, gray-left-nav.png
>
>
> The current design of the Ref Guide was essentially copied from a
> Jekyll-based documentation theme
> (https://idratherbewriting.com/documentation-theme-jekyll/), which had a
> couple important benefits for that time:
> * It was well-documented and since I had little experience with Jekyll and
> its Liquid templates and since I was the one doing it, I wanted to make it as
> easy on myself as possible
> * It was designed for documentation specifically so took care of all the
> things like inter-page navigation, etc.
> * It helped us get from Confluence to our current system quickly
> It had some drawbacks, though:
> * It wasted a lot of space on the page
> * The theme was built for Markdown files, so did not take advantage of the
> features of the {{jekyll-asciidoc}} plugin we use (the in-page TOC being one
> big example - the plugin could create it at build time, but the theme
> included JS to do it as the page loads, so we use the JS)
> * It had a lot of JS and overlapping CSS files. While it used Bootstrap it
> used a customized CSS on top of it for theming that made modifications
> complex (it was hard to figure out how exactly a change would behave)
> * With all the stuff I'd changed in my bumbling way just to get things to
> work back then, I broke a lot of the stuff Bootstrap is supposed to give us
> in terms of responsiveness and making the Guide usable even on smaller screen
> sizes.
> After upgrading the Asciidoctor components in SOLR-12786 and stopping the PDF
> (SOLR-13782), I wanted to try to set us up for a more flexible system. We
> need it for things like Joel's work on the visual guide for streaming
> expressions (SOLR-13105), and in order to implement other ideas we might have
> on how to present information in the future.
> I view this issue as a phase 1 of an overall redesign that I've already
> started in a local branch. I'll explain in a comment the changes I've already
> made, and will use this issue to create and push a branch where we can
> discuss in more detail.
> Phase 1 here will be under-the-hood CSS/JS changes + overall page layout
> changes.
> Phase 2 (issue TBD) will be a wholesale re-organization of all the pages of
> the Guide.
> Phase 3 (issue TBD) will explore moving us from Jekyll to another static site
> generator that is better suited for our content format, file types, and build
> conventions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14441) Upgrade Tika to 1.24.1
mibo created SOLR-14441: --- Summary: Upgrade Tika to 1.24.1 Key: SOLR-14441 URL: https://issues.apache.org/jira/browse/SOLR-14441 Project: Solr Issue Type: Task Security Level: Public (Default Security Level. Issues are Public) Affects Versions: 8.5 Reporter: mibo Assignee: Erick Erickson Fix For: 8.6 Upgrade Apache Tika to new released 1.24 to handle [CVE-2020-1950|https://nvd.nist.gov/vuln/detail/CVE-2020-1950]. Created [PR #1383|https://github.com/apache/lucene-solr/pull/1383] but afterwards I found https://issues.apache.org/jira/browse/SOLR-14054 and it looks like an update is much more complicated. I someone support me I will update my contribution. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14441) Upgrade Tika to 1.24.1
[ https://issues.apache.org/jira/browse/SOLR-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mibo resolved SOLR-14441. - Resolution: Duplicate Sorry, wrongly created duplicate of: https://issues.apache.org/jira/browse/SOLR-14439 > Upgrade Tika to 1.24.1 > -- > > Key: SOLR-14441 > URL: https://issues.apache.org/jira/browse/SOLR-14441 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 8.5 >Reporter: mibo >Assignee: Erick Erickson >Priority: Minor > Fix For: 8.6 > > > Upgrade Apache Tika to new released 1.24 to handle > [CVE-2020-1950|https://nvd.nist.gov/vuln/detail/CVE-2020-1950]. > Created [PR #1383|https://github.com/apache/lucene-solr/pull/1383] but > afterwards I found https://issues.apache.org/jira/browse/SOLR-14054 and it > looks like an update is much more complicated. > I someone support me I will update my contribution. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-11005) inconsistency when maxShardsPerNode used along with policies
[
https://issues.apache.org/jira/browse/SOLR-11005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cassandra Targett updated SOLR-11005:
-
Fix Version/s: (was: 7.3)
(was: 8.0)
> inconsistency when maxShardsPerNode used along with policies
>
>
> Key: SOLR-11005
> URL: https://issues.apache.org/jira/browse/SOLR-11005
> Project: Solr
> Issue Type: Improvement
> Components: AutoScaling, SolrCloud
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
>
> The attribute maxShardsPerNode conflicts with the conditions in the new
> Policy framework
> for example , I can say maxShardsPerNode=5 and I can have a policy
> {code}
> { replica:"<3" , shard: "#ANY", node:"#ANY"}
> {code}
> So, it makes no sense to persist this attribute in collection state.json .
> Ideally, we would like to keep this as a part of the policy and policy only.
> h3. proposed new behavior
> if the new policy framework is being used {maxShardsPerNode} should result in
> creating a new collection specific policy with the correct condition. for
> example, if a collection "x" is created with the parameter
> {{maxShardsPerNode=2}} we will create a new policy in autoscaling.json
> {code}
> {
> "policies":{
> "x_COLL_POLICY" : [{replica:"<3", shard:"#ANY" , node:"ANY"}]
> }
> }
> {code}
> this policy will be referred to in the state.json. There will be no attribute
> called {{maxShardsPerNode}} persisted to the state.json.
> if there is already a policy being specified for the collection, solr should
> throw an error asking the user to edit the policy directly
> h3.the name is bad
> We must rename the attribute {{maxShardsPerNode}} to {{maxReplicasPerNode}}.
> This should be a backward compatible change. The old name will continue to
> work and the API would give a friendly warning if the old name is used
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14276) At various times when we restart Solr or following a hard reboot, Solr generates multiple replicas beyond the defined amount
[ https://issues.apache.org/jira/browse/SOLR-14276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gerlowski resolved SOLR-14276. Resolution: Invalid Hi Jason, As a community we use the JIRA bugtracker for known or suspected bugs (ideally with steps for reproduction). Since this is more of a question or request for help, your questions are better off directed at the solr-user mailing list (solr-u...@lucene.apache.org). That's also much more widely read, so posting there isn't just to satisfy our arbitrary rules - it's also where you're much more likely to get an answer to your question. See [here|https://lucene.apache.org/solr/community.html#mailing-lists-irc] for information about subscribing or posting to our mailing lists. > At various times when we restart Solr or following a hard reboot, Solr > generates multiple replicas beyond the defined amount > > > Key: SOLR-14276 > URL: https://issues.apache.org/jira/browse/SOLR-14276 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.2.1 >Reporter: Jason Randall >Priority: Major > > Our cluster is set up so that each daily log shard has 3 replicas. It should > create one replica for each of the sites in the cluster and then host one of > the replicas on each site. We have days in which we get only 1 or 2 replicas > instead of 3. We have days in which we get multiple replicas. The replicas > are often randomly dispersed amongst the nodes in the cluster. So we might > have 5 hosted on node 2 and 3 on node 3 but maybe none on node 1 for that > day. Can you provide possible scenarios as to why this may happen? And how to > resolve the issue? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9087) Should the BKD tree use a fixed maxPointsInLeafNode?
[ https://issues.apache.org/jira/browse/LUCENE-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ignacio Vera updated LUCENE-9087: - Attachment: Study of BKD tree performance with different values for max points per leaf.pdf Status: Open (was: Open) I have done an experiment about the effect of changing the default of maxPointsInLeafNode (attached the results in the issue). After this experience I would like to propose: * It makes sense to change the way we construct trees, so we always construct unbalanced trees as we are doing for the 1D case. This gives us more control abut the expected performance. * Lower the default number of max points in leaf node to 512. Even though in many cases lowering even further can give greater performance, it would not behave so well in edge cases. > Should the BKD tree use a fixed maxPointsInLeafNode? > - > > Key: LUCENE-9087 > URL: https://issues.apache.org/jira/browse/LUCENE-9087 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Ignacio Vera >Priority: Major > Attachments: Study of BKD tree performance with different values for > max points per leaf.pdf > > > Currently the BKD tree uses a fixed maxPointsInLeafNode provided in the > constructor. For the current default codec the value is set to 1024. This is > a good compromise between memory usage and performance of the BKD tree. > Lowering this value can increase search performance but it has a penalty in > memory usage. Now that the BKD tree can be load off-heap, this can be less of > a concern. Note that lowering too much that value can hurt performance as > well as the tree becomes too deep and benefits are gone. > For data types that use the tree as an effective R-tree (ranges and shapes > datatypes) the benefits are larger as it can minimise the overlap between > leaf nodes. > Finally, creating too many leaf nodes can be dangerous at write time as > memory usage depends on the number of leaf nodes created. The writer creates > a long array of length = numberOfLeafNodes. > What I am wondering here is if we can improve this situation in order to > create the most efficient tree? My current ideas are: > > * We can adapt the points per leaf depending on that number so we create a > tree with the best depth and best points per leaf. Note that for the for 1D > case we have an upper estimation of the number of points that the tree will > be indexing. > * Add a mechanism so field types can easily define their best points per > leaf. In the case, field types like ranges or shapes can define its own value > to minimise overlap. > * Maybe the default is just too high now that we can load the tree off-heap. > > Any thoughts? > > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob commented on pull request #1463: SOLR-14440 Cert Auth plugin
madrob commented on pull request #1463: URL: https://github.com/apache/lucene-solr/pull/1463#issuecomment-620725921 > Would it be useful to be able to configure from where to pull the username that will be passed in to Solr I think so, but that is probably better with something like SOLR-10814 in place so that we can keep the full subject name and also define a "short name". > I'm not familiar with the Principal class being wrapped here so maybe it has what we need, but as I understand there is more than one way to add a userid to a cert. The principal class works about how you'd expect w.r.t. `getName()`, which is what RuleBasedAuthorizationPlugin uses, and will return something like `CN=Solr User,OU=Engineering,O=Example Inc.,C=US`. Added that to the docs to clarify. > Wrt Admin UI I suspect it to "just work" if you have the right cert. Good ideas, I'll look into this and see if we need to make additions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14442) bin/solr to attempt jstack before killing hung Solr instance
Christine Poerschke created SOLR-14442:
--
Summary: bin/solr to attempt jstack before killing hung Solr
instance
Key: SOLR-14442
URL: https://issues.apache.org/jira/browse/SOLR-14442
Project: Solr
Issue Type: Improvement
Reporter: Christine Poerschke
Assignee: Christine Poerschke
If a Solr instance did not respond to the 'stop' command in a timely manner
then the {{bin/solr}} script will attempt to forcefully kill it:
[https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/solr/bin/solr#L859]
Gathering of information (e.g. a jstack of the java process) before the kill
command may be helpful in determining why the instance did not stop as expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] iverase opened a new pull request #1464: LUCENE-9087: Build always trees with full leaves and lower the default value for maxPointsPerLeafNode
iverase opened a new pull request #1464: URL: https://github.com/apache/lucene-solr/pull/1464 This commit changes the logic used to build BKD trees to always construct trees with full leaves (except the last one). This change gives more control in the expected behaviour of the tree. In addition the special logic the we have for 1D trees constructs the same type of trees, therefore removing some discrepancy. In addition, this commit lowers the default for maxPointsPerLeafNode from 1024 to 512 and simplifies the logic for rotating tree leaves. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14442) bin/solr to attempt jstack before killing hung Solr instance
[
https://issues.apache.org/jira/browse/SOLR-14442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christine Poerschke updated SOLR-14442:
---
Attachment: SOLR-14442.patch
> bin/solr to attempt jstack before killing hung Solr instance
>
>
> Key: SOLR-14442
> URL: https://issues.apache.org/jira/browse/SOLR-14442
> Project: Solr
> Issue Type: Improvement
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: SOLR-14442.patch
>
>
> If a Solr instance did not respond to the 'stop' command in a timely manner
> then the {{bin/solr}} script will attempt to forcefully kill it:
> [https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/solr/bin/solr#L859]
> Gathering of information (e.g. a jstack of the java process) before the kill
> command may be helpful in determining why the instance did not stop as
> expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14442) bin/solr to attempt jstack before killing hung Solr instance
[
https://issues.apache.org/jira/browse/SOLR-14442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094680#comment-17094680
]
Christine Poerschke commented on SOLR-14442:
Attached work-in-progress patch to invite initial comments, if any. I haven't
tested the change as yet and also the Windows {{solr.cmd}} equivalent to
{{solr}} is still missing.
> bin/solr to attempt jstack before killing hung Solr instance
>
>
> Key: SOLR-14442
> URL: https://issues.apache.org/jira/browse/SOLR-14442
> Project: Solr
> Issue Type: Improvement
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: SOLR-14442.patch
>
>
> If a Solr instance did not respond to the 'stop' command in a timely manner
> then the {{bin/solr}} script will attempt to forcefully kill it:
> [https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/solr/bin/solr#L859]
> Gathering of information (e.g. a jstack of the java process) before the kill
> command may be helpful in determining why the instance did not stop as
> expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9087) Should the BKD tree use a fixed maxPointsInLeafNode?
[ https://issues.apache.org/jira/browse/LUCENE-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094692#comment-17094692 ] Adrien Grand commented on LUCENE-9087: -- +1 I like the idea of making the 1D and multi-dimensional cases more consistent, I suspect that it will also make it easier to reason about the compression that we can apply within a leaf, or make it possible to use more efficient decoding techniques that are more similar to what we do in postings. I also support the idea of moving to 512 points per leaf. Your analysis suggests it's a better trade-off. Given how we try to build balanced trees today, you might end up with between 513 and 1024 points per leaf , so it wouldn't be a risky change. > Should the BKD tree use a fixed maxPointsInLeafNode? > - > > Key: LUCENE-9087 > URL: https://issues.apache.org/jira/browse/LUCENE-9087 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Ignacio Vera >Priority: Major > Attachments: Study of BKD tree performance with different values for > max points per leaf.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently the BKD tree uses a fixed maxPointsInLeafNode provided in the > constructor. For the current default codec the value is set to 1024. This is > a good compromise between memory usage and performance of the BKD tree. > Lowering this value can increase search performance but it has a penalty in > memory usage. Now that the BKD tree can be load off-heap, this can be less of > a concern. Note that lowering too much that value can hurt performance as > well as the tree becomes too deep and benefits are gone. > For data types that use the tree as an effective R-tree (ranges and shapes > datatypes) the benefits are larger as it can minimise the overlap between > leaf nodes. > Finally, creating too many leaf nodes can be dangerous at write time as > memory usage depends on the number of leaf nodes created. The writer creates > a long array of length = numberOfLeafNodes. > What I am wondering here is if we can improve this situation in order to > create the most efficient tree? My current ideas are: > > * We can adapt the points per leaf depending on that number so we create a > tree with the best depth and best points per leaf. Note that for the for 1D > case we have an upper estimation of the number of points that the tree will > be indexing. > * Add a mechanism so field types can easily define their best points per > leaf. In the case, field types like ranges or shapes can define its own value > to minimise overlap. > * Maybe the default is just too high now that we can load the tree off-heap. > > Any thoughts? > > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] tflobbe commented on a change in pull request #1456: SOLR-13289: Support for BlockMax WAND
tflobbe commented on a change in pull request #1456:
URL: https://github.com/apache/lucene-solr/pull/1456#discussion_r416796194
##
File path: solr/core/src/java/org/apache/solr/search/MaxScoreCollector.java
##
@@ -39,7 +39,7 @@ public float getMaxScore() {
public ScoreMode scoreMode() {
// Should be TOP_SCORES but this would wrap the scorer unnecessarily since
// this collector is only used in a MultiCollector.
-return ScoreMode.COMPLETE;
+return ScoreMode.TOP_SCORES;
Review comment:
@jpountz, What did you mean with this comment? MultiCollector will set
the `scoreMode` to `COMPLETE` in the case of the main collector being something
other than `TOP_SCORES`, right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14173) Ref Guide Redesign
[
https://issues.apache.org/jira/browse/SOLR-14173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094756#comment-17094756
]
Cassandra Targett commented on SOLR-14173:
--
I thought of another sort of compromise here - I'll commit this to master, and
then share the solr-ref-guide-master Jenkins build link with the user community
for feedback before backporting to branch_8_x. If it's roundly despised, I can
revert it.
> Ref Guide Redesign
> --
>
> Key: SOLR-14173
> URL: https://issues.apache.org/jira/browse/SOLR-14173
> Project: Solr
> Issue Type: Improvement
> Components: documentation
>Reporter: Cassandra Targett
>Assignee: Cassandra Targett
>Priority: Major
> Attachments: SOLR-14173.patch, blue-left-nav.png, gray-left-nav.png
>
>
> The current design of the Ref Guide was essentially copied from a
> Jekyll-based documentation theme
> (https://idratherbewriting.com/documentation-theme-jekyll/), which had a
> couple important benefits for that time:
> * It was well-documented and since I had little experience with Jekyll and
> its Liquid templates and since I was the one doing it, I wanted to make it as
> easy on myself as possible
> * It was designed for documentation specifically so took care of all the
> things like inter-page navigation, etc.
> * It helped us get from Confluence to our current system quickly
> It had some drawbacks, though:
> * It wasted a lot of space on the page
> * The theme was built for Markdown files, so did not take advantage of the
> features of the {{jekyll-asciidoc}} plugin we use (the in-page TOC being one
> big example - the plugin could create it at build time, but the theme
> included JS to do it as the page loads, so we use the JS)
> * It had a lot of JS and overlapping CSS files. While it used Bootstrap it
> used a customized CSS on top of it for theming that made modifications
> complex (it was hard to figure out how exactly a change would behave)
> * With all the stuff I'd changed in my bumbling way just to get things to
> work back then, I broke a lot of the stuff Bootstrap is supposed to give us
> in terms of responsiveness and making the Guide usable even on smaller screen
> sizes.
> After upgrading the Asciidoctor components in SOLR-12786 and stopping the PDF
> (SOLR-13782), I wanted to try to set us up for a more flexible system. We
> need it for things like Joel's work on the visual guide for streaming
> expressions (SOLR-13105), and in order to implement other ideas we might have
> on how to present information in the future.
> I view this issue as a phase 1 of an overall redesign that I've already
> started in a local branch. I'll explain in a comment the changes I've already
> made, and will use this issue to create and push a branch where we can
> discuss in more detail.
> Phase 1 here will be under-the-hood CSS/JS changes + overall page layout
> changes.
> Phase 2 (issue TBD) will be a wholesale re-organization of all the pages of
> the Guide.
> Phase 3 (issue TBD) will explore moving us from Jekyll to another static site
> generator that is better suited for our content format, file types, and build
> conventions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Closed] (SOLR-13663) XML Query Parser to Support SpanPositionRangeQuery
[ https://issues.apache.org/jira/browse/SOLR-13663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Khludnev closed SOLR-13663. --- > XML Query Parser to Support SpanPositionRangeQuery > -- > > Key: SOLR-13663 > URL: https://issues.apache.org/jira/browse/SOLR-13663 > Project: Solr > Issue Type: Improvement > Components: query parsers >Affects Versions: 8.2 >Reporter: Alessandro Benedetti >Assignee: Mikhail Khludnev >Priority: Major > Fix For: 8.3 > > Attachments: SOLR-13663.patch, SOLR-13663.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Currently the XML Query Parser support a vast array of span queries, > including the SpanFirstQuery, but it doesn't support the generic > SpanPositionRangeQuery. > < SpanPositionRange start="2" end="3"> > prejudice > > > Scope of this issue is to introduce the related builder and allow the > possibility to build such queries. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14173) Ref Guide Redesign
[
https://issues.apache.org/jira/browse/SOLR-14173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cassandra Targett updated SOLR-14173:
-
Attachment: SOLR-14173.patch
> Ref Guide Redesign
> --
>
> Key: SOLR-14173
> URL: https://issues.apache.org/jira/browse/SOLR-14173
> Project: Solr
> Issue Type: Improvement
> Components: documentation
>Reporter: Cassandra Targett
>Assignee: Cassandra Targett
>Priority: Major
> Attachments: SOLR-14173.patch, SOLR-14173.patch, blue-left-nav.png,
> gray-left-nav.png
>
>
> The current design of the Ref Guide was essentially copied from a
> Jekyll-based documentation theme
> (https://idratherbewriting.com/documentation-theme-jekyll/), which had a
> couple important benefits for that time:
> * It was well-documented and since I had little experience with Jekyll and
> its Liquid templates and since I was the one doing it, I wanted to make it as
> easy on myself as possible
> * It was designed for documentation specifically so took care of all the
> things like inter-page navigation, etc.
> * It helped us get from Confluence to our current system quickly
> It had some drawbacks, though:
> * It wasted a lot of space on the page
> * The theme was built for Markdown files, so did not take advantage of the
> features of the {{jekyll-asciidoc}} plugin we use (the in-page TOC being one
> big example - the plugin could create it at build time, but the theme
> included JS to do it as the page loads, so we use the JS)
> * It had a lot of JS and overlapping CSS files. While it used Bootstrap it
> used a customized CSS on top of it for theming that made modifications
> complex (it was hard to figure out how exactly a change would behave)
> * With all the stuff I'd changed in my bumbling way just to get things to
> work back then, I broke a lot of the stuff Bootstrap is supposed to give us
> in terms of responsiveness and making the Guide usable even on smaller screen
> sizes.
> After upgrading the Asciidoctor components in SOLR-12786 and stopping the PDF
> (SOLR-13782), I wanted to try to set us up for a more flexible system. We
> need it for things like Joel's work on the visual guide for streaming
> expressions (SOLR-13105), and in order to implement other ideas we might have
> on how to present information in the future.
> I view this issue as a phase 1 of an overall redesign that I've already
> started in a local branch. I'll explain in a comment the changes I've already
> made, and will use this issue to create and push a branch where we can
> discuss in more detail.
> Phase 1 here will be under-the-hood CSS/JS changes + overall page layout
> changes.
> Phase 2 (issue TBD) will be a wholesale re-organization of all the pages of
> the Guide.
> Phase 3 (issue TBD) will explore moving us from Jekyll to another static site
> generator that is better suited for our content format, file types, and build
> conventions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14173) Ref Guide Redesign
[
https://issues.apache.org/jira/browse/SOLR-14173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094759#comment-17094759
]
Cassandra Targett commented on SOLR-14173:
--
Also attached a new patch with the changes I'm committing today.
> Ref Guide Redesign
> --
>
> Key: SOLR-14173
> URL: https://issues.apache.org/jira/browse/SOLR-14173
> Project: Solr
> Issue Type: Improvement
> Components: documentation
>Reporter: Cassandra Targett
>Assignee: Cassandra Targett
>Priority: Major
> Attachments: SOLR-14173.patch, SOLR-14173.patch, blue-left-nav.png,
> gray-left-nav.png
>
>
> The current design of the Ref Guide was essentially copied from a
> Jekyll-based documentation theme
> (https://idratherbewriting.com/documentation-theme-jekyll/), which had a
> couple important benefits for that time:
> * It was well-documented and since I had little experience with Jekyll and
> its Liquid templates and since I was the one doing it, I wanted to make it as
> easy on myself as possible
> * It was designed for documentation specifically so took care of all the
> things like inter-page navigation, etc.
> * It helped us get from Confluence to our current system quickly
> It had some drawbacks, though:
> * It wasted a lot of space on the page
> * The theme was built for Markdown files, so did not take advantage of the
> features of the {{jekyll-asciidoc}} plugin we use (the in-page TOC being one
> big example - the plugin could create it at build time, but the theme
> included JS to do it as the page loads, so we use the JS)
> * It had a lot of JS and overlapping CSS files. While it used Bootstrap it
> used a customized CSS on top of it for theming that made modifications
> complex (it was hard to figure out how exactly a change would behave)
> * With all the stuff I'd changed in my bumbling way just to get things to
> work back then, I broke a lot of the stuff Bootstrap is supposed to give us
> in terms of responsiveness and making the Guide usable even on smaller screen
> sizes.
> After upgrading the Asciidoctor components in SOLR-12786 and stopping the PDF
> (SOLR-13782), I wanted to try to set us up for a more flexible system. We
> need it for things like Joel's work on the visual guide for streaming
> expressions (SOLR-13105), and in order to implement other ideas we might have
> on how to present information in the future.
> I view this issue as a phase 1 of an overall redesign that I've already
> started in a local branch. I'll explain in a comment the changes I've already
> made, and will use this issue to create and push a branch where we can
> discuss in more detail.
> Phase 1 here will be under-the-hood CSS/JS changes + overall page layout
> changes.
> Phase 2 (issue TBD) will be a wholesale re-organization of all the pages of
> the Guide.
> Phase 3 (issue TBD) will explore moving us from Jekyll to another static site
> generator that is better suited for our content format, file types, and build
> conventions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mkhludnev commented on pull request #812: [SOLR-13663] Span Positional Range Support in XML Query Parser + test
mkhludnev commented on pull request #812: URL: https://github.com/apache/lucene-solr/pull/812#issuecomment-620777554 @alessandrobenedetti would you mind to close this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] alessandrobenedetti commented on pull request #812: [SOLR-13663] Span Positional Range Support in XML Query Parser + test
alessandrobenedetti commented on pull request #812: URL: https://github.com/apache/lucene-solr/pull/812#issuecomment-620779177 Done! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14412) NPE in MetricsHistoryHandler when running single node in cloud mode with SSL
[
https://issues.apache.org/jira/browse/SOLR-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094764#comment-17094764
]
ASF subversion and git services commented on SOLR-14412:
Commit 0fc5179c6a7247b2a6df93459cd29aec4d60658d in lucene-solr's branch
refs/heads/master from Mike Drob
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=0fc5179 ]
SOLR-14412 only set ssl props when ssl enabled
> NPE in MetricsHistoryHandler when running single node in cloud mode with SSL
>
>
> Key: SOLR-14412
> URL: https://issues.apache.org/jira/browse/SOLR-14412
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: metrics
>Affects Versions: master (9.0)
> Environment: 9.0.0-SNAPSHOT, SSL enabled (self-signed certificate),
> cloud mode.
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
> Fix For: master (9.0)
>
>
> {noformat}
> 2020-04-17 15:35:19.335 INFO (MetricsHistoryHandler-22-thread-1) [ ]
> o.a.s.c.s.i.SolrClientNodeStateProvider Error on getting remote info, trying
> again: IOException occurred when talking to server at:
> http://localhost:8983/solr
> 2020-04-17 15:35:19.842 INFO (MetricsHistoryHandler-22-thread-1) [ ]
> o.a.s.c.s.i.SolrClientNodeStateProvider Error on getting remote info, trying
> again: IOException occurred when talking to server at:
> http://localhost:8983/solr
> 2020-04-17 15:35:20.346 INFO (MetricsHistoryHandler-22-thread-1) [ ]
> o.a.s.c.s.i.SolrClientNodeStateProvider Error on getting remote info, trying
> again: IOException occurred when talking to server at:
> http://localhost:8983/solr
> 2020-04-17 15:35:20.851 WARN (MetricsHistoryHandler-22-thread-1) [ ]
> o.a.s.c.s.i.SolrClientNodeStateProvider could not get tags from node
> localhost:8983_solr => java.lang.NullPointerException
> at
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.lambda$fetchReplicaMetrics$7(SolrClientNodeStateProvider.java:226)
> java.lang.NullPointerException: null
> at
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.lambda$fetchReplicaMetrics$7(SolrClientNodeStateProvider.java:226)
> ~[solr-solrj-9.0.0-SNAPSHOT.jar:9.0.0-SNAPSHOT
> 74ecc13816fb6aae6e512e2e9d815459e235a120 - mdrob - 2020-04-17 10:28:44]
> at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
> at
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetrics(SolrClientNodeStateProvider.java:225)
> ~[solr-solrj-9.0.0-SNAPSHOT.jar:9.0.0-SNAPSHOT
> 74ecc13816fb6aae6e512e2e9d815459e235a120 - mdrob - 2020-04-17 10:28:44]
> at
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$AutoScalingSnitch.getRemoteInfo(SolrClientNodeStateProvider.java:271)
> ~[solr-solrj-9.0.0-SNAPSHOT.jar:9.0.0-SNAPSHOT
> 74ecc13816fb6aae6e512e2e9d815459e235a120 - mdrob - 2020-04-17 10:28:44]
> at
> org.apache.solr.common.cloud.rule.ImplicitSnitch.getTags(ImplicitSnitch.java:75)
> ~[solr-solrj-9.0.0-SNAPSHOT.jar:9.0.0-SNAPSHOT
> 74ecc13816fb6aae6e512e2e9d815459e235a120 - mdrob - 2020-04-17 10:28:44]
> at
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchTagValues(SolrClientNodeStateProvider.java:139)
> ~[solr-solrj-9.0.0-SNAPSHOT.jar:9.0.0-SNAPSHOT
> 74ecc13816fb6aae6e512e2e9d815459e235a120 - mdrob - 2020-04-17 10:28:44]
> at
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getNodeValues(SolrClientNodeStateProvider.java:128)
> ~[solr-solrj-9.0.0-SNAPSHOT.jar:9.0.0-SNAPSHOT
> 74ecc13816fb6aae6e512e2e9d815459e235a120 - mdrob - 2020-04-17 10:28:44]
> at
> org.apache.solr.handler.admin.MetricsHistoryHandler.collectGlobalMetrics(MetricsHistoryHandler.java:506)
> ~[solr-core-9.0.0-SNAPSHOT.jar:9.0.0-SNAPSHOT
> 74ecc13816fb6aae6e512e2e9d815459e235a120 - mdrob - 2020-04-17 10:28:44]
> at
> org.apache.solr.handler.admin.MetricsHistoryHandler.collectMetrics(MetricsHistoryHandler.java:378)
> ~[solr-core-9.0.0-SNAPSHOT.jar:9.0.0-SNAPSHOT
> 74ecc13816fb6aae6e512e2e9d815459e235a120 - mdrob - 2020-04-17 10:28:44]
> at
> org.apache.solr.handler.admin.MetricsHistoryHandler.lambda$new$0(MetricsHistoryHandler.java:235)
> ~[solr-core-9.0.0-SNAPSHOT.jar:9.0.0-SNAPSHOT
> 74ecc13816fb6aae6e512e2e9d815459e235a120 - mdrob - 2020-04-17 10:28:44]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> ~[?:?]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> ~[?:?]
> at
> java.uti
[jira] [Created] (SOLR-14443) Make SolrLogPostTool resilient to unexpected requests
Jason Gerlowski created SOLR-14443: -- Summary: Make SolrLogPostTool resilient to unexpected requests Key: SOLR-14443 URL: https://issues.apache.org/jira/browse/SOLR-14443 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: scripts and tools Affects Versions: master (9.0) Reporter: Jason Gerlowski Assignee: Jason Gerlowski When SolrLogPostTool parses log messages corresponding to incoming requests, it sets various predefined fields based on the parameters on the request. e.g. it sets a rows_i field, a wt_s field, and so on. This logic works for most requests, but if the log-parser encounters requests with multiple of these params (e.g. rows), it will blithely add them to the SolrInputDocument, and error out when Solr rejects the eventual update request because it is attempting to put multiple values into a single-valued field. We can do two things to fix this. # Make SolrLogPostTool's "posting" code resilient to individual update failures. It doesn't make any sense to crash the entire posting routine just because one batch (or one log message) was malformed. # Tweak the field parsing logic to be more resilient to the specific "redundant query params" case I encountered specifically here. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14173) Ref Guide Redesign
[
https://issues.apache.org/jira/browse/SOLR-14173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094802#comment-17094802
]
ASF subversion and git services commented on SOLR-14173:
Commit f4eb586ef6470772115e2c70276e379c99aec583 in lucene-solr's branch
refs/heads/master from Cassandra Targett
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=f4eb586 ]
SOLR-14173: Ref Guide Redesign: upgrade bootstrap; change layout; consolidate
CSS. See issue for list of changes.
> Ref Guide Redesign
> --
>
> Key: SOLR-14173
> URL: https://issues.apache.org/jira/browse/SOLR-14173
> Project: Solr
> Issue Type: Improvement
> Components: documentation
>Reporter: Cassandra Targett
>Assignee: Cassandra Targett
>Priority: Major
> Attachments: SOLR-14173.patch, SOLR-14173.patch, blue-left-nav.png,
> gray-left-nav.png
>
>
> The current design of the Ref Guide was essentially copied from a
> Jekyll-based documentation theme
> (https://idratherbewriting.com/documentation-theme-jekyll/), which had a
> couple important benefits for that time:
> * It was well-documented and since I had little experience with Jekyll and
> its Liquid templates and since I was the one doing it, I wanted to make it as
> easy on myself as possible
> * It was designed for documentation specifically so took care of all the
> things like inter-page navigation, etc.
> * It helped us get from Confluence to our current system quickly
> It had some drawbacks, though:
> * It wasted a lot of space on the page
> * The theme was built for Markdown files, so did not take advantage of the
> features of the {{jekyll-asciidoc}} plugin we use (the in-page TOC being one
> big example - the plugin could create it at build time, but the theme
> included JS to do it as the page loads, so we use the JS)
> * It had a lot of JS and overlapping CSS files. While it used Bootstrap it
> used a customized CSS on top of it for theming that made modifications
> complex (it was hard to figure out how exactly a change would behave)
> * With all the stuff I'd changed in my bumbling way just to get things to
> work back then, I broke a lot of the stuff Bootstrap is supposed to give us
> in terms of responsiveness and making the Guide usable even on smaller screen
> sizes.
> After upgrading the Asciidoctor components in SOLR-12786 and stopping the PDF
> (SOLR-13782), I wanted to try to set us up for a more flexible system. We
> need it for things like Joel's work on the visual guide for streaming
> expressions (SOLR-13105), and in order to implement other ideas we might have
> on how to present information in the future.
> I view this issue as a phase 1 of an overall redesign that I've already
> started in a local branch. I'll explain in a comment the changes I've already
> made, and will use this issue to create and push a branch where we can
> discuss in more detail.
> Phase 1 here will be under-the-hood CSS/JS changes + overall page layout
> changes.
> Phase 2 (issue TBD) will be a wholesale re-organization of all the pages of
> the Guide.
> Phase 3 (issue TBD) will explore moving us from Jekyll to another static site
> generator that is better suited for our content format, file types, and build
> conventions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9089) FST.Builder with fluent-style constructor
[ https://issues.apache.org/jira/browse/LUCENE-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094819#comment-17094819 ] Dawid Weiss commented on LUCENE-9089: - Go ahead, Tomoko. Thank you. > FST.Builder with fluent-style constructor > - > > Key: LUCENE-9089 > URL: https://issues.apache.org/jira/browse/LUCENE-9089 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Bruno Roustant >Assignee: Bruno Roustant >Priority: Minor > Fix For: master (9.0) > > Attachments: fix-fst-package-summary.patch > > Time Spent: 2.5h > Remaining Estimate: 0h > > A first step in a try to make the FST code easier to read and evolve. This > step is just about the FST Builder constructor. > By making it fluent, the many calls to it are simplified and it becomes easy > to spot the intent and special param tuning. > No functional change. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14444) Ref Guide Redesign Phase 2: Page Organization Overhaul
Cassandra Targett created SOLR-1: Summary: Ref Guide Redesign Phase 2: Page Organization Overhaul Key: SOLR-1 URL: https://issues.apache.org/jira/browse/SOLR-1 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: documentation Reporter: Cassandra Targett Assignee: Cassandra Targett SOLR-14173 changed the look & feel of the Ref Guide, and left one glaring problem unresolved: our top-level categories are far too many for that sort of design. This issue will track a full reconsideration of the Ref Guide organization. I have a couple of thoughts and will update this issue with those when I get started on it; for now this is a placeholder issue for future work. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14173) Ref Guide Redesign Phase 1: Page Design
[
https://issues.apache.org/jira/browse/SOLR-14173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cassandra Targett updated SOLR-14173:
-
Summary: Ref Guide Redesign Phase 1: Page Design (was: Ref Guide Redesign)
> Ref Guide Redesign Phase 1: Page Design
> ---
>
> Key: SOLR-14173
> URL: https://issues.apache.org/jira/browse/SOLR-14173
> Project: Solr
> Issue Type: Improvement
> Components: documentation
>Reporter: Cassandra Targett
>Assignee: Cassandra Targett
>Priority: Major
> Attachments: SOLR-14173.patch, SOLR-14173.patch, blue-left-nav.png,
> gray-left-nav.png
>
>
> The current design of the Ref Guide was essentially copied from a
> Jekyll-based documentation theme
> (https://idratherbewriting.com/documentation-theme-jekyll/), which had a
> couple important benefits for that time:
> * It was well-documented and since I had little experience with Jekyll and
> its Liquid templates and since I was the one doing it, I wanted to make it as
> easy on myself as possible
> * It was designed for documentation specifically so took care of all the
> things like inter-page navigation, etc.
> * It helped us get from Confluence to our current system quickly
> It had some drawbacks, though:
> * It wasted a lot of space on the page
> * The theme was built for Markdown files, so did not take advantage of the
> features of the {{jekyll-asciidoc}} plugin we use (the in-page TOC being one
> big example - the plugin could create it at build time, but the theme
> included JS to do it as the page loads, so we use the JS)
> * It had a lot of JS and overlapping CSS files. While it used Bootstrap it
> used a customized CSS on top of it for theming that made modifications
> complex (it was hard to figure out how exactly a change would behave)
> * With all the stuff I'd changed in my bumbling way just to get things to
> work back then, I broke a lot of the stuff Bootstrap is supposed to give us
> in terms of responsiveness and making the Guide usable even on smaller screen
> sizes.
> After upgrading the Asciidoctor components in SOLR-12786 and stopping the PDF
> (SOLR-13782), I wanted to try to set us up for a more flexible system. We
> need it for things like Joel's work on the visual guide for streaming
> expressions (SOLR-13105), and in order to implement other ideas we might have
> on how to present information in the future.
> I view this issue as a phase 1 of an overall redesign that I've already
> started in a local branch. I'll explain in a comment the changes I've already
> made, and will use this issue to create and push a branch where we can
> discuss in more detail.
> Phase 1 here will be under-the-hood CSS/JS changes + overall page layout
> changes.
> Phase 2 (issue TBD) will be a wholesale re-organization of all the pages of
> the Guide.
> Phase 3 (issue TBD) will explore moving us from Jekyll to another static site
> generator that is better suited for our content format, file types, and build
> conventions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14173) Ref Guide Redesign Phase 1: Page Design
[
https://issues.apache.org/jira/browse/SOLR-14173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cassandra Targett updated SOLR-14173:
-
Description:
The current design of the Ref Guide was essentially copied from a Jekyll-based
documentation theme
(https://idratherbewriting.com/documentation-theme-jekyll/), which had a couple
important benefits for that time:
* It was well-documented and since I had little experience with Jekyll and its
Liquid templates and since I was the one doing it, I wanted to make it as easy
on myself as possible
* It was designed for documentation specifically so took care of all the things
like inter-page navigation, etc.
* It helped us get from Confluence to our current system quickly
It had some drawbacks, though:
* It wasted a lot of space on the page
* The theme was built for Markdown files, so did not take advantage of the
features of the {{jekyll-asciidoc}} plugin we use (the in-page TOC being one
big example - the plugin could create it at build time, but the theme included
JS to do it as the page loads, so we use the JS)
* It had a lot of JS and overlapping CSS files. While it used Bootstrap it used
a customized CSS on top of it for theming that made modifications complex (it
was hard to figure out how exactly a change would behave)
* With all the stuff I'd changed in my bumbling way just to get things to work
back then, I broke a lot of the stuff Bootstrap is supposed to give us in terms
of responsiveness and making the Guide usable even on smaller screen sizes.
After upgrading the Asciidoctor components in SOLR-12786 and stopping the PDF
(SOLR-13782), I wanted to try to set us up for a more flexible system. We need
it for things like Joel's work on the visual guide for streaming expressions
(SOLR-13105), and in order to implement other ideas we might have on how to
present information in the future.
I view this issue as a phase 1 of an overall redesign that I've already started
in a local branch. I'll explain in a comment the changes I've already made, and
will use this issue to create and push a branch where we can discuss in more
detail.
Phase 1 here will be under-the-hood CSS/JS changes + overall page layout
changes.
Phase 2 (SOLR-1) will be a wholesale re-organization of all the pages of
the Guide.
Phase 3 (issue TBD) will explore moving us from Jekyll to another static site
generator that is better suited for our content format, file types, and build
conventions.
was:
The current design of the Ref Guide was essentially copied from a Jekyll-based
documentation theme
(https://idratherbewriting.com/documentation-theme-jekyll/), which had a couple
important benefits for that time:
* It was well-documented and since I had little experience with Jekyll and its
Liquid templates and since I was the one doing it, I wanted to make it as easy
on myself as possible
* It was designed for documentation specifically so took care of all the things
like inter-page navigation, etc.
* It helped us get from Confluence to our current system quickly
It had some drawbacks, though:
* It wasted a lot of space on the page
* The theme was built for Markdown files, so did not take advantage of the
features of the {{jekyll-asciidoc}} plugin we use (the in-page TOC being one
big example - the plugin could create it at build time, but the theme included
JS to do it as the page loads, so we use the JS)
* It had a lot of JS and overlapping CSS files. While it used Bootstrap it used
a customized CSS on top of it for theming that made modifications complex (it
was hard to figure out how exactly a change would behave)
* With all the stuff I'd changed in my bumbling way just to get things to work
back then, I broke a lot of the stuff Bootstrap is supposed to give us in terms
of responsiveness and making the Guide usable even on smaller screen sizes.
After upgrading the Asciidoctor components in SOLR-12786 and stopping the PDF
(SOLR-13782), I wanted to try to set us up for a more flexible system. We need
it for things like Joel's work on the visual guide for streaming expressions
(SOLR-13105), and in order to implement other ideas we might have on how to
present information in the future.
I view this issue as a phase 1 of an overall redesign that I've already started
in a local branch. I'll explain in a comment the changes I've already made, and
will use this issue to create and push a branch where we can discuss in more
detail.
Phase 1 here will be under-the-hood CSS/JS changes + overall page layout
changes.
Phase 2 (issue TBD) will be a wholesale re-organization of all the pages of the
Guide.
Phase 3 (issue TBD) will explore moving us from Jekyll to another static site
generator that is better suited for our content format, file types, and build
conventions.
> Ref Guide Redesign Phase 1: Page Design
> ---
>
>
[jira] [Created] (LUCENE-9349) Avoid parsing all terms in TermInSetQuery.visit()
Alan Woodward created LUCENE-9349: - Summary: Avoid parsing all terms in TermInSetQuery.visit() Key: LUCENE-9349 URL: https://issues.apache.org/jira/browse/LUCENE-9349 Project: Lucene - Core Issue Type: Improvement Reporter: Alan Woodward Assignee: Alan Woodward TermInSetQuery currently iterates through all its prefix-encoded terms in order to build an array to pass back to its visitor when visit() is called. This seems like a waste, particularly when the visitor is not actually consuming the terms (for example, when doing a clause-count check before executing a search). Instead TermInSetQuery should use consumeTermsMatching(), and we should change the signature of this method so that it takes a BytesRunAutomaton supplier to allow for lazy instantiation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] romseygeek opened a new pull request #1465: LUCENE-9349: TermInSetQuery should use consumeMatchingTerms in visit()
romseygeek opened a new pull request #1465: URL: https://github.com/apache/lucene-solr/pull/1465 TermInSetQuery currently iterates through all its prefix-encoded terms in order to build an array to pass back to its visitor when visit() is called. This seems like a waste, particularly when the visitor is not actually consuming the terms (for example, when doing a clause-count check before executing a search). This commit changes TermInSetQuery to use consumeTermsMatching(), and also changes the signature of this method so that it takes a BytesRunAutomaton supplier to allow for lazy instantiation. In addition, IndexSearcher's clause count check wasn't counting leaves that called consumeTermsMatching(). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] romseygeek commented on pull request #1465: LUCENE-9349: TermInSetQuery should use consumeMatchingTerms in visit()
romseygeek commented on pull request #1465: URL: https://github.com/apache/lucene-solr/pull/1465#issuecomment-620825699 This will also make FuzzyQuery's visit() method nicer if we switch back to not holding its automaton directly on the query. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8811) Add maximum clause count check to IndexSearcher rather than BooleanQuery
[ https://issues.apache.org/jira/browse/LUCENE-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094845#comment-17094845 ] Alan Woodward commented on LUCENE-8811: --- I opened LUCENE-9349 to address TermInSetQuery's visit() implementation. > Add maximum clause count check to IndexSearcher rather than BooleanQuery > > > Key: LUCENE-8811 > URL: https://issues.apache.org/jira/browse/LUCENE-8811 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Assignee: Alan Woodward >Priority: Minor > Fix For: master (9.0) > > Attachments: LUCENE-8811.patch, LUCENE-8811.patch, LUCENE-8811.patch, > LUCENE-8811.patch, LUCENE-8811.patch, LUCENE-8811.patch > > > Currently we only check whether boolean queries have too many clauses. > However there are other ways that queries may have too many clauses, for > instance if you have boolean queries that have themselves inner boolean > queries. > Could we use the new Query visitor API to move this check from BooleanQuery > to IndexSearcher in order to make this check more consistent across queries? > See for instance LUCENE-8810 where a rewrite rule caused the maximum clause > count to be hit even though the total number of leaf queries remained the > same. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14428) FuzzyQuery has severe memory usage in 8.5
[
https://issues.apache.org/jira/browse/SOLR-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094850#comment-17094850
]
Mike Drob commented on SOLR-14428:
--
bq. I don't think we need to give queries the ability to produce cache keys.
Queries are abstractions for an information need, which is typically something
that users would type in a search box. If users can express an information need
in a few characters, then there is no reason why the Query representation would
take much more memory
I've been thinking about this and I think this is the most intuitive path.
bq. I think we can try building the automata in `getTermsEnum()`, and adding
some laziness in `visit()` to prevent it being built unnecessarily. I'll open a
PR.
My PR mostly does this already, in addition to the cache key piece. If you take
a look, it might make for a reasonable starting point.
> FuzzyQuery has severe memory usage in 8.5
> -
>
> Key: SOLR-14428
> URL: https://issues.apache.org/jira/browse/SOLR-14428
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.5, 8.5.1
>Reporter: Colvin Cowie
>Assignee: Andrzej Bialecki
>Priority: Major
> Attachments: FuzzyHammer.java, SOLR-14428-WeakReferences.patch,
> image-2020-04-23-09-18-06-070.png, image-2020-04-24-20-09-31-179.png,
> screenshot-2.png, screenshot-3.png, screenshot-4.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I sent this to the mailing list
> I'm moving from 8.3.1 to 8.5.1, and started getting Out Of Memory Errors
> while running our normal tests. After profiling it was clear that the
> majority of the heap was allocated through FuzzyQuery.
> LUCENE-9068 moved construction of the automata from the FuzzyTermsEnum to the
> FuzzyQuery's constructor.
> I created a little test ( [^FuzzyHammer.java] ) that fires off fuzzy queries
> from random UUID strings for 5 minutes
> {code}
> FIELD_NAME + ":" + UUID.randomUUID().toString().replace("-", "") + "~2"
> {code}
> When running against a vanilla Solr 8.31 and 8.4.1 there is no problem, while
> the memory usage has increased drastically on 8.5.0 and 8.5.1.
> Comparison of heap usage while running the attached test against Solr 8.3.1
> and 8.5.1 with a single (empty) shard and 4GB heap:
> !image-2020-04-23-09-18-06-070.png!
> And with 4 shards on 8.4.1 and 8.5.0:
> !screenshot-2.png!
> I'm guessing that the memory might be being leaked if the FuzzyQuery objects
> are referenced from the cache, while the FuzzyTermsEnum would not have been.
> Query Result Cache on 8.5.1:
> !screenshot-3.png!
> ~316mb in the cache
> QRC on 8.3.1
> !screenshot-4.png!
> <1mb
> With an empty cache, running this query
> _field_s:e41848af85d24ac197c71db6888e17bc~2_ results in the following memory
> allocation
> {noformat}
> 8.3.1: CACHE.searcher.queryResultCache.ramBytesUsed: 1520
> 8.5.1: CACHE.searcher.queryResultCache.ramBytesUsed:648855
> {noformat}
> ~1 gives 98253 and ~0 gives 6339 on 8.5.1. 8.3.1 is constant at 1520
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9087) Should the BKD tree use a fixed maxPointsInLeafNode?
[ https://issues.apache.org/jira/browse/LUCENE-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated LUCENE-9087: - Description: Currently the BKD tree uses a fixed maxPointsInLeafNode provided in the constructor. For the current default codec the value is set to 1024. This is a good compromise between memory usage and performance of the BKD tree. Lowering this value can increase search performance but it has a penalty in memory usage. Now that the BKD tree can be load off-heap, this can be less of a concern. Note that lowering too much that value can hurt performance as well as the tree becomes too deep and benefits are gone. For data types that use the tree as an effective R-tree (ranges and shapes datatypes) the benefits are larger as it can minimise the overlap between leaf nodes. Finally, creating too many leaf nodes can be dangerous at write time as memory usage depends on the number of leaf nodes created. The writer creates a long array of length = numberOfLeafNodes. What I am wondering here is if we can improve this situation in order to create the most efficient tree? My current ideas are: * We can adapt the points per leaf depending on that number so we create a tree with the best depth and best points per leaf. Note that for the for 1D case we have an upper estimation of the number of points that the tree will be indexing. * Add a mechanism so field types can easily define their best points per leaf. In the case, field types like ranges or shapes can define its own value to minimise overlap. * Maybe the default is just too high now that we can load the tree off-heap. Any thoughts? was: Currently the BKD tree uses a fixed maxPointsInLeafNode provided in the constructor. For the current default codec the value is set to 1024. This is a good compromise between memory usage and performance of the BKD tree. Lowering this value can increase search performance but it has a penalty in memory usage. Now that the BKD tree can be load off-heap, this can be less of a concern. Note that lowering too much that value can hurt performance as well as the tree becomes too deep and benefits are gone. For data types that use the tree as an effective R-tree (ranges and shapes datatypes) the benefits are larger as it can minimise the overlap between leaf nodes. Finally, creating too many leaf nodes can be dangerous at write time as memory usage depends on the number of leaf nodes created. The writer creates a long array of length = numberOfLeafNodes. What I am wondering here is if we can improve this situation in order to create the most efficient tree? My current ideas are: * We can adapt the points per leaf depending on that number so we create a tree with the best depth and best points per leaf. Note that for the for 1D case we have an upper estimation of the number of points that the tree will be indexing. * Add a mechanism so field types can easily define their best points per leaf. In the case, field types like ranges or shapes can define its own value to minimise overlap. * Maybe the default is just too high now that we can load the tree off-heap. Any thoughts? > Should the BKD tree use a fixed maxPointsInLeafNode? > - > > Key: LUCENE-9087 > URL: https://issues.apache.org/jira/browse/LUCENE-9087 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Ignacio Vera >Priority: Major > Attachments: Study of BKD tree performance with different values for > max points per leaf.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently the BKD tree uses a fixed maxPointsInLeafNode provided in the > constructor. For the current default codec the value is set to 1024. This is > a good compromise between memory usage and performance of the BKD tree. > Lowering this value can increase search performance but it has a penalty in > memory usage. Now that the BKD tree can be load off-heap, this can be less of > a concern. Note that lowering too much that value can hurt performance as > well as the tree becomes too deep and benefits are gone. > For data types that use the tree as an effective R-tree (ranges and shapes > datatypes) the benefits are larger as it can minimise the overlap between > leaf nodes. > Finally, creating too many leaf nodes can be dangerous at write time as > memory usage depends on the number of leaf nodes created. The writer creates > a long array of length = numberOfLeafNodes. > What I am wondering here is if we can improve this situation in order to > create the most efficient tree? My current ideas are: > > * We can adapt the points per leaf depending on that number so we create a > tree with the best depth and best points per leaf. Note that for the for 1D > cas
[GitHub] [lucene-solr] mkhludnev commented on pull request #1465: LUCENE-9349: TermInSetQuery should use consumeMatchingTerms in visit()
mkhludnev commented on pull request #1465: URL: https://github.com/apache/lucene-solr/pull/1465#issuecomment-620839186 ok. +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dragonsinth commented on a change in pull request #1446: SOLR-14425: Aliases: ZK sync() needs to be synchronous
dragonsinth commented on a change in pull request #1446:
URL: https://github.com/apache/lucene-solr/pull/1446#discussion_r416945913
##
File path: solr/solrj/src/java/org/apache/solr/common/cloud/SolrZkClient.java
##
@@ -843,6 +847,31 @@ public void downloadFromZK(String zkPath, Path dir) throws
IOException {
ZkMaintenanceUtils.downloadFromZK(this, zkPath, dir);
}
+ /**
+ * Ensures we can subsequently read the most up-to-date state of the
provided {@code path} and all data below
+ * it, after this method completes.
+ * This should not be called a lot; there may be some delay. Consider
alternative approaches It's often better to try to devise a way to
+ * "watch" for state changed instead of calling this.
+ */
+ public void sync(String path) {
+// zookeeper.sync is asynchronous; we need to wait till it's done
+CompletableFuture future = new CompletableFuture<>();
+keeper.sync(path, (rc, path1, ctx) -> future.complete(rc), null);
+try {
+ Integer status = future.get(zkClientTimeout, TimeUnit.MILLISECONDS);
Review comment:
Dumb question from something who hasn't dug into ZK client code in Java
in ages, but if you're waiting on the answer anyway, why use a future and then
wait it instead of just calling a synchronous method to begin with? Or is more
like, "I'm waiting on potentially lots and lots of answers, I just need it to
be up to date with any previous operations"?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #1446: SOLR-14425: Aliases: ZK sync() needs to be synchronous
dsmiley commented on a change in pull request #1446:
URL: https://github.com/apache/lucene-solr/pull/1446#discussion_r416947064
##
File path: solr/solrj/src/java/org/apache/solr/common/cloud/SolrZkClient.java
##
@@ -843,6 +847,31 @@ public void downloadFromZK(String zkPath, Path dir) throws
IOException {
ZkMaintenanceUtils.downloadFromZK(this, zkPath, dir);
}
+ /**
+ * Ensures we can subsequently read the most up-to-date state of the
provided {@code path} and all data below
+ * it, after this method completes.
+ * This should not be called a lot; there may be some delay. Consider
alternative approaches It's often better to try to devise a way to
+ * "watch" for state changed instead of calling this.
+ */
+ public void sync(String path) {
+// zookeeper.sync is asynchronous; we need to wait till it's done
+CompletableFuture future = new CompletableFuture<>();
+keeper.sync(path, (rc, path1, ctx) -> future.complete(rc), null);
+try {
+ Integer status = future.get(zkClientTimeout, TimeUnit.MILLISECONDS);
Review comment:
Why synchronous method do you refer to? If there was one that
accomplishes this task here, I'd call it! keeper.sync() is an asynchronous
method.
##
File path: solr/solrj/src/java/org/apache/solr/common/cloud/SolrZkClient.java
##
@@ -843,6 +847,31 @@ public void downloadFromZK(String zkPath, Path dir) throws
IOException {
ZkMaintenanceUtils.downloadFromZK(this, zkPath, dir);
}
+ /**
+ * Ensures we can subsequently read the most up-to-date state of the
provided {@code path} and all data below
+ * it, after this method completes.
+ * This should not be called a lot; there may be some delay. Consider
alternative approaches It's often better to try to devise a way to
+ * "watch" for state changed instead of calling this.
+ */
+ public void sync(String path) {
+// zookeeper.sync is asynchronous; we need to wait till it's done
+CompletableFuture future = new CompletableFuture<>();
+keeper.sync(path, (rc, path1, ctx) -> future.complete(rc), null);
+try {
+ Integer status = future.get(zkClientTimeout, TimeUnit.MILLISECONDS);
Review comment:
What synchronous method do you refer to? If there was one that
accomplishes this task here, I'd call it! keeper.sync() is an asynchronous
method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] tflobbe commented on a change in pull request #1456: SOLR-13289: Support for BlockMax WAND
tflobbe commented on a change in pull request #1456:
URL: https://github.com/apache/lucene-solr/pull/1456#discussion_r416956723
##
File path:
solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
##
@@ -401,6 +403,14 @@ public void process(ResponseBuilder rb) throws IOException
doProcessUngroupedSearch(rb, cmd, result);
}
+ private int getMinExactHits(SolrParams params) {
+long minExactHits = params.getLong(CommonParams.MIN_EXACT_HITS,
Integer.MAX_VALUE);
Review comment:
I plan to move the default value to a solrconfig config in a future PR,
for now, MAX_VALUE (disabled)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #1456: SOLR-13289: Support for BlockMax WAND
jpountz commented on a change in pull request #1456:
URL: https://github.com/apache/lucene-solr/pull/1456#discussion_r416972950
##
File path: solr/core/src/java/org/apache/solr/search/MaxScoreCollector.java
##
@@ -39,7 +39,7 @@ public float getMaxScore() {
public ScoreMode scoreMode() {
// Should be TOP_SCORES but this would wrap the scorer unnecessarily since
// this collector is only used in a MultiCollector.
-return ScoreMode.COMPLETE;
+return ScoreMode.TOP_SCORES;
Review comment:
Yeah I think that at some point we tried not to wrap the scorer when all
score modes were `COMPLETE` but apparently now we do, so this comment is stale.
https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/search/MultiCollector.java#L158
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Reopened] (LUCENE-9191) Fix linefiledocs compression or replace in tests
[
https://issues.apache.org/jira/browse/LUCENE-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris M. Hostetter reopened LUCENE-9191:
Apache jenkins found some reproducing failures originating from
{{BasePostingsFormatTestCase.testInvertedWrite}} that seem suspiciuosly related
to this issue.
The seeds alone don't seem to reproduce for me, but i didn't try downloading &
using the same
{{tests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt}}
used by jenkins...
https://builds.apache.org/view/L/view/Lucene/job/Lucene-Solr-NightlyTests-master/2170/
{noformat}
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant
get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestDirectPostingsFormat -Dtests.method=testInvertedWrite
-Dtests.seed=172C6414BE5E2A2C -Dtests.multiplier=2 -Dtests.nightly=true
-Dtests.s
low=true
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
-Dtests.locale=shi-Tfng-MA -Dtests.timezone=SystemV/MST7 -Dtests.
asserts=true -Dtests.file.encoding=US-ASCII
[junit4] ERROR 0.03s J2 | TestDirectPostingsFormat.testInvertedWrite <<<
[junit4]> Throwable #1: java.nio.charset.MalformedInputException: Input
length = 1
[junit4]>at
__randomizedtesting.SeedInfo.seed([172C6414BE5E2A2C:E5829DFC005A1F0]:0)
[junit4]>at
java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274)
[junit4]>at
java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
[junit4]>at
java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
[junit4]>at
java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
[junit4]>at
java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
[junit4]>at
java.base/java.io.BufferedReader.readLine(BufferedReader.java:326)
[junit4]>at
java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
[junit4]>at
org.apache.lucene.util.LineFileDocs.open(LineFileDocs.java:175)
[junit4]>at
org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:65)
[junit4]>at
org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:69)
[junit4]>at
org.apache.lucene.index.BasePostingsFormatTestCase.testInvertedWrite(BasePostingsFormatTestCase.java:540)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4]>at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4]>at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
[junit4]>at java.base/java.lang.Thread.run(Thread.java:834)
[junit4] 1> [_0.cfe, _0.cfs, _0.si, segments_1]
...
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant
get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestBloomPostingsFormat -Dtests.method=testInvertedWrite
-Dtests.seed=172C6414BE5E2A2C -Dtests.multiplier=2 -Dtests.nightly=true
-Dtests.slow=true
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
-Dtests.locale=vai-Vaii -Dtests.timezone=SystemV/HST10 -Dtests.asserts=true
-Dtests.file.encoding=US-ASCII
[junit4] ERROR 0.01s J1 | TestBloomPostingsFormat.testInvertedWrite <<<
[junit4]> Throwable #1: java.nio.charset.MalformedInputException: Input
length = 1
[junit4]>at
__randomizedtesting.SeedInfo.seed([172C6414BE5E2A2C:E5829DFC005A1F0]:0)
[junit4]>at
java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274)
[junit4]>at
java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
[junit4]>at
java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
[junit4]>at
java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
[junit4]>at
java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
[junit4]>at
java.base/java.io.BufferedReader.readLine(BufferedReader.java:326)
[junit4]>at
java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
[junit4]>at
org.apache.lucene.util.LineFileDocs.open(LineFileDocs.java:175)
[junit4]>at
org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:65)
[junit4]>at
org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:69)
[jira] [Comment Edited] (LUCENE-9191) Fix linefiledocs compression or replace in tests
[
https://issues.apache.org/jira/browse/LUCENE-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094921#comment-17094921
]
Chris M. Hostetter edited comment on LUCENE-9191 at 4/28/20, 11:19 PM:
---
Apache jenkins found some reproducing failures originating from
{{BasePostingsFormatTestCase.testInvertedWrite}} that seem suspiciuosly related
to this issue.
The seeds alone don't seem to reproduce for me, but i didn't try downloading &
using the same
{{tests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt}}
used by jenkins...
https://builds.apache.org/view/L/view/Lucene/job/Lucene-Solr-NightlyTests-master/2170/
{noformat}
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url
> https://gitbox.apache.org/repos/asf/lucene-solr.git # timeout=10
Fetching upstream changes from
https://gitbox.apache.org/repos/asf/lucene-solr.git
> git --version # timeout=10
> git fetch --tags --progress
> https://gitbox.apache.org/repos/asf/lucene-solr.git
> +refs/heads/*:refs/remotes/origin/*
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision e0c06ee6a6db925efa40b6633869c800d5745261
(refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f e0c06ee6a6db925efa40b6633869c800d5745261
Commit message: "LUCENE-9191: make LineFileDocs random seeking more efficient
by recording safe skip points in the concatenated gzip'd chunks"
> git rev-list --no-walk c94770c2b9c00ccdc2d617d595d62f85a332dc0c # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Cleaning up
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data
Updating http://svn.apache.org/repos/asf/lucene/test-data at revision
'2020-04-22T04:32:15.464 +'
At revision 1876810
No emails were triggered.
[checkout] $ /home/jenkins/tools/ant/apache-ant-1.8.4/bin/ant -file build.xml
-Dtests.multiplier=2
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data
/enwiki.random.lines.txt jenkins-nightly
Buildfile:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml
jenkins-nightly:
-print-java-info:
[java-info] java version "11.0.4"
[java-info] Java(TM) SE Runtime Environment (11.0.4+10-LTS, Oracle Corporation)
[java-info] Java HotSpot(TM) 64-Bit Server VM (11.0.4+10-LTS, Oracle
Corporation)
[java-info] Test args: [-XX:TieredStopAtLevel=1]
...
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant
get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestDirectPostingsFormat -Dtests.method=testInvertedWrite
-Dtests.seed=172C6414BE5E2A2C -Dtests.multiplier=2 -Dtests.nightly=true
-Dtests.s
low=true
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
-Dtests.locale=shi-Tfng-MA -Dtests.timezone=SystemV/MST7 -Dtests.
asserts=true -Dtests.file.encoding=US-ASCII
[junit4] ERROR 0.03s J2 | TestDirectPostingsFormat.testInvertedWrite <<<
[junit4]> Throwable #1: java.nio.charset.MalformedInputException: Input
length = 1
[junit4]>at
__randomizedtesting.SeedInfo.seed([172C6414BE5E2A2C:E5829DFC005A1F0]:0)
[junit4]>at
java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274)
[junit4]>at
java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
[junit4]>at
java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
[junit4]>at
java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
[junit4]>at
java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
[junit4]>at
java.base/java.io.BufferedReader.readLine(BufferedReader.java:326)
[junit4]>at
java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
[junit4]>at
org.apache.lucene.util.LineFileDocs.open(LineFileDocs.java:175)
[junit4]>at
org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:65)
[junit4]>at
org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:69)
[junit4]>at
org.apache.lucene.index.BasePostingsFormatTestCase.testInvertedWrite(BasePostingsFormatTestCase.java:540)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
[jira] [Comment Edited] (LUCENE-9191) Fix linefiledocs compression or replace in tests
[
https://issues.apache.org/jira/browse/LUCENE-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094921#comment-17094921
]
Chris M. Hostetter edited comment on LUCENE-9191 at 4/28/20, 11:21 PM:
---
Apache jenkins found some reproducing failures originating from
{{BasePostingsFormatTestCase.testInvertedWrite}} that seem suspiciuosly related
to this issue.
The seeds alone don't seem to reproduce for me, but i didn't try downloading &
using the same
{{tests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt}}
used by jenkins...
https://builds.apache.org/view/L/view/Lucene/job/Lucene-Solr-NightlyTests-master/2170/
{noformat}
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url
> https://gitbox.apache.org/repos/asf/lucene-solr.git # timeout=10
Fetching upstream changes from
https://gitbox.apache.org/repos/asf/lucene-solr.git
> git --version # timeout=10
> git fetch --tags --progress
> https://gitbox.apache.org/repos/asf/lucene-solr.git
> +refs/heads/*:refs/remotes/origin/*
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision e0c06ee6a6db925efa40b6633869c800d5745261
(refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f e0c06ee6a6db925efa40b6633869c800d5745261
Commit message: "LUCENE-9191: make LineFileDocs random seeking more efficient
by recording safe skip points in the concatenated gzip'd chunks"
> git rev-list --no-walk c94770c2b9c00ccdc2d617d595d62f85a332dc0c # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Cleaning up
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data
Updating http://svn.apache.org/repos/asf/lucene/test-data at revision
'2020-04-22T04:32:15.464 +'
At revision 1876810
No emails were triggered.
[checkout] $ /home/jenkins/tools/ant/apache-ant-1.8.4/bin/ant -file build.xml
-Dtests.multiplier=2
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data
/enwiki.random.lines.txt jenkins-nightly
Buildfile:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml
jenkins-nightly:
-print-java-info:
[java-info] java version "11.0.4"
[java-info] Java(TM) SE Runtime Environment (11.0.4+10-LTS, Oracle Corporation)
[java-info] Java HotSpot(TM) 64-Bit Server VM (11.0.4+10-LTS, Oracle
Corporation)
[java-info] Test args: [-XX:TieredStopAtLevel=1]
...
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant
get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestDirectPostingsFormat -Dtests.method=testInvertedWrite
-Dtests.seed=172C6414BE5E2A2C -Dtests.multiplier=2 -Dtests.nightly=true
-Dtests.s
low=true
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
-Dtests.locale=shi-Tfng-MA -Dtests.timezone=SystemV/MST7 -Dtests.
asserts=true -Dtests.file.encoding=US-ASCII
[junit4] ERROR 0.03s J2 | TestDirectPostingsFormat.testInvertedWrite <<<
[junit4]> Throwable #1: java.nio.charset.MalformedInputException: Input
length = 1
[junit4]>at
__randomizedtesting.SeedInfo.seed([172C6414BE5E2A2C:E5829DFC005A1F0]:0)
[junit4]>at
java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274)
[junit4]>at
java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
[junit4]>at
java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
[junit4]>at
java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
[junit4]>at
java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
[junit4]>at
java.base/java.io.BufferedReader.readLine(BufferedReader.java:326)
[junit4]>at
java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
[junit4]>at
org.apache.lucene.util.LineFileDocs.open(LineFileDocs.java:175)
[junit4]>at
org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:65)
[junit4]>at
org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:69)
[junit4]>at
org.apache.lucene.index.BasePostingsFormatTestCase.testInvertedWrite(BasePostingsFormatTestCase.java:540)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
[jira] [Commented] (SOLR-13289) Support for BlockMax WAND
[
https://issues.apache.org/jira/browse/SOLR-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094923#comment-17094923
]
Tomas Eduardo Fernandez Lobbe commented on SOLR-13289:
--
This is how the relation comes back when using json response writer:
{code}
"response": {
"numFound": 21,
"start": 0,
"hitCountRelation": "GREATER_THAN_OR_EQUAL_TO",
"docs": [
...
{code}
and xml
{code:xml}
...
{code}
When using SolrJ the {{SolrDocumentList}} contains the relation, and can be
accessed via the {{getHitCountRelation()}} method.
> Support for BlockMax WAND
> -
>
> Key: SOLR-13289
> URL: https://issues.apache.org/jira/browse/SOLR-13289
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Tomas Eduardo Fernandez Lobbe
>Priority: Major
> Attachments: SOLR-13289.patch, SOLR-13289.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> LUCENE-8135 introduced BlockMax WAND as a major speed improvement. Need to
> expose this via Solr. When enabled, the numFound returned will not be exact.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob commented on pull request #807: Remove solr.jetty.https.port when SSL is not used
madrob commented on pull request #807: URL: https://github.com/apache/lucene-solr/pull/807#issuecomment-620909328 @janhoy I didn't see this PR until you made your comment, but had to make the same changes as part of SOLR-14412. We can certainly add a CHANGES entry (and credit upendrasoft) if you think it's worthwhile, but the PR itself is probably safe to close. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob edited a comment on pull request #807: Remove solr.jetty.https.port when SSL is not used
madrob edited a comment on pull request #807: URL: https://github.com/apache/lucene-solr/pull/807#issuecomment-620909328 @janhoy I didn't see this PR until you made your comment, but had to make the same changes as part of SOLR-14412 (0fc5179c6a7). We can certainly add a CHANGES entry (and credit upendrasoft) if you think it's worthwhile, but the PR itself is probably safe to close. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9191) Fix linefiledocs compression or replace in tests
[
https://issues.apache.org/jira/browse/LUCENE-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094930#comment-17094930
]
Chris M. Hostetter commented on LUCENE-9191:
Similar root causes in other tests in other builds w/diff seeds...
{noformat}
Checking out Revision ecc98e8698a3ce8efa51712686697c0f33afab4d
(refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f ecc98e8698a3ce8efa51712686697c0f33afab4d
Commit message: "LUCENE-7788: fail precommit on unparameterised log messages
and examine for wasted work/objects"
> git rev-list --no-walk 03363f413f2134594b012175deb3f10ec9384400 # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Cleaning up
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/test-data
Updating http://svn.apache.org/repos/asf/lucene/test-data at revision
'2020-04-24T17:44:24.647 +'
At revision 1876938
No emails were triggered.
[checkout] $ /home/jenkins/tools/ant/apache-ant-1.8.4/bin/ant -file build.xml
-Dtests.multiplier=2
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
BadApples-NightlyTests-master/test-data/enwiki.random.lines.txt
jenkins-nightly-badapples
Buildfile:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/build.xml
jenkins-nightly-badapples:
-print-java-info:
[java-info] java version "11.0.4"
[java-info] Java(TM) SE Runtime Environment (11.0.4+10-LTS, Oracle Corporation)
[java-info] Java HotSpot(TM) 64-Bit Server VM (11.0.4+10-LTS, Oracle
Corporation)
[java-info] Test args: [-XX:TieredStopAtLevel=1]
...
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestSameScoresWithThreads -Dtests.method=test
-Dtests.seed=6B89EFC89E940CDC -Dtests.multiplier=2 -Dtests.nightly=true
-Dtests.slow=true -Dtests.badapples=true
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/test-data/enwiki.random.lines.txt
-Dtests.locale=ar-SY -Dtests.timezone=Africa/Dar_es_Salaam
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.02s J1 | TestSameScoresWithThreads.test <<<
[junit4]> Throwable #1: java.nio.charset.MalformedInputException: Input
length = 1
[junit4]>at
__randomizedtesting.SeedInfo.seed([6B89EFC89E940CDC:E3DDD01230686124]:0)
[junit4]>at
java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274)
[junit4]>at
java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
[junit4]>at
java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
[junit4]>at
java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
[junit4]>at
java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
[junit4]>at
java.base/java.io.BufferedReader.readLine(BufferedReader.java:326)
[junit4]>at
java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
[junit4]>at
org.apache.lucene.util.LineFileDocs.open(LineFileDocs.java:175)
[junit4]>at
org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:65)
[junit4]>at
org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:69)
[junit4]>at
org.apache.lucene.search.TestSameScoresWithThreads.test(TestSameScoresWithThreads.java:49)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4]>at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4]>at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
[junit4]>at java.base/java.lang.Thread.run(Thread.java:834)
...
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestAllFilesHaveCodecHeader -Dtests.method=test
-Dtests.seed=6B89EFC89E940CDC -Dtests.multiplier=2 -Dtests.nightly=true
-Dtests.slow=true -Dtests.badapples=true
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/test-data/enwiki.random.lines.txt
-Dtests.locale=ar-LY -Dtests.timezone=Africa/Windhoek -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J1 | TestAllFilesHaveCodecHeader.test <<<
[junit4]> Throwable #1: java.nio.charset.MalformedInputException: Input
length = 1
[junit4]>at
__randomizedtesting.SeedInfo.seed([6B89EFC89E940CDC:E3DDD01230686124]:0)
[junit4]>at
java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274)
[junit4]>at
java.base
[jira] [Commented] (LUCENE-7788) fail precommit on unparameterised log messages and examine for wasted work/objects
[ https://issues.apache.org/jira/browse/LUCENE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17095141#comment-17095141 ] Dawid Weiss commented on LUCENE-7788: - Erick there are "nocommit" substrings in validate-log-calls.gradle - these strings should be removed from master, I guess? > fail precommit on unparameterised log messages and examine for wasted > work/objects > -- > > Key: LUCENE-7788 > URL: https://issues.apache.org/jira/browse/LUCENE-7788 > Project: Lucene - Core > Issue Type: Task >Reporter: Christine Poerschke >Assignee: Erick Erickson >Priority: Minor > Attachments: LUCENE-7788.patch, LUCENE-7788.patch, gradle_only.patch, > gradle_only.patch > > Time Spent: 50m > Remaining Estimate: 0h > > SOLR-10415 would be removing existing unparameterised log.trace messages use > and once that is in place then this ticket's one-line change would be for > 'ant precommit' to reject any future unparameterised log.trace message use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
