[GitHub] [lucene-solr] s1monw commented on a change in pull request #1573: Cleanup TermsHashPerField
s1monw commented on a change in pull request #1573:
URL: https://github.com/apache/lucene-solr/pull/1573#discussion_r439803910
##
File path:
lucene/core/src/java/org/apache/lucene/index/FreqProxTermsWriterPerField.java
##
@@ -207,8 +202,6 @@ public void newPostingsArray() {
@Override
ParallelPostingsArray createPostingsArray(int size) {
-IndexOptions indexOptions = fieldInfo.getIndexOptions();
-assert indexOptions != IndexOptions.NONE;
Review comment:
I moved it in a better place in the ctor of the base class. I think
that's enough?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on a change in pull request #1573: Cleanup TermsHashPerField
s1monw commented on a change in pull request #1573:
URL: https://github.com/apache/lucene-solr/pull/1573#discussion_r439804315
##
File path:
lucene/core/src/java/org/apache/lucene/index/FreqProxTermsWriterPerField.java
##
@@ -56,12 +56,6 @@ public FreqProxTermsWriterPerField(FieldInvertState
invertState, TermsHash terms
@Override
void finish() throws IOException {
super.finish();
-sumDocFreq += fieldState.uniqueTermCount;
-sumTotalTermFreq += fieldState.length;
Review comment:
`sumDocFreq` and `sumTotalTermFreq` are unused. They were used in
`FreqProxFields` in the past but not anymore for a while now. I removed their
commented out usage so you can see it in a followup commit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on a change in pull request #1573: Cleanup TermsHashPerField
s1monw commented on a change in pull request #1573:
URL: https://github.com/apache/lucene-solr/pull/1573#discussion_r439804443
##
File path: lucene/core/src/java/org/apache/lucene/index/TermsHashPerField.java
##
@@ -19,203 +19,207 @@
import java.io.IOException;
-import org.apache.lucene.analysis.tokenattributes.TermFrequencyAttribute;
-import org.apache.lucene.analysis.tokenattributes.TermToBytesRefAttribute;
import org.apache.lucene.util.ByteBlockPool;
+import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.BytesRefHash.BytesStartArray;
import org.apache.lucene.util.BytesRefHash;
import org.apache.lucene.util.Counter;
import org.apache.lucene.util.IntBlockPool;
+/**
+ * This class allows to store streams of information per term without knowing
+ * the size of the stream ahead of time. Each stream typically encodes one
level
+ * of information like term frequency per document or term proximity.
Internally
+ * this class allocates a linked list of slices that can be read by a {@link
ByteSliceReader}
+ * for each term. Terms are first deduplicated in a {@link BytesRefHash} once
this is done
+ * internal data-structures point to the current offset of each stream that
can be written to.
+ */
abstract class TermsHashPerField implements Comparable {
private static final int HASH_INIT_SIZE = 4;
- final TermsHash termsHash;
-
- final TermsHashPerField nextPerField;
- protected final DocumentsWriterPerThread.DocState docState;
- protected final FieldInvertState fieldState;
- TermToBytesRefAttribute termAtt;
- protected TermFrequencyAttribute termFreqAtt;
-
- // Copied from our perThread
- final IntBlockPool intPool;
+ private final TermsHashPerField nextPerField;
+ private final IntBlockPool intPool;
final ByteBlockPool bytePool;
- final ByteBlockPool termBytePool;
-
- final int streamCount;
- final int numPostingInt;
-
- protected final FieldInfo fieldInfo;
-
- final BytesRefHash bytesHash;
+ // for each term we store an integer per stream that points into the
bytePool above
+ // the address is updated once data is written to the stream to point to the
next free offset
+ // this the terms stream. The start address for the stream is stored in
postingsArray.byteStarts[termId]
+ // This is initialized in the #addTerm method, either to a brand new per
term stream if the term is new or
+ // to the addresses where the term stream was written to when we saw it the
last time.
+ private int[] termStreamAddressBuffer;
+ private int streamAddressOffset;
+ private final int streamCount;
+ private final String fieldName;
+ final IndexOptions indexOptions;
+ /* This stores the actual term bytes for postings and offsets into the
parent hash in the case that this
+ * TermsHashPerField is hashing term vectors.*/
+ private final BytesRefHash bytesHash;
ParallelPostingsArray postingsArray;
- private final Counter bytesUsed;
+ private int lastDocID; // only with assert
/** streamCount: how many streams this field stores per term.
* E.g. doc(+freq) is 1 stream, prox+offset is a second. */
-
- public TermsHashPerField(int streamCount, FieldInvertState fieldState,
TermsHash termsHash, TermsHashPerField nextPerField, FieldInfo fieldInfo) {
-intPool = termsHash.intPool;
-bytePool = termsHash.bytePool;
-termBytePool = termsHash.termBytePool;
-docState = termsHash.docState;
-this.termsHash = termsHash;
-bytesUsed = termsHash.bytesUsed;
-this.fieldState = fieldState;
+ TermsHashPerField(int streamCount, IntBlockPool intPool, ByteBlockPool
bytePool, ByteBlockPool termBytePool,
+Counter bytesUsed, TermsHashPerField nextPerField, String
fieldName, IndexOptions indexOptions) {
+this.intPool = intPool;
+this.bytePool = bytePool;
this.streamCount = streamCount;
-numPostingInt = 2*streamCount;
-this.fieldInfo = fieldInfo;
+this.fieldName = fieldName;
this.nextPerField = nextPerField;
+assert indexOptions != IndexOptions.NONE;
Review comment:
yeah :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on a change in pull request #1573: Cleanup TermsHashPerField
s1monw commented on a change in pull request #1573:
URL: https://github.com/apache/lucene-solr/pull/1573#discussion_r439804390
##
File path:
lucene/core/src/java/org/apache/lucene/index/TermVectorsConsumerPerField.java
##
@@ -222,7 +234,7 @@ void writeProx(TermVectorsPostingsArray postings, int
termID) {
}
@Override
- void newTerm(final int termID) {
+ void newTerm(final int termID, final int docID) {
Review comment:
that's correct.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on pull request #1573: Cleanup TermsHashPerField
s1monw commented on pull request #1573: URL: https://github.com/apache/lucene-solr/pull/1573#issuecomment-643735813 > I think we should run indexing throughput benchmark to make sure there's no real impact here. I'll do that. +1 thanks @mikemccand This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on a change in pull request #1573: Cleanup TermsHashPerField
s1monw commented on a change in pull request #1573:
URL: https://github.com/apache/lucene-solr/pull/1573#discussion_r439805381
##
File path:
lucene/core/src/java/org/apache/lucene/index/FreqProxTermsWriterPerField.java
##
@@ -56,12 +56,6 @@ public FreqProxTermsWriterPerField(FieldInvertState
invertState, TermsHash terms
@Override
void finish() throws IOException {
super.finish();
-sumDocFreq += fieldState.uniqueTermCount;
-sumTotalTermFreq += fieldState.length;
Review comment:
see this
https://github.com/apache/lucene-solr/pull/1573/files#diff-aa6c5376b6b755262430916164fd0088L84
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14567) Fix or suppress remaining warnings in solrj
[ https://issues.apache.org/jira/browse/SOLR-14567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135133#comment-17135133 ] ASF subversion and git services commented on SOLR-14567: Commit c7c17b79928dc9b08a0568eb2103401ceeca4649 in lucene-solr's branch refs/heads/branch_8x from Erick Erickson [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c7c17b7 ] SOLR-14567: Fix or suppress remaining warnings in solrj > Fix or suppress remaining warnings in solrj > --- > > Key: SOLR-14567 > URL: https://issues.apache.org/jira/browse/SOLR-14567 > Project: Solr > Issue Type: Sub-task >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > > This is another place where the number of warnings per directory is getting > too small to do individually, so I'll do them all in a bunch. > Note: this will exclude autoscaling. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14567) Fix or suppress remaining warnings in solrj
[ https://issues.apache.org/jira/browse/SOLR-14567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135134#comment-17135134 ] ASF subversion and git services commented on SOLR-14567: Commit 4e90e48ac29ee38662c04fcf7815d5170ceb2669 in lucene-solr's branch refs/heads/master from Erick Erickson [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=4e90e48 ] SOLR-14567: Fix or suppress remaining warnings in solrj > Fix or suppress remaining warnings in solrj > --- > > Key: SOLR-14567 > URL: https://issues.apache.org/jira/browse/SOLR-14567 > Project: Solr > Issue Type: Sub-task >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > > This is another place where the number of warnings per directory is getting > too small to do individually, so I'll do them all in a bunch. > Note: this will exclude autoscaling. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14567) Fix or suppress remaining warnings in solrj
[ https://issues.apache.org/jira/browse/SOLR-14567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson resolved SOLR-14567. --- Fix Version/s: 8.6 Resolution: Fixed > Fix or suppress remaining warnings in solrj > --- > > Key: SOLR-14567 > URL: https://issues.apache.org/jira/browse/SOLR-14567 > Project: Solr > Issue Type: Sub-task >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > > This is another place where the number of warnings per directory is getting > too small to do individually, so I'll do them all in a bunch. > Note: this will exclude autoscaling. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9403) tune BufferedChecksum.DEFAULT_BUFFERSIZE
[
https://issues.apache.org/jira/browse/LUCENE-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Muir updated LUCENE-9403:
Attachment: LUCENE-9403.patch
> tune BufferedChecksum.DEFAULT_BUFFERSIZE
>
>
> Key: LUCENE-9403
> URL: https://issues.apache.org/jira/browse/LUCENE-9403
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Robert Muir
>Priority: Major
> Attachments: LUCENE-9403.patch
>
>
> This is currently set to 256 bytes, so that's the amount of data we pass to
> crc.update() at once.
> I tried different sizes with https://github.com/benalexau/hash-bench and
> JDK14:
> {noformat}
> HashBench.withArray crc32-jre 256 avgt5 81.349 ± 8.364
> ns/op
> HashBench.withArray crc32-jre 512 avgt5 95.204 ± 10.057
> ns/op
> HashBench.withArray crc32-jre 1024 avgt5 120.081 ± 8.471
> ns/op
> HashBench.withArray crc32-jre 2048 avgt5 173.505 ± 8.857
> ns/op
> HashBench.withArray crc32-jre 8192 avgt5 487.721 ± 11.435
> ns/op
> {noformat}
> based on this let's bump the buffersize from 256 to 1024? I think we want to
> avoid huge buffers but still keep the CPU overhead low. It only impacts
> ChecksumIndexInputs (e.g. speed of checkIntegrity() calls at merge) because
> IndexOutputs do not need this buffer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9403) tune BufferedChecksum.DEFAULT_BUFFERSIZE
Robert Muir created LUCENE-9403:
---
Summary: tune BufferedChecksum.DEFAULT_BUFFERSIZE
Key: LUCENE-9403
URL: https://issues.apache.org/jira/browse/LUCENE-9403
Project: Lucene - Core
Issue Type: Improvement
Reporter: Robert Muir
Attachments: LUCENE-9403.patch
This is currently set to 256 bytes, so that's the amount of data we pass to
crc.update() at once.
I tried different sizes with https://github.com/benalexau/hash-bench and JDK14:
{noformat}
HashBench.withArray crc32-jre 256 avgt5 81.349 ± 8.364
ns/op
HashBench.withArray crc32-jre 512 avgt5 95.204 ± 10.057
ns/op
HashBench.withArray crc32-jre 1024 avgt5 120.081 ± 8.471
ns/op
HashBench.withArray crc32-jre 2048 avgt5 173.505 ± 8.857
ns/op
HashBench.withArray crc32-jre 8192 avgt5 487.721 ± 11.435
ns/op
{noformat}
based on this let's bump the buffersize from 256 to 1024? I think we want to
avoid huge buffers but still keep the CPU overhead low. It only impacts
ChecksumIndexInputs (e.g. speed of checkIntegrity() calls at merge) because
IndexOutputs do not need this buffer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14541) Ensure classes that implement equals implement hashCode or suppress warnings
[ https://issues.apache.org/jira/browse/SOLR-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135231#comment-17135231 ] Erick Erickson commented on SOLR-14541: --- This branch has the hashCode commented out, ping me if you need help before I get to back to this. I want to do all these in a wodge. > Ensure classes that implement equals implement hashCode or suppress warnings > > > Key: SOLR-14541 > URL: https://issues.apache.org/jira/browse/SOLR-14541 > Project: Solr > Issue Type: Sub-task >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Attachments: 0001-SOLR-14541-add-hashCode-for-some-classes.patch, > 0002-SOLR-14541-add-hashCode-for-some-classes-in-autoscal.patch, > 0003-SOLR-14541-add-hashCode-or-remove-equals-for-some-cl.patch > > > While looking at warnings, I found that the following classes generate this > warning: > *overrides equals, but neither it nor any superclass overrides hashCode > method* > I can suppress the warning, but this has been a source of errors in the past > so I'm reluctant to just do that blindly. > NOTE: The Lucene one should probably be it's own Jira if it's going to have > hashCode implemented, but here for triage. > What I need for each method is for someone who has a clue about that > particular code to render an opinion that we can safely suppress the warning > or to provide a hashCode method. > Some of these have been here for a very long time and were implemented by > people no longer active... > lucene/suggest/src/java/org/apache/lucene/search/spell/LuceneLevenshteinDistance.java:39 > solr/solrj/src/java/org/apache/solr/common/cloud/ZkNodeProps.java:34 > solr/solrj/src/java/org/apache/solr/common/cloud/Replica.java:26 > solr/solrj/src/java/org/apache/solr/common/cloud/DocCollection.java:49 > solr/core/src/java/org/apache/solr/cloud/rule/Rule.java:277 > solr/core/src/java/org/apache/solr/pkg/PackageAPI.java:177 > solr/core/src/java/org/apache/solr/packagemanager/SolrPackageInstance.java:31 > > Noble Paul says it's OK to suppress warnings for these: > solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/VersionedData.java:31 > > solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/AutoScalingConfig.java:61 > > solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/AutoScalingConfig.java:150 > > solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/AutoScalingConfig.java:252 > > solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/AutoScalingConfig.java:45 > > solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java:73 > > solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Preference.java:32 > > solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/ReplicaInfo.java:39 > > Joel Bernstein says it's OK to suppress warnings for these: > > solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/ReplicaCount.java:27 > > solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamExpression.java:25 > > solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamExpressionNamedParameter.java:23 > > solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/CloudSolrStream.java:467 > > solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/DeepRandomStream.java:417 > > solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamExpressionValue.java:22 > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14566) Record "NOW" on "coordinator" log messages
[
https://issues.apache.org/jira/browse/SOLR-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135245#comment-17135245
]
Robert Muir commented on SOLR-14566:
{quote}
An alternative approach would be to create a qid or query-uuid when the
coordinator starts its work that can be logged everywhere. This provides a
stronger expectation around uniqueness, but would require UUID generation on
the coordinator, which may be non-negligible work at high QPS (maybe? I have no
idea). It also loses the neatness of reusing data already present on the shard
requests.
{quote}
I don't think a UUID is needed: you just want it to be unique per request for
correlation (can use ips and ports and request counters and stuff). There are
some good recommendations from haproxy docs:
https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#4-unique-id-format.
I usually set it as X-Request-ID header and its usually easy to plumb to the
logs in any programming language. in the worst case you can just set a
threadlocal variable and use that.
> Record "NOW" on "coordinator" log messages
> --
>
> Key: SOLR-14566
> URL: https://issues.apache.org/jira/browse/SOLR-14566
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, in SolrCore.java we log each search request that comes through
> each core as it is finishing. This includes the path, query-params, QTime,
> and status. In the case of a distributed search both the "coordinator" node
> and each of the per-shard requests produce a log message.
> When Solr is fielding many identical queries, such as those created by a
> healthcheck or dashboard, it can be hard when examining logs to link the
> per-shard requests with the "cooordinator" request that came in upstream.
> One thing that would make this easier is if the {{NOW}} param added to
> per-shard requests is also included in the log message from the
> "coordinator". While {{NOW}} isn't unique strictly speaking, it often is in
> practice, and along with the query-params would allow debuggers to associate
> shard requests with coordinator requests a large majority of the time.
> An alternative approach would be to create a {{qid}} or {{query-uuid}} when
> the coordinator starts its work that can be logged everywhere. This provides
> a stronger expectation around uniqueness, but would require UUID generation
> on the coordinator, which may be non-negligible work at high QPS (maybe? I
> have no idea). It also loses the neatness of reusing data already present on
> the shard requests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9403) tune BufferedChecksum.DEFAULT_BUFFERSIZE
[
https://issues.apache.org/jira/browse/LUCENE-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135263#comment-17135263
]
Michael McCandless commented on LUCENE-9403:
+1 looks compelling!
> tune BufferedChecksum.DEFAULT_BUFFERSIZE
>
>
> Key: LUCENE-9403
> URL: https://issues.apache.org/jira/browse/LUCENE-9403
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Robert Muir
>Priority: Major
> Attachments: LUCENE-9403.patch
>
>
> This is currently set to 256 bytes, so that's the amount of data we pass to
> crc.update() at once.
> I tried different sizes with https://github.com/benalexau/hash-bench and
> JDK14:
> {noformat}
> HashBench.withArray crc32-jre 256 avgt5 81.349 ± 8.364
> ns/op
> HashBench.withArray crc32-jre 512 avgt5 95.204 ± 10.057
> ns/op
> HashBench.withArray crc32-jre 1024 avgt5 120.081 ± 8.471
> ns/op
> HashBench.withArray crc32-jre 2048 avgt5 173.505 ± 8.857
> ns/op
> HashBench.withArray crc32-jre 8192 avgt5 487.721 ± 11.435
> ns/op
> {noformat}
> based on this let's bump the buffersize from 256 to 1024? I think we want to
> avoid huge buffers but still keep the CPU overhead low. It only impacts
> ChecksumIndexInputs (e.g. speed of checkIntegrity() calls at merge) because
> IndexOutputs do not need this buffer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1573: Cleanup TermsHashPerField
dweiss commented on a change in pull request #1573:
URL: https://github.com/apache/lucene-solr/pull/1573#discussion_r439857921
##
File path: lucene/core/src/java/org/apache/lucene/index/TermsHashPerField.java
##
@@ -19,203 +19,207 @@
import java.io.IOException;
-import org.apache.lucene.analysis.tokenattributes.TermFrequencyAttribute;
-import org.apache.lucene.analysis.tokenattributes.TermToBytesRefAttribute;
import org.apache.lucene.util.ByteBlockPool;
+import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.BytesRefHash.BytesStartArray;
import org.apache.lucene.util.BytesRefHash;
import org.apache.lucene.util.Counter;
import org.apache.lucene.util.IntBlockPool;
+/**
+ * This class stores streams of information per term without knowing
+ * the size of the stream ahead of time. Each stream typically encodes one
level
+ * of information like term frequency per document or term proximity.
Internally
+ * this class allocates a linked list of slices that can be read by a {@link
ByteSliceReader}
+ * for each term. Terms are first deduplicated in a {@link BytesRefHash} once
this is done
+ * internal data-structures point to the current offset of each stream that
can be written to.
+ */
abstract class TermsHashPerField implements Comparable {
private static final int HASH_INIT_SIZE = 4;
- final TermsHash termsHash;
-
- final TermsHashPerField nextPerField;
- protected final DocumentsWriterPerThread.DocState docState;
- protected final FieldInvertState fieldState;
- TermToBytesRefAttribute termAtt;
- protected TermFrequencyAttribute termFreqAtt;
-
- // Copied from our perThread
- final IntBlockPool intPool;
+ private final TermsHashPerField nextPerField;
+ private final IntBlockPool intPool;
final ByteBlockPool bytePool;
- final ByteBlockPool termBytePool;
-
- final int streamCount;
- final int numPostingInt;
-
- protected final FieldInfo fieldInfo;
-
- final BytesRefHash bytesHash;
+ // for each term we store an integer per stream that points into the
bytePool above
+ // the address is updated once data is written to the stream to point to the
next free offset
+ // this the terms stream. The start address for the stream is stored in
postingsArray.byteStarts[termId]
Review comment:
this the?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1573: Cleanup TermsHashPerField
dweiss commented on a change in pull request #1573:
URL: https://github.com/apache/lucene-solr/pull/1573#discussion_r439858046
##
File path: lucene/core/src/java/org/apache/lucene/index/TermsHashPerField.java
##
@@ -19,203 +19,207 @@
import java.io.IOException;
-import org.apache.lucene.analysis.tokenattributes.TermFrequencyAttribute;
-import org.apache.lucene.analysis.tokenattributes.TermToBytesRefAttribute;
import org.apache.lucene.util.ByteBlockPool;
+import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.BytesRefHash.BytesStartArray;
import org.apache.lucene.util.BytesRefHash;
import org.apache.lucene.util.Counter;
import org.apache.lucene.util.IntBlockPool;
+/**
+ * This class stores streams of information per term without knowing
+ * the size of the stream ahead of time. Each stream typically encodes one
level
+ * of information like term frequency per document or term proximity.
Internally
+ * this class allocates a linked list of slices that can be read by a {@link
ByteSliceReader}
+ * for each term. Terms are first deduplicated in a {@link BytesRefHash} once
this is done
+ * internal data-structures point to the current offset of each stream that
can be written to.
+ */
abstract class TermsHashPerField implements Comparable {
private static final int HASH_INIT_SIZE = 4;
- final TermsHash termsHash;
-
- final TermsHashPerField nextPerField;
- protected final DocumentsWriterPerThread.DocState docState;
- protected final FieldInvertState fieldState;
- TermToBytesRefAttribute termAtt;
- protected TermFrequencyAttribute termFreqAtt;
-
- // Copied from our perThread
- final IntBlockPool intPool;
+ private final TermsHashPerField nextPerField;
+ private final IntBlockPool intPool;
final ByteBlockPool bytePool;
- final ByteBlockPool termBytePool;
-
- final int streamCount;
- final int numPostingInt;
-
- protected final FieldInfo fieldInfo;
-
- final BytesRefHash bytesHash;
+ // for each term we store an integer per stream that points into the
bytePool above
+ // the address is updated once data is written to the stream to point to the
next free offset
+ // this the terms stream. The start address for the stream is stored in
postingsArray.byteStarts[termId]
+ // This is initialized in the #addTerm method, either to a brand new per
term stream if the term is new or
+ // to the addresses where the term stream was written to when we saw it the
last time.
+ private int[] termStreamAddressBuffer;
+ private int streamAddressOffset;
+ private final int streamCount;
+ private final String fieldName;
+ final IndexOptions indexOptions;
+ /* This stores the actual term bytes for postings and offsets into the
parent hash in the case that this
+ * TermsHashPerField is hashing term vectors.*/
+ private final BytesRefHash bytesHash;
ParallelPostingsArray postingsArray;
- private final Counter bytesUsed;
+ private int lastDocID; // only with assert
/** streamCount: how many streams this field stores per term.
* E.g. doc(+freq) is 1 stream, prox+offset is a second. */
-
- public TermsHashPerField(int streamCount, FieldInvertState fieldState,
TermsHash termsHash, TermsHashPerField nextPerField, FieldInfo fieldInfo) {
-intPool = termsHash.intPool;
-bytePool = termsHash.bytePool;
-termBytePool = termsHash.termBytePool;
-docState = termsHash.docState;
-this.termsHash = termsHash;
-bytesUsed = termsHash.bytesUsed;
-this.fieldState = fieldState;
+ TermsHashPerField(int streamCount, IntBlockPool intPool, ByteBlockPool
bytePool, ByteBlockPool termBytePool,
+Counter bytesUsed, TermsHashPerField nextPerField, String
fieldName, IndexOptions indexOptions) {
+this.intPool = intPool;
+this.bytePool = bytePool;
this.streamCount = streamCount;
-numPostingInt = 2*streamCount;
-this.fieldInfo = fieldInfo;
+this.fieldName = fieldName;
this.nextPerField = nextPerField;
+assert indexOptions != IndexOptions.NONE;
+this.indexOptions = indexOptions;
PostingsBytesStartArray byteStarts = new PostingsBytesStartArray(this,
bytesUsed);
bytesHash = new BytesRefHash(termBytePool, HASH_INIT_SIZE, byteStarts);
}
void reset() {
bytesHash.clear(false);
+sortedTermIDs = null;
if (nextPerField != null) {
nextPerField.reset();
}
}
- public void initReader(ByteSliceReader reader, int termID, int stream) {
+ final void initReader(ByteSliceReader reader, int termID, int stream) {
assert stream < streamCount;
-int intStart = postingsArray.intStarts[termID];
-final int[] ints = intPool.buffers[intStart >>
IntBlockPool.INT_BLOCK_SHIFT];
-final int upto = intStart & IntBlockPool.INT_BLOCK_MASK;
+int streamStartOffset = postingsArray.addressOffset[termID];
+final int[] streamAddressBuffer = intPool.buffers[streamStartOffset >>
IntBlockPool.INT_BLOCK_SHIFT];
+final int offsetInAddressBuff
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #1572: SOLR-14561 CoreAdminAPI's parameters instanceDir and dataDir are now validated
dsmiley commented on a change in pull request #1572:
URL: https://github.com/apache/lucene-solr/pull/1572#discussion_r439789761
##
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##
@@ -338,6 +339,19 @@ public CoreContainer(NodeConfig config, CoresLocator
locator, boolean asyncSolrC
ExecutorUtil.newMDCAwareCachedThreadPool(
cfg.getReplayUpdatesThreads(),
new SolrNamedThreadFactory("replayUpdatesExecutor")));
+
+this.allowPaths = new java.util.HashSet<>();
+this.allowPaths.add(cfg.getSolrHome().toAbsolutePath());
Review comment:
paths in cfg are already absolute -- thanks to me recently :-)
##
File path: solr/core/src/test-files/solr/solr-50-all.xml
##
@@ -24,6 +24,7 @@
testConfigSetsHandler
testManagementPath
testSharedLib
+ /tmp,/home/john
Review comment:
Since this file is a NamedList, shouldn't we be using `
[jira] [Commented] (SOLR-14556) Fix or suppress warnings in solrj/cloud/autoscaling
[ https://issues.apache.org/jira/browse/SOLR-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135278#comment-17135278 ] ASF subversion and git services commented on SOLR-14556: Commit c8bb3bdb524ea609d20ce566944d54a2a648b7ce in lucene-solr's branch refs/heads/branch_8x from Erick Erickson [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c8bb3bd ] SOLR-14556: Fix or suppress warnings in solrj/cloud/autoscaling > Fix or suppress warnings in solrj/cloud/autoscaling > --- > > Key: SOLR-14556 > URL: https://issues.apache.org/jira/browse/SOLR-14556 > Project: Solr > Issue Type: Sub-task >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14556) Fix or suppress warnings in solrj/cloud/autoscaling
[ https://issues.apache.org/jira/browse/SOLR-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135277#comment-17135277 ] ASF subversion and git services commented on SOLR-14556: Commit 8426dc7a6a3b6b059b38bf0d3fd22a3ef61a45b6 in lucene-solr's branch refs/heads/master from Erick Erickson [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=8426dc7 ] SOLR-14556: Fix or suppress warnings in solrj/cloud/autoscaling > Fix or suppress warnings in solrj/cloud/autoscaling > --- > > Key: SOLR-14556 > URL: https://issues.apache.org/jira/browse/SOLR-14556 > Project: Solr > Issue Type: Sub-task >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14556) Fix or suppress warnings in solrj/cloud/autoscaling
[ https://issues.apache.org/jira/browse/SOLR-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson resolved SOLR-14556. --- Fix Version/s: 8.6 Resolution: Fixed They said it couldn't be done! > Fix or suppress warnings in solrj/cloud/autoscaling > --- > > Key: SOLR-14556 > URL: https://issues.apache.org/jira/browse/SOLR-14556 > Project: Solr > Issue Type: Sub-task >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14479) Fix or suppress warnings in solr/analysis
[ https://issues.apache.org/jira/browse/SOLR-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson resolved SOLR-14479. --- Resolution: Fixed Taken care of this in other JIRAs > Fix or suppress warnings in solr/analysis > - > > Key: SOLR-14479 > URL: https://issues.apache.org/jira/browse/SOLR-14479 > Project: Solr > Issue Type: Sub-task > Components: Build >Reporter: Erick Erickson >Assignee: Gus Heck >Priority: Major > > [~gus] Ask and ye shall receive. > Here's how I'd like to approach this: > * Let's start with solr/core, one subdirectory at a time. > * See SOLR-14474 for how we want to address auxiliary classes, especially > the question to move them to their own file or nest them. It'll be fuzzy > until we get some more experience. > * Let's just clean everything up _except_ deprecations. My thinking here is > that there will be a bunch of code changes that we can/should backport to 8x > to clean up the warnings. Deprecations will be (probably) 9.0 only so > there'll be fewer problems with maintaining the two branches if we leave > deprecations out of the mix for the present. > * Err on the side of adding @SuppressWarnings rather than code changes for > this phase. If it's reasonably safe to change the code (say by adding ) do > so, but substantive changes are too likely to have unintended consequences. > I'd like to reach a consensus on what changes are "safe", that'll probably be > an ongoing discussion as we run into them for a while. > * I expect there'll be a certain amount of stepping on each other's toes, no > doubt to clean some things up in one of the subdirectories we'll have to > change something in an ancestor directory, but we can deal with those as they > come up, probably that'll just mean merging the current master with the fork > we're working on... > Let me know what you think or if you'd like to change the approach. > Oh, and all I did here was take the first subdirectory of solr/core that I > found, feel free to take on something else. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14569) HTTP 401 when searching on alias in secured Solr
Isabelle Giguere created SOLR-14569: --- Summary: HTTP 401 when searching on alias in secured Solr Key: SOLR-14569 URL: https://issues.apache.org/jira/browse/SOLR-14569 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: Authentication Affects Versions: 8.5, master (9.0) Environment: Unit test on master branch (9x) built on Windows 10 with Java 11 Solr 8.5.0 instance running on CentOS 7.7 with Java 11 Reporter: Isabelle Giguere The issue was first noticed on an instance of Solr 8.5.0, after securing Solr with security.json. Searching on a single collection returns the expected results, but searching on an alias returns HTTP 401. Note that this issue is not reproduced when the collections are created using the _default configuration. The attached patch includes a unit test that reproduces the issue. The unit test is added to the test class that was originally part of the patch to fix SOLR-13510. I also attach: - our product-specific Solr configuration, modified to remove irrelevant plugins and fields - security.json with user 'admin' (pwd 'admin') -- Note that forwardCredentials true or false does not modify the behavior To test with this configuration: - Download and unzip Solr 8.5.0 - Modify ./bin/solr.in.sh : -- ZK_HOST (optional) -- SOLR_AUTH_TYPE="basic" -- SOLR_AUTHENTICATION_OPTS="-Dbasicauth=admin:admin" - Upload security.json into Zookeeper -- ./bin/solr zk cp file:/path/to/security.json zk:/path/to/solr/security.json [-z :[/]] - Start Solr in cloud mode -- ./bin/solr -c - Upload the provided configuration - ./bin/solr zk upconfig -z :[/] -n conf_en -d /path/to/folder/conf/ - Create 2 collections using the uploaded configuration -- test1, test2 - Create an alias grouping the 2 collections -- test = test1, test2 - Query (/select?q=*:*) one collection -- results in successful Solr response - Query the alias (/select?q=*:*) -- results in HTTP 401 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14569) HTTP 401 when searching on alias in secured Solr
[ https://issues.apache.org/jira/browse/SOLR-14569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Isabelle Giguere updated SOLR-14569: Description: The issue was first noticed on an instance of Solr 8.5.0, after securing Solr with security.json. Searching on a single collection returns the expected results, but searching on an alias returns HTTP 401. Note that this issue is not reproduced when the collections are created using the _default configuration. The attached patch includes a unit test that reproduces the issue. Patch applies on master branch (9x). The unit test is added to the test class that was originally part of the patch to fix SOLR-13510. I also attach: - our product-specific Solr configuration, modified to remove irrelevant plugins and fields - security.json with user 'admin' (pwd 'admin') -- Note that forwardCredentials true or false does not modify the behavior To test with this configuration: - Download and unzip Solr 8.5.0 - Modify ./bin/solr.in.sh : -- ZK_HOST (optional) -- SOLR_AUTH_TYPE="basic" -- SOLR_AUTHENTICATION_OPTS="-Dbasicauth=admin:admin" - Upload security.json into Zookeeper -- ./bin/solr zk cp file:/path/to/security.json zk:/path/to/solr/security.json [-z :[/]] - Start Solr in cloud mode -- ./bin/solr -c - Upload the provided configuration - ./bin/solr zk upconfig -z :[/] -n conf_en -d /path/to/folder/conf/ - Create 2 collections using the uploaded configuration -- test1, test2 - Create an alias grouping the 2 collections -- test = test1, test2 - Query (/select?q=*:*) one collection -- results in successful Solr response - Query the alias (/select?q=*:*) -- results in HTTP 401 was: The issue was first noticed on an instance of Solr 8.5.0, after securing Solr with security.json. Searching on a single collection returns the expected results, but searching on an alias returns HTTP 401. Note that this issue is not reproduced when the collections are created using the _default configuration. The attached patch includes a unit test that reproduces the issue. The unit test is added to the test class that was originally part of the patch to fix SOLR-13510. I also attach: - our product-specific Solr configuration, modified to remove irrelevant plugins and fields - security.json with user 'admin' (pwd 'admin') -- Note that forwardCredentials true or false does not modify the behavior To test with this configuration: - Download and unzip Solr 8.5.0 - Modify ./bin/solr.in.sh : -- ZK_HOST (optional) -- SOLR_AUTH_TYPE="basic" -- SOLR_AUTHENTICATION_OPTS="-Dbasicauth=admin:admin" - Upload security.json into Zookeeper -- ./bin/solr zk cp file:/path/to/security.json zk:/path/to/solr/security.json [-z :[/]] - Start Solr in cloud mode -- ./bin/solr -c - Upload the provided configuration - ./bin/solr zk upconfig -z :[/] -n conf_en -d /path/to/folder/conf/ - Create 2 collections using the uploaded configuration -- test1, test2 - Create an alias grouping the 2 collections -- test = test1, test2 - Query (/select?q=*:*) one collection -- results in successful Solr response - Query the alias (/select?q=*:*) -- results in HTTP 401 > HTTP 401 when searching on alias in secured Solr > > > Key: SOLR-14569 > URL: https://issues.apache.org/jira/browse/SOLR-14569 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Authentication >Affects Versions: master (9.0), 8.5 > Environment: Unit test on master branch (9x) built on Windows 10 with > Java 11 > Solr 8.5.0 instance running on CentOS 7.7 with Java 11 >Reporter: Isabelle Giguere >Priority: Major > Attachments: SOLR-14569.patch > > > The issue was first noticed on an instance of Solr 8.5.0, after securing Solr > with security.json. > Searching on a single collection returns the expected results, but searching > on an alias returns HTTP 401. > Note that this issue is not reproduced when the collections are created using > the _default configuration. > The attached patch includes a unit test that reproduces the issue. Patch > applies on master branch (9x). > The unit test is added to the test class that was originally part of the > patch to fix SOLR-13510. > I also attach: > - our product-specific Solr configuration, modified to remove irrelevant > plugins and fields > - security.json with user 'admin' (pwd 'admin') > -- Note that forwardCredentials true or false does not modify the behavior > To test with this configuration: > - Download and unzip Solr 8.5.0 > - Modify ./bin/solr.in.sh : > -- ZK_HOST (optional) > -- SOLR_AUTH_TYPE="basic" > -- SOLR_AUTHENTICATION_OPTS="-Dbasicauth=admin:admin" > - Upload security.json into Zookeeper > -- ./bin/solr zk cp file:/path/to/security.json > zk:/path/to/solr/security.json
[jira] [Updated] (SOLR-14569) HTTP 401 when searching on alias in secured Solr
[ https://issues.apache.org/jira/browse/SOLR-14569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Isabelle Giguere updated SOLR-14569: Attachment: SOLR-14569.patch > HTTP 401 when searching on alias in secured Solr > > > Key: SOLR-14569 > URL: https://issues.apache.org/jira/browse/SOLR-14569 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Authentication >Affects Versions: master (9.0), 8.5 > Environment: Unit test on master branch (9x) built on Windows 10 with > Java 11 > Solr 8.5.0 instance running on CentOS 7.7 with Java 11 >Reporter: Isabelle Giguere >Priority: Major > Attachments: SOLR-14569.patch > > > The issue was first noticed on an instance of Solr 8.5.0, after securing Solr > with security.json. > Searching on a single collection returns the expected results, but searching > on an alias returns HTTP 401. > Note that this issue is not reproduced when the collections are created using > the _default configuration. > The attached patch includes a unit test that reproduces the issue. The unit > test is added to the test class that was originally part of the patch to fix > SOLR-13510. > I also attach: > - our product-specific Solr configuration, modified to remove irrelevant > plugins and fields > - security.json with user 'admin' (pwd 'admin') > -- Note that forwardCredentials true or false does not modify the behavior > To test with this configuration: > - Download and unzip Solr 8.5.0 > - Modify ./bin/solr.in.sh : > -- ZK_HOST (optional) > -- SOLR_AUTH_TYPE="basic" > -- SOLR_AUTHENTICATION_OPTS="-Dbasicauth=admin:admin" > - Upload security.json into Zookeeper > -- ./bin/solr zk cp file:/path/to/security.json > zk:/path/to/solr/security.json [-z :[/]] > - Start Solr in cloud mode > -- ./bin/solr -c > - Upload the provided configuration > - ./bin/solr zk upconfig -z :[/] -n conf_en -d > /path/to/folder/conf/ > - Create 2 collections using the uploaded configuration > -- test1, test2 > - Create an alias grouping the 2 collections > -- test = test1, test2 > - Query (/select?q=*:*) one collection > -- results in successful Solr response > - Query the alias (/select?q=*:*) > -- results in HTTP 401 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14569) HTTP 401 when searching on alias in secured Solr
[ https://issues.apache.org/jira/browse/SOLR-14569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Isabelle Giguere updated SOLR-14569: Attachment: security.json > HTTP 401 when searching on alias in secured Solr > > > Key: SOLR-14569 > URL: https://issues.apache.org/jira/browse/SOLR-14569 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Authentication >Affects Versions: master (9.0), 8.5 > Environment: Unit test on master branch (9x) built on Windows 10 with > Java 11 > Solr 8.5.0 instance running on CentOS 7.7 with Java 11 >Reporter: Isabelle Giguere >Priority: Major > Attachments: SOLR-14569.patch, security.json > > > The issue was first noticed on an instance of Solr 8.5.0, after securing Solr > with security.json. > Searching on a single collection returns the expected results, but searching > on an alias returns HTTP 401. > Note that this issue is not reproduced when the collections are created using > the _default configuration. > The attached patch includes a unit test that reproduces the issue. Patch > applies on master branch (9x). > The unit test is added to the test class that was originally part of the > patch to fix SOLR-13510. > I also attach: > - our product-specific Solr configuration, modified to remove irrelevant > plugins and fields > - security.json with user 'admin' (pwd 'admin') > -- Note that forwardCredentials true or false does not modify the behavior > To test with this configuration: > - Download and unzip Solr 8.5.0 > - Modify ./bin/solr.in.sh : > -- ZK_HOST (optional) > -- SOLR_AUTH_TYPE="basic" > -- SOLR_AUTHENTICATION_OPTS="-Dbasicauth=admin:admin" > - Upload security.json into Zookeeper > -- ./bin/solr zk cp file:/path/to/security.json > zk:/path/to/solr/security.json [-z :[/]] > - Start Solr in cloud mode > -- ./bin/solr -c > - Upload the provided configuration > - ./bin/solr zk upconfig -z :[/] -n conf_en -d > /path/to/folder/conf/ > - Create 2 collections using the uploaded configuration > -- test1, test2 > - Create an alias grouping the 2 collections > -- test = test1, test2 > - Query (/select?q=*:*) one collection > -- results in successful Solr response > - Query the alias (/select?q=*:*) > -- results in HTTP 401 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14569) HTTP 401 when searching on alias in secured Solr
[ https://issues.apache.org/jira/browse/SOLR-14569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Isabelle Giguere updated SOLR-14569: Attachment: solr_conf.zip > HTTP 401 when searching on alias in secured Solr > > > Key: SOLR-14569 > URL: https://issues.apache.org/jira/browse/SOLR-14569 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Authentication >Affects Versions: master (9.0), 8.5 > Environment: Unit test on master branch (9x) built on Windows 10 with > Java 11 > Solr 8.5.0 instance running on CentOS 7.7 with Java 11 >Reporter: Isabelle Giguere >Priority: Major > Attachments: SOLR-14569.patch, security.json, solr_conf.zip > > > The issue was first noticed on an instance of Solr 8.5.0, after securing Solr > with security.json. > Searching on a single collection returns the expected results, but searching > on an alias returns HTTP 401. > Note that this issue is not reproduced when the collections are created using > the _default configuration. > The attached patch includes a unit test that reproduces the issue. Patch > applies on master branch (9x). > The unit test is added to the test class that was originally part of the > patch to fix SOLR-13510. > I also attach: > - our product-specific Solr configuration, modified to remove irrelevant > plugins and fields > - security.json with user 'admin' (pwd 'admin') > -- Note that forwardCredentials true or false does not modify the behavior > To test with this configuration: > - Download and unzip Solr 8.5.0 > - Modify ./bin/solr.in.sh : > -- ZK_HOST (optional) > -- SOLR_AUTH_TYPE="basic" > -- SOLR_AUTHENTICATION_OPTS="-Dbasicauth=admin:admin" > - Upload security.json into Zookeeper > -- ./bin/solr zk cp file:/path/to/security.json > zk:/path/to/solr/security.json [-z :[/]] > - Start Solr in cloud mode > -- ./bin/solr -c > - Upload the provided configuration > - ./bin/solr zk upconfig -z :[/] -n conf_en -d > /path/to/folder/conf/ > - Create 2 collections using the uploaded configuration > -- test1, test2 > - Create an alias grouping the 2 collections > -- test = test1, test2 > - Query (/select?q=*:*) one collection > -- results in successful Solr response > - Query the alias (/select?q=*:*) > -- results in HTTP 401 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14569) HTTP 401 when searching on alias in secured Solr
[ https://issues.apache.org/jira/browse/SOLR-14569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Isabelle Giguere updated SOLR-14569: Description: The issue was first noticed on an instance of Solr 8.5.0, after securing Solr with security.json. Searching on a single collection returns the expected results, but searching on an alias returns HTTP 401. *Note that this issue is not reproduced when the collections are created using the _default configuration.* The attached patch includes a unit test that reproduces the issue. *Patch applies on master branch (9x)*: Do not include in the regular build ! The test is failing to illustrate this issue. The unit test is added to the test class that was originally part of the patch to fix SOLR-13510. I also attach: - our product-specific Solr configuration, modified to remove irrelevant plugins and fields - security.json with user 'admin' (pwd 'admin') -- Note that forwardCredentials true or false does not modify the behavior To test with this configuration: - Download and unzip Solr 8.5.0 - Modify ./bin/solr.in.sh : -- ZK_HOST (optional) -- SOLR_AUTH_TYPE="basic" -- SOLR_AUTHENTICATION_OPTS="-Dbasicauth=admin:admin" - Upload security.json into Zookeeper -- ./bin/solr zk cp file:/path/to/security.json zk:/path/to/solr/security.json [-z :[/]] - Start Solr in cloud mode -- ./bin/solr -c - Upload the provided configuration - ./bin/solr zk upconfig -z :[/] -n conf_en -d /path/to/folder/conf/ - Create 2 collections using the uploaded configuration -- test1, test2 - Create an alias grouping the 2 collections -- test = test1, test2 - Query (/select?q=*:*) one collection -- results in successful Solr response - Query the alias (/select?q=*:*) -- results in HTTP 401 was: The issue was first noticed on an instance of Solr 8.5.0, after securing Solr with security.json. Searching on a single collection returns the expected results, but searching on an alias returns HTTP 401. Note that this issue is not reproduced when the collections are created using the _default configuration. The attached patch includes a unit test that reproduces the issue. Patch applies on master branch (9x). The unit test is added to the test class that was originally part of the patch to fix SOLR-13510. I also attach: - our product-specific Solr configuration, modified to remove irrelevant plugins and fields - security.json with user 'admin' (pwd 'admin') -- Note that forwardCredentials true or false does not modify the behavior To test with this configuration: - Download and unzip Solr 8.5.0 - Modify ./bin/solr.in.sh : -- ZK_HOST (optional) -- SOLR_AUTH_TYPE="basic" -- SOLR_AUTHENTICATION_OPTS="-Dbasicauth=admin:admin" - Upload security.json into Zookeeper -- ./bin/solr zk cp file:/path/to/security.json zk:/path/to/solr/security.json [-z :[/]] - Start Solr in cloud mode -- ./bin/solr -c - Upload the provided configuration - ./bin/solr zk upconfig -z :[/] -n conf_en -d /path/to/folder/conf/ - Create 2 collections using the uploaded configuration -- test1, test2 - Create an alias grouping the 2 collections -- test = test1, test2 - Query (/select?q=*:*) one collection -- results in successful Solr response - Query the alias (/select?q=*:*) -- results in HTTP 401 > HTTP 401 when searching on alias in secured Solr > > > Key: SOLR-14569 > URL: https://issues.apache.org/jira/browse/SOLR-14569 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Authentication >Affects Versions: master (9.0), 8.5 > Environment: Unit test on master branch (9x) built on Windows 10 with > Java 11 > Solr 8.5.0 instance running on CentOS 7.7 with Java 11 >Reporter: Isabelle Giguere >Priority: Major > Attachments: SOLR-14569.patch, security.json, solr_conf.zip > > > The issue was first noticed on an instance of Solr 8.5.0, after securing Solr > with security.json. > Searching on a single collection returns the expected results, but searching > on an alias returns HTTP 401. > *Note that this issue is not reproduced when the collections are created > using the _default configuration.* > The attached patch includes a unit test that reproduces the issue. > *Patch applies on master branch (9x)*: Do not include in the regular build ! > The test is failing to illustrate this issue. > The unit test is added to the test class that was originally part of the > patch to fix SOLR-13510. > I also attach: > - our product-specific Solr configuration, modified to remove irrelevant > plugins and fields > - security.json with user 'admin' (pwd 'admin') > -- Note that forwardCredentials true or false does not modify the behavior > To test with this configuration: > - Download and unzip Solr 8.5.0 > - Modify ./bin/s
[jira] [Updated] (SOLR-14569) HTTP 401 when searching on alias in secured Solr
[ https://issues.apache.org/jira/browse/SOLR-14569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Isabelle Giguere updated SOLR-14569: Description: The issue was first noticed on an instance of Solr 8.5.0, after securing Solr with security.json. Searching on a single collection returns the expected results, but searching on an alias returns HTTP 401. *Note that this issue is not reproduced when the collections are created using the _default configuration.* The attached patch includes a unit test that reproduces the issue. *Patch applies on master branch (9x)*: Do not include in the regular build ! The test is failing to illustrate this issue. The unit test is added to the test class that was originally part of the patch to fix SOLR-13510. I also attach: - our product-specific Solr configuration, modified to remove irrelevant plugins and fields - security.json with user 'admin' (pwd 'admin') -- Note that forwardCredentials true or false does not modify the behavior To test with this configuration: - Download and unzip Solr 8.5.0 - Modify ./bin/solr.in.sh : -- ZK_HOST (optional) -- SOLR_AUTH_TYPE="basic" -- SOLR_AUTHENTICATION_OPTS="-Dbasicauth=admin:admin" - Upload security.json into Zookeeper -- ./bin/solr zk cp file:/path/to/security.json zk:/path/to/solr/security.json [-z :[/]] - Start Solr in cloud mode -- ./bin/solr -c - Upload the provided configuration - ./bin/solr zk upconfig -z :[/] -n conf_en -d /path/to/folder/conf/ - Create 2 collections using the uploaded configuration -- test1, test2 - Create an alias grouping the 2 collections -- test = test1, test2 - Query (/select?q=*:*) one collection -- results in successful Solr response - Query the alias (/select?q=*:*) -- results in HTTP 401 There is no need to add documents to observe the issue. was: The issue was first noticed on an instance of Solr 8.5.0, after securing Solr with security.json. Searching on a single collection returns the expected results, but searching on an alias returns HTTP 401. *Note that this issue is not reproduced when the collections are created using the _default configuration.* The attached patch includes a unit test that reproduces the issue. *Patch applies on master branch (9x)*: Do not include in the regular build ! The test is failing to illustrate this issue. The unit test is added to the test class that was originally part of the patch to fix SOLR-13510. I also attach: - our product-specific Solr configuration, modified to remove irrelevant plugins and fields - security.json with user 'admin' (pwd 'admin') -- Note that forwardCredentials true or false does not modify the behavior To test with this configuration: - Download and unzip Solr 8.5.0 - Modify ./bin/solr.in.sh : -- ZK_HOST (optional) -- SOLR_AUTH_TYPE="basic" -- SOLR_AUTHENTICATION_OPTS="-Dbasicauth=admin:admin" - Upload security.json into Zookeeper -- ./bin/solr zk cp file:/path/to/security.json zk:/path/to/solr/security.json [-z :[/]] - Start Solr in cloud mode -- ./bin/solr -c - Upload the provided configuration - ./bin/solr zk upconfig -z :[/] -n conf_en -d /path/to/folder/conf/ - Create 2 collections using the uploaded configuration -- test1, test2 - Create an alias grouping the 2 collections -- test = test1, test2 - Query (/select?q=*:*) one collection -- results in successful Solr response - Query the alias (/select?q=*:*) -- results in HTTP 401 > HTTP 401 when searching on alias in secured Solr > > > Key: SOLR-14569 > URL: https://issues.apache.org/jira/browse/SOLR-14569 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Authentication >Affects Versions: master (9.0), 8.5 > Environment: Unit test on master branch (9x) built on Windows 10 with > Java 11 > Solr 8.5.0 instance running on CentOS 7.7 with Java 11 >Reporter: Isabelle Giguere >Priority: Major > Attachments: SOLR-14569.patch, security.json, solr_conf.zip > > > The issue was first noticed on an instance of Solr 8.5.0, after securing Solr > with security.json. > Searching on a single collection returns the expected results, but searching > on an alias returns HTTP 401. > *Note that this issue is not reproduced when the collections are created > using the _default configuration.* > The attached patch includes a unit test that reproduces the issue. > *Patch applies on master branch (9x)*: Do not include in the regular build ! > The test is failing to illustrate this issue. > The unit test is added to the test class that was originally part of the > patch to fix SOLR-13510. > I also attach: > - our product-specific Solr configuration, modified to remove irrelevant > plugins and fields > - security.json with user 'admin' (pwd 'admin') > -- Note that forwar
[jira] [Updated] (SOLR-14569) HTTP 401 when searching on alias in secured Solr
[ https://issues.apache.org/jira/browse/SOLR-14569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Isabelle Giguere updated SOLR-14569: Description: The issue was first noticed on an instance of Solr 8.5.0, after securing Solr with security.json. Searching on a single collection returns the expected results, but searching on an alias returns HTTP 401. *Note that this issue is not reproduced when the collections are created using the _default configuration.* The attached patch includes a unit test that reproduces the issue. *Patch applies on master branch (9x)*: Do not include in the regular build ! The test is failing to illustrate this issue. The unit test is added to the test class that was originally part of the patch to fix SOLR-13510. I also attach: - our product-specific Solr configuration, modified to remove irrelevant plugins and fields - security.json with user 'admin' (pwd 'admin') -- Note that forwardCredentials true or false does not modify the behavior To test with this configuration: - Download and unzip Solr 8.5.0 - Modify ./bin/solr.in.sh : -- ZK_HOST (optional) -- SOLR_AUTH_TYPE="basic" -- SOLR_AUTHENTICATION_OPTS="-Dbasicauth=admin:admin" - Upload security.json into Zookeeper -- ./bin/solr zk cp file:/path/to/security.json zk:/path/to/solr/security.json [-z :[/]] - Start Solr in cloud mode -- ./bin/solr -c - Upload the provided configuration - ./bin/solr zk upconfig -z :[/] -n conf_en -d /path/to/folder/conf/ - Create 2 collections using the uploaded configuration -- test1, test2 - Create an alias grouping the 2 collections -- test = test1, test2 - Query (/select?q=\*:\*) one collection -- results in successful Solr response - Query the alias (/select?q=\*:\*) -- results in HTTP 401 There is no need to add documents to observe the issue. was: The issue was first noticed on an instance of Solr 8.5.0, after securing Solr with security.json. Searching on a single collection returns the expected results, but searching on an alias returns HTTP 401. *Note that this issue is not reproduced when the collections are created using the _default configuration.* The attached patch includes a unit test that reproduces the issue. *Patch applies on master branch (9x)*: Do not include in the regular build ! The test is failing to illustrate this issue. The unit test is added to the test class that was originally part of the patch to fix SOLR-13510. I also attach: - our product-specific Solr configuration, modified to remove irrelevant plugins and fields - security.json with user 'admin' (pwd 'admin') -- Note that forwardCredentials true or false does not modify the behavior To test with this configuration: - Download and unzip Solr 8.5.0 - Modify ./bin/solr.in.sh : -- ZK_HOST (optional) -- SOLR_AUTH_TYPE="basic" -- SOLR_AUTHENTICATION_OPTS="-Dbasicauth=admin:admin" - Upload security.json into Zookeeper -- ./bin/solr zk cp file:/path/to/security.json zk:/path/to/solr/security.json [-z :[/]] - Start Solr in cloud mode -- ./bin/solr -c - Upload the provided configuration - ./bin/solr zk upconfig -z :[/] -n conf_en -d /path/to/folder/conf/ - Create 2 collections using the uploaded configuration -- test1, test2 - Create an alias grouping the 2 collections -- test = test1, test2 - Query (/select?q=*:*) one collection -- results in successful Solr response - Query the alias (/select?q=*:*) -- results in HTTP 401 There is no need to add documents to observe the issue. > HTTP 401 when searching on alias in secured Solr > > > Key: SOLR-14569 > URL: https://issues.apache.org/jira/browse/SOLR-14569 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Authentication >Affects Versions: master (9.0), 8.5 > Environment: Unit test on master branch (9x) built on Windows 10 with > Java 11 > Solr 8.5.0 instance running on CentOS 7.7 with Java 11 >Reporter: Isabelle Giguere >Priority: Major > Attachments: SOLR-14569.patch, security.json, solr_conf.zip > > > The issue was first noticed on an instance of Solr 8.5.0, after securing Solr > with security.json. > Searching on a single collection returns the expected results, but searching > on an alias returns HTTP 401. > *Note that this issue is not reproduced when the collections are created > using the _default configuration.* > The attached patch includes a unit test that reproduces the issue. > *Patch applies on master branch (9x)*: Do not include in the regular build ! > The test is failing to illustrate this issue. > The unit test is added to the test class that was originally part of the > patch to fix SOLR-13510. > I also attach: > - our product-specific Solr configuration, modified to remove irrelevant > plugins and fields > - security
[GitHub] [lucene-solr] dsmiley commented on pull request #1514: SOLR-13749: Change cross-collection join query syntax to {!join method=crossCollection ...}
dsmiley commented on pull request #1514: URL: https://github.com/apache/lucene-solr/pull/1514#issuecomment-643826837 @danmfox Still missing Ref Guide changes to account for the refactor This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9404) simplify checksum calculation of ByteBuffersIndexOutput
Robert Muir created LUCENE-9404: --- Summary: simplify checksum calculation of ByteBuffersIndexOutput Key: LUCENE-9404 URL: https://issues.apache.org/jira/browse/LUCENE-9404 Project: Lucene - Core Issue Type: Improvement Reporter: Robert Muir Attachments: LUCENE-9404.patch I think this class can avoid its current logic/copying and just call CRC32.update(ByteBuffer) which is optimized for both array and direct buffers? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9404) simplify checksum calculation of ByteBuffersIndexOutput
[ https://issues.apache.org/jira/browse/LUCENE-9404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-9404: Attachment: LUCENE-9404.patch > simplify checksum calculation of ByteBuffersIndexOutput > --- > > Key: LUCENE-9404 > URL: https://issues.apache.org/jira/browse/LUCENE-9404 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Robert Muir >Priority: Major > Attachments: LUCENE-9404.patch > > > I think this class can avoid its current logic/copying and just call > CRC32.update(ByteBuffer) which is optimized for both array and direct buffers? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9404) simplify checksum calculation of ByteBuffersIndexOutput
[ https://issues.apache.org/jira/browse/LUCENE-9404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135303#comment-17135303 ] Robert Muir commented on LUCENE-9404: - I thought about this when looking at benchmark results from https://github.com/benalexau/hash-bench and playing with LUCENE-9403 The crc code screams for the different cases, and this JDK method already has the .hasArray etc logic and optimizations. > simplify checksum calculation of ByteBuffersIndexOutput > --- > > Key: LUCENE-9404 > URL: https://issues.apache.org/jira/browse/LUCENE-9404 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Robert Muir >Priority: Major > Attachments: LUCENE-9404.patch > > > I think this class can avoid its current logic/copying and just call > CRC32.update(ByteBuffer) which is optimized for both array and direct buffers? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on a change in pull request #1572: SOLR-14561 CoreAdminAPI's parameters instanceDir and dataDir are now validated
janhoy commented on a change in pull request #1572:
URL: https://github.com/apache/lucene-solr/pull/1572#discussion_r439883240
##
File path: solr/core/src/test-files/solr/solr-50-all.xml
##
@@ -24,6 +24,7 @@
testConfigSetsHandler
testManagementPath
testSharedLib
+ /tmp,/home/john
Review comment:
It could, but to make it convenient to set it as a Java Sysprop, I kept
it as a simple string. De default `solr.xml` defines it as
`${solr.allowPaths:}` so that `-Dsolr.allowPaths=foo,bar,baz` will work. I'm
not aware of any way to specify a native array as Java Sysprop?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on a change in pull request #1572: SOLR-14561 CoreAdminAPI's parameters instanceDir and dataDir are now validated
janhoy commented on a change in pull request #1572:
URL: https://github.com/apache/lucene-solr/pull/1572#discussion_r439884150
##
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##
@@ -338,6 +339,19 @@ public CoreContainer(NodeConfig config, CoresLocator
locator, boolean asyncSolrC
ExecutorUtil.newMDCAwareCachedThreadPool(
cfg.getReplayUpdatesThreads(),
new SolrNamedThreadFactory("replayUpdatesExecutor")));
+
+this.allowPaths = new java.util.HashSet<>();
+this.allowPaths.add(cfg.getSolrHome().toAbsolutePath());
Review comment:
Ok, simplified it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #1572: SOLR-14561 CoreAdminAPI's parameters instanceDir and dataDir are now validated
dsmiley commented on a change in pull request #1572: URL: https://github.com/apache/lucene-solr/pull/1572#discussion_r439936737 ## File path: solr/core/src/test-files/solr/solr-50-all.xml ## @@ -24,6 +24,7 @@ testConfigSetsHandler testManagementPath testSharedLib + /tmp,/home/john Review comment: ok... I suppose both could be supported provided that the consumer checked with the aid of some new utility method that doesn't exist like `Utils.toListOfString(Object, char separator) : String` but that might be a bit much. I suspect we have other places where we trivially use a delimited string and fail to leverage NamedList. Your call; no strong opinion here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9404) simplify checksum calculation of ByteBuffersIndexOutput
[ https://issues.apache.org/jira/browse/LUCENE-9404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135457#comment-17135457 ] Dawid Weiss commented on LUCENE-9404: - I remember clearly this had to do with update(ByteBuffer) version being slower in performance tests... but I look at the sources of OpenJDK now and see it's implemented in pretty much the same way [1] since Java 9 (me looking completely puzzled). The choice of built-in Checksum class was arbitrary (felt like the simplest choice without bringing in other dependencies or copying code). Please feel free to commit it in, strange. [1] https://docs.oracle.com/javase/9/docs/api/java/util/zip/Checksum.html#update-java.nio.ByteBuffer- > simplify checksum calculation of ByteBuffersIndexOutput > --- > > Key: LUCENE-9404 > URL: https://issues.apache.org/jira/browse/LUCENE-9404 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Robert Muir >Priority: Major > Attachments: LUCENE-9404.patch > > > I think this class can avoid its current logic/copying and just call > CRC32.update(ByteBuffer) which is optimized for both array and direct buffers? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9404) simplify checksum calculation of ByteBuffersIndexOutput
[ https://issues.apache.org/jira/browse/LUCENE-9404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135473#comment-17135473 ] Uwe Schindler commented on LUCENE-9404: --- [~dweiss]: The Java 9 version is completely new, so in branch_8x we were not able to use it. Maybe that was the reason, we just forgot to update it for Lucene 9. > simplify checksum calculation of ByteBuffersIndexOutput > --- > > Key: LUCENE-9404 > URL: https://issues.apache.org/jira/browse/LUCENE-9404 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Robert Muir >Priority: Major > Attachments: LUCENE-9404.patch > > > I think this class can avoid its current logic/copying and just call > CRC32.update(ByteBuffer) which is optimized for both array and direct buffers? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9404) simplify checksum calculation of ByteBuffersIndexOutput
[ https://issues.apache.org/jira/browse/LUCENE-9404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135495#comment-17135495 ] Dawid Weiss commented on LUCENE-9404: - Could be, I can't remember. Let's just switch. > simplify checksum calculation of ByteBuffersIndexOutput > --- > > Key: LUCENE-9404 > URL: https://issues.apache.org/jira/browse/LUCENE-9404 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Robert Muir >Priority: Major > Attachments: LUCENE-9404.patch > > > I think this class can avoid its current logic/copying and just call > CRC32.update(ByteBuffer) which is optimized for both array and direct buffers? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
