[jira] [Commented] (LUCENE-10424) Optimize the "everything matches" case for count query in PointRangeQuery
[ https://issues.apache.org/jira/browse/LUCENE-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494445#comment-17494445 ] Lu Xugang commented on LUCENE-10424: Or maybe such additional consideration should be in PointRangeQuery#rewrite() while PointRangeQuery could be rewrited to MatchAllDocsQuery? > Optimize the "everything matches" case for count query in PointRangeQuery > - > > Key: LUCENE-10424 > URL: https://issues.apache.org/jira/browse/LUCENE-10424 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: 9.1 >Reporter: Lu Xugang >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > In Implement of Weight#count in PointRangeQuery, Whether additional > consideration is needed that when PointValues#getDocCount() == > IndexReader#maxDoc() and the range's lower bound is less that the field's min > value and the range's upper bound is greater than the field's max value, then > return reader.maxDoc() directly? -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] (LUCENE-10424) Optimize the "everything matches" case for count query in PointRangeQuery
[ https://issues.apache.org/jira/browse/LUCENE-10424 ] Lu Xugang deleted comment on LUCENE-10424: was (Author: chrislu): Or maybe such additional consideration should be in PointRangeQuery#rewrite() while PointRangeQuery could be rewrited to MatchAllDocsQuery? > Optimize the "everything matches" case for count query in PointRangeQuery > - > > Key: LUCENE-10424 > URL: https://issues.apache.org/jira/browse/LUCENE-10424 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: 9.1 >Reporter: Lu Xugang >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > In Implement of Weight#count in PointRangeQuery, Whether additional > consideration is needed that when PointValues#getDocCount() == > IndexReader#maxDoc() and the range's lower bound is less that the field's min > value and the range's upper bound is greater than the field's max value, then > return reader.maxDoc() directly? -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mayya-sharipova commented on a change in pull request #690: LUCENE-10408: Fix vector values iteration bug
mayya-sharipova commented on a change in pull request #690:
URL: https://github.com/apache/lucene/pull/690#discussion_r809803977
##
File path:
lucene/core/src/java/org/apache/lucene/codecs/lucene91/Lucene91HnswVectorsReader.java
##
@@ -475,11 +475,10 @@ public int advance(int target) {
}
}
- assert ord <= size;
- if (ord == size) {
-doc = NO_MORE_DOCS;
- } else {
+ if (ord < size) {
Review comment:
Great notice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mogui commented on pull request #679: Monitor Improvements LUCENE-10422
mogui commented on pull request #679: URL: https://github.com/apache/lucene/pull/679#issuecomment-1044239703 OK I've moved things around, making QueryIndex an abstract class sharing common code for both implementations ReadonlyQueryIndex have the same logic of in-memory laoding of parsed queries, but without locking handling and without purge executor. I've made it usable by enabling purging cache so it can be done externally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mogui edited a comment on pull request #679: Monitor Improvements LUCENE-10422
mogui edited a comment on pull request #679: URL: https://github.com/apache/lucene/pull/679#issuecomment-1044239703 @romseygeek OK I've moved things around, making QueryIndex an abstract class sharing common code for both implementations ReadonlyQueryIndex have the same logic of in-memory laoding of parsed queries, but without locking handling and without purge executor. I've made it usable by enabling purging cache so it can be done externally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] iverase commented on a change in pull request #691: LUCENE-10424: Optimize the "everything matches" case for count query in PointRangeQuery
iverase commented on a change in pull request #691:
URL: https://github.com/apache/lucene/pull/691#discussion_r809843886
##
File path: lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java
##
@@ -391,6 +391,14 @@ public int count(LeafReaderContext context) throws
IOException {
&& numDims == 1
&& values.getDocCount() == values.size()) {
// if all documents have at-most one point
+ if (values.getDocCount() == reader.maxDoc()) {
+final byte[] fieldPackedLower = values.getMinPackedValue();
Review comment:
Is this check really needed? I guess this will work:
```
if (relate(values.getMinPackedValue, values.getMaxPackedValue()) ==
Relation.CELL_WITHIN) {
return values.getDocCount()
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] LuXugang commented on a change in pull request #691: LUCENE-10424: Optimize the "everything matches" case for count query in PointRangeQuery
LuXugang commented on a change in pull request #691:
URL: https://github.com/apache/lucene/pull/691#discussion_r809895342
##
File path: lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java
##
@@ -391,6 +391,14 @@ public int count(LeafReaderContext context) throws
IOException {
&& numDims == 1
&& values.getDocCount() == values.size()) {
// if all documents have at-most one point
+ if (values.getDocCount() == reader.maxDoc()) {
+final byte[] fieldPackedLower = values.getMinPackedValue();
Review comment:
Simply and much more thoughtful！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] iverase opened a new pull request #692: LUCENE-10311: Different implementations of DocIdSetBuilder for points and terms
iverase opened a new pull request #692:
URL: https://github.com/apache/lucene/pull/692
The idea in this PR is the following:
1) Extract the sparse structure to collect docIds from docIdSetBuilder in a
class called Buffers so it can be used by the different implementations.
2) Add a new DocIdSetBuilder implementation for points called
`PointsDocIdSetBuilder` with the following API:
```
/**
* Utility class to efficiently add many docs in one go.
*
* @see PointsDocIdSetBuilder#grow
*/
public abstract static class BulkAdder {
public abstract void add(int doc);
public void add(DocIdSetIterator iterator) throws IOException {
int docID;
while ((docID = iterator.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) {
add(docID);
}
}
}
/**
* Reserve space and return a {@link BulkAdder} object that can be used to
visit up to {@code
* numPoints} points.
*/
public BulkAdder grow(long numPoints);
```
3) DocIdSet builder API looks like that after the extraction of the points
API:
```
/**
* Add the content of the provided {@link DocIdSetIterator} to this
builder. NOTE: if you need to
* build a {@link DocIdSet} out of a single {@link DocIdSetIterator}, you
should rather use {@link
* RoaringDocIdSet.Builder}.
*/
public void add(DocIdSetIterator iter) throws IOException;
/** Add a single document to this builder. */
public void add(int doc);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] iverase commented on a change in pull request #692: LUCENE-10311: Different implementations of DocIdSetBuilder for points and terms
iverase commented on a change in pull request #692:
URL: https://github.com/apache/lucene/pull/692#discussion_r809966771
##
File path: lucene/core/src/java/org/apache/lucene/util/DocIdSetBuilder.java
##
@@ -167,99 +85,44 @@ public void add(DocIdSetIterator iter) throws IOException {
return;
}
Review comment:
I haven't change the implementation here but this looks buggy for me. If
the bitSet is not null we don't update counter any more in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] iverase commented on pull request #691: LUCENE-10424: Optimize the "everything matches" case for count query in PointRangeQuery
iverase commented on pull request #691: URL: https://github.com/apache/lucene/pull/691#issuecomment-1044480879 Could you add an entry in CHANGES.txt? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] LuXugang commented on pull request #691: LUCENE-10424: Optimize the "everything matches" case for count query in PointRangeQuery
LuXugang commented on pull request #691: URL: https://github.com/apache/lucene/pull/691#issuecomment-1044529977 > Could you add an entry in CHANGES.txt? > Could you add an entry in CHANGES.txt? DONE -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-10427) OLAP likewise rollup during segment merge process
Suhan Mao created LUCENE-10427:
--
Summary: OLAP likewise rollup during segment merge process
Key: LUCENE-10427
URL: https://issues.apache.org/jira/browse/LUCENE-10427
Project: Lucene - Core
Issue Type: New Feature
Reporter: Suhan Mao
Currently, many OLAP engines support rollup feature like
clickhouse(AggregateMergeTree)/druid.
Rollup definition: [https://athena.ecs.csus.edu/~mei/olap/OLAPoperations.php]
One of the way to do rollup is to merge the same dimension fields and do
sum()/min()/max() operation on metric fields. This can significantly reduce the
size of the data and speed up the query a lot.
*Abstraction of how to do*
# Define rollup logic: which is dimensions and metrics.
# Rollup definition for each metric field: max/min/sum ...
# index sorting should the the same as dimension fields.
# We will do rollup calculation during segment merge just like other OLAP
engine do.
*Assume the scenario*
If we use ES to ingest realtime raw temperature data every minutes of each
sensor device along with many dimension information. User may want to query the
latest data like "what is the max temperature of some device within some/latest
hour" or "what is the max temperature of some city(dimension) within
some/latest hour"
In that way, we can define such fields and rollup definition:
# event_hour(round to hour granularity)
# device_id(dimension)
# city_id(dimension)
# temperature(metrics, max/min rollup logic)
The raw data will periodically be rolled up to the hour granularity during
segment merge process, which should save 60x storage ideally in the end.
For each sensor device, within one hour, 60 rows will be merged into 1 row
ideally.
*How we do rollup in segment merge*
bucket: docs should belong to the same bucket if the dimension values are all
the same.
# For docvalues merge, we send the normal mappedDocId if we encounter a new
bucket in DocIDMerger.
# Since the index sorting fields are the same with dimension fields. if we
encounter more docs in the same bucket, We emit special mappedDocId from
DocIDMerger .
# In DocValuesConsumer.mergeNumericField, if we meet special mappedDocId, we
do a rollup calculation on metric fields and fold the result value to the first
doc in the bucket. The calculation just like a streaming merge sort rollup.
# We discard all the special mappedDocId docs because the metrics is already
folded to the first doc of in the bucket.
# In BKD/posting index structure, we discard all the special mappedDocId docs
and no rollup calculation is needed. It should be simple.
*How to define the logic*
{code:java}
public class RollupMergeConfig {
private List dimensionNames;
private List aggregateFields;
}
public class RollupMergeAggregateField {
private String name;
private RollupMergeAggregateType aggregateType;
}
public enum RollupMergeAggregateType {
COUNT,
SUM,
MIN,
MAX,
CARDINALITY // if data sketch is stored in binary doc values, we can do a
union logic
}{code}
I have written the initial code in a basic level. I can submit the complete PR
if you think this feature is good to try.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10427) OLAP likewise rollup during segment merge process
[
https://issues.apache.org/jira/browse/LUCENE-10427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suhan Mao updated LUCENE-10427:
---
Description:
Currently, many OLAP engines support rollup feature like
clickhouse(AggregateMergeTree)/druid.
Rollup definition: [https://athena.ecs.csus.edu/~mei/olap/OLAPoperations.php]
One of the way to do rollup is to merge the same dimension buckets and do
sum()/min()/max() operation on metric fields during segment compact/merge
process. This can significantly reduce the size of the data and speed up the
query a lot.
*Abstraction of how to do*
# Define rollup logic: which is dimensions and metrics.
# Rollup definition for each metric field: max/min/sum ...
# index sorting should the the same as dimension fields.
# We will do rollup calculation during segment merge just like other OLAP
engine do.
*Assume the scenario*
If we use ES to ingest realtime raw temperature data every minutes of each
sensor device along with many dimension information. User may want to query the
latest data like "what is the max temperature of some device within some/latest
hour" or "what is the max temperature of some city(dimension) within
some/latest hour"
In that way, we can define such fields and rollup definition:
# event_hour(round to hour granularity)
# device_id(dimension)
# city_id(dimension)
# temperature(metrics, max/min rollup logic)
The raw data will periodically be rolled up to the hour granularity during
segment merge process, which should save 60x storage ideally in the end.
For each sensor device, within one hour, 60 rows will be merged into 1 row
ideally.
*How we do rollup in segment merge*
bucket: docs should belong to the same bucket if the dimension values are all
the same.
# For docvalues merge, we send the normal mappedDocId if we encounter a new
bucket in DocIDMerger.
# Since the index sorting fields are the same with dimension fields. if we
encounter more docs in the same bucket, We emit special mappedDocId from
DocIDMerger .
# In DocValuesConsumer.mergeNumericField, if we meet special mappedDocId, we
do a rollup calculation on metric fields and fold the result value to the first
doc in the bucket. The calculation just like a streaming merge sort rollup.
# We discard all the special mappedDocId docs because the metrics is already
folded to the first doc of in the bucket.
# In BKD/posting index structure, we discard all the special mappedDocId docs
and no rollup calculation is needed. It should be simple.
*How to define the logic*
{code:java}
public class RollupMergeConfig {
private List dimensionNames;
private List aggregateFields;
}
public class RollupMergeAggregateField {
private String name;
private RollupMergeAggregateType aggregateType;
}
public enum RollupMergeAggregateType {
COUNT,
SUM,
MIN,
MAX,
CARDINALITY // if data sketch is stored in binary doc values, we can do a
union logic
}{code}
I have written the initial code in a basic level. I can submit the complete PR
if you think this feature is good to try.
was:
Currently, many OLAP engines support rollup feature like
clickhouse(AggregateMergeTree)/druid.
Rollup definition: [https://athena.ecs.csus.edu/~mei/olap/OLAPoperations.php]
One of the way to do rollup is to merge the same dimension fields and do
sum()/min()/max() operation on metric fields. This can significantly reduce the
size of the data and speed up the query a lot.
*Abstraction of how to do*
# Define rollup logic: which is dimensions and metrics.
# Rollup definition for each metric field: max/min/sum ...
# index sorting should the the same as dimension fields.
# We will do rollup calculation during segment merge just like other OLAP
engine do.
*Assume the scenario*
If we use ES to ingest realtime raw temperature data every minutes of each
sensor device along with many dimension information. User may want to query the
latest data like "what is the max temperature of some device within some/latest
hour" or "what is the max temperature of some city(dimension) within
some/latest hour"
In that way, we can define such fields and rollup definition:
# event_hour(round to hour granularity)
# device_id(dimension)
# city_id(dimension)
# temperature(metrics, max/min rollup logic)
The raw data will periodically be rolled up to the hour granularity during
segment merge process, which should save 60x storage ideally in the end.
For each sensor device, within one hour, 60 rows will be merged into 1 row
ideally.
*How we do rollup in segment merge*
bucket: docs should belong to the same bucket if the dimension values are all
the same.
# For docvalues merge, we send the normal mappedDocId if we encounter a new
bucket in DocIDMerger.
# Since the index sorting fields are the same with dimension fields. if we
encounter more docs in the same bucket, We emit special
[jira] [Updated] (LUCENE-10427) OLAP likewise rollup during segment merge process
[
https://issues.apache.org/jira/browse/LUCENE-10427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suhan Mao updated LUCENE-10427:
---
Description:
Currently, many OLAP engines support rollup feature like
clickhouse(AggregateMergeTree)/druid.
Rollup definition: [https://athena.ecs.csus.edu/~mei/olap/OLAPoperations.php]
One of the way to do rollup is to merge the same dimension buckets into one and
do sum()/min()/max() operation on metric fields during segment compact/merge
process. This can significantly reduce the size of the data and speed up the
query a lot.
*Abstraction of how to do*
# Define rollup logic: which is dimensions and metrics.
# Rollup definition for each metric field: max/min/sum ...
# index sorting should the the same as dimension fields.
# We will do rollup calculation during segment merge just like other OLAP
engine do.
*Assume the scenario*
If we use ES to ingest realtime raw temperature data every minutes of each
sensor device along with many dimension information. User may want to query the
latest data like "what is the max temperature of some device within some/latest
hour" or "what is the max temperature of some city(dimension) within
some/latest hour"
In that way, we can define such fields and rollup definition:
# event_hour(round to hour granularity)
# device_id(dimension)
# city_id(dimension)
# temperature(metrics, max/min rollup logic)
The raw data will periodically be rolled up to the hour granularity during
segment merge process, which should save 60x storage ideally in the end.
For each sensor device, within one hour, 60 rows will be merged into 1 row
ideally.
*How we do rollup in segment merge*
bucket: docs should belong to the same bucket if the dimension values are all
the same.
# For docvalues merge, we send the normal mappedDocId if we encounter a new
bucket in DocIDMerger.
# Since the index sorting fields are the same with dimension fields. if we
encounter more docs in the same bucket, We emit special mappedDocId from
DocIDMerger .
# In DocValuesConsumer.mergeNumericField, if we meet special mappedDocId, we
do a rollup calculation on metric fields and fold the result value to the first
doc in the bucket. The calculation just like a streaming merge sort rollup.
# We discard all the special mappedDocId docs because the metrics is already
folded to the first doc of in the bucket.
# In BKD/posting index structure, we discard all the special mappedDocId docs
and no rollup calculation is needed. It should be simple.
*How to define the logic*
{code:java}
public class RollupMergeConfig {
private List dimensionNames;
private List aggregateFields;
}
public class RollupMergeAggregateField {
private String name;
private RollupMergeAggregateType aggregateType;
}
public enum RollupMergeAggregateType {
COUNT,
SUM,
MIN,
MAX,
CARDINALITY // if data sketch is stored in binary doc values, we can do a
union logic
}{code}
I have written the initial code in a basic level. I can submit the complete PR
if you think this feature is good to try.
was:
Currently, many OLAP engines support rollup feature like
clickhouse(AggregateMergeTree)/druid.
Rollup definition: [https://athena.ecs.csus.edu/~mei/olap/OLAPoperations.php]
One of the way to do rollup is to merge the same dimension buckets and do
sum()/min()/max() operation on metric fields during segment compact/merge
process. This can significantly reduce the size of the data and speed up the
query a lot.
*Abstraction of how to do*
# Define rollup logic: which is dimensions and metrics.
# Rollup definition for each metric field: max/min/sum ...
# index sorting should the the same as dimension fields.
# We will do rollup calculation during segment merge just like other OLAP
engine do.
*Assume the scenario*
If we use ES to ingest realtime raw temperature data every minutes of each
sensor device along with many dimension information. User may want to query the
latest data like "what is the max temperature of some device within some/latest
hour" or "what is the max temperature of some city(dimension) within
some/latest hour"
In that way, we can define such fields and rollup definition:
# event_hour(round to hour granularity)
# device_id(dimension)
# city_id(dimension)
# temperature(metrics, max/min rollup logic)
The raw data will periodically be rolled up to the hour granularity during
segment merge process, which should save 60x storage ideally in the end.
For each sensor device, within one hour, 60 rows will be merged into 1 row
ideally.
*How we do rollup in segment merge*
bucket: docs should belong to the same bucket if the dimension values are all
the same.
# For docvalues merge, we send the normal mappedDocId if we encounter a new
bucket in DocIDMerger.
# Since the index sorting fields are the same with dimension fields. if we
encounte
[jira] [Updated] (LUCENE-10427) OLAP likewise rollup during segment merge process
[
https://issues.apache.org/jira/browse/LUCENE-10427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suhan Mao updated LUCENE-10427:
---
Description:
Currently, many OLAP engines support rollup feature like
clickhouse(AggregateMergeTree)/druid.
Rollup definition: [https://athena.ecs.csus.edu/~mei/olap/OLAPoperations.php]
One of the way to do rollup is to merge the same dimension buckets into one and
do sum()/min()/max() operation on metric fields during segment compact/merge
process. This can significantly reduce the size of the data and speed up the
query a lot.
*Abstraction of how to do*
# Define rollup logic: which is dimensions and metrics.
# Rollup definition for each metric field: max/min/sum ...
# index sorting should the the same as dimension fields.
# We will do rollup calculation during segment merge just like other OLAP
engine do.
*Assume the scenario*
We use ES to ingest realtime raw temperature data every minutes of each sensor
device along with many dimension information. User may want to query the data
like "what is the max temperature of some device within some/latest hour" or
"what is the max temperature of some city within some/latest hour"
In that way, we can define such fields and rollup definition:
# event_hour(round to hour granularity)
# device_id(dimension)
# city_id(dimension)
# temperature(metrics, max/min rollup logic)
The raw data will periodically be rolled up to the hour granularity during
segment merge process, which should save 60x storage ideally in the end.
*How we do rollup in segment merge*
bucket: docs should belong to the same bucket if the dimension values are all
the same.
# For docvalues merge, we send the normal mappedDocId if we encounter a new
bucket in DocIDMerger.
# Since the index sorting fields are the same with dimension fields. if we
encounter more docs in the same bucket, We emit special mappedDocId from
DocIDMerger .
# In DocValuesConsumer.mergeNumericField, if we meet special mappedDocId, we
do a rollup calculation on metric fields and fold the result value to the first
doc in the bucket. The calculation just like a streaming merge sort rollup.
# We discard all the special mappedDocId docs because the metrics is already
folded to the first doc of in the bucket.
# In BKD/posting index structure, we discard all the special mappedDocId docs
and no rollup calculation is needed. It should be simple.
*How to define the logic*
{code:java}
public class RollupMergeConfig {
private List dimensionNames;
private List aggregateFields;
}
public class RollupMergeAggregateField {
private String name;
private RollupMergeAggregateType aggregateType;
}
public enum RollupMergeAggregateType {
COUNT,
SUM,
MIN,
MAX,
CARDINALITY // if data sketch is stored in binary doc values, we can do a
union logic
}{code}
I have written the initial code in a basic level. I can submit the complete PR
if you think this feature is good to try.
was:
Currently, many OLAP engines support rollup feature like
clickhouse(AggregateMergeTree)/druid.
Rollup definition: [https://athena.ecs.csus.edu/~mei/olap/OLAPoperations.php]
One of the way to do rollup is to merge the same dimension buckets into one and
do sum()/min()/max() operation on metric fields during segment compact/merge
process. This can significantly reduce the size of the data and speed up the
query a lot.
*Abstraction of how to do*
# Define rollup logic: which is dimensions and metrics.
# Rollup definition for each metric field: max/min/sum ...
# index sorting should the the same as dimension fields.
# We will do rollup calculation during segment merge just like other OLAP
engine do.
*Assume the scenario*
If we use ES to ingest realtime raw temperature data every minutes of each
sensor device along with many dimension information. User may want to query the
latest data like "what is the max temperature of some device within some/latest
hour" or "what is the max temperature of some city(dimension) within
some/latest hour"
In that way, we can define such fields and rollup definition:
# event_hour(round to hour granularity)
# device_id(dimension)
# city_id(dimension)
# temperature(metrics, max/min rollup logic)
The raw data will periodically be rolled up to the hour granularity during
segment merge process, which should save 60x storage ideally in the end.
For each sensor device, within one hour, 60 rows will be merged into 1 row
ideally.
*How we do rollup in segment merge*
bucket: docs should belong to the same bucket if the dimension values are all
the same.
# For docvalues merge, we send the normal mappedDocId if we encounter a new
bucket in DocIDMerger.
# Since the index sorting fields are the same with dimension fields. if we
encounter more docs in the same bucket, We emit special mappedDocId from
DocIDMerger .
# In DocValuesCons
[jira] [Updated] (LUCENE-10427) OLAP likewise rollup during segment merge process
[
https://issues.apache.org/jira/browse/LUCENE-10427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suhan Mao updated LUCENE-10427:
---
Description:
Currently, many OLAP engines support rollup feature like
clickhouse(AggregateMergeTree)/druid.
Rollup definition: [https://athena.ecs.csus.edu/~mei/olap/OLAPoperations.php]
One of the way to do rollup is to merge the same dimension buckets into one and
do sum()/min()/max() operation on metric fields during segment compact/merge
process. This can significantly reduce the size of the data and speed up the
query a lot.
*Abstraction of how to do*
# Define rollup logic: which is dimensions and metrics.
# Rollup definition for each metric field: max/min/sum ...
# index sorting should the the same as dimension fields.
# We will do rollup calculation during segment merge just like other OLAP
engine do.
*Assume the scenario*
We use ES to ingest realtime raw temperature data every minutes of each sensor
device along with many dimension information. User may want to query the data
like "what is the max temperature of some device within some/latest hour" or
"what is the max temperature of some city within some/latest hour"
In that way, we can define such fields and rollup definition:
# event_hour(round to hour granularity)
# device_id(dimension)
# city_id(dimension)
# temperature(metrics, max/min rollup logic)
The raw data will periodically be rolled up to the hour granularity during
segment merge process, which should save 60x storage ideally in the end.
*How we do rollup in segment merge*
bucket: docs should belong to the same bucket if the dimension values are all
the same.
# For docvalues merge, we send the normal mappedDocId if we encounter a new
bucket in DocIDMerger.
# Since the index sorting fields are the same with dimension fields. if we
encounter more docs in the same bucket, We emit special mappedDocId from
DocIDMerger .
# In DocValuesConsumer.mergeNumericField, if we meet special mappedDocId, we
do a rollup calculation on metric fields and fold the result value to the first
doc in the bucket. The calculation just like a streaming merge sort rollup.
# We discard all the special mappedDocId docs because the metrics is already
folded to the first doc of in the bucket.
# In BKD/posting structure, we discard all the special mappedDocId docs and
only place the first doc id within a bucket in the BKD/posting data. It should
be simple.
*How to define the logic*
{code:java}
public class RollupMergeConfig {
private List dimensionNames;
private List aggregateFields;
}
public class RollupMergeAggregateField {
private String name;
private RollupMergeAggregateType aggregateType;
}
public enum RollupMergeAggregateType {
COUNT,
SUM,
MIN,
MAX,
CARDINALITY // if data sketch is stored in binary doc values, we can do a
union logic
}{code}
I have written the initial code in a basic level. I can submit the complete PR
if you think this feature is good to try.
was:
Currently, many OLAP engines support rollup feature like
clickhouse(AggregateMergeTree)/druid.
Rollup definition: [https://athena.ecs.csus.edu/~mei/olap/OLAPoperations.php]
One of the way to do rollup is to merge the same dimension buckets into one and
do sum()/min()/max() operation on metric fields during segment compact/merge
process. This can significantly reduce the size of the data and speed up the
query a lot.
*Abstraction of how to do*
# Define rollup logic: which is dimensions and metrics.
# Rollup definition for each metric field: max/min/sum ...
# index sorting should the the same as dimension fields.
# We will do rollup calculation during segment merge just like other OLAP
engine do.
*Assume the scenario*
We use ES to ingest realtime raw temperature data every minutes of each sensor
device along with many dimension information. User may want to query the data
like "what is the max temperature of some device within some/latest hour" or
"what is the max temperature of some city within some/latest hour"
In that way, we can define such fields and rollup definition:
# event_hour(round to hour granularity)
# device_id(dimension)
# city_id(dimension)
# temperature(metrics, max/min rollup logic)
The raw data will periodically be rolled up to the hour granularity during
segment merge process, which should save 60x storage ideally in the end.
*How we do rollup in segment merge*
bucket: docs should belong to the same bucket if the dimension values are all
the same.
# For docvalues merge, we send the normal mappedDocId if we encounter a new
bucket in DocIDMerger.
# Since the index sorting fields are the same with dimension fields. if we
encounter more docs in the same bucket, We emit special mappedDocId from
DocIDMerger .
# In DocValuesConsumer.mergeNumericField, if we meet special mappedDocId, we
do a rollup calcu
[GitHub] [lucene] jtibshirani merged pull request #690: LUCENE-10408: Fix vector values iteration bug
jtibshirani merged pull request #690: URL: https://github.com/apache/lucene/pull/690 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10408) Better dense encoding of doc Ids in Lucene91HnswVectorsFormat
[ https://issues.apache.org/jira/browse/LUCENE-10408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494799#comment-17494799 ] ASF subversion and git services commented on LUCENE-10408: -- Commit f0d17e94d98730bbe94904321e186c75cb8d845b in lucene's branch refs/heads/main from Julie Tibshirani [ https://gitbox.apache.org/repos/asf?p=lucene.git;h=f0d17e9 ] LUCENE-10408: Fix vector values iteration bug (#690) Now that there is special logic to handle the dense case, we need to adjust some assertions in VectorValues#advance. > Better dense encoding of doc Ids in Lucene91HnswVectorsFormat > - > > Key: LUCENE-10408 > URL: https://issues.apache.org/jira/browse/LUCENE-10408 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Mayya Sharipova >Assignee: Mayya Sharipova >Priority: Minor > Fix For: 9.1 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > Currently we write doc Ids of all documents that have vectors as is. We > should improve their encoding either using delta encoding or bitset. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10408) Better dense encoding of doc Ids in Lucene91HnswVectorsFormat
[ https://issues.apache.org/jira/browse/LUCENE-10408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494801#comment-17494801 ] ASF subversion and git services commented on LUCENE-10408: -- Commit fcc384fdafcbd7fb86bb9a2dff566515895abcff in lucene's branch refs/heads/branch_9x from Julie Tibshirani [ https://gitbox.apache.org/repos/asf?p=lucene.git;h=fcc384f ] LUCENE-10408: Fix vector values iteration bug (#690) Now that there is special logic to handle the dense case, we need to adjust some assertions in VectorValues#advance. > Better dense encoding of doc Ids in Lucene91HnswVectorsFormat > - > > Key: LUCENE-10408 > URL: https://issues.apache.org/jira/browse/LUCENE-10408 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Mayya Sharipova >Assignee: Mayya Sharipova >Priority: Minor > Fix For: 9.1 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > Currently we write doc Ids of all documents that have vectors as is. We > should improve their encoding either using delta encoding or bitset. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-10428) getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge leading to busy threads in infinite loop
Ankit Jain created LUCENE-10428: --- Summary: getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge leading to busy threads in infinite loop Key: LUCENE-10428 URL: https://issues.apache.org/jira/browse/LUCENE-10428 Project: Lucene - Core Issue Type: Bug Components: core/query/scoring, core/search Reporter: Ankit Jain Customers complained about high CPU for Elasticsearch cluster in production. We noticed that few search requests were stuck for long time ``` % curl -s localhost:9200/_cat/tasks?v indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205 AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075 emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270 emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h ``` Flame graphs indicated that CPU time is mostly going into *getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some live JVM debugging found that org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had around 4 million invocations every second Figured out the values of some parameters from live debugging: ``` minScoreSum = 3.5541441 minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) = 3.554144322872162 returnObj scoreSumUpperBound = 3.5541444 Math.ulp(minScoreSum) = 2.3841858E-7 ``` -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10428) getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge leading to busy threads in infinite loop
[
https://issues.apache.org/jira/browse/LUCENE-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Jain updated LUCENE-10428:
Description:
Customers complained about high CPU for Elasticsearch cluster in production. We
noticed that few search requests were stuck for long time
{{{quote}% curl -s localhost:9200/_cat/tasks?v
indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h{quote}}}
Flame graphs indicated that CPU time is mostly going into
*getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some live
JVM debugging found that
org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
around 4 million invocations every second
Figured out the values of some parameters from live debugging:
{{minScoreSum = 3.5541441
minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
3.554144322872162
returnObj scoreSumUpperBound = 3.5541444
Math.ulp(minScoreSum) = 2.3841858E-7}}
was:
Customers complained about high CPU for Elasticsearch cluster in production. We
noticed that few search requests were stuck for long time
```
% curl -s localhost:9200/_cat/tasks?v
indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h
```
Flame graphs indicated that CPU time is mostly going into
*getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some live
JVM debugging found that
org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
around 4 million invocations every second
Figured out the values of some parameters from live debugging:
```
minScoreSum = 3.5541441
minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
3.554144322872162
returnObj scoreSumUpperBound = 3.5541444
Math.ulp(minScoreSum) = 2.3841858E-7
```
> getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge
> leading to busy threads in infinite loop
> -
>
> Key: LUCENE-10428
> URL: https://issues.apache.org/jira/browse/LUCENE-10428
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/query/scoring, core/search
>Reporter: Ankit Jain
>Priority: Major
>
> Customers complained about high CPU for Elasticsearch cluster in production.
> We noticed that few search requests were stuck for long time
> {{{quote}% curl -s localhost:9200/_cat/tasks?v
> indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
> AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
> emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
> emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h{quote}}}
> Flame graphs indicated that CPU time is mostly going into
> *getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some
> live JVM debugging found that
> org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
> around 4 million invocations every second
> Figured out the values of some parameters from live debugging:
> {{minScoreSum = 3.5541441
> minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
> 3.554144322872162
> returnObj scoreSumUpperBound = 3.5541444
> Math.ulp(minScoreSum) = 2.3841858E-7}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10428) getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge leading to busy threads in infinite loop
[
https://issues.apache.org/jira/browse/LUCENE-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Jain updated LUCENE-10428:
Description:
Customers complained about high CPU for Elasticsearch cluster in production. We
noticed that few search requests were stuck for long time
{quote}% curl -s localhost:9200/_cat/tasks?v
indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h{quote}
Flame graphs indicated that CPU time is mostly going into
*getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some live
JVM debugging found that
org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
around 4 million invocations every second
Figured out the values of some parameters from live debugging:
{quote}minScoreSum = 3.5541441
minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
3.554144322872162
returnObj scoreSumUpperBound = 3.5541444
Math.ulp(minScoreSum) = 2.3841858E-7{quote}
was:
Customers complained about high CPU for Elasticsearch cluster in production. We
noticed that few search requests were stuck for long time
{{{quote}% curl -s localhost:9200/_cat/tasks?v
indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h{quote}}}
Flame graphs indicated that CPU time is mostly going into
*getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some live
JVM debugging found that
org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
around 4 million invocations every second
Figured out the values of some parameters from live debugging:
{{minScoreSum = 3.5541441
minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
3.554144322872162
returnObj scoreSumUpperBound = 3.5541444
Math.ulp(minScoreSum) = 2.3841858E-7}}
> getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge
> leading to busy threads in infinite loop
> -
>
> Key: LUCENE-10428
> URL: https://issues.apache.org/jira/browse/LUCENE-10428
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/query/scoring, core/search
>Reporter: Ankit Jain
>Priority: Major
>
> Customers complained about high CPU for Elasticsearch cluster in production.
> We noticed that few search requests were stuck for long time
> {quote}% curl -s localhost:9200/_cat/tasks?v
> indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
> AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
> emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
> emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h{quote}
> Flame graphs indicated that CPU time is mostly going into
> *getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some
> live JVM debugging found that
> org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
> around 4 million invocations every second
> Figured out the values of some parameters from live debugging:
> {quote}minScoreSum = 3.5541441
> minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
> 3.554144322872162
> returnObj scoreSumUpperBound = 3.5541444
> Math.ulp(minScoreSum) = 2.3841858E-7{quote}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10428) getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge leading to busy threads in infinite loop
[
https://issues.apache.org/jira/browse/LUCENE-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Jain updated LUCENE-10428:
Description:
Customers complained about high CPU for Elasticsearch cluster in production. We
noticed that few search requests were stuck for long time
{code:java}
% curl -s localhost:9200/_cat/tasks?v
indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h
{code}
Flame graphs indicated that CPU time is mostly going into
*getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some live
JVM debugging found that
org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
around 4 million invocations every second
Figured out the values of some parameters from live debugging:
{code:java}
minScoreSum = 3.5541441
minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
3.554144322872162
returnObj scoreSumUpperBound = 3.5541444
Math.ulp(minScoreSum) = 2.3841858E-7
{code}
was:
Customers complained about high CPU for Elasticsearch cluster in production. We
noticed that few search requests were stuck for long time
{quote}% curl -s localhost:9200/_cat/tasks?v
indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h{quote}
Flame graphs indicated that CPU time is mostly going into
*getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some live
JVM debugging found that
org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
around 4 million invocations every second
Figured out the values of some parameters from live debugging:
{quote}minScoreSum = 3.5541441
minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
3.554144322872162
returnObj scoreSumUpperBound = 3.5541444
Math.ulp(minScoreSum) = 2.3841858E-7{quote}
> getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge
> leading to busy threads in infinite loop
> -
>
> Key: LUCENE-10428
> URL: https://issues.apache.org/jira/browse/LUCENE-10428
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/query/scoring, core/search
>Reporter: Ankit Jain
>Priority: Major
>
> Customers complained about high CPU for Elasticsearch cluster in production.
> We noticed that few search requests were stuck for long time
> {code:java}
> % curl -s localhost:9200/_cat/tasks?v
> indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
> AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
> emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
> emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h
> {code}
> Flame graphs indicated that CPU time is mostly going into
> *getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some
> live JVM debugging found that
> org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
> around 4 million invocations every second
> Figured out the values of some parameters from live debugging:
> {code:java}
> minScoreSum = 3.5541441
> minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
> 3.554144322872162
> returnObj scoreSumUpperBound = 3.5541444
> Math.ulp(minScoreSum) = 2.3841858E-7
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10428) getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge leading to busy threads in infinite loop
[
https://issues.apache.org/jira/browse/LUCENE-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Jain updated LUCENE-10428:
Description:
Customers complained about high CPU for Elasticsearch cluster in production. We
noticed that few search requests were stuck for long time
{code:java}
% curl -s localhost:9200/_cat/tasks?v
indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h
{code}
Flame graphs indicated that CPU time is mostly going into
*getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some live
JVM debugging found that
org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
around 4 million invocations every second
Figured out the values of some parameters from live debugging:
{code:java}
minScoreSum = 3.5541441
minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
3.554144322872162
returnObj scoreSumUpperBound = 3.5541444
Math.ulp(minScoreSum) = 2.3841858E-7
{code}
was:
Customers complained about high CPU for Elasticsearch cluster in production. We
noticed that few search requests were stuck for long time
{code:java}
% curl -s localhost:9200/_cat/tasks?v
indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h
{code}
Flame graphs indicated that CPU time is mostly going into
*getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some live
JVM debugging found that
org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
around 4 million invocations every second
Figured out the values of some parameters from live debugging:
{code:java}
minScoreSum = 3.5541441
minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
3.554144322872162
returnObj scoreSumUpperBound = 3.5541444
Math.ulp(minScoreSum) = 2.3841858E-7
{code}
> getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge
> leading to busy threads in infinite loop
> -
>
> Key: LUCENE-10428
> URL: https://issues.apache.org/jira/browse/LUCENE-10428
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/query/scoring, core/search
>Reporter: Ankit Jain
>Priority: Major
>
> Customers complained about high CPU for Elasticsearch cluster in production.
> We noticed that few search requests were stuck for long time
> {code:java}
> % curl -s localhost:9200/_cat/tasks?v
> indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
> AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
> emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
> emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h
> {code}
> Flame graphs indicated that CPU time is mostly going into
> *getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some
> live JVM debugging found that
> org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
> around 4 million invocations every second
> Figured out the values of some parameters from live debugging:
> {code:java}
> minScoreSum = 3.5541441
> minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
> 3.554144322872162
> returnObj scoreSumUpperBound = 3.5541444
> Math.ulp(minScoreSum) = 2.3841858E-7
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10428) getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge leading to busy threads in infinite loop
[
https://issues.apache.org/jira/browse/LUCENE-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Jain updated LUCENE-10428:
Attachment: Flame_graph.png
> getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge
> leading to busy threads in infinite loop
> -
>
> Key: LUCENE-10428
> URL: https://issues.apache.org/jira/browse/LUCENE-10428
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/query/scoring, core/search
>Reporter: Ankit Jain
>Priority: Major
> Attachments: Flame_graph.png
>
>
> Customers complained about high CPU for Elasticsearch cluster in production.
> We noticed that few search requests were stuck for long time
> {code:java}
> % curl -s localhost:9200/_cat/tasks?v
> indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
> AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
> emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
> emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h
> {code}
> Flame graphs indicated that CPU time is mostly going into
> *getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some
> live JVM debugging found that
> org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
> around 4 million invocations every second
> Figured out the values of some parameters from live debugging:
> {code:java}
> minScoreSum = 3.5541441
> minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
> 3.554144322872162
> returnObj scoreSumUpperBound = 3.5541444
> Math.ulp(minScoreSum) = 2.3841858E-7
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10428) getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge leading to busy threads in infinite loop
[
https://issues.apache.org/jira/browse/LUCENE-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Jain updated LUCENE-10428:
Description:
Customers complained about high CPU for Elasticsearch cluster in production. We
noticed that few search requests were stuck for long time
{code:java}
% curl -s localhost:9200/_cat/tasks?v
indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h
{code}
Flame graphs indicated that CPU time is mostly going into
*getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some live
JVM debugging found that
org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
around 4 million invocations every second
Figured out the values of some parameters from live debugging:
{code:java}
minScoreSum = 3.5541441
minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
3.554144322872162
returnObj scoreSumUpperBound = 3.5541444
Math.ulp(minScoreSum) = 2.3841858E-7
{code}
Example code snippet:
{code:java}
double sumOfOtherMaxScores = 3.554144322872162;
double minScoreSum = 3.5541441;
float minScore = (float) (minScoreSum - sumOfOtherMaxScores);
while (scoreSumUpperBound(minScore + sumOfOtherMaxScores) >
minScoreSum) {
minScore -= Math.ulp(minScoreSum);
System.out.printf("%.20f, %.100f\n", minScore,
Math.ulp(minScoreSum));
}
{code}
was:
Customers complained about high CPU for Elasticsearch cluster in production. We
noticed that few search requests were stuck for long time
{code:java}
% curl -s localhost:9200/_cat/tasks?v
indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h
{code}
Flame graphs indicated that CPU time is mostly going into
*getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some live
JVM debugging found that
org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
around 4 million invocations every second
Figured out the values of some parameters from live debugging:
{code:java}
minScoreSum = 3.5541441
minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
3.554144322872162
returnObj scoreSumUpperBound = 3.5541444
Math.ulp(minScoreSum) = 2.3841858E-7
{code}
> getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge
> leading to busy threads in infinite loop
> -
>
> Key: LUCENE-10428
> URL: https://issues.apache.org/jira/browse/LUCENE-10428
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/query/scoring, core/search
>Reporter: Ankit Jain
>Priority: Major
> Attachments: Flame_graph.png
>
>
> Customers complained about high CPU for Elasticsearch cluster in production.
> We noticed that few search requests were stuck for long time
> {code:java}
> % curl -s localhost:9200/_cat/tasks?v
> indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
> AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
> emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
> emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h
> {code}
> Flame graphs indicated that CPU time is mostly going into
> *getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some
> live JVM debugging found that
> org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
> around 4 million invocations every second
> Figured out the values of some parameters from live debugging:
> {code:java}
> minScoreSum = 3.5541441
> minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
> 3.554144322872162
> returnObj scoreSumUpperBound = 3.5541444
> Math.ulp(minScoreSum) = 2.3841858E-7
> {code}
> Example code snippet:
> {code:java}
> double sumOfOtherMaxScores = 3.554144322872162;
>
[jira] [Updated] (LUCENE-10428) getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge leading to busy threads in infinite loop
[
https://issues.apache.org/jira/browse/LUCENE-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Jain updated LUCENE-10428:
Description:
Customers complained about high CPU for Elasticsearch cluster in production. We
noticed that few search requests were stuck for long time
{code:java}
% curl -s localhost:9200/_cat/tasks?v
indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h
{code}
Flame graphs indicated that CPU time is mostly going into
*getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some live
JVM debugging found that
org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
around 4 million invocations every second
Figured out the values of some parameters from live debugging:
{code:java}
minScoreSum = 3.5541441
minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
3.554144322872162
returnObj scoreSumUpperBound = 3.5541444
Math.ulp(minScoreSum) = 2.3841858E-7
{code}
Example code snippet:
{code:java}
double sumOfOtherMaxScores = 3.554144322872162;
double minScoreSum = 3.5541441;
float minScore = (float) (minScoreSum - sumOfOtherMaxScores);
while (scoreSumUpperBound(minScore + sumOfOtherMaxScores) > minScoreSum) {
minScore -= Math.ulp(minScoreSum);
System.out.printf("%.20f, %.100f\n", minScore, Math.ulp(minScoreSum));
}
{code}
was:
Customers complained about high CPU for Elasticsearch cluster in production. We
noticed that few search requests were stuck for long time
{code:java}
% curl -s localhost:9200/_cat/tasks?v
indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h
{code}
Flame graphs indicated that CPU time is mostly going into
*getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some live
JVM debugging found that
org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
around 4 million invocations every second
Figured out the values of some parameters from live debugging:
{code:java}
minScoreSum = 3.5541441
minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
3.554144322872162
returnObj scoreSumUpperBound = 3.5541444
Math.ulp(minScoreSum) = 2.3841858E-7
{code}
Example code snippet:
{code:java}
double sumOfOtherMaxScores = 3.554144322872162;
double minScoreSum = 3.5541441;
float minScore = (float) (minScoreSum - sumOfOtherMaxScores);
while (scoreSumUpperBound(minScore + sumOfOtherMaxScores) >
minScoreSum) {
minScore -= Math.ulp(minScoreSum);
System.out.printf("%.20f, %.100f\n", minScore,
Math.ulp(minScoreSum));
}
{code}
> getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge
> leading to busy threads in infinite loop
> -
>
> Key: LUCENE-10428
> URL: https://issues.apache.org/jira/browse/LUCENE-10428
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/query/scoring, core/search
>Reporter: Ankit Jain
>Priority: Major
> Attachments: Flame_graph.png
>
>
> Customers complained about high CPU for Elasticsearch cluster in production.
> We noticed that few search requests were stuck for long time
> {code:java}
> % curl -s localhost:9200/_cat/tasks?v
> indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
> AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
> emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
> emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h
> {code}
> Flame graphs indicated that CPU time is mostly going into
> *getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some
> live JVM debugging found that
> org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
> around 4 million invocations every se
[jira] [Commented] (LUCENE-10428) getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge leading to busy threads in infinite loop
[
https://issues.apache.org/jira/browse/LUCENE-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494872#comment-17494872
]
Daniel Doubrovkine commented on LUCENE-10428:
-
This is Elasticsearch version 7.9 (lucene 8.6.0), fyi. Don't have a
reproducible search query that leads to this infinite loop.
> getMinCompetitiveScore method in MaxScoreSumPropagator fails to converge
> leading to busy threads in infinite loop
> -
>
> Key: LUCENE-10428
> URL: https://issues.apache.org/jira/browse/LUCENE-10428
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/query/scoring, core/search
>Reporter: Ankit Jain
>Priority: Major
> Attachments: Flame_graph.png
>
>
> Customers complained about high CPU for Elasticsearch cluster in production.
> We noticed that few search requests were stuck for long time
> {code:java}
> % curl -s localhost:9200/_cat/tasks?v
> indices:data/read/search[phase/query] AmMLzDQ4RrOJievRDeGFZw:569205
> AmMLzDQ4RrOJievRDeGFZw:569204 direct1645195007282 14:36:47 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:502075
> emjWc5bUTG6lgnCGLulq-Q:502074 direct1645195037259 14:37:17 6.2h
> indices:data/read/search[phase/query] emjWc5bUTG6lgnCGLulq-Q:583270
> emjWc5bUTG6lgnCGLulq-Q:583269 direct1645201316981 16:21:56 4.5h
> {code}
> Flame graphs indicated that CPU time is mostly going into
> *getMinCompetitiveScore method in MaxScoreSumPropagator*. After doing some
> live JVM debugging found that
> org.apache.lucene.search.MaxScoreSumPropagator.scoreSumUpperBound method had
> around 4 million invocations every second
> Figured out the values of some parameters from live debugging:
> {code:java}
> minScoreSum = 3.5541441
> minScore + sumOfOtherMaxScores (params[0] scoreSumUpperBound) =
> 3.554144322872162
> returnObj scoreSumUpperBound = 3.5541444
> Math.ulp(minScoreSum) = 2.3841858E-7
> {code}
> Example code snippet:
> {code:java}
> double sumOfOtherMaxScores = 3.554144322872162;
> double minScoreSum = 3.5541441;
> float minScore = (float) (minScoreSum - sumOfOtherMaxScores);
> while (scoreSumUpperBound(minScore + sumOfOtherMaxScores) > minScoreSum) {
> minScore -= Math.ulp(minScoreSum);
> System.out.printf("%.20f, %.100f\n", minScore, Math.ulp(minScoreSum));
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta opened a new pull request #693: LUCENE-10400: revise binary dictionaries' constructor in nori
mocobeta opened a new pull request #693: URL: https://github.com/apache/lucene/pull/693 This applies the same changes as #643 to Nori. - Add new constructors that take Path object to load external resources. - Add tests for the new constructors. - Deprecate the old constructors to switch the resource location (classpath/file path). - Always use Class.getResourceAsStream() to load class resources. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] mocobeta commented on pull request #643: LUCENE-10400: revise binary dictionaries' constructor in kuromoji
mocobeta commented on pull request #643: URL: https://github.com/apache/lucene/pull/643#issuecomment-1045724183 We need to apply the same changes here to Nori. I opened #693. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
