epotyom commented on code in PR #14204: URL: https://github.com/apache/lucene/pull/14204#discussion_r1958022460
########## lucene/facet/src/java/org/apache/lucene/facet/histogram/HistogramCollector.java: ########## @@ -0,0 +1,252 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.facet.histogram; + +import java.io.IOException; +import org.apache.lucene.index.DocValues; +import org.apache.lucene.index.DocValuesSkipper; +import org.apache.lucene.index.DocValuesType; +import org.apache.lucene.index.FieldInfo; +import org.apache.lucene.index.LeafReaderContext; +import org.apache.lucene.index.NumericDocValues; +import org.apache.lucene.index.SortedNumericDocValues; +import org.apache.lucene.internal.hppc.LongIntHashMap; +import org.apache.lucene.search.CollectionTerminatedException; +import org.apache.lucene.search.Collector; +import org.apache.lucene.search.LeafCollector; +import org.apache.lucene.search.Scorable; +import org.apache.lucene.search.ScoreMode; + +final class HistogramCollector implements Collector { Review Comment: > Then it would be nice to make ordinals `long`s rather than `int` It's an interesting idea. The issue is many facet implementations don't need `long` ordinals, e.g. Taxonomy or SSDV facets counting use `int`s, even though at least in some cases they index as `long`s. It looks like using `long` for them might be wasteful. Overall, using `long` for something that is essential a group id in a grouping mechanism seems excessive. At the same time `long` facet ordinals can simplify and improve performance for some FacetCutter implementations, in particular `LongValueFacetCutter` . Also, as you've mentioned, it is FacetRecorder responsibility to keep counts in a dense data structure, so it might be fine to move to `long`. In any case it requires a separate effort, and I think we should run [luceneutil #325](https://github.com/mikemccand/luceneutil/pull/325) and Amazon internal perf tests before making the decision. I can create an issue for it. > ordinals seem to be expected to be positive Yes, it is also a limitation in the current API. IIRC the only thing that relies on it is `OrdinalIterator#NO_MORE_ORDS = -1`. We can probably reserve some other value for it, e.g. `Long.MAX_VALUE` or `MIN_VALUE` , it should work for most cases, including histogram since bucketWidth has to be greater than 2. It's a bit fragile for LongValueFacetCutter - we'd have to throw runtime error is the value in the index is `NO_MORE_ORDS`. Although the implementation is already fragile as it uses LongIntHashMap which [size is limited by an array size](https://github.com/apache/lucene/blob/05fc9a834d85fdbefbaf4545fb34ab0f38c0c2a7/lucene/core/src/java/org/apache/lucene/internal/hppc/HashContainers.java#L48). So I suppose it's not too terrible to not allow NO_MORE_ORDS value when counting. > Separately I played with the quick/dirty benchmark I had created, which seems to have got a bit more than 2x slower. I'd like to look into it - maybe there is something we can improve. Are you running runGeoBench.cmd to get results? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
