gsmiller commented on a change in pull request #509:
URL: https://github.com/apache/lucene/pull/509#discussion_r780546237
##########
File path:
lucene/facet/src/java/org/apache/lucene/facet/sortedset/DefaultSortedSetDocValuesReaderState.java
##########
@@ -51,20 +54,42 @@
private final Map<String, OrdinalMap> cachedOrdMaps = new HashMap<>();
+ private final FacetsConfig config;
+
+ /*** Used for hierarchical dims ***/
Review comment:
Oh I see. That's fine then. I assumed it was an IDE/spotless issue since
I saw some other places that looked a little more funky.
##########
File path: lucene/facet/src/java/org/apache/lucene/facet/FacetsConfig.java
##########
@@ -471,19 +471,36 @@ private void indexDrillDownTerms(
private void processSSDVFacetFields(
Map<String, List<SortedSetDocValuesFacetField>> byField, Document doc) {
+
for (Map.Entry<String, List<SortedSetDocValuesFacetField>> ent :
byField.entrySet()) {
String indexFieldName = ent.getKey();
for (SortedSetDocValuesFacetField facetField : ent.getValue()) {
- FacetLabel facetLabel = new FacetLabel(facetField.dim,
facetField.label);
- String fullPath = pathToString(facetLabel.components,
facetLabel.length);
-
- // For facet counts:
- doc.add(new SortedSetDocValuesField(indexFieldName, new
BytesRef(fullPath)));
-
+ FacetLabel facetLabel = new FacetLabel(facetField.dim,
facetField.path);
+ DimConfig dimConfig = getDimConfig(facetField.dim);
+ if (dimConfig.hierarchical) {
+ for (int i = 0; i < facetLabel.length; i++) {
+ String fullPath = pathToString(facetLabel.components, i + 1);
+ // For facet counts:
+ doc.add(new SortedSetDocValuesField(indexFieldName, new
BytesRef(fullPath)));
+ }
+ // For drill-down:
Review comment:
Looks like a straggler comment line leftover.
##########
File path:
lucene/facet/src/java/org/apache/lucene/facet/sortedset/SortedSetDocValuesFacetCounts.java
##########
@@ -317,10 +345,19 @@ public Number getSpecificValue(String dim, String...
path) throws IOException {
public List<FacetResult> getAllDims(int topN) throws IOException {
List<FacetResult> results = new ArrayList<>();
- for (Map.Entry<String, OrdRange> ent :
state.getPrefixToOrdRange().entrySet()) {
- FacetResult fr = getDim(ent.getKey(), ent.getValue(), topN);
- if (fr != null) {
- results.add(fr);
+ for (String dim : state.getDims()) {
+ if (stateConfig.getDimConfig(dim).hierarchical) {
+ DimTree dimTree = state.getDimTree(dim);
+ FacetResult fr = getDim(dim, emptyPath, dimTree.dimStartOrd,
dimTree.iterator(), topN);
Review comment:
I can't respond to our previous comment thread because it seems to think
the comment is outdated or something. But to pick up that thread, you'd said:
>If we pass -1 in the code would just use dimCount for the value, which
could be inaccurate in the multi-valued case. Given that, I think it would be
better to pass in dimStartOrd so we can guarantee that hierarchical SSDV facets
have accurate multi-valued counts, given that we are already doing ancestral
indexing. What do you think?
... and that makes total sense. I got my wired crossed a little on this.
Separately (but related to my confusion), is `dimStartOrd` really just the
`dimOrd`? Do you think it would be more readable to rename this? Or maybe I'm
still confused about this?
##########
File path:
lucene/facet/src/java/org/apache/lucene/facet/sortedset/SortedSetDocValuesFacetField.java
##########
@@ -40,20 +41,33 @@
/** Dimension. */
public final String dim;
- /** Label. */
- public final String label;
+ /** Path. */
+ public final String[] path;
+
+ /** String form of path. */
Review comment:
Could you add a `@deprecated` message to the javadoc as well please?
Maybe something like:
```
/**
* String form of path.
*
* @deprecated This field will be removed in a future version. {@link
* FacetsConfig#pathToString(String[])} can be applied to {@code path}
as a replacement if
* string path is desired.
*/
```
##########
File path:
lucene/facet/src/java/org/apache/lucene/facet/sortedset/DefaultSortedSetDocValuesReaderState.java
##########
@@ -79,38 +105,139 @@ public DefaultSortedSetDocValuesReaderState(IndexReader
reader, String field) th
}
valueCount = (int) dv.getValueCount();
- // TODO: we can make this more efficient if eg we can be
- // "involved" when OrdinalMap is being created? Ie see
- // each term/ord it's assigning as it goes...
- String lastDim = null;
- int startOrd = -1;
-
- // TODO: this approach can work for full hierarchy?;
- // TaxoReader can't do this since ords are not in
- // "sorted order" ... but we should generalize this to
- // support arbitrary hierarchy:
- for (int ord = 0; ord < valueCount; ord++) {
- final BytesRef term = dv.lookupOrd(ord);
+ int ord = 0;
+ while (ord != valueCount) {
+ BytesRef term = dv.lookupOrd(ord);
String[] components = FacetsConfig.stringToPath(term.utf8ToString());
- if (components.length != 2) {
- throw new IllegalArgumentException(
- "this class can only handle 2 level hierarchy (dim/value); got: "
- + Arrays.toString(components)
- + " "
- + term.utf8ToString());
+ String dim = components[0];
+ if (config != null && config.getDimConfig(dim).hierarchical) {
+ ord = createOneHierarchicalFacetDimState(dv, ord) + 1;
+ } else {
+ ord = createOneFlatFacetDimState(dv, ord) + 1;
}
- if (!components[0].equals(lastDim)) {
- if (lastDim != null) {
- prefixToOrdRange.put(lastDim, new OrdRange(startOrd, ord - 1));
+ }
+ }
+
+ // returns last ord of dimension
+ private int createOneHierarchicalFacetDimState(SortedSetDocValues dv, int
dimStartOrd)
+ throws IOException {
+ List<Boolean> hasChildren = new ArrayList<>();
+ List<Integer> siblings = new ArrayList<>();
+
+ // stack of paths with unfulfilled siblings
+ Stack<OrdAndComponent> siblingStack = new Stack<>();
+
+ int dimEndOrd = dimStartOrd;
+
+ BytesRef nextTerm = dv.lookupOrd(dimEndOrd);
+ String[] nextComponents =
FacetsConfig.stringToPath(nextTerm.utf8ToString());
+ String dim = nextComponents[0];
+
+ while (true) {
+ String[] components = nextComponents;
+
+ int ord = dimEndOrd - dimStartOrd;
+
+ while (siblingStack.empty() == false
+ && siblingStack.peek().component.length >= components.length) {
+ OrdAndComponent possibleSibling = siblingStack.pop();
+ if (possibleSibling.component.length == components.length) {
+ // lengths are equal, all non-siblings of equal length will have
already been popped off
+ // so this must be sibling
+ siblings.set(possibleSibling.ord, ord);
}
- startOrd = ord;
- lastDim = components[0];
}
+
+ if (dimEndOrd + 1 == valueCount) {
+ // current ord needs to be added, can't have children or siblings
+ siblings.add(-1);
+ hasChildren.add(false);
+ break;
+ }
+
+ nextTerm = dv.lookupOrd(dimEndOrd + 1);
+ nextComponents = FacetsConfig.stringToPath(nextTerm.utf8ToString());
+
+ if (nextComponents[0].equals(components[0]) == false) {
+ // current ord needs to be added, can't have children or siblings
+ siblings.add(-1);
+ hasChildren.add(false);
+ break;
+ }
+
+ if (components.length < nextComponents.length) {
+ // next ord must be a direct child of current ord, this is because we
are indexing all
+ // ancestral paths
+ hasChildren.add(ord, true);
+ // we don't know if this ord has a sibling or where it's sibling could
be yet
+ siblingStack.push(new OrdAndComponent(ord, components));
+ // we still add INVALID_ORDINAL, which will be replaced if a valid
sibling is found
+ siblings.add(ord, INVALID_ORDINAL);
+ } else if (components.length == nextComponents.length) {
+ // next ord must be a sibling of current and there are no direct
children or current, this
Review comment:
typo: "no direct children _of_ current" instead of "no direct children
or current"
##########
File path:
lucene/facet/src/java/org/apache/lucene/facet/sortedset/SortedSetDocValuesReaderState.java
##########
@@ -62,15 +166,28 @@ protected SortedSetDocValuesReaderState() {}
/** Indexed field we are reading. */
public abstract String getField();
+ /** Returns top-level index reader. */
+ public abstract IndexReader getReader();
+
+ /** Number of unique labels. */
+ public abstract int getSize();
+
+ /** Returns the associated facet config. */
+ public abstract FacetsConfig getFacetConfig();
Review comment:
minor: For naming consistency, maybe make "facet" plural (so
`getFacetsConfig`). Sorry to be so nit-picky on this but since it's a public
API, I'm being a little pedantic.
##########
File path:
lucene/facet/src/java/org/apache/lucene/facet/sortedset/DefaultSortedSetDocValuesReaderState.java
##########
@@ -51,20 +55,42 @@
private final Map<String, OrdinalMap> cachedOrdMaps = new HashMap<>();
+ private final FacetsConfig config;
+
+ /** Used for hierarchical dims. */
+ private final Map<String, DimTree> prefixToDimTree = new HashMap<>();
+
+ /** Used for flat dims. */
private final Map<String, OrdRange> prefixToOrdRange = new HashMap<>();
/**
- * Creates this, pulling doc values from the default {@link
+ * Creates this with a config, pulling doc values from the default {@link
+ * FacetsConfig#DEFAULT_INDEX_FIELD_NAME}.
+ */
+ public DefaultSortedSetDocValuesReaderState(IndexReader reader, FacetsConfig
config)
+ throws IOException {
+ this(reader, FacetsConfig.DEFAULT_INDEX_FIELD_NAME, config);
+ }
+
+ /**
+ * Creates this without a config, pulling doc values from the default {@link
* FacetsConfig#DEFAULT_INDEX_FIELD_NAME}.
*/
public DefaultSortedSetDocValuesReaderState(IndexReader reader) throws
IOException {
- this(reader, FacetsConfig.DEFAULT_INDEX_FIELD_NAME);
+ this(reader, FacetsConfig.DEFAULT_INDEX_FIELD_NAME, null);
}
- /** Creates this, pulling doc values from the specified field. */
+ /** Creates this without a config, pulling doc values from the specified
field. */
public DefaultSortedSetDocValuesReaderState(IndexReader reader, String
field) throws IOException {
+ this(reader, field, null);
+ }
+
+ /** Creates this, pulling doc values from the specified field. */
+ public DefaultSortedSetDocValuesReaderState(IndexReader reader, String
field, FacetsConfig config)
+ throws IOException {
this.field = field;
this.reader = reader;
+ this.config = Objects.requireNonNullElse(config, new FacetsConfig());
Review comment:
Hmm, I don't think we want to default this here. It would be good to
know that the user didn't provide a `FacetsConfig` instance later on instead of
just assuming some default. Can we let this be `null` and be defensive against
that case elsewhere?
Separately, we might want to deprecate the ability to instantiate this class
without providing the config. Let's tackle that as a separate issue though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]