glawson0 commented on a change in pull request #157:
URL: https://github.com/apache/lucene/pull/157#discussion_r655951001
##########
File path:
lucene/analysis/common/src/java/org/apache/lucene/analysis/core/FlattenGraphFilter.java
##########
@@ -284,27 +316,27 @@ public boolean incrementToken() throws IOException {
InputNode src = inputNodes.get(inputFrom);
if (src.node == -1) {
- // This means the "from" node of this token was never seen as a "to"
node,
- // which should only happen if we just crossed a hole. This is a
challenging
- // case for us because we normally rely on the full dependencies
expressed
- // by the arcs to assign outgoing node IDs. It would be better if
tokens
- // were never dropped but instead just marked deleted with a new
- // TermDeletedAttribute (boolean valued) ... but until that future,
we have
Review comment:
Working on this change has made me realize how valuable not removing
tokens would be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
