rmuir commented on code in PR #14583:
URL: https://github.com/apache/lucene/pull/14583#discussion_r2083651585
##########
lucene/analysis/common/src/java/org/apache/lucene/analysis/classic/ClassicTokenizerImpl.java:
##########
@@ -438,6 +436,16 @@ public final void setBufferSize(int numChars) {
this.zzReader = in;
}
+ /** Returns the maximum size of the scanner buffer, which limits the size of
tokens. */
+ private int zzMaxBufferLen() {
+ return Integer.MAX_VALUE;
+ }
+
+ /** Whether the scanner buffer can grow to accommodate a larger token. */
+ private boolean zzCanGrow() {
+ return true;
+ }
Review Comment:
I understand it now. these buffer limits/control are new features that we
probably want to adopt (and remove our "skeletons"). first I want to get tests
passing before attempting it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
