rmuir commented on code in PR #14583:
URL: https://github.com/apache/lucene/pull/14583#discussion_r2067794017
##########
lucene/analysis/common/src/java/org/apache/lucene/analysis/classic/ClassicTokenizerImpl.java:
##########
@@ -438,6 +436,16 @@ public final void setBufferSize(int numChars) {
this.zzReader = in;
}
+ /** Returns the maximum size of the scanner buffer, which limits the size of
tokens. */
+ private int zzMaxBufferLen() {
+ return Integer.MAX_VALUE;
+ }
+
+ /** Whether the scanner buffer can grow to accommodate a larger token. */
+ private boolean zzCanGrow() {
+ return true;
+ }
Review Comment:
it is also suspicious that these are unused methods. Maybe it just has to do
with how the code emitter works and no check happens unless you override these
defaults.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
