hidingmyname opened a new issue, #16927:
URL: https://github.com/apache/lucene/issues/16927
### Summary
The `yyreset(Reader)` javadoc in both of Lucene's JFlex skeletons says
"Lexical state is set to {@code ZZ_INITIAL}". No `ZZ_INITIAL` exists. The
initial lexical state constant is `YYINITIAL`, and that's what the method
body
a few lines further down actually assigns.
Because the skeletons are templates, the same sentence is copied into every
generated scanner.
### Where
`gradle/regenerate/jflex/skeleton.default.txt:155-171`:
```java
/**
* Resets the scanner to read from a new input stream.
*
* <p>Does not close the old reader.
*
* <p>All internal variables are reset, the old input stream <b>cannot</b>
be reused (internal
* buffer is discarded and lost). Lexical state is set to {@code
ZZ_INITIAL}.
*
* <p>Internal scan buffer is resized down to its initial length, if it
has grown.
*
* @param reader The new input stream.
*/
public final void yyreset(java.io.Reader reader) {
zzReader = reader;
zzEOFDone = false;
yyResetPosition();
zzLexicalState = YYINITIAL;
```
`gradle/regenerate/jflex/skeleton.disable.buffer.expansion.txt:157` has the
identical text.
Generated copies on `main`:
-
`lucene/core/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java:826`
-
`lucene/analysis/common/src/java/org/apache/lucene/analysis/charfilter/HTMLStripCharFilter.java:5990`
-
`lucene/analysis/common/src/java/org/apache/lucene/analysis/classic/ClassicTokenizerImpl.java:540`
-
`lucene/analysis/common/src/java/org/apache/lucene/analysis/email/UAX29URLEmailTokenizerImpl.java:33704`
-
`lucene/analysis/common/src/java/org/apache/lucene/analysis/wikipedia/WikipediaTokenizerImpl.java:693`
Each of those declares `public static final int YYINITIAL = 0;` (e.g.
`StandardTokenizerImpl.java:54`), and none declares `ZZ_INITIAL`.
### Why it matters
To be upfront about scope: this is javadoc on generated scanner classes, and
most users never read it. The only one that's really public-facing is
`HTMLStripCharFilter`, whose generated javadoc ends up in the analysis-common
API docs. So this is low priority.
It's still a doc that names something you can't find. Someone checking what
state the scanner is in after a reset searches for `ZZ_INITIAL` and gets no
hits, and `{@code}` (rather than `{@link}`) means javadoc never flags it.
One process note: the generated `.java` files are covered by checksums, so
the
fix has to go into the two skeleton files followed by a regenerate. Editing
the
generated files by hand would fail the checksum check.
### Suggested fix
In both `skeleton.default.txt` and `skeleton.disable.buffer.expansion.txt`:
```diff
* <p>All internal variables are reset, the old input stream
<b>cannot</b> be reused (internal
- * buffer is discarded and lost). Lexical state is set to {@code
ZZ_INITIAL}.
+ * buffer is discarded and lost). Lexical state is set to {@code
YYINITIAL}.
```
then regenerate the affected scanners and their checksums:
```
gradlew :lucene:core:regenerateStandardTokenizer
gradlew :lucene:analysis:common:regenerateClassicTokenizer
gradlew :lucene:analysis:common:regenerateHTMLStripCharFilter
gradlew :lucene:analysis:common:regenerateUAX29URLEmailTokenizer
gradlew :lucene:analysis:common:regenerateWikipediaTokenizer
```
The wording comes from JFlex's stock skeleton. Its current
`jflex/src/main/resources/jflex/skeleton.default` still says `ZZ_INITIAL`, so
it could be reported there as well, but Lucene keeps its own skeleton copies
and can fix them on its own.
### Upstream reference
Lucene.NET ported these generated scanners and fixed the text in
apache/lucenenet@`d4b9c00ef` ("Lucene.Net.Analysis SWEEP: fix for various
broken XML comments"):
```diff
---
a/src/Lucene.Net.Analysis.Common/Analysis/CharFilter/HTMLStripCharFilter.cs
+++
b/src/Lucene.Net.Analysis.Common/Analysis/CharFilter/HTMLStripCharFilter.cs
@@ -31061,7 +31061,7 @@ private void YyClose()
/// <para/>
/// All internal variables are reset, the old input stream
/// <b>cannot</b> be reused (internal buffer is discarded and lost).
- /// Lexical state is set to <see cref="ZZ_INITIAL"/>.
+ /// Lexical state is set to <see cref="YYINITIAL"/>.
/// <para/>
/// Internal scan buffer is resized down to its initial length, if
it has grown.
/// </summary>
```
Current Lucene.NET uses `YYINITIAL` throughout its ported scanners, e.g.
`ClassicTokenizerImpl.cs` and `StandardTokenizerImpl.cs`:
```csharp
/// Lexical state is set to <see cref="YYINITIAL"/>.
```
I can submit a PR for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
