kuramitsu commented on code in PR #12885:
URL: https://github.com/apache/lucene/pull/12885#discussion_r1421180794
##########
lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseReadingFormFilter.java:
##########
@@ -43,10 +43,30 @@ public JapaneseReadingFormFilter(TokenStream input) {
this(input, false);
}
+ private boolean isAllHiragana(CharSequence s) {
+ int len = s.length();
+ for (int i = 0; i < len; i++) {
+ char ch = s.charAt(i);
+ if (ch < 0x3040 || ch > 0x3097) {
+ return false;
+ }
+ }
+ return true;
+ }
+
@Override
public boolean incrementToken() throws IOException {
if (input.incrementToken()) {
String reading = readingAttr.getReading();
+ if (reading == null & isAllHiragana(termAttr)) {
Review Comment:
Fixed to support partial hiragana.
However, it is difficult to describe a test for oov terms that contain a
mixture of hiragana and katakana, because the system seems to basically split
different character types for oov terms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
