rmuir commented on issue #15365: URL: https://github.com/apache/lucene/issues/15365#issuecomment-5896698870
@msokolov I think solving anything would require a biggish packaging change, so I'd be inclined to just remove the 11.0 milestone from this issue. Currently we supply the ipadic dictionary along with the kuromoji jar. It is managed by gradle regeneration tasks, and the size is reasonable. But there are other dictionaries (naist, unidic). These other dictionaries can produce very large artifacts. If we wanted to provide pre-built support (JARs), we'd have to rearrange them, somehow make it pluggable/selectable at runtime. Today it is possible to regenerate and recompile lucene-kuromoji from sources, using naist as an alternative dictionary to ipadic. It will download it on the fly and generate different stuff in src/resources and then you can make a new jar that uses naist instead of ipadic. for unidic we can't even yet do that: see the linked PR for some of that work. As far as pluggability/packaging that is more friendly than "recompile with alternative dictionary from sources", I really don't know the best way to do it. Some of the generated artifacts might be so large that we might have to look at git LFS or other changes, if we want to store them in git. I cant remember which one is the really big one, unidic or naist, but one of them is orders of magnitude larger than the current ipadic. Separately, we do some tricks and preprocessing so that the current ipadic is of reasonable size. I remember @uschindler and I just furiously hacked until we felt we had compressed everything :) But some of these might have assumptions that don't hold with other dictionaries. in short, the other dictionaries arent "exactly" drop-in compatible. For example look at linked PR and you can see even the features might be different! I don't think we should close the linked PR as we might be able to iterate from it too, none of the ideas look bad, we just have to "come up with a plan" * I think supporting additional dictionaries is a good thing * We need to be careful about generalizing the code when working with these dictionaries * I don't know if/how we should make them runtime pluggable * I don't know if/how we should alter packaging -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
