Hi Tino,
I don't oppose the idea of adding support for compiling dix-format lexicons
with HFST and implementing it for existing language modules, but we'd still
need to process the transducers with lt-proc because of the tokenisation
bug.¹ If the HFST tokenisation bug were to disappear, then what you
propose for the long term might make sense. In any case, it couldn't hurt
to add .dix support in hfst-comp as a GSoC project idea—that'd be a good
step in the right direction. Maybe if someone ends up working on this and
has extra time, they could investigate the tokenisation bug as well.
¹ https://sourceforge.net/p/hfst/bugs/59/ seems to suggest that this was
fixed, and I'm unable to reproduce it at the moment. I remember something
about undesired side-effects of the implementation, but I don't remember
now what those were. Perhaps Tommi or Fran remembers why we still use
lt-proc for most HFST modules?
--
Jonathan
2018-02-16 4:50 GMT-05:00 Tino Didriksen <[email protected]>:
> We have these 3 tasks about adding features to lttoolbox:
>
> - http://wiki.apertium.org/wiki/Ideas_for_Google_Summer_of_
> Code#Robust_tokenisation_in_lttoolbox
> - http://wiki.apertium.org/wiki/Ideas_for_Google_Summer_of_
> Code#Extend_lttoolbox_to_have_the_power_of_HFST
> - http://wiki.apertium.org/wiki/Ideas_for_Google_Summer_of_
> Code#Add_weights_to_lttoolbox
>
> Why don't we instead add .dix format support to HFST, add weights to that
> format, drop lttoolbox, and just use HFST wholesale? It can do all the
> things we want - just need dix support.
>
> That sounds like a fraction of the work and we don't duplicate code.
>
> -- Tino Didriksen
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Apertium-stuff mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/apertium-stuff
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Apertium-stuff mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/apertium-stuff
