On Wed, Apr 21, 2010 at 1:18 PM, Chris Hostetter <hossman_luc...@fucit.org>wrote: > > Strictly speaking: you haven't "ditched" stemmers altogether -- you've > ditched *algorithmic* stemmers and moved to a *dictionary* based stemmer > -- but it's still a stemmer. > > (i just don't want people reading this thread to be confused about > terminology) > > I agree, and dictionary-based stemming has its own set of problems. While its easy to look at the "faults" of some algorithmic stemmer, in truth its only purpose is to cause related forms of the word to conflate to the same form, and hopefully avoiding unrelated terms from conflating to this form.
A dictionary-based stemmer is out-of-date the day you put it into production: languages aren't static. For example, you can't expect a dictionary-based stemmer to properly deal with forms like "googling" or "tweets" that have recently slipped into English vocabulary, but an algorithmic stemmer will likely deal with these just fine. -- Robert Muir rcm...@gmail.com
