Re: Making stemming dynamic at query time

2007-12-20 Thread Erick Erickson
Well, you *still* have to store the stemmed and unstemmed version in your index, otherwise you can't distinguish between, say, run and running because you'd have indexed run both times. But you could think about using "special tokenizing". That is, for a word that's stemmed, index a "stem form".

Re: Making stemming dynamic at query time

2007-12-20 Thread Kamran Shadkhast
Kamran, I think Bertrand's suggestion is the only possible solution. I can't think of a way you can not stem at index time and make it an option at search time. If you look at and understand low-level/basic indexing and term matching process, I think you'll see why this seems impossible. But

RE: Making stemming dynamic at query time

2007-12-20 Thread Steven A Rowe
@lucene.apache.org > Sent: Wednesday, December 19, 2007 9:46:10 AM > Subject: Re: Making stemming dynamic at query time > > > The easiest is probably to have two copies of your field, using > , one stemmed and one not, and search in one or the other. > > -Bertrand > > Ye

Re: Making stemming dynamic at query time

2007-12-20 Thread Otis Gospodnetic
M Subject: Re: Making stemming dynamic at query time The easiest is probably to have two copies of your field, using , one stemmed and one not, and search in one or the other. -Bertrand Yes, I knew this, but it costs me too much, in my case having more than 65M records and saving most of the fi

Re: Making stemming dynamic at query time

2007-12-19 Thread Kamran Shadkhast
The easiest is probably to have two copies of your field, using , one stemmed and one not, and search in one or the other. -Bertrand Yes, I knew this, but it costs me too much, in my case having more than 65M records and saving most of the fields inside the index for highlighting purpose does no

Re: Making stemming dynamic at query time

2007-12-18 Thread Bertrand Delacretaz
On Dec 18, 2007 9:41 PM, Kamran Shadkhast <[EMAIL PROTECTED]> wrote: > ...it would be great if we could dynamiclly control this during > search if we want to search with stemming or not The easiest is probably to have two copies of your field, using , one stemmed and one not, and search in on