Re: Facets with an IDF concept

2009-10-10 Thread Lance Norskog
o me, terms that appear in >> all documents aren't really that interesting. I'm thinking of using a >> combination of document count (in the result set, not globally) and term >> frequency (in the result set, not globally) to come up with a facet sort >> order. >&g

Re: Facets with an IDF concept

2009-10-09 Thread Asif Rahman
sort > order. > > Wojtek > -- > View this message in context: > http://www.nabble.com/Facets-with-an-IDF-concept-tp24071160p24959192.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- Asif Rahman Lead Engineer - NewsCred a...@newscred.com http://platform.newscred.com

Re: Facets with an IDF concept

2009-08-13 Thread wojtekpia
) to come up with a facet sort order. Wojtek -- View this message in context: http://www.nabble.com/Facets-with-an-IDF-concept-tp24071160p24959192.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Facets with an IDF concept

2009-06-23 Thread Grant Ingersoll
On Jun 23, 2009, at 6:23 PM, Chris Hostetter wrote: : Regardless of the semantics, it doesn't sound like DF would give you what you : want. It could be entirely possible that in some short timespan the number of : docs on Iran could match up w/ the number on Obama (maybe not for that :

Re: Facets with an IDF concept

2009-06-23 Thread Otis Gospodnetic
lr - Nutch - Original Message > From: Asif Rahman > To: solr-user@lucene.apache.org > Sent: Tuesday, June 23, 2009 8:05:48 AM > Subject: Re: Facets with an IDF concept > > Hi Grant, > > I'll give a real life example of the problem that we are trying to solve. >

Re: Facets with an IDF concept

2009-06-23 Thread Chris Hostetter
: Regardless of the semantics, it doesn't sound like DF would give you what you : want. It could be entirely possible that in some short timespan the number of : docs on Iran could match up w/ the number on Obama (maybe not for that : particular example) in which case your "hot" item would no lon

Re: Facets with an IDF concept

2009-06-23 Thread Grant Ingersoll
On Jun 23, 2009, at 8:05 AM, Asif Rahman wrote: Hi Grant, I'll give a real life example of the problem that we are trying to solve. We index a large number of current news articles on a continuing basis. We tag these articles with news topics (e.g. Barack Obama, Iran, etc.). We then

Re: Facets with an IDF concept

2009-06-23 Thread Ian Holsman
Asif Rahman wrote: Hi Grant, I'll give a real life example of the problem that we are trying to solve. We index a large number of current news articles on a continuing basis. We tag these articles with news topics (e.g. Barack Obama, Iran, etc.). We then use these tags to facet our queries.

Re: Facets with an IDF concept

2009-06-23 Thread Asif Rahman
Hi Grant, I'll give a real life example of the problem that we are trying to solve. We index a large number of current news articles on a continuing basis. We tag these articles with news topics (e.g. Barack Obama, Iran, etc.). We then use these tags to facet our queries. For example, we might

Re: Facets with an IDF concept

2009-06-23 Thread Asif Rahman
Hi Kent, Your problem is close cousin of the problem that we're tackling. We have experience the same problem as you when calculating facets on MoreLikeThis queries, since those queries tend to match a lot of documents. We used one of the solutions that you mentioned, rank cutoff, to solve it.

Re: Facets with an IDF concept

2009-06-23 Thread Grant Ingersoll
On Jun 23, 2009, at 3:58 AM, Asif Rahman wrote: Hi again, I guess nobody has used facets in the way I described below before. Do any of the experts have any ideas as to how to do this efficiently and correctly? Any thoughts would be greatly appreciated. Thanks, Asif On Wed, Jun 17, 200

Re: Facets with an IDF concept

2009-06-23 Thread Kent Fitch
Hi Asif, I was holding back because we have a similar problem, but we're not sure how best to approach it, or even whether approaching it at all is the right thing to do. Background: - large index (~35m documents) - about 120k on these include full text book contents plus metadata, the rest are j

Re: Facets with an IDF concept

2009-06-23 Thread Asif Rahman
Hi again, I guess nobody has used facets in the way I described below before. Do any of the experts have any ideas as to how to do this efficiently and correctly? Any thoughts would be greatly appreciated. Thanks, Asif On Wed, Jun 17, 2009 at 12:42 PM, Asif Rahman wrote: > Hi all, > > We ha

Facets with an IDF concept

2009-06-17 Thread Asif Rahman
Hi all, We have an index of news articles that are tagged with news topics. Currently, we use solr facets to see which topics are popular for a given query or time period. I'd like to apply the concept of IDF to the facet counts so as to penalize the topics that occur broadly through our index. I