Hi Itay, in IR research there’s a long tradition (TREC and alike) for measuring the effectiveness of search engines. In this context it is measured by using a so called test collection, which consists of three things: 1. Documents 2. Topics i.e. information needs/queries of users for these documents 3. relevance assessment data i.e. which documents are relevant for which topics
Using the results your search engine returns for each topic, one can calculate Precision and Recall or - depending on the context and use case the searching engine is running in - other measures might be more appropriate. For example in an enterprise context it might be the case that only one document is possible to fulfill the searchers information need. If you find this to be the case for a lot of information needs, different measures like MRR might be more suitable. Moreover there’s also the question on whether you are aiming for high precision or high recall. Balancing both is a hard task and it is up to you and your users needs to find out what is more important for them. E.g. is it critical to not have certain documents? Than high recall might be your aim... If you are aiming for high precision you could, for example, measure something like Precision@10, which means how many relevant documents are in the top 10 documents returned. For this you don’t need relevance assessment data. However, what you need in any case is an idea what your users are searching for and generate possible test queries from this and also have an idea what documents are relevant. Finally, I thinks it’s not possible to give a recommended percentage cause it depends a lot on your context. Hope this helps, Florian > Am 27.07.2017 um 15:20 schrieb Itay K <itayk...@gmail.com>: > > Hi, > > I'm trying to measure Precision and recall for a search engine which is > crawling data sources of an organization. > > Are there any best practices regrading these indexes and specific > industries (e.g. for financial organizations, the recommended percentage > for precision and recall is ~60%). > > Is there any best practice in general for the recommended percentage? > > I read an article from 2005 regrading measured precision and recall for web > search engines but unfortunately my use case isn't a web application and I > believe that since than a lot has changed. > > thanks
