Folks: We are building a search capability into our web and plan to use Solr. While we have the initial prototype version up and running on Solr 1.2, we are now turning our attention to sizing/scalability.
Our app in brief: We get merchant sku files (in either xml/csv) which we process and index and make available to our site visitors to search. Our current plan calls for us to support approx 10,000 merchants each with an average of 50,000 sku's. This will make a total of approx 500 Million SKUs. In addition, we assume that on a daily basis approx 5-10% of the SKUs need to be updated (either added/deleted/modified). (Assume each sku will be approx 4K) Here are a few questions that we are thinking about and would value any insights you all may have: a) Should we have just one giant master index (containing all the sku's) and then have multiple slaves to handle the search queries? In this case, the master index will be approx 2 TB in size. Not being an expert in solr/lucene, I am thinking that this may be a bad idea to let one index become so large. What size limit should we assume for each index? b) Or, should we partition the 10,000 merchants into N buckets and have a master index for each of the N buckets? We could partition the merchants depending on their type or some other simple algorithm. Then, we could have slaves setup for each of the N masters. The trick here will be to partition the merchants carefully. Ideally we would like a search for any product type to hit only one index but this may not be possible always. For example, a search for "Harry Potter" may result in hits in "books", "dvds", "memorabilia", etc etc. With N masters we will have to plan for having a distributed search across the N indices (and then some mechanism for weighting the results across the results that come back). Any recommendations for a distributed search solution? I saw some references to Katta. Is this viable? In the extreme case, we could have one master for each of the merchants (if there are 10000 merchants there will be 10,000 master indices). The advantage here is that indices will have to be updated only for every merchant who submits a new data file. The others remain unchanged. c) By the way, for those of you who have deployed solr on a production environment can you give me your hardware configuration and the rough number of search queries that can be handled per second by a single solr instance -- assuming a dedicated box? d) Our plan is to release a beta version Spring 2009. Should we plan on using Solr 1.2 or else move to solr 1.3 now? Any insights/thoughts/whitepapers will be greatly appreciated! Cheers, Bill
