Newbie Question: Master Index or 100s Small Index
We run a central database of 14M (and growing) photos with dates, captions, keywords, etc. We currently upgrading from old Lucene Servers to latest Solr running with a couple of dedicated servers (6 core, 36GB, 500SSD). Planning on using Solr Cloud. We take in thousands of changes each day (big and small) so indexing may be a bigger problem than searching. My question is an architecture one. These photos are currently indexed and searched in three ways. 1: The 14M pictures from above are split into a few hundred indexes that feed a single website. This means index sizes of between 100 and 500,000 entries each. 2: 95% of these same photos are also wanted for searching on a global site. Index size of 12M plus. 3: 80% of these same photos are also required for smaller group sites. Index sizes of between 400K and 4M. We currently make changes the single indexes and then merge into groups and global. Due to the size of the numbers, is it worth changing or not. Is it quicker/better to just have one big 14M index and filter the complexities for each website or is it better to still maintain hundreds of indexes so we are searching smaller one. Bear in mind, we get thousands of changes a day PLUS very busy search servers. Thanks Col -- View this message in context: http://lucene.472066.n3.nabble.com/Newbie-Question-Master-Index-or-100s-Small-Index-tp4125407.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Newbie Question: Master Index or 100s Small Index
Hi Toke Thanks for replying. My question is really regarding index architecture. One big or many small (with merged big ones) We probably get 5-10K photos added each day. Others are updated, some are deleted. Updates need to happen quite fast (e.g. within minutes of our Databases receiving them). In terms of bytes, each photo has a up to 1.5KB of data. Special requirements are search by date range, text, date range and text. Plus some boolean filtering. All results can be sorted by date or filename. -- View this message in context: http://lucene.472066.n3.nabble.com/Newbie-Question-Master-Index-or-100s-Small-Index-tp4125407p4125429.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Newbie Question: Master Index or 100s Small Index
Hi Toke Our current configuration Lucene 2.(something) with RAILO/CFML app server. 10K drives, Quad Core, 16GB, Two servers. But the indexing and searching are starting to fail and our developer is no longer with us so it is quicker to rebuild than fix all the code. Our existing config is lots of indexes with merges into the larger ones. They are still running very fast but indexing is causing us issues. Thanks -- View this message in context: http://lucene.472066.n3.nabble.com/Newbie-Question-Master-Index-or-100s-Small-Index-tp4125407p4125447.html Sent from the Solr - User mailing list archive at Nabble.com.
