Mahmoud Almokadem [prog.mahm...@gmail.com] wrote: > I've the same index with a bit different schema and 200M documents, > installed on 3 r3.xlarge (30GB RAM, and 600 General Purpose SSD). The size > of index is about 1.5TB, have many updates every 5 minutes, complex queries > and faceting with response time of 100ms that is acceptable for us.
So you have Setup 1: 3 * (30GB RAM + 600GB SSD) for a total of 1.5TB index 200M docs. Acceptable performance. Setup 2: 3 * (60GB RAM + 1TB SSD + 500GB SSD) for a total of 3.3TB 350M docs. Poor performance. The only real difference, besides doubling everything, is the LVM? I understand why you find that to be the culprit, but from what I can read, the overhead should not be anywhere near enough to result in the performance drop you are describing. Could it be that some snapshotting or backup was running when you tested? Splitting your shards and doubling the number of machines, as you suggest, would result in Setup 3: 6 * (60GB RAM + 600GB SSD) for a total of 3.3TB 350M docs. which would be remarkable similar to your setup 1. I think that would be the next logical step, unless you can easily do a temporary boost of your IOPS. BTW: You are getting dangerously close to your storage limits here - it seems that a single large merge could make you run out of space. - Toke Eskildsen
