We have a large (100M) index where we add about 1M new docs per day. We want to keep index at a constant size so the oldest ones are removed and/or archived each day (so index contains around 100 days of data). What is the best way to do this? We still want to keep older data in some archive index, not just delete it (so is it possible to export older segments, etc. into some other index?). If we have some daily job to delete old data, I assume we'd need to optimize the index to actually remove and free space, but that will require very large (and slow) replication after optimize which will probably not work out well for so large an index. Is there some way to shard the data or other best practice?
Thanks Bob
