Otis Gospodnetic <otis_gospodne...@yahoo.com> wrote on 01/22/2010 12:20:45 AM: > I'm missing the bigger context of this thread here, but from the > snippet below - sure, commits cause in-memory index to get written > to disk, that causes some IO, and that *could* affect search *if* > queries are running on the same box. When index and/or query volume > is high, one typically puts indexing and searching on different servers.
After some more research, I realize that what we're trying to do is essentially near-real-time processing. I have data collection that is near-real-time and I'm trying to avoid arbitrary delays pushing the data into the index so that the data collection doesn't stall. On the search side, we don't have a lot of search traffic but would like it to be responsive when it comes in. We also dynamically purge old data to keep the storage requirements within limits. So, basically I'm trying to have the system tuned so that this all works well :-) I'm trying to keep search on a single system to keep the costs down as well. One thing I'm trying now is to put an intermediary in so that updates can be asynchronous. Then my data collection processes can continue without waiting for unpredictable index merges. Thanks, Jerry
