On 3/3/2017 10:17 AM, Sales wrote: > I am not sure how best to handle this. We use the data import handle to > re-sync all our data on a daily basis, takes 1-2 hours depending on system > load. It is set up to commit at the end, so, the old index remains until it’s > done, and, we lose no access while the import is happening. > > But, we now want to update certain fields in the index, but still regen > daily. So, it would seem we might need to autocommit, and, soft commit > potentially. When we enabled those, during the index, the data disappeared > since it kept soft committing during the import process, I see no way to > avoid soft commits during the import. But soft commits would appear to be > needed for the (non import) updates to the index. > > I realize the import could happen while an update is done, but we can > actually avoid those. So, that is not an issue (one or two might go through, > but, we will redo those updates once the index is done, that part is all > handled.
Erick's solution of using aliases to swap a live index and a build index is one very good way to go. It does involve some additional complexity that you may not be ready for. Only you will know whether that's something you can implement easily. Collection aliasing was implemented in Solr 4.2 by SOLR-4497, so 4.10 should definitely have it. You have indicated that you have a way to avoid doing updates during the full import. Because of this, you do have another option that is likely much easier for you to implement: Set the "commitWithin" parameter on each update request. This works almost identically to autoSoftCommit, but only after a request is made. As long as there are never any of these updates during a full import, these commits cannot affect that import. No matter what is happening, you should have autoCommit (not autoSoftCommit) configured with openSearcher set to false. This will ensure transaction log rollover, without affecting change visibility. I recommend a maxTime of one to five minutes for this. You'll see 15 seconds as the recommended value in many places. https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Thanks Shawn
