Re: Inconsistent results for facet queries

2017-10-12 Thread Chris Ulicny
I'm not sure if that method is viable for reindexing and fetching the whole collection at once for us, but unless there is something inherent in that process which happens at the collection level, we could do it a few shards at a time since it is a multi-tenant setup. I'll see if we can setup a sm

Re: Inconsistent results for facet queries

2017-10-12 Thread Erick Erickson
(1) It doesn't matter whether it "affect only segments being merged". You can't get accurate information if different segments have different expectations. (2) I strongly doubt it. The problem is that the "tainted" segments' meta-data is still read when merging. If the segment consisted of _only_

Re: Inconsistent results for facet queries

2017-10-12 Thread Chris Ulicny
We tested the query on all replicas for the given shard, and they all have the same issue. So deleting and adding another replica won't fix the problem since the leader is exhibiting the behavior as well. I believe the second replica was moved (new one added, old one deleted) between nodes and so w

Re: Inconsistent results for facet queries

2017-10-12 Thread Erick Erickson
Never mind. Anything that didn't merge old segments, just threw them away when empty (which was my idea) would possibly require as much disk space as the index currently occupied, so doesn't help your disk-constrained situation. Best, Erick On Thu, Oct 12, 2017 at 8:06 AM, Erick Erickson wrote:

Re: Inconsistent results for facet queries

2017-10-12 Thread Erick Erickson
If it's _only_ on a particular replica, here's what you could do: Just DELETEREPLICA on it, then ADDREPLICA to bring it back. You can define the "node" parameter on ADDREPLICA to get it back on the same node. Then the normal replication process would pull the entire index down from the leader. My

Re: Inconsistent results for facet queries

2017-10-12 Thread Chris Ulicny
I thought that decision would come back to bite us somehow. At the time, we didn't have enough space available to do a fresh reindex alongside the old collection, so the only course of action available was to index over the old one, and the vast majority of its use worked as expected. We're planni

Re: Inconsistent results for facet queries

2017-10-11 Thread Erick Erickson
bq: ...but the collection wasn't emptied first This is what I'd suspect is the problem. Here's the issue: Segments aren't merged identically on all replicas. So at some point you had this field indexed without docValues, changed that and re-indexed. But the segment merging could "read" the fir

Inconsistent results for facet queries

2017-10-11 Thread Chris Ulicny
Hi, We've run into a strange issue with our deployment of solrcloud 6.3.0. Essentially, a standard facet query on a string field usually comes back empty when it shouldn't. However, every now and again the query actually returns the correct values. This is only affecting a single shard in our setu