Some very interesting scenarious cropping up here. Sounds to me like a minor architectural changes will be in order for them to be addressed.
What's the usual delay from stage one to stage 2. IF that were some kind of constant value, old versions could be kept around for approx doulbe that time. You'd need one of two things: A/ Last accessed by Stage one stored per record ~or~ B/ A regular MVCC system and garbage collection. Dennis Gearon Signature Warning ---------------- It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036' EARTH has a Right To Life, otherwise we all die. --- On Tue, 10/12/10, Jan Høydahl / Cominvent <jan....@cominvent.com> wrote: > From: Jan Høydahl / Cominvent <jan....@cominvent.com> > Subject: Re: LuceneRevolution - NoSQL: A comparison > To: solr-user@lucene.apache.org > Date: Tuesday, October 12, 2010, 9:11 AM > This is a different issue. You are > seeing the latency between master index update and > replication to slave(s). > Solve this by pointing your monitoring script directly to > slave instead of master. > > What this thread is about is a potential difference in > state during the execution of a single sharded query, not > due to master/slave but due to the index being updated > between phase 1 and phase 2. > > I'm pretty sure the 2nd phase to fetch doc-summaries goes > directly to same server as first phase. But what if you > stick a LB in between? Then perhaps the first phase may go > to master and second to slave? > > -- > Jan Høydahl, search solution architect > Cominvent AS - www.cominvent.com > > On 12. okt. 2010, at 17.12, Shawn Heisey wrote: > > > On 10/11/2010 6:32 PM, Peter Keegan wrote: > >> When Solr does a distributed search across shards, > it does this in 2 phases > >> (correct me if I'm wrong): > >> > >> 1. 1st query to get the docIds and facet counts > >> 2. 2nd query to retrieve the stored fields of the > top hits > >> > >> The problem here is that the index could change > between (1) and (2), so it's > >> not an atomic transaction. If the stored fields > were kept outside of Lucene, > >> only the first query would be necessary. However, > this would mean that the > >> external NoSQL data store would have to be > synchronized with the Lucene > >> index, which might present its own problems. (I'm > just throwing this out for > >> discussion) > > > > I've got a related issue that I have run into because > of my use of a load balancer. > > > > I have a total of seven shards, each of which has a > replica. I've got one set of machines set up as > brokers that have the shards parameter in the standard > request handler. Queries are sent to the load > balancer, which sends it to one of the brokers. The > shards parameter sends requests back to the load balancer to > be ultimately sent to an actual server. > > > > I have a monitoring script that retrieves the latest > document and alarms if it's older than ten minutes. > Something that happens on occasion: > > > > 1) An update is made to the master (happens every two > minutes). > > 2) Monitoring script requests newest document. > > 3) Initial request is sent to master, finds ID. > > 4) Second request is sent to the slave, document not > found. > > 5) Up to 15 seconds later, the slave replicates. > > > > I solved this problem by having the monitoring script > try several times on failure, waiting a few seconds on each > loop. Do I need to be terribly concerned about this > impacting real queries? > > > > I do not actually need to load balance, I have slave > servers purely for failover. Currently the load > balancer has a 3 to 1 weight ratio favoring the slaves, > which I plan to increase. At one time I had the master > set up as a backup rather than a lower weight target, but > haproxy seemed to take longer to recover from failures in > that mode. I will have to do some more comprehensive > testing. If there's a better solution than haproxy > that works with heartbeat, I can change that. > > > > Thanks, > > Shawn > > > >
