Hi Stephen, Thanks for the update.
Regarding SOLR-9527 - I think we need a unit test for verifying "createNodeSet" functionality. I will spend some time on it in next couple of days. Also regarding #2, i also found similar issue (doc count mismatch after restore) while testing with a large collection (~50GB index size). I have opened SOLR-9598 to track this. Please take a look and comment if you have any insight. -Hrishikesh On Tue, Nov 8, 2016 at 12:54 PM, Stephen Weiss <steve.we...@wgsn.com> wrote: > Just wanted to note that we tested out the patch from SOLR-9527 and it > worked perfectly for the balancing issue - thank you so much for that! > > As for issue #2, we've resorted to doing a hard commit, stopping all > indexing against the index, and then taking the backup, and we have a > reasonably good success rate with that. The system is set up to > automatically delete and retry the backup/restore process if the cores > don't match, so that's allowed us to smooth over that problem and get this > process out into production. We've been using it for several weeks now > without any major issue! > > We just looked because Solr 6.3 was out, and wanted to know if we could > upgrade without patching again, but it appears this ticket hasn't gone > anywhere yet. I know one users' testing is probably not enough, but given > that it seems the patch works just fine, are there any plans to merge it > into release yet? > > -- > Steve > > On Tue, Oct 4, 2016 at 6:46 PM, Stephen Lewis <sle...@panopto.com<mailto: > sle...@panopto.com>> wrote: > Hi All, > > I have been experiencing error#1 too with the current branch_6_2 build. I > started noticing after I applied my patch to that branch< > https://issues.apache.org/jira/browse/SOLR-9527> (on issue #2), but it > appears to occur without the patch as well. I haven't seen this issue with > solr 6.1.0 despite extensive testing. I haven't confirmed if this occurs on > the official 6.2.0 release build. I will try to confirm and gather more > data soon. > > As with Stephen Weiss, I also am not seeing any errors logged in the index > after backup and the task is marked as succeeded. However, after each > backup which is missing a large amount of data, the restore command fails, > in the sense that the collection is created, but the initialized cores are > blank and the logs contain errors about "incomplete segments". I will try > to research further and get back with more data soon. > > > > On Mon, Sep 26, 2016 at 11:26 AM, Hrishikesh Gadre <gadre.s...@gmail.com > <mailto:gadre.s...@gmail.com>> wrote: > Hi Stephen, > > regarding #1, can you verify following steps during backup/restore? > > - Before backup command, make sure to run a "hard" commit on the original > collection. The backup operation will capture only hard committed data. > - After restore command, check the Solr web UI to verify that all replicas > of the new (or restored) collection are in the "active" state. During my > testing, I found that when one or more replicas are in "recovery" state, > the doc count of the restored collection doesn't match the doc count of the > original collection. But after the recovery is complete, the doc counts > match. I will file a JIRA to fix this issue. > > Thanks > Hrishikesh > > On Mon, Sep 26, 2016 at 9:34 AM, Stephen Weiss <steve.we...@wgsn.com > <mailto:steve.we...@wgsn.com>> wrote: > > > #2 - that's great news. I'll try to patch it in and test it out. > > > > #1 - In all cases, the backup and restore both appear successful. There > > are no failure messages for any of the shards, no warnings, etc - I > didn't > > even realize at first that data was missing until I noticed differences > in > > some of the query results when we were testing. Either manual restore of > > the data or using the restore API (with all data on one node), we see the > > same, so I think it's more a problem in the backup process than the > restore > > process. > > > > If there's any kind of debugging output we can provide that can help > solve > > this, let me know. > > > > -- > > Steve > > > > On Sun, Sep 25, 2016 at 7:17 PM, Hrishikesh Gadre <gadre.s...@gmail.com > <mailto:gadre.s...@gmail.com>> > > wrote: > > > >> Hi Steve, > >> > >> Regarding the 2nd issue, a JIRA is already created and patch is uploaded > >> (SOLR-9527). Can someone review and commit the patch? > >> > >> Regarding 1st issue, does backup command succeeds? Also do you see any > >> warning/error log messages? How about the restore command? > >> > >> Thanks > >> Hrishikesh > >> > >> > >> > >> On Sat, Sep 24, 2016 at 12:14 PM, Stephen Weiss <steve.we...@wgsn.com > <mailto:steve.we...@wgsn.com>> > >> wrote: > >> > >>> Hi everyone, > >>> > >>> We're very excited about SolrCloud's new backup / restore collection > >>> APIs, which should introduce some major new efficiencies into our > indexing > >>> workflow. Unfortunately, we've run into some snags with it that are > >>> preventing us from moving into production. I was hoping someone on the > >>> list could help. > >>> > >>> 1) Data inconsistencies > >>> > >>> There seems to be a problem getting all the data consistently. > >>> Sometimes, the backup will contain all of the data in the collection, > and > >>> sometimes, large portions of the collection (as much as 40%) will be > >>> missing. We haven't quite figured out what might cause this yet, > although > >>> one thing I've noticed is the chances of success are greater when we > are > >>> only backing up one collection at a time. Unfortunately, for our > workflow, > >>> it will be difficult to make that work, and there still doesn't seem > to be > >>> a guarantee of success either way. > >>> > >>> 2) Shards are not distributed > >>> > >>> To make matters worse, for some reason, any type of restore operation > >>> always seems to put all shards of the collection on the same node. > We've > >>> tried setting maxShardsPerNode to 1 in the restore command, but this > has no > >>> effect. We are seeing the same behavior on both 6.1 and 6.2.1. No > matter > >>> what we do, all the shards always go to the same node - and it's not > even > >>> the node that we execute the restore request on, but oddly enough, a > >>> totally different node, and always the same one (the 4th one). It > should > >>> be noted that all nodes of our 8 node cloud are up and totally > functional > >>> when this happens. > >>> > >>> To work around this, we wrote up a quick script to create an empty > >>> collection, which always distributes itself across the cloud quite well > >>> (another indication that there's nothing wrong with the nodes > themselves), > >>> and then we rsync the individual shards' data into the empty shards and > >>> reload the collection. This works fine, however, because of the data > >>> inconsistencies mentioned above, we can't really move forward anyway. > >>> > >>> > >>> Problem #2, we have a reasonable workaround for, but problem #1 we do > >>> not. If anyone has any thoughts about either of these problems, I > would be > >>> very grateful. Thanks! > >>> > >>> -- > >>> Steve > >>> > >>> ________________________________ > >>> > >>> WGSN is a global foresight business. Our experts provide deep insight > >>> and analysis of consumer, fashion and design trends. We inspire our > clients > >>> to plan and trade their range with unparalleled confidence and > accuracy. > >>> Together, we Create Tomorrow. > >>> > >>> WGSN<http://www.wgsn.com/> is part of WGSN Limited, comprising of > >>> market-leading products including WGSN.com<http://www.wgsn.com>, WGSN > >>> Lifestyle & Interiors<http://www.wgsn.com/en/lifestyle-interiors>, > WGSN > >>> INstock<http://www.wgsninstock.com/>, WGSN StyleTrial< > >>> http://www.wgsn.com/en/styletrial/> and WGSN Mindset< > >>> http://www.wgsn.com/en/services/consultancy/>, our bespoke consultancy > >>> services. > >>> > >>> The information in or attached to this email is confidential and may be > >>> legally privileged. If you are not the intended recipient of this > message, > >>> any use, disclosure, copying, distribution or any action taken in > reliance > >>> on it is prohibited and may be unlawful. If you have received this > message > >>> in error, please notify the sender immediately by return email and > delete > >>> this message and any copies from your computer and network. WGSN does > not > >>> warrant that this email and any attachments are free from viruses and > >>> accepts no liability for any loss resulting from infected email > >>> transmissions. > >>> > >>> WGSN reserves the right to monitor all email through its networks. Any > >>> views expressed may be those of the originator and not necessarily of > WGSN. > >>> WGSN is powered by Ascential plc<http://www.ascential.com>, which > >>> transforms knowledge businesses to deliver exceptional performance. > >>> > >>> Please be advised all phone calls may be recorded for training and > >>> quality purposes and by accepting and/or making calls from and/or to > us you > >>> acknowledge and agree to calls being recorded. > >>> > >>> WGSN Limited, Company number 4858491 > >>> > >>> registered address: > >>> > >>> Ascential plc, The Prow, 1 Wilder Walk, London W1B 5AP > >>> > >>> WGSN Inc., tax ID 04-3851246, registered office c/o National Registered > >>> Agents, Inc., 160 Greentree Drive, Suite 101, Dover DE 19904, United > States > >>> > >>> 4C Serviços de Informação Ltda., CNPJ/MF (Taxpayer's Register): > >>> 15.536.968/0001-04, Address: Avenida Cidade Jardim, 377, 7˚ andar CEP > >>> 01453-000, Itaim Bibi, São Paulo > >>> > >>> 4C Business Information Consulting (Shanghai) Co., Ltd, > >>> 富新商务信息咨询（上海）有限公司, registered address Unit 4810/4811, 48/F Tower 1, > Grand > >>> Gateway, 1 Hong Qiao Road, Xuhui District, Shanghai > >>> > >> > >> > > > > ------------------------------ > > > > > WGSN is a global foresight business. Our experts provide deep insight and > > analysis of consumer, fashion and design trends. We inspire our clients > to > > plan and trade their range with unparalleled confidence and accuracy. > > Together, we Create Tomorrow. > > > > WGSN <http://www.wgsn.com/> is part of WGSN Limited, comprising of > > market-leading products including WGSN.com <http://www.wgsn.com>, WGSN > > Lifestyle & Interiors <http://www.wgsn.com/en/lifestyle-interiors>, WGSN > > INstock <http://www.wgsninstock.com/>, WGSN StyleTrial > > <http://www.wgsn.com/en/styletrial/> and WGSN Mindset > > <http://www.wgsn.com/en/services/consultancy/>, our bespoke consultancy > > services. > > > > The information in or attached to this email is confidential and may be > > legally privileged. If you are not the intended recipient of this > message, > > any use, disclosure, copying, distribution or any action taken in > reliance > > on it is prohibited and may be unlawful. If you have received this > message > > in error, please notify the sender immediately by return email and delete > > this message and any copies from your computer and network. WGSN does not > > warrant that this email and any attachments are free from viruses and > > accepts no liability for any loss resulting from infected email > > transmissions. > > > > WGSN reserves the right to monitor all email through its networks. Any > > views expressed may be those of the originator and not necessarily of > WGSN. > > WGSN is powered by Ascential plc <http://www.ascential.com>, which > > transforms knowledge businesses to deliver exceptional performance. > > > > Please be advised all phone calls may be recorded for training and > quality > > purposes and by accepting and/or making calls from and/or to us you > > acknowledge and agree to calls being recorded. > > > > WGSN Limited, Company number 4858491 > > > > registered address: > > > > Ascential plc, The Prow, 1 Wilder Walk, London W1B 5AP > > > > WGSN Inc., tax ID 04-3851246, registered office c/o National Registered > > Agents, Inc., 160 Greentree Drive, Suite 101, Dover DE 19904, United > States > > > > 4C Serviços de Informação Ltda., CNPJ/MF (Taxpayer's Register): > > 15.536.968/0001-04, Address: Avenida Cidade Jardim, 377, 7˚ andar CEP > > 01453-000, Itaim Bibi, São Paulo > > > > 4C Business Information Consulting (Shanghai) Co., Ltd, 富新商务信息咨询（上海）有限公司, > > registered address Unit 4810/4811, 48/F Tower 1, Grand Gateway, 1 Hong > Qiao > > Road, Xuhui District, Shanghai > > > > > > -- > Stephen > > (206)753-9320<tel:%28206%29753-9320> > stephen-lewis.net<http://stephen-lewis.net> > > > ________________________________ > > WGSN is a global foresight business. Our experts provide deep insight and > analysis of consumer, fashion and design trends. We inspire our clients to > plan and trade their range with unparalleled confidence and accuracy. > Together, we Create Tomorrow. > > WGSN<http://www.wgsn.com/> is part of WGSN Limited, comprising of > market-leading products including WGSN.com<http://www.wgsn.com>, WGSN > Lifestyle & Interiors<http://www.wgsn.com/en/lifestyle-interiors>, WGSN > INstock<http://www.wgsninstock.com/>, WGSN StyleTrial<http://www.wgsn. > com/en/styletrial/> and WGSN Mindset<http://www.wgsn.com/ > en/services/consultancy/>, our bespoke consultancy services. > > The information in or attached to this email is confidential and may be > legally privileged. If you are not the intended recipient of this message, > any use, disclosure, copying, distribution or any action taken in reliance > on it is prohibited and may be unlawful. If you have received this message > in error, please notify the sender immediately by return email and delete > this message and any copies from your computer and network. WGSN does not > warrant that this email and any attachments are free from viruses and > accepts no liability for any loss resulting from infected email > transmissions. > > WGSN reserves the right to monitor all email through its networks. Any > views expressed may be those of the originator and not necessarily of WGSN. > WGSN is powered by Ascential plc<http://www.ascential.com>, which > transforms knowledge businesses to deliver exceptional performance. > > Please be advised all phone calls may be recorded for training and quality > purposes and by accepting and/or making calls from and/or to us you > acknowledge and agree to calls being recorded. > > WGSN Limited, Company number 4858491 > > registered address: > > Ascential plc, The Prow, 1 Wilder Walk, London W1B 5AP > > WGSN Inc., tax ID 04-3851246, registered office c/o National Registered > Agents, Inc., 160 Greentree Drive, Suite 101, Dover DE 19904, United States > > 4C Serviços de Informação Ltda., CNPJ/MF (Taxpayer's Register): > 15.536.968/0001-04, Address: Avenida Cidade Jardim, 377, 7˚ andar CEP > 01453-000, Itaim Bibi, São Paulo > > 4C Business Information Consulting (Shanghai) Co., Ltd, 富新商务信息咨询（上海）有限公司, > registered address Unit 4810/4811, 48/F Tower 1, Grand Gateway, 1 Hong Qiao > Road, Xuhui District, Shanghai >
