Hi Luke, Sorry for the late response and thanks for following up. I haven't seen it happen since. At this point I'm going to wait and see if it happens again and hopefully get more details about what might be causing it.
Daniel On Thu, Feb 9, 2017 at 1:02 PM, Luke Bakken <[email protected]> wrote: > Hi Daniel - > > I don't have any ideas at this point. Has this scenario happened again? > > -- > Luke Bakken > Engineer > [email protected] > > > On Wed, Jan 25, 2017 at 2:11 PM, Daniel Miller <[email protected]> wrote: > > Thanks for the quick response, Luke. > > > > There is nothing unusual about the keys. The format is a name + UUID + > some > > other random URL-encoded charaters, like most other keys in our cluster. > > > > There are no errors near the time of the incident in any of the logs (the > > last [error] is from over a month before). I see lots of messages like > this > > in console.log: > > > > /var/log/riak/console.log > > 2017-01-20 15:38:10.184 [info] > > <0.22902.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 2 keys > during > > active anti-entropy exchange of > > {776422744832042175295707567380525354192214163456,3} between > > {776422744832042175295707567380525354192214163456,'riak- > [email protected]'} > > and > > {822094670998632891489572718402909198556462055424,'riak- > [email protected]'} > > 2017-01-20 15:40:39.640 [info] > > <0.21789.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 1 keys > during > > active anti-entropy exchange of > > {936274486415109681974235595958868809467081785344,3} between > > {959110449498405040071168171470060731649205731328,'riak- > [email protected]'} > > and > > {981946412581700398168100746981252653831329677312,'riak- > [email protected]'} > > 2017-01-20 15:46:40.918 [info] > > <0.13986.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 2 keys > during > > active anti-entropy exchange of > > {662242929415565384811044689824565743281594433536,3} between > > {685078892498860742907977265335757665463718379520,'riak- > [email protected]'} > > and > > {707914855582156101004909840846949587645842325504,'riak- > [email protected]'} > > 2017-01-20 15:48:25.597 [info] > > <0.29943.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 2 keys > during > > active anti-entropy exchange of > > {776422744832042175295707567380525354192214163456,3} between > > {776422744832042175295707567380525354192214163456,'riak- > [email protected]'} > > and > > {799258707915337533392640142891717276374338109440,'riak- > [email protected]'} > > > > Thanks! > > Daniel > > > > > > > > On Wed, Jan 25, 2017 at 9:45 AM, Luke Bakken <[email protected]> wrote: > >> > >> Hi Daniel - > >> > >> This is a strange scenario. I recommend looking at all of the log > >> files for "[error]" or other entries at about the same time as these > >> PUTs or 404 responses. > >> > >> Is there anything unusual about the key being used? > >> -- > >> Luke Bakken > >> Engineer > >> [email protected] > >> > >> > >> On Wed, Jan 25, 2017 at 6:40 AM, Daniel Miller <[email protected]> > wrote: > >> > I have a 9-node Riak CS cluster that has been working flawlessly for > >> > about 3 > >> > months. The cluster configuration, including backend and bucket > >> > parameters > >> > such as N-value are using default settings. I'm using the S3 API to > >> > communicate with the cluster. > >> > > >> > Within the past week I had an issue where two objects were PUT > resulting > >> > in > >> > a 200 (success) response, but all subsequent GET requests for those > two > >> > keys > >> > return status of 404 (not found). Other than the fact that they are > now > >> > missing, there was nothing out of the ordinary with these particular > to > >> > PUTs. Maybe I'm missing something, but this seems like a scenario that > >> > should never happen. All information included here about PUTs and GETs > >> > comes > >> > from reviewing the CS access logs. Both objects were PUT on the same > >> > node, > >> > however GET requests returning 404 have been observed on all nodes. > >> > There is > >> > plenty of other traffic on the cluster involving GETs and PUTs that > are > >> > not > >> > failing. I'm unsure of how to troubleshoot further to find out what > may > >> > have > >> > happened to those objects and why they are now missing. What is the > best > >> > approach to figure out why an object that was successfully PUT seems > to > >> > be > >> > missing? > >> > > >> > Thanks! > >> > Daniel Miller > >> > > >> > _______________________________________________ > >> > riak-users mailing list > >> > [email protected] > >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >> > > > > > >
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
