Streamsession's timeout is not reasonable
As the preview message, see below, after some hours rebuild failure, we found
it is due to timeout.
The transfer side incoming socket read timeout( as
streaming_socket_timeout_in_ms default one hours), then the whole
streamsession fail.
As rebuild going the transfer rate will slow down, the transferring file can't
accomplish in the timeout time. The transfer side didn't receive any byte
(expected RECEIVED message), then the incoming socket raised timeout.
As incoming and outgoing belong to the streamsession, To determine timeout，we
can't test incoming alone, as outgoing is streaming(transferring file is
continue especially large file, low speed). In other words, when file is
transferring, we can't raise timeout.
Question again:
Will re-rebuild rebuild all rang of tokens which belong to the node or just
rebuild the rest rang of tokens from last rebuild.(since last rebuild we get
some data).
Please excuse me for my poor English.
===
At 2015-11-21 01:07:05, "wateray" wrote:
>we want deploy one more data-center for data safe.
>As we rebuild one node's data from the old DC, after some hours rebuild
>failure due to network fault.
>I can restart rebuild surely,but I'm afraid restart rebuild,
>is it rebuild all rang of tokens which belong to the node or just rebuild the
>rest rang of tokens from last rebuild.(since last rebuild we get some data).
>
>As I view the source, I see this code.
>
>class RangeStreamer method getRangeFetchMap
>
>private static Multimap>
>getRangeFetchMap(Multimap, InetAddress> rangesWithSources,
>Collection sourceFilters, String keyspace)
>{
>Multimap> rangeFetchMapMap =
> HashMultimap.create();
>for (Range range : rangesWithSources.keySet())
>{
>boolean foundSource = false;
>
>outer:
>for (InetAddress address : rangesWithSources.get(range))
>{
>if (address.equals(FBUtilities.getBroadcastAddress()))
>{
>// If localhost is a source, we have found one, but we
> don't add it to the map to avoid streaming locally
>foundSource = true;
>continue;
>}
>
>for (ISourceFilter filter : sourceFilters)
>{
>if (!filter.shouldInclude(address))
>continue outer;
>}
>
>rangeFetchMapMap.put(address, range);
>foundSource = true;
>break; // ensure we only stream from one other node for each
> range
>}
>
>if (!foundSource)
>throw new IllegalStateException("unable to find sufficient
> sources for streaming range " + range + " in keyspace " + keyspace);
>}
>
>return rangeFetchMapMap;
>}
>
>The bold lines ，when found the address is localhost, It continue to find
>others and then put into the rangeFetchMapMap。
>I think the continue key word should be break, if it just want rebuild the
>data it doesn't have. Is it right?
>
>
>Best regards!
>
>
>
>
>
>
Re: UML sequence diagrams on Wiki for explaining read/write path
I just remembered... the new Materialized View support in 3.0 - writes to the materialized views get triggered when a write occurs to the base table. That needs to be in the write path flow/description as well. -- Jack Krupansky On Mon, Nov 30, 2015 at 9:30 PM, Michael Edge wrote: > Thanks for the feedback guys. I've made the updates. > > On 1 December 2015 at 00:56, Jack Krupansky > wrote: > > > Great stuff! > > > > You wrote "When the replica node comes back online the coordinator node > > will send the data to the replica node", which is partially true - if the > > replica comes back online within the timeout window of three hours. So, > you > > probably want to say something like: > > > > "If the replica node comes back online within the hinted handoff window > the > > coordinator node will send the data to the replica node, otherwise the > > replica node will need to be repaired." > > > > And maybe mention the configuration of the window. Change "The data is > > stored for a default period of 3 hours" to "The data is stored for a > > default period of 3 hours, configurable using the > > max_hint_window_in_ms property > > in cassandra.yaml." > > > > -- Jack Krupansky > > > > On Mon, Nov 30, 2015 at 2:24 AM, Michael Edge > > wrote: > > > > > Write path docs updated on Wiki - please review diagram/text and let me > > > have your comments (or update text in place). > > > > > > https://wiki.apache.org/cassandra/WritePathForUsers > > > > > > Cheers, > > > > > > Michael > > > > > > On 26 November 2015 at 11:25, Michael Shuler > > > wrote: > > > > > > > On 11/25/2015 07:36 PM, Michael Edge wrote: > > > > > I'd like to update the read/write path description on the wiki (see > > > link > > > > > below) by adding a couple of UML sequence diagrams I drew a while > > ago. > > > I > > > > > think they are much better than long textual descriptions for > > > describing > > > > > the order of operations on components. Before publishing them I'd > > > prefer > > > > a > > > > > knowledgeable volunteer to review them to ensure they are accurate. > > > > > > > > > > Let me know if you're interested and I'll send you a copy. > > > > > > > > > > https://wiki.apache.org/cassandra/ArchitectureOverview > > > > > > > > I don't think the mailing list likes attachments, so throw them up > on a > > > > web server somewhere and send the list link(s) to your diagrams. > > > > Alternatively, go ahead and post them on the wiki and ask for > feedback. > > > > > > > > -- > > > > Kind regards, > > > > Michael > > > > > > > > > >
Re: UML sequence diagrams on Wiki for explaining read/write path
Yes - good point. I've updated the text for the write path page below - could you please review? Once my understanding is correct I'll update the sequence diagram to match. https://wiki.apache.org/cassandra/WritePathForUsers On 1 December 2015 at 22:51, Jack Krupansky wrote: > I just remembered... the new Materialized View support in 3.0 - writes to > the materialized views get triggered when a write occurs to the base table. > That needs to be in the write path flow/description as well. > > -- Jack Krupansky > > On Mon, Nov 30, 2015 at 9:30 PM, Michael Edge > wrote: > > > Thanks for the feedback guys. I've made the updates. > > > > On 1 December 2015 at 00:56, Jack Krupansky > > wrote: > > > > > Great stuff! > > > > > > You wrote "When the replica node comes back online the coordinator node > > > will send the data to the replica node", which is partially true - if > the > > > replica comes back online within the timeout window of three hours. So, > > you > > > probably want to say something like: > > > > > > "If the replica node comes back online within the hinted handoff window > > the > > > coordinator node will send the data to the replica node, otherwise the > > > replica node will need to be repaired." > > > > > > And maybe mention the configuration of the window. Change "The data is > > > stored for a default period of 3 hours" to "The data is stored for a > > > default period of 3 hours, configurable using the > > > max_hint_window_in_ms property > > > in cassandra.yaml." > > > > > > -- Jack Krupansky > > > > > > On Mon, Nov 30, 2015 at 2:24 AM, Michael Edge > > > wrote: > > > > > > > Write path docs updated on Wiki - please review diagram/text and let > me > > > > have your comments (or update text in place). > > > > > > > > https://wiki.apache.org/cassandra/WritePathForUsers > > > > > > > > Cheers, > > > > > > > > Michael > > > > > > > > On 26 November 2015 at 11:25, Michael Shuler > > > > > wrote: > > > > > > > > > On 11/25/2015 07:36 PM, Michael Edge wrote: > > > > > > I'd like to update the read/write path description on the wiki > (see > > > > link > > > > > > below) by adding a couple of UML sequence diagrams I drew a while > > > ago. > > > > I > > > > > > think they are much better than long textual descriptions for > > > > describing > > > > > > the order of operations on components. Before publishing them I'd > > > > prefer > > > > > a > > > > > > knowledgeable volunteer to review them to ensure they are > accurate. > > > > > > > > > > > > Let me know if you're interested and I'll send you a copy. > > > > > > > > > > > > https://wiki.apache.org/cassandra/ArchitectureOverview > > > > > > > > > > I don't think the mailing list likes attachments, so throw them up > > on a > > > > > web server somewhere and send the list link(s) to your diagrams. > > > > > Alternatively, go ahead and post them on the wiki and ask for > > feedback. > > > > > > > > > > -- > > > > > Kind regards, > > > > > Michael > > > > > > > > > > > > > > >
