How does a node know it's fully joined to the ring?
I'm working on a bug report in my Deleting Compaction Strategy project. Some of the provided deletion strategies allow for a series of deletion rules to be added to a CQL table, before compaction begins, a snapshot of the current rules are read out of the CQL table and a deletion plan is produced that applies for the life of that compaction. The bug report (at https://github.com/protectwise/cassandra-util/issues/2) is that if compaction begins while a node is first starting up, before it has joined the ring, the attempt to read the rules will fail with an AssertionError (it's an attempt to call QueryProcessor.process at local quorum before the node has joined the ring). My question is, how can I detect when a node has fully joined the ring, and it's safe to call QueryProcessor.process without just trying, failing, and catching (and even then how would I know that it's because the node is still starting up vs some other source of problems)? I had thought that something might set a flag during the startup process to indicate that the node thinks it is fully started up, but as far as I see nothing like that is tracked. I took a stab at what this could look like here: https://github.com/protectwise/cassandra-util/blob/93dfb02157446e2e37427f1d4846c6039cc97099/deleting-compaction-strategy/src/main/java/com/protectwise/cassandra/retrospect/deletion/QueryHelper.java#L31-L36 This approach seems cumbersome and fragile, and I feel like I'm missing something. How else could I tell whether it's safe to call out to QueryProcessor.process yet? Thanks in advance, -e
Re: How does a node know it's fully joined to the ring?
Probably want to do something like SystemKeyspace.bootstrapComplete() On Thu, Jan 5, 2017 at 1:58 PM, Eric Stevens wrote: > I'm working on a bug report in my Deleting Compaction Strategy project. > > Some of the provided deletion strategies allow for a series of deletion > rules to be added to a CQL table, before compaction begins, a snapshot of > the current rules are read out of the CQL table and a deletion plan is > produced that applies for the life of that compaction. > > The bug report (at https://github.com/protectwise/cassandra-util/issues/2) > is that if compaction begins while a node is first starting up, before it > has joined the ring, the attempt to read the rules will fail with an > AssertionError (it's an attempt to call QueryProcessor.process at local > quorum before the node has joined the ring). > > My question is, how can I detect when a node has fully joined the ring, and > it's safe to call QueryProcessor.process without just trying, failing, and > catching (and even then how would I know that it's because the node is > still starting up vs some other source of problems)? I had thought that > something might set a flag during the startup process to indicate that the > node thinks it is fully started up, but as far as I see nothing like that > is tracked. > > I took a stab at what this could look like here: > https://github.com/protectwise/cassandra-util/blob/ > 93dfb02157446e2e37427f1d4846c6039cc97099/deleting- > compaction-strategy/src/main/java/com/protectwise/ > cassandra/retrospect/deletion/QueryHelper.java#L31-L36 > > This approach seems cumbersome and fragile, and I feel like I'm missing > something. How else could I tell whether it's safe to call out to > QueryProcessor.process yet? > > Thanks in advance, > -e >
Re: How does a node know it's fully joined to the ring?
Thanks for the quick response. Looking at it, I'm not sure if it's quite the right thing for me. Maybe I used the wrong terminology; it actually doesn't matter if the node is still bootstrapping, as long as it could query other nodes in the cluster. On Thu, Jan 5, 2017 at 1:03 PM Brandon Williams wrote: > Probably want to do something like SystemKeyspace.bootstrapComplete() > > On Thu, Jan 5, 2017 at 1:58 PM, Eric Stevens wrote: > > > I'm working on a bug report in my Deleting Compaction Strategy project. > > > > Some of the provided deletion strategies allow for a series of deletion > > rules to be added to a CQL table, before compaction begins, a snapshot of > > the current rules are read out of the CQL table and a deletion plan is > > produced that applies for the life of that compaction. > > > > The bug report (at > https://github.com/protectwise/cassandra-util/issues/2) > > is that if compaction begins while a node is first starting up, before it > > has joined the ring, the attempt to read the rules will fail with an > > AssertionError (it's an attempt to call QueryProcessor.process at local > > quorum before the node has joined the ring). > > > > My question is, how can I detect when a node has fully joined the ring, > and > > it's safe to call QueryProcessor.process without just trying, failing, > and > > catching (and even then how would I know that it's because the node is > > still starting up vs some other source of problems)? I had thought that > > something might set a flag during the startup process to indicate that > the > > node thinks it is fully started up, but as far as I see nothing like that > > is tracked. > > > > I took a stab at what this could look like here: > > https://github.com/protectwise/cassandra-util/blob/ > > 93dfb02157446e2e37427f1d4846c6039cc97099/deleting- > > compaction-strategy/src/main/java/com/protectwise/ > > cassandra/retrospect/deletion/QueryHelper.java#L31-L36 > > > > This approach seems cumbersome and fragile, and I feel like I'm missing > > something. How else could I tell whether it's safe to call out to > > QueryProcessor.process yet? > > > > Thanks in advance, > > -e > > >
Re: How does a node know it's fully joined to the ring?
We don't bind thrift/CQL until bootstrap is complete, so that's not possible. But, if you want to be sure, you can check StorageService.isRPCServerRunning and isNativeTransportRunning. On Thu, Jan 5, 2017 at 3:17 PM, Eric Stevens wrote: > Thanks for the quick response. > > Looking at it, I'm not sure if it's quite the right thing for me. Maybe I > used the wrong terminology; it actually doesn't matter if the node is still > bootstrapping, as long as it could query other nodes in the cluster. > > On Thu, Jan 5, 2017 at 1:03 PM Brandon Williams wrote: > > > Probably want to do something like SystemKeyspace.bootstrapComplete() > > > > On Thu, Jan 5, 2017 at 1:58 PM, Eric Stevens wrote: > > > > > I'm working on a bug report in my Deleting Compaction Strategy project. > > > > > > Some of the provided deletion strategies allow for a series of deletion > > > rules to be added to a CQL table, before compaction begins, a snapshot > of > > > the current rules are read out of the CQL table and a deletion plan is > > > produced that applies for the life of that compaction. > > > > > > The bug report (at > > https://github.com/protectwise/cassandra-util/issues/2) > > > is that if compaction begins while a node is first starting up, before > it > > > has joined the ring, the attempt to read the rules will fail with an > > > AssertionError (it's an attempt to call QueryProcessor.process at local > > > quorum before the node has joined the ring). > > > > > > My question is, how can I detect when a node has fully joined the ring, > > and > > > it's safe to call QueryProcessor.process without just trying, failing, > > and > > > catching (and even then how would I know that it's because the node is > > > still starting up vs some other source of problems)? I had thought > that > > > something might set a flag during the startup process to indicate that > > the > > > node thinks it is fully started up, but as far as I see nothing like > that > > > is tracked. > > > > > > I took a stab at what this could look like here: > > > https://github.com/protectwise/cassandra-util/blob/ > > > 93dfb02157446e2e37427f1d4846c6039cc97099/deleting- > > > compaction-strategy/src/main/java/com/protectwise/ > > > cassandra/retrospect/deletion/QueryHelper.java#L31-L36 > > > > > > This approach seems cumbersome and fragile, and I feel like I'm missing > > > something. How else could I tell whether it's safe to call out to > > > QueryProcessor.process yet? > > > > > > Thanks in advance, > > > -e > > > > > >
Re: How does a node know it's fully joined to the ring?
Perfect, thanks! On Thu, Jan 5, 2017 at 2:26 PM Brandon Williams wrote: > We don't bind thrift/CQL until bootstrap is complete, so that's not > possible. But, if you want to be sure, you can check > StorageService.isRPCServerRunning and isNativeTransportRunning. > > On Thu, Jan 5, 2017 at 3:17 PM, Eric Stevens wrote: > > > Thanks for the quick response. > > > > Looking at it, I'm not sure if it's quite the right thing for me. Maybe > I > > used the wrong terminology; it actually doesn't matter if the node is > still > > bootstrapping, as long as it could query other nodes in the cluster. > > > > On Thu, Jan 5, 2017 at 1:03 PM Brandon Williams > wrote: > > > > > Probably want to do something like SystemKeyspace.bootstrapComplete() > > > > > > On Thu, Jan 5, 2017 at 1:58 PM, Eric Stevens > wrote: > > > > > > > I'm working on a bug report in my Deleting Compaction Strategy > project. > > > > > > > > Some of the provided deletion strategies allow for a series of > deletion > > > > rules to be added to a CQL table, before compaction begins, a > snapshot > > of > > > > the current rules are read out of the CQL table and a deletion plan > is > > > > produced that applies for the life of that compaction. > > > > > > > > The bug report (at > > > https://github.com/protectwise/cassandra-util/issues/2) > > > > is that if compaction begins while a node is first starting up, > before > > it > > > > has joined the ring, the attempt to read the rules will fail with an > > > > AssertionError (it's an attempt to call QueryProcessor.process at > local > > > > quorum before the node has joined the ring). > > > > > > > > My question is, how can I detect when a node has fully joined the > ring, > > > and > > > > it's safe to call QueryProcessor.process without just trying, > failing, > > > and > > > > catching (and even then how would I know that it's because the node > is > > > > still starting up vs some other source of problems)? I had thought > > that > > > > something might set a flag during the startup process to indicate > that > > > the > > > > node thinks it is fully started up, but as far as I see nothing like > > that > > > > is tracked. > > > > > > > > I took a stab at what this could look like here: > > > > https://github.com/protectwise/cassandra-util/blob/ > > > > 93dfb02157446e2e37427f1d4846c6039cc97099/deleting- > > > > compaction-strategy/src/main/java/com/protectwise/ > > > > cassandra/retrospect/deletion/QueryHelper.java#L31-L36 > > > > > > > > This approach seems cumbersome and fragile, and I feel like I'm > missing > > > > something. How else could I tell whether it's safe to call out to > > > > QueryProcessor.process yet? > > > > > > > > Thanks in advance, > > > > -e > > > > > > > > > >
