Materialized Views during node replace
Hi We recently replaced a down node containing 90 GB of data. The streaming process completed within 1 hour as we could see a sufficient amount of data being streamed on the data disk but the node remained in UJ state and it took 7 more hours to move to UN state. We could not figure out anything from system logs. Our DB schema contains 3 materialized views over the main table, could that be the reason for this? If yes how can we be sure of it and if not then how do we figure out what went wrong ? Thanks! -- Shaurya Gupta
[DISCUSSION] Should we mark DROP COMPACT STORAGE as experimental
Hi everybody, There are a significant amount of issues with DROP COMPACT STORAGE that can be pretty surprising for users. To name a few: * Some hidden columns will show up changing the resultset returned for wildcard queries * As COMPACT tables did not have primary key liveness there empty rows inserted AFTER the ALTER will be returned whereas the one inserted before the ALTER will not. * Also due to the lack of primary key liveness the amount of SSTables being read will increase resulting in slower queries * After DROP COMPACT it becomes possible to ALTER the table in a way that makes all the row disappears * There is a loss of functionality around null clustering when dropping compact storage (CASSANDRA-16069) In my opinion DROP COMPACT STORAGE is not ready for production use unless users fully understand what they are doing. By consequence, I am wondering if we should not mark it as experimental as we did for the Materialized Views (CASSANDRA-13959). What is your opinion?
Re: Materialized Views during node replace
I can't recall if view builds affect the bootstrap but index builds definitely do. It's a bit late now but you could've run nodetool compactionstats to see what tasks were running at the time and those need to complete before the bootstrap is considered complete for the node to go into UP/NORMAL status. Cheers!
Re: Obfuscation of passwords in audit loging, in or not in 4.0?
> On 4 Jun 2021, at 03:44, Jonathan Koppenhofer wrote: > > +1 to this being a serious bug. As a large user, if we used internal > passwords, this would completely prevent me from using Cassandra native > audit log capabilities. Disabling DCL is not a great option, as DCL is > probably the most needed auditable event. > > If this is on by default (not sure of default settings) I also assume this > would be classified as an immediate CVE... Right? +1, I would think so too. Shipping a brand new, non-experimental feature with a security hole like this feels counter to our goal of releases being prod ready in .0, so I'm +1 on including it in an rc/ga > I don't directly > contribute, so I can't talk too much, but I can't see how 4.0.0 could go GA > with this. > > > On Thu, Jun 3, 2021, 7:24 PM Sumanth Pasupuleti < > sumanth.pasupuleti...@gmail.com> wrote: > >>> I am on the side of "this sounds like a really bad bug" for the audit >> pieces, maybe less so than FQL. Anyone using audit for real probably has >> meaningful audit requirements, which means they're in an industry where >> they get audited for security, which means logging passwords is a big deal. >> >> +1. Given we are shipping audit logging feature for the first time with >> 4.0, it would be great if this rather low complex patch can be included in >> the 4.0 RC and thereby ship a "complete feature". >> >> On Thu, Jun 3, 2021 at 4:04 PM Vinay Chella >> wrote: >> I think it can be argued that this is a pretty serious bug for a newly >>> introduced feature, and qualifies for inclusion in an RC, but I don’t >>> personally have a strong opinion on if this should happen. >>> +1 >>> One more point - if we keep the workaround, that should be documented >>> with big red letters for the users. >>> >>> Yes, heavy +1, if we are not merging it. Another idea, if we are not >>> merging this in, is to put DCL(CREATE ROLE/USER, ALTER ROLE/USER etc.,) >>> queries in the default configuration (cassandra.yaml) exclude list to >> avoid >>> oops for operators, since that is the only query type that log passwords >> in >>> plain text and all other places they are not. >>> >>> >>> Thanks, >>> Vinay >>> >>> >>> On Thu, Jun 3, 2021 at 3:58 PM bened...@apache.org >>> wrote: >>> I think it can be argued that this is a pretty serious bug for a newly introduced feature, and qualifies for inclusion in an RC, but I don’t personally have a strong opinion on if this should happen. I can’t imagine how this would be an _exception_ for inclusion in 4.0.1 though. From: Mick Semb Wever Date: Thursday, 3 June 2021 at 22:45 To: dev@cassandra.apache.org Subject: Re: Obfuscation of passwords in audit loging, in or not in >> 4.0? Thanks for raising this Stefan. > While I humbly think this is 4.0-worthy, the process we have, as far > as I know, is that there should be only critical fixes in 4.0 so I > guess this will go to 4.0.1, right? Or does this qualify to go to 4.0 > still? > I believe the question here is whether this patch is worthy of an >>> exception to go to 4.0.x. (i.e. 4.0.1) At this point in time all improvements would be by default slated for >> 4.x (i.e. 4.1) It does not qualify as a RC critical bug for 4.0.0. Looking at the patch it is simple, and one could almost consider it a security fix on a new 4.0 feature, so I'd say it's a valid exception >> for 4.0.x. Keen to hear what others think. And how we should go about requesting >>> such exceptions for non-bugs during each annual release cycle. >>> >> - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: [DISCUSSION] Should we mark DROP COMPACT STORAGE as experimental
This seems reasonable to me, but it raises a question of roadmap. My understanding is that we are deprecating compact storage, and will remove it in a future release (or have already partially removed it? I forget). Do these issues then constitute a blocking issue for GA, or do we modify our roadmap, or do we stipulate that users must upgrade to a future patch version of 4.0 before going to 4.next/5.0? From: Benjamin Lerer Date: Friday, 4 June 2021 at 09:53 To: dev@cassandra.apache.org Subject: [DISCUSSION] Should we mark DROP COMPACT STORAGE as experimental Hi everybody, There are a significant amount of issues with DROP COMPACT STORAGE that can be pretty surprising for users. To name a few: * Some hidden columns will show up changing the resultset returned for wildcard queries * As COMPACT tables did not have primary key liveness there empty rows inserted AFTER the ALTER will be returned whereas the one inserted before the ALTER will not. * Also due to the lack of primary key liveness the amount of SSTables being read will increase resulting in slower queries * After DROP COMPACT it becomes possible to ALTER the table in a way that makes all the row disappears * There is a loss of functionality around null clustering when dropping compact storage (CASSANDRA-16069) In my opinion DROP COMPACT STORAGE is not ready for production use unless users fully understand what they are doing. By consequence, I am wondering if we should not mark it as experimental as we did for the Materialized Views (CASSANDRA-13959). What is your opinion?
Re: Materialized Views during node replace
Thanks! Why are MVs being treated differently here ? While node replace was ongoing sstables for MV also got streamed. Is rebuilding the MV required to ensure consistency between main table and MVs on the replaced node ? On Fri, Jun 4, 2021 at 2:25 PM Erick Ramirez wrote: > I can't recall if view builds affect the bootstrap but index builds > definitely do. It's a bit late now but you could've run nodetool > compactionstats to see what tasks were running at the time and those need > to complete before the bootstrap is considered complete for the node to go > into UP/NORMAL status. Cheers! > -- Shaurya Gupta
Re: Obfuscation of passwords in audit loging, in or not in 4.0?
On Fri, Jun 4, 2021 at 4:32 AM Sam Tunnicliffe wrote: > Shipping a brand new, non-experimental feature with a security hole like this > feels > counter to our goal of releases being prod ready in .0, so I'm +1 on > including it in > an rc/ga I think I have to agree here. We can ship a complete feature, we can remove it and not ship it, but what is not acceptable is shipping it in a broken and potentially dangerous state. - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: Obfuscation of passwords in audit loging, in or not in 4.0?
+1, please, reclassify it as a bug. Thank you Stefan On Fri, 4 Jun 2021 at 9:13, Brandon Williams wrote: > On Fri, Jun 4, 2021 at 4:32 AM Sam Tunnicliffe wrote: > > Shipping a brand new, non-experimental feature with a security hole like > this feels > > counter to our goal of releases being prod ready in .0, so I'm +1 on > including it in > > an rc/ga > > I think I have to agree here. We can ship a complete feature, we can > remove it and not ship it, but what is not acceptable is shipping it > in a broken and potentially dangerous state. > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: [DISCUSSION] Should we mark DROP COMPACT STORAGE as experimental
+1, thank you Benjamin! There is already a warning in the docs but marking it as experimental sounds more reasonable at this point. Thank you for all your investigations and work done On Fri, 4 Jun 2021 at 5:43, bened...@apache.org wrote: > This seems reasonable to me, but it raises a question of roadmap. My > understanding is that we are deprecating compact storage, and will remove > it in a future release (or have already partially removed it? I forget). Do > these issues then constitute a blocking issue for GA, or do we modify our > roadmap, or do we stipulate that users must upgrade to a future patch > version of 4.0 before going to 4.next/5.0? > > > From: Benjamin Lerer > Date: Friday, 4 June 2021 at 09:53 > To: dev@cassandra.apache.org > Subject: [DISCUSSION] Should we mark DROP COMPACT STORAGE as experimental > Hi everybody, > > There are a significant amount of issues with DROP COMPACT STORAGE that can > be pretty surprising for users. > To name a few: > * Some hidden columns will show up changing the resultset returned for > wildcard queries > * As COMPACT tables did not have primary key liveness there empty rows > inserted AFTER the ALTER will be returned whereas the one inserted before > the ALTER will not. > * Also due to the lack of primary key liveness the amount of SSTables being > read will increase resulting in slower queries > * After DROP COMPACT it becomes possible to ALTER the table in a way that > makes all the row disappears > * There is a loss of functionality around null clustering when dropping > compact storage (CASSANDRA-16069) > > In my opinion DROP COMPACT STORAGE is not ready for production use unless > users fully understand what they are doing. > By consequence, I am wondering if we should not mark it as experimental as > we did for the Materialized Views (CASSANDRA-13959). > > What is your opinion? >
Re: [DISCUSSION] Should we mark DROP COMPACT STORAGE as experimental
+1 On Fri, Jun 4, 2021, 3:53 AM Benjamin Lerer wrote: > Hi everybody, > > There are a significant amount of issues with DROP COMPACT STORAGE that can > be pretty surprising for users. > To name a few: > * Some hidden columns will show up changing the resultset returned for > wildcard queries > * As COMPACT tables did not have primary key liveness there empty rows > inserted AFTER the ALTER will be returned whereas the one inserted before > the ALTER will not. > * Also due to the lack of primary key liveness the amount of SSTables being > read will increase resulting in slower queries > * After DROP COMPACT it becomes possible to ALTER the table in a way that > makes all the row disappears > * There is a loss of functionality around null clustering when dropping > compact storage (CASSANDRA-16069) > > In my opinion DROP COMPACT STORAGE is not ready for production use unless > users fully understand what they are doing. > By consequence, I am wondering if we should not mark it as experimental as > we did for the Materialized Views (CASSANDRA-13959). > > What is your opinion? >
Re: [DISCUSSION] Should we mark DROP COMPACT STORAGE as experimental
> > not ready for production use unless users fully understand what they are > doing. This statement stood out to me - in my opinion we should think carefully about the surface area of the user interfaces on new features before we add more cognitive burden to our users. We already have plenty of "foot-guns" in the project and should only add more if absolutely necessary. Further, marking this as experimental would be another feature we've released and then retroactively marked as experimental; that's a habit we should not get into. On balance, my .02 is the benefits to our end users and operators of getting 4.0 to GA outweigh the costs of flagging this as experimental now so I'm a +1 to the flagging idea, but I think there's some valuable lessons for us to learn in retrospect from not just this feature but others like it in the past. Curious to hear Alex' thoughts about this situation in particular as author of C-10857. I recall that being a pretty painful slog so apologies in advance for picking at this scab. :) On Fri, Jun 4, 2021 at 9:44 AM Brandon Williams wrote: > +1 > > On Fri, Jun 4, 2021, 3:53 AM Benjamin Lerer wrote: > > > Hi everybody, > > > > There are a significant amount of issues with DROP COMPACT STORAGE that > can > > be pretty surprising for users. > > To name a few: > > * Some hidden columns will show up changing the resultset returned for > > wildcard queries > > * As COMPACT tables did not have primary key liveness there empty rows > > inserted AFTER the ALTER will be returned whereas the one inserted before > > the ALTER will not. > > * Also due to the lack of primary key liveness the amount of SSTables > being > > read will increase resulting in slower queries > > * After DROP COMPACT it becomes possible to ALTER the table in a way that > > makes all the row disappears > > * There is a loss of functionality around null clustering when dropping > > compact storage (CASSANDRA-16069) > > > > In my opinion DROP COMPACT STORAGE is not ready for production use unless > > users fully understand what they are doing. > > By consequence, I am wondering if we should not mark it as experimental > as > > we did for the Materialized Views (CASSANDRA-13959). > > > > What is your opinion? > > >
Re: Obfuscation of passwords in audit loging, in or not in 4.0?
Hi, ok, so this will make it to 4.0 then. I would re-iterate on FQL logging though. What is our decision? Should these passwords be clearly visible or we should obfuscate them too? I am trying to close all remaining questions, while I do get that passwords in audit are for sure problematic, I do not think that I have a clear agreement what we should do with FQL yet. Thank you On Fri, 4 Jun 2021 at 15:22, Ekaterina Dimitrova wrote: > > +1, please, reclassify it as a bug. > Thank you Stefan > > On Fri, 4 Jun 2021 at 9:13, Brandon Williams wrote: > > > On Fri, Jun 4, 2021 at 4:32 AM Sam Tunnicliffe wrote: > > > Shipping a brand new, non-experimental feature with a security hole like > > this feels > > > counter to our goal of releases being prod ready in .0, so I'm +1 on > > including it in > > > an rc/ga > > > > I think I have to agree here. We can ship a complete feature, we can > > remove it and not ship it, but what is not acceptable is shipping it > > in a broken and potentially dangerous state. > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: Welcome Dinesh Joshi as Cassandra PMC member
Congrats Dinesh! On Thu, Jun 3, 2021 at 3:59 PM Patrick McFadin wrote: > This is great. Congratulations Dinesh! > > On Thu, Jun 3, 2021 at 11:51 AM Jordan West wrote: > > > Congratulations Dinesh! > > > > Jordan > > > > On Thu, Jun 3, 2021 at 1:40 AM Mick Semb Wever wrote: > > > > > Congrats Dinesh. Thanks for all the help given and offered whenever it > is > > > needed! > > > > > > On Wed, 2 Jun 2021 at 18:16, Benjamin Lerer wrote: > > > > > > > The PMC's members are pleased to announce that Dinesh Joshi has > > accepted > > > > the invitation to become a PMC member. > > > > > > > > Thanks a lot, Dinesh, for everything you have done for the project > all > > > > these years. > > > > > > > > Congratulations and welcome > > > > > > > > The Apache Cassandra PMC members > > > > > > > > > > -- Dikang
Re: Obfuscation of passwords in audit loging, in or not in 4.0?
On Fri, Jun 4, 2021 at 10:32 AM Stefan Miklosovic wrote: > I would re-iterate on FQL logging though. What is our decision? Should > these passwords be clearly visible or we should obfuscate them too? I don't think it ever makes sense to log a password in plaintext, so my feeling is we should obfuscate there as well. - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
