Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-24 Thread Blake Eggleston
Those are both fair points. Once you start dealing with data loss though, maintaining guarantees is often impossible, so I’m not sure that torn writes or updated timestamps are dealbreakers, but I’m fine with tabling option 2 for now and seeing if we can figure out something better. Regarding t

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-23 Thread Runtian Liu
In the second option, we use the repair timestamp to re-update any cell or row we want to fix in the base table. This approach is problematic because it alters the write time of user-supplied data. Although Cassandra does not allow users to set timestamps for LWT writes, users may still rely on the

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-23 Thread Blake Eggleston
> Sorry, Blake—I was traveling last week and couldn’t reply to your email > sooner. No worries, I’ll be taking some time off soon as well. > I don’t think the first or second option is ideal. We should treat the base > table as the source of truth. Modifying it—or forcing an update on it, even

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-22 Thread Runtian Liu
Sorry, Blake—I was traveling last week and couldn’t reply to your email sooner. > First - we interpret view data with higher timestamps than the base table as data that’s missing from the base and replicate it into the base table. The timestamp of the missing data may be below the paxos timestamp

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-12 Thread Blake Eggleston
Got it, thanks for clearing that up. I’d seen the strict liveness code around but didn’t realize it was MV related and hadn’t dug into what it did or how it worked. I think you’re right about the row liveness update working for extra data with timestamps lower than the most recent base table u

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-11 Thread Paulo Motta
> I’m not sure if this is the only edge case—there may be other issues as well. I’m also unsure whether we should redesign the tombstone handling for MVs, since that would involve changes to the storage engine. To minimize impact there, the original proposal was to rebuild the affected ranges usin

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-11 Thread Runtian Liu
The current design leverages strict liveness to shadow the old view row. When the view-indexed value changes from 'a' to 'b', no tombstone is written; instead, the old row is marked as expired by updating its liveness info with the timestamp of the change. If the column is later set back to 'a', th

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-11 Thread Blake Eggleston
That’s a good point, although as described I don’t think that could ever work properly, even in normal operation. Either we’re misunderstanding something, or this is a flaw in the current MV design. Assuming changing the view indexed column results in a tombstone being applied to the view row

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-11 Thread Runtian Liu
> In the case of a missed update, we'll have a new value and we can send a tombstone to the view with the timestamp of the most recent update. > then something has gone wrong and we should issue a tombstone using the paxos repair timestamp as the tombstone timestamp. The current MV implementation

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-10 Thread Blake Eggleston
> Extra row in MV (assuming the tombstone is gone in the base table) — how > should we fix this? > > > This would mean that the base table had either updated or deleted a row and the view didn't receive the corresponding delete. In the case of a missed update, we'll have a new value and we

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-10 Thread Runtian Liu
Okay, let’s put the efficiency discussion on hold for now. I want to make sure the actual repair process after detecting inconsistencies will work with the index-based solution. When a mismatch is detected, the MV replica will need to stream its index file to the base table replica. The base table

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-08 Thread Blake Eggleston
> hopefully we can come up with a solution that everyone agrees on. I’m sure we can, I think we’ve been making good progress > My main concern with the index-based solution is the overhead it adds to the > hot path, as well as having to build indexes periodically. So the additional overhead of

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-07 Thread Runtian Liu
Thanks, Blake. I’m open to iterating on the design, and hopefully we can come up with a solution that everyone agrees on. My main concern with the index-based solution is the overhead it adds to the hot path, as well as having to build indexes periodically. As mentioned earlier, this MV repair sho

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-06 Thread Blake Eggleston
> I don't see any outcome here that is good for the community though. Either > Runtian caves and adopts your design that he (and I) consider inferior, or he > is prevented from contributing this work. Hey Runtian, fwiw, these aren't the only 2 options. This isn’t a competition. We can collabora

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-06 Thread Benedict Elliott Smith
Hmm, I am very surprised as I helped write that and I distinctly recall a specific goal was avoiding binding vetoes as they're so toxic. Ok, I guess you can block this work if you like. I don't see any outcome here that is good for the community though. Either Runtian caves and adopts your desi

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-06 Thread Blake Eggleston
Thanks for the updated table Runtian, I think it misses the point though. The problem with snapshot based repair is that it degrades operators ability to react to data problems by imposing a significant upfront processing burden on repair, and that it doesn’t scale well with cluster size, as I i

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-06 Thread Blake Eggleston
Hi Benedict, that’s actually not true. Here’s a link to the project governance page: _https://cwiki.apache.org/confluence/display/CASSANDRA/Cassandra+Project+Governance_ The CEP section says: “*Once the proposal is finalized and any major committer dissent reconciled, call a [VOTE] on the ML

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-06 Thread Benedict Elliott Smith
Without intending to get into further discussion, I would note that I think we already have the necessary facilities to enable compaction of the sstables being used to construct a snapshot, so that additional disk space is not required for snapshots. We use this already today to compact incoming

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-06 Thread Runtian Liu
Thanks, Blake and Jon, for your feedback—I really appreciate your time on this topic and your efforts to help make this CEP a success. Throughout this discussion, we've explored many interesting problems, and your input helped me better understand how the index-based solution would work. Now that b

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-06 Thread Benedict Elliott Smith
> but the snapshot repair design is not a viable path forward. It’s the first > iteration of a repair design. We’ve proposed a second iteration, and we’re > open to a third iteration. I shan't be participating further in discussion, but I want to make a point of order. The CEP process has no ve

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-04 Thread Blake Eggleston
You can detect and fix the mismatch in a single round of repair, but the amount of work needed to do it is _significantly_ higher with snapshot repair. Consider a case where we have a 300 node cluster w/ RF 3, where each view partition contains entries mapping to every token range in the cluster

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-04 Thread Runtian Liu
> We potentially have to do it several times on each node, depending on the size of the range. Smaller ranges increase the size of the board exponentially, larger ranges increase the number of SSTables that would be involved in each compaction. As described in the CEP example, this can be handled

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-04 Thread Jon Haddad
> This isn’t really the whole story. The amount of wasted scans on index repairs is negligible. If a difference is detected with snapshot repairs though, you have to read the entire partition from both the view and base table to calculate what needs to be fixed. You nailed it. When the base table

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-06-03 Thread Blake Eggleston
> Adds overhead in the hot path due to maintaining indexes. Extra memory needed > during write path and compaction. I’d make the same argument about the overhead of maintaining the index that Jon just made about the disk space required. The relatively predictable overhead of maintaining the ind

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-27 Thread Benedict
Are you and Blake proposing to implement it?On 27 May 2025, at 19:37, Jon Haddad wrote:If the goal of this proposal is to move MVs to a state of being production ready, then it shouldn't have glaring holes.  We've already identified that the proposed global snapshots have these issues:1. The data

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-21 Thread Benedict
It’s an additional piece of work. If you need to be able to rebuild this data, then you need the original proposal either way. This proposal to maintain a live updating snapshot is therefore an additional feature on top of the MVP proposed.I don’t think this new proposal is fully fleshed out, I hav

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-21 Thread Jon Haddad
I agree with Blake. These are perfectly reasonable discussions to have up front. Snapshot based repair has a huge downside in that you're repairing data that's days or weeks old. There's going to be issues that arise from that especially since the deletes that are recorded on the MV aren't going

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-21 Thread Benedict
Depending how long the grid structure takes to build, there is perhaps anyway value in being able to update the snapshot after construction, so that when the repair is performed it is as up to date as possible. But, I don’t think this is trivial? I have some ideas how this might be done but they ar

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-20 Thread Jon Haddad
More questions and thoughts... * Consistency question: In the case where a base table gets a corrupt SSTable and is scrubbed, when it repairs against the view, without tracking the deletes against the secondary table, do we end up pushing the lack of data into the MV? * I threw out the idea earli

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-19 Thread Runtian Liu
> You don’t need to duplicate the full data set in the index, you just need enough info to detect that something is missing. Could you please explain how this would work? If we build Merkle trees or compute hashes at the SSTable level, how would this case be handled? For example, consider the follo

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-19 Thread Jon Haddad
We could also track the deletes that need to be made to the view, in another SSTable component on the base. That way you can actually do repair with tombstones. On Mon, May 19, 2025 at 11:37 AM Blake Eggleston wrote: > If we went the storage attached route then I think you’d just need more >

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-18 Thread Runtian Liu
> If you had a custom SAI index or something, this isn’t something you’d need to worry about This is what I missed. I think this could be a potential solution, but comparing indexes alone isn’t sufficient—it only handles cases where the MV has extra or missing rows. It doesn’t catch data mismatche

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-18 Thread Jaydeep Chovatia
>Isn’t the reality here is that repairing a single partition in the base table is potentially a full cluster-wide scan of the MV if you also want to detect rows in the MV that don’t exist in the base table (eg resurrection or a missed delete) Exactly. Since materialized views (MVs) are partitioned

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-18 Thread Jeff Jirsa
Isn’t the reality here is that repairing a single partition in the base table is potentially a full cluster-wide scan of the MV if you also want to detect rows in the MV that don’t exist in the base table (eg resurrection or a missed delete)There’s no getting around that. Keeping an extra index doe

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-17 Thread Runtian Liu
For each row, when calculating its hash, we first need to merge all the SSTables that contain that row. We cannot attach a Merkle tree directly to each SSTable, because merged Merkle trees would produce different hash values for the same data if the compaction states differ. On Sat, May 17, 2025 a

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-17 Thread Jon Haddad
Could we could do that for regular repair as well? which would make a validation possible with barely any IO? Sstable attached merkle trees? On Sat, May 17, 2025 at 5:36 PM Jon Haddad wrote: > What if you built the merkle tree for each sstable as a storage attached > index? > > Then your rep

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-17 Thread Jon Haddad
What if you built the merkle tree for each sstable as a storage attached index? Then your repair is merging merkle tables. On Sat, May 17, 2025 at 4:57 PM Runtian Liu wrote: > > I think you could exploit this to improve your MV repair design. Instead > of taking global snapshots and persisting

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-17 Thread Runtian Liu
> I think you could exploit this to improve your MV repair design. Instead of taking global snapshots and persisting merkle trees, you could implement a set of secondary indexes on the base and view tables that you could quickly compare the contents of for repair. We actually considered this appro

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-17 Thread Jon Haddad
Yeah, this is exactly what i suggested in a different part of the thread. The transformative repair could be done against the local index, and the local index can repair against the global index. It opens up a lot of possibilities, query wise, as well. On Sat, May 17, 2025 at 1:47 PM Blake Eggle

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-16 Thread Runtian Liu
Unfortunately, no. When building Merkle trees for small token ranges in the base table, those ranges may span the entire MV token range. As a result, we need to scan the entire MV to generate all the necessary Merkle trees. For efficiency, we perform this as a single pass over the entire table rath

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-16 Thread Jon Haddad
I spoke too soon - endless questions are not over :) Since the data that's going to be repaired only covers a range, I wonder if it makes sense to have the ability to issue a minimalist snapshot that only hardlinks SSTables that are in a token range. Based on what you (Runtian) have said above, o

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-16 Thread Jon Haddad
Thats the critical bit i was missing, thank you Blake. I guess we’d need to have unlimited height trees then, since you’d need to be able to update the hashes of individual partitions, and we’d also need to propagate the hashes up every time as well. I’m curious what the cost will look like with t

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-15 Thread Jon Haddad
One last thing. I'm pretty sure building the tree requires the keys be added in token order: https://github.com/apache/cassandra/blob/08946652434edbce38a6395e71d4068898ea13fa/src/java/org/apache/cassandra/repair/Validator.java#L173 Which definitely introduces a bit of a problem, given that the tr

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-15 Thread Jon Haddad
There's a lot here that's still confusing to me. Maybe you can help me understand it better? Apologies in advance for the text wall :) I'll use this schema as an example: - CREATE TABLE test.t1 ( id int PRIMARY KEY, v1 int ); create MATERIALIZED VIEW test_mv as SELECT v1, id f

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-15 Thread Runtian Liu
The previous table compared the complexity of full repair and MV repair when reconciling one dataset with another. In production, we typically use a replication factor of 3 in one datacenter. This means full repair involves 3n rows, while MV repair involves comparing 6n rows (base + MV). Below is a

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-15 Thread Jon Haddad
> They are not two unordered sets, but rather two sets ordered by different keys. I think this is a distinction without a difference. Merkle tree repair works because the ordering of the data is mostly the same across nodes. On Thu, May 15, 2025 at 9:27 AM Runtian Liu wrote: > > what we're try

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-15 Thread Runtian Liu
> what we're trying to achieve here is comparing two massive unordered sets. They are not two unordered sets, but rather two sets ordered by different keys. This means that when building Merkle trees for the base table and the materialized view (MV), we need to use different strategies to ensure t

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-15 Thread Josh McKenzie
> I think in order to address this, the view should be propagated to the base > replicas *after* it's accepted by all or a majority of base replicas. This is > where I think mutation tracking could probably help. Yeah, the idea of "don't reflect in the MV until you hit the CL the user requested

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-15 Thread Paulo Motta
> I think requiring a rebuild is a deal breaker for most teams. In most instances it would be having to also expand the cluster to handle the additional disk requirements. It turns an inconsistency problem into a major operational headache that can take weeks to resolve. Agreed. The rebuild wou

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-15 Thread Jon Haddad
I think requiring a rebuild is a deal breaker for most teams. In most instances it would be having to also expand the cluster to handle the additional disk requirements. It turns an inconsistency problem into a major operational headache that can take weeks to resolve. On Thu, May 15, 2025 a

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-15 Thread Paulo Motta
> There's bi-directional entropy issues with MV's - either orphaned view data or missing view data; that's why you kind of need a "bi-directional ETL" to make sure the 2 agree with each other. While normal repair would resolve the "missing data in MV" case, it wouldn't resolve the "data in MV that'

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-15 Thread Josh McKenzie
There's bi-directional entropy issues with MV's - either orphaned view data or missing view data; that's why you kind of need a "bi-directional ETL" to make sure the 2 agree with each other. While normal repair would resolve the "missing data in MV" case, it wouldn't resolve the "data in MV that

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-14 Thread Blake Eggleston
Maybe, I’m not really familiar enough with how “classic” MV repair works to say. You can’t mix normal repair and mutation reconciliation in the current incarnation of mutation tracking though, so I wouldn’t assume it would work with MVs. On Wed, May 14, 2025, at 11:29 AM, Jon Haddad wrote: > In

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-14 Thread Jon Haddad
In the case of bitrot / losing an SSTable, wouldn't a normal repair (just the MV against the other nodes) resolve the issue? On Wed, May 14, 2025 at 11:27 AM Blake Eggleston wrote: > Mutation tracking is definitely an approach you could take for MVs. > Mutation reconciliation could be extended t

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-14 Thread Blake Eggleston
Mutation tracking is definitely an approach you could take for MVs. Mutation reconciliation could be extended to ensure all changes have been replicated to the views. When a base table received a mutation w/ an id it would generate a view update. If you block marking a given mutation id as recon

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-14 Thread Jon Haddad
I was thinking along the lines of mutation tracking too, but I have to admit I haven't spent much time on reading through it, it's probably time I did. I'd read up on it, thanks for bringing it up. One thing to consider is that 5TB is not particularly dense anymore, in the world of Cassandra 5+.

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-14 Thread Paulo Motta
I don't see mutation tracking [1] mentioned in this thread or in the CEP-48 description. Not sure this would fit into the scope of this initial CEP, but I have a feeling that mutation tracking could be potentially helpful to reconcile base tables and views ? For example, when both base and view up

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-14 Thread Paulo Motta
> - The first thing I notice is that we're talking about repairing the entire table across the entire cluster all in one go. It's been a *long* time since I tried to do a full repair of an entire table without using sub-ranges. Is anyone here even doing that with clusters of non-trivial size? Ho

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-14 Thread Jon Haddad
Putting this another way - what we're trying to achieve here is comparing two massive unordered sets. I believe the worst case is a O(N^2) complexity, and N can be in the billions. I really think we need a completely different approach here than how repair currently works. For me to be a +1 on t

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-14 Thread Jon Haddad
I've got several concerns around this repair process. - The first thing I notice is that we're talking about repairing the entire table across the entire cluster all in one go. It's been a *long* time since I tried to do a full repair of an entire table without using sub-ranges. Is anyone here e

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-13 Thread Runtian Liu
> Looking at the details of the CEP it seems to describe Paxos as PaxosV1, but PaxosV2 works slightly differently (it can read during the prepare phase). I assume that supporting Paxos means supporting both V1 and V2 for materialized views? We are going to support Paxos V2. The CEP is not clear on

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-12 Thread Jaydeep Chovatia
>Like something doesn't add up here because if it always includes the base table's primary key columns that means The requirement for materialized views (MVs) to include the base table's primary key appears to be primarily a syntactic constraint specific to Apache Cassandra. For instance, in Dynam

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-12 Thread Jon Haddad
> Or compaction hasn’t made a mistake, or cell merge reconciliation hasn’t made a mistake, or volume bitrot hasn’t caused you to lose a file. > Repair isnt’ just about “have all transaction commits landed”. It’s “is the data correct N days after it’s written”. Don't forget about restoring from a b

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-12 Thread Benedict Elliott Smith
> Like something doesn't add up here because if it always includes the base > table's primary key columns that means they could be storage attached by just > forbidding additional columns and there doesn't seem to be much utility in > including additional columns in the primary key? You can re-

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-12 Thread Jeff Jirsa
> On May 12, 2025, at 8:10 AM, Ariel Weisberg wrote: > > Hi, > > I think it's worth taking a step back and looking at the current MV > restrictions which are pretty onerous. > > A view must have a primary key and that primary key must conform to the > following restrictions: > it must conta

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-12 Thread Ariel Weisberg
Hi, I think it's worth taking a step back and looking at the current MV restrictions which are pretty onerous. A view must have a primary key and that primary key must conform to the following restrictions: • it must contain all the primary key columns of the base table. This ensures that eve

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-11 Thread Josh McKenzie
> This makes me really, really concerned with how it scales, and how likely it > is to be able to schedule automatically without blowing up. It seems to me that resource-aware throttling would be the solution here, or from a more primitive case, just hard bounding threadpool size, throughput ra

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-09 Thread Runtian Liu
I’ve added a new section on isolation and consistency . In our current design, materialized-view tables stay eventually consistent,

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-09 Thread David Capwell
> The MV repair tool in Cassandra is intended to address inconsistencies that > may occur in materialized views due to various factors. This component is the > most complex and demanding part of the development effort, representing > roughly 70% of the overall work. > but I am very concerned ab

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-09 Thread Jeff Jirsa
> On May 9, 2025, at 12:59 PM, Ariel Weisberg wrote: > > > I am *big* fan of getting repair really working with MVs. It does seem > problematic that the number of merkle trees will be equal to the number of > ranges in the cluster and repair of MVs would become an all node operation. > How

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-09 Thread Ariel Weisberg
Hi, Great to see MVs getting some attention and it's a good time to start addressing their shortcomings. Looking at the details of the CEP it seems to describe Paxos as PaxosV1, but PaxosV2 works slightly differently (it can read during the prepare phase). I assume that supporting Paxos means

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-09 Thread Benedict
I should add that I’m in favour of this proposal in principle, and support the proposal to utilise Paxos.On 9 May 2025, at 08:21, Benedict Elliott Smith wrote:I’d also like to explore a bit further the isolation guarantees we’re promising with "Strict Consistency Mode” - and the protocol details.

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-09 Thread Benedict Elliott Smith
I’d also like to explore a bit further the isolation guarantees we’re promising with "Strict Consistency Mode” - and the protocol details. By strict, do we mean linearizable? Either way, we should state the guarantees explicitly so we can evaluate whether the protocol can meet them. Also, if the

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-08 Thread Jeff Jirsa
Setting aside the paxos vs accord conversation (though admittedly my first question would have been “why not accord”), I’m curious from folks who have thought about this how you’re thinking about correctness of repairI ask because I have seen far more data resurrection cases than I have lost write

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-08 Thread Runtian Liu
Here’s my perspective: #1 Accord vs. LWT round trips Based on the insights shared by the Accord experts, it appears that implementing MV using Accord can achieve a comparable number of round trips as the LWT solution proposed in CEP-48. Additionally, it seems that the number of WAN RTTs might be

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-08 Thread Jon Haddad
Based on David and Blake’s responses, it sounds like we don’t need to block on anything. I realize you may be making a broader point, but in this instance it sounds like there’s nothing here preventing an accord based MV implementation. Now that i understand more about how it would be done, it als

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-08 Thread Josh McKenzie
> IMHO, focus should be on accord-based MVs. Even if that means it's blocked > on first adding support for multiple conditions. > Strongly disagree here. We should develop features to be as loosely coupled w/one another as possible w/an eye towards future compatibility and leverage but not blo

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-08 Thread Mick Semb Wever
> Curious what others think though. I'm +1 on the spirit of getting MVs to > a stable point, but not convinced this is the best approach. > IMHO, focus should be on accord-based MVs. Even if that means it's blocked on first adding support for multiple conditions.

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-07 Thread David Capwell
> I think the primary argument *against* Accord is that the syntax isn't > expressive enough to be able to address multiple conditions in MVs. For each > field that's updated, you'll need to know if you want to add that update into > the transaction, and you'd need to check if it was modified.

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-07 Thread Blake Eggleston
> Yes, you need to read the original row before the transaction begins in order > to get the initial state, but could be done at local one by the coordinator, > reading itself. The performance overhead of an additional, local one read > should be significantly less than a Paxos transaction that

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-07 Thread Jon Haddad
Glad to see folks are looking to improve MVs. Definitely one of the areas we need some attention paid to. Do you have a patch already for this? We haven't had a discussion yet about winding down new development in trunk but IMO we should probably stop merging big things in soon and focus on gett

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-07 Thread guo Maxwell
After thinking about it, if you want to use accord for synchronization in the future, you need to modify the base table attribute " transactional_mode = 'full' ". If the user's base table does not want to use accord, do you plan to force the modification of this attribute? Runtian Liu 于2025年5月7日周

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-06 Thread Runtian Liu
Thanks for the questions. A few clarifications: - *Performance impact & opt-in model:* The new MV synchronization mechanism is fully opt-in. We understand that LWT-backed writes may introduce performance overhead, so users who prefer higher throughput over strict consistency can co

Re: [DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-06 Thread guo Maxwell
If the entire write operation involves additional LWTs to change the MV, it is uncertain whether users can accept the performance loss of such write operations. If this CEP is finally accepted, I think users should at least be given the choice of whether to use the old method or the new method, be

[DISCUSS] CEP-48: First-Class Materialized View Support

2025-05-06 Thread Runtian Liu
Hi everyone, We’d like to propose a new Cassandra Enhancement Proposal: CEP-48: First-Class Materialized View Support . This CEP focuses on addressing the long-standing consistency issues in th