Right, exactly. Which (I think) makes the object store about as valuable as an ephemeral disk if you don't keep everything on there. It's a tradeoff I'd never use given the cost / benefit.
Does that mean you agree that we should focus on writethrough cache mode first? Jon On Fri, Apr 11, 2025 at 2:09 PM Jeff Jirsa <jji...@gmail.com> wrote: > > > > On Apr 11, 2025, at 1:15 PM, Jon Haddad <j...@rustyrazorblade.com> wrote: > > > > > > I also keep running up against my concern about treating object store as > a write back cache instead of write through. "Tiering" data off has real > consequences for the user, the big one being data loss, especially with > regards to tombstones. I think this is a pretty serious foot gun. It's > the same problem we originally had with JBOD, where we could have > tombstones on one disk and the shadowed data on the other. Losing one disk > results in data getting resurrected. Anthony covered it in a blog post [1] > and I believe CASSANDRA-6696 was the JIRA that addressed the problem. > Introducing tiering would essentially bring this problem back. > > > If you lose one disk, you shoot the instance. We have to stop pretending > you can have partial failures. That’s it. That’s the fix. You don’t get to > lose part of a machine and pretend it’s still viable. Just like losing a a > commit log segment or losing an object in a bucket, if you lose one object, > you throw it away or you’ve resurrected data / violated consistency. > > > >
