Hi all, thank you for your replies.
@Yakir ntpd service is a prerequisite, but sometimes even NTP servers malfunction or misbehave if not properly configured with multiple redundant NTP upstream sources. @Abe this is a fresh approach that I haven't thought about! However, I perceive the table migration as even more complex than manipulating the SStables offline. Depending on the cluster size and the complexity of the application logic, such migration can become impractical from both an operational and cost perspective (coordinating double writes/reads, verifying completeness of the backfill, determining the migration completion, managing the performance impact throughout, etc.). @Jeff thank you for your replies. CASSANDRA-18352 as a prevention measure looks promising, I am having a look. CASSANDRA-12106 is very interesting, but also difficult to use with thousands or millions of affected partition keys. I'm primarily looking for a permanent recovery procedure. Your proposal of implementing a custom mechanism for SStable scrub sounds like a viable alternative. I assume the same logic could be applied with offline SStable manipulation followed by sstableloader loading. Are there practical benefits of one approach over the other? My main concern with either procedure is avoiding resurrection of the problematic cells. If we scrub nodes one at a time, we would need to disable auto-repairs and read-repair to prevent un-scrubbed replicas from re-propagating the future timestamps. Would we also need to discard pending hints generated before the scrub? Are there any other precautionary measures needed to ensure the scrubbed data doesn't get overwritten by stale replicas? Regards, Panagiotis -----Original Message----- From: Jeff Jirsa <[email protected]> Sent: Wednesday, July 15, 2026 9:28 PM To: cassandra <[email protected]> Subject: Re: Recovery options for data written with far-future timestamps due to NTP clock jump That’s a good point. CASSANDRA-12106 would let you blacklist those partitions if you know they’re invalid. That’s probably strictly easier than truncating the table. > On Jul 15, 2026, at 9:01 AM, Abe Ratnofsky <[email protected]> wrote: > > Read repair will propagate the future-writes, so if you scrub them from one > replica and have concurrent reads, they can re-appear. > > Depending on your requirements you might also be able to post-filter the > future-writes at query time, by including SELECT WRITETIME(col) in your read > queries and skipping them. > > On Wed, Jul 15, 2026, at 8:39 AM, Jeff Jirsa wrote: >> Double writing / double reading gets messy because it requires an >> awful lot of work to ensure it’s correct as you move data around. >> >> Just fixing it during compaction or scrub doesnt require app changes, >> just disable repair, start rewriting the sstables, enable repair >> again when you’re done. >> >> Especially if 18352 gives you a cassandra-internal “is this a legal >> timestamp” predicate, that seems way easier than changing all the app >> logic. >> >> >>> On Jul 15, 2026, at 8:21 AM, Abe Ratnofsky <[email protected]> wrote: >>> >>> For recovery, I’d recommend approaching this as a regular table migration, >>> where you start writing to a new table and backfill using >>> cassandra-analytics, filtering out all cells with WRITETIME in the distant >>> future. >>> >>> Freeze your backups so they don’t get purged. You might be able to recover >>> some overwritten values from backup SSTables if they weren’t compacted yet. >>> >>> On Wed, Jul 15, 2026, at 8:16 AM, Jeff Jirsa wrote: >>>> It’s not that it’s not running, it’s that ntpd in startup phases >>>> can have big steps in some configs. Also alert on large clock >>>> drift. If you run cassandra, you need alerts on ntpd >>>> >>>>> On Jul 15, 2026, at 8:13 AM, Yakir Gibraltar <[email protected]> wrote: >>>>> >>>>> Configure Cassandra to start after ntpd service , easy on systemd. >>>>> >>>>> Thank you, Yakir Gibraltar >>>>> >>>>> On Wed, Jul 15, 2026, 10:47 Panagiotis Melidis via user >>>>> <[email protected]> wrote: >>>>>> Hi all,____ >>>>>> __ __ >>>>>> I'm looking for guidance on a scenario involving NTP clock skew >>>>>> and its impact on Cassandra data.____ __ __ If an NTP malfunction >>>>>> causes the system clock on multiple Cassandra nodes to jump far >>>>>> into the future (e.g., 100 years), any writes processed during >>>>>> that window will carry far-future timestamps. Once the clock is >>>>>> corrected, these entries effectively become permanent and shadow >>>>>> any new write/delete in the affected partition keys.____ __ __ >>>>>> The only viable recovery approach at scale I've been able to >>>>>> identify is truncating the affected tables.____ 1. Are there any >>>>>> other known viable recovery procedures that don't require a full >>>>>> truncate?____ 2. Are there any known existing or planned >>>>>> preventive measures in Cassandra to guard against such >>>>>> scenarios?____ __ __ Any pointers to prior discussions, tooling, >>>>>> or patches would be greatly appreciated.____ __ __ Thanks.____
