[
https://issues.apache.org/jira/browse/CASSANDRA-21245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18071113#comment-18071113
]
Matt Byrd edited comment on CASSANDRA-21245 at 4/4/26 10:49 PM:
----------------------------------------------------------------
I've attached the CI run to the ticket also.
I'm actually out of office next week but will be back on 12th April, but have
attached the CI run also.
Were we thinking of perhaps fixing this on 5.0 also? we likely have something
which cherry-picks fairly cleanly there too.
As for reviewers any folks who feel comfortable or keen?
CC [~marcuse] [~dnk] [~jonmeredith] [~jmckenzie]
was (Author: mbyrd):
I've attached the CI run to the ticket also.
I'm actually out of office next week but will be back on 12th April, but have
attached the CI run also.
Were we thinking of perhaps fixing this on 5.0 also? we likely have something
which cherry-picks fairly cleanly there too.
As for reviewers any folks who feel comfortable or keen?
CC [~marcuse][~dnk][~jonmeredith][~jmckenzie]
> Uncompressed size is being used for compressed tables in maintenance
> operations
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-21245
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21245
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: vasya b
> Assignee: Dmitry Konstantinov
> Priority: Normal
> Attachments: cassandra.yaml, ci_summary.html, debug.log, go.mod,
> main.go
>
>
> Using a compressed table can lead to a state where uncompressed table size is
> bigger
> than the whole volume size, while it shouldn't be a problem it leads to
> compaction poblems, as other compaction task are being denied.
> f.e. one can easily create a compressed table with
> {code:java}
> CREATE TABLE IF NOT EXISTS ` + keyspace + `.` + table + ` (
> pk bigint,
> data text,
> PRIMARY KEY (pk)
> ) WITH compression = {'class': 'DeflateCompressor'}
> {code}
> (or any other compression)
> Insert a bunch of data which is going to be compressed quite well (f.e. a
> bunch of zeros or any other single char * 1024 *1024) * N where N is some big
> number
> one would expect that such table will be compacted without any problems.
> but one will get:
> {code:java}
> WARN [CompactionExecutor:8] 2026-03-20 14:25:17,256 CompactionTask.java:434
> - Not enough space for compaction (78c5c7c0-244f-11f1-9e8a-9937f6f9033a) of
> zerotohero.bulk_data, estimated sstables = 1, expected write size = 2686214
> ERROR [CompactionExecutor:8] 2026-03-20 14:25:17,256
> JVMStabilityInspector.java:70 - Exception in thread
> Thread[CompactionExecutor:8,5,CompactionExecutor]
> java.lang.RuntimeException: Not enough space for compaction
> (78c5c7c0-244f-11f1-9e8a-9937f6f9033a) of zerotohero.bulk_data, estimated
> sstables = 1, expected write size = 2686214
> at
> org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:436)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:148)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:94)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:374)
> at
> org.apache.cassandra.concurrent.FutureTask$3.call(FutureTask.java:141)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:840)
> WARN [CompactionExecutor:9] 2026-03-20 15:21:27,534 CompactionTask.java:440
> - Not enough space for compaction 51834c70-2457-11f1-9e8a-9937f6f9033a,
> 71.31226MiB estimated. Reducing scope.
> DEBUG [CompactionExecutor:8] 2026-03-20 15:25:27,447 Directories.java:550 -
> FileStore /mnt/cassandra (/dev/sdb1) has 36221336371 bytes available,
> checking if we can write 95979501107 bytes
> WARN [CompactionExecutor:8] 2026-03-20 15:25:27,447 Directories.java:553 -
> FileStore /mnt/cassandra (/dev/sdb1) has only 33.73 GiB available, but 89.39
> GiB is needed
> WARN [CompactionExecutor:8] 2026-03-20 15:25:27,447 CompactionTask.java:434
> - Not enough space for compaction (e08281c0-2457-11f1-9e8a-9937f6f9033a) of
> zerotohero.bulk_data_zstd, estimated sstables = 1, expected write size =
> 2195528
> ERROR [CompactionExecutor:8] 2026-03-20 15:25:27,447
> JVMStabilityInspector.java:70 - Exception in thread
> Thread[CompactionExecutor:8,5,CompactionExecutor]
> java.lang.RuntimeException: Not enough space for compaction
> (e08281c0-2457-11f1-9e8a-9937f6f9033a) of zerotohero.bulk_data_zstd,
> estimated sstables = 1, expected write size = 2195528
> at
> org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:436)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:148)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:94)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:374)
> at
> org.apache.cassandra.concurrent.FutureTask$3.call(FutureTask.java:141)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:840)
> DEBUG [CompactionExecutor:8] 2026-03-20 15:25:27,447 HeapUtils.java:133 -
> Heap dump creation on uncaught exceptions is disabled.
> ERROR [CompactionExecutor:12] 2026-03-20 16:09:13,125
> JVMStabilityInspector.java:70 - Exception in thread
> Thread[CompactionExecutor:12,5,CompactionExecutor]
> java.lang.RuntimeException: Not enough space for compaction
> (fd9ba330-245d-11f1-9e8a-9937f6f9033a) of zerotohero.bulk_data_deflate,
> estimated sstables = 1, expected write size = 2200455
> at
> org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:436)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:148)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:94)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:374)
> at
> org.apache.cassandra.concurrent.FutureTask$3.call(FutureTask.java:141)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:840)
> DEBUG [CompactionExecutor:12] 2026-03-20 16:09:13,126 HeapUtils.java:133 -
> Heap dump creation on uncaught exceptions is disabled. {code}
> while there is enough space:
> {code:java}
> root@zerotohero:~# df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sdb1 49G 4.5G 42G 10% /mnt/cassandra
> root@zerotohero:~# nodetool compactionstats -H
> concurrent compactors 2
>
> pending tasks 31
>
> zerotohero bulk_data 31
>
> compactions completed 5009
>
> data compacted 1.1 TiB
>
> compactions aborted 174
>
> compactions reduced 1
>
> sstables dropped from compaction 51
>
> 15 minute rate 5.33/minute
>
> mean rate 1226.87/hour
>
> compaction throughput (MiB/s) 64.0
>
> id compaction type keyspace table
> completed total unit progress
> c73b9e00-2451-11f1-9e8a-9937f6f9033a Compaction zerotohero bulk_data
> 58.16 GiB 136.33 GiB bytes 42.66%
> active compaction remaining time 0h20m50s {code}
> simple reproducer in go included
> cassandra version tested: 5.0.6, 5.0.7
> 4.1.11 works fine
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
