[
https://issues.apache.org/jira/browse/CASSANDRA-21245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18068004#comment-18068004
]
vasya b edited comment on CASSANDRA-21245 at 3/24/26 2:02 PM:
--------------------------------------------------------------
5.0.7:
{code:java}
concurrent compactors 2
pending tasks 22
zerotohero bulk_data_deflate 22
compactions completed 2378
data compacted 32.82 GiB
compactions aborted 47
compactions reduced 0
sstables dropped from compaction 0
15 minute rate 42.09/minute
mean rate 3981.42/hour
compaction throughput (MiB/s) 64.0
id compaction type keyspace table
completed total unit progress
1dcc51b0-2781-11f1-9986-a70799cee76a Compaction zerotohero
bulk_data_deflate 16.11 GiB 31.83 GiB bytes 50.63%
active compaction remaining time 0h04m11s {code}
tablestats:
{code:java}
Total number of tables: 1
----------------
Keyspace: zerotohero
Read Count: 0
Read Latency: NaN ms
Write Count: 103688
Write Latency: 0.24912840444410153 ms
Pending Flushes: 0
Table: bulk_data_deflate
SSTable count: 150
Old SSTable count: 0
Max SSTable size: 41.670MiB
Space used (live): 507 MiB
Space used (total): 507 MiB
Space used by snapshots (total): 0 bytes
Off heap memory used (total): 387.16 MiB
SSTable Compression Ratio: 0.00440
Number of partitions (estimate): 103745
Memtable cell count: 324
Memtable data size: 324.01 MiB
Memtable off heap memory used: 324.02 MiB
Memtable switch count: 203
Speculative retries: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 103688
Local write latency: 0.250 ms
Local read/write ratio: 0.00000
Pending flushes: 0
Percent repaired: 0.0
Bytes repaired: 0B
Bytes unrepaired: 100.944GiB
Bytes pending repair: 0B
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 168.36 KiB
Bloom filter off heap memory used: 167.19 KiB
Index summary off heap memory used: 0 bytes
Compression metadata off heap memory used: 62.97 MiB
Compacted partition minimum bytes: 943128
Compacted partition maximum bytes: 1131752
Compacted partition mean bytes: 1131752
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Droppable tombstone ratio: 0.00000----------------
{code}
df -h:
{code:java}
/dev/sdb1 9.8G 555M 8.7G 6% /mnt/cassandra {code}
[^debug.log]
table:
{code:java}
CREATE TABLE zerotohero.bulk_data_deflate (
pk bigint PRIMARY KEY,
data text
) WITH additional_write_policy = '99p'
AND allow_auto_snapshot = true
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND cdc = false
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.UnifiedCompactionStrategy',
'scaling_parameters': 'T4'}
AND compression = {'chunk_length_in_kb': '16', 'class':
'org.apache.cassandra.io.compress.DeflateCompressor'}
AND memtable = 'default'
AND crc_check_chance = 1.0
AND default_time_to_live = 0
AND extensions = {}
AND gc_grace_seconds = 864000
AND incremental_backups = true
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair = 'BLOCKING'
AND speculative_retry = '99p';
{code}
was (Author: JIRAUSER312838):
5.0.7:
{code:java}
concurrent compactors 2
pending tasks 22
zerotohero bulk_data_deflate 22
compactions completed 2378
data compacted 32.82 GiB
compactions aborted 47
compactions reduced 0
sstables dropped from compaction 0
15 minute rate 42.09/minute
mean rate 3981.42/hour
compaction throughput (MiB/s) 64.0
id compaction type keyspace table
completed total unit progress
1dcc51b0-2781-11f1-9986-a70799cee76a Compaction zerotohero
bulk_data_deflate 16.11 GiB 31.83 GiB bytes 50.63%
active compaction remaining time 0h04m11s {code}
tablestats:
{code:java}
Total number of tables: 1
----------------
Keyspace: zerotohero
Read Count: 0
Read Latency: NaN ms
Write Count: 103688
Write Latency: 0.24912840444410153 ms
Pending Flushes: 0
Table: bulk_data_deflate
SSTable count: 150
Old SSTable count: 0
Max SSTable size: 41.670MiB
Space used (live): 507 MiB
Space used (total): 507 MiB
Space used by snapshots (total): 0 bytes
Off heap memory used (total): 387.16 MiB
SSTable Compression Ratio: 0.00440
Number of partitions (estimate): 103745
Memtable cell count: 324
Memtable data size: 324.01 MiB
Memtable off heap memory used: 324.02 MiB
Memtable switch count: 203
Speculative retries: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 103688
Local write latency: 0.250 ms
Local read/write ratio: 0.00000
Pending flushes: 0
Percent repaired: 0.0
Bytes repaired: 0B
Bytes unrepaired: 100.944GiB
Bytes pending repair: 0B
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 168.36 KiB
Bloom filter off heap memory used: 167.19 KiB
Index summary off heap memory used: 0 bytes
Compression metadata off heap memory used: 62.97 MiB
Compacted partition minimum bytes: 943128
Compacted partition maximum bytes: 1131752
Compacted partition mean bytes: 1131752
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Droppable tombstone ratio: 0.00000----------------
{code}
df -h:
{code:java}
/dev/sdb1 9.8G 555M 8.7G 6% /mnt/cassandra {code}
[^debug.log]
> Uncompressed size is being used for compressed tables in maintenance
> operations
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-21245
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21245
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: vasya b
> Priority: Normal
> Attachments: cassandra.yaml, debug.log, go.mod, main.go
>
>
> Using a compressed table can lead to a state where uncompressed table size is
> bigger
> than the whole volume size, while it shouldn't be a problem it leads to
> compaction poblems, as other compaction task are being denied.
> f.e. one can easily create a compressed table with
> {code:java}
> CREATE TABLE IF NOT EXISTS ` + keyspace + `.` + table + ` (
> pk bigint,
> data text,
> PRIMARY KEY (pk)
> ) WITH compression = {'class': 'DeflateCompressor'}
> {code}
> (or any other compression)
> Insert a bunch of data which is going to be compressed quite well (f.e. a
> bunch of zeros or any other single char * 1024 *1024) * N where N is some big
> number
> one would expect that such table will be compacted without any problems.
> but one will get:
> {code:java}
> WARN [CompactionExecutor:8] 2026-03-20 14:25:17,256 CompactionTask.java:434
> - Not enough space for compaction (78c5c7c0-244f-11f1-9e8a-9937f6f9033a) of
> zerotohero.bulk_data, estimated sstables = 1, expected write size = 2686214
> ERROR [CompactionExecutor:8] 2026-03-20 14:25:17,256
> JVMStabilityInspector.java:70 - Exception in thread
> Thread[CompactionExecutor:8,5,CompactionExecutor]
> java.lang.RuntimeException: Not enough space for compaction
> (78c5c7c0-244f-11f1-9e8a-9937f6f9033a) of zerotohero.bulk_data, estimated
> sstables = 1, expected write size = 2686214
> at
> org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:436)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:148)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:94)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:374)
> at
> org.apache.cassandra.concurrent.FutureTask$3.call(FutureTask.java:141)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:840)
> WARN [CompactionExecutor:9] 2026-03-20 15:21:27,534 CompactionTask.java:440
> - Not enough space for compaction 51834c70-2457-11f1-9e8a-9937f6f9033a,
> 71.31226MiB estimated. Reducing scope.
> DEBUG [CompactionExecutor:8] 2026-03-20 15:25:27,447 Directories.java:550 -
> FileStore /mnt/cassandra (/dev/sdb1) has 36221336371 bytes available,
> checking if we can write 95979501107 bytes
> WARN [CompactionExecutor:8] 2026-03-20 15:25:27,447 Directories.java:553 -
> FileStore /mnt/cassandra (/dev/sdb1) has only 33.73 GiB available, but 89.39
> GiB is needed
> WARN [CompactionExecutor:8] 2026-03-20 15:25:27,447 CompactionTask.java:434
> - Not enough space for compaction (e08281c0-2457-11f1-9e8a-9937f6f9033a) of
> zerotohero.bulk_data_zstd, estimated sstables = 1, expected write size =
> 2195528
> ERROR [CompactionExecutor:8] 2026-03-20 15:25:27,447
> JVMStabilityInspector.java:70 - Exception in thread
> Thread[CompactionExecutor:8,5,CompactionExecutor]
> java.lang.RuntimeException: Not enough space for compaction
> (e08281c0-2457-11f1-9e8a-9937f6f9033a) of zerotohero.bulk_data_zstd,
> estimated sstables = 1, expected write size = 2195528
> at
> org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:436)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:148)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:94)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:374)
> at
> org.apache.cassandra.concurrent.FutureTask$3.call(FutureTask.java:141)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:840)
> DEBUG [CompactionExecutor:8] 2026-03-20 15:25:27,447 HeapUtils.java:133 -
> Heap dump creation on uncaught exceptions is disabled.
> ERROR [CompactionExecutor:12] 2026-03-20 16:09:13,125
> JVMStabilityInspector.java:70 - Exception in thread
> Thread[CompactionExecutor:12,5,CompactionExecutor]
> java.lang.RuntimeException: Not enough space for compaction
> (fd9ba330-245d-11f1-9e8a-9937f6f9033a) of zerotohero.bulk_data_deflate,
> estimated sstables = 1, expected write size = 2200455
> at
> org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:436)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:148)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:94)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:374)
> at
> org.apache.cassandra.concurrent.FutureTask$3.call(FutureTask.java:141)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:840)
> DEBUG [CompactionExecutor:12] 2026-03-20 16:09:13,126 HeapUtils.java:133 -
> Heap dump creation on uncaught exceptions is disabled. {code}
> while there is enough space:
> {code:java}
> root@zerotohero:~# df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sdb1 49G 4.5G 42G 10% /mnt/cassandra
> root@zerotohero:~# nodetool compactionstats -H
> concurrent compactors 2
>
> pending tasks 31
>
> zerotohero bulk_data 31
>
> compactions completed 5009
>
> data compacted 1.1 TiB
>
> compactions aborted 174
>
> compactions reduced 1
>
> sstables dropped from compaction 51
>
> 15 minute rate 5.33/minute
>
> mean rate 1226.87/hour
>
> compaction throughput (MiB/s) 64.0
>
> id compaction type keyspace table
> completed total unit progress
> c73b9e00-2451-11f1-9e8a-9937f6f9033a Compaction zerotohero bulk_data
> 58.16 GiB 136.33 GiB bytes 42.66%
> active compaction remaining time 0h20m50s {code}
> simple reproducer in go included
> cassandra version tested: 5.0.6, 5.0.7
> 4.1.11 works fine
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
