Klay created CASSANDRA-19623:
--------------------------------
Summary: Read fail with "illegal RT bounds sequence" after
migrating data from 2.2.19 to 3.0.30
Key: CASSANDRA-19623
URL: https://issues.apache.org/jira/browse/CASSANDRA-19623
Project: Cassandra
Issue Type: Bug
Components: Legacy/Local Write-Read Paths
Reporter: Klay
Attachments: cassandra.yaml, data.tar.gz, system.log
After migrating data from 2.2.19 to 3.0.30, reading the legacy data in 3.0.30
would fail with the following exception.
{code:java}
INFO [main] 2024-05-08 00:42:42,397 CassandraDaemon.java:653 - Startup complete
ERROR [SharedPool-Worker-2] 2024-05-08 00:42:49,975
AbstractLocalAwareExecutorService.java:166 - Uncaught exception on thread
Thread[SharedPool-Worker-2,10,main]
java.lang.RuntimeException: java.lang.IllegalStateException: SSTABLE
UnfilteredRowIterator for ks.tb has an illegal RT bounds sequence: unexpected
end bound or boundary Marker INCL_START_BOUND(1)@1715128933051779/1715128933
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2656)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.IllegalStateException: SSTABLE UnfilteredRowIterator for
ks.tb has an illegal RT bounds sequence: unexpected end bound or boundary
Marker INCL_START_BOUND(1)@1715128933051779/1715128933
at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.ise(RTBoundValidator.java:120)
at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.applyToMarker(RTBoundValidator.java:87)
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:144)
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77)
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:297)
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187)
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:180)
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:176)
at
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76)
at
org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:347)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1914)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2652)
... 5 common frames omitted {code}
h1. Reproduce
1. Start up cassandra-2.2.19, single node. Set column_index_size_in_kb in
cassandra.yaml to 1.
{code:java}
// cassandra.yaml
column_index_size_in_kb: 1 {code}
2. Execute the following commands (I masked out all data in the INSERT command
using long string of A to make it simple)
{code:java}
CREATE KEYSPACE ks WITH REPLICATION = { 'class' : 'SimpleStrategy',
'replication_factor' : 1 };
CREATE TABLE ks.tb (c3 INT, c7 INT, c0 TEXT,c6 set<TEXT>, PRIMARY KEY (c3, c7,
c0));DELETE FROM ks.tb WHERE c3 = 1 AND c7 = 1;
DELETE FROM ks.tb WHERE c3 = 1 AND c7 = 1 AND c0 = 'e';INSERT INTO ks.tb (c7,
c3, c6, c0) VALUES
(1,1,{'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA','AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA','AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'},'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA');
{code}
Perform a read in the old version, it works normally
{code:java}
cqlsh> SELECT c7, c0, c3 FROM ks.tb WHERE c3 = 1 AND c7 = 1 ORDER BY c7 DESC;
c7 | c0 | c3
----+--------------------------------------------------+----
1 | AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA | 1(1 rows) {code}
3. Drain and stop 2.2.19.
{code:java}
bin/nodetool -h ::FFFF:127.0.0.1 drain
bin/nodetool -h ::FFFF:127.0.0.1 stopdaemon; {code}
4. Copy data into 3.0.30 and start up (using default config is okay), read the
legacy data with the following read command
{code:java}
SELECT c7, c0, c3 FROM ks.tb WHERE c3 = 1 AND c7 = 1 ORDER BY c7 DESC; {code}
It would fails and cassandra throws an error in system.log.
{code:java}
➜ apache-cassandra-3.0.30 bin/cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.0.30-SNAPSHOT | CQL spec 3.4.0 | Native protocol v4]
Use HELP for help.
cqlsh> SELECT c7, c0, c3 FROM ks.tb WHERE c3 = 1 AND c7 = 1 ORDER BY c7 DESC;
ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read]
message="Operation failed - received 0 responses and 1 failures"
info={'failures': 1, 'received_responses': 0, 'required_responses': 1,
'consistency': 'ONE'} {code}
system.log
{code:java}
INFO [main] 2024-05-08 00:52:11,649 CassandraDaemon.java:653 - Startup complete
ERROR [SharedPool-Worker-2] 2024-05-08 00:52:13,695
AbstractLocalAwareExecutorService.java:166 - Uncaught exception on thread
Thread[SharedPool-Worker-2,10,main]
java.lang.RuntimeException: java.lang.IllegalStateException: SSTABLE
UnfilteredRowIterator for ks.tb has an illegal RT bounds sequence: unexpected
end bound or boundary Marker INCL_START_BOUND(1)@1715129489466439/1715129489
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2656)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.IllegalStateException: SSTABLE UnfilteredRowIterator for
ks.tb has an illegal RT bounds sequence: unexpected end bound or boundary
Marker INCL_START_BOUND(1)@1715129489466439/1715129489
at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.ise(RTBoundValidator.java:120)
at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.applyToMarker(RTBoundValidator.java:87)
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:144)
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77)
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:297)
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187)
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:180)
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:176)
at
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76)
at
org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:347)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1914)
at org.apache.cassandra.service {code}
I attached the data.tar.gz file generated from the 2.2.19, using 3.0.30 to
start with it and execute the above read command can directly reproduce it.
I also attached the cassandra.yaml file used in 2.2.19.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
