date:20101117

questions about SSTableExport/Import

2010-11-17 Thread aaron morton

I was trying to help this guy 
http://www.mail-archive.com/u...@cassandra.apache.org/msg07297.html who seemed 
to have troubles loading a json file. And I started taking a look at 
SSTableExport and SSTableImport.

SSTableExport does not encode any information about the Column sub type 
(ExpiringColumn or DeletedColumn). It records isMarkedForDelete(), the 
timestamp and the localDeletionTime as the col value if its a DeletedColumn. 
SSTableImport then calls either cf.addColumn() or cf.addTombstone() based on 
the deleted flag. 

First question is is the code in SSTableImport.addToStandardCF() correct to 
call cf.addColumn() if when the column was serialised it was 
isMarkedForDelete() ?

Next is it OK to lose the fact that a column is an ExpiringColumn (and its ttl) 
when it's exported to json? 

On my local machine I modified the unit test for SSTableExport as below and the 
assertion that the col was not returned failed.

Thanks
Aaron

diff --git a/test/unit/org/apache/cassandra/tools/SSTableExportTest.java 
b/test/unit/org/apache/cassandra/tools/SSTableExportTest.java
index 6f79f62..53d2a9c 100644
--- a/test/unit/org/apache/cassandra/tools/SSTableExportTest.java
+++ b/test/unit/org/apache/cassandra/tools/SSTableExportTest.java
@@ -179,6 +179,7 @@ public class SSTableExportTest extends SchemaLoader
 
 // Add rowA
 cfamily.addColumn(new QueryPath("Standard1", null, 
ByteBufferUtil.bytes("name")), ByteBufferUtil.bytes("val"), 1);
+cfamily.addColumn(new QueryPath("Standard1", null, 
ByteBufferUtil.bytes("ttl")), ByteBufferUtil.bytes("val"), 1, 1);
 writer.append(Util.dk("rowA"), cfamily);
 cfamily.clear();
 
@@ -187,6 +188,15 @@ public class SSTableExportTest extends SchemaLoader
 writer.append(Util.dk("rowExclude"), cfamily);
 cfamily.clear();
 
+//make sure the ttl col has expired
+try
+{
+Thread.sleep(1500);
+}
+catch (InterruptedException e)
+{
+throw new AssertionError(e);
+}
 SSTableReader reader = writer.closeAndOpenReader();
 
 // Export to JSON and verify
@@ -203,6 +213,11 @@ public class SSTableExportTest extends SchemaLoader
 assertTrue(cf != null);
 
assertTrue(cf.getColumn(ByteBufferUtil.bytes("name")).value().equals(ByteBuffer.wrap(hexToBytes("76616c";
 
+qf = QueryFilter.getNamesFilter(Util.dk("rowA"), new 
QueryPath("Standard1", null, null), ByteBufferUtil.bytes("ttl"));
+cf = qf.getSSTableColumnIterator(reader).getColumnFamily();
+assertTrue(cf != null);
+assertTrue(cf.getColumn(ByteBufferUtil.bytes("ttl")) == null);
+

Re: questions about SSTableExport/Import

2010-11-17 Thread Jonathan Ellis

On Wed, Nov 17, 2010 at 5:19 AM, aaron morton  wrote:
> I was trying to help this guy 
> http://www.mail-archive.com/u...@cassandra.apache.org/msg07297.html who 
> seemed to have troubles loading a json file. And I started taking a look at 
> SSTableExport and SSTableImport.
>
> SSTableExport does not encode any information about the Column sub type 
> (ExpiringColumn or DeletedColumn). It records isMarkedForDelete(), the 
> timestamp and the localDeletionTime as the col value if its a DeletedColumn. 
> SSTableImport then calls either cf.addColumn() or cf.addTombstone() based on 
> the deleted flag.
>
> First question is is the code in SSTableImport.addToStandardCF() correct to 
> call cf.addColumn() if when the column was serialised it was 
> isMarkedForDelete() ?

That does look backwards.

> Next is it OK to lose the fact that a column is an ExpiringColumn (and its 
> ttl) when it's exported to json?

It's okay in the sense that we make no effort to support TTL in json
export/import.  You're welcome to add that, but I don't consider it
being missing a bug.

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: CASSANDRA-1472 (bitmap indexes)

2010-11-17 Thread dragos cernahoschi

Back. I've tested the keys index pagination once again. 0.7 head. Smaller
data set: 1 million rows. It seems there are still some issues:

1. *test*: query on one column, count: 1000, expected number of distinct
results: 48251
*result*: 5 pages of 1000 results, than, after the 6th page, the results
begin to repeat, I would expect that repetition begins after the 48251-th
row

2. *test*: query on 3 columns, count: 10 (count 100, count 1000 failed with
time out)
*result*: 1 page of 10 results, than second page => time out

3. There are queries with combinations of 2, 3 columns that fail right away
with time out (count 10, 100).

Dragos

On Mon, Nov 15, 2010 at 2:29 PM, Jonathan Ellis  wrote:

> On Mon, Nov 15, 2010 at 5:57 AM, dragos cernahoschi
>  wrote:
> > I've tested 0.7-beta3 branch index feature without the 1472 patch. The
> > queries on more than one column works better than the patched version,
> but
> > definitely not correctly.
>
> Please test 0.7 branch head, as you can see from the CHANGES there
> have been a lot of fixes.
>
> > 1.
> > 2.
> > 4.
>
> Should be fixed in head.
>
> > 3. Is there any example on the pagination feature? (without knowing the
> > expected number of rows).
>
> Same way you paginate through range slices or columns within a row,
> set start to the last result you got w/ previous query.
>
> > Will the get_indexed_slices return an empty list when there is no more
> > results?
>
> No, all queries are start-inclusive.
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Build failed in Hudson: Cassandra-0.7 #11

2010-11-17 Thread Apache Hudson Server

See 

Changes:

[jbellis] merge from 0.6

--
[...truncated 1593 lines...]
[junit] Testsuite: 
org.apache.cassandra.dht.CollatingOrderPreservingPartitionerTest
[junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 15.174 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.dht.OrderPreservingPartitionerTest
[junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 8.091 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.dht.RandomPartitionerTest
[junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 4.086 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.dht.RangeTest
[junit] Tests run: 12, Failures: 0, Errors: 0, Time elapsed: 0.498 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.gms.ArrivalWindowTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.135 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.gms.GossipDigestTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.071 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.hadoop.ColumnFamilyInputFormatTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.216 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.BloomFilterTrackerTest
[junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.376 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.LazilyCompactedRowTest
[junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 20.461 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.sstable.LegacySSTableTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.718 sec
[junit] 
[junit] - Standard Error -
[junit]  WARN 20:20:45,499 Invalid file '.svn' in data directory 

[junit] -  ---
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.sstable.SSTableReaderTest
[junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 5.927 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.sstable.SSTableTest
[junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 7.196 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.sstable.SSTableWriterTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.329 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.util.BufferedRandomAccessFileTest
[junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.729 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.locator.DynamicEndpointSnitchTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.222 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.locator.NetworkTopologyStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.516 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: 
org.apache.cassandra.locator.OldNetworkTopologyStrategyTest
[junit] Tests run: 3, Failures: 0,

Build failed in Hudson: Cassandra-0.7 #12

2010-11-17 Thread Apache Hudson Server

See 

Changes:

[jbellis] index scan treats missing columns as not matching secondary 
expressions
patch by jbellis; reviewed by Stu Hood for CASSANDRA-1745

--
[...truncated 1576 lines...]
[junit] Testsuite: 
org.apache.cassandra.dht.CollatingOrderPreservingPartitionerTest
[junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 15.37 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.dht.OrderPreservingPartitionerTest
[junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 8.069 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.dht.RandomPartitionerTest
[junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 3.824 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.dht.RangeTest
[junit] Tests run: 12, Failures: 0, Errors: 0, Time elapsed: 0.521 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.gms.ArrivalWindowTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.138 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.gms.GossipDigestTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.069 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.hadoop.ColumnFamilyInputFormatTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.223 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.BloomFilterTrackerTest
[junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.393 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.LazilyCompactedRowTest
[junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 21.101 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.sstable.LegacySSTableTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.705 sec
[junit] 
[junit] - Standard Error -
[junit]  WARN 20:53:58,002 Invalid file '.svn' in data directory 

[junit] -  ---
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.sstable.SSTableReaderTest
[junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 6.4 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.sstable.SSTableTest
[junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 7.031 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.sstable.SSTableWriterTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.283 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.util.BufferedRandomAccessFileTest
[junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.767 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.locator.DynamicEndpointSnitchTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.246 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.locator.NetworkTopologyStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.488 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit

Build failed in Hudson: Cassandra-0.7 #13

2010-11-17 Thread Apache Hudson Server

See 

Changes:

[jbellis] truncate includes secondary indexes
patch by jbellis; reviewed by gdusbabek for CASSANDRA-1747

--
[...truncated 1621 lines...]
[junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.374 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.LazilyCompactedRowTest
[junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 3.006 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.sstable.LegacySSTableTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.705 sec
[junit] 
[junit] - Standard Error -
[junit]  WARN 00:12:46,151 Invalid file '.svn' in data directory 

[junit] -  ---
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.sstable.SSTableReaderTest
[junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 1.279 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.sstable.SSTableTest
[junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 6.751 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.sstable.SSTableWriterTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.771 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.io.util.BufferedRandomAccessFileTest
[junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.11 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.locator.DynamicEndpointSnitchTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.224 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.locator.NetworkTopologyStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.492 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: 
org.apache.cassandra.locator.OldNetworkTopologyStrategyTest
[junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.438 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: 
org.apache.cassandra.locator.ReplicationStrategyEndpointCacheTest
[junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.748 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.locator.SimpleStrategyTest
[junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.791 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.locator.TokenMetadataTest
[junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.493 sec
[junit] 
[junit] Cobertura: Loaded information on 956 classes.
[junit] Cobertura: Saved information on 956 classes.
[junit] Testsuite: org.apache.cassandra.service.AntiEntropyServiceTest
[junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 4.024 sec
[junit] 
[junit] - Standard Error -
[junit]  WARN 00:13:39,508 Generated random token 
Token(bytes[31b0966e16f19aaa4acfcee8b32cd34b]). Random tokens will result in an 
unbalanced ring; see http://wiki.apache.org/cassandra/Operations
[junit] -  ---
[junit] Exception in thread "Thread-5" java.lang.RuntimeException: 
java.lang.NullPointerException
[junit] at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
[junit] at 
org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:162)
[junit] at 
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:66)
[junit] at 
or

Hudson build is back to normal : Cassandra-0.7 #14

2010-11-17 Thread Apache Hudson Server

See

questions about SSTableExport/Import

Re: questions about SSTableExport/Import

Re: CASSANDRA-1472 (bitmap indexes)

Build failed in Hudson: Cassandra-0.7 #11

Build failed in Hudson: Cassandra-0.7 #12

Build failed in Hudson: Cassandra-0.7 #13

Hudson build is back to normal : Cassandra-0.7 #14

7 matches

Site Navigation

Mail list logo

Footer information