findbugs

[Radim Kolar]

I used findbugs on cassandra and it returns 69 possible errors.

most problematic part of code is CQL - lot of null pointer problems there

some interesting errors:

C:/apache-nutch/eclipse/cassandra/src/java/org/apache/cassandra/service/AntiEntropyService.java:916 
Condition.await() not in loop in 
org.apache.cassandra.service.AntiEntropyService$RepairSession$RepairJob.addTree(AntiEntropyService$TreeRequest, 
MerkleTree)


C:/apache-nutch/eclipse/cassandra/src/java/org/apache/cassandra/service/StorageProxy.java:370 
Null passed for nonnull parameter of 
org.apache.cassandra.utils.UUIDGen.decompose(UUID) in 
org.apache.cassandra.service.StorageProxy$5.runMayThrow()


C:/apache-nutch/eclipse/cassandra/src/java/org/apache/cassandra/db/compaction/CompactionManager.java:157 
org.apache.cassandra.db.compaction.CompactionManager$2.call() does not 
release lock on all paths


C:/apache-nutch/eclipse/cassandra/src/java/org/apache/cassandra/db/compaction/CompactionManager.java:248 
org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow() 
does not release lock on all paths


C:/apache-nutch/eclipse/cassandra/src/java/org/apache/cassandra/utils/SimpleCondition.java:34 
Monitor wait() called on a Condition in 
org.apache.cassandra.utils.SimpleCondition.await() ** important! **


C:/apache-nutch/eclipse/cassandra/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java:224 
Result of integer multiplication cast to long in 
org.apache.cassandra.db.compaction.LeveledManifest.maxBytesForLevel(int)

Re: findbugs

[Radim Kolar]

The line numbers here don't appear to match with trunk.

you are right, it was from old trunk 415 commits old. It was just demo 
of findbugs, for serious use developers should install findbugs maven 
plugin or eclipse plugin (preferred).

Re: findbugs

[Radim Kolar]

Dne 23.7.2012 16:34, Zoltan Farkas napsal(a):

In general, I prefer integrating findbugs into the build process and fail the 
build if issues are found. I am a strong believer in this approach, increases 
the quality of the project significantly.
Thats true, i am currently in process of fixing findbugs discovered 
problems in my other project (120k lines). i was amazed how much bugs 
was left in the code even if we have quite good code coverage.

Re: findbugs

[Radim Kolar]

was any decision about findbugs made? you do not consider code style 
recommended by findbugs as good practice which should be followed?


I can submit few findbugs patches, but it will probably turns into 
flamewar WE vs FINDBUGS like there:

https://issues.apache.org/jira/browse/HADOOP-8619

findbugs problems are pretty easy to fix and there are just 70 of them, 
it could be done in two days.


I do not care about findbugs+cas-dev issue much because i need to fork 
cassandra anyway to get performance patches there. Its just matter of 
schedule for me if 1 should feed you fb patches before i fork it.

Re: findbugs

[Radim Kolar]

i am using maven to build cassandra. i didnt have in mind to contribute 
build system because you are not interested in maven. In maven you just 
call findbugs plugin, nothing special to contribute. I had in mind patch 
fixing various FB discovered problems. but because its difficult to post 
it as many JIRA tickets, it will not be much useful. Maybe post report 
to list?


you can create exlude list for FB with list of known false positives, 
its an xml file.




  

Re: findbugs

[Radim Kolar]

Dne 30.7.2012 16:52, Jonathan Ellis napsal(a):

Is Jenkins smart enough to be able to say, "I know we had X findbugs
warnings previously, which are known to be false positives, but now
there are X+1" ?

yes. Look at hadoop project pre-commit check builds.

customizable size tiered compaction

[Radim Kolar]

I am interested in experiments with size tiered compaction, because i 
get sstables which are never compacted because no other sstable is close 
to their size, i have plans to experiment with bucket ratio which is 
currently 50-150 percent to make it 33-200 percent.


Its all about changing constants at lines 170-171

https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java

if ((size > (oldAverageSize / 2) && size < (3 * oldAverageSize) / 2)

|| (size < minSSTableSize && oldAverageSize < minSSTableSize))

to make these constant configurable, i suppose something similar to 
"min_sstable_size" is needed. Its min_sstable_size configurable by CF or 
its node wide setup?

Re: customizable size tiered compaction

[Radim Kolar]

Dne 22.9.2012 16:01, Jonathan Ellis napsal(a):

It's a compaction strategy option, so it's cluster-wide for that strategy.

coded in https://issues.apache.org/jira/browse/CASSANDRA-4704

maximum sstable size

[Radim Kolar]

its possible to implement maximum sstable size for 
tieredcompactionpolicy without much code changes?


I am using it in lucene with really good performance effect, max size is 
4 GB, dataset total size is 30 GB. It prevents lucene from creating too 
big segment which takes too long to be merged with newer segments.

Re: maximum sstable size

[Radim Kolar]

done
https://issues.apache.org/jira/browse/CASSANDRA-4897

Re: maximum sstable size

[Radim Kolar]

Dne 4.11.2012 1:24, Edward Capriolo napsal(a):

I have another ticket open for this.

which one

Re: findbugs

[Radim Kolar]

Dne 30.7.2012 16:47, Edward Capriolo napsal(a):

I am sure no one would have an issue with an optional findbugs target.

https://issues.apache.org/jira/browse/CASSANDRA-4891
here you have optional findbugs target.

slf4j

[Radim Kolar]

instead of this:

   if (logger.isDebugEnabled())
logger.debug("INDEX LOAD TIME for " + descriptor + ": " + 
(System.currentTimeMillis() - start) + " ms.");


do this:
logger.debug("INDEX LOAD TIME for {} : {} ms.", descriptor, 
(System.currentTimeMillis() - start));


easier to read.

Re: Hector 0.8.0-2 update fails with : "All host pools marked down. Retry burden pushed out to client."

[Radim Kolar]

after 1 year experience with Hector, i would recommend: stay away from 
hector if possible.

Re: Hector 0.8.0-2 update fails with : "All host pools marked down. Retry burden pushed out to client."

[Radim Kolar]

Dne 3.12.2012 9:15, Bisht, Jaikrit napsal(a):

Hi there,

What have been the problems with Hector?

problems with improper detection of down nodes
problems with improper detection of timeouts
some lost updates due to bad timestamp generation, spliting into more 
mutators helped.

lack of support for all data types

max_compaction_threshold removed - bad move

[Radim Kolar]

removing max_compaction_threshold in 1.2 was bad move, keeping it low 
helps compaction throughput because it lowers number of disk seeks.


if you have redhat linux, check during install category "performance 
tools" or something like that, you will get tools for disk monitoring. 
Learn to use them.

Re: max_compaction_threshold removed - bad move

[Radim Kolar]

if this was renamed then update your documentation:

http://www.datastax.com/docs/1.2/configuration/storage_configuration

real leveldb vs cassandra leveldb

[Radim Kolar]

real leveldb is better in lot of areas:

L0 are 1/10 of L1 sstable size
tables can be promoted to upper levels if no merging is needed (there is 
hole)
variable number of sstables per level, but it tries to keep 1:10:100 
sstable ratios. Not hard requirement
very important - better hash function. murmur and md5 hashes are 
unsuitable for ldb because it turns key sequence into more or less 
random noise. Changing hash function to leveldb one gives about 8 times 
speed increase during seq. writes because far less table merges is needed
better merge policy: merge 1 table + up to 10 tables from next level 
into next + 2 level.

Re: real leveldb vs cassandra leveldb

[Radim Kolar]

Dne 13.2.2013 16:32, Jonathan Ellis napsal(a):

The only point here that would make a difference in practice is
leveldb using a worse hash function.  For us it's not worth making
partitioning worse to make compaction better.
then use two hash functions. one for spliting rows to nodes and second 
for index inside leveldb.


too much compactions is major problem of leveldb implementation in 
cassandra. Cassandra implements initial leveldb design ignoring 
improvements in about last 2 years in leveldb. You can try it yourself 
by benchmarking original and current leveldb.

Re: real leveldb vs cassandra leveldb

[Radim Kolar]

Dne 13.2.2013 16:32, Jonathan Ellis napsal(a):

The only point here that would make a difference in practice is
leveldb using a worse hash function.
how do you know that it would not make difference in practice. i have 
implemented some optimalization from leveldb to cassandra - different L0 
level - 12 tables of 1/10 size L1, faster promotion to next level and 
variable number of sstables per level and it performs faster for write 
heavy workload.


i do not understand why you are not interested in performance 
optimalizations. For example yesterday i did buffered sstable writing 
and it turned to be significant perfomance advantage using 1mb buffer 
changed test time from 1m50s to 0m40s. Another significant gain is to 
use read ahead in compactions and replace bucket based size tiered 
compaction to compaction strategy used in lucene - it did not produce 
immortal sstables like cassandra does.


i have quite high demands for perfomance, my test environment for new 
project has about 1 bilion new rows per day, in production it will be 
about 30 times higher, so what is worth coding for me might not be worth 
coding for you.

Re: Time to roll 1.1.12?

[Radim Kolar]

  * fsync leveled manifest to avoid corruption (CASSANDRA-5535)
you sure that this does not have performance impact? most filesystems 
sync all their data not just one file. write to .new file and then do 
rename.

cassie

[Radim Kolar]

http://wiki.apache.org/cassandra/ClientOptionsThrift

add cassie - https://github.com/twitter/cassie

EOL info

[Radim Kolar]

Dne 22.5.2013 18:22, Brandon Williams napsal(a):

I don't see a 1.1.13 ever happening
do you have some page with EOL information something like 
http://www.freebsd.org/security/


lifetime is about 6 months per major release?

manifest fsync

[Radim Kolar]

should be fine, the manifest is not rewritten that often
its rewritten after each sstable flush? databases should do fsync only 
in checkpoint.


Fsync scenario in WAFL is that hard checkpoint is done after predefined 
number of log segments. On checkpoint fsync everything and then write 
checkpoint marker to log, then remove checkpointed logs.

cassandra vnodes

[Radim Kolar]

cassandra vnodes are just implementation of consistent hashing or there 
are some improvements to make similar sized split sizes?


I decided to implement it in my cassandra too, but i am using zookeeper 
for cluster management.

wiki bootstrap doc

[Radim Kolar]

http://wiki.apache.org/cassandra/Operations*

*
To bootstrap a node, turn _AutoBootstrap_ on in the configuration file, 
and start it. *


*its called *auto_bootstrap *in config file

Re: cassandra vnodes

[Radim Kolar]

I decided to implement it in my cassandra too, but i am using 
zookeeper for cluster management.
I scraped idea of consistent hashing with random tokens. There is too 
much variance with effective ranges allocated to nodes in large cluster, 
you need to have lot of ranges which is quite large overhead because of 
suboptimal implementation in current Cassandra.


I decided to go for mtrees on top of clock allocator. Because mtrees can 
overlap they can take care of replication as well.


there is couple of bugs in Cassandra 1.2 after you enable vnodes, but 
you will never warn user about vnodes not production quality, don't you? 
Look at Redhat, some features are marked "Tech preview". Your thinking 
is: ppl should test it before putting it into production. You can't 
expect user to test all possible code paths.

Apache Cassandra 2.0.0 rc1

[Radim Kolar]

did you package it correctly? something seems to be missing

ERROR 20:27:11,578 Internal error processing batch_mutate
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.cassandra.

triggers.TriggerExecutor
at 
org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageP

roxy.java:535)
at 
org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.

java:1062)

Re: Apache Cassandra 2.0.0 rc1

[Radim Kolar]

Dne 10.8.2013 21:30, Brandon Williams napsal(a):

Make a conf/triggers directory and that will fix it. We fixed this in trunk 
already.

yes, that fixed it.

2.0 is considerably slower then 1.2 for cpu bound tasks, average 
throughput is -15% at 50 threads.

2.0 with 20 threads burst thruput with is just 1/2 of 1.2 at 50 threads.

Re: Apache Cassandra 2.0.0 rc1

[Radim Kolar]

thrift, sync

Re: [VOTE] Release Apache Cassandra 2.0.0-rc2

[Radim Kolar]

what about failing shuffle?

CASSANDRA-5876 
https://issues.apache.org/jira/browse/CASSANDRA-5873

avro in binary distro

[Radim Kolar]

Cassandra binary distribution contains avro-1.4.0-sources-fixes.jar 
which is source code jar. It is packaged by mistake?

slow read performance with leveldb compactor

[Radim Kolar]

Lets say i have this:

{
  "generations" : [ {
"generation" : 0,
"members" : [ 650, 651, 652, 653, 654 ]
  }, {
"generation" : 1,
"members" : [ 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648 ]
  }, {
"generation" : 2,
"members" : [ 566, 575, 576, 578, 579, 580, 582, 583, 584, 586, 
587, 588, 590, 591, 592, 593, 595, 596, 597, 599, 600, 601, 603, 604, 
605, 607, 608, 609, 611, 612, 613, 615, 616, 617, 618, 620, 621, 622, 623 ]


If i understand leveldb design documents correctly then key can be in 
any lvl0 sstable (standard bloom filters will be used) but for lvl1+ its 
enough to check only sstable containing key range because they do not 
overlap.


So we need to check/merge all L0 tables + one from L1 + one from L2. 
Because number of sstables needed to check is low It should not be more 
performance heavy than checking similar number of sstables (all L0 + 2 
levels) with old compactor.


This is probably not done yet because my current read performance is 
3-4x slower than expected. because i have currently 60 sstables in 
lvl0-2 and read latency 48.4ms on 7200 rpm drive which is about 5.3 
seeks per read.

Better CF stats

[Radim Kolar]

I need more detailed CF stats. Currently CASS supports Read/write stats 
and cache hit ratio.


I am interested in:

1. key not found: like get cf['non-existent-key']
2. hits to tombstone, row existed but it is tombstoned now

It this easy enough to implement?

Re: [VOTE] Release Apache Cassandra 1.0.3

[Radim Kolar]

-1 unless  (CASSANDRA-3466) is included

Re: [VOTE] Release Apache Cassandra 1.0.3

[Radim Kolar]

-1

fix for (CASSANDRA-3466) is included (no exception this time) but hints 
are not delivered to other node:


 INFO [GossipTasks:1] 2011-11-12 15:05:35,001 Gossiper.java (line 759) 
InetAddress /**.99.40 is now dead.
 WARN [pool-1-thread-1] 2011-11-12 15:06:11,514 Memtable.java (line 
169) setting live ratio to minimum of 1.0 instead of 0.5090741051924046
 INFO [pool-1-thread-1] 2011-11-12 15:06:11,514 Memtable.java (line 
179) CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio 
is 1.0 (just-counted was 1.0).  calculation took 2ms for 64 columns
 WARN [pool-1-thread-1] 2011-11-12 15:06:55,967 Memtable.java (line 
169) setting live ratio to minimum of 1.0 instead of 0.42475558504755584
 INFO [pool-1-thread-1] 2011-11-12 15:06:55,967 Memtable.java (line 
179) CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio 
is 1.0 (just-counted was 1.0).  calculation took 3ms for 128 columns


[hints are available, now node is back again]

 INFO [GossipStage:1] 2011-11-12 15:08:10,038 Gossiper.java (line 777) 
Node /*.99.40 has restarted, now UP
 INFO [GossipStage:1] 2011-11-12 15:08:10,038 Gossiper.java (line 745) 
InetAddress /.99.40 is now UP
 INFO [GossipStage:1] 2011-11-12 15:08:10,039 StorageService.java (line 
871) Node /*.40 state jump to normal
 INFO [OptionalTasks:1] 2011-11-12 15:12:32,869 MeteredFlusher.java 
(line 62) flushing high-traffic column family CFS(Keyspace='rapidshare', 
ColumnFamily='resultcache') (estimated 142007760 bytes)
 INFO [OptionalTasks:1] 2011-11-12 15:12:32,869 ColumnFamilyStore.java 
(line 688) Enqueuing flush of 
Memtable-resultcache@1897226876(1775097/142007760 serialized/live bytes, 
10852 ops)
 INFO [FlushWriter:8] 2011-11-12 15:12:32,870 Memtable.java (line 239) 
Writing Memtable-resultcache@1897226876(1775097/142007760 
serialized/live bytes, 10852 ops)
 INFO [FlushWriter:8] 2011-11-12 15:12:33,081 Memtable.java (line 275) 
Completed flushing 
/usr/local/cassandra/data/rapidshare/resultcache-hb-2238-Data.db 
(2066384 bytes)


[no message about hints delivery for 8 minutes after node is back up 
again]. I tested it with inserting test data and they are not available 
on other node - problem is not lack of message. Please test it too.

Re: [VOTE] Release Apache Cassandra 1.0.3

[Radim Kolar]

-1. fix for (CASSANDRA-3466) is included (no exception this time) but 
hints are not delivered to other node:

anybody tested this? Did included fix worked for you?

Re: [VOTE] Release Apache Cassandra 1.0.3

[Radim Kolar]

I'm not sure why hints are not working for you. You might have hit some 
other issue. Some suggestions:


1. Verify that HintsColumnFamily actually contains some data with 
cassandra-cli and "list HintsColumnFamily"

yes

2. Try restarting the node containing the hints to check if that gets 
your hints delivered (gossip state related issue?).
 INFO [HintedHandoff:1] 2011-11-14 12:35:15,557 
HintedHandOffManager.java (line 334) Finished hinted handoff of 0 rows 
to endpoint /***99.40
Worked. This is improvement. at least node tried to deliver hints. So 
conclusion was that keeping gossip state still has some issues?


3. but hints table is not empty after that. majority of hints 
disappeared but this was left in table. Take a look at it.


[default@system] list HintsColumnFamily;
Using default limit of 100
---
RowKey: 4241c76b735b154119e2dd30
=> (super_column=9b7873500eb411e1b1504fb874bb,
 (column=6b6579, value=4f6e646120436869726f, 
timestamp=1321270466319, ttl=864000)
 (column=6d75746174696f6e, 
value=000a72617069647368617265000a4f6e646120436869726f000103ea0103ea800080010007726573756c74730200093a804eca37420004b1b040cb95980008, 
timestamp=1321270466309, ttl=864000)
 (column=7461626c65, value=72617069647368617265, 
timestamp=1321270466319, ttl=864000)
 (column=76657273696f6e, value=0003, timestamp=1321270466319, 
ttl=864000))

=> (super_column=9d7a84e00eb411e1b1504fb874bb,
 (column=6b6579, value=7069676c6574, timestamp=1321270469678, 
ttl=864000)
 (column=6d75746174696f6e, 
value=000a7261706964736861726500067069676c6574000103e90103e94ec0fcc50004b1b040fef5f0, 
timestamp=1321270469678, ttl=864000)
 (column=7461626c65, value=72617069647368617265, 
timestamp=1321270469678, ttl=864000)
 (column=76657273696f6e, value=0003, timestamp=1321270469678, 
ttl=864000))


1 Row Returned.
Elapsed time: 5 msec(s).

hintedhandoff in 1.0.3

[Radim Kolar]

I suspect these partial/invalid hints are left over from a failed 
hints delivery from before you upgraded to 1.0.3 and not something 
created by 1.0.3.
Try to clear HintsColumnFamily (by removing the sstables for example) 
first and then see if you still can reproduce this issue afterwards.
it still does not work, but there is an improvement over 1.0.2. 
currently it looks like this:


 INFO [HintedHandoff:1] 2011-11-15 06:42:59,794 
HintedHandOffManager.java (line 334) Finished hinted handoff of 0 rows 
to endpoint /


but there is lot of data not delivered. forced compaction shows it:

112) Compacting 
[SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily-hb-63-Data.db'), 
SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily-hb-61-Data.db')]
 INFO [CompactionExecutor:41] 2011-11-15 07:08:24,034 
CompactionController.java (line 129) Compacting large row 
system/HintsColumnFamily:0758480e325aa09951c555e1249a69ba (237411479 
bytes) incrementally
 INFO [CompactionExecutor:41] 2011-11-15 07:08:33,418 
CompactionTask.java (line 213) Compacted to 
[/var/lib/cassandra/data/system/HintsColumnFamily-hb-64-Data.db,].  
237,411,643 to 237,411,505 (~99% of original) bytes for 1 keys at 
24.091642MB/s.  Time: 9,398ms.


after node restart it still shows 0 rows delivered. hints should be 
newly written by 1.0.3. i deleted old rows from older versions in hints 
- only 2 tombstones left there before another batch of hints testing.


hints inside are not from supercolumn CF. I remember to have 2 rows in 
HintsColumnFamily, one was successfully delivered and this one seems to 
be stuck. These hints are older than 1 hour.


Did you tested outgoing hints delivery with more than row there? its 
probably 1 key = 1 hints delivery session and if node is back up for 
while then second hints session is created. Try to do your testing with 
about 500k hints with multiple sessions.

Re: hintedhandoff in 1.0.3

[Radim Kolar]

Same problem on other node:  2 keys in HintsColumnFamily. One delivered, 
one left.


 INFO [HintedHandoff:1] 2011-11-15 10:31:53,181 
HintedHandOffManager.java (line 268) Started hinted handoff for token: 
99070591730234615865843651857942052864
 INFO [HintedHandoff:1] 2011-11-15 10:32:49,385 ColumnFamilyStore.java 
(line 688) Enqueuing flush of 
Memtable-HintsColumnFamily@797897458(1674737/2093421 serialized/live 
bytes, 6176 ops)
 INFO [FlushWriter:5] 2011-11-15 10:32:49,386 Memtable.java (line 239) 
Writing Memtable-HintsColumnFamily@797897458(1674737/2093421 
serialized/live bytes, 6176 ops)
 INFO [CompactionExecutor:10] 2011-11-15 10:32:49,387 
CompactionTask.java (line 112) Compacting 
[SSTableReader(path='/usr/local/cassandra/data/system/HintsColumnFamily-hb-754-Data.db'), 
SSTableReader(path='/usr/local/cassandra/data/system/HintsColumnFamily-hb-752-Data.db')]
 INFO [FlushWriter:5] 2011-11-15 10:32:49,523 Memtable.java (line 275) 
Completed flushing 
/usr/local/cassandra/data/system/HintsColumnFamily-hb-755-Data.db 
(1888357 bytes)
 INFO [CompactionExecutor:10] 2011-11-15 10:32:49,820 
CompactionTask.java (line 213) Compacted to 
[/usr/local/cassandra/data/system/HintsColumnFamily-hb-756-Data.db,].  
19,913,818 to 19,913,392 (~99% of original) bytes for 2 keys at 
43.960395MB/s.  Time: 432ms.
 INFO [HintedHandoff:1] 2011-11-15 10:32:49,820 
HintedHandOffManager.java (line 334) Finished hinted handoff of 5796 rows

Re: How is Cassandra being used?

[Radim Kolar]

ppl hate EHCache and Quartz for doing this.

Re: How is Cassandra being used?

[Radim Kolar]

Dne 16.11.2011 23:58, Bill napsal(a):
We'll turn this off, and would possibly patch it out of the code. 
That's not to say it wouldn't be useful to others.
we patch out of code spyware in ehcache and quartz too. This is only way 
to be sure that it will not be enabled by configuration mistake.


We had similar stuff in our software (just version check) and had to 
remove it after 14 days of customer complains. As correctly stated in 
comments - you will get results from self selecting group and this group 
will not include most important deployments running cassandra in 
production.


There is zero benefit for company allowing cassandra to spy how many 
nodes, data, rows and CF they have and its bad idea to implement spyware 
into application used for working with sensitive data.

Re: hintedhandoff in 1.0.3

[Radim Kolar]

Dne 16.11.2011 18:17, Jonathan Ellis napsal(a):

Keys in HCF are nodes it has hints for.
Because it is 2 node cluster then it must write HH to himself and that 
explains why after second node gets back again. HH for it are delivered 
and cleaned but HH with second key are never delivered.

Re: 1.1 freeze approaching

[Radim Kolar]

Just a reminder that for us to meet our four-month major release
schedule (i.e., 1.1 = Feb 18),
can you make release cycle slower? its better to have more new features 
and do major upgrades less often. it saves time needed for testing and 
migrations.

major version release schedule

[Radim Kolar]

http://www.mail-archive.com/dev@cassandra.apache.org/msg01549.html

I read it but things are different now because magic 1.0 is out. If you 
implement 1.0 and put it into production, you really do not want to 
retest app on new version every 4 months and its unlikely that you will 
get migration approved by management unless you present clear benefits 
for such migration. Compression was nice new feature of 1.0 but it was 
rejected by lot of IT managers  as "too risky" for now.


While you can test application quite easily, testing cluster stability 
is way harder in test environment because its not usually possible to 
fully replicate workload and data volume in test environment and 
migration back is difficult because Cassandra currently does not have 
tool for fast sstable downgrade (1.0 -> 0.8).


For production use long time between major releases is better. I would 
double time between major releases, maybe not for 1.1/1.2 but later for 
sure.  Take look at postgresql project, they release 1 major version per 
year and they support 4 major versions for bugfixes and older postgresql 
versions are still common in production.


Did you asked people running mission critical workloads about their 
opinion? Another possibility is to use ISV like Datastax to provide long 
term support.

Re: major version release schedule

[Radim Kolar]

Nobody's forcing you to upgrade.  If you want twice as much time
between upgrading, just wait for 1.2.
Currently 1.0 branch is still less stable then 0.8, i still get OOM on 
some nodes. Adding 1.1 feature set on top will make it less stable.

It's also worth noting that waiting for 2x as many features for freeze
will result in MORE than 2x as much complexity for tracking down
regressions.
Then make releases from 2 branches - stable and dev. Its common practice 
used in lot of software projects.



 I really don't see why anyone would feel forced to upgrade just because a new 
version is available.

you will get less likely bugfixes if new branch comes out. Also client 
libraries needs some time to catch release.

Re: Cassandra has moved to Git

[Radim Kolar]

git://git.apache.org/cassandra.git

this still works?

Re: Cassandra has moved to Git

[Radim Kolar]

Dne 5.1.2012 7:22, Peter Schuller napsal(a):

(And btw, major +1 on the transition to git!)

please fix github mirror already.

Re: RFC: Cassandra Virtual Nodes

[Radim Kolar]

I don't like that every node will have same portion of data.

1. We are using nodes with different HW sizes (number of disks)
2.  especially with ordered partitioner there tends to be hotspots and 
you must assign smaller portion of data to nodes holding hotspots

Re: RFC: Cassandra Virtual Nodes

[Radim Kolar]

Hi Radim,

The number of virtual nodes for each host would be configurable by the
user, in much the same way that initial_token is configurable now. A host
taking a larger number of virtual nodes (tokens) would have proportionately
more of the data. This is how we anticipate support for heterogeneity in
cluster hardware.
Yes, but this is good only for random partitioner. For ordered you need 
to be able split token space on highly loaded servers. With virtual 
tokens it will move load to random node.
What if random node will be also hotspot node? Administration will be 
more difficult because you don't know where workload lands after you 
reduce number of tokens held by node.

maven 3 build system

[Radim Kolar]

In general, maintaining the pom is something that can fall off the 
C* devs
Maven is really easy tool once you get it going and gain necessary 
knowledge. It is really well integrated in Eclipse,  in Jenkins and 
there are plugins for nearly anything and writing your plugins is easy 
and you can embed Groovy if you are lazy to write own plugin. While 
migration to Maven 3 can take a while, it is worth it in long run.

Re: [VOTE] Release Mojo's Cassandra Maven Plugin 1.0.0-1

[Radim Kolar]

 I'd like to release version 1.1.0-1 of Mojo's Cassandra Maven Plugin


What is this plugin supposed to do?

Re: [VOTE] Release Apache Cassandra 1.0.10

[Radim Kolar]

Dne 4.5.2012 18:33, Sylvain Lebresne napsal(a):

CASSANDRA-4116 is kind of a big deal

its bugfix or improvement?

make default download cassandra 1.0

[Radim Kolar]

because cassandra 1.0 is not sufficiently stable, what about to make 
cassandra 1.0 default download and add bottom line - cassandra 1.0 is 
also available.

I seen this in other projects.

Re: make default download cassandra 1.0

[Radim Kolar]

message was wrong, It should be cass 1.1 vs 1.0. Cassandra 1.1 needs 
some time to stabilize. It took months to get cassandra 1.0 stable after 
it was released.


Reworked schema changes in cass 1.1 produces some really weird bugs like 
disappearing entire keyspace (data are still there). I think that new 
cassandra users should not be used as beta testers.


Other SW projects have similar problems for example dovecot 2.1 is 
offered as default download even it is not stable enough for production.

Re: PerRowSecondaryIndex

[Radim Kolar]

Dne 27.5.2012 12:54, Fábio Caldas napsal(a):

by the way ... Solr with Cassandra was a great idea .. I´m using it
and loving ...

how much data you have stored in solr

Re: Cassandra in memory key index

[Radim Kolar]

If you are interested I can help, I used the FST on a Hadoop project
to implement a fast map side range join.

create JIRA item with patch attached, i will test it.

Re: Cassandra in memory key index

[Radim Kolar]

Dne 8.6.2012 21:19, Jason Rutherglen napsal(a):

Ok looks like the IndexSummary encapsulates everything, I can start with
hacking that.

do memory part first. i want to test it on existing serialized index data.