Re: Send reads concurrently

[altanis]

I found the solution to this problem. I was sending the same message to
all nodes, with the same ID, and that was apparently causing problems
somehow. It is solved anyway.

Alexander

> Thank you for your answer, however I am pretty sure that's not it. I have
> a small-two node cluster for development testing, and I have loaded it
> with data in a way that responses to my queries usually have about 5
> short rows (which I think is not very much).
>
> First of all, if I do it like this, it works:
>
> for (AbstractBounds range : ranges)
> {
> 
>
> for (InetAddress endpoint : entry.getValue())
> {
> MessagingService.instance.sendRR(message, endpoint, handler);
> }
>
> List response = handler.get();
> }
>
> So it's not a matter or rpc_timeout. However, just in case, I increased
> rpc_timeout to 4 ms (which is longer than any of my queries take with
> the above method), and the problem still occurred.
>
> My problem is that I don't want to wait for all the results for one range
> before sending the request to another one.
>
> I also added some debugging to QuorumResponseHandler.get() and I found out
> where exactly the problem occurs. This (QRH, line 64 in rc1):
>
> success = condition.await(timeout, TimeUnit.MILLISECONDS);
>
> is never true with my way.
>
> The puzzling thing is that I noticed strongRead does exactly the same
> thing that I want to do, but it obviously works, and I can't see what I am
> doing differently. One thing to note, maybe, is that I even send a request
> to the same node that sends the requests, and I am wondering if there is
> some kind of deadlock between the different threads that handle the verb
> and the requests.
>
> Alexander
>
>> I would guess that sending a bunch of range requests simultaneously
>> overwhelms the targets (range scans are expensive), so you're timing
>> out simply because it couldn't finish all of them within rpc_timeout.
>>
>> Solution: don't do that, or increase rpc_timeout.
>>
>> On Wed, Jan 12, 2011 at 3:03 AM,   wrote:
>>> Hello,
>>>
>>> I am reading through getRangeSlice() in StorageProxy, and I am trying
>>> to
>>> do roughly the same thing for a join operation I am trying to implement
>>> in
>>> Cassandra.
>>>
>>> I see that getRangeSlice() loops through all available ranges, and for
>>> each range, it sends a request to the applicable nodes and then handles
>>> their answer *before* advancing to the next range. That may be fine for
>>> getRangeSlice(), but I'd like to send requests for all ranges at once
>>> and
>>> collect and handle the responses asynchronoysly (when they arrive).
>>>
>>> I tried the following (pseudocode mixed with code):
>>>
>>> 
>>>
>>> for (AbstractBounds range : ranges)
>>> {
>>>    
>>>
>>>    for (InetAddress endpoint : entry.getValue())
>>>    {
>>>        MessagingService.instance.sendRR(message, endpoint, handler);
>>>    }
>>>
>>>    handlers.add(handler);
>>> }
>>>
>>> for(QRH handler : handlers)
>>> {
>>>    List response = handler.get();
>>> }
>>>
>>> However my client gets a TimedOutException and I think Cassandra blocks
>>> during the resolve() in get().
>>>
>>> I am using 0.7.0rc1, however I don't think this code would have changed
>>> much.
>>>
>>> Any ideas?
>>>
>>> Alexander
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>>
>>
>
>

Re: Time for 1.0

[Eric Evans]

On Thu, 2011-01-13 at 19:20 -0800, Jonathan Ellis wrote:
> > -0
> >
> > I've said it elsewhere, but the only reason to fuss about a 1.0, is 
> > that it is loaded with special meaning.
> 
> Right: that's what we should be doing.  Up to and including the start
> of 0.6 you almost had to have a committer on staff to run Cassandra in
> production.  That's much less true at the end of 0.6 and the start of
> 0.7.  (As Paul says, I think we could have legitimately called 0.7,
> 1.0, but better late than never.)

As we've already seen in this thread, your criteria for a 1.0 isn't
universally accepted, and neither is your assessment that Cassandra
meets it.

Don't get me wrong, I'm not saying that your criteria is incorrect.  I'm
saying that "1.0" has a special yet different meaning for everyone.
Christening a 1.0 will send a message, and many will receive the wrong
one.

It's also worth pointing out that dev@ is a pretty narrow audience, I'm
guessing we'll see even greater variation elsewhere.

> > I'd rather drop the leading the 0 and continue to number releases
> > sequentially the way we have.  If our < 1 versioning is signaling a
> lack
> > of readiness, and if >= 1 is a necessary gate, then 8.0 should work
> > equally as well.  Better in fact, 8 times better!
> 
> This defeats the purpose of changing the numbering, since by calling
> it 8.0 you're saying "all those other major releases we did were just
> as > 1.0 production ready as this one," so you're signaling nothing at
> all.  Which I think was your somewhat cynical point. :)

It would be my intention to say, "the 0 never meant anything", and "this
is our 8th release", which I think is better than saying, "this is our
first release".

FWIW, the other suggestions (date-based versions like 2011.01, etc),
work equally as well for me. :)

-- 
Eric Evans
eev...@rackspace.com

Re: Time for 1.0

[Ryan King]

On Thu, Jan 13, 2011 at 7:32 PM, Jonathan Ellis  wrote:
>...
> In other words, at some point you have so many production users that
> it's silly to pretend it's ready for 1.0.  I'd say we've passed that
> point.

Did you mean to say "silly to pretend it's *not* ready for 1.0"?
Otherwise, I don't understand.

> I'm on board with this, to the point that Riptano is hiring a
> full-time QA engineer to contribute here.

Like I said at the outset, I don't care so much about what the version
is called as long as the quality continues to improve.

-ryan

Re: Time for 1.0

[Jonathan Ellis]

On Fri, Jan 14, 2011 at 11:24 AM, Ryan King  wrote:
> On Thu, Jan 13, 2011 at 7:32 PM, Jonathan Ellis  wrote:
>>...
>> In other words, at some point you have so many production users that
>> it's silly to pretend it's ready for 1.0.  I'd say we've passed that
>> point.
>
> Did you mean to say "silly to pretend it's *not* ready for 1.0"?
> Otherwise, I don't understand.

Yes, that is what I meant. :)

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: Time for 1.0

[Vijay]

+1...

Regards,




On Tue, Jan 11, 2011 at 5:35 PM, Jonathan Ellis  wrote:

> Way back in Nov 09, we did a users survey and asked what features
> people wanted to see.  Here was my summary of the responses:
>
> http://www.mail-archive.com/cassandra-user@incubator.apache.org/msg01446.html
>
> Looking at that, we've done essentially all of them.  I think we can
> make a strong case that our next release should be 1.0; it's
> production ready, it's reasonably feature-complete, it's documented,
> and we know what our upgrade path story is.
>
> The list--
>
> Load balancing: basics done;
> https://issues.apache.org/jira/browse/CASSANDRA-1427 is open to
> improve it
>
> Decommission: done
>
> Map/reduce support: done
>
> ColumnFamily / Keyspace definitions w/o restart: done
>
> Design documentation: started at
> http://wiki.apache.org/cassandra/ArchitectureInternals
>
> Insert multiple rows at once: done
>
> Remove_slice_range / remove_key_range: turned out to be a *lot* harder
> than it looks at first.  Postponed indefinitely.
>
> Secondary indexing: done
>
> Caching: done (with some enhancements possible such as
> https://issues.apache.org/jira/browse/CASSANDRA-1969 and
> https://issues.apache.org/jira/browse/CASSANDRA-1956)
>
> Bulk delete (truncate): done
>
> I would add,
>
> User documentation: done (http://www.riptano.com/docs)
>
> Large row support: done
>
> Improved replication strategies and more sophisticated ConsistencyLevels:
> done
>
> Efficient bootstrap/streaming: done
>
> Flow control: done
>
> Network-level compatibility between releases: scheduled
> (https://issues.apache.org/jira/browse/CASSANDRA-1015)
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Re: Time for 1.0

[SriSatish Ambati]

+1 On making unit tests & distributed tests robust
(with & without ec2)

Sri

On Thu, Jan 13, 2011 at 1:46 PM, Ryan King  wrote:

> I'm a -1 on naming the next release 1.0 because I don't think it has
> the quality that 1.0 implies, but to be honest I don't really care
> that much. The version numbers don't really effect those that of use
> that are running production clusters. Calling it 1.0 won't make it any
> more stable or faster.
>
> Also, before we say that "everything people want in 1.0 is done",
> perhaps we need to do that survey again. A lot of people have joined
> the community since 0.5 days and their needs should probably be
> considered in this situation. Also, those of use who've been around
> have new things we care about. Of course this will always be true and
> at some point we need to draw a line in the sand and put the 1.0 stamp
> on it– I just feel that that time has not come yet (but, like I said I
> don't really care that much because it won't affect me).
>
> Regardless of what we call the next major release there's at least 2
> things I'd like to see happen:
>
> 1. make the distributed test suite more reliable (its admittedly flaky
> on ec2) and flesh it out to include all distributed functionality. We
> shouldn't run a distributed system without distributed tests. We'll
> work on the flakiness, but we need people to write tests (and
> reviewers to require tests).
> 2. I think we should change how we plan releases. I'll send another
> email about this soon.
>
> -ryan
>
> On Tue, Jan 11, 2011 at 5:35 PM, Jonathan Ellis  wrote:
> > Way back in Nov 09, we did a users survey and asked what features
> > people wanted to see.  Here was my summary of the responses:
> >
> http://www.mail-archive.com/cassandra-user@incubator.apache.org/msg01446.html
> >
> > Looking at that, we've done essentially all of them.  I think we can
> > make a strong case that our next release should be 1.0; it's
> > production ready, it's reasonably feature-complete, it's documented,
> > and we know what our upgrade path story is.
> >
> > The list--
> >
> > Load balancing: basics done;
> > https://issues.apache.org/jira/browse/CASSANDRA-1427 is open to
> > improve it
> >
> > Decommission: done
> >
> > Map/reduce support: done
> >
> > ColumnFamily / Keyspace definitions w/o restart: done
> >
> > Design documentation: started at
> > http://wiki.apache.org/cassandra/ArchitectureInternals
> >
> > Insert multiple rows at once: done
> >
> > Remove_slice_range / remove_key_range: turned out to be a *lot* harder
> > than it looks at first.  Postponed indefinitely.
> >
> > Secondary indexing: done
> >
> > Caching: done (with some enhancements possible such as
> > https://issues.apache.org/jira/browse/CASSANDRA-1969 and
> > https://issues.apache.org/jira/browse/CASSANDRA-1956)
> >
> > Bulk delete (truncate): done
> >
> > I would add,
> >
> > User documentation: done (http://www.riptano.com/docs)
> >
> > Large row support: done
> >
> > Improved replication strategies and more sophisticated ConsistencyLevels:
> done
> >
> > Efficient bootstrap/streaming: done
> >
> > Flow control: done
> >
> > Network-level compatibility between releases: scheduled
> > (https://issues.apache.org/jira/browse/CASSANDRA-1015)
> >
> > --
> > Jonathan Ellis
> > Project Chair, Apache Cassandra
> > co-founder of Riptano, the source for professional Cassandra support
> > http://riptano.com
> >
>

Re: Time for 1.0

[Jeremy Hanna]

+1 on starting a more predictable release cycle for Cassandra and doing more 
multi-node testing.  I don't care at all about what version number it is.

On Jan 11, 2011, at 7:35 PM, Jonathan Ellis wrote:

> Way back in Nov 09, we did a users survey and asked what features
> people wanted to see.  Here was my summary of the responses:
> http://www.mail-archive.com/cassandra-user@incubator.apache.org/msg01446.html
> 
> Looking at that, we've done essentially all of them.  I think we can
> make a strong case that our next release should be 1.0; it's
> production ready, it's reasonably feature-complete, it's documented,
> and we know what our upgrade path story is.
> 
> The list--
> 
> Load balancing: basics done;
> https://issues.apache.org/jira/browse/CASSANDRA-1427 is open to
> improve it
> 
> Decommission: done
> 
> Map/reduce support: done
> 
> ColumnFamily / Keyspace definitions w/o restart: done
> 
> Design documentation: started at
> http://wiki.apache.org/cassandra/ArchitectureInternals
> 
> Insert multiple rows at once: done
> 
> Remove_slice_range / remove_key_range: turned out to be a *lot* harder
> than it looks at first.  Postponed indefinitely.
> 
> Secondary indexing: done
> 
> Caching: done (with some enhancements possible such as
> https://issues.apache.org/jira/browse/CASSANDRA-1969 and
> https://issues.apache.org/jira/browse/CASSANDRA-1956)
> 
> Bulk delete (truncate): done
> 
> I would add,
> 
> User documentation: done (http://www.riptano.com/docs)
> 
> Large row support: done
> 
> Improved replication strategies and more sophisticated ConsistencyLevels: done
> 
> Efficient bootstrap/streaming: done
> 
> Flow control: done
> 
> Network-level compatibility between releases: scheduled
> (https://issues.apache.org/jira/browse/CASSANDRA-1015)
> 
> -- 
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com

Re: Proposal: fixed release schedule

[Eric Evans]

On Thu, 2011-01-13 at 14:32 -0800, Ryan King wrote:
> # Fixed schedule
> 
> We should set a fixed schedule and stick to it. Anything features not
> ready at branch time won't make it and will be disabled in the stable
> branch.

In general, I agree, if we want a cadence, we're going to have to exert
some pressure to release at an agreed upon time.  But, as Jonathan says,
we need to be flexible.

Even on this project, there have been times when we've put a clock on a
stable release, and had problems with people rushing in poorly tested
changes.

> # Trunk-first
> 
> Everyone on chrome commits to trunk first. I think the important
> change we could make is to keep everyone closer to trunk. We spend a
> good deal of effort back-porting patches between major versions. I
> think we should make the major versions less different. This would
> mean letting them live for shorter amounts of time and possibly making
> them bugfix only. Currently we add new features in stable branches,
> but I think if we made the major release schedule more predictable
> people would be more comfortable with letting their new feature wait
> until the next major version.

I'll call your keep-close-to-trunk, and raise you a:

*  branch as soon as all planned features land in trunk
*  iteratively release betas w/ bug-fixes only
*  release candidates are actual candidates for the final (not releases)

The time between creating the branch and making the release should be as
short as practical, and there should be no backward incompatible changes
between the first beta, and final release.

> We should be more liberal about committing things to trunk early and
> iterating on them there (rather than iterating on them in patches). If
> the features are unstable we can either hide them behind configuration
> flags or remove them when we cut a stable branch.

As Jonathan said elsewhere in this thread, Git is finally within reach,
and would solve many of the current technical limitations with our
current process.  I'd favor that as a first-step.

> # Automate all tests
> 
> I think the only way that we can keep people close to trunk and stay
> stable is to build automated tests for *everything*. All code should
> be exercised by thorough unit tests and distributed black-box tests.
> Every regression should get a test.

Absolutely.

> Chrome has a 6 week cycle. I think ours would be more like 4 months
> for major releases.
> 
> Whatever we do, I think the schedule needs to be more predictable,
> which means that the contents of each release will be less predictable
> (since its "whatever's ready at the appointed time"). Like the Chrome
> presentation mentioned the idea isn't raw speed, but predictable
> release schedules.

Yeah.  Sounds good.

-- 
Eric Evans
eev...@rackspace.com

Cassandra-Maven-Plugin

[Stephen Connolly]

OK,

I nearly have the Cassandra-Maven-Plugin ready.

It has the following goals:
  run: launches Cassandra in the foreground and blocks until you press
^C at which point Maven terminates. Use-case: Running integration
tests from your IDE. Live development from your IDE.

  start: launches Cassandra in the background. Cassandra will be torn
down when Maven ends or if the stop goal is called. Use-case: Running
integration tests from Maven. Live development from your IDE with e.g.
jetty

  clean: Clears out the Cassandra database directory in
${basedir}/target/cassandra. Use-case: Resetting the dataset.

  load: Runs the cassandra-cli with a file as input.  Use-case:
Creating Keyspaces & pre-populating the dataset

  stop: Shuts down the background Cassandra instance started by start.
Use-case: Running integration tests from Maven.

So for example, if you are developing a web application using Maven
you would use a command like:

mvn cassandra:clean cassandra:start cassandra:load jetty:run

which would start up cassandra with a clean dataset and then start up
jetty (which presumably connects via a client library to cassandra).

Similarly, you can use cassandra-maven-plugin, jetty-maven-plugin,
maven-failsafe-plugin and selenium-maven-plugin to run web integration
tests as part of your build.

So I have some questions:

1. Is there a standard file extension for the scripts that get passed
to cassandra-cli?

2. Is there any other obvious goal I have missed out on?

There is a small bit of tidy-up left and then I just have to add some
integration tests and the site documentation.  Once I have all that in
place I will raise a JIRA with the full source code against CASSANDRA
and hopefully a friendly committer will pick it up and commit it into
the tree. While waiting for a committer testers will be welcome.

If it gets accepted I will then see about getting it released and
published on central.

Expect to see the JIRA sometime Monday or Tuesday.

-Stephen

Build failed in Hudson: Cassandra-0.7 #159

[Apache Hudson Server]

See 

Changes:

[jake] zero-copy reads
Patch by Pavel Yaskevich; Reviewed by Jake Luciani for CASSANDRA-1714

[jake] Fix regression caused by CASSANDRA-1530
patch by Kelvin Kakugawa; reviewed by Jake Luciani for CASSANDRA-1986

--
[...truncated 810 lines...]
AUbin/cassandra-cli.bat
AUbin/json2sstable
AUbin/sstable2json
A bin/config-converter
AUbin/cassandra.in.sh
A bin/json2sstable.bat
A bin/sstable2json.bat
AUREADME.txt
A .rat-excludes
 U.
At revision 1059227
[cassandra] $ /home/hudson/tools/ant/latest/bin/ant -Dversion=$BUILD_ID 
-Dcobertura.dir=/usr/share/java clean release test
Buildfile: 


clean:

ivy-download:
 [echo] Downloading Ivy...
[mkdir] Created dir: 

  [get] Getting: 
http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
  [get] To: 


ivy-init:
[mkdir] Created dir: 


ivy-retrieve-build:
No ivy:settings found for the default reference 'ivy.instance'.  A default 
instance will be used
[ivy:resolve] :: Ivy 2.0.0-rc2 - 20081028224207 :: http://ant.apache.org/ivy/ ::
:: loading settings :: file = 

[ivy:resolve] :: resolving dependencies :: 
apache-cassandra#cassandra;work...@vesta.apache.org
[ivy:resolve]   confs: [default]
[ivy:resolve]   found com.thoughtworks.paranamer#paranamer-ant;2.1 in ibiblio
[ivy:resolve]   found com.thoughtworks.paranamer#paranamer-generator;2.1 in 
ibiblio
[ivy:resolve]   found com.thoughtworks.qdox#qdox;1.10 in ibiblio
[ivy:resolve]   found asm#asm;3.2 in ibiblio
[ivy:resolve]   found ant#ant;1.6.5 in ibiblio
[ivy:resolve]   found junit#junit;4.6 in ibiblio
[ivy:resolve]   found commons-logging#commons-logging;1.1.1 in chain
[ivy:resolve]   found log4j#log4j;1.2.12 in ibiblio
[ivy:resolve]   found logkit#logkit;1.0.1 in ibiblio
[ivy:resolve]   found avalon-framework#avalon-framework;4.1.3 in ibiblio
[ivy:resolve]   found javax.servlet#servlet-api;2.3 in ibiblio
[ivy:resolve]   found org.apache.rat#apache-rat;0.6 in ibiblio
[ivy:resolve]   found org.apache.rat#apache-rat-core;0.6 in chain
[ivy:resolve]   found commons-collections#commons-collections;3.2 in chain
[ivy:resolve]   found commons-lang#commons-lang;2.1 in chain
[ivy:resolve]   found commons-cli#commons-cli;1.1 in chain
[ivy:resolve]   found org.apache.rat#apache-rat-tasks;0.6 in chain
[ivy:resolve]   found com.cloudera.hadoop#hadoop-core;0.20.2-320 in cloudera
[ivy:resolve]   found commons-cli#commons-cli;1.2 in ibiblio
[ivy:resolve]   found xmlenc#xmlenc;0.52 in ibiblio
[ivy:resolve]   found commons-httpclient#commons-httpclient;3.0.1 in ibiblio
[ivy:resolve]   found commons-codec#commons-codec;1.3 in chain
[ivy:resolve]   found commons-net#commons-net;1.4.1 in ibiblio
[ivy:resolve]   found oro#oro;2.0.8 in chain
[ivy:resolve]   found org.mortbay.jetty#jetty;6.1.14 in ibiblio
[ivy:resolve]   found org.mortbay.jetty#jetty-util;6.1.14 in ibiblio
[ivy:resolve]   found org.mortbay.jetty#servlet-api-2.5;6.1.14 in ibiblio
[ivy:resolve]   found tomcat#jasper-runtime;5.5.12 in ibiblio
[ivy:resolve]   found tomcat#jasper-compiler;5.5.12 in ibiblio
[ivy:resolve]   found org.mortbay.jetty#jsp-api-2.1;6.1.14 in ibiblio
[ivy:resolve]   found org.mortbay.jetty#jsp-2.1;6.1.14 in ibiblio
[ivy:resolve]   found org.eclipse.jdt#core;3.1.1 in ibiblio
[ivy:resolve]   found commons-el#commons-el;1.0 in ibiblio
[ivy:resolve]   found net.java.dev.jets3t#jets3t;0.7.1 in ibiblio
[ivy:resolve]   found net.sf.kosmosfs#kfs;0.3 in ibiblio
[ivy:resolve]   found hsqldb#hsqldb;1.8.0.10 in ibiblio
[ivy:resolve]   found com.cloudera.hadoop#hadoop-streaming;0.20.2-320 in 
cloudera
[ivy:resolve]   found net.sf.jopt-simple#jopt-simple;3.2 in ibiblio
[ivy:resolve]   found net.java.dev.jna#jna;3.2.7 in java.net2
[ivy:resolve] :: resolution report :: resolve 1478ms :: artifacts dl 163ms
[ivy:resolve]   :: evicted modules:
[ivy:resolve]   commons-cli#commons-cli;1.1 by [commons-cli#commons-cli;1.2] in 
[default]
[ivy:resolve]   junit#junit;3.8.1 by [junit#junit;4.6] in [default]
[ivy:resolve]   commons-logging#commons-logging;1.0.3 by 
[commons-logging#commons-logging;1.1.1] in [default]
[ivy:resolve]   commons-codec#commons-codec;1.2 by 
[commons-codec#commons-codec;1.3] in [default]
[ivy:resolve]   commons-httpclient#commons-httpclient;3.1 by 
[commons-httpclient#commons-httpclient;3.0.1] in [default]
-
|  |modules||   artifac

Hudson build is back to normal : Cassandra-0.7 #160

[Apache Hudson Server]

See