Client side issues with implementing new Collections support in 1.2
I am in the process of implementing collections support in the JDBC Driver and have run in to a few issues. Specifically: 1. There does not seem to be a notion of a reflexive o.a.c.cql.jdbc.* class for the new collection classes in o.a.c.db.marshall. Is that intentional or was that to be "left to be implemented student", which for me is now. I am trying to make sure I understand to broader design so I do not go off half cocked. 2. In order to do a proper job on the client side of translating the intent of the collection declarations that are built on the server side, the server must send enough data back in the thrift (or native) stream to reconstitute it properly on the client side. Unless I am missing something (very possible) I think the underlying base types of the data need to be in the encoded (ByteBuffer) value in addition to the literal values (encoded as a JSON string payload). JSON does not have a rich enough vocabulary to imply all of the CQL types without this additional information. I think it is as simple as appending this information adjacent to the JSON literal and letting the compose/decompose methods deal with the exact content issues. I am happy to cut a JIRA and provide a patch to handle the issues but want to air out the issues before solving something that messes with the bigger picture of the overall design intent that I may be missing.
Re: Client side issues with implementing new Collections support in 1.2
If the data type comes across in the "generic" part of the type name in the metadata, that should be enough for the client side to do its job. (i.e. ListType or MapType ) I'm happy to change the way to access the compose/decompose fucnctions and handle the JDBC specific stuff based on Type name and not have a Jdbc specific set of methods. Just commit in trunk and we will get going. On Mon, Aug 6, 2012 at 11:20 AM, Jonathan Ellis wrote: > On Mon, Aug 6, 2012 at 10:15 AM, Jonathan Ellis wrote: > > That said, it should still be possible to decode correctly by using > > the type information from CqlMetadata.value_types. > > Hmm, apparently we only send "set", not "set". Probably a bug, > but maybe it will be simpler to just switch to a more type-aware > encoding as Sylvain suggested. > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >
Fwd:
http://www.cossardavinci.com/ffryd3.php
Re: Reoganizing drivers
On Jul 7, 2011, at 10:53 AM, Eric Evans wrote: > On Wed, 2011-07-06 at 13:33 -0500, Jonathan Ellis wrote: >> - the git mirror won't pick up anything under drivers/ > > Has there been any effort made to have INFRA add it? > It was separated out to make it look/act like an independent project. Cant we > get an additional GIT project for drivers? > >> - building the Java drivers is fragile and complicated, and there's a >> lot of duplication with the "main" ant build > > Fragile how so? Because of the build-dependency on Cassandra (and/or > how it is satisfied)? Yes. > > What duplication are you referring to? I don't see much beyond all of > the boilerplate you'd see between any two ant projects. >Wow I would hardly call the current build for C* boilerplate. It is quite >comprehensive but very elaborate and complicate. A lot of clever expertise has >been applied. >Duplicating the necessary functionality and reaching into another existing >source tree is just complicated and error prone. Not impossible, but fragile >seems a good >description. > >> - patches that affect both Cassandra and JDBC are cumbersome since >> they have to be committed separately (e.g. >> https://issues.apache.org/jira/browse/CASSANDRA-2857) > > Well, the idea of moving it was to be able to treat it as a separate > project (more or less), so it follows that you'd have to independently > patch anything using AbstractCassandraDaemon. >This is a problem that cant be helped if they are separate projects. RIght? > > Well, it follows that if we change an API that any project using it will > need to be updated as well. Since the idea behind moving the drivers > was to be able to treat them as separate projects, it follows that we'd > have to do it here as well. > >> I'm inclined to think we should move it back to trunk (but not have >> multiple versions for 0.8 branch). We can still tag/branch separately >> from there. > > What do you mean by "but not have multiple versions for 0.8 branch"? >I like the idea of it being being built as a separate project. But I also like >it being in the main trunk. Is it possible to have both? >Can't we keep the /drivers code in the trunk and just have separate Ant tasks >for building the driver parts independent of the tasks for for the server? > Another thought would be to keep it in the separate tree as it is now, and > dumb down the driver build to just build the JAR artifacts (binary, source, > and javadoc). and do all testing and integration work in the server-side > trunk by using just the Driver jar as a dependency for the tests in the > server tree? > > -- > Eric Evans > eev...@rackspace.com >
Re: CqlResult in CassandraConnection
The class itself is not public, so it is generally protected from misuse, but it is a good recommendation to remove the public modifier on those non-interface imethods as well. I'll see to it. On Aug 28, 2011, at 11:35 AM, Eric Evans wrote: > On Sun, Aug 28, 2011 at 3:47 AM, Vivek Mishra wrote: >> Recently i can see changes made in jdbc connection API. >> >> Wondering why are we returning CqlResult from CassandraConnection, ideally >> it should return ResultSet as jdbc api. >> >> Any thoughts? > > The execute methods aren't a part of the java.sql.Connection > interface, but they are public, and so shouldn't be returning Thrift > structs. Maybe we just need to drop the public modifier. > > Can you open a ticket Vivek? > > -- > Eric Evans > Acunu | http://www.acunu.com | @acunu
Re: CqlResult in CassandraConnection
The reason they are set up that was is to clearly delineate between methods that there are no plans to implement any time in the near future; it minimizes the distraction in the classes that get a lot of activity. It is not necessary, but it was done that way when I started looking at the code and I recognized it as a clever approach and I followed its example. As to being confusing, I guess the abstract keyword was inadvertently omitted along the way and got propagated; but the intended effect is not hampered by the omission. I'll clean it up with the addition of the keyword to improve on its clarity. As to class level hierarchy, I am not sure I understand the reference? Could you elaborate? As to plans: Broad plans for 1.0 timeframe target are published in CASSANDRA-2876. The implementation of the prepared statement improvements (CASSANDRA-2885) is waiting on required support of CQL in various forms. I am also unclear as to what you are asking for in the final paragraph? Changes in each of those classes is inevitable to support new features but there are no plans (by me) for any other (alternate) classes. Perhaps you are implying they could be marked final? Note CassandraPreparedStatement is a sub-class of CassandraStatement. On Aug 28, 2011, at 1:33 PM, Vivek Mishra wrote: > I was looking at AbstractCassandraConnection,AbstractResultSet and > AbstractStatement classes. Name looks to me quite confusing as none of them > is an abstract class. 1 more thin, any specific reason for creating class > level hierarchy? > -2876 > Plans for any other implementation/s of CassandraConnection, ResultSet and > Statement sub class? > > Vovel > > > > ____ > From: Rick Shaw > To: dev@cassandra.apache.org > Cc: Vivek Mishra > Sent: Sunday, August 28, 2011 9:39 PM > Subject: Re: CqlResult in CassandraConnection > > The class itself is not public, so it is generally protected from misuse, but > it is a good recommendation to remove the public modifier on those > non-interface imethods as well. I'll see to it. > > On Aug 28, 2011, at 11:35 AM, Eric Evans wrote: > >> On Sun, Aug 28, 2011 at 3:47 AM, Vivek Mishra wrote: >>> Recently i can see changes made in jdbc connection API. >>> >>> Wondering why are we returning CqlResult from CassandraConnection, ideally >>> it should return ResultSet as jdbc api. >>> >>> Any thoughts? >> >> The execute methods aren't a part of the java.sql.Connection >> interface, but they are public, and so shouldn't be returning Thrift >> structs. Maybe we just need to drop the public modifier. >> >> Can you open a ticket Vivek? >> >> -- >> Eric Evans >> Acunu | http://www.acunu.com | @acunu
Re: CqlResult in CassandraConnection
No. I know of none. I will however, make sure the the abstract keyword is added to the classes you have mentioned. On Aug 29, 2011, at 2:17 AM, Vivek Mishra wrote: > Thanks Rick. > > > As to class level hierarchy, I am not sure I understand the reference? Could > you elaborate? > > As we have got 1 super class and sub class in place for ( > AbstractCassandraConnection,AbstractResultSet and AbstractStatement). So > thought to ask for if any other implementation is on roadmap? > > > Vivek > > > > ____ > From: Rick Shaw > To: dev@cassandra.apache.org; Vivek Mishra > Sent: Monday, August 29, 2011 12:59 AM > Subject: Re: CqlResult in CassandraConnection > > The reason they are set up that was is to clearly delineate between methods > that there are no plans to implement any time in the near future; it > minimizes the distraction in the classes that get a lot of activity. It is > not necessary, but it was done that way when I started looking at the code > and I recognized it as a clever approach and I followed its example. As to > being confusing, I guess the abstract keyword was inadvertently omitted along > the way and got propagated; but the intended effect is not hampered by the > omission. I'll clean it up with the addition of the keyword to improve on its > clarity. > > As to class level hierarchy, I am not sure I understand the reference? Could > you elaborate? > > As to plans: Broad plans for 1.0 timeframe target are published in > CASSANDRA-2876. The implementation of the prepared statement improvements > (CASSANDRA-2885) is waiting on required support of CQL in various forms. > > I am also unclear as to what you are asking for in the final paragraph? > Changes in each of those classes is inevitable to support new features but > there are no plans (by me) for any other (alternate) classes. Perhaps you > are implying they could be marked final? Note CassandraPreparedStatement is a > sub-class of CassandraStatement. > > On Aug 28, 2011, at 1:33 PM, Vivek Mishra wrote: > >> I was looking at AbstractCassandraConnection,AbstractResultSet and >> AbstractStatement classes. Name looks to me quite confusing as none of them >> is an abstract class. 1 more thin, any specific reason for creating class >> level hierarchy? >> -2876 >> Plans for any other implementation/s of CassandraConnection, ResultSet and >> Statement sub class? >> >> Vovel >> >> >> >> >> From: Rick Shaw >> To: dev@cassandra.apache.org >> Cc: Vivek Mishra >> Sent: Sunday, August 28, 2011 9:39 PM >> Subject: Re: CqlResult in CassandraConnection >> >> The class itself is not public, so it is generally protected from misuse, >> but it is a good recommendation to remove the public modifier on those >> non-interface imethods as well. I'll see to it. >> >> On Aug 28, 2011, at 11:35 AM, Eric Evans wrote: >> >>> On Sun, Aug 28, 2011 at 3:47 AM, Vivek Mishra >>> wrote: >>>> Recently i can see changes made in jdbc connection API. >>>> >>>> Wondering why are we returning CqlResult from CassandraConnection, ideally >>>> it should return ResultSet as jdbc api. >>>> >>>> Any thoughts? >>> >>> The execute methods aren't a part of the java.sql.Connection >>> interface, but they are public, and so shouldn't be returning Thrift >>> structs. Maybe we just need to drop the public modifier. >>> >>> Can you open a ticket Vivek? >>> >>> -- >>> Eric Evans >>> Acunu | http://www.acunu.com | @acunu
Re: CQL Drivers
For what it is worth, my preference would be to have unit tests that would form a regression testing package in the tree with the client sources. Ideally the build package (whether dedicated or mixed in with the server) would have specific tasks to build, test and install/deploy devoted to the individual client in question. I suggest that the client should not need to rely on anything but JAR files and local configuration to do the build and test. If they needed to be coordinated with a server build then perhaps some utility tasks to copy current jars for the server build site to the client build site would be useful. Config files local to the client build/test would point to test C* configs as appropriate. I think that makes me favor option #3. On Sep 2, 2011, at 4:52 PM, Eric Evans wrote: > On Thu, Sep 1, 2011 at 9:08 AM, Eric Evans wrote: >> On Wed, Aug 31, 2011 at 10:58 PM, Jonathan Ellis wrote: >>> On Wed, Aug 31, 2011 at 4:24 PM, Eric Evans wrote: CASSANDRA-2936 is in progress (patches attached), but is there any reason not to get started with the Python driver now? >>> >>> Heads up that test/system/test_cql.py depends on the Python driver. >>> It should probably be moved to the Python driver's test suite. (Which >>> then needs the same kind of "start a Cassandra server" ability that >>> the Cassandra system tests have.) >> >> I posed a similar question about the JDBC driver in CASSANDRA-2936. >> >> Should these tests be considered functional tests of Cassandra, and >> left be left where they are? I know that was my intention WRT >> test_cql.py (the driver itself has a few tests of its own that do not >> require a server). >> >> I guess I don't have a strong opinion either way, though it seems >> easier to say that Cassandra's tests will require you to have a driver >> installed, versus having to configure the build/test of a driver to >> point to a Cassandra tree. > > No opinions on this? To summarize, the options as I see them are: > > 1. Keep the tests that query Cassandra, as-is, with the drivers they use. > 2. Leave the tests that query Cassandra with Cassandra, (i.e. consider > them Cassandra tests). > 3. Keep the tests that query Cassandra, but alter them to be > idempotent and to connect to an existent node, instead of spinning > up/down an instance in test setup/teardown. > > (1) requires that you have some way of pointing the build at a local > copy of a Cassandra tree (as I remember it, there were some that > didn't care for this). (2) means that you'd need the corresponding > driver installed (or an embedded copy as the case may be) in order to > run those tests. (3) seems to be most consistent with how other > people do it for code that connects to some service. > > -- > Eric Evans > Acunu | http://www.acunu.com | @acunu
Re: Proposal: Moving CQL drivers
I worry that if the issue management is moved from the main JIRA for C* sponsored drivers there will be synergy and awareness lost between client and server. Did the Hector and Pelops folks see it as a good thing to have their own? As a small convenience to the Maven inclined among us, perhaps it would be useful and non threatening to maintain the directory structure for the JDBC driver in default Maven directory structure format? This would be exceedingly helpful for simple source control checkout of the code base for those that would like to add their own POM to build and test, and not effect a default ANT build in any way. Yes I know it is possible to remap all the various directory resources but if it start out using the Maven defaults both camps can easily interoperate without a lot of special casing as it should not really matter to an ANT build design. On Wed, Sep 7, 2011 at 10:49 AM, Stephen Connolly < stephen.alan.conno...@gmail.com> wrote: > On 7 September 2011 10:12, Eric Evans wrote: > > On Wed, Sep 7, 2011 at 8:22 AM, Stephen Connolly > > wrote: > >> On 6 September 2011 18:34, Vivek Mishra wrote: > >>> Sounds good moving to github. > >>> 1 quick question, what about JIRAs already raised w.r.t drivers? Not > sure but is it possible to integrate these new projects with current JIRA > flow? > >>> > >>> Planning to make these new projects based on maven build process?(As > that might be helpful in case of any quick release required for any sub > module). > >> > >> Ha! > >> > >> I would be genuinely surprised if that were to happen. > >> > >> I think there is a greater chance of seeing C* itself being built with > >> maven than the drivers... > > > > Really? I see the exact opposite, (and it's probably no secret how I > > feel about Maven). > > > > Hmmm, well here's my view, the only ones where Maven makes sense are > the JVM based drivers. Most of the JVM based ones can be simplified > down to the JDBC driver... and as Eric is the driver of the JDBC > driver and Eric's opinions on Maven are well known... > > >> P.S. > >> I will wait to be asked for my opinion on how this could be addresses > >> using Maven as a build tool. > > > > Is that, "Through the use of copious amounts of XML markup, storage, > > and network bandwidth?". :) > > > > Meh, the XML would be an order of magnitude less than the current ANT > build script. > > Storage... well it depends which storage you are talking about, it > would be a choice of storage in .svn or storage in ~/.m2 > > Network bandwidth... only if people follow poor -SNAPSHOT strategies > and are constantly deploying -SNAPSHOTs to the remote repo > > > Sorry, I couldn't resist. > > Neither could I ;-) > > > > >> The stated preference of the C* > >> developers is to use ANT. I am happy that Maven ANT Tasks is being > >> used over IVY, and happy that the artifacts are being pushed to > >> central, after that it doesn't matter what the build tool used is, as > >> long as the published poms are good (and last time I fine tuned them > >> they were) and as long as stuff gets into central, I am fine. > > > > > > > > -- > > Eric Evans > > Acunu | http://www.acunu.com | @acunu > > >
Re: How is Cassandra being used?
Speaking from the perspective of a large corporation with many and varied privacy and IP protection requirements it will be totally impossible to send ANY kind of data into the public network from a production Data Center. I expect most companies with similar concerns would want to be ASSURED that this could not happen by removing such code altogether. If you consider such code in the product you should make an easy way to configure it out completely not just disable it. I don't think you will get enough valuable information for the heartache it will cause your internal corporate champions. r. On Nov 15, 2011, at 9:01 PM, Jonathan Ellis wrote: > On Tue, Nov 15, 2011 at 7:02 PM, Eric Evans wrote: >> I think this is potentially quite dangerous; There are a lot people >> who get very twitchy at the idea of software that Phones Home. I've >> seen this so many times, and in all cases it was for software a lot >> less sensitive than a database. > > True, but unlike most Home Phoners, ours will be out there in the open > and you can see exactly what it's sending (or not, if you disable it). > I'm sure there's other examples in the wild of this, but the only one > I can think of is popcorn [1]. > > More broadly, my sense is that people are getting used to the idea > that it's okay to give away anonymous statistics as part of the price > of "free," although YMMclearlyV. I am, after all, a Windows user. :) > >> I'm sure you've already considered this though, you're already talking >> about anonymity, and transparency, and what I assume is neutrality of >> the collection endpoint (can apache actually provide a VM; is that a >> thing?). > > Yes, they provide Ubuntu or FreeBSD VMs. > >> I'm just afraid that we'll scare people off before they can >> be properly convinced that it's all on the up-and-up. > > How would you propose addressing this? > >> I'm curious to see what others think, but at the moment I'm hovering >> somewhere around a -0 if it were opt-in (off by default). > > I'm okay with opt-in if you think that's useful as a first step to > ease the twitchiness you mention, but longer term I think it's only > really useful if it's on by default. There's a lot of research that > shows that people tend to stick with whatever is the path of least > resistance [2], and specifically, my experience with Cassandra users > is exactly that -- one reason we've spent so much effort getting > defaults so good is because almost nobody goes beyond that. > > [1] http://popcon.debian.org/ > [2] > http://www.richmondfed.org/publications/research/region_focus/2007/winter/pdf/feature2.pdf > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com
Re: How is Cassandra being used?
Yes of course. But then an alarm would sound, emails to management would be sent, and the violator of the corporate policy would be hunted down and dealt with accordingly... The point is that the responsible parties are expected to NOT do such things in the first place. I expect many shops are quite a bit less concerned with such things but I expect the larger and more exposed companies will just not want to opt in, and may need to prove to their betters that it is not only not enabled but the code has been actively removed or disabled. I'm not trying to be a buzz-kill; I am adding a parochial perspective. r. On Nov 15, 2011, at 11:09 PM, Jonathan Ellis wrote: > On Tue, Nov 15, 2011 at 10:02 PM, Rick Shaw wrote: >> Speaking from the perspective of a large corporation with many and varied >> privacy and IP protection requirements it will be totally impossible to send >> ANY kind of data into the public network from a production Data Center. > > At the risk of getting off-topic, wouldn't such corporations prevent > this the easy way, with a firewall? > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com
Re: CQL support for compound columns
+1 on Gamma
+1 on haveing capability to specify a value.
My only reservation is the choice of the keword "TABLE", which is going to
be a source of continued confusion.
On Fri, Dec 30, 2011 at 11:58 AM, Jonathan Ellis wrote:
> I think we're closing in on something workable.
>
> Dropping TRANSPOSED from Gamma as redundant with respect to the
> composite PRIMARY KEY definition.
>
> Should we support column values in non-sparse rows by adding a
> VALUE(column_name) section?
>
> CREATE TABLE timeline (
>user_id int,
>posted_at uuid,
>body string,
>posted_by string,
>PRIMARY KEY(user_id, posted_at, posted_by),
>VALUE(body)
> );
>
> (Open to better suggestions for that keyword.)
>
>
> On Thu, Dec 29, 2011 at 3:13 PM, Eric Evans wrote:
> > On Thu, Dec 29, 2011 at 12:04 PM, Jonathan Ellis
> wrote:
> >> I've updated the wiki page at
> >> http://wiki.apache.org/cassandra/Cassandra2474 with a more in-depth
> >> Background section that hopefully clears up where I'm going with this
> >> sparse/dense business.
> >>
> >> Eric mentioned on IRC that he's uneasy about the PRIMARY KEY syntax
> >> implicitly using the first element of PRIMARY KEY as the row key. We
> >> could make it explicit with another WITH option to the TRANSPOSED
> >> clause:
> >>
> >> {{{
> >> CREATE TABLE timeline (
> >>user_id int,
> >>posted_at uuid,
> >>column string,
> >>value blob,
> >>PRIMARY KEY(user_id, posted_at)
> >> ) TRANSPOSED WITH ROW KEY(user_id)
> >> }}}
> >>
> >> This makes things more verbose (this would be a required clause) but
> >> I'm okay with that if consensus is that being explicit here is better.
> >
> > I think that was a reaction to an earlier iteration. Assuming that
> > the only place where order matters is in that primary key definition,
> > then I think it makes sense without the "... WITH ROW KEY..." bit.
> >
> >
> >
> > --
> > Eric Evans
> > Acunu | http://www.acunu.com | @acunu
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: [VOTE] Release Mojo's Cassandra Maven Plugin 1.0.0-1
+1 Great plugin. Integral part of Cassandra-JDBC testing. On May 2, 2012, at 7:15 AM, Stephen Connolly wrote: > Hi, > > I'd like to release version 1.1.0-1 of Mojo's Cassandra Maven Plugin > to sync up with the 1.1.0 release of Apache Cassandra. > > We solved 2 issues: > http://jira.codehaus.org/secure/ReleaseNote.jspa?projectId=12121&version=17926 > > Staging Repository: > https://nexus.codehaus.org/content/repositories/orgcodehausmojo-068/ > > Site: > http://mojo.codehaus.org/cassandra-maven-plugin/index.html > > SCM Tag: > https://svn.codehaus.org/mojo/tags/cassandra-maven-plugin-1.1.0-1@16519 > > [ ] +1 Yeah! fire ahead oh and the blind man on the galloping horse > says it looks fine too. > [ ] 0 Mehhh! like I care, I don't have any opinions either, I'd > follow somebody else if only I could decide who > [ ] -1 No! wait up there I have issues (in general like, ya know, > and being a trouble-maker is only one of them) > > The vote is open for 72h and will succeed by lazy consensus. > > Guide to testing staged releases: > http://maven.apache.org/guides/development/guide-testing-releases.html > > Cheers > > -Stephen > > P.S. > In the interest of ensuring (more is) better testing, this vote is > also open to subscribers of the dev and u...@cassandra.apache.org > mailing lists.
