Re: Cassandra CI Status

[Mick Semb Wever]

> If I don't hear any objection, I'll commit this. Off this, as it 
> aggregates test reports, it's now possible to start test posting emails 
> with the test report summary, as well as bringing in the dtest builds 
> into the pipeline. 


Based on the pipeline approach I've gotten notifications to slack and email 
working.

Does anyone object if I send these to #cassandra-builds (a brand new slack 
room) and to bui...@cassandra.apache.org ?

This is not meant as anything perfect or finished, just to get something out 
there, on which a pragmatic discussion can continue…


An example of the email is below.
-- 

Build complete: Cassandra-devbranch-pipeline #92 [UNSTABLE]

GENERAL INFO

BUILD UNSTABLE
Build URL: https://builds.apache.org/job/Cassandra-devbranch-pipeline/92/
Project: Cassandra-devbranch-pipeline
Date of build: Mon, 13 Jan 2020 13:11:16 +
Build duration: 1 hr 16 min and counting



JUNIT RESULTS

Name: (root) Failed: 14 test(s), Passed: 0 test(s), Skipped: 0 test(s), Total: 
14 test(s)Failed: …
Name: junit.framework Failed: 1 test(s), Passed: 0 test(s), Skipped: 42 
test(s), Total: 43 test(s)Failed: 
junit.framework.TestSuite.org.apache.cassandra.io.sstable.CQLSSTableWriterTest-cdc
Name: org.apache.cassandra.audit Failed: 0 test(s), Passed: 285 test(s), 
Skipped: 0 test(s), Total: 285 test(s)
Name: org.apache.cassandra.auth Failed: 0 test(s), Passed: 119 test(s), 
Skipped: 0 test(s), Total: 119 test(s)
Name: org.apache.cassandra.auth.jmx Failed: 0 test(s), Passed: 89 test(s), 
Skipped: 0 test(s), Total: 89 test(s)
Name: org.apache.cassandra.batchlog Failed: 0 test(s), Passed: 37 test(s), 
Skipped: 0 test(s), Total: 37 test(s)
Name: org.apache.cassandra.cache Failed: 0 test(s), Passed: 16 test(s), 
Skipped: 0 test(s), Total: 16 test(s)
Name: org.apache.cassandra.concurrent Failed: 0 test(s), Passed: 38 test(s), 
Skipped: 0 test(s), Total: 38 test(s)
Name: org.apache.cassandra.config Failed: 0 test(s), Passed: 40 test(s), 
Skipped: 0 test(s), Total: 40 test(s)
Name: org.apache.cassandra.cql.jdbc Failed: 0 test(s), Passed: 4 test(s), 
Skipped: 0 test(s), Total: 4 test(s)
Name: org.apache.cassandra.cql3 Failed: 0 test(s), Passed: 832 test(s), 
Skipped: 16 test(s), Total: 848 test(s)
Name: org.apache.cassandra.cql3.conditions Failed: 0 test(s), Passed: 16 
test(s), Skipped: 0 test(s), Total: 16 test(s)
Name: org.apache.cassandra.cql3.functions Failed: 0 test(s), Passed: 124 
test(s), Skipped: 0 test(s), Total: 124 test(s)
Name: org.apache.cassandra.cql3.restrictions Failed: 0 test(s), Passed: 92 
test(s), Skipped: 0 test(s), Total: 92 test(s)
Name: org.apache.cassandra.cql3.selection Failed: 0 test(s), Passed: 40 
test(s), Skipped: 0 test(s), Total: 40 test(s)
Name: org.apache.cassandra.cql3.statements Failed: 0 test(s), Passed: 48 
test(s), Skipped: 0 test(s), Total: 48 test(s)
Name: org.apache.cassandra.cql3.validation.entities Failed: 0 test(s), Passed: 
1416 test(s), Skipped: 4 test(s), Total: 1420 test(s)
Name: org.apache.cassandra.cql3.validation.miscellaneous Failed: 0 test(s), 
Passed: 224 test(s), Skipped: 0 test(s), Total: 224 test(s)
Name: org.apache.cassandra.cql3.validation.operations Failed: 0 test(s), 
Passed: 1564 test(s), Skipped: 0 test(s), Total: 1564 test(s)
Name: org.apache.cassandra.db Failed: 0 test(s), Passed: 1434 test(s), Skipped: 
2 test(s), Total: 1436 test(s)
Name: org.apache.cassandra.db.aggregation Failed: 0 test(s), Passed: 24 
test(s), Skipped: 0 test(s), Total: 24 test(s)
Name: org.apache.cassandra.db.columniterator Failed: 0 test(s), Passed: 4 
test(s), Skipped: 0 test(s), Total: 4 test(s)
Name: org.apache.cassandra.db.commitlog Failed: 0 test(s), Passed: 1452 
test(s), Skipped: 4 test(s), Total: 1456 test(s)
Name: org.apache.cassandra.db.compaction Failed: 0 test(s), Passed: 692 
test(s), Skipped: 4 test(s), Total: 696 test(s)
Name: org.apache.cassandra.db.composites Failed: 0 test(s), Passed: 12 test(s), 
Skipped: 0 test(s), Total: 12 test(s)
Name: org.apache.cassandra.db.context Failed: 0 test(s), Passed: 32 test(s), 
Skipped: 0 test(s), Total: 32 test(s)
Name: org.apache.cassandra.db.filter Failed: 0 test(s), Passed: 28 test(s), 
Skipped: 0 test(s), Total: 28 test(s)
Name: org.apache.cassandra.db.lifecycle Failed: 0 test(s), Passed: 268 test(s), 
Skipped: 0 test(s), Total: 268 test(s)
Name: org.apache.cassandra.db.marshal Failed: 0 test(s), Passed: 444 test(s), 
Skipped: 0 test(s), Total: 444 test(s)
Name: org.apache.cassandra.db.monitoring Failed: 0 test(s), Passed: 60 test(s), 
Skipped: 0 test(s), Total: 60 test(s)
Name: org.apache.cassandra.db.partition Failed: 0 test(s), Passed: 40 test(s), 
Skipped: 0 test(s), Total: 40 test(s)
Name: org.apache.cassandra.db.partitions Failed: 0 test(s), Passed: 32 test(s), 
Skipped: 0 test(s), Total: 32 test(s)
Name: org.apache.cassandra.db.repair Failed: 1 test(s), Passed: 87 test(s), 
Skipped: 0 test(s), Total: 88 test(s)Failed: 
org.apache.cassandra.db.repair.PendingAntiCompactionBytemanTest.testException

Amazon MCS contribution plan

[Gouws, Almero]

Hi Cassandra devs,

My name is Almero Gouws, I am the head of engineering for Amazon Managed Apache 
Cassandra Service (MCS). My team and I are eager to begin contributing to 
Apache Cassandra, and I want to share some of the contribution ideas we have 
planned. The goal of this thread is to get your input and guidance on the best 
way to accomplish these efforts.

SigV4 plugins
We have developed Java client and server authentication plugins which make use 
of Amazon Signature Version 4 [1]. We use these plugins in production to 
leverage short term session-based credentials, as well as AWS Identity and 
Access Management Roles [2], when authenticating to MCS. We are looking to 
open-source both the client and server plugins, so that self-managed Cassandra 
clusters could benefit as well. Is this something the community would 
interested in? If so, should we aim to add the plugins to the main project, or 
host them in a separate repo? We'll make the code available for review as soon 
as we can.

Infrastructure
I have noticed that there is a need for infrastructure to run automated tests 
for Cassandra. We'd like to help so that the tests can be run regularly with a 
reasonable running time. We believe that 15 m4.2xlarge [3] EC2 instances could 
help accomplish this, and we'd be happy to make this capacity available. Can 
you help us validate that this is the appropriate capacity, and guide us into 
how it should be configured?

CodeGuru
At re:Invent 2019 we announced Amazon CodeGuru [4] in preview. CodeGuru is a 
machine learning service for automated code reviews and application performance 
recommendations. We are looking to enable CodeGuru support on the Apache 
Cassandra GitHub repo to assist with reviewing pull requests. CodeGuru is not 
meant to be a replacement for developer reviews, instead it gives specific 
recommendations to fix or improve code. If added to the repository, CodeGuru 
would still be an optional tool and it will not require reviewers to change 
their existing workflow. Is this something the community would be interested in?

I'm looking forward to hearing from you and working with the community to 
benefit all users of Apache Cassandra. I'd also like to introduce Derek 
Chen-Becker (dchen...@amazon.com), a senior 
engineer on Amazon MCS. Along with myself, Derek will be available on the user 
and dev mailing lists and we will regularly be in the slack channels as well.

-Almero

[1] https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html
[2] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html
[3] https://aws.amazon.com/ec2/instance-types/m5/
[4] https://aws.amazon.com/codeguru/

Re: Amazon MCS contribution plan

[Dinesh Joshi]

Hi Almero,

Great to see your interest in contributing to the project. Currently, the 
Apache Cassandra community is in a code freeze[0] to ensure that we stabilize 
Cassandra 4.0 so the focus is going to be on testing. Here's a blog post[1] on 
how we're validating Cassandra. Anything that your team could help out test 
Cassandra 4.0 and stabilize it would be helpful to the community.

Thanks,

Dinesh

[0] http://cassandra.apache.org/doc/latest/development/patches.html#code-freeze
[1] https://cassandra.apache.org/blog/2018/08/21/testing_apache_cassandra.html

> On Jan 13, 2020, at 3:32 PM, Gouws, Almero  wrote:
> 
> Hi Cassandra devs,
> 
> My name is Almero Gouws, I am the head of engineering for Amazon Managed 
> Apache Cassandra Service (MCS). My team and I are eager to begin contributing 
> to Apache Cassandra, and I want to share some of the contribution ideas we 
> have planned. The goal of this thread is to get your input and guidance on 
> the best way to accomplish these efforts.
> 
> SigV4 plugins
> We have developed Java client and server authentication plugins which make 
> use of Amazon Signature Version 4 [1]. We use these plugins in production to 
> leverage short term session-based credentials, as well as AWS Identity and 
> Access Management Roles [2], when authenticating to MCS. We are looking to 
> open-source both the client and server plugins, so that self-managed 
> Cassandra clusters could benefit as well. Is this something the community 
> would interested in? If so, should we aim to add the plugins to the main 
> project, or host them in a separate repo? We'll make the code available for 
> review as soon as we can.
> 
> Infrastructure
> I have noticed that there is a need for infrastructure to run automated tests 
> for Cassandra. We'd like to help so that the tests can be run regularly with 
> a reasonable running time. We believe that 15 m4.2xlarge [3] EC2 instances 
> could help accomplish this, and we'd be happy to make this capacity 
> available. Can you help us validate that this is the appropriate capacity, 
> and guide us into how it should be configured?
> 
> CodeGuru
> At re:Invent 2019 we announced Amazon CodeGuru [4] in preview. CodeGuru is a 
> machine learning service for automated code reviews and application 
> performance recommendations. We are looking to enable CodeGuru support on the 
> Apache Cassandra GitHub repo to assist with reviewing pull requests. CodeGuru 
> is not meant to be a replacement for developer reviews, instead it gives 
> specific recommendations to fix or improve code. If added to the repository, 
> CodeGuru would still be an optional tool and it will not require reviewers to 
> change their existing workflow. Is this something the community would be 
> interested in?
> 
> I'm looking forward to hearing from you and working with the community to 
> benefit all users of Apache Cassandra. I'd also like to introduce Derek 
> Chen-Becker (dchen...@amazon.com), a senior 
> engineer on Amazon MCS. Along with myself, Derek will be available on the 
> user and dev mailing lists and we will regularly be in the slack channels as 
> well.
> 
> -Almero
> 
> [1] https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html
> [2] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html
> [3] https://aws.amazon.com/ec2/instance-types/m5/
> [4] https://aws.amazon.com/codeguru/
> 
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Apache Cassandra Contributor Meeting

[Patrick McFadin]

Hi everyone,

In order to catch up on what's happening here, here's the establishing
thread:
https://lists.apache.org/thread.html/aa54420a43671c00392978f2b0920bc6926ca9ba1e61a486ad39fb21%40%3Cdev.cassandra.apache.org%3E

Key points that Scott Andreas proposed in the initial email was

Motivation for such a meeting
1. We currently have Slack, JIRA and emails however an agenda driven video
meeting can help facilitate alignment within the community.
2. This will give an opportunity to the community to summarize past
progress and talk about future tasks.
3. Agenda notes can serve as newsletters for the community.

To that, I humbly offer my services as a community organizer to help with
the logistics and setup. I'm happy to say this is finally happening and I
apologize this has taken so long. I saw some of the examples mentioned in
the original thread for other open source projects and I "borrowed" heavily
from them.

I created a page in the Cassandra Confluence page to hopefully centralize
both logistics and records of each call. You can fine it here:
https://cwiki.apache.org/confluence/display/CASSANDRA/Apache+Cassandra+Contributor+Meeting

The meetings are on Zoom and set to be wide open. Anyone can join via
computer or phone. I'm using a tier that allows for 100 participants. If we
need more, I can change the type of meeting but it's more of a pain for
logistics. We can try this and see how it goes. Once the meeting starts
I'll hit record, I'll post the video on YouTube and add the link to the
notes. All meeting notes for each agenda items can live in the doc above
and remain as a permanent record. After the meeting, I'll send the notes
link to the dev list as a reminder that it happened to anyone subscribed.

If you have agenda items, please edit the Confluence page and add your name
and what you would like discussed.

My contribution here is as an organizer. Please feel free to email or Slack
if you need anything. Most important, a video meet is an alpha product and
we'll learn a lot from the first time trying. I'll try to keep note of
things to improve in the doc.

See you there,

Patrick

Re: Apache Cassandra Contributor Meeting

[Patrick McFadin]

And I sent this without saying when. Let me save you a click on the
confluence link.

January 21, 1PM PST

On Mon, Jan 13, 2020 at 5:28 PM Patrick McFadin  wrote:

> Hi everyone,
>
> In order to catch up on what's happening here, here's the establishing
> thread:
> https://lists.apache.org/thread.html/aa54420a43671c00392978f2b0920bc6926ca9ba1e61a486ad39fb21%40%3Cdev.cassandra.apache.org%3E
>
> Key points that Scott Andreas proposed in the initial email was
>
> Motivation for such a meeting
> 1. We currently have Slack, JIRA and emails however an agenda driven video
> meeting can help facilitate alignment within the community.
> 2. This will give an opportunity to the community to summarize past
> progress and talk about future tasks.
> 3. Agenda notes can serve as newsletters for the community.
>
> To that, I humbly offer my services as a community organizer to help with
> the logistics and setup. I'm happy to say this is finally happening and I
> apologize this has taken so long. I saw some of the examples mentioned in
> the original thread for other open source projects and I "borrowed" heavily
> from them.
>
> I created a page in the Cassandra Confluence page to hopefully centralize
> both logistics and records of each call. You can fine it here:
> https://cwiki.apache.org/confluence/display/CASSANDRA/Apache+Cassandra+Contributor+Meeting
>
> The meetings are on Zoom and set to be wide open. Anyone can join via
> computer or phone. I'm using a tier that allows for 100 participants. If we
> need more, I can change the type of meeting but it's more of a pain for
> logistics. We can try this and see how it goes. Once the meeting starts
> I'll hit record, I'll post the video on YouTube and add the link to the
> notes. All meeting notes for each agenda items can live in the doc above
> and remain as a permanent record. After the meeting, I'll send the notes
> link to the dev list as a reminder that it happened to anyone subscribed.
>
> If you have agenda items, please edit the Confluence page and add your
> name and what you would like discussed.
>
> My contribution here is as an organizer. Please feel free to email or
> Slack if you need anything. Most important, a video meet is an alpha
> product and we'll learn a lot from the first time trying. I'll try to keep
> note of things to improve in the doc.
>
> See you there,
>
> Patrick
>

Re: Amazon MCS contribution plan

[Mick Semb Wever]

> Infrastructure
> I have noticed that there is a need for infrastructure to run automated 
> tests for Cassandra. We'd like to help so that the tests can be run 
> regularly with a reasonable running time. We believe that 15 m4.2xlarge 
> [3] EC2 instances could help accomplish this, and we'd be happy to make 
> this capacity available. Can you help us validate that this is the 
> appropriate capacity, and guide us into how it should be configured?


This would be an immense contribution. Thank you Almero (and everyone at AWS).

I think others that have had internal infrastructure for running the dtests 
will have much better input about what the ideal instance type is, but it is my 
understanding that servers with 32gb ram meets the minimum specs for our 
largest dtests.  

Michael? Aleksey? Sam? Alex? Blake?

The ticket https://jira.apache.org/jira/browse/INFRA-14153 describes the 
process the last time infrastructure was added.

The ASF-slaves.txt doc also helps explain a bit.
https://github.com/apache/cassandra-builds/blob/master/ASF-slaves.txt

Though cassandra1-7 are currently MIA, and some of the info there is out of 
date.

I'd be more than happy to take charge in coordinating to get these servers in, 
if it all gets agreed upon.

regards,
Mick

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org