Very good, here is the latest revision with updated Mentors.
Sunil and I have been added to the IPMC as well.
Welcome Madhawa and thanks for stepping up as a Mentor for Livy!
Abstract
Livy is a web service that exposes a REST interface for managing long
running Apache Spark contexts in your cluster. With Livy, new applications
can be built on top of Apache Spark that require fine grained interaction
with many Spark contexts [1].
While this project has been well regarded and used in many contexts as the
defacto standard API to Spark environments, it has been incubating for over
5 years without graduation to a TLP and it has become difficult to
impossible for fixes and improvements to be contributed as the current
community seems to have moved on.
There has been discussion regarding retirement of this podling where there
seems to be some increasing interest in joining and reviving the community
[2].
The intent of this proposal is to avoid retiring a well regarded, actively
used and rather mature project by reviving the PPMC and community with new
folks that have a vested interest in the project and health of the
community.
Proposal
We propose to revive the PPMC with a set of contributors and maintainers as
mentors, PPMC members and committers.
The retirement DISCUSS thread [2] has shown a growing interest in providing
new committers and bringing improvements and fixes from organization’s
internally maintained forks back to a revived community.
General Approach to Revival:
-
Add new Mentors
-
Larry McCay, lmc...@apache.org , Cloudera
-
Sunil Govindan, sun...@apache.org, Cloudera
-
Jean-Baptiste Onofré, jbono...@apache.org, Talend
-
Madhawa Gunasekara, madhaw...@gmail.com, Independent
-
Add new Committers/PPMC
-
Larry McCay, lmc...@apache.org, Cloudera
-
Vinod Kumar Vavilapalli, vino...@cloudera.com, Cloudera
-
Imran Rashid - iras...@apache.org, Cloudera
-
Gyorgy Gal, ggal ,gal.gyo...@gmail.com, Cloudera
-
Wing Yew Poon, wyp...@cloudera.com, Cloudera
-
Xilang Yan, xilang....@gmail.com, Shopee
-
Jianzhen Wu, myjianz...@gmail.com, Shopee
-
Nagella Jagadeewara Rao, jnage...@visa.com, Visa
-
Pralab Kumar, pralk...@visa.com, Visa
-
Prasad Shrikant, shrikant....@gmail.com, Visa
-
Brahma Reddy Battula, bra...@apache.org, Visa
-
Invite existing PPMC members to opt-in or otherwise go emeritus
-
Jean-Baptiste Onofré, jbono...@apache.org, Talend (opted-in via
Retirement DISCUSS thread [2])
-
Invite existing Committers to opt-out or otherwise continue
-
Establish Roadmap via follow up DISCUSS thread
-
Known Improvements from Forks which will need proposals and
discussion:
-
Adding HA for Livy
-
Updating security capabilities (eg. kerberos for jdbc, fixing bugs
in encryption)
-
Expanding the support for kubernetes
-
Responding to CVEs in dependencies (eg. log4j, thrift)
-
Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
-
Support multi Spark versions
-
Implemented a metrics system for Livy
-
Support customize batch/interactive session lifecycle event
handler, default log event with log4j, very helpful for troubleshooting
-
Optimize log to track which session id the log message came from,
also very helpful for troubleshooting
-
Support customize Spark config optimization rules, can be used to
optimize config for users’ job
-
A set of command line tool which can be used to replace Spark’s
spark-submit, pyspark, spark-sql but actually submit
application in Livy
-
We are planning to implement a JDBC state store, and allow multi
Livy Thrift sessions to share one backend Spark application
in the next few
months.
-
These items and others that are brought to community may need
consolidation or multiple configurable options and will need to
be part of
the discussion
-
One-pager Livy Improvement Proposals (LIP) may make sense to drive
these discussions and convergence
-
Feature Branch Strategy for large changes
-
Large features are hard to review we will need to define a
strategy
-
Determine the Improvements to be delivered across first 3 Releases
with Target Release Dates
-
Ensure CVE and Dependency management hygiene is in place
The above approach will usher the community back to an active status with a
Roadmap of 3 or more release plans and security hygiene in place.
Development Practices
The Livy project follows a review before commit philosophy. Every commit
automatically runs through the unit tests and generates coverage reports
presented as a pull request comment. Our experience with this process leads
us to believe that it helps ease new contributors into the project. They
get feedback quickly on common mistakes, lowering the burden on reviewers.
Those same reviewers get to lead by example, showing the new contributors
that we value feedback within our community even when changes are done by
more experienced folks. Taken from the original Apache Livy Proposal [1],
this should continue to be true. As mentioned, Livy is a mature project and
as such RTC is the most appropriate for continued quality and awareness.
1.
Original Apache Livy Proposal
https://cwiki.apache.org/confluence/display/incubator/LivyProposal
2.
Retirement DISCUSS thread
https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <madhaw...@gmail.com>
wrote:
> Hi Larry,
>
> I'm an IPMC Member. madhawa30 at gmail dot com is my preferred email
> address.
> apache id: madhawa
>
> Thanks,
> Madhawa
>
>
> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <lmc...@apache.org> wrote:
>
> > Hi Madhawa -
> >
> > That's awesome!
> > Are you already a member of IPMC?
> > If not, are you an ASF member?
> > If you are an ASF member you can request that you be added as an IPMC
> > member.
> >
> > Can you provide your company affiliation for the proposal and preferred
> > email?
> >
> > thanks!
> >
> > --larry
> >
> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <madhaw...@gmail.com>
> > wrote:
> >
> > > Hi Larry,
> > >
> > > I'm interested in working with Livy and would like to join as a Mentor.
> > >
> > > Thanks,
> > > Madhawa
> > >
> > >
> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <lmc...@apache.org> wrote:
> > >
> > > > Sorry, I missed commenting on this:
> > > >
> > > > "There is also no concept as an emeritus PPMC member at the ASF."
> > > >
> > > > I assume that we can remove PPMC members that do not opt-in
> explicitly
> > at
> > > > this point.
> > > > They will have every opportunity to rejoin.
> > > >
> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <lmc...@apache.org>
> > wrote:
> > > >
> > > > > I will ask in a separate thread, @Justin Mclean <
> justinmcl...@me.com
> > >
> > > -
> > > > > thanks.
> > > > > Adding JB adds another company and we are certainly open to anyone
> > else
> > > > > that would like to join as a mentor.
> > > > > At the end of the day, the mentors are for instilling the Apache
> Way
> > > > > knowledge and steering toward graduation.
> > > > > I feel that this diversity, while nice to have, is less important
> > than
> > > > > that of the PPMC and committers for the long term health of the
> > > > community.
> > > > >
> > > > > We need to push this podling to graduation as quickly as possible
> > since
> > > > it
> > > > > is rather mature and needs to get to the next level.
> > > > >
> > > > > Again, any potential Mentors that would like to join are more than
> > > > welcome.
> > > > >
> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
> > > jus...@classsoftware.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> I’m sorry, but Imran Rashid can’t be a mentor for the project as
> > they
> > > > are
> > > > >> not an IPMC member. Currently, both Sunil and Larry (as they are
> ASF
> > > > >> members) need to ask to join the IPMC and NOTICE sent to the ASF
> > > board.
> > > > I
> > > > >> would also prefer that mentors come from different companies.
> > > > >>
> > > > >> There is also no concept as an emeritus PPMC member at the ASF.
> > > > >>
> > > > >> Kind Regards,
> > > > >> Justin
> > > > >>
> > > > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > > > >> For additional commands, e-mail:
> general-h...@incubator.apache.org
> > > > >>
> > > > >>
> > > >
> > >
> >
>
