I would characterize my vote as 0. I really don’t care either way. Just sharing I think they have no value in a release.
> On Jan 16, 2020, at 6:08 PM, Owen Nichols <onich...@pivotal.io> wrote: > > Geode PMC has 52 members. If this were a vote, it looks like the results > would have been: > +1: 2 (Anthony, Dan) > -1: 1 (Jake) > > If the next release manager were to go ahead and put geode-benchmarks in the > Geode 1.12.0 source release, at least 3 PMC members would need to be willing > to vote +1. So it sounds like we need a few more of the other 49 PMC members > to weigh in on this discussion. > > To summarize so far: > > Proposal: > - add a geode-benchmarks-n.n.n-src.tgz artifact to all Geode releases going > forward, starting with 1.12.0 > > Arguments in favor: > - why not? > - it’s already public > - we should default to including all things > - it might be of interest to the user community > - it might encourage contributions back to further improve it > - it is required by CI, which is already included > - Apache mandates that source releases must include test code too > > Arguments against: > - doing nothing is less work > - it will burden PMC members with additional work to validate and vote on RCs > - nobody outside the dev community has asked for it to be included > - maybe it’s not ready > - maybe it’s not documented well enough > - it’s not needed to use Geode > - Apache's legal separation between dev stuff and public release stuff > - legal or license review may be not have been conducted yet > > >>> On Jan 16, 2020, at 4:48 PM, Dan Smith <dsm...@pivotal.io> wrote: >>> >>> If geode-benchmarks is included, that implies that an RC cannot be >> approved until reviewers can successfully run the benchmark suite from the >> geode-benchmarks source distribution. Is that what we want? >> >> I think it would be sufficient to run the tests of the benchmarks, eg >> ./gradlew test >> >>> Deploying CI pipelines and running Benchmarks seems like a prime example >> of things we’d be happy to help others in the community with on the dev >> list — but not something we would expect questions about on the user list. >> >> I think it would be valuable to share our benchmarks with the geode user >> community. The benchmark framework itself (the harness module) is a fairly >> generic benchmarking framework than can be used to benchmark anything that >> can be spun up using java. The geode-benchmark module has geode benchmarks >> that could be used for testing specific hardware, for example. >> >> -Dan >> >>> On Thu, Jan 16, 2020 at 12:37 PM Owen Nichols <onich...@pivotal.io> wrote: >>> >>> When voting on RC candidates, PMC members "are required to download the >>> signed source code package, compile it as provided, and test the resulting >>> executable on their own platform”. >>> >>> If geode-benchmarks is included, that implies that an RC cannot be >>> approved until reviewers can successfully run the benchmark suite from the >>> geode-benchmarks source distribution. Is that what we want? >>> >>> Similarly, if CI is included, that seems to imply that an RC cannot be >>> approved until reviewers can stand up their own pipeline from the geode/ci >>> source distribution. Is that what we want? >>> >>> So far there doesn’t seem to be consensus on what to include in a Geode >>> source release, but let’s keep in mind that anything we add to the release >>> becomes an Act Of The Foundation and is held to a higher standard. Apache >>> makes a clear distinction between between development activity and official >>> releases to the public. Development activity is anything that should stay >>> within the dev list. Deploying CI pipelines and running Benchmarks seems >>> like a prime example of things we’d be happy to help others in the >>> community with on the dev list — but not something we would expect >>> questions about on the user list. >>> >>>> On Jan 16, 2020, at 10:23 AM, Dan Smith <dsm...@pivotal.io> wrote: >>>> >>>> We are supposed to be including all of the source necessary to test Geode >>>> in the source release [1] - I think that would include benchmarks as >>> well. >>>> >>>> I don't really see any compelling reason *not* to include the benchmarks, >>>> let's go ahead and get them into our source release! >>>> >>>> [1] >>>> >>> http://www.apache.org/legal/release-policy.html#what-must-every-release-contain >>>> >>>> On Wed, Jan 15, 2020 at 10:26 PM Owen Nichols <onich...@pivotal.io> >>> wrote: >>>> >>>>> +1 for no changes >>>>> >>>>> On Wed, Jan 15, 2020 at 5:57 PM Jacob Barrett <jbarr...@pivotal.io> >>> wrote: >>>>> >>>>>> We can live in areas of gray that don’t require any changes. Nobody is >>>>>> asking for benchmarks so let’s not do work to add them. Nobody is >>>>>> complaining they CI is included so let’s not do work to remove them. Is >>>>> it >>>>>> ideal, meh... >>>>>> >>>>>>> On Jan 15, 2020, at 5:50 PM, Mark Hanson <mhan...@pivotal.io> wrote: >>>>>>> >>>>>>> Just my two cents. >>>>>>> >>>>>>> I think that we should probably strip CI into a separate repo. The key >>>>>> indicator is that if something were wrong in the CI yaml, would I hold >>> a >>>>>> release for that? I think no. So that suggests to me it is a separate >>>>>> thing. Same goes for benchmarks. If we were failing a benchmark I would >>>>> be >>>>>> concerned, but if the script were broken, would I hold the release? I >>>>> think >>>>>> no as well. >>>>>>> >>>>>>> I think that says that the CI code should also be a separate repo. >>>>>>> >>>>>>> Thanks, >>>>>>> Mark >>>>>>> >>>>>>>> On Jan 14, 2020, at 10:21 PM, Jacob Barrett <jbarr...@pivotal.io> >>>>>> wrote: >>>>>>>> >>>>>>>> Until someone outside of the geode ci community is asking for it I >>>>> just >>>>>> don’t see utility in going through the motions of making a release for >>>>> it. >>>>>>>> >>>>>>>>>> On Jan 14, 2020, at 10:13 PM, Owen Nichols <onich...@pivotal.io> >>>>>> wrote: >>>>>>>>> >>>>>>>>> The source is already public, so on some level a source release is >>>>> no >>>>>> different from a git tag. Benchmarks has matured enough that I think >>> it >>>>>> makes sense to at least start branching and tagging the >>> geode-benchmarks >>>>>> repo to capture exactly what was used in that Geode release. >>>>>>>>> >>>>>>>>> Others in the dev and user community may find the benchmarks useful >>>>> in >>>>>> other ways than we use them. While our focus for CI is on tuning for >>>>>> repeatability, someone else might just want a load generator to break >>> in >>>>> a >>>>>> new cluster or get some rough numbers. Some might want to get under >>> the >>>>>> hood and tinker and tune, or contribute their own benchmarks, with the >>>>>> understanding that it’s not a turnkey or standalone product, but a tool >>>>>> that requires getting your hands dirty. >>>>>>>>> >>>>>>>>> Would a “1 page” readme with a few tips on “how to run on a laptop” >>>>> be >>>>>> enough to let other interested contributors help get geode-benchmarks >>> to >>>>> a >>>>>> “better state”? >>>>>>>>> >>>>>>>>>> On Jan 14, 2020, at 9:38 PM, Jacob Barrett <jbarr...@pivotal.io> >>>>>> wrote: >>>>>>>>>> >>>>>>>>>> I don’t think the benchmarks provide any material benefit to a user >>>>>> in their current state. They are heavily tuned for our CI process which >>>>>> relies on very beefy machines. Usage on other hardware will require >>> more >>>>>> tuning. I don’t think it’s worth putting the source in the release >>> until >>>>>> they are in a better state. >>>>>>>>>> >>>>>>>>>> -Jake >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> On Jan 14, 2020, at 4:14 PM, Dan Smith <dsm...@pivotal.io> >>> wrote: >>>>>>>>>>> >>>>>>>>>>> On Tue, Jan 14, 2020 at 4:11 PM Owen Nichols < >>> onich...@pivotal.io >>>>>> >>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> I believe the desire is to include the source code for >>>>>> geode-benchmarks as >>>>>>>>>>>> part of the official geode release, much like how we include >>>>>> geode-examples. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Oh! I thought you meant running the benchmarks in the release >>>>>> pipeline - I >>>>>>>>>>> think last release we were running them but decided they were too >>>>>> flaky to >>>>>>>>>>> use. >>>>>>>>>>> >>>>>>>>>>> +1 to including the benchmark source in the source release. >>>>>>>>>>> >>>>>>>>>>> -Dan >>>>>>>>> >>>>>>> >>>>>> >>>>> >>> >>> >
