Re: Intra-project dependencies

[Benedict]

The answer to all your questions is “like any other library” - this is a 
procedural hack to ease development. There are alternative isomorphic hacks, 
like compiling source jars from Accord and including them in the C* tree, if it 
helps your mental model.

> you stated that a goal was to avoid maintaining multiple branches.

No, I stated that a goal was to *decouple* development of Accord from C*. I 
don’t see why you would take that to mean there are no branches of Accord, as 
that would quite clearly be incompatible with the C* release strategy.



> On 17 Jan 2023, at 07:36, Mick Semb Wever  wrote:
> 
> 
>>> … extrapolating this experience to multiple C* versions
> 
> To include forward-merging, bisecting old history, etc etc. that's a leap of 
> faith that I believe deserves the discussion.
> 
>>> - patches are off submodule SHAs, not the submodule's HEAD,
>> 
>> A SHA would point to the HEAD of a given branch, at the time of merge, just 
>> by SHA? I’ve no idea what you imagine here, but this just ensures that a 
>> given SHA of the importing project continues to compile correctly when it is 
>> no longer HEAD. It does not mean there’s no HEAD that corresponds directly 
>> to the SHA of the importing project’s HEAD.
> 
> 
> That wasn't my concern. Rather that you need to know in advance when the SHA 
> is not HEAD. You can't commit off a past SHA. Once you find out (and how does 
> this happen?) that the submodule code is not HEAD what do you then do? What 
> if fast-forwarding the submodule to HEAD's SHA breaks things, do you now have 
> to fix that or introduce branching in the submodule? If the submodule doesn't 
> have releases, is it doing versioning, and if not how are branches 
> distinguished? 
> 
> Arn't these all fair enquiries to raise? 
> 
>>> - you need to be making commits to all branches (and forward merging) 
>>> anyway to update submodule SHAs,
>> 
>> Exactly as you would any library upgrade?
> 
> 
> Correct. submodules does not solve/remove the need to commit to multiple 
> branches and forward merge.
> Furthermore submodules means at least one additional commit, and possibly 
> twice as many commits.
>  
> 
>>> - if development is active on trunk, and then you need an update on an 
>>> older branch, you have to accommodate to backporting all those trunk 
>>> changes (or introduce the same branching in the submodule),
>> 
>> If you do feature development against Accord then you will obviously branch 
>> it? You would only make bug fixes to a bug fix branch. I’m not sure what you 
>> think is wrong here.
> 
> 
> That's not obvious, you stated that a goal was to avoid maintaining multiple 
> branches. Sure there's benefits to a lazy branching approach, but it 
> contradicts your initial motivations and introduces methodology changes that 
> are worth pointing out. What happens when there are multiple consumers of 
> Accord, and (like the situation we face with jamm) its HEAD is well in front 
> of anything C* is using.
> 
> As Henrik states, the underlying problem doesn't change, we're just choosing 
> between trade-offs. My concern is that we're not even doing a very good job 
> of choosing between the trade-offs. Based on past experiences with 
> submodules: that started with great excitement and led to tears and 
> frustration after a few years; I'm only pushing for a more thorough 
> discussion and proposal.
> 
>  
> 

Re: Merging CEP-15 to trunk

[Benedict]

> but the pre-commit gateway here is higher than the previous tickets being 
> worked on

Which tickets, and why?

> On 17 Jan 2023, at 07:43, Mick Semb Wever  wrote:
> 
> 
> 
> 
>> Could you file a bug report with more detail about which classes you think 
>> are lacking adequate documentation in each project, and what you would like 
>> to see? 
> 
> 
> I suggest instead that we open a task ticket for the merge.
> 
> I 100% agree with merging work incrementally, under a feature flag, but the 
> pre-commit gateway here is higher than the previous tickets being worked on. 
> API changes, pre-commit test results, and high (/entry) level comments, all 
> deserve any extra eyeballs available.
> 

Re: Intra-project dependencies

[Derek Chen-Becker]

I'd like to go back to Benedict's initial point: if we have a new
consensus protocol that other projects would potentially be interested
in, then by all means it should be its own project. Let's start with
that as a basis for discussion, because from my reading it seems like
people might be disagreeing with that initial premise.

If we agree that Accord should be independent, I'm +1 for git
submodules primarily because that's a standard way of doing things and
I don't think we need yet another bespoke solution to a problem that
hundreds, if not thousands of other software projects encounter. I've
worked with lots of projects using submodules and while they're not a
panacea, they've never been a significant problem to work with.

It's also a little confusing to see people argue about HEAD in
relation to any of this, since that's just an alias to the latest
commit for a given branch. In every project I've worked with that uses
submodules you would never use HEAD, because the submodule itself
already records the *exact* commit associated with the parent.

Cheers,

Derek

On Tue, Jan 17, 2023 at 2:28 AM Benedict  wrote:
>
> The answer to all your questions is “like any other library” - this is a 
> procedural hack to ease development. There are alternative isomorphic hacks, 
> like compiling source jars from Accord and including them in the C* tree, if 
> it helps your mental model.
>
> > you stated that a goal was to avoid maintaining multiple branches.
>
> No, I stated that a goal was to *decouple* development of Accord from C*. I 
> don’t see why you would take that to mean there are no branches of Accord, as 
> that would quite clearly be incompatible with the C* release strategy.
>
>
>
> On 17 Jan 2023, at 07:36, Mick Semb Wever  wrote:
>
> 
>>
>> … extrapolating this experience to multiple C* versions
>
>
> To include forward-merging, bisecting old history, etc etc. that's a leap of 
> faith that I believe deserves the discussion.
>
>> - patches are off submodule SHAs, not the submodule's HEAD,
>>
>>
>> A SHA would point to the HEAD of a given branch, at the time of merge, just 
>> by SHA? I’ve no idea what you imagine here, but this just ensures that a 
>> given SHA of the importing project continues to compile correctly when it is 
>> no longer HEAD. It does not mean there’s no HEAD that corresponds directly 
>> to the SHA of the importing project’s HEAD.
>
>
>
> That wasn't my concern. Rather that you need to know in advance when the SHA 
> is not HEAD. You can't commit off a past SHA. Once you find out (and how does 
> this happen?) that the submodule code is not HEAD what do you then do? What 
> if fast-forwarding the submodule to HEAD's SHA breaks things, do you now have 
> to fix that or introduce branching in the submodule? If the submodule doesn't 
> have releases, is it doing versioning, and if not how are branches 
> distinguished?
>
> Arn't these all fair enquiries to raise?
>
>> - you need to be making commits to all branches (and forward merging) anyway 
>> to update submodule SHAs,
>>
>>
>> Exactly as you would any library upgrade?
>
>
>
> Correct. submodules does not solve/remove the need to commit to multiple 
> branches and forward merge.
> Furthermore submodules means at least one additional commit, and possibly 
> twice as many commits.
>
>
>> - if development is active on trunk, and then you need an update on an older 
>> branch, you have to accommodate to backporting all those trunk changes (or 
>> introduce the same branching in the submodule),
>>
>>
>> If you do feature development against Accord then you will obviously branch 
>> it? You would only make bug fixes to a bug fix branch. I’m not sure what you 
>> think is wrong here.
>
>
>
> That's not obvious, you stated that a goal was to avoid maintaining multiple 
> branches. Sure there's benefits to a lazy branching approach, but it 
> contradicts your initial motivations and introduces methodology changes that 
> are worth pointing out. What happens when there are multiple consumers of 
> Accord, and (like the situation we face with jamm) its HEAD is well in front 
> of anything C* is using.
>
> As Henrik states, the underlying problem doesn't change, we're just choosing 
> between trade-offs. My concern is that we're not even doing a very good job 
> of choosing between the trade-offs. Based on past experiences with 
> submodules: that started with great excitement and led to tears and 
> frustration after a few years; I'm only pushing for a more thorough 
> discussion and proposal.
>
>
>
>


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Intra-project dependencies

[Derek Chen-Becker]

Actually, re-reading the thread, I think I missed the initial point
brought up and got lost in the discussion specific to submodules. What
is the technical reason for bringing Accord in-tree? While I think
submodules are the best way to include source in-tree, I'm not sure
this is actually the correct thing to do in this case. Don't we
already have mechanisms to deal with snapshot versions of library
dependencies in the build? Do we need release votes for snapshots?

Thanks,

Derek

On Tue, Jan 17, 2023 at 9:25 AM Derek Chen-Becker  wrote:
>
> I'd like to go back to Benedict's initial point: if we have a new
> consensus protocol that other projects would potentially be interested
> in, then by all means it should be its own project. Let's start with
> that as a basis for discussion, because from my reading it seems like
> people might be disagreeing with that initial premise.
>
> If we agree that Accord should be independent, I'm +1 for git
> submodules primarily because that's a standard way of doing things and
> I don't think we need yet another bespoke solution to a problem that
> hundreds, if not thousands of other software projects encounter. I've
> worked with lots of projects using submodules and while they're not a
> panacea, they've never been a significant problem to work with.
>
> It's also a little confusing to see people argue about HEAD in
> relation to any of this, since that's just an alias to the latest
> commit for a given branch. In every project I've worked with that uses
> submodules you would never use HEAD, because the submodule itself
> already records the *exact* commit associated with the parent.
>
> Cheers,
>
> Derek
>
> On Tue, Jan 17, 2023 at 2:28 AM Benedict  wrote:
> >
> > The answer to all your questions is “like any other library” - this is a 
> > procedural hack to ease development. There are alternative isomorphic 
> > hacks, like compiling source jars from Accord and including them in the C* 
> > tree, if it helps your mental model.
> >
> > > you stated that a goal was to avoid maintaining multiple branches.
> >
> > No, I stated that a goal was to *decouple* development of Accord from C*. I 
> > don’t see why you would take that to mean there are no branches of Accord, 
> > as that would quite clearly be incompatible with the C* release strategy.
> >
> >
> >
> > On 17 Jan 2023, at 07:36, Mick Semb Wever  wrote:
> >
> > 
> >>
> >> … extrapolating this experience to multiple C* versions
> >
> >
> > To include forward-merging, bisecting old history, etc etc. that's a leap 
> > of faith that I believe deserves the discussion.
> >
> >> - patches are off submodule SHAs, not the submodule's HEAD,
> >>
> >>
> >> A SHA would point to the HEAD of a given branch, at the time of merge, 
> >> just by SHA? I’ve no idea what you imagine here, but this just ensures 
> >> that a given SHA of the importing project continues to compile correctly 
> >> when it is no longer HEAD. It does not mean there’s no HEAD that 
> >> corresponds directly to the SHA of the importing project’s HEAD.
> >
> >
> >
> > That wasn't my concern. Rather that you need to know in advance when the 
> > SHA is not HEAD. You can't commit off a past SHA. Once you find out (and 
> > how does this happen?) that the submodule code is not HEAD what do you then 
> > do? What if fast-forwarding the submodule to HEAD's SHA breaks things, do 
> > you now have to fix that or introduce branching in the submodule? If the 
> > submodule doesn't have releases, is it doing versioning, and if not how are 
> > branches distinguished?
> >
> > Arn't these all fair enquiries to raise?
> >
> >> - you need to be making commits to all branches (and forward merging) 
> >> anyway to update submodule SHAs,
> >>
> >>
> >> Exactly as you would any library upgrade?
> >
> >
> >
> > Correct. submodules does not solve/remove the need to commit to multiple 
> > branches and forward merge.
> > Furthermore submodules means at least one additional commit, and possibly 
> > twice as many commits.
> >
> >
> >> - if development is active on trunk, and then you need an update on an 
> >> older branch, you have to accommodate to backporting all those trunk 
> >> changes (or introduce the same branching in the submodule),
> >>
> >>
> >> If you do feature development against Accord then you will obviously 
> >> branch it? You would only make bug fixes to a bug fix branch. I’m not sure 
> >> what you think is wrong here.
> >
> >
> >
> > That's not obvious, you stated that a goal was to avoid maintaining 
> > multiple branches. Sure there's benefits to a lazy branching approach, but 
> > it contradicts your initial motivations and introduces methodology changes 
> > that are worth pointing out. What happens when there are multiple consumers 
> > of Accord, and (like the situation we face with jamm) its HEAD is well in 
> > front of anything C* is using.
> >
> > As Henrik states, the underlying problem doesn't change, we're just 
> > choosing between trade-offs

Re: Intra-project dependencies

[Henrik Ingo]

Hi Derek

Somewhat of a newcomer myself, it seems the answers to your excellent
questions are:

 * We don't all agree with the premise that Accord will attract substantial
outside interest. Even so, my personal opinion is that whether that happens
or not, it's not wrong to aspire toward or plan for such a future.

 * Yes, just using Accord as a library dependency would be the normal thing
to do, but that introduces a need to create Accord releases to match
Cassandra releases. Since ASF mandates a 3 day voting process to release
software artifacts, this creates a lot of bureaucratic overhead, which is
why this otherwise sane alternative is nobody's favorite. (Cassandra
releases cannot or should not depend on snapshot releases of libraries.

 * So we are discussing various alternatives that keep Accord separate,
while at the same time recording some link about which exact version of
Accord was checked out.

henrik

On Tue, Jan 17, 2023 at 7:23 PM Derek Chen-Becker 
wrote:

> Actually, re-reading the thread, I think I missed the initial point
> brought up and got lost in the discussion specific to submodules. What
> is the technical reason for bringing Accord in-tree? While I think
> submodules are the best way to include source in-tree, I'm not sure
> this is actually the correct thing to do in this case. Don't we
> already have mechanisms to deal with snapshot versions of library
> dependencies in the build? Do we need release votes for snapshots?
>
> Thanks,
>
> Derek
>
> On Tue, Jan 17, 2023 at 9:25 AM Derek Chen-Becker 
> wrote:
> >
> > I'd like to go back to Benedict's initial point: if we have a new
> > consensus protocol that other projects would potentially be interested
> > in, then by all means it should be its own project. Let's start with
> > that as a basis for discussion, because from my reading it seems like
> > people might be disagreeing with that initial premise.
> >
> > If we agree that Accord should be independent, I'm +1 for git
> > submodules primarily because that's a standard way of doing things and
> > I don't think we need yet another bespoke solution to a problem that
> > hundreds, if not thousands of other software projects encounter. I've
> > worked with lots of projects using submodules and while they're not a
> > panacea, they've never been a significant problem to work with.
> >
> > It's also a little confusing to see people argue about HEAD in
> > relation to any of this, since that's just an alias to the latest
> > commit for a given branch. In every project I've worked with that uses
> > submodules you would never use HEAD, because the submodule itself
> > already records the *exact* commit associated with the parent.
> >
> > Cheers,
> >
> > Derek
> >
> > On Tue, Jan 17, 2023 at 2:28 AM Benedict  wrote:
> > >
> > > The answer to all your questions is “like any other library” - this is
> a procedural hack to ease development. There are alternative isomorphic
> hacks, like compiling source jars from Accord and including them in the C*
> tree, if it helps your mental model.
> > >
> > > > you stated that a goal was to avoid maintaining multiple branches.
> > >
> > > No, I stated that a goal was to *decouple* development of Accord from
> C*. I don’t see why you would take that to mean there are no branches of
> Accord, as that would quite clearly be incompatible with the C* release
> strategy.
> > >
> > >
> > >
> > > On 17 Jan 2023, at 07:36, Mick Semb Wever  wrote:
> > >
> > > 
> > >>
> > >> … extrapolating this experience to multiple C* versions
> > >
> > >
> > > To include forward-merging, bisecting old history, etc etc. that's a
> leap of faith that I believe deserves the discussion.
> > >
> > >> - patches are off submodule SHAs, not the submodule's HEAD,
> > >>
> > >>
> > >> A SHA would point to the HEAD of a given branch, at the time of
> merge, just by SHA? I’ve no idea what you imagine here, but this just
> ensures that a given SHA of the importing project continues to compile
> correctly when it is no longer HEAD. It does not mean there’s no HEAD that
> corresponds directly to the SHA of the importing project’s HEAD.
> > >
> > >
> > >
> > > That wasn't my concern. Rather that you need to know in advance when
> the SHA is not HEAD. You can't commit off a past SHA. Once you find out
> (and how does this happen?) that the submodule code is not HEAD what do you
> then do? What if fast-forwarding the submodule to HEAD's SHA breaks things,
> do you now have to fix that or introduce branching in the submodule? If the
> submodule doesn't have releases, is it doing versioning, and if not how are
> branches distinguished?
> > >
> > > Arn't these all fair enquiries to raise?
> > >
> > >> - you need to be making commits to all branches (and forward merging)
> anyway to update submodule SHAs,
> > >>
> > >>
> > >> Exactly as you would any library upgrade?
> > >
> > >
> > >
> > > Correct. submodules does not solve/remove the need to commit to
> multiple branches and forward merge.
> 

Re: Intra-project dependencies

[Josh McKenzie]

Is there any reason we couldn't "bundle" a release vote to include both an 
Accord release and ASF C* in one voting round as a combined release? My reading 
of the release process w/the ASF doesn't speak to that (if anything it implies 
this might be a valid approach):

https://www.apache.org/legal/release-policy.html#release-approval

> Every ASF release MUST contain one or more source packages,

On Tue, Jan 17, 2023, at 4:03 PM, Henrik Ingo wrote:
> Hi Derek
> 
> Somewhat of a newcomer myself, it seems the answers to your excellent 
> questions are:
> 
>  * We don't all agree with the premise that Accord will attract substantial 
> outside interest. Even so, my personal opinion is that whether that happens 
> or not, it's not wrong to aspire toward or plan for such a future.
> 
>  * Yes, just using Accord as a library dependency would be the normal thing 
> to do, but that introduces a need to create Accord releases to match 
> Cassandra releases. Since ASF mandates a 3 day voting process to release 
> software artifacts, this creates a lot of bureaucratic overhead, which is why 
> this otherwise sane alternative is nobody's favorite. (Cassandra releases 
> cannot or should not depend on snapshot releases of libraries.
> 
>  * So we are discussing various alternatives that keep Accord separate, while 
> at the same time recording some link about which exact version of Accord was 
> checked out.
> 
> henrik
> 
> On Tue, Jan 17, 2023 at 7:23 PM Derek Chen-Becker  
> wrote:
>> Actually, re-reading the thread, I think I missed the initial point
>> brought up and got lost in the discussion specific to submodules. What
>> is the technical reason for bringing Accord in-tree? While I think
>> submodules are the best way to include source in-tree, I'm not sure
>> this is actually the correct thing to do in this case. Don't we
>> already have mechanisms to deal with snapshot versions of library
>> dependencies in the build? Do we need release votes for snapshots?
>> 
>> Thanks,
>> 
>> Derek
>> 
>> On Tue, Jan 17, 2023 at 9:25 AM Derek Chen-Becker  
>> wrote:
>> >
>> > I'd like to go back to Benedict's initial point: if we have a new
>> > consensus protocol that other projects would potentially be interested
>> > in, then by all means it should be its own project. Let's start with
>> > that as a basis for discussion, because from my reading it seems like
>> > people might be disagreeing with that initial premise.
>> >
>> > If we agree that Accord should be independent, I'm +1 for git
>> > submodules primarily because that's a standard way of doing things and
>> > I don't think we need yet another bespoke solution to a problem that
>> > hundreds, if not thousands of other software projects encounter. I've
>> > worked with lots of projects using submodules and while they're not a
>> > panacea, they've never been a significant problem to work with.
>> >
>> > It's also a little confusing to see people argue about HEAD in
>> > relation to any of this, since that's just an alias to the latest
>> > commit for a given branch. In every project I've worked with that uses
>> > submodules you would never use HEAD, because the submodule itself
>> > already records the *exact* commit associated with the parent.
>> >
>> > Cheers,
>> >
>> > Derek
>> >
>> > On Tue, Jan 17, 2023 at 2:28 AM Benedict  wrote:
>> > >
>> > > The answer to all your questions is “like any other library” - this is a 
>> > > procedural hack to ease development. There are alternative isomorphic 
>> > > hacks, like compiling source jars from Accord and including them in the 
>> > > C* tree, if it helps your mental model.
>> > >
>> > > > you stated that a goal was to avoid maintaining multiple branches.
>> > >
>> > > No, I stated that a goal was to *decouple* development of Accord from 
>> > > C*. I don’t see why you would take that to mean there are no branches of 
>> > > Accord, as that would quite clearly be incompatible with the C* release 
>> > > strategy.
>> > >
>> > >
>> > >
>> > > On 17 Jan 2023, at 07:36, Mick Semb Wever  wrote:
>> > >
>> > > 
>> > >>
>> > >> … extrapolating this experience to multiple C* versions
>> > >
>> > >
>> > > To include forward-merging, bisecting old history, etc etc. that's a 
>> > > leap of faith that I believe deserves the discussion.
>> > >
>> > >> - patches are off submodule SHAs, not the submodule's HEAD,
>> > >>
>> > >>
>> > >> A SHA would point to the HEAD of a given branch, at the time of merge, 
>> > >> just by SHA? I’ve no idea what you imagine here, but this just ensures 
>> > >> that a given SHA of the importing project continues to compile 
>> > >> correctly when it is no longer HEAD. It does not mean there’s no HEAD 
>> > >> that corresponds directly to the SHA of the importing project’s HEAD.
>> > >
>> > >
>> > >
>> > > That wasn't my concern. Rather that you need to know in advance when the 
>> > > SHA is not HEAD. You can't commit off a past SHA. Once you find out (and 
>> > > how does this happen?) that the

Re: Intra-project dependencies

[Benedict]

I am certainly not proposing any certainty about outside interest, but I think as the only full implementation of a leaderless protocol in existence, as well as an open source pluggable distributed transaction protocol, the chance of some interest is not vanishingly small (once it is proven in Cassandra). Whether that will translate to any useful interest is even less certain, but I think it would be valuable if it transpires.It’s also not the only reason to develop this project as a library, and wasn’t one of the reasons given when we discussed this when the proposal was discussed initially. The main reasons were ensuring that the transaction system in Cassandra was not tied to Accord, and to allow better testing in isolation.Regarding the use of snapshots, this isn’t impossible Henrik - I floated this as an option. But besides the additional overhead during development, this does not maintain reproducible builds, as the snapshot can change. Perhaps we could introduce permanent snapshots for a given SHA, but at that point it seems to just make more sense to use submodules. Including snapshots inside the lib directory until release would seem to be fine, and then perform parallel release votes.On 17 Jan 2023, at 21:04, Henrik Ingo  wrote:Hi DerekSomewhat of a newcomer myself, it seems the answers to your excellent questions are: * We don't all agree with the premise that Accord will attract substantial outside interest. Even so, my personal opinion is that whether that happens or not, it's not wrong to aspire toward or plan for such a future. * Yes, just using Accord as a library dependency would be the normal thing to do, but that introduces a need to create Accord releases to match Cassandra releases. Since ASF mandates a 3 day voting process to release software artifacts, this creates a lot of bureaucratic overhead, which is why this otherwise sane alternative is nobody's favorite. (Cassandra releases cannot or should not depend on snapshot releases of libraries. * So we are discussing various alternatives that keep Accord separate, while at the same time recording some link about which exact version of Accord was checked out.henrikOn Tue, Jan 17, 2023 at 7:23 PM Derek Chen-Becker  wrote:Actually, re-reading the thread, I think I missed the initial point
brought up and got lost in the discussion specific to submodules. What
is the technical reason for bringing Accord in-tree? While I think
submodules are the best way to include source in-tree, I'm not sure
this is actually the correct thing to do in this case. Don't we
already have mechanisms to deal with snapshot versions of library
dependencies in the build? Do we need release votes for snapshots?

Thanks,

Derek

On Tue, Jan 17, 2023 at 9:25 AM Derek Chen-Becker  wrote:
>
> I'd like to go back to Benedict's initial point: if we have a new
> consensus protocol that other projects would potentially be interested
> in, then by all means it should be its own project. Let's start with
> that as a basis for discussion, because from my reading it seems like
> people might be disagreeing with that initial premise.
>
> If we agree that Accord should be independent, I'm +1 for git
> submodules primarily because that's a standard way of doing things and
> I don't think we need yet another bespoke solution to a problem that
> hundreds, if not thousands of other software projects encounter. I've
> worked with lots of projects using submodules and while they're not a
> panacea, they've never been a significant problem to work with.
>
> It's also a little confusing to see people argue about HEAD in
> relation to any of this, since that's just an alias to the latest
> commit for a given branch. In every project I've worked with that uses
> submodules you would never use HEAD, because the submodule itself
> already records the *exact* commit associated with the parent.
>
> Cheers,
>
> Derek
>
> On Tue, Jan 17, 2023 at 2:28 AM Benedict  wrote:
> >
> > The answer to all your questions is “like any other library” - this is a procedural hack to ease development. There are alternative isomorphic hacks, like compiling source jars from Accord and including them in the C* tree, if it helps your mental model.
> >
> > > you stated that a goal was to avoid maintaining multiple branches.
> >
> > No, I stated that a goal was to *decouple* development of Accord from C*. I don’t see why you would take that to mean there are no branches of Accord, as that would quite clearly be incompatible with the C* release strategy.
> >
> >
> >
> > On 17 Jan 2023, at 07:36, Mick Semb Wever  wrote:
> >
> > 
> >>
> >> … extrapolating this experience to multiple C* versions
> >
> >
> > To include forward-merging, bisecting old history, etc etc. that's a leap of faith that I believe deserves the discussion.
> >
> >> - patches are off submodule SHAs, not the submodule's HEAD,
> >>
> >>
> >> A SHA would point to the 

Re: Intra-project dependencies

[Mick Semb Wever]

>
> Regarding the use of snapshots, this isn’t impossible Henrik - I floated
> this as an option. But besides the additional overhead during development,
> this does not maintain reproducible builds, as the snapshot can change.
>

You would reference the snapshot dependency by the timestamped snapshot.
This makes it a reproducible build.

We have done this with dtest-api already, and there's already a comment
explaining it:
https://github.com/apache/cassandra/blob/trunk/.build/build-resolver.xml#L59-L60


It introduces some overhead when bisecting to go from the snapshot's
timestamp to the other repo's SHA (this is easily solvable by putting the
SHA inside the jarfile).

I don't see the problem of letting trunk use snapshots during the annual
development cycle, if we accept the overhead of cutting all library
releases before we cut the first alpha/beta.

FTR, i'm sitting on the fence between this and submodules. There's many dev
tasks we do, and different approaches have different pain points. The
amount of dev happening in the library also matters. I also agree with
Derek that linking in the source code into in-tree is a significant thing
to do, just to avoid the rigamaroles of dependency management.

Josh, bundling releases gets tricky in that you need to include the library
sources, because the cassandra release is essentially being voted on
(because it has been built) with non-released dependencies.

Re: Intra-project dependencies

[Josh McKenzie]

> Josh, bundling releases gets tricky in that you need to include the library 
> sources, because the cassandra release is essentially being voted on (because 
> it has been built) with non-released dependencies.
Arguably, one shouldn't vote on a release of Accord unless there's something 
that's integrated it and shown it's working. Through that lens it doesn't make 
sense to release those dependencies w/out the parent, nor the parent without 
the dependency.

Not a hill I'm willing to die on but at least out of the gate, seems like a way 
we could streamline the process of cutting releases until someone / something 
external starts exerting influence on Accord.

On Tue, Jan 17, 2023, at 4:39 PM, Mick Semb Wever wrote:
>> Regarding the use of snapshots, this isn’t impossible Henrik - I floated 
>> this as an option. But besides the additional overhead during development, 
>> this does not maintain reproducible builds, as the snapshot can change. 
> 
> 
> You would reference the snapshot dependency by the timestamped snapshot. This 
> makes it a reproducible build.
> 
> We have done this with dtest-api already, and there's already a comment 
> explaining it:
> https://github.com/apache/cassandra/blob/trunk/.build/build-resolver.xml#L59-L60
>  
> 
> It introduces some overhead when bisecting to go from the snapshot's 
> timestamp to the other repo's SHA (this is easily solvable by putting the SHA 
> inside the jarfile).
> 
> I don't see the problem of letting trunk use snapshots during the annual 
> development cycle, if we accept the overhead of cutting all library releases 
> before we cut the first alpha/beta.
> 
> FTR, i'm sitting on the fence between this and submodules. There's many dev 
> tasks we do, and different approaches have different pain points. The amount 
> of dev happening in the library also matters. I also agree with Derek that 
> linking in the source code into in-tree is a significant thing to do, just to 
> avoid the rigamaroles of dependency management.
> 
> Josh, bundling releases gets tricky in that you need to include the library 
> sources, because the cassandra release is essentially being voted on (because 
> it has been built) with non-released dependencies.

Re: Intra-project dependencies

[Benedict]

> You would reference the snapshot dependency by the timestamped snapshot. This 
> makes it a reproducible build.

How confident are we that the repository will not alter or delete them?

> linking in the source code into in-tree is a significant thing to do

Could you explain why? I thought your preferred alternative was merging the 
source trees permanently

> On 17 Jan 2023, at 21:40, Mick Semb Wever  wrote:
> 
> 
>> Regarding the use of snapshots, this isn’t impossible Henrik - I floated 
>> this as an option. But besides the additional overhead during development, 
>> this does not maintain reproducible builds, as the snapshot can change. 
> 
> You would reference the snapshot dependency by the timestamped snapshot. This 
> makes it a reproducible build.
> 
> We have done this with dtest-api already, and there's already a comment 
> explaining it:
> https://github.com/apache/cassandra/blob/trunk/.build/build-resolver.xml#L59-L60
>  
> 
> It introduces some overhead when bisecting to go from the snapshot's 
> timestamp to the other repo's SHA (this is easily solvable by putting the SHA 
> inside the jarfile).
> 
> I don't see the problem of letting trunk use snapshots during the annual 
> development cycle, if we accept the overhead of cutting all library releases 
> before we cut the first alpha/beta.
> 
> FTR, i'm sitting on the fence between this and submodules. There's many dev 
> tasks we do, and different approaches have different pain points. The amount 
> of dev happening in the library also matters. I also agree with Derek that 
> linking in the source code into in-tree is a significant thing to do, just to 
> avoid the rigamaroles of dependency management.
> 
> Josh, bundling releases gets tricky in that you need to include the library 
> sources, because the cassandra release is essentially being voted on (because 
> it has been built) with non-released dependencies.

Re: Intra-project dependencies

[Henrik Ingo]

On Tue, Jan 17, 2023 at 11:40 PM Mick Semb Wever  wrote:

>
>> It introduces some overhead when bisecting to go from the snapshot's
> timestamp to the other repo's SHA (this is easily solvable by putting the
> SHA inside the jarfile).
>

Whatever system we choose, the link should be the SHA. It shouldn't be
necessary for a human to lookup the necessary parameters based on some
mapping to other parameters.

A basic first order requirement: Restarting an old build in Jenkins should
rerun the exact same version of all modules.

More complex requirement: Starting a Jenkins build from cassandra commit
123abc, should checkout/download/use the correct versions of  other modules
at the time 123abc was committed.

henrik


-- 

Henrik Ingo

c. +358 40 569 7354

w. www.datastax.com