Re: CASSANDRA-18941 produce size bounded SSTables from CQLSSTableWriter

[Josh McKenzie]

> to 4.0 and up to trunk
Think the proposal is all supported branches from 4.0 up.

+1 here.

On Mon, Oct 23, 2023, at 10:19 PM, guo Maxwell wrote:
> +1, but I want to know why only trunk and 4.0 ? not all the versions 
> involved, like 4.1 ,5.0 。
> 
> Francisco Guerrero  于2023年10月24日周二 07:47写道：
>> +1 (nb). I think this is a great addition to offline tools that use SSTable 
>> writer in general.
>> 
>> On 2023/10/23 23:21:13 Yifan Cai wrote:
>> > Hi,
>> > 
>> > I want to propose merging the patch in CASSANDRA-18941 to 4.0 and up to
>> > trunk and hope we are all OK with it.
>> > 
>> > In CASSANDRA-18941, I am adding the capability to produce size-bounded
>> > SSTables in CQLSSTableWriter for sorted data. It can greatly benefit
>> > Cassandra Analytics (https://github.com/apache/cassandra-analytics) for
>> > bulk writing SSTables, since it avoids buffering and sorting on flush,
>> > given the data source is sorted already in the bulk write process.
>> > Cassandra Analytics supports Cassandra 4.0 and depends on the cassandra-all
>> > 4.0.x library. Therefore, we are mostly interested in using the new
>> > capability in 4.0.
>> > 
>> > CQLSSTableWriter is only used in offline tools and never in the code path
>> > of Cassandra server.
>> > 
>> > Any objections to merging the patch to 4.0 and up to trunk?
>> > 
>> > - Yifan
>> >

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Benjamin Lerer]

My biggest concern with the 5.1 suggestion is that it makes the review of
TCM far more complicated than it should be. Even if all TCM patches were
fully reviewed by committers that I fully trust, due to the patch size and
the impact of the changes it feels safer to me to have a second round of
review now than the dust is settling. Merging TCM and Accord in a 5.1
branch and doing the review there will make things harder than it should.

Le lun. 23 oct. 2023 à 23:02, Dinesh Joshi  a écrit :

> I have a strong preference to move out the 5.0 date to have accord and
> TCM. I don’t see the point in shipping 5.0 without these features
> especially if 5.1 is going to follow close behind it.
>
> Dinesh
>
> On Oct 23, 2023, at 4:52 AM, Mick Semb Wever  wrote:
>
> 
>
> The TCM work (CEP-21) is in its review stage but being well past our
> cut-off date¹ for merging, and now jeopardising 5.0 GA efforts, I would
> like to propose the following.
>
> We merge TCM and Accord only to trunk.  Then branch cassandra-5.1 and cut
> an immediate 5.1-alpha1 release.
>
> I see this as a win-win scenario for us, considering our current
> situation.  (Though it is unfortunate that Accord is included in this
> scenario because we agreed it to be based upon TCM.)
>
> This will mean…
>  - We get to focus on getting 5.0 to beta and GA, which already has a ton
> of features users want.
>  - We get an alpha release with TCM and Accord into users hands quickly
> for broader testing and feedback.
>  - We isolate GA efforts on TCM and Accord – giving oss and downstream
> engineers time and patience reviewing and testing.  TCM will be the biggest
> patch ever to land in C*.
>  - Give users a choice for a more incremental upgrade approach, given just
> how many new features we're putting on them in one year.
>  - 5.1 w/ TCM and Accord will maintain its upgrade compatibility with all
> 4.x versions, just as if it had landed in 5.0.
>
>
> The risks/costs this introduces are
>  - If we cannot stabilise TCM and/or Accord on the cassandra-5.1 branch,
> and at some point decide to undo this work, while we can throw away the
> cassandra-5.1 branch we would need to do a bit of work reverting the
> changes in trunk.  This is a _very_ edge case, as confidence levels on the
> design and implementation of both are already tested and high.
>  - We will have to maintain an additional branch.  I propose that we treat
> the 5.1 branch in the same maintenance window as 5.0 (like we have with 3.0
> and 3.11).  This also adds the merge path overhead.
>  - Reviewing of TCM and Accord will continue to happen post-merge.  This
> is not our normal practice, but this work will have already received its
> two +1s from committers, and such ongoing review effort is akin to GA
> stabilisation work on release branches.
>
>
> I see no other ok solution in front of us that gets us at least both the
> 5.0 beta and TCM+Accord alpha releases this year.  Keeping in mind users
> demand to start experimenting with these features, and our Cassandra Summit
> in December.
>
>
> 1) https://lists.apache.org/thread/9c5cnn57c7oqw8wzo3zs0dkrm4f17lm3
>
>
>

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Mick Semb Wever]

On Tue, 24 Oct 2023 at 12:25, Benjamin Lerer  wrote:

> My biggest concern with the 5.1 suggestion is that it makes the review of
> TCM far more complicated than it should be. Even if all TCM patches were
> fully reviewed by committers that I fully trust, due to the patch size and
> the impact of the changes it feels safer to me to have a second round of
> review now than the dust is settling. Merging TCM and Accord in a 5.1
> branch and doing the review there will make things harder than it should.
>


To be clear, in a cassandra-5.1 branch we would be performing this second
round of review post-merge, which as you say makes it more complicated.

But if we do this in the cassandra-5.0 branch, this second round of
review is then to be done pre-merge – blocking both tcm and accord from any
preview release that users can start playing with (i.e. users that want to
play with this will have to build it themselves from the feature branch).

Can you provide an estimate on how long we need for this second round of
review, best-case and worst case scenarios ?

CASSANDRA-16565

[Claude Warren, Jr via dev]

I am working on https://issues.apache.org/jira/browse/CASSANDRA-16565 and
have a small testing program that executes the sigar and equivalent OSHI
methods to verify that they are the same.  I would like to have this run on
various platforms.

I have tgz with all the libraries and code as well as a run.sh bash script
to compile and execute it.

Is there someplace I can stash the tgz that others can download it from?
The file info is : 6269066 okt 24 12:46 compare_oshi_sigar.tgz

OSHI does not implement all the methods that Sigar does and there is a
difference in the bitness (32/64 bit) of the results.

*Maximum Virtual Memory*

Sigar dates  from the time of 32 bit OS and so when checking for things
like maximum virtual memory it returns -1 (INFINITE) for any value over
0x7fff.

OSHI on the other hand is 64 bit aware and will return long values for the
maximum virtual memory.  Looking at tools like "ulimit" it converts
anything over 0x7fff to the text "infinite" in the return.

To handle this I set the expected Virtual memory to be 0x7FFFl and
accept any value >= that or  -1 as  the acceptable value.

*Maximum Processes*

This appears to be the value of "ulimit -u" which is not supported in
OSHI.  However, on Linux (and I think Mac OS) we can make a call to the
bash interpreter to return "uname -u".  On other systems I log that we
don't have a way to get this value and return the standard default max
processes of 1024.  This will cause the "warnIfRunningInDegradedMode" to
warn about possible degradation.

Claude

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Brandon Williams]

The concern I have with this is that is a big slippery 'if' that
involves development time estimation, and if it tends to take longer
than we estimate (as these things tend to do), then I can see a future
where 5.0 is delayed until the middle of 2024, and I really don't want
that to happen.  Maybe it won't be a glamorous release but shipping
5.0 mitigates our worst case scenario.

Kind Regards,
Brandon

On Mon, Oct 23, 2023 at 4:02 PM Dinesh Joshi  wrote:
>
> I have a strong preference to move out the 5.0 date to have accord and TCM. I 
> don’t see the point in shipping 5.0 without these features especially if 5.1 
> is going to follow close behind it.
>
> Dinesh
>
> On Oct 23, 2023, at 4:52 AM, Mick Semb Wever  wrote:
>
> 
>
> The TCM work (CEP-21) is in its review stage but being well past our cut-off 
> date¹ for merging, and now jeopardising 5.0 GA efforts, I would like to 
> propose the following.
>
> We merge TCM and Accord only to trunk.  Then branch cassandra-5.1 and cut an 
> immediate 5.1-alpha1 release.
>
> I see this as a win-win scenario for us, considering our current situation.  
> (Though it is unfortunate that Accord is included in this scenario because we 
> agreed it to be based upon TCM.)
>
> This will mean…
>  - We get to focus on getting 5.0 to beta and GA, which already has a ton of 
> features users want.
>  - We get an alpha release with TCM and Accord into users hands quickly for 
> broader testing and feedback.
>  - We isolate GA efforts on TCM and Accord – giving oss and downstream 
> engineers time and patience reviewing and testing.  TCM will be the biggest 
> patch ever to land in C*.
>  - Give users a choice for a more incremental upgrade approach, given just 
> how many new features we're putting on them in one year.
>  - 5.1 w/ TCM and Accord will maintain its upgrade compatibility with all 4.x 
> versions, just as if it had landed in 5.0.
>
>
> The risks/costs this introduces are
>  - If we cannot stabilise TCM and/or Accord on the cassandra-5.1 branch, and 
> at some point decide to undo this work, while we can throw away the 
> cassandra-5.1 branch we would need to do a bit of work reverting the changes 
> in trunk.  This is a _very_ edge case, as confidence levels on the design and 
> implementation of both are already tested and high.
>  - We will have to maintain an additional branch.  I propose that we treat 
> the 5.1 branch in the same maintenance window as 5.0 (like we have with 3.0 
> and 3.11).  This also adds the merge path overhead.
>  - Reviewing of TCM and Accord will continue to happen post-merge.  This is 
> not our normal practice, but this work will have already received its two +1s 
> from committers, and such ongoing review effort is akin to GA stabilisation 
> work on release branches.
>
>
> I see no other ok solution in front of us that gets us at least both the 5.0 
> beta and TCM+Accord alpha releases this year.  Keeping in mind users demand 
> to start experimenting with these features, and our Cassandra Summit in 
> December.
>
>
> 1) https://lists.apache.org/thread/9c5cnn57c7oqw8wzo3zs0dkrm4f17lm3
>
>

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Josh McKenzie]

> Maybe it won't be a glamorous release but shipping
> 5.0 mitigates our worst case scenario.
I disagree with this characterization of 5.0 personally. UCS, SAI, Trie 
memtables and sstables, maybe vector ANN if the sub-tasks on C-18715 are 
accurate, all combine to make 5.0 a pretty glamorous release IMO independent of 
TCM and Accord. Accord is a true paradigm-shift game-changer so it's easy to 
think of 5.0 as uneventful in comparison, and TCM helps resolve one of the 
biggest pain-points in our system for over a decade, but I think 5.0 is a very 
meaty release in its own right today.

Anyway - I agree with you Brandon re: timelines. If things take longer than 
we'd hope (which, if I think back, they do roughly 100% of the time on this 
project), blocking on these features could both lead to a significant delay in 
5.0 going out as well as increasing pressure and risk of burnout on the folks 
working on it. While I believe we all need some balanced urgency to do our best 
work, being under the gun for something with a hard deadline or having an 
entire project drag along blocked on you is not where I want any of us to be.

Part of why we talked about going to primarily annual calendar-based releases 
was to avoid precisely this situation, where something that *feels* right at 
the cusp of merging leads us to delay a release repeatedly. We discussed this a 
couple times this year:
1: https://lists.apache.org/thread/9c5cnn57c7oqw8wzo3zs0dkrm4f17lm3, where Mick 
proposed a "soft-freeze" for everything w/out exception and 1st week October 
"hard-freeze", and there was assumed to be lazy consensus
2: https://lists.apache.org/thread/mzj3dq8b7mzf60k6mkby88b9n9ywmsgw, where we 
kept along with what we discussed in 1 but added in CEP-30 to be waivered in as 
well.

So. We're at a crossroads here where we need to either follow through with what 
we all agreed to earlier this year, or acknowledge that our best intention of 
calendar-based releases can't stand up to our optimism and desire to get these 
features into the next major.

There's no immediate obvious better path to me in terms of what's best for our 
users. This is a situation of risk tolerance with a lot of unknowns that could 
go either way.

Any light that folks active on TCM and Accord could shed in terms of their best 
and worst-case scenarios on timelines for those features might help us narrow 
this down a bit. Otherwise, I'm inclined to defer to our past selves and fall 
back to "we agreed to yearly calendar releases for good reason. Let's stick to 
our guns."

On Tue, Oct 24, 2023, at 9:37 AM, Brandon Williams wrote:
> The concern I have with this is that is a big slippery 'if' that
> involves development time estimation, and if it tends to take longer
> than we estimate (as these things tend to do), then I can see a future
> where 5.0 is delayed until the middle of 2024, and I really don't want
> that to happen.  Maybe it won't be a glamorous release but shipping
> 5.0 mitigates our worst case scenario.
> 
> Kind Regards,
> Brandon
> 
> On Mon, Oct 23, 2023 at 4:02 PM Dinesh Joshi  wrote:
> >
> > I have a strong preference to move out the 5.0 date to have accord and TCM. 
> > I don’t see the point in shipping 5.0 without these features especially if 
> > 5.1 is going to follow close behind it.
> >
> > Dinesh
> >
> > On Oct 23, 2023, at 4:52 AM, Mick Semb Wever  wrote:
> >
> > 
> >
> > The TCM work (CEP-21) is in its review stage but being well past our 
> > cut-off date¹ for merging, and now jeopardising 5.0 GA efforts, I would 
> > like to propose the following.
> >
> > We merge TCM and Accord only to trunk.  Then branch cassandra-5.1 and cut 
> > an immediate 5.1-alpha1 release.
> >
> > I see this as a win-win scenario for us, considering our current situation. 
> >  (Though it is unfortunate that Accord is included in this scenario because 
> > we agreed it to be based upon TCM.)
> >
> > This will mean…
> >  - We get to focus on getting 5.0 to beta and GA, which already has a ton 
> > of features users want.
> >  - We get an alpha release with TCM and Accord into users hands quickly for 
> > broader testing and feedback.
> >  - We isolate GA efforts on TCM and Accord – giving oss and downstream 
> > engineers time and patience reviewing and testing.  TCM will be the biggest 
> > patch ever to land in C*.
> >  - Give users a choice for a more incremental upgrade approach, given just 
> > how many new features we're putting on them in one year.
> >  - 5.1 w/ TCM and Accord will maintain its upgrade compatibility with all 
> > 4.x versions, just as if it had landed in 5.0.
> >
> >
> > The risks/costs this introduces are
> >  - If we cannot stabilise TCM and/or Accord on the cassandra-5.1 branch, 
> > and at some point decide to undo this work, while we can throw away the 
> > cassandra-5.1 branch we would need to do a bit of work reverting the 
> > changes in trunk.  This is a _very_ edge case, as confidence levels on the 
> > design and imple

Re: CASSANDRA-16565

[Brandon Williams]

On Tue, Oct 24, 2023 at 7:48 AM Claude Warren, Jr via dev
 wrote:
>
> Is there someplace I can stash the tgz that others can download it from? The 
> file info is : 6269066 okt 24 12:46 compare_oshi_sigar.tgz

Is posting a github branch not sufficient?

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Jonathan Ellis]

On Tue, Oct 24, 2023 at 9:23 AM Josh McKenzie  wrote:

> Maybe it won't be a glamorous release but shipping
> 5.0 mitigates our worst case scenario.
>
> I disagree with this characterization of 5.0 personally. UCS, SAI, Trie
> memtables and sstables, maybe vector ANN if the sub-tasks on C-18715 are
> accurate, all combine to make 5.0 a pretty glamorous release IMO
> independent of TCM and Accord. Accord is a true paradigm-shift game-changer
> so it's easy to think of 5.0 as uneventful in comparison, and TCM helps
> resolve one of the biggest pain-points in our system for over a decade, but
> I think 5.0 is a very meaty release in its own right today.
>

Yes.  SAI will make a huge difference for almost everyone, and ANN will be
even more relevant to a smaller subset.

Let's not get sucked back into "we can't do fast releases because we need
to wait for major features, and we have to wait for major features because
it will be a long time before the next release."

-- 
Jonathan Ellis
co-founder, http://www.datastax.com
@spyced

Re: CASSANDRA-18941 produce size bounded SSTables from CQLSSTableWriter

[Francisco Guerrero]

I understood that the intention is for this to land in 4.0 all the way up to 
trunk (including 4.1 and 5.0).

On 2023/10/24 02:19:41 guo Maxwell wrote:
> +1, but I want to know why only trunk and 4.0 ? not all the versions
> involved, like 4.1 ,5.0 。
> 
> Francisco Guerrero  于2023年10月24日周二 07:47写道：
> 
> > +1 (nb). I think this is a great addition to offline tools that use
> > SSTable writer in general.
> >
> > On 2023/10/23 23:21:13 Yifan Cai wrote:
> > > Hi,
> > >
> > > I want to propose merging the patch in CASSANDRA-18941 to 4.0 and up to
> > > trunk and hope we are all OK with it.
> > >
> > > In CASSANDRA-18941, I am adding the capability to produce size-bounded
> > > SSTables in CQLSSTableWriter for sorted data. It can greatly benefit
> > > Cassandra Analytics (https://github.com/apache/cassandra-analytics) for
> > > bulk writing SSTables, since it avoids buffering and sorting on flush,
> > > given the data source is sorted already in the bulk write process.
> > > Cassandra Analytics supports Cassandra 4.0 and depends on the
> > cassandra-all
> > > 4.0.x library. Therefore, we are mostly interested in using the new
> > > capability in 4.0.
> > >
> > > CQLSSTableWriter is only used in offline tools and never in the code path
> > > of Cassandra server.
> > >
> > > Any objections to merging the patch to 4.0 and up to trunk?
> > >
> > > - Yifan
> > >
> >
> 

Re: CASSANDRA-18941 produce size bounded SSTables from CQLSSTableWriter

[Brandon Williams]

+1

Kind Regards,
Brandon

On Mon, Oct 23, 2023 at 6:22 PM Yifan Cai  wrote:
>
> Hi,
>
> I want to propose merging the patch in CASSANDRA-18941 to 4.0 and up to trunk 
> and hope we are all OK with it.
>
> In CASSANDRA-18941, I am adding the capability to produce size-bounded 
> SSTables in CQLSSTableWriter for sorted data. It can greatly benefit 
> Cassandra Analytics (https://github.com/apache/cassandra-analytics) for bulk 
> writing SSTables, since it avoids buffering and sorting on flush, given the 
> data source is sorted already in the bulk write process. Cassandra Analytics 
> supports Cassandra 4.0 and depends on the cassandra-all 4.0.x library. 
> Therefore, we are mostly interested in using the new capability in 4.0.
>
> CQLSSTableWriter is only used in offline tools and never in the code path of 
> Cassandra server.
>
> Any objections to merging the patch to 4.0 and up to trunk?
>
> - Yifan

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Patrick McFadin]

+1 to what you are saying, Josh. Based on the last survey, yes, everyone
was excited about Accord, but SAI and UCS were pretty high on the list.

Benedict and I had a good conversation last night, and now I understand
more essential details for this conversation. TCM is taking far more work
than initially scoped, and Accord depends on a stable TCM. TCM is months
behind and that's a critical fact, and one I personally just learned of. I
thought things were wrapping up this month, and we were in the testing
phase. I get why that's a topic we are dancing around. Nobody wants to say
ship dates are slipping because that's part of our culture. It's
disappointing and, if new information, an unwelcome surprise, but none of
us should be angry or in a blamey mood because I guarantee every one of us
has shipped the code late. My reaction yesterday was based on an incorrect
assumption. Now that I have a better picture, my point of view is changing.

Josh's point about what's best for users is crucial. Users deserve stable
code with a regular cadence of features that make their lives easier. If we
put 5.0 on hold for TCM + Accord, users will get neither for a very long
time. And I mentioned a disaster yesterday. A bigger disaster would be
shipping Accord with a major bug that causes data loss, eroding community
trust. Accord has to be the most bulletproof of all bulletproof features.
The pressure to ship is only going to increase and that's fertile ground
for that sort of bug.

So, taking a step back and with a clearer picture, I support the 5.0 + 5.1
plan mainly because I don't think 5.1 is (or should be) a fast follow.

For the user community, the communication should be straightforward. TCM +
Accord are turning out to be much more complicated than was originally
scoped, and for good reasons. Our first principle is to provide a stable
and reliable system, so as a result, we'll be de-coupling TCM + Accord from
5.0 into a 5.1 branch, which is available in parallel to 5.0 while
additional hardening and testing is done. We can communicate this in a blog
post.,

To make this much more palatable to our use community, if we can get a
build and docker image available ASAP with Accord, it will allow developers
to start playing with the syntax. Up to this point, that hasn't been widely
available unless you compile the code yourself. Developers need to
understand how this will work in an application, and up to this point, the
syntax is text they see in my slides. We need to get some hands-on and that
will get our user community engaged on Accord this calendar year. The
feedback may even uncover some critical changes we'll need to make. Lack of
access to Accord by developers is a critical problem we can fix soon and
there will be plenty of excitement there and start building use cases
before the final code ships.

I'm bummed but realistic. It sucks that I won't have a pony for Christmas,
but maybe one for my birthday?

Patrick



On Tue, Oct 24, 2023 at 7:23 AM Josh McKenzie  wrote:

> Maybe it won't be a glamorous release but shipping
> 5.0 mitigates our worst case scenario.
>
> I disagree with this characterization of 5.0 personally. UCS, SAI, Trie
> memtables and sstables, maybe vector ANN if the sub-tasks on C-18715 are
> accurate, all combine to make 5.0 a pretty glamorous release IMO
> independent of TCM and Accord. Accord is a true paradigm-shift game-changer
> so it's easy to think of 5.0 as uneventful in comparison, and TCM helps
> resolve one of the biggest pain-points in our system for over a decade, but
> I think 5.0 is a very meaty release in its own right today.
>
> Anyway - I agree with you Brandon re: timelines. If things take longer
> than we'd hope (which, if I think back, they do roughly 100% of the time on
> this project), blocking on these features could both lead to a significant
> delay in 5.0 going out as well as increasing pressure and risk of burnout
> on the folks working on it. While I believe we all need some balanced
> urgency to do our best work, being under the gun for something with a hard
> deadline or having an entire project drag along blocked on you is not where
> I want any of us to be.
>
> Part of why we talked about going to primarily annual calendar-based
> releases was to avoid precisely this situation, where something that
> *feels* right at the cusp of merging leads us to delay a release
> repeatedly. We discussed this a couple times this year:
> 1: https://lists.apache.org/thread/9c5cnn57c7oqw8wzo3zs0dkrm4f17lm3,
> where Mick proposed a "soft-freeze" for everything w/out exception and 1st
> week October "hard-freeze", and there was assumed to be lazy consensus
> 2: https://lists.apache.org/thread/mzj3dq8b7mzf60k6mkby88b9n9ywmsgw,
> where we kept along with what we discussed in 1 but added in CEP-30 to be
> waivered in as well.
>
> So. We're at a crossroads here where we need to either follow through with
> what we all agreed to earlier this year, or acknowledge that our best
> intenti

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Jon Haddad]

I guess at the end of the day, shipping a release with a bunch of awesome 
features is better than holding it back.  If there's 2 big releases in 6 months 
the community isn't any worse off.  

We either ship something, or nothing, and something is probably better.

Jon


On 2023/10/24 16:27:04 Patrick McFadin wrote:
> +1 to what you are saying, Josh. Based on the last survey, yes, everyone
> was excited about Accord, but SAI and UCS were pretty high on the list.
> 
> Benedict and I had a good conversation last night, and now I understand
> more essential details for this conversation. TCM is taking far more work
> than initially scoped, and Accord depends on a stable TCM. TCM is months
> behind and that's a critical fact, and one I personally just learned of. I
> thought things were wrapping up this month, and we were in the testing
> phase. I get why that's a topic we are dancing around. Nobody wants to say
> ship dates are slipping because that's part of our culture. It's
> disappointing and, if new information, an unwelcome surprise, but none of
> us should be angry or in a blamey mood because I guarantee every one of us
> has shipped the code late. My reaction yesterday was based on an incorrect
> assumption. Now that I have a better picture, my point of view is changing.
> 
> Josh's point about what's best for users is crucial. Users deserve stable
> code with a regular cadence of features that make their lives easier. If we
> put 5.0 on hold for TCM + Accord, users will get neither for a very long
> time. And I mentioned a disaster yesterday. A bigger disaster would be
> shipping Accord with a major bug that causes data loss, eroding community
> trust. Accord has to be the most bulletproof of all bulletproof features.
> The pressure to ship is only going to increase and that's fertile ground
> for that sort of bug.
> 
> So, taking a step back and with a clearer picture, I support the 5.0 + 5.1
> plan mainly because I don't think 5.1 is (or should be) a fast follow.
> 
> For the user community, the communication should be straightforward. TCM +
> Accord are turning out to be much more complicated than was originally
> scoped, and for good reasons. Our first principle is to provide a stable
> and reliable system, so as a result, we'll be de-coupling TCM + Accord from
> 5.0 into a 5.1 branch, which is available in parallel to 5.0 while
> additional hardening and testing is done. We can communicate this in a blog
> post.,
> 
> To make this much more palatable to our use community, if we can get a
> build and docker image available ASAP with Accord, it will allow developers
> to start playing with the syntax. Up to this point, that hasn't been widely
> available unless you compile the code yourself. Developers need to
> understand how this will work in an application, and up to this point, the
> syntax is text they see in my slides. We need to get some hands-on and that
> will get our user community engaged on Accord this calendar year. The
> feedback may even uncover some critical changes we'll need to make. Lack of
> access to Accord by developers is a critical problem we can fix soon and
> there will be plenty of excitement there and start building use cases
> before the final code ships.
> 
> I'm bummed but realistic. It sucks that I won't have a pony for Christmas,
> but maybe one for my birthday?
> 
> Patrick
> 
> 
> 
> On Tue, Oct 24, 2023 at 7:23 AM Josh McKenzie  wrote:
> 
> > Maybe it won't be a glamorous release but shipping
> > 5.0 mitigates our worst case scenario.
> >
> > I disagree with this characterization of 5.0 personally. UCS, SAI, Trie
> > memtables and sstables, maybe vector ANN if the sub-tasks on C-18715 are
> > accurate, all combine to make 5.0 a pretty glamorous release IMO
> > independent of TCM and Accord. Accord is a true paradigm-shift game-changer
> > so it's easy to think of 5.0 as uneventful in comparison, and TCM helps
> > resolve one of the biggest pain-points in our system for over a decade, but
> > I think 5.0 is a very meaty release in its own right today.
> >
> > Anyway - I agree with you Brandon re: timelines. If things take longer
> > than we'd hope (which, if I think back, they do roughly 100% of the time on
> > this project), blocking on these features could both lead to a significant
> > delay in 5.0 going out as well as increasing pressure and risk of burnout
> > on the folks working on it. While I believe we all need some balanced
> > urgency to do our best work, being under the gun for something with a hard
> > deadline or having an entire project drag along blocked on you is not where
> > I want any of us to be.
> >
> > Part of why we talked about going to primarily annual calendar-based
> > releases was to avoid precisely this situation, where something that
> > *feels* right at the cusp of merging leads us to delay a release
> > repeatedly. We discussed this a couple times this year:
> > 1: https://lists.apache.org/thread/9c5cnn57c7oqw8wzo3zs0dkrm4f17lm3,

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Jeremiah Jordan]

 If we decide to go the route of not merging TCM to the 5.0 branch.  Do we
actually need to immediately cut a 5.1 branch?  Can we work on stabilizing
things while it is in trunk and cut the 5.1 branch when we actually think
we are near releasing?  I don’t see any reason we can not cut “preview”
artifacts from trunk?

-Jeremiah

On Oct 24, 2023 at 11:54:25 AM, Jon Haddad 
wrote:

> I guess at the end of the day, shipping a release with a bunch of awesome
> features is better than holding it back.  If there's 2 big releases in 6
> months the community isn't any worse off.
>
> We either ship something, or nothing, and something is probably better.
>
> Jon
>
>
> On 2023/10/24 16:27:04 Patrick McFadin wrote:
>
> +1 to what you are saying, Josh. Based on the last survey, yes, everyone
>
> was excited about Accord, but SAI and UCS were pretty high on the list.
>
>
> Benedict and I had a good conversation last night, and now I understand
>
> more essential details for this conversation. TCM is taking far more work
>
> than initially scoped, and Accord depends on a stable TCM. TCM is months
>
> behind and that's a critical fact, and one I personally just learned of. I
>
> thought things were wrapping up this month, and we were in the testing
>
> phase. I get why that's a topic we are dancing around. Nobody wants to say
>
> ship dates are slipping because that's part of our culture. It's
>
> disappointing and, if new information, an unwelcome surprise, but none of
>
> us should be angry or in a blamey mood because I guarantee every one of us
>
> has shipped the code late. My reaction yesterday was based on an incorrect
>
> assumption. Now that I have a better picture, my point of view is changing.
>
>
> Josh's point about what's best for users is crucial. Users deserve stable
>
> code with a regular cadence of features that make their lives easier. If we
>
> put 5.0 on hold for TCM + Accord, users will get neither for a very long
>
> time. And I mentioned a disaster yesterday. A bigger disaster would be
>
> shipping Accord with a major bug that causes data loss, eroding community
>
> trust. Accord has to be the most bulletproof of all bulletproof features.
>
> The pressure to ship is only going to increase and that's fertile ground
>
> for that sort of bug.
>
>
> So, taking a step back and with a clearer picture, I support the 5.0 + 5.1
>
> plan mainly because I don't think 5.1 is (or should be) a fast follow.
>
>
> For the user community, the communication should be straightforward. TCM +
>
> Accord are turning out to be much more complicated than was originally
>
> scoped, and for good reasons. Our first principle is to provide a stable
>
> and reliable system, so as a result, we'll be de-coupling TCM + Accord from
>
> 5.0 into a 5.1 branch, which is available in parallel to 5.0 while
>
> additional hardening and testing is done. We can communicate this in a blog
>
> post.,
>
>
> To make this much more palatable to our use community, if we can get a
>
> build and docker image available ASAP with Accord, it will allow developers
>
> to start playing with the syntax. Up to this point, that hasn't been widely
>
> available unless you compile the code yourself. Developers need to
>
> understand how this will work in an application, and up to this point, the
>
> syntax is text they see in my slides. We need to get some hands-on and that
>
> will get our user community engaged on Accord this calendar year. The
>
> feedback may even uncover some critical changes we'll need to make. Lack of
>
> access to Accord by developers is a critical problem we can fix soon and
>
> there will be plenty of excitement there and start building use cases
>
> before the final code ships.
>
>
> I'm bummed but realistic. It sucks that I won't have a pony for Christmas,
>
> but maybe one for my birthday?
>
>
> Patrick
>
>
>
>
> On Tue, Oct 24, 2023 at 7:23 AM Josh McKenzie 
> wrote:
>
>
> > Maybe it won't be a glamorous release but shipping
>
> > 5.0 mitigates our worst case scenario.
>
> >
>
> > I disagree with this characterization of 5.0 personally. UCS, SAI, Trie
>
> > memtables and sstables, maybe vector ANN if the sub-tasks on C-18715 are
>
> > accurate, all combine to make 5.0 a pretty glamorous release IMO
>
> > independent of TCM and Accord. Accord is a true paradigm-shift
> game-changer
>
> > so it's easy to think of 5.0 as uneventful in comparison, and TCM helps
>
> > resolve one of the biggest pain-points in our system for over a decade,
> but
>
> > I think 5.0 is a very meaty release in its own right today.
>
> >
>
> > Anyway - I agree with you Brandon re: timelines. If things take longer
>
> > than we'd hope (which, if I think back, they do roughly 100% of the time
> on
>
> > this project), blocking on these features could both lead to a
> significant
>
> > delay in 5.0 going out as well as increasing pressure and risk of burnout
>
> > on the folks working on it. While I believe we all need some balanced
>
> > urgency to do our

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Patrick McFadin]

I would like to have something for developers to use ASAP to try the Accord
syntax. Very few people have seen it, and I think there's a learning curve
we can start earlier.

It's my understanding that ASF policy is that it needs to be a project
release to create a docker image.

On Tue, Oct 24, 2023 at 11:54 AM Jeremiah Jordan 
wrote:

> If we decide to go the route of not merging TCM to the 5.0 branch.  Do we
> actually need to immediately cut a 5.1 branch?  Can we work on stabilizing
> things while it is in trunk and cut the 5.1 branch when we actually think
> we are near releasing?  I don’t see any reason we can not cut “preview”
> artifacts from trunk?
>
> -Jeremiah
>
> On Oct 24, 2023 at 11:54:25 AM, Jon Haddad 
> wrote:
>
>> I guess at the end of the day, shipping a release with a bunch of awesome
>> features is better than holding it back.  If there's 2 big releases in 6
>> months the community isn't any worse off.
>>
>> We either ship something, or nothing, and something is probably better.
>>
>> Jon
>>
>>
>> On 2023/10/24 16:27:04 Patrick McFadin wrote:
>>
>> +1 to what you are saying, Josh. Based on the last survey, yes, everyone
>>
>> was excited about Accord, but SAI and UCS were pretty high on the list.
>>
>>
>> Benedict and I had a good conversation last night, and now I understand
>>
>> more essential details for this conversation. TCM is taking far more work
>>
>> than initially scoped, and Accord depends on a stable TCM. TCM is months
>>
>> behind and that's a critical fact, and one I personally just learned of. I
>>
>> thought things were wrapping up this month, and we were in the testing
>>
>> phase. I get why that's a topic we are dancing around. Nobody wants to say
>>
>> ship dates are slipping because that's part of our culture. It's
>>
>> disappointing and, if new information, an unwelcome surprise, but none of
>>
>> us should be angry or in a blamey mood because I guarantee every one of us
>>
>> has shipped the code late. My reaction yesterday was based on an incorrect
>>
>> assumption. Now that I have a better picture, my point of view is
>> changing.
>>
>>
>> Josh's point about what's best for users is crucial. Users deserve stable
>>
>> code with a regular cadence of features that make their lives easier. If
>> we
>>
>> put 5.0 on hold for TCM + Accord, users will get neither for a very long
>>
>> time. And I mentioned a disaster yesterday. A bigger disaster would be
>>
>> shipping Accord with a major bug that causes data loss, eroding community
>>
>> trust. Accord has to be the most bulletproof of all bulletproof features.
>>
>> The pressure to ship is only going to increase and that's fertile ground
>>
>> for that sort of bug.
>>
>>
>> So, taking a step back and with a clearer picture, I support the 5.0 + 5.1
>>
>> plan mainly because I don't think 5.1 is (or should be) a fast follow.
>>
>>
>> For the user community, the communication should be straightforward. TCM +
>>
>> Accord are turning out to be much more complicated than was originally
>>
>> scoped, and for good reasons. Our first principle is to provide a stable
>>
>> and reliable system, so as a result, we'll be de-coupling TCM + Accord
>> from
>>
>> 5.0 into a 5.1 branch, which is available in parallel to 5.0 while
>>
>> additional hardening and testing is done. We can communicate this in a
>> blog
>>
>> post.,
>>
>>
>> To make this much more palatable to our use community, if we can get a
>>
>> build and docker image available ASAP with Accord, it will allow
>> developers
>>
>> to start playing with the syntax. Up to this point, that hasn't been
>> widely
>>
>> available unless you compile the code yourself. Developers need to
>>
>> understand how this will work in an application, and up to this point, the
>>
>> syntax is text they see in my slides. We need to get some hands-on and
>> that
>>
>> will get our user community engaged on Accord this calendar year. The
>>
>> feedback may even uncover some critical changes we'll need to make. Lack
>> of
>>
>> access to Accord by developers is a critical problem we can fix soon and
>>
>> there will be plenty of excitement there and start building use cases
>>
>> before the final code ships.
>>
>>
>> I'm bummed but realistic. It sucks that I won't have a pony for Christmas,
>>
>> but maybe one for my birthday?
>>
>>
>> Patrick
>>
>>
>>
>>
>> On Tue, Oct 24, 2023 at 7:23 AM Josh McKenzie 
>> wrote:
>>
>>
>> > Maybe it won't be a glamorous release but shipping
>>
>> > 5.0 mitigates our worst case scenario.
>>
>> >
>>
>> > I disagree with this characterization of 5.0 personally. UCS, SAI, Trie
>>
>> > memtables and sstables, maybe vector ANN if the sub-tasks on C-18715 are
>>
>> > accurate, all combine to make 5.0 a pretty glamorous release IMO
>>
>> > independent of TCM and Accord. Accord is a true paradigm-shift
>> game-changer
>>
>> > so it's easy to think of 5.0 as uneventful in comparison, and TCM helps
>>
>> > resolve one of the biggest pain-points in our system for over 

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Jeremiah Jordan]

 In order for the project to advertise the release outside the dev@ list it
needs to be a formal release.  That just means that there was a release
vote and at least 3 PMC members +1’ed it, and there are more +1 than there
are -1, and we follow all the normal release rules.  The ASF release
process doesn’t care what branch you cut the artifacts from or what version
you call it.

So the project can cut artifacts for and release a 5.1-alpha1,
5.1-dev-preview1, what ever we want to version this thing, from trunk or
any other branch name we want.

-Jeremiah

On Oct 24, 2023 at 2:03:41 PM, Patrick McFadin  wrote:

> I would like to have something for developers to use ASAP to try the
> Accord syntax. Very few people have seen it, and I think there's a learning
> curve we can start earlier.
>
> It's my understanding that ASF policy is that it needs to be a project
> release to create a docker image.
>
> On Tue, Oct 24, 2023 at 11:54 AM Jeremiah Jordan <
> jeremiah.jor...@gmail.com> wrote:
>
>> If we decide to go the route of not merging TCM to the 5.0 branch.  Do we
>> actually need to immediately cut a 5.1 branch?  Can we work on stabilizing
>> things while it is in trunk and cut the 5.1 branch when we actually think
>> we are near releasing?  I don’t see any reason we can not cut “preview”
>> artifacts from trunk?
>>
>> -Jeremiah
>>
>> On Oct 24, 2023 at 11:54:25 AM, Jon Haddad 
>> wrote:
>>
>>> I guess at the end of the day, shipping a release with a bunch of
>>> awesome features is better than holding it back.  If there's 2 big releases
>>> in 6 months the community isn't any worse off.
>>>
>>> We either ship something, or nothing, and something is probably better.
>>>
>>> Jon
>>>
>>>
>>> On 2023/10/24 16:27:04 Patrick McFadin wrote:
>>>
>>> +1 to what you are saying, Josh. Based on the last survey, yes, everyone
>>>
>>> was excited about Accord, but SAI and UCS were pretty high on the list.
>>>
>>>
>>> Benedict and I had a good conversation last night, and now I understand
>>>
>>> more essential details for this conversation. TCM is taking far more work
>>>
>>> than initially scoped, and Accord depends on a stable TCM. TCM is months
>>>
>>> behind and that's a critical fact, and one I personally just learned of.
>>> I
>>>
>>> thought things were wrapping up this month, and we were in the testing
>>>
>>> phase. I get why that's a topic we are dancing around. Nobody wants to
>>> say
>>>
>>> ship dates are slipping because that's part of our culture. It's
>>>
>>> disappointing and, if new information, an unwelcome surprise, but none of
>>>
>>> us should be angry or in a blamey mood because I guarantee every one of
>>> us
>>>
>>> has shipped the code late. My reaction yesterday was based on an
>>> incorrect
>>>
>>> assumption. Now that I have a better picture, my point of view is
>>> changing.
>>>
>>>
>>> Josh's point about what's best for users is crucial. Users deserve stable
>>>
>>> code with a regular cadence of features that make their lives easier. If
>>> we
>>>
>>> put 5.0 on hold for TCM + Accord, users will get neither for a very long
>>>
>>> time. And I mentioned a disaster yesterday. A bigger disaster would be
>>>
>>> shipping Accord with a major bug that causes data loss, eroding community
>>>
>>> trust. Accord has to be the most bulletproof of all bulletproof features.
>>>
>>> The pressure to ship is only going to increase and that's fertile ground
>>>
>>> for that sort of bug.
>>>
>>>
>>> So, taking a step back and with a clearer picture, I support the 5.0 +
>>> 5.1
>>>
>>> plan mainly because I don't think 5.1 is (or should be) a fast follow.
>>>
>>>
>>> For the user community, the communication should be straightforward. TCM
>>> +
>>>
>>> Accord are turning out to be much more complicated than was originally
>>>
>>> scoped, and for good reasons. Our first principle is to provide a stable
>>>
>>> and reliable system, so as a result, we'll be de-coupling TCM + Accord
>>> from
>>>
>>> 5.0 into a 5.1 branch, which is available in parallel to 5.0 while
>>>
>>> additional hardening and testing is done. We can communicate this in a
>>> blog
>>>
>>> post.,
>>>
>>>
>>> To make this much more palatable to our use community, if we can get a
>>>
>>> build and docker image available ASAP with Accord, it will allow
>>> developers
>>>
>>> to start playing with the syntax. Up to this point, that hasn't been
>>> widely
>>>
>>> available unless you compile the code yourself. Developers need to
>>>
>>> understand how this will work in an application, and up to this point,
>>> the
>>>
>>> syntax is text they see in my slides. We need to get some hands-on and
>>> that
>>>
>>> will get our user community engaged on Accord this calendar year. The
>>>
>>> feedback may even uncover some critical changes we'll need to make. Lack
>>> of
>>>
>>> access to Accord by developers is a critical problem we can fix soon and
>>>
>>> there will be plenty of excitement there and start building use cases
>>>
>>> before the final code

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Josh McKenzie]

> In order for the project to advertise the release outside the dev@ list it 
> needs to be a formal release.
That's my reading as well:
https://www.apache.org/legal/release-policy.html#release-definition

I wonder if there'd be value in us having a cronned job that'd do nightly 
docker container builds on trunk + feature branches, archived for N days, and 
we make that generally known to the dev@ list here so folks that want to poke 
at the current state of trunk or other branches could do so with very low 
friction. We'd probably see more engagement on feature branches if it was 
turn-key easy for other C* devs to spin the up and check them out.

For what you're talking about here Patrick (a docker image for folks outside 
the dev@ audience and more user-facing), we'd want to vote on it and go through 
the formal process.

On Tue, Oct 24, 2023, at 3:10 PM, Jeremiah Jordan wrote:
> In order for the project to advertise the release outside the dev@ list it 
> needs to be a formal release.  That just means that there was a release vote 
> and at least 3 PMC members +1’ed it, and there are more +1 than there are -1, 
> and we follow all the normal release rules.  The ASF release process doesn’t 
> care what branch you cut the artifacts from or what version you call it.
> 
> So the project can cut artifacts for and release a 5.1-alpha1, 
> 5.1-dev-preview1, what ever we want to version this thing, from trunk or any 
> other branch name we want.
> 
> -Jeremiah
> 
> On Oct 24, 2023 at 2:03:41 PM, Patrick McFadin  wrote:
>> I would like to have something for developers to use ASAP to try the Accord 
>> syntax. Very few people have seen it, and I think there's a learning curve 
>> we can start earlier.
>> 
>> It's my understanding that ASF policy is that it needs to be a project 
>> release to create a docker image.
>> 
>> On Tue, Oct 24, 2023 at 11:54 AM Jeremiah Jordan  
>> wrote:
>>> If we decide to go the route of not merging TCM to the 5.0 branch.  Do we 
>>> actually need to immediately cut a 5.1 branch?  Can we work on stabilizing 
>>> things while it is in trunk and cut the 5.1 branch when we actually think 
>>> we are near releasing?  I don’t see any reason we can not cut “preview” 
>>> artifacts from trunk?
>>> 
>>> -Jeremiah
>>> 
>>> On Oct 24, 2023 at 11:54:25 AM, Jon Haddad  
>>> wrote:
 I guess at the end of the day, shipping a release with a bunch of awesome 
 features is better than holding it back.  If there's 2 big releases in 6 
 months the community isn't any worse off.  
 
 We either ship something, or nothing, and something is probably better.
 
 Jon
 
 
 On 2023/10/24 16:27:04 Patrick McFadin wrote:
> +1 to what you are saying, Josh. Based on the last survey, yes, everyone
> was excited about Accord, but SAI and UCS were pretty high on the list.
> 
> Benedict and I had a good conversation last night, and now I understand
> more essential details for this conversation. TCM is taking far more work
> than initially scoped, and Accord depends on a stable TCM. TCM is months
> behind and that's a critical fact, and one I personally just learned of. I
> thought things were wrapping up this month, and we were in the testing
> phase. I get why that's a topic we are dancing around. Nobody wants to say
> ship dates are slipping because that's part of our culture. It's
> disappointing and, if new information, an unwelcome surprise, but none of
> us should be angry or in a blamey mood because I guarantee every one of us
> has shipped the code late. My reaction yesterday was based on an incorrect
> assumption. Now that I have a better picture, my point of view is 
> changing.
> 
> Josh's point about what's best for users is crucial. Users deserve stable
> code with a regular cadence of features that make their lives easier. If 
> we
> put 5.0 on hold for TCM + Accord, users will get neither for a very long
> time. And I mentioned a disaster yesterday. A bigger disaster would be
> shipping Accord with a major bug that causes data loss, eroding community
> trust. Accord has to be the most bulletproof of all bulletproof features.
> The pressure to ship is only going to increase and that's fertile ground
> for that sort of bug.
> 
> So, taking a step back and with a clearer picture, I support the 5.0 + 5.1
> plan mainly because I don't think 5.1 is (or should be) a fast follow.
> 
> For the user community, the communication should be straightforward. TCM +
> Accord are turning out to be much more complicated than was originally
> scoped, and for good reasons. Our first principle is to provide a stable
> and reliable system, so as a result, we'll be de-coupling TCM + Accord 
> from
> 5.0 into a 5.1 branch, which is available in parallel to 5.0 while
> additional hardening and testing is done. We can communicate this in a 
> 

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Patrick McFadin]

Let me make that really easy. Hell yes

Not everybody runs CCM, I've tried but I've met resistance.

Compiling your own version usually involves me saying the words "Yes, ant
realclean exists. I'm not trolling you"

docker pull  works on every OS and curates a single node experience.



On Tue, Oct 24, 2023 at 12:37 PM Josh McKenzie  wrote:

> In order for the project to advertise the release outside the dev@ list
> it needs to be a formal release.
>
> That's my reading as well:
> https://www.apache.org/legal/release-policy.html#release-definition
>
> I wonder if there'd be value in us having a cronned job that'd do nightly
> docker container builds on trunk + feature branches, archived for N days,
> and we make that generally known to the dev@ list here so folks that want
> to poke at the current state of trunk or other branches could do so with
> very low friction. We'd probably see more engagement on feature branches if
> it was turn-key easy for other C* devs to spin the up and check them out.
>
> For what you're talking about here Patrick (a docker image for folks
> outside the dev@ audience and more user-facing), we'd want to vote on it
> and go through the formal process.
>
> On Tue, Oct 24, 2023, at 3:10 PM, Jeremiah Jordan wrote:
>
> In order for the project to advertise the release outside the dev@ list
> it needs to be a formal release.  That just means that there was a release
> vote and at least 3 PMC members +1’ed it, and there are more +1 than there
> are -1, and we follow all the normal release rules.  The ASF release
> process doesn’t care what branch you cut the artifacts from or what version
> you call it.
>
> So the project can cut artifacts for and release a 5.1-alpha1,
> 5.1-dev-preview1, what ever we want to version this thing, from trunk or
> any other branch name we want.
>
> -Jeremiah
>
> On Oct 24, 2023 at 2:03:41 PM, Patrick McFadin  wrote:
>
> I would like to have something for developers to use ASAP to try the
> Accord syntax. Very few people have seen it, and I think there's a learning
> curve we can start earlier.
>
> It's my understanding that ASF policy is that it needs to be a project
> release to create a docker image.
>
> On Tue, Oct 24, 2023 at 11:54 AM Jeremiah Jordan <
> jeremiah.jor...@gmail.com> wrote:
>
> If we decide to go the route of not merging TCM to the 5.0 branch.  Do we
> actually need to immediately cut a 5.1 branch?  Can we work on stabilizing
> things while it is in trunk and cut the 5.1 branch when we actually think
> we are near releasing?  I don’t see any reason we can not cut “preview”
> artifacts from trunk?
>
> -Jeremiah
>
> On Oct 24, 2023 at 11:54:25 AM, Jon Haddad 
> wrote:
>
> I guess at the end of the day, shipping a release with a bunch of awesome
> features is better than holding it back.  If there's 2 big releases in 6
> months the community isn't any worse off.
>
> We either ship something, or nothing, and something is probably better.
>
> Jon
>
>
> On 2023/10/24 16:27:04 Patrick McFadin wrote:
>
> +1 to what you are saying, Josh. Based on the last survey, yes, everyone
>
> was excited about Accord, but SAI and UCS were pretty high on the list.
>
>
> Benedict and I had a good conversation last night, and now I understand
>
> more essential details for this conversation. TCM is taking far more work
>
> than initially scoped, and Accord depends on a stable TCM. TCM is months
>
> behind and that's a critical fact, and one I personally just learned of. I
>
> thought things were wrapping up this month, and we were in the testing
>
> phase. I get why that's a topic we are dancing around. Nobody wants to say
>
> ship dates are slipping because that's part of our culture. It's
>
> disappointing and, if new information, an unwelcome surprise, but none of
>
> us should be angry or in a blamey mood because I guarantee every one of us
>
> has shipped the code late. My reaction yesterday was based on an incorrect
>
> assumption. Now that I have a better picture, my point of view is changing.
>
>
> Josh's point about what's best for users is crucial. Users deserve stable
>
> code with a regular cadence of features that make their lives easier. If we
>
> put 5.0 on hold for TCM + Accord, users will get neither for a very long
>
> time. And I mentioned a disaster yesterday. A bigger disaster would be
>
> shipping Accord with a major bug that causes data loss, eroding community
>
> trust. Accord has to be the most bulletproof of all bulletproof features.
>
> The pressure to ship is only going to increase and that's fertile ground
>
> for that sort of bug.
>
>
> So, taking a step back and with a clearer picture, I support the 5.0 + 5.1
>
> plan mainly because I don't think 5.1 is (or should be) a fast follow.
>
>
> For the user community, the communication should be straightforward. TCM +
>
> Accord are turning out to be much more complicated than was originally
>
> scoped, and for good reasons. Our first principle is to provide a stable
>
> and reliable s

Re: CASSANDRA-18941 produce size bounded SSTables from CQLSSTableWriter

[Chris Lohfink]

+1

On Tue, Oct 24, 2023 at 11:24 AM Brandon Williams  wrote:

> +1
>
> Kind Regards,
> Brandon
>
> On Mon, Oct 23, 2023 at 6:22 PM Yifan Cai  wrote:
> >
> > Hi,
> >
> > I want to propose merging the patch in CASSANDRA-18941 to 4.0 and up to
> trunk and hope we are all OK with it.
> >
> > In CASSANDRA-18941, I am adding the capability to produce size-bounded
> SSTables in CQLSSTableWriter for sorted data. It can greatly benefit
> Cassandra Analytics (https://github.com/apache/cassandra-analytics) for
> bulk writing SSTables, since it avoids buffering and sorting on flush,
> given the data source is sorted already in the bulk write process.
> Cassandra Analytics supports Cassandra 4.0 and depends on the cassandra-all
> 4.0.x library. Therefore, we are mostly interested in using the new
> capability in 4.0.
> >
> > CQLSSTableWriter is only used in offline tools and never in the code
> path of Cassandra server.
> >
> > Any objections to merging the patch to 4.0 and up to trunk?
> >
> > - Yifan
>

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[lorinapoland]

I'm certainly a fan of docker images as a dev tool! LorinaSent from my Verizon, 
Samsung Galaxy smartphone
 Original message From: Patrick McFadin  
Date: 10/24/23  13:31  (GMT-08:00) To: dev@cassandra.apache.org Subject: Re: 
Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1) 
Let me make that really easy. Hell yesNot everybody runs CCM, I've tried but 
I've met resistance. Compiling your own version usually involves me saying the 
words "Yes, ant realclean exists. I'm not trolling you" docker pull  
works on every OS and curates a single node experience. On Tue, Oct 24, 2023 at 
12:37 PM Josh McKenzie  wrote:In order for the project to 
advertise the release outside the dev@ list it needs to be a formal 
release.That's my reading as 
well:https://www.apache.org/legal/release-policy.html#release-definitionI 
wonder if there'd be value in us having a cronned job that'd do nightly docker 
container builds on trunk + feature branches, archived for N days, and we make 
that generally known to the dev@ list here so folks that want to poke at the 
current state of trunk or other branches could do so with very low friction. 
We'd probably see more engagement on feature branches if it was turn-key easy 
for other C* devs to spin the up and check them out.For what you're talking 
about here Patrick (a docker image for folks outside the dev@ audience and more 
user-facing), we'd want to vote on it and go through the formal process.On Tue, 
Oct 24, 2023, at 3:10 PM, Jeremiah Jordan wrote:In order for the project to 
advertise the release outside the dev@ list it needs to be a formal release.  
That just means that there was a release vote and at least 3 PMC members +1’ed 
it, and there are more +1 than there are -1, and we follow all the normal 
release rules.  The ASF release process doesn’t care what branch you cut the 
artifacts from or what version you call it.So the project can cut artifacts for 
and release a 5.1-alpha1, 5.1-dev-preview1, what ever we want to version this 
thing, from trunk or any other branch name we want.-JeremiahOn Oct 24, 2023 at 
2:03:41 PM, Patrick McFadin  wrote:I would like to have 
something for developers to use ASAP to try the Accord syntax. Very few people 
have seen it, and I think there's a learning curve we can start earlier.It's my 
understanding that ASF policy is that it needs to be a project release to 
create a docker image.On Tue, Oct 24, 2023 at 11:54 AM Jeremiah Jordan 
 wrote:If we decide to go the route of not merging 
TCM to the 5.0 branch.  Do we actually need to immediately cut a 5.1 branch?  
Can we work on stabilizing things while it is in trunk and cut the 5.1 branch 
when we actually think we are near releasing?  I don’t see any reason we can 
not cut “preview” artifacts from trunk?-JeremiahOn Oct 24, 2023 at 11:54:25 AM, 
Jon Haddad  wrote:I guess at the end of the day, 
shipping a release with a bunch of awesome features is better than holding it 
back.  If there's 2 big releases in 6 months the community isn't any worse off. 
 We either ship something, or nothing, and something is probably better.JonOn 
2023/10/24 16:27:04 Patrick McFadin wrote:+1 to what you are saying, Josh. 
Based on the last survey, yes, everyonewas excited about Accord, but SAI and 
UCS were pretty high on the list.Benedict and I had a good conversation last 
night, and now I understandmore essential details for this conversation. TCM is 
taking far more workthan initially scoped, and Accord depends on a stable TCM. 
TCM is monthsbehind and that's a critical fact, and one I personally just 
learned of. Ithought things were wrapping up this month, and we were in the 
testingphase. I get why that's a topic we are dancing around. Nobody wants to 
sayship dates are slipping because that's part of our culture. 
It'sdisappointing and, if new information, an unwelcome surprise, but none ofus 
should be angry or in a blamey mood because I guarantee every one of ushas 
shipped the code late. My reaction yesterday was based on an 
incorrectassumption. Now that I have a better picture, my point of view is 
changing.Josh's point about what's best for users is crucial. Users deserve 
stablecode with a regular cadence of features that make their lives easier. If 
weput 5.0 on hold for TCM + Accord, users will get neither for a very longtime. 
And I mentioned a disaster yesterday. A bigger disaster would beshipping Accord 
with a major bug that causes data loss, eroding communitytrust. Accord has to 
be the most bulletproof of all bulletproof features.The pressure to ship is 
only going to increase and that's fertile groundfor that sort of bug.So, taking 
a step back and with a clearer picture, I support the 5.0 + 5.1plan mainly 
because I don't think 5.1 is (or should be) a fast follow.For the user 
community, the communication should be straightforward. TCM +Accord are turning 
out to be much more complicated than was originallyscoped, and for good 
reasons

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Brandon Williams]

The catch here is that we don't publish docker images currently.  The
C* docker images available are not made by us.

Kind Regards,
Brandon

On Tue, Oct 24, 2023 at 3:31 PM Patrick McFadin  wrote:
>
> Let me make that really easy. Hell yes
>
> Not everybody runs CCM, I've tried but I've met resistance.
>
> Compiling your own version usually involves me saying the words "Yes, ant 
> realclean exists. I'm not trolling you"
>
> docker pull  works on every OS and curates a single node experience.
>
>
>
> On Tue, Oct 24, 2023 at 12:37 PM Josh McKenzie  wrote:
>>
>> In order for the project to advertise the release outside the dev@ list it 
>> needs to be a formal release.
>>
>> That's my reading as well:
>> https://www.apache.org/legal/release-policy.html#release-definition
>>
>> I wonder if there'd be value in us having a cronned job that'd do nightly 
>> docker container builds on trunk + feature branches, archived for N days, 
>> and we make that generally known to the dev@ list here so folks that want to 
>> poke at the current state of trunk or other branches could do so with very 
>> low friction. We'd probably see more engagement on feature branches if it 
>> was turn-key easy for other C* devs to spin the up and check them out.
>>
>> For what you're talking about here Patrick (a docker image for folks outside 
>> the dev@ audience and more user-facing), we'd want to vote on it and go 
>> through the formal process.
>>
>> On Tue, Oct 24, 2023, at 3:10 PM, Jeremiah Jordan wrote:
>>
>> In order for the project to advertise the release outside the dev@ list it 
>> needs to be a formal release.  That just means that there was a release vote 
>> and at least 3 PMC members +1’ed it, and there are more +1 than there are 
>> -1, and we follow all the normal release rules.  The ASF release process 
>> doesn’t care what branch you cut the artifacts from or what version you call 
>> it.
>>
>> So the project can cut artifacts for and release a 5.1-alpha1, 
>> 5.1-dev-preview1, what ever we want to version this thing, from trunk or any 
>> other branch name we want.
>>
>> -Jeremiah
>>
>> On Oct 24, 2023 at 2:03:41 PM, Patrick McFadin  wrote:
>>
>> I would like to have something for developers to use ASAP to try the Accord 
>> syntax. Very few people have seen it, and I think there's a learning curve 
>> we can start earlier.
>>
>> It's my understanding that ASF policy is that it needs to be a project 
>> release to create a docker image.
>>
>> On Tue, Oct 24, 2023 at 11:54 AM Jeremiah Jordan  
>> wrote:
>>
>> If we decide to go the route of not merging TCM to the 5.0 branch.  Do we 
>> actually need to immediately cut a 5.1 branch?  Can we work on stabilizing 
>> things while it is in trunk and cut the 5.1 branch when we actually think we 
>> are near releasing?  I don’t see any reason we can not cut “preview” 
>> artifacts from trunk?
>>
>> -Jeremiah
>>
>> On Oct 24, 2023 at 11:54:25 AM, Jon Haddad  
>> wrote:
>>
>> I guess at the end of the day, shipping a release with a bunch of awesome 
>> features is better than holding it back.  If there's 2 big releases in 6 
>> months the community isn't any worse off.
>>
>> We either ship something, or nothing, and something is probably better.
>>
>> Jon
>>
>>
>> On 2023/10/24 16:27:04 Patrick McFadin wrote:
>>
>> +1 to what you are saying, Josh. Based on the last survey, yes, everyone
>>
>> was excited about Accord, but SAI and UCS were pretty high on the list.
>>
>>
>> Benedict and I had a good conversation last night, and now I understand
>>
>> more essential details for this conversation. TCM is taking far more work
>>
>> than initially scoped, and Accord depends on a stable TCM. TCM is months
>>
>> behind and that's a critical fact, and one I personally just learned of. I
>>
>> thought things were wrapping up this month, and we were in the testing
>>
>> phase. I get why that's a topic we are dancing around. Nobody wants to say
>>
>> ship dates are slipping because that's part of our culture. It's
>>
>> disappointing and, if new information, an unwelcome surprise, but none of
>>
>> us should be angry or in a blamey mood because I guarantee every one of us
>>
>> has shipped the code late. My reaction yesterday was based on an incorrect
>>
>> assumption. Now that I have a better picture, my point of view is changing.
>>
>>
>> Josh's point about what's best for users is crucial. Users deserve stable
>>
>> code with a regular cadence of features that make their lives easier. If we
>>
>> put 5.0 on hold for TCM + Accord, users will get neither for a very long
>>
>> time. And I mentioned a disaster yesterday. A bigger disaster would be
>>
>> shipping Accord with a major bug that causes data loss, eroding community
>>
>> trust. Accord has to be the most bulletproof of all bulletproof features.
>>
>> The pressure to ship is only going to increase and that's fertile ground
>>
>> for that sort of bug.
>>
>>
>> So, taking a step back and with a clearer picture, I support the 5.0 + 

Re: [EXTERNAL] Re: [DISCUSS] Add JVector as a dependency for CEP-30

[Benedict]

[LEGAL-656] Application of Generative AI policy to dependencies - ASF JIRAissues.apache.orgLegal’s opinion is that this is not an acceptable workaround to the policy.On 22 Sep 2023, at 23:51, German Eichberger via dev  wrote:






+1 with taking it to legal




As anyone else I enjoy speculating about legal stuff and I think for jars you probably need possible deniability aka no paper trail that we knowingly... but that horse is out of the barn. So really interested in what legal says
🙂




If you can stomach non Java here is an alternate DiskANN implementation: 
microsoft/DiskANN: Graph-structured Indices for Scalable, Fast, Fresh and Filtered Approximate Nearest Neighbor Search (github.com)




Thanks,

German





From: Josh McKenzie 
Sent: Friday, September 22, 2023 7:43 AM
To: dev 
Subject: [EXTERNAL] Re: [DISCUSS] Add JVector as a dependency for CEP-30
 


I highly doubt liability works like that in all jurisdictions

That's a fantastic point. When speculating there, I overlooked the fact that there are literally dozens of legal jurisdictions in which this project is used and the foundation operates.


As a PMC let's take this to legal.


On Fri, Sep 22, 2023, at 9:16 AM, Jeff Jirsa wrote:

To do that, the cassandra PMC can open a legal JIRA and ask for a (durable, concrete) opinion.




On Fri, Sep 22, 2023 at 5:59 AM Benedict  wrote:






my understanding is that with the former the liability rests on the provider of the lib to ensure it's in compliance with their claims to copyright



I highly doubt liability works like that in all jurisdictions, even if it might in some. I can even think of some historic cases related to Linux where patent trolls went after users of Linux, though I’m not sure where that got
 to and I don’t remember all the details.


But anyway, none of us are lawyers and we shouldn’t be depending on this kind of analysis. At minimum we should invite legal to proffer an opinion on whether dependencies are a valid loophole to the policy.







On 22 Sep 2023, at 13:48, J. D. Jordan  wrote:





This Gen AI generated code use thread should probably be its own mailing list DISCUSS thread?  It applies to all source code we take in, and accept copyright assignment of, not to jars we depend on and not only to vector related
 code contributions.



On Sep 22, 2023, at 7:29 AM, Josh McKenzie  wrote:



So if we're going to chat about GenAI on this thread here, 2 things:

A dependency we pull in != a code contribution (I am not a lawyer but my understanding is that with the former the liability rests on the provider of the lib to ensure it's in compliance with their claims to copyright and it's not
 sticky). Easier to transition to a different dep if there's something API compatible or similar.
With code contributions we take in, we take on some exposure in terms of copyright and infringement. git revert can be painful.

For this thread, here's an excerpt from the ASF policy:


a recommended practice when using generative AI tooling is to use tools with features that identify any included content that is similar to parts of the tool’s training data, as well as the license
 of that content.

Given the above, code generated in whole or in part using AI can be contributed if the contributor ensures that:


The terms and conditions of the generative AI tool do not place
 any restrictions on use of the output that would be inconsistent with the Open Source Definition (e.g., ChatGPT’s terms are inconsistent).

At least one of the following conditions
 is met:


The output is not copyrightable subject matter (and would not
 be even if produced by a human)

No third party materials are included in the output

Any third party materials that are included in the output are
 being used with permission (e.g., under a compatible open source license) of the third party copyright holders and in compliance with the applicable license terms


A contributor obtain reasonable
 certainty that conditions 2.2 or 2.3 are met if the AI tool itself provides sufficient information about materials that may have been copied, or from code scanning results


E.g. AWS CodeWhisperer recently added a feature that provides
 notice and attribution



When providing contributions authored using generative AI tooling, a recommended practice is for contributors to indicate the tooling used to create the contribution. This should be included as a token
 in the source control commit message, for example including the phrase “Generated-by



I think the real challenge right now is ensuring that the output from an LLM doesn't include a string of tokens that's identical to something in its input training dataset if it's trained on non-permissively licensed inputs. That
 plus the risk of, at least in the US, the courts landing on the side of saying that not only is the output of generative AI not copyrightable, but that there's legal liability on either the

Re: [DISCUSS] Backport CASSANDRA-18816 to 5.0? Add support for repair coordinator to retry messages that timeout

[David Capwell]

I sat down to add IR messages to the mix… given how positive the feedback was 
for other repair messages I assume people are still ok with this new IR work 
going to 5.0 as well, if not please let me know here (will send a patch 
tomorrow).

Once I send out the patch 100% of repair messages have retry logic

> On Sep 26, 2023, at 12:08 PM, David Capwell  wrote:
> 
> Thanks all for the feedback!  The patch has 2 +1s on trunk and back ported to 
> 5.0, making sure it’s stable now; I plan to merge early this week.
> 
>> On Sep 21, 2023, at 2:07 PM, Ekaterina Dimitrova  
>> wrote:
>> 
>> +1 from me too. Moreover, this work has started as part of the test efforts 
>> and identifying weak points during the 4.0 testing, if I recall correctly. 
>> 5.0 sounds like a good place to land. Thank you David and everyone else 
>> involved for your efforts!
>> 
>> On Thu, 21 Sep 2023 at 1:01, Berenguer Blasi > > wrote:
>>> +1 I agree with Brandon. It's more like a bug imo.
>>> On 20/9/23 21:42, Caleb Rackliffe wrote:
 +1 on a 5.0 backport
 
 On Wed, Sep 20, 2023 at 2:26 PM Brandon Williams >>> > wrote:
> I think it could be argued that not retrying messages is a bug, I am
> +1 on including this in 5.0.
> 
> Kind Regards,
> Brandon
> 
> On Tue, Sep 19, 2023 at 1:16 PM David Capwell  > wrote:
> >
> > To try to get repair more stable, I added optional retry logic (patch 
> > is still in review) to a handful of critical repair verbs.  This patch 
> > is disabled by default but allows you to opt-in to retries so ephemeral 
> > issues don’t cause a repair to fail after running for a long time 
> > (assuming they resolve within the retry window). There are 2 protocol 
> > level changes to enable this: VALIDATION_RSP and SYNC_RSP now send an 
> > ACK (if the sender doesn’t attach a callback, these ACKs get ignored in 
> > all versions; see org.apache.cassandra.net 
> > .ResponseVerbHandler#doVerb and 
> > Verb.REPAIR_RSP).  Given that we have already forked, I believe we 
> > would need to give a waiver to allow this patch due to this change.
> >
> > The patch was written on trunk, but figured back porting 5.0 would be 
> > rather trivial and this was brought up during the review, so floating 
> > this to a wider audience.
> >
> > If you look at the patch you will see that it is very large, but this 
> > is only to make testing of repair coordination easier and 
> > deterministic, the biggest code changes are:
> >
> > 1) Moving from ActiveRepairService.instance to 
> > ActiveRepairService.instance() (this is the main reason so many files 
> > were touched; this was needed so unit tests don’t load the whole world)
> > 2) Repair no longer reaches into global space and instead is provided 
> > the subsystems needed to perform repair; this change is local to repair 
> > code
> >
> > Both of these changes were only for testing as they allow us to 
> > simulate 1k repairs in around 15 seconds with 100% deterministic 
> > execution.
> 

Re: CASSANDRA-18941 produce size bounded SSTables from CQLSSTableWriter

[guo Maxwell]

😄

Chris Lohfink  于2023年10月25日周三 05:02写道：

> +1
>
> On Tue, Oct 24, 2023 at 11:24 AM Brandon Williams 
> wrote:
>
>> +1
>>
>> Kind Regards,
>> Brandon
>>
>> On Mon, Oct 23, 2023 at 6:22 PM Yifan Cai  wrote:
>> >
>> > Hi,
>> >
>> > I want to propose merging the patch in CASSANDRA-18941 to 4.0 and up to
>> trunk and hope we are all OK with it.
>> >
>> > In CASSANDRA-18941, I am adding the capability to produce size-bounded
>> SSTables in CQLSSTableWriter for sorted data. It can greatly benefit
>> Cassandra Analytics (https://github.com/apache/cassandra-analytics) for
>> bulk writing SSTables, since it avoids buffering and sorting on flush,
>> given the data source is sorted already in the bulk write process.
>> Cassandra Analytics supports Cassandra 4.0 and depends on the cassandra-all
>> 4.0.x library. Therefore, we are mostly interested in using the new
>> capability in 4.0.
>> >
>> > CQLSSTableWriter is only used in offline tools and never in the code
>> path of Cassandra server.
>> >
>> > Any objections to merging the patch to 4.0 and up to trunk?
>> >
>> > - Yifan
>>
>

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Berenguer Blasi]

I also think there's many good new features in 5.0 already they'd make a 
good release even on their own. My 2 cts.


On 24/10/23 23:20, Brandon Williams wrote:

The catch here is that we don't publish docker images currently.  The
C* docker images available are not made by us.

Kind Regards,
Brandon

On Tue, Oct 24, 2023 at 3:31 PM Patrick McFadin  wrote:

Let me make that really easy. Hell yes

Not everybody runs CCM, I've tried but I've met resistance.

Compiling your own version usually involves me saying the words "Yes, ant realclean 
exists. I'm not trolling you"

docker pull  works on every OS and curates a single node experience.



On Tue, Oct 24, 2023 at 12:37 PM Josh McKenzie  wrote:

In order for the project to advertise the release outside the dev@ list it 
needs to be a formal release.

That's my reading as well:
https://www.apache.org/legal/release-policy.html#release-definition

I wonder if there'd be value in us having a cronned job that'd do nightly 
docker container builds on trunk + feature branches, archived for N days, and 
we make that generally known to the dev@ list here so folks that want to poke 
at the current state of trunk or other branches could do so with very low 
friction. We'd probably see more engagement on feature branches if it was 
turn-key easy for other C* devs to spin the up and check them out.

For what you're talking about here Patrick (a docker image for folks outside 
the dev@ audience and more user-facing), we'd want to vote on it and go through 
the formal process.

On Tue, Oct 24, 2023, at 3:10 PM, Jeremiah Jordan wrote:

In order for the project to advertise the release outside the dev@ list it 
needs to be a formal release.  That just means that there was a release vote 
and at least 3 PMC members +1’ed it, and there are more +1 than there are -1, 
and we follow all the normal release rules.  The ASF release process doesn’t 
care what branch you cut the artifacts from or what version you call it.

So the project can cut artifacts for and release a 5.1-alpha1, 
5.1-dev-preview1, what ever we want to version this thing, from trunk or any 
other branch name we want.

-Jeremiah

On Oct 24, 2023 at 2:03:41 PM, Patrick McFadin  wrote:

I would like to have something for developers to use ASAP to try the Accord 
syntax. Very few people have seen it, and I think there's a learning curve we 
can start earlier.

It's my understanding that ASF policy is that it needs to be a project release 
to create a docker image.

On Tue, Oct 24, 2023 at 11:54 AM Jeremiah Jordan  
wrote:

If we decide to go the route of not merging TCM to the 5.0 branch.  Do we 
actually need to immediately cut a 5.1 branch?  Can we work on stabilizing 
things while it is in trunk and cut the 5.1 branch when we actually think we 
are near releasing?  I don’t see any reason we can not cut “preview” artifacts 
from trunk?

-Jeremiah

On Oct 24, 2023 at 11:54:25 AM, Jon Haddad  wrote:

I guess at the end of the day, shipping a release with a bunch of awesome 
features is better than holding it back.  If there's 2 big releases in 6 months 
the community isn't any worse off.

We either ship something, or nothing, and something is probably better.

Jon


On 2023/10/24 16:27:04 Patrick McFadin wrote:

+1 to what you are saying, Josh. Based on the last survey, yes, everyone

was excited about Accord, but SAI and UCS were pretty high on the list.


Benedict and I had a good conversation last night, and now I understand

more essential details for this conversation. TCM is taking far more work

than initially scoped, and Accord depends on a stable TCM. TCM is months

behind and that's a critical fact, and one I personally just learned of. I

thought things were wrapping up this month, and we were in the testing

phase. I get why that's a topic we are dancing around. Nobody wants to say

ship dates are slipping because that's part of our culture. It's

disappointing and, if new information, an unwelcome surprise, but none of

us should be angry or in a blamey mood because I guarantee every one of us

has shipped the code late. My reaction yesterday was based on an incorrect

assumption. Now that I have a better picture, my point of view is changing.


Josh's point about what's best for users is crucial. Users deserve stable

code with a regular cadence of features that make their lives easier. If we

put 5.0 on hold for TCM + Accord, users will get neither for a very long

time. And I mentioned a disaster yesterday. A bigger disaster would be

shipping Accord with a major bug that causes data loss, eroding community

trust. Accord has to be the most bulletproof of all bulletproof features.

The pressure to ship is only going to increase and that's fertile ground

for that sort of bug.


So, taking a step back and with a clearer picture, I support the 5.0 + 5.1

plan mainly because I don't think 5.1 is (or should be) a fast follow.


For the user community, the communication should be straightforward. TCM +