Re: CI's working, and repeatable !!
On Mon, 29 Apr 2024 at 04:08, Josh McKenzie wrote: > I want to offer a heartfelt thanks to all involved for the focus and > energy that went into this! > … I also think there'd be significant value in more of us moving towards > using the JenkinsFile where at all possible. > Appreciated Josh. Some small details to add… We now cache our docker images in apache.jfrog.org to get around dockerhub pull rate limits. The images for dockerfiles found under .build/docker/ and the alpine image we use for utility are found there, just prefix their image name with " apache.jfrog.io/cassan-docker/" e.g. "apache.jfrog.io/cassan-docker/alpine::3.19.1" Automating deployment (build+push) of these images is CASSANDRA-18931 The in-tree scripts for building and testing, found under .build/, are a pile of bash poo. (Bash translates to shit in Norwegian, yes). They are a residual of their incarnation in cassandra-builds. It would be awesome to rewrite the bulk of them to something cleaner, and it is now possible to do so, because all changes to .build/ are part of the CI you run. (Keep in mind CASSANDRA-18942 is in-flight.)
Re: discuss: add to_human_size function
FYI, I added both to_human_size and to_human_duration (1), (2). I try my luck with asking for a reviewer. It is also tested / documented etc. (1) https://github.com/apache/cassandra/pull/3239/files (2) https://github.com/apache/cassandra/blob/f35ed228145fae3edb4325d29464f0d950d13511/doc/modules/cassandra/pages/developing/cql/functions.adoc#human-helper-functions On Thu, Apr 25, 2024 at 6:20 PM Ekaterina Dimitrova wrote: > All I say is we should be careful not to open the door for someone to be > able to set for a parameter in cassandra.yaml 512MiB and convert it to 0 > GiB internally while changing those classes. Loss of precision and weird > settings. As long as that pandora box stays closed, all good 👍🏻 > > I do support this new function addition proposed here, thank you! > > On Thu, 25 Apr 2024 at 7:31, Jon Haddad wrote: > >> I can’t see a good reason not to support it. Seems like extra work to >> avoid with no benefit. >> >> — >> >> Jon Haddad >> Rustyrazorblade Consulting >> rustyrazorblade.com >> >> >> On Thu, Apr 25, 2024 at 7:16 AM Štefan Miklošovič < >> stefan.mikloso...@gmail.com> wrote: >> >>> Can you elaborate on intentionally not supporting some conversions? Are >>> we safe to base these conversions on DataStorageUnit? We have set of units >>> from BYTES to GIBIBYTES and respective methods on them which convert from >>> that unit to whatever else. Is this OK to be used for the purposes of this >>> feature? I would expect that once we have units like these and methods on >>> them to convert from-to, it can be reused in wherever else. >>> >>> On Thu, Apr 25, 2024 at 4:06 PM Ekaterina Dimitrova < >>> e.dimitr...@gmail.com> wrote: >>> All I am saying is be careful with adding those conversions not to end up used while setting our configuration. Thanks 🙏 On Thu, 25 Apr 2024 at 6:53, Štefan Miklošovič < stefan.mikloso...@gmail.com> wrote: > Well, technically I do not need DataStorageSpec at all. All I need is > DataStorageUnit for that matter. That can convert from one unit to another > easily. > > We can omit tebibytes, that's just fine. People would need to live > with gibibytes at most in cqlsh output. They would not get 5 TiB but 5120 > GiB, I guess that is just enough to have a picture of what magnitude that > value looks like. > > On Thu, Apr 25, 2024 at 3:36 PM Ekaterina Dimitrova < > e.dimitr...@gmail.com> wrote: > >> Quick comment: >> >> DataRateSpec, DataStorageSpec, or DurationSpec >> - we intentionally do not support going smaller to bigger size in >> those classes which are specific for cassandra.yaml - precision issues. >> Please keep it that way. That is why the notion of min unit was added in >> cassandra.yaml for parameters that are internally represented in a bigger >> unit. >> >> I am not sure that people want to add TiB. There was explicit >> agreement what units we will allow in cassandra.yaml. I suspect any new >> units should be approved on the ML >> >> Hope this helps >> >> >> >> On Thu, 25 Apr 2024 at 5:55, Claude Warren, Jr via dev < >> dev@cassandra.apache.org> wrote: >> >>> TiB is not yet in DataStorageSpec (perhaps we should add it). >>> >>> A quick review tells me that all the units are unique across the 3 >>> specs. As long as we guarantee that in the future the method you >>> propose >>> should be easily expandable to the other specs. >>> >>> +1 to this idea. >>> >>> On Thu, Apr 25, 2024 at 12:26 PM Štefan Miklošovič < >>> stefan.mikloso...@gmail.com> wrote: >>> That is a very interesting point, Claude. My so-far implementation is using FileUtils.stringifyFileSize which is just dividing a value by a respective divisor based on how big a value is. While this works, it will prevent you from specifying what unit you want that value to be converted to as well as it will prevent you from specifying what unit a value you provided is of. So, for example, if a column is known to be in kibibytes and we want that to be converted into gibibytes, that won't be possible because that function will think that a value is in bytes. It would be more appropriate to have something like this: to_human_size(val) -> alias to FileUtils.stringifyFileSize, without any source nor target unit, it will consider it to be in bytes and it will convert it like in FileUtils.stringifyFileSize to_human_size(val, 'MiB') -> alias for to_human_size(val, 'B', 'MiB') to_human_size(val, 'GiB') -> alias for to_human_size(val, 'B', 'GiB') the first argument is the source unit, the second argument is target unit to_human_size(val, 'B', 'MiB
Re: [DISCUSS] CEP-40: Data Transfer Using Cassandra Sidecar for Live Migrating Instances
I just wanted to chime in here about a recently introduced feature in Sidecar. Sidecar has now the ability to restore data from S3 (although the restores are for bulk write jobs coming from the Cassandra Analytics library). The ability to restore data from S3 (or any other cloud provider) can be extended for other use cases such as live migration, backup and restore, and others. On 2024/04/25 17:57:16 Venkata Hari Krishna Nukala wrote: > I have updated the CEP to use binary level file digest verification. > > In the next iteration, I am going to address the below point. > > I would like to see more abstraction of how the files get moved / put in > place with the proposed solution being the default implementation. That > would allow others to plug in alternatives means of data movement like > pulling down backups from S3 or rsync, etc. > > Thanks! > Hari > > On Wed, Apr 24, 2024 at 1:24 AM Patrick McFadin wrote: > > > I finally got a chance to digest this CEP and am happy to see it raised. > > This feature has been left to the end user for far too long. > > > > It might get roasted for scope creep, but here goes. Related and something > > that I've heard for years is the ability to migrate a single keyspace away > > from a set of hardware... online. Similar problem but a lot more > > coordination. > > - Create a Keyspace in Cluster B mimicking keyspace in Cluster A > > - Establish replication between keyspaces and sync schema > > - Move data from Cluster A to B > > - Decommission keyspace in Cluster A > > > > In many cases, multiple tenants present cause the cluster to overpressure. > > The best solution in that case is to migrate the largest keyspace to a > > dedicated cluster. > > > > Live migration but a bit more complicated. No chance of doing this > > manually without some serious brain surgery on c* and downtime. > > > > Patrick > > > > > > On Tue, Apr 23, 2024 at 11:37 AM Venkata Hari Krishna Nukala < > > n.v.harikrishna.apa...@gmail.com> wrote: > > > >> Thank you all for the inputs and apologies for the late reply. I see good > >> points raised in this discussion. *Please allow me to reply to each > >> point individually.* > >> > >> To start with, let me focus on the point raised by Scott & Jon about file > >> content verification at the destination with the source in this reply. > >> Agree that just verifying the file name + size is not fool proof. The > >> reason why I called out binary level verification out of initial scope is > >> because of these two reasons: 1) Calculating digest for each file may > >> increase CPU utilisation and 2) Disk would also be under pressure as > >> complete disk content will also be read to calculate digest. As called out > >> in the discussion, I think we can't compromise on binary level check for > >> these two reasons. Let me update the CEP to include binary level > >> verification. During implementation, it can probably be made optional so > >> that it can be skipped if someone doesn't want it. > >> > >> Thanks! > >> Hari > >> > >> On Mon, Apr 22, 2024 at 4:40 AM Slater, Ben via dev < > >> dev@cassandra.apache.org> wrote: > >> > >>> We use backup/restore for our implementation of this concept. It has the > >>> added benefit that the backup / restore path gets exercised much more > >>> regularly than it would in normal operations, finding edge case bugs at a > >>> time when you still have other ways of recovering rather than in a full > >>> disaster scenario. > >>> > >>> > >>> > >>> Cheers > >>> > >>> Ben > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> *From: *Jordan West > >>> *Date: *Sunday, 21 April 2024 at 05:38 > >>> *To: *dev@cassandra.apache.org > >>> *Subject: *Re: [DISCUSS] CEP-40: Data Transfer Using Cassandra Sidecar > >>> for Live Migrating Instances > >>> > >>> *EXTERNAL EMAIL - USE CAUTION when clicking links or attachments * > >>> > >>> > >>> > >>> I do really like the framing of replacing a node is restoring a node and > >>> then kicking off a replace. That is effectively what we do internally. > >>> > >>> > >>> > >>> I also agree we should be able to do data movement well both internal to > >>> Cassandra and externally for a variety of reasons. > >>> > >>> > >>> > >>> We’ve seen great performance with “ZCS+TLS” even though it’s not full > >>> zero copy — nodes that previously took *days* to replace now take a few > >>> hours. But we have seen it put pressure on nodes and drive up latencies > >>> which is the main reason we still rely on an external data movement system > >>> by default — falling back to ZCS+TLS as needed. > >>> > >>> > >>> > >>> Jordan > >>> > >>> > >>> > >>> On Fri, Apr 19, 2024 at 19:15 Jon Haddad wrote: > >>> > >>> Jeff, this is probably the best explanation and justification of the > >>> idea that I've heard so far. > >>> > >>> > >>> > >>> I like it because > >>> > >>> > >>> > >>> 1) we really should have something official for backups > >>> > >>> 2) backups / object store would be great for analyti
Re: [DISCUSS] CEP-40: Data Transfer Using Cassandra Sidecar for Live Migrating Instances
On Tue, Apr 23, 2024 at 11:37 AM Venkata Hari Krishna Nukala < n.v.harikrishna.apa...@gmail.com> wrote: > reason why I called out binary level verification out of initial scope is > because of these two reasons: 1) Calculating digest for each file may > increase CPU utilisation and 2) Disk would also be under pressure as > complete disk content will also be read to calculate digest. As called out > in the discussion, I think we can't > We should have a digest / checksum for each of the file components computed and stored on disk so this doesn't need to be recomputed each time. Most files / components are immutable and therefore their checksum won't change. There are some components which may be mutated and therefore their checksum may need to be recomputed. However, data integrity is not something we can compromise on. On the receiving node, CPU utilization is not a big issue as that node isn't servicing traffic. I was too lazy to dig into the code and someone who is more familiar with the SSTable components / file format can help shed light on checksums.
Re: [DISCUSS] Donating easy-cass-stress to the project
> So: besides Jon, who in the community expects/desires to maintain this going > forward? I have been maintaining a fork for years, so don’t mind helping maintain this project. > On Apr 28, 2024, at 4:08 AM, Mick Semb Wever wrote: > >> A separate subproject like dtest and the Java driver would maybe help >> address concerns with introducing a gradle build system and Kotlin. > > > > Nit, dtest is a separate repository, not a subproject. The Java driver is > one repository to be in the Drivers subproject. Esoteric maybe, but ASF > terminology we need to get right :-) > > To your actual point (IIUC), it can be a separate repository and not a > separate subproject. This permits it to be kotlin+gradle, while not having > the formal subproject procedures. It still needs 3 responsible committers > from the get-go to show sustainability. Would easy-cass-stress have > releases, or always be a codebase users work directly with ? > > Can/Should we first demote cassandra-stress by moving it out to a separate > repo ? > ( Can its imports work off non-snapshot dependencies ? ) > It might feel like an extra prerequisite step to introduce, but maybe it > helps move the needle forward and make this conversation a bit easier/obvious. >
Re: CI's working, and repeatable !!
💯! Amazing work - thanks so much for posting the details, Mick, and Josh
is right on. Kinda bummed I haven't been following C* CI dev, being more on
the ops side lately. Posting this up has me intrigued, so I may just have
to go poke around some and scratch an itch :)
Warm regards,
Michael
On Sun, Apr 28, 2024 at 9:08 PM Josh McKenzie wrote:
> A huge amount of work and time went into this and it's going to have a big
> impact on the project. I want to offer a heartfelt thanks to all involved
> for the focus and energy that went into this!
>
> As the author of the system David lovingly dubbed "JoshCI" (/sigh), I
> definitely want to see us all move to converge as much as possible on the
> CI code we're running. While I remain convinced something like
> CASSANDRA-18731 is vital for hygiene in the long run (unit testing our CI,
> declaratively defining atoms of build logic independently from flow), I
> also think there'd be significant value in more of us moving towards using
> the JenkinsFile where at all possible.
>
> Seriously - thanks again for all this work everyone. CI on Cassandra is a
> Big Data Problem, and not an easy one.
>
> On Sun, Apr 28, 2024, at 10:22 AM, Mick Semb Wever wrote:
>
>
> Good news.
>
> CI on 5.0 and trunk is working again, after an unexpected 6 weeks
> hiatus (and a string of general problems since last year).
> This includes pre-commit for 5.0 and trunk working again.
>
>
> More info…
>
> From 5.0 we now have in-tree a Jenkinsfile that only relies on the in-tree
> scripts – it does not depend upon cassandra-builds and all the individual
> dsl created stage jobs. This aligns how pre-commit and post-commit works.
> More importantly, it makes our CI repeatable regardless of the fork/branch
> of the code, or the jenkins installation.
>
> For 5.0+ pre-commit use the Cassandra-devbranch-5 and make sure your patch
> is after sha 3c85def
> The jenkinsfile now comes with pre-defined profiles, it's recommended to
> use "skinny" until you need the final "pre-commit". You can also use the
> custom profile with a regexp when you need just specific test types.
> See https://ci-cassandra.apache.org/job/Cassandra-devbranch-5/build
>
> For pre-commit on older branches, you now use Cassandra-devbranch-before-5
>
> For both pre- and post-commit builds, each build now creates two new
> sharable artefacts: ci_summary.html and results_details.tar.xz
> These are based on what apple contributors were sharing from builds from
> their internal CI system. The format and contents of these files is
> expected to evolve.
>
> Each build now archives its results and logs all under one location in
> nightlies.
>
> e.g. https://nightlies.apache.org/cassandra/Cassandra-5.0/227/
>
>
>
> The post-commit pipeline profile remains *very* heavy, at 130k+ tests.
> These were previously ramped up to include everything in their pipelines,
> given everything that's happening in both branches. So they take time and
> saturate everything they touch. We need to re-evaluate what we need to be
> testing to alleviate this. There'll also be a new pattern of timeouts and
> infra/script -related flakies, as happens whenever there's such a
> significant change, all the patience and help possible is appreciated!
>
>
>
> Now that the jenkinsfile can now be used on any jenkins server for any
> fork/branch, the next work-in-progress is CASSANDRA-18145, to be able to
> run the full pipeline with a single command line (given a k8s context
> (~/.kube/config)).
>
> We already have most of this working – it's possible to make a clone
> ci-cassandra.apache.org on k8s using this wip helm chart:
> https://github.com/thelastpickle/Cassius
> And we are already using this on an auto-scaling gke k8s cluster – you
> might have seen me posting the ci_summary.html and results_details.tar.xz
> files to tickets for pre-commit CI instead of using the ci-cassandra.a.o or
> circleci pre-commit liks. Already, we have a full pipeline time down to
> two hours and less than a third of the cost of CircleCI, and there's lhf to
> further improve this. For serious pre-commit testing we are still missing
> and need repeatable test runs, ref CASSANDRA-18942. On all this I'd like
> to give a special shout out to Aleksandr Volochnev who was instrumental in
> the final (and helm based) work of 18145 which was needed to be able to
> test its prerequisite ticket CASSANDRA-18594 – ci-cassandra.a.o would not
> be running again today without his recent time spent on it.
>
> On a separate note, this new jenkinsfile is designed in preparation for
> CASSANDRA-18731 ('Add declarative root CI structure'), to make it easier to
> define profiles, tests, and their infrastructural requirements.
>
>
> To the community…
> We are now in a place where we are looking and requesting further
> donations of servers to the ci-cassandra.apache.org jenkins cluster. We
> can now also use cloud/instance credits to host auto-scaling k8s-based
> ci-cassandra.a.o clones that would b
