upgrade sstable selection

[Claude Warren, Jr via dev]

Greetings,

I am working on the downgradesstables code and seem to have a problem with
ordering of the downgrade or perhaps the Directories.SSTableLister

I lifted the code from upgradesstables to select the files to downgrade.
The only difference in the code that selects the files to downgrade is the
actual selection of the file.  There is no change to the ordering of the
files that are evaluated for inclusion.  Yet I think the downgrade ordering
is incorrect.

My process is to start 3.1 version to create the tables and then use the
4.0 code base to run the standaloneupgrader and then the
standalonedowngrader

When running the standaloneupgrader on system local I see the following
{{noformat}}
Found 3 sstables that need upgrading.
Upgrading
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-1-big-Data.db')
Upgrade of
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-1-big-Data.db')
complete.
Upgrading
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-2-big-Data.db')
Upgrade of
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-2-big-Data.db')
complete.
Upgrading
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-3-big-Data.db')
Upgrade of
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-3-big-Data.db')
complete.
{{noformat}}

when running the standalonedowngrader is see
{{noformat}}
Found 3 sstables that need downgrading.
Downgrading
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-6-big-Data.db')
Downgrade of
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-6-big-Data.db')
complete.
Downgrading
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db')
Downgrade of
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db')
complete.
Downgrading
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db')
Downgrade of
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db')
complete.
{{noformat}}}

Note the order of the generations in the downgrader (I have seen similar
out of order issues with the upgrader, but infrequently)

The difference between the upgrader and downgrader code in the questionable
section (
https://github.com/Claudenw/cassandra/blob/CASSANDRA-8928/src/java/org/apache/cassandra/tools/StandaloneDowngrader.java#:~:text=new%20ArrayList%3C%3E()%3B-,//%20Downgrade%20sstables,%7D,-int%20numSSTables%20%3D)
is on line 101 where the files are selected and put into a list.  I think
this means that the Directories.SSTableLister on occasion returns files in
the incorrect order during a call to lister.list().entrySet()

I believe that the files are processed in the order specified and that the
generations get switched around.  This is evidenced by the file size of the
Data file associated with the generations as it moves through the process.
In this case we expect the nb-6 to become ma-7 as per the output from the
run.  In actuality we want nb-6 to be nb-9.

{{noformat}}
  ma   nb  ma
1  212 4  2128  212
2   64 5   709   70
3 4876 6 48837 4883
{{noformat}}

So now my question, has anyone seen the behaviour before?

Oh, to make things more interesting I am using Docker images of 3.1 and a
modified 4.0 that turns off the execution so I can just run the upgrade and
downgrade.

Any help would be appreciated,
Claude

Re: upgrade sstable selection

[Brandon Williams]

> I think this means that the Directories.SSTableLister on occasion returns 
> files in the incorrect order during a call to lister.list().entrySet()

This seems easy enough to verify by looping it and examining the results.

Kind Regards,
Brandon

On Tue, Jan 10, 2023 at 4:44 AM Claude Warren, Jr via dev
 wrote:
>
> Greetings,
>
> I am working on the downgradesstables code and seem to have a problem with 
> ordering of the downgrade or perhaps the Directories.SSTableLister
>
> I lifted the code from upgradesstables to select the files to downgrade.  The 
> only difference in the code that selects the files to downgrade is the actual 
> selection of the file.  There is no change to the ordering of the files that 
> are evaluated for inclusion.  Yet I think the downgrade ordering is incorrect.
>
> My process is to start 3.1 version to create the tables and then use the 4.0 
> code base to run the standaloneupgrader and then the standalonedowngrader
>
> When running the standaloneupgrader on system local I see the following
> {{noformat}}
> Found 3 sstables that need upgrading.
> Upgrading 
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-1-big-Data.db')
> Upgrade of 
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-1-big-Data.db')
>  complete.
> Upgrading 
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-2-big-Data.db')
> Upgrade of 
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-2-big-Data.db')
>  complete.
> Upgrading 
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-3-big-Data.db')
> Upgrade of 
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-3-big-Data.db')
>  complete.
> {{noformat}}
>
> when running the standalonedowngrader is see
> {{noformat}}
> Found 3 sstables that need downgrading.
> Downgrading 
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-6-big-Data.db')
> Downgrade of 
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-6-big-Data.db')
>  complete.
> Downgrading 
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db')
> Downgrade of 
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db')
>  complete.
> Downgrading 
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db')
> Downgrade of 
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db')
>  complete.
> {{noformat}}}
>
> Note the order of the generations in the downgrader (I have seen similar out 
> of order issues with the upgrader, but infrequently)
>
> The difference between the upgrader and downgrader code in the questionable 
> section 
> (https://github.com/Claudenw/cassandra/blob/CASSANDRA-8928/src/java/org/apache/cassandra/tools/StandaloneDowngrader.java#:~:text=new%20ArrayList%3C%3E()%3B-,//%20Downgrade%20sstables,%7D,-int%20numSSTables%20%3D)
>  is on line 101 where the files are selected and put into a list.  I think 
> this means that the Directories.SSTableLister on occasion returns files in 
> the incorrect order during a call to lister.list().entrySet()
>
> I believe that the files are processed in the order specified and that the 
> generations get switched around.  This is evidenced by the file size of the 
> Data file associated with the generations as it moves through the process.  
> In this case we expect the nb-6 to become ma-7 as per the output from the 
> run.  In actuality we want nb-6 to be nb-9.
>
> {{noformat}}
>   ma   nb  ma
> 1  212 4  2128  212
> 2   64 5   709   70
> 3 4876 6 48837 4883
> {{noformat}}
>
> So now my question, has anyone seen the behaviour before?
>
> Oh, to make things more interesting I am using Docker images of 3.1 and a 
> modified 4.0 that turns off the execution so I can just run the upgrade and 
> downgrade.
>
> Any help would be appreciated,
> Claude

Re: upgrade sstable selection

[Claude Warren, Jr via dev]

Actually since the Directories.SSTableLister stores the Components in a
HashMap indexed by the Descriptor.  Since the upgrade/downgrade code
retrieves the list in hash order there is no guarantee that they will be in
order.  I suspect that this is a bug.

On Tue, Jan 10, 2023 at 12:34 PM Brandon Williams  wrote:

> > I think this means that the Directories.SSTableLister on occasion
> returns files in the incorrect order during a call to
> lister.list().entrySet()
>
> This seems easy enough to verify by looping it and examining the results.
>
> Kind Regards,
> Brandon
>
> On Tue, Jan 10, 2023 at 4:44 AM Claude Warren, Jr via dev
>  wrote:
> >
> > Greetings,
> >
> > I am working on the downgradesstables code and seem to have a problem
> with ordering of the downgrade or perhaps the Directories.SSTableLister
> >
> > I lifted the code from upgradesstables to select the files to
> downgrade.  The only difference in the code that selects the files to
> downgrade is the actual selection of the file.  There is no change to the
> ordering of the files that are evaluated for inclusion.  Yet I think the
> downgrade ordering is incorrect.
> >
> > My process is to start 3.1 version to create the tables and then use the
> 4.0 code base to run the standaloneupgrader and then the
> standalonedowngrader
> >
> > When running the standaloneupgrader on system local I see the following
> > {{noformat}}
> > Found 3 sstables that need upgrading.
> > Upgrading
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-1-big-Data.db')
> > Upgrade of
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-1-big-Data.db')
> complete.
> > Upgrading
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-2-big-Data.db')
> > Upgrade of
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-2-big-Data.db')
> complete.
> > Upgrading
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-3-big-Data.db')
> > Upgrade of
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-3-big-Data.db')
> complete.
> > {{noformat}}
> >
> > when running the standalonedowngrader is see
> > {{noformat}}
> > Found 3 sstables that need downgrading.
> > Downgrading
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-6-big-Data.db')
> > Downgrade of
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-6-big-Data.db')
> complete.
> > Downgrading
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db')
> > Downgrade of
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db')
> complete.
> > Downgrading
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db')
> > Downgrade of
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db')
> complete.
> > {{noformat}}}
> >
> > Note the order of the generations in the downgrader (I have seen similar
> out of order issues with the upgrader, but infrequently)
> >
> > The difference between the upgrader and downgrader code in the
> questionable section (
> https://github.com/Claudenw/cassandra/blob/CASSANDRA-8928/src/java/org/apache/cassandra/tools/StandaloneDowngrader.java#:~:text=new%20ArrayList%3C%3E()%3B-,//%20Downgrade%20sstables,%7D,-int%20numSSTables%20%3D)
> is on line 101 where the files are selected and put into a list.  I think
> this means that the Directories.SSTableLister on occasion returns files in
> the incorrect order during a call to lister.list().entrySet()
> >
> > I believe that the files are processed in the order specified and that
> the generations get switched around.  This is evidenced by the file size of
> the Data file associated with the generations as it moves through the
> process.  In this case we expect the nb-6 to become ma-7 as per the output
> from the run.  In actuality we want nb-6 to be nb-9.
> >
> > {{noformat}}
> >   ma   nb  ma
> > 1  212 4  2128  212
> > 2   64 5   709   70
> > 3 4876 6 48837 4883
> > {{noformat}}
> >
> > So now my question, has anyone seen the behaviour before?
> >
> > Oh, to make things more interesting I am using Docker images of 3.1 and
> a modified 4.0 that turns off the execution so I can just run the upgrade
> and downgrade.
> >
> > Any help would be appreciated,
> > Claude
>

Re: [DISCUSS] Taking another(other(other)) stab at performance testing

[Henrik Ingo]

Since I cited several papers in my essay below, I might as well add the
latest one, which describes our use of automatic change point detection
inside Datastax. We've indirectly been testing Cassandra 4.0 already over a
year with this method, as we use change detection against an internal fork
of 4.0.

https://arxiv.org/abs/2301.03034

henrik

On Sun, Jan 8, 2023 at 5:12 AM Henrik Ingo  wrote:

> Hi Josh, all
>
> I'm sitting at an airport, so rather than participating in the comment
> threads in the doc, I will just post some high level principles I've
> derived during my own long career in performance testing.
>
> Infra:
>  - It's a common myth that you need to use on premise HW because cloud HW
> is noisy.
>  - Most likely the opposite is true: A small cluster of lab hardware runs
> the risk of some sysadmin with root access manually modifying the servers
> and leave them in an inconsistent configuration. Otoh a public cloud is
> configured with infrastructure as code, so every change is tracked in
> version control.
>  - Four part article on how we tuned EC2 at my previous employer: 1
> ,
> 2
> ,
> 3
> 
> , 4
> 
> .
>  - Trust no one, measure everything. For example, don't  trust that what
> I'm writing here is true. Run sysbench against your HW, then you have first
> hand observations.
>  - Specifically using EC2 has an additional benefit that the instance
> types can be considered well known and standard HW configurations more than
> any on premise system.
>
> Performance testing is regression testing
>  - Important: Run perf tests with the nightly build. Make sure your HW
> configuration is repeatable and low variability from day to day.
>  - Less important / later:
>  - Using complciated benchmarks (tpcc...) that try to model a real
> world app. These can take weeks to develop, each.
>  - Having lots of different benchmarks for "coverage".
>  - Adding the above two together: Running a simple key-value test (e.g.
> YCSB) every night in an automated and repeatable way, and storing the
> result - whatever is considered relevant - so that you end up with a
> timeseries is a great start and I'd take this over that complicated
> "representative" benchmark any day.
>  - Use change detection to automatically and deterministically flag
> statistically significant change points (regressions).
>  - Literature: detecting-performance-regressions-with-datastax-hunter
> 
> ,
>  - Literature: Fallout: Distributed Systems Testing as a Service
> 
>  Automated system performance testing at MongoDB
> 
>
>
> Common gotchas:
>  - Testing with a small data set that fits entirely in RAM. A good dataset
> is 5x the RAM available to the DB process. Or you just test with the size a
> real production server would be running - at Datastax we have tests that
> use a 1TB and 1.5TB data set, because those tend to be standard maximum
> sizes (per node) at customers.
>  - The test runtime is too short. IT depends on the database what is a
> good test duration. The goal is to reach stable state. But for an LSM
> database like Cassandra this can be hard. For other databases I worked
> with, the default is typically to flush every 15 to 60 seconds, and the
> test duration should be a multiple of those (3 to 5 min).
>  - Naive comparisons to determine whether a test result is a regression or
> not. For example benchmarking the new release against the stable version,
> one run each, and reporting the result as "fact". Or comparing today's
> result with yesterday's.
> '
>
> Building perf testing systems following the above principles have had a
> lot of positive impact in my projects. For example, at my previous employer
> we caught 17 significant regressions during the 1 year long development
> cycle of the next major version. (see my paper above)  Otoh after the GA
> release, during the next year users only reported 1 significant performance
> regression. That is to say, the perf testing of nightly builds caught all
> but one regressions in the new major version.
>
> henrik
>
>
>
>
> On Fri, Dec 30, 2022 at 7:41 AM Josh McKenzie 
> wrote:
>
>> There was a really interesting presentation from the Lucene folks at
>> ApacheCon about how they're doing perf regression testing. That combined
>> with some recent contributors wanting to get involved on some performance
>> work and not havi

Re: Cassandra CI Status 2023-01-07

[Josh McKenzie]

> I don't believe it warrants a CEP, speak up if you disagree. 
I agree with this but I'm also biased having been working w/you on this for a 
bit.

My instinct is that most folks on the project want CI that works consistently, 
quickly, and is minimally complex to modify. So the less disruptive and more 
well documented and streamlined we can make interacting with this process the 
better. 

On Mon, Jan 9, 2023, at 2:06 PM, Mick Semb Wever wrote:
>Happy 2023 everyone!
> 
> With only four months in front of us before the first 5.0 release I'm
> hoping we can re-energize our focus on CI and Stable Trunk.
> 
> This post covers the following
> * Recap of CI improvements
> * State of Affair
> * The Butler (Build Lead)
> * Proposal for a Repeatable Containerised CI
> 
> and it calls for the following actions
> ** we need you to sign up for a week's rotation as Build Lead !
> ** please reply in-thread any CI issues I've forgotten,
> ** does CASSANDRA-18137 warrant a CEP?
> 
> 
> *** Recap of CI improvements
> 
> It's been over two years since my last CI Status post, with Adam and
> Josh covering much of it in their general Status emails (which are
> deeply appreciated).  I'm hoping we can continue with both, given
> their importance to a successful 5.0 release and the debt cost we face
> otherwise going from the initial alpha release to the eventual GA.
> 
> 
> We have made good efforts on moving towards a Stable Trunk.
> Special mentions to
> - improving parity between CircleCI and ci-cassandra.a.o (CASSANDRA-17930)
> - introducing Butler and the Build Lead role
> - pre-commit workflow, and automated multiplexing, in CircleCI
> (CASSANDRA-16625)
> - single digit flaky failures per build on 4.0, 4.1 and trunk
> ci-cassandra.a.o !!
> - CircleCI is as stable on Large as XLarge containers (CASSANDRA-18127)
> 
> 
> *** State of Affair
> 
> None of our CI systems are consistently green yet.  Flakies occur in
> both CircleCI and ci-cassandra.a.o  . We had to lower the 4.1 release
> CI criteria to accept three consequential green runs on CircleCI, as
> it would have been unlikely to achieve the same on ci-cassandra.a.o.
> While the flakey rate is lower than 4.0, the higher number of tests we
> run is making it harder to get those green runs.
> 
> Despite the overhead we continue to face with flakies and getting
> major releases out, 4.1 saw fewer releases to GA than 4.0, I think all
> will agree things are improving.  But the challenge in front of us up
> to the 5.0 release is huge with nine CEPs slated to land.  Pre-commit
> and post-commit CI needs investing in if we want our stable trunk
> efforts to continue to improve.
> 
> 
> *** The Butler (Build Lead)
> 
> The introduction of Butler and the Build Lead was a wonderful
> improvement to our CI efforts.  It has brought a lot of hygiene in
> listing out flakies as they happened.  Noted that this has in-turn
> increased the burden in getting our major releases out, but that's to
> be seen as a one-off cost.  This initiative lost traction and
> volunteers mid last year.
> 
> We really need you to take part in the Build Lead weekly rotation.
> 
> I've signed myself up for this week, please jump in and sign yourself
> up for the weeks ahead.  If you are a coach/manager for a team, please
> permit and encourage your engineers to be involved in this activity,
> it shouldn't be more than an hour over the week.  Further instructions
> found at https://cwiki.apache.org/confluence/display/CASSANDRA/Build+Lead
> 
> If it's your first time being a Build Lead the community is here to
> help you, just reach out.  It's also a great way into our community
> for newcomers!
> 
> When it comes to Butler it's UX of history is a bit clumsy.  TIL that
> you can indeed list the full history of failures per test, see 'Full
> History' under a test page*.  Please use this information to help
> create jira tickets on flakies, specifically the versions it applies
> to and the rough rate of failure so far observed.
> 
> *) e.g. 
> https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-trunk/failure/snapshot_test/TestArchiveCommitlog/test_archive_commitlog_point_in_time_ln
> 
> 
> *** Proposal for a Repeatable Containerised CI
> 
> Building on what Josh writes in his "Cassandra project status, Year in
> Review Holiday Edition" post, and many discussions offline with many
> folk, I've written up the ticket epic for creating a reproducible
> containerised ci-cassandra.a.o
> 
> Please read https://issues.apache.org/jira/browse/CASSANDRA-18137
> 
> The tl;dr of it is to create a script that, using the jenkins k8s
> operator, can set up a ci-cassandra.a.o clone in your k8s context.
> 
> The ticket is lengthy, despite being in bullet form.  I don't believe
> it warrants a CEP, speak up if you disagree.  The idea is to provide
> us a turnkey solution: the jenkins k8s operator based script (create
> ci-cassandra.a.o clone, run pipeline, save results, tear down clone);
> to bring our ex