Re: upgrade sstable selection

[Jacek Lewandowski]

Hi,

It does not look like returning an unsorted map is a bug. Whenever we want
to sort sstables according to their generation id, we do that explicitly,
for example in leveled compactor.

Though, as you pointed out, the order should be retained when upgrading
sstables because otherwise things like the mentioned compactor would get
wrong order.

I suggest creating a ticket and fix the upgrader code by explicitly
ordering sstables by their id. The comparator for SSTableId is available in
SSTableIdFactory.COMPARATOR.

Thanks,
- - -- --- -  -
Jacek Lewandowski


On Tue, Jan 10, 2023 at 2:26 PM Claude Warren, Jr via dev <
dev@cassandra.apache.org> wrote:

> Actually since the Directories.SSTableLister stores the Components in a
> HashMap indexed by the Descriptor.  Since the upgrade/downgrade code
> retrieves the list in hash order there is no guarantee that they will be in
> order.  I suspect that this is a bug.
>
> On Tue, Jan 10, 2023 at 12:34 PM Brandon Williams 
> wrote:
>
>> > I think this means that the Directories.SSTableLister on occasion
>> returns files in the incorrect order during a call to
>> lister.list().entrySet()
>>
>> This seems easy enough to verify by looping it and examining the results.
>>
>> Kind Regards,
>> Brandon
>>
>> On Tue, Jan 10, 2023 at 4:44 AM Claude Warren, Jr via dev
>>  wrote:
>> >
>> > Greetings,
>> >
>> > I am working on the downgradesstables code and seem to have a problem
>> with ordering of the downgrade or perhaps the Directories.SSTableLister
>> >
>> > I lifted the code from upgradesstables to select the files to
>> downgrade.  The only difference in the code that selects the files to
>> downgrade is the actual selection of the file.  There is no change to the
>> ordering of the files that are evaluated for inclusion.  Yet I think the
>> downgrade ordering is incorrect.
>> >
>> > My process is to start 3.1 version to create the tables and then use
>> the 4.0 code base to run the standaloneupgrader and then the
>> standalonedowngrader
>> >
>> > When running the standaloneupgrader on system local I see the following
>> > {{noformat}}
>> > Found 3 sstables that need upgrading.
>> > Upgrading
>> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-1-big-Data.db')
>> > Upgrade of
>> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-1-big-Data.db')
>> complete.
>> > Upgrading
>> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-2-big-Data.db')
>> > Upgrade of
>> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-2-big-Data.db')
>> complete.
>> > Upgrading
>> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-3-big-Data.db')
>> > Upgrade of
>> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-3-big-Data.db')
>> complete.
>> > {{noformat}}
>> >
>> > when running the standalonedowngrader is see
>> > {{noformat}}
>> > Found 3 sstables that need downgrading.
>> > Downgrading
>> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-6-big-Data.db')
>> > Downgrade of
>> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-6-big-Data.db')
>> complete.
>> > Downgrading
>> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db')
>> > Downgrade of
>> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db')
>> complete.
>> > Downgrading
>> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db')
>> > Downgrade of
>> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db')
>> complete.
>> > {{noformat}}}
>> >
>> > Note the order of the generations in the downgrader (I have seen
>> similar out of order issues with the upgrader, but infrequently)
>> >
>> > The difference between the upgrader and downgrader code in the
>> questionable section (
>> https://github.com/Claudenw/cassandra/blob/CASSANDRA-8928/src/java/org/apache/cassandra/tools/StandaloneDowngrader.java#:~:text=new%20ArrayList%3C%3E()%3B-,//%20Downgrade%20sstables,%7D,-int%20numSSTables%20%3D)
>> is on line 101 where the files are selected and put into a list.  I think
>> this means that the Directories.SSTableLister on occasion returns files in
>> the incorrect order during a call to lister.list().entrySet()
>> >
>> > I believe that the files are processed in the order specified and that
>> the generations get switched around.  This is evidenced by the file size of
>> the Data file associated with the generations as it moves through the
>> process.  In this case we expect the nb-6 to become ma-7 as per the output
>> from the run.  In actuality we want nb-6 to be

GSoD 2023

[Deepak Vohra via dev]

Lorina,
Happy New Year.
You mentioned this year you might apply. As I am interested in being the Tech 
Writer, a reminder to apply as applications are to open from January to March 
according to 
https://sites.google.com/view/gsoc-sod-fosdem-2023/google-season-of-docs. 
Detail yet to be posted for  GSOD 2023.
regards,Deepak

Introducing mockito-inline library among test dependencies

[Miklosovic, Stefan]

Hi list,

the test for (1) is using mockito-inline dependency for mocking static methods 
as mockito-core is not able to do that on its own. mockito-inline was not part 
of our test dependencies prior this work. I want to ask if we are all OK with 
being able to mock static methods from now on with the help of this library. 

Please tell me if we are mocking static methods already by some other (to me 
yet unknown) mean so we do not include this unnecessarily.

G:A:V is org.mockito:mockito-inline:4.7.0

(1) https://issues.apache.org/jira/browse/CASSANDRA-14361

Thanks

Re: Introducing mockito-inline library among test dependencies

[David Capwell]

+1. We already use mockito.  Also that library is basically empty, its just 
defining configs for extensions (see 
https://github.com/mockito/mockito/tree/main/subprojects/inline/src/main/resources/mockito-extensions
 
)

> On Jan 11, 2023, at 12:02 PM, Miklosovic, Stefan 
>  wrote:
> 
> Hi list,
> 
> the test for (1) is using mockito-inline dependency for mocking static 
> methods as mockito-core is not able to do that on its own. mockito-inline was 
> not part of our test dependencies prior this work. I want to ask if we are 
> all OK with being able to mock static methods from now on with the help of 
> this library. 
> 
> Please tell me if we are mocking static methods already by some other (to me 
> yet unknown) mean so we do not include this unnecessarily.
> 
> G:A:V is org.mockito:mockito-inline:4.7.0
> 
> (1) https://issues.apache.org/jira/browse/CASSANDRA-14361
> 
> Thanks