CommitLogReaderTest

[Angelo Polo]

Hi C* Devs,

I've got two questions relating to the CommitLogReader.

1.
I maintain the Cassandra port for FreeBSD. For C* 3.11.5 (OpenJDK
1.8.0_222, FreeBSD 12.1-RELEASE-p1 GENERIC amd64), there are currently test
failures in org.apache.cassandra.db.commitlog.CommitLogReaderTest with zero
mutations read in each case. Only testReadCount passes and it looks like
that's because the number of generated samples in the test is smaller than
the others. Increase the sample size to around 300 and it will fail too;
similarly, the others pass with a smaller sample size.

The tests are failing because CommitLogDescriptor#readHeader returns null,
a result of the checksum check failing: the checksum read from the file
(CommitLogDescriptor:175) is zero. Similarly version, id, and
parameterLength are all zero and from what I can tell the commit log in
these instances is just all zeros.

Any idea why a "large" number of mutations could cause a problem with the
commit log? Has this been seen and fixed on any of the officially supported
platforms (Linux, Darwin)?

2.
Because of #1 above, I started playing around with CommitLogReaderTest on
Linux (OpenJDK 1.8.0_232, Ubuntu 18.04). Checked out the 3.11.5 release
commit and changed just samples and readCount in
CommitLogReaderTest#testReadFromMidpoint as follows:

public void testReadFromMidpoint() throws Throwable
{
int samples = 100;  // formerly 1000
int readCount = 60; // formerly 500

Shouldn't this fail uniformly every time with "Expected 60 seen mutations,
got: 50"? It produces somewhat random results:

[junit-timeout] Testcase:
testReadFromMidpoint(org.apache.cassandra.db.commitlog.CommitLogReaderTest):
 FAILED
[junit-timeout] Expected 60 seen mutations, got: 55 expected:<60> but
was:<55>
[junit-timeout] junit.framework.AssertionFailedError: Expected 60 seen
mutations, got: 55 expected:<60> but was:<55>
[junit-timeout] at
org.apache.cassandra.db.commitlog.CommitLogReaderTest.testReadFromMidpoint(CommitLogReaderTest.java:111)

[junit-timeout] Testcase:
testReadFromMidpoint(org.apache.cassandra.db.commitlog.CommitLogReaderTest):
 FAILED
[junit-timeout] Expected 60 seen mutations, got: 57 expected:<60> but
was:<57>
[junit-timeout] junit.framework.AssertionFailedError: Expected 60 seen
mutations, got: 57 expected:<60> but was:<57>
[junit-timeout] at
org.apache.cassandra.db.commitlog.CommitLogReaderTest.testReadFromMidpoint(CommitLogReaderTest.java:111)

Including, occasionally, the anticipated answer :)
[junit-timeout] Testcase:
testReadFromMidpoint(org.apache.cassandra.db.commitlog.CommitLogReaderTest):
 FAILED
[junit-timeout] Expected 60 seen mutations, got: 50 expected:<60> but
was:<50>
[junit-timeout] junit.framework.AssertionFailedError: Expected 60 seen
mutations, got: 50 expected:<60> but was:<50>
[junit-timeout] at
org.apache.cassandra.db.commitlog.CommitLogReaderTest.testReadFromMidpoint(CommitLogReaderTest.java:111)

Shall I open a bug in JIRA for this or have I misunderstood how
testReadFromMidpoint works?

Thanks,
Angelo Polo

Is i386 still officially supported?

[Angelo Polo]

Is i386 still officially supported in 3.11.x and 4.0? I could swear I saw a
statement on cassandra.apache.org to the effect that C* was supposed to run
on the i386 architecture, but can't find it anymore.

i386 and  x86 are listed in
java/org/apache/cassandra/utils/Architecture.java and they're *not* listed
in the comment in that file listing architectures excluded from official
support - not sure what status that confers.

Thanks,
Angelo Polo

Re: Is i386 still officially supported?

[Angelo Polo]

`ant test` made it through several hundred test suites before it died:
apache-cassandra-4.0-alpha4-src/build.xml:1927: java.lang.OutOfMemoryError:
Java heap space

Some individual tests also error out with an OOM, e.g.
IndexSummaryTest#testLargeIndexSummary.

Best,
Angelo

On Mon, May 25, 2020 at 11:58 PM Nate McCall  wrote:

> If you had an environment available, I would be real curious about what
> happens when you start it up, particularly when we started writing larger
> SSTables.
>
> That said, I don't think we've fielded this question in like 8yrs, so I
> doubt there is any interest in supporting it if it doesnt just kind of work
> anyway.
>
> Hopefully someone else has some more concrete details.
>
> On Tue, May 26, 2020 at 5:07 AM Angelo Polo 
> wrote:
>
> > Is i386 still officially supported in 3.11.x and 4.0? I could swear I
> saw a
> > statement on cassandra.apache.org to the effect that C* was supposed to
> > run
> > on the i386 architecture, but can't find it anymore.
> >
> > i386 and  x86 are listed in
> > java/org/apache/cassandra/utils/Architecture.java and they're *not*
> listed
> > in the comment in that file listing architectures excluded from official
> > support - not sure what status that confers.
> >
> > Thanks,
> > Angelo Polo
> >
>

Cassandra 4.0-beta1 available on FreeBSD

[Angelo Polo]

Cassandra 4.0-beta1 is now available on FreeBSD.

You can find information about the port here:
https://www.freshports.org/databases/cassandra4/

The beta can be installed from an up-to-date ports tree under
databases/cassandra4.

Best,
Angelo

Re: [DISCUSS] When to stop supporting Python 2

[Angelo Polo]

Since python2 was completely removed from FreeBSD at the end of 2020, this
is a good idea from my perspective.

On a related note, since cassandra3 on FreeBSD was facing removal due to
the python2 dependency, I back-ported python3 compatibility to the 3.11
branch.
I've just created CASSANDRA-16403 to upstream this.

Best,
Angelo

https://issues.apache.org/jira/browse/CASSANDRA-16403

On Fri, Jan 22, 2021 at 9:14 PM Adam Holmberg 
wrote:

> As you may recall, CASSANDRA-10190 [1] introduced Python 3 support for
> cqlsh. This change will be landing in 4.0. In the course of development and
> discussion spanning years, it was decided to retain support for Python 2.
> In the meantime, Python 2 sunsetted (a year ago [2]). I hadn't seen a
> discussion about whether we intend to carry on support for Python 2, so I'm
> raising one here.
>
> 4.0 is a major release and we have an opportunity to drop support at this
> milestone. It has been mentioned that it will not be acceptable to do in a
> minor or patch release, so if it's not done for 4.0, we will need to wait
> for the next major. I do understand that many in the project would like
> majors on a more frequent interval post-4.0, but at this time we don't know
> when that will be.
>
> I advocate for dropping support ASAP. I expect that users should not be
> inconvenienced by this -- I am not aware of a major distro that has not had
> python3 for years. Dropping python2 support does not mean that we would do
> work to rip out python2-compatible code, just that we wouldn't advertise
> support and any package requirements would be adjusted. We benefit by
> removing the need to test multiple runtimes, and we wouldn't be concerned
> with fixing python2-specific issues that may arise on the EOL runtime [3].
>
> I look forward to the discussion.
>
> --
> Adam Holmberg
> e. adam.holmb...@datastax.com
> w. www.datastax.com
>
> [1] https://issues.apache.org/jira/browse/CASSANDRA-10190
> [2] https://www.python.org/doc/sunset-python-2/
> [3] https://issues.apache.org/jira/browse/CASSANDRA-16400
>

Re: Invitation to participate in a survey about Apache Cassandra

[Angelo Polo]

Hi Tan,

How, when, and to whom will the results be published?

Best,
Angelo

On Mon, Mar 1, 2021 at 10:32 PM Tan, J.  wrote:

> Dear Cassandra contributor,
>
> We are doing research on understanding how developers manage a special kind
> of Technical Debt in *Java.*
>
> We kindly ask 15-20 minutes of your time to fill out our survey. To help
> you decide whether to fill it in, we clarify two points.
>
> “Why should I answer this survey?”
>
> Your participation is essential for us to correctly understand how
> developers manage Technical Debt.
>
> “What is in it for me?”
>
> Your valuable contributions to *Cassandra* are part of the information we
> analyzed for this study. Thus, if you help us further by answering
> this survey, there are two immediate benefits:
>
>- you help to improve the efficiency of maintaining the quality of
>*Cassandra*.
>- the results will be used to propose recommendations to manage
>technical debt and create tool support.
>
> Here is the link to the survey
> <
> https://docs.google.com/forms/d/e/1FAIpQLSebaVkO0SyHog7tPohoQ6nrsC_ZiRRFIXIx8egUbwfjpDzKsA/viewform?usp=sf_link
> >
> .
>
> Thank you for your time and attention.
>
> Kind regards,
> Jie Tan, Daniel Feitosa and Paris Avgeriou
>
> Software Engineering and Architecture group 
> Faculty of Science and Engineering
> University of Groningen, the Netherlands
>

Re: [DISCUSS] Remove support for `test.runners` and `testparallel`

[Angelo Polo]

Docker doesn't run natively on FreeBSD (though work is underway to enable
that). It's possible to run Docker Machine inside VirtualBox so maybe
that's workable, otherwise I suppose I can live without parallel testing
for now since I'm probably the only one.

Best,
Angelo

On Tue, Apr 13, 2021 at 10:59 AM Mick Semb Wever  wrote:

> > +1 after chatting with Mick who clarified the picture for me. Thx Mick.
>
> 👍
>
> I'm +1 as well to removing test.runner and testparallel support, from
> all branches.
>
> CASSANDRA-16595 has been created.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: [DISCUSSION] Attracting new contributors

[Angelo Polo]

Might also want to check among the tickets opened by non-committers and
still awaiting an assignee. E.g.

*assignee is EMPTY AND reporter not in membersOf(Committers)*

There are patches/pull-requests there too.

Best,
Angelo

On Thu, Apr 29, 2021 at 1:51 PM Benjamin Lerer  wrote:

> >
> > Berenguer created this board to help to track newcomers contributions:
> >
> >
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=463&quickFilter=2088
>
>
> My apologies, the board was not accessible to most persons. I solved that
> and everybody should have access to it now.
>
> Some committers are appearing in the list because they do not belong to the
> correct JIRA groups. I opened a ticket to INFRA to solve that problem (
> INFRA-21808 ).
>
> Nevertheless we have obviously a serious issue because even after removing
> the committers we have more than 80 non committer patches waiting for
> reviews. I will try to go through them in the coming weeks to see what
> happened with those patches and what we can do about them.
>
> My hope is that with this new board we can ensure that we provide a timely
> response to newcomers tickets.
>
> Le jeu. 29 avr. 2021 à 13:42, Benjamin Lerer  a écrit :
>
> > Thanks for the feedback Aleksei,
> >
> > A good way of doing that
> >> is having some rewards. It might be smth material like a T-Shirt (I
> >> remember getting a T-Shirt on C* v2 release was nice; obviously not for
> >> a single commit, but for multiple - depends on the budget; or top 10 the
> >> most active external contributors) or smth free like a virtual badge,
> >> being posted in an annual list of contributors or similar.
> >
> >
> > It seems that we will need to find some sponsors for some t-shirt ;-) We
> > definitely need to have some T-shirts for 4.0!!!
> >
> >   If there is a need I can volunteer/kick off any of the above points.
> >>
> >
> > Please do. Feel free to fire some discussions on the dev list to discuss
> > each of those points. Simply take care to do it for one subject at a time
> > as people might have some trouble to follow all the discussions going on
> > otherwise.
> >
> > Le jeu. 29 avr. 2021 à 13:23, Benedict Elliott Smith <
> bened...@apache.org>
> > a écrit :
> >
> >> Thanks Aleksei,
> >>
> >> Some of these are great points, but to respond specifically to the
> >> checkstyle suggestion: I hope to kick off some (minor) discussion around
> >> codestyle soon to modernise our guide, however I would personally prefer
> >> that code style enforcement remains relatively light touch. Some obvious
> >> things could be enforced by checkstyle (such as the braces), and we
> should
> >> investigate that*, but I would hate for the project to get _too_
> mechanical
> >> about the way code is structured.
> >>
> >> I've been fairly opposed to the upheaval caused by changing build
> >> tooling, but you're right that the barrier to booting up your IDE is a
> big
> >> part of the contribution overhead for newbies, so perhaps we should take
> >> another look at it.
> >>
> >> * I hope to utilise checkstyle soon to prohibit certain specific code
> >> patterns too, but that’s for a much later discussion
> >>
> >>
> >> On 29/04/2021, 12:05, "Aleksei Zotov"  wrote:
> >>
> >> Hi Benjamin,
> >>
> >> I'd like to put in my two cents as well.
> >>
> >> There were many great suggestions related to the communication and
> >> process. They make sense to me, however, I'd like to look at the
> >> problem
> >> from another perspective.
> >>
> >> First of all, let me share my perception on the opensource
> >> activities.
> >> There are two main reasons why people may want to contribute: 1)
> they
> >> experience a problem on the current project 2) any kind of
> >> volunteering.
> >> The first reason is clear, those contributors are not going to stick
> >> around because they just need to solve their particular problem.
> >>
> >> The second group of people is our target. In fact, there could be
> >> numerous reasons why people want to contribute (feel bored, get new
> >> experience, improve resume, etc), but despite the particular
> >> motivation
> >> point, people should feel positive of what they are doing. For that
> >> we
> >> should make sure they feel a part of the team/process and their work
> >> appreciated.
> >>
> >> The first point is related to many suggestions that have been
> already
> >> brought. I feel the most important here is timely replies (even
> >> "sorry,
> >> we're busy these days, I'll review/respond in two weeks / after xxx
> >> version is released" is much better than silence). Such "follow up"
> >> responses do not address the original queries, but they help the
> >> external contributors to keep courage and remove uncertainty related
> >> to
> >> the lack of transparency (it might not be clear: a) whether the
> >> request
> >> is still on someone'

4.0-rc1 from source - CqlshTest failure: Python driver not installed

[Angelo Polo]

Hey there,

When running bin/cqlsh or the test CqlshTest#testKeyspaceRequired, I'm
getting an error : "Python Cassandra driver not installed, or not on
PYTHONPATH." The suggestion to install the driver with pip is in the test
failure output (below), but is pre-installing the driver a new requirement
for using cqlsh? In earlier releases up through 4.0-beta4 the driver was
bundled. For example in 3.11.10
$ find . -name "*driver*"
./doc/source/getting_started/drivers.rst
./lib/cassandra-driver-core-3.0.1-shaded.jar
*./lib/cassandra-driver-internal-only-3.10.zip*
*./lib/cassandra-driver-internal-only-3.11.0-bb96859b.zip*
./lib/licenses/cassandra-driver-3.0.1.txt

Whereas 4.0-rc1 only has the Java jars.
$ find . -name "*driver*"
./doc/source/getting_started/drivers.rst
./build/lib/jars/cassandra-driver-core-3.11.0-shaded.jar
./build/dist/lib/cassandra-driver-core-3.11.0-shaded.jar
./lib/cassandra-driver-core-3.11.0-shaded.jar

Wasn't able to find mention of a new driver requirement here:
https://cassandra.apache.org/download/
https://cassandra.apache.org/doc/latest/getting_started/installing.html

Have I got some build misconfiguration or have requirements changed due to
the whole dependency bundling question that recently came up?

Thanks,
Angelo


Test output:

[junit-timeout] Testsuite: org.apache.cassandra.tools.cqlsh.CqlshTest Tests
run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 14.872 sec
[junit-timeout]
[junit-timeout] Testcase:
testKeyspaceRequired(org.apache.cassandra.tools.cqlsh.CqlshTest): FAILED
[junit-timeout]
[junit-timeout] Expected: a string containing "No keyspace has been
specified" ignoring case
[junit-timeout]  but: was "
[junit-timeout] Python Cassandra driver not installed, or not on PYTHONPATH.
[junit-timeout] You might try "pip install cassandra-driver".
[junit-timeout]
[junit-timeout] Python: /usr/local/bin/python3.7
[junit-timeout] Module load path:
['/wrkdirs/usr/ports/databases/cassandra4/work/apache-cassandra-4.0-rc1-src/bin/../lib/geomet-0.1.0.zip',
'/wrkdirs/usr/ports/databases/cassandra4/work/apache-cassandra-4.0-rc1-src/bin/../lib/six-1.12.0-py2.py3-none-any.zip',
'/wrkdirs/usr/ports/databases/cassandra4/work/apache-cassandra-4.0-rc1-src/bin/../lib/futures-2.1.6-py2.py3-none-any.zip',
'/wrkdirs/usr/ports/databases/cassandra4/work/apache-cassandra-4.0-rc1-src/bin',
'/usr/local/lib/python37.zip', '/usr/local/lib/python3.7',
'/usr/local/lib/python3.7/lib-dynload',
'/usr/local/lib/python3.7/site-packages']
[junit-timeout]
[junit-timeout] Error: No module named 'cassandra'
[junit-timeout]
[junit-timeout] "

Re: 4.0-rc1 from source - CqlshTest failure: Python driver not installed

[Angelo Polo]

Below I'd included a partial listing of the lib/ directory... build has
happened prior to attempting the tests or using cqlsh.
But I take it that you're also not expecting that I would need to manually
install the python driver?

Also don't see the python driver as an explicit dependency in build.xml,
only as something not to delete in the "realclean" target. Should it have
been downloaded as a transitive dep of something else?

Angelo


On Wed, May 26, 2021 at 3:49 PM Brandon Williams  wrote:

> The lib directory will be empty until you build, then it gets populated
> now.
>
> On Wed, May 26, 2021, 8:31 AM Angelo Polo 
> wrote:
>
> > Hey there,
> >
> > When running bin/cqlsh or the test CqlshTest#testKeyspaceRequired, I'm
> > getting an error : "Python Cassandra driver not installed, or not on
> > PYTHONPATH." The suggestion to install the driver with pip is in the test
> > failure output (below), but is pre-installing the driver a new
> requirement
> > for using cqlsh? In earlier releases up through 4.0-beta4 the driver was
> > bundled. For example in 3.11.10
> > $ find . -name "*driver*"
> > ./doc/source/getting_started/drivers.rst
> > ./lib/cassandra-driver-core-3.0.1-shaded.jar
> > *./lib/cassandra-driver-internal-only-3.10.zip*
> > *./lib/cassandra-driver-internal-only-3.11.0-bb96859b.zip*
> > ./lib/licenses/cassandra-driver-3.0.1.txt
> >
> > Whereas 4.0-rc1 only has the Java jars.
> > $ find . -name "*driver*"
> > ./doc/source/getting_started/drivers.rst
> > ./build/lib/jars/cassandra-driver-core-3.11.0-shaded.jar
> > ./build/dist/lib/cassandra-driver-core-3.11.0-shaded.jar
> > ./lib/cassandra-driver-core-3.11.0-shaded.jar
> >
> > Wasn't able to find mention of a new driver requirement here:
> > https://cassandra.apache.org/download/
> > https://cassandra.apache.org/doc/latest/getting_started/installing.html
> >
> > Have I got some build misconfiguration or have requirements changed due
> to
> > the whole dependency bundling question that recently came up?
> >
> > Thanks,
> > Angelo
> >
> > 
> > Test output:
> >
> > [junit-timeout] Testsuite: org.apache.cassandra.tools.cqlsh.CqlshTest
> Tests
> > run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 14.872 sec
> > [junit-timeout]
> > [junit-timeout] Testcase:
> > testKeyspaceRequired(org.apache.cassandra.tools.cqlsh.CqlshTest): FAILED
> > [junit-timeout]
> > [junit-timeout] Expected: a string containing "No keyspace has been
> > specified" ignoring case
> > [junit-timeout]  but: was "
> > [junit-timeout] Python Cassandra driver not installed, or not on
> > PYTHONPATH.
> > [junit-timeout] You might try "pip install cassandra-driver".
> > [junit-timeout]
> > [junit-timeout] Python: /usr/local/bin/python3.7
> > [junit-timeout] Module load path:
> >
> >
> ['/wrkdirs/usr/ports/databases/cassandra4/work/apache-cassandra-4.0-rc1-src/bin/../lib/geomet-0.1.0.zip',
> >
> >
> '/wrkdirs/usr/ports/databases/cassandra4/work/apache-cassandra-4.0-rc1-src/bin/../lib/six-1.12.0-py2.py3-none-any.zip',
> >
> >
> '/wrkdirs/usr/ports/databases/cassandra4/work/apache-cassandra-4.0-rc1-src/bin/../lib/futures-2.1.6-py2.py3-none-any.zip',
> >
> >
> '/wrkdirs/usr/ports/databases/cassandra4/work/apache-cassandra-4.0-rc1-src/bin',
> > '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7',
> > '/usr/local/lib/python3.7/lib-dynload',
> > '/usr/local/lib/python3.7/site-packages']
> > [junit-timeout]
> > [junit-timeout] Error: No module named 'cassandra'
> > [junit-timeout]
> > [junit-timeout] "
> >
>

Re: 4.0-rc1 from source - CqlshTest failure: Python driver not installed

[Angelo Polo]

Thanks for the reply Brandon.
I've opened CASSANDRA-16700, but I'll leave it to others more familiar with
the driver versions to decide the exact dependency to set.

Best,
Angelo

On Wed, May 26, 2021 at 5:00 PM Brandon Williams  wrote:

> On Wed, May 26, 2021 at 9:05 AM Angelo Polo 
> wrote:
> > But I take it that you're also not expecting that I would need to
> manually
> > install the python driver?
>
> I'm not, but I checked the rc1 source archive and indeed, the python
> driver never gets pulled in so cqlsh is not usable.
>
> > Also don't see the python driver as an explicit dependency in build.xml,
> > only as something not to delete in the "realclean" target. Should it have
> > been downloaded as a transitive dep of something else?
>
> It should have, but it looks like this is a bug as I just tried
> building rc2 and it has the same problem.  Can you please file a jira?
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>