Have we considered static type checking for our python libs?

[Joshua McKenzie]

Relevant links:
1) Optional static typing for python:
https://docs.python.org/3/library/typing.html
2) Mypy static type checker for python: https://github.com/python/mypy

So the question - has anyone given any serious thought to introducing type
hints and a static type checker in ccm and python dtests? A search on dev
ponymail doesn't turn up anything.

I've used it pretty extensively in the past and found it incredibly helpful
combined with other linters in surfacing troublesome edge cases, and also
found it accelerated development quite a bit.

Any thoughts on the topic for or against?

~Josh

Re: Have we considered static type checking for our python libs?

[Brandon Williams]

I have yet to encounter this class of problem in the dtests.

On Wed, Jan 26, 2022 at 6:59 AM Joshua McKenzie  wrote:
>
> Relevant links:
> 1) Optional static typing for python: 
> https://docs.python.org/3/library/typing.html
> 2) Mypy static type checker for python: https://github.com/python/mypy
>
> So the question - has anyone given any serious thought to introducing type 
> hints and a static type checker in ccm and python dtests? A search on dev 
> ponymail doesn't turn up anything.
>
> I've used it pretty extensively in the past and found it incredibly helpful 
> combined with other linters in surfacing troublesome edge cases, and also 
> found it accelerated development quite a bit.
>
> Any thoughts on the topic for or against?
>
> ~Josh

Re: Have we considered static type checking for our python libs?

[bened...@apache.org]

I was sort of hoping we would retire the python dtests before long, at least in 
large part (probably not ever entirely, but 99%).

I think many of them could be migrated to in-jvm dtests without much effort. I 
would hate to expend loads of effort modernising them when the same effort 
could see them superseded by much better versions of the same test.


From: Joshua McKenzie 
Date: Wednesday, 26 January 2022 at 12:59
To: dev 
Subject: Have we considered static type checking for our python libs?
Relevant links:
1) Optional static typing for python: 
https://docs.python.org/3/library/typing.html
2) Mypy static type checker for python: https://github.com/python/mypy

So the question - has anyone given any serious thought to introducing type 
hints and a static type checker in ccm and python dtests? A search on dev 
ponymail doesn't turn up anything.

I've used it pretty extensively in the past and found it incredibly helpful 
combined with other linters in surfacing troublesome edge cases, and also found 
it accelerated development quite a bit.

Any thoughts on the topic for or against?

~Josh

Re: Have we considered static type checking for our python libs?

[Joshua McKenzie]

>
> I have yet to encounter this class of problem in the dtests.

It's more about development velocity and convenience than about preventing
defects in our case, since we're not abusing duck-typing everywhere. Every
time I have to work on python dtests (for instance, when doing build lead
work and looking at flaky tests) it's a little irritating and I think of
this.

 I would hate to expend loads of effort modernising them when the same
> effort could see them superseded by much better versions of the same test.

I completely agree, however this is something someone would have to take on
as an effort and I don't believe I've seen anybody step up yet. At the
current rate we're going to be dragging along the python dtests into
perpetuity.


On Wed, Jan 26, 2022 at 8:16 AM bened...@apache.org 
wrote:

> I was sort of hoping we would retire the python dtests before long, at
> least in large part (probably not ever entirely, but 99%).
>
>
>
> I think many of them could be migrated to in-jvm dtests without much
> effort. I would hate to expend loads of effort modernising them when the
> same effort could see them superseded by much better versions of the same
> test.
>
>
>
>
>
> *From: *Joshua McKenzie 
> *Date: *Wednesday, 26 January 2022 at 12:59
> *To: *dev 
> *Subject: *Have we considered static type checking for our python libs?
>
> Relevant links:
>
> 1) Optional static typing for python:
> https://docs.python.org/3/library/typing.html
>
> 2) Mypy static type checker for python: https://github.com/python/mypy
>
>
>
> So the question - has anyone given any serious thought to introducing type
> hints and a static type checker in ccm and python dtests? A search on dev
> ponymail doesn't turn up anything.
>
>
>
> I've used it pretty extensively in the past and found it incredibly
> helpful combined with other linters in surfacing troublesome edge cases,
> and also found it accelerated development quite a bit.
>
>
>
> Any thoughts on the topic for or against?
>
>
>
> ~Josh
>

Re: Have we considered static type checking for our python libs?

[bened...@apache.org]

We could set this as a broad goal of the project, and like the build lead role 
could each volunteer to adopt a test every X weeks. We would have migrated in 
no time, I expect, with this kind of concerted effort, and might not even 
notice a significant penalty to other ongoing work.

Last time I ported a dtest it was a very easy thing to do.

I might even venture to predict that it might payoff with lower development 
overhead, as we can run our tests much more quickly, and debug failures much 
more easily.

From: Joshua McKenzie 
Date: Wednesday, 26 January 2022 at 13:40
To: dev 
Subject: Re: Have we considered static type checking for our python libs?
I have yet to encounter this class of problem in the dtests.
It's more about development velocity and convenience than about preventing 
defects in our case, since we're not abusing duck-typing everywhere. Every time 
I have to work on python dtests (for instance, when doing build lead work and 
looking at flaky tests) it's a little irritating and I think of this.

 I would hate to expend loads of effort modernising them when the same effort 
could see them superseded by much better versions of the same test.
I completely agree, however this is something someone would have to take on as 
an effort and I don't believe I've seen anybody step up yet. At the current 
rate we're going to be dragging along the python dtests into perpetuity.


On Wed, Jan 26, 2022 at 8:16 AM bened...@apache.org 
mailto:bened...@apache.org>> wrote:
I was sort of hoping we would retire the python dtests before long, at least in 
large part (probably not ever entirely, but 99%).

I think many of them could be migrated to in-jvm dtests without much effort. I 
would hate to expend loads of effort modernising them when the same effort 
could see them superseded by much better versions of the same test.


From: Joshua McKenzie mailto:jmcken...@apache.org>>
Date: Wednesday, 26 January 2022 at 12:59
To: dev mailto:dev@cassandra.apache.org>>
Subject: Have we considered static type checking for our python libs?
Relevant links:
1) Optional static typing for python: 
https://docs.python.org/3/library/typing.html
2) Mypy static type checker for python: https://github.com/python/mypy

So the question - has anyone given any serious thought to introducing type 
hints and a static type checker in ccm and python dtests? A search on dev 
ponymail doesn't turn up anything.

I've used it pretty extensively in the past and found it incredibly helpful 
combined with other linters in surfacing troublesome edge cases, and also found 
it accelerated development quite a bit.

Any thoughts on the topic for or against?

~Josh

Re: Have we considered static type checking for our python libs?

[Joshua McKenzie]

Could be a very fruitful source of LHF tickets to highlight in the biweekly
email and would be pretty trivial to integrate this into the build lead
role (getting an epic and jira tickets created to port tests over, etc).

we can run our tests much more quickly, and debug failures much more easily.

Please Yes. If we can get away from python upgrade tests I think all our
lives would be improved.

I like it.


On Wed, Jan 26, 2022 at 8:42 AM bened...@apache.org 
wrote:

> We could set this as a broad goal of the project, and like the build lead
> role could each volunteer to adopt a test every X weeks. We would have
> migrated in no time, I expect, with this kind of concerted effort, and
> might not even notice a significant penalty to other ongoing work.
>
>
>
> Last time I ported a dtest it was a very easy thing to do.
>
>
>
> I might even venture to predict that it might payoff with lower
> development overhead, as we can run our tests much more quickly, and debug
> failures much more easily.
>
>
>
> *From: *Joshua McKenzie 
> *Date: *Wednesday, 26 January 2022 at 13:40
> *To: *dev 
> *Subject: *Re: Have we considered static type checking for our python
> libs?
>
> I have yet to encounter this class of problem in the dtests.
>
> It's more about development velocity and convenience than about preventing
> defects in our case, since we're not abusing duck-typing everywhere. Every
> time I have to work on python dtests (for instance, when doing build lead
> work and looking at flaky tests) it's a little irritating and I think of
> this.
>
>
>
>  I would hate to expend loads of effort modernising them when the same
> effort could see them superseded by much better versions of the same test.
>
> I completely agree, however this is something someone would have to take
> on as an effort and I don't believe I've seen anybody step up yet. At the
> current rate we're going to be dragging along the python dtests into
> perpetuity.
>
>
>
>
>
> On Wed, Jan 26, 2022 at 8:16 AM bened...@apache.org 
> wrote:
>
> I was sort of hoping we would retire the python dtests before long, at
> least in large part (probably not ever entirely, but 99%).
>
>
>
> I think many of them could be migrated to in-jvm dtests without much
> effort. I would hate to expend loads of effort modernising them when the
> same effort could see them superseded by much better versions of the same
> test.
>
>
>
>
>
> *From: *Joshua McKenzie 
> *Date: *Wednesday, 26 January 2022 at 12:59
> *To: *dev 
> *Subject: *Have we considered static type checking for our python libs?
>
> Relevant links:
>
> 1) Optional static typing for python:
> https://docs.python.org/3/library/typing.html
>
> 2) Mypy static type checker for python: https://github.com/python/mypy
>
>
>
> So the question - has anyone given any serious thought to introducing type
> hints and a static type checker in ccm and python dtests? A search on dev
> ponymail doesn't turn up anything.
>
>
>
> I've used it pretty extensively in the past and found it incredibly
> helpful combined with other linters in surfacing troublesome edge cases,
> and also found it accelerated development quite a bit.
>
>
>
> Any thoughts on the topic for or against?
>
>
>
> ~Josh
>
>

Re: Have we considered static type checking for our python libs?

[Andrés de la Peña]

Last time I ported dtests during the 4.0 quality test epic there wasn't
support for virtual nodes in in-jvm dtests. We have many Python dtests
depending on vnodes that can't be totally ported if we still don't have
support for vnodes, I don't know if it's still the case.

On Wed, 26 Jan 2022 at 14:02, Joshua McKenzie  wrote:

> Could be a very fruitful source of LHF tickets to highlight in the
> biweekly email and would be pretty trivial to integrate this into the build
> lead role (getting an epic and jira tickets created to port tests over,
> etc).
>
> we can run our tests much more quickly, and debug failures much more
>> easily.
>
> Please Yes. If we can get away from python upgrade tests I think all our
> lives would be improved.
>
> I like it.
>
>
> On Wed, Jan 26, 2022 at 8:42 AM bened...@apache.org 
> wrote:
>
>> We could set this as a broad goal of the project, and like the build lead
>> role could each volunteer to adopt a test every X weeks. We would have
>> migrated in no time, I expect, with this kind of concerted effort, and
>> might not even notice a significant penalty to other ongoing work.
>>
>>
>>
>> Last time I ported a dtest it was a very easy thing to do.
>>
>>
>>
>> I might even venture to predict that it might payoff with lower
>> development overhead, as we can run our tests much more quickly, and debug
>> failures much more easily.
>>
>>
>>
>> *From: *Joshua McKenzie 
>> *Date: *Wednesday, 26 January 2022 at 13:40
>> *To: *dev 
>> *Subject: *Re: Have we considered static type checking for our python
>> libs?
>>
>> I have yet to encounter this class of problem in the dtests.
>>
>> It's more about development velocity and convenience than about
>> preventing defects in our case, since we're not abusing duck-typing
>> everywhere. Every time I have to work on python dtests (for instance, when
>> doing build lead work and looking at flaky tests) it's a little irritating
>> and I think of this.
>>
>>
>>
>>  I would hate to expend loads of effort modernising them when the same
>> effort could see them superseded by much better versions of the same test.
>>
>> I completely agree, however this is something someone would have to take
>> on as an effort and I don't believe I've seen anybody step up yet. At the
>> current rate we're going to be dragging along the python dtests into
>> perpetuity.
>>
>>
>>
>>
>>
>> On Wed, Jan 26, 2022 at 8:16 AM bened...@apache.org 
>> wrote:
>>
>> I was sort of hoping we would retire the python dtests before long, at
>> least in large part (probably not ever entirely, but 99%).
>>
>>
>>
>> I think many of them could be migrated to in-jvm dtests without much
>> effort. I would hate to expend loads of effort modernising them when the
>> same effort could see them superseded by much better versions of the same
>> test.
>>
>>
>>
>>
>>
>> *From: *Joshua McKenzie 
>> *Date: *Wednesday, 26 January 2022 at 12:59
>> *To: *dev 
>> *Subject: *Have we considered static type checking for our python libs?
>>
>> Relevant links:
>>
>> 1) Optional static typing for python:
>> https://docs.python.org/3/library/typing.html
>>
>> 2) Mypy static type checker for python: https://github.com/python/mypy
>>
>>
>>
>> So the question - has anyone given any serious thought to introducing
>> type hints and a static type checker in ccm and python dtests? A search on
>> dev ponymail doesn't turn up anything.
>>
>>
>>
>> I've used it pretty extensively in the past and found it incredibly
>> helpful combined with other linters in surfacing troublesome edge cases,
>> and also found it accelerated development quite a bit.
>>
>>
>>
>> Any thoughts on the topic for or against?
>>
>>
>>
>> ~Josh
>>
>>

Re: Have we considered static type checking for our python libs?

[bened...@apache.org]

I don’t think this support would be very hard to add, if this is the only 
stumbling block.

From: Andrés de la Peña 
Date: Wednesday, 26 January 2022 at 14:10
To: dev@cassandra.apache.org 
Subject: Re: Have we considered static type checking for our python libs?
Last time I ported dtests during the 4.0 quality test epic there wasn't support 
for virtual nodes in in-jvm dtests. We have many Python dtests depending on 
vnodes that can't be totally ported if we still don't have support for vnodes, 
I don't know if it's still the case.

On Wed, 26 Jan 2022 at 14:02, Joshua McKenzie 
mailto:jmcken...@apache.org>> wrote:
Could be a very fruitful source of LHF tickets to highlight in the biweekly 
email and would be pretty trivial to integrate this into the build lead role 
(getting an epic and jira tickets created to port tests over, etc).

we can run our tests much more quickly, and debug failures much more easily.
Please Yes. If we can get away from python upgrade tests I think all our lives 
would be improved.

I like it.


On Wed, Jan 26, 2022 at 8:42 AM bened...@apache.org 
mailto:bened...@apache.org>> wrote:
We could set this as a broad goal of the project, and like the build lead role 
could each volunteer to adopt a test every X weeks. We would have migrated in 
no time, I expect, with this kind of concerted effort, and might not even 
notice a significant penalty to other ongoing work.

Last time I ported a dtest it was a very easy thing to do.

I might even venture to predict that it might payoff with lower development 
overhead, as we can run our tests much more quickly, and debug failures much 
more easily.

From: Joshua McKenzie mailto:jmcken...@apache.org>>
Date: Wednesday, 26 January 2022 at 13:40
To: dev mailto:dev@cassandra.apache.org>>
Subject: Re: Have we considered static type checking for our python libs?
I have yet to encounter this class of problem in the dtests.
It's more about development velocity and convenience than about preventing 
defects in our case, since we're not abusing duck-typing everywhere. Every time 
I have to work on python dtests (for instance, when doing build lead work and 
looking at flaky tests) it's a little irritating and I think of this.

 I would hate to expend loads of effort modernising them when the same effort 
could see them superseded by much better versions of the same test.
I completely agree, however this is something someone would have to take on as 
an effort and I don't believe I've seen anybody step up yet. At the current 
rate we're going to be dragging along the python dtests into perpetuity.


On Wed, Jan 26, 2022 at 8:16 AM bened...@apache.org 
mailto:bened...@apache.org>> wrote:
I was sort of hoping we would retire the python dtests before long, at least in 
large part (probably not ever entirely, but 99%).

I think many of them could be migrated to in-jvm dtests without much effort. I 
would hate to expend loads of effort modernising them when the same effort 
could see them superseded by much better versions of the same test.


From: Joshua McKenzie mailto:jmcken...@apache.org>>
Date: Wednesday, 26 January 2022 at 12:59
To: dev mailto:dev@cassandra.apache.org>>
Subject: Have we considered static type checking for our python libs?
Relevant links:
1) Optional static typing for python: 
https://docs.python.org/3/library/typing.html
2) Mypy static type checker for python: https://github.com/python/mypy

So the question - has anyone given any serious thought to introducing type 
hints and a static type checker in ccm and python dtests? A search on dev 
ponymail doesn't turn up anything.

I've used it pretty extensively in the past and found it incredibly helpful 
combined with other linters in surfacing troublesome edge cases, and also found 
it accelerated development quite a bit.

Any thoughts on the topic for or against?

~Josh

Re: Have we considered static type checking for our python libs?

[Brandon Williams]

On Wed, Jan 26, 2022 at 7:43 AM bened...@apache.org  wrote:
> I might even venture to predict that it might payoff with lower development 
> overhead, as we can run our tests much more quickly, and debug failures much 
> more easily.

I don't think in practice these will happen at all, let alone 'much
more.'  Python execution is nowhere near the bottleneck, not that
either of these would speed it up significantly.  I'm unable to think
of an instance where typing in python could have helped me, at least
in the dtest tickets I've worked on.  Maybe someone with more
experience has a different estimation?

Re: Have we considered static type checking for our python libs?

[bened...@apache.org]

Python execution is obviously not a bottleneck, but cluster startup/shutdown 
and lengthy waits due to lack of visibility to state changes in the tests quite 
probably are. The python tests are also IME quite rudimentary compared to their 
java equivalents, and more brittle, due to worse tooling for e.g. data 
generation, interfering with message delivery etc.

They are also comparatively poorly maintained, due to most on the project not 
wanting to touch them beyond what is necessary. It would be hugely beneficial 
to migrate to a platform the majority uses, and that can produce more powerful 
tests that are also able to execute faster (for reasons of integration, not 
language).


From: Brandon Williams 
Date: Wednesday, 26 January 2022 at 16:09
To: dev 
Subject: Re: Have we considered static type checking for our python libs?
On Wed, Jan 26, 2022 at 7:43 AM bened...@apache.org  wrote:
> I might even venture to predict that it might payoff with lower development 
> overhead, as we can run our tests much more quickly, and debug failures much 
> more easily.

I don't think in practice these will happen at all, let alone 'much
more.'  Python execution is nowhere near the bottleneck, not that
either of these would speed it up significantly.  I'm unable to think
of an instance where typing in python could have helped me, at least
in the dtest tickets I've worked on.  Maybe someone with more
experience has a different estimation?

Re: Have we considered static type checking for our python libs?

[Joshua McKenzie]

>
> I'm unable to think of an instance where typing in python could have
> helped me,

Discoverability, IDE integration, more primitives to describe your intent
in your code to other maintainers, and shifting a certain class of runtime
errors to being "merge-time" errors all come to mind.

There's a significant implication to the "rigorous vs. fast and loose"
continuum with code you write using something like type hints and mypy vs.
classic python, and we have an admitted problem with rigor in that codebase
at this point.

Admittedly, changes w/regards to typing and such aren't going to do much to
address the deeper problems of non-deterministic execution, leaning heavily
on sleeps for timing in tests, etc

On Wed, Jan 26, 2022 at 11:41 AM bened...@apache.org 
wrote:

> Python execution is obviously not a bottleneck, but cluster
> startup/shutdown and lengthy waits due to lack of visibility to state
> changes in the tests quite probably are. The python tests are also IME
> quite rudimentary compared to their java equivalents, and more brittle, due
> to worse tooling for e.g. data generation, interfering with message
> delivery etc.
>
>
>
> They are also comparatively poorly maintained, due to most on the project
> not wanting to touch them beyond what is necessary. It would be hugely
> beneficial to migrate to a platform the majority uses, and that can produce
> more powerful tests that are also able to execute faster (for reasons of
> integration, not language).
>
>
>
>
>
> *From: *Brandon Williams 
> *Date: *Wednesday, 26 January 2022 at 16:09
> *To: *dev 
> *Subject: *Re: Have we considered static type checking for our python
> libs?
>
> On Wed, Jan 26, 2022 at 7:43 AM bened...@apache.org 
> wrote:
> > I might even venture to predict that it might payoff with lower
> development overhead, as we can run our tests much more quickly, and debug
> failures much more easily.
>
> I don't think in practice these will happen at all, let alone 'much
> more.'  Python execution is nowhere near the bottleneck, not that
> either of these would speed it up significantly.  I'm unable to think
> of an instance where typing in python could have helped me, at least
> in the dtest tickets I've worked on.  Maybe someone with more
> experience has a different estimation?
>

Re: Have we considered static type checking for our python libs?

[Brandon Williams]

On Wed, Jan 26, 2022 at 2:17 PM Joshua McKenzie  wrote:
>>
>> I'm unable to think of an instance where typing in python could have helped 
>> me,
>
> Discoverability, IDE integration, more primitives to describe your intent in 
> your code to other maintainers, and shifting a certain class of runtime 
> errors to being "merge-time" errors all come to mind.

I meant concretely from actually working on the tests, not a theoretical list.

Re: Have we considered static type checking for our python libs?

[Joshua McKenzie]

>
> I meant concretely from actually working on the tests, not a theoretical
> list.

I think that's fair. I'll see if I can't whip up an example of how it'd
mutate a single test and the API boundary between that and ccm. Should be a
quick but helpful exercise in seeing how it'd materially change things if
at all.

My experience was working with manipulating large quantities of data from
JIRA that came back in a huge variety of types; was a real pain in python.
Probably less surface area in any given test in the dtest suite interacting
with ccm.



On Wed, Jan 26, 2022 at 3:23 PM Brandon Williams  wrote:

> On Wed, Jan 26, 2022 at 2:17 PM Joshua McKenzie 
> wrote:
> >>
> >> I'm unable to think of an instance where typing in python could have
> helped me,
> >
> > Discoverability, IDE integration, more primitives to describe your
> intent in your code to other maintainers, and shifting a certain class of
> runtime errors to being "merge-time" errors all come to mind.
>
> I meant concretely from actually working on the tests, not a theoretical
> list.
>

Re: Have we considered static type checking for our python libs?

[Berenguer Blasi]

We'll have node reusage in dtests soon once CI is green again and 16951 
merges (in review). #justfyi


On 27/1/22 0:28, Joshua McKenzie wrote:


I meant concretely from actually working on the tests, not a
theoretical list.

I think that's fair. I'll see if I can't whip up an example of how 
it'd mutate a single test and the API boundary between that and ccm. 
Should be a quick but helpful exercise in seeing how it'd materially 
change things if at all.


My experience was working with manipulating large quantities of data 
from JIRA that came back in a huge variety of types; was a real pain 
in python. Probably less surface area in any given test in the dtest 
suite interacting with ccm.




On Wed, Jan 26, 2022 at 3:23 PM Brandon Williams  wrote:

On Wed, Jan 26, 2022 at 2:17 PM Joshua McKenzie
 wrote:
>>
>> I'm unable to think of an instance where typing in python could
have helped me,
>
> Discoverability, IDE integration, more primitives to describe
your intent in your code to other maintainers, and shifting a
certain class of runtime errors to being "merge-time" errors all
come to mind.

I meant concretely from actually working on the tests, not a
theoretical list.