[Python-Dev] Concurrent.futures: no type discovery for PyCharm
I am using concurrent.futures to parallelize independent tasks on multiple cores once in a while. Each time I have a difficulty remembering the specific syntax and have to look it up in old code or google. I would much prefer to be able to find the source through the PyCharm and have autocompletion. It takes adding two lines to the __init__.py of concurrent.futures: (insert on line 19) from .process import ProcessPoolExecutor from .thread import ThreadPoolExecutor I would also guess that it would make the __getattr__ redundant? Am I missing something or can this change be done this way and would indeed be an improvement? Best Regards, -- Ilya Kamenshchikov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Concurrent.futures: no type discovery for PyCharm
alright, so would an import under TYPE_CHECKING guard be an option? like: from typing import TYPE_CHECKING if TYPE_CHECKING: from .process import ProcessPoolExecutor from .thread import ThreadPoolExecutor Perhaps we can have both clarity and performance. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Concurrent.futures: no type discovery for PyCharm
How would we answer the same question if it was not a part of stdlib? I am not sure it is fair to expect of Pycharm to parse / execute the __getattr__ on modules, as more elaborate implementation could even contain different types per some condition at the runtime or anything at all. The code: TYPE_CHECKING = False if TYPE_CHECKING: from .process import ProcessPoolExecutor from .thread import ThreadPoolExecutor works for type checking in PyCharm and is fast. This is how stdlib can be an example to how side libraries can be implemented. If we can agree that this is the only clear, performant and sufficient code - then perhaps modifying mypy is a reasonable price to pay. Perhaps this particular case can be just patched locally by PyCharm /JetBrains, but what is a general solution to this class of problems? Best Regards, -- Ilya Kamenshchikov On Tue, Apr 23, 2019 at 7:05 PM Guido van Rossum wrote: > In any case I think this should be filed (by the OP) as an issue against > JetBrains' PyCharm issue tracker. Who knows they may be able to > special-case this in a jiffy. I don't think we should add any clever hacks > to the stdlib for this. > > On Tue, Apr 23, 2019 at 9:59 AM Nathaniel Smith wrote: > >> On Tue, Apr 23, 2019, 05:09 Andrew Svetlov >> wrote: >> >>> I agree that `from typing import TYPE_CHECKING` is not desirable from >>> the import time reduction perspective. >>> >>> From my understanding code completion *can* be based on type hinting >>> to avoid actual code execution. >>> That's why I've mentioned that typeshed already has the correct type >>> information. >>> >>> if TYPE_CHECKING: >>> import ... >>> >>> requires mypy modification. >>> >>> if False: >>> import ... >>> >>> Works right now for stdlib (mypy ignores stdlib code but uses typeshed >>> anyway) but looks a little cryptic. >>> Requires a comprehensive comment at least. >>> >> >> Last time I looked at this, I'm pretty sure `if False` broke at least one >> popular static analysis tool (ie it was clever enough to ignore everything >> inside `if False`) – I think either pylint or jedi? >> >> I'd suggest checking any clever hacks against at least: mypy, >> pylint/astroid, jedi, pyflakes, and pycharm. They all have their own static >> analysis engines, and each one has its own idiosyncratic quirks. >> >> We've struggled with this a *lot* in trio, and eventually ended up giving >> up on all forms of dynamic export cleverness; we've even banned the use of >> __all__ entirely. Static analysis has gotten good enough that users won't >> accept it not working, but it hasn't gotten good enough to handle anything >> but the simplest static exports in a reliable way: >> https://github.com/python-trio/trio/pull/316 >> https://github.com/python-trio/trio/issues/542 >> >> The stdlib has more leeway because when tools don't work on the stdlib >> then they tend to eventually add workarounds. I'm just saying, think twice >> before diving into clever hacks to workaround static analysis limits, and >> if you're going to do it then be careful to be thorough. You're basically >> relying on undocumented bugs, and it gets really messy really quickly. >> >> -n >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > -- > --Guido van Rossum (python.org/~guido) > *Pronouns: he/him/his **(why is my pronoun here?)* > <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Concurrent.futures: no type discovery for PyCharm
Ok thanks for explaining. I will proceed by trying it with typeshed. Best Regards, -- Ilya Kamenshchikov On Tue, Apr 23, 2019 at 9:44 PM Ivan Levkivskyi wrote: > Mypy doesn't use source code of stlib for analysis and instead uses stub > files from typeshed. IIUC PyCharm can also do that (i.e. use the typeshed > stubs). > The whole idea of typeshed is to avoid changing stlib solely for the sake > of static analysis. Please open an issue on typeshed an/or PyCharm tracker. > > -- > Ivan > > > > On Tue, 23 Apr 2019 at 20:38, Ilya Kamenshchikov > wrote: > >> How would we answer the same question if it was not a part of stdlib? >> I am not sure it is fair to expect of Pycharm to parse / execute the >> __getattr__ on modules, as more elaborate implementation could even contain >> different types per some condition at the runtime or anything at all. >> The code: >> >> TYPE_CHECKING = False >> if TYPE_CHECKING: >> from .process import ProcessPoolExecutor >> from .thread import ThreadPoolExecutor >> >> works for type checking in PyCharm and is fast. >> >> This is how stdlib can be an example to how side libraries can be >> implemented. If we can agree that this is the only clear, performant and >> sufficient code - then perhaps modifying mypy is a reasonable price to pay. >> >> Perhaps this particular case can be just patched locally by PyCharm >> /JetBrains, but what is a general solution to this class of problems? >> >> Best Regards, >> -- >> Ilya Kamenshchikov >> >> >> On Tue, Apr 23, 2019 at 7:05 PM Guido van Rossum >> wrote: >> >>> In any case I think this should be filed (by the OP) as an issue against >>> JetBrains' PyCharm issue tracker. Who knows they may be able to >>> special-case this in a jiffy. I don't think we should add any clever hacks >>> to the stdlib for this. >>> >>> On Tue, Apr 23, 2019 at 9:59 AM Nathaniel Smith wrote: >>> >>>> On Tue, Apr 23, 2019, 05:09 Andrew Svetlov >>>> wrote: >>>> >>>>> I agree that `from typing import TYPE_CHECKING` is not desirable from >>>>> the import time reduction perspective. >>>>> >>>>> From my understanding code completion *can* be based on type hinting >>>>> to avoid actual code execution. >>>>> That's why I've mentioned that typeshed already has the correct type >>>>> information. >>>>> >>>>> if TYPE_CHECKING: >>>>> import ... >>>>> >>>>> requires mypy modification. >>>>> >>>>> if False: >>>>> import ... >>>>> >>>>> Works right now for stdlib (mypy ignores stdlib code but uses typeshed >>>>> anyway) but looks a little cryptic. >>>>> Requires a comprehensive comment at least. >>>>> >>>> >>>> Last time I looked at this, I'm pretty sure `if False` broke at least >>>> one popular static analysis tool (ie it was clever enough to ignore >>>> everything inside `if False`) – I think either pylint or jedi? >>>> >>>> I'd suggest checking any clever hacks against at least: mypy, >>>> pylint/astroid, jedi, pyflakes, and pycharm. They all have their own static >>>> analysis engines, and each one has its own idiosyncratic quirks. >>>> >>>> We've struggled with this a *lot* in trio, and eventually ended up >>>> giving up on all forms of dynamic export cleverness; we've even banned the >>>> use of __all__ entirely. Static analysis has gotten good enough that users >>>> won't accept it not working, but it hasn't gotten good enough to handle >>>> anything but the simplest static exports in a reliable way: >>>> https://github.com/python-trio/trio/pull/316 >>>> https://github.com/python-trio/trio/issues/542 >>>> >>>> The stdlib has more leeway because when tools don't work on the stdlib >>>> then they tend to eventually add workarounds. I'm just saying, think twice >>>> before diving into clever hacks to workaround static analysis limits, and >>>> if you're going to do it then be careful to be thorough. You're basically >>>> relying on undocumented bugs, and it gets really messy really quickly. >>>> >>>> -n >>>> ___ >>>> Python-Dev mailing list >>>> Python-Dev@python.org >>>> https://mail.python.org/mailman/listinfo/python-dev >>>> Unsubscribe: >>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org >>>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> *Pronouns: he/him/his **(why is my pronoun here?)* >>> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> >>> >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com >> > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] reversed enumerate
Hi,
I needed reversed(enumerate(x: list)) in my code, and have discovered that
it wound't work. This is disappointing because operation is well defined.
It is also well defined for str type, range, and - in principle, but not
yet in practice - on dictionary iterators - keys(), values(), items() as
dictionaries are ordered now.
It would also be well defined on any user type implementing __iter__,
__len__, __reversed__ - think numpy arrays, some pandas dataframes, tensors.
That's plenty of usecases, therefore I guess it would be quite useful to
avoid hacky / inefficient solutions like described here:
https://code.activestate.com/lists/python-list/706205/.
If deemed useful, I would be interested in implementing this, maybe
together with __reversed__ on dict keys, values, items.
Best Regards,
--
Ilya Kamen
---
p.s.
*Sketch* of what I am proposing:
class reversible_enumerate:
def __init__(self, iterable):
self.iterable = iterable
self.ctr = 0
def __iter__(self):
for e in self.iterable:
yield self.ctr, e
self.ctr += 1
def __reversed__(self):
try:
ri = reversed(self.iterable)
except Exception as e:
raise Exception(
"enumerate can only be reversed if iterable to
enumerate can be reversed and has defined length."
) from e
try:
l = len(self.iterable)
except Exception as e:
raise Exception(
"enumerate can only be reversed if iterable to
enumerate can be reversed and has defined length."
) from e
indexes = range(l-1, -1, -1)
for i, e in zip(indexes, ri):
yield i, e
for i, c in reversed(reversible_enumerate("Hello World")):
print(i, c)
for i, c in reversed(reversible_enumerate([11, 22, 33])):
print(i, c)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/NDDKDUDHE3J7SR7IPO3QXCVFSGE4BAV4/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: reversed enumerate
Re: st...@pearwood.info > It isn't really well-defined, since enumerate can operate on infinite > iterators, and you cannot reverse an infinite stream. It is well defined on any iterable that itself is reversible and has defined length. For standard types that's lists, strings, dictionary iterators and some of the collections (deque, dictionary-based counter). Re: storch...@gmail.com > Could you please provide evidence that this feature would be quite > useful? How much usecases can you find in the stdlib? I needed this while solving a CS problem https://leetcode.com/problems/container-with-most-water/, which, I agree, does not tell how useful it is in production projects. Couldn't find any usecase in stdlib. I could find a usecase in scikit-learn: https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/pipeline.py#L639 in case file gets changed, here is a copy: for i in reversed(range(len(estimators))): name = names[i] if name in namecount: names[i] += "-%d" % namecount[name] namecount[name] -= 1 It can indeed be that one doesn't have to reverse enumeration very often in practice. It can also be that people just express their ideas differently knowing it doesn't work in straight-forward way. Best Regards, -- Ilya Kamenshchikov ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/W26NX33MDJSDKKEEOWWYFMZN6UKTJUOK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Python-Dev Digest, Vol 201, Issue 1
One potentially serious question: what should `enumerate.__reversed__`
> do when given a starting value?
> reversed(enumerate('abc', 1))
> Should that yield...?
> # treat the start value as a start value
> (1, 'c'), (0, 'b'), (-1, 'a')
> # treat the start value as an end value
> (3, 'c'), (2, 'b'), (1, 'a')
> Something else?
> My preference would be to treat the starting value as an ending value.
My idea is largely that as long as any iterable has defined length and
order, it should be reversible, and this should yield the same as if I did
this reverse countdown with some pointer.
It just so happens that `enumerate` depends fully in this regard on the
iterable it enumerates, if it has defined length, then enumerate will also
have defined length. Maybe enumerate should also start defining `len`
whenever underlying iterable supports it.
Regarding the starting value, I think it is important to keep symmetry:
enumerate('abc', 1) is [ (1, 'a'), (2, 'b'), (3, 'c') ], therefore
reversed(enumerate('abc', 1)) should be [(3, 'c'), (2, 'b'), (1, 'a')]
Best Regards,
--
Ilya Kamen
On Thu, Apr 2, 2020 at 5:55 AM wrote:
> Send Python-Dev mailing list submissions to
> python-dev@python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> or, via email, send a message with subject or body 'help' to
> python-dev-requ...@python.org
>
> You can reach the person managing the list at
> python-dev-ow...@python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Python-Dev digest..."
>
> Today's Topics:
>
>1. Re: reversed enumerate (Steven D'Aprano)
>2. Re: [Python-ideas] Re: reversed enumerate (Andrew Barnert)
>3. Re: [Python-ideas] Re: reversed enumerate (Andrew Barnert)
>
>
> --
>
> Date: Thu, 2 Apr 2020 13:20:02 +1100
> From: Steven D'Aprano
> Subject: [Python-Dev] Re: reversed enumerate
> To: python-dev@python.org
> Cc: python-id...@python.org
> Message-ID: <20200402022000.gc16...@ando.pearwood.info>
> Content-Type: text/plain; charset=us-ascii
>
> Hi Ilya,
>
> I'm not sure that this mailing list (Python-Dev) is the right place for
> this discussion, I think that Python-Ideas (CCed) is the correct place.
>
> For the benefit of Python-Ideas, I have left your entire post below, to
> establish context.
>
> [Ilya]
> > I needed reversed(enumerate(x: list)) in my code, and have discovered
> > that it wound't work. This is disappointing because operation is well
> > defined.
>
> It isn't really well-defined, since enumerate can operate on infinite
> iterators, and you cannot reverse an infinite stream. Consider:
>
> def values():
> while True:
> yield random.random()
>
> a, b = reversed(enumerate(values())
>
> What should the first pair of (a, b) be?
>
> However, having said that, I think that your idea is not unreasonable.
> `enumerate(it)` in the most general case isn't reversable, but if `it`
> is reversable and sized, there's no reason why `enumerate(it)` shouldn't
> be too.
>
> My personal opinion is that this is a fairly obvious and straightforward
> enhancement, one which (hopefully!) shouldn't require much, if any,
> debate. I don't think we need a new class for this, I think enhancing
> enumerate to be reversable if its underlying iterator is reversable
> makes good sense.
>
> But if you can show some concrete use-cases, especially one or two from
> the standard library, that would help your case. Or some other languages
> which offer this functionality as standard.
>
> On the other hand, I think that there is a fairly lightweight work
> around. Define a helper function:
>
> def countdown(n):
> while True:
> yield n
> n -= 1
>
> then call it like this:
>
> # reversed(enumerate(seq))
> zip(countdown(len(seq)-1), reversed(seq)))
>
> So it isn't terribly hard to work around this. But I agree that it would
> be nice if enumerate encapsulated this for the caller.
>
> One potentially serious question: what should `enumerate.__reversed__`
> do when given a starting value?
>
> reversed(enumerate('abc', 1))
>
> Should that yield...?
>
> # treat the start value as a start value
> (1, 'c'), (0, 'b'), (-1, &#
