Re: [Python-Dev] Another case for frozendict
On Wed, Jul 16, 2014 at 7:47 AM, R. David Murray wrote:
> After I hit send on my previous message, I thought more about your
> example. One of the issues here is that modifying the dict breaks an
> invariant of the API. I have a similar situation in the email module,
> and I used the same solution you did in regex: always return a new dict.
> It would be nice to be able to return a frozendict instead of having the
> overhead of building a new dict on each call. That by itself might not
> be enough reason. But, if the user wants to use the data in modified form
> elsewhere, they would then have to construct a new regular dict out of it,
> making the decision to vary the data from what matches the state of the
> object it came from an explicit one. That seems to fit the Python zen
> ("explicit is better than implicit").
>
> So I'm changing my mind, and do consider this a valid use case, even
> absent the crash.
+1
A simple implementation is pretty straight-forward:
class FrozenDict(Mapping):
def __init__(self, *args, **kwargs):
self._map = dict(*args, **kwargs)
self._hash = ...
def __hash__(self):
return self._hash
def __len__(self):
return len(self._map)
def __iter__(self):
yield from self._map
def __getitem__(self, key):
return self._map[key]
This is actually something I've used before on a number of occasions.
Having it in the stdlib would be nice (though that alone is not
sufficient for inclusion :)). If there is a valid use case for a
frozen dict type in other stdlib modules, I'd consider that a pretty
good justification for adding it.
Incidentally, collections.abc.Mapping is the only one of the 6
container ABCs that does not have a concrete implementation (not
counting types.MappingProxyType which is only a proxy).
-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github
On Sun, Nov 30, 2014 at 6:25 PM, Donald Stufft wrote: >The technical benefits mostly come from Github generally being a higher > quality product than it’s competitors, both FOSS and not. Here's a solution to allow contribution via PR while not requiring anything to switch VCS or hosting: 1. Set up mirrors of a desired repo on any hosting providers we choose. 2. Set up a webhook for PRs that automatically creates/re-uses a tracker ticket with the diff from the PR. The workflow does not change for the committer, but it gets easier to contribute. I did something like this for juju (https://github.com/juju/juju) when we switched to github, weren't satisfied with their code review tool, and switched to something else. We have a web hook that automatically creates a review request for new PRs and updates the review request when the PR gets updated. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github
On Sun, Nov 30, 2014 at 6:40 PM, Donald Stufft wrote: > I’m not sure if it got lost in the discussion or if it was purposely left > out. However I did come up with another idea, where we enable people to make > PRs against these repositories with PR integration within roundup. Using the > fact that it’s trivial to turn a PR into a patch core contributors (and the > “single source of truth”) for the repositories can remain Mercurial with > core contributors needing to download a .patch file from Github instead of a > .patch from from Roundup. This could allow non-committers to use git if they > want, including PRs but without moving things around. Hah. I just had a similar idea. > > The obvious cost is that since the committer side of things is still using > the existing tooling there’s no “Merge button” or the other committer > benefits of Github, it would strictly be enabling people who aren’t > committing directly to the repository to use git and Github. This is not an added cost. It's just the status quo and something that can be addressed separately. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github
On Sun, Nov 30, 2014 at 6:44 PM, Donald Stufft wrote: > Yea this is essentially what I meant. We already have “unofficial” mirrors > for PEPs and CPython itself on Github that are updated a few times a day. > It wouldn’t be very difficult I think to make them official mirrors and > update them immediately after a push. > > Then just some integration with Roundup would enable people to send PRs > on Github. Exactly. In my mind this eliminates almost all the controversy in this discussion. The question of switching hosting provider or DVCS becomes something to discuss separately (and probably just drop since at that point it doesn't improve much over the status quo). Of course, it does not address the more pressing concern of how to get contributions landed more quickly/steadily. :( However, that is largely what Nick addresses in PEP 462. :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github
On Tue, Dec 2, 2014 at 6:24 AM, Nick Coghlan wrote: > P.S. I'll also bring up some of the RFEs raised in this discussion > around making it possible for folks to submit pull requests via > GitHub/BitBucket, even if the master repositories are hosted on PSF > infrastructure. In case it helps with any GH/BB-to-roundup/reitveld integration we might do, I've already done something similar for GH-to-reviewboard at work. All the code is on-line: https://bitbucket.org/ericsnowcurrently/rb_webhooks_extension -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github
On Tue, Dec 2, 2014 at 9:50 AM, Brett Cannon wrote: > So I was waiting for Nick to say what he wanted to do for the peps repo > since I view it as I get 2/3 of the choices and he gets the other third. > > The way I view it, the options are: > > Move to GitHub > Move to Bitbucket > Improve our current tooling (either through new hosting setup and/or adding > first-world support for downloading PRs from GitHub/Bitbucket) I'd argue that option #3 here is somewhat orthogonal to switching hosting. It makes sense regardless unless we plan on ditching roundup and reitveld (to which I'd be opposed). > > Regardless of what we do, I think we should graduate the mirrors on GitHub > and Bitbucket to "official" -- for the proposed repos and cpython -- and get > their repos updating per-push instead of as a cron job. I also think we > should also flip on any CI we can (e.g. turn on Travis for GitHub along with > coveralls support using coverage.py's encodings trick). This will get us the > most accessible repo backups as well as the widest tool coverage for > contributors to assist them in their contributions (heck, even if we just > get regular coverage reports for Python that would be a great win out of all > of this). +1 to all of this. Doing this would allow us to move forward with GH/BB-roundup/reitveld integration (option #3) sooner rather than later, regardless of moving to other hosting. > So do people want PEPs or experimentation first? +1 to PEPs. It's basically already happening. I'd like to see where 474/481/etc. end up, particularly with what Nick brought up earlier. Furthermore, I'm not sure how effectively we can experiment when it comes to moving hosting. There's overhead involved that biases the outcome and in part contributes to the momentum of the initial experimental conditions. I doubt any external solution is going to prove drastically better than another, making it harder to justify the effort to move yet again. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] My thinking about the development process
be affected, and also got a test coverage report > 7. Core developer does a code review > 8. Contributor updates their code based on the code review and the updated > patch gets pulled by bugs.python.org automatically and CI runs again > 9. Once the patch is acceptable and assuming the patch applies cleanly to > all versions to commit to, the core developer clicks a "Commit" button, > fills in a commit message and NEWS entry, and everything gets committed (if > the patch can't apply cleanly then the core developer does it the > old-fashion way, or maybe auto-generate a new PR which can be manually > touched up so it does apply cleanly?) 6-9 sounds a lot like PEP 462. :) This seems like the part the would win us the most. > > Basically the ideal scenario lets contributors use whatever tools and > platforms that they want and provides as much automated support as possible > to make sure their code is tip-top before and during code review while core > developers can review and commit patches so easily that they can do their > job from a beach with a tablet and some WiFi. Sign me up! > > ## Where the current proposed solutions seem to fall short > ### GitHub/Bitbucket > Basically GitHub/Bitbucket is a win for contributors but doesn't buy core > developers that much. GitHub/Bitbucket gives contributors the easy cloning, > drive-by patches, CI, and PRs. Core developers get a code review tool -- I'm > counting Rietveld as deprecated after Guido's comments about the code's > maintenance issues -- and push-button commits **only for single branch > changes**. But for any patch that crosses branches we don't really gain > anything. At best core developers tell a contributor "please send your PR > against 3.4", push-button merge it, update a local clone, merge from 3.4 to > default, do the usual stuff, commit, and then push; that still keeps me off > the beach, though, so that doesn't get us the whole way. This will probably be one of the trickiest parts. > You could force > people to submit two PRs, but I don't see that flying. Maybe some tool could > be written that automatically handles the merge/commit across branches once > the initial PR is in? Or automatically create a PR that core developers can > touch up as necessary and then accept that as well? Regardless, some > solution is necessary to handle branch-crossing PRs. > > As for GitHub vs. Bitbucket, I personally don't care. I like GitHub's > interface more, but that's personal taste. I like hg more than git, but > that's also personal taste (and I consider a transition from hg to git a > hassle but not a deal-breaker but also not a win). It is unfortunate, > though, that under this scenario we would have to choose only one platform. > > It's also unfortunate both are closed-source, but that's not a deal-breaker, > just a knock against if the decision is close. > > ### Our own infrastructure > The shortcoming here is the need for developers, developers, developers! > Everything outlined in the ideal scenario is totally doable on our own > infrastructure with enough code and time (donated/paid-for infrastructure > shouldn't be an issue). But historically that code and time has not > materialized. Our code review tool is a fork that probably should be > replaced as only Martin von Löwis can maintain it. Basically Ezio Melotti > maintains the issue tracker's code. Doing something about those two tools is something to consider. Would it be out of scope for this discussion or any resulting PEPS? I have opinions here, but I'd rather not sidetrack the discussion. > We don't exactly have a ton of people > constantly going "I'm so bored because everything for Python's development > infrastructure gets sorted so quickly!" A perfect example is that R. David > Murray came up with a nice update for our workflow after PyCon but then ran > out of time after mostly defining it and nothing ever became of it (maybe we > can rectify that at PyCon?). Eric Snow has pointed out how he has written > similar code for pulling PRs from I think GitHub to another code review > tool, but that doesn't magically make it work in our infrastructure or get > someone to write it and help maintain it (no offense, Eric). None taken. I was thinking the same thing when I wrote that. :) > > IOW our infrastructure can do anything, but it can't run on hopes and > dreams. Commitments from many people to making this happen by a certain > deadline will be needed so as to not allow it to drag on forever. People > would also have to commit to continued maintenance to make this viable > long-term. > > # Next steps > I'm thinking first dra
Re: [Python-Dev] datetime nanosecond support (ctd?)
On Wed, Dec 17, 2014 at 7:52 PM, Matthieu Bec wrote: > > > Attached patch defines a new type struct_timespec for the time module. A new > capsule exports the type along with to/from converters - opening a bridge > for C, and for example the datetime module. I'd recommend opening a new issue in the bug tracker (bugs.python.org) and attach the patch there. Attaching it to an email is a good way for it to get lost and forgotten. :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 468 (Ordered kwargs)
On Sat, Jan 24, 2015 at 3:50 AM, Maciej Fijalkowski wrote: > I would like to point out that we implemented rhettingers idea in PyPy > that makes all the dicts ordered by default and we don't have any > adverse performance effects (in fact, there is quite significant > memory saving coming from it). The measurments on CPython could be > different, but in principle OrderedDict can be implemented as > efficiently as normal dict. > > Writeup: > http://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html > > Previous discussion: > https://mail.python.org/pipermail/python-dev/2012-December/123028.html Cool. I'll add a note to the PEP. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import Fails in setup.py On Android
On Mon, Feb 2, 2015 at 12:36 PM, Cyd Haselton wrote: > After fixing a segfault issue (many thanks Ryan) I'm back to the same issue > I was having with Python 2.7.8; the newly built python throws an undefined > reference to dlopen when running setup.py...specifically when importing > just-built extensions > > I've managed to narrow the problem down to the following line: > > importlib._bootstrap._SpecMethods(spec).load() That call is where modules are created (and executed) via the loader designated for the module by the import machinery. So a problem here may simply reflect a problem with the loader. Which module is being imported when you run into trouble? Is it a C-extension module? Also keep in mind that basically everything in importlib._bootstrap is frozen into your Python binary when you build Python (or should be). So if you are seeing any errors related to something missing or broken in importlib._bootstrap, it could be related to the build step where importlib gets frozen. Either way, more info (e.g. a traceback) would be great if you need more help. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Building on Windows - importlib.h changed
On Fri, Feb 13, 2015 at 3:49 PM, Zachary Ware wrote: > On Fri, Feb 13, 2015 at 4:09 PM, Paul Moore wrote: >> I'm working on a patch for the Python launcher. I built Python >> (current tip, on MS Windows, with VS 2015), and I've just noticed that >> hg status shows: >> hg status -mard >> M Doc\using\windows.rst >> M PC\launcher.c >> M Python\importlib.h >> >> I didn't change importlib.h, and I don't see that the changes I did >> make would affect importlib. I presume I'm OK *not* including the >> importlib.h change in my patch? > > Yes, importlib.h changes should never be included in a patch (it would Unless they should. :) E.g. you modified importlib/_bootstrap.py, the marshal format, bytecodes, etc. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 9:34 AM, Brett Cannon wrote: > Not specifying the optimization level when it is at 0 > - > > It has been suggested that for the common case of when the > optimizations are at level 0 that the entire part of the file name > relating to the optimization level be left out. This would allow for > file names of ``.pyc`` files to go unchanged, potentially leading to > less backwards-compatibility issues (although Python 3.5 introduces a > new magic number for bytecode so all bytecode files will have to be > regenerated regardless of the outcome of this PEP). > > It would also allow a potentially redundant bit of information to be > left out of the file name if an implementation of Python did not > allow for optimizing bytecode. This would only occur, though, if the > interpreter didn't support ``-O`` **and** didn't implement the ast > module, else users could implement their own optimizations. The presence of the "opt-0" part in the filename could imply that there are other supported optimization levels, even when there aren't. So leaving that out when optimizations aren't supported may be a good idea. Perhaps add sys.implementation.supports_optimization or something like that? Then only leave "opt-0" off if the implementation does not support any optimization. --eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Final call for PEP 488: eliminating PYO files
On Mar 21, 2015 7:44 AM, "Brett Cannon" wrote: > > Thanks! PEP 488 is now marked as accepted. I expect I will have PEP 488 implemented before the PyCon sprints are over (work will be tracked in http://bugs.python.org/issue23731). > > On Fri, Mar 20, 2015 at 8:06 PM Guido van Rossum wrote: >> >> Awesome, that's what I was hoping. Accepted! Congrats and thank you very much for writing the PEP and guiding the discussion. Congratulations Brett! This is a welcome change. I'll be sure to give you a review. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)
On Fri, Apr 3, 2015 at 6:44 AM, Martin Teichmann wrote: >> When I first wrote PEP 422 I was of the view that "Python 2 allows >> class definition postprocessing injection, we should allow it in >> Python 3 as well". I've since changed my view to "Having to declare >> post-processing of a class definition up front as a decorator, base >> class or metaclass is a good thing for readability, as otherwise >> there's nothing obvious when reading a class definition that tells you >> whether or not postprocessing may happen, so you have to assume its >> possible for *every* class definition". > > Nick, I couldn't agree more with you, yet I think PJ actually brought > up a very interesting point. Post-processing is a very common thing > these days, and has been re-written so many times that I think it is > about time that something like it should be in the standard library. > > I'm less thinking about decorated methods, more about descriptors. > They always have the problem that they don't know which attribute they > belong to, so every author of a framework that defines descriptors > writes a metaclass which goes through all the descriptors and tells > them their attribute name. > > I propose to have some metaclass in the standard library that does > that. I think it would fit nicely in my metaclass module proposed in > PEP 487. > > It would basically do the following: > > class Metaclass(type): > def __init__(self, name, bases, dict): > super().__init__(name, bases, dict) > for k, v in dict.items(): > if hasattr(v, "__post_process__"): > v.__post_process__(k, self) > > So each descriptor could define a __post_process__ hook that tells > it the attribute name and also the class it belongs to. This would for > sure also work for decorated methods. > > This should mature on PyPI, then introduced into the standard library, > and if demand is really that high, maybe even be introduced into > type.__init__. It should be noted that this can also be easily written > as a PEP 487 class using __subclass_init__, I just used the classical > metaclass notion as I guess people are more used to that. > > This proposal can actually be seen as an extension to the __class__ > and super() mechanism of normal methods: methods currently have the > priviledge to know which classes they are defined in, while descriptors > don't. So we could unify all this by giving functions a __post_process__ > method which sets the __class__ in the function body. This is about the > same as what happened when functions got a __get__ method to turn > them into object methods. I've felt for a long time that it would be helpful in some situations to have a reverse descriptor protocol. What you are describing it a strict subset of that concept (which is fine). It may be worth considering a method name that would also be used for that more generic reverse descriptor, rather than having 2 names for the same thing. Even if such a protocol never materializes, the name borrowed here would still be informative. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)
On Sat, Apr 4, 2015 at 6:40 PM, Greg Ewing wrote: > Eric Snow wrote: >> >> I've felt for a long time that it would be helpful in some situations >> to have a reverse descriptor protocol. > > Can you elaborate on what you mean by that? Sure. It's more python-ideas territory (I posted about it a few years back). The idea is to allow an object the opportunity to handle being bound to a name. So if a type defines __bound__ (or similar) then it will be called for the instance being bound to a name: type(obj).__bound__(obj, name). There would also be an __unbound__, but that is less of an issue here. I'm still not convinced such a reverse descriptor protocol is practical as a general approach (though no less than the current descriptor protocol). However, I do see a distinct correspondence with the "__post_process__" method being considered here. So I wanted to point out the possibility of a more general approach for the sake of its impact on the name and semantics of a descriptor post-process method. While I expect __post_process__ would be called at a different place than __bound__, the responsibility of both would still be identical. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)
On Fri, Apr 3, 2015 at 6:44 AM, Martin Teichmann wrote: > Nick, I couldn't agree more with you, yet I think PJ actually brought > up a very interesting point. Post-processing is a very common thing > these days, and has been re-written so many times that I think it is > about time that something like it should be in the standard library. Here's another approach that would help. Support a mechanism for inheriting class decorators. Classes would have an attribute like __subclass_decorators__ that would hold a tuple of all the inherited decorators. They would be applied, in order, in __build_class__ before any explicit decorators are. One way to accomplish this is with a meta-decorator, e.g. "classutil.inherited_decorator". You would decorate a class decorator with it: def inherited_decorator(deco): def new_deco(cls): cls = deco(cls) try: inherited = cls.__subclass_decorators__ except AttributeError: cls.__subclass_decorators__ = (deco,) else: cls.__subclass_decorators__ = inherited + (deco,) return cls @inherited_decorator def register(cls): registry.add(cls) return cls @register class X: ... The downside to the meta-decorator is that is isn't apparent when the class decorator is used that it will be inherited. It could also be used directly to make it more apparent: def register(cls): registry.add(cls) return cls @inherited(register) class X: ... However, that doesn't read well. Syntax would be better, but is a harder sell and a little grittier: def register(cls): registry.add(cls) return cls @@register class X: -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Should instances really be able to dictate the "existence" of special methods?
_PyObject_LookupSpecial is used in place of obj.__getattribute__ for looking up special methods. (As far as I recall it is not exposed in the stdlib, e.g. inspect.getattr_special.) Correct me if I'm wrong (please!), but there are two key reasons: * access to special methods in spite of obj.__getattribute__ * speed While _PyObject_LookupSpecial does not do lookup on obj.__dict__ or call obj.__getattr__, it does resolve descriptors. This is important particularly since special methods will nearly always be some kind of descriptor. However, one consequence of this is that instances can influence whether or not some capability, as relates to the special method, is available. This is accomplished by the descriptor's __get__ raising AttributeError. My question is: was this intentional? Considering the obscure bugs that can result (e.g. where did the AttributeError come from?), it seems more likely that it is an oversight of an obscure corner case. If that is the case then it would be nice if we could fix _PyObject_LookupSpecial to chain any AttributeError coming from descr.__get__ into a RuntimeError. However, I doubt we could get away with that at this point. Also, while it may be appropriate in general to allow instances to dictate the availability of attributes/methods (e.g. through __getattribute__, __getattr__, or descriptors), I'm not convinced it makes sense for special methods. We are already explicitly disconnecting special methods from instances in _PyObject_LookupSpecial (except in the case of descriptors...). -eric p.s. I also find it a bit strange that instances have any say at all in which methods (i.e. behavior) are *available*. Certainly instances influence behavior, but I always find their impact on method availability to be surprising. Conceptually for me instances are all about state and classes about behavior (driven by state). However, it is very rarely that I run into code that takes advantage of the opportunity. :) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should instances really be able to dictate the "existence" of special methods?
On Mon, Apr 20, 2015 at 2:20 AM, Guido van Rossum wrote: > (I suppose this new thread is a result of some research you did regarding > the thread complaining about callable()?) Yep. :) > On Sun, Apr 19, 2015 at 4:03 PM, Eric Snow > wrote: >> >> _PyObject_LookupSpecial is used in place of obj.__getattribute__ for >> looking up special methods. (As far as I recall it is not exposed in >> the stdlib, e.g. inspect.getattr_special.) Correct me if I'm wrong >> (please!), but there are two key reasons: >> >> * access to special methods in spite of obj.__getattribute__ >> * speed > > Good question! I don't have an easy pointer to the original discussion, but > I do recall that this was introduced in response to some issues with the > original behavior, which looked up dunder methods on the instance and relied > on the general mechanism for binding it to the instance. I don't think the > reason was to circumvent __getattribute__, but your second bullet rings > true: for every +, -, * etc. there would be a (usually failing) lookup in > the instance dict before searching the class dict and then the base classes > etc. There may also have been some confusion where people would e.g. assign > a function of two arguments to x.__add__ and would be disappointed to find > out it was called with only one argument. I think there were some folks who > wanted to fix this by somehow "binding" such calls to the instance (since > there's no easy way otherwise to get the first argument) but I thought the > use case was sufficiently odd that it was better to avoid it altogether. > > In any case, it's not just an optimization -- it's an intentional (though > obscure) feature. Thanks for explaining. >> While _PyObject_LookupSpecial does not do lookup on obj.__dict__ or >> call obj.__getattr__, it does resolve descriptors. This is important >> particularly since special methods will nearly always be some kind of >> descriptor. However, one consequence of this is that instances can >> influence whether or not some capability, as relates to the special >> method, is available. This is accomplished by the descriptor's >> __get__ raising AttributeError. > > Well, it's not really the instance that raises AttributeError -- it's the > descriptor, which is a separate class (usually but not always a builtin > class, such as property or classmethod). And the descriptor is "owned" by > the class. Sure. That's what I meant. :) The instance can influence what the descriptor returns. >> My question is: was this intentional? Considering the obscure bugs >> that can result (e.g. where did the AttributeError come from?), it >> seems more likely that it is an oversight of an obscure corner case. > > I'm not sure what you would do to avoid this. You can't very well declare > that a descriptor's __get__ method must not raise AttributeError. It could > be implemented in Python and it could just hit a bug or something. Right. And such a bug will be misinterpreted and obscured and hard to unravel. I ran into this a while back with pickle (which still does lookup for special methods on the instance). Ultimately it's the same old problem of not knowing how to interpret an exception that may have bubbled up from some other layer. Like I said, I don't think there's anything to be done about it either way. I just got the feeling that in the case of special methods, the descriptor part of lookup should not expect AttributeError to come out of the getter. So I wanted to see if my intuition was correct even if the point is essentially irrelevant. :) At this point, though, I think my intuition wasn't quite right, though I still don't think a descriptor's getter is the right place to raise AttributeError. > But > perhaps I'm misunderstanding the situation you're describing? > >> >> If that is the case then it would be nice if we could fix >> _PyObject_LookupSpecial to chain any AttributeError coming from >> descr.__get__ into a RuntimeError. However, I doubt we could get away >> with that at this point. > > Yeah, I think that ship has sailed. It also seems to be hardly worth trying > to control "double fault" situations like this. (It's not really a double > fault, but it reeks like it.) > > I wonder if maybe you're feeling inspired by PEP 479? But that's really a > much more special case, and I don't really want to start down a whole > cascade of trying to "fix" all cases where an AttributeError could be raised > due to a problem in the user's lookup code. Nah. It isn't about fixing all the ca
Re: [Python-Dev] Should instances really be able to dictate the "existence" of special methods?
On Mon, Apr 20, 2015 at 4:37 AM, Guido van Rossum wrote: > OK, so I think there isn't anything we can or should do here. Yes, it's > possible that type(x).__add__ succeeds but x.__add__ fails. That's how you > spell descriptor. :-) You could also use a random number generator in > __getattribube__... Cool. That's pretty much what I figured. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Type hints -- a mediocre programmer's reaction
On Mon, Apr 20, 2015 at 1:35 PM, Łukasz Langa wrote: > Yeah, so agreed, this is pretty busy. For such cases, reformatting makes > it less confusing (see: Screenshot 1). > > > While it helps, this sort of best-practice is still unsettled (and apparently not obvious). In the short term it would make more sense to recommend using stub files for all the reason Harry enumerated. Once the best practices are nailed down through experience with stub files, then we can make recommendations regarding inline type hints. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 554 v2 (new "interpreters" module)
I've updated the PEP in response to some excellent feedback. Thanks to all who helped. Most notably, I've added a basic object passing mechanism. I've included the PEP below. Please let me know what you think. Thanks! -eric PEP: 554 Title: Multiple Interpreters in the Stdlib Author: Eric Snow Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2017-09-05 Python-Version: 3.7 Post-History: Abstract This proposal introduces the stdlib ``interpreters`` module. It exposes the basic functionality of subinterpreters that already exists in the C-API. Each subinterpreter runs with its own state (see ``Interpreter Isolation`` below). The module will be "provisional", as described by PEP 411. Rationale = Running code in multiple interpreters provides a useful level of isolation within the same process. This can be leveraged in number of ways. Furthermore, subinterpreters provide a well-defined framework in which such isolation may extended. CPython has supported subinterpreters, with increasing levels of support, since version 1.5. While the feature has the potential to be a powerful tool, subinterpreters have suffered from neglect because they are not available directly from Python. Exposing the existing functionality in the stdlib will help reverse the situation. This proposal is focused on enabling the fundamental capability of multiple isolated interpreters in the same Python process. This is a new area for Python so there is relative uncertainly about the best tools to provide as companions to subinterpreters. Thus we minimize the functionality we add in the proposal as much as possible. Concerns * "subinterpreters are not worth the trouble" Some have argued that subinterpreters do not add sufficient benefit to justify making them an official part of Python. Adding features to the language (or stdlib) has a cost in increasing the size of the language. So it must pay for itself. In this case, subinterpreters provide a novel concurrency model focused on isolated threads of execution. Furthermore, they present an opportunity for changes in CPython that will allow simulateous use of multiple CPU cores (currently prevented by the GIL). Alternatives to subinterpreters include threading, async, and multiprocessing. Threading is limited by the GIL and async isn't the right solution for every problem (nor for every person). Multiprocessing is likewise valuable in some but not all situations. Direct IPC (rather than via the multiprocessing module) provides similar benefits but with the same caveat. Notably, subinterpreters are not intended as a replacement for any of the above. Certainly they overlap in some areas, but the benefits of subinterpreters include isolation and (potentially) performance. In particular, subinterpreters provide a direct route to an alternate concurrency model (e.g. CSP) which has found success elsewhere and will appeal to some Python users. That is the core value that the ``interpreters`` module will provide. * "stdlib support for subinterpreters adds extra burden on C extension authors" In the ``Interpreter Isolation`` section below we identify ways in which isolation in CPython's subinterpreters is incomplete. Most notable is extension modules that use C globals to store internal state. PEP 3121 and PEP 489 provide a solution for most of the problem, but one still remains. [petr-c-ext]_ Until that is resolved, C extension authors will face extra difficulty to support subinterpreters. Consequently, projects that publish extension modules may face an increased maintenance burden as their users start using subinterpreters, where their modules may break. This situation is limited to modules that use C globals (or use libraries that use C globals) to store internal state. Ultimately this comes down to a question of how often it will be a problem in practice: how many projects would be affected, how often their users will be affected, what the additional maintenance burden will be for projects, and what the overall benefit of subinterpreters is to offset those costs. The position of this PEP is that the actual extra maintenance burden will be small and well below the threshold at which subinterpreters are worth it. Proposal The ``interpreters`` module will be added to the stdlib. It will provide a high-level interface to subinterpreters and wrap the low-level ``_interpreters`` module. The proposed API is inspired by the ``threading`` module. The module provides the following functions: ``list()``:: Return a list of all existing interpreters. ``get_current()``:: Return the currently running interpreter. ``get_main()``:: Return the main interpreter. ``create()``:: Initialize a new Python interpreter and return it. The interpreter will be created in the current thre
Re: [Python-Dev] PEP 554 v2 (new "interpreters" module)
On Fri, Sep 8, 2017 at 4:28 PM, Stefan Krah wrote: > The most promising model to me is to put *all* globals in a tls structure > and cache the whole structure. Extrapolating from my experiences with the > context, this might have a slowdown of "only" 4%. Yeah, this is actually something I've been exploring and was one of the motivations for consolidating the C globals. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v2 (new "interpreters" module)
On Sat, Sep 9, 2017 at 1:04 AM, Paul Moore wrote: > On 9 September 2017 at 00:04, Eric Snow wrote: >>add_recv_fifo(name=None): >>add_send_fifo(name=None): > > Personally, I *always* read these names backwards - from the POV of > the caller. So when I see "add_send_fifo", I then expect to be able to > send stuff on the returned FIFO (i.e., I get a writer back). But > that's not how it works. > > I may be alone in this - I've had similar problems in the past with > how people name pipes, for example - but I thought I'd raise it while > there's still a possibility that it's not just me and the names can be > changed. Yeah, those names are bugging me too. I'm stewing over possible alternatives. It's the one piece of the PEP that I want to address before I re-post to the list. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v2 (new "interpreters" module)
On Fri, Sep 8, 2017 at 8:54 PM, Michel Desmoulin wrote: > Le 09/09/2017 à 01:28, Stefan Krah a écrit : >> Still, the argument "who uses subinterpreters?" of course still remains. > > For now, nobody. But if we expose it and web frameworks manage to create > workers as fast as multiprocessing and as cheap as threading, you will > find a lot of people starting to want to use it. Note that subinterpreters share the GIL currently, and the objects you can pass between interpreters will be quite restricted initially. However, the ultimate goal is to stop sharing the GIL between interpreters and to broaden the types that can be passed between interpreters. Regardless, the isolation inherent to subinterpreters provides benefits immediately. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v2 (new "interpreters" module)
On Sat, Sep 9, 2017 at 4:45 AM, Antoine Pitrou wrote: > How does the other interpreter get the FIFO "tied" to it? > Is it `get_current().get_fifo(name)`? Something else? Yep, that's it. I've added some examples to the PEP to illustrate. That said, I'm re-working the method names since they confuse me too. :) > How does it get to learn the *name*? The main way is that the programmer hard-codes it. I'd expect close proximity between the code that adds the FIFO and the code that get's run in the subinterpreter. > > Why are FIFOs unidirectional? Does it really make them significantly > cheaper? With bidirectional pipes, the whole recv/send confusion would > be avoided. > > As a point of comparison, for default for `multiprocessing.Pipe()` is > to create a bidirectional pipe (*), yet I'm sure a multiprocessing Pipe > has more overhead than an in-process FIFO. The choice wasn't about cost or performance. In part it's a result of my experience with Go's channels. Bidirectional channels are problematic in certain situations, including when it relates to which goroutine is responsible for managing the channel's lifetime. Also, bidirectional channels require both the reader and the writer (of the code) to keep track of the two roles that the channel plays. It's easy to forget about one or the other, and accidentally do the wrong thing. Unidirectional channels resolve all those problems. Granted, the FIFOs from the PEP aren't the same as Go's channels. They are more rudimentary and they lack the support provided by a static compiler (e.g. deadlock detection), but my intention is that they provide all the functionality we need as a building block. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v2 (new "interpreters" module)
On Sat, Sep 9, 2017 at 5:05 AM, Antoine Pitrou wrote: > On Fri, 8 Sep 2017 16:04:27 -0700, Eric Snow > wrote: >> ``list()``:: > > It's called ``enumerate()`` in the threading module. Not sure there's > a point in choosing a different name here. Yeah, in the first version of the PEP it was called "enumerate()". I changed it to "list()" at Raymond's recommendation. The reasoning is that it's less confusing to most people that way. TBH, I'd rather leave it "list()", but could be swayed. Perhaps it would be enough for the PEP to not mention any relationship to "threading"? >> The current interpreter (which called ``run()``) will block >> until the subinterpreter finishes running the requested code. Any >> uncaught exception in that code will bubble up to the current >> interpreter. > > Why does it block? How is concurrency supposed to be achieved in that > model? It would be more flexible if run(code) returned an object that > can later be waited on. Something like... a Future :-) I expect this is more a problem with my description than with the feature. :) I've already re-written this bit to be more clear. It's not that the thread blocks. It's more like a function call, where the current frame is paused while the call is executed. Then it returns to the calling frame. Likewise the interpreter in the current thread gets swapped out with the target interpreter, where the code gets run, and then the original interpreter gets swapped back in. This is how you do it in the C-API and it made sense (to me) to do it the same way in Python. > > And why guarantee that it executes in the "current OS thread"? > I would say you don't want to specify where it executes exactly, as it > opens the door for more sophisticated implementations (such as > automatic assignment of subinterpreters inside a pool of threads). Again, I had explained this poorly in the PEP. The key thing here is that subinterpreters don't do anything special relative to threading. If you want to call "Interpreter.run()" in a thread then you stick it in a "threading.Thread". If you want to auto-assign to a pool of threads then you treat it like any other function you would auto-assign to a pool of threads. >>get_fifo(name): >>list_fifos(): > > If fifos are uniquely named, why not return a name->fifo mapping? I suppose we could. Then we could get rid of "get_fifo()" too. I'm still mulling over the right API for the FIFO parts of the PEP. > >> ``FIFOReader(name)``:: >> [...] > > I don't think the method naming choice is very adequate here. The API > model for the FIFO objects can either be a (threading or > multiprocessing) Queue or a multiprocessing Pipe. > > - if a Queue, then it should have a get() / put() pair of methods > - if a Pipe, then it should have a recv() / send() pair of methods > > Now, since Queues are multi-producer multi-consumer, while Pipes are > single-producer single-consumer (they aren't "synchronized"), the > better analogy seems to the multiprocessing Pipe here, so I would vote > for recv() / send(). > > But, in any case, definitely not a pop() / push() pair. Thanks for pointing that out. Prior art, FTW! I'll factor that in. > Has any thought been given to how FIFOs could integrate with async code > driven by an event loop (e.g. asyncio)? I think the model of executing > several asyncio (or Tornado) applications each in their own > subinterpreter may prove quite interesting to reconcile multi-core > concurrency with ease of programming. That would require the FIFOs to > be able to synchronize on something an event loop can wait on (probably > a file descriptor?). Personally I've given pretty much no thought to the relationship with async. TBH, the FIFO parts of the PEP were added only recently and haven't fully baked yet. I'd be interested more feedback on async relative to PEP, not just with the FIFO bits; my experience with async is pretty limited thus far. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v2 (new "interpreters" module)
On Sat, Sep 9, 2017 at 11:04 AM, Nathaniel Smith wrote: > This phrase "bubble up" here is doing a lot of work :-). Can you elaborate > on what you mean? The text now makes it seem like the exception will just > pass from one interpreter into another, but that seems impossible I've updated the PEP to clarify. > it'd mean sharing not just arbitrary user defined exception classes but full > frame objects... My current implementation does the right thing currently. However, it would probably struggle under a more tightly controlled inter-interpreter boundary. I'll take a look and also update the PEP. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 554 v3 (new interpreters module)
I've updated PEP 554 in response to feedback. (thanks all!) There are a few unresolved points (some of them added to the Open Questions section), but the current PEP has changed enough that I wanted to get it out there first. Notably changed: * the API relative to object passing has changed somewhat drastically (hopefully simpler and easier to understand), replacing "FIFO" with "channel" * added an examples section * added an open questions section * added a rejected ideas section * added more items to the deferred functionality section * the rationale section has moved down below the examples Please let me know what you think. I'm especially interested in feedback about the channels. Thanks! -eric PEP: 554 Title: Multiple Interpreters in the Stdlib Author: Eric Snow Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2017-09-05 Python-Version: 3.7 Post-History: Abstract CPython has supported subinterpreters, with increasing levels of support, since version 1.5. The feature has been available via the C-API. [c-api]_ Subinterpreters operate in `relative isolation from one another `_, which provides the basis for an `alternative concurrency model `_. This proposal introduces the stdlib ``interpreters`` module. The module will be `provisional `_. It exposes the basic functionality of subinterpreters already provided by the C-API. Proposal The ``interpreters`` module will be added to the stdlib. It will provide a high-level interface to subinterpreters and wrap the low-level ``_interpreters`` module. The proposed API is inspired by the ``threading`` module. See the `Examples`_ section for concrete usage and use cases. API for interpreters The module provides the following functions: ``list_all()``:: Return a list of all existing interpreters. ``get_current()``:: Return the currently running interpreter. ``create()``:: Initialize a new Python interpreter and return it. The interpreter will be created in the current thread and will remain idle until something is run in it. The interpreter may be used in any thread and will run in whichever thread calls ``interp.run()``. The module also provides the following class: ``Interpreter(id)``:: id: The interpreter's ID (read-only). is_running(): Return whether or not the interpreter is currently executing code. Calling this on the current interpreter will always return True. destroy(): Finalize and destroy the interpreter. This may not be called on an already running interpreter. Doing so results in a RuntimeError. run(source_str, /, **shared): Run the provided Python source code in the interpreter. Any keyword arguments are added to the interpreter's execution namespace. If any of the values are not supported for sharing between interpreters then RuntimeError gets raised. Currently only channels (see "create_channel()" below) are supported. This may not be called on an already running interpreter. Doing so results in a RuntimeError. A "run()" call is quite similar to any other function call. Once it completes, the code that called "run()" continues executing (in the original interpreter). Likewise, if there is any uncaught exception, it propagates into the code where "run()" was called. The big difference is that "run()" executes the code in an entirely different interpreter, with entirely separate state. The state of the current interpreter in the current OS thread is swapped out with the state of the target interpreter (the one that will execute the code). When the target finishes executing, the original interpreter gets swapped back in and its execution resumes. So calling "run()" will effectively cause the current Python thread to pause. Sometimes you won't want that pause, in which case you should make the "run()" call in another thread. To do so, add a function that calls "run()" and then run that function in a normal "threading.Thread". Note that interpreter's state is never reset, neither before "run()" executes the code nor after. Thus the interpreter state is preserved between calls to "run()". This includes "sys.modules", the "builtins" module, and the internal state of C extension modules. Also note that "run()" executes in the namespace of the "__main__" module, just like scripts, the REPL, "-m", and "-c". Just as the interpreter's state is not ever reset, the "__main__" module is never reset. Yo
Re: [Python-Dev] PEP 554 v3 (new interpreters module)
Thanks for the feedback, Antoine. Sorry for the delay; it's been a busy week for me. I just pushed an updated PEP to the repo. Once I've sorted out the question of passing bytes through channels I plan on posting the PEP to the list again for another round of discussion. In the meantime, I've replied below in-line. -eric On Mon, Sep 18, 2017 at 4:46 AM, Antoine Pitrou wrote: > First my high-level opinion about the PEP: the CSP model can probably > be already implemented using Queues. To me, the interesting promise of > subinterpreters is if they allow to remove the GIL while sharing memory > for big objects (such as Numpy arrays). This means the PEP should > probably focus on potential concurrency improvements rather than try to > faithfully follow the CSP model. Please elaborate. I'm interested in understanding what you mean here. Do you have some subinterpreter-based concurrency improvements in mind? What aspect of CSP is the PEP following too faithfully? >> ``list_all()``:: >> >>Return a list of all existing interpreters. > > See my naming proposal in the previous thread. Sorry, your previous comment slipped through the cracks. You suggested: As for the naming, let's make it both unconfusing and explicit? How about three functions: `all_interpreters()`, `running_interpreters()` and `idle_interpreters()`, for example? As to "all_interpreters()", I suppose it's the difference between "interpreters.all_interpreters()" and "interpreters.list_all()". To me the latter looks better. As to "running_interpreters()" and "idle_interpreters()", I'm not sure what the benefit would be. You can compose either list manually with a simple comprehension: [interp for interp in interpreters.list_all() if interp.is_running()] [interp for interp in interpreters.list_all() if not interp.is_running()] >>run(source_str, /, **shared): >> >> Run the provided Python source code in the interpreter. Any >> keyword arguments are added to the interpreter's execution >> namespace. > > "Execution namespace" specifically means the __main__ module in the > target interpreter, right? Right. It's explained in more detail a little further down and elsewhere in the PEP. I've updated the PEP to explicitly mention __main__ here too. >> If any of the values are not supported for sharing >> between interpreters then RuntimeError gets raised. Currently >> only channels (see "create_channel()" below) are supported. >> >> This may not be called on an already running interpreter. Doing >> so results in a RuntimeError. > > I would distinguish between both error cases: RuntimeError for calling > run() on an already running interpreter, ValueError for values which > are not supported for sharing. Good point. >> Likewise, if there is any uncaught >> exception, it propagates into the code where "run()" was called. > > That makes it a bit harder to differentiate with errors raised by run() > itself (see above), though how much of an annoyance this is remains > unclear. The more litigious implication, though, is that it forces the > interpreter to support migration of arbitrary objects from one > interpreter to another (since a traceback keeps all local variables > alive). Yeah, the proposal to propagate exceptions out of the subinterpreter is still rather weak. I've added some notes the the PEP about this open issue. >> The mechanism for passing objects between interpreters is through >> channels. A channel is a simplex FIFO similar to a pipe. The main >> difference is that channels can be associated with zero or more >> interpreters on either end. > > So it seems channels have become more complicated now? Is it important > to support multi-producer multi-consumer channels? To me it made the API simpler. The change did introduce the "close()" method, which I suppose could be confusing. However, I'm sure that in practice it won't be. In contrast, the FIFO/pipe-based API that I had before required passing names around, required more calls, required managing the channel/interpreter relationship more carefully, and made it hard to follow that relationship. >> Unlike queues, which are also many-to-many, >> channels have no buffer. > > How does it work? Does send() block until someone else calls recv()? > That does not sound like a good idea to me. Correct "send()" blocks until the other end receives (if ever). Likewise "recv()" blocks until the other end sends. This specific behavior is probably the main thing I borrowed from CSP. It is *the* synchronization mechanism. Given the isolated nature of subinterpreters, I consider using this concept from CSP to be a good fit. > I don't think it's a > coincidence that the most varied kinds of I/O (from socket or file IO > to threading Queues to multiprocessing Pipes) have non-blocking send(). Interestingly, you can set sockets to blocking mode, in which case send() will block until ther
Re: [Python-Dev] PEP 554 v3 (new interpreters module)
On Thu, Sep 14, 2017 at 8:44 PM, Nick Coghlan wrote: > Not really, because the only way to ensure object separation (i.e no > refcounted objects accessible from multiple interpreters at once) with > a bytes-based API would be to either: > > 1. Always copy (eliminating most of the low overhead communications > benefits that subinterpreters may offer over multiple processes) > 2. Make the bytes implementation more complicated by allowing multiple > bytes objects to share the same underlying storage while presenting as > distinct objects in different interpreters > 3. Make the output on the receiving side not actually a bytes object, > but instead a view onto memory owned by another object in a different > interpreter (a "memory view", one might say) 4. Pass Bytes through directly. The only problem of which I'm aware is that when Py_DECREF() triggers Bytes.__del__(), it happens in the current interpreter, which may not be the "owner" (i.e. allocated the object). So the solution would be to make PyBytesType.tp_free() effectively run as a "pending call" under the owner. This would require two things: 1. a new PyBytesObject.owner field (PyInterpreterState *), or a separate owner table, which would be set when the object is passed through a channel 2. a Py_AddPendingCall() that targets a specific interpreter (which I expect would be desirable regardless) Then, when the object has an owner, PyBytesType.tp_free() would add a pending call on the owner to call PyObject_Del() on the Bytes object. The catch is that currently "pending" calls (via Py_AddPendingCall) are run only in the main thread of the main interpreter. We'd need a similar mechanism that targets a specific interpreter . > By contrast, if we allow an actual bytes object to be shared, then > either every INCREF or DECREF on that bytes object becomes a > synchronisation point, or else we end up needing some kind of > secondary per-interpreter refcount where the interpreter doesn't drop > its shared reference to the original object in its source interpreter > until the internal refcount in the borrowing interpreter drops to > zero. There shouldn't be a need to synchronize on INCREF. If both interpreters have at least 1 reference then either one adding a reference shouldn't be a problem. If only one interpreter has a reference then the other won't be adding any references. If neither has a reference then neither is going to add any references. Perhaps I've missed something. Under what circumstances would INCREF happen while the refcount is 0? On DECREF there shouldn't be a problem except possibly with a small race between decrementing the refcount and checking for a refcount of 0. We could address that several different ways, including allowing the pending call to get queued only once (or being a noop the second time). FWIW, I'm not opposed to the CIV/memoryview approach, but want to make sure we really can't use Bytes before going down that route. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v3 (new interpreters module)
After having looked it over, I'm leaning toward supporting buffering,
as well as not blocking by default. Neither adds much complexity to
the implementation.
On Sat, Sep 23, 2017 at 5:45 AM, Antoine Pitrou wrote:
> On Fri, 22 Sep 2017 19:09:01 -0600
> Eric Snow wrote:
>> > send() blocking until someone else calls recv() is not only bad for
>> > performance,
>>
>> What is the performance problem?
>
> Intuitively, there must be some kind of context switch (interpreter
> switch?) at each send() call to let the other end receive the data,
> since you don't have any internal buffering.
There would be an internal size-1 buffer.
>> (FWIW, CSP
>> provides rigorous guarantees about deadlock detection (which Go
>> leverages), though I'm not sure how much benefit that can offer such a
>> dynamic language as Python.)
>
> Hmm... deadlock detection is one thing, but when detected you must still
> solve those deadlock issues, right?
Yeah, I haven't given much thought into how we could leverage that
capability but my
gut feeling is that we won't have much opportunity to do so. :)
>> I'm not sure I understand your concern here. Perhaps I used the word
>> "sharing" too ambiguously? By "sharing" I mean that the two actors
>> have read access to something that at least one of them can modify.
>> If they both only have read-only access then it's effectively the same
>> as if they are not sharing.
>
> Right. What I mean is that you *can* share very simple "data" under
> the form of synchronization primitives. You may want to synchronize
> your interpreters even they don't share user-visible memory areas. The
> point of synchronization is not only to avoid memory corruption but
> also to regulate and orchestrate processing amongst multiple workers
> (for example processes or interpreters). For example, a semaphore is
> an easy way to implement "I want no more than N workers to do this
> thing at the same time" ("this thing" can be something such as disk
> I/O).
I'm still not convinced that sharing synchronization primitives is
important enough to be worth including it in the PEP. It can be added
later, or via an extension module in the meantime. To that end, I'll
add a mechanism to the PEP for third-party types to indicate that they
can be passed through channels. Something like
"obj.__channel_support__ = True".
-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v3 (new interpreters module)
On Mon, Oct 2, 2017 at 9:31 PM, Eric Snow wrote: > On DECREF there shouldn't be a problem except possibly with a small > race between decrementing the refcount and checking for a refcount of > 0. We could address that several different ways, including allowing > the pending call to get queued only once (or being a noop the second > time). Alternately, the channel could own a reference and DECREF it in the owning interpreter once the refcount reaches 1. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v3 (new interpreters module)
On Mon, Sep 25, 2017 at 8:42 PM, Nathaniel Smith wrote: > It's fairly reasonable to implement a mutex using a CSP-style > unbuffered channel (send = acquire, receive = release). And the same > trick turns a channel with a fixed-size buffer into a bounded > semaphore. It won't be as efficient as a modern specialized mutex > implementation, of course, but it's workable. > > Unfortunately while technically you can construct a buffered channel > out of an unbuffered channel, the construction's pretty unreasonable > (it needs two dedicated threads per channel). Yeah, if threading's synchronization primitives make sense between interpreters then we'll add direct support. Using channels for that isn't a good option. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v3 (new interpreters module)
On Wed, Sep 27, 2017 at 1:26 AM, Nick Coghlan wrote:
> It's also the case that unlike Go channels, which were designed from
> scratch on the basis of implementing pure CSP,
FWIW, Go's channels (and goroutines) don't implement pure CSP. They
provide a variant that the Go authors felt was more in-line with the
language's flavor. The channels in the PEP aim to support a more pure
implementation.
> Python has an
> established behavioural precedent in the APIs of queue.Queue and
> collections.deque: they're unbounded by default, and you have to opt
> in to making them bounded.
Right. That's part of why I'm leaning toward support for buffered channels.
> While the article title is clickbaity,
> http://www.jtolds.com/writing/2016/03/go-channels-are-bad-and-you-should-feel-bad/
> actually has a good discussion of this point. Search for "compose" to
> find the relevant section ("Channels don’t compose well with other
> concurrency primitives").
>
> The specific problem cited is that only offering unbuffered or
> bounded-buffer channels means that every send call becomes a potential
> deadlock scenario, as all that needs to happen is for you to be
> holding a different synchronisation primitive when the send call
> blocks.
Yeah, that blog post was a reference for me as I was designing the
PEP's channels.
> The fact that the proposal now allows for M:N sender:receiver
> relationships (just as queue.Queue does with threads) makes that
> problem worse, since you may now have variability not only on the
> message consumption side, but also on the message production side.
>
> Consider this example where you have an event processing thread pool
> that we're attempting to isolate from blocking IO by using channels
> rather than coroutines.
>
> Desired flow:
>
> 1. Listener thread receives external message from socket
> 2. Listener thread files message for processing on receive channel
> 3. Listener thread returns to blocking on the receive socket
>
> 4. Processing thread picks up message from receive channel
> 5. Processing thread processes message
> 6. Processing thread puts reply on the send channel
>
> 7. Sending thread picks up message from send channel
> 8. Sending thread makes a blocking network send call to transmit the message
> 9. Sending thread returns to blocking on the send channel
>
> When queue.Queue is used to pass the messages between threads, such an
> arrangement will be effectively non-blocking as long as the send rate
> is greater than or equal to the receive rate. However, the GIL means
> it won't exploit all available cores, even if we create multiple
> processing threads: you have to switch to multiprocessing for that,
> with all the extra overhead that entails.
>
> So I see the essential premise of PEP 554 as being to ask the question
> "If each of these threads was running its own *interpreter*, could we
> use Sans IO style protocols with interpreter channels to separate
> internally "synchronous" processing threads from separate IO threads
> operating at system boundaries, without having to make the entire
> application pervasively asynchronous?"
+1
> If channels are an unbuffered blocking primitive, then we don't get
> that benefit: even when there are additional receive messages to be
> processed, the processing thread will block until the previous send
> has completed. Switching the listener and sender threads over to
> asynchronous IO would help with that, but they'd also end up having to
> implement their own message buffering to manage the lack of buffering
> in the core channel primitive.
>
> By contrast, if the core channels are designed to offer an unbounded
> buffer by default, then you can get close-to-CSP semantics just by
> setting the buffer size to 1 (it's still not exactly CSP, since that
> has a buffer size of 0, but you at least get the semantics of having
> to alternate sending and receiving of messages).
Yep, I came to the same conclusion.
>> By the way, I do think efficiency is a concern here. Otherwise
>> subinterpreters don't even have a point (just use multiprocessing).
>
> Agreed, and I think the interaction between the threading module and
> the interpreters module is one we're going to have to explicitly call
> out as being covered by the provisional status of the interpreters
> module, as I think it could be incredibly valuable to be able to send
> at least some threading objects through channels, and have them be an
> interpreter-specific reference to a common underlying sync primitive.
Agreed. I'll add a note to the PEP.
-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v3 (new interpreters module)
On Tue, Oct 3, 2017 at 5:00 AM, Antoine Pitrou wrote: > On Mon, 2 Oct 2017 22:15:01 -0400 > Eric Snow wrote: >> >> I'm still not convinced that sharing synchronization primitives is >> important enough to be worth including it in the PEP. It can be added >> later, or via an extension module in the meantime. To that end, I'll >> add a mechanism to the PEP for third-party types to indicate that they >> can be passed through channels. Something like >> "obj.__channel_support__ = True". > > How would that work? If it's simply a matter of flipping a bit, why > don't we do it for all objects? The type would also have to be safe to share between interpreters. :) Eventually I'd like to make that work for all immutable objects (and immutable containers thereof), but until then each type must be adapted individually. The PEP starts off with just Bytes. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v3 (new interpreters module)
On Tue, Oct 3, 2017 at 11:36 PM, Nick Coghlan wrote: > The problem relates to the fact that there aren't any memory barriers > around CPython's INCREF operations (they're implemented as an ordinary > C post-increment operation), so you can get the following scenario: > > * thread on CPU A has the sole reference (ob_refcnt=1) > * thread on CPU B acquires a new reference, but hasn't pushed the > updated ob_refcnt value back to the shared memory cache yet > * original thread on CPU A drops its reference, *thinks* the refcnt is > now zero, and deletes the object > * bad things now happen in CPU B as the thread running there tries to > use a deleted object :) I'm not clear on where we'd run into this problem with channels. Mirroring your scenario: * interpreter A (in thread on CPU A) INCREFs the object (the GIL is still held) * interp A sends the object to the channel * interp B (in thread on CPU B) receives the object from the channel * the new reference is held until interp B DECREFs the object >From what I see, at no point do we get a refcount of 0, such that there would be a race on the object being deleted. The only problem I'm aware of (it dawned on me last night), is in the case that the interpreter that created the object gets deleted before the object does. In that case we can't pass the deletion back to the original interpreter. (I don't think this problem is necessarily exclusive to the solution I've proposed for Bytes.) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v3 (new interpreters module)
On Tue, Oct 3, 2017 at 8:55 AM, Antoine Pitrou wrote:
> I think we need a sharing protocol, not just a flag. We also need to
> think carefully about that protocol, so that it does not imply
> unnecessary memory copies. Therefore I think the protocol should be
> something like the buffer protocol, that allows to acquire and release
> a set of shared memory areas, but without imposing any semantics onto
> those memory areas (each type implementing its own semantics). And
> there needs to be a dedicated reference counting for object shares, so
> that the original object can be notified when all its shares have
> vanished.
I've come to agree. :) I actually came to the same conclusion tonight
before I'd been able to read through your message carefully. My idea
is below. Your suggestion about protecting shared memory areas is
something to discuss further, though I'm not sure it's strictly
necessary yet (before we stop sharing the GIL).
On Wed, Oct 4, 2017 at 7:41 PM, Nick Coghlan wrote:
> Having the sending interpreter do the INCREF just changes the problem
> to be a memory leak waiting to happen rather than an access-after-free
> issue, since the problematic non-synchronised scenario then becomes:
>
> * thread on CPU A has two references (ob_refcnt=2)
> * it sends a reference to a thread on CPU B via a channel
> * thread on CPU A releases its reference (ob_refcnt=1)
> * updated ob_refcnt value hasn't made it back to the shared memory cache yet
> * thread on CPU B releases its reference (ob_refcnt=1)
> * both threads have released their reference, but the refcnt is still
> 1 -> object leaks!
>
> We simply can't have INCREFs and DECREFs happening in different
> threads without some way of ensuring cache coherency for *both*
> operations - otherwise we risk either the refcount going to zero when
> it shouldn't, or *not* going to zero when it should.
>
> The current CPython implementation relies on the process global GIL
> for that purpose, so none of these problems will show up until you
> start trying to replace that with per-interpreter locks.
>
> Free threaded reference counting relies on (expensive) atomic
> increments & decrements.
Right. I'm not sure why I was missing that, but I'm clear now.
Below is a rough idea of what I think may work instead (the result of
much tossing and turning in bed*).
While we're still sharing a GIL between interpreters:
Channel.send(obj): # in interp A
incref(obj)
if type(obj).tp_share == NULL:
raise ValueError("not a shareable type")
ch.objects.append(obj)
Channel.recv(): # in interp B
orig = ch.objects.pop(0)
obj = orig.tp_share()
return obj
bytes.tp_share():
return self
After we move to not sharing the GIL between interpreters:
Channel.send(obj): # in interp A
incref(obj)
if type(obj).tp_share == NULL:
raise ValueError("not a shareable type")
set_owner(obj) # obj.owner or add an obj -> interp entry to global table
ch.objects.append(obj)
Channel.recv(): # in interp B
orig = ch.objects.pop(0)
obj = orig.tp_share()
set_shared(obj, orig) # add to a global table
return obj
bytes.tp_share():
obj = blank_bytes(len(self))
obj.ob_sval = self.ob_sval # hand-wavy memory sharing
return obj
bytes.tp_free(): # under no-shared-GIL:
# most of this could be pulled into a macro for re-use
orig = lookup_shared(self)
if orig != NULL:
current = release_LIL()
interp = lookup_owner(orig)
acquire_LIL(interp)
decref(orig)
release_LIL(interp)
acquire_LIL(current)
# clear shared/owner tables
# clear/release self.ob_sval
free(self)
The CIV approach could be facilitated through something like a new
SharedBuffer type, or through a separate BufferViewChannel, etc.
Most notably, this approach avoids hard-coding specific type support
into channels and should work out fine under no-shared-GIL
subinterpreters. One nice thing about the tp_share slot is that it
makes it much easier (along with C-API for managing the global
owned/shared tables) to implement other types that are legal to pass
through channels. Such could be provided via extension modules.
Numpy arrays could be made to support it, if that's your thing.
Antoine could give tp_share to locks and semaphores. :) Of course,
any such types would have to ensure that they are actually safe to
share between intepreters without a GIL between them...
For PEP 554, I'd only propose the tp_share slot and its use in
Channel.send()/.recv(). The parts related to global tables and memory
sharing and tp_free() wouldn't be necessary until we stop sharing the
GIL between interpreters. However, I believe that tp_share would make
us ready for that.
-eric
* I should know by now that some ideas sound better in the middle of
the night than they do the next day, but this idea is keeping me awake
so I'll risk it! :)
___
Python-Dev
Re: [Python-Dev] PEP 554 v3 (new interpreters module)
On Thu, Oct 5, 2017 at 4:57 AM, Nick Coghlan wrote: > This would be hard to get to work reliably, because "orig.tp_share()" would > be running in the receiving interpreter, but all the attributes of "orig" > would have been allocated by the sending interpreter. It gets more reliable > if it's *Channel.send* that calls tp_share() though, but moving the call to > the sending side makes it clear that a tp_share protocol would still need to > rely on a more primitive set of "shareable objects" that were the permitted > return values from the tp_share call. The point of running tp_share() in the receiving interpreter is to force allocation under that interpreter, so that GC applies there. I agree that you basically can't do anything in tp_share() that would affect the sending interpreter, including INCREF and DECREF. Since we INCREFed in send(), we know that the we have a safe reference, so we don't have to worry about that part in tp_share(). We would only be able to do low-level things (like the buffer protocol) that don't interact with the original object's interpreter. Given that this is a quite low-level tp slot and low-level functionality, I'd expect that a sufficiently clear entry (i.e. warning) in the docs would be enough for the few that dare. >From my perspective adding the tp_share slot allows for much more experimentation with object sharing (right now, long before we get to considering how to stop sharing the GIL) by us *and* third parties. None of the alternatives seem to offer the same opportunity while still working out *after* we stop sharing the GIL. > > And that's the real pay-off that comes from defining this in terms of the > memoryview protocol: Py_buffer structs *aren't* Python objects, so it's only > a regular C struct that gets passed across the interpreter boundary (the > reference to the original objects gets carried along passively as part of > the CIV - it never gets *used* in the receiving interpreter). Yeah, the (PEP 3118) buffer protocol offers precedent in a number of ways that are applicable to channels here. I'm simply reticent to lock PEP 554 into such a specific solution as the buffer-specific CIV. I'm trying to accommodate anticipated future needs while keeping the PEP as simple and basic as possible. It's driving me nuts! :P Things were *much* simpler before I added Channels to the PEP. :) > >> >> bytes.tp_share(): >> obj = blank_bytes(len(self)) >> obj.ob_sval = self.ob_sval # hand-wavy memory sharing >> return obj > > > This is effectively reinventing memoryview, while trying to pretend it's an > ordinary bytes object. Don't reinvent memoryview :) > >> >> bytes.tp_free(): # under no-shared-GIL: >> # most of this could be pulled into a macro for re-use >> orig = lookup_shared(self) >> if orig != NULL: >> current = release_LIL() >> interp = lookup_owner(orig) >> acquire_LIL(interp) >> decref(orig) >> release_LIL(interp) >> acquire_LIL(current) >> # clear shared/owner tables >> # clear/release self.ob_sval >> free(self) > > > I don't think we should be touching the behaviour of core builtins solely to > enable message passing to subinterpreters without a shared GIL. Keep in mind that I included the above as a possible solution using tp_share() that would work *after* we stop sharing the GIL. My point is that with tp_share() we have a solution that works now *and* will work later. I don't care how we use tp_share to do so. :) I long to be able to say in the PEP that you can pass bytes through the channel and get bytes on the other side. That said, I'm not sure how this could be made to work without involving tp_free(). If that is really off the table (even in the simplest possible ways) then I don't think there is a way to actually share objects of builtin types between interpreters other than through views like CIV. We could still support tp_share() for the sake of third parties, which would facilitate that simplicity I was aiming for in sending data between interpreters, as well as leaving the door open for nearly all the same experimentation. However, I expect that most *uses* of channels will involve builtin types, particularly as we start off, so having to rely on view types for builtins would add not-insignificant awkwardness to using channels. I'd still like to avoid that if possible, so let's not rush to completely close the door on small modifications to tp_free for builtins. :) Regardless, I still (after a night's rest and a day of not thinking about it) consider tp_share() to be the solution I'd been hoping we'd find, whether or not we can apply it to builtin types. > > The simplest possible variant of CIVs that I can think of would be able to > avoid that outcome by being a memoryview subclass, since they just need to > hold the extra reference to the original interpreter, and include some logic > to swtich interpreters at the appropriate time. > >
Re: [Python-Dev] The current dict is not an "OrderedDict"
On Nov 7, 2017 08:12, "INADA Naoki" wrote: Additionally, class namespace should keep insertion order. It's language spec from 3.6. So we should have two mode for such optimization. It makes dict more complicated. FWIW, PEP 520 (Preserving Class Attribute Definition Order) originally specified leaving the class namespace alone. Instead, the default class *definition* namespace was changed to OrderedDict, and the ordering from that namespace was stored as a tuple of names in a new __definition_order__ attribute on classes. That approach effectively decoupled the final class namespace from the proposed feature. If it's an issue now then we might consider reviving __definition_order__ (which, as a bonus, has other minor benefits). However, I expect we will make the current dict implementation's behavior official, which renders any changes unnecessary. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python initialization and embedded Python
On Nov 18, 2017 19:20, "Nick Coghlan" wrote: OK, in that case I think the answer to Victor's question is: 1. Breaking calling Py_DecodeLocale() before calling Py_Initialize() is a compatibility break with the API implied by our own usage examples, and we'll need to revert the breakage for 3.7, +1 The break was certainly unintentional. :/ Fortunately, Py_DecodeLocale() should be the only "Process-wide parameter" needing repair. I suppose, PyMem_RawMalloc() and PyMem_RawFree() *could* be considered too, but my understanding is that they aren't really intended for direct use (especially pre-init). and ensure at least one release's worth of DeprecationWarning before requiring either the use of an alternative API (where the caller controls the memory management), or else a new lower level pre-initialization API (i.e. making `PyRuntime_Initialize` a public API) There shouldn't be a need to deprecate anything, right? We just need to restore the pre-init behavior of Py_DecodeLocale. 2. We should provide a consolidated list of these functions in the C API initialization docs +1 PyMem_Raw*() do not belong in that group, right? Again, my understanding is that they aren't intended for direct third-party use (are they even a part of the C-API?), and particularly pre-init. That Py_DecodeLocale() can use PyMem_RawMalloc() pre-init is an implementation detail. 3. We should add more test cases to _testembed.c that ensure they all work correctly prior to Py_Initialize (some of them are already tested there, but definitely not all of them) +1 -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python initialization and embedded Python
On Mon, Nov 20, 2017 at 8:43 AM, Victor Stinner wrote: > 2017-11-20 16:31 GMT+01:00 Eric Snow : >> That Py_DecodeLocale() can use PyMem_RawMalloc() pre-init is an >> implementation detail. > > Py_DecodeLocale() uses PyMem_RawMalloc(), and so its result must be > freed by PyMem_RawFree(). It's part the documentation. Ah, I'd missed that. Thanks for pointing it out. > > I'm not sure that I understood correctly. Do you agree to move "PyMem" > globals back to Objects/obmalloc.c? (to allow to call > PyMem_RawMalloc() before Py_Initialize()) I'm okay with that if we can't find another way. However, shouldn't we be able to statically initialize the raw allocator in _PyRuntime, much as we were doing before in obmalloc.c? I have a rough PR up: https://github.com/python/cpython/pull/4481 Also, I opened https://bugs.python.org/issue32096 for the regression. Thanks for bringing it up. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python initialization and embedded Python
On Mon, Nov 20, 2017 at 3:03 PM, Victor Stinner wrote: > To statically initialize PyMemAllocatorEx fields, you need to export a > lot of allocator functions. I would prefer to not do that. > > [snip] > > The rules to choose the allocator to each domain are also complex > depending if pymalloc is enabled, debug hooks are enabled by default, > etc. The memory allocator is also linked to _PyMem_Debug which is not > currently in Include/internals/ but Objects/obmalloc.c. I'm not suggesting supporting the full machinery. Rather, as my PR demonstrates, we can statically initialize the minimum needed to support pre-init use of PyMem_RawMalloc() and PyMem_RawFree(). The allocators will be fully initialized once the runtime is initialized (i.e. once Py_Initialize() is called), just as they are now. FWIW, I'm not sure that's the best approach. See my notes in https://bugs.python.org/issue32096. > > I understand that moving global variables to _PyRuntime helps to > clarify how these variables are initialized and then finalized, but > memory allocators are a complex corner case. Agreed. I spent a large portion of my time getting the allocators right when working on the original _PyRuntime patch. It's tricky code. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can Python guarantee the order of keyword-only parameters?
On Mon, Nov 27, 2017 at 10:05 AM, Larry Hastings wrote: > I'd like inspect.signature to guarantee that the order of keyword-only > parameters always matches the order they were declared in. Technically this > isn't a language feature, it's a library feature. But making this guarantee > would require that CPython internally cooperate, so it's kind of a language > feature too. > > Does this sound reasonable? Would it need a PEP? I'm hoping for "yes" and > "no", respectively. +1 There is definitely significant information in source code text that gets thrown away in some cases, which I'm generally in favor of preserving (see PEP 468 and PEP 520). The use case here is unclear to me, but the desired guarantee is effectively the status quo and it is a minor guarantee as well, so I don't see the harm. Furthermore, I don't see a need for a PEP given the small scale and impact. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 554 v4 (new interpreters module)
Hi all, I've finally updated PEP 554. Feedback would be most welcome. The PEP is in a pretty good place now and I hope to we're close to a decision to accept it. :) In addition to resolving the open questions, I've also made the following changes to the PEP: * put an API summary at the top and moved the full API description down * add the "is_shareable()" function to indicate if an object can be shared * added None as a shareable object Regarding the open questions: * "Leaking exceptions across interpreters" I chose to go with an approach that effectively creates a traceback.TracebackException proxy of the original exception, wraps that in a RuntimeError, and raises that in the calling interpreter. Raising an exception that safely preserves the original exception and traceback seems like the most intuitive behavior (to me, as a user). The only alternative that made sense is to fully duplicate the exception and traceback (minus stack frames) in the calling interpreter, which is probably overkill and likely to be confusing. * "Initial support for buffers in channels" I chose to add a "SendChannel.send_buffer(obj)" method for this. Supporting buffer objects from the beginning makes sense, opening good experimentation opportunities for a valuable set of users. Supporting buffer objects separately and explicitly helps set clear expectations for users. I decided not to go with a separate class (e.g. MemChannel) as it didn't seem like there's enough difference to warrant keeping them strictly separate. FWIW, I'm still strongly in favor of support for passing (copies of) bytes objects via channels. Passing objects to SendChannel.send() is obvious. Limiting it, for now, to bytes (and None) helps us avoid tying ourselves strongly to any particular implementation (it seems like all the reservations were relative to the implementation). So I do not see a reason to wait. * "Pass channels explicitly to run()?" I've applied the suggested solution (make "channels" an explicit keyword argument). -eric I've include the latest full text (https://www.python.org/dev/peps/pep-0554/) below: +++++ PEP: 554 Title: Multiple Interpreters in the Stdlib Author: Eric Snow Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2017-09-05 Python-Version: 3.7 Post-History: 07-Sep-2017, 08-Sep-2017, 13-Sep-2017, 05-Dec-2017 Abstract CPython has supported multiple interpreters in the same process (AKA "subinterpreters") since version 1.5. The feature has been available via the C-API. [c-api]_ Subinterpreters operate in `relative isolation from one another `_, which provides the basis for an `alternative concurrency model `_. This proposal introduces the stdlib ``interpreters`` module. The module will be `provisional `_. It exposes the basic functionality of subinterpreters already provided by the C-API, along with new functionality for sharing data between interpreters. Proposal The ``interpreters`` module will be added to the stdlib. It will provide a high-level interface to subinterpreters and wrap a new low-level ``_interpreters`` (in the same was as the ``threading`` module). See the `Examples`_ section for concrete usage and use cases. Along with exposing the existing (in CPython) subinterpreter support, the module will also provide a mechanism for sharing data between interpreters. This mechanism centers around "channels", which are similar to queues and pipes. Note that *objects* are not shared between interpreters since they are tied to the interpreter in which they were created. Instead, the objects' *data* is passed between interpreters. See the `Shared data`_ section for more details about sharing between interpreters. At first only the following types will be supported for sharing: * None * bytes * PEP 3118 buffer objects (via ``send_buffer()``) Support for other basic types (e.g. int, Ellipsis) will be added later. API summary for interpreters module --- Here is a summary of the API for the ``interpreters`` module. For a more in-depth explanation of the proposed classes and functions, see the `"interpreters" Module API`_ section below. For creating and using interpreters: +--+--+ | signature| description | ++=+==+ | list_all() -> [Intepreter] | Get all existing interpreters. | +--+--+ | get_current() -> Interpreter | Get the currently running interpreter. | +-
Re: [Python-Dev] PEP 554 v4 (new interpreters module)
On Dec 5, 2017 23:49, "Nick Coghlan" wrote: Nice updates! I like this version. Great! :) My one suggestion here would be to consider a dedicated exception type like "interpreters.SubinterpreterError", rather than re-using RuntimeError directly. That way you can put the extracted traceback on a named attribute, and retain the option of potentially adding subinterpreter awareness to the traceback module in the future. Yeah, I already have a deferred idea item for this. :). TBH, I was on the fence about a dedicated exception type, so you've nudged me on board. :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v4 (new interpreters module)
On Dec 6, 2017 20:31, "Guido van Rossum" wrote: If the point is just to be able to test the existing API better, no PEP is needed, right? It would be an unsupported, undocumented API. In the short term that's one major goal. In the long term the functionality provided by the PEP is a prerequisite for other concurrency-related features, and targeting 3.8 for that is fine. :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v4 (new interpreters module)
On Dec 6, 2017 21:14, "Guido van Rossum" wrote: OK, then please just change the PEP's Version: header to 3.8. Will do. Have a nice vacation! :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 -- Context Variables
Overall, I like this PEP. It's definitely easier to follow conceptually than PEP 550. Thanks for taking the time to re-think the idea. I have a few comments in-line below. -eric On Tue, Dec 12, 2017 at 10:33 AM, Yury Selivanov wrote: > This is a new proposal to implement context storage in Python. +1 This is something I've had on my back burner for years. Getting this right is non-trivial, so having a stdlib implementation will help open up clean solutions in a number of use cases that are currently addressed in more error-prone ways. > > It's a successor of PEP 550 and builds on some of its API ideas and > datastructures. Contrary to PEP 550 though, this proposal only focuses > on adding new APIs and implementing support for it in asyncio. There > are no changes to the interpreter or to the behaviour of generator or > coroutine objects. Do you have any plans to revisit extension of the concept to generators and coroutine objects? I agree they can be addressed separately, if necessary. TBH, I'd expect this PEP to provide an approach that allows such applications of the concept to effectively be implementation details that can be supported later. > Abstract > > > This PEP proposes the new ``contextvars`` module and a set of new > CPython C APIs to support context variables. This concept is > similar to thread-local variables but, unlike TLS, it allows s/it allows/it also allows/ > correctly keeping track of values per asynchronous task, e.g. > ``asyncio.Task``. > > [snip] > > Rationale > = > > Thread-local variables are insufficient for asynchronous tasks which > execute concurrently in the same OS thread. Any context manager that > needs to save and restore a context value and uses > ``threading.local()``, will have its context values bleed to other > code unexpectedly when used in async/await code. FWIW, I'd consider the concept to extend to all execution contexts in the interpreter, of which threads and async/await are the only kinds we have currently. That said, I don't see us adding any new kinds of execution context so what you've said is entirely satisfactory. :) > > [snip] > > Introduction > > > [snip] > > Specification > = > > A new standard library module ``contextvars`` is added Why not add this to contextlib instead of adding a new module? IIRC this was discussed relative to PEP 550, but I don't remember the reason. Regardless, it would be worth mentioning somewhere in the PEP. > with the > following APIs: > > 1. ``get_context() -> Context`` function is used to get the current >``Context`` object for the current OS thread. > > 2. ``ContextVar`` class to declare and access context variables. It may be worth explaining somewhere in the PEP the reason why you've chosen to add ContextVar instead of adding a new keyword (e.g. "context", a la global and nonlocal) to do roughly the same thing. Consider that execution contexts are very much a language-level concept, a close sibling to scope. Driving that via a keyword would a reasonable approach, particularly since it introduces less coupling between a language-level feature and a stdlib module. (Making it a builtin would sort of help with that too, but a keyword would seem like a better fit.) A keyword would obviate the need for explicitly calling .get() and .set(). FWIW, I agree with not adding a new keyword. To me context variables are a low-level tool for library authors to implement their high-level APIs. ContextVar, with its explicit .get() and .set() methods is a good fit for that and better communicates the conceptual intent of the feature. However, it would still be worth explicitly mentioning the alternate keyword-based approach in the PEP. > > 3. ``Context`` class encapsulates context state. Every OS thread >stores a reference to its current ``Context`` instance. >It is not possible to control that reference manually. >Instead, the ``Context.run(callable, *args)`` method is used to run >Python code in another context. I'd call that "Context.call()" since its for callables. Did you have a specific reason for calling it "run" instead? > > FWIW, I think there are some helpers you could add that library authors would appreciate. However, they aren't critical so I'll hold off and maybe post about them later. :) > contextvars.ContextVar > -- > > The ``ContextVar`` class has the following constructor signature: > ``ContextVar(name, *, default=no_default)``. The ``name`` parameter > is used only for introspection and debug purposes. It doesn't need to be required then, right? > [snip] > > ``ContextVar.set(value) -> Token`` is used to set a new value for > the context variable in the current ``Context``:: > > # Set the variable 'var' to 1 in the current context. > var.set(1) > > ``contextvars.Token`` is an opaque object that should be used to > restore the ``ContextVar`` to its previous value, or remove it from > the
Re: [Python-Dev] PEP 567 -- Context Variables
On Tue, Dec 12, 2017 at 4:49 PM, Victor Stinner wrote: >> The ``Token`` API exists to make the current proposal forward >> compatible with :pep:`550`, in case there is demand to support >> context variables in generators and asynchronous generators in the >> future. > > Cool. I like the idea of starting with something simple in Python 3.7. > Then extend it in Python 3.8 or later (support generators), if it > becomes popular, once the first simple (but "incomplete", without > generators) implementation is battle-tested. +1 for starting with a basic API and building on that. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
On Fri, Dec 15, 2017 at 8:53 AM, Guido van Rossum wrote: > Make it so. "Dict keeps insertion order" is the ruling. Thanks! Does that include preserving order after deletion? -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Make __class_getitem__ a class method
On Fri, Dec 15, 2017 at 11:35 AM, Serhiy Storchaka wrote: > In this case I suggest to make __class_getitem__ an automatic class method > like __init_subclass__. +1 I was just about to suggest the same thing. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pep-0557 dataclasses top level module vs part of collections?
On Thu, Dec 21, 2017 at 4:21 PM, Gregory P. Smith wrote: > It seems a suggested use is "from dataclasses import dataclass" > > But people are already familiar with "from collections import namedtuple" > which suggests to me that "from collections import dataclass" would be a > more natural sounding API addition. FWIW, I'd consider this a good time to add a new top-level classtools/classutils module (a la functools). There are plenty of other things that would fit there that we've shoved into other places. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [python-committers] Welcome the 3.8 and 3.9 Release Manager - Łukasz Langa!
On Sat, Jan 27, 2018 at 2:02 PM, Barry Warsaw wrote: > please welcome your next release manager… > > Łukasz Langa! Congrats, Łukasz! (or condolences? ) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How is the GitHub workflow working for people?
On Thu, Feb 22, 2018 at 9:30 AM, Paul Moore wrote: > My experience on pip is that automated style review is helpful for > avoiding debates over subjective details. This is the allure of Go's official linting tools. Nobody is happy with *all* the style choices but there isn't any room to fuss about it so people don't. Without that overhead, folks are more apt to lint their changes. (At least, that was my experience after several years working on projects written in Go.) One nice thing is that it frees you up to argue about other things. :) > But it does result in a > certain level of "tweak to satisfy the style checker" churn in PRs. > That can be frustrating when CI takes a long time to run. I had exactly that experience on one particularly large Go project (on GitHub, with slow CI, driven by bots). To make matters worse, the project had a dozen people actively working on it, meaning a high potential that your PR would not apply cleanly if you took too long to merge it. So, coupled with slow CI, linting failures were particularly onerous. We all got in the habit of locally running the linter frequently. IIRC, I even set up VIM to run the linter whenever I saved. FWIW, I'm fine with a bot that leaves a message (or a review) if there are linting issues. Regardless, we should careful about adding any extra overhead to our workflow, particularly since the move to GH has been driven by the desire to reduce overhead. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 572: Assignment Expressions
On Tue, Apr 17, 2018 at 7:55 AM, Steve Dower wrote: > Agree with Paul. The PEP is well thought out and well presented, but I > really don’t think we need this in Python (and I say this as someone who > uses it regularly in C/C#). > > -1 on the idea; no disrespect intended toward to people who did a lot of > work on it. Same here. I'm more interested in having anonymous blocks (i.e. scopes), along the lines of PEP 3150, though it's currently #2 on my wish list. :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] assignment expressions: an alternative proposal
Thanks for thinking this through, Yury. :) FWIW, I'm still unconvinced that an assignment expression is worth it. It's hard to say, though, without seeing how much folks would actually use it (and I don't have my own time machine unfortunately). IIRC, in the past several proposed syntax (e.g. decorators) were in the same boat and in retrospect turned out to be a strongly positive addition to the language. :) Comments in-line below. -eric On Tue, Apr 24, 2018 at 7:38 AM, Yury Selivanov wrote: > I propose to use the following syntax for assignment expressions: > > ( NAME = expr ) > > [snip] > > 1. Only NAME token is allowed as a single target. This seems reasonable and does keep assignment expressions easier for the reader to find. At the same time, there have been some arguments elsewhere in favor of tuple unpacking as the other obvious use case. Presumably that would not be supported under this rule. > 2. Parenthesis are required. Similar to rule #1, this would make assignment expressions more obvious to readers, which is a good thing. > > 3. Most importantly: it is *not* allowed to mask names in the current > local scope. I was about to leave this counter example for accidentally-typed-one-equal-sign-instead-of-two bugs: if (y == 3): print(y) # vs. if (y = 3): print(y) Then it dawned on me that your rule #3 solves this. :) > [snip] > > py> f = lambda x: x * 10 > py> [[(y = f(x)), x/y] for x in range(1,5)] > [[10, 0.1], [20, 0.1], [30, 0.1], [40, 0.1]] > > [snip] > > I believe that this syntax is the best of both worlds: it allows to > write succinct code just like PEP 572, but without introducing a new > ':=' operator. This is the main point of this alternate proposal, right? It certainly seems reasonable that we not add another assignment syntax. On the other hand, having ":=" as a distinct syntax for assignment expressions is close enough to the existing syntax that it doesn't really add any real extra burden to readers, while being more searchable and visually distinct. If we were to add assignment expressions I'd probably favor ":=". Regardless, your 3 rules would benefit either syntax. Nick may have a point that the rules might be an excessive burden, but I don't think it's too big a deal since the restrictions are few (and align with the most likely usage) and are limited to syntax so the compiler will be quick to point mistakes. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] (Looking for) A Retrospective on the Move to Python 3
In pondering our approach to future Python major releases, I found myself considering the experience we've had with Python 3. The whole Py3k effort predates my involvement in the community so I missed a bunch of context about the motivations, decisions, and challenges. While I've pieced some of that together over the years now since I've been around, I've certainly seen much of the aftermath. For me, at least, it would be helpful to have a bit more insight into the history. :) With that in mind, it would be worth having an informational PEP with an authoritative retrospective on the lessons learned from the Python 3 effort (and transition). Consider it a sort of autobiography, "memoirs on the python-dev change to Python 3". :) At this point the transition has settled in enough that we should be able to present a relatively objective (and consistent) view, while we're not so far removed that we've forgotten anything important. :) If such a document already exists then I'd love a pointer to it. The document would benefit (among others): * python-dev (by giving us a clear viewpoint to inform decisions about future releases) * new-comers to Python that want more insight into the language * folks transitioning from 2 to 3 * communities that have (or think they have) problems similar to those we faced in Python 2 The PEP doesn't even have to be done all at once, nor by one person. In fact, there are many viewpoints that would add value to the document. Hence it would probably make sense to encourage broad participation and then have a single editor to effect a single voice in the document. The contents of the retrospective document should probably cover a broad range of topics, since there's so much to learn from the move to Python 3. To give an indication of what I mean, I've included a rough outline at the bottom of this message. So...I typically strongly avoid making proposals that I'm not willing to execute. However, in this case I simply do not have enough experience in the history to feel comfortable doing a good job of it in a reasonable amount of time (which matters due to the tendency of valuable info to fade away). :/ I have no expectation that someone will pick this up, though I do hope since the benefit would be significant. My apologies in advance if this wasted anyone's time. -eric I'd hope to see something along the lines of (at least) the following, in rough order: * a concise summary of the document at the top (very meta, I know :) ) + what were we solving? + what was the solution? + why do it that way? + what went right? + what went wrong? + impact on the community + impact on core dev contribution * timeline * key players (and level of involvement) + old guard core devs + new guard + folks brought on for Py3k (e.g. IIRC a swarm of Googlers dove in) + non-core-devs * motivations * expectations (e.g. time frames, community reaction) * corresponding results * a summary of what we did * alternative approaches * what went right (and was it on purpose :) ) * what went wrong (e.g. io) and why * how the Py3k project differed from normal python-dev workflow (e.g. pace, decision-making, communications) * lasting impact of python-dev * key things that would have been better if done differently * key decisions/planning (mostly a priori to the release work) + scope of backward compatibility + process (using PEPs with PEPs 30xx guiding) + schedule + specific changes (i.e. PEPs 31xx) + what was left out (and why) + plans to help library and app authors transition (e.g. 2to3) + feature/schedule overlap with Python 2 (i.e. 2.6 and 2.7) + the language moratorium * things that got missed and why + unicode/bytes in some stdlib modules (and builtins?) * things that were overdone (and how that got missed) + unicode/bytes in some stdlib modules (and builtins?) * (last but not least) challenges faced by folks working to transition their exiting code to Python 3 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] (Looking for) A Retrospective on the Move to Python 3
On Thu, Apr 26, 2018 at 10:25 AM, Eric Snow wrote: > In pondering our approach to future Python major releases, I found > myself considering the experience we've had with Python 3. The whole > Py3k effort predates my involvement in the community so I missed a > bunch of context about the motivations, decisions, and challenges. > While I've pieced some of that together over the years now since I've > been around, I've certainly seen much of the aftermath. For me, at > least, it would be helpful to have a bit more insight into the > history. :) As to motivation, I suppose (since I wasn't involved yet) that it was roughly: let's bite the bullet and fix unicode with a backward-incompatible release which led to: well, we're already going to break backward compatibility so we might as well get any breakage we're planning over with now so folks only have to fix their code for this release and then to: oh, and while we're at it let's clean up a bunch of cruft that's built up over the years At least that's my impression. :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Every Release Can Be a Mini "Python 4000", Within Reason (was (name := expression) doesn't fit the narrative of PEP 20)
On Thu, Apr 26, 2018 at 12:52 AM, Greg Ewing wrote: > [snip] > here we would be *creating* one (two different assignment > operators with overlapping use cases) that we won't be > able to get rid of without a Python 4000 (that Guido has > promised won't happen). The edict about "Python 4000" is more about not repeating what happened with Python 3 than strictly prohibiting breaking backward compatibility. [1] The way I understand it, the problem with Py3k was that so many things changed in a backward-incompatible way. Folks could have coped with the unicode change as the big one for Python 2.7 (or 2.8 or 3.0 or whatever it ended up being). However, so many other things changed all at once that the burden to move to Python 3 became daunting. This included a number of back-compat-breaking syntax changes. Unless I've missed something, there's no prohibition against deprecating things (and then later removing them) or other breaks in backward compatibility. We certainly avoid it, with good reason. However, when we do break compatibility, the thing we want to avoid is introducing too many such changes all at once (and to keep in mind that such changes can add up to the same detriment when viewed in aggregate across multiple releases). That said, syntax is definitely a trickier target when it comes to breaking backward compatibility. So we have to be especially careful about adding it in the first place. I suppose that's a big part of the reason for the strong reaction to the "binding expressions" proposal. -eric [1] I'm hopeful we can consolidate a retrospective on Python 3 in a PEP: https://mail.python.org/pipermail/python-dev/2018-April/153131.html ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] (Looking for) A Retrospective on the Move to Python 3
On Thu, Apr 26, 2018 at 10:25 AM, Eric Snow wrote: > In pondering our approach to future Python major releases, I found > myself considering the experience we've had with Python 3. The whole > Py3k effort predates my involvement in the community so I missed a > bunch of context about the motivations, decisions, and challenges. > While I've pieced some of that together over the years now since I've > been around, I've certainly seen much of the aftermath. For me, at > least, it would be helpful to have a bit more insight into the > history. :) Thanks, all, for the responses. :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 394: Allow the `python` command to not be installed (and other minor edits)
On Fri, Apr 27, 2018 at 7:23 PM, Nick Coghlan wrote: > The key missing piece for doing that would be to define how we'd want a `py` > launcher to work on *nix systems, and then provide that as part of CPython > 3.8+ (and potentially backport it to a 3.7x maintenance release). I was thinking along the same lines just today, with an email forthcoming. Now I don't have to write it. :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Every Release Can Be a Mini "Python 4000", Within Reason (was (name := expression) doesn't fit the narrative of PEP 20)
FWIW, this thread is about what "Python 4000" means and does not mean. Namely, Python feature deprecation and removal is not prohibited but the bar is high (as always), especially for syntax. While I appreciate the level of interest in certain under-consideration proposals, you'd be better served by continuing discussion about that proposal in other threads. Thanks! -eric On Mon, Apr 30, 2018 at 7:25 PM, Terry Reedy wrote: > On 4/30/2018 4:00 PM, Jeff Allen wrote: > >> They were not "statements", but "formulas" while '=' was assignment (sec >> 8) *and* comparison (sec 10B). So conversely to our worry, they actually >> wanted users to think of assignment initially as a mathematical formula >> (page 2) in order to exploit the similarity to a familiar concept, albeit >> a=a+i makes no sense from this perspective. > > > When explaining iterative algorithms, such as Newton's method, > mathematicians write things like a' = a+1 or a(sub i+1 or sub new) = f(a(sub > i or sub old)) . For computer, we drop the super/subscript. Or one can > write more circuitously, > anew = update(aold) > aold = anew > The abbreviations should be explained when teaching loops. > > For proving that the body of a loop maintains a loop constant, one may > reinstate the super- or sub-script. > > -- > Terry Jan Reedy > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slow down...
On Sun, May 6, 2018 at 8:25 PM, Nick Coghlan wrote: > I'm inclined to agree that a Python 3.8 PEP in the spirit of the PEP 3003 > language moratorium could be a very good idea. Note that the PEP specifically applies to "syntax, semantics, and built-ins". Here's the full abstract [1]: This PEP proposes a temporary moratorium (suspension) of all changes to the Python language syntax, semantics, and built-ins for a period of at least two years from the release of Python 3.1. In particular, the moratorium would include Python 3.2 (to be released 18-24 months after 3.1) but allow Python 3.3 (assuming it is not released prematurely) to once again include language changes. This suspension of features is designed to allow non-CPython implementations to "catch up" to the core implementation of the language, help ease adoption of Python 3.x, and provide a more stable base for the community. -eric [1] https://www.python.org/dev/peps/pep-3003/#abstract ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How to watch buildbots?
On Wed, May 30, 2018 at 7:36 AM, Nick Coghlan wrote: > There are a few key details here: > > 1. We currently need to run post-merge CI anyway, as we're not doing > linearised commits (where core devs just approve a change without merging > it, and then a gating system like Zuul ensures that the tests are run > against the latest combination of the target branch and the PR before > merging the change) This is more of a concern when non-conflicting PRs against the same (or related) code are active at the same time. For the CPython code base this isn't as much of a problem, right? Under normal circumstances [fresh] active PRs typically do not run afoul of each other. Furthermore, during peak-activity events (like sprints) folks tend to keep a closer eye on the buildbots. I suppose old-but-still-active PRs that previously passed CI could cause a problem. However, it would be unlikely for such a PR to sit for a long time without needing changes before merging, whether to address reviewer concerns or to resolve merge conflicts. So post-merge CI (or merge gating) doesn't seem like much of a factor for us. In that regard I'd consider the buildbots more that sufficient. > 2. Since the buildbots are running on donated dedicated machines (rather > than throwaway instances from a dynamic CI provider), we need to review the > code before we let it run on the contributed systems > 3. The buildbot instances run *1* build at a time, ...where each build incorporates potentially several merged PRs... > which would lead to major > PR merging bottlenecks during sprints if we made them a gating requirement Agreed. There's enough of a delay already when watching the buildbots post-merge (especially some of them). :) > 4. For the vast majority of PRs, the post-merge cross-platform testing is a > formality, since the code being modified is using lower level cross-platform > APIs elsewhere in the standard library, so if it works on Windows, Linux, > and Mac OS X, it will work everywhere Python runs This is especially true of changes proposed by non-core contributors. It is also very true for buildbots with the OS/hardware combos that match CI. That said, when working with the C-API you can end up breaking things on the less common OSes and hardware platforms. So *those* buildbots are invaluable. I'm dealing with that right now. > 5. We generally don't *want* to burden new contributors with the task of > dealing with the less common (or harder to target) platforms outside the big > 3 - when they do break, it often takes a non-trivial amount of platform > knowledge to understand what's different about the platform in question As hinted above, I would not expect new contributors to provide patches very often (if ever) that would have the potential to cause buildbot failures but not fail under CI. So this point seems somewhat moot. :) > P.S. That said, if VSTS or Travis were to offer FreeBSD as an option for > pre-merge CI, I'd suggest we enable it, at least in an advisory capacity - > it's a better check against Linux-specific assumptions creeping into the > code base than Mac OS X, since the latter is regularly different enough from > other *nix systems that we need to give it dedicated code paths. +1 -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stable ABI
I've pointed out in bpo-21142 the similar script I added last year to track C globals: https://github.com/python/cpython/tree/master/Tools/c-globals -eric On Mon, Jun 4, 2018 at 1:17 AM, Ronald Oussoren wrote: > > > On 4 Jun 2018, at 08:35, Ronald Oussoren wrote: > > > > On 3 Jun 2018, at 17:04, Eric V. Smith wrote: > > On 6/3/2018 10:55 AM, Christian Tismer wrote: > > On 03.06.18 13:18, Ronald Oussoren wrote: > > > > On 3 Jun 2018, at 12:03, Christian Tismer wrote: > > ... > > > I have written a script that scans all relevant header files > and analyses all sections which are reachable in the limited API > context. > All macros that don't begin with an underscore which contain > a "->tp_" string are the locations which will break. > > I found exactly 7 locations where this is the case. > > My PR will contain the 7 fixes plus the analysis script > to go into tools. Preparind that in the evening. > > > Having tests would still be nice to detect changes to the stable ABI when > they are made. > > Writing those tests is quite some work though, especially if those at least > smoke test the limited ABI by compiling snippets the use all symbols that > should be exposed by the limited ABI. Writing those tests should be fairly > simple for someone that knows how to write C extensions, but is some work. > > Writing a tests that complain when the headers expose symbols that shouldn’t > be exposed is harder, due to the need to parse headers (either by hacking > something together using regular expressions, or by using tools like gccxml > or clang’s C API). > > What do you mean? > My script does that with all "tp_*" type fields. > What else would you want to check? > > > I think Ronald is saying we're trying to answer a few questions: > > 1. Did we accidentally drop anything from the stable ABI? > > 2. Did we add anything to the stable ABI that we didn't mean to? > > 3. (and one of mine): Does the stable ABI already contain things that we > don't expect it to? > > > That’s correct. There have been instances of the second item over the year, > and not all of them have been caught before releases. What doesn’t help for > all of these is that the stable ABI documentation says that every documented > symbol is part of the stable ABI unless there’s explicit documentation to > the contrary. This makes researching if functions are intended to be part of > the stable ABI harder. > > And also: > > 4. Does the stable ABI actually work? > > Christian’s script finds cases where exposed names don’t actually work when > you try to use them. > > > To reply to myself, the gist below is a very crude version of what I was > trying to suggest: > > https://gist.github.com/ronaldoussoren/fe4f80351a7ee72c245025df7b2ef1ed#file-gistfile1-txt > > The gist is far from usable, but shows some tests that check that symbols in > the stable ABI can be used, and tests that everything exported in the stable > ABI is actually tested. > > Again, the code in the gist is a crude hack and I have currently no plans to > turn this into something that could be added to the testsuite. > > Ronald > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2018 Python Language Summit coverage
Thanks for doing this Jake. -eric On Wed, Jun 6, 2018 at 3:56 PM, Jake Edge wrote: > > Hola python-dev, > > I have been remiss in posting about my coverage from this year's Python > Language Summit -- not to mention remiss in getting it all written up. > But I am about half-way done with the sessions from this year. > > I am posting SubscriberLinks for articles that are still behind the > paywall. LWN subscribers can always see our content right away; one > week after they are published in a weekly edition, they become freely > available for everyone. SubscriberLinks are a way around the paywall. > Please feel free to share the SubscriberLinks I am posting here. > > The starting point is here: https://lwn.net/Articles/754152/ That is > an overview article with links to the articles. It will be updated as > I add more articles. Here is what we have so far: > > - Subinterpreter support for Python https://lwn.net/Articles/754162/ > > - Modifying the Python object model https://lwn.net/Articles/754163/ > > - A Gilectomy update https://lwn.net/Articles/754577/ > > - Using GitHub Issues for Python https://lwn.net/Articles/754779/ > > - Shortening the Python release schedule > https://lwn.net/Articles/755224/ > > - Unplugging old batteries > https://lwn.net/SubscriberLink/755229/df78cf17181dbdca/ > > Hopefully I captured things reasonably well -- if you have corrections > or clarifications (or just comments :) , I would recommend posting them > as comments on the article. > > I will post an update soon with the next round (with luck, all of the > rest of them). > > enjoy! > > jake > > -- > Jake Edge - LWN - j...@lwn.net - http://lwn.net > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] segfaults due to hash randomization in C OrderedDict
(see http://bugs.python.org/issue16991) I an working on resolving an intermittent segfault that my C OrderedDict patch introduces. The failure happens in test_configparser (RawConfigParser uses OrderedDict internally), but only sporadically. However, Ned pointed out to me that it appears to be related to hash randomization, which I have verified. I'm looking into it. In the meantime, here's a specific question. What would lead to the pattern of failures I'm seeing? I've verified that the segfault happens consistently for certain hash randomization seeds and never for the rest. I don't immediately recognize the pattern but expect that it would shed some light on where the problem lies. I ran the following command with the OrderedDict patch applied: for i in `seq 1 100`; do echo $i; PYTHONHASHSEED=$i ./python -m test.regrtest -m test_basic test_configparser ; done Through 100 I get segfaults with seeds of 7, 15, 35, 37, 39, 40, 42, 47, 50, 66, 67, 85, 87, 88, and 92. I expect the distribution across all seeds is uniform, but I haven't verified that. Thoughts? -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] segfaults due to hash randomization in C OrderedDict
Good catch. Unfortunately, sticking "keys = ((PyDictObject *)od)->ma_keys;" right after "hash = ..." did not make a difference. I still get the same segfault. -eric On Thu, May 21, 2015 at 11:17 AM, MRAB wrote: > On 2015-05-21 15:55, Eric Snow wrote: >> >> (see http://bugs.python.org/issue16991) >> >> I an working on resolving an intermittent segfault that my C >> OrderedDict patch introduces. The failure happens in >> test_configparser (RawConfigParser uses OrderedDict internally), but >> only sporadically. However, Ned pointed out to me that it appears to >> be related to hash randomization, which I have verified. I'm looking >> into it. >> >> In the meantime, here's a specific question. What would lead to the >> pattern of failures I'm seeing? I've verified that the segfault >> happens consistently for certain hash randomization seeds and never >> for the rest. I don't immediately recognize the pattern but expect >> that it would shed some light on where the problem lies. I ran the >> following command with the OrderedDict patch applied: >> >>for i in `seq 1 100`; do echo $i; PYTHONHASHSEED=$i ./python -m >> test.regrtest -m test_basic test_configparser ; done >> >> Through 100 I get segfaults with seeds of 7, 15, 35, 37, 39, 40, 42, >> 47, 50, 66, 67, 85, 87, 88, and 92. I expect the distribution across >> all seeds is uniform, but I haven't verified that. >> >> Thoughts? >> > In "_odict_get_index", for example (there are others), you're caching > "ma_keys": > > PyDictKeysObject *keys = ((PyDictObject *)od)->ma_keys; > > If it resizes, you go back to the label "start", which is after that > line, but could "ma_keys" change when it's resized? > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] segfaults due to hash randomization in C OrderedDict
On Thu, May 21, 2015 at 4:06 PM, MRAB wrote: > On 2015-05-21 22:52, Eric Snow wrote: >> Good catch. Unfortunately, sticking "keys = ((PyDictObject >> *)od)->ma_keys;" right after "hash = ..." did not make a difference. >> I still get the same segfault. > > So, does it change sometimes? The segfault is consistent if I use the same seed (e.g. 7): PYTHONHASHSEED=7 ./python -m test.regrtest -m test_basic test_configparser Some seeds always segfault and some seeds never segfault. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] segfaults due to hash randomization in C OrderedDict
On Thu, May 21, 2015 at 4:41 PM, MRAB wrote: > On 2015-05-21 23:17, Eric Snow wrote: >> The segfault is consistent if I use the same seed (e.g. 7): >> >> PYTHONHASHSEED=7 ./python -m test.regrtest -m test_basic >> test_configparser >> >> Some seeds always segfault and some seeds never segfault. >> > OK, another thought. > > In "_odict_get_index" again, you say that if the hash has changed, the dict > might've > been resized, but could the dict be resized _without_ the hash changing? > > Could the value of "keys" still become invalid even if the hash is the same? Good question. The only way I can see here that the dict would resize is during re-entrance to the interpreter eval loop via Python code potentially triggered through the PyObject_Hash call. Also, there's no check for a changed hash. The code compares the size of ma_keys (effectively the dict keys hash table) against the size of of the odict "fast nodes" table. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] segfaults due to hash randomization in C OrderedDict
On Thu, May 21, 2015 at 5:55 PM, MRAB wrote: > I'm not looking at the use of "PyTuple_Pack". As I understand it, > "PyTuple_Pack" borrows the > references of the objects passed, and when the tuple itself is DECREFed, > those objects will be > DECREFed >From the docs [1] it seems that PyTuple_Pack does not steal any references and it returns a new reference. Perhaps you were thinking of PyTuple_SetItem (and PyTuple_SET_ITEM)? [1] https://docs.python.org/3.5//c-api/tuple.html > > "odict_reduce" calls "PyTuple_Pack", passing 1 or 2 references to Py_None > which aren't INCREFed > first, so could there be a bug there? (There might be similar issues in > other functions.) Alas, I don't think it is. :( I'll point out that the configparser test in question does a lot of resizes. It may be that the problem only surfaces after many resizes and apparently only for certain hash randomization seeds. At the moment I'm looking at how hash randomization impacts resizing. I'm certainly seeing that the resizes happen at different item counts depending on the seed. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] segfaults due to hash randomization in C OrderedDict
On Thu, May 21, 2015 at 6:22 PM, MRAB wrote: > Oh, well, I'll keep looking... Thanks! -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] segfaults due to hash randomization in C OrderedDict
On Thu, May 21, 2015 at 6:22 PM, MRAB wrote: > Oh, well, I'll keep looking... I've posted some data to http://bugs.python.org/issue16991 that I hope will shed some light on the issue. We can continue the conversation there. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Accepting PEP 489 (Multi-phase extension module initialization)
Hi all, After extended discussion over the last several months on import-sig, the resulting proposal for multi-phase (PEP 451) extension module initialization has finalized. The resulting PEP provides a clean, straight-forward, and backward-compatible way to import extension modules using ModuleSpecs. With that in mind and given the improvement it provides, PEP 489 is now accepted. I want to thank Petr, Nick, and Stefan for the time, thought, and effort they put into the proposal (and implementation). It was a disappointment to me when, at the time, we couldn't find a good way to apply PEP 451 to builtins and extension modules. So thanks for easing my anxiety! -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] peps: PEP 489: The PEP is accepted.
On Sat, May 23, 2015 at 2:22 PM, Brett Cannon wrote: > Are you also going to check the code in or is someone else doing it? Nick already did: http://bugs.python.org/issue24268 https://hg.python.org/cpython/rev/e729b946cc03 :) -eric > > > On Fri, May 22, 2015, 17:47 eric.snow wrote: >> >> https://hg.python.org/peps/rev/1fbc23a1078c >> changeset: 5874:1fbc23a1078c >> user:Eric Snow >> date:Fri May 22 15:45:38 2015 -0600 >> summary: >> PEP 489: The PEP is accepted. >> >> files: >> pep-0489.txt | 11 --- >> 1 files changed, 8 insertions(+), 3 deletions(-) >> >> >> diff --git a/pep-0489.txt b/pep-0489.txt >> --- a/pep-0489.txt >> +++ b/pep-0489.txt >> @@ -7,13 +7,13 @@ >> Nick Coghlan >> BDFL-Delegate: Eric Snow >> Discussions-To: import-...@python.org >> -Status: Draft >> +Status: Final >> Type: Standards Track >> Content-Type: text/x-rst >> Created: 11-Aug-2013 >> Python-Version: 3.5 >> -Post-History: 23-Aug-2013, 20-Feb-2015, 16-Apr-2015 >> -Resolution: >> +Post-History: 23-Aug-2013, 20-Feb-2015, 16-Apr-2015, 7-May-2015, >> 18-May-2015 >> +Resolution: >> https://mail.python.org/pipermail/python-dev/2015-May/140108.html >> >> >> Abstract >> @@ -730,8 +730,13 @@ >> >> * PyModuleDef_Slot >> >> +Other changes: >> + >> PyModuleDef.m_reload changes to PyModuleDef.m_slots. >> >> +``BuiltinImporter`` and ``ExtensionFileLoader`` will now implement >> +``create_module`` and ``exec_module``. >> + >> The internal ``_imp`` module will have backwards incompatible changes: >> ``create_builtin``, ``create_dynamic``, and ``exec_dynamic`` will be >> added; >> ``init_builtin``, ``load_dynamic`` will be removed. >> >> -- >> Repository URL: https://hg.python.org/peps >> ___ >> Python-checkins mailing list >> python-check...@python.org >> https://mail.python.org/mailman/listinfo/python-checkins > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Preserving the definition order of class namespaces.
tl;dr Are there any objections to making making the default cls.__prepare__ return OrderedDict instead of dict (and preserve that order in a list on the class)? A couple years ago [1][2] I proposed making class definition namespaces use OrderedDict by default. Said Guido [3]: I'm fine with doing this by default for a class namespace; the type of cls.__dict__ is already a non-dict (it's a proxy) and it's unlikely to have 100,000 entries. It turns out making cls.__dict__ an OrderedDict isn't reasonably tractable (due to the concrete API v. subclasses), but really that isn't what I was looking for anyway. Regardless, since it's been a while I just want to run the proposal by the group again. I'm hopeful about landing my C implementation of OrderedDict [4] in the next few days. Also, I have a patch up [5] that implements using OrderedDict for class definitions. So mostly I just want to double check that I'm still good to go. Just to be clear on what I'm proposing specifically, I've summed it up below. -eric - Currently if you want to preserve the order of a class definition you have to use a metaclass with a __prepare__ method (see PEP 3115). However, as that PEP points out [6], the common case for __prepare__ is to use OrderedDict. I'm proposing that we return OrderedDict() by default from __prepare__. Considering the common case, we should also expose that definition order on the class afterward since otherwise the extra information from the class definition namespace is discarded (type.__new__ copies it into a dict which is then used for cls.__dict__). So the key changes are: * use OrderedDict by default for class definition namespace (e.g. from type.__prepare__) * expose that definition order as cls.__definition_order__ (a list) (Note that I will not be changing the actual type of cls.__dict__ (i.e. tp_dict) which will remain a dict.) The effect of the change would be that the following are basically equivalent (relative to the the definition namespace): class Meta(type): @classmethod. def __prepare__(meta, *args, **kwargs): return OrderedDict() class SpamOld(metaclass=Meta): a = 1 b = 2 c = 3 __definition_order__ = list(locals()) class SpamNew: a = 1 b = 2 c = 3 assert SpamOld.__definition__order == SpamNew.__definition_order__ The key differences are: * for SpamNew you don't need to use a metaclass [7][8] * for SpamNew you don't need to rely on the behavior of locals() * for SpamNew the class definition isn't cluttered with extra boilerplate for __definition_order__ * class decorators that care about definition order [9] don't have to require that classes like SpamNew manually preserve that order somehow The patch for the change is pretty minimal. [5] Also, Nick Coghlan recently expressed that he favored using OrderedDict by default over the alternative presented by PEP 422/487. [10] [1] https://mail.python.org/pipermail/python-ideas/2013-February/019690.html [2] https://mail.python.org/pipermail/python-dev/2013-June/127103.html [3] Guido: https://mail.python.org/pipermail/python-ideas/2013-February/019704.html [4] http://bugs.python.org/issue16991 [5] http://bugs.python.org/issue24254 [6] see the "Alternate Proposals" section of https://www.python.org/dev/peps/pep-3115/ [7] PEPs 422 and 487 relatedly focus on the benefits of reducing the need to use metaclasses [8] https://mail.python.org/pipermail/python-ideas/2013-February/019706.html [9] see "Key points" on https://mail.python.org/pipermail/python-dev/2013-February/124439.html [10] Nick: https://mail.python.org/pipermail/python-ideas/2015-March/032254.html ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Preserving the definition order of class namespaces.
On May 23, 2015 10:47 PM, "Guido van Rossum" wrote: > > How will __definition_order__ be set in the case where __prepare__ doesn't return an OrderedDict? Or where a custom metaclass's __new__ calls its superclass's __new__ with a plain dict? (I just wrote some code that does that. :-) I was planning on setting it to None if the order is not available. At the moment that's just a check for OrderedDict. -eric > > On Sat, May 23, 2015 at 7:38 PM, Nick Coghlan wrote: >> >> On 24 May 2015 at 12:04, Nick Coghlan wrote: >> > On 24 May 2015 at 11:15, Eric Snow wrote: >> >> tl;dr Are there any objections to making making the default >> >> cls.__prepare__ return OrderedDict instead of dict (and preserve that >> >> order in a list on the class)? >> >> >> >> A couple years ago [1][2] I proposed making class definition >> >> namespaces use OrderedDict by default. Said Guido [3]: >> >> >> >> I'm fine with doing this by default for a class namespace; the type of >> >> cls.__dict__ is already a non-dict (it's a proxy) and it's unlikely to >> >> have 100,000 entries. >> >> >> >> It turns out making cls.__dict__ an OrderedDict isn't reasonably >> >> tractable (due to the concrete API v. subclasses), but really that >> >> isn't what I was looking for anyway. >> >> >> >> Regardless, since it's been a while I just want to run the proposal by >> >> the group again. I'm hopeful about landing my C implementation of >> >> OrderedDict [4] in the next few days. Also, I have a patch up [5] >> >> that implements using OrderedDict for class definitions. So mostly I >> >> just want to double check that I'm still good to go. >> > >> > While it isn't controversial (since you already have the +1 from >> > Guido), it's worth writing up the change as a PEP for 3.6 anyway, >> > since that then provides clearer guidance to alternate implementations >> > that they're going to need to change the way their class namespace >> > evaluation works for 3.6. >> >> Eric clarified for me that Larry was considering granting a feature >> freeze exemption to defer landing this to beta 2 while Eric tracked >> down a segfault bug in the current patch that provides a C >> implementation of OrderedDict. That sounds like a nicer approach than >> what I did for PEP 489 (where I checked in an initial version that I >> knew still had a refleak bug in it), so +1 from me for going down that >> path. >> >> A top level section in the What's New would cover my concerns >> regarding making sure folks are suitably aware of the change (as I >> believe leaving it out of the original 2.6 What's New document was the >> real problem with making people aware of the addition of zip archive >> and directory execution support). >> >> Regards, >> Nick. >> >> -- >> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Preserving the definition order of class namespaces.
On May 24, 2015 3:35 AM, "Nick Coghlan" wrote: > Is it specifically necessary to save the order by default? Metaclasses > would be able to access the ordered namespace in their __new__ method > regardless, and for 3.6, I still like the __init_subclass__ hook idea > proposed in PEP 487, which includes passing the original namespace to > the new hook. > > So while I'm sold on the value of making class execution namespaces > ordered by default, I'm not yet sold on the idea of *remembering* that > order without opting in to doing so in the metaclass. > > If we leave __definition_order__ out for the time being then, for the > vast majority of code, the fact that the ephemeral namespace used to > evaluate the class body switched from being a basic dictionary to an > ordered one would be a hidden implementation detail, rather than > making all type objects a little bigger. It's too late for 3.5 to negotiate much so I'll try to make my case here for __definition_order__ one last time. If that's not sufficient then I'll defer further discussion to 3.6. My premise for storing the definition order on the class is that Guido was okay with using OrderedDict for cls.__dict__, which is a bigger change. Regardless, there are two reasons why it makes sense: * If it makes sense to use OrderedDict by default for class definition then it makes sense to preserve the extra information OrderedDict provides. * As I noted at the beginning of the thread, you could still preserve that info manually, but that makes it less convenient for library authors. If you still think that's not enough justification then we can table __definition_order__ for now. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Preserving the definition order of class namespaces.
On May 24, 2015 4:52 PM, "Nick Coghlan" wrote: > > > On 25 May 2015 07:26, "Guido van Rossum" wrote: > > > > On Sun, May 24, 2015 at 1:36 PM, Eric Snow wrote: > >> If you still think that's not enough justification then we can table __definition_order__ for now. > > > > > > Let's table it. It's hard to compare alternatives on a single dimension of "which is a bigger change". Sounds good. > > Right, it isn't that I think __definition_order__ is necessarily a bad idea, I just suspect it's redundant if we end up going ahead with __init_subclass__ (which would allow a base class to opt in to preserving the definition order, either of all fields or selected ones), > and the latter change is definitely out of scope for 3.5 at this point. > > There are also other open questions, like whether or not dir() should respect the order when reporting attribute names, or if dict_proxy should respect the order when iterating. Yeah, I'll start up a thread on python-ideas once I've gotten the other stuff wrapped up. Thanks for the feedback. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Preserving the definition order of class namespaces.
On Mon, May 25, 2015 at 1:33 AM, Antoine Pitrou wrote: > On Sat, 23 May 2015 20:14:56 -0700 > Larry Hastings wrote: >> Yeah, I'm willing to grant the feature freeze exception, assuming he can >> find general approval from the community (and assuming he still has >> Guido's blessing). I just wanted a little more sunlight on the topic, >> rather than rushing to check it in. > > Given the pain that has gone into making the patch segfault- and > reference leak-free, and given it adds a lot of complication in the > data types area, I'm frankly uneasy with having this land after the > feature freeze. It's a sure recipe to *add* instability rather than > remove it. Well, the exception for C OrderedDict itself is a separate matter. I chose to be more public than I could have been about the last remaining bugs in the interest of getting them resolved a bit faster. At this point I wouldn't consider C OrderedDict to add a whole lot of risk to 3.5. That said, at this point landing it in 3.5 it doesn't matter to me much because my main motivator (__definition_order__) isn't landing in 3.5. The fact that 3.6 is already open to new features eases the sting a bit. I'd still prefer to land OrdereDict-by-default class definition namespaces in 3.5, which is dependent on C OrderedDict, but alone that isn't as important to me for 3.5 as cls.__definition_order__ was. Regardless, I know there were a few folks (e.g. Yury) that wanted to see C OrderedDict in 3.5 and there may be others that would really like OrderedDict-by-default in 3.5 (Nick?). Since Larry already gave an exception, I'd still be glad to land both in 3.5 if Yury (or others) wants to make that case. The patches will be ready. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Preserving the definition order of class namespaces.
On Mon, May 25, 2015 at 2:40 PM, Terry Reedy wrote: > On 5/25/2015 3:40 PM, Eric Snow wrote: >> Since Larry already gave an exception, > > Conditional on 'general approval of the community'. Unless I misunderstood him, Larry gave me an unconditional exception for OrderedDict itself (as long as it is in before beta 2.) The condition only applied to making OrderedDict the default class definition namespace and adding cls.__definition_order__. Furthermore, the condition related to the semantic changes to Python, not to concerns about destabilizing Python. I don't mean to imply that Larry can't retract (or modify) the exceptions he's given me. In fact, if there is sufficient risk of de-stabilizing the release then I'd expect him to do so. However, ultimately that's his call as release manager; and I do not believe there is any greater risk now than what I explained to him in our discussions leading up to the exceptions I received. -eric > >> I'd still be glad to land both in 3.5 if Yury (or >> >> others) wants to make that case. The patches will be ready. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Preserving the definition order of class namespaces.
On Mon, May 25, 2015 at 6:30 PM, Larry Hastings wrote: > Eric asked for one for this C reimplementation of OrderedDict; the coding > was done, the debugging wasn't. > > And yes, as Eric said, I made separate pronouncements. I said COrderedDict > could go in as long as it was in before beta 2; "the other work" of > __definition_order__ and switching type_prepare and __build_class__ to using > ordered dicts I made conditional on "general approval of the community." > The latter has already been tabled for now. > > > So, in all three cases it's work that's been under development for a while. > These people did this work out of the kindness of their hearts, to make > Python better. As a community we want to encourage that and make sure these > developers know we appreciate their efforts. These people would be happier > if the work shipped in 3.5 as opposed to 3.6 so it got into user's hands > sooner. > > Also, in Serhiy and Eric's cases, these are reimplementations of existing > Python libraries in C. On the one hand, that means we should have good > regression test coverage in the library--which it seems like we do, as both > of them are debugging problems uncovered by the regression tests. This > gives us a little more confidence that the work is good. On the other hand, > it does mean there's a higher chance of destabilization, as there's already > an installed base using these libraries. (As opposed to something new like > math.isclose which has no installed base.) So yes this could introduce bugs > that will impact existing users. > > > Bottom line: while an important part job of my job is saying "no", I also > feel like an important part of my job is saying "yes". On balance, what > will be best for Python? In these cases, I think "yes" is better. My > feeling is, let's check it in (before beta 2), and if it causes problems > during the betas / rcs we can back them out. Thanks, Larry. As to the conditional exceptions, I'm just going to drop those entirely in favor of getting them in 3.6. I'll still pursue OrderedDict for 3.5b2 though (and hope to land it this week). -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On Thu, May 28, 2015 at 5:01 PM, Paul Sokolovsky wrote: > That said, making a demo of self-contained webapp server in 350-400K is > definitely on my TODO list (package support for frozen modules is the > only blocker for that). It may be worth taking this over to import-...@python.org for more discussion. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Profile Guided Optimization active by-default
On Aug 22, 2015 9:02 AM, "Patrascu, Alecsandru" < alecsandru.patra...@intel.com> wrote: [snip] > For instance, as shown from attached sample performance results from the Grand Unified Python Benchmark, >20% speed up was observed. Are you referring to the tests in the benchmarks repo? [1] How does the real-world performance improvement compare with other languages you are targeting for optimization? And thanks for working on this! I have several more questions: What sorts of future changes in CPython's code might interfere with your optimizations? What future additions might stand to benefit? What changes in existing code might improve optimization opportunities? What is the added maintenance burden of the optimizations on CPython, if any? What is the performance impact on non-Intel architectures? What about older Intel architectures? ...and future ones? What is Intel's commitment to supporting these (or other) optimizations in the future? How is the practical EOL of the optimizations managed? Finally, +1 on adding an opt-in Makefile target rather than enabling the optimizations by default. Thanks again! -eric [1] https://hg.python.org/benchmarks/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Profile Guided Optimization active by-default
On Aug 24, 2015 3:51 PM, "Stewart, David C" wrote: > > (Sorry about the format here - I honestly just subscribed to Python-dev so > be gentle ...) :) > > > Date: Sat, 22 Aug 2015 11:25:59 -0600 > > From: Eric Snow > > >On Aug 22, 2015 9:02 AM, "Patrascu, Alecsandru" >intel.com <https://mail.python.org/mailman/listinfo/python-dev>> > >wrote:[snip]> For instance, as shown from attached sample performance > >results from theGrand Unified Python Benchmark, >20% speed up was > >observed. > > > > > > Eric I'm the manager of Intel's server scripting language optimization > team, so I'll answer from that perspective. Thanks, David! > > >Are you referring to the tests in the benchmarks repo? [1] How does the > >real-world performance improvement compare with otherlanguages you are > >targeting for optimization? > > Yes, we're using [1]. > > We're seeing up to 10% improvement on Swift (a project in OpenStack) on > some architectures using the ssbench workload, which is as close to > real-world as we can get. Cool. > Relative to other languages we target, this is > quite good actually. For example, Java's Hotspot JIT is driven by > profiling at its core so it's hard to distinguish the value profiling > alone brings. Interesting. So pypy (with it's profiling JIT) would be in a similar boat, potentially. > We have seen a nice boost on PHP running Wordpress using > PGO, but not as impressive as Python and Swift. Nice. Presumably this reflects some of the choices we've made on the level of complexity in the interpreter source. > > By the way, I think letting the compiler optimize the code is a good > strategy. Not the only strategy we want to use, but it seems like one we > could do more of. > > > And thanks for working on this! I have several more questions: What > >sorts of future changes in CPython's code might interfere with > >youroptimizations? > > > > > > We're also looking at other source-level optimizations, like the CGOTO > patch Vamsi submitted in June. Some of these may reduce the value of PGO, > but in general it's nice to let the compiler do some optimization for you. > > > What future additions might stand to benefit? > > > > It's a good question. Our intent is to continue to evaluate and measure > different training workloads for improvement. In other words, as with any > good open source project, this patch should improve things a lot and > should be accepted upstream, but we will continue to make it better. > > > What changes in existing code might improve optimization opportunities? > > > > > > We intend to continue to work on source-level optimizations and measuring > them against GUPB and Swift. Thanks! These sorts of contribution has far-reaching positive effects. > > > What is the added maintenance burden of the optimizations on CPython, > >ifany? > > > > > > I think the answer is none. Our goal was to introduce performance > improvements without adding to maintenance effort. > > >What is the performance impact on non-Intel architectures? What > >aboutolder Intel architectures? ...and future ones? > > > > > > We should modify the patch to make it for Intel only, since we're not > evaluating non-Intel architectures. Unfortunately for us, I suspect that > older Intel CPUs might benefit more than current and future ones. Future > architectures will benefit from other enabling work we're planning. That's fine though. At the least you're setting the stage for future work, including building a relationship here. :) > > > What is Intel's commitment to supporting these (or other) optimizations > >inthe future? How is the practical EOL of the optimizations managed? > > > > > > As with any corporation's budgeting process, it's hard to know exactly > what my managers will let me spend money on. :-) But we're definitely > convinced of the value of dynamic languages for servers and the need to > work on optimization. As far as I have visibility, it appears to be > holding true. Sounds good. > > > Finally, +1 on adding an opt-in Makefile target rather than enabling > >theoptimizations by default. > > > > > > Frankly since Ubuntu has been running this way for past two years, I think > it's fine to make it opt-in, but eventually I hope it can be the default > once we're happy with it. Given the reaction here that sounds reasonable. Thanks for answering these questions and to your team for getting involved! -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] type(obj) vs. obj.__class__
In a recent tracker issue about OrderedDict [1] we've had some discussion about the use of type(od) as a replacement for od.__class__. It came up because the pure Python implementation of OrderedDict uses self.__class__ in 3 different methods (__repr__, __reduce__, and copy). The patch in that issue changes the C implementation to use Py_TYPE(). [2] So I wanted to get some feedback on the practical implications of such a change and if we need to clarify the difference more formally. In this specific case [3] there are 3 questions: * Should __repr__() for a stdlib class use type(self).__name__ or self.__class__.__name__? * Should __reduce__() return type(self) or self.__class__? * Should copy() use type(self) or self.__class__? The more general question of when we use type(obj) vs. obj.__class__ applies to both the language and to all the stdlib as I expect consistency there would result in fewer surprises. I realize that there are some places where using obj.__class__ makes more sense (e.g. for some proxy support). There are other places where using type(obj) is the way to go (e.g. special method lookup). However, the difference is muddled enough that usage is inconsistent in the stdlib. For example, C-implemented types use Py_TYPE() almost exclusively. So, would it make sense to establish some concrete guidelines about when to use type(obj) vs. obj.__class__? If so, what would those be? It may also be helpful to enumerate use cases for "type(obj) is not obj.__class__". -eric [1] http://bugs.python.org/issue25410 [2] I'm going to open a separate thread about the issue of compatibility and C accelerated types. [3] https://hg.python.org/cpython/file/default/Lib/collections/__init__.py#l238 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] compatibility for C-accelerated types
A recent discussion in a tracker issue [1] brought up the matter of compatibility between the pure Python implementation of OrderedDict and the new C implementation. In working on that port I stuck as closely as possible to the Python implementation. This meant some parts of the code are bit more complex than they would be otherwise. (Serhiy has been kind enough to do some cleanup.) Compatibility was one of the fundamental goals of the porting effort. Not only does compatibility make sense but it's also specifically required by PEP 399 [2]: Any new accelerated code must act as a drop-in replacement as close to the pure Python implementation as reasonable. Technical details of the VM providing the accelerated code are allowed to differ as necessary, e.g., a class being a type when implemented in C. For the most part I have questions about what is "reasonable", specifically in relation to OrderedDict. I've already opened up a separate thread related to my main question: type(obj) vs. obj.__class__. [3] In the tracker issue, Serhiy pointed out: There is no a difference. io, pickle, ElementTree, bz2, virtually all accelerator classes was created as replacements of pure Python implementations. All C implementations use Py_TYPE(self) for repr() and pickling. I think this deviation is common and acceptable. In a review comment on the associated patch he said: Isn't type(self) is always the same as self.__class__ for pure Python class? If right, then this change doesn't have any effect. To which he later replied: It is the same if you assigned the __class__ attribute, but can be different if set __class__ in the subclass declaration. So it isn't clear if that is a compatibility break or how much so it might be. Serhiy also noted that, as of 3.5 [4], you can no longer assign to obj.__class__ for instances of subclasses of builtin (non-heap) types. So that is another place where the two OrderedDict implementations differ. I expect there are a few others in dark corner cases. On the tracker he notes another OrderedDict compatibility break: Backward compatibility related to __class__ assignment was already broken in C implementation. In 3.4 following code works: >>> from collections import * >>> class foo(OrderedDict): ... def bark(self): return "spam" ... >>> class bar(OrderedDict): ... pass ... >>> od = bar() >>> od.__class__ = foo >>> od.bark() 'spam' In 3.5 it doesn't. As PEP 399 says, we should go as far "as reasonable" in the pursuit of compatibility. At the same time, I feel not insignificant responsibility for *any* incompatibility that comes from the C implementation of OrderedDict. The corner cases impacted by the above compatibility concerns are borderline enough that I wanted to get some feedback. Thanks! -eric [1] http://bugs.python.org/issue25410 [2] https://www.python.org/dev/peps/pep-0399/ [3] https://mail.python.org/pipermail/python-dev/2015-October/141953.html [4] http://bugs.python.org/issue24912 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compatibility for C-accelerated types
On Mon, Oct 19, 2015 at 3:00 PM, Guido van Rossum wrote: > Apart from Serhiy's detraction of the 3.5 bug report there wasn't any > discussion in this thread. I also don't really see any specific questions, > so maybe you don't have any. Are you just asking whether it's okay to merge > your code? Or are you asking for more code review? Basically, I've had all my questions answered. PEP 399 covers the matter well enough. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compatibility for C-accelerated types
On Tue, Oct 20, 2015 at 2:38 AM, Maciej Fijalkowski wrote: > For what is worth, that level of differences already exists on pypy > and it's really hard to get the *exact* same semantics if things are > implemented in python vs C or the other way around. > > Example list of differences (which I think OrderedDict already breaks > if moved to C): > > * do methods like items call special methods like __getitem__ (I think > it's undecided anyway) > > * what happens if you take a method and rebind it to another subclass, > does it automatically become a method (there are differences between > built in and pure python) > > * atomicity of operations. Some operations used to be non-atomic in > Python will be atomic now. > > I personally think those (and the __class__ issue) are unavoidable Yeah, I figured as much. Thanks for pointing those out. Perhaps it would be useful to enumerate specific cases like these in PEP 399? They could go near the part that says "as close to the pure Python implementation as reasonable". -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compatibility for C-accelerated types
On Tue, Oct 20, 2015 at 9:10 AM, Guido van Rossum wrote: > Please go ahead and update PEP 399. Will do. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] Daily reference leaks (d7e490db8d54): sum=61494
On Wed, Oct 21, 2015 at 9:32 AM, Raymond Hettinger wrote: > I'm re-opening > https://mail.python.org/pipermail/python-dev/2015-October/141993.html Presumably you meant http://bugs.python.org/issue24379. :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytcode "magic tag"
On Wed, Oct 28, 2015 at 6:35 AM, Eric V. Smith wrote:
> In issue 25483 I'm adding an opcode to make f-string formatting more
> robust and faster. As part of that, I'm bumping the .pyc magic number.
>
> While doing that, I notice Lib/importlib/_bootstrap_external.h includes
> this comment:
>
> # Starting with the adoption of PEP 3147 in Python 3.2, every bump in magic
> # number also includes a new "magic tag", i.e. a human readable string used
> # to represent the magic number in __pycache__ directories. When you change
> # the magic number, you must also set a new unique magic tag. Generally
> this
> # can be named after the Python major version of the magic number bump, but
> # it can really be anything, as long as it's different than anything else
> # that's come before. The tags are included in the following table,
> starting
> # with Python 3.2a0.
>
> The "following table" is a comment, that contains a few references to
> the tag "cpython-", specifically cpython-32. It doesn't seem
> that the tag is routinely updated in the comment.
>
> sys.implementation.cache_tag returns 'cpython-36', and is in fact
> implemented as 'cpython-{PY_MAJOR_VERSION}{PY_MINOR_VERSION}'.
>
> Do I need to do anything else? Unlike what the comment in
> _boostrap_external.py suggests, this "magic tag" will not change every
> time a bytecode is added, but only on every minor release. Which implies
> that if we have a micro release that adds an opcode, we'll in fact break
> the promise in the comment.
You will want to bump the number on the following line:
https://hg.python.org/cpython/file/default/Lib/importlib/_bootstrap_external.py#l231
-eric
>
> From my understanding on how this tag is used, this wouldn't be a
> problem (because the magic number in the file also changes). But I want
> to make sure I'm not misunderstanding something. I think the comment
> above is probably just misleading.
>
> Eric.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytcode "magic tag"
On Wed, Oct 28, 2015 at 8:28 AM, Eric V. Smith wrote: > Thanks. That part I've done (but forgot to mention). I was just > concerned about the "magic tag" part, which Barry cleared up. Ah, I misread. :) Yeah, that comment is out of date. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python environment registration in the Windows Registry
On Tue, Feb 2, 2016 at 10:15 PM, Steve Dower wrote: > I was throwing around some ideas with colleagues about how we detect Python > installations on Windows from within Visual Studio, and it came up that > there are many Python distros that install into different locations but > write the same registry entries. (I knew about this, of course, but this > time I decided to do something.) > > [snip] > > So here is a rough proposal to standardise the registry keys that can be set > on Windows in a way that (a) lets other installers besides the official ones > have equal footing, (b) provides consistent search and resolution semantics > for tools, and (c) includes slightly more rich metadata (such as display > names and URLs). Presented in PEP-like form here, but if feedback suggests > just putting it in the docs I'm okay with that too. It is fully backwards > compatible with official releases of Python (at least back to 2.5, possibly > further) and does not require modifications to Python or the official > installer - it is purely codifying a superset of what we already do. > > Any and all feedback welcomed, especially from the owners of other distros, > Python implementations or tools on the list. Just wanted to quickly point out another use of the WIndows registry in Python: WindowsRegistryFinder [1]. This is an import "meta-path" finder that locates modules declared (*not* defined) in the registry. I'm not familiar with the Windows registry nor do I know if anyone is using this finder. That said, ISTM the finder's use of the registry does not face quite the same challenges you've described in the proposal. I expect Martin von Löwis could explain more as he was involved with adding the finder. Just wanted to throw that out there, particularly if there's a chance of the finder's registry keys conflicting in some way. -eric [1] https://hg.python.org/cpython/file/5873cfb42ebe/Lib/importlib/_bootstrap_external.py#l570 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accepted: PEP 493 (HTTPS verification migration tools for Python 2.7)
On Wed, Mar 2, 2016 at 8:02 AM, Barry Warsaw wrote: > As BDFL-Delegate, I am officially accepting PEP 493. > > Congratulations Nick, Robert, and MAL. I want to personally thank Nick for > taking my concerns into consideration, and re-working the PEP to resolve > them. Thanks also to everyone who contributed to the discussion. Yeah, congrats! And thanks for taking on something that (in my mind) isn't the most exciting thing to work on. :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] When should pathlib stop being provisional?
On Apr 6, 2016 14:00, "Barry Warsaw" wrote: > Aside from the name of the attribute (though I'm partial to __path__), Ahem, pkg.__path__. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol
On Apr 7, 2016 1:22 AM, "Georg Brandl" wrote: > > On 04/06/2016 07:26 PM, Brett Cannon wrote: > > 1. Name: __path__, __fspath__, or something else? > > __path__ is already taken as a module attribute, so I would avoid it. > __fspath__ is fine with me, although the more explicit variants are also > ok. It's not like you need to read/write it constantly (that's the goal). +1 I also think that __ospath__ may be more correct since it is an OS-dependent representation, e.g. slash vs. backslash. > > > 2. Method or attribute? (changes what kind of one-liner you might use in > > libraries, but I think historically all protocols have been methods and the > > serialized string representation might be costly to build) > > An attribute would be somewhat inconsistent with the special-method lookup rules > (looked up on the type, not the instance), so a method is probably a better > choice. I was just about to point this out. The deviation by pickle (lookup on instance rather than type) has been a source of pain. > > > 3. Built-in? (name is dependent on #1 if we add one) > > I don't think it warrants a builtin. I'd place it as a function in pathlib. +1 > > > 4. Add the method/attribute to str? (I assume so, much like __index__() is on > > int, but I have not seen it explicitly stated so I would rather clarify it) > > +1. +1 > > > 5. Expand the C API to have something like PyObject_Path()? > > +1 (with _Py_ at first) since you're going to need it in a lot of C functions. +1 -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Other pathlib improvements? was: When should pathlib stop being provisional?
On Apr 6, 2016 11:11 PM, "Raymond Hettinger" wrote: > Having worked through the API when it is first released, I find it to be highly forgettable (i.e. I have to re-read the docs each time I've revisited it). Agreed, though it's arguably better than argparse, logging, unittest, or several other stdlib modules. To some extent the challenge with those is the complexity of the problem space. Furthermore, the key for any sufficiently complex module is that the common-case usage is intuitive and simple enough. Some stdlib modules do a better job of that than others. :/ How much would you say that any of that applies to pathlib? What about relative to other similar packages on the cheeseshop? Regardless, are there any specific improvements you'd recommend while the module is still provisional? Are your concerns a matter of structure vs. naming? Usability vs. (intuitive) discoverability? -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, Apr 8, 2016 at 12:25 PM, Brett Cannon wrote: > I personally still like __ospath__ as well. Same here. The strings are essentially an OS-dependent serialization, rather than related to a particular file system. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, Apr 8, 2016 at 3:57 PM, Eric Snow wrote: > On Fri, Apr 8, 2016 at 12:25 PM, Brett Cannon wrote: >> I personally still like __ospath__ as well. > > Same here. The strings are essentially an OS-dependent serialization, > rather than related to a particular file system. Hmm. It's important to note the distinction between a standardized representation of a path and the OS-dependent representation. That is essentially the same distinction as provided by Go's "path" vs. "path/fliepath" packages. pathlib provides an abstraction of FS paths, but does it provide a standardized representation? From what I can tell you only ever get some OS-dependent representation. All this matters because it impacts the value returned from __ospath__(). Should it return the string representation of the path for the current OS or some standardized representation? I'd expect the former. However, if that is the expectation then something like pathlib.PureWindowsPath will give you the wrong thing if your current OS is linux. pathlib.PureWindowsPath.__ospath__() would have to fail or first internally convert to pathlib.PurePosixPath? On the other hand, it seems like the caller should be in charge of deciding the required meaning. That implies that returning a standardized representation or even something like a pathlib.PureGenericPath would be more appropriate. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
