Re: [Python-Dev] os.walk() is going to be *fast* with scandir
A small tip from my bzr days - cd into the directory before scanning it - especially if you'll end up statting more than a fraction of the files, or are recursing - otherwise the VFS does a traversal for each path you directly stat / recurse into. This can become a dominating factor in some workloads (I shaved several hundred milliseconds off of bzr stat on kernel trees doing this). -Rob On 10 August 2014 15:57, Nick Coghlan wrote: > On 10 August 2014 13:20, Antoine Pitrou wrote: >> Le 09/08/2014 12:43, Ben Hoyt a écrit : >> >>> Just thought I'd share some of my excitement about how fast the all-C >>> version [1] of os.scandir() is turning out to be. >>> >>> Below are the results of my scandir / walk benchmark run with three >>> different versions. I'm using an SSD, which seems to make it >>> especially faster than listdir / walk. Note that benchmark results can >>> vary a lot, depending on operating system, file system, hard drive >>> type, and the OS's caching state. >>> >>> Anyway, os.walk() can be FIFTY times as fast using os.scandir(). >> >> >> Very nice results, thank you :-) > > Indeed! > > This may actually motivate me to start working on a redesign of > walkdir at some point, with scandir and DirEntry objects as the basis. > My original approach was just too slow to be useful in practice (at > least when working with trees on the scale of a full Fedora or RHEL > build hosted on an NFS share). > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/robertc%40robertcollins.net -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] os.walk() is going to be *fast* with scandir
Indeed - my suggestion is applicable to people using the library -Rob On 10 Aug 2014 18:21, "Larry Hastings" wrote: > On 08/09/2014 10:40 PM, Robert Collins wrote: > > A small tip from my bzr days - cd into the directory before scanning it > > > I doubt that's permissible for a library function like os.scandir(). > > > */arry* > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/robertc%40robertcollins.net > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] web-sig mailing list moderating every post?
Ugh - this was in my mailbox shortly after the moderator action email from mailman: "No, this looks like the spam filter. Don't know what triggered it. Or why it went to you. But the list moderation is turned off (except for non-members posting to the list), and you yourself are not moderated, so... Bill" - nothing to see here, move right along, and sorry for the noise. -Rob On 21 September 2014 10:19, Robert Collins wrote: > I'm not sure of the right place to bring this up - I tried to on the > web-sig list itself, but the moderator rejected the post. > > What I tried to post there was > > """Looks like *every* post to web-sig gets manually moderated. That seems > like it will make discussion rather hard: can we get that changed (or > is there some historical need for it - if so, perhaps we should use > python-dev or some other list) ?""" > > -Rob > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] web-sig mailing list moderating every post?
I'm not sure of the right place to bring this up - I tried to on the web-sig list itself, but the moderator rejected the post. What I tried to post there was """Looks like *every* post to web-sig gets manually moderated. That seems like it will make discussion rather hard: can we get that changed (or is there some historical need for it - if so, perhaps we should use python-dev or some other list) ?""" -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encoding of PyFrameObject members
On 9 February 2015 at 09:11, Maciej Fijalkowski wrote: > Hi Francis > > Feel free to steal most of vmprof code, it should generally work > without requiring to patch cpython (python 3 patches appreciated :-). > As far as timer goes - it seems not to be going anywhere, I would > rather use a background thread or something What about setting a flag when the signal arrives and checking it at the next bytecode evaluation or something? -Rob ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [python-committers] Do we need to sign Windows files with GnuPG?
On 4 April 2015 at 11:14, Steve Dower wrote: > The thing is, that's exactly the same goodness as Authenticode gives, except > everyone gets that for free and meanwhile you're the only one who has > admitted to using GPG on Windows :) > > Basically, what I want to hear is that GPG sigs provide significantly better > protection than hashes (and I can provide better than MD5 for all files if > it's useful), taking into consideration that (I assume) I'd have to obtain a > signing key for GPG and unless there's a CA involved like there is for > Authenticode, there's no existing trust in that key. GPG sigs will provide protection against replay attacks [unless we're proposing to revoke signatures on old point releases with known security vulnerabilities - something that Window software vendors tend not to do because of the dramatic and immediate effect on the deployed base...] This is not relevant for things we're hosting on SSL, but is if anyone is mirroring our installers around. They dont' seem to be so perhaps its a bit 'meh'. OTOH I also think there is value in consistency: signing all our artifacts makes checking back on them later easier, should we need to. One question, if you will - I don't think this was asked so far - is authenticode verifiable from Linux, without Windows? And does it work for users of WINE ? -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unittest test discovery and namespace packages
On 17 April 2015 at 19:40, Alex Shkop wrote: > Hello! > > There's an issue considering test discovery in unittest module. Basically it > is about unittest module that doesn't find tests in namespace packages. For > more info see issue http://bugs.python.org/issue23882. > > I'm willing to make a patch for this bug. But I need help to formulate how > test discovery should work. > > Documentation states that all importable modules that match pattern will be > loaded. This means that test modules inside namespace packages should be > loaded too. But enabling this would change things drastically. For example > now, running > > python -m unittest > > inside cpython source root does nothing. If we will enable test discovery > inside namespace packages then this command will start running the whole > python test suite in Lib/test/. I don't think that 'scan the global namespace' makes a sensible default definition. The behaviour of discovery with namespace packages today requires some key to select the namespace - either a locally discovered directory, which happens to be a namespace package, or the name of the package to process. Since discovery is recursive, sub namespace packages should work, but I note there are no explicit tests to this effect. I'm sorry I didn't respond earlier on the tracker, didn't see the issue in my inbox for some reason. Lets discuss there. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Type hints -- a mediocre programmer's reaction
On 21 April 2015 at 08:07, Guido van Rossum wrote: > The situation is possibly even bleaker (or happier, depending on your > position :-) for inline type hints in 3rd party packages -- few package > authors will be satisfied with supporting only Python 3.5 and later. True, > you can support Python 3.2 and up by declaring the 3rd party typing package > as a dependency (unless Python 3.5+ is detected), but I don't expect this to > become a popular approach overnight. mypy has a codec for 2.x which strips type annotations - https://github.com/JukkaL/mypy/tree/master/mypy/codec - while you can't run mypy under 2.x, you can run it under 3.x to perform the analysis, and ones code still runs under 2.x. Another route - the one I've been experimenting with as I get familiar with mypy - is to just use type comments exclusively. Function type comments currently break, but that seems like a fairly shallow bug to me, rather than something that shouldn't work. The advantage of that route is that editors which make comments appear in subtle colours, makes the type hints be unobtrusive without specific syntax colouring support. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Type hints -- a mediocre programmer's reaction
On 21 April 2015 at 08:10, Eric Snow wrote: > > > > While it helps, this sort of best-practice is still unsettled (and apparently > not obvious). In the short term it would make more sense to recommend using > stub files for all the reason Harry enumerated. Once the best practices are > nailed down through experience with stub files, then we can make > recommendations regarding inline type hints. > > -eric Forgive my ignorance, but can stub files can't annotate variables within functions? E.g. AIUI if there is a stub file, it is used in the static analysis instead of the actual source. Likely I've got it modelled wrong in my head :) -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Type hints -- a mediocre programmer's reaction
On 21 April 2015 at 08:50, Harry Percival wrote: >> stub files are only used to type-check *users* of a module. If you want a >> module itself to be type-checked you have to use inline type hints > > is this a fundamental limitation, or just the current state of tooling? AIUI its the fundamental design. Stubs don't annotate python code, they *are* annotated code themselves. They aren't merged with the observed code at all. Could they be? Possibly. I don't know how much work that would be. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Type hints -- a mediocre programmer's reaction
On 21 April 2015 at 10:02, Ian Cordasco wrote: > > > > So I've generally stayed out of this but I feel there is some context that > people are missing in general. > > First, allow me to provide some context: I maintain a /lot/ of Python > code[1] and nearly all of it is designed to be compatible with Pythons 2.6, > 2.7, 3.2, 3.3, 3.4 (and eventually 3.5) and sometimes 2.5 (depending on the > project). If I want to improve a developer's experience with some of that > code using Type Hints I will essentially have no way to do that unless I > write the code with the annotations and ship versions with annotations > stripped and other versions with annotations? That's a lot more overhead. If > I could provide the annotations in stubs that means that only the people who > care about using them will have to use them. 2.5? I'm so sorry :). Being in approximately the same boat, I definitely want to be able to improve the developer experience. That said, with one key exception (str/bytes/unicode) Python code generally has the same type on all versions. Sure it might be imported from somewhere else, and you're restricted to the common subset of APIs, but the types in use don't vary per-python. So - as long as your *developers* can run mypy on 3.2+, they can benefit from type checking. mypy itself requires 3.2+ to run, but programs with type annotations should be able to run on all those python versions you mention. Now, what is the minimum barrier for entry? Nothing :) - at the moment every file can be analysed, and mypy ships with a bunch of stubs that describe the stdlib. So - you'll get some benefit immediately, where bad use of stdlib routines is happening. Constraining the types of functions gets you better errors (because you are expressing intent rather than what-might-happen which the inference has to work from otherwise. In particular, constraining the type of *inputs* can let bad callers be detected, rather than the engine assuming they are valid-until-a-contradiction-occurs. You can do that with stub files: put them in repo A, and add them to the MYPYPATH when working on repo B which calls the code in repo A. You can also add those stubs to repo B, but I wouldn't do that because then they will skew vs repo A. A further step up would be to annotate A in its code, rather than using stubs. That way, developers of repo A will be warned about bad uses of other repo A code. But - if you have stubs *or* annotations in-line in repo A, *everyone* changing repo A needs to care. Because if the code mismatches the stub, the folk that do care will now be unable to use repo A correctly - their type checker will complain about valid uses, and fail to complain about more invalid uses. I'm particularly interested in mypy for OpenStack because for some repos > 10% of reported bugs are type mismatch errors which mypy may well have avoided. > Is it more overhead to manage twice the number of files? Yes. Do I feel it > would be worth it to not overly complicate how these packages are released? > Yes. > Further, there are far more reasons to make stubs the baseline (in my > opinion) the biggest reason of all is that people want to provide stubs for > popular yet unmaintained libraries as third party packages. Should everyone > using PIL be using Pillow? Of course. Does that mean they'll migrate or be > allowed to migrate? No. Should they be able to benefit from this? Yes the > should. The only way for PIL users to be able to do that is if stub files > can be packaged separately for PIL and distributed by someone else. stubs can certainly be packaged and distributed separately. That doesn't make the case that we should use stubs for projects that are opting in. > I think while the authors are currently seeing stubs as a necessary *evil* > they're missing points where they're a better backwards compatible solution > for people who want to give users with capable IDEs the ability to use stub > (or hint) files. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Type hints -- a mediocre programmer's reaction
On 22 April 2015 at 04:28, Guido van Rossum wrote: > On Tue, Apr 21, 2015 at 12:49 AM, Antoine Pitrou > wrote: >> >> On Mon, 20 Apr 2015 20:43:38 -0400 >> "R. David Murray" wrote: >> > +1 to this from me too. I'm afraid that means I'm -1 on the PEP. >> > >> > I didn't write this in my earlier email because I wasn't sure about it, >> > but my gut reaction after reading Harry's email was "if type annotations >> > are used in the stdlib, I'll probably stop contributing". That doesn't >> > mean that's *true*, but that's the first time I've ever had that >> > thought, so it is probably worth sharing. >> >> I think it would be nice to know what the PEP means for daily stdlib >> development. If patches have to carry typing information each time they >> add/enhance an API that's an addition burden. If typing is done >> separately by interested people then it sounds like it wouldn't have >> much of an impact on everyone else's workflow. > > > This point will be moot until new code appears in the stdlib whose author > likes type hints. As I said, we won't be converting existing code to add > type hints (I'm actively against that for the stdlib, for reasons I've > explained already). > > *If* type hints prove useful, I expect that adding type hints **to code that > deserves them** is treated no different in the workflow than adding tests or > docs. I.e. something that is the right thing to do because it has obvious > benefits for users and/or future maintainers. If at some point running a > type checker over the stdlib as part of continuous integration become > routine, type hints can also replace certain silly tests. > > Until some point in a possible but distant future when we're all thinking > back fondly about the argument we're currently having, it will be the choice > of the author of new (and *only* new) stdlib modules whether and how to use > type hints. Such a hypothetical author would also be reviewing updates to > "their" module and point out lack of type hints just like you might point > out an incomplete docstring, an outdated comment, or a missing test. (The > type checker would be responsible for pointing out bugs. :-P ) What about major changes to existing modules? I have a backlog of intended feature uplifts from testtools into unittest - if the type hints thing works out I am likely to put them into testtools. Whats your view on type hints to such *new code* in existing modules? -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Type hints -- a mediocre programmer's reaction
On 22 April 2015 at 08:26, Guido van Rossum wrote: > In the end this should be up to you and the reviewers, but for such a > venerable module like unittest I'd be hesitant to be an early adopter. I'd > also expect that much of unittest is too dynamic in nature to benefit from > type hints. But maybe you should just try to use them for testtools and see > for yourself how beneficial or cumbersome they are in that particular case? Exactly yes. I've been experimenting recently with mypy to see. So far I've regressed backthrough 4 repos (unittest2, testtools, traceback2, linecache2) to get something small enough to work and experiment with. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 10 May 2015 at 11:44, Chris Angelico wrote: > On Sun, May 10, 2015 at 4:13 AM, M.-A. Lemburg wrote: >> By providing a way to intentionally switch off the new default, >> we do make people aware of the risks and that's good enough, >> while still maintaining the contract people rightly expect of >> patch level releases of Python. > > Just as long as it's the sysadmin, and NOT some random attacker over > the internet, who has the power to downgrade security. Environment > variables can be attacked in various ways. They can, and the bash fun was very good evidence of that. OTOH if someones environment is at risk, PATH and PYTHONPATH are already very effective attack vectors. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can Python guarantee the order of keyword-only parameters?
Plus 1 from me. I'm not 100% sure the signature / inspect backport does this, but as you say, it should be trivial to do, to whatever extent the python version we're hosted on does it. Rob On 28 Nov. 2017 07:14, "Larry Hastings" wrote: > > > First, a thirty-second refresher, so we're all using the same terminology: > > A *parameter* is a declared input variable to a function. > An *argument* is a value passed into a function. (*Arguments* are stored > in *parameters.*) > > So in the example "def foo(clonk): pass; foo(3)", clonk is a parameter, > and 3 is an argument. ++ > > > Keyword-only arguments were conceived of as being unordered. They're > stored in a dictionary--by convention called **kwargs--and dictionaries > didn't preserve order. But knowing the order of arguments is occasionally > very useful. PEP 468 proposed that Python preserve the order of > keyword-only arguments in kwargs. This became easy with the > order-preserving dictionaries added to Python 3.6. I don't recall the > order of events, but in the end PEP 468 was accepted, and as of 3.6 Python > guarantees order in **kwargs. > > But that's arguments. What about parameters? > > Although this isn't as directly impactful, the order of keyword-only > parameters *is* visible to the programmer. The best way to see a > function's parameters is with inspect.signature, although there's also the > deprecated inspect.getfullargspec; in CPython you can also directly examine > fn.__code__.co_varnames. Two of these methods present their data in a way > that preserves order for all parameters, including keyword-only > parameters--and the third one is deprecated. > > Python must (and does) guarantee the order of positional and > positional-or-keyword parameters, because it uses position to map arguments > to parameters when the function is called. But conceptually this isn't > necessary for keyword-only parameters because their position is > irrelevant. I only see one place in the language & library that addresses > the ordering of keyword-only parameters, by way of omission. The PEP for > inspect.signature (PEP 362) says that when comparing two signatures for > equality, their positional and positional-or-keyword parameters must be in > the same order. It makes a point of *not* requiring that the two > functions' keyword-only parameters be in the same order. > > For every currently supported version of Python 3, inspect.signature and > fn.__code__.co_varnames preserve the order of keyword-only parameters. > This isn't surprising; it's basically the same code path implementing those > as the two types of positional-relevant parameters, so the most > straightforward implementation would naturally preserve their order. It's > just not guaranteed. > > I'd like inspect.signature to guarantee that the order of keyword-only > parameters always matches the order they were declared in. Technically > this isn't a language feature, it's a library feature. But making this > guarantee would require that CPython internally cooperate, so it's kind of > a language feature too. > > Does this sound reasonable? Would it need a PEP? I'm hoping for "yes" > and "no", respectively. > > > Three final notes: > >- Yes, I do have a use case. I'm using inspect.signature metadata to >mechanically map arguments from an external domain (command-line arguments) >to a Python function. Relying on the declaration order of keyword-only >parameters would elegantly solve one small problem. >- I asked Armin Rigo about PyPy's support for Python 3. He said it >should already maintain the order of keyword-only parameters, and if I ever >catch it not maintaining them in order I should file a bug. I assert that >making this guarantee would be nearly zero effort for any Python >implementation--I bet they all already behave this way, all they need is a >test case and some documentation. >- One can extend this concept to functools.partial and >inspect.Signature.bind: should its transformations of keyword-only >parameters also maintain order in a consistent way? I suspect the answer >there is much the same--there's an obvious way it should behave, it almost >certainly already behaves that way, but it doesn't guarantee it. I don't >think I need this for my use case. > > > > */arry* > > ++ Yes, that means "Argument Clinic" should really have been called > "Parameter Clinic". But the "Parameter Clinic" sketch is nowhere near as > funny. > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > robertc%40robertcollins.net > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-
Re: [Python-Dev] PEP 574 -- Pickle protocol 5 with out-of-band data
One question.. On Thu., 29 Mar. 2018, 07:42 Antoine Pitrou, wrote: > ... > === > > Mutability > -- > > PEP 3118 buffers [#pep-3118]_ can be readonly or writable. Some objects, > such as Numpy arrays, need to be backed by a mutable buffer for full > operation. Pickle consumers that use the ``buffer_callback`` and > ``buffers`` > arguments will have to be careful to recreate mutable buffers. When doing > I/O, this implies using buffer-passing API variants such as ``readinto`` > (which are also often preferrable for performance). > > Data sharing > > > If you pickle and then unpickle an object in the same process, passing > out-of-band buffer views, then the unpickled object may be backed by the > same buffer as the original pickled object. > > For example, it might be reasonable to implement reduction of a Numpy array > as follows (crucial metadata such as shapes is omitted for simplicity):: > >class ndarray: > > def __reduce_ex__(self, protocol): > if protocol == 5: > return numpy.frombuffer, (PickleBuffer(self), self.dtype) > # Legacy code for earlier protocols omitted > > Then simply passing the PickleBuffer around from ``dumps`` to ``loads`` > will produce a new Numpy array sharing the same underlying memory as the > original Numpy object (and, incidentally, keeping it alive):: This seems incompatible with v4 semantics. There, a loads plus dumps combination is approximately a deep copy. This isn't. Sometimes. Sometimes it is. Other than that way, I like it. Rob > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How do people like the early 3.5 branch?
On 17 June 2015 at 11:26, Larry Hastings wrote: > > > A quick look through the checkin logs suggests that there's literally > nothing happening in 3.6 right now. All the checkins are merges. > > Is anyone expecting to do work in 3.6 soon? Or did the early branch just > create a bunch of make-work for everybody? > > Monitoring the progress of our experiment, When I next get tuits, it will be on 3.6; I like the branch early even though I haven't used it. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Not getting mail from the issue tracker
Firstly, a big sorry to all those unittest issues I haven't commented on. Turns out I simply don't get mail from the issue tracker. :(. Who should I speak to to try and debug this? In the interim, if you want me to look at an issue please ping me on IRC (lifeless) or mail me directly. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Freeze exception for http://bugs.python.org/issue23661 ?
So unittest.mock regressed during 3.5, and I found out when I released the mock backport. The regression is pretty shallow - I've applied the fix to 3.6, its a one-liner and comes with a patch. Whats the process for getting this into 3.5? Its likely to affect a lot of folk using mock (pretty much every OpenStack project got git with it when I released mock 1.1). -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Freeze exception for http://bugs.python.org/issue23661 ?
On 14 July 2015 at 14:25, R. David Murray wrote: > On Tue, 14 Jul 2015 14:01:25 +1200, Robert Collins > wrote: >> So unittest.mock regressed during 3.5, and I found out when I released >> the mock backport. >> >> The regression is pretty shallow - I've applied the fix to 3.6, its a >> one-liner and comes with a patch. >> >> Whats the process for getting this into 3.5? Its likely to affect a >> lot of folk using mock (pretty much every OpenStack project got git >> with it when I released mock 1.1). > > 3.5 hasn't been released yet. The patch ideally would have gone into > 3.5 first, then been merged to 3.6. As it is, you'll apply it to > 3.5, and then do a null merge to 3.6. It will get released in the > next 3.5 beta. What I'm unclear on is the approval process for doing ^. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How far to go with user-friendliness
On 15 July 2015 at 02:06, Paul Moore wrote: > On 14 July 2015 at 14:51, Florian Bruhin wrote: >> * Steven D'Aprano [2015-07-14 23:41:56 +1000]: ... >> With the patch, an AttributeError is raised if you call something >> starting with assert or assret instead. > > In retrospect, this seems like a mistake in the design. Much like > namedtuple, mocks should carefully separate "actual" methods from > mocked ones (in the case of namedtuple, from tuple element names). If > Guido would let us use the time machine, I'd argue that maybe the > special methods should be _assert_called_with (or something similar). Well. I'd go further and just separate the APIs. mock.assert_called_with(a_mock, *args, **kwargs) mock can know how to poke under the covers (e.g. using __Mock_assert_called_with) without leaking it into the users objects. > Given that it's way too late to take that path, I can see the value of > trying to detect common errors. Particularly as the result of failing > to do so is an apparently-passing test. We can add a new API and gradually deprecate the old one. With the presence of 'mock' as a rolling backport, this can be used by folk on Python 3.3 and 3.4 so they don't get locked into one release of Python or another. > In effect, this patch is "reserving" all attributes starting with > "assert" or "assret" as actual methods of the mock object, and not > mocked attributes. Yes, and thats ugly. OTOH it caught hundreds of useless tests in OpenStack when this got ported into mock 1.1.0. > Reserving "assert" seems fair. > Reserving "assret" seems odd, as people say why just this > mis-spelling? Is there any specific evidence that this typo happens > more often "in the wild" than any other? Given that the original issue > was raised by Michael Foord (the author of mock), I'd be inclined to > assume that he'd encountered evidence to that effect. > > So ultimately I'm +1 on reserving "assert" (given that a more radical > fix isn't possible) and +0 on adding "assret" (simply on the basis > that someone more knowledgeable than me says it makes sense). Since assret is solely a 'you may not use this' case, I think we can remove the check for that quite trivially, at any point we want to. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How far to go with user-friendliness
On 15 July 2015 at 07:39, Paul Moore wrote: > On 14 July 2015 at 20:27, Robert Collins wrote: >>> In effect, this patch is "reserving" all attributes starting with >>> "assert" or "assret" as actual methods of the mock object, and not >>> mocked attributes. >> >> Yes, and thats ugly. OTOH it caught hundreds of useless tests in >> OpenStack when this got ported into mock 1.1.0. > > ... which I guess counts as strong evidence that this *is* a common > typo, at least in certain contexts. For clarity: None of the caught failures were assret as far as I know. They were things like assert_called_onec_with, or assert_called. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How far to go with user-friendliness
On 15 July 2015 at 09:41, A.M. Kuchling wrote: > On Tue, Jul 14, 2015 at 09:53:33AM -0700, Ethan Furman wrote: >> Part of writing tests is making sure they fail (and for the right reason) -- >> proper testing of the tests would reveal such a typo. > > And there are other failure modes for writing tests that succeed but > are not testing what you think. For example, you might re-use the > same method name: > >def test_connection(self): ># Never executed >... > >... 200 lines and 10 other test methods later ... > >def test_connection(self): >... > > Or misuse assertRaises: > >with self.assertRaises(TypeError): >1 + "a" ># Second statement never reached >[] + 'b' > > I don't think unittest can protect its users from such things. It can't, but there is a sliding scale of API usability, and we should try to be up the good end of that :). http://sweng.the-davies.net/Home/rustys-api-design-manifesto -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How far to go with user-friendliness
On 15 July 2015 at 10:05, Ethan Furman wrote: > On 07/14/2015 02:53 PM, Robert Collins wrote: ... >>> I don't think unittest can protect its users from such things. >> >> >> It can't, but there is a sliding scale of API usability, and we should >> try to be up the good end of that :). > > > I hope you're not suggesting that supporting misspellings, and thereby > ruling out the proper use of an otherwise fine variable name, is at the good > end of that scale? I'm not supporting the misspelling thing - see my suggestion earlier in this thread to move the mock assertions to standalone functions, removing the bug in that area *entirely* and eventually removing the check for method names starting with assert from mock entirely. What I am doing is rejecting the argument that because we can't fix every mis-use users might make, we therefore should not fix the cases where we can fix it. For clarity, I think we should: - remove the assret check, it is I think spurious. - add a set of functions to the mock module that should be used in preference to Mock.assert* - mark the Mock.assert* functions as PendingDeprecation - in 3.6 move the PendingDeprecation to Deprecated - in 3.7 remove the Mock.assert* functions and the check for method names beginning with assert entirely. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Consenting adults considered beneficial [was: How far to go with user-friendliness]
On 15 July 2015 at 15:00, Stephen J. Turnbull wrote:
> Robert Collins writes:
>
> > What I am doing is rejecting the argument that because we can't fix
> > every mis-use users might make, we therefore should not fix the cases
> > where we can fix it.
>
> This involves a value judgment, every time a new fix is proposed, as
> to whether it's a mis-use that deserves fixing or a permitted-to-
> consenting-adults behavior. IMO, it's a bad idea to institutionalize
> that kind of bikeshedding, especially when such "fixes" involve
> overriding user choices that are permitted everywhere else.
I'm thoroughly confused by this.
> Arbitrary choices that *some* users want to be protected from ("stop
> me before I 'assret' again!") belong in linters, not in Python or the
> stdlib.
I agree with this.
> To be frank, I think you have the Pythonic approach exactly backwards
> here (though I am no authority on Pythonicity). ISTM that in general
> Python takes the attitude that if a particular "mis-use" seems to be
> common, then we should figure out what it is about Python that
> encourages that "mistake", or makes an otherwise arbitrary user choice
> into a "mistake", and fix Python -- not restrict users.
>
> Of course that's not always possible, but that's the first choice
> AIUI.
And these two paragraphs confuse me again.
Lets be really specific here.
Mock today has the following behaviour:
x = Mock()
x.foo()
x.bar()
...
all will just work and are mock methods that magically appear on demand.
And it includes some methods:
x.assert_called_with()
which are not mock methods, can't be mocked, and if you make a typo
you got *no* signal back to say that you'd messed up, until the patch
that added assert_ and assret_ prefix checking was added.
Which part of that API is Pythonic?
I rejected an argument that just because some APIs are are
intrinsically able to be misused, that we should not try to write
better APIs.
I then gave an plan, for this case, which appears to have been
enthusiastically recieved by a bunch of long-time Python devs.
In what way is that unPythonic behaviour or design?
Confusedly-yrs.
Rob
--
Robert Collins
Distinguished Technologist
HP Converged Cloud
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How far to go with user-friendliness
On 15 July 2015 at 12:59, Nick Coghlan wrote: > > There is zero urgency here, so nothing needs to change for 3.5. > Robert's plan is a fine one to propose for 3.6 (and the PyPI mock > backport). Right - the bad API goes back to the very beginning. I'm not planning on writing the new thing I sketched, though it should be straight forward if someone wishes to do so. I'll probably file a ticket in the tracker asking for it once this thread quiesces. -Rob ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How far to go with user-friendliness
On 15 July 2015 at 19:17, Antoine Pitrou wrote: > On Wed, 15 Jul 2015 10:22:14 +1200 > Robert Collins wrote: >> >> For clarity, I think we should: >> - remove the assret check, it is I think spurious. >> - add a set of functions to the mock module that should be used in >> preference to Mock.assert* >> - mark the Mock.assert* functions as PendingDeprecation >> - in 3.6 move the PendingDeprecation to Deprecated >> - in 3.7 remove the Mock.assert* functions and the check for method >> names beginning with assert entirely. > > I think removing them is a bit too strong. There's software out there > that would like to have cross-version-compatible test suites. Which they can do using 'mock'. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How far to go with user-friendliness
On 17 Jul 2015 08:34, "Michael Foord" wrote: > > > > On Wednesday, 15 July 2015, Robert Collins wrote: > > On 15 July 2015 at 12:59, Nick Coghlan wrote: > >> > >> There is zero urgency here, so nothing needs to change for 3.5. > >> Robert's plan is a fine one to propose for 3.6 (and the PyPI mock > >> backport). > > > > Right - the bad API goes back to the very beginning. I'm not planning > > > I disagree it's a bad api. It's part of why mock was so easy to use and part of why it was so successful. With the new check for non-existent assert methods it's no longer dangerous and so doesn't need fixing. Could you help me understand how the presence of assert... on the mock vs in the mock module affected ease of use? -Rob ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How far to go with user-friendliness
On 18 July 2015 at 15:19, Nick Coghlan wrote: > > This change *doesn't really matter* in the grand scheme things, but would > require a non-zero amount of time and effort to reverse, so unless you're > offering one of the unittest maintainers a contract gig to change it back, > let it go. s/unittest/mock :). But yes. Currently only Michael is listed under 'experts' in the devguide for unittest.mock. I've taken up the baton to keep the backport up to date, to keep the ecosystem healthy, but I've no specific plans to hack on mock per se. I think we'd probably benefit from more names there :) I know Kushal and Berker have been doing things in the stdlib. But there's a tonne of important work to do before worrying about tweaking a patch which was peer reviewed and went through the entirely normal development process to address a critical usability issue with mock. Which, judging from this thread, a bunch of folk don't actually understand in the first place. [I mean no disrespect here, but there have been multiple explanations trying to cover the distinction about what is actually going on, and I'm well over them]. So - folk interested in unittest.mock. If you want to help, going through the 200 open issues in https://github.com/testing-cabal/mock, starting with the oldest, assessing whether they are: - still relevant - present only in the backport (leave them where they are) - or in 3.6 as well (in which case open a new ticket at https://bugs.python.org/ linked to the github issue, and either close the github issue or label it upstream (or both)). THAT would be valuable, and improve users experience of unittest.mock [and mock] much more than making a_mock.assret_called_once *not error*. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How do we tell if we're helping or hindering the core development process? (was Re: How far to go with user-friendliness)
On 21 July 2015 at 00:34, Ben Finney wrote: > Paul Moore writes: > >> Again, I'm sorry to pick on one sentence out of context, but it cut >> straight to my biggest fear when doing a commit (on any project) - >> what if, after all the worrying and consideration I put into doing >> this commit, people disagree with me (or worse still, I made a >> mistake)? Will I be able to justify what I decided? > > That seems quite healthy to me. On a collaborative project with effects > far beyond oneself, yes, any change *should* be able to be justified > when challenged. Depending on what you mean by justification , this leaves no leeway for hunches, feelings, intuition, or grey area changes. It's also a terrible way to treat people that are contributing their own time and effort: assume good faith is a much better starting point. I think its reasonable to say that any change should be open to post hoc discussion. Thats how we learn, but justification and challenging is an adversarial framing, and one I'm flatly uninterested in. Life is too short to volunteer on adversarial activies. > That isn't a mandate to challenge every change, of course. It does mean > that every change should be considered in light of “Can I justify > this, if challenged?” > > So what you describe sounds like a healthy barrier: one which works to > keep out unjustifiable changes. I don't understand why thats healthy or useful. > What is needed is to have both that *and* the support of the community > so it's not a barrier to the *contributors*. The contributors should not > feel excluded merely because some of their changes might need to be. I don't think that contributors are a problem here. But, I'm going to dig into that more in reply to Nick. >> Hmm, maybe I'd better hold off and let someone else make the >> decision... > > What of the (obvious, to me) option to retain the authority to make the > decision, but take the doubt as a sign that one should consult with > others before making the decision? > > That is, there's no need to feel that one shouldn't make the decision. > But perhaps one shouldn't make it solely on one's own experience or > insight. Get others involved, even non-committers, and discuss it, and > understand the issue better. With that improved basis, then make the > decision. As a case study, this discussion where something like 90% of the kibbitzing demonstrated a clear lack of understanding of the very behaviour of Mock, *and* the change in question was discussed, *prior* to it being done, *and* users have welcomed itI must conclude that either: - this discussion was the exception to prove your general rule (but why would we derive that general rule from this discussion) - the general rule isn't general. > Am I naive to think that's desirable for PYthon core committers? What's the goal here: what actual problem are we trying to solve for? More contributors? A better Python more people can use? Keeping up with the contributions we've already received but not actioned? [...] Like: pick one thing. What we /really/ want to achieve, then lets look at what will let us get there. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How do we tell if we're helping or hindering the core development process? (was Re: How far to go with user-friendliness)
On 21 July 2015 at 19:40, Nick Coghlan wrote: > On 20 July 2015 at 22:34, Ben Finney wrote: >> Paul Moore writes: >> >>> Again, I'm sorry to pick on one sentence out of context, but it cut >>> straight to my biggest fear when doing a commit (on any project) - >>> what if, after all the worrying and consideration I put into doing >>> this commit, people disagree with me (or worse still, I made a >>> mistake)? Will I be able to justify what I decided? >> >> That seems quite healthy to me. On a collaborative project with effects >> far beyond oneself, yes, any change *should* be able to be justified >> when challenged. > > No, that's not how this works: if folks are thinking that being a > Python user, or even a CPython core developer, means that we're > entitled to micromanage core developers by demanding extensive > explanations for any arbitrary commit we choose, they're thoroughly > mistaken. Only Guido has that privilege, and one of the reasons he's > as respected as he is is his willingness to trust the experience and > expertise of others and only rarely exercise his absolute authority. I wouldn't even agree that Guido is entitled to micromanage us. He is certainly entitled to demand explanations etc if he feels appropriate - but if that turned into micromanaging, I think we'd be in a very bad place. > All of this is why the chart that I believe should be worrying people > is the topmost one on this page: > http://bugs.python.org/issue?@template=stats > > Both the number of open issues and the number of open issues with > patches are steadily trending upwards. That means the bottleneck in > the current process *isn't* getting patches written in the first > place, it's getting them up to the appropriate standards and applied. Perhaps. > Yet the answer to the problem isn't a simple "recruit more core > developers", as the existing core developers are *also* the bottleneck > in the review and mentoring process for *new* core developers. sidebar: So, anyone here familiar with theory of constraints - e.g. 'The Goal', 'Critical Chain' etc? Might be interesting to do some brainstorming in that shape if folk are interest. Having open idle patches is I think generally acknowledged as a poor thing. I think of them as inventory in a manufacturing plant: they take up space, they take effort to create, they offer no value [until they're actually shipped to users], and they have direct negatives (tracker is harder to work with due to the signal-to-noise ratio, perception of the project suffers, contributors make the rational discussion not to contribute further...). Lets assume that our actual goal is to ship new Python versions, offering more and better things to users. AFAIK we're not lacking any infrastructure resources - we have enough buildbots, we have freely available compilers for all platforms. > In my view, those stats are a useful tool we can use to ask ourselves > "Am I actually helping with this contribution, or at the very least, > not causing harm?": I like this approach :). But - can we go further, can we actively protect core committer time such that they waste less of it? Adding core committers won't help if the problem isn't the number of committers, but rather the amount of the time that they can devote to Python that actually gets spent on committer-activities. > * helping core developers that have time to work on "CPython in > general" rather than specific projects of interest to them to focus > their attention more effectively may help make those stats better (and > it would be even better if we could credit such triaging efforts > appropriately) Iterating on a patch someone else put up can help. Making sure its passing tests, trying it with ecosystem projects and giving feedback. > * exploring ways to extract contribution metrics from Roundup so we > can have a more reliable detection mechanism for active and consistent > contributors than the "Mark 1 core developer memory, aka the > notoriously unreliable human brain" may help make those stats better OTOH, make sure that what we measure provokes things that help our goal :). Double edged sword this. > Make no mistake, sustainable open source development is a *genuinely > hard problem*. We're in a much better position than most projects > (being one of the top 5 programming languages in the world has its > benefits), but we're still effectively running a giant multinational > collaborative software development project with close to zero formal > management support. While their are good examples we can (and are) > learning from, improving the support structures for an already wildly > successful open source project without alienating existing > contributors isn't a task that comes with an instruction manual :) +1 -Rob ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-d
Re: [Python-Dev] How far to go with user-friendliness
On 22 July 2015 at 03:47, Tim Golden wrote: > On 20/07/2015 19:48, Christie Wilson wrote: >> I am terrified of replying to this thread since so many folks on it seem >> unhappy that it is continuing, but I want to +1 what Erik said. > > Don't be terrified :) > > But do understand that, in general, and especially for this > already-noisy thread, the right place for arguments supporting a change > or a reversion is usually on the issue tracker: > > https://bugs.python.org/ > > I don't know whether Robert's opened an issue to propose his solution, > but if not, you could open one and add him as nosy. I did: http://bugs.python.org/issue24651 -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reminder: Python 3.5 beta 4 is tagged in one week
Cool. http://bugs.python.org/issue21750 is in a bad state right now.
I landed a patch to fix it, which when exposed to users had some
defects. I'm working on a better patch now, but need to either roll
the prior patch completely back, or get the new one landed before the
next beta. I hope to have that up for review later today {fingers
crossed} - will that be soon enough, or should I look up how to easily
revert stuff out with hg?
-Rob
On 18 July 2015 at 22:24, Larry Hastings wrote:
>
>
> Approximately a week from when I post this, I'll be tagging Python 3.5 beta
> 4, which is the last beta before we go to release candidates. Please wind
> up all your bug fixes soon, I'd really like checkins to 3.5 to stop soon.
>
> And a minor reminder: when we hit Release Candidate 1, I'll be switching the
> canonical repo for 3.5 to a public Bitbucket repo. Any bug fixes that go in
> between RC 1 and final will only be merged using Bitbucket "pull requests".
>
> The new workflow experiment continues,
>
>
> /arry
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/robertc%40robertcollins.net
>
--
Robert Collins
Distinguished Technologist
HP Converged Cloud
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reminder: Python 3.5 beta 4 is tagged in one week
On 22 July 2015 at 05:08, Larry Hastings wrote:
>
>
> On 07/21/2015 06:35 PM, Robert Collins wrote:
>
> Cool. http://bugs.python.org/issue21750 is in a bad state right now.
>
> I landed a patch to fix it, which when exposed to users had some
> defects. I'm working on a better patch now, but need to either roll
> the prior patch completely back, or get the new one landed before the
> next beta. I hope to have that up for review later today {fingers
> crossed} - will that be soon enough, or should I look up how to easily
> revert stuff out with hg?
>
>
> If you want to undo it, "hg backout" is the command you want. In general
> it's best to not check in broken stuff.
Thanks. And yes, naturally - we didn't realise it was broken. Passing
tests != fit for purpose.
-Rob
--
Robert Collins
Distinguished Technologist
HP Converged Cloud
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reminder: Python 3.5 beta 4 is tagged in one week
On 22 July 2015 at 08:07, Robert Collins wrote:
> On 22 July 2015 at 05:08, Larry Hastings wrote:
>>
>>
>> On 07/21/2015 06:35 PM, Robert Collins wrote:
>>
>> Cool. http://bugs.python.org/issue21750 is in a bad state right now.
>>
>> I landed a patch to fix it, which when exposed to users had some
>> defects. I'm working on a better patch now, but need to either roll
>> the prior patch completely back, or get the new one landed before the
>> next beta. I hope to have that up for review later today {fingers
>> crossed} - will that be soon enough, or should I look up how to easily
>> revert stuff out with hg?
>>
>>
>> If you want to undo it, "hg backout" is the command you want. In general
>> it's best to not check in broken stuff.
>
> Thanks. And yes, naturally - we didn't realise it was broken. Passing
> tests != fit for purpose.
21750 is now sorted out in the cpython repo.
I have a separate question for you - issue2091 has a good patch on it,
but would you like it added to 3.5?
It makes a broken combination of file modes - rU+ - a clean error, and
tweaks the existing exception text for U + writing modes.
-Rob
--
Robert Collins
Distinguished Technologist
HP Converged Cloud
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Burning down the backlog.
On 21 July 2015 at 19:40, Nick Coghlan wrote: > All of this is why the chart that I believe should be worrying people > is the topmost one on this page: > http://bugs.python.org/issue?@template=stats > > Both the number of open issues and the number of open issues with > patches are steadily trending upwards. That means the bottleneck in > the current process *isn't* getting patches written in the first > place, it's getting them up to the appropriate standards and applied. > Yet the answer to the problem isn't a simple "recruit more core > developers", as the existing core developers are *also* the bottleneck > in the review and mentoring process for *new* core developers. Those charts doesn't show patches in 'commit-review' - http://bugs.python.org/issue?%40columns=title&%40columns=id&stage=5&%40columns=activity&%40sort=activity&status=1&%40columns=status&%40pagesize=50&%40startwith=0&%40sortdir=on&%40action=search There are only 45 of those patches. AIUI - and I'm very new to core here - anyone in triagers can get patches up to commit-review status. I think we should set a goal to keep inventory low here - e.g. review and either bounce back to patch review, or commit, in less than a month. Now - a month isn't super low, but we have lots of stuff greater than a month. For my part, I'm going to pick up more or less one thing a day and review it, but I think it would be great if other committers were to also to do this: if we had 5 of us doing 1 a day, I think we'd burn down this 45 patch backlog rapidly without significant individual cost. At which point, we can fairly say to folk doing triage that we're ready for patches :) -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How do we tell if we're helping or hindering the core development process?
On 29 July 2015 at 02:17, Ben Finney wrote: > Paul Moore writes: > >> On 28 July 2015 at 13:35, Ben Finney wrote: >> > People can, do, and probably must make many decisions through >> > non-rational processes. I don't propose to change that. >> >> Good. >> >> > Choices can be made that, when challenged, lack compelling rational >> > justification. I do propose that such a challenge should be taken as a >> > healthy desire to improve Python, not a personal attack. >> >> While that is fine, you appear unwilling to accept the possibility >> that people may not have the time/energy to develop a detailed >> rational justification for a change that they have made, and demanding >> that they do so when they are offering the time they do give on a >> volunteer basis, is what I claim is unacceptable. > > I've said many times now that's not what I'm advocating. > > I've made a clear distinction between the need to *be able to* justify a > change, versus arbitrary demands to do so by arbitrary members. > > The latter is what you're arguing against, and of course I agree. I've > never advocated that. I'm arguing against the former. Being able to survive a crowd sourced grilling on any arbitrary change would be quite the chilling effect, and its a level of backpressure that the committers who engaged in this discussion have rejected. Some have rejected contributing *at all* as a result of the discussion. Others, like me, are telling you that you're wrong, that we don't accept that we can be called up for any odd commit and asked to justify it to anyone. There is a social contract around our commits - and it does permit enquiry and discussion, but not with the degree of heat or antagonism that was present in this thread. AND Not by uninformed folk: If you're going to second guess the onus is on you to educate yourself about the issue first. This particular one, for instance, requires going back through the history of mock right to its founding in 2007, and walking forward through the merge into the stdlib in Python 3,3 (because its popular) and finally the realisation that large chunks of peoples code were silently not testing what was desired and the fixing of that. Discussing the thing we discussed *in that context* is a very different discussion to what we had, where every second message was someone misunderstanding what the issue is and chiming in to say that this is surprising and unPythonic and against the Zen and oh my. >> The issue is not one of your motives in asking for explanations - it's >> the implication that you are entitled to require others to *provide* >> those explanations, to whatever level of detail *you* require. > > Hopefully this repetition is enough: I do not claim any such entitlement. If you don't claim such entitlement, who does? Whose entitlement are you arguing for? If its Guido's, I think we can stop arguing - sure, he is entitled to ask for a lot, but I don't want to argue about what entitlements someone else has: they can argue on their own. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] updating ensurepip to include wheel
So, pip 7.0 depends on the wheel module for its automatic wheel building, and installing pip from get-pip.py, or the bundled copy in virtualenvs will automatically install wheel. But ensurepip doesn't bundle wheel, so we're actually installing a slightly crippled pip 7.1, which will lead to folk having a poorer experience. Is this a simple bug, or do we need to update the PEP? -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] updating ensurepip to include wheel
On 6 August 2015 at 15:04, Nick Coghlan wrote: > On 6 August 2015 at 09:29, Victor Stinner wrote: >> Le 5 août 2015 17:12, "Nick Coghlan" a écrit : >>> A hard dependency on wheel wouldn't fit into the same category - when >>> folks are using a build pipeline to minimise the installation >>> footprint on production systems, the wheel package itself has no >>> business being installed anywhere other than developer systems and >>> build servers. >> >> I'm quite sure that virtualenv is used to deploy python on production. >> >> Pip 7 automatically creates wheel packages when no build wheel package is >> available on PyPI. Examples numpy and any pure python package only providing >> a tarball. >> >> For me it makes sense to embed wheel in ensurepip and to install wheel on >> production systems (to install pacakes, not to build them). > > pip can install from wheels just fine without the wheel package being > present - that's how ensurepip already works. pip can also do this without setuptools being installed; yet we bundle setuptools with pip in ensurepip. I am thus confused :). When I consider the harm to a production pipeline that using setuptools can cause (in that it triggers easy_install, and easy_install has AFAIK none of the security improvements pip has added over the last couple years), I find the acceptance of setuptools, but non-acceptance of wheel flummoxing. > The wheel package itself is only needed in order to support the > setuptools "bdist_wheel" command, which then allows pip to implicitly > cache wheel files when installing from an sdist. > > Installing from sdist in production is a *fundamentally bad idea*, > because it means you have to have a build toolchain on your production > servers. One of the benefits of the wheel format and projects like > devpi is that it makes it easier to discourage people from doing that. > Even without getting into Linux containers and tools like pyp2rpm, > it's also possible to create an entire virtualenv on a build server, > bundle that up as an RPM or DEB file, and use the system package > manager to do the production deployment. Yes: but the logic chain from 'its a bad idea' to 'we don't include wheel but we do include setuptools' is the bit I'm having a hard time with. > However, production Linux servers aren't the only case we need to care > about, and there's a strong user experience argument to be made for > providing wheel by default upstream, and telling downstream > redistributors that care about the distinction to do the necessary > disentangling to make it easy to have "build dependency free" > production images. > > We've learned from experience that things go far more smoothly if we > thrash out those kinds of platform dependent behavioural differences > *before* we inflict them on end users, rather than having downstream > redistributors tackle foreseeable problems independently of both each > other and upstream :) > > Hence my request for a PEP - I can see why adding wheel to the > ensurepip bundle would be a good idea for upstream, but I can also see > why it's a near certainty downstream Linux distros (including Fedora) > would take it out again in at least some situations to better meet the Does Fedora also take out setuptools? If not, why not? > needs of *our* user base. (Since RPM has weak dependency support now, > we'd likely make python-wheel a "Recommends:" dependency, rather than > a "Requires:" dependency - still installed by default, but easy to > omit if not wanted or needed) So, a new PEP? -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] updating ensurepip to include wheel
On 7 August 2015 at 03:28, Donald Stufft wrote: > >> On Aug 6, 2015, at 5:04 AM, Robert Collins wrote: >> >> Yes: but the logic chain from 'its a bad idea' to 'we don't include >> wheel but we do include setuptools' is the bit I'm having a hard time >> with. > > > In my opinion, it’s the severity of how crippled their experience is without > that particular thing installed. > > In the case of wheel not being installed they lose the ability to have an > implicit wheel cache and to run ``pip wheel``. This makes pip less good but, > unless they are running ``pip wheel`` everything is still fully functioning. > > In the case of setuptools they lose the ability to ``pip install`` when there > isn’t a wheel available and the ability to run ``pip wheel``. This is making > pip completely unusable for a lot of people, and if we did not pre-install > setup tools the number one thing people would do is to ``pip install > setuptools``, most likely while bitching under their breath about the command > that just failed because they tried to install from sdist. > > So it’s really just “how bad are we going to break people’s expectations”. So - I was in a talk at PyCon AU about conda[*], and the author believed they were using the latest pip with all the latest caching features, but their experience (16 minute installs) wasn't that. I dug into that with them after the talk, and it was due to Conda not installing wheel by default. Certainly the framing of ensurepip as 'this installs pip' is going to be confusing and misleading if it doesn't install pip the way get-pip.py (or virtualenv) install pip, leading to confusion such as that. Given the inconsequential impact of installing wheel, I see only harm in holding it back, and only benefits in adding it. All the harm from having source builds comes in with setuptools ;). -Rob *) https://www.youtube.com/watch?v=Fqknoni5aX0 -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP needed for http://bugs.python.org/issue9232 ?
So, there's a patch on issue 9232 - allow trailing commas in function definitions - but there's been enough debate that I suspect we need a PEP. Would love it if someone could correct me, but I'd like to be able to either categorically say 'no' and close the ticket, or 'yes and this is what needs to happen next'. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Burning down the backlog.
On 26 July 2015 at 07:28, Robert Collins wrote: > On 21 July 2015 at 19:40, Nick Coghlan wrote: > >> All of this is why the chart that I believe should be worrying people >> is the topmost one on this page: >> http://bugs.python.org/issue?@template=stats >> >> Both the number of open issues and the number of open issues with >> patches are steadily trending upwards. That means the bottleneck in >> the current process *isn't* getting patches written in the first >> place, it's getting them up to the appropriate standards and applied. >> Yet the answer to the problem isn't a simple "recruit more core >> developers", as the existing core developers are *also* the bottleneck >> in the review and mentoring process for *new* core developers. > > Those charts doesn't show patches in 'commit-review' - > http://bugs.python.org/issue?%40columns=title&%40columns=id&stage=5&%40columns=activity&%40sort=activity&status=1&%40columns=status&%40pagesize=50&%40startwith=0&%40sortdir=on&%40action=search > > There are only 45 of those patches. > > AIUI - and I'm very new to core here - anyone in triagers can get > patches up to commit-review status. > > I think we should set a goal to keep inventory low here - e.g. review > and either bounce back to patch review, or commit, in less than a > month. Now - a month isn't super low, but we have lots of stuff > greater than a month. > > For my part, I'm going to pick up more or less one thing a day and > review it, but I think it would be great if other committers were to > also to do this: if we had 5 of us doing 1 a day, I think we'd burn > down this 45 patch backlog rapidly without significant individual > cost. At which point, we can fairly say to folk doing triage that > we're ready for patches :) We're down to 9 such patches, and reading through them today there are none that I felt comfortable moving forward: either their state is unclear, or they are waiting for action from a *specific* core. However - 9 isn't a bad number for 'patches that the triagers think are ready to commit' inventory. So yay!. Also - triagers, thank you for feeding patches through the process. Please keep it up :) -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)
On 26 November 2015 at 08:57, Barry Warsaw wrote: > There's a lot to process in this thread, but as I see it, the issue breaks > down to these questions: > > * How should PEP 493 be implemented? > > * What should the default be? > > * How should PEP 493 be worded to express the right tone to redistributors? > > Let me take on the implementation details here. > > On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote: > >>I would still find having built-in support for the recommendations >>in the Python stdlib a better approach > > As would I. For what its worth: a PEP telling distributors to patch the standard library is really distasteful to me. We've spent a long time trying to build close relations such that when something doesn't work distributors can share their needs with us and we can make Python out of the box be a good fit. This seems to fly in the exact opposite direction: we're explicitly making it so that Python builds on these vendor's platforms will not be the same as you get by checking out the Python source code. Ugh. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pypi simple index
On 18 December 2015 at 06:13, Carlos Barera wrote: > Hi, > > I'm using install_requires in setup.py to specify a specific package my > project is dependant on. > When running python setup.py install, apparently the simple index is used > as an older package is taken from pypi. While > What's happening here is that easy-install is triggering - which does not support wheels. Use 'pip install .' instead. > in https://pypi.python.org/pypi, there's a newer package. > When installing directly using pip, the latest package is installed > successfully. > I noticed that the new package is only available as a wheel and older > versions of setup tools won't install wheels for install_requires. > However, upgrading setuptools didn't help. > > Several questions: > 1. What's the difference between the pypi simple index and the general > pypi index? > The '/simple' API is for machine consumption, /pypi is for humans, other than that there should be not be any difference. > 2. Why is setup.py defaulting to the simple index? > Because it is the only index :). > 3. How can I make the setup.py triggered install use the main pypi index > instead of simple > You can't - the issue is not the index being consulted, but your use of 'python setup.py install' which does not support wheels. Cheers, Rob ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Update PEP 7 to require curly braces in C
+1 from me on requiring them. On 18 January 2016 at 11:19, Brett Cannon wrote: > > > On Sun, 17 Jan 2016, 13:59 Ethan Furman wrote: >> >> On 01/17/2016 11:10 AM, Brett Cannon wrote: >> >> > https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the >> > curly braces is purely an aesthetic thing while leaving them out can >> > lead to actual bugs. >> >> Not sure what that sentence actually says, but +1 on making them >> mandatory. > > > > Yeah, bad phrasing on my part. What I meant to say is leaving them off is an > aesthetic thing while requiring them is a bug prevention thing. When it > comes to writing C code I always vote for practicality over aesthetics. > >> >> -- >> ~Ethan~ >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brett%40python.org > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/robertc%40robertcollins.net > -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] thoughts on backporting __wrapped__ to 2.7?
I'm working on teaching funcsigs - the backport of inspect.signature - better handling for wrapped functions, and the key enabler to do that is capturing the wrapped function in __wrapped__. I'm wondering what folks thoughts are on backporting that to 2.7 - seems cleaner than monkeypatching functools.wraps, which would tend to be subject to import ordering races and general ick. I'll likely prep such a monkeypatch for folk that are stuck on older versions of 2.7 anyhow... so its not a huge win... -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thoughts on backporting __wrapped__ to 2.7?
Sadly that has the ordering bug of assigning __wrapped__ first and appears a little unmaintained based on the bug tracker :( On 5 Apr 2016 8:10 PM, "Victor Stinner" wrote: > See https://pypi.python.org/pypi/functools32 for the functools backport > for Python 2.7. > > Victor > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)
On 11 April 2016 at 13:49, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 04/10/2016 06:31 PM, Jon Ribbens wrote: >> Unless someone knows a way to get to an object's __dict__ or its type >> without using vars() or type() or underscore attributes... > > Hmm, 'classmethod'-wrapped functions get passed the type. yeah, but to access that you need to assign the descriptor to the type - circular loop. If you can arrange that assignment its easy: thetype = [] class gettype: def __get__(self, obj, type=None): thetype.append((obj, type)) return None classIwant.query = gettype() classIwant().query thetype[0][1]... but you've already gotten to classIwant there. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thoughts on backporting __wrapped__ to 2.7?
On 6 April 2016 at 15:03, Stephen J. Turnbull wrote: > Robert Collins writes: > > > Sadly that has the ordering bug of assigning __wrapped__ first and appears > > a little unmaintained based on the bug tracker :( > > You can fix two problems with one patch, then! > Not really - taking over a project is somewhat long winded; it would be centralising yet another backport which may-or-may-not-be-a-good-thing, and I'm not exactly overflowing with spare tuits. If someone wants to do it - great, more power to them, but the last thing we need is to move it from one unmaintained spot to another unmaintained spot. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] git history conundrum
Thank you! If I understand correctly this is just the hg style branch backport consequence, multiple copies of a change. Should be safe to skip those. Rob On Sun, 28 Apr 2019, 07:11 Chris Withers, wrote: > Hi All, > > I'm in the process of bringing the mock backport up to date, but this > has got me stumped: > > $ git log --oneline --no-merges > 5943ea76d529f9ea18c73a61e10c6f53bdcc864f.. -- Lib/unittest/mock.py > Lib/unittest/test/testmock/ | tail > 362f058a89 Issue #28735: Fixed the comparison of mock.MagickMock with > mock.ANY. > d9c956fb23 Issue #20804: The unittest.mock.sentinel attributes now > preserve their identity when they are copied or pickled. > 84b6fb0eea Fix unittest.mock._Call: don't ignore name > 161a4dd495 Issue #28919: Simplify _copy_func_details() in unittest.mock > ac5084b6c7 Fixes issue28380: unittest.mock Mock autospec functions now > properly support assert_called, assert_not_called, and assert_called_once. > 0be894b2f6 Issue #27895: Spelling fixes (Contributed by Ville Skyttä). > 15f44ab043 Issue #27895: Spelling fixes (Contributed by Ville Skyttä). > d4583d7fea Issue #26750: use inspect.isdatadescriptor instead of our own > _is_data_descriptor(). > 9854789efe Issue #26750: unittest.mock.create_autospec() now works > properly for subclasses of property() and other data descriptors. > 204bf0b9ae English spelling and grammar fixes > > Right, so I've merged up to 15f44ab043, what comes next? > > $ git log --oneline --no-merges 15f44ab043.. -- Lib/unittest/mock.py > Lib/unittest/test/testmock/ | tail -n 3 > 161a4dd495 Issue #28919: Simplify _copy_func_details() in unittest.mock > ac5084b6c7 Fixes issue28380: unittest.mock Mock autospec functions now > properly support assert_called, assert_not_called, and assert_called_once. > 0be894b2f6 Issue #27895: Spelling fixes (Contributed by Ville Skyttä). > > Okay, no idea why 0be894b2f6 is there, appears to be a totally identical > commit to 15f44ab043, so let's skip it: > > $ git log --oneline --no-merges 0be894b2f6.. -- Lib/unittest/mock.py > Lib/unittest/test/testmock/ | tail -n 3 > 161a4dd495 Issue #28919: Simplify _copy_func_details() in unittest.mock > ac5084b6c7 Fixes issue28380: unittest.mock Mock autospec functions now > properly support assert_called, assert_not_called, and assert_called_once. > 15f44ab043 Issue #27895: Spelling fixes (Contributed by Ville Skyttä). > > Wat?! Why is 15f44ab043 showing up again?! > > What's the git subtlety I'm missing here? > > Chris > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/robertc%40robertcollins.net > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] git history conundrum
Share your own username with Michael or I and we'll add you there. Rob On Mon, 29 Apr 2019, 09:55 Chris Withers, wrote: > On 28/04/2019 22:21, Robert Collins wrote: > > Thank you! > > Thank me when we get there ;-) Currently in Dec 2018 with a wonderful > Py2 failure: > > == > ERROR: test_autospec_getattr_partial_function > (mock.tests.testhelpers.SpecSignatureTest) > -- > Traceback (most recent call last): >File "mock/tests/testhelpers.py", line 973, in > test_autospec_getattr_partial_function > autospec = create_autospec(proxy) >File "mock/mock.py", line 2392, in create_autospec > for entry in dir(spec): > TypeError: __dir__() must return a list, not str > > Once we're done, I'll need a username/password that can write to > https://pypi.org/project/mock/ ... > > > If I understand correctly this is just the hg style branch backport > > consequence, multiple copies of a change. Should be safe to skip those. > > Yep, current script I've been using is here, high level highlighted: > > https://github.com/cjw296/mock/blob/backporting/backport.py#L102-L125 > > cheers, > > Chris > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files
They were never needed 😁 Removal is fine with me. On Wed, 1 May 2019, 09:27 Chris Withers, wrote: > Hi All, > > I have a crazy idea of getting unittest.mock up to 100% code coverage. > > I noticed at the bottom of all of the test files in testmock/, there's a: > > if __name__ == '__main__': > unittest.main() > > ...block. > > How would people feel about these going away? I don't *think* they're > needed now that we have unittest discover, but thought I'd ask. > > Chris > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/robertc%40robertcollins.net > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library
This vector exists today for all new stdlib modules: once added, any existing dependency could include that name to cater it to be imported on prior python versions. Rob On Wed, 22 May 2019, 17:03 Stephen J. Turnbull, < turnbull.stephen...@u.tsukuba.ac.jp> wrote: > Christian Heimes writes: > > > It's all open source. It's up to the Python community to adopt > > packages and provide them on PyPI. > > > > Python core will not maintain and distribute the packages. I'll > > merely provide a repository with packages to help kick-starting the > > process. > > This looks to me like an opening to a special class of supply chain > attacks. I realize that PyPI is not yet particularly robust to such > attacks, and we have seen "similar name" attacks (malware uploaded > under a name similar to a popular package). ISTM that this approach > to implementing the PEP will enable "identical name" attacks. (By > download count, stdlib packages are as popular as Python. :-) > > It now appears that there's been substantial pushback against removing > packages that could be characterized as "obsolete and superseded but > still in use", so this may not be a sufficient great risk to be worth > addressing. I guess this post is already a warning to those who are > taking care of the "similar name" malware that this class of attacks > will be opened up. > > One thing we *could* do that would require moderate effort would be to > put them up on PyPI ourselves, and require that would-be maintainers > be given a (light) vetting before handing over the keys. (Maybe just > require that they be subscribers to the Dead Parrot SIG? :-) > > Steve > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/robertc%40robertcollins.net > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On 28 February 2013 05:51, Michael Foord wrote: > Hello all, > > PyCon, and the Python Language Summit, is nearly upon us. We have a good > number of people confirmed to attend. If you are intending to come to the > language summit but haven't let me know please do so. > > The agenda of topics for discussion so far includes the following: > > * A report on pypy status - Maciej and Armin > * Jython and IronPython status reports - Dino / Frank > * Packaging (Doug Hellmann and Monty Taylor at least) > * Cleaning up interpreter initialisation (both in hopes of finding areas > to rationalise and hence speed things up, as well as making things > more embedding friendly). Nick Coghlan > * Adding new async capabilities to the standard library (Guido) > * cffi and the standard library - Maciej > * flufl.enum and the standard library - Barry Warsaw > * The argument clinic - Larry Hastings > > If you have other items you'd like to discuss please let me know and I can > add them to the agenda. I'd like to talk about overhauling - not tweaking, overhauling - the standard library testing facilities. -Rob -- Robert Collins Distinguished Technologist HP Cloud Services ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On 4 March 2013 18:54, Guido van Rossum wrote: > On Sun, Mar 3, 2013 at 9:24 PM, Robert Collins > wrote: >> I'd like to talk about overhauling - not tweaking, overhauling - the >> standard library testing facilities. > > That seems like too big a topic and too vague a description to discuss > usefully. Perhaps you have a specific proposal? Or at least just a use > case that's poorly covered? I have both - I have a draft implementation for a new test result API (and forwards and backwards compat code etc), and use cases that drive it. I started a thread here - http://lists.idyll.org/pipermail/testing-in-python/2013-February/005434.html , with blog posts https://rbtcollins.wordpress.com/2013/02/14/time-to-revise-the-subunit-protocol/ https://rbtcollins.wordpress.com/2013/02/15/more-subunit-needs/ https://rbtcollins.wordpress.com/2013/02/19/first-experience-implementing-streamresult/ https://rbtcollins.wordpress.com/2013/02/23/simpler-is-better/ They are focused on subunit, but much of subunit's friction has been due to issues encountered from the stdlibrary TestResult API - in particular three things: - the single-active-test model that the current API (or at least implementation) has. - the expectation that all test outcomes will originate from the same interpreter (or something with a live traceback object) - the inability to supply details about errors other than the exception All of which start to bite rather deep when working on massively parallel test environments. It is of course possible for subunit and related tools to run their own implementation, but it seems ideal to me to have a common API which regular unittest, nose, py.test and others can all agree on and use : better reuse for pretty printers, GUI displays and the like depend on some common API. > TBH, your choice of words is ambiguous -- are you interested in > overhauling the facilities for testing *of* the standard library (i.e. > the 'test' package), or the testing facilities *provided by* the > standard library (i.e. the unittest module)? Sorry! Testing facilities provided by the standard library. They should naturally facilitate testing of the standard library too. -Rob > -- > --Guido van Rossum (python.org/~guido) -- Robert Collins Distinguished Technologist HP Cloud Services ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On 4 March 2013 19:40, Nick Coghlan wrote: > Your feedback on http://bugs.python.org/issue16997 would be greatly > appreciated. Done directly to Antoine on IRC the other day in a conversation with him and Michael about the compatability impact of subtests. Happy to do a full code review if that would be useful. -Rob -- Robert Collins Distinguished Technologist HP Cloud Services ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On 5 March 2013 05:34, Brett Cannon wrote: > > > > On Mon, Mar 4, 2013 at 11:29 AM, Barry Warsaw wrote: >> >> On Mar 04, 2013, at 07:26 PM, Robert Collins wrote: >> >> >It is of course possible for subunit and related tools to run their >> >own implementation, but it seems ideal to me to have a common API >> >which regular unittest, nose, py.test and others can all agree on and >> >use : better reuse for pretty printers, GUI displays and the like >> >depend on some common API. >> >> And One True Way of invoking and/or discovering how to invoke, a package's >> test suite. > > > How does unittest's test discovery not solve that? Three reasons a) There are some bugs (all filed I think) - I intend to hack on these in the near future - that prevent discovery working at all for some use cases. b) discovery requires magic parameters that are project specific (e.g. is it 'discover .' or 'discover . lib' to run it). This is arguably a setup.py/packaging entrypoint issue. c) Test suites written for e.g. Twisted, or nose, or other non-stdunit-runner-compatible test runners will fail to execute even when discovered correctly. There are ways to solve this without addressing a/b/c - just defining a standard command to run that signals success/failure with it's exit code. Packages can export a particular flavour of that in their setup.py if they have exceptional needs, and do nothing in the common case. That doesn't solve 'how to introspect a package test suite' but for distro packagers - and large scale CI integration - that doesn't matter. For instance testrepository offers a setuptools extension to let it be used trivially, I believe nose does something similar. Having something that would let *any* test suite spit out folk's favourite test protocol de jour would be brilliant of course :). [junit-xml, subunit, TAP, ...] -Rob -- Robert Collins Distinguished Technologist HP Cloud Services ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On 5 March 2013 05:51, Barry Warsaw wrote: > I should have added "from the command line". E.g. is it: > > $ python -m unittest discover > $ python setup.py test > $ python setup.py nosetests > $ python -m nose test > $ nosetests-X.Y $ testr run :) > Besides having a multitude of choices, there's almost no way to automatically > discover (e.g. by metadata inspection or some such) how to invoke the tests. > You're often lucky if there's a README.test and it's still accurate. If there is a .testr.conf you can run 'testr init; testr run'. Thats the defined entry point for testr, and .testr.conf can specify running make, or setup.py build or whatever else is needed to run tests. I would love to see a declaritive interface so that you can tell that is what you should run. -Rob -- Robert Collins Distinguished Technologist HP Cloud Services ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] built-in Python test runner (was: Python Language Summit at PyCon: Agenda)
On 5 March 2013 10:26, Eli Bendersky wrote: > [Splitting into a separate thread] > > Do we really need to overthink something that requires a trivial alias to > set up for one's own convenience? The big thing is automated tools, not developers. When distributors want to redistribute packages they want to be sure they work. Running the tests is a pretty good signal for that, but having every package slightly different adds to the work they need to do. Being able to do 'setup.py test' consistently, everywhere - that would be great. -Rob -- Robert Collins Distinguished Technologist HP Cloud Services ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] built-in Python test runner (was: Python Language Summit at PyCon: Agenda)
On 5 March 2013 12:49, Eli Bendersky wrote: > > On Mon, Mar 4, 2013 at 2:14 PM, Barry Warsaw wrote: >> >> On Mar 05, 2013, at 11:01 AM, Robert Collins wrote: >> >> >The big thing is automated tools, not developers. >> >> Exactly. > > I don't understand. Is "python -m unittest discover" too much typing for > automatic tools? If anything, it's much more portable across Python versions > since any new coommand/script won't be added before 3.4, while the longer > version works in 2.7 and 3.2+ It isn't about length. It is about knowing that *that* is what to type (and btw that exact command cannot run twisted's tests, among many other projects tests). Perhaps we are talking about different things. A top level script to run tests is interesting, but orthogonal to the thing Barry was asking for. -Rob -- Robert Collins Distinguished Technologist HP Cloud Services ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On 5 March 2013 13:21, Michael Foord wrote: > > We can certainly talk about it - although as Guido says, something specific > may be easier to have a useful discussion about. > > Reading through your blog articles it seemed like a whole lot of subunit > context was required to understand the specific > proposal you're making for the TestResult. It also *seems* like you're > redesigning the TestResult for a single use case > (distributed testing) with an api that looks quite "odd" for anything that > isn't that use case. I'd rather see how we can > make the TestResult play *better* with those requirements. That discussion > probably belongs in another thread - or at > the summit. Right - all I wanted was to flag that you and I and any other interested parties should discuss this at the summit :). -Rob -- Robert Collins Distinguished Technologist HP Cloud Services ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] built-in Python test runner (was: Python Language Summit at PyCon: Agenda)
On 5 March 2013 13:35, Eli Bendersky wrote: > Perhaps :-) > I'm specifically referring to a new top-level script that will run all > unittests in discovery mode from the current directory, as a shortcut to > "python -m unittest discover". ISTM this is at leas in part what was > discussed, and my email was in this context. So that is interesting, but its not sufficient to meet the automation need Barry is calling out, unless all test suites can be run by 'python -m unittest discover' with no additional parameters [and a pretty large subset cannot]. -Rob -- Robert Collins Distinguished Technologist HP Cloud Services ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On 5 March 2013 13:50, Michael Foord wrote: >> Right - all I wanted was to flag that you and I and any other >> interested parties should discuss this at the summit :). > > I've added a testing topic to the agenda. At the very least you could outline > your streaming test result proposal, or kick off a meta discussion. We'll > probably time limit the discussion so some specific focus will make it more > productive - or maybe you can get a feel for how open to major changes in > this area other python devs are. Cool. I can step through the core use cases and differences to what TestResult is in pretty short order. We can spider out from there as folk desire. -Rob -- Robert Collins Distinguished Technologist HP Cloud Services ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] built-in Python test runner (was: Python Language Summit at PyCon: Agenda)
On 5 March 2013 20:02, Lennart Regebro wrote: > On Tue, Mar 5, 2013 at 1:41 AM, Robert Collins > wrote: >> So that is interesting, but its not sufficient to meet the automation >> need Barry is calling out, unless all test suites can be run by >> 'python -m unittest discover' with no additional parameters [and a >> pretty large subset cannot]. > > But can they be changed so they are? That's gotta be the important bit. In principle maybe. Need to talk with the trial developers, nose developers, py.test developers etc - to get consensus on a number of internal API friction points. > What's needed here is not a tool that can run all unittests in > existence, but an official way for automated tools to run tests, with > the ability for any test and test framework to hook into that, so that > you can run any test suite automatically from an automated tool. The, > once that mechanism has been identified/implemented, we need to tell > everybody to do this. I think the command line is the right place to do that - declare as metadata the command line to run a packages tests. > I don't care much what that mechanism is, but I think the easiest way > to get there is to tell people to extend distutils with a test command > (or use Distribute) and perhaps add such a command in 3.4 that will do > the unittest discover thingy. I remember looking into zope.testrunner > hooking into that mechanism as well, but I don't remember what the > outcome was. Agreed. -Rob -- Robert Collins Distinguished Technologist HP Cloud Services ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] casefolding in pathlib (PEP 428)
On 12 April 2013 09:18, Oleg Broytman wrote: > On Thu, Apr 11, 2013 at 02:11:21PM -0700, Guido van Rossum > wrote: >> - the case-folding algorithm on some filesystems is burned into the >> disk when the disk is formatted > >Into the partition, I guess, not the physical disc? CDROMs - Joliet IIRC - so yes, physical disc. -Rob -- Robert Collins Distinguished Technologist HP Cloud Services ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stop using timeit, use perf.timeit!
On 11 June 2016 at 04:09, Victor Stinner wrote: ..> We should design a CLI command to do timeit+compare at once. http://judge.readthedocs.io/en/latest/ might offer some inspiration There's also ministat - https://www.freebsd.org/cgi/man.cgi?query=ministat&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?
On 16 Jun 2016 6:55 PM, "Larry Hastings" wrote: > > > Why do you call it only "semi-fixed"? As far as I understand it, the semantics of os.urandom() in 3.5.2rc1 are indistinguishable from reading from /dev/urandom directly, except it may not need to use a file handle. Which is a contract change. Someone testing in E.g. a chroot could have a different device on /dev/urandom, and now they will need to intercept syscalls for the same effect. Personally I think this is fine, but assuming i see Barry's point correctly, it is indeed but the same as it was. -rob ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing memoryview object patch from Python 2.7
On 14 December 2016 at 01:26, Sesha Narayanan Subbiah wrote: > Hello > > > I have some implementation that currently uses python 2.6.4, which I m > trying to upgrade to Python 2.7.6. After upgrade, I get the following error: > > > "expected string or Unicode object, memoryview found" > > > On checking further, I could find that memory view object has been back > ported to python 2.7 using this patch: > > > https://bugs.python.org/issue2396 > > > I would like to know if it is safe to revert this patch alone from Python > 2.7.6, or do we know if there are any other dependencies? I'm not sure - if you're going to run with old, custom, builds of Python, you're probably best served by testing comprehensively for this yourself. That said, I have to presume that the error you're getting is from some code that should be changed anyway, and will need to be changed when you move to Python 3. Please remember that Python 2.7.6 was released in 2013 - there have been many security issues since then, including some of the most egregious SSL issues ever, which should prompt you to run the latest 2.7 branch (if you're unable to migrate straight to 3.x. -Rob ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing memoryview object patch from Python 2.7
On 14 December 2016 at 18:10, Sesha Narayanan Subbiah wrote: > Hi Rob > > Thanks for your reply. > > From http://legacy.python.org/download/, I could see that the current > production releases are Python 3.4 and Python 2.7.6. Nope - https://www.python.org/downloads/ - 2.7.12 and 3.5.2 are current. The 'legacy' domain there was from a site revamp, I think its causing confusion at this point and we should look at retiring it completely. > Since we use python for some our legacy applications, we don't want to > switch to Python 3.0 right now. Moreover, since Python 2.6 is not supported > anymore, we want to upgrade to Python 2.7. > Do you suggest I should use Python 2.7.12 which is the latest version in 2.7 > series? I picked up 2.7.6, since it was listed as production release and > assumed it is the most stable version. If you can, 3.5.2 is where to switch to. If that won't work, 2.7.12 yes. -Rob ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecate `from __future__ import unicode_literals`?
On 17 December 2016 at 08:24, Guido van Rossum wrote: > I am beginning to think that `from __future__ import unicode_literals` does > more harm than good. I don't recall exactly why we introduced it, but with > the restoration of u"" literals in Python 3.3 we have a much better story > for writing straddling code that is unicode-correct. > > The problem is that the future import does both too much and not enough -- > it does too much because it changes literals to unicode even in contexts > where there is no benefit (e.g. the argument to getattr() -- I still hear of > code that breaks due to this occasionally) and at the same time it doesn't > do anything for strings that you read from files, receive from the network, > or even from other files that don't use the future import. > > I wonder if we can add an official note to the 2.7 docs recommending against > it? (And maybe even to the 3.x docs if it's mentioned there at all.) I think thats a good idea. I've found u"" to be entirely sufficient and very robust. Perhaps also have python2 -3 report on it? -Rob ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 unittest does not support namespace packages for discovering
On 24 March 2017 at 04:59, INADA Naoki wrote: > And this issue is relating to it too: http://bugs.python.org/issue29716 > > In short, "namespace package" is for make it possible to `pip install > foo_bar foo_baz`, > when foo_bar provides `foo.bar` and foo_baz provides `foo.baz` > package. (foo is namespace package). > > If unittests searches normal directly, it may walk deep into very > large tree containing > millions of directories. I don't like it. That is a risk, OTOH I think the failure to do what folk expect is a bigger risk. -Rob ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: Request to postpone some Python 3.9 incompatible changes to Python 3.10
On Fri, 24 Jan 2020 at 14:46, Ethan Furman wrote: > On 01/23/2020 03:36 PM, Barry Warsaw wrote: > > On Jan 23, 2020, at 14:03, Victor Stinner wrote: > > >> It's not only about specific changes, but more a discussion about a > >> general policy to decide if a deprecated feature should stay until > >> 3.10, or if it's ok to remove it in 3.9. > > > > Given that we’ve changed the release cadence to one major release per > year, it doesn’t seem that much of a burden to revert and carry these > changes forward into Python 3.10. And if it helps with the migration off > of Python 2.7, then +1 from me. > > I suspect anyone who's waited this long to switch to Python 3 will be okay > with staying on 3.8 for a few years. > The issue isn't end users, it is projects who will then be incompatible with 3.10 for "a few years." -Rob ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/D7RVF7E7FKEZ46NW4YQWJLRCAGYESU7R/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Set PyEval_SetTrace to method from class
Hi Leandro, it seems that your email is probably more about how to use the existing PyEval_SetTrace API, not about developing or changing Python itself. May I direct you to the "Got a Python problem or question?" section from https://www.python.org/about/help/ web page, where you are likely to get a good answer. Cheers, Rob On Sun, 5 Apr 2020 at 15:09, Leandro Müller wrote: > Hello. > > I need to set a PyEval_SetTrace with a new class instance. > All examples use simple function. > PyEval_SetTrace(trace_trampoline, obj); > > Following my problem: > I start three threads, but I need to check trace C just one thread. > If I run a while on C trace on thread, it stops all threads. > > So, I've been thinking to run Trace with a new instance class C. > > Any suggestions? > > > Att. > > Leandro Müller > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/CDH7BTAYPJ6QRTXI32QRJVEPKWXEPYF4/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZJ32HUJP7T6MCZCZ2UP2GYON2PO37ROJ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 622 (match statement) playground
On Thu, 2 Jul 2020 at 09:28, Matthias Bussonnier wrote: > > It's still weird user experience as if you swap case .z and case z you don't > get the Unbound error anymore. SO it can work w/o global. For some value of work: if z comes before .z, the .z branch will never get evaluated, because the binding of z will take precedence, making the .z branch dead code. This is something I would expect linters to pick up on (eventually), or possibly the compiler itself. And if you don't put z first, anything that hits the .z branch will throw the UnboundLocalError. -Rob ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S5TLBC4G42S5VGPVOT7DIILMSON6MIO7/ Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-Dev] test_io fails on test_1686475
On Sun, 2009-03-01 at 23:04 +0100, Cesare Di Mauro wrote: > Running the test suite with Python 2.6.1 32 bit (compiled in DEBUG mode > with Visual Studio Express Edition 2008) on Vista x64, I've got an assert > error: > > test_1686475 (__main__.StatAttributeTests) ... Assertion failed: > (__int64)(int)((in / 1000) - secs_between_epochs) == ((in / 1000) > - secs_between_epochs), file ..\Modules\posixmodule.c, line 790 > > I have no idea about this failure. Any hint? [I haven't checked the test yet, let me know if my guess is wrong :)]. It looks to me that its asserting something to do with stat times; note that windows has up to a 2 second granularity for stat times on files - you may be hitting some code that assumes a sane file system :) -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unittest package
On Thu, 2009-04-02 at 16:58 -0500, Michael Foord wrote: > Hello all, > > The unittest module is around 1500 lines of code now, and the tests are > 3000 lines. > > It would be much easier to maintain as a package rather than a module. > Shall I work on a suggested structure or are there objections in principle? > > Obviously all the functionality would still be available from the > top-level unittest namespace (for backwards compatibility). > > Michael I'd like to see this; jmls' testtools package has a layout for this which is quite nice. -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Sun, 2009-04-05 at 18:08 +0900, Stephen J. Turnbull wrote: > Steve Holden writes: > > > Not only that, but the Cygwin packaging system appears to be extremely > > difficult to organize a package for. > > Really? I Don't Do Windows[tm], but the people who did installers and > stuff for XEmacs releases never had problems with it. It was much > more painful to create the .exe-style Windows installers. Back when I was maintaining setup.exe was when XEmacs started using setup.exe to do installers; it must have been fairly straight forward because we first heard of it when it was complete :). The following may have changed, but I doubt it has changed dramatically - the setup.exe system is kindof trivial: There is a .lst file which is a .INI format file listing packages and direct dependencies. - each package is a .tar.(gz|bz2) which is unpacked on disk, and [optional] post-install, pre-removal scripts inside the tarball. Doing an installer for something not part of Cygwin requires a one-time fork of the setup.exe program, to change the master source for .lst files, and thats about it. Beyond that its all maintaining whatever set of packages and dependencies you have. If you are installing things for Cygwin itself you can just depend directly on things Cygwin ships in your .lst file; and not ship a setup.exe at all - setup.exe can source from many places to satisfy dependencies. -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Rethinking intern() and its data structure
On Thu, 2009-04-09 at 21:26 -0700, Guido van Rossum wrote: > Just to add some skepticism, has anyone done any kind of > instrumentation of bzr start-up behavior? We sure have. 'bzr --profile-imports' reports on the time to import different modules (both cumulative and individually). We have a lazy module loader that allows us to defer loading modules we might not use (though if they are needed we are in fact going to pay for loading them eventually). We monkeypatch the standard library where modules we want are unreasonably expensive to import (for instance by making a regex we wouldn't use be lazy compiled rather than compiled at import time). > IIRC every time I was asked > to reduce the start-up cost of some Python app, the cause was too many > imports, and the solution was either to speed up import itself (.pyc > files were the first thing ever that came out of that -- importing > from a single .zip file is one of the more recent tricks) or to reduce > the number of modules imported at start-up (or both :-). Heavy-weight > frameworks are usually the root cause, but usually there's nothing > that can be done about that by the time you've reached this point. So, > amen on the good luck, but please start with a bit of analysis. Certainly, import time is part of it: robe...@lifeless-64:~$ python -m timeit -s 'import sys; import bzrlib.errors' "del sys.modules['bzrlib.errors']; import bzrlib.errors" 10 loops, best of 3: 18.7 msec per loop (errors.py is 3027 lines long with 347 exception classes). We've also looked lower - python does a lot of stat operations search for imports and determining if the pyc is up to date; these appear to only really matter on cold-cache imports (but they matter a lot then); in hot-cache situations they are insignificant. Uhm, there's probably more - but I just wanted to note that we have done quite a bit of analysis. I think a large chunk of our problem is having too much code loaded when only a small fraction will be used in any one operation. Consider importing bzrlib errors - 10% of the startup time for 'bzr help'. In any operation only a few of those exceptions will be used - and typically 0. -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Rethinking intern() and its data structure
On Fri, 2009-04-10 at 11:52 +, Antoine Pitrou wrote: > Robert Collins canonical.com> writes: > > > > (errors.py is 3027 lines long with 347 exception classes). > > 347 exception classes? Perhaps your framework is over-engineered. > > Similarly, when using a heavy Web framework, reloading a Web app can take > several seconds... but I won't blame Python for that. Well, we've added exceptions as we needed them. This isn't much different to errno in C programs; the errno range has expanded as people have wanted to signal that specific situations have arisen. The key thing for us is to have both something that can be caught (for library users of bzrlib) and something that can be formatted with variable substitution (for displaying to users). If there are better ways to approach this in python than what we've done, that would be great. -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces
On Mon, 2009-04-27 at 22:25 -0700, Glenn Linderman wrote: > > Indeed, that was the missing piece. I'd forgotten about the > encodings > that use escape sequences, rather than UTF-8, and DBCS. I don't > think > those encodings are permitted by POSIX file systems, but I suppose > they > could sneak in via Environment variable values, and the like. This may already have been discussed, and if so I apologise for the for the noise. Does the PEP take into consideration the normalising behaviour of Mac OSX ? We've had some ongoing challenges in bzr related to this with bzr. -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
On Sun, 2009-08-30 at 16:28 -0700, Brett Cannon wrote: > > > My question is how important is this functionality? Do I really need > to go through and add an argument to marshal.loads or some new > function just to set co_filename to something that someone explicitly > set in a .pyc file? Or I can let this go and have this be the one > place where builtins.__import__ and importlib.__import__ differ and > just not worry about it? Just to be clear, this would show up if I: had a python tree built and run stuff from it symlinked to that tree from somewhere else ran stuff from that somewhere else - because the pyc is already on disk? Thats been an invaluable 'wtf' debugging tool at various times, because the odd provenance of the path in the pyc makes it extremely clear that what is being loaded isn't what one had thought was being loaded. OTOH, always showing the path that the pyc was *actually found at* would fix the weirdness that occurs when you mv a python tree from one place to another. -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] time.clock() on windows
On Thu, 2009-10-22 at 15:21 +1100, Mark Hammond wrote: > I'd be very surprised if any applications rely on > the fact that each process starts counting at zero, so if someone can > come up with a high-res counter which avoids that artifact I'd expect it > could be used. Could you offset it by the system time on the first call? -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bug 7183 and Python 2.6.4
On Thu, 2009-10-22 at 13:16 -0400, Tres Seaver wrote: ... > That being said, I can't this bug as a release blocker: people can > either upgrade to super-current Boost, or stick with 2.6.2 until they can. Thats the challenge Ubuntu faces: https://bugs.edge.launchpad.net/ubuntu/+source/boost1.38/+bug/457688 We've just announced our Karmic RC, boost 1.40 isn't released, and python 2.6.3 doesn't work with a released boost :( -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bug 7183 and Python 2.6.4
On Thu, 2009-10-22 at 21:27 +0100, Paul Moore wrote: > 2009/10/22 Robert Collins : > > On Thu, 2009-10-22 at 13:16 -0400, Tres Seaver wrote: > > ... > >> That being said, I can't this bug as a release blocker: people can > >> either upgrade to super-current Boost, or stick with 2.6.2 until they can. > > > > Thats the challenge Ubuntu faces: > > https://bugs.edge.launchpad.net/ubuntu/+source/boost1.38/+bug/457688 > > > > We've just announced our Karmic RC, boost 1.40 isn't released, and > > python 2.6.3 doesn't work with a released boost :( > > ?? Boost 1.40 was released on 27th August, according to www.boost.org... Oh bah, teach me to trust comments in bug reports :) Still, a full release is high risk, one of the Ubuntu devs is looking for the single patch they added to fix things. -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bizarre mtime behaviour
On Sun, 2009-11-01 at 12:03 -0500, Eric Smith wrote: > Antoine Pitrou wrote: > > Adam Olsen gmail.com> writes: > >> Looks like an OS bug to me. Linux I'm guessing? > > > > Yes, but only on certain boxes. I could never reproduce on my home box. > > RDM (David)'s buildbot is a Gentoo vserver with a reiserfs filesystem. > > You'll occasionally see something similar on Windows boxes running FAT, > because the timestamps stored in the filesystem have a 2 second > granularity there. Not sure if this might be similar, or just a reiserfs > issue. Some years ago there was a bug where the dentry cache held timestamps more granular than those that the filesystem could represent. We saw this cause build failures which were troubling to reproduce :) I haven't seen Linux arbitrarily invent time travel so far:). The FAT rounding issue is a possibility, but I didn't think reiserfs was short that much precision. I'd check that the work area you had really was reiser, not a mounted AT partition, and if its not look up the ReiserFS precision for mtime. -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyPI comments and ratings, *really*?
On Thu, 2009-11-12 at 08:25 -0600, Barry Warsaw wrote: > > > That's distressing. For better or worse PyPI is the central > repository of 3rd party packages. It should be easy, desirable, fun > and socially encouraged to get your packages there. Its already socially encouraged: heck, if package foo is not on PyPI, it doesn't exist. As an author who hosts code elsewhere, I really want to get feedback on my packages. I don't really see the point of having a comment system in PyPI ((*for me*), but I would really like to be able to have a link to the appropriate web page for folk that want to ask questions. To be clear, I mean a specific link, not just the 'link to a website'. -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setUpClass and setUpModule in unittest
On Sat, 2010-02-13 at 01:04 +, Michael Foord wrote:
> > However from this example I *cannot* guess whether those resources are
> > set up and torn down per test or per test class.
> This particular example is the equivalent of setUpClass - so by
> declaring the resource as a class attribute it will created before the
> first test for the class is run and disposed of after the last test for
> the class.
>
> You could *also* create a single resource and share it between several
> test classes, or even across classes in several modules, and have it
> created and disposed of at the right point. I've copied Rob Collins in
> on this email in case I've misunderstood.
Yes, precisely.
> > Also the notation
> >
> >resources = [('workdir', MyTempDir())]
> >
> > looks pretty ugly -- if 'workdir' ends up being an instance attribute,
> > why not make it a dict instead of a list of tuples? Or even better, a
> > could each resource become a class variable?
I have two key 'todos' planned in the notation for testresources:
- I want to make a decorator to let individual tests (rather
than /just/ class scope as currently happens) get resources easily.
Something like
@resource(workdir=MyTempDir())
def test_foo(self):
pass
Secondly, I want to make the class level 'resources' list into a dict.
It was a list initially, for misguided reasons that no longer apply, and
it would be much nicer and clearer as a dict.
> >
> I guess we could introspect the class for every attribute that is a
> resource, but I prefer some way of explicitly declaring which resources
> a TestCase is using.
I could see doing the below as an alternative:
@resource(workdir=MyTempDir())
class TestFoo(TestCase):
...
I'm not personally very keen on inspecting everything in self.__dict__,
I suspect it would tickle bugs in other unittest extensions. However I'm
not really /against/ it - I don't think it will result in bad test
behaviour or isolation issues. So if users would like it, lets do it.
-Rob
signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setUpClass and setUpModule in unittest
On Fri, 2010-02-12 at 12:27 -0800, Guido van Rossum wrote: > On Fri, Feb 12, 2010 at 12:20 PM, wrote: > > The idea is that you're declaring what the tests need in order to work. > > You're not explicitly defining the order in which things are set up and torn > > down. That is left up to another part of the library to determine. > > > > One such other part, OptimisingTestSuite, will look at *all* of your tests > > and find an order which involves the least redundant effort. > > So is there a way to associate a "cost" with a resource? I may have > one resource which is simply a /tmp subdirectory (very cheap) and > another that requires starting a database service (very expensive). From the pydoc: :ivar setUpCost: The relative cost to construct a resource of this type. One good approach is to set this to the number of seconds it normally takes to set up the resource. :ivar tearDownCost: The relative cost to tear down a resource of this type. One good approach is to set this to the number of seconds it normally takes to tear down the resource. > > You might have something else that breaks up the test run across multiple > > processes and uses the resource declarations to run all tests requiring one > > thing in one process and all tests requiring another thing somewhere else. > > I admire the approach, though I am skeptical. We have a thing to split > up tests at Google which looks at past running times for tests to make > an informed opinion. Have you thought of that? I think thats a great way to do it; in fact doing the same thing to assign setup and teardown costs would be lovely; I should write some glue to do that automatically for people. -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setUpClass and setUpModule in unittest
On Sat, 2010-02-13 at 10:42 +, Antoine Pitrou wrote:
> Robert Collins robertcollins.net> writes:
> >
> > I'm not personally very keen on inspecting everything in self.__dict__,
> > I suspect it would tickle bugs in other unittest extensions. However I'm
> > not really /against/ it - I don't think it will result in bad test
> > behaviour or isolation issues. So if users would like it, lets do it.
>
> Why not take all resource_XXX attributes?
Sure, though if we're just introspecting I don't see much risk
difference between resource_XXX and all XXX. As I say above I'm happy to
do it if folk think it will be nice.
> By the way, how does a given test access the allocated resource? Say, the DB
> connection. Does it become an attribute of the test case instance?
yes. Given
class Foo(TestCase):
resources = {'thing', MyResourceManager()}
def test_foo(self):
self.thing
self.thing will access the resource returned by MyResourceManager.make()
-Rob
signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setUpClass and setUpModule in unittest
On Tue, 2010-02-16 at 02:09 -0500, Glyph Lefkowitz wrote: > > > > In a recent message he was talking about either breaking > > > compatibility with TestSuite implementations that override run(), > > > or test-reordering - both of which I consider important, core > > > features of the unittest module. > > > > Well, by "breaking compatibility with custom TestSuite > > implementations that override run" I mean that is one possible place > > to put the functionality. Code that does override it will *not* stop > > working, it just won't support the new features. > > > > > Ah, I see. This doesn't sound *too* bad, but I'd personally prefer it > if the distinction were a bit more clearly drawn. I'd like frameworks > to be able to implement extension functionality without having to > first stub out functionality. In other words, if I want a test suite > without setUpClass, I'd prefer to avoid having an abstraction > inversion. +1 > > If we chose this implementation strategy there would be no > > compatibility issues for existing tests / frameworks that don't use > > the new features. > > That's very good to hear. It does however get tougher to be 'stdlib compatible' for frameworks that extend the stdlib - at least with how extensions work today. > > If tests do want to use the new features then the framework authors > > will need to ensure they are compatible with them. This seems like a > > reasonable trade-off to me. We can ensure that it is easy to write > > custom TestSuite objects that work with earlier versions of unittest > > but are also compatible with setUpClass in 2.7 (and document the > > recipe - although I expect it will just mean that TestSuite.run > > should call a single method if it exists). > > > > > This is something that I hope Jonathan Lange or Robert Collins will > chime in to comment on: expanding the protocol between suite and test > is an area which is fraught with peril, but it seems like it's > something that test framework authors always want to do. (Personally, > *I* really want to do it because I want to be able to run things > asynchronously, so the semantics of 'run()' need to change pretty > dramatically to support that...) It might be good to eventually > develop a general mechanism for this, rather than building up an > ad-hoc list of test-feature compatibility recipes which involve a list > of if hasattr(...): foo(); checks in every suite implementation. Please have a look at the testtools.TestCase.run - its incomplete, but its working towards making it possible for trial to not need to replace run, but instead provide a couple of hooks (registered during setUp) to handle what you need. What it currently offers is catching additional exceptions for you, which is a common form of extension. bzrlib is using this quite successfully, and we deleted a lot of code that overlapped the stdlib unittest run(). > > Perhaps a better idea might be to also add startTest and stopTest > > methods to TestSuite so that frameworks can build in features like > > timing tests (etc) without having to override run itself. This is > > already possible in the TestResult of course, which is a more common > > extensibility point in *my* experience. > > > > I think timing and monitoring tests can mostly be done in the > TestResult class; those were bad examples. There's stuff like > synthesizing arguments for test methods, or deciding to repeat a > potentially flaky test method before reporting a failure, which are > not possible to do from the result. I'm not sure that startTest and > stopTest hooks help with those features, the ones which really need > suites; it would seem it mostly gives you a hook to do stuff that > could already be done in TestResult anyway. Also its not really possible to 'run one thing' around a test at the moment - theres no good place (without changing tests or doing somewhat convoluted stuff) to have custom code sit in the stack above the test code - this makes it harder to handle: - profiling - drop-into-a-debugger - $other use case This is also in my hit-list of things to solve-and-propose-for-stdlib-unittest that I blogged about a while back. -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Sun, 2010-02-28 at 12:21 -0800, Brett Cannon wrote: > > Actually it's four: name/__init__.py, name/__init__.pyc, name.py, and > then name.pyc. And just so people have terminology to go with all of > this, this search is what the finder does to say whether it can or > cannot handle the requested module. Aren't there also: name.so namemodule.so ? -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Mon, 2010-03-01 at 12:35 +1300, Greg Ewing wrote: > > Yes, although that would then incur higher stat overheads for > people distributing .pyc files. There doesn't seem to be a > way of pleasing everyone. > > This is all assuming that the extra stat calls are actually > a problem. Does anyone have any evidence that they would > really take significant time compared to loading the module? > Once you've looked for one file in a given directory, looking > for another one in the same directory ought to be quite fast, > since all the relevant directory blocks will be in the > filesystem cache. We've done a bunch of testing in bzrlib. Basic things are: - statting /is/ expensive *if* you don't use the result. - loading code is the main cost *once* you have a hot disk cache Specifically, stats for files that are *not present* incur page-in costs for the dentries needed to determine the file is absent. In the special case of probing for $name.$ext1, ...$ext2, ...$ext3, you generally hit the same pages and don't incur additional page in costs. (you'll hit the same page in most file systems when you look for the second and third entries). In most file systems stats for files that *are present* also incur a page-in for the inode of the file. If you then do not read the file, this is I/O that doesn't really gain anything. Being able to disable .py file usage completely - so that only foo.pyc and foo/__init__.pyc are probed for, could have a very noticable change in the cold cache startup time. # Startup time for bzr (cold cache): $ drop-caches $ time bzr --no-plugins revno 5061 real0m8.875s user0m0.210s sys 0m0.140s # Hot cache $ time bzr --no-plugins revno 5061 real0m0.307s user0m0.250s sys 0m0.040s (revno is a small command that reads a small amount of data - just enough to trigger demand loading of the core repository layers and so on). strace timings for those two operations: cold cache: $ strace -c bzr --no-plugins revno 5061 % time seconds usecs/call callserrors syscall -- --- --- - - 56.340.04 76 527 read 28.980.020573 9 2273 1905 open 14.430.010248 14 734 625 stat 0.150.000107 0 533 fstat ... hot cache: % time seconds usecs/call callserrors syscall -- --- --- - - 45.100.000368 92 4 getdents 19.490.000159 0 527 read 16.910.000138 1 163 munmap 10.050.82 254 mprotect 8.460.69 0 2273 1905 open 0.000.00 0 8 write 0.000.00 0 367 close 0.000.00 0 734 625 stat ... Cheers, Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Sun, 2010-02-28 at 18:11 -0800, Brett Cannon wrote: ... > So then looking for a .pyc alongside a .py or vice versa > should be almost free, and we shouldn't be worrying about > it. > But that is making the assumption that all filesystems operate this > way (.e.g does NFS have the same performance characteristics?). NFS doesn't cache pages, rather it caches individual entries. I do not know if adjacent data is pre-populated into an NFS client cache. I rather suspect not, but the general point is true - many filesystems cache by pages, not all. And some have unsorted lists or might have hash tables rather than b-trees or sorted lists, where locality of reference doesn't help at all. (VFAT is unsorted list, IIRC). > hot cache: > % time seconds usecs/call callserrors > syscall > -- --- --- - - > > 45.100.000368 92 4 > getdents > > 0.000.00 0 734 625 stat > > > Further supporting the idea that stat calls are negligible > once the cache is warmed up. > > > But that's the point: once it's warmed up. This is not the case when > executing a script once every once in a while compared to something > bzr where you are most likely going to execute the command multiple > times within a small timeframe. bzr would /love/ cold cache times to come down. They are one of the most glaring performance differences between a large C program, and a large python program. Even though the second run is fast, it hurts the first time you run 'bzr st'. (**) -Rob (**) To the extent that I've seriously considered an import hook to disable normal imports under bzrlib, and special case how the search works so that we only load pyc's and assume all imports are absolute. signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] patch for review: unittest ImportError handling
I'd really like to see a fix that works with loadTestsFromNames - generating failing tests, for instance, and the failing tests having the full import error string in them. This doesn't preclude raising ImportError from loadTestFromName, and in fact I'd encourage that as a step towards the aforementioned loadTestsFromNames change. It is a change with the existing misguided behaviour; sometimes thats better to do than preserving compat; as you say a new API might be best overall though. I'd go with a keyword arg rather than a new function, I think. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Sun, May 23, 2010 at 10:15 PM, Brian Quinlan wrote: >> Also, you can't fix bugs except by >> releasing new versions of Python. Therefore the API must be completely >> stable, and the product virtually bugfree before it should be in >> stdlib. The best way of ensuring that is to release it as a separate >> module on PyPI, and let it stabilize for a couple of years. > > Yeah but that model isn't likely to work with this package. > Cheers, > Brian Forgive my ignorance, but why do you say that that model won't work with this package? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bytes / unicode
Also, url's are bytestrings - by definition; if the standard library has made them unicode objects in 3, I expect a lot of pain in the webserver space. -Rob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bytes / unicode
2010/6/21 Stephen J. Turnbull : > Robert Collins writes: > > > Also, url's are bytestrings - by definition; > > Eh? RFC 3896 explicitly says ?Definitions of Managed Objects for the DS3/E3 Interface Type Perhaps you mean 3986 ? :) > A URI is an identifier consisting of a sequence of characters > matching the syntax rule named in Section 3. > > (where the phrase "sequence of characters" appears in all ancestors I > found back to RFC 1738), and Sure, ok, let me unpack what I meant just a little. An abstract URI is neither unicode nor bytes per se - see section 1.2.1 " A URI is a sequence of characters from a very limited set: the letters of the basic Latin alphabet, digits, and a few special characters. " URI interpretation is fairly strictly separated between producers and consumers. A consumer can manipulate a url with other url fragments - e.g. doing urljoin. But it needs to keep the url as a url and not try to decode it to a unicode representation. The producer of the url however, can decode via whatever heuristics it wants - because it defines the encoding used to go from unicode to URL encoding. As an example, if I give the uri "http://server/%c3%83";, rendering that as http://server/Ã is able to lead to transcription errors and reinterpretation problems unless you know - out of band - that the server is using utf8 to encode. Conversely if someone enters in http://server/Ã in their browser window, choosing utf8 or their local encoding is quite arbitrary and able to not match how the server would represent that resource. Beyond that, producers can do odd things - like when there are a series of servers stacked and forwarding requests amongst themselves - where they generate different parts of the same URL using different encodings. > 2. Characters > > The URI syntax provides a method of encoding data, presumably for > the sake of identifying a resource, as a sequence of characters. > The URI characters are, in turn, frequently encoded as octets for > transport or presentation. This specification does not mandate any > particular character encoding for mapping between URI characters > and the octets used to store or transmit those characters. When a > URI appears in a protocol element, the character encoding is > defined by that protocol; without such a definition, a URI is > assumed to be in the same character encoding as the surrounding > text. Thats true, but its been taken out of context; the set of characters permitted in a URL is a strict subset of characters found in ASCII; there is a BNF that defines it and it is quite precise. While it doesn't define a set of octets, it also doesn't define support for unicode characters - individual schemes need to define the mapping used between characters define as safe and those that get percent encoded. E.g. unicode (abstract) -> utf8 -> percent encoded. See also the section on comparing URL's - Unicode isn't at all relevant. > > if the standard library has made them unicode objects in 3, I > > expect a lot of pain in the webserver space. > > Yup. But pain is inevitable if people are treating URIs (whether URLs > or otherwise) as octet sequences. Then your base URL is gonna be > b'mailto:step...@xemacs.org', but the natural thing the UI will want > to do is > > formurl = baseurl + '?subject=うるさいやつだなぁ…' > > IMO, the UI is right. "Something" like the above "ought" to work. I wish it would. The problem is not in Python here though - and casually handwaving will exacerbate it, not fix it. Modelling URL's as string like things is great from a convenience perspective, but, like file paths, they are much more complex difficult. For your particular case, subject contains characters outside the URL specification, so someone needs to choose an encoding to get them into a sequence-of-bytes-that-can-be-percent-escaped. Section 2.5, identifying data goes into this to some degree. Note a trap - the last paragraph says 'when a *NEW* URI scheme...' (emphasis mine). Existing schemes do not mandate UTF8, which is why the producer/consumer split matters. I spent a few minutes looking, but its lost in the minutiae somewhere - HTTP does not specify UTF8 (though I wish it would) for its URI's, and std66 is the generic definition and rules for new URI schemes, preserving intact the mistake of HTTP. > So the function that actually handles composing the URL should take a > string (ie, unicode), and do all escaping. The UI code should not > need to know about escaping. If nothing escapes except the function > that puts the URL in composed form, and that function always escapes, > life is easy. Arg. The problem is very similar to the file system problem: - We get given
Re: [Python-Dev] bytes / unicode
On Wed, Jun 23, 2010 at 6:09 AM, M.-A. Lemburg wrote:
>> return constant.encode('utf-8')
>>
>> So now you can write x.split(literal_as('&', x)).
>
> This polymorphism is what we used in Python2 a lot to write
> code that works for both Unicode and 8-bit strings.
>
> Unfortunately, this no longer works as easily in Python3 due
> to the literals sometimes having the wrong type and using
> such a helper function slows things down a lot.
I didn't work in 2 either - see for instance the traceback module with
an Exception with unicode args and a non-ascii file path - the file
path is in its bytes form, the string joining logic triggers an
implicit upcast and *boom*.
> Too bad we can't add such porting enhancements to Python2 anymore
Perhaps a 'py3compat' module on pypi, with things like the py._builtin
reraise helper and so forth ?
-Rob
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bytes / unicode
On Wed, Jun 23, 2010 at 12:25 PM, Glyph Lefkowitz wrote: > I can also appreciate what's been said in this thread a bunch of times: to my > knowledge, nobody has actually shown a profile of an application where > encoding is significant overhead. I believe that encoding _will_ be a > significant overhead for some applications (and actually I think it will be > very significant for some applications that I work on), but optimizations > should really be implemented once that's been demonstrated, so that there's a > better understanding of what the overhead is, exactly. Is memory a big deal? > Is CPU? Is it both? Do you want to tune for the tradeoff? etc, etc. > Clever data-structures seem premature until someone has a good idea of all > those things. bzr has a cache of decoded strings in it precisely because decode is slow. We accept slowness encoding to the users locale because thats typically much less data to examine than we've examined while generating the commit/diff/whatever. We also face memory pressure on a regular basis, and that has been, at least partly, due to UCS4 - our translation cache helps there because we have less duplicate UCS4 strings. You're welcome to dig deeper into this, but I don't have more detail paged into my head at the moment. -Rob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python equivalents in stdlib Was: Include datetime.py in stdlib or not?
On pypi - testscenarios; Its been discussed on TIP before. Its a 'run a function to parameterise some tests' API, it changes the id() of the test to include the parameters, and it can be hooked in via load_tests quite trivially. Cheers, Rob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
