Re: [Python-Dev] _PyThreadState_Current

2016-01-18 Thread Victor Stinner
Hum, you can try to lie and define Py_BUILD_CORE? Victor 2016-01-18 21:18 GMT+01:00 Maciej Fijalkowski : > Hi > > change in between 3.5.0 and 3.5.1 (hiding _PyThreadState_Current and > pyatomic.h) broke vmprof. The problem is that as a profile, vmprof can > really encounter _PyThreadState_Current

Re: [Python-Dev] PEP 509: Add a private version to dict

2016-01-18 Thread Victor Stinner
Is someone opposed to this PEP 509? The main complain was the change on the public Python API, but the PEP doesn't change the Python API anymore. I'm not aware of any remaining issue on this PEP. Victor 2016-01-11 17:49 GMT+01:00 Victor Stinner : > Hi, > > After a first ro

Re: [Python-Dev] PEP 510: Specialize functions with guards

2016-01-19 Thread Victor Stinner
Oh, I think that the PEP 510 lacks two functions to: * remove a specific specialized code * remove all specialized code Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://ma

[Python-Dev] Reference cycle on the module dict (globals())

2016-01-19 Thread Victor Stinner
Hi, While working on my FAT Python optimizer project, I found an annoying bug in my code. When at least one guard is created with a reference to the global namespace (globals(), the module dictionary), objects of the module are no more removed at exit. Example: --- import sys class MessageAtExit

Re: [Python-Dev] _PyThreadState_Current

2016-01-19 Thread Victor Stinner
Since it's a regression introduced in Python 3.5.1, I propose to introduce a new private function _PyThreadState_FastGet() to reintroduce the feature: https://bugs.python.org/issue26154 Using afunction instead of using directly the variable hides how atomic variables are implemented and so avoid c

Re: [Python-Dev] Reference cycle on the module dict (globals())

2016-01-19 Thread Victor Stinner
Hi, 2016-01-19 11:39 GMT+01:00 Petr Viktorin : >> Did I miss something obvious, or is it a known issue of the garbage >> collector on modules? > > The default type flags are for objects that don't store references. > Since you're creating a mutable container, you need to set > Py_TPFLAGS_HAVE_GC.

[Python-Dev] Devguide: Add a table summarizing status of Python branches

2016-01-20 Thread Victor Stinner
Hi, I proposed a patch for the devguide to give the current status of all Python branches: active, bugfix, security only, end-of-line, with their end-of-life when applicable (past date or scheduled date) http://bugs.python.org/issue26165 What do you think? Does it look correct? We will have to u

Re: [Python-Dev] Devguide: Add a table summarizing status of Python branches

2016-01-20 Thread Victor Stinner
I pushed my table, it will be online in a few hours (I don't know when the devguide is recompiled?): http://docs.python.org/devguide/triaging.html#generating-special-links-in-a-comment By the way, it would be super cool to rebuild the PEPs with a post-commit hook server-side, rather than having to

Re: [Python-Dev] Devguide: Add a table summarizing status of Python branches

2016-01-20 Thread Victor Stinner
2016-01-20 23:01 GMT+01:00 Brett Cannon : > This is a proposed optional, future feature leading from moving to GitHub: > https://www.python.org/dev/peps/pep-0512/#web-hooks-for-re-generating-web-content I'm using the free service ReadTheDocs.org and it's really impressive how fast it is to update

Re: [Python-Dev] PEP 509: Add a private version to dict

2016-01-20 Thread Victor Stinner
Hi, 2016-01-20 22:18 GMT+01:00 Glenn Linderman : > On 1/20/2016 12:50 PM, Brett Cannon wrote: >> >> A global (shared between all dicts) unit64 ma_version is actually quite >> reliable -- if a program does 1,000,000 dict modifications per second, >> it would take it 600,000 years till wrap-around.

Re: [Python-Dev] PEP 509: Add a private version to dict

2016-01-20 Thread Victor Stinner
2016-01-21 1:08 GMT+01:00 Brett Cannon : > On Wed, 20 Jan 2016 at 15:46 Victor Stinner >> The worst case is when a value different than the watched value is >> modified between each guard check. In this case, we always need a dict >> lookup. An heuristic can be chosen to deci

Re: [Python-Dev] Devguide: Add a table summarizing status of Python branches

2016-01-20 Thread Victor Stinner
2016-01-21 1:09 GMT+01:00 Brett Cannon : > On Wed, 20 Jan 2016 at 14:28 Victor Stinner >> I'm using the free service ReadTheDocs.org and it's really impressive >> how fast it is to update the HTML page after a push. It's usually less >> than 10 seconds. > &

Re: [Python-Dev] Devguide: Add a table summarizing status of Python branches

2016-01-20 Thread Victor Stinner
2016-01-20 22:22 GMT+01:00 Victor Stinner : > I pushed my table, it will be online in a few hours (I don't know when > the devguide is recompiled?): > http://docs.python.org/devguide/triaging.html#generating-special-links-in-a-comment Hum ok, it takes more than a few hours in fact.

Re: [Python-Dev] PEP 509: Add a private version to dict

2016-01-21 Thread Victor Stinner
2016-01-21 2:54 GMT+01:00 Andrew Barnert : > This idea worries me. I'm not sure why, but I think because of threading. > After all, it's pretty rare for two threads to both want to work on the same > dict, but very, very common for two threads to both want to work on _any_ > dict. So, imagine so

Re: [Python-Dev] PEP 509: Add a private version to dict

2016-01-21 Thread Victor Stinner
2016-01-21 6:08 GMT+01:00 Yury Selivanov : > Yeah, I think that's what we agreed on: > https://mail.python.org/pipermail/python-dev/2016-January/142837.html > > The only advantage of ma_extra pointer is that it allows to add more stuff > to dicts in the future. I don't agree on ma_extra since I do

Re: [Python-Dev] Devguide: Add a table summarizing status of Python branches

2016-01-22 Thread Victor Stinner
2016-01-21 19:42 GMT+01:00 Paul Moore : > Minor nit, the status column says "end of life", but the text below > the table uses the term "end of line" Ooops, it's a funny typo. Fixed. Thanks for the report! Victor ___ Python-Dev mailing list Python-Dev@p

Re: [Python-Dev] Devguide: Add a table summarizing status of Python branches

2016-01-22 Thread Victor Stinner
2016-01-21 18:18 GMT+01:00 Brett Cannon : > It's live: https://docs.python.org/devguide/#status-of-python-branches There is a very strange bug in this website. This URL shows the table: https://docs.python.org/devguide/ This URL doesn't show the table: https://docs.python.org/devguide/index.html

Re: [Python-Dev] Buildbot timing out - test suite failure - test_socket issue with UDP6?

2016-01-23 Thread Victor Stinner
3600 seconds is the maximum duration of a single test file. We may reduce it since a single test file should not take longer than 30 min. Maybe we can do better and put the timeout on a single test function. Victor ___ Python-Dev mailing list Python-Dev@

[Python-Dev] FAT Python (lack of) performance

2016-01-25 Thread Victor Stinner
Hi, Summary: FAT Python is not faster, but it will be ;-) -- When I started the FAT Python as a fork of CPython 3.6, I put everything in the same repository. Last weeks, I focused on splitting my giant patch (10k lines) into small reviewable patches. I wrote 3 PEP (509 dict version, 510 function

Re: [Python-Dev] FAT Python (lack of) performance

2016-01-25 Thread Victor Stinner
Hi, 2016-01-25 22:20 GMT+01:00 Ludovic Gasc : > Just thanks for this big contribution. > And maybe this project could give new ideas to optimize Python, who knows ? Sorry for my long email. I should try to summarize next time :-) In short: FAT Python is not fast today, but it will be faster if yo

Re: [Python-Dev] FAT Python (lack of) performance

2016-01-25 Thread Victor Stinner
Hi, 2016-01-25 23:28 GMT+01:00 Andrew Barnert : > On Jan 25, 2016, at 13:43, Victor Stinner wrote: >> >> According to microbenchmarks, the most promising optimizations are >> functions inlining (Python function calls are slow :-/) and specialize >> the code for the typ

Re: [Python-Dev] FAT Python (lack of) performance

2016-01-25 Thread Victor Stinner
2016-01-25 22:51 GMT+01:00 Sven R. Kunze : > - they provide a great infrastructure for optimizing CPython AND > extending/experimenting Python as an ecosystem I hope that these API will create more optimizer projects than just fatoptimizer. For example, I expect more specialized optimizers like n

Re: [Python-Dev] FAT Python (lack of) performance

2016-01-26 Thread Victor Stinner
Hi, 2016-01-26 3:21 GMT+01:00 INADA Naoki : > How can I help your work? I don't know exactly yet, but I started to write a documentation to explain how to contribute: http://faster-cpython.readthedocs.org/fat_python.html#how-can-you-contribute You may contact me directly ;-) Victor

Re: [Python-Dev] [Python-checkins] Daily reference leaks (cbd4a6a2657e): sum=134

2016-01-27 Thread Victor Stinner
Hi, I pushed a fix before you sent your message. At least test_ast should be fixed. https://hg.python.org/cpython/rev/c5df914e73ad FYI I'm unable to reproduce the test_collections leak. Victor Le mardi 26 janvier 2016, Brett Cannon a écrit : > Looks like Victor's ast.Constant change introduc

Re: [Python-Dev] FAT Python (lack of) performance

2016-01-27 Thread Victor Stinner
Paul Moore : > On 27 January 2016 at 05:23, Sjoerd Job Postmus wrote: >> On Mon, Jan 25, 2016 at 11:58:12PM +0100, Victor Stinner wrote: >>> ... >>> Oh, they are a lot of things to do! ... >> >> Just wondering, do you also need a set of (abusive) test-cases which

Re: [Python-Dev] Buildbot timing out - test suite failure - test_socket issue with UDP6?

2016-01-27 Thread Victor Stinner
2016-01-23 7:03 GMT+01:00 Chris Angelico : > I just had a major crash on the system that hosts the > angelico-debian-amd64 buildbot, and as usual, checked it carefully > after bringing everything up. It seems now to be timing out after an > hour of operation: > > http://buildbot.python.org/all/buil

Re: [Python-Dev] Fun with ancient unsupported platforms

2016-01-28 Thread Victor Stinner
We slowly remove old platforms, but only if the code specific to these old platforms is annoying to maintain. For example, I wrote the change: https://hg.python.org/cpython/rev/a1605d2508af """ Issue #22591: Drop support of MS-DOS Drop support of MS-DOS, especially of the DJGPP compiler (MS-DOS p

Re: [Python-Dev] Speeding up CPython 5-10%

2016-02-02 Thread Victor Stinner
Hi, I'm back for the FOSDEM event at Bruxelles, it was really cool. I gave talk about FAT Python and I got good feedback. But friends told me that people now have expectations on FAT Python. It looks like people care of Python performance :-) FYI the slides of my talk: https://github.com/haypo/co

Re: [Python-Dev] Opcode cache in ceval loop

2016-02-02 Thread Victor Stinner
Hi, Maybe it's worth to write a PEP to summarize all your changes to optimize CPython? It would avoid to have to follow different threads on the mailing lists, different issues on the bug tracker, with external links to GitHub gists, etc. Your code changes critical parts of Python: code object str

Re: [Python-Dev] Opcode cache in ceval loop

2016-02-02 Thread Victor Stinner
2016-02-02 20:23 GMT+01:00 Yury Selivanov : > Alright, I modified the code to optimize ALL code objects, and ran unit > tests with the above tests excluded: > > -- Max process mem (ru_maxrss) = 131858432 > -- Opcode cache number of objects = 42109 > -- Opcode cache total extra mem= 1090110

[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-03 Thread Victor Stinner
Hi, There is an old discussion about the performance of PyMem_Malloc() memory allocator. CPython is stressing a lot memory allocators. Last time I made statistics, it was for the PEP 454: "For example, the Python test suites calls malloc() , realloc() or free() 270,000 times per second in average.

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-03 Thread Victor Stinner
on.org/issue26249 for the longer context. 2016-02-03 22:03 GMT+01:00 Victor Stinner : > Does anyone recall the rationale to have two families to memory allocators? I asked Mercurial, and I found the change addind PyMem_Malloc(): --- branch: legacy-trunk user:Guido van Rossum date:

Re: [Python-Dev] speed.python.org

2016-02-04 Thread Victor Stinner
Great! 2016-02-04 7:48 GMT+01:00 Zachary Ware : > I'm happy to announce that speed.python.org is finally functional! > There's not much there yet, as each benchmark builder has only sent > one result so far (and one of those involved a bit of cheating on my > part), but it's there. > > There are l

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-04 Thread Victor Stinner
Hi, 2016-02-04 11:17 GMT+01:00 M.-A. Lemburg : >> Do you see any drawback of using pymalloc for PyMem_Malloc()? > > Yes: You cannot free memory allocated using pymalloc with the > standard C lib free(). That's not completly new. If Python is compiled in debug mode, you get a fatal error with a h

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-04 Thread Victor Stinner
Thanks for your feedback, you are asking good questions :-) 2016-02-04 13:54 GMT+01:00 M.-A. Lemburg : >> There are 536 calls to the functions PyMem_Malloc(), PyMem_Realloc() >> and PyMem_Free(). >> >> I would prefer to modify a single place having to replace 536 calls :-/ > > You have a point the

[Python-Dev] Google Summer of Code

2016-02-07 Thread Victor Stinner
Hi, I would like to propose the FAT Python project subject to the Google Summer of Code: https://developers.google.com/open-source/gsoc/ I have a long list of optimization ideas for fatoptimizer: http://fatoptimizer.readthedocs.org/en/latest/todo.html The fatoptimizer project is written in pure

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-08 Thread Victor Stinner
2016-02-04 15:05 GMT+01:00 M.-A. Lemburg : > Sometimes, yes, but we also do allocations for e.g. > parsing values in Python argument tuples (e.g. using > "es" or "et"): > > https://docs.python.org/3.6/c-api/arg.html > > We do document to use PyMem_Free() on those; not sure whether > everyone does t

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-08 Thread Victor Stinner
2016-02-07 9:22 GMT+01:00 Stefan Behnel : > Note that the PyObject_Malloc() functions have never been documented. Yeah, there is an old bug to track this: http://bugs.python.org/issue20064 > And, for example, the "what's new in 2.5" document says: > > """ > Python’s API has many different functio

[Python-Dev] Windows: Remove support of bytes filenames in the os module?

2016-02-08 Thread Victor Stinner
Hi, Since 3.3, functions of the os module started to emit DeprecationWarning when called with bytes filenames. The rationale is quite simple: Windows native type for filenames is Unicode, and the Windows has a weird behaviour when you use bytes. For example, os.listdir(b'.') gives you paths which

Re: [Python-Dev] Windows: Remove support of bytes filenames in the os module?

2016-02-08 Thread Victor Stinner
2016-02-08 15:32 GMT+01:00 Victor Stinner : > Since 3.3, functions of the os module started to emit > DeprecationWarning when called with bytes filenames. > (...) > Recently, an user complained that os.walk() doesn't work with bytes on > Windows anymore: > (...) It

[Python-Dev] Issue #26204: compiler now emits a SyntaxWarning on constant statement

2016-02-08 Thread Victor Stinner
Hi, I changed the Python compiler to ignore any kind "constant expressions", whereas it only ignored strings and integers before: http://bugs.python.org/issue26204 The compiler now also emits a SyntaxWarning on such case. IMHO the warning can help to detect bugs for developers who just learnt Pyt

Re: [Python-Dev] Issue #26204: compiler now emits a SyntaxWarning on constant statement

2016-02-08 Thread Victor Stinner
Le 8 févr. 2016 8:14 PM, "Guido van Rossum" a écrit : > Hum. I'm not excited by this idea. It is not bad syntax. Do you see an use case for "constant statements" other than strings and ellipsis? Such statement does nothing. Previously the compiler emited LOAD_CONST+POP_TOP. GCC also emits a war

Re: [Python-Dev] Issue #26204: compiler now emits a SyntaxWarning on constant statement

2016-02-08 Thread Victor Stinner
Le 8 févr. 2016 9:34 PM, "Guido van Rossum" a écrit : > If you want to do linter integration that should probably be > integrated with the user's editor, like it is in PyCharm, and IIUC > people can do this in e.g. Emacs, Sublime or Vim as well. Leave the > interpreter alone. In GCC, warnings are

Re: [Python-Dev] Issue #26204: compiler now emits a SyntaxWarning on constant statement

2016-02-08 Thread Victor Stinner
Le 8 févr. 2016 9:10 PM, "Alexander Walters" a écrit : > > I am not keen on a SyntaxWarning. Either something is python syntax, or it is not. Oh I forgot to mention that Python already emits SyntaxWarning, on "assert True" for example. Victor ___ Pyth

Re: [Python-Dev] Issue #26204: compiler now emits a SyntaxWarning on constant statement

2016-02-08 Thread Victor Stinner
2016-02-08 22:28 GMT+01:00 Alexander Walters : > What incantation do you need to do to make that behavior apparent? I didn't know. I just checked. It's assert used with a non-empty tuple: >>> assert ("tuple",) :1: SyntaxWarning: assertion is always true, perhaps remove parentheses? Victor __

Re: [Python-Dev] Windows: Remove support of bytes filenames in the os module?

2016-02-09 Thread Victor Stinner
2016-02-09 1:37 GMT+01:00 eryk sun : > For example, in codepage 932 (Japanese), it's an error if a lead byte > (i.e. 0x81-0x9F, 0xE0-0xFC) is followed by a trailing byte with a > value less than 0x40 (note that ASCII 0-9 is 0x30-0x39, so this is not > uncommon). In this case the ANSI API substitute

Re: [Python-Dev] Windows: Remove support of bytes filenames in the os module?

2016-02-09 Thread Victor Stinner
2016-02-09 1:37 GMT+01:00 eryk sun : > For example, in codepage 932 (Japanese), it's an error if a lead byte > (i.e. 0x81-0x9F, 0xE0-0xFC) is followed by a trailing byte with a > value less than 0x40 (note that ASCII 0-9 is 0x30-0x39, so this is not > uncommon). In this case the ANSI API substitute

Re: [Python-Dev] Issue #26204: compiler now emits a SyntaxWarning on constant statement

2016-02-09 Thread Victor Stinner
2016-02-09 10:57 GMT+01:00 Joseph Martinot-Lagarde : > I frequently use 1/0 as a quick break in a script or a program (it's even > more useful with post-mortem debugging). Would it be considered as a > constant and ignored instead of raising a ZeroDivisionError ? "self.x - self.y" and "1/0" are no

Re: [Python-Dev] Windows: Remove support of bytes filenames in the os module?

2016-02-09 Thread Victor Stinner
Hi, 2016-02-08 18:02 GMT+01:00 Brett Cannon : > If Unicode string don't work in Python 2 then what is Python 2/3 to do as a > cross-platform solution if we completely remove bytes support in Python 3? > Wouldn't that mean there is no common type between Python 2 & 3 that one can > use which will w

Re: [Python-Dev] Windows: Remove support of bytes filenames in the os module?

2016-02-09 Thread Victor Stinner
2016-02-08 19:26 GMT+01:00 Paul Moore : > On 8 February 2016 at 14:32, Victor Stinner wrote: >> Since 3.3, functions of the os module started to emit >> DeprecationWarning when called with bytes filenames. > > Everywhere? Or just on Windows? I can't tell from your email

Re: [Python-Dev] Windows: Remove support of bytes filenames in the os module?

2016-02-09 Thread Victor Stinner
Le mercredi 10 février 2016, Steve Dower a écrit : > > I really don't like the idea of not being able to use bytes in cross > platform code. Unless it's become feasible to use Unicode for lossless > filenames on Linux - last I heard it wasn't. > The point of my email is that even on Python 3, use

Re: [Python-Dev] Windows: Remove support of bytes filenames in theos module?

2016-02-10 Thread Victor Stinner
2016-02-10 9:30 GMT+01:00 Paul Moore : > Whether removing the bytes interface is feasible, given that there's > then no way that works across Python 2 and 3 of writing code that > manipulates the sort of bytes-that-use-multiple-encodings data that > you mention, is a separate issue. It's annoying

Re: [Python-Dev] Windows: Remove support of bytes filenames in theos module?

2016-02-10 Thread Victor Stinner
2016-02-10 11:18 GMT+01:00 Steven D'Aprano : > [steve@ando ~]$ python3.3 -c 'print(open(b"/tmp/abc\xD8\x01", "r").read())' > Hello World > > [steve@ando ~]$ python3.3 -c 'print(open("/tmp/abc\xD8\x01", "r").read())' > Traceback (most recent call last): > File "", line 1, in > FileNotFoundError:

Re: [Python-Dev] PEP 515: Underscores in Numeric Literals

2016-02-10 Thread Victor Stinner
It looks like the implementation https://bugs.python.org/issue26331 only changes the Python parser. What about other functions converting strings to numbers at runtime like int(str) and float(str)? Paul also asked for Decimal(str). Victor ___ Python-Dev

Re: [Python-Dev] PEP 515: Underscores in Numeric Literals

2016-02-11 Thread Victor Stinner
2016-02-11 9:11 GMT+01:00 Georg Brandl : > On 02/11/2016 12:04 AM, Victor Stinner wrote: >> It looks like the implementation https://bugs.python.org/issue26331 >> only changes the Python parser. >> >> What about other functions converting strings to numbers at runtime

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-12 Thread Victor Stinner
ping? 2016-02-08 15:18 GMT+01:00 Victor Stinner : > 2016-02-04 15:05 GMT+01:00 M.-A. Lemburg : >> Sometimes, yes, but we also do allocations for e.g. >> parsing values in Python argument tuples (e.g. using >> "es" or "et"): >> >> https://docs.

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-12 Thread Victor Stinner
Hi, 2016-02-12 14:31 GMT+01:00 M.-A. Lemburg : > Sorry, your email must gotten lost in my inbox. no problemo > Yes, but those are part of the stdlib. You'd need to check > a few C extensions which are not tested as part of the stdlib, > e.g. numpy, scipy, lxml, pillow, etc. (esp. ones which imp

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-03-09 Thread Victor Stinner
2016-02-08 15:18 GMT+01:00 Victor Stinner : >> Perhaps if you add some guards somewhere :-) > > We have runtime checks but only implemented in debug mode for efficiency. > > By the way, I proposed once to add an environment variable to allow to > enable these checks withou

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-03-14 Thread Victor Stinner
2016-03-09 18:54 GMT+01:00 Brett Cannon : >>> https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks >> >> The main advantage of this variable is that you don't have to >> recompile Python in debug mode to benefit of these checks. > > I just wanted to say this all sounds awesome! Tha

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-03-14 Thread Victor Stinner
2016-02-12 14:31 GMT+01:00 M.-A. Lemburg : >>> If your program has bugs, you can use a debug build of Python 3.5 to >>> detect misusage of the API. > > Yes, but people don't necessarily do this, e.g. I have > for a very long time ignored debug builds completely > and when I started to try them, I f

[Python-Dev] GSoC: looking for a student to help on FAT Python

2016-03-19 Thread Victor Stinner
Hi, I am now looking for a Google Summer of Code (GSoC) student to help me of my FAT Python project, a new static optimizer for CPython 3.6 using specialization with guards. The FAT Python project is already fully functional, the code is written and tested. I need help to implement new efficient

[Python-Dev] Make the warnings module extensible

2016-03-19 Thread Victor Stinner
Hi, I have an API question for you. I would like to add a new parameter the the showwarning() function of the warnings module. Problem: it's not possible to do that without breaking the backward compatibility (when an application replaces warnings.showwarning(), the warnings allows and promotes t

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-03-25 Thread Victor Stinner
PYTHONMALLOC=debug? Victor 2016-03-15 0:19 GMT+01:00 Victor Stinner : > 2016-02-12 14:31 GMT+01:00 M.-A. Lemburg : >>>> If your program has bugs, you can use a debug build of Python 3.5 to >>>> detect misusage of the API. >> >> Yes, but people don't necessarily

Re: [Python-Dev] Scandir module seeking new maintainer

2016-03-29 Thread Victor Stinner
Hi, 2016-03-29 2:55 GMT+02:00 Ben Hoyt : > I'm the author and current maintainer of the scandir module (Python 3.5's > os.scandir but for Python 2.x and 3.x before 3.5). But I've taken on a few > new non-programming projects and found I don't have time for the little > improvements and fixes that

Re: [Python-Dev] Not receiving bug tracker emails

2016-03-29 Thread Victor Stinner
same for me, i'm using using gmail with a @gmail.com email. Victor 2016-03-30 1:30 GMT+02:00 Martin Panter : > For the last ~36 hours I have stopped receiving emails for messages > posted in the bug tracker. Is anyone else having this problem? Has > anything changed recently? > > I have had it se

[Python-Dev] The next major Python version will be Python 8

2016-03-31 Thread Victor Stinner
Hi, Python 3 becomes more and more popular and is close to a dangerous point where it can become popular that Python 2. The PSF decided that it's time to elaborate a new secret plan to ensure that Python users suffer again with a new major release breaking all their legacy code. The PSF is happy

Re: [Python-Dev] Not receiving bug tracker emails

2016-04-02 Thread Victor Stinner
Any progress on the issue? Victor Le jeudi 31 mars 2016, Martin Panter a écrit : > On 30 March 2016 at 13:30, R. David Murray > wrote: > > Anyone know how to find out what changed from Google's POV? As far as > > we know nothing changed at the bugs end, but it is certainly possible > > that s

Re: [Python-Dev] Py_SETREF vs. Py_XSETREF

2016-04-04 Thread Victor Stinner
If some dev don't want to use the single macro for good or bad reasons, it's maybe better to have two macros to generalize their usage. The macro makes to C code shorter and easier to review. Victor ___ Python-Dev mailing list Python-Dev@python.org https

Re: [Python-Dev] thoughts on backporting __wrapped__ to 2.7?

2016-04-05 Thread Victor Stinner
See https://pypi.python.org/pypi/functools32 for the functools backport for Python 2.7. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/pyt

Re: [Python-Dev] summary: a Path protocol

2016-04-07 Thread Victor Stinner
Sorry, I don't have time to read the whole discussion. What is the problem with adding a __str__ to pathlib? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.or

Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)

2016-04-08 Thread Victor Stinner
Please write a new PEP. The topic looks to be discussed since many months by many different people on different mailing list. A PEP is a good standard to take a decision and it became clear that a decision must be taken for pathlib. Victor ___ Python-De

Re: [Python-Dev] Other pathlib improvements? was: When should pathlib stop being provisional?

2016-04-08 Thread Victor Stinner
FYI the doc of the builtin functions is the #1 in stats of docs python.org. I also read this doc every week, even if I consider that I know well Python. IMHO it's not an issue to regulary read the doc. Victor ___ Python-Dev mailing list Python-Dev@pytho

Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)

2016-04-08 Thread Victor Stinner
I like __fspath__ because it looks like os.fsencode() and os.fsdecode(). Please no builtin function, we have enough of them, but make sure that the __fspath__ is accepted in all functions expecting a filename. If you consider that a function would make your change simpler, I suggest to add os.fsp

Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)

2016-04-09 Thread Victor Stinner
os.DirEntry doesn't support bytes: os.scandir() only accept str. It's a deliberate choice. I strongly suggest to only support Unicode for filenames in Python 3. So __fspath__ must only return str, or a TypeError must be raised. Victor ___ Python-Dev mai

Re: [Python-Dev] Question about the current implementation of str

2016-04-09 Thread Victor Stinner
Le 9 avr. 2016 03:04, "Larry Hastings" a écrit : > Although the str object is immutable from Python's perspective, the C object itself is mutable. For example, for dynamically-created strings the hash field may be lazy-computed and cached inside the object. Yes, the hash is computed once on dema

Re: [Python-Dev] Question about the current implementation of str

2016-04-09 Thread Victor Stinner
2016-04-09 9:52 GMT+02:00 Victor Stinner : > But the hash is used as an heuristic to decide if a string is "immutable" or > not, the refcount is also used by the heuristic. If the string is immutable, > an operation like resize must create a new string. I'm talking abo

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

2016-04-09 Thread Victor Stinner
Please don't loose time trying yet another sandbox inside CPython. It's just a waste of time. It's broken by design. Please read my email about my attempt (pysandbox): https://lwn.net/Articles/574323/ And the LWN article: https://lwn.net/Articles/574215/ There are a lot of safe ways to run CPyth

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

2016-04-11 Thread Victor Stinner
2016-04-10 18:43 GMT+02:00 Jon Ribbens : > On Sat, Apr 09, 2016 at 02:43:19PM +0200, Victor Stinner wrote: >>Please don't loose time trying yet another sandbox inside CPython. It's >>just a waste of time. It's broken by design. >> >>Please

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

2016-04-11 Thread Victor Stinner
2016-04-11 21:00 GMT+02:00 Brett Cannon : > I'm -0 on allowing __fspath__ to return bytes, but we can see what others > think. With the PEP 383, a bytes filename can be stored as str using the surrogateescape error handler. So DirEntry can convert a bytes path to str using os.fsdecode(). A "byte

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

2016-04-11 Thread Victor Stinner
Le 11 avr. 2016 11:11 PM, "Ethan Furman" a écrit : > So my concern in such a case is what happens if we pass this SE string somewhere else: a UTF-8 file, or over a socket, or into a database? Does this have issues that we wouldn't face if we just used bytes? "SE string" are returned by os.listdir

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

2016-04-12 Thread Victor Stinner
2016-04-12 13:10 GMT+02:00 Jon Ribbens : > No, it's a matter of reducing the whitelist. I must admit that > I don't understand in what way this is not already clear. Look: > > >>> len(unsafe._SAFE_MODULES) > 23 You don't understand that even if the visible "Python scope", "Python namespace", o

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

2016-04-12 Thread Victor Stinner
2016-04-08 16:18 GMT+02:00 Jon Ribbens : > I've made another attempt at Python sandboxing, which does something > which I've not seen tried before - using the 'ast' module to do static > analysis of the untrusted code before it's executed, to prevent most > of the sneaky tricks that have been used

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

2016-04-12 Thread Victor Stinner
2016-04-12 13:38 GMT+02:00 Maciej Fijalkowski : > (...) you end up with either a > completely unusable python (the python that can't run anything is > trivially secure) Yeah, that's the obvious question: what's the purpose of such very limited Python subset, for example something limited to int wi

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

2016-04-12 Thread Victor Stinner
2016-04-12 14:18 GMT+02:00 Jon Ribbens : > The question is: with a minimal (or empty) set of builtins, and a > restriction on ast.Name and ast.Attribute nodes, can exec/eval be > made 'safe' so they cannot execute code outside the sandbox. According to multiple exploits listed in this thread, no,

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

2016-04-12 Thread Victor Stinner
2016-04-12 14:16 GMT+02:00 Victor Stinner : > I read your code and the code of CPython. I found many issues. > (...) > The exploit is based on two things: > > * update_wrapper() is used to get the secret attribute using the real > getattr() function > * update_wrapper() + A.__

[Python-Dev] Most 3.x buildbots are green again, please don't break them and watch them!

2016-04-13 Thread Victor Stinner
Hi, Last months, most 3.x buildbots failed randomly. Some of them were always failing. I spent some time to fix almost all Windows and Linux buildbots. There were a lot of different issues. So please try to not break buildbots again and remind to watch them sometimes: http://buildbot.python.o

[Python-Dev] Wordcode: new regular bytecode using 16-bit units

2016-04-13 Thread Victor Stinner
Hi, In the middle of recent discussions about Python performance, it was discussed to change the Python bytecode. Serhiy proposed to reuse MicroPython short bytecode to reduce the disk space and reduce the memory footprint. Demur Rumed proposes a different change to use a regular bytecode using 1

Re: [Python-Dev] Wordcode: new regular bytecode using 16-bit units

2016-04-13 Thread Victor Stinner
2016-04-13 23:02 GMT+02:00 Eric Fahlgren : > Percentage of 1-byte args= 96.80% Yeah, I expected such high ratio. Good news that you confirm it. > Non-argument ops =53,719 > One-byte args= 368,787 > Multi-byte args =12,191 Again, only a very

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

2016-04-13 Thread Victor Stinner
Le mercredi 13 avril 2016, Brett Cannon a écrit : > > All of this is demonstrated in > https://gist.github.com/brettcannon/b3719f54715787d54a206bc011869aa1 by > the various possibilities. In the end it's not a corner case because the > definition of __fspath__ will be such that there's no ambiguit

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

2016-04-13 Thread Victor Stinner
Oops sorry, I forgot to add that I have no strong opinion on the type (I only have a minor preference for str only). Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.p

Re: [Python-Dev] Wordcode: new regular bytecode using 16-bit units

2016-04-13 Thread Victor Stinner
Le mercredi 13 avril 2016, Ryan Gonzalez a écrit : > What is the value of HAS_ARG going to be now? > I asked Demur to keep HAS_ARG(). Not really for backward compatibility, but for the dis module: to keep a nice assembler. There are also debug traces in ceval.c which use it. For ceval.c, we mig

Re: [Python-Dev] pathlib - current status of discussions

2016-04-13 Thread Victor Stinner
Oh, since others voted, I will also vote and explain my vote. I like choice 1, str only, because it's very well defined. In Python 3, Unicode is simply the native type for text. It's accepted by almost all functions. In other emails, I also explained that Unicode is fine to store undecodable filen

Re: [Python-Dev] Wordcode: new regular bytecode using 16-bit units

2016-04-13 Thread Victor Stinner
2016-04-14 0:11 GMT+02:00 Ryan Gonzalez : > So code that depends on iterating through bytecode via HAS_ARG is going to > break... Sure. This change is backward incompatible for applications parsing bytecode in C or Python. That's why the patch also has to update the dis module. I don't see how yo

Re: [Python-Dev] Most 3.x buildbots are green again, please don't break them and watch them!

2016-04-14 Thread Victor Stinner
Le 14 avr. 2016 11:16 AM, "Serhiy Storchaka" a écrit : > A desirable but nonexistent feature is to write emails to authors of commits that broke buildbots. How hard to implement this? Yeah I also had this idea since many years but buildbots were quite unstable. Maybe we should be more strict to c

Re: [Python-Dev] Bytes path

2016-04-14 Thread Victor Stinner
IMHO it's more a side effect of the implementation than a deliberate choice. For new code which really want to support bytes paths, I suggest to only accept bytes and bytes subclasses. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail

Re: [Python-Dev] Not receiving bug tracker emails

2016-04-14 Thread Victor Stinner
Le 14 avr. 2016 10:53 AM, "Serhiy Storchaka" a écrit : > Most bug tracker emails still went in the Spam folder. I have a filter for Roundap emails, but there is no any mark that I can use for filtering Rietveld emails. I'm using the base URL of Rietveld and match it in the mail body. Gmail filter

Re: [Python-Dev] Not receiving bug tracker emails

2016-04-14 Thread Victor Stinner
2016-04-14 13:01 GMT+02:00 Serhiy Storchaka : > But this filter is not quite robust, for example it will cause this mail to > be moved to the folder for Rietveld reviews. Right, it's just a workaround since I'm unable to fix the root cause (emails marked as spam which looks like a configuration is

Re: [Python-Dev] Wordcode: new regular bytecode using 16-bit units

2016-04-14 Thread Victor Stinner
Le jeudi 14 avril 2016, Nick Coghlan a écrit : > > > IHMO it's not a big deal to update these projects for the future > > Python 3.6. I can even help them to support the new bytecode format. > > We've also had previous discussions on adding a "minimum viable > bytecode editing" API to the standard

Re: [Python-Dev] pathlib - current status of discussions

2016-04-14 Thread Victor Stinner
2016-04-13 19:10 GMT+02:00 Brett Cannon : > https://gist.github.com/brettcannon/b3719f54715787d54a206bc011869aa1 has the > four potential approaches implemented (although it doesn't follow the > "separate functions" approach some are proposing and instead goes with the > allow_bytes approach I orig

Re: [Python-Dev] pathlib - current status of discussions

2016-04-14 Thread Victor Stinner
2016-04-14 15:40 GMT+02:00 Nick Coghlan : >> I consider that the final goal of the whole discussion is to support >> something like: >> >> path = os.path.join(pathlib_path, "str_path", direntry) > > That's not a *new* problem though, it already exists if you pass in a > mix of bytes and str: >

<    1   2   3   4   5   6   7   8   9   10   >