[issue13836] Define key failed
New submission from olivier : Hi, I tried to define new key in python idle and then python 2.5.1 failed to launch. What I did : I defined a new key, applied and changed my mind, removed my key set named 'ole'. I launched cmd C:\Python25\Lib\idlelib>..\..\python idle.py got dozens of messages : Warning: configHandler.py - IdleConf.GetOption - problem retrieving configration option 'redo' from section 'ole'. returning default value: '' Finally, this opened python idle so I went again to configure idle/keys and saw that my key set was still there. I selected again "Delete custom Key set" and now it's ok. -- messages: 151771 nosy: olivier57 priority: normal severity: normal status: open title: Define key failed versions: Python 2.7 ___ Python tracker <http://bugs.python.org/issue13836> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9098] MSYS build fails with `S_IXGRP' undeclared
Olivier Refalo added the comment: hum, your patch actually works on MSYS ! ok.. so I am pretty much having the very some issue. Could not find platform dependent libraries Consider setting $PYTHONHOME to [:] Fatal Python error: Py_Initialize: unable to load the file system codec LookupError: no codec search functions registered: can't find encoding -- nosy: +orefalo versions: +Python 3.2 -Python 2.6 ___ Python tracker <http://bugs.python.org/issue9098> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2504] Add gettext.pgettext() and variants support
Olivier Berten added the comment: Pleeese ;-) -- nosy: +olivier-berten ___ Python tracker <http://bugs.python.org/issue2504> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1698] urlparse and usernames containing @
New submission from Olivier Croquette:
Some servers allow the @ character is usernames. It gives URLs like:
ftp://[EMAIL PROTECTED]@host/dir
[EMAIL PROTECTED] could for example by an email address.
I am not sure if this is RFC compliant. What's sure is that is makes
trouble with urlparse:
>>> from urlparse import urlparse
>>> p = urlparse("ftp://[EMAIL PROTECTED]@host2/dir")
>>> print p.username
user
>>> print p.hostname
[EMAIL PROTECTED]
By using rsplit instead of split in lib/python2.5/urlparse.py, the
problem can be solved.
--
components: Library (Lib)
messages: 58990
nosy: ocroquette
severity: normal
status: open
title: urlparse and usernames containing @
type: behavior
versions: Python 2.5
__
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue1698>
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1698] urlparse and usernames containing @
Olivier Croquette added the comment: Hi! Thanks for the reply! The problem right now is that urlparse parses silently an URL which is not compliant, but does the wrong thing with it (since usernames can contain @, and hostname can not, it's a more logical thing to parse from the right using rsplit instead of split). I see two possibilities to address that: 1. return a parse error if the URL contains two raw @ This way users and app developers will notice the problem rapidly 2. still accept this malformed URLs, but do what the user expects Both solutions seem to me better than the current behaviour, so I would say a change is necessary anyways. PS: will urlparse transform an encoded "@" in the username when .username is called, or does the application have to decode explicitely? Olivier __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1698> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1698] urlparse and usernames containing @
Olivier Croquette added the comment: See also the related bug on duplicity: http://savannah.nongnu.org/bugs/?21475 __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1698> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1698] urlparse and usernames containing @
Olivier Croquette added the comment: And about the decoding, sorry, it's clear from your snippets that urlparse doesn't do it: >>> print q.username user%40xyz Maybe it should do it, I am not sure. What do you think? It would save work for the module user. __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1698> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4548] OptionParser : Weird comportement in args processing
New submission from Olivier Hervieu <[EMAIL PROTECTED]>:
Hi guys.. i found something strange on the behavior of OptionParser
If I have this sample code :
import sys
from optparse import OptionParser
if __name__ == '__main__':
parser = OptionParser()
parser.add_option("-p", "--p", help="The P of python", default=None)
parser.add_option("-y", "--y", help="The Y of python", default=None)
(options,args) = parser.parse_args(sys.argv)
print options
and i execute :
myFile.py -p -y
the options.p will be -y and options.y will be None
worst.. if my user say :
myFile.py -p -y 'thon'
options.p will be -y and options.y will be None...
In all case I think that, if i do
myFile.py -p -y
options -p and options -y must be None
and if i want to put -y in options.p i say something like
myFile.py -p "-y" so i can do after myFile.py -p "-y" -y "thon".
--
components: Extension Modules
messages: 77028
nosy: ohervieu
severity: normal
status: open
title: OptionParser : Weird comportement in args processing
type: behavior
versions: Python 2.5, Python 2.5.3
___
Python tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue4548>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45232] ascii codec is used by default when LANG is not set
New submission from Olivier Delhomme :
$ python3 --version
Python 3.6.4
Setting LANG to en_US.UTF8 works like a charm
$ export LANG=en_US.UTF8
$ python3
Python 3.6.4 (default, Jan 11 2018, 16:45:55)
[GCC 4.8.5] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> machaine='help me if you can'
>>>
>>>
>>> print('{}'.format(machaine))
>>>
>>>
help me if you can
Unsetting LANG shell variable fails the program:
$ unset LANG
$ python3
Python 3.6.4 (default, Jan 11 2018, 16:45:55)
[GCC 4.8.5] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> machaine='help me if you can'
File "", line 0
^
SyntaxError: 'ascii' codec can't decode byte 0xc3 in position 10: ordinal not
in range(128)
Setting LANG inside the program does not change this behavior:
$ unset LANG
$ python3
Python 3.6.4 (default, Jan 11 2018, 16:45:55)
[GCC 4.8.5] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ['LANG'] = 'en_US.UTF8'
>>> machaine='help me if you can'
File "", line 0
^
SyntaxError: 'ascii' codec can't decode byte 0xc3 in position 10: ordinal not
in range(128)
Is this an expected behavior ? How can I force an utf8 codec ?
--
components: Interpreter Core
messages: 402046
nosy: od-cea
priority: normal
severity: normal
status: open
title: ascii codec is used by default when LANG is not set
type: behavior
versions: Python 3.6
___
Python tracker
<https://bugs.python.org/issue45232>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45232] ascii codec is used by default when LANG is not set
Olivier Delhomme added the comment: Hi Marc-Andre, Please note that setting PYTHONUTF8 with "export PYTHONUTF8=1": * Is external to the program and user dependent * It does not seems to work on my use case: $ unset LANG $ export PYTHONUTF8=1 $ python3 Python 3.6.4 (default, Jan 11 2018, 16:45:55) [GCC 4.8.5] on linux Type "help", "copyright", "credits" or "license" for more information. >>> machaine='help me if you can' File "", line 0 ^ SyntaxError: 'ascii' codec can't decode byte 0xc3 in position 10: ordinal not in range(128) Regards, Olivier. -- ___ Python tracker <https://bugs.python.org/issue45232> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45232] ascii codec is used by default when LANG is not set
Olivier Delhomme added the comment: >> Hi Marc-Andre, >> >> Please note that setting PYTHONUTF8 with "export PYTHONUTF8=1": >> >> * Is external to the program and user dependent >> * It does not seems to work on my use case: >> >>$ unset LANG >>$ export PYTHONUTF8=1 >>$ python3 >>Python 3.6.4 (default, Jan 11 2018, 16:45:55) >>[GCC 4.8.5] on linux >>Type "help", "copyright", "credits" or "license" for more information. >>>>> machaine='help me if you can' >> File "", line 0 >> >> ^ >> SyntaxError: 'ascii' codec can't decode byte 0xc3 in position 10: >> ordinal not in range(128) > > UTF-8 mode is only supported in Python 3.7 and later: > > https://docs.python.org/3/whatsnew/3.7.html#whatsnew37-pep540 Oh. Thanks. $ unset LANG $ export PYTHONUTF8=1 $ python3 Python 3.7.5 (default, Dec 24 2019, 08:52:13) [GCC 4.8.5] on linux Type "help", "copyright", "credits" or "license" for more information. >>> machaine='help me if you can' >>> From the code point of view: $ unset LANG $ unset PYTHONUTF8 $ python3 Python 3.7.5 (default, Dec 24 2019, 08:52:13) [GCC 4.8.5] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.environ['PYTHONUTF8'] = '1' >>> machaine='help me if you can' >>> Even better: $ unset LANG $ unset PYTHONUTF8 $ python3 Python 3.7.5 (default, Dec 24 2019, 08:52:13) [GCC 4.8.5] on linux Type "help", "copyright", "credits" or "license" for more information. >>> machaine='help me if you can' >>> Works as expected. Thank you very much. You can close this bug report. Regards, Olivier. -- ___ Python tracker <https://bugs.python.org/issue45232> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35805] email package folds msg-id identifiers using RFC2047 encoded words where it must not
Olivier Dony added the comment: With regard to msg349895, is there any chance this fix could be considered for backport? I imagine you could view it as a new feature, but it seems to be the only official fix we have for the fact that Python 3 generates invalid SMTP messages. And that's not a minor problem because many popular MTAs (GMail, Outlook, etc.) will rewrite non-RFC-conformant Message IDs, causing the original ID to be lost and missing in subsequent replies. This breaks an important mechanism to support email threads. To this day, several Linux distributions still ship 3.6 or 3.7, even in their latest LTS, and users and vendors are stuck with supporting those for a while. Thanks! -- nosy: +odo2 ___ Python tracker <https://bugs.python.org/issue35805> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35805] email package folds msg-id identifiers using RFC2047 encoded words where it must not
Olivier Dony added the comment: Further, under Python 3.8 the issue is not fully solved, as other identification headers are still being folded in a non-RFC-conformant manner (see OP for RFC references). This was indicated on the original PR by the author: https://github.com/python/cpython/pull/13397#issuecomment-493618544 It is less severe of a problem than for Message-ID, but still means that MTA/MUA may fail to recognize the threading structure because identifiers are lost. Is it better to open a new issue for this? # Example on 3.8.2: the `In-Reply-To` header is RFC2047-folded. Python 3.8.2 (default, Jul 16 2020, 14:00:26) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import email.message >>> import email.policy >>> msg = email.message.EmailMessage(policy=email.policy.SMTP) >>> msg['Message-Id'] = >>> '<929227342217024.1596730490.324691772460938-example-30661-some.refere...@test-123.example.com>' >>> msg['In-Reply-To'] = >>> '<92922734221723.1596730568.324691772460444-another-30661-parent.refere...@test-123.example.com>' >>> print(msg.as_string()) Message-Id: <929227342217024.1596730490.324691772460938-example-30661-some.refere...@test-123.example.com> In-Reply-To: =?utf-8?q?=3C92922734221723=2E1596730568=2E324691772460444-anot?= =?utf-8?q?her-30661-parent=2Ereference=40test-123=2Eexample=2Ecom=3E?= -- ___ Python tracker <https://bugs.python.org/issue35805> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35805] email package folds msg-id identifiers using RFC2047 encoded words where it must not
Olivier Dony added the comment: Somehow the message identifiers in the code sample got messed up in previous comment, here's the actual code, for what it's worth ;-) https://gist.github.com/odony/0323eab303dad2077c1277076ecc3733 -- ___ Python tracker <https://bugs.python.org/issue35805> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37351] Drop libpython38.a from Windows release
Olivier Croquette added the comment: I don't know what version of gendef is meant, but the one from MSYS2 / MinGW64 doesn't output the result on stdout, but rather writes the file "python38.def" itself. So the commands are the following: cd libs gendef ..\python38.dll dlltool --dllname python38.dll --def python38.def --output-lib libpython38.a -- nosy: +ocroquette ___ Python tracker <https://bugs.python.org/issue37351> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35900] Add pickler hook for the user to customize the serialization of user defined functions and types.
Olivier Grisel added the comment: Adding such a hook would make it possible to reimplement cloudpickle.CloudPickler by deriving from the fast _pickle.Pickler class (instead of the slow pickle._Pickler as done currently). This would mean rewriting most of the CloudPickler method to only rely on a save_reduce-style design instead of directly calling pickle._Pickler.write and pickle._Pickler.save. This is tedious but doable. There is however a blocker with the current way closures are set: when we pickle a dynamically defined function (e.g. lambda, nested function or function in __main__), we currently use a direct call to memoize (https://github.com/cloudpipe/cloudpickle/blob/v0.7.0/cloudpickle/cloudpickle.py#L594) so as to be able to refer to the function itself in its own closure without causing an infinite loop in CloudPickler.dump. This also makes possible to pickle mutually recursive functions. The easiest way to avoid having to call memoize explicitly would be to be able to pass the full __closure__ attribute in the state dict of the reduce call. Indeed the save_reduce function calls memoize automatically after saving the reconstructor and its args but prior to saving the state: https://github.com/python/cpython/blob/v3.7.2/Modules/_pickle.c#L3903-L3931 It would therefore be possible to pass a (state, slotstate) tuple with the closure in slotstate that so it could be reconstructed at unpickling time with a setattr: https://github.com/python/cpython/blob/v3.7.2/Modules/_pickle.c#L6258-L6272 However, it is currently not possible to setattr __closure__ at the moment. We can only set individual closure cell contents (which is not compatible with the setattr state trick described above). To summarize, we need to implement the setter function for the __closure__ attribute of functions and methods to make it natural to reimplement the CloudPickler by inheriting from _pickle.Pickler using the hook described in this issue. -- nosy: +Olivier.Grisel ___ Python tracker <https://bugs.python.org/issue35900> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29738] Fix memory leak in SSLSocket.getpeercert()
New submission from Olivier Vielpeau: The code snippet in #25569 reproduces the memory leak with Python 3.6.0 and 2.7.13. The current memory leak is a regression that was introduced in #26470. Going to attach a PR on github that fixes the issue shortly. -- assignee: christian.heimes components: SSL messages: 289130 nosy: christian.heimes, olivielpeau priority: normal severity: normal status: open title: Fix memory leak in SSLSocket.getpeercert() type: resource usage versions: Python 2.7, Python 3.6, Python 3.7 ___ Python tracker <http://bugs.python.org/issue29738> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29738] Fix memory leak in SSLSocket.getpeercert()
Changes by Olivier Vielpeau : -- pull_requests: +433 ___ Python tracker <http://bugs.python.org/issue29738> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29738] Fix memory leak in SSLSocket.getpeercert()
Olivier Vielpeau added the comment: I've attached the PR on Github and signed the CLA, is there anything else needed from me to get this reviewed? Thanks! -- ___ Python tracker <http://bugs.python.org/issue29738> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29738] Fix memory leak in SSLSocket.getpeercert()
Olivier Vielpeau added the comment: Thnaks for the reviews and the merge! :) -- ___ Python tracker <http://bugs.python.org/issue29738> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36867] Make semaphore_tracker track other system resources
Olivier Grisel added the comment:
As Victor said, the `time.sleep(1.0)` might lead to Heisen failures. I am not
sure how to write proper strong synchronization in this case but we could
instead go for something intermediate such as the following pattern:
...
p.terminate()
p.wait()
for i in range(60):
try:
shared_memory.SharedMemory(name, create=False)
except FileNotFoundError:
# the OS successfully collected the segment as expected
break
time.sleep(1.0) # wait for the OS to collect the segment
else:
raise AssertionError(f"Failed to collect shared_memory segment
{name}")
What do you think?
--
nosy: +Olivier.Grisel
___
Python tracker
<https://bugs.python.org/issue36867>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in the standard library should be sanitized on fork
Olivier Chédru added the comment: FWIW, I encountered the same kind of issue when using the mkstemp() function: under the hood, it calls gettempdir() and this one is protected by a lock too. Current thread 0x7ff10231f700 (most recent call first): File "/usr/lib/python3.5/tempfile.py", line 432 in gettempdir File "/usr/lib/python3.5/tempfile.py", line 269 in _sanitize_params File "/usr/lib/python3.5/tempfile.py", line 474 in mkstemp -- nosy: +ochedru ___ Python tracker <https://bugs.python.org/issue6721> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
New submission from Olivier Grisel : I noticed that both pickle.Pickler (C version) and pickle._Pickler (Python version) make unnecessary memory copies when dumping large str, bytes and bytearray objects. This is caused by unnecessary concatenation of the opcode and size header with the large bytes payload prior to calling self.write. For protocol 4, an additional copy is caused by the framing mechanism. I will submit a pull request to fix the issue for the Python version. I am not sure how to test this properly. The BigmemPickleTests seems to be skipped on my 16 GB laptop. -- components: Library (Lib) messages: 305975 nosy: Olivier.Grisel, pitrou priority: normal severity: normal status: open title: pickle.dump allocates unnecessary temporary bytes / str type: resource usage versions: Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel added the comment: I wrote a script to monitor the memory when dumping 2GB of data with python master (C pickler and Python pickler): ``` (py37) ogrisel@ici:~/code/cpython$ python ~/tmp/large_pickle_dump.py Allocating source data... => peak memory usage: 2.014 GB Dumping to disk... done in 5.141s => peak memory usage: 4.014 GB (py37) ogrisel@ici:~/code/cpython$ python ~/tmp/large_pickle_dump.py --use-pypickle Allocating source data... => peak memory usage: 2.014 GB Dumping to disk... done in 5.046s => peak memory usage: 5.955 GB ``` This is using protocol 4. Note that the C pickler is only making 1 useless memory copy instead of 2 for the Python pickler (one for the concatenation and the other because of the framing mechanism of protocol 4). Here the output with the Python pickler fixed in python/cpython#4353: ``` (py37) ogrisel@ici:~/code/cpython$ python ~/tmp/large_pickle_dump.py --use-pypickle Allocating source data... => peak memory usage: 2.014 GB Dumping to disk... done in 6.138s => peak memory usage: 2.014 GB ``` Basically the 2 spurious memory copies of the Python pickler with protocol 4 are gone. Here is the script: https://gist.github.com/ogrisel/0e7b3282c84ae4a581f3b9ec1d84b45a -- ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel added the comment: Note that the time difference is not significant. I rerun the last command I got: ``` (py37) ogrisel@ici:~/code/cpython$ python ~/tmp/large_pickle_dump.py --use-pypickle Allocating source data... => peak memory usage: 2.014 GB Dumping to disk... done in 4.187s => peak memory usage: 2.014 GB ``` -- ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel added the comment: More benchmarks with the unix time command: ``` (py37) ogrisel@ici:~/code/cpython$ git checkout master Switched to branch 'master' Your branch is up-to-date with 'origin/master'. (py37) ogrisel@ici:~/code/cpython$ time python ~/tmp/large_pickle_dump.py --use-pypickle Allocating source data... => peak memory usage: 2.014 GB Dumping to disk... done in 10.677s => peak memory usage: 5.936 GB real0m11.068s user0m0.940s sys 0m5.204s (py37) ogrisel@ici:~/code/cpython$ time python ~/tmp/large_pickle_dump.py --use-pypickle Allocating source data... => peak memory usage: 2.014 GB Dumping to disk... done in 5.089s => peak memory usage: 5.978 GB real0m5.367s user0m0.840s sys 0m4.660s (py37) ogrisel@ici:~/code/cpython$ git checkout issue-31993-pypickle-dump-mem-optim Switched to branch 'issue-31993-pypickle-dump-mem-optim' (py37) ogrisel@ici:~/code/cpython$ time python ~/tmp/large_pickle_dump.py --use-pypickle Allocating source data... => peak memory usage: 2.014 GB Dumping to disk... done in 6.974s => peak memory usage: 2.014 GB real0m7.300s user0m0.368s sys 0m4.640s (py37) ogrisel@ici:~/code/cpython$ time python ~/tmp/large_pickle_dump.py --use-pypickle Allocating source data... => peak memory usage: 2.014 GB Dumping to disk... done in 10.873s => peak memory usage: 2.014 GB real0m11.178s user0m0.324s sys 0m5.100s (py37) ogrisel@ici:~/code/cpython$ time python ~/tmp/large_pickle_dump.py --use-pypickle Allocating source data... => peak memory usage: 2.014 GB Dumping to disk... done in 4.233s => peak memory usage: 2.014 GB real0m4.574s user0m0.396s sys 0m4.368s ``` User time is always better in the PR than on master but is also much slower than system time (disk access) in any case. System time is much less deterministic. -- ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel added the comment: In my last comment, I also reported the user times (not spend in OS level disk access stuff): the code of the PR is on the order of 300-400ms while master is around 800ms or more. -- ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel added the comment: I have pushed a new version of the code that now has a 10% overhead for small bytes (instead of 40% previously). It could be possible to optimize further but I think that would render the code much less readable so I would be tempted to keep it this way. Please let me know what you think. -- ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel added the comment: Actually, I think this can still be improved while keeping it readable. Let me try again :) -- ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel added the comment: Alright, the last version has now ~4% overhead for small bytes. -- ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel added the comment: BTW, I am looking at the C implementation at the moment. I think I can do it. -- ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel added the comment: I have tried to implement the direct write bypass for the C version of the pickler but I get a segfault in a Py_INCREF on obj during the call to memo_put(self, obj) after the call to _Pickler_write_large_bytes. Here is the diff of my current version of the patch: https://github.com/ogrisel/cpython/commit/4e093ad6993616a9f16e863b72bf2d2e37bc27b4 I am new to the Python C-API so I would appreciate some help on this one. -- ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel added the comment: Alright, I found the source of my refcounting bug. I updated the PR to include the C version of the dump for PyBytes. I ran Serhiy's microbenchmarks on the C version and I could not detect any overhead on small bytes objects while I get a ~20x speedup (and no-memory copy) on large bytes objects as expected. I would like to update the `write_utf8` function but I would need to find a way to wrap `const char* data` as a PyBytes instance without making a memory copy to be able to pass it to my `_Pickle_write_large_bytes`. I browsed the C-API documentation but I could not understand how to do that. Also I would appreciate any feedback on the code style or things that could be improved in my PR. -- ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel added the comment: Thanks Antoine, I updated my code to what you suggested. -- ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel added the comment: > While we are here, wouldn't be worth to flush the buffer in the C > implementation to the disk always after committing a frame? This will save a > memory when dump a lot of small objects. I think it's a good idea. The C pickler would behave more like the Python pickler. I think framing was intended this way initially. Antoine what do you think? -- ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel added the comment: Flushing the buffer at each frame commit will cause a medium-sized write every 64kB on average (instead of one big write at the end). So that might actually cause a performance regression for some users if the individual file-object writes induce significant overhead. In practice though, latency inducing file objects like filesystem-backed ones are likely to derive from the [BufferedWriter](https://docs.python.org/3/library/io.html#io.BufferedWriter) base class and the only latency we should really care about it the one induced by the write call overhead itself in which case the 64kB frame / buffer size should be enough. -- ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel added the comment: Shall we close this issue now that the PR has been merged to master? -- ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel added the comment: Thanks for the very helpful feedback and guidance during the review. -- ___ Python tracker <https://bugs.python.org/issue31993> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18352] collections.Counter with added attributes are not deepcopied properly.
New submission from Olivier Gagnon: The following code shows that the Counter is not deepcopied properly. The same code with an user defined class or a dict is copied with the "b" attribute. import collections import copy count = collections.Counter() count.b = 3 print(count.b) # prints 3 count_copy = copy.deepcopy(count) print(count_copy.b) # raise AttributeError: 'Counter' object has no attribute 'b' -- components: Library (Lib) messages: 192239 nosy: Olivier.Gagnon priority: normal severity: normal status: open title: collections.Counter with added attributes are not deepcopied properly. type: behavior versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue18352> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18033] Example for Profile Module shows incorrect method
Olivier Hervieu added the comment: Here is a fixed version for python2.7. Using StringIO instead of io module fixes the problem pointed by Ezio. The print_stats method dumps the stats either on sys.stdout if `Stats` class is declared without a stream specification or in the given stream parameter (the name print_stats may be a bit misleading). -- nosy: +ohe Added file: http://bugs.python.org/file30760/issue18033_py2.patch ___ Python tracker <http://bugs.python.org/issue18033> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18352] collections.Counter with added attributes are not deepcopied properly.
Olivier Gagnon added the comment: The dictionary and the set do not give the freedom to add dynamic attributes to them. I agree that the Counter should have the same behaviour. However, this will raise the same bug when we inherit from a Counter object. >>> class mylist(list): pass ... >>> l = mylist() >>> l.foo = "bar" >>> c = copy.deepcopy(l) >>> print(c.foo) # prints bar >>> class myCounter(Counter): pass ... >>> original = myCounter() >>> original.foo = "bar" >>> c = copy.deepcopy(original) >>> c.foo Traceback (most recent call last): File "", line 1, in AttributeError: 'myCounter' object has no attribute 'foo' The reduction function should still copy every dynamic attribute of the object. -- ___ Python tracker <http://bugs.python.org/issue18352> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18352] collections.Counter with added attributes are not deepcopied properly.
Olivier Gagnon added the comment:
I can understand that the current behaviour can be correct in regard with the
added attributes of the object. However, should I open a new issue for the
following inheritance behaviour which the reduce function affects also.
class myCounter(Counter):
def __init__(self, bar, *args):
self.foo = bar
super().__init__(*args)
class myDict(dict):
def __init__(self, bar, *args):
self.foo = bar
super().__init__(*args)
c = myCounter("bar")
l = myDict("bar")
print(c.foo) # prints bar
print(l.foo) # prints bar
cc = copy.copy(c)
ll = copy.copy(l)
print(cc.foo) # prints {}
print(ll.foo) # prints bar
--
___
Python tracker
<http://bugs.python.org/issue18352>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18352] collections.Counter with added attributes are not deepcopied properly.
Olivier Gagnon added the comment: Yes I do have code that break because of this behaviour. I'm doing evolutionary algorithm using a framework called DEAP. This framework creates a type called individual at the runtime by subclassing a container and adding it a fitness attribute. Those individual are copied as not to modify every indivual when we work on a single one. AFAIK the only container that can't be used right now is the counter because the fitness is not copied. I'm sure I can come up with a hack to have this behaviour, but it does clash with other standard container type and there is no mention anywhere that the Counter should be different than every other container type in the python standard library. -- ___ Python tracker <http://bugs.python.org/issue18352> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17560] problem using multiprocessing with really big objects?
Olivier Grisel added the comment: I have implemented a custom subclass of the multiprocessing Pool to be able plug custom pickling strategy for this specific use case in joblib: https://github.com/joblib/joblib/blob/master/joblib/pool.py#L327 In particular it can: - detect mmap-backed numpy - transform large memory backed numpy arrays into numpy.memmap instances prior to pickling using the /dev/shm partition when available or TMPDIR otherwise. Here is some doc: https://github.com/joblib/joblib/blob/master/doc/parallel_numpy.rst I could submit the part that makes it possible to customize the picklers of multiprocessing.pool.Pool instance to the standard library if people are interested. The numpy specific stuff would stay in third party projects such as joblib but at least that would make it easier for people to plug their own optimizations without having to override half of the multiprocessing class hierarchy. -- nosy: +Olivier.Grisel ___ Python tracker <http://bugs.python.org/issue17560> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17560] problem using multiprocessing with really big objects?
Olivier Grisel added the comment: I forgot to end a sentence in my last comment: - detect mmap-backed numpy should read: - detect mmap-backed numpy arrays and pickle only the filename and other buffer metadata to reconstruct a mmap-backed array in the worker processes instead of copying the data around. -- ___ Python tracker <http://bugs.python.org/issue17560> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17560] problem using multiprocessing with really big objects?
Olivier Grisel added the comment: > In 3.3 you can do > > from multiprocessing.forking import ForkingPickler > ForkingPickler.register(MyType, reduce_MyType) > > Is this sufficient for you needs? This is private (and its definition has > moved in 3.4) but it could be made public. Indeed I forgot that the multiprocessing pickler was made already made pluggable in Python 3.3. I needed backward compat for python 2.6 in joblib hence I had to rewrite a bunch of the class hierarchy. -- ___ Python tracker <http://bugs.python.org/issue17560> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Olivier Grisel added the comment: Related question: is there any good reason that would prevent to pass a custom `start_method` kwarg to the `Pool` constructor to make it use an alternative `Popen` instance (that is an instance different from the `multiprocessing._Popen` singleton)? This would allow libraries such as joblib to keep minimal side effect by try to impact the default multiprocessing runtime as low as possible. -- nosy: +Olivier.Grisel ___ Python tracker <http://bugs.python.org/issue18999> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Olivier Grisel added the comment: > Maybe it would be better to have separate contexts for each start method. > That way joblib could use the forkserver context without interfering with the > rest of the user's program. Yes in general it would be great if libraries could customize the multiprocessing default options without impacting any of the module singletons. That also include the ForkingPickler registry for custom: now it's a class attribute. It would be great to be able to scope custom reducers registration to a given multiprocessing.Pool or multiprocessing.Process instance. Now how to implement that kind of isolation: it could either be done by adding new constructor parameters or new public methods to the Process and Pool classes to be able to customize their behavior while sticking to the OOP paradigm if possible or by using a context manager as you suggested. I am not sure which option is best. Prototyping both is probably the best way to feel the tradeoffs. -- ___ Python tracker <http://bugs.python.org/issue18999> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Olivier Grisel added the comment: The process pool executor [1] from the concurrent futures API would be suitable to explicitly start and stop the helper process for the `forkserver` mode. [1] http://docs.python.org/3.4/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor The point would be to have as few state as possible encoded in the multiprocessing module (and its singletons) and move that state information to be directly managed by multiprocessing Process and Pool class instances so that libraries could customize the behavior (start_method, executable, ForkingPIckler reducers registry and so on) without mutating the state of the multiprocessing module singletons. -- ___ Python tracker <http://bugs.python.org/issue18999> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
python-bugs-list@python.org
New submission from olivier-mattelaer : Hi Everyone, I have found a strange behavior of the import command for the routine readline: The commands (put in the file test.py) is simply: import readline print readline.__doc__ If I run this programs "normally" (i.e. python2.x test.py) everything runs normally: [tmp]$ python2.7 tmp.py Importing this module enables command line editing using GNU readline. But if I launched it in the following way: python2.X test.py & It's stops before finishing the import command (so no output even if I print the output into a file). This doesn't raise any error just stop the program. I tried to add a try...except...finally... but even the finally statement is not executed. Of course, I realize that readline is not that interesting to load with that options. So I would be more than happy if you could indicate me a way to either detect that the user launch the program with '&' or a way to avoid readline to stop the program in this situations. Thanks a lot, Olivier -- assignee: ronaldoussoren components: Macintosh messages: 161450 nosy: olivier-mattelaer, ronaldoussoren priority: normal severity: normal status: open title: 'import readline' fails when launching with '&' type: crash versions: Python 2.6, Python 2.7 ___ Python tracker <http://bugs.python.org/issue14892> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1276] LookupError: unknown encoding: X-MAC-JAPANESE
Olivier Berten added the comment: Any idea why mac cjk encodings still aren't available in Python 2.7 and 3.2 ? -- components: -Build, Demos and Tools, Library (Lib), Macintosh nosy: +ezio.melotti, olivier-berten versions: +Python 2.7, Python 3.2 -Python 2.6, Python 3.0 ___ Python tracker <http://bugs.python.org/issue1276> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14892] 'import readline' hangs when launching with '&' on BSD and OS X
olivier-mattelaer added the comment: Thanks a lot Ronald. Cheers, Olivier -- status: pending -> open ___ Python tracker <http://bugs.python.org/issue14892> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Olivier Grisel added the comment: Richard Oudkerk: thanks for the clarification, that makes sense. I don't have the time either in the coming month, maybe later. -- ___ Python tracker <http://bugs.python.org/issue18999> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19851] reload problem with submodule
Olivier Grisel added the comment: I tested the patch on the current HEAD and it fixes a regression introduced between 3.3 and 3.4b1 that prevented to build scipy from source with "pip install scipy". -- nosy: +Olivier.Grisel ___ Python tracker <http://bugs.python.org/issue19851> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21905] RuntimeError in pickle.whichmodule when sys.modules if mutated
New submission from Olivier Grisel: `pickle.whichmodule` performs an iteration over `sys.modules` and tries to perform `getattr` calls on those modules. Unfortunately some modules such as those from the `six.moves` dynamic module can trigger imports when calling `getattr` on them, hence mutating the `sys.modules` dict and causing a `RuntimeError: dictionary changed size during iteration`. This would also render `pickle.whichmodule` more thread-safe and cause concurrent thread perform new module imports and `whichmodule` calls. The attach patch protect the iterator by copying the dict items into a fixed list. I could write a tests involving dynamic module definitions as done in `six.moves` but it sounds very complicated for such a trivial fix. -- components: Library (Lib) files: pickle_whichmodule.patch keywords: patch messages: 222099 nosy: Olivier.Grisel priority: normal severity: normal status: open title: RuntimeError in pickle.whichmodule when sys.modules if mutated type: crash versions: Python 3.4, Python 3.5 Added file: http://bugs.python.org/file35830/pickle_whichmodule.patch ___ Python tracker <http://bugs.python.org/issue21905> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21905] RuntimeError in pickle.whichmodule when sys.modules if mutated
Olivier Grisel added the comment: New version of the patch to add an inline comment. -- Added file: http://bugs.python.org/file35841/pickle_whichmodule_20140703.patch ___ Python tracker <http://bugs.python.org/issue21905> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21905] RuntimeError in pickle.whichmodule when sys.modules if mutated
Olivier Grisel added the comment: No problem. Thanks Antoine for the review! -- ___ Python tracker <http://bugs.python.org/issue21905> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19946] multiprocessing crash with forkserver or spawn when run from a non ".py" ending script
New submission from Olivier Grisel:
Here is a simple python program that uses the new forkserver feature introduced
in 3.4b1:
name: checkforkserver.py
"""
import multiprocessing
import os
def do(i):
print(i, os.getpid())
def test_forkserver():
mp = multiprocessing.get_context('forkserver')
mp.Pool(2).map(do, range(3))
if __name__ == "__main__":
test_forkserver()
"""
When running this using the "python check_forkserver.py" command everything
works as expected.
When running this using the nosetests launcher ("nosetests -s
check_forkserver.py"), I get:
"""
Traceback (most recent call last):
File "", line 1, in
File "/opt/Python-HEAD/lib/python3.4/multiprocessing/forkserver.py", line
141, in main
spawn.import_main_path(main_path)
File "/opt/Python-HEAD/lib/python3.4/multiprocessing/spawn.py", line 252, in
import_main_path
methods.init_module_attrs(main_module)
File "", line 1051, in init_module_attrs
AttributeError: 'NoneType' object has no attribute 'loader'
"""
Indeed, the spec variable in multiprocessing/spawn.py's import_main_path
function is None as the nosetests script is not a regular python module: in
particular is does not have a ".py" extension.
If I copy or symlink or renamed the "nosetests" script as "nosetests.py" in the
same folder, this works as expected. I am not familiar enough with the
importlib machinery to suggest a fix for this bug.
Also there is a typo in the comment: "causing a psuedo fork bomb" => "causing a
pseudo fork bomb".
Note: I am running CPython head updated today.
--
components: Library (Lib)
messages: 205810
nosy: Olivier.Grisel
priority: normal
severity: normal
status: open
title: multiprocessing crash with forkserver or spawn when run from a non
".py" ending script
versions: Python 3.4
___
Python tracker
<http://bugs.python.org/issue19946>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19946] multiprocessing crash with forkserver or spawn when run from a non ".py" ending script
Changes by Olivier Grisel : -- type: -> crash ___ Python tracker <http://bugs.python.org/issue19946> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19946] multiprocessing crash with forkserver or spawn when run from a non ".py" ending script
Olivier Grisel added the comment: > So the question is exactly what module is being passed to > importlib.find_spec() and why isn't it finding a spec/loader for that module. The module is the `nosetests` python script. module_name == 'nosetests' in this case. However, nosetests is not considered an importable module because of the missing '.py' extension in the filename. > Did this code work in Python 3.3? This code did not exist in Python 3.3. -- ___ Python tracker <http://bugs.python.org/issue19946> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19946] Have multiprocessing raise ImportError when spawning a process that can't find the "main" module
Olivier Grisel added the comment: I agree that a failure to lookup the module should raise an explicit exception. > Second, there is no way that 'nosetests' will ever succeed as an import > since, as Oliver pointed out, it doesn't end in '.py' or any other > identifiable way for a finder to know it can handle the file. So this is not > a bug and simply a side-effect of how import works. The only way around it > would be to symlink nosetests to nosetests.py or to somehow pre-emptively set > up 'nosetests' for supported importing. I don't agree that (unix) Python programs that don't end with ".py" should be modified to have multiprocessing work correctly. I think it should be the multiprocessing responsibility to transparently find out how to spawn the new process independently of the fact that the program ends in '.py' or not. Note: the fork mode works always under unix (with or without the ".py" extension). The spawn mode always work under windows as AFAIK there is no way to have Python programs that don't end in .py under windows and furthermore I think multiprocessing does execute the __main__ under windows (but I haven't tested if it's still the case in Python HEAD). -- ___ Python tracker <http://bugs.python.org/issue19946> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19946] Have multiprocessing raise ImportError when spawning a process that can't find the "main" module
Olivier Grisel added the comment:
> what is sys.modules['__main__'] and sys.modules['__main__'].__file__ if you
> run under nose?
$ cat check_stuff.py
import sys
def test_main():
print("sys.modules['__main__']=%r"
% sys.modules['__main__'])
print("sys.modules['__main__'].__file__=%r"
% sys.modules['__main__'].__file__)
if __name__ == '__main__':
test_main()
(pyhead) ogrisel@is146148:~/tmp$ python check_stuff.py
sys.modules['__main__']=
sys.modules['__main__'].__file__='check_stuff.py'
(pyhead) ogrisel@is146148:~/tmp$ nosetests -s check_stuff.py
sys.modules['__main__']=
sys.modules['__main__'].__file__='/volatile/ogrisel/envs/pyhead/bin/nosetests'
.
--
Ran 1 test in 0.001s
OK
--
___
Python tracker
<http://bugs.python.org/issue19946>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19946] Have multiprocessing raise ImportError when spawning a process that can't find the "main" module
Olivier Grisel added the comment: Note however that the problem is not specific to nose. If I rename my initial 'check_forserver.py' script to 'check_forserver', add the '#!/usr/bin/env python' header and make it 'chmod +x' I get the same crash. So the problem is related to the fact that under posix, valid Python programs can be executable scripts without the '.py' extension. -- ___ Python tracker <http://bugs.python.org/issue19946> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19946] Have multiprocessing raise ImportError when spawning a process that can't find the "main" module
Olivier Grisel added the comment: Here is a patch that uses `imp.load_source` when the first importlib name-based lookup fails. Apparently it fixes the issue on my box but I am not sure whether this is the correct way to do it. -- keywords: +patch Added file: http://bugs.python.org/file33091/issue19946.diff ___ Python tracker <http://bugs.python.org/issue19946> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19946] Have multiprocessing raise ImportError when spawning a process that can't find the "main" module
Olivier Grisel added the comment: Why has this issue been closed? Won't the spawn and forkserver mode work in Python 3.4 for Python program started by a Python script (which is probably the majority of programs written in Python under unix)? Is there any reason not to use the `imp.load_source` code I put in my patch as a temporary workaround if the cleaner runpy.run_path solution is too tricky to implement for the Python 3.4 release time frame? -- ___ Python tracker <http://bugs.python.org/issue19946> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19946] Have multiprocessing raise ImportError when spawning a process that can't find the "main" module
Olivier Grisel added the comment: > The semantics are not going to change in python 3.4 and will just stay as > they were in Python 3.3. Well the semantics do change: in Python 3.3 the spawn and forkserver modes did not exist at all. The "spawn" mode existed but only implicitly and only under Windows. So Python 3.4 is introducing a new feature for POSIX systems that will only work in the rare cases where the Python program is launched by a ".py" ending script. Would running the `imp.load_source` trick only if `sys.platform != "win32"` be a viable way to preserve the semantics of Python 3.3 under the windows while not introducing a partially broken feature in Python 3.4? -- ___ Python tracker <http://bugs.python.org/issue19946> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19946] Have multiprocessing raise ImportError when spawning a process that can't find the "main" module
Olivier Grisel added the comment:
I can wait (or monkey-patch the stuff I need as a temporary workaround in my
code). My worry is that Python 3.4 will introduce a new feature that is very
crash-prone.
Take this simple program that uses the newly introduced `get_context` function
(the same problem happens with `set_start_method`):
filename: mytool
"""
#!/usr/bin/env python
from multiprocessing import freeze_support, get_context
def compute_stuff(i):
# in real life you could use a lib that uses threads
# like cuda and that would crash with the default 'fork'
# mode under POSIX
return i ** 2
if __name__ == "__main__":
freeze_support()
ctx = get_context('spawn')
ctx.Pool(4).map(compute_stuff, range(8))
"""
If you chmod +x this file and run it with ./mytool, the user will get an
infinitely running process that keeps displaying on stderr:
"""
Traceback (most recent call last):
File "", line 1, in
File "/opt/Python-HEAD/lib/python3.4/multiprocessing/spawn.py", line 96, in
spawn_main
exitcode = _main(fd)
File "/opt/Python-HEAD/lib/python3.4/multiprocessing/spawn.py", line 105, in
_main
prepare(preparation_data)
File "/opt/Python-HEAD/lib/python3.4/multiprocessing/spawn.py", line 210, in
prepare
import_main_path(data['main_path'])
File "/opt/Python-HEAD/lib/python3.4/multiprocessing/spawn.py", line 256, in
import_main_path
raise ImportError(name=main_name)
ImportError
Traceback (most recent call last):
File "", line 1, in
File "/opt/Python-HEAD/lib/python3.4/multiprocessing/spawn.py", line 96, in
spawn_main
exitcode = _main(fd)
File "/opt/Python-HEAD/lib/python3.4/multiprocessing/spawn.py", line 105, in
_main
prepare(preparation_data)
File "/opt/Python-HEAD/lib/python3.4/multiprocessing/spawn.py", line 210, in
prepare
import_main_path(data['main_path'])
File "/opt/Python-HEAD/lib/python3.4/multiprocessing/spawn.py", line 256, in
import_main_path
raise ImportError(name=main_name)
ImportError
...
"""
until the user kills the process. Is there really nothing we can do to avoid
releasing Python 3.4 with this bug?
--
___
Python tracker
<http://bugs.python.org/issue19946>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19946] Have multiprocessing raise ImportError when spawning a process that can't find the "main" module
Olivier Grisel added the comment: For Python 3.4: Maybe rather than raising ImportError, we could issue warning to notify the users that names from the __main__ namespace could not be loaded and make the init_module_attrs return early. This way a multiprocessing program that only calls functions defined in non-main namespaces could still use the new "start method" feature introduced in Python 3.4 while not changing the Python 3.3 semantics for windows programs and not relying on any deprecated hack. -- ___ Python tracker <http://bugs.python.org/issue19946> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19946] Handle a non-importable __main__ in multiprocessing
Olivier Grisel added the comment: I applied issue19946_pep_451_multiprocessing_v2.diff and I confirm that it fixes the problem that I reported initially. -- ___ Python tracker <http://bugs.python.org/issue19946> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9062] urllib.urlopen crashes when launched from a thread
New submission from Olivier Berten : I'm writing SwatchBooker <https://launchpad.net/swatchbooker>, an app that's (among other things) reading some data from the web. When urllib.urlopen is called first from within a secondary thread, the app crashes (or freezes). If it's been called first from the main thread, there's no problem. You can reproduce that by downloading http://launchpad.net/swatchbooker/trunk/0.7/+download/SwatchBooker-0.7.noinstall.zip 1. Launch swatchbooker.pyw 2. Open from web 3. Select ICI Dulux, Pantone or RAL 4. Cancel 5. Close the program 6. Relaunch swatchbooker.pyw 7. Open from web 8. Choose any swatch from ICI Dulux, Pantone or RAL (Don't click ColorCharts.org or Digital Colour Atlas!) 9. OK 10. Crash For ColorCharts.org and Digital Colour Atlas, the list itself is loaded through urllib.urlopen from the main thread so it makes the rest go smoothly... -- components: None messages: 108458 nosy: olivier-berten priority: normal severity: normal status: open title: urllib.urlopen crashes when launched from a thread versions: Python 2.6 ___ Python tracker <http://bugs.python.org/issue9062> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17760] No i18n of IDLE's interface in french
New submission from Olivier Berger: The IDLE UI isn't internationalized, AFAICS. This doesn't help when teachning Python to non-english native speakers. While learning basic english skills is no problem for wanabe Python hackers, it isn't so for young programmers being tought informatics through Python classes. Maybe this is a FAQ... but I couldn't discriminate enough my search terms to find prior reports :-/ Thanks in advance. -- components: IDLE messages: 187086 nosy: olberger priority: normal severity: normal status: open title: No i18n of IDLE's interface in french type: enhancement versions: Python 2.7 ___ Python tracker <http://bugs.python.org/issue17760> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17776] IDLE Internationalization
Olivier Berger added the comment: Excellent. I've started playing with pygettext and msgfmt and it looks like this works, from the initial tests I've made -- nosy: +olberger ___ Python tracker <http://bugs.python.org/issue17776> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23067] Export readline forced_update_display
Olivier Matz added the comment: Hi, I'm also interrested in this feature. Indeed, exporting rl_forced_update_display() is the only way I've found to make readline.set_completion_display_matches_hook() working properly. Without forcing the redisplay, the prompt is not displayed. This seems to be the proper function to call, because that's the one called at the end of display_matches() in libreadline code, which is the default function for displaying matches. The attached file is a simple example code derived from http://pymotw.com/2/readline/ that shows that the patch is working. -- nosy: +zer0 Added file: http://bugs.python.org/file38162/test-rdline.py ___ Python tracker <http://bugs.python.org/issue23067> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23067] Export readline forced_update_display
Olivier Matz added the comment: By the way, I have my own implementation of the patch that did before checking the issue tracker. Instead of adding an argument to readline.redisplay(), it adds a new function readline.forced_update_display(). I attach the patch for reference, I don't know what is the preferred way to add this feature. -- Added file: http://bugs.python.org/file38163/readline_add_forced_update_display.patch ___ Python tracker <http://bugs.python.org/issue23067> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43112] SOABI on Linux does not distinguish between GNU libc and musl libc
Change by Olivier Le Floch : -- nosy: +olivierlefloch ___ Python tracker <https://bugs.python.org/issue43112> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26527] CGI library - Using unicode in header fields
New submission from Olivier Le Moign: According to RFC5987 (http://tools.ietf.org/html/rfc5987), it's possible to use other encoding than ASCII in header fields. Specifically in the CGI library, posting files with non-ASCII characters will lead the header to be (for example) filename*=utf-8"x" which is not recognised: l 513 if 'filename' in pdict: self.filename = pdict['filename'] self._binary_file = self.filename is not None The file will thus be treated as a string. The correction isn't too big but being a total newbie, I'm a bit scared to suggest a patch. -- components: Unicode messages: 261491 nosy: ezio.melotti, haypo, olemoign priority: normal severity: normal status: open title: CGI library - Using unicode in header fields type: behavior versions: Python 3.5 ___ Python tracker <http://bugs.python.org/issue26527> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26527] CGI library - Using unicode in header fields
Olivier Le Moign added the comment: I guess this is fixed by https://pypi.python.org/pypi/rfc6266. Could have looked better, sorry. -- ___ Python tracker <http://bugs.python.org/issue26527> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
