[issue5092] weird memory usage in multiprocessing module
New submission from Jerzy : Hi I am using the multiprocessing mudule and I found a very weird thing. It seems that that the result of one fragment of the code depends on the fragment of the code that is after it, which should not happen. My script looks like this import time import multiprocessing import sys def f(): sys.stderr.write(str(len(l))+"\n") print len(l) #del l while(True): time.sleep(1) l=[] for i in range(2*1000*1000): l.append(str(i)) process = multiprocessing.Process(target=f) process.start() while(True): time.sleep(1) And its output is as expected: 200 200 but when I uncoment the 'del l' line I get: File "/home/jerzyo/programs/python2.6/Python-2.6.1/Lib/multiprocessing/process.py", line 231, in _bootstrap self.run() File "/home/jerzyo/programs/python2.6/Python-2.6.1/Lib/multiprocessing/process.py", line 88, in run self._target(*self._args, **self._kwargs) File "bin/momory.py", line 6, in f sys.stderr.write(str(len(l))+"\n") UnboundLocalError: local variable 'l' referenced before assignment How is that? The line that deletes l is after the printing line. How python interpreter knows that l will be deleted. This is a very anomalus behaviour and should never happen. By the way. Is there any way to free some parts of memory in child process. Suppose I wand to create 100 child processes that do not use the l list. How can I avoid making 100 copies of l in child processes. That is my firs post and won't come here very often, so please answer also to my email (if it is not automaic). I am running python 2.6.1 on ubuntu 8.04 32bit. jerzy -- messages: 80731 nosy: Orlowski severity: normal status: open title: weird memory usage in multiprocessing module type: resource usage versions: Python 2.6 ___ Python tracker <http://bugs.python.org/issue5092> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5092] weird memory usage in multiprocessing module
Jerzy added the comment: I still do not understand what is going on when python executed thic code. I have a local variable l in my parent process. When I create a child process, program makes first makes a copy of memory. Than what? I am sure that l still exists in child process because 1. It can be printed 2. It has still a lot of memory allocated for it You say that l does not exist as a local variable in child process. Is it global? How can I dealocate it in child process? Jerzy Martin v. Löwis pisze: > Martin v. Löwis added the comment: > > As David says, this is not a bug. del l indicates that there is a local > variable to be deleled, but when the del statement is executed, there is > no local variable. The error message is confusing in this case: there > actually is no later assignment to l (in the function at all). > Typically, when you have an unbound local, it is because of a later > assignment, such as > > def foo(): > a = l + 1 > l = 2 > > In this specific example, there is no later assignment - yet it is still > an unbound local. > > So that you get the exception is not a bug. > > I was going to suggest that the error message could be better, but I > can't think of any other error message that is better and still correct, > hence closing it as won't fix. > > -- > nosy: +loewis > resolution: -> wont fix > status: open -> closed > > ___ > Python tracker > <http://bugs.python.org/issue5092> > ___ > > > ___ Python tracker <http://bugs.python.org/issue5092> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5092] weird memory usage in multiprocessing module
Jerzy added the comment: OK, I see and if don't want l to exist in f() I have to: def f(): pass def a(): l=[] f() a() Jurek Martin v. Löwis wrote: > Martin v. Löwis added the comment: > >> I still do not understand what is going on when python executed thic >> code. I have a local variable l in my parent process. > > No, you don't. It's a global variable, not a local one. > >> When I create a >> child process, program makes first makes a copy of memory. Than what? > > It doesn't have to do anything with the multiprocessing at all. > For comparison, just run the Python script > > def f(): > del l > l = [] > f() > > It produces the same error, with no multiprocessing involved. > > ___ > Python tracker > <http://bugs.python.org/issue5092> > ___ > > > ___ Python tracker <http://bugs.python.org/issue5092> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5092] weird memory usage in multiprocessing module
Jerzy added the comment: And anyway, for me it's not OK if something in a code of a function like 'del' affect how variables are affected in whole function. It is really illogical. There code is in lines and line are one below another. The logical way is that a line of code affects the program ONLY when it is executed and ONLY from the time it is executed. A statement that is not executed (python never reach the place) should not affect the program in ANY way. You may think what you think, but for me it is a big bug in the heart of python Jerzy Martin v. Löwis wrote: > Martin v. Löwis added the comment: > >> I still do not understand what is going on when python executed thic >> code. I have a local variable l in my parent process. > > No, you don't. It's a global variable, not a local one. > >> When I create a >> child process, program makes first makes a copy of memory. Than what? > > It doesn't have to do anything with the multiprocessing at all. > For comparison, just run the Python script > > def f(): > del l > l = [] > f() > > It produces the same error, with no multiprocessing involved. > > ___ > Python tracker > <http://bugs.python.org/issue5092> > ___ > > > ___ Python tracker <http://bugs.python.org/issue5092> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5092] weird memory usage in multiprocessing module
Jerzy added the comment:
I am not an expert. But for me it is much better.
If you cannot delete the global variable in a function (del makes the
variable local anyway). So trying to delete a global variable should
raise an exception "Cannot delete a global variable" or something like
that. In a function variable should be global till the place when you
define a local one.
Example:
a='Something'
def f():
print a #prints the global variable a
del a #Make an exception that a is global so it cannot be deleted
a='anotherthing' #make a local a
print a #print local a
del a#delete local a
print a #print global a
f()
Also, if there are two variable (global and local) with seme name, there
should be a way to access either of them like 'print loc(a)' and 'print
glob(a)'. This is just a suggestion
Another way of resolving the problem would be making it impossible to
make a local variable when there is anothe one with the same name.
David W. Lambert pisze:
> David W. Lambert added the comment:
>
> The alternative is unreasonable. I doubt you'd be happy with this:
>
>
> a = 'Something'
>
> def variable_both_global_and_local()->Exception('No good!'):
> del a# delete a from global name space
> a = 'anotherthing' # define a in local name space
>
> ___
> Python tracker
> <http://bugs.python.org/issue5092>
> ___
>
>
>
___
Python tracker
<http://bugs.python.org/issue5092>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6832] Outputting unicode crushes when printing to file on Linux
New submission from Jerzy : Hi When I am outputting unicode strings to terminal my script works OK, but when I redirect it to file I get a crash: $ python mailing/message_sender.py -l Bia Białystok $ python mailing/message_sender.py -l Bia > ~/tmp/aaa.txt Traceback (most recent call last): File "mailing/message_sender.py", line 71, in list_groups(unicode(args[0],'utf-8')) File "mailing/message_sender.py", line 53, in list_groups print group[1].name UnicodeEncodeError: 'ascii' codec can't encode character u'\u0142' in position 3: ordinal not in range(128) -- components: Unicode messages: 92196 nosy: Orlowski severity: normal status: open title: Outputting unicode crushes when printing to file on Linux type: crash versions: Python 2.6 ___ Python tracker <http://bugs.python.org/issue6832> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6832] Outputting unicode crushes when printing to file on Linux
Jerzy added the comment: I know how to make it work. The question is why outputting to file makes it crush when outputting to terminal does not. I have never seen "$program > file" behaving in a different way than "$program" in any other language Jerzy Orlowski Benjamin Peterson wrote: > Benjamin Peterson added the comment: > > You have to use an encoding that's not ascii then. > > -- > nosy: +benjamin.peterson > resolution: -> works for me > status: open -> closed > > ___ > Python tracker > <http://bugs.python.org/issue6832> > ___ > > > > -- ___ Python tracker <http://bugs.python.org/issue6832> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6832] Outputting unicode crushes when printing to file on Linux
Jerzy added the comment: Well, I would suggest using the terminal encoding as default one when redirecting. In my opinion sys.stdin and sys.stdout should always have the terminal encoding Alternatively you could make the function sys.setdefaultencoding() visible to change it in a reasonable way Jerzy Georg Brandl wrote: > Georg Brandl added the comment: > > When output goes to a terminal, Python can determine its encoding. For a > file, it cannot, therefore it refuses to guess. > > Also, many programs behave differently when used with redirection; > namely, all those that use `isatty()` to determine if stdout is a terminal. > > -- > nosy: +georg.brandl > > ___ > Python tracker > <http://bugs.python.org/issue6832> > ___ > > > > -- ___ Python tracker <http://bugs.python.org/issue6832> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6832] Outputting unicode crushes when printing to file on Linux
Jerzy added the comment: OK, I give up. The problem is that one might test a program on terminal and think that everything is running OK and then spend a reasonable amount of time trying to find the problem later Another approach: couldn't utf8 be set as default encoding for all inputs and outputs? I know that some of my questions are caused by the fact that I do not understand how python works. But You have to bear in mind that most of the people don't. Such behaviour of Python (see also http://bugs.python.org/issue5092) is illogical in the "common sense" for standard poeple. If interpreter does something illogical for me, I am more eager to switch to another language. Jerzy Martin v. Löwis wrote: > Martin v. Löwis added the comment: > > Using the terminal encoding for sys.stdout does not work in the general > case, as a (background) process may not *have* a controlling terminal > (such as a CGI script, a cron job, or a Windows service). That Python > recognizes the terminal encoding is primarily a convenience feature for > the interactive mode. > > Exposing sys.setdefaultencoding is not implementable in a reasonable way. > > -- > nosy: +loewis > > ___ > Python tracker > <http://bugs.python.org/issue6832> > ___ > > > > -- ___ Python tracker <http://bugs.python.org/issue6832> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6832] Outputting unicode crushes when printing to file on Linux
Jerzy added the comment: good point! I will give it a try Jerzy Martin v. Löwis wrote: > Martin v. Löwis added the comment: > > If you want to switch to a different language, consider switching to > Python 3. There, all strings are Unicode strings, and files opened in > text mode always use the locale encoding. > > -- > > ___ > Python tracker > <http://bugs.python.org/issue6832> > ___ > > > > -- ___ Python tracker <http://bugs.python.org/issue6832> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9742] Python 2.7: math module fails to build on Solaris 9
Jerzy Kozera added the comment: Running gcc -Wl,-R/usr/local/lib,-R/usr/lib -o python Python/pymath.o Modules/python.o libpython2.7.a -lresolv -lsocket -lnsl -lrt -ldl -lpthread -lm mv build/lib.solaris-2.8-sun4u-2.7/math_failed.so build/lib.solaris-2.8-sun4u-2.7/math.so seems to have made math module import correctly and work: bash-2.03$ ./python Python 2.7 (r27:82500, Nov 23 2010, 14:49:30) [GCC 3.4.6] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> import math >>> math.floor(2.4) 2.0 I suppose it's more a workaround than a solution, but hopefully it makes using math module possible and confirms the suggestion there might be something wrong with ar/gcc linking the .a file. -- nosy: +Jerzy.Kozera ___ Python tracker <http://bugs.python.org/issue9742> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7978] SocketServer doesn't handle syscall interruption
Jerzy Kozera added the comment: I've updated the patch according to suggestions from Gregory P. Smith. Thanks to a change from #12555 (PEP 3151) now just checking for OSError is enough. (I've decided to use mocked select() instead of calling alarm() to avoid depending on timing.) -- nosy: +Jerzy.Kozera Added file: http://bugs.python.org/file25144/socketserver_eintr_20120406.diff ___ Python tracker <http://bugs.python.org/issue7978> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7978] SocketServer doesn't handle syscall interruption
Jerzy Kozera added the comment: I forgot to mention my patch is 3.3-only, sorry - it depends on changes from #12555 (http://hg.python.org/cpython/rev/41a1de81ef2b#l18.21 to be precise). To support 3.2 and 2.7: (1) select.error must be caught as in the original patch, (2) e.args[0] must be used - select.error doesn't have 'errno' attribute. Should I prepare the patch for 3.2 and 2.7? Regarding not updating the timeout, it was already mentioned above. Though as an afterthought, it might be worrying that if the process is receiving repeated signals with interval between them less than timeout, we might fall into infinite loop of select() when it should timeout, but that is probably very obscure case. -- ___ Python tracker <http://bugs.python.org/issue7978> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6518] Enable 'with' statement in ossaudiodev module
New submission from Jerzy Jalocha N :
Actually, it is not possible to use the 'with' statement in the
ossaudiodev module:
>>> import ossaudiodev
>>> with ossaudiodev.open('/dev/dsp', 'r') as device:
... pass
...
Traceback (most recent call last):
File "", line 1 in
AttributeError: 'ossaudodev.oss_audio_device' object has no attribute
'__exit__'
In order to provide a similar interface as standard Python files, and
encourage safe coding practices, the 'with' statement should be
supported in the ossaudiodev module.
Thanks.
--
components: Extension Modules
messages: 90697
nosy: jjalocha
severity: normal
status: open
title: Enable 'with' statement in ossaudiodev module
type: feature request
versions: Python 2.6, Python 2.7, Python 3.0, Python 3.1, Python 3.2
___
Python tracker
<http://bugs.python.org/issue6518>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6519] Reorder 'with' statement for files in Python Tutorial
New submission from Jerzy Jalocha N : Actually, the Python Tutorial recommends the use of the 'with' statement in Section 7.2.1. "Methods of File Objects": > It is good practice to use the with keyword when dealing with file > objects. [etc.] But the example and description are at the very bottom of this very large section, and are easily missed by new Python users. If this suggestion is to be taken seriously, I suggest putting this information at a more prominent place, somewhere at the top of the short section 7.2. "Reading and Writing Files". -- assignee: georg.brandl components: Documentation messages: 90698 nosy: georg.brandl, jjalocha severity: normal status: open title: Reorder 'with' statement for files in Python Tutorial type: feature request versions: Python 2.6, Python 2.7, Python 3.0, Python 3.1, Python 3.2 ___ Python tracker <http://bugs.python.org/issue6519> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
