Re: [Python-Dev] [RELEASED] Python 2.7.5
On 5/16/2013 1:18 AM, Ben Hoyt wrote: Thanks, Benjamin -- that's great! This may not be a python-dev question exactly. But on Windows, is it safe to update to 2.7.5 on top of 2.7.4 (at C:\Python27) using the .msi installer? In other words, will it update/add/remove all the files correctly? What if python.exe is running? Yes, I update all the time, but without python running. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 2.7.5
This may not be a python-dev question exactly. But on Windows, is it > safe to update to 2.7.5 on top of 2.7.4 (at C:\Python27) using the .msi >> installer? In other words, will it update/add/remove all the files >> correctly? What if python.exe is running? >> > > Yes, I update all the time, but without python running. Great to know -- thanks. -Ben ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 2.7.5
>> Yes, I update all the time, but without python running. FYI, I tried this just now with Python 2.7.4 running, and the installer nicely tells you that "some files that need to be updated are currently in use ... the following applications are using files, please close them and click Retry ... python.exe (Process Id: 5388)". So you can't do it while python.exe is running, but at least it notifies you and gives you the option to retry. Good work, whoever did this installer. -Ben ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 3.2.5 and Python 3.3.2
16.05.13 08:20, Georg Brandl написав(ла): On behalf of the Python development team, I am pleased to announce the releases of Python 3.2.5 and 3.3.2. The releases fix a few regressions in 3.2.4 and 3.3.1 in the zipfile, gzip and xml.sax modules. Details can be found in the changelogs: It seems that I'm the main culprit of this releases. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 2.7.5
Am 16.05.13 10:42, schrieb Ben Hoyt: > FYI, I tried this just now with Python 2.7.4 running, and the > installer nicely tells you that "some files that need to be updated > are currently in use ... the following applications are using files, > please close them and click Retry ... python.exe (Process Id: 5388)". > > So you can't do it while python.exe is running, but at least it > notifies you and gives you the option to retry. Good work, whoever did > this installer. This specific feature is part of the MSI technology itself, so the honor goes to Microsoft in this case. They also have an advanced feature where the installer can tell the running application to terminate, and then restart after installation (since Vista, IIRC). Unfortunately, this doesn't apply to Python, as a "safe restart" is typically not feasible. FWIW, I'm the one who put together the Python installer. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 3.2.5 and Python 3.3.2
2013/5/16 Serhiy Storchaka : > 16.05.13 08:20, Georg Brandl написав(ла): >> >> On behalf of the Python development team, I am pleased to announce the >> releases of Python 3.2.5 and 3.3.2. >> >> The releases fix a few regressions in 3.2.4 and 3.3.1 in the zipfile, gzip >> and xml.sax modules. Details can be found in the changelogs: > > > It seems that I'm the main culprit of this releases. Well, when I look at the changelogs, what strikes me more is that you're the author of *many* fixes, and also a lot of new features/improvements. So I wouldn't feel bad if I were you, this kind of things happens (and it certainly did to me). Cheers, Charles ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 3.2.5 and Python 3.3.2
2013/5/16 Serhiy Storchaka : > 16.05.13 08:20, Georg Brandl написав(ла): > >> On behalf of the Python development team, I am pleased to announce the >> releases of Python 3.2.5 and 3.3.2. >> >> The releases fix a few regressions in 3.2.4 and 3.3.1 in the zipfile, gzip >> and xml.sax modules. Details can be found in the changelogs: > > > It seems that I'm the main culprit of this releases. You've now passed your Python-dev initiation. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On May 15, 2013, at 06:06 PM, Tres Seaver wrote: >On 05/15/2013 04:58 PM, Barry Warsaw wrote: >> This leads me to hypothesize that the bug is due to an as yet >> unidentified race condition during installation of Python source code >> on Ubuntu, which is normally when we automatically byte compile the >> source to .pyc files. > >Any chance you are using 'detox' or the equivalent to run tests on >mutliple interpreters in parallel? The only "bad marshall data" errors I >have seen lately seemed to be provoked by that kind of practice. Nope. PyPI's detox isn't even available in Ubuntu currently. (The detox package in Ubuntu is something else.) Tests should only be run at package build time, not installation time, and the byte compiling of source files at installation time *should* be single threaded and single process. We've since found a few cases where Python 3.3 pyc files are probably corrupted, so that shoots down my theory about a race condition on reading/writing pyc files, since 3.3 implements atomic-rename and *should* be immune to that kind of thing. It's still a mystery though. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBCAAGBQJRlP3SAAoJEBJutWOnSwa/Xy8QAI6Ul/s0nF+rac8nGy6fieLB FHHEfmjIgj6MkyUUw/zcbR48ELiOPkkV+GM4HJnY/H2ZG9vZqwuQYWFI0cgIBmxj EmHfKK7OTdaaHZeNTt83RDwGRLUS/gYXQ7JVikGyFnSbftfmUoN/y/ndlX5DX1hT ecDHVtXCH/ti/kcOWe2OlMABONZQPW0qYB7/0PiCCmaOxulqUsz20Ofy8SfWmSPd Rbig5i8fSnI98dkLVUzyy1tbUkdRkLBro/hawu1V9y7qVkoYx1Jz6p8XkQLp3jES m22m+6CLrnD39HxvJGGNkIaYmu5xTW+rK/Li8OrfOKx6QVIZ+XQRJFkiXnKmiezk sMYv/psySWJ/BSImsQOSt/sLHJWAGh8fkMIBpx9tI3BWMvyMkI0Hs9l7JyQn0moo oSTNb9AbgRSrkh0rVv4fhOhd1Ir3LXYTGwwYE5+o7tB/Pp0AKi2tX/XTBctDpy86 xqNHOaCV0hRA2Y+/C2QAAA7LRruP0yv10DfkciVUHR7UzbXgViICEEUizGmnkni7 utGg9EDk5VcSeg2ySxhX9Uj3E2M/ijOuYpXUJ2Gwd4UNUT5XGK1+6i2JTO2pEQOM HqhsGqk4WJsfEBTIrAt4NSxZyEuQ2nRV3MIsNaVCDp1FDySZWt3Cckq8hkZ/6vOM 7ncE6aG1cJgq4WKErvCk =BMFW -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On May 16, 2013, at 08:33 AM, Nick Coghlan wrote: >Personally, I would be suspicious of developmental web services doing >auto-reloading while an installer is recompiling the world. I don't have >enough context to be sure how plausible that is as a possible explanation, >though. It's possible that some Python written system service is getting invoked during the bytecompile-on-installation phase. But now that we've found cases of this problem with Python 3.3, I'm less inclined to follow that line of reasoning since it implements atomic rename. Unless there's a flaw in importlib we haven't identified yet. -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Am 16.05.2013 17:40, schrieb Barry Warsaw: > We've since found a few cases where Python 3.3 pyc files are > probably corrupted, so that shoots down my theory about a race > condition on reading/writing pyc files, since 3.3 implements > atomic-rename and *should* be immune to that kind of thing. > > It's still a mystery though. Are you able to reproduce the issue? Perhaps you could use inotify to track down file activity. It shouldn't affect timing much and you can track if more than one process it writing to the same file. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQIcBAEBCAAGBQJRlQe4AAoJEMeIxMHUVQ1F7K8P/1QoH8/iJP60dHQHfU12AYFY nUu1ztRmwTSx0eYEumwsRF5iUuNse7kFzYM0u02lEmZyuk34hLtBBcnNGA0wJ4mZ SdiXL1ZA2levg9Qlr8cPQqgnlm9aXnIazQKbUJ+/MOGBdTPBemMunMyMSsNg5ENT gLEVb/lufNssAoo+M0QKq9EjE2xSQEsFjUDM575KHbq006EzdHp7on2xQ20pJzLc iq/qWAFh+kjS42Udk9luvAKy3iGJcGXnG9AY0hkLBh8tQYhISWplsBo5wiigZLyv PZ0tbh5h3bsi80FjDlSfVPFOzlt34xI6tPRONUj/XWLPfvBCGzwqjYGc5Z6+CkAF pPAamF0ntwq76mNBl9EABAY1q85SgEU+toft8KQdxm+SHuKINJc95R8x6ypsnYIQ Ol+L5nUy+zV3vCJe9TM2U2cUB/UWHLM0qGTSYowLqTXtv+1Y+J55g63kOLkfCrnF znVXMU5FMotlh6i1rK/uwBttJ+NdjOTL0+eVbVqm39bBA6PU7UgANNNSIWVPCbfu HwucdVwkY932TxiVpWZBSPVLQmjNHIOlVj8uFIkhBnEeWSYkpIu+wV4f+Gc048AP 7EYYMMHTdlodNdKRlr+ksczhJvO67STjKH0a+vB2fro/wjuxwcqD7A0qG0PfvURg 7vofW0fs9lKap0wk9DsT =v709 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On May 16, 2013, at 06:22 PM, Christian Heimes wrote: >Are you able to reproduce the issue? Perhaps you could use inotify to >track down file activity. It shouldn't affect timing much and you can >track if more than one process it writing to the same file. Sadly, no. I've never seen it on any of my own desktops or servers, and none of the bug reporters have been able to provide a reproducible test case. I tried some smaller test cases to see if I could trigger pyc writing race conditions, but failed to reproduce it there either. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBCAAGBQJRlQuDAAoJEBJutWOnSwa/03AP/04BxVqqm0nrGIdtuKDej3MW 4qhXxbYExgpcFpclesFw479TatGCh3hBwDoosrYdk5Lrf8Bwa9FGUNbRJozdAGmE wAB1vq30mGm+2QtBuVPoXu3xrNWGGmUUtI0yzBSwnvxlfuIzsLkibZPMIMfdVCi+ f3/LSldowWx0DJp8V5TUq4GIhOfe3yccgIxMU55YbDj8cplzFJuBuBtO4DOGsoFI IPlFLwGPG503Nju37zzdkoq3Xkw4Og+vXtXsCv/rhAWIqnZgKYNF/CLv0dolZWFy GhjM5bfQtUWwxH6Ng1Wl2kcuCVmF1/vD2vTUCsgpA4qQc0nYrTy/q1OPho72x40o DvvaVHueDqH7N1xm64KL75sFxu6QDIniBbgV7gklU1z6P6ZVADwoilon8HC9FnJN w5I0sYLTnIHxUIrM0h0wi517gQTZHTSF0bQxKqynNV+PrZBprvB9lEkYCpy5tV0s LEqf+oUwXvGIOZ6Nmv2MyjQb0xajxHmzz+RO1qQ3R4tbiQjwGoqc43CrlxhVduJh 1VGM6b7ysZ2iwyJG+q0aVi9YSaStzzUvMPO2F+HTmE+r3MvgdTcKQQzLDuRF6LfV 74eWwtHBpiJuvdBG37uDQj5bU/oLWiYyfM52vASgHB4zoKOx0EUxAd1Wf5nyxc1E Bo0G3kYwbFaNvSnwcJZw =a4x0 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On 05/16/2013 09:38 AM, Barry Warsaw wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On May 16, 2013, at 06:22 PM, Christian Heimes wrote: Are you able to reproduce the issue? Perhaps you could use inotify to track down file activity. It shouldn't affect timing much and you can track if more than one process it writing to the same file. Sadly, no. I've never seen it on any of my own desktops or servers, and none of the bug reporters have been able to provide a reproducible test case. I tried some smaller test cases to see if I could trigger pyc writing race conditions, but failed to reproduce it there either. Is it happening on the same machines? If so, perhaps a daemon to monitor those files and then scream and shout when one changes. Might help track down what's going on at the time. (Yeah, that does sound like saying 'inotify' but with more words...) -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On May 16, 2013, at 09:44 AM, Ethan Furman wrote: >Is it happening on the same machines? If so, perhaps a daemon to monitor >those files and then scream and shout when one changes. Might help track >down what's going on at the time. (Yeah, that does sound like saying >'inotify' but with more words...) No, it's all different kinds of machines, at different times, on different files. So far, there's no rhyme or reason to the corruptions that I can tell. We're trying to instrument things to collect more data when these failures do occur. -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Thu, May 16, 2013 at 11:04 AM, Barry Warsaw wrote: > On May 16, 2013, at 09:44 AM, Ethan Furman wrote: > > >Is it happening on the same machines? If so, perhaps a daemon to monitor > >those files and then scream and shout when one changes. Might help track > >down what's going on at the time. (Yeah, that does sound like saying > >'inotify' but with more words...) > > No, it's all different kinds of machines, at different times, on different > files. So far, there's no rhyme or reason to the corruptions that I can > tell. We're trying to instrument things to collect more data when these > failures do occur. > Even on machines with ECC ram and reliable storage, not owned by l33t gam0rzs weenies who overclock things? -gps ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On 5/16/2013 2:04 PM, Barry Warsaw wrote: No, it's all different kinds of machines, at different times, on different files. So far, there's no rhyme or reason to the corruptions that I can tell. If the corruption only happens on Ubuntu, that would constitute 'rhyme' ;-). I realize that asking for reports on other systems is part of the reason you posted, but I don't remember seeing any others yet. We're trying to instrument things to collect more data when these failures do occur. Do failures only occur during compileall process? (or whatever substitute you use). At the end of py_compile.complile, after the with block that opens, writes, flushes, and closes, you could add with open(cfile, 'rb') as fc: This would be a high-level write and verify. Verify would be a bit faster if marshal.dump were replaced by marshal.dumps + write to keep alive the string version of the code object. Then the codeobject comparison in the verify step would be replaced by string comparison. You could also read and verify (by unmarshal) after the compile-all process (faster than importing). Terry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
This reminds me of the following bug, which can happen when two processes are both writing the .pyc file and a third is reading it. First some background. When writing a .pyc file, we use the following strategy: - open the file for writing - write a dummy header (four null bytes) - write the .py file's mtime - write the marshalled code object - replace the dummy heaer with the correct magic word Even py_compile.py (used by compileall.py) uses this strategy. When reading a .pyc file, we ignore it when the magic word isn't there (or when the mtime doesn't match that of the .py file exactly), and then we will write it back like described above. Now consider the following scenario. It involves *three* processes. - Two unrelated processes both start and want to import the same module. - They both see the .pyc file is missing/corrupt and decide to write it. - The first process finishing writing the file, writing the correct header. - Now a third process wants to import the module, sees the valid header, and starts reading the file. - However, while this is going on, the second process gets ready to write the file. - The second process truncates the file, writes the dummy header, and then stalls. - At this point the third process (which thought it was reading a valid file) sees an unexpected EOF because the file has been truncated. Now, this would explain the EOFError, but not necessarily the ValueError with "unknown type code". However, it looks like marshal doesn't always check for EOF immediately (sometimes it calls getc() without checking the result, and sometimes it doesn't check the error state after calling r_string()), so I think all the errors are actually explainable from this scenario. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Thu, May 16, 2013 at 5:19 PM, Guido van Rossum wrote: > This reminds me of the following bug, which can happen when two > processes are both writing the .pyc file and a third is reading it. > First some background. > > When writing a .pyc file, we use the following strategy: > - open the file for writing > - write a dummy header (four null bytes) > - write the .py file's mtime > - write the marshalled code object > - replace the dummy heaer with the correct magic word > Just so people know, this is how we used to do it. In importlib we write the entire file to a temp file and then to an atomic rename. > Even py_compile.py (used by compileall.py) uses this strategy. py_compile as of Python 3.4 now just uses importlib directly, so it matches its semantics. -Brett > > When reading a .pyc file, we ignore it when the magic word isn't there > (or when the mtime doesn't match that of the .py file exactly), and > then we will write it back like described above. > > Now consider the following scenario. It involves *three* processes. > > - Two unrelated processes both start and want to import the same module. > - They both see the .pyc file is missing/corrupt and decide to write it. > - The first process finishing writing the file, writing the correct header. > - Now a third process wants to import the module, sees the valid > header, and starts reading the file. > - However, while this is going on, the second process gets ready to > write the file. > - The second process truncates the file, writes the dummy header, and > then stalls. > - At this point the third process (which thought it was reading a > valid file) sees an unexpected EOF because the file has been > truncated. > > Now, this would explain the EOFError, but not necessarily the > ValueError with "unknown type code". However, it looks like marshal > doesn't always check for EOF immediately (sometimes it calls getc() > without checking the result, and sometimes it doesn't check the error > state after calling r_string()), so I think all the errors are > actually explainable from this scenario. > > -- > --Guido van Rossum (python.org/~guido) > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
I still suspect this might explain most of what Barry saw, if not all. — Sent from Mailbox On Thu, May 16, 2013 at 2:36 PM, Brett Cannon wrote: > On Thu, May 16, 2013 at 5:19 PM, Guido van Rossum wrote: >> This reminds me of the following bug, which can happen when two >> processes are both writing the .pyc file and a third is reading it. >> First some background. >> >> When writing a .pyc file, we use the following strategy: >> - open the file for writing >> - write a dummy header (four null bytes) >> - write the .py file's mtime >> - write the marshalled code object >> - replace the dummy heaer with the correct magic word >> > Just so people know, this is how we used to do it. In importlib we > write the entire file to a temp file and then to an atomic rename. >> Even py_compile.py (used by compileall.py) uses this strategy. > py_compile as of Python 3.4 now just uses importlib directly, so it > matches its semantics. > -Brett >> >> When reading a .pyc file, we ignore it when the magic word isn't there >> (or when the mtime doesn't match that of the .py file exactly), and >> then we will write it back like described above. >> >> Now consider the following scenario. It involves *three* processes. >> >> - Two unrelated processes both start and want to import the same module. >> - They both see the .pyc file is missing/corrupt and decide to write it. >> - The first process finishing writing the file, writing the correct header. >> - Now a third process wants to import the module, sees the valid >> header, and starts reading the file. >> - However, while this is going on, the second process gets ready to >> write the file. >> - The second process truncates the file, writes the dummy header, and >> then stalls. >> - At this point the third process (which thought it was reading a >> valid file) sees an unexpected EOF because the file has been >> truncated. >> >> Now, this would explain the EOFError, but not necessarily the >> ValueError with "unknown type code". However, it looks like marshal >> doesn't always check for EOF immediately (sometimes it calls getc() >> without checking the result, and sometimes it doesn't check the error >> state after calling r_string()), so I think all the errors are >> actually explainable from this scenario. >> >> -- >> --Guido van Rossum (python.org/~guido) >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/brett%40python.org___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Thu, May 16, 2013 at 5:40 PM, Guido van Rossum wrote: > I still suspect this might explain most of what Barry saw, if not all. Quite possible, especially since he is seeing more issues on 3.2 than 3.3. Just wanted to fill people in on how 3.3 onwards does things is all. -Brett > — > Sent from Mailbox > > > On Thu, May 16, 2013 at 2:36 PM, Brett Cannon wrote: >> >> On Thu, May 16, 2013 at 5:19 PM, Guido van Rossum >> wrote: >> > This reminds me of the following bug, which can happen when two >> > processes are both writing the .pyc file and a third is reading it. >> > First some background. >> > >> > When writing a .pyc file, we use the following strategy: >> >> > - open the file for writing >> > - write a dummy header (four null bytes) >> > - write the .py file's mtime >> > - write the marshalled code object >> > - replace the dummy heaer with the correct magic word >> > >> >> Just so people know, this is how we used to do it. In importlib we >> write the entire file to a temp file and then to an atomic rename. >> >> > Even py_compile.py (used by compileall.py) uses this strategy. >> >> py_compile as of Python 3.4 now just uses importlib directly, so it >> matches its semantics. >> >> -Brett >> >> > >> > When reading a .pyc file, we ignore it when the magic word isn't there >> > (or when the mtime doesn't match that of the .py file exactly), and >> > then we will write it back like described above. >> > >> > Now consider the following scenario. It involves *three* processes. >> > >> > - Two unrelated processes both start and want to import the same module. >> > - They both see the .pyc file is missing/corrupt and decide to write it. >> > - The first process finishing writing the file, writing the correct >> > header. >> > - Now a third process wants to import the module, sees the valid >> > header, and starts reading the file. >> > - However, while this is going on, the second process gets ready to >> > write the file. >> > - The second process truncates the file, writes the dummy header, and >> > then stalls. >> > - At this point the third process (which thought it was reading a >> > valid file) sees an unexpected EOF because the file has been >> > truncated. >> > >> > Now, this would explain the EOFError, but not necessarily the >> > ValueError with "unknown type code". However, it looks like marshal >> > doesn't always check for EOF immediately (sometimes it calls getc() >> > without checking the result, and sometimes it doesn't check the error >> > state after calling r_string()), so I think all the errors are >> > actually explainable from this scenario. >> > >> > -- >> > --Guido van Rossum (python.org/~guido) >> > ___ >> > Python-Dev mailing list >> > Python-Dev@python.org >> > http://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On 5/16/2013 5:30 PM, Brett Cannon wrote: On Thu, May 16, 2013 at 5:19 PM, Guido van Rossum wrote: This reminds me of the following bug, which can happen when two processes are both writing the .pyc file and a third is reading it. First some background. When writing a .pyc file, we use the following strategy: - open the file for writing - write a dummy header (four null bytes) - write the .py file's mtime - write the marshalled code object - replace the dummy heaer with the correct magic word Just so people know, this is how we used to do it. In importlib we write the entire file to a temp file and then to an atomic rename. Even py_compile.py (used by compileall.py) uses this strategy. py_compile as of Python 3.4 now just uses importlib directly, so it matches its semantics. But in 3.3, it still is as Guido describes, even though importlib is improved. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Thu, May 16, 2013 at 11:19 PM, Guido van Rossum wrote: > This reminds me of the following bug, which can happen when two > processes are both writing the .pyc file and a third is reading it. > First some background. > > When writing a .pyc file, we use the following strategy: > - open the file for writing > - write a dummy header (four null bytes) > - write the .py file's mtime > - write the marshalled code object > - replace the dummy heaer with the correct magic word > > Even py_compile.py (used by compileall.py) uses this strategy. > > When reading a .pyc file, we ignore it when the magic word isn't there > (or when the mtime doesn't match that of the .py file exactly), and > then we will write it back like described above. > > Now consider the following scenario. It involves *three* processes. > > - Two unrelated processes both start and want to import the same module. > - They both see the .pyc file is missing/corrupt and decide to write it. > - The first process finishing writing the file, writing the correct header. > - Now a third process wants to import the module, sees the valid > header, and starts reading the file. > - However, while this is going on, the second process gets ready to > write the file. > - The second process truncates the file, writes the dummy header, and > then stalls. > - At this point the third process (which thought it was reading a > valid file) sees an unexpected EOF because the file has been > truncated. > > Now, this would explain the EOFError, but not necessarily the > ValueError with "unknown type code". The 'unknown type codes' can also be explained if the two processes writing to the .pyc files are *different Python versions*. As you may recall, at Google we used to use modified Python interpreters that used '.pyc-2.2', '.pyc-2.4', etc, for the pyc extension. That was because otherwise different Python versions would keep overwriting the .pyc files of shared Python modules, and "at Google scale" it caused all manner of problems... I guess Ubuntu is approaching Google scale ;-) (The decision to rename to an awkward extension broke a lot of third-party tools; it was made before I -- or you, for that matter -- joined Google... Now we just turn on -B by default :) > However, it looks like marshal > doesn't always check for EOF immediately (sometimes it calls getc() > without checking the result, and sometimes it doesn't check the error > state after calling r_string()), so I think all the errors are > actually explainable from this scenario. > > -- > --Guido van Rossum (python.org/~guido) > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/thomas%40python.org > -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
Guido van Rossum wrote: This reminds me of the following bug, which can happen when two processes are both writing the .pyc file and a third is reading it. ... I think all the errors are actually explainable from this scenario. The second writer will still carry on to write a valid .pyc file, though, won't it? So this wouldn't result in a permanently broken .pyc file being left behind, which is what the original problem description seemed say was happening. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Thu, May 16, 2013 at 3:27 PM, Greg Ewing wrote: > Guido van Rossum wrote: >> >> This reminds me of the following bug, which can happen when two >> processes are both writing the .pyc file and a third is reading it. >> ... I think all the errors are >> >> actually explainable from this scenario. > > > The second writer will still carry on to write a valid > .pyc file, though, won't it? So this wouldn't result in > a permanently broken .pyc file being left behind, which > is what the original problem description seemed say > was happening. >From the evidence that is not completely clear to me. Thomas Wouters' scenario with two different Python versions writing the same .pyc file could cause that; I don't know if Barry has ruled that possibility out yet. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On 17 May 2013 08:37, "Guido van Rossum" wrote: > > On Thu, May 16, 2013 at 3:27 PM, Greg Ewing wrote: > > Guido van Rossum wrote: > >> > >> This reminds me of the following bug, which can happen when two > >> processes are both writing the .pyc file and a third is reading it. > >> ... I think all the errors are > >> > >> actually explainable from this scenario. > > > > > > The second writer will still carry on to write a valid > > .pyc file, though, won't it? So this wouldn't result in > > a permanently broken .pyc file being left behind, which > > is what the original problem description seemed say > > was happening. > > From the evidence that is not completely clear to me. > > Thomas Wouters' scenario with two different Python versions writing > the same .pyc file could cause that; I don't know if Barry has ruled > that possibility out yet. 3.2 uses __pycache__, so it should only potentially conflict within the same version. I haven't heard any rumblings about anything like this in Fedora or RHEL, so my suspicions still lean towards a Debian or Ubuntu specific background service somehow managing to interfere. However, I'll ask explicitly on the Fedora Python list to see if anyone has encountered anything similar. Cheers, Nick. > > -- > --Guido van Rossum (python.org/~guido) > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Why is nb_inplace_add copied to sq_inplace_concat?
I have encountered what I believe to be a bug but I'm sure there is some
reason things are done as they are and I am hoping someone can shed some light
or confirm it is indeed a bug.
As a bit of background I have a c++ class that I use sip to generate python
bindings. The potential python bug manifests itself as:
>>> rl = RecordList()
>>> rl += []
>>> rl
NotImplemented
The bindings fill in nb_inplace_add which appears to be implemented properly,
returning a new reference to Py_NotImplemented if the right hand argument is
not as expected.
Where things appear to go wrong is that PyNumber_InPlaceAdd, after getting a
NotImplemented return value from nb_inplace_add, then attempts to call
sq_inplace_concat. From reading the code it appears that sq_inplace_concat is
not supposed to return NotImplemented, instead it should set an exception and
return null if the right hand arg is not supported. In my case
sq_inplace_concat ends up being the same function as nb_inplace_add, which
results in the buggy behavior.
When I figured this out I tried to find out why sq_inplace_concat was set to
the
same function as nb_inplace_add, and ended up having to set a watchpoint in
gdb which finally gave me the answer that python itself is setting
sq_inplace_concat during type creation in the various functions in
typeobject.c. Stack trace is below.
I don't really understand what the fixup_slot_dispatchers function is doing,
but it does seem like there must be a bug either in what it's doing, or in
PyNumber_InPlaceAdd's handling of a NotImplemented return value from
sq_inplace_concat.
Thanks,
Matt
Python 2.7.3 (default, Jan 2 2013, 13:56:14)
[GCC 4.7.2] on linux2
Stack trace where a watch on sq->sq_inplace_concat reveals the change:
Hardware watchpoint 5: *(binaryfunc *) 0xcf6f88
Old value = (binaryfunc) 0
New value = (binaryfunc) 0x74d41c78
#0 update_one_slot.25588 (type=type@entry=0xcf6c70, p=0x86ba90) at
../Objects/typeobject.c:6203
#1 0x004b96d0 in fixup_slot_dispatchers (type=0xcf6c70) at
../Objects/typeobject.c:6299
#2 type_new.part.40 (kwds=, args=0x0, metatype=) at ../Objects/typeobject.c:2464
#3 type_new.25999 (metatype=, args=0x0, kwds=)
at ../Objects/typeobject.c:2048
#4 0x00463c08 in type_call.25547 (type=0x765953a0,
args=('RecordList', (,),
{'__module__': 'blur.Stone'}), kwds=0x0) at ../Objects/typeobject.c:721
#5 0x004644eb in PyObject_Call (func=,
arg=, kw=) at ../Objects/abstract.c:2529
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: fix compilation on Windows
On 5/16/2013 4:17 PM, victor.stinner wrote: summary: fix compilation on Windows That fixed my problem with compiling 3.4, 32 bit, Win 7. Thanks. But I cannot compile 3.3 python_d since May 6. In fact, there are more errors now than 8 hours ago. 7 things failed to build instead of 5 (3 is normal for me, given the lack of some dependencies). I believe the following is new. Red error box with .../p33/PCBuild/make_versioninfo_d.exe is not a valid Win32 application. The VS gui output box has "Please verify that you have sufficient rights to run this command." Some more errors: 10>..\PC\pylauncher.rc(16): error RC2104: undefined keyword or key name: FIELD3 10> 9> symtable.c 9>..\Python\symtable.c(1245): error C2143: syntax error : missing ';' before 'type' 9>..\Python\symtable.c(1246): error C2065: 'cur' : undeclared identifier 9>..\Python\symtable.c(1248): error C2065: 'cur' : undeclared identifier 9>..\Python\symtable.c(1253): error C2065: 'cur' : undeclared identifier 23> Traceback (most recent call last): 23>File "build_ssl.py", line 253, in 23> main() 23>File "build_ssl.py", line 187, in main 23> os.chdir(ssl_dir) 23> FileNotFoundError: [WinError 2] The system cannot find the file specified: '..\\..\\openssl-1.0.1e' Earlier, about 4 other files had several warnings. I do no see the warnings now because they compiled then and have not changed. Errors are more urgent, but should warnings be ignored? Terry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why is nb_inplace_add copied to sq_inplace_concat?
On Fri, May 17, 2013 at 9:17 AM, Matt Newell wrote: > I don't really understand what the fixup_slot_dispatchers function is doing, > but it does seem like there must be a bug either in what it's doing, or in > PyNumber_InPlaceAdd's handling of a NotImplemented return value from > sq_inplace_concat. I didn't read your post in detail, but operand precedence in CPython is known to be broken for types which only populate the sq_* slots without also populating the corresponding nb_* slots: http://bugs.python.org/issue11477 The bug doesn't affect types implemented in Python, as the interpreter always populates both slots (I believe Cython also populated both slots for types defined that way). I made one attempt at fixing it (by changing the fallback handling in abstract.c) but it turned out to be completely unmaintainable (and didn't really work right anyway). There's another suggested approach that would likely work better (automatically populating the nb_* slots with delegation wrappers and losing the fallback code in abstract.c entirely), but it still needs a patch (the test cases from my failed attempt may still prove useful, though). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why is nb_inplace_add copied to sq_inplace_concat?
On Fri, May 17, 2013 at 1:38 PM, Nick Coghlan wrote: > On Fri, May 17, 2013 at 9:17 AM, Matt Newell wrote: >> I don't really understand what the fixup_slot_dispatchers function is doing, >> but it does seem like there must be a bug either in what it's doing, or in >> PyNumber_InPlaceAdd's handling of a NotImplemented return value from >> sq_inplace_concat. > > I didn't read your post in detail, but operand precedence in CPython > is known to be broken for types which only populate the sq_* slots > without also populating the corresponding nb_* slots: > http://bugs.python.org/issue11477 Oops, I meant to state that one of the consequences of the bug is that returning NotImplemented from the sq_* methods doesn't work at all - it's never checked and thus never turned into a TypeError. That's why changing to delegation from the nb_* slots is the most promising approach - all that handling is there and correct for the numeric types, but pure sequence types (which can only be created from C code) bypass that handling. I *did* read enough of the original post to know that was the symptom you were seeing, I just failed to mention that in my initial reply... Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/16/2013 06:59 PM, Nick Coghlan wrote: > 3.2 uses __pycache__, so it should only potentially conflict within > the same version. > > I haven't heard any rumblings about anything like this in Fedora or > RHEL, so my suspicions still lean towards a Debian or Ubuntu specific > background service somehow managing to interfere. However, I'll ask > explicitly on the Fedora Python list to see if anyone has encountered > anything similar. I can confirm at least that I have seen this problem within the last two weeks on Ubuntu boxes unrelated to the thw Debian / Ubuntu build infrastruction. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGVqIgACgkQ+gerLs4ltQ6ksACePs7jO1TynGm3kNodpV4lPA2b VbgAoNNHMmQhJQhOvxuHMO/LFyv+Umho =KNdc -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why is nb_inplace_add copied to sq_inplace_concat?
On Thursday, May 16, 2013 08:41:32 PM you wrote: > On Fri, May 17, 2013 at 1:38 PM, Nick Coghlan wrote: > > On Fri, May 17, 2013 at 9:17 AM, Matt Newell wrote: > >> I don't really understand what the fixup_slot_dispatchers function is > >> doing, but it does seem like there must be a bug either in what it's > >> doing, or in PyNumber_InPlaceAdd's handling of a NotImplemented return > >> value from sq_inplace_concat. > > > > I didn't read your post in detail, but operand precedence in CPython > > is known to be broken for types which only populate the sq_* slots > > without also populating the corresponding nb_* slots: > > http://bugs.python.org/issue11477 In this case it's the other way around. Only nb_inplace_add is populated, and python forces the buggy behavior that you describe below by copying nb_inplace_add to sq_inplace_concat. > > Oops, I meant to state that one of the consequences of the bug is that > returning NotImplemented from the sq_* methods doesn't work at all - > it's never checked and thus never turned into a TypeError. That's why > changing to delegation from the nb_* slots is the most promising > approach - all that handling is there and correct for the numeric > types, but pure sequence types (which can only be created from C code) > bypass that handling. > > I *did* read enough of the original post to know that was the symptom > you were seeing, I just failed to mention that in my initial reply... > I read through the bug and it looks like whatever solution you choose will fix this problem also. In the meantime I guess the solution for me is to always define sq_inplace_concat with a function that simply raises a TypeError. Hmm, even simpler would be to reset sq_inplace_concat to 0 after python sets it. I actually tested the latter in gdb and it gave the correct results. I'll just have to keep an eye out to make sure my workaround doesn't break things when the real fix gets into python. Matt ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Thu, 16 May 2013 11:42:30 -0400 Barry Warsaw wrote: > On May 16, 2013, at 08:33 AM, Nick Coghlan wrote: > > >Personally, I would be suspicious of developmental web services doing > >auto-reloading while an installer is recompiling the world. I don't have > >enough context to be sure how plausible that is as a possible explanation, > >though. > > It's possible that some Python written system service is getting invoked > during the bytecompile-on-installation phase. But now that we've found cases > of this problem with Python 3.3, I'm less inclined to follow that line of > reasoning since it implements atomic rename. Please try to reproduce it by adding e.g. some sleep() calls in the middle of the writing routine. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
