Re: [Python-Dev] Special file "nul" in Windows and os.stat
2007/11/3, "Martin v. Löwis" <[EMAIL PROTECTED]>: > > GetFileAttributes() doesn't return those, just the FAT filesystem > > attributes. GetFileSize and GetFileTime fail. > > Ok, so how does msvcrt stat() manage to fill these fields if those > functions fail? Beyond the question to this specific question, I do not like the inconsistency of windows with itself during time and versions. As Mask Hammond said, I think that we should rely on what windows is saying to us as strict as possible. If windows change its behaviour, ok, I do not think that we need to "patch" these behaviour holes. What do you think? Is a mistake to adhere to windows behaviour? Regards, -- .Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Special file "nul" in Windows and os.stat
> As Mask Hammond said, I think that we should rely on what windows is > saying to us as strict as possible. If windows change its behaviour, > ok, I do not think that we need to "patch" these behaviour holes. > > What do you think? Is a mistake to adhere to windows behaviour? We certainly should rely on the Windows behavior. The next question then is: What exactly *is* "the Windows behavior". Windows is not just inconsistent across versions, but apparently so even within a single version. IIUC, GetFileAttributes and FindFirstFile both claim that NUL exists, whereas GetFileAttributesEx claims that it doesn't exist, all in a single version, and all is Windows API. Please understand that Python 2.4 *also* adheres to Windows behavior. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Tracker Issues
ACTIVITY SUMMARY (10/30/07 - 11/06/07) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1323 open (+16) / 11557 closed (+19) / 12880 total (+35) Open issues with patches: 419 Average duration of open issues: 683 days. Median duration of open issues: 785 days. Open Issues Breakdown open 1318 (+16) pending 5 ( +0) Issues Created Or Reopened (35) ___ python 2.4.4 fails on solaris (sun4u sparc SUNW,Sun-Fire-880)10/30/07 http://bugs.python.org/issue1363created theoryno3 os.lstat documentation error 10/31/07 CLOSED http://bugs.python.org/issue1364created billiejoex bytes() constructor 10/31/07 CLOSED http://bugs.python.org/issue1365created tiran popen spawned process may not write to stdout under windows 10/31/07 http://bugs.python.org/issue1366created pmezard mkdir+chdir problem in multiple threads 10/31/07 CLOSED http://bugs.python.org/issue1367created anonyprog Bug tracker link in about tutorial page is wrong 11/01/07 CLOSED http://bugs.python.org/issue1368created ksjohnson Reference to Python24 path in Python 2.5 doc 11/01/07 CLOSED http://bugs.python.org/issue1369created ksjohnson Doc changes left over after mega-merge from trunk11/01/07 CLOSED http://bugs.python.org/issue1370created gvanrossum Two bsddb tests temporarily commented out in py3k branch 11/01/07 CLOSED http://bugs.python.org/issue1371created gvanrossum py3k zlibmodule.c: int overflow in PyZlib_decompress 11/02/07 http://bugs.python.org/issue1372created PeterW patch turn off socket timeout in test_xmlrpc 11/02/07 CLOSED http://bugs.python.org/issue1373created hupp py3k, patch IDLE - minor FormatParagraph bug fix 11/02/07 http://bugs.python.org/issue1374created taleinat patch hotshot IndexError when loading stats11/02/07 http://bugs.python.org/issue1375created ratsberg uu module catches a wrong exception type 11/02/07 CLOSED http://bugs.python.org/issue1376created billiejoex test_import breaks on Linux 11/02/07 http://bugs.python.org/issue1377created gvanrossum py3k fromfd() and dup() for _socket on WIndows11/03/07 http://bugs.python.org/issue1378created roudkerk patch reloading imported modules sometimes fail with 'parent not in sy 11/03/07 CLOSED http://bugs.python.org/issue1379created Paul Pogonyshev py3k, patch fix for test_asynchat and test_asyncore on pep3137 branch11/03/07 CLOSED http://bugs.python.org/issue1380created hupp py3k, patch cmath is numerically unsound 11/03/07 http://bugs.python.org/issue1381created inducer py3k-pep3137: patch for test_ctypes 11/04/07 CLOS
Re: [Python-Dev] Special file "nul" in Windows and os.stat
2007/11/6, "Martin v. Löwis" <[EMAIL PROTECTED]>: > We certainly should rely on the Windows behavior. The next question then > is: What exactly *is* "the Windows behavior". Windows is not just > inconsistent across versions, but apparently so even within a single > version. +1 for QOTW > IIUC, GetFileAttributes and FindFirstFile both claim that NUL exists, > whereas GetFileAttributesEx claims that it doesn't exist, all in a > single version, and all is Windows API. > > Please understand that Python 2.4 *also* adheres to Windows behavior. So, in Py2.4 we adhered to windows behaviour in one way, and in 2.5 we adhere to windows behaviour in other way. As Windows is inconsistant with itself, we got a behaviour change right? If yes, we have three paths to follow... leave 2.5 as is and say that the behaviour change is ok (windows fault), change 2.5 to use the same API than 2.4 and get the same behaviour, or hardwire the behaviour for this set of special files... What do you think we should do? Thanks! -- .Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Special file "nul" in Windows and os.stat
Martin v. Löwis schreef: >>> That doesn't really answer the question, though - you merely state >>> that Python 2.4 calls the CRT, but then my question is still what >>> kernel32 functions are called to have stat on NUL succeed. >>> > Sure - but what does stat then do when passed NUL? AFAIK then it doesn't fill in the size and time fields of the structure (or sets them to a useless/invalid value). (See http://msdn2.microsoft.com/en-us/library/14h5k7ff(vs.71).aspx) > >> GetFileAttributes() doesn't return those, just the FAT filesystem >> attributes. GetFileSize and GetFileTime fail. > > Ok, so how does msvcrt stat() manage to fill these fields if those > functions fail? See above: if stat() (_stat() actually) is called on NUL (or another device), I don't think it does anything useful with these fields. -- The saddest aspect of life right now is that science gathers knowledge faster than society gathers wisdom. -- Isaac Asimov Roel Schroeven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Special file "nul" in Windows and os.stat
> If yes, we have three paths to follow... leave 2.5 as is and say that > the behaviour change is ok (windows fault), change 2.5 to use the same > API than 2.4 and get the same behaviour, or hardwire the behaviour for > this set of special files... As you said before, we should avoid hardwiring things. > What do you think we should do? I think we should try to follow the behavior of 2.4. To do that, we still have to find out what precisely the behavior of 2.4 is (and then perhaps we might decide to not follow it when we know what it is). Unfortunately, it seems that none of us is both capable and has sufficient time to research what the 2.4 behavior actually is; I'd like to emphasize that I think no changes should be made until the behavior is fully understood, which it currently isn't. So I suggest to take no action until somebody comes along who has both the time and the knowledge to resolve the issue. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Special file "nul" in Windows and os.stat
2007/11/6, "Martin v. Löwis" <[EMAIL PROTECTED]>: > Unfortunately, it seems that none of us is both capable and has > sufficient time to research what the 2.4 behavior actually is; > I'd like to emphasize that I think no changes should be made until > the behavior is fully understood, which it currently isn't. > > So I suggest to take no action until somebody comes along who > has both the time and the knowledge to resolve the issue. Yes, I can try things on Windows, but don't have a development enviroment there, so I'll leave the bug open. Thanks everybody! -- .Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Special file "nul" in Windows and os.stat
>> Sure - but what does stat then do when passed NUL? > > AFAIK then it doesn't fill in the size and time fields of the structure > (or sets them to a useless/invalid value). > > (See http://msdn2.microsoft.com/en-us/library/14h5k7ff(vs.71).aspx) What specifically on that page tells me how the fields get filled for NUL? If you are referring to the "if path refers to a device..." sentence: how does it determine that NUL is a device? > See above: if stat() (_stat() actually) is called on NUL (or another > device), I don't think it does anything useful with these fields. You mean, it does nothing documented... AFAICT from the code, it always fills in something. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Special file "nul" in Windows and os.stat
On 06/11/2007, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> > See above: if stat() (_stat() actually) is called on NUL (or another
> > device), I don't think it does anything useful with these fields.
>
> You mean, it does nothing documented... AFAICT from the code, it always
> fills in something.
>From my reading of the CRT source code, _stat() uses FindFirstFile().
This in turn appears to return a valid result on "nul" -
win32api.FindFile, which is a thin wrapper round FindFirstFile etc,
returns
>>> win32api.FindFiles("nul")
[(32, , ,
, 0L, 0L, 0L, 0L, 'nul
', '')]
32 is FILE_ATTRIBUTE_ARCHIVE, the times are the epoch, and everything
else is null.
This is on my machine, using the Windows Server 2003 SP1 CRT source
code. How consistent it is across versions, or anything else, I can't
say :-(
Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Special file "nul" in Windows and os.stat
>>From my reading of the CRT source code, _stat() uses FindFirstFile().
> This in turn appears to return a valid result on "nul" -
> win32api.FindFile, which is a thin wrapper round FindFirstFile etc,
> returns
>
win32api.FindFiles("nul")
> [(32, , ,
> , 0L, 0L, 0L, 0L, 'nul
> ', '')]
Ok. I would still like to avoid calling FindFirstFile *first*, i.e.
"normally" use GetFileAttributesEx first, and only fall back to
FindFirstFile if that gives an error. Such fallback already occurs
if the GetFileAttributesEx error was ERROR_SHARING_VIOLATION.
So is there any good way to determine that the GetFileAttributesError
was caused by using a "reserved" file name. It seems that the error
is ERROR_INVALID_PARAMETER, but that would also be issued if you
have an otherwise-invalid file name (e.g. one including wild cards),
right?
> This is on my machine, using the Windows Server 2003 SP1 CRT source
> code. How consistent it is across versions, or anything else, I can't
> say :-(
Thanks, that helps already.
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Special file "nul" in Windows and os.stat
Martin v. Löwis schreef: >>> Sure - but what does stat then do when passed NUL? >> AFAIK then it doesn't fill in the size and time fields of the structure >> (or sets them to a useless/invalid value). >> >> (See http://msdn2.microsoft.com/en-us/library/14h5k7ff(vs.71).aspx) > > What specifically on that page tells me how the fields get filled > for NUL? If you are referring to the "if path refers to a device..." > sentence: Yes, I was > how does it determine that NUL is a device? I'm not sure. I suppose it just calls GetFileSize() etc. The docs for GetFileSize() say "You cannot use the GetFileSize function with a handle of a nonseeking device such as a pipe or a communications device. To determine the file type for hFile, use the GetFileType function." GetFileType() on its turn returns one of: FILE_TYPE_UNKNOWN: The type of the specified file is unknown. FILE_TYPE_DISK: The specified file is a disk file. FILE_TYPE_CHAR: The specified file is a character file, typically an LPT device or a console. FILE_TYPE_PIPE: The specified file is either a named or anonymous pipe. But I don't know where it is specified which names refer to devices. I tried to query all device names with QueryDosDevice() (with a very simple C program; I can give you the code though I doubt it's useful), but that returns 208 names (on my system); much more than the LPT1, CON, NUL etc. It also includes stuff like, on my system, "Dritek_KB_Filter", "Sony Ericsson T610 Series Bluetooth (TM) Modem #2" etc. I've tried calling _stat() on those names and it returns -1 meaning "File not found". That behavior is clearly different from CON, NUL etc. I thought I've seen the complete list on MSDN some time before, but I can't find it now. >> See above: if stat() (_stat() actually) is called on NUL (or another >> device), I don't think it does anything useful with these fields. > > You mean, it does nothing documented... AFAICT from the code, it always > fills in something. Yes, it returns 0x in the time fields and 0 in the size field (at least on my system, Windows XP SP1). I made another small C++ program to see what _stat() does (again, I can give you the code if you want). With a normal file: $ ./stat stat.cpp gid 0 atime 1194169674 = 2007-11-04T10:47:54 ctime 1194167463 = 2007-11-04T10:11:03 dev 2 ino 0 mode0x81b6 mtime 1194381734 = 2007-11-06T21:42:14 nlink 1 rdev2 size1342 uid 0 With a device: $ ./stat NUL gid 0 atime 4294967295 = invalid time ctime 4294967295 = invalid time dev 2 ino 0 mode0x81b6 mtime 4294967295 = invalid time nlink 1 rdev2 size0 uid 0 (The $ and ./ is because I ran the program from an msys shell) (it says "invalid time" because localtime() returns NULL) In summary, I'm afraid all of this doesn't really help very much... -- The saddest aspect of life right now is that science gathers knowledge faster than society gathers wisdom. -- Isaac Asimov Roel Schroeven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Special file "nul" in Windows and os.stat
Martin v. Löwis wrote:
>> So, the question is what we should do?:
>
> Before this question can be answered, I think we need to fully
> understand what precisely is happening in 2.4, and what precisely
> is happening in 2.5.
>
Seeing this thread drag on was enough to get me to find out what
changed. The implementation of "os.stat" has been changed. In 2.4.4,
_stati64/_wstati64 were called directly, but in 2.5.1, a new pair of
functions were wrote win32_stat/win32_wstat.
_stati64/_wstati64 (as others have noted) fallback onto the use of
FindFirstFile. win32_stat/win32_wstat use some functions called
Py_GetFileAttributesExA/Py_GetFileAttributesExW which ultimately use
GetFileAttributesA/GetFileAttributesW.
The change to this implementation is r42230 with the Misc/NEWS comment
saying:
- Use Win32 API to implement os.stat/fstat. As a result, subsecond
timestamps are reported, the limit on path name lengths is
removed, and stat reports WindowsError now (instead of OSError).
As to the specifics of what FindFirstFile* does with the values, I
tested this quickly with ctypes on 'nul' (or any of the other special
files):
cAlternameFileName:
cFileName: nul
dwFileAttributes: 32
dwReserved0: 0
dwReserved1: 0
ftCreationTime: (dwLowDateTime: 0, dwHighDateTime: 0)
ftLastAccessTime: (dwLowDateTime: 0, dwHighDateTime: 0)
ftLastWriteTime: (dwLowDateTime: 0, dwHighDateTime: 0)
nFileSizeHigh: 0
nFileSizeLow: 0
In order to keep the higher accuracy timestamps for normal files and to
maintain the old behavior, my recommendation would be that the existing
implementation of win32_stat/win32_wstat be extended to use
FindFileFirst if GetFileAttributes* fails. I would be willing to do the
legwork for such a patch if everyone agrees this is the appropriate
solution.
* As an aside, Martin, I find the argument that "hard-wiring is bad" to
be against what is actually occurring in the posixmodule. For that
matter, the S_IFEXEC flag is hardwired to path in (*.bat, *.cmd, *.exe,
*.com) despite the fact that the platform says it is really anything in
the list of os.getenv('PATHEXT'), but I suppose that is a bug for
another day.
-Scott
--
Scott Dial
[EMAIL PROTECTED]
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Special file "nul" in Windows and os.stat
> In order to keep the higher accuracy timestamps for normal files and to
> maintain the old behavior, my recommendation would be that the existing
> implementation of win32_stat/win32_wstat be extended to use
> FindFileFirst if GetFileAttributes* fails. I would be willing to do the
> legwork for such a patch if everyone agrees this is the appropriate
> solution.
That, in general, might be wrong. os.stat("*.txt") should fail, even
though FindFirstFile will succeed when passed that string.
That was my primary motivation for not using FindFirstFile by default:
it may succeed even if the string being passed does not denote a file
name.
> * As an aside, Martin, I find the argument that "hard-wiring is bad" to
> be against what is actually occurring in the posixmodule. For that
> matter, the S_IFEXEC flag is hardwired to path in (*.bat, *.cmd, *.exe,
> *.com) despite the fact that the platform says it is really anything in
> the list of os.getenv('PATHEXT'), but I suppose that is a bug for
> another day.
No. See the source of the C library - the algorithm Python uses now is
(or should be) the same as the one of the C library. Of course, you
may argue that then msvcrt has the same bug.
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Special file "nul" in Windows and os.stat
""Martin v. Löwis"" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] |> If yes, we have three paths to follow... leave 2.5 as is and say that | > the behaviour change is ok (windows fault), change 2.5 to use the same | > API than 2.4 and get the same behaviour, or hardwire the behaviour for | > this set of special files... | | As you said before, we should avoid hardwiring things. | | > What do you think we should do? | | I think we should try to follow the behavior of 2.4. To do that, we | still have to find out what precisely the behavior of 2.4 is (and | then perhaps we might decide to not follow it when we know what it | is). | | Unfortunately, it seems that none of us is both capable and has | sufficient time to research what the 2.4 behavior actually is; | I'd like to emphasize that I think no changes should be made until | the behavior is fully understood, which it currently isn't. | | So I suggest to take no action until somebody comes along who | has both the time and the knowledge to resolve the issue. Perhaps a note should be added to the docs that the 'existence' of 'nul', etc, is inconsistent in Windows and hence, at present, in Python. In part, it seems to me, that anyone doing Windows-specific stuff should decide for themselves whether 'nul' exists for their purposes or not. Hence the problem should only arise when receiving a filename from an external source. Maybe the special Windows module should have an 'isdevice' function if it does not already to test such. I agree that there are more generally useful things for you to do. Windows is sometimes a maddening platform to work on. tjr ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Request for inclusion in 2.5.2 (5-for-1)
Mike, thanks for reviewing the bugs and for creating a patch. Good work! On Nov 2, 2007 2:23 AM, Nick Coghlan <[EMAIL PROTECTED]> wrote: > Mike Klaas wrote: > > http://bugs.python.org/issue1705170: reproduced. Conjecture as to > > why it is occurring, but I don't know the guts well enough to propose > > a decent fix. > > I've fixed this on the trunk (I'm afraid I have no opinion on the patch > you're interested in though. > > Neal - where does the 2.5 branch stand at the moment? This would be a > simple fix to slip into 2.5.2 if there's still time. I'm not sure if this was before my last status mail. We are hoping to get a release candidate out in a week or two. I agree with Guido that this isn't a bugfix and therefore shouldn't be backported to 2.5. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Special file "nul" in Windows and os.stat
Martin v. Löwis wrote:
>> In order to keep the higher accuracy timestamps for normal files and to
>> maintain the old behavior, my recommendation would be that the existing
>> implementation of win32_stat/win32_wstat be extended to use
>> FindFileFirst if GetFileAttributes* fails. I would be willing to do the
>> legwork for such a patch if everyone agrees this is the appropriate
>> solution.
>
> That, in general, might be wrong. os.stat("*.txt") should fail, even
> though FindFirstFile will succeed when passed that string.
>
Sorry, I meant to imply that we would guard FindFirstFile in the same
manner that _stati64 and friends do already (using strpbrk/wcspbrk to
search for "?*" in the path). At that point, you have essentially
duplicated the CRT code with the added improvement of using
GetFileAttributes* to retrieve the high-precision timestamps. So, I
think my opinion has changed now to say: first, use GetFileAttributes*,
and if that fails use _stati64/_wstati64.
> That was my primary motivation for not using FindFirstFile by default:
> it may succeed even if the string being passed does not denote a file
> name.
While I understand your reasoning, I thought we were letting the
platform decide what are and are not files. This bug appeared because we
are imposing our own notion of what a file is or is not, probably only
by ignorance of the differences of GetFileAttribute* and FindFirstFile.
So, my suggestion is basically a compromise of keeping higher precision
timestamps for paths where GetFileAttribute* works and retaining the old
behavior for all others.
>
>> * As an aside, Martin, I find the argument that "hard-wiring is bad" to
>> be against what is actually occurring in the posixmodule. For that
>> matter, the S_IFEXEC flag is hardwired to path in (*.bat, *.cmd, *.exe,
>> *.com) despite the fact that the platform says it is really anything in
>> the list of os.getenv('PATHEXT'), but I suppose that is a bug for
>> another day.
>
> No. See the source of the C library - the algorithm Python uses now is
> (or should be) the same as the one of the C library. Of course, you
> may argue that then msvcrt has the same bug.
>
I concede and apologies, I didn't read the code for converting
attributes to mode flags. I don't imagine (m)any people care about this
flag on the win32 platform anyways.
-Scott
--
Scott Dial
[EMAIL PROTECTED]
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
