[issue18314] Have os.unlink remove junction points
New submission from Kim Gräsman: os.unlink currently raises a WindowsError (Access Denied) if I attempt to unlink an NTFS junction point. It looks trivial to allow Py_DeleteFileW [1] to remove junction points as well as proper symbolic links, as far as I can tell. For example, the ntfslink-python library [2] only checks if both FILE_ATTRIBUTE_DIRECTORY and FILE_ATTRIBUTE_REPARSE_POINT are set. RemoveDirectoryW is documented to handle junction points transparently, so it should just be a matter of passing the path on if it's a junction point or a symbolic link. My motivation for this is that I have used external tools to create junction points, and am now switching to symbolic links. When deleting a directory, I need to do: try: os.unlink(link_path) except WindowsError as detail: # BACKWARD COMPATIBILITY HACK if detail.winerror == 5: _delete_junction_point(link_path) else: raise which is a little funky. It seems like os.unlink semantics work just as well for junction points, even if they can't be created with os.symlink. Love it/hate it? [1] http://hg.python.org/cpython/file/44f455e6163d/Modules/posixmodule.c#l4105 [2] https://github.com/Juntalis/ntfslink-python/blob/2f6ff903f9b22942de8aa93a32a3d817124f359e/ntfslink/internals/__init__.py#L32 -- components: Windows messages: 191945 nosy: Kim.Gräsman priority: normal severity: normal status: open title: Have os.unlink remove junction points type: behavior versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: Also, I believe the reason os.unlink raises "access denied" is because a junction point does not currently qualify as a directory && link, so its path is passed directly to DeleteFileW, which in turn refuses to delete a directory. -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: This comment outlines how to tell junction points from other mount points: http://www.codeproject.com/Articles/21202/Reparse-Points-in-Vista?msg=3651130#xx3651130xx This should port straight into Py_DeleteFileW. Would anyone be interested in a patch? -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: Victor, Junction points are like links between directories only. They've been around since the NTFS that came with Windows 2000, but integration with OS tools has been generally poor (e.g. Explorer wouldn't see the difference between a junction point and a regular folder.) As of Windows 7, this works better. -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: Thanks for pushing this forward! Do you have links to the failing bots I could take a look at? -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: Thanks! At first I suspected 32 vs 64 bit, but the failing bots cover both... One thing that stands out to me as risky is the memcmp() against "\\??\\", which could overrun a short src_path buffer. But I don't think that would fail here. I must have made some mistake with the REPARSE_DATA_BUFFER, but I can't see anything off hand. What are our debugging options? -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: Aha, that might cause trouble. I think you should add a memset() to sero out the newly allocated buffer also, I think I may have used calloc to be able to assume it was initialized with zeros. -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: Sorry, that wasn't clear. I mean if you change allocator _from_ calloc, make sure the buffer is zeroed out after allocation. -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: By the way, is PyMem_RawMalloc/PyMem_RawFree preferred for memory allocation across the board? If so, I can just prepare a new patch for you with that changed, zero-initialization in place and the prefix-overrun fixed. I might get to it tonight. -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: > Also, since the setup of the reparse header is such an underdocumented > nightmare, please add as much commentary as possible around the choice > of allocations & offsets. I'll try. It might turn into a novel. > (BTW I'm not convinced that the PyMem change was the problem since > the PyMem_Raw* functions simply hand off to malloc/free unless > there's a custom allocator. > Unless of course the missing memset-0 was significant here). I think it might be, there was a message in the log that DeviceIoControl failed: stty: standard input: Inappropriate ioctl for device That could be attributed to garbage in the buffer. I'll be back with a revised patch, and we can work from there. Thanks for your help! -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: Here's a new attempt, please let me know if this works out better. Changes: - Switched to CRT string functions (wcsncmp, wcscpy) instead of Windows lstrxxxW. There was no lstrncmpW. - Switched to PyMem_Raw(Malloc|Free) and added explicit memset after allocation - Better error handling (check arguments for NULL, check memory allocation) - Fix possible overrun when checking if src_path starts with "\??\" - Extensive commentary to describe the buffer sizing Hope this works out better. I already have ideas for improvements, but I think we can try to get this in place first. -- Added file: http://bugs.python.org/file35095/unlink_junctions_r2.patch ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: Thanks for helping me land this! eryksun: interesting, thanks! I seem to remember having problems without the \??\ prefix, but it could have been something else causing it (filling the buffer to DeviceIoControl's satisfaction was... challenging.) I have some ideas for small improvements, and I could try to fold the removal of \??\ into that. Do I just open a new enhancement issue? -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: _delete_junction_point currently shells out to a command-line tool, junction.exe, from SysInternals. That ran fine on XP. As I understand it, RemoveDirectoryW on XP also takes care of junction points, but I'll find a machine to verify. -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: Attached is a patch that considers directory symlinks and junction points equivalent. I'm struggling with the test -- would it be acceptable to only run this test on platforms that have mklink /j (Windows Vista and higher)? I've looked at programmatic means of creating junction points, but it involves enough Win32 interop to make it a candidate for a module all by itself (it's REALLY messy.) Any ideas? Thanks! -- keywords: +patch Added file: http://bugs.python.org/file31915/unlink_junction.patch ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: Gentle ping. -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: I didn't know about _winapi; looks like a good place! It looks like it exposes the Windows API pretty faithfully, but the junction point stuff feels like it would be better represented as a simple _winapi.CreateJunctionPoint(source, target) rather than attempting to expose DeviceIoControl and associated structures. I'll try to cook up a patch and we can argue about details based on that :-) Thanks! -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: I really needed the well-wishing with regard to buffer sizing :-) Here's a patch for a couple of fronts: - Teach os.unlink about junction points - Introduce _winapi.CreateJunction - Introduce a new test suite in test_os.py for junction points I pulled the definition of _REPARSE_DATA_BUFFER out into a new header called winreparse.h. I'd appreciate critical review of _winapi.CreateJunction to make sure I haven't missed anything. I'm not familiar with the Python/C interop idioms, so I might have missed something wrt arguments/return values handling. Happy new year! -- Added file: http://bugs.python.org/file33288/unlink-junctions.patch ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: Thanks! There's another thing I would appreciate having somebody else test: creating and removing junctions in a non-elevated prompt. I haven't been able to, my IT department has trouble understanding the value of least-privilege. -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: ping -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18314] Have os.unlink remove junction points
Kim Gräsman added the comment: ping -- ___ Python tracker <http://bugs.python.org/issue18314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17233] http.client header debug output format
New submission from Kim Gräsman: Python 3.2.3 on 64-bit Windows 7. When I set debuglevel on HTTPConnection to 1, the output seems jumbled, and I'm having trouble interpreting it. Attached is a full, anonymized log from a conversation I was troubleshooting. Here's an excerpt: send: b'GET /a HTTP/1.1\r\nHost: localhost:55380\r\nAccept-Encoding: identity\r\n\r\n' reply: 'HTTP/1.1 503 Service unavailable\r\n' header: Connection header: Content-Length header: Content-Type header: Date send: b'GET /a HTTP/1.1\r\nHost: localhost:55380\r\nAccept-Encoding: identity\r\n\r\n' reply: 'HTTP/1.1 503 Service unavailable\r\n' - Does line 3, starting with header:, show headers for the request or response? I'm guessing response, but it didn't occur to me until just now, after a full day of looking at it. - It would be nice if the header dump showed both header key and value - There's a trailing "send:" at the end of the header: line, shouldn't that be on its own line? Overall, not printing a newline after these debug statements makes it really hard to combine with other print debugging, as unrelated prints show up at the end of every line of httpclient's debug output. I find "header" a bit confusing as a prefix, as it doesn't say whether it's request or response headers. Maybe change the prefix to "response-headers:" and also add "request-headers:", even if those are visible in the "send:" lines? Alternatively, why not show the response headers in the "reply:" dump? Would you consider patches to address these concerns? Thank you! -- components: Library (Lib) files: httpclient.debuglevel.txt messages: 182367 nosy: Kim.Gräsman priority: normal severity: normal status: open title: http.client header debug output format type: behavior versions: Python 3.2 Added file: http://bugs.python.org/file29120/httpclient.debuglevel.txt ___ Python tracker <http://bugs.python.org/issue17233> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17233] http.client header debug output format
Kim Gräsman added the comment: Thanks for your response! > I agree that send: and reply: should be formatted the same, > so the reply: line should include the headers *with* the > values. OK, yeah, that would avoid having to specify request/response for headers as well, I think. > The current header line seems to be truncated after Date, > without even its ':', It doesn't look like there are any trailing colons on header names. Rather, all header names have a "header: " prefix. As you can see, it's pretty hard to parse :-) > and it is definitely missing a newline. Glad we agree! > Are you claiming that there are other missing newlines? It's not visible in the attached log, but if you add print() statements in a script using http.client, the output usually ends up at the end of one of http.client's log lines. So I think every log output is missing a newline. > I don't understand the lines with > send: bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00\x00') We're sending binary data in our HTTP body, so this is the actual content. For some reason, though, the fact that it's binary seems to force it onto its own line. > Patches are good, but you might wait a few days > for other comments as I am definitely not an http expert. OK, thanks again! -- ___ Python tracker <http://bugs.python.org/issue17233> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17233] http.client header debug output format
Kim Gräsman added the comment: Thanks! I filed this so long ago that I'd forgotten about it. I ran the code in question with your patch manually hacked into my Python installation (3.4.1) and output is much easier to digest now. I don't have a strong opinion on key names -- I haven't needed this code path since I filed the bug :) -- it's probably fine as-is if nobody wants to take a stab at it. Anyway, thanks for this, I can confirm that it solves my problem, and it would be nice if it went into trunk. -- ___ Python tracker <http://bugs.python.org/issue17233> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
