[Python-Dev] str() on memoryview of bytearray failing on py3k
Hi all, I'm not sure if the following is a bug I should report or simply an artifact of the implementation and/or simply the way things work on py3k: % py30 -c "str(memoryview(bytearray((1,2,3" Traceback (most recent call last): File "", line 1, in TypeError: __str__ returned non-string (type bytes) Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] str() on memoryview of bytearray failing on py3k
On 8/02/2009 10:21 AM, Antoine Pitrou wrote: Mark Hammond gmail.com> writes: I'm not sure if the following is a bug I should report or simply an artifact of the implementation and/or simply the way things work on py3k: [...] It's a bug. http://bugs.python.org/issue5182 Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
On 11/03/2009 1:55 PM, Guido van Rossum wrote: On Tue, Mar 10, 2009 at 7:45 PM, Christian Heimes wrote: Antoine Pitrou wrote: Christian Heimes cheimes.de> writes: ... Let's not think too Unix-specific. If we add such an API it should do something on Windows too -- the app shouldn't have to test for the presence of the API. (And thus the API probably shouldn't be called fsync.) This is especially true given Windows has recently introduced a transactional API for NTFS. Although the tone is - err - gushing - it (a) should give some information on what is available, and (b) was high on my google search list http://msdn.microsoft.com/en-us/magazine/cc163388.aspx http://msdn.microsoft.com/en-us/library/aa363764(VS.85).aspx Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On 25/03/2009 6:51 AM, "Martin v. Löwis" wrote: Mike Driscoll did some work providing Windows installers for various Python packages and extension modules, and people were amused that he provided executable installers for pure Python libraries. But I saw that as a sensible decision, since it meant that Windows users (and even Windows Python application developers) used the same mechanism to install everything. The Windows story is indeed sad, as none of the Windows packaging formats provides support for dependencies (MSI has some support, but as far as I understand it, it's pretty useless). But yes, for Windows, you want .exe or .msi installers, not something proprietary. Isn't this discussion slightly at cross-purposes? * py2exe doesn't create *installers*, just the target application. It (poorly) manages dependencies on the build machine. There doesn't seem to be a need for runtime dependency management in this tool. * The creation of an *installer* is something quite different. An installer for a py2exe based tool also doesn't need dependency management. An installer for a pure-python package that made no attempt to bundle dependencies might be nice, but I don't quite see how that falls outside the scope of distutils/setuptools/etc. In other words, I don't see why the installer can't bootstrap the 'normal' dependency management which would be used if the package was installed any other way or on other platforms. * distutils already has the ability to create Windows installer executables for pure-python apps/libs. I agree it would be nice if it was an MSI but that is an implementation detail rather than implementation requirement. How were Mike's packages fundamentally different than that? Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Test failures under Windows?
On 25/03/2009 7:34 AM, "Martin v. Löwis" wrote: I don't quite remember the -n flag, but I believe that Kristjan just removed all that stuff, ie. there is now no way to block CRT assertions anymore. I wasn't paying close attention, so maybe there's some other mechanism in place at this point? Unfortunately, not. There was a long debate, I lost, and Kristjan removed all the changes to support this setup, claiming that it is good to get assertions :-( Note that I previously opened http://bugs.python.org/issue5116 with a patch to enable this to be controlled from Python via the msvcrt module. This would enable the test suite to disable assertions for its entire run. Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Test failures under Windows?
On 25/03/2009 10:05 AM, David Bolen wrote: Kristján Valur Jónsson writes: Now, I know that this msvc behaviour can be disabled, but it was decided that it was not appropriate to meddle with runtime flags of the whole process for python. I must have missed that discussion, but I can't see what the problem is if such an override only occurs during the buildbot non-interactive test runs? Isn't that what using -n in the buildbot test.bat did? The issue was that Python unconditionally changed the behaviour of the CRT, not only during the test suite. I'm inclined to agree that Python itself (ie, pythonxx.dll on windows) has no business changing these flags, but it should be up to the application - ie, either python.exe itself, or exposed to the python app - the latter of which is what my patch offers. Note however that my patch doesn't cause any .py code to actually change the flags, just offers the ability for them to do so. Curt's suggestion of redirecting to a file is better still - I'll look at tweaking my patch to also offer that capability... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Test failures under Windows?
On 25/03/2009 7:58 PM, David Bolen wrote: Mark Hammond writes: The issue was that Python unconditionally changed the behaviour of the CRT, not only during the test suite. Hmm... I was more or less referring to the state of the py3k tree as of, say, r57823 back in 2007. I was referring to the issue described in http://bugs.python.org/issue4804, which notes that exceptions.c unconditionally called: _CrtSetReportMode(_CRT_ASSERT, 0); which has the effect of disabling crt assertions that would otherwise result in a modal dialog. That bug remains open though, so maybe I'm confused about the problem which started this discussion... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Test failures under Windows?
I'm going to poke my contacts at Microsoft and ask them if there is a way to disable popups like this for a process that runs unattended and/or is running as a windows service. There is, and Curt pointed out one strategy for achieving this without losing the checks it provides... > Curt's suggestion of redirecting to a file is better still - I'll look > at tweaking my patch to also offer that capability... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] splitting out bdist_* (was: interminable 'setuptools' thread)
On 28/03/2009 7:49 AM, gl...@divmod.com wrote: Perhaps bdist_wininst/_msi could be donated to the py2exe project if they would be willing to maintain it, and the new project for _deb and _rpm could be called "py2packman" or something. As mentioned, it isn't really a natural fit - but regardless, py2exe is struggling to maintain itself at the moment... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unladen-Swallow: A faster python
On 29/03/2009 1:41 AM, andrew cooke wrote: Mark Hammond wrote: On 28/03/2009 9:50 PM, andrew cooke wrote: Tim Roberts wrote: [...] IronPython has certainly shown that Python can be successfully implemented in a JIT compiled VM in a performant way, but it has issues running C extension modules. I'll be curious to see where this project goes. given the comments on python-dev i wonder if this is the first indication that python is going to split into separate implementations for windows and unix (via .net and llvm, respectively)? What comments are they? There is no indication that unladen-swallow is fundamentally broken for Windows, just temporarily broken due to the lack of windows developers/contributors... the comments you are referring to - that windows is not a priority, that they currently have no testing on windows machines, etc. i quote, for example: "None of the three of us have Windows machines, nor do we particularly want to acquire them :), and Windows support isn't going to be a big priority." Saying-no-to-fud ly, which part of "i wonder" don't you understand? i'm not saying it is true, i'm just discussing the possibility. Fair enough - although its reasonable for people to look for some indication of what you are saying to have some basis in reality, surely? > i am getting a little tired of people here acting so defensively... Which part of my reply did you think was defensive? I wasn't suggesting such a split would necessarily be a bad idea, just that there was no evidence of it. IMO discussing such a split would create a huge amount of FUD, generating a huge amount of wasted energy if it wasn't true. > i'm discussing a programming language, not the size of your dick. Wow, talk about jumping to conclusions :) Is there something you feel the need to get off your chest? Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial?
On 6/04/2009 12:13 AM, Dirkjan Ochtman wrote: I have a stab at an author map at http://dirkjan.ochtman.nl/author-map. Could use some review, but it seems like a good start. Just to be clear, what input would you like on that map? I'm listed twice: mark.hammond = Mark Hammond mhammond = Mark Hammond but that email address isn't the address normally associated with any checkins I make, nor the address in the comments of the ssh keys I use (which is mhamm...@skippinet.com.au) The addresses given are valid though, so I'm not sure what kind of review or feedback you are after. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dropping bytes "support" in json
[Dropping email sig] On 11/04/2009 1:06 PM, "Martin v. Löwis" wrote: However, I really think that this question cannot be answered by reading the RFC. It should be answered by verifying how people use the json library in 2.x. In the absence of anything more formal, here are 2 anecdotes: * The python-twitter package seems to: - Use dumps() mainly to get string objects. It uses it both for __str__, and for an API called 'AsJsonString' - the intent of this seems to be to provide strings for the consumer of the twitter API - its not clear how such consumers would use them. Note that this API doesn't seem to need to 'write' json objects, else I suspect they would then be expecting dumps to return bytes to put on the wire. They expect loads to accept the bytes they are reading directly off the wire. * couchdb's wrappers use these functions purely as bytes - they are either decoding an application/json object from the bits they read, or they are encoding it to use directly in the body of a request (or even directly in the URL of the request!) I find myself conflicted. On one hand I believe the most common use of json will be to exchange data with something inherently byte-based. On the other hand though, json itself seems to be naturally "stringy" and the most natural interface for a casual user would be strings. I'm personally leaning slightly towards strings, putting the burden on bytes-users of json to explicitly use the appropriate encoding, even in cases where it *must* be utf8. On the other hand, I'm too lazy to dig back through this large thread, but I seem to recall a suggestion that using bytes would be significantly faster. If that is true, I'd be happy to settle for bytes as I believe the most common *actual* use of json will be via things like the twitter and couch libraries - and may even be a key bottleneck for such libraries - so people will not be directly exposed to its interface... Mark Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dropping bytes "support" in json
On 11/04/2009 6:12 PM, Antoine Pitrou wrote: Martin v. Löwis v.loewis.de> writes: Not sure whether it would be *significantly* faster, but yes, Bob wrote an accelerator for parsing out of a byte string to make it really fast; IIRC, he claims that it is faster than pickling. Isn't premature optimization the root of all evil? Besides, the fact that many values in a typical JSON object will be strings, and must be encoded from/decoded to unicode objects in py3k, suggests that accepting/outputting unicode as default is the laziest (i.e. the best) choice performance-wise. I don't see it as premature optimization, but rather trying to ensure the interface/api best suits the actual use cases. But you don't have to trust me: look at the quick numbers I've posted. The py3k version (in the str-only incarnation I've proposed) is sometimes actually faster than the trunk version: http://mail.python.org/pipermail/python-dev/2009-April/088498.html But if all *actual* use-cases involve moving to and from utf8 encoded bytes, I'm not sure that little example is particularly useful. In those use-cases, I'd be surprised if there wasn't significant time and space benefits in not asking apps to use an 'intermediate' string object before getting the bytes they need, particularly when the payload may be a significant size. Assuming the above is all true, I'd see choosing bytes less as a premature optimization and more a design choice which best supports actual use. So to my mind the only real question is whether the above *is* true, or if there are common use-cases which don't involve utf8-off/on-the-wire... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposed: add support for UNC paths to all functions in ntpath
Larry Hastings wrote: Counting the votes for http://bugs.python.org/issue5799 : +1 from Mark Hammond (via private mail) +1 from Paul Moore (via the tracker) +1 from Tim Golden (in Python-ideas, though what he literally said was "I'm up for it") +1 from Michael Foord +1 from Eric Smith There have been no other votes. Is that enough consensus for it to go in? If so, are there any core developers who could help me get it in before the 3.1 feature freeze? The patch should be in good shape; it has unit tests and updated documentation. I've taken the liberty of explicitly CCing Martin just incase he missed the thread with all the noise regarding PEP383. If there are no objections from Martin or anyone else here, please feel free to assign it to me (and mail if I haven't taken action by the day before the beta freeze...) Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposed: add support for UNC paths to all functions in ntpath
Eric Smith wrote: Mark: I've reviewed this and it looks okay to me. Thanks Eric - I've now applied that patch. As you mentioned in a followup to the bug: | Thanks for looking at this, Mark. If we could only assign issues to | Python 3.2 and 3.3 to change the pending deprecation warning to a real | one, and to remove the function entirely, we'd be all set! I'm always | worried we'll forget these things. (for reference; the patch introduces a PendingDeprecationWarning for ntpath.uncpath) The bug tracker doesn't have these future versions available yet - is there some other way these things should be tracked? I fear simply opening a new bug without a reasonable 'trigger' will linger way beyond the next few versions... Thanks, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial, linefeeds, and Visual Studio
Paul Moore wrote: 2009/6/5 Nick Coghlan : Dirkjan Ochtman wrote: Essentially, pcbuild.sln is a binary file, and should be treated as such. Maybe it's an error in the Subversion setup that it's treated as text at all... Yes, it certainly seems that way. Except it isn't a binary file - it's a text file with CRLF line endings. Why would we throw away the chance to get meaningful diffs when Subversion can easily get the line endings correct? Fair point - I hadn't thought of it in that context. I wonder if Mercurial can treat it as a specialised form of binary file with custom diff/merge behaviour (which just happens to be "treat as text")? Of course, if that also requires client-side configuration, it's no better solution than win32text. I've a feeling I've seen discussions of versioned metadata (things like file properties) in Mercurial at some point in the past, but my Google skills are failing me at the moment... I've looked into this recently. IIUC: * There has been some limited support expressed for having win32text look in a normal versioned file for hints about the current repo. * Once concern here was the fact that such config files are somewhat 'untrusted', but the hg config parsing code has recently been reimplemented to support the concept of 'trusted' versus 'untrusted' options. I'm not sure what this means in practice though. * There isn't currently much activity on win32text. None of the core hg devs seem to use Windows, so while they are receptive to Windows patches, it might be necessary to be quite noisy to get attention. I recently submitted some changes to the filtering for Windows which were accepted without any angst, and the ability to use a versioned file for per-repo rules is something I'd like too. I believe that if a few Windows users got together and agreed on both semantics and implementation we could get such an enhancement landed in the core of hg... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
On 3/07/2009 6:42 AM, Dirkjan Ochtman wrote: In response to some rumblings on python-committers and just to request more feedback, a progress report. I know it's long, I've tried to put to keep it concise and chunked, though. Although this has come up in the past, I don't recall a resolution. What is your plan to handle svn:eol-style? We have some files in the tree which need that support and it isn't clear to me how that would work with the existing win32text extension provided with current mercurial releases. (I've an outstanding patch to hg which should address some of these issues, but without the 'rules' being versioned I fear that would still fall short.) Even more generally, how will you suggest Windows users work? Will local files, in general, have windows line endings or unix? If the latter, will there be hooks in-place to prevent editors on Windows 'accidently' mixing eol styles? If so, this cycles back to the first question - how would we know which files get treated that way? Thanks, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
On 3/07/2009 9:28 PM, Dirkjan Ochtman wrote: On Fri, Jul 3, 2009 at 01:01, Mark Hammond wrote: Although this has come up in the past, I don't recall a resolution. What is your plan to handle svn:eol-style? We have some files in the tree which need that support and it isn't clear to me how that would work with the existing win32text extension provided with current mercurial releases. (I've an outstanding patch to hg which should address some of these issues, but without the 'rules' being versioned I fear that would still fall short.) What files would need what? Are there any files that really need to be \r\n on Windows and \n on Unix (and possibly \r on Mac)? I remember one file was discussed separately, but I think the outcome there was that it could just always be \r\n (since it wasn't used at all on non-Windows platforms). Anyway, knowing specific requirements (or where to find them) would help here. Even more generally, how will you suggest Windows users work? Will local files, in general, have windows line endings or unix? If the latter, will there be hooks in-place to prevent editors on Windows 'accidently' mixing eol styles? If so, this cycles back to the first question - how would we know which files get treated that way? There will be a server-side hook to check whitespace. People will also be able to install it for commit-time. I think just using \n by default everywhere is a good default (though I almost always use Windows client machine, I do all nearly all of my development through a terminal on several Linux boxen), except where it isn't. So we must work without effective EOL support? I fear we will end up like the mozilla hg repo with some files in windows line endings and some with linux. While my editing tools are good enough to preserve existing EOL styles, I've found myself accidentally checking in new \r\n terminated files in a repo which otherwise uses \n line endings. IMO, SVN's EOL support was better than no EOL support. People who want to use can set up the win32text stuff to get \r\n on Windows if they feel they need that -- we can provide information about that in the dev FAQ (although it would be nice if someone else who was more familiar with it -- like yourself! -- would write it). This is exactly why I was asking for your advice - I can't work out how to work effectively with win32text as it stands myself, so remain stuck accidently checking in files with inappropriate line endings and stuck working out how to move pywin32's CVS repo with abandoning the very primitive EOL safety it offers... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
On 4/07/2009 12:04 AM, Nick Coghlan wrote: However, I expect that would still be painful to work with for Windows developers, even if it prevented any line ending problems from making their way into the main repository. I believe that is where the win32text extensions can help. Looking at the Wiki page for win32text [1], I believe it would be a matter of configuring the extension to encode and decode all files with the extensions: *.py *.pyw *.h *.c *.in *.rst *.asdl That said, I don't see a way to tell win32text to also translate files which don't have an extension at all (e.g. NEWS or ACKS), and there doesn't seem to be a way to tell it to *skip* files matching certain patterns (if there was, we could just tell it to ignore extensions like .bat, .sln, .vcproj, .vsprops, .dps, .dsw, .wse, .ico, .bmp and convert everything else) The big problem I have with win32text is that the rules are not versioned, meaning we will rely on every (windows only?) developer manually enabling the win32text extension, then manually adding these rules to an unversioned hgrc file. Should the rules change over time, we will need to communicate that to every developer. Contrast this with, say, SVN, where the rule is a versioned property on the file. Further, if a repo *already* has mixed line endings and the file doesn't appear in the rules, win32text makes *lots* of noise regarding existing line endings and to "review your rules before you commit." In effect, it seems to me that win32text will only work effectively if *all* developers using a repo are using it, or none are, or other steps have been put in place to ensure \r\n line endings in the repo are otherwise impossible. I've completely failed to use win32text on the mozilla hg repo for exactly this reason. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
On 3/07/2009 11:43 PM, Dirkjan Ochtman wrote: On Fri, Jul 3, 2009 at 15:31, Mark Hammond wrote: So we must work without effective EOL support? I fear we will end up like the mozilla hg repo with some files in windows line endings and some with linux. While my editing tools are good enough to preserve existing EOL styles, I've found myself accidentally checking in new \r\n terminated files in a repo which otherwise uses \n line endings. IMO, SVN's EOL support was better than no EOL support. This is why we'll have hooks -- to prevent you from pushing changesets with inappropriate, to say the least, and, if you're willing to do a little bit of extra work, to prevent you from committing them. This is exactly why I was asking for your advice - I can't work out how to work effectively with win32text as it stands myself, so remain stuck accidently checking in files with inappropriate line endings and stuck working out how to move pywin32's CVS repo with abandoning the very primitive EOL safety it offers... As long as the difference between \r\n- and \n-based files is clear and can be reasoned about, I don't see why having some of both (I'm assuming an overwhelming majority will have one, and only a few the other) is a big problem. But feel free to enlighten me! I'm surprised it isn't obvious. Think about it this way: CVS had basic EOL support and SVN gave slightly better support, and we leveraged this support in both those tools. The end result is a very clean repo with consistent line endings and a distinct lack of developer friction between the platforms. Your position seems to be that we *never* needed EOL conversion, especially in SVN which offered decent hooks, and that having a source repo with a mismatch and unpredictable line endings isn't really a problem at all. Even just at face value, it seems clear we have taken full advantage of that feature for around 1.5 decades, so to suggest we never actually needed it, or at least don't need it now for some reason, seems slightly disingenuous... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
On 3/07/2009 11:43 PM, Dirkjan Ochtman wrote: As long as the difference between \r\n- and \n-based files is clear and can be reasoned about, I don't see why having some of both (I'm assuming an overwhelming majority will have one, and only a few the other) is a big problem. But feel free to enlighten me! In that case, and given that I expect more Windows users to be clueless about EOL issues than users of other operating systems, I propose we check in all files initially with Windows line endings Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
On 4/07/2009 11:20 AM, Nick Coghlan wrote: I didn't realise Mercurial didn't have a way for a repository to provide versioned extension settings, but in this case I think using our own server side hook can deal with the problem (either adding it to the existing whitespace hook that checks for tab/space sanity in Python files, or else as a separate hook that just checks line endings). Such a hook should be able to base the determination of which files it checks for \r\n line endings on a normal versioned file in the repository. For everything that *isn't* mentioned in that file, the hook would check that it had \n line endings. I believe that would address both of your concerns: 1. A server side hook means that the rules are enforced on everyone regardless of whether they use win32text or not (e.g no Linux developers corrupting a .dsp file by opening it and saving it with \n line endings) 2. The server side hooks rules will be versioned, allowing people to copy them to their hgrc file so that win32text will correctly handle the line endings for them. That sounds like it should work - presumably mention of this should go into the PEP. (While that is great for Python, it is a shame I probably can't use the same strategy for other hg repos I care about as I don't have access to server side hooks - but that if off-topic for this list.) Also, on another read of the win32text instructions, I believe the configuration actually *does* support negative filters, it just doesn't explain them very well (it only mentions them in the context of overriding default settings in the config file for a specific repository). I think a .hgrc file along the following lines might do what we want (I'm not developing on Windows though, so I haven't actually tested it): This *appears* to be correct at first glance, but in practice it doesn't interact well with wildcards specifications - particularly '**=cleverencode'. I started a thread on the ML and submitted a patch a few months ago to fix this and it was accepted, but sadly it seemed to have dropped off the queue somewhere. The patch now conflicts, and I've promised to resubmit it when I find time. But even with that in place it doesn't address the more general problem that when only *some* developers use the extension - mixed mode files are still very possible, at which point win32text starts reporting enough spurious conflicts to become unusable for me. Eg, doing a clean checkout of mozilla with win32text enabled results in a working tree with hundreds of files reporting every line in the file has changed. It is worth noting that win32text also works on Linux, and that in theory, win32text should be able to allow Linux users to work seamlessly with a \r\n repo, in the same way it promises to allow Windows users to work with \n repos. But in practice, it is simply never used by Linux developers, even in the hg core dev team. Patches or comments regarding win32text are always met with "but no one here ever uses it". If they did, and some of the core hg team tried to experiment with win32text and mixed line ending repos the problems would, I believe, become self-evident... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
On 4/07/2009 12:30 PM, Nick Coghlan wrote: ... I think there needs to be a solid answer in place for these use cases before the actual migration to Mercurial takes place. A hand-waving "use win32text" isn't enough - it needs to be "use win32text with these exact settings" (with server side hook support to enforce the rules). And since Mercurial doesn't even allow us to say "this is a binary file" the way CVS used to I'm currently not seeing any way for that to happen except for win32text to be updated to correctly handle wild cards in combination with negative filters. I agree with your conclusion. My ruminating on this over the last few months leaves me thinking this would involve: * my older 'accepted but then lost' hg patch to allow an explicit 'none' rule for a single file to override wildcards. * win32text be enhanced to use a normal versioned file in the root of the repo, much like hgingore, where a project can maintain project wide rules. * win32text be enhanced such that all python developers, regardless of platform, are willing to use this extension, even if the majority of files happen to use their native line ending (sauce for the goose is sauce for the gander, and all that...) * commit hooks be implemented to enforce this - but this should not be necessary if the above was implemented and socially enforced. Cheers, Mark Cheers, Nick. [1] http://mail.python.org/pipermail/python-dev/2006-March/062225.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
Sorry Dirkjan - I just noticed I didn't CC you on this mail originally. I'm wondering if you have any more thoughts on these EOL issues and if there is anything I can do to help? Cheers, Mark On 4/07/2009 2:03 PM, Mark Hammond wrote: On 4/07/2009 12:30 PM, Nick Coghlan wrote: ... I think there needs to be a solid answer in place for these use cases before the actual migration to Mercurial takes place. A hand-waving "use win32text" isn't enough - it needs to be "use win32text with these exact settings" (with server side hook support to enforce the rules). And since Mercurial doesn't even allow us to say "this is a binary file" the way CVS used to I'm currently not seeing any way for that to happen except for win32text to be updated to correctly handle wild cards in combination with negative filters. I agree with your conclusion. My ruminating on this over the last few months leaves me thinking this would involve: * my older 'accepted but then lost' hg patch to allow an explicit 'none' rule for a single file to override wildcards. * win32text be enhanced to use a normal versioned file in the root of the repo, much like hgingore, where a project can maintain project wide rules. * win32text be enhanced such that all python developers, regardless of platform, are willing to use this extension, even if the majority of files happen to use their native line ending (sauce for the goose is sauce for the gander, and all that...) * commit hooks be implemented to enforce this - but this should not be necessary if the above was implemented and socially enforced. Cheers, Mark Cheers, Nick. [1] http://mail.python.org/pipermail/python-dev/2006-March/062225.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: the eol-type issue
On 4/08/2009 7:20 PM, Nick Coghlan wrote: Dirkjan Ochtman wrote: * commit hooks be implemented to enforce this - but this should not be necessary if the above was implemented and socially enforced. You seem to advocate a two-step approach: enforce line endings through win32text, catch any errors that slipped through in a hook (commit hook is an optional first line of defense, changegroup hooks on the server to protect the rest of the world). I think inverting that approach would be better: have strict hooks on the server to prevent people from pushing inappropriate EOLs, and provide help on configuring win32text as an extra help for developers on Windows who use editors that work better with \r\n. That leaves people to pick their own weapon of choice against propagation of \r\n (e.g. better editor, commit hooks, whatever) while still making sure no inappropriate line endings land in the python.org repositories. It also seems to fit well with the whole consenting adults thing (but that might just be me). It's about not treating Windows developers as second class citizens. Their platform uses \r\n as its native line ending format, so they Thanks Nick; I didn't want to be the only one saying that. There is a fine line between asserting reasonable requirements for Windows users and being obstructionist and unhelpful, and I'm trying to stay on the former side :) should be able to work in that format without any hassles by following some simple instructions (such as "ensure you have version X of the Windows hg client, enable the win32text extension and configure it in such-and-such a way"). Not "oh, yeah, that's an issue but if you search the Intarwebs there are a few different things you can do that kinda sorta work but are a bit fragile and klunky". The precise order the two issues (server side enforcement and client side assistance) are dealt with doesn't really matter because *both* issues need to be addressed before we migrate. I'm not that happy with the server being the primary line of defense. Let's say I make a branch of the hg repo, myself and a few others work on it committing as we go, then attempt to merge back upstream. Let's say some of the early commits on that clone introduced "bad" line endings. I'm guessing I would be forced to make a number of whitespace-only checkins to normalize the line-endings before it could merge - and these checkins would then be in the history forever. Or I could attempt to recreate the clone by somehow "replaying" the commits with line endings corrected. Either way, the situation doesn't seem good. win32text needs to be usable on non-Windows clients so that tarballs generated on a *nix machine get the line endings right in the Windows-only files. I agree. It isn't fair to make this windows users problem. It would be like me proposing the repo get imported with \r\n line endings, enforce that with server side hooks, and let non-Windows users worry about the ramifications of that - somehow I doubt that would fly - so neither should it fly for Windows users... I'm more than willing to help on this; I haven't resurrected my stale patch because I find win32text only 1/2 a solution that doesn't work in practice. Therefore that patch is as stale for me as it is anyone. However, if a plan is put in place which offers a full solution and the hg developers are committed to it, I promise I'll put my hand up to help with implementation in a fairly timely manner... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: the eol-type issue
On 5/08/2009 3:56 PM, Ben Finney wrote: Mark Hammond writes: Let's say I make a branch of the hg repo, myself and a few others work on it committing as we go, then attempt to merge back upstream. Let's say some of the early commits on that clone introduced "bad" line endings. I'm guessing I would be forced to make a number of whitespace-only checkins to normalize the line-endings before it could merge - and these checkins would then be in the history forever. What is wrong with that? I mean, if that is the actual sequence of events, why should the history not reflect that? The problem is the sequence of events happened in the first place. An extra burden is placed on the developer that will quickly get tiresome. I wouldn't personally be happy if that workflow became the norm. Either way, the situation doesn't seem good. I see this assertion made often, so I'm not saying you are necessarily wrong to make it. I just don't see a justification for making it (and, without justification, I would say it *is* wrong to make it). *shrug* - in my opinion, the fact the developer is faced with that hurdle in their workflow is justification enough to say that developer's situation "doesn't seem good" and should have been prevented from happening by the tool much earlier than proposed. Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: the eol-type issue
On 5/08/2009 4:50 PM, Ben Finney wrote: Mark Hammond writes: On 5/08/2009 3:56 PM, Ben Finney wrote: Mark Hammond writes: Let's say I make a branch of the hg repo, myself and a few others work on it committing as we go, then attempt to merge back upstream. Let's say some of the early commits on that clone introduced "bad" line endings. […] The problem is the sequence of events happened in the first place. An extra burden is placed on the developer that will quickly get tiresome. I wouldn't personally be happy if that workflow became the norm. Ah, okay. In that case, the ultimate “problem” is that OS vendors entrenched their incompatible line-ending conventions instead of choosing a single standard. Any line-ending burden borne by developers is a result of that. Yeah - this happened around 1964 if wikipedia is any guide. If things were different, they'd be different. However, we live with the legacy of that stupid set of decisions and have no real option to resolve it permanently short of deprecating entire vistas of tools (or even entire operating systems). Agreed - so let's not solve it permanently. ... It's not a simple thing to solve, and many clever people have tried over the decades. As already mentioned in this thread, a capability similar to what svn or cvs offers would be sufficient. While a DVCS does offer unique challenges, it seems to me that doing something at commit time without requiring magic hooks be configured would go a long way to addressing the problem. Magic hooks on the official repo would then be considered the final fallback defense, but should rarely be invoked. At some point, the decision about how to handle line endings in cross-platform data needs to be punted to a human for a context-sensitive assessment, since (as can be seen) the above list of requirements is internally inconsistent and can't be relegated to a one-size-fits-all algorithm. I'm not sure what point you are trying to make, but I believe it *is* possible for a solution to be found here which will keep Windows users happy. I'm guessing you haven't had much practical experience with this problem, so probably don't see this is clearly as Windows users do. Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: the eol-type issue
On 5/08/2009 5:35 PM, "Martin v. Löwis" wrote: Now, the specific outcome of the process means that more work needs to be done. So we have a *second* PEP, and we have a lack of volunteers that help implementing it. The second PEP hasn't been approved yet (as it isn't complete, yet), so migration to hg is stalled. The primary volunteer (Dirkjan) has indicated that he can't help with that specific issue, so other volunteers need to step forward, or we cannot move to hg. I don't recall Dirkjan saying he can't help with that issue - was it a lack of time, or a lack of understanding the problem/lack of a Windows environment? The problem I see is a lack of agreement about exactly what the solution entails. I believe there is general agreement win32text needs to be enhanced to support versioned 'rules'. But even with that, the only option I see is a truly cross-platform extension to implement these rules which every Python committer, regardless of operating-system, is expected to use - but that doesn't seem the consensus. As mentioned, I'm willing to lend manpower for this once there is agreement on something workable... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: the eol-type issue
On 5/08/2009 6:00 PM, Ben Finney wrote: Mark Hammond writes: As already mentioned in this thread, a capability similar to what svn or cvs offers would be sufficient. That capability presented by centralised VCSen is entirely dependent on the fact that they *are* centralised. Using a distributed VCS means the same capability doesn't apply. Why do you say that (without justification I might add ) about this issue? While a DVCS does offer unique challenges, it seems to me that doing something at commit time without requiring magic hooks be configured would go a long way to addressing the problem. The hand-waving “doing something” is exactly what needs to be solved. I think you have been mis-reading this thread. It is quite clear what 'doing something' means in this context - it means implement the human-defined rules for the line-ending policy for the repository. Magic hooks on the official repo would then be considered the final fallback defense, but should rarely be invoked. Right, so that's “capability similar to centralised VCS” out of consideration; I'm glad we agree in the end. I'm afraid you have lost me again, as clearly we don't agree on what useful things can be done at local commit time. I'm not sure what point you are trying to make That I disagree with your position. You seem to think that the problem has an obvious solution, which is not true; and that choice of a distributed VCS should be delayed until the problem is solved, which I don't agree with. Fair enough - but it seems clear to enough of us that we can make progress and meet the requirements of the people actually impacted. but I believe it *is* possible for a solution to be found here which will keep Windows users happy. I'm guessing you haven't had much practical experience with this problem, so probably don't see this is clearly as Windows users do. Your guess is incorrect; I've been bitten time and again by this problem in many different contexts, enough to know that it's not obvious what the “right” solution is. Sorry about that - but that was the only way I could explain you not seeing how such a solution can work. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: the eol-type issue
On 5/08/2009 6:25 PM, Dirkjan Ochtman wrote: On Wed, Aug 5, 2009 at 01:43, Mark Hammond wrote: Thanks Nick; I didn't want to be the only one saying that. There is a fine line between asserting reasonable requirements for Windows users and being obstructionist and unhelpful, and I'm trying to stay on the former side :) I'm not trying to be obstructionist and unhelpful (I hope that should be obvious). It is, and I hope I didn't imply otherwise. On the other hand, I'm working from the point of view of hg, which has two assumptions: - we're a distributed system, there's fairly little we can assume about clients - we exchange checksummed byte streams (even if we have some tools that assume those streams are code) - because of the previous point, there's one native (and therefore better, in a sense) serialization of what you consider "structured" data The first point means, for example, there will always be some clients who don't have win32text enabled, no matter what, so you can't rely on it, which is why I want to make the server hooks the primary line of defense, and view the client-side tools as helper tools (to make it easy not to trigger the server-side hooks). That doesn't mean I think Windows users are second-rate, or anything like that! In general I agree - although I think we can enforce a "social contract" which puts requirements on people who commit to the Python repository - and therefore we can consider the server-side hooks a "secondary" defense. IOW, the system (including the social aspects of the system) are setup such that the server-side hooks are very rarely called upon. I'm not that happy with the server being the primary line of defense. Let's say I make a branch of the hg repo, myself and a few others work on it committing as we go, then attempt to merge back upstream. Let's say some of the early commits on that clone introduced "bad" line endings. I'm guessing I would be forced to make a number of whitespace-only checkins to normalize the line-endings before it could merge - and these checkins would then be in the history forever. Or I could attempt to recreate the clone by somehow "replaying" the commits with line endings corrected. Either way, the situation doesn't seem good. I don't think either is bad. With all due respect, I suspect that is because you don't expect to see the issue regularly. This proposal still leaves the problem squarely in the lap of Windows users and imposes a burden on them that would probably be considered unreasonable if the situation was reversed. I'm yet to work on a hg repository without mixed line endings. If I understand correctly, every such repository would have involved a developer checking in locally, than at some point in the future pushing these changes upstream. I really really don't want hg to tell me at this final step that I need to perform whitespace only fixes purely because I am running Windows. I understand we are discussing how win32text can offer that - but I must object to your assertion that the situation I described isn't bad when you hit it. Well, I'd be happy to help convince the hg crew to accept whatever we come up with, but I'm not sure I'm the best person to come up with it. It sounds like a versioned .hgeols would help a bunch of issues, but I have the feeling you know that better than me, so I'm hoping you can come up with a concrete proposal on what should change in win32text to fix all the problems you see. Actually, I think it is easy to make this problem much easier to understand; mandate every platform should use win32text, then start collating the issues people, including yourself, will no doubt face. I'm happy to get this ball rolling, but again, don't want this left purely in the domain of "it is a windows problem" - it isn't. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: the eol-type issue
On 5/08/2009 7:09 PM, Dirkjan Ochtman wrote: I'm not sure how win32text will provide anything other than performance degradation for non-Windows developers, but if there's functionality to be had, I'm happy to mandate its use on every platform. I see two practical outcomes of such a mandate: * line-ending rules are enforced for local checkins, even for linux users, even though such 'accidental' inappropriate line-ending checkins should be much rarer than for windows. * practical problems faced by Windows users, including any performance considerations, are shared by the community and therefore addressed as a community, thereby ensuring all platforms are considered as important as any other. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: the eol-type issue
On 5/08/2009 8:04 PM, Paul Moore wrote: 2009/8/5 "Martin v. Löwis": With such a setup, using the hook would be truly optional on Unix, as it only ever checks and never converts. So if you manage to mess up, and don't have the hook installed on Unix, you lose when trying to push. That will teach you to be more careful in the future, or to install the hook (which hopefully becomes built into Mercurial at some point). Given that my preference is to use Unix-style EOL for "text" files on Windows, as every text editor I use (barring notepad!) understands LF format, Most tools that I use will tend to not mix EOL styles in a single file, but will tend to create \r\n line endings for new files I create. Most hg repos I come across don't have mixed line endings within individual files, so I can only guess these files were accidentally introduced in the same way (and indeed I have personally done this.) I'm hoping to be part of the solution instead of part of the problem :) > it seems to me that this proposal also means that the hook would be optional for me. Technically it would be optional for everyone, of course. However, the solution should be such that everyone, regardless of personal preference, is willing to take the hit. For example, if the repo is converted using \r\n line endings natively, then Windows users would need to take no action either and puts the onus back on you (given your stated preferences) to configure the tool appropriately. I assume you would have no objection to that and would be happy to make that tool optional for me? That suits me fine - I'd prefer to avoid having hooks that are required for Python checkouts, as that means I have to remember to configure them on each clone (IIUC). Configuring on each clone would certainly be sub-optimal, so the proposal is this configuration be stored in a versioned file in the repo. Of course, this implies that your proposal only requires any action by the user in the case of Windows users whose text editing tools insist on CRLF format text files (sources, etc). Is that really a large group of developers? (I honestly don't know). It applies to all files that aren't "native" EOL style - there are just less of them regularly modified than those that are so marked. I suspect that there is something missing from your proposal, as if this were the case, then the problem appears to be limited to a very small group of developers. Maybe it's Visual Studio that insists on CRLF for source files? (I don't know, as I don't use the VS editor). If that's the case, then maybe a VS hook would be an alternative approach? (I can't imagine such a hook would be an *easier* approach, I only mention it because it makes it clearer where the issue lies). I must concede that Windows developers are the minority here - but assuming we want a level playing field, I don't see how that changes the underlying issue... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: the eol-type issue
On 5/08/2009 8:14 PM, Dirkjan Ochtman wrote: endings. Typically, in my case, that was either Notepad2 (an awesomely light-weight Notepad replacement) or Komodo (Edit). That solved all of my issues, so I haven't had a need for win32text so far. FWIW, I use komodo and scite as my primary editors, and as mentioned, am personally responsible for accidentally checking in \r\n files into what should be a \n repo. I am slowly and painfully learning to be more careful - IMO, I shouldn't need to... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: the eol-type issue
On 5/08/2009 9:28 PM, Dirkjan Ochtman wrote: On Wed, Aug 5, 2009 at 13:19, Mark Hammond wrote: Configuring on each clone would certainly be sub-optimal, so the proposal is this configuration be stored in a versioned file in the repo. Even if we do that, enabling hg extensions will still need to be done locally -- although it can be done per-user/box instead of per-clone. That is completely fine, and not unlike SVN where a per-user/box setting generally needs to be set once - but after that everything "just works". Windows developers don't mind taking a hit once ;) The dev guide can make it clear what the expectations are... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: the eol-type issue
On 6/08/2009 12:28 AM, Stephen J. Turnbull wrote: Mark Hammond writes: > I'm not sure what point you are trying to make, but I believe it *is* > possible for a solution to be found here which will keep Windows users > happy. I'm guessing you haven't had much practical experience with this > problem, so probably don't see this is clearly as Windows users do. Mercurial is not only open source, it's written in Python. The problem is known to be hard in a practical sense, the existing solutions (written by non-Windows developers, of course) are judged to be insufficient by Windows users, and the non-Windows developers "probably don't see this is clearly as Windows users do." I think the implication is obvious. There will be no good solution until Windows users develop it. I don't see a good reason to wait for that. My conclusion is different. I'm not sure of the history of win32text, but it most certainly is now squarely in the hands of Windows users. Patches to win32text, or even general discussion is usually met with silence, and when prodded, the response is "sorry - we don't use that - it is a Windows problem." As a result, we end up in the position we are in now - win32text is great in theory but doesn't work in practice, attempts to make it work are met with indifference, and the "problem" stays squarely with Windows users. Non Windows users remain oblivious to the pain, Windows users stop bothering with the extension, and the repository post-commit hooks then cause different pain. Hence my conclusion that the answer is for any such support to be developed in conjunction with Windows users, but also in such a way that the solution works, almost identically, for non Windows users. By insisting all platforms eat the same dog-food, there is much more chance the glaringly obvious (to Windows users) issues are addressed. > I do see good reason for non-Windows users to put up with some inconvenience during the "beta" phase of implementing that solution; it's important enough to be fast-tracked, and doesn't need to be perfect for everybody to be tried (though it should not be allowed to endanger repo content, which seems unlikely but needs care since it's a potential disaster). And on the flip-side, I accept we may migrate without the agreed solution fully implemented - I'm happy to accept commitments about what *will* be done even if it isn't a reality for a short while... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: help needed
On 18/08/2009 6:20 PM, Dirkjan Ochtman wrote: On Tue, Aug 18, 2009 at 10:12, "Martin v. Löwis" wrote: In this thread, I'd like to collect things that ought to be done but where Dirkjan has indicated that he would prefer if somebody else did it. I think the most important item here is currently the win32text stuff. Mark Hammond said he would work on this; Mark, when do you have time for this? Then I could set apart some time for it as well. I can make time, somewhat spasmodically, starting fairly soon. Might I suggest that as a first task I can resurrect my old stale patch, and you can arrange to install win32text locally and start experimenting with how mixed line-endings can work for you. Once we are all playing in the same ballpark I think we should be able to make good progress. I-said-ballpark-yet-I-call-myself-an-aussie? ly, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: help needed
[Adjusted the CCs...] On 19/08/2009 8:21 AM, Dj Gilcrease wrote: On Tue, Aug 18, 2009 at 2:12 AM, "Martin v. Löwis" wrote: The second item is line conversion hooks. Dj Gilcrease has posted a solution which he considers a hack himself. Mark Hammond has also volunteered, but it seems some volunteer needs to be "in charge", keeping track of a proposed solution until everybody agrees that it is a good solution. It may be that two solutions are necessary: a short-term one, that operates as a hook and has limitations, and a long-term one, that improves the hook system of Mercurial to implement the proper functionality (which then might get shipped with Mercurial in a cross-platform manner). My solution is a hack because the hooks in Mercurial need to be modified to support it properly, I would be happy to help work on this as it is a situation I run into all the time in my own projects. I can never seem to get all the developers to enable the hooks, and one of them always commits with improper line endings =P Maybe you can enumerate what you think needs to change in mercurial, then once we have a plan in place it will be clearer who can do what. I'm resurrecting my patch to support a filter called 'none' (which is turning out to be harder than I thought). Off the top of my head, it would the following would give us a pretty solid solution: * Finish my patch for 'none' as a filter, so '**=cleverencode' can be reasonably used (currently you can't specify specific files *not* have cleverencode, making it unsuitable in practice without the concept of 'none') * Add support for versioned 'filter rules' - eg, /.hgfilters or similar. * This might be pushing my luck, but: add 'defensive' support to core hg for this feature - if /.hgfilters exists, hg should refuse to operate on the working tree unless the win32text extension is enabled. Note that this last point still leaves win32text optional for hg itself - but if the owner of a repository has explicitly 'opted in' for win32text support, hg can still assist in refusing to screw the tree. The hg user has the option of enabling that extension, declining to use that repository, or arguing with the owner of the repo about use of the feature in the first place. Is there something I'm missing? Or maybe a better way to have hg enforce a repository's policy while not inflicting pain on hg users who don't want to ever think about windows? Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: help needed
On 22/08/2009 12:19 AM, Dirkjan Ochtman wrote:
On Fri, Aug 21, 2009 at 16:10, Dj Gilcrease wrote:
I like this, though maybe .hgextensions since it would contain
versioned rules and the actual required extension. The extra sub
directories are not really required IMHO, you just have a hgrc file
that works the same as the local hgrc file except it only looks in the
.hgextensions directory for the correct extension so for python we
could have something like
[extensions]
format_enforcer =
Enabling extensions in a versioned file is not going to fly.
I like Stephen and Nick's discussion higher in this thread, but wonder
if some middle ground couldn't work.
Instead of [extensions], just have a place to list the required
extensions - eg;
Something like ~/.hgrules having:
[config] # or maybe [rules] ?
required_extensions = win32text, some_pydev_specific_extension
[encode]
{rules for encoding}
[pydev]
some_custom_property_for_our_custom_ext = 1
... etc ...
(Note I am not proposing we need out own pydev_specific_extension, I
just included it here to try and show the more general concept)
This way you aren't *enabling* extensions in this versioned file, just
listing rules about what extensions must be enabled. From core hg's
POV, it doesn't care if the required extensions relate to windows line
endings or re-encoding images - it just honours the wishes of the repo
owner.
From earlier in the thread, Dirkjan writes:
> The [concept of hg enforing required extensions] might indeed be
> hard to get into the core, but seems like a good idea to try.
From my POV, this would be required in some form or another before such
a scheme could actually work. Without it we end up with an improved
win32text (good!) but in practice still have the same problems we have
discussed in this thread which would make it unsuitable for us who
actually try and use it, particularly as a general solution for projects
with any kind of windows focus or community.
Given you are a core hg committer and well known in the community, would
you be willing to start a thread with the hg developers about this
issue? If something like this can't get into the core, I will drop any
expectations of it becoming a viable general solution for windows
focused projects, so would limit the work I am willing to invest to the
commitments I've made here.
Thanks!
Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: help needed
On 22/08/2009 12:10 AM, Dj Gilcrease wrote: On Fri, Aug 21, 2009 at 1:16 AM, Mark Hammond wrote: Maybe you can enumerate what you think needs to change in mercurial, then once we have a plan in place it will be clearer who can do what. The encode/decode hooks need to be passed the filename they are working on so you can have an ignore list, this is why I consider my method a hack since I am using a precommit hook to do conversion since I am able to find out which file I am working on and make sure it is not in an ignore list. There also needs to be a way to have required and version controlled extensions. I think this is the exact issue my 'none' patch addresses. Your filters can say: [encode] *.dsp=none: **=cleverencode: The end result should be that anything with 'none:' forms what you call an ignore list. Would that not meet your requirements? Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: help needed
On 22/08/2009 2:46 PM, Stephen J. Turnbull wrote:
Mark Hammond writes:
> Something like ~/.hgrules having:
Surely you mean $PROJECTROOT/.hgrules?
Indeed.
> [config] # or maybe [rules] ?
> required_extensions = win32text, some_pydev_specific_extension
[extensions]
required_for_commit = win32text,some_other_ext
That might require a change to hg's ini file semantics if currently it
refuses to parse [extension] sections in versioned hgrcs.
Yes - I'm not proposing specific names for sections etc - I'm more
interested in getting the concepts across, and fully expect the hg guys
will have their own opinions and make final decisions on the exact spelling.
Note the change in name: I'm not sure exactly what the semantics
should be, but surely we want to allow browsing the repository,
branching, etc without enabling any extensions.
> [Encode]
> {rules for encoding}
No, there must be a way to indicate that "this is a section for a
specific extension". Bare [Encode] will be seen as polluting the
global namespace, and will get a lot of pushback, I think.
Possibly - although I would expect the existing section names be reused
when applied to a versioned file, I'd be more than happy for the hg guys
to declare new names are appropriate for this.
> This way you aren't *enabling* extensions in this versioned file,
True, but how many people will just download the extension and enable
it?
In the ideal world, exactly as many people who would read the Python
developer guide, then download and install the extension based purely on
that. IOW, it is Python itself setting the policy, so people need to
make their own decisions based on that, regardless of whether the tool
enforces it or not.
This would open a door to "social engineering". (Personally, *I*
am not opposed to it on those grounds, but as devil's advocate I do
want to mention that as an argument you might run into.)
> just listing rules about what extensions must be enabled. From
> core hg's POV, it doesn't care if the required extensions relate to
> windows line endings or re-encoding images - it just honours the
> wishes of the repo owner.
If it refuses the user's request, it should issue a message to the
effect of "Please enable win32text, which is required in."
Agreed.
Thanks,
Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: help needed
On 22/08/2009 6:52 PM, Stephen J. Turnbull wrote: Mark Hammond writes: > On 22/08/2009 2:46 PM, Stephen J. Turnbull wrote: > Possibly - although I would expect the existing section names be reused > when applied to a versioned file, I'd be more than happy for the hg guys > to declare new names are appropriate for this. If there's already an [Encode] section, that's different. (I don't details, I'm not that big a Mercurial fan.) But you'd still need a way to differentiate win32text rules from other encoding rules. As mentioned in my previous post, I'm trying to avoid bike-shedding what the hg guys are better placed to decree. How they choose to spell these options is something for hg to decide, and I doubt my opinion matters enough to bother sharing, let alone advocating. > >> This way you aren't *enabling* extensions in this versioned file, > > > > True, but how many people will just download the extension and enable > > it? > > In the ideal world, exactly as many people who would read the Python > developer guide, then download and install the extension based purely on > that. IOW, it is Python itself setting the policy, so people need to > make their own decisions based on that, regardless of whether the tool > enforces it or not. You're missing the point. I'm not talking about whether it will work for Python, I'm talking about the worry that somebody will post a way cool Python branch and require a private extension, which everybody will just automatically install and enable, which extension then proceeds to phone home to Spammer Haven, Inc. with the contents of your email contact list. That's what I mean by "social engineering," and why I worry about policy pushback from Mercurial HQ. No, you are missing the point - social engineering doesn't require tool support - tools simply make certain things easier. Maybe that's more paranoid than they are But it can't hurt your cause to be ready for that kind of worry. If this becomes seen as 'my' cause, I suspect it will run out of steam very quickly. I truly hope python-dev, as a community, takes some ownership of this issue or I predict the effort will fizzle out without a workable solution. There seem to be a number of people who agree the status-quo isn't acceptable, so I'm not sure what would happen in that case... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: help needed
On 22/08/2009 7:09 PM, "Martin v. Löwis" wrote: From my POV, this would be required in some form or another before such a scheme could actually work. Without it we end up with an improved win32text (good!) I still think this would be actually bad. Instead, a new extension should be written, with a name that does not have "win32" as a substring, and that has no provision for guessing line breaks by inspecting files. To be clear, you are suggesting: * Having hg enforce an extension as required is good. * Python adopting win32text as that extension would be bad - instead another extension with different semantics (ie, no guessing based on file content) should be used, and enforced, instead. Or have I misunderstood? Assuming I am correct, I am inclined to agree - win32text may be "good enough" in the short term, but it is far from ideal. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: help needed
On 23/08/2009 5:25 PM, "Martin v. Löwis" wrote: If this becomes seen as 'my' cause, I suspect it will run out of steam very quickly. I truly hope python-dev, as a community, takes some ownership of this issue That certainly won't happen. python-dev, as a community, has never ever taken ownership of anything. It's always individuals who take ownership. I believe ownership of a task and ownership of a cause are somewhat different. In other words, I'm happy to take ownership of a number as tasks relating to this cause, but if the general feeling is that it is my cause rather than *our* cause, then I will probably opt-out - I'm taking these tasks on at this moment purely because I believe it *is* a common cause. So you essentially say that you want somebody else (but not you) take ownership - which, of course, is certainly fine. Hence my call for volunteers. Hence my volunteering and the time I am currently spending. There seem to be a number of people who agree the status-quo isn't acceptable, so I'm not sure what would happen in that case... My prediction is that it will depend on whether workable code is available by the time a decision is made to migrate. If code is available, then migration will happen (no matter whether the code has an owner); if no code is available, migration will stall. Right - I guess we are all still struggling with exactly what "workable code" means in this context. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: help needed
On 18/08/2009 6:20 PM, Dirkjan Ochtman wrote: On Tue, Aug 18, 2009 at 10:12, "Martin v. Löwis" wrote: In this thread, I'd like to collect things that ought to be done but where Dirkjan has indicated that he would prefer if somebody else did it. I think the most important item here is currently the win32text stuff. Mark Hammond said he would work on this; Mark, when do you have time for this? Then I could set apart some time for it as well. Have stalled a bit on the fine-grained branch processing, hope to move that forward tomorrow. I'm afraid I've also stalled on this task and I need some help to get things moving again. 1) I've stalled on the 'none:' patch I promised to resurrect. While doing this, I re-discovered that the tests for win32text appear to check win32 line endings are used by win32text on *all* platforms, not just Windows. I asked for advice from Dirkjan who referred me to the mercurual-devel list, but my request of slightly over a week ago remains unanswered (http://selenic.com/pipermail/mercurial-devel/2009-August/014873.html) - maybe I just need to be more patient... Further, Martin's comments in this thread indicate he believes a new extension will be necessary rather than 'fixing' win32text. If this is the direction we take, it may mean the none: patch, which targets the implementation of win32text, is no longer necessary anyway. 2) These same recent discussions about an entirely new extension and no clear indication of our expectations regarding what the tool actually enforces means I'm not sure how to make a start on the more general issue. I also fear that should I try to make a start on this, it will still wind up fruitless - eg, it seems any work targeting win32text specifically would have been wasted, so I'd really like to see a consensus on what needs to be done before attempting to start it. So in short, I'm still offering to work on this issue - I just don't know what that currently entails. Thanks, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: help needed
On 30/08/2009 9:37 PM, Martin Geisler wrote: Mark Hammond writes: 1) I've stalled on the 'none:' patch I promised to resurrect. While doing this, I re-discovered that the tests for win32text appear to check win32 line endings are used by win32text on *all* platforms, not just Windows. I think it is only Patrick Mezard who knows how to run (parts of) the test suite on Windows. I asked for advice from Dirkjan who referred me to the mercurual-devel list, but my request of slightly over a week ago remains unanswered (http://selenic.com/pipermail/mercurial-devel/2009-August/014873.html) - maybe I just need to be more patient... Oh no, that's usually the wrong tactic :-) I've been too busy for real Mercurial work the last couple of weeks, but you should not feel bad about poking us if you don't get a reply. Or come to the IRC channel (#mercurial on irc.freenode.net) where Dirkjan (djc) and myself (mg) hang out when it's daytime in Europe. To be fair, I did mail Dirkjan directly who referred me to the -develop list, which I did with a CC to him and a private mail asking for some help should the mail fall on deaf ears as I feared it would. There really is only so far I'm willing to poke and prod people when I'm well aware we are all volunteers. Further, Martin's comments in this thread indicate he believes a new extension will be necessary rather than 'fixing' win32text. If this is the direction we take, it may mean the none: patch, which targets the implementation of win32text, is no longer necessary anyway. I suggested a new extension for two reasons: ... Thanks, and that does indeed sound excellent. However, this is going a fair way beyond the original scope I signed up for. While I was willing to help implement some new features into an existing extension, taking on the design and implementation of an entire new extension is something I'm not willing to undertake. I don't think such an extension should even come from the Python community or it will end up being a python-only extension - or at best, will need to run the gauntlet of 2 bike-shedding sessions from both the Python and hg communities which will waste much time. What is the hope of an EOL extension which meets our requirements coming directly out of the hg community? If that hope is small, where does that leave us? Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] time.clock() on windows
On 22/10/2009 8:52 AM, Scott Dial wrote: Curt Hagenlocher wrote: But it makes more sense to understand why someone chose to implement time.clock() on Windows the way they did -- this seems very broken to me, and I think it should be changed. Some SVN detective work takes this to all the way back to r7713 (1997-04-02). The original implementation checked by Guido and attributed to Mark Hammond. So, we should ask Mark why he did that. The thread seems to be at http://groups.google.com/group/comp.lang.python/browse_frm/thread/be32478a4b8e77b6/816d6228119a3474 (although I do seem to recall more discussion of the patch which I currently can't find). I'd be very surprised if any applications rely on the fact that each process starts counting at zero, so if someone can come up with a high-res counter which avoids that artifact I'd expect it could be used. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] time.clock() on windows
On 22/10/2009 8:47 PM, Kristján Valur Jónsson wrote:
The point in question seems to be this this (from the thread):
* Need some sort of static "start value", which is set when the
process starts, so I can return to Python in seconds. An easy hack
is to set this the first time clock() is called, but then it wont
reflect any sort of real time - but would be useful for relative
times...
But the argumentation is flawed.
It was made in the context of the APIs available to implement this. The
code is short-and-sweet in timemodule.c, so please do go ahead and fix
my flawed reasoning.
For reference:
#if defined(MS_WINDOWS) && !defined(__BORLANDC__)
/* Due to Mark Hammond and Tim Peters */
static PyObject *
time_clock(PyObject *self, PyObject *unused)
{
static LARGE_INTEGER ctrStart;
static double divisor = 0.0;
LARGE_INTEGER now;
double diff;
if (divisor == 0.0) {
LARGE_INTEGER freq;
QueryPerformanceCounter(&ctrStart);
if (!QueryPerformanceFrequency(&freq) || freq.QuadPart == 0) {
/* Unlikely to happen - this works on all intel
machines at least! Revert to clock() */
return PyFloat_FromDouble(((double)clock()) /
CLOCKS_PER_SEC);
}
divisor = (double)freq.QuadPart;
}
QueryPerformanceCounter(&now);
diff = (double)(now.QuadPart - ctrStart.QuadPart);
return PyFloat_FromDouble(diff / divisor);
}
Cheers,
Mark.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] time.clock() on windows
On 22/10/2009 3:45 PM, Robert Collins wrote: On Thu, 2009-10-22 at 15:21 +1100, Mark Hammond wrote: I'd be very surprised if any applications rely on the fact that each process starts counting at zero, so if someone can come up with a high-res counter which avoids that artifact I'd expect it could be used. Could you offset it by the system time on the first call? Off the top of my head, I guess that depends on the actual accuracy required (ie, how many clock ticks elapse between querying the time and the high-resolution timer). Starting at 0 works fine for profiling in a single process, the predominant use-case when this was done; I guess it depends on the specific requirements and time-intervals being dealt with in the cross-process case which determines how suitable that might be? Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3003 - Python Language Moratorium
On 4/11/2009 4:35 AM, Guido van Rossum wrote: I've checked draft (!) PEP 3003, "Python Language Moratorium", into SVN. As authors I've listed Jesse, Brett and myself. Good move, +1. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] First draft of "sysconfig"
On 15/12/2009 2:07 PM, David Lyon wrote: Hi Tarek, Is there anything in this proposal for windows developers ? Just that I can't see anything that would help us... So I understand - help doing what? For me, the terminology isn't anything a windows developer could really understand. It presumes that the developer understands the python implementation. A developer might not understand all those details and might not be interested to learn. That seems true for all operating systems, not just Windows. The vast majority of Python programmers will probably never need to use this feature, but those who do will need to understand enough of the python implementation to make sense of the values returned regardless of the OS. I accept that the terminology is good on linux.. but it's near meaningless on windows - for me - anyway. I think it is fine. If you are really looking for properties specific to the operating system (eg, the location of the start menu, desktop, appdata folders etc) I don't think they belong in this PEP; they are a property of the OS install/version, not of the specific Python install/version. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] First draft of "sysconfig"
On 15/12/2009 2:42 PM, David Lyon wrote: On Tue, 15 Dec 2009 14:40:56 +1100, Mark Hammond wrote: I think it is fine. If you are really looking for properties specific to the operating system (eg, the location of the start menu, desktop, appdata folders etc) But under windows, an application developer might (as in probably would) like to install an application in \Program Files\someapp rather than hidden in the bowels of the python interpretor. I agree - but in that case you are talking about an application built with Python - that is a different set of requirements. But regardless of where Python itself lives the returned values will be correct; they will not reflect the operating system preference for various other folders, but will correctly return the paths they are documented as returning. IOW, this isn't designed for applications which happen to be written in Python. There might be a case for such a module to be created, but this PEP doesn't attempt to solve that particular problem. They might like their data in "Application Data", which is where support people get trained to look for application data. Not down in \pythonX.Y\ ... Nothing is stopping them from doing that - but this PEP isn't intended to provide that information. I don't think they belong in this PEP; they are a property of the OS install/version, not of the specific Python install/version. Yes - exactly. My point. The operating system says that programs should be installed into "Program Files" and data into "Application Data". Why can not python respect the operating system directions for well behaved apps? It does - many applications written in Python exist which do exactly that. If a user really wants to install Python itself under "\Program Files", sysconfig should correctly reflect that. Python doesn't care where it is installed, or even when it is (sensibly) bundled with an app which is designed to be "stand-alone". You are after operating system properties - I understand your need to know those paths (and there are already reasonable Windows specific ways to get them), but sysconfig isn't trying to solve that for you and I agree it should not attempt to. Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] First draft of "sysconfig"
On 15/12/2009 3:09 PM, David Lyon wrote: On Tue, 15 Dec 2009 15:05:18 +1100, Mark Hammond wrote: But under windows, an application developer might (as in probably would) like to install an application in \Program Files\someapp rather than hidden in the bowels of the python interpretor. I agree - but in that case you are talking about an application built with Python - that is a different set of requirements. Building an application with python.. that's right. Of course. Why not? I'm missing your point - many applications exist written in Python. IOW, this isn't designed for applications which happen to be written in Python. There might be a case for such a module to be created, but this PEP doesn't attempt to solve that particular problem. But programmers might want to write an application with python. It doesn't seem like such an edge-case thing to do. They can, and they have. So again your point is lost on me. They might like their data in "Application Data", which is where support people get trained to look for application data. Not down in \pythonX.Y\ ... Nothing is stopping them from doing that - but this PEP isn't intended to provide that information. Distutils is stopping them. I don't agree with that and I can present many applications as evidence. You yourself mentioned mercurial and it looks for mercurial.ini in the user's appdata directory. Regardless, this discussion isn't about distutils. It does - many applications written in Python exist which do exactly that. Yes. And they don't use any of the built in facilities, under windows. To continue the mercurial example - mercurial will not use sysconfig to determine where to look for mercurial.ini on *any* operating system. sysconfig is not about solving that particular problem. So under windows, then, what is it trying to solve? Thats what I am asking. The same thing it is trying to solve for non-Windows users - various threads here have articulated this well. You needn't feel bad about not having such use-cases yourself - that simply means sysconfig isn't targetted at you - it isn't targetted at application developers on any operating system. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385 progress report
Hi Dirkjan, On 8/02/2010 8:35 AM, Dirkjan Ochtman wrote: ... In fact, a few weeks ago I talked to Brett and we figured that we should probably pin down a deadline. We discussed aiming at May 1, and at this time I think that should be feasible. That also seems to coincide with the release of 2.7b2, though, so maybe we need to do it one week later (or sooner?). Anyway, we figured that a weekend would probably be a good time. If we manage to find a good date, I'll put it in the PEP. Isn't setting a date premature while outstanding issues remain without a timetable for their resolution? As for the current state of The Dreaded EOL Issue, there is an extension which seems to be provide all the needed features, but it appears there are some nasty corner cases still to be fixed. Martin Geisler has been working on it over the sprint, but I think there's more work to be done here. Anyone who wants to jump in would be quite welcome (maybe Martin will clarify here what exactly the remaining issues are). See http://mercurial.selenic.com/wiki/EOLTranslationPlan#TODO - of particular note: * There are transient errors in the tests which Martin is yet to identify. These tests do not require windows to reproduce or fix. * The mercurial tests do not run on Windows. Given the above, most sane Windows developers would hold off on "live" testing of the extension until at least the first issue is resolved - but the second issue makes it very difficult for them to help resolve that. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows registry path not ignored with Py_IgnoreEnvironmentFlag set
On 2/06/2010 11:32 AM, Farshid Lashkari wrote: Hello, I noticed that if Py_IgnoreEnvironmentFlag is enabled, the Windows registry is still used to initialize sys.path during startup. Is this an oversight or intentional? I guess it falls somewhere in the middle - the flag refers to the 'environment' so I believe it hasn't really been considered as applying to the registry - IOW, the reference to 'environment' probably refers to the specific 'environment variables' rather than the more general 'execution environment'. I assumed one of the intentions of this flag is to prevent embedded Python interpreters from being affected by other Python installations. Ignoring the Window registry as well as environment variables seems to make sense in this situation. I agree. If this is an oversight, would it be too late to have this fixed in Python 2.7? Others will have opinions which carry more weight than mine, but I see no reason it should not be fixed for *some* Python version. Assuming no objections from anyone else, I suggest the best way to get this to happen in the short to medium term would be to open a bug with a patch. A bug without a patch would also be worthwhile but would almost certainly cause it to be pushed back to a future 3.x version... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows
On 4/08/2010 11:08 AM, Steve Holden wrote: It's a little disappointing to discover that despite the relatively large number of developers who have received MSDN licenses from Microsoft, none if us have the time to make sure that the buildbots are green for the 2.6.6 release. I wonder if anyone can think of a way we can get some Windows skillz into the group that could assist at ties like this. Some brainstorming might find a way through. I never go looking at the buildbots to look for problems - maybe some way of explicitly bringing such failures to peoples attention would be good - I don't recall seeing anything recently on python-dev which would prompt me to take a look. Visiting http://www.python.org/dev/buildbot/2.6/ shows a single Windows buildbot that seems to have been green for the last few builds - am I looking in the wrong place? Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 384 status
On 31/08/2010 7:54 AM, Nick Coghlan wrote: On Tue, Aug 31, 2010 at 12:47 AM, Michael Foord wrote: An extension compiled for one version of Python will be linked against the version of the C runtime used by that version of Python (if it is compiled with the same version of Visual Studio of course). If the extension binary is to remain compatible with a later version of Python, compiled against a different version of the C runtime, then it *must* be possible for multiple C runtimes to be loaded. If the stable ABI doesn't allow this then binaries will *still* have to be recompiled when we update the version of Visual Studio used to compile Python - defeating the purpose of the PEP. Right? If this is the case then I agree that it should be explicit in the PEP. Ah, I knew it was explicit in the PEP somewhere. The following is currently mentioned in the "Excluded Functions" section: "In addition, functions expecting FILE* are not part of the ABI, to avoid depending on a specific version of the Microsoft C runtime DLL on Windows." It would be interesting to know how, in practice, these FILE pointers come to life. In my experience they are generally obtained via fopen. If that is broadly true, then a middle-ground may be for Python to expose something like Py_fopen, Py_fclose and a PyFILE opaque "handle". API elements which currently take a FILE * could be exposed using a PyFILE * in the ABI. People who didn't care about this level of portability could continue to use the non-ABI FILE * functions, but people who do could use Py_fopen/Py_fclose in place of fopen/fclose but otherwise work as now. A downside is that as mentioned, in practice these FILE pointers may come from more than simply fopen, so few people would be able to leverage this. It also assumes that people open files before handing them to Python, but otherwise don't use that file - it would be a slippery-slope to wind up with Py_fread etc. Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Guido writes: > As Phillip understood, I meant the environment to include the > filesystem (and on Windows, the registry -- in fact, Python on Windows > *has* exactly such a mechanism in the registry, although I believe > it's rarely used these days -- it was done by Mark Hammond to support > COM servers I believe.) It is rarely used these days due to the fact it is truly global to the machine. These days, it is not uncommon to have multiple copies of the same Python version installed on the same box - generally installed privately into an application by the vendor - eg, Komodo and Open Office both do this to some degree. The problem with a global registry is that *all* Python installations honoured it. This meant bugs in the vendor applications, as their 'import foo' did *not* locate the foo module inside their directory, but instead loaded a completely unrelated one, which promptly crashed. A mechanism similar to .pth files, where the "declaration" of a module's location is stored privately to the installation seems a more workable approach. Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python+XUL
> Has anyone heard the Python+XUL community cry "I'm not dead > yet!" or are > they really dead? I haven't seen mentions of new work in these areas > lately. XUL in general seems to have died (so many broken > links on XUL > pages). Was this just a fad? If not, and if there's some > really useful > of it (especially if in Python) please point it out. This work is still ongoing. The primary docs are at http://developer.mozilla.org/en/docs/PyXPCOM, and the #pyxpcom channel on irc.mozilla.org does sometimes see some action. The mozilla trunk has Python in XUL working, and is being used by at least a few people. The ongoing work is fairly sporadic, but is currently aimed at closer integration of Javascript and Python with the DOM - notably enabling "native" attributes on DOM objects to be accessed by other languages. Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 30XZ: Simplified Parsing
Please add my -1 to the chorus here, for the same reasons already expressed.
Cheers,
Mark
> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED]
> ]On Behalf
> Of Jim Jewett
> Sent: Monday, 30 April 2007 1:29 PM
> To: Python 3000; Python Dev
> Subject: [Python-Dev] PEP 30XZ: Simplified Parsing
>
>
> PEP: 30xz
> Title: Simplified Parsing
> Version: $Revision$
> Last-Modified: $Date$
> Author: Jim J. Jewett <[EMAIL PROTECTED]>
> Status: Draft
> Type: Standards Track
> Content-Type: text/plain
> Created: 29-Apr-2007
> Post-History: 29-Apr-2007
>
>
> Abstract
>
> Python initially inherited its parsing from C. While this has
> been generally useful, there are some remnants which have been
> less useful for python, and should be eliminated.
>
> + Implicit String concatenation
>
> + Line continuation with "\"
>
> + 034 as an octal number (== decimal 28). Note that this is
> listed only for completeness; the decision to raise an
> Exception for leading zeros has already been made in the
> context of PEP XXX, about adding a binary literal.
>
>
> Rationale for Removing Implicit String Concatenation
>
> Implicit String concatentation can lead to confusing, or even
> silent, errors. [1]
>
> def f(arg1, arg2=None): pass
>
> f("abc" "def") # forgot the comma, no warning ...
> # silently becomes f("abcdef", None)
>
> or, using the scons build framework,
>
> sourceFiles = [
> 'foo.c',
> 'bar.c',
> #...many lines omitted...
> 'q1000x.c']
>
> It's a common mistake to leave off a comma, and then
> scons complains
> that it can't find 'foo.cbar.c'. This is pretty
> bewildering behavior
> even if you *are* a Python programmer, and not everyone here is.
>
> Note that in C, the implicit concatenation is more
> justified; there
> is no other way to join strings without (at least) a
> function call.
>
> In Python, strings are objects which support the __add__ operator;
> it is possible to write:
>
> "abc" + "def"
>
> Because these are literals, this addition can still be optimized
> away by the compiler.
>
> Guido indicated [2] that this change should be handled by
> PEP, because
> there were a few edge cases with other string operators,
> such as the %.
> The resolution is to treat them the same as today.
>
> ("abc %s def" + "ghi" % var) # fails like today.
> # raises TypeError because of
> # precedence. (% before +)
>
> ("abc" + "def %s ghi" % var) # works like today;
> precedence makes
> # the optimization more
> difficult to
> # recognize, but does
> not change the
> # semantics.
>
> ("abc %s def" + "ghi") % var # works like today, because of
> # precedence: () before %
> # CPython compiler can already
> # add the literals at
> compile-time.
>
>
> Rationale for Removing Explicit Line Continuation
>
> A terminal "\" indicates that the logical line is continued on the
> following physical line (after whitespace).
>
> Note that a non-terminal "\" does not have this meaning,
> even if the
> only additional characters are invisible whitespace.
> (Python depends
> heavily on *visible* whitespace at the beginning of a
> line; it does
> not otherwise depend on *invisible* terminal whitespace.) Adding
> whitespace after a "\" will typically cause a syntax error rather
> than a silent bug, but it still isn't desirable.
>
> The reason to keep "\" is that occasionally code looks better with
> a "\" than with a () pair.
>
> assert True, (
> "This Paren is goofy")
>
> But realistically, that paren is no worse than a "\". The only
> advantage of "\" is that it is slightly more familiar to users of
> C-based languages. These same languages all also support line
> continuation with (), so reading code will not be a problem, and
> there will be one less rule to learn for people entirely new to
> programming.
>
>
> Rationale for Removing Implicit Octal Literals
>
> This decision should be covered by PEP ???, on numeric literals.
> It is mentioned here only for completeness.
>
> C treats integers beginning with "0" as octal, rather
> than decimal.
> Historically, Python has inherited this usage. This has caused
> quite a few annoying bugs for people who forgot the rule, and
> tried to line up their constants.
>
> a = 123
> b = 024 # really only 20, because octal
> c = 245
>
> In Python 3.0, the second line will instead raise a SyntaxError,
> because of the
Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream
On 5/03/2011 8:21 AM, "Martin v. Löwis" wrote: ... As for Windows support: we currently don't install a python3.exe binary, let alone python2.exe or pythonw2.exe (or is that python2w.exe?). I'll adjust the installer if the PEP asks me to. For the reasons discussed, I'm -0 on the change (i.e. double-clicking .py will continue to launch the most-recently installed Python, rather than the "right" one, and setting PYTHONPATH will continue to break installations). I agree with the -0 - I simply don't think it will, in practice, make anyone's life much easier. To run python2 and python3 based scripts in the same environment already requires some manual work by the machine owner (both directories will need to be added to the path) so the additional burden of some other steps (eg, .bat files, doskey alaises etc) doesn't seem that great. There is also a small risk of confusion - people may believe python.exe and python2.exe/python3.exe have different purposes and be confused about when to use which. I think this discussion should be divorced from this PEP and taken up with the discussion about the PATH and the "last installed wins" issue Martin mentions - only all of them taken together will "fix" this issue - not that I personally consider it particularly broken - more like "sub-optimal" :) Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream
On 6/03/2011 12:37 AM, Michael Foord wrote: On 05/03/2011 07:02, Nick Coghlan wrote: On Sat, Mar 5, 2011 at 10:47 AM, Mark Hammond wrote: I think this discussion should be divorced from this PEP and taken up with the discussion about the PATH and the "last installed wins" issue Martin mentions - only all of them taken together will "fix" this issue - not that I personally consider it particularly broken - more like "sub-optimal" :) I updated the draft PEP, explicitly bringing Mac OS X and Cygwin within the scope of the recommendation, and excluding PYTHON* environment variable considerations and Windows-related proposals. The Windows section does include a hit-list that may serve as a useful starting point if someone else felt like starting a Windows specific PEP, though. The issues are relevant to Windows (there are *other* issues as well but that doesn't mean that the same issue doesn't apply). Guido also said he would like to see Windows addressed. What issues specifically? If I look at the current PEP language, it refers to "the default version of the Python interpreter" and "all new code that needs to invoke the Python interpreter". Neither of these apply in any meaningful way to Windows. It could be argued that the latter could include .bat files which use Python but that sounds like a theoretical problem (ie, I haven't heard much noise about that) where PEP394 is addressing an observed practical problem. On Windows it seems to be just about 'python2' and 'python3' doing the correct thing in a default command prompt, which while desirable ignores the broader issues (eg, the file associations and anything else "windowsy" and not driven by the command-prompt) I'll be happy to try and come to some consensus with Martin (and others) on what we *can* do for Windows (as similar to the other platforms as possible would be my goal) and provide appropriate text for the PEP. IIUC, the PEP language is referring to links which point to a specific version of Python and that there is no suggestion a 'python3' will live in the Python 3 binary tree. If that is correct and assuming we don't want to investigate using links on Windows, I'd suggest the best analogy to the *nix situation could be simply for the installers to generate "python2.bat" and "python3.bat" files and put them in System32 with a "last installed wins" policy. The fact this doesn't help users installing Python "just for me" is fine too from the POV of matching *nix systems - a user installing a private Python build on *nix also doesn't get the python2 and python3 conveniences, nor the automatic PATH convenience. But in practice I expect this would *not* be OK - which just highlights the risks of trying to generalize a specific observed problem on one OS to others. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream
On 6/03/2011 11:51 PM, Dj Gilcrease wrote: > Why not modify the windows installers to install into C:\Python\X.Y > and have the .bat files generated in C:\Python which is what I have > been doing manually since py25. I just add C:\Python to the system > Path then create/modify the bat files for new versions of python I > install. That sounds like a reasonable scheme people may choose to use - however, it doesn't really address the basic issue - something still needs to add c:\Python to PATH, and the scheme itself doesn't really necessitate the common parent directory - ie, it is fundamentally the same (if less "clean") as the existing "c:\pythonx" with a (eg) "c:\python-cmd" or "c:\python-scripts" directory with the .bat files - especially given the various installations can be located via the registry. ... > Something I have been thinking about recently though is outside the > scope of the pep is writing a python.exe, to replace the python.bat, > that would try to read the shebang line of the file to send it to the > right version of python. Then I just associate py files with the > dispatcher exe and everything should work as intended. But where would such a python.exe live and how would that directory end up on the PATH? On the more general idea though, it could have legs as it would solve the file association issue for files which include the shebang and arrange for the status-quo (or better) for files which don't... But this sounds like a different PEP ;) Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream
On 7/03/2011 1:55 AM, Paul Moore wrote: On 6 March 2011 02:33, Mark Hammond wrote: IIUC, the PEP language is referring to links which point to a specific version of Python and that there is no suggestion a 'python3' will live in the Python 3 binary tree. If that is correct and assuming we don't want to investigate using links on Windows, I'd suggest the best analogy to the *nix situation could be simply for the installers to generate "python2.bat" and "python3.bat" files and put them in System32 with a "last installed wins" policy. No, no, no! Please no! Bat files are a lousy way of writing wrappers on Windows, as they don't nest. I've lost count of the number of times I've been tripped up by putting a series of commands into a batch file, only to have them fail because somewhere in the middle is a bat file wrapper which makes the "outer" bat file stop part way through :-( To be clear, I was suggesting that using .bat files in system32 is a close analogy to the *nix situation - I didn't mean to advocate for it to actually happen :) Further, I see the creation of a python3.exe in the Python install directory as quite different than the *nix situation (where the 'python3' link is not in the install dir, but instead in a 'system' dir). IOW, I was trying to point out the solution to the problem on *nix doesn't translate that well to Windows, so Windows should not be considered as part of this PEP. That isn't to suggest Windows should not be considered at all. That said though, I'm only -0 on python2.exe/python3.exe - I don't think it will hurt, but also don't think it will help that much in practice. It may also turn out to be unnecessary should a "complete" solution be implemented - eg, a "python launcher" which (a) read the shebang lines and (b) allowed something like "python -3" on the command-line would render both python3.exe and requests to have multiple installed Python versions on the PATH redundant. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream
On 7/03/2011 10:04 AM, Michael Foord wrote: Paul Moore was +1 on Windows being included. Mark did accept that some of the changes were "desirable", but was also concerned they didn't address all the issues on Windows. I *would* like to see all the issues addressed but I think that is outside the scope of this PEP. I simply think Windows is outside the scope of this PEP - the title is "The "python" command on Unix-Like Systems" - IMO it should be kept short and focus on exactly what the title says. ... A generic Python launcher that could be used for the file association and honours the Unix style shebang line, delegating to the appropriate version of python, is an interesting idea but out of the scope of this PEP. As above - I think Windows should be kept out of scope. It would be interesting to see if Mark, or others, are strongly opposed to these ideas or have strong preferences. If the biggest downside is really that it still leaves some issues unresolved then lets do this and tackle the other issues separately. As mentioned in my other email, I'm -0 as I think they are slightly misguided. Why not leave the PEP with "Unix-like" in the title to Unix-like systems and create one with Windows in the title for Windows? :) Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream
On 7/03/2011 3:30 PM, Glenn Linderman wrote: I'm only against the overhead of a helper written in Python, since it would have to launch Python (some explicit version) to run the helper script, and then launch the "right" version of Python to execute the real script. You mention a thin executable wrapper, and I have no problem with the overhead of that, probably. Yeah - I think any such wrapper could be a reasonably small C program - although I guess that depends on the decided functionality :) ... Since Windows does not use a #! line, then the solutions for (2) must be different. The ones that I can think of are: (A) declare the Python version in the name of the script file. By doing so in the extension, additional Windows associations could be created to launch different versions of Python. Switching versions would be as simple as renaming the file, changing the extension to include a different version. However, various tools would have to learn about additional extensions that mean Python (syntax directed editors, etc.) And it would be hard to have a script in a module, unless the importer recognized all those extensions too. So this "solution" has ripple effects that make it unattractive, although it is simple to implement the basic feature. Agreed - I find this unattractive. (B) declare the Python version in the content of the script file. This cures most of the ripple effects of the above, but requires a "launcher" or "wrapper" program to be designed, implemented, and installed. There are a variety of subsolutions for different ways of declaring the version. Without putting too much thought into it, I think a simple scheme could work where the path must either be "/usr/bin/python[23]?" or a fully-qualified path to a Python executable. IIUC, this should allow most scripts to have a shebang line that "does the right thing" on *nix and Windows systems while still offering maximum flexibility on Windows. By default, for Python, the wrapper would get associated only with .py files. There would probably also need to be the same for ".pyw", but it would be almost identical to the .py handling. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream
On 7/03/2011 4:34 PM, Nick Coghlan wrote: On Mon, Mar 7, 2011 at 3:19 PM, Mark Hammond wrote: Without putting too much thought into it, I think a simple scheme could work where the path must either be "/usr/bin/python[23]?" or a fully-qualified path to a Python executable. IIUC, this should allow most scripts to have a shebang line that "does the right thing" on *nix and Windows systems while still offering maximum flexibility on Windows. Such a scheme should be able to handle "/usr/bin/env python2" and variants as well. Agreed - that is just my ignorance showing through :) Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream
On 7/03/2011 9:33 PM, Paul Moore wrote: That sounds like a fairly cool idea. So if I follow what you're suggesting, we'd have a single python.exe, probably installed in system32, which did the necessary command line juggling and shebang parsing, then simply redirected to the appropriate Python interpreter? Presumably that launcher would be pretty version independent, so (a) the one that gets installed with Python 3.3 would support older versions even though they didn't include the launcher themselves, (b) overwriting the launcher when a new version of Python is installed wouldn't be too big a deal, and (c) it could be released as a standalone package for people with only older versions of Python installed? Yup - although I think a pythonw.exe launcher would be needed too (for the same reasons we need python.exe and pythonw.exe today) I like this idea. If I had the spare time (I don't :-() I'd work on this myself. +1 from me. Cool - I may have a go at this over the next few weekends... :) Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream
On 8/03/2011 7:33 AM, Michael Foord wrote: A python launcher as you describe is a *great* idea. A few concerns: * we're missing an opportunity to do something easy (Martin is happy to modify the installer and says it is easy) for something that may or may not happen Don't let my -0 stop anyone :) * will you call it python.exe? will it be installed by the python installer? - I doubt calling it python.exe will fly, but I'm not sure. If so what will you call what is currently 'python.exe'? - if not then "python foo.py" on the command line will *still* not work... Calling it python.exe would make the most sense for people who don't look behind the curtain, but I agree it could potentially be confusing for people. Further, we would need to ensure we didn't create an infinite loop where the launcher python.exe found a python.exe it thought was an appropriate sub-process, but where it turns out it is actually another launcher. Having it installed by the Python installer also makes sense to me but I'd be very interested in Martin's take on this (and also on everything else we are discussing here). * we're (yet again) making instructions for running stuff on Windows *different* to other platforms (and making tutorials written for other pythons "not work" in certain ways) Actually, I'd argue we are making them more similar as we would be honoring a shebang line and allowing the exact same mechanism be used to denote the major version of Python needed. * as I work with multiple platforms it would be really nice if the same invocations worked across all of them - whilst I say again that I really like the idea of the launcher it doesn't conflict with the other suggestions (creating multiple binaries) and as you (Mark) say it wouldn't hurt... So why not do both? We could create the extra binaries to bring Python on Windows inline with the unix conventions for command line invocations, and the new launcher can follow on as a nice addition. Note that the discussions about the Python installer adding to the PATH won't be *ended* by the creation of the installer. A typical install on a Unix-like system adds various other utilities to the path that merely adding the top-level of your Python install on Windows still doesn't add. In particular distutils installed scripts go into a subdirectory of your Python install. The distutils scripts are a good point. To be honest, apart from the "file association" issue, I can't see much advantage in doing both - if all Python directories end up on your path such that "python3 foo.py" magically works, then the launcher script is adding complexity without bringing much to the table. Further, I'm somewhat skeptical that the file associations are used by that many people in the real world - currently when you double-click on a script you get a temp console created, and should a traceback be raised, you get to see it for about 2ms before the console is destroyed. Some people may be .pyw to avoid that, but then their script has to go to extraordinary lengths to display such errors in a UI of some kind. I suspect most people just find it more convenient to launch such scripts from a console. Maybe a quick poll on python-list would be reasonable... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream
On 8/03/2011 10:15 AM, Greg Ewing wrote: Mark Hammond wrote: Yup - although I think a pythonw.exe launcher would be needed too Couldn't the launcher look at the extension of the file being launched to decide about this? Nope - the launcher itself must be marked as "console" or "windows", and hence would suffer the exact same problem which needs us to have python.exe and pythonw.exe in the first place (ie, in some cases you get a "temp" console where you don't want one, and in other cases you get a new console where you wanted an existing one to be used) Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream
On 9/03/2011 1:43 PM, Glenn Linderman wrote: I'm of the opinion that attempting to parse a Unix #! line, and intuit what would be the equivalent on Windows is unnecessarily complex and error prone, and assumes that the variant systems are configured using the same guidelines (which the Python community may espouse, but may not be followed by all distributions, sysadmins, or users). That's why I propose a different line for Windows... it is as simple as the long-proven Unix #! line, but imposes no restrictions on or requirements that there be a #! line; it has more flexibility in that it could invoke different versions or provide different options on Unix and Windows if necessary for some environments. #!/usr/bin/env python2.6 -B #@c:\python26\python2.6.exe or #!/usr/bin/python2.5 #@"C:\Program Files (x86)\IronPython 2.6 for .NET 4.0\ipy.exe" I don't think that buys us much. A script with "#!/usr/bin/env python" could be distributed with an expectation it will work across various different machines (and possibly even different operating systems). A script with "c:\..." could not be distributed and expected to work reliably anywhere. ie, any script with a #! line (or even a #@ line) with a fully qualified Windows path can only be expected to work on a single machine - so there is no need to support both #! and #@ as the script is not even cross-machine portable, let alone cross-platform portable. The only way to expect a #! line to work across machines would be to have a "virtual" path - eg just "python2.6" without any path specifier at all. In that case, I see no problem with reusing the #! from *nix systems and treating "/usr/bin" etc as a "virtual" specifier on Windows. If people find a need on Windows to add a fully-qualified path to this line (whatever the spelling), they are implicitly saying this script works only on the current machine. IOW, as soon as someone has: #!/usr/bin/env python2.6 -B #@c:\python26\python2.6.exe in their script, the script is targeted at exactly 1 specific machine, so why not just reuse the #! syntax? OTOH, if an existing script has: #!/usr/bin/env python2.6 -B They are attempting to declare in a portable way that Python 2.6 is necessary - so why force them to add a #@ line to make it work on Windows when the #! line is the only clue Windows needs to make it work automagically? Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream
On 9/03/2011 5:05 PM, Glenn Linderman wrote: Standard installation paths are accepted by about 99% of the users, so embedding standard installation paths can work well for that batch of users. Of course, Windows changes the standard path periodically, so that it different from versions prior to and including WinXP versus versions after WinXP. And they had no standard before WinNT (or if they did, few followed it). Your comment does point out a possible need for multiple standard installation paths just for different versions of Windows, though :( The python installer ignores "Program Files" - as much as anything I'm concerned about the drive letter - IIUC, Python will default to installing on the system drive, which may not be C: ... Your premise that using a standard installation path in #@ restricts the script to be targeted to one machine is fallacious, so the conclusions are also. Sorry - I should have said "some machines with Python installed by the installer". Contrast this to the option of looking up the registry and you have "all machines with Python installed by the installer". I'm not going to quibble about the percentages in the first category as I have no data to guide me, but I'm reasonably confident about the "all" in the second. ... The #! line is clearly is NOT "the only clue Windows needs to make it work automagically" or we wouldn't be having this discussion at all. Sorry - I should have been clear - I meant "the only clue a windows specific Python launcher needs". > And it is not at all clear if a Windows machine contains Jython, IronPython, and Cython which one should be launched by a launcher. I think it is reasonably clear: it would be the same version which would be launched on most *nix systems. Are there any systems where "/usr/bin/python" would use Jython, for example? You could, of course, argue that python-dev is only concerned with CPython, and the launcher can be specific to CPython, and that argument might carry the day, but the CPython registry lookup necessary to make that happen doesn't help the users of alternate implementations a bit, forcing each of them to implement their own launcher as well... I'd have no problem with the Windows launcher supporting other implementations - I expect something like "/usr/bin/jython" would be reasonable. How we locate the installed jython is indeed an interesting problem, but I'd guess it is surmountable - especially if we can get the buy-in of these alternate implementations for their future releases. #!/usr/bin/env python2.6 #@"C:\Program Files\Env.exe" ipy2.6.3.exe As above, the C: assumes Windows is installed on C: - that may be common, but not universal. Further, a 64bit machine would probably want "\Program Files (x86)" (which is becoming more and more common) in .py files and #!/usr/bin/env python2.6 #@"C:\Program Files\Env.exe" ipy2.6.3w.exe in .pyw files. I'm slightly skeptical about that - .pyw files are surely rare on *nix systems and as we've discussed, pyw files are generally not meant to be cmd-line scripts. IOW, I'm skeptical scripts exist which are meant to work with '/usr/bin/env python2.6' on *nix and pythonw.exe on windows. ... So no matter what is done, it won't solve world hunger. Probably that's why Windows users have been left high and dry for years in this regard... there is no "system" Python on it, out of the box, so the only users are those that are smart enough to download and install and configure things, and those who use a Python embedded into an application, at the cost of a Python installation per application, because there has been no "system" Python, and because Windows users are not perceived to be capable of, or willing to bother with, downloading and installing various requisite dependencies. The resulting vacuum has been filled by people with different points of view, because there has been no technique implemented, none declared to be "best", and anarchy and variant workarounds have abounded. I'm glad solving world hunger is out of scope for this :) I understand your position but my personal opinion is that simple support for #! is more desirable. I'd be happy to go with the consensus though... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bugs in thread_nt.h
These issues are best put in the tracker so they don't get lost -
especially at the moment with lots of regulars at pycon.
It would also be good to know if there is an actual behaviour bug caused
by this (ie, what problems can be observed which are caused by the
current code?)
Cheers,
Mark
On 10/03/2011 12:25 PM, Sturla Molden wrote:
Atomic operations (InterlockedCompareExchange, et al.) are used on the
field 'owned' in NRMUTEX. These methods require the memory to be aligned
on 32-byte boundaries. They also require the volatile qualifer. Three
small changes are therefore needed (see below).
Regards,
Sturla Molden
typedef struct NRMUTEX {
volatile LONG owned ; /* Bugfix: remember volatile */
DWORD thread_id ;
HANDLE hevent ;
} NRMUTEX, *PNRMUTEX;
NRMUTEX
AllocNonRecursiveMutex(void)
{
PNRMUTEX mutex = (PNRMUTEX)_aligned_malloc(sizeof(NRMUTEX),32) ; /*
Bugfix: align to 32-bytes */
if (mutex && !InitializeNonRecursiveMutex(mutex))
{
free(mutex) ;
mutex = NULL ;
}
return mutex ;
}
void
FreeNonRecursiveMutex(PNRMUTEX mutex)
{
if (mutex)
{
DeleteNonRecursiveMutex(mutex) ;
_aligned_free(mutex) ; /* Bugfix: align to 32-bytes */
}
}
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/skippy.hammond%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
Hi all, During the huge thread about PEP 394, a suggestion was made that a "launcher" for Python on Windows could be implemented which would allow for some of the guidelines in that PEP to apply to the Windows version of Python. I've attached the first draft of a PEP for such a launcher and put the first draft of the "reference implementation" (implemented in Python 3) at http://starship.python.net/crew/mhammond/downloads/python_launcher.py The reference implementation is fairly complete WRT the features described in the PEP. However, I'm sure there will be comments on both the PEP itself and the implementation, so I'm putting them up for discussion now rather than spending more time polishing things which may end up changing. I'll start work on a C implementation once there is broad agreement on the functionality in the reference implementation. All comments welcome, including, but not limited to: * Is this really PEP material, or will turning the PEP into a regular spec be suitable? * If it is a PEP, is it "Standards Track" or "Informational"? * Does the functionality described strike the right balance between simplicity and usefulness? * Does the strategy have general support from Martin, who as the person making Python distributions would need to be involved in at least some aspects of it (specifically, having the installer MSI install the launcher and update the file associations to reference it). Thanks, Mark PEP: ?? Title: Python launcher for Windows Version: $Revision$ Last-Modified: $Date$ Author: Mark Hammond Status: Draft Type: Standards Track or Informational ? Content-Type: text/x-rst Created: 15-Mar-2011 Abstract This PEP describes a Python launcher for the Windows platform. A Python launcher is a single executable which uses a number of heuristics to locate a Python executable and launch it with a specified command line. Rationale Unix-like operating systems (referred to simply as "Unix" in this PEP) allow scripts to be executed as if they were executable images by examining the script for a "shebang" line which specified the actual executable to be used to run the script. This is described in detail in the evecve(2) man page [1]. Additionally, these operating systems provide symbolic-links to Python executables in well-known directories. For example, many systems will have a link /usr/bin/python which references a particular version of Python installed under the operating-system. These symbolic links allow Python to be executed without regard for where Python it actually installed on the machine (eg, without requiring the path where Python is actually installed to be referenced in the shebang line or in the PATH. PEP 394 'The "python" command on Unix-Like Systems' [2] describes additional conventions for more fine-grained specification of a particular Python version. These 2 facilities combined allow for a portable and somewhat predictable way of both starting Python interactively and for allowing Python scripts to execute. This PEP describes an implementation of a launcher which can offer the same benefits for Python on the Windows platform. An overview of the launcher. This PEP outlines the general functionality and key guidelines of a launcher for Windows. It is accompianied by reference implemention, written in Python, which defines the detailed implementation. Over time, changes to the implementation may be desired - if the changes adhere to the guidelines in this PEP and have been made following the standard Python development model, this PEP need not change. In other words, this PEP makes no attempt to describe in detail every feature offered by the launcher, but instead to offer guidelines the launcher should adhere to. The actual implementation of the launcher will be written in C and will not link directly or indirectly with Python, but the Python based reference implementation should always remain the canonical description of the functionality. Any changes proposed to the launcher functionality should first be made and agreed to in the Python implementation, after which time the C implementation can be changed to reflect the new reference. Any deviations between the functionality of the reference implementation versus the C implementation due should imply the C implementation is in error. It is expected that 2 versions of the launcher will be needed - one which is a console program and one which is a "windows" (ie, GUI) program. These 2 launchers correspond to the 'python.exe' and 'pythonw.exe' executables which currently ship with Python. The console launcher will be named 'py.exe'
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
Thanks for the feedback! On 19/03/2011 7:44 PM, Glenn Linderman wrote: Not all of the ideas below are complementary to each other, some are either or, to allow different thoughts to be inspired or different directions to be taken. Thanks for starting a PEP. On 3/18/2011 11:02 PM, Mark Hammond wrote: The launcher should be as simple as possible (but no simpler.) The launcher could be simpler if it isn't used for launching interactive interpreters as well as script references via Windows associations (more about that after the next quote). The launcher could be simpler if it just read through the file of its first parameter until it finds a line starting with #@ (process as a Windows version of Unix #!) or starting without a # (error case). This avoids the need parse such lines. I don't agree with that at all. The rules for the shebang parsing are very simple - see the man-page I reference in the pep - the file must start with the characters '#!' and the entire command must be < 127 chars. The only real "parsing" needed is to check if the specified command starts with one of 2 fixed strings. I believe this is simpler than parsing multiple lines of the file. > Remember, the typical Windows user is not likely to place a #! line in their scripts in the first place, so teaching them what a Unix #! line is, and how the parameter after it should be something that Windows doesn't even use, and the launcher has to work hard to interpret, is not as simple as possible. I disagree with various aspects of that - the "typical Windows user" is not going to add a shebang or variation of to their Python source files, period. IMO, the kind of user who would is already somewhat likely to know of the shebang convention, so it would not be foreign. Those remaining who are not familiar with it can simply be pointed at existing docs etc for the shebang line and their new knowledge now works on more than Windows (and more than Python on non-Windows platforms) The launcher could be simpler if the Python installer placed versioned Python executables on the PATH. Unfortunately, historically it hasn't. If it did, would, or the launcher installer would place them there for pre-existing versions of Python, then the launcher could work by launching the appropriate version of Python, expecting Windows to find it on the PATH. The PEP doesn't address the level of internal complexity of the launcher necessary to find where Python has been installed, neither for CPython, nor for the alternate implementations to be supported. The PEP intentionally doesn't, but the implementation does - it already does exactly that for CPython. Other implementation may need a different strategy, but we can cross that bridge when we come to it. The launcher could be simpler if a directory \usr\bin were created under Windows Program Files, placed on the PATH, and %ProgramFiles% prepended to the Unix #! line, with the Python/Jython/Cython installers placing appropriately versioned executables in that directory. Could even start a trend for programs ported from Unix. One could even place an "env" program there, for more simplicity. Again, I disagree - I think in practice the code would be more complex, and having Python assert it "owns" such directories and executables is a can of worms we should avoid. * When used to launch an interactive Python interpreter, the launcher will support the first command-line argument optionally be a version specifier in the form "-n[.n]" (where n is a single integer) to nominate a specific version be used. For example, while "py.exe" may locate and launch the latest Python 2.x implementation installed, a command-line such as "py.exe -3" could specify the latest Python 3.x implementation be launched, while "py.exe -2.6" could specify Python 2.6 be located and launched. If a Python 2.x implementation is desired to be launched with the -3 flag, the command-line would need to be similar to "py.exe -2 -3" (or the specific version of Python could obviously be launched manually without use of this launcher.) I think that a python launcher that is "on the PATH" that could be used to launch an interactive Python, should be different than one that is used to launch .py[w] scripts. I believe that if you know the Python you want is already on the PATH, you should just use 'python.exe' instead of 'py.exe'. I don't see any reason to use this launcher for interactive Python sessions where this is the case. 1) python should be invoked interactively by typing "python" or "pythonX[.Y]" at the CMD prompt, not "py". This can be done without a launcher, if appropriate versioned pythons are placed on th
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
On 20/03/2011 4:15 AM, Dj Gilcrease wrote: On Sat, Mar 19, 2011 at 4:44 AM, Glenn Linderman wrote: 2) If the launcher provides command line options for the "benefit" of launching interactive Python interpreters, those command line options can have data puns with script names, or can conflict with actual Python options. I believe Python 2 already has a -3 option, for example. FWIW, the point about Python 2 supporting a -3 arg is explicitly mentioned in the PEP. Windows users are not trained that "-" introduces an option syntax, but rather "/". Most _programmer_ users would probably be aware of "-" as an option syntax, but Python is used as a language for non-programmers in some circles, and few Windows non-programmers understand "/" much less "-" and not even command lines very well. So not using a launcher for launching interactive Python sidesteps all that: Python itself is introduced and taught, and no need to teach about (or even have) launcher options that could possibly conflict and confuse, in addition to Python options that may conflict with script names already. (I have seen lots of Windows users use leading punctuation characters on filenames to affect sort order and grouping of files in My Documents, not knowing they can create subdirectories/subfolders, or not wanting to bother with them, since all their applications default to opening things from My Documents.) The command line options I disagree with as well. If the user wants to test a script that has a shebang of python2 with python3 then they should explicitly launch it that way just like you would have to do on *nix See my reply to Glenn on this - the optional -N.N argument must be the first arg, as must the file with the shebang - so if you specify -N.N, no shebang processing is done at all. I expect that in general, the key advantage of shebang parsing will be when the script is launched via file association - eg, entering the command "foo.py" directly at the command-prompt or double-clicking from explorer. Windows itself will lookup the association and execute "py.exe foo.py" - there is no opportunity for the user to arrange for arguments to be inserted in that command (only at the tail). If a user explicitly specifies the command "py.exe -3 foo.py", I personally think it is fine the shebang line is ignored as the user has expressed a clear intention that a specific version of Python be used. No I think there should be at max 2 environment variables and they should be explicitly set by the user not added by default PYTHON_2 If set would override the latest version of python2 that is launched via a shebang line ending in python or python2 but not python2.x PYTHON_3 if set would override the latest version of python3 that is launched via a shebang line ending in python3 but not python3.x The reference implementation currently supports 3 environment variables - while they have different names, 2 of them work as you describe above. The third is used to nominate how a simple 'python' with no version qualifiers should be treated - I wanted the ability for the simple 'python' to execute a python 3.x version if desired by the user. Thank you Mark for writing up this pep. It is almost identical to the one I was working up but covers more details and actually provides an implementation example. Other then the command line args I agree with everything covered. Thanks for the feedback! Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
On 20/03/2011 5:49 AM, "Martin v. Löwis" wrote: * Is this really PEP material, or will turning the PEP into a regular spec be suitable? It's PEP material if it is contentious, which I believe it is. Of course it is - this is python-dev . I, for example, will find issues with it if the implementation uses CreateProcess at some point - the launcher should IMO run Python directly, rather than creating a new process. This, in turn, will probably mean that you cannot actually implement the PEP in Python. Out of curiosity, what is your objection to having the child process? I guess it must be the system resources needed for the parent process while it waits for the child to terminate so the exit code can be reflected correctly "up stream"? I would have 3 main problems with dynamically loading Python into the launcher process: * It may be difficult or impossible to do for implementations other than CPython. * There are subtle differences when Python is loaded by an executable in the "install" directory versus by an executable that is not. Specifically, I'm thinking about the differences in how the default sys.path is populated. * The registry doesn't currently tell us exactly where pythonxx.dll can be located, but does tell is where python.exe is. Thus things would get a little more complex as we go snooping for the location of the DLL. Note that these are almost certainly surmountable, but I wonder if the benefits outweigh the costs. * If it is a PEP, is it "Standards Track" or "Informational"? If you are proposing to change the CPython code base (which I think you should), it's standards track. If you are talking about a tool provided by you on PyPI, you actually don't need to discuss it with anybody. Thanks - "Standards Track" it is then :) FWIW, I have no interest in making this a stand-alone project or distribution. If there is agreement that this shouldn't be part of Python itself (which may well be a reasonable decision to make), I'll let the idea die. * Does the functionality described strike the right balance between simplicity and usefulness? It leaves a number of issues still open, so it's hard to tell yet. For example, the 32-bit vs. 64-bit issue should be resolved (which I think comes down to having 4 binaries). This is an interesting one and one which I don't have a firm opinion about - but the soft opinion I have is something along the lines of "the functionality should not be dictated by the bittedness of the launcher itself". IOW, if we decide that a 64bit implementation of Python should be chosen over a 32bit implementation of the exact same version, this should be done by both the 32bit and 64bit launchers. Of course, I welcome more thoughts on this and could be easily swayed. Plus, it should talk about installation of multiple copies of the same Python version, e.g. from ActivePython, EPD, etc. Good point - this is something I haven't considered at all. Any thoughts on this from anyone? Notice that the PEP doesn't talk about file associations yet (but it should). Actually, it does say '...the "console" version of the launcher should be associated with .py files and the "windows" version associated with .pyw files.' What extra level of detail do you feel is necessary? I'm fine with the strategy, but I feel that the devil's in the detail. Indeed! But general approval for the strategy is an encouraging first step :) Thanks! Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
On 20/03/2011 6:22 PM, Glenn Linderman wrote: On 3/19/2011 7:38 PM, Mark Hammond wrote: ... A Windows user who has only learned Python 2.x programming would not necessarily have ever heard of execve, would not realize execve(2) means it is from the 2nd chapter of the Unix man pages, meaning an API call, would not know C programming, would not know what #include means, would not understand syntax like "const" or "[]". The description is a bit friendlier, if they get that far, but they may still be quite confused by wondering how to create and provide argv and envp in the right forms, wondering what SIGCHLD, PID, set-user-ID bit, SIG_IGN, SIG_DFL, SIGTRAP, set-group-ID, effective id, a.out, binary executable, shared library stubs, ld.so(8), ELF executable, PT_INTERP, and /lib/ld-unix.so.1 (and .2) are. Sorry, I didn't mean to imply we would refer people to the execve man page, but simply that the man page defines the behaviour from the POV of the PEP. Ah, but Mark! You are confusing (and maybe I do in some of my comments too) the complexity of the launcher versus the complexity of describing the launcher. I believe it will be simple to describe it -as you mention, the execve man page needs only a couple of sentences - I believe with just a few more it can be documented with the same clarity and quality as the rest of the Python docs for Windows users. The PEP could therefore be clearer by discussing the CPython implementation strategy, even if it points out that other implementations may need a different strategy. I believe it states fairly clearly that the reference implementation defines much of this, and the reference implementation I referred to does indeed implement (and therefore define) it. The PEP does make mention of other implementations. So I think we are agreeing here, mostly. And you mentioned having the CPython implementation PythonX.Y get on the PATH somehow. So just keep the python launcher off the PATH, remove its then unnecessary option syntax, and we agree totally :) Not quite - my point is that *if* a CPython implementation is on the path (which many users arrange), then there is no advantage to using the launcher for that specific Python. I personally don't have any CPython implementation on my path (actually, that isn't totally true - cygwin's Python is for my command-prompts), and therefore would find the behaviour described in the launcher incredibly useful. This scratches my itch and unless the option does harm for people who choose not to use it, I see no problem with supporting it. Does anyone see any benefit to using a launcher, if PythonX.Y is on the PATH? I don't. Fair enough. But users who do not have a PythonX.Y on the PATH (even if they have another X.Y on it) will. Per the beginning of this email, I think you need to stay far away from referencing the execve man page in a description of a Windows launcher, except perhaps as a footnote. The PEP is not intended to be user documentation, but if it is to be used in that way I'd be happy to document it directly in the PEP. Not using the launcher from the command line, and it not having any available options, would sidestep this issue nicely. I don't see what issue needs to be side-stepped. IMO the feature can be clearly described (even if my initial PEP didn't do so) and easily avoided if not desired. No, the workaround of using fully-qualified executable names in the shebang line makes the script now dysfunctional on Unix. No - as the PEP says, if it can be expected to work on Unix, it should be expected to work on Windows. Otherwise, the script is inherently dysfunctional on Unix, so the fully-qualified path is perfectly reasonable. I don't think you'll get "always", because of environments that use different implementations on different platforms. To get always, you need the flexibility to specify implementation and version for each platform separately... two lines. Sorry, but I just don't go for this at all. Please don't continue to advocate for that in a PEP I am sponsoring - just propose your own. Let's enumerate... and maybe if we keep talking we'll increase the agreement list and decrease the disagreement list. New ideas in this message: 1) If launcher doesn't find a #!/#@ line, it should run python2 to support scripts without such lines. Sorry if the PEP didn't make it clear, but that is how the reference implementation works. You might like to try it. Agreements: 1) Python needs PythonX.Y to be on the PATH Nope. 2) Need a launcher to handle Windows associations that otherwise only allow one version for one extension 3) Need a separate launcher for .py and .pyw Yep. 4) Support for as many environments as possible, and as many installations as possible for each environment. Nope - "supports standard Python
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
On 20/03/2011 8:36 PM, Terry Reedy wrote: On 3/20/2011 3:22 AM, Glenn Linderman wrote: On 3/19/2011 7:38 PM, Mark Hammond wrote: [snip] As both a writer and reader, I would like to just add, for instance, #! python3 (or 3.3 or whatever) and have the launcher do the 'right thing'. If there was general agreement, I'd be happy for the PEP to support that variation. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
On 20/03/2011 8:37 PM, "Martin v. Löwis" wrote: ... ... problems with creating child processes: - applications using the debug API, PSAPI, etc. will be confused if the action all happens in a child process. I can accept that they have to adjust, though. Some of these uses probably shouldn't use the launcher directly - eg, ISAPI apps and COM objects which have a separate registration step could register a specific python.exe - the installation script itself could by launched by the launcher, but that script could continue to use sys.executable to record the specific python.exe. A side effect of this though is that if a newer version of python is installed such that "py.exe register.py" now used the later version, the existing registration would still use the older version until re-registered. I think that is reasonable though. - TerminateProcess must work correctly, and I see no way to achieve that (i.e. terminating the parent process won't terminate the child) I understand this concern (and meant to say this in reply to Eugene's mail), but wonder if it isn't similar to the above - that if some program wants to launch python and note the id of the child process so it can terminate the child later, then python.exe could continue to be used. OTOH, if you are terminating the process by using Process Manager or something else to find a 'python.exe', that will continue to work and the parent py.exe will automatically silently terminate. That wouldn't bother me. * There are subtle differences when Python is loaded by an executable in the "install" directory versus by an executable that is not. Specifically, I'm thinking about the differences in how the default sys.path is populated. This would need to be fixed. I.e. the launcher would have to arrange it so it gives the illusion of having been started from the executable. It isn't immediately obvious how that could be done though - although I admit I haven't looked into it. But in general, I'd be happy for py.exe to avoid CreateProcess for CPython if that could be reasonably done and was the consensus... Thanks, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
Following up myself here... On 20/03/2011 9:25 PM, Mark Hammond wrote: On 20/03/2011 8:37 PM, "Martin v. Löwis" wrote: ... Some of these uses probably shouldn't use the launcher directly - eg, ISAPI apps and COM objects which have a separate registration step could register a specific python.exe - the installation script itself could by launched by the launcher, but that script could continue to use sys.executable to record the specific python.exe. The above raises an interesting question - if the launcher executed Python in-process, what would sys.executable be? I can imagine there are few scenarios where it would be desirable to have it refer to the launcher and a number of scenarios where it would be undesirable and possibly break existing scripts. Consider when a script with a "../python3" shebang line is started via the launcher, but that script itself uses sys.executable to launch its own child. If sys.executable was the launcher, that child would likely be started using a python 2.x interpreter... Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
On 20/03/2011 11:56 PM, Dj Gilcrease wrote: ... Before Mark wrote up this pep I had started experimenting with how to make the launcher and I was able to get it to launch python while exiting py.exe and as far as I could tell it behaved just as if I had launched the app directly by double clicking it. In addition to Martin's point, this approach would mean the exit code of the child process probably isn't available to whoever started the launcher. For example, a batch file that tried to use something like "if errorlevel ..." to check the result of the Python process would probably not work as expected. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
On 21/03/2011 10:32 AM, "Martin v. Löwis" wrote: The above raises an interesting question - if the launcher executed Python in-process, what would sys.executable be? I can imagine there are few scenarios where it would be desirable to have it refer to the launcher and a number of scenarios where it would be undesirable and possibly break existing scripts. Interesting question. What is it in COM and ISAPI applications? ISAPI isn't a good example - that uses DLLs. For COM objects hosted in a .exe it will be pythonw.exe. For Windows services it will be pythonservice.exe (which is part of pywin32 and is located simply by assuming it is next to win32service.pyd) Regardless, I don't think they are very likely to break - my concern is more for "normal" scripts which construct a child process cmdline/argv using sys.executable... Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
On 21/03/2011 11:10 AM, "Martin v. Löwis" wrote: Am 21.03.2011 00:52, schrieb Mark Hammond: On 21/03/2011 10:32 AM, "Martin v. Löwis" wrote: The above raises an interesting question - if the launcher executed Python in-process, what would sys.executable be? I can imagine there are few scenarios where it would be desirable to have it refer to the launcher and a number of scenarios where it would be undesirable and possibly break existing scripts. Interesting question. What is it in COM and ISAPI applications? ISAPI isn't a good example - that uses DLLs. Actually, this is precisely the example i was looking at. For COM objects hosted in a .exe it will be pythonw.exe. For Windows services it will be pythonservice.exe (which is part of pywin32 and is located simply by assuming it is next to win32service.pyd) Regardless, I don't think they are very likely to break - my concern is more for "normal" scripts which construct a child process cmdline/argv using sys.executable... If we find out how other cases of just loading the DLL deal with sys.executable, it may help us in guiding how it should be set in the launcher. I guess it is all about context; in an ISAPI application or in a COM object implemented in a DLL, sys.executable will be the name of the host process (which is determined in PC/getpathp.c by calling GetModuleFileName with a NULL handle). People writing ISAPI apps or COM objects are likely to understand the implications of them existing in some process which isn't python[w].exe so will ignore sys.executable and use some other technique to locate a python[w].exe if they need it. "Normal" scripts are less likely to do so - a code search at google for sys.executable shows it used in ways which would possibly break if sys.executable referred to a launcher and the launcher didn't use the "current" version of Python as the default. Often these are in tests or setup.py, but we would need to avoid any such breakage. [Which isn't to suggest we can't avoid such breakage with an "in-process launcher" - it just isn't immediately clear how we would do so] Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
On 21/03/2011 11:15 AM, "Martin v. Löwis" wrote: Am 21.03.2011 00:43, schrieb Greg Ewing: Mark Hammond wrote: The above raises an interesting question - if the launcher executed Python in-process, what would sys.executable be? I think it should be the actual Python executing at that moment, not the launcher. This is the least change from current behaviour and therefore least likely to break existing code. I remain -1 on any proposal that uses subprocesses. It absolutely must be the launcher that is running Python. In fact, I'd call it "python.exe". For clarity, could you please tell us which scenarios you have in mind that cause you to take that position, and why those scenarios couldn't ignore the existence of the launcher and stick with doing exactly what they do now (which presumably is to use python.exe directly)? Thanks, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
On 21/03/2011 12:27 PM, "Martin v. Löwis" wrote: I remain -1 on any proposal that uses subprocesses. It absolutely must be the launcher that is running Python. In fact, I'd call it "python.exe". For clarity, could you please tell us which scenarios you have in mind that cause you to take that position, and why those scenarios couldn't ignore the existence of the launcher and stick with doing exactly what they do now (which presumably is to use python.exe directly)? I'm primarily bothered about the failure to implement TerminateProcess correctly. I don't actually know what use cases would be affected, other then saying that anything launching py.exe could be affect, in particular applications using ShellExecuteEx. I don't think it is feasible to change them all to launch something different instead; some may be out of our control. Potential candidates would be web servers. This fear seems overblown to me as I doubt any such programs exist. A program could not use ShellExecuteEx on an arbitrary file extension and hope to obtain a handle it can use to kill the process - the docs make it clear no handle is returned in a number of common cases and implies that even if you get a handle back you should not kill it (as it explicitly states that a single process may be the target for any number of such calls.) For the same reason, you can't even wait on it unless you know the implementation of the associated program. So this scenario would need to be a program which used ShellExecuteEx only to launch Python programs and therefore felt it could rely on the ability to (a) always get a handle back and (b) use TerminateProcess to kill it. ISTM the likelihood of such a program existing is so low that it should not kill the chances of this PEP being accepted if the sub-process model is the only reasonable way to implement the launcher. ie, let's say we are forced to choose between the following 3 options: * No launcher at all (the status-quo), causing demonstrable breakage in Windows file associations whenever Python 2.x and Python 3.x scripts exist on the same box. * An in-process launcher which caused breakage in a number of reasonably common scenarios for Python programmers, and such breakage could easily be demonstrated. * An out-of-process launcher which caused breakage for the hypothetical program mentioned above, of which no instance can be found and no breakage actually demonstrated. I personally would conclude that the last option is the least worst scenario by a wide margin. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
On 21/03/2011 3:50 PM, Greg Ewing wrote: Martin v. Löwis wrote: Windows doesn't support exec. Hmmm. In that case, if the launcher works by loading a pythonXY.dll, I'd say that sys.executable should point to whatever version of python.exe corresponds to that dll. Generally, things should be made to look as much as possible as if that python.exe had been executed in the first place. Yeah, that would have to be the plan - if there are ways it *doesn't* look like python.exe had been executed it is likely to cause problems. Initially I thought we could just adjust sys.executable after it is set to the "inappropriate" default value, but that would be tricky - this is called by Python during Py_Main() - which also executes the specified script. Thus, the launcher would either: * Call Py_Initialize(), adjust sys.executable, then call Py_Main(). I'm not sure what side-effects this would have though - Py_Main does a number of things before calling Py_Initialize() and having it called before those things are done might have other side-effects. * Re-implement Py_Main and adjust sys.executable immediately after it is set. This would be a huge burden given the size of that routine and the number of different versions we want to support. The first option is the only one I'd really consider reasonable and I'll probably experiment with it once we have enough agreement that the C implementation can proceed - but I'm not confident enough we can make it work correctly to base the entire success or failure of the PEP on that detail. Any other ideas/thoughts about this? Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
On 21/03/2011 1:04 PM, "Martin v. Löwis" wrote: Can you please add a summary of this discussion to the PEP? (also, can you please check in the PEP, and > give it a number?) OK, I'll check it in once I get a PEP number allocated as per PEP1, updated to reflect some of the discussions in this thread. Should I also check the reference implementation in? Maybe next to the PEP text as "pep--reference.py"? Thanks, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
On 22/03/2011 12:04 AM, Paul Moore wrote: I haven't had time to read the PEP yet, so my apologies if this is made explicit there, but is the launcher expected to be solely for implementing file associations? I thought there had been discussions of using it to start the interactive interpreter, and for it having command line arguments (implying direct command line usage). If it can be used directly, there are many other scenarios that might be impacted. Consider a service implemented using SRVANY which uses the launcher. Stopping the service kills the launcher, leaving Python (and the script, ie the service) running... The intention is for the script to be used to interactively start a Python interpreter for the convenience of the developers working interactively. It is not intended to replace all current uses of python.exe though, hence the different name. Problems with things like SVRANY will be caused if those things use file associations to implicitly launch the executable associated with .py files - but those problems are caused by having py.exe associated with .py files, not with the fact that py.exe can also be used to launch python interactively. IOW, SVRANY type use-cases should continue to use python.exe. If problems are caused due to the change in associations, those problems would be caused even if py.exe did not have facilities designed for interactive use. FWIW, I do accept that if an in-process model was used there may be less problems if use-cases like SVRANY happened to use py.exe, so should be supported if possible. But while updating the PEP last night to reference 64bit and 32bit considerations, this idea hit another snag. I changed the PEP to say: On 64bit Windows with both 32bit and 64bit implementations of the same (major.minor) Python version installed, the 64bit version will always be preferred. This will be true for both 32bit and 64bit implementations of the launcher - a 32bit launcher will prefer to execute a 64bit Python installation of the same version if available. This is so the behaviour of the launcher can be predicted knowing only what versions are installed on the PC and without regard to the order in which they were installed. I think that is sensible as needing to know the order of past installations to predict the behaviour is undesirable. However, that would also be impossible to achieve in an in-process model - a 32bit launcher could not load a 64bit Python into its process, and vice-versa. Even if the wording of the above paragraph changes, if there remains a case where a 64bit launcher needs to load a 32bit Python or vice-versa, the same issue will arise. I'm starting to think the only reasonable compromise will be to use child processes but use the win32 "job" API to arrange for Windows to kill the child should be the parent be killed - this will need more research and experimentation though - see http://msdn.microsoft.com/en-us/library/ms684161%28v=vs.85%29.aspx for an overview. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Module version variable
On 22/03/2011 8:09 AM, Barry Warsaw wrote: I started an Informational PEP on this at Pycon, and will try to finish a draft of it this week. (I'm claiming 396 for it.) We nearly had a race for that number! FYI, I just checked in pep-0397 leaving 396 for you :) Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Second draft: PEP397: Python launcher for Windows
Hi all, I've made some changes to the draft PEP and checked it into the PEP repository as PEP397. The reference implementation is currently being tracked at http://bugs.python.org/issue11629. Most of the changes should not be too controversial, but of note: * I've added lots of discussion about the child-process issue and the wording now states, basically, that in-process is not feasible, but if we use the win32 Job API we can arrange for the child to be killed with the parent on Windows XP and later. The reference implementation demonstrates this thanks to an example from Curt Hagenlocher. * 64bit and 32bit considerations have been added, along with basic information about other distros and implementations. * The somewhat controversial optional first command-line parameter which allows you to specify a particular version for interactive use has grown an almost-certain-to-be-controversial option to specify the bittedness of the Python you wish to start. I've added this mainly as it would scratch my personal itch as I tend to have 32bit and 64bit implementations of most versions available and while this isn't particularly common, I'm going to attempt to defend it on the principle of "just don't use it if you don't like it" :) Note the reference implementation does not yet implement this, nor does it implement the general policy about 32bit vs 64bit considerations. Please let me know what you think and what objections/comments/etc remain. Thanks, Mark PEP: 397 Title: Python launcher for Windows Version: $Revision$ Last-Modified: $Date$ Author: Mark Hammond Status: Draft Type: Standards Track or Informational ? Content-Type: text/x-rst Created: 15-Mar-2011 Abstract This PEP describes a Python launcher for the Windows platform. A Python launcher is a single executable which uses a number of heuristics to locate a Python executable and launch it with a specified command line. Rationale Windows provides "file associations" so an executable can be associated with an extension, allowing for scripts to be executed directly in some contexts (eg., double-clicking the file in Windows Explorer.) Until now, a strategy of "last installed Python wins" has been used and while not ideal, has generally been workable due to the conservative changes in Python 2.x releases. As Python 3.x scripts are often syntactically incompatible with Python 2.x scripts, a different strategy must be used to allow files with a '.py' extension to use a different executable based on the Python version the script targets. This will be done by borrowing the existing practices of another operating system - scripts will be able to nominate the version of Python they need by way of a "shebang" line, as described below. Unix-like operating systems (referred to simply as "Unix" in this PEP) allow scripts to be executed as if they were executable images by examining the script for a "shebang" line which specifies the actual executable to be used to run the script. This is described in detail in the evecve(2) man page [1] and while user documentation will be created for this feature, for the purposes of this PEP that man page describes a valid shebang line. Additionally, these operating systems provide symbolic-links to Python executables in well-known directories. For example, many systems will have a link /usr/bin/python which references a particular version of Python installed under the operating-system. These symbolic links allow Python to be executed without regard for where Python it actually installed on the machine (eg., without requiring the path where Python is actually installed to be referenced in the shebang line or in the PATH.) PEP 394 'The "python" command on Unix-Like Systems' [2] describes additional conventions for more fine-grained specification of a particular Python version. These 2 facilities combined allow for a portable and somewhat predictable way of both starting Python interactively and for allowing Python scripts to execute. This PEP describes an implementation of a launcher which can offer the same benefits for Python on the Windows platform and therefore allows the launcher to be the executable associated with '.py' files to support multiple Python versions concurrently. While this PEP offers the ability to use a shebang line which should work on both Windows and Unix, this is not the primary motivation for this PEP - the primary motivation is to allow a specific version to be specified without inventing new syntax or conventions to describe it. An overview of the launcher. This PEP outlines the general functionality and key guidelines of a launch
Re: [Python-Dev] Second draft: PEP397: Python launcher for Windows
On 23/03/2011 6:12 AM, Michael Foord wrote: On 22/03/2011 07:21, Mark Hammond wrote: Hi all, I've made some changes to the draft PEP and checked it into the PEP repository as PEP397. The reference implementation is currently being tracked at http://bugs.python.org/issue11629. Hey Mark, One way of supporting alternative implementations (that may not even have a standard install directory) is allowing configuration. e.g. config.ini [paths] ironpython = c:\Program Files\IronPython 2.7\ipy.exe ironpython2.7 = c:\Program Files\IronPython 2.7\ipy.exe jython = c:\Users\foobar\jython2.5\jython.exe Hi Michael, I'd have no problem with that in general, but how would you feel about letting the PEP stand as it is without this additional requirement and then treat this as an additional feature to be thrashed out separately? I intentionally worded the PEP to specifically allow these kinds of features to be added outside the PEP process. For example, I guess the name of the INI file wouldn't be config.ini, and I guess there might need to be a strategy to allow it to exist in multiple places for when users want this feature but don't have write access to the location of py.exe. Then people might want it to be in the cwd, or in any parent of the cwd, etc... Further, it might also be possible to support this with simple environment variables which might wind up being just as (or more) reasonable (the concept of per-user environment variables already exist and a UI already exists for editing them, and it would allow different cmd-prompts to have different "rules" with minimal complexity) - not that I am necessarily advocating this - I'd just prefer the PEP to not get bogged down with those kinds of issues. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second draft: PEP397: Python launcher for Windows
On 24/03/2011 4:34 AM, Ethan Furman wrote: Michael Foord wrote: To be honest I would be happy with registry entries instead as the alternative implementations can then add to their installers (or provide a script) to add the right registry entry. I'm partial to the config file method myself. I have this vague hope where this tool could end up being a shebang-script-launching tool for other scripting languages as well. That is an explicit non-goal for the launcher - but if it happens to be able to work that way for you, that's great :) If you guys (or anyone) would like to agree on some precise rules for both the location of the config file and its contents and express this as a patch to the PEP text, I have no problem supporting it in the implementations. I'd like to insist that the format of the config file was such that the GetPrivateProfileString() Windows function could be used to extract the data (eg, only '=' can be used to separate the name/value pair, case-insensitive and no support for string interpolation) as I have no interest in writing my own config file parser in C :) Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second draft: PEP397: Python launcher for Windows
On 24/03/2011 1:20 PM, Michael Foord wrote: On 24/03/2011 00:44, Dj Gilcrease wrote: On Wed, Mar 23, 2011 at 8:14 PM, Mark Hammond wrote: If you guys (or anyone) would like to agree on some precise rules for both the location of the config file and its contents and express this as a patch to the PEP text, I have no problem supporting it in the implementations. I'd like to insist that the format of the config file was such that the GetPrivateProfileString() Windows function could be used to extract the data (eg, only '=' can be used to separate the name/value pair, case-insensitive and no support for string interpolation) as I have no interest in writing my own config file parser in C :) In the user directory much like TortoiseHG adds a murcurial.ini file to the users directory for basic globa config the launch could look for a python.ini there and use to to add known paths or version overrides on a user by user basis. A single global location (for shared installs) or a single per-user location for per-user installs would seem to be sensible if the config file route is chosen. My concern with that would be that an administrator may install Python, but the user may not have write access to that global location, leaving that user unable to customize the launcher. OTOH, that is how things work on Unix today - such users could not change what /usr/bin/python points to). Always using a per-user location would mean there is no opportunity to have global settings, but it is unclear which is the lesser of 2 evils. Supporting both might even be reasonable if the rules are very straightforward. I'd still very much like to see the change to the wording of the PEP to describe this feature though, otherwise I fear different people will have different assumptions about exactly what it does and how it does it... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] .hgignore including site-packages and scripts directories?
I'm wondering if it is a reasonable idea to have .hgignore exclude all files from 'Lib/site-packages' and 'Scripts'? As I install packages into my source builds, a 'hg status' lists *many* files in both those directories forcing me to scroll up a number of pages to see files which have actually changed. IIUC, listing a directory in .hgignore doesn't preclude files from that directory being added to hg, and doesn't prevent files in those directories already under hg from being detected as changed. The only downside I can see if that if new files are added to those directories which should be added to hg, a simple "hg st" will not show it - someone must remember and explicitly add it. However, ISTM those files are already likely to be missed given the large amount of noise 'hg st' shows in that directory - the files are likely to be in the middle of a very long list which my brain will be trained to habitually skip over. The number of new files which legitimately need to be added to those directories seem so small that this risk seems worthwhile. Any thoughts? Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] .hgignore including site-packages and scripts directories?
On 30/03/2011 12:09 PM, R. David Murray wrote: On Wed, 30 Mar 2011 11:11:45 +1100, Mark Hammond wrote: I'm wondering if it is a reasonable idea to have .hgignore exclude all files from 'Lib/site-packages' and 'Scripts'? As I install packages into my source builds, a 'hg status' lists *many* files in both those directories forcing me to scroll up a number of pages to see files which have actually changed. I hardly ever install things into my source build. The first time I've done that, in fact, was to run coverage. Windows doesn't really have an install process integrated into the build, so it is probably fairly common there. The solution is to add such directories and/or files to your personal ignore list See the 'ignore' entry under 'ui' in the hgrc documentation. Yeah - but I was wondering if it could be made more convenient by default given the downside seems quite small... Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] .hgignore including site-packages and scripts directories?
On 30/03/2011 1:37 PM, R. David Murray wrote: On Wed, 30 Mar 2011 12:17:05 +1100, Mark Hammond wrote: On 30/03/2011 12:09 PM, R. David Murray wrote: The solution is to add such directories and/or files to your personal ignore list See the 'ignore' entry under 'ui' in the hgrc documentation. Yeah - but I was wondering if it could be made more convenient by default given the downside seems quite small... I suppose I wouldn't care about site-packages. Nothing except the existing README should ever get checked in there, I think. And I don't seem to have a 'Scripts' directory, just Tools/scripts, which shouldn't be ignored. Is Scripts windows specific? (I also have a build/scripts, but build is ignored.) Yeah, "Scripts" is indeed Windows specific - which I admit I had forgotten until a couple of hours ago when debugging why a script using virtualenv failed on Windows due to assuming stuff went into a 'bin' directory and not the 'Scripts' directory. The directory is normally populated by the distutils 'install' command, easy_install, etc Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Non-code changes on "old" branches
Hi, There are a couple of changes I'd like to make and would like some guidance on policy: http://bugs.python.org/issue6498 is a documentation bug which exists in Python 2.6 and later. The patch in that bug touches the docs and a comment in one source file. Is it acceptable to push that change to the 2.6 branch, or should I start with 2.7? My request re .hgignore from yesterday didn't get any complaints, so I intend opening a bug, asking for review here and if I don't get objections in a day or so, pushing the change. This really should go all the way back to 2.5 even though that release has long been closed. Is it acceptable to push a change to .hgignore to the 2.5 branch? If not, where should I start with such a change? I ask these questions primarily as the dev guide tells me I should forward-port all changes - thus, I need to know the earliest versions I can use before I can even start the process... Thanks, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] sys.settrace: behavior doesn't match docs
On 2/05/2011 9:27 PM, Ned Batchelder wrote: ... Maybe the fact no one noticed the docs were wrong proves that no one ever tried returning None from a local trace function. Or if they did, they should have complained by now. IMO, if the behaviour regresses from how it is documented and how it previously worked and no reports of the regression exist, we should just fix it without regard to people relying on the "new" functionality... Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Updated version of PEP-0397 - Python launcher for Windows.
Hi all, I've updated PEP-0397 to try and address some of the comments from the last draft. I've checked the new version into hg, so you can find a full diff there, but the key items I've changed are: * Spelled out the "version qualifier" rules for the shebang lines. * Spelled out some customization options, both for fine-tuning the specific Python version selected and for supporting other Python implementations via "custom" commands. * Indicated the launcher is not supported at all on Win2k or earlier. * Removed some cruft. The new version is attached and I welcome all comments, including bike-shedding on the environment variable names and INI section/value names. Note that the reference implementation has not changed - I'll update that once there is general agreement on the functionality described in the PEP. Thanks, Mark PEP: 397 Title: Python launcher for Windows Version: $Revision$ Last-Modified: $Date$ Author: Mark Hammond Status: Draft Type: Standards Track Content-Type: text/plain Created: 15-Mar-2011 Post-History: 17-May-2011, 15-Mar-2011 Abstract This PEP describes a Python launcher for the Windows platform. A Python launcher is a single executable which uses a number of heuristics to locate a Python executable and launch it with a specified command line. Rationale Windows provides "file associations" so an executable can be associated with an extension, allowing for scripts to be executed directly in some contexts (eg., double-clicking the file in Windows Explorer.) Until now, a strategy of "last installed Python wins" has been used and while not ideal, has generally been workable due to the conservative changes in Python 2.x releases. As Python 3.x scripts are often syntactically incompatible with Python 2.x scripts, a different strategy must be used to allow files with a '.py' extension to use a different executable based on the Python version the script targets. This will be done by borrowing the existing practices of another operating system - scripts will be able to nominate the version of Python they need by way of a "shebang" line, as described below. Unix-like operating systems (referred to simply as "Unix" in this PEP) allow scripts to be executed as if they were executable images by examining the script for a "shebang" line which specifies the actual executable to be used to run the script. This is described in detail in the evecve(2) man page [1] and while user documentation will be created for this feature, for the purposes of this PEP that man page describes a valid shebang line. Additionally, these operating systems provide symbolic-links to Python executables in well-known directories. For example, many systems will have a link /usr/bin/python which references a particular version of Python installed under the operating-system. These symbolic links allow Python to be executed without regard for where Python it actually installed on the machine (eg., without requiring the path where Python is actually installed to be referenced in the shebang line or in the PATH.) PEP 394 'The "python" command on Unix-Like Systems' [2] describes additional conventions for more fine-grained specification of a particular Python version. These 2 facilities combined allow for a portable and somewhat predictable way of both starting Python interactively and for allowing Python scripts to execute. This PEP describes an implementation of a launcher which can offer the same benefits for Python on the Windows platform and therefore allows the launcher to be the executable associated with '.py' files to support multiple Python versions concurrently. While this PEP offers the ability to use a shebang line which should work on both Windows and Unix, this is not the primary motivation for this PEP - the primary motivation is to allow a specific version to be specified without inventing new syntax or conventions to describe it. An overview of the launcher. This PEP outlines the general functionality and key guidelines of a launcher for Windows. It is accompanied by a reference implementation [3], written in Python, which defines the detailed implementation. Over time, changes to the implementation may be desired - if the changes adhere to the guidelines in this PEP and have been made following the standard Python development model this PEP need not change. In other words, this PEP makes no attempt to describe in detail every feature offered by the launcher but instead to offer guidelines the launcher should adhere to. The actual implementation of the launcher will be written in C and will not link directly or i
Re: [Python-Dev] PEP 397 (Python launcher for Windows) reference implementation
On 30/06/2011 3:00 AM, Vinay Sajip wrote: PEP 397 (Python launcher for Windows) has a reference implementation in Python. Does anyone know of a C implementation, or is planning/working on one? I realise this is the final objective, so such implementation might be premature, but perhaps someone has been experimenting ... Not yet - my last update of the PEP has made the existing reference implementation out-of-date, so I want to work on that before starting on the C version. However, seeing as my note about the most recent PEP update attracted zero comments, I admit I lost any sense of urgency on actually doing this... I'll make an effort to update that reference impl in the next week or so (but obviously anyone else is free to help ;) Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
