Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[Nick Coghlan]

On Sat, Dec 8, 2012 at 4:46 PM, PJ Eby  wrote:

> So if package A includes a "Conflicts: B" declaration, I recommend the
> following:
>
> * An attempt to install A with B already present refuses to install A
> without a warning and confirmation
> * An attempt to install B informs the user of the conflict, and
> optionally offers to uninstall A
>
> In this way, any collateral damage to B is avoided, while still making
> the intended "lack of support" declaration clear.
>
> How does that sound?
>

No, that's not the way it works. A conflict is always symmetric, no matter
who declares it.

The beneficiary of these notifications is the aggregator attempting to
build a systematically coherent system, rather than one with latent
incompatibilities waiting to bite them at run time. It doesn't *matter* if
"A conflicts with B" or "B conflicts with A", you cannot have a system with
both of them installed that will be supported by the developers of both A
*and* B.

Now, this beneficiary *may* be the packagers for a Linux distribution, but
it may also be a larger Python distribution (ActiveState, EPD, etc), a web
application developer, a desktop application developer, a system integrator
for a large-scale distributed system, or anyone else that combines and
deploys an integrated set of packages (even those a developer installs on
their personal workstation).

It's up to the user to decide who they want to believe. Now, it may be
that, for a given use case, the end user doesn't actually care about the
potential conflict (e.g. they've done their own research and determined
that the conflicting behaviour doesn't affect their system) - that's then a
design decision in the installation tools as to whether or not they want to
make it easy for users to override the metadata. In the Linux distro case,
the installer *and* most of the metadata are largely provided by the same
people, so yum/rpm/etc generally *don't* make it easy to install
conflicting packages. Python installers are in a different situation
though, so forced installs are likely to be an expected feature (in fact, I
expect the more likely outcome given the status quo is that the default
behaviour will be a warning at installation time with an option to request
enforcement of "no conflicts").

Building integrated systems *is hard*. Pretending projects can't conflict
just because they're both written in Python isn't sensible, and neither is
it sensible to avoid warning users about the the potential for latent
defects when particular packages are used in combination.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Sat, Dec 8, 2012 at 5:06 AM, Nick Coghlan  wrote:
> On Sat, Dec 8, 2012 at 4:46 PM, PJ Eby  wrote:
>>
>> So if package A includes a "Conflicts: B" declaration, I recommend the
>> following:
>>
>> * An attempt to install A with B already present refuses to install A
>> without a warning and confirmation
>> * An attempt to install B informs the user of the conflict, and
>> optionally offers to uninstall A
>>
>> In this way, any collateral damage to B is avoided, while still making
>> the intended "lack of support" declaration clear.
>>
>> How does that sound?
>
>
> No, that's not the way it works. A conflict is always symmetric, no matter
> who declares it.

But that *precisely contradicts* what you said in your previous email:

> It's to allow a project to say
> *they don't support* installing in parallel with another package.

Just because A doesn't support being installed next to B, doesn't mean
B doesn't support being installed next to A.  B might work just fine
with A installed, and even be explicitly supported by the author of B.
 Why should the author of A get to decide what happens to B?  Just
because I trust A about A, doesn't mean I should have to trust them
about B.

Look, I really don't care about the individual fields' definitions
that much.  I care about only one thing: A shouldn't get to (de facto)
dictate what happens to B.  If you *really* want the behavior to be
symmetrical, then it should *only* be symmetrical if both A and B
*agree* they are in conflict.  (i.e., both refer to the other in their
conflict fields).  Otherwise, it should only be a warning.

There are tons of other things that I could argue here about the
positions you've laid out.  But all I *really* care about is that we
not define fields in such a way as to permit or encourage
inter-package warfare -- intentional or not.  Solutions acceptable to
me include (in no particular order):

* Make declarations affect only the declarer (as with Obsoleted-By)
* Make declarations only warn users, not block installation or result
in uninstallation
* Have no automated action at all, and document them as intended for
downstream repackagers only
* Toss the field entirely
* Make the field include a context (e.g. a distro name), so that only
tools explicitly told you're operating in that context pay attention
* Use the new metadata extension vocabularies to define hints for
specific downstream packaging tools and systems
* Replace "conflicts" with a specification of resources actually used
by the project, so that such conflicts can be automatically detected
without needing to target a specific project

And there are probably others I haven't thought of yet.  If you can be
clearer about what it is you want from the Conflicts field *other*
than just wanting it to stay as is (or perhaps *why* you would like to
have the Python infrastructure side with project A over project B,
irrespective of which project is A and which one is B), then perhaps I
can come up with others.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] python 3.3 module test failures on FreeBSD 9 amd64

[A G]

Hello All,


I am successfully compiling python 3.3 on FreeBSD 9.0 amd64 architecture.
When I run the tests, I get these two test failures ( I trimmed out all the
output from test cases that returned ok):

test_saltedcrypt (test.test_crypt.CryptTestCase) ... FAIL

==
FAIL: test_saltedcrypt (test.test_crypt.CryptTestCase)
--
Traceback (most recent call last):
  File "/usr/home/adam/Python-3.3.0/Lib/test/test_crypt.py", line 23, in
test_saltedcrypt
self.assertEqual(len(pw), method.total_size)
AssertionError: 60 != 13

--
Ran 4 tests in 0.063s


==
FAIL: test_setegid (test.test_os.PosixUidGidTests)
--
Traceback (most recent call last):
  File "/usr/home/adam/Python-3.3.0/Lib/test/test_os.py", line 1211, in
test_setegid
self.assertRaises(os.error, os.setegid, 0)
AssertionError: OSError not raised by setegid

==
FAIL: test_setgid (test.test_os.PosixUidGidTests)
--
Traceback (most recent call last):
  File "/usr/home/adam/Python-3.3.0/Lib/test/test_os.py", line 1199, in
test_setgid
self.assertRaises(os.error, os.setgid, 0)
AssertionError: OSError not raised by setgid

==
FAIL: test_setregid (test.test_os.PosixUidGidTests)
--
Traceback (most recent call last):
  File "/usr/home/adam/Python-3.3.0/Lib/test/test_os.py", line 1231, in
test_setregid
self.assertRaises(os.error, os.setregid, 0, 0)
AssertionError: OSError not raised by setregid

--
Ran 124 tests in 2.035s

FAILED (failures=3, skipped=23)
Warning -- threading._dangling was modified by test_os
test test_os failed
2 tests failed:
test_crypt test_os


My question is, are these known? Is it safe for me to use this compiled
version of python, or will I run into problems if I use those two modules
or any code that depends on those? Ideally I'd like to resolve these and
submit a port for python3.3 since the most recent FreeBSD port is stuck on
3.2.


Thanks!
Adam
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[MRAB]

On 2012-12-08 20:18, PJ Eby wrote:

On Sat, Dec 8, 2012 at 5:06 AM, Nick Coghlan  wrote:

On Sat, Dec 8, 2012 at 4:46 PM, PJ Eby  wrote:


So if package A includes a "Conflicts: B" declaration, I recommend the
following:

* An attempt to install A with B already present refuses to install A
without a warning and confirmation
* An attempt to install B informs the user of the conflict, and
optionally offers to uninstall A

In this way, any collateral damage to B is avoided, while still making
the intended "lack of support" declaration clear.

How does that sound?



No, that's not the way it works. A conflict is always symmetric, no matter
who declares it.


But that *precisely contradicts* what you said in your previous email:


It's to allow a project to say
*they don't support* installing in parallel with another package.


Just because A doesn't support being installed next to B, doesn't mean
B doesn't support being installed next to A.  B might work just fine
with A installed, and even be explicitly supported by the author of B.
  Why should the author of A get to decide what happens to B?  Just
because I trust A about A, doesn't mean I should have to trust them
about B.


[snip]
If package A says that it conflicts with package B, it may or may not
be symmetrical, because it's possible that package B has been updated
since the author of package A discovered the conflict, so it's
important that the user is told which package is complaining about the
conflict, the one that is being installed or the one that is already
installed.

It may also be helpful if the package that includes the "Conflicts"
declaration specifies which version of the other package it was last
tested against in case there is a more recent version of the other
package that does not cause the conflict, or, indeed, that there's a
more recent version of the package that includes the "Conflicts"
declaration that does not cause the conflict.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Emacs users: hg-tools-grep

[Barry Warsaw]

Hark fellow Emacsers.  All you unenlightened heathens can stop reading now.

A few years ago, my colleague Jono Lange wrote probably the best little chunk
of Emacs lisp ever.  `M-x bzr-tools-grep` lets you easily search a Bazaar
repository for a case-sensitive string, providing you with a nice *grep*
buffer which you can scroll through.  When you find a code sample you want to
look at, C-c C-c visits the file and plops you right at the matching line.
You *only* grep through files under version control, so you get to ignore
generated files, and compilation artifacts, etc.

Of course, this doesn't help you for working on the Python code base, because
Mercurial.  I finally whipped up this straight up rip of Jono's code to work
with hg.  I'm actually embarrassed to put a copyright on this thing, and would
happily just donate it to Jono, drop it in Python's Misc directory, or slip it
like a lump of coal into the xmas stocking of whoever wants to "maintain" it
for the next 20 years.

But anyway, it's already proven enormously helpful to me, so here it is.

Cheers,
-Barry

P.S. Who wants to abuse Jono and Matthew's copyright again and provide a
 git version?

;; Copyright (c) 2012 Barry A. Warsaw
;;
;; Permission is hereby granted, free of charge, to any person obtaining
;; a copy of this software and associated documentation files (the
;; "Software"), to deal in the Software without restriction, including
;; without limitation the rights to use, copy, modify, merge, publish,
;; distribute, sublicense, and/or sell copies of the Software, and to
;; permit persons to whom the Software is furnished to do so, subject to
;; the following conditions:
;;
;; The above copyright notice and this permission notice shall be
;; included in all copies or substantial portions of the Software.
;;
;; THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
;; EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
;; MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
;; NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
;; LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
;; OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
;; WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

;; This code is based on bzr-tools.el
;; Copyright (c) 2008-2009 Jonathan Lange, Matthew Lefkowitz

(provide 'hg-tools)

(defconst hg-tools-grep-command
  "hg locate --print0 | xargs -0 grep -In %s"
  "The command used for grepping files using hg. See `hg-tools-grep'.")

;; Run 'code' at the root of the branch which dirname is in.
(defmacro hg-tools-at-branch-root (dirname &rest code)
  `(let ((default-directory (locate-dominating-file (expand-file-name ,dirname) 
".hg"))) ,@code))


(defun hg-tools-grep (expression dirname)
  "Search a branch for `expression'. If there's a C-u prefix, prompt for 
`dirname'."
  (interactive
   (let* ((string (read-string "Search for: "))
  (dir (if (null current-prefix-arg)
   default-directory
 (read-directory-name (format "Search for %s in: " string)
 (list string dir)))
  (hg-tools-at-branch-root dirname
   (grep-find (format hg-tools-grep-command (shell-quote-argument 
expression)


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 12/08/2012 05:06 AM, Nick Coghlan wrote:

> Building integrated systems *is hard*. Pretending projects can't
> conflict just because they're both written in Python isn't sensible,
> and neither is it sensible to avoid warning users about the the
> potential for latent defects when particular packages are used in
> combination.

Building such systems is *too hard* to deletgate to the maintainers of
every Python distribution registered on the Cheeseshop:  there is too
much policy involved for the ha'penn'orth of mechanism we are discussing
here (decentralized inter-project metadata) to support.

Such metadata *cannot* be useful in the general sense, but only in the
context of a "curated" collection of packages, where the *curator* (not
the upstream package authors) makes the choices.



Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlDDwioACgkQ+gerLs4ltQ4rOACghpN5x+k0w0Umn20AG1WOvYkq
KQsAnibXQtbTnmbrPaMaVEfLH7W496lk
=WAh9
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3.3 module test failures on FreeBSD 9 amd64

[Stefan Krah]

A G  wrote:
> ==
> FAIL: test_saltedcrypt (test.test_crypt.CryptTestCase)
> --
> Traceback (most recent call last):
>   File "/usr/home/adam/Python-3.3.0/Lib/test/test_crypt.py", line 23, in
> test_saltedcrypt
>     self.assertEqual(len(pw), method.total_size)
> AssertionError: 60 != 13

This isn't known. Probably just a test assumption that is too strict. You
can open an issue at http://bugs.python.org/ .


> ==
> FAIL: test_setegid (test.test_os.PosixUidGidTests)
> --
> Traceback (most recent call last):
>   File "/usr/home/adam/Python-3.3.0/Lib/test/test_os.py", line 1211, in
> test_setegid
>     self.assertRaises(os.error, os.setegid, 0)
> AssertionError: OSError not raised by setegid


This is harmless and occurs if the test user is in the wheel group, see:

http://bugs.python.org/issue14110


Stefan Krah


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Conflicts [was Re: Keyword meanings [was: Accept just PEP-0426]]

[Steven D'Aprano]

On 09/12/12 08:14, MRAB wrote:


If package A says that it conflicts with package B, it may or may not
be symmetrical, because it's possible that package B has been updated
since the author of package A discovered the conflict, so it's
important that the user is told which package is complaining about the
conflict, the one that is being installed or the one that is already
installed.


I must admit than in reading this thread, I'm having a bit of trouble
understanding why merely *installing* packages should lead to conflicts.

Assuming that two software packages Spam and Ham install into directories
Spam and Ham, how can merely having them installed side-by-side lead to a
conflict?

I can see how running or importing Spam and Ham together might lead to
problems. And I can see that if package Spam wants to install into
directory Ham, that would be bad. But who does that?

Have I just demonstrated my naivety when it comes to packaging?

Under what circumstances would two well-behaved packages with different
names conflict?



--
Steven
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Conflicts [was Re: Keyword meanings [was: Accept just PEP-0426]]

[Chris Angelico]

On Sun, Dec 9, 2012 at 12:15 PM, Steven D'Aprano  wrote:
> Assuming that two software packages Spam and Ham install into directories
> Spam and Ham, how can merely having them installed side-by-side lead to a
> conflict?
>
> I can see how running or importing Spam and Ham together might lead to
> problems. And I can see that if package Spam wants to install into
> directory Ham, that would be bad. But who does that?

If two packages Spam and Ham both define a module Jam, then the one
that gets loaded will depend on the search path. That would be one
form of conflict.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Conflicts [was Re: Keyword meanings [was: Accept just PEP-0426]]

[Steven D'Aprano]

On 09/12/12 12:32, Chris Angelico wrote:

On Sun, Dec 9, 2012 at 12:15 PM, Steven D'Aprano  wrote:

Assuming that two software packages Spam and Ham install into directories
Spam and Ham, how can merely having them installed side-by-side lead to a
conflict?

I can see how running or importing Spam and Ham together might lead to
problems. And I can see that if package Spam wants to install into
directory Ham, that would be bad. But who does that?


If two packages Spam and Ham both define a module Jam, then the one
that gets loaded will depend on the search path. That would be one
form of conflict.


import Spam.Jam

import Ham.Jam

What am I missing?


Why would a software package called "Spam" install a top-level module called
"Jam" rather than "Spam"? Isn't the whole point of Python packages to solve
this namespace problem?



--
Steven
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Conflicts [was Re: Keyword meanings [was: Accept just PEP-0426]]

[Donald Stufft]

On Saturday, December 8, 2012 at 9:11 PM, Steven D'Aprano wrote:
> Why would a software package called "Spam" install a top-level module called
> "Jam" rather than "Spam"? Isn't the whole point of Python packages to solve
> this namespace problem?
> 
Conflicts doesn't really solve file based conflicts as PJ Elby has pointed out 
tools
need to detect that circumstance already. But to answer this question no, there
is no required mapping between Project names (what your thing is called on PyPI)
and python package names (what you import).

Something named Spam on PyPI could provide multiple python packages, named
whatever it was they wanted to be named. 

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Conflicts [was Re: Keyword meanings [was: Accept just PEP-0426]]

[Chris Angelico]

On Sun, Dec 9, 2012 at 1:11 PM, Steven D'Aprano  wrote:
> On 09/12/12 12:32, Chris Angelico wrote:
>>
>> On Sun, Dec 9, 2012 at 12:15 PM, Steven D'Aprano
>> wrote:
>>>
>>> Assuming that two software packages Spam and Ham install into directories
>>> Spam and Ham, how can merely having them installed side-by-side lead to a
>>> conflict?
>>>
>>> I can see how running or importing Spam and Ham together might lead to
>>> problems. And I can see that if package Spam wants to install into
>>> directory Ham, that would be bad. But who does that?
>>
>>
>> If two packages Spam and Ham both define a module Jam, then the one
>> that gets loaded will depend on the search path. That would be one
>> form of conflict.
>
>
> import Spam.Jam
>
> import Ham.Jam
>
> What am I missing?
>
>
> Why would a software package called "Spam" install a top-level module called
> "Jam" rather than "Spam"? Isn't the whole point of Python packages to solve
> this namespace problem?

That would require/demand that the software package MUST define a
module with its own name, and MUST NOT define any other top-level
modules, and also that package names MUST be unique. (RFC 2119
keywords.) That would work, as long as those restrictions are
acceptable.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Conflicts [was Re: Keyword meanings [was: Accept just PEP-0426]]

[MRAB]

On 2012-12-09 01:15, Steven D'Aprano wrote:

On 09/12/12 08:14, MRAB wrote:


If package A says that it conflicts with package B, it may or may not
be symmetrical, because it's possible that package B has been updated
since the author of package A discovered the conflict, so it's
important that the user is told which package is complaining about the
conflict, the one that is being installed or the one that is already
installed.


I must admit than in reading this thread, I'm having a bit of trouble
understanding why merely *installing* packages should lead to conflicts.


[snip]
Personally speaking, I was thinking more about possible problems at
runtime due to functional conflicts, but it could apply to any
(undefined) conflict.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3.3 module test failures on FreeBSD 9 amd64

[Trent Nelson]

On Sat, Dec 08, 2012 at 01:07:43PM -0800, A G wrote:
> Ideally I'd like to resolve these and submit a port for python3.3 since
> the most recent FreeBSD port is stuck on 3.2.

FWIW, the FreeBSD Python port maintainer, Kubilay Kocak, is active
on #python-dev@freenode under the nick 'koobs'.  He has been working
on the 3.3 port.  I'd recommend liaising with him in order to avoid
duplicating any effort.

Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3145 (With Contents)

[Glyph]

On Dec 7, 2012, at 5:10 PM, anatoly techtonik  wrote:

> What about reading from other file descriptors?  subprocess.Popen allows 
> arbitrary file descriptors to be used.  Is there any provision here for 
> reading and writing non-blocking from or to those?
> 
> On Windows it is WriteFile/ReadFile and PeekNamedPipe. On Linux it is select. 
> Of course a test is needed, but why it should not just work?


This is exactly why the provision needs to be made explicitly.

On Windows it is WriteFile and ReadFile and PeekNamedPipe - unless the handle 
is a socket in which case it needs to be WSARecv.  Or maybe it's some other 
weird thing - like, maybe a mailslot - and you need to call a different API.

On *nix it really shouldn't be select.  select cannot wait upon a file 
descriptor whose value is greater than FD_SETSIZE, which means it sets a hard 
(and small) limit on the number of things that a process which wants to use 
this facility can be doing.

On the other hand, if you hard-code another arbitrary limit like this into the 
stdlib subprocess module, it will just be another great reason why Twisted's 
spawnProcess is the best and everyone should use it instead, so be my guest ;-).

-glyph___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3145 (With Contents)

[Gregory P. Smith]

I'm really not sure what this PEP is trying to get at given that it
contains no examples and sounds from the descriptions to be adding a
complicated api on top of something that already, IMNSHO, has too much it
(subprocess.Popen).

Regardless, any user can use the stdout/err/in file objects with their own
code that handles them asynchronously (yes that can be painful but that is
what is required for _any_ socket or pipe I/O you don't want to block on).

It *sounds* to me like this entire PEP could be written and released as a
third party module on PyPI that offers a subprocess.Popen subclass adding
some more convenient non-blocking APIs.  That's where I'd start if I were
interested in this as a future feature.

-gps



On Fri, Dec 7, 2012 at 5:10 PM, anatoly techtonik wrote:

> On Tue, Sep 15, 2009 at 9:24 PM,  wrote:
>
>> On 04:25 pm, eric.pru...@gmail.com wrote:
>>
>>> I'm bumping this PEP again in hopes of getting some feedback.
>>>
>>
> This is useful, indeed. ActiveState recipe for this has 10 votes, which is
> high for ActiveState (and such hardcore topic FWIW).
>
>
>> On Tue, Sep 8, 2009 at 23:52, Eric Pruitt  wrote:
>>>
 PEP: 3145
 Title: Asynchronous I/O For subprocess.Popen
 Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson
 Type: Standards Track
 Content-Type: text/plain
 Created: 04-Aug-2009
 Python-Version: 3.2

 Abstract:

In its present form, the subprocess.Popen implementation is prone to
dead-locking and blocking of the parent Python script while waiting
 on data
from the child process.

 Motivation:

A search for "python asynchronous subprocess" will turn up numerous
accounts of people wanting to execute a child process and
 communicate with
it from time to time reading only the data that is available instead
 of
blocking to wait for the program to produce data [1] [2] [3].  The
 current
behavior of the subprocess module is that when a user sends or
 receives
data via the stdin, stderr and stdout file objects, dead locks are
 common
and documented [4] [5].  While communicate can be used to alleviate
 some of
the buffering issues, it will still cause the parent process to
 block while
attempting to read data when none is available to be read from the
 child
process.

 Rationale:

There is a documented need for asynchronous, non-blocking
 functionality in
subprocess.Popen [6] [7] [2] [3].  Inclusion of the code would
 improve the
utility of the Python standard library that can be used on Unix
 based and
Windows builds of Python.  Practically every I/O object in Python
 has a
file-like wrapper of some sort.  Sockets already act as such and for
strings there is StringIO.  Popen can be made to act like a file by
 simply
using the methods attached the the subprocess.Popen.stderr, stdout
 and
stdin file-like objects.  But when using the read and write methods
 of
those options, you do not have the benefit of asynchronous I/O.  In
 the
proposed solution the wrapper wraps the asynchronous methods to
 mimic a
file object.

 Reference Implementation:

I have been maintaining a Google Code repository that contains all
 of my
changes including tests and documentation [9] as well as blog
 detailing
the problems I have come across in the development process [10].

I have been working on implementing non-blocking asynchronous I/O in
 the
subprocess.Popen module as well as a wrapper class for
 subprocess.Popen
that makes it so that an executed process can take the place of a
 file by
duplicating all of the methods and attributes that file objects have.

>>>
>> "Non-blocking" and "asynchronous" are actually two different things. From
>> the rest of this PEP, I think only a non-blocking API is being introduced.
>>  I haven't looked beyond the PEP, though, so I might be missing something.
>
>
> I suggest renaming http://www.python.org/dev/peps/pep-3145/ to
> 'Non-blocking I/O for subprocess' and continue. IMHO on this stage is where
> examples with deadlocks that occur with current subprocess
> implementation are badly needed.
>
> There are two base functions that have been added to the
 subprocess.Popen
class: Popen.send and Popen._recv, each with two separate
 implementations,
one for Windows and one for Unix based systems.  The Windows
implementation uses ctypes to access the functions needed to control
 pipes
in the kernel 32 DLL in an asynchronous manner.  On Unix based
 systems,
the Python interface for file control serves the same purpose.  The
different implementations of Popen.send and Popen._recv have

Re: [Python-Dev] PEP 3145 (With Contents)

[Gregory P. Smith]

On Sat, Dec 8, 2012 at 8:14 PM, Glyph  wrote:

>
> On Dec 7, 2012, at 5:10 PM, anatoly techtonik  wrote:
>
> What about reading from other file descriptors?  subprocess.Popen allows
>> arbitrary file descriptors to be used.  Is there any provision here for
>> reading and writing non-blocking from or to those?
>
>
> On Windows it is WriteFile/ReadFile and PeekNamedPipe. On Linux it is
> select. Of course a test is needed, but why it should not just work?
>
>
> This is exactly why the provision needs to be made explicitly.
>
> On Windows it is WriteFile and ReadFile and PeekNamedPipe - unless the
> handle is a socket in which case it needs to be WSARecv.  Or maybe it's
> some other weird thing - like, maybe a mailslot - and you need to call a
> different API.
>
> On *nix it really shouldn't be select.  select cannot wait upon a file
> descriptor whose *value* is greater than FD_SETSIZE, which means it sets
> a hard (and small) limit on the number of things that a process which wants
> to use this facility can be doing.
>

Nobody should ever touch select() this decade.  poll() exists everywhere
that matters.


> On the other hand, if you hard-code another arbitrary limit like this into
> the stdlib subprocess module, it will just be another great reason why
> Twisted's spawnProcess is the best and everyone should use it instead, so
> be my guest ;-).
>
>
Is twisted's spawnProcess thread safe and async signal safe by using
restricted C code for everything between the fork() and exec()?  I'm not
familiar enough with the twisted codebase to find things easily in it but
I'm not seeing such an extension module within twisted and the code in
http://twistedmatrix.com/trac/browser/trunk/twisted/internet/process.pycertainly
is not safe.  Just sayin'. :)

Python >= 3.2 along with the
http://pypi.python.org/pypi/subprocess32/backport for use on 2.x get
this right.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3145 (With Contents)

[Glyph]

On Dec 8, 2012, at 8:37 PM, Gregory P. Smith  wrote:

> Is twisted's spawnProcess thread safe and async signal safe by using 
> restricted C code for everything between the fork() and exec()?  I'm not 
> familiar enough with the twisted codebase to find things easily in it but I'm 
> not seeing such an extension module within twisted and the code in 
> http://twistedmatrix.com/trac/browser/trunk/twisted/internet/process.py 
> certainly is not safe.  Just sayin'. :)


It's on the agenda: .

-glyph___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[Nick Coghlan]

On Sun, Dec 9, 2012 at 8:41 AM, Tres Seaver  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 12/08/2012 05:06 AM, Nick Coghlan wrote:
>
> > Building integrated systems *is hard*. Pretending projects can't
> > conflict just because they're both written in Python isn't sensible,
> > and neither is it sensible to avoid warning users about the the
> > potential for latent defects when particular packages are used in
> > combination.
>
> Building such systems is *too hard* to deletgate to the maintainers of
> every Python distribution registered on the Cheeseshop:  there is too
> much policy involved for the ha'penn'orth of mechanism we are discussing
> here (decentralized inter-project metadata) to support.
>
> Such metadata *cannot* be useful in the general sense, but only in the
> context of a "curated" collection of packages, where the *curator* (not
> the upstream package authors) makes the choices.
>

The authors of major projects are often in a good position to know when
they conflict with other high profile projects and thus can't be used
reliably in the same system. Now, *most* of the time, if there's a genuine
conflict between two Python packages, it's going to be at install time -
two projects attempting to install the same file obviously can't coexist on
a single system (distribute and setuptools, for example, conflict at this
level - they both want to own the "setuptools" and "easy_install" names).
However, Python has plenty of other global state too (the codec registry,
the import system, monkeypatching), and there is potential for conflict
over underlying OS level resources.

So let's look at the case of collections of Python packages that *are*
curated. Maybe I'm a Linux distro packager, looking to automate the
conversion to distro packages. Maybe I'm a toolsmith for a large
corporation trying to build a curated set of packages for internal use
(clearly indicating to my internal users which ones don't play nicely with
each other and thus shouldn't be used together in the same project).

Regardless of the reason, I'm the curator for a collection of Python
packages. How shall I express the conflicts I have identified? Shall I go
invent my own metadata system? Shall I be forced to choose a particular
platform-specific dependency management system? How shall upstream authors
communicate to *me* the conflicts that they're already aware of?

Or, hey, there's this nice shiny cross-platform dependency management
system *right here*. Maybe they'll be nice enough to consider handling *my*
use case as well, even if it's a use case *they* don't care about.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Conflicts [was Re: Keyword meanings [was: Accept just PEP-0426]]

[PJ Eby]

On Sat, Dec 8, 2012 at 10:22 PM, MRAB  wrote:
> On 2012-12-09 01:15, Steven D'Aprano wrote:
>>
>> On 09/12/12 08:14, MRAB wrote:
>>
>>> If package A says that it conflicts with package B, it may or may not
>>> be symmetrical, because it's possible that package B has been updated
>>> since the author of package A discovered the conflict, so it's
>>> important that the user is told which package is complaining about the
>>> conflict, the one that is being installed or the one that is already
>>> installed.
>>
>>
>> I must admit than in reading this thread, I'm having a bit of trouble
>> understanding why merely *installing* packages should lead to conflicts.
>>
> [snip]
> Personally speaking, I was thinking more about possible problems at
> runtime due to functional conflicts, but it could apply to any
> (undefined) conflict.

If it's for a runtime functional conflict, there's no need for
installation tools to worry about it, except perhaps in the case where
a single project C depends on *both* A and B, where A and B conflict
with each other.  Apart from that piece of information, there is no
way to know that the code will ever even be imported at the same time.
 (And even then, it's just a hint of the possibility, not a
guarantee.)

Nick, OTOH, says that the purpose of the field is to declare that mere
side-by-side installation invalidates developer support for the
configuration.

However, the widespread confusion (conflicts?) over what exactly the
field is supposed to mean and when it should be used suggests that its
charter is not nearly as clear as it should be.

It seems perhaps it is suffering from the so-called "Illusion of
Transparency", wherein everybody looks at it and thinks that it
*obviously* means X, and only a fool could think otherwise...  except
that everyone has a *different* value of X in mind.

That's why I keep asking for specific, concrete use cases.  At this
point, for the field to make any sense, there needs to be some better
idea of what a "runtime" or "undefined" conflict is.  Apart from file
conflicts, has anybody identified a single PyPI package that would
make use of this field?  If so, what *is* that example, and what is
the nature of the conflict?

Do any of the distro folks know of a Python project tagged as
conflicting with another for their distro, where the conflict does
*not* involve any files in conflict?

(And the conflict is not specific to the distro's packaging of that
project and the project in conflict?  i.e., that it would have
actually been possible and/or meaningful for the upstream developer to
have flagged the conflict in the project's metadata, given the
proposed metadata standard?)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[Nick Coghlan]

On Sun, Dec 9, 2012 at 6:18 AM, PJ Eby  wrote:

> On Sat, Dec 8, 2012 at 5:06 AM, Nick Coghlan  wrote:
> > On Sat, Dec 8, 2012 at 4:46 PM, PJ Eby  wrote:
> >>
> >> So if package A includes a "Conflicts: B" declaration, I recommend the
> >> following:
> >>
> >> * An attempt to install A with B already present refuses to install A
> >> without a warning and confirmation
> >> * An attempt to install B informs the user of the conflict, and
> >> optionally offers to uninstall A
> >>
> >> In this way, any collateral damage to B is avoided, while still making
> >> the intended "lack of support" declaration clear.
> >>
> >> How does that sound?
> >
> >
> > No, that's not the way it works. A conflict is always symmetric, no
> matter
> > who declares it.
>
> But that *precisely contradicts* what you said in your previous email:
>
> > It's to allow a project to say
> > *they don't support* installing in parallel with another package.
>
> Just because A doesn't support being installed next to B, doesn't mean
> B doesn't support being installed next to A.  B might work just fine
> with A installed, and even be explicitly supported by the author of B.
>  Why should the author of A get to decide what happens to B?  Just
> because I trust A about A, doesn't mean I should have to trust them
> about B.
>

If I'm installing both A *and* B, I want to know if *either* project
doesn't support that configuration. The order in which they get installed
should *not* have any impact on my finding out that I am using one of my
dependencies in an unsupported way that may cause me unanticipated problems
further down the line.

The author of A *doesn't* get to decide what happens to B, *I* do. They're
merely providing a heads up that they believe there are problems when using
their project in conjunction with B. My options will be:
- use them both anyway (e.g. perhaps after doing some research, I may find
out the conflict relates solely to a feature of B that I'm not using, so I
simply update my project documentation to say "do not use feature X from
project B, as it conflicts with dependency A")
- choose to continue using A, find another solution for B
- choose to continue using B, find another solution for A

As a concrete example, there are projects out there that are known not to
work with gevent's socket monkeypatching, but people don't know that until
they try it and it blows up in their face.

I now agree that *enforcing* a conflicts field at install time in a Python
installer doesn't make any sense, since the nature of Python means it will
often be easy to sidestep any such issues once you're aware of their
existence (e.g. by avoiding gevent's monkeypatching features and using
threads to interact with the uncooperative synchronous library, or by
splitting your application into multiple processes, some using gevent and
others synchronous sockets). I also believe that *any* Conflicts
declaration *should* be backed up with an explicit explanation and
rationale for that conflict declaration in the project documentation.

Making it impossible to document runtime conflicts in metadata doesn't make
those conflicts go away - it just means they will continue to be documented
in an ad hoc manner on project web sites (if they get documented at all),
making the job of package curation unnecessarily more difficult (since
there is no standard way to document runtime conflicts). Adding a metadata
field doesn't make sure such known conflicts *will* be documented, but it
least makes it possible.

So, I still like the idea of including a Conflicts field, but think a few
points should be made clear:
- the Conflicts field would be for documenting other distributions which
have known issues working together in the same process and thus constitute
an unsupported configuration
- this field would be aimed at package *users*, rather than at installation
tools (although it would still be good if they installation tools supported
scanning a set of packages for known conflicts)
- any use of this field should be backed up with a more detailed
explanation in the project documentation

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Conflicts [was Re: Keyword meanings [was: Accept just PEP-0426]]

[Nick Coghlan]

On Sun, Dec 9, 2012 at 3:48 PM, PJ Eby  wrote:

> That's why I keep asking for specific, concrete use cases.  At this
> point, for the field to make any sense, there needs to be some better
> idea of what a "runtime" or "undefined" conflict is.  Apart from file
> conflicts, has anybody identified a single PyPI package that would
> make use of this field?  If so, what *is* that example, and what is
> the nature of the conflict?
>

The best current example I know of is whether or not a given package is
gevent compatible. At the moment, you have to try it and see, or hope the
project developers have a note somewhere saying whether or not it works.
"Incompatible" might be a better field name than "Conflicts" for that use
case, though.

You've persuaded me that any installer based notification of runtime
conflicts should at most be a warning (or even a separate query), since the
user has so many options for dealing with it (including the typical case
where the two components are simply never used in the same process).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com