[Python-Dev] Use dlopen() on Darwin/OS X to load extensions?
Hi folks, I submitted a patch a little while ago to led Python on Darwin/OS X use the same code path to load extensions it uses on most other Unix- like platforms. (The reasons for this are several, and mentioned in the patch: http://sourceforge.net/tracker/index.php? func=detail&aid=1454844&group_id=5470&atid=305470 ). Anyhow, IMO if this patch is to be included at all (I rather think it should, and will happily discuss that on this list or on the patch comments to clarify why this is so), it probably ought to make it into python 2.5 earlier rather than later. While I'm almost certain that these changes will cause no issues (as Apple officially encourages use of dlopen() as a drop-in replacement for the officially 'discouraged' NeXT-derived functions that Python now uses to load extensions), it would seem more prudent to give a low-level change like this plenty of time to settle out in case I am wrong. I've run this by the python-mac folks, and there seemed to be some assent, or at least no complaint. Bob Ippolito appeared to think that this approach was the best to making Python on the Mac load extensions properly in some corner cases (see the patch description for more details), but he hasn't weighted in for a while. Sorry if it's bad form to ask about patches one has submitted -- let me know if that sort of discussion should be kept strictly on the patch tracker. Zach Pincus Program in Biomedical Informatics and Department of Biochemistry Stanford University School of Medicine ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use dlopen() on Darwin/OS X to load extensions?
> Thanks for reminding us about this issue. > Unfortunately, without an explicit ok from one of the Mac maintainers, > I don't want to add this myself. If you can get Bob, Ronald, or Jack > to say ok, I will apply the patch ASAP. I have a Mac OS X.4 box and > can test it, but don't know the suitability of the patch. Fair enough -- this seems reasonable. Now, there is one issue with this all that some general feedback from Python-Dev would be helpful with: how best to test such a patch? Specifically, this patch would change a core python code path. Now, I can see no reason why it would break anything -- but we know how flimsy such arguments are. More strong evidence is that python builds and tests flawlessly with this patch. Given that many of the tests involve loading C extension libs, that's a good sign. Moreover, I've been using patched versions of 2.4 and 2.5 for some time, and loading fairly extensive libs (numpy/scipy, as well as the more exotic extensions that drove me to uncover this problem before), all without issue. But it would be good to have a specific benchmark to know nothing will break. I personally sort of feel that if dlopen() works once or twice, it will probably always work, but there are those who probably understand better the failure modes of opening shared libs as python extensions, and could suggest some good things to test. Any thoughts? Zach ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use dlopen() on Darwin/OS X to load extensions?
On Apr 4, 2006, at 1:29 AM, Martin v. Löwis wrote:
> Zachary Pincus wrote:
>> Specifically, this patch would change a core python code path.
>
> Why do you think so? I believe Python always passes absolute paths
> to dlopen, so any path resolution dlopen might do should be
> irrelevant.
> *If* you can get dlopen to look at directories outside sys.path,
> that would be a serious problem. You can use ktrace to find out
> what places it looks at.
Sorry, I meant path in a more metaphoric sense; e.g. on OS X /
Darwin, Python used to go through the code in dynload_next.c every
time it loaded an extension, now under the proposed patch it goes
through dynload_shlib.c.
>
>> But it would be good to have a specific benchmark to know nothing
>> will break. I personally sort of feel that if dlopen() works once or
>> twice, it will probably always work, but there are those who probably
>> understand better the failure modes of opening shared libs as python
>> extensions, and could suggest some good things to test.
>
> Running the test suite should already exercise this code a lot.
> You should run the test suite both in "working copy" mode, and
> in "make install" mode; if you know how to produce a Mac installer,
> testing it in such an installer ("framework mode"?) could also
> be done.
I'll do this, thanks.
Zach
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use dlopen() on Darwin/OS X to load extensions?
Hello folks, I just ran all the test iterations Martin suggested on Py2.5a1. That is, I did a normal build and ran 'make test', then installed and ran 'import test.regrtest; test.regrtest.main()', and then I did the whole thing over again with a framework build and install. All four test runs worked out fine. So at least on 10.4.5, this patch seems to be as tested as it's going to be. I've had some reports that it works fine on 10.3, and the patch doesn't change anything on 10.2. So given Bob's previous OK, this is probably ready to be applied -- unless anyone else has further concerns? Zach PS. I should mention as an aside that test_startfile.py is reported as 'failing unexpectedly on darwin', but since startfile is a windows thing, it really should be added to the expected tests in Lib/test/ regrtest.py. My patch didn't mess this up, though -- the startfile test is absent from the 'exclude' list in the SVN repository. On Apr 4, 2006, at 11:57 AM, Bob Ippolito wrote: >> No, it's fine. Thanks for reminding us about this issue. >> Unfortunately, without an explicit ok from one of the Mac >> maintainers, >> I don't want to add this myself. If you can get Bob, Ronald, or Jack >> to say ok, I will apply the patch ASAP. I have a Mac OS X.4 box and >> can test it, but don't know the suitability of the patch. > > The patch has my OK (I gave it a while ago on pythonmac-sig). > > -bob > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3102: Keyword-only arguments
Some thoughts from a lurker, largely concerning syntax; discount as
you wish.
First:
> Keyword-only arguments are not required to have a default value.
> Since Python requires that all arguments be bound to a value,
> and since the only way to bind a value to a keyword-only argument
> is via keyword, such arguments are therefore 'required keyword'
> arguments. Such arguments must be supplied by the caller, and
> they must be supplied via keyword.
So what would this look like?
def foo(*args, kw1, kw2):
That seems a bit odd, as my natural expectation wouldn't be to see
kw1 ands kw2 as required, no-default keyword args, but as misplaced
positional args.
Perhaps this might be a little better?
def foo(*args, kw1=, kw2=):
I'm rather not sure. At least it makes it clear that kw1 and kw2 are
keyword arguments, and that they have no default values.
Though, I'm kind of neutral on the whole bit -- in my mind "keyword
args" and "default-value args" are pretty conflated and I can't think
of any compelling reasons why they shouldn't be. It's visually handy
when looking at some code to see keywords and be able to say "ok,
those are the optional args for changing the handling of the main
args". I'm not sure where the big win with required keyword args is.
For that matter, why not "default positional args"? I think everyone
will agree that seems a bit odd, but it isn't too much odder than
"required keyword args". (Not that I'm for the former! I'm just
pointing out that if the latter is OK, there's no huge reason why the
former wouldn't be, and that is in my mind a flaw.)
Second:
> def compare(a, b, *, key=None):
This syntax seems a bit odd to me, as well. I always "understood" the
*args syntax by analogy to globbing -- the asterisk means to "take
all the rest", in some sense in both a shell glob and *args.
In this syntax, the asterisk, though given a position in the comma-
separated list, doesn't mean "take the rest and put it in this
position." It means "stop taking things before this position", which
is a bit odd, in terms of items *in* an argument list.
I grant that it makes sense as a derivation from "*ignore"-type
solutions, but as a standalone syntax it feels off. How about
something like:
def compare(a, b; key=None):
The semicolon is sort of like a comma but more forceful; it ends a
phrase. This seems like a logical (and easily comprehended by
analogy) use here -- ending the "positional arguments" phrase.
Zach Pincus
Program in Biomedical Informatics and Department of Biochemistry
Stanford University School of Medicine
On Apr 29, 2006, at 11:27 AM, Talin wrote:
> PEP: 3102
> Title: Keyword-Only Arguments
> Version: $Revision$
> Last-Modified: $Date$
> Author: Talin
> Status: Draft
> Type: Standards
> Content-Type: text/plain
> Created: 22-Apr-2006
> Python-Version: 3.0
> Post-History:
>
>
> Abstract
>
> This PEP proposes a change to the way that function arguments are
> assigned to named parameter slots. In particular, it enables the
> declaration of "keyword-only" arguments: arguments that can only
> be supplied by keyword and which will never be automatically
> filled in by a positional argument.
>
>
> Rationale
>
> The current Python function-calling paradigm allows arguments to
> be specified either by position or by keyword. An argument
> can be
> filled in either explicitly by name, or implicitly by position.
>
> There are often cases where it is desirable for a function to
> take
> a variable number of arguments. The Python language supports
> this
> using the 'varargs' syntax ('*name'), which specifies that any
> 'left over' arguments be passed into the varargs parameter as a
> tuple.
>
> One limitation on this is that currently, all of the regular
> argument slots must be filled before the vararg slot can be.
>
> This is not always desirable. One can easily envision a function
> which takes a variable number of arguments, but also takes one
> or more 'options' in the form of keyword arguments. Currently,
> the only way to do this is to define both a varargs argument,
> and a 'keywords' argument (**kwargs), and then manually extract
> the desired keywords from the dictionary.
>
>
> Specification
>
> Syntactically, the proposed changes are fairly simple. The first
> change is to allow regular arguments to appear after a varargs
> argument:
>
> def sortwords(*wordlist, case_sensitive=False):
> ...
>
> This function accepts any number of positional arguments, and it
> also accepts a keyword option called 'case_sensitive'. This
> option will never be filled in by a positional argument, but
> must be explicitly specified by name.
>
> Keyword-only arguments are not required to have a default value.
> Since Python requires that all arg
Re: [Python-Dev] PEP 3101: Advanced String Formatting
I'm not sure about introducing a special syntax for accessing
dictionary entries, array elements and/or object attributes *within a
string formatter*... much less an overloaded one that differs from
how these elements are accessed in "regular python".
> Compound names are a sequence of simple names seperated by
> periods:
>
> "My name is {0.name} :-\{\}".format(dict(name='Fred'))
>
> Compound names can be used to access specific dictionary entries,
> array elements, or object attributes. In the above example, the
> '{0.name}' field refers to the dictionary entry 'name' within
> positional argument 0.
Barring ambiguity about whether .name would mean the "name" attribute
or the "name" dictionary entry if both were defined, I'm not sure I
really see the point. How is:
d = {last:'foo', first:'bar'}
"My last name is {0.last}, my first name is {0.first}.".format(d)
really that big a win over:
d = {last:'foo', first:'bar'}
"My last name is {0}, my first name is {1}.".format(d['last'], d
['first'])
Plus, the in-string syntax is limited -- e.g. what if I want to call
a function on an attribute? Unless you want to re-implement all
python syntax within the formatters, someone will always be able to
level these sort of complaints. Better, IMO, to provide none of that
than a restricted subset of the language -- especially if the syntax
looks and works differently from real python.
Zach Pincus
Program in Biomedical Informatics and Department of Biochemistry
Stanford University School of Medicine
On Apr 29, 2006, at 11:24 AM, Talin wrote:
> PEP: 3101
> Title: Advanced String Formatting
> Version: $Revision$
> Last-Modified: $Date$
> Author: Talin
> Status: Draft
> Type: Standards
> Content-Type: text/plain
> Created: 16-Apr-2006
> Python-Version: 3.0
> Post-History:
>
>
> Abstract
>
> This PEP proposes a new system for built-in string formatting
> operations, intended as a replacement for the existing '%' string
> formatting operator.
>
>
> Rationale
>
> Python currently provides two methods of string interpolation:
>
> - The '%' operator for strings. [1]
>
> - The string.Template module. [2]
>
> The scope of this PEP will be restricted to proposals for
> built-in
> string formatting operations (in other words, methods of the
> built-in string type).
>
> The '%' operator is primarily limited by the fact that it is a
> binary operator, and therefore can take at most two arguments.
> One of those arguments is already dedicated to the format string,
> leaving all other variables to be squeezed into the remaining
> argument. The current practice is to use either a dictionary
> or a
> tuple as the second argument, but as many people have commented
> [3], this lacks flexibility. The "all or nothing" approach
> (meaning that one must choose between only positional arguments,
> or only named arguments) is felt to be overly constraining.
>
> While there is some overlap between this proposal and
> string.Template, it is felt that each serves a distinct need,
> and that one does not obviate the other. In any case,
> string.Template will not be discussed here.
>
>
> Specification
>
> The specification will consist of 4 parts:
>
> - Specification of a set of methods to be added to the built-in
>string class.
>
> - Specification of a new syntax for format strings.
>
> - Specification of a new set of class methods to control the
>formatting and conversion of objects.
>
> - Specification of an API for user-defined formatting classes.
>
>
> String Methods
>
> The build-in string class will gain a new method, 'format',
> which takes takes an arbitrary number of positional and keyword
> arguments:
>
> "The story of {0}, {1}, and {c}".format(a, b, c=d)
>
> Within a format string, each positional argument is identified
> with a number, starting from zero, so in the above example,
> 'a' is
> argument 0 and 'b' is argument 1. Each keyword argument is
> identified by its keyword name, so in the above example, 'c' is
> used to refer to the third argument.
>
> The result of the format call is an object of the same type
> (string or unicode) as the format string.
>
>
> Format Strings
>
> Brace characters ('curly braces') are used to indicate a
> replacement field within the string:
>
> "My name is {0}".format('Fred')
>
> The result of this is the string:
>
> "My name is Fred"
>
> Braces can be escaped using a backslash:
>
> "My name is {0} :-\{\}".format('Fred')
>
> Which would produce:
>
> "My name is Fred :-{}"
>
> The element within the braces is called a 'field'. Fields
> consist
> of a name, which can either be simple or compound, and an
> optional
> 'conversion spec
