after embedding and extending python (using swig) problem importing (non-core) modules
Hi Folks, I currenty extended some of my C++ functionality to python and also embedded python to use python functionality in my C++ system (and use as well these extended functions). While this works fine with the core python functionality, as soon as I run a script (on the embedded system) which tries to import modules which are not in the core system, like "xml" or "re", it fails and says it cannot find the related dll (for example in the case of re the _sre.pyd). It works fine if i run the script with the 'regular' interpreter. It does not look like a path-problem to me, so I'm clueless right now. I could work around this extending some functions to python and use them, but then I lose what I was aiming for, the "power" of python and python-modules. Do i have to tell the embedded python system somehow where to look for extension dll's? (it even does not work if i have the pyd files in the (same) folder than the system where I start my embedded python program. It would be great if someone would have an idea, at least how to get me started on this. thanks a lot in advance, -stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: after embedding and extending python (using swig) problem importing (non-core) modules
thanks a lot for the quick answer. I had to provide the debug-versions, since I was in debug mode, like you already expected! thanks a lot again! -stefan -- http://mail.python.org/mailman/listinfo/python-list
Howto get the argument defaults of a function using an AST-object?
Hi, for a school project I'm trying to analyze some python modules. I just struggeld over a little detail I cannot solve: shortened example of walking trough an AST with my own visitor: class MyVisitor: def visitFunction(self, t): # Here I do not know know how to associate the default values in t.defaults # with the arguments in t.argnames params = '???' # howto join the right args with their default??? print "def %s(%s)" % (t.name, params) fname = 'test.py' ast = compiler.parseFile(fname) compiler.walk(ast, MyVisitor()) I can only get a list with the arguments or the defaults. The following examples would return the same: def foo(x, y=True) def foo(x=True, y) Anyone an idea? Thanks, Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Howto get the argument defaults of a function using an AST-object?
Fredrik Lundh schrieb: > > > def foo(x, y=True) > > def foo(x=True, y) > > the latter is a syntax error. > ... > > the defaults array contains the defaults for the len(defaults) last arguments. And again the own stupidity: I didn't check all my testcases for syntax errors. Thanks a lot, Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Help with implementing callback functions using ctypes
dieter, 09.05.2013 07:54:
> jamadagni writes:
>> ...
> I cannot help you with "ctypes". But, if you might be able to use
> "cython", then calling callbacks is not too difficult
+1 for using Cython. It also has (multi-)source level gdb support, which
greatly helps in debugging crashes like this one.
http://docs.cython.org/src/userguide/debugging.html
> (you can find an example in e.g. my "dm.xmlsec.binding").
An "official" example is here:
https://github.com/cython/cython/tree/master/Demos/callback
> Note, however, that properly handling the GIL ("Global Interpreter Lock")
> may be of great importance when the Python/C boundary is crossed --
> at least when you intend to use your code in a multi thread environment.
Cython makes this really easy. You can use the "with" statement to release
the GIL when you don't need it, and the compiler will produce errors when
you try to do things that require the GIL while you don't own it.
Stefan
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python C-API: how to define nested classes?
Serge WEINSTOCK, 16.05.2013 10:55: > I'm currently writing a C extension module for python using the "raw" C-API. > I would like to be able to define "nested classes" like in the following > python code > > > class A: > class B: > def __init__(self): > self.i = 2 > def __init__(self): > self.b = A.B() > a = A() > b = A.B() > print(a.b.i) > print(b.i) > > > How can I create such nested class with the C-API? Assuming you really mean Python classes and not extension types, you might get away with defining them separately and then assigning A.B = B Recent Python versions added a __qualname__ attribute for classes that you might want to fix up in this case. If you need better compatibility, consider writing your code in Cython. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Short-circuit Logic
On 2013-05-30 08:29:41 +, Steven D'Aprano said: On Thu, 30 May 2013 10:22:02 +0300, Jussi Piitulainen wrote: I wonder why floating-point errors are not routinely discussed in terms of ulps (units in last position). ... That is an excellent question! ... I have a module that works with ULPs. I may clean it up and publish it. Would there be interest in seeing it in the standard library? ... I am definitely interested seeing this in the python standard library. But as I continued to read the lines following your proposal and the excellent article from Bruce pointed to by Carlos on this thread, maybe a package on pypi first grounding somewhat the presumably massive discussion thread on python-ideas :-?) All the best, Stefan. -- http://mail.python.org/mailman/listinfo/python-list
Re: How to check for threads being finished?
Irmen de Jong, 05.07.2013 19:12: > On 5-7-2013 18:59, Steven D'Aprano wrote: >> I then block until the threads are all done: >> >> while any(t.isAlive() for t in threads): >> pass >> >> >> Is that the right way to wait for the threads to be done? Should I stick >> a call to time.sleep() inside the while loop? If so, how long should I >> sleep? That's probably an unanswerable question, but some guidelines on >> choosing the sleep time will be appreciated. >> > > I think your while loop busy-waits until the threads are completed. > Do this instead: > > for t in threads: t.join()# optionally pass a timeout to join A related stdlib tool that many people don't seem to know is the thread pool in concurrent.futures. It supports both waiting for all threads to finish as well as iterating over results as they come in. It also comes with a parallel map() implementation. http://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor-example http://docs.python.org/3/library/concurrent.futures.html#module-functions New in Py3.2, but there's also a backport AFAIR. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Explain your acronyms (RSI?)
Rotwang, 06.07.2013 21:51: > On 06/07/2013 20:38, Terry Reedy wrote: >> "rms has crippling RSI" (anonymous, as quoted by Skip). >> [...] >> Let us try Google. Type in RSI and it offers 'RSI medications' as a >> choice. Sound good, as it will eliminate all the companies with those >> initials. The two standard medical meanings of RSI seem to be Rapid >> Sequence Intubation and Rapid Sequence Induction. But those are >> procedures, not chronic syndromes. So I still do not know what the >> original poster, as quoted by Skip, meant. > > Repetitive strain injury, I assume. Not sure if you're joking but over here > the top 7 hits for "RSI" on Google, as well as the three ads that precede > them, are repetitive strain injury-related. Both of you might want to delete your browser cookies, log out of your Google accounts, and then retry. Maybe disabling JavaScript helps. Or enabling the Privacy Mode in your browser. Or try a different browser all together. Or a different search engine. Google has lots of ways to detect who's asking. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Babel i18n package has new maintainers
Hi, I've been looking for a Python package for formatting international dates, numbers and monetary values in a web context for a couple of days now, but the only thing that I could find that looked suitable at first sight was the Babel package, of which the last release dates back two years, with stale localisation data and missing Py3 support. http://pythonhosted.org/Babel https://pypi.python.org/pypi/Babel Now, after searching a bit more, I found that Armin Ronacher and Alex Morega have recently taken over that package. https://groups.google.com/forum/#!msg/python-babel/WImO1u-Q8GA/j62ABuuwTpAJ They're now developing the code on github and porting it to Py3.3. https://github.com/mitsuhiko/babel Since it took me quite a while to figure this out, I thought I'd post this here to make others aware of it. I think it's a very important tool to have, and I'm sure Armin and Alex can use some help and testing by others to get a new release out. There's always stuff like reading through and updating the documentation, finding bugs, testing with your existing code base, etc., that everyone can help with. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Best Scripting Language for Embedded Work?
Christian Gollwitzer, 10.07.2013 09:03: > http://www.lua.org/ > > Very compact (a static binary is about ~200K), clean synatx, relatively > fast. OTOH, the standard library is of course not so extensive as for Tcl > or Python. "not so extensive" is a rather bold understatement. ISTM that most projects that use Lua build their own eco-system around it, or integrate Lua into their existing eco-system, with the obvious drawbacks for code reuse across different projects and user bases. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Mypy
Fábio Santos, 11.07.2013 10:16: > Guido tweeted that yesterday. It seems interesting. Although I'm not > comfortable using a subset of the language. > > They seem to want to kill the GIL. This could get much more popular when > they do. Then don't forget to also take a look at Cython. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Best Scripting Language for Embedded Work?
David T. Ashley, 12.07.2013 03:19: > On Wed, 10 Jul 2013 14:38:51 -0500, Johann Hibschman wrote: > >> David T. Ashley writes: >> >>> We develop embedded software for 32-bit micros using Windows as the >>> development platform. >> ... >>> I know that Tcl/Tk would do all of the above, but what about Python? >>> Any other alternatives? >> >> Given that list, I'd say just use Tcl and be done. You could force the >> square peg of python into that round hole, but I doubt it'd be worth the >> effort. > > I tend to agree with you. I'm not sure how go monolithic with Python. PyRun? py2exe? Just to name two. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Play Ogg Files
Devyn Collier Johnson, 20.07.2013 03:06:
> I am making a chatbot that I host on Launchpad.net/neobot. I am currently
> converting the engine from BASH code to Python3. I need to convert this for
> cross-platform compatibility. I do not need to use Mplayer; I just show the
> below code to give others a better idea what I am doing. I would prefer to
> be Python3 independent; I do not want to use the system shell. I am fine
> with using Python3 modules like Pygame (if there is a py3 module). As long
> as the code is fast, efficient, and simple without depending on the system
> shell or external apps, that would be nice. I also need the code to execute
> while the rest of the script continues running.
>
> jobs = multiprocessing.Process(SEND = subprocess.getoutput('mplayer
> -nogui -nolirc -noar -quiet ./conf/boot.ogg')) #Boot sound#
Well, since you mentioned it already, have you actually looked at pygame?
It should be able to do what you want. There's also pyaudio, which is more
specialised to, well, audio. A web search for python and ogg might provide
more.
Stefan
--
http://mail.python.org/mailman/listinfo/python-list
Re: How can I make this piece of code even faster?
[email protected], 21.07.2013 12:48: > El domingo, 21 de julio de 2013 12:31:42 UTC+2, Steven D'Aprano escribió: >> [steve@ando ~]$ python3.3 -m timeit -s "x = 2.357e7" "x**0.5" >> 100 loops, best of 3: 0.319 usec per loop >> [steve@ando ~]$ python3.3 -m timeit -s "x = 2.357e7" -s "from math import >> sqrt" "sqrt(x)" >> 1000 loops, best of 3: 0.172 usec per loop >> >> How exactly are you timing the code? > > I'm timing the whole program with cProfile. Removing math.sqrt() from a > function and using **(1/2) instead cut the execution time for a significant > amount (~0.035 to ~0.020). With or without the profiler running? Note that profiling will slow down your code (especially function calls), often significantly and sometimes even in such an unbalanced way that it visibly changes its execution profile. Always make sure you validate your code changes with benchmarks, outside of the profiler. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Play Ogg Files
Devyn Collier Johnson, 20.07.2013 14:25:
> On 07/20/2013 12:21 AM, Stefan Behnel wrote:
>> Devyn Collier Johnson, 20.07.2013 03:06:
>>> I am making a chatbot that I host on Launchpad.net/neobot. I am currently
>>> converting the engine from BASH code to Python3. I need to convert this for
>>> cross-platform compatibility. I do not need to use Mplayer; I just show the
>>> below code to give others a better idea what I am doing. I would prefer to
>>> be Python3 independent; I do not want to use the system shell. I am fine
>>> with using Python3 modules like Pygame (if there is a py3 module). As long
>>> as the code is fast, efficient, and simple without depending on the system
>>> shell or external apps, that would be nice. I also need the code to execute
>>> while the rest of the script continues running.
>>>
>>> jobs = multiprocessing.Process(SEND = subprocess.getoutput('mplayer
>>> -nogui -nolirc -noar -quiet ./conf/boot.ogg')) #Boot sound#
>> Well, since you mentioned it already, have you actually looked at pygame?
>> It should be able to do what you want. There's also pyaudio, which is more
>> specialised to, well, audio. A web search for python and ogg might provide
>> more.
>
> Thanks Stefan! I have not heard of Pyaudio; I will look into that. As for
> Pygame, I have not been able to find any good documentation for playing
> audio files.
A quick duckduckgo search gave me this, at least:
http://www.pygame.org/docs/ref/mixer.html
> Plus, I recently learned that Pygame is not Python3 compatible.
Looks like it's your lucky day:
http://www.pygame.org/wiki/FrequentlyAskedQuestions#Does%20Pygame%20work%20with%20Python%203?
Stefan
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python 3: dict & dict.keys()
Chris Angelico, 24.07.2013 18:34: > On Thu, Jul 25, 2013 at 1:57 AM, Ethan Furman wrote: >> On 07/24/2013 05:51 AM, Oscar Benjamin wrote: >>> What do you mean? Why would you want to create a temporary list just to >>> iterate over it explicitly or implicitly (set, >>> sorted, max,...)? >> >> You wouldn't. But you don't need .keys() for that either as you can just >> use the dict itself. > > Side point: Why is iterating over a dict equivalent to .keys() rather > than .items()? It feels odd that, with both options viable, the > implicit version iterates over half the dict instead of all of it. > Obviously it can't be changed now, even if .items() were the better > choice, but I'm curious as to the reason for the decision. The reason is that you can easily get at the values when iterating over the keys, or simply decide not to care about them and be happy with the keys only. Note that there are also many use cases that need all keys but not all values. If iteration always returned an item tuple by default, many use cases would have to resort to using .keys() in order to be efficient. And for the simple case, you'd have to type more, either the additional .keys() or the useless tuple unpacking. So, the reasoning is that iteration should do the basic thing that still allows you to do everything, instead of doing everything and pushing unnecessary work on the users by default. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 3: dict & dict.keys()
Peter Otten, 24.07.2013 08:23: > Ethan Furman wrote: >> So, my question boils down to: in Python 3 how is dict.keys() different >> from dict? What are the use cases? > > I just grepped through /usr/lib/python3, and could not identify a single > line where some_object.keys() wasn't either wrapped in a list (or set, > sorted, max) call, or iterated over. In case that directory mainly consists of the standard library, then you shouldn't forget that most of the code in there predates Python 3 by ages and was only adapted to work with the new syntax/semantics, not rewritten in a "better" way. Even if it's not just the stdlib, that argument still holds. There is still fairly little code out there that was specifically written for Py2.6+, as opposed to just being adapted. > To me it looks like views are a solution waiting for a problem. They reduce the API overhead. Previously, you needed values() and itervalues(), with values() being not more than a special case of what itervalues() provides anyway. Now it's just one method that gives you everything. It simply has corrected the tradeoff from two special purpose APIs to one general purpose API, that's all. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Dihedral
Fábio Santos, 16.07.2013 00:54: >> On 07/15/2013 08:36 AM, Steven D'Aprano wrote: >>> >>> Devyn, >>> >>> 8 Dihedral is our resident bot, not a human being. Nobody knows who >>> controls it, and why they are running it, but we are pretty certain that >>> it is a bot responding mechanically to keywords in people's posts. >>> >>> It's a very clever bot, but still a bot. About one post in four is >>> meaningless jargon, the other three are relevant enough to fool people >>> into thinking that maybe it is a human being. It had me fooled for a long >>> time. > > Does this mean he passes the Turing test? I'd say that "it" appears more correct. Or is there any indication of a specific bot gender? (I sure might have missed it...) Note that being of a specific gender is definitely not required to pass the Turing test. I'm not even sure pretending to have one is required. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 3: dict & dict.keys()
Ethan Furman, 24.07.2013 20:31: > On 07/24/2013 10:23 AM, Stefan Behnel wrote: >> Peter Otten, 24.07.2013 08:23: >>> Ethan Furman wrote: >>>> >>>> So, my question boils down to: in Python 3 how is dict.keys() different >>>> from dict? What are the use cases? >>> >>> To me it looks like views are a solution waiting for a problem. >> >> They reduce the API overhead. Previously, you needed values() and >> itervalues(), with values() being not more than a special case of what >> itervalues() provides anyway. Now it's just one method that gives you >> everything. It simply has corrected the tradeoff from two special purpose >> APIs to one general purpose API, that's all. > > I started this thread for two reasons: > > 1) Increase awareness that using `list(dict)` is a cross-version > replacement for `dict.keys()` > > 2) Hopefully learn something about when a view is useful. > > So far #2 is pretty much a failure. I think the question is: how else would you implement an interface that doesn't restrict itself to returning a list? I mean, previously, the following was totally inefficient in terms of memory: value in d.values() It now avoids creating an intermediate list copy of the values, thus running with no additional memory overhead (well, a constant, ok, but definitely not linear) and keeps users from resorting to the much more unfriendly for v in d.itervalues(): if v == value: return True else: return False in order to achieve the same thing. You can now even efficiently do this for items, i.e. (key, value) in d.items() That's equivalent to "d[key] == value", but uses a different protocol, meaning that you don't have to make a copy of the dict items in order to pass it into something that works on a set or iterable of 2-tuples (which is a way more generic interface than requiring a dict as input). These things chain much more cleanly now, without first having to explain the difference between items() and iteritems() and when to use which. It's all about replacing the old copy-to-list interface by something that is efficiently processable step by step. All of this started back when iterators became a part of the language, then generators, and now dict views. They may not be the hugest feature ever, but they definitely fit into the language much better and much more cleanly than the old copy-to-list way. Ask yourself, if they had been there in Python 1.x, would you even have thought about making the iter*() methods a part of the language? Would you really have wanted a shorter way to create a list of dict values than list(d.values())? Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: embedded python and threading
David M. Cotter, 26.07.2013 08:15: > in my app i initialize python on the main thread, then immediately call > PyEval_SaveThread() because i do no further python stuff on the main thread. > > then, for each script i want to run, i use boost::threads to create a new > thread, then on that thread i "ensure" the GIL, do my stuff, then release it. > > so, to test concurrency, on my first background thread, i do an infinite loop > that just logs "i'm alive", then calls sleep(0.25) > > so that thread continues to run forever (with it's GIL ensured) > > according to the doc: "In order to emulate concurrency of execution, the > interpreter regularly tries to switch threads" > > so i figure i can run another thread that does a single print statement: > >> ensure gil >> print my thing >> release gil > > and this DOES run. however, after releasing it's gil, i guess the interpeter > gets back to the first back thread, but then has this error immediately: > > 9: Traceback (most recent call last): > 9: File "", line 70, in ? > 9: File "", line 55, in main > 9: AttributeError: 'builtin_function_or_method' object has no attribute > 'sleep' > > suddenly the sleep module has been unloaded?? huh? i thought the thread > state had been preserved? You didn't show your code, but as a wild guess, maybe you did from time import time instead of import time somewhere? If not, please provide the exact example code that you are running. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: embedded python and threading
David M. Cotter, 26.07.2013 19:28: > DOH! as my second thread, i had been using a sample script that i had > copy-pasted without much looking at it. guess what? it prints the time. and > yes, it did "from time import time", which explains it all. Ah, and you were using the same globals dict for both scripts, I guess? That explains it then. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: collections.Counter surprisingly slow
Steven D'Aprano, 28.07.2013 22:51: > Calling Counter ends up calling essentially this code: > > for elem in iterable: > self[elem] = self.get(elem, 0) + 1 > > (although micro-optimized), where "iterable" is your data (lines). > Calling the get method has higher overhead than dict[key], that will also > contribute. It comes with a C accelerator (at least in Py3.4dev), but it seems like that stumbles a bit over its own feet. The accelerator function special cases the (exact) dict type, but the Counter class is a subtype of dict and thus takes the generic path, which makes it benefit a bit less than possible. Look for _count_elements() in http://hg.python.org/cpython/file/tip/Modules/_collectionsmodule.c Nevertheless, even the generic C code path looks fast enough in general. I think the problem is just that the OP used Python 2.7, which doesn't have this accelerator function. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: collections.Counter surprisingly slow
Serhiy Storchaka, 29.07.2013 21:37: > 29.07.13 20:19, Ian Kelly написав(ла): >> On Mon, Jul 29, 2013 at 5:49 AM, Joshua Landau wrote: >>> Also, couldn't Counter just extend from defaultdict? >> >> It could, but I expect the C helper function in 3.4 will be faster >> since it doesn't even need to call __missing__ in the first place. > > I'm surprised, but the Counter constructor with commented out import of > this accelerator is faster (at least for some data). Read my post. The accelerator doesn't take the fast path for dicts as Counter is only a subtype of dict, not exactly a dict. That means that it raises and catches a KeyError exception for each new value that it finds, and that is apparently more costly than the overhead of calling get(). So, my expectation is that it's faster for highly repetitive data and slower for mostly unique data. Maybe a "fast_dict_lookup" option for the accelerator that forces the fast path would fix this. The Counter class, just like many (most?) other subtypes of dict, definitely doesn't need the fallback behaviour. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: collections.Counter surprisingly slow
Stefan Behnel, 30.07.2013 08:39: > Serhiy Storchaka, 29.07.2013 21:37: >> 29.07.13 20:19, Ian Kelly написав(ла): >>> On Mon, Jul 29, 2013 at 5:49 AM, Joshua Landau wrote: >>>> Also, couldn't Counter just extend from defaultdict? >>> >>> It could, but I expect the C helper function in 3.4 will be faster >>> since it doesn't even need to call __missing__ in the first place. >> >> I'm surprised, but the Counter constructor with commented out import of >> this accelerator is faster (at least for some data). > > Read my post. The accelerator doesn't take the fast path for dicts as > Counter is only a subtype of dict, not exactly a dict. That means that it > raises and catches a KeyError exception for each new value that it finds, > and that is apparently more costly than the overhead of calling get(). > > So, my expectation is that it's faster for highly repetitive data and > slower for mostly unique data. > > Maybe a "fast_dict_lookup" option for the accelerator that forces the fast > path would fix this. The Counter class, just like many (most?) other > subtypes of dict, definitely doesn't need the fallback behaviour. Or rather drop the fallback path completely. It's not worth having code duplication if it's not predictable up-front (before looking at the data) if it will help or not. http://bugs.python.org/issue18594 Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Checking compatibility of a script across Python versions automatically
Andrew Berg, 18.06.2012 21:24: > Are there any tools out there that will parse a script and tell me if it > is compatible with an arbitrary version of Python and highlight any > incompatibilities? I need to check a few of my scripts that target 3.2 > to see if I can make them compatible with 3.0 and 3.1 if they aren't > already. I found pyqver, but it isn't accurate (at least for 3.2/3.3 > scripts) and hasn't been updated in 2 years. I could look over the docs > and do it manually, but one of the scripts isn't small, so I'd prefer > not to. My advice: write a good test suite for your code and use something like tox to run it under the various Python versions that you want to support. No static analysis tool will ever be able to find all portability problems. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Is python a interpreted or compiled language?
Dave Angel, 21.06.2012 02:53: > On 06/20/2012 07:30 PM, gmspro wrote: >> Is python a interpreted or compiled language? > > Ian has given you a good answer. But since you mention java, I'd like > to point out a few things that are different between the two > environments. He and I are describing CPython; jython and other > implementations don't use .pyc files, and they behave differently. In other words: it's not the language that is interpreted or compiled, it's an implementation that interprets or compiles a language. It may do so in various degrees of interpretation and compilation, such as JIT compilation of otherwise interpreted code. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Can't understand python C apis
gmspro, 23.06.2012 09:02: > I'm trying to understand the source code of python and how it works > internally. > But i can't understand the python C apis. > Too much macro calling there and python C api. > I can't understand those. > I've also read the doc for python C api. > > What should i do? Which file's code should i read to understand those > PyObject or other type and other C apis? The first thing to ask yourself is: why do you want to understand it? What is the thing you are trying to do with it? Once you've answered that, it'll be easy to tell you where to look. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: How can i call array_length to get the length of array object?
gmspro, 24.06.2012 10:01: > Why are some methods/functions named in this way in python? __len__ > > underscoreunderscoreNAMEunderscoreunderscore > > Is there any speciality of naming such methods? Yes. Look up "special methods" in the documentation. You may have noticed the correspondence between len() and __len__(). That is Python's way of allowing you to implement this kind of generic functionality (sometimes referred to as a protocol). You also asked why len() is a function instead of a method. Don't you find it much easier to use one function for everything than to look up and sometimes even learn one method for each kind of object you are dealing with? Python prefers simplicity here. You want the length? Use len(). Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Why are these files names started with underscore?
gmspro, 24.06.2012 10:42: > I see there are some files here started with underscore(_) , is there any > convention of something for it? > > http://hg.python.org/cpython/file/3b7230997425/Modules > > _bisectmodule.c > [...] > _winapi.c > > Is there any convention or something? Yes. Please read the CPython developer guide (click on "core development" on the Python front page). Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Why is python source code not available on github?
gmspro, 24.06.2012 02:16: > Why is python source code not available on github? > > Make it available on github so that we can git clone and work on source code. github != git. You can use git to work on the sources if you wish. Just install a Mercurial plugin for it and clone the code from the source repository. However, if you want to contribute to the project, it's still worth considering to work with Mercurial. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Why is python source code not available on github?
Peter Otten <[email protected]> wrote: > gmspro wrote: > > Why is python source code not available on github? Why should every free software project be available on a single proprietary platform? Also, see: http://arstechnica.com/business/2012/03/hacker-commandeers-github-to-prove-vuln-in-ruby/ Stefan Krah -- http://mail.python.org/mailman/listinfo/python-list
Re: Faster way to map numpy arrays
Saurabh Kabra, 25.06.2012 05:37: > I have written a script to map a 2D numpy array(A) onto another array(B) of > different dimension. more than one element (of array A) are summed and > mapped to each element of array B. To achieve this I create a list where I > store the index of array A to be mapped to array B. The list is the > dimension of array B (if one can technically say that) and each element is > a list of indices to be summed. Then I parse this list with a nested loop > and compute each element of array B. > > Because of the nested loop and the big arrays the process takes a minute or > so. My question is: is there a more elegant and significantly faster way of > doing this in python? I'm sure there's a way to do this kind of transformation more efficiently in NumPy. I faintly recall that you can use one array to index into another, something like that might do the trick already. In any case, using a NumPy array also for the mapping matrix sounds like a straight forward thing to try. But you might also want to take a look at Cython. It sounds like a problem where a trivial Cython implementation would seriously boost the performance. http://docs.cython.org/src/tutorial/numpy.html Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Why has python3 been created as a seperate language where there is still python2.7 ?
gmspro, 24.06.2012 05:46: > Why has python3 been created as a seperate language where there is still > python2.7 ? > > What's the benifit to make python3 over python2.7 ? I have read this though: > http://docs.python.org/release/3.0.1/whatsnew/3.0.html > > What's wrong editing/customizing/changin python2.7 instead of making a > seperate language? > > What's wrong working with python2.7? > > As python3 is not backward compatible, so many packages built on python2.7 > will be gone someday. Or you have to re-write/upgrade to python3. That's a > tedious/labourious task. > > So after 5 years will we get another python4 as seperate language? > > Any answer will be highly appreciated. Note that this topic has been discussed in full and overly full length several times on this list. You may want to read up on it in the archives. I'm not sure if you're just trolling (posting suggestive questions with well-known answers is a well established troll metier) or if you really want an answer to your questions, but here's a short answer anyway. The intention of Py3 was to deliberately break backwards compatibility in order to clean up the language. The situation is not as bad as you seem to think, a huge amount of packages have been ported to Python 3 already and/or work happily with both language dialects. It's not an either-or kind of thing, you can have both with a little effort. And, no, there won't be a Py4 in 5 years. The established release time frame is way longer. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Why has python3 been created as a seperate language where there is still python2.7 ?
Jeremiah Dodds, 26.06.2012 07:04: > [email protected] writes: > >> On Monday, June 25, 2012 5:10:47 AM UTC-5, Michiel Overtoom wrote: >>> It has not. Python2 and Python3 are very similar. It's not like if >>> you learn Python using version 2, you have to relearn the language >>> when you want to switch Python3. The syntax is the same, only >>> 'print' is a function instead of a statement. >> >> However, there is something to be said for "old habits die hard". I myself >> lament every time i must type->(, then blah, then->) AGAIN!. My fingers are >> hardwired for the old print statement. Damned that Guido and his mind games! >> >> http://www.youtube.com/watch?v=-Ny42Mdg5qo > > I'm of the opinion that the solution to this involves configuring your editor. But wasn't the whole idea behind Python to be accessible for innocent users? Isn't the one and only acceptable goal of a language cleanup that those who do not have the expert knowledge to configure the editor of their choice should finally be able to wholeheartedly type down their Pythonish code without having the parser allude them to the one holy way from time to time? It clearly cannot have achieved that goal, given the amount of traffic on this list, can it? Maybe we should add a remote error reporting mode to Python that sends all syntax error messages not only to the local screen but also directly to the PSF so that they can fund developers who are able to delete that error message from the interpreter based on real world statistical evidence. That would eventually make the language match the powerset of what all users have in their fingers. A very worthy goal, if you ask me. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Why has python3 been created as a seperate language where there is still python2.7 ?
Devin Jeanpierre, 26.06.2012 08:15: > On Mon, Jun 25, 2012 at 11:35 PM, Steven D'Aprano >> Making print a statement in the first place was a mistake, but >> fortunately it was a simple enough mistake to rectify once the need for >> backward compatibility was relaxed. > > Hmmm, why is the function so much better than the statement? You like > using it in expressions? Or is it that you like passing it in as a > callback? First of all, the statement has a rather special syntax that is not obvious and practically non-extensible. It also has hidden semantics that are hard to explain and mixes formatting with output - soft-space, anyone? The function is straight forward, configurable, does one thing, works with help() and doesn't get in the way. And something as rarely[1] used as a print simply doesn't deserve special syntax. Oh, and, yes, you can even pass it into some code as callback, although I rarely had a need for that. Stefan [1] Seriously, it's not very helpful in interactive mode and too simplistic to be used in application code. Even scripts often work better with logging than with prints. -- http://mail.python.org/mailman/listinfo/python-list
Re: Frustrating circular bytes issue
J, 26.06.2012 18:30:
> def _reader(self, file, size=4096, delimiter=r"\n{2,}"):
> buffer_old = ""
> while True:
> buffer_new = file.read()
> print(type(buffer_new))
> if not buffer_new:
> break
> lines = re.split(delimiter, buffer_old + buffer_new)
"delimiter" is a Unicode string, which makes the regular expression a
Unicode regex that can't work on a byte string.
> buffer_old = lines.pop(-1)
>
> for line in lines:
> yield line
>
> yield buffer_old
>
>
> (the print statement is something I put in to verify the problem.
>
> So stepping through this, when _reader executes, it executes read() on
> the opened filehandle. Originally, it read in 4096 byte chunks, I
> removed that to test a theory. It creates buffer_new with the output
> of the read.
>
> Running type() on buffer_new tells me that it's a bytes object.
Stefan
--
http://mail.python.org/mailman/listinfo/python-list
Re: Why has python3 been created as a seperate language where there is still python2.7 ?
Chris Angelico, 27.06.2012 13:02: > On Wed, Jun 27, 2012 at 8:25 PM, Christian Tismer wrote: >> I think, for the small importance of the print statement in code, it >> would have made the transition easier, if python 3 was as flexible >> as python 2.7, with a symmetric >> >> "from __past__ import print_statement" construct. > > For how long? Will Python require, in perpetuity, the code to support > this? Must the print statement be enhanced when the print function is? > What about bug fixes? How much dev time is required to enable backward > compatibility past a boundary across which backward compatibility was > not promised? And if there's a limit to the duration of this __past__ > directive, when should it be and what should happen after that point? > > Much easier to simply say no. I concur. For supporting something that shouldn't have been there in the first place, such as a print statement in production code, it's way too much overhead to keep it working despite of significant changes in the I/O layer that went into Python 3.x, including the removal of the soft-space "feature". For comparison, the revival of the "u" string prefix in Py3.3 is a simple change in the parser grammar that's easy to maintain but that has a huge impact on the Py3 compatibility of code that accepts to drop support for Py2.5 and earlier (as well as Py3.[012]) but wants to keep working in Py2.[67] (which supports the opposite "b" prefix). We've been supporting that in Cython for a while and it worked out really well so far. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Why has python3 been created as a seperate language where there is still python2.7 ?
Christian Tismer, 27.06.2012 15:15: > print, function or not, is not important enough to enforce a rewrite > everywhere because of syntax error. That hides the real semantic > changes which _are_ important. > > So what I would have done is to let it work in an imperfect way. People > then have the chance to rewrite with the print function, where it makes > sense. But old packages which need to be changed, only because they > have lots of > > if DEBUG: > print xxx, yyy, ... > > could just stay unchanged or migrated later after the real meat has > been properly tested etc. Well, at least a SyntaxError makes it easy to find. And even if you don't intend to use 2to3, you can still run it once to generate a patch for you that only fixes up "print". In many cases, that will also make it work in Py2 in such an "imperfect" way, either by doing the right thing already or by printing out a tuple instead of a space separated string. If it's only for debug output (which, as I said, would better be served by the logging module), I can't see why that would be all that unacceptable. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Why has python3 been created as a seperate language where there is still python2.7 ?
Serhiy Storchaka, 28.06.2012 07:36: > On 28.06.12 00:14, Terry Reedy wrote: >> Another prediction: people who code Python without reading the manual, >> at least not for new features, will learn about 'u' somehow (such as by >> reading this list) and may do either of the following, both of which are >> bad. >> >> 1. They will confuse themselves by thinking that 'u' actually means >> somethings. They may then confuse others by writing about its supposed >> meaning. This might get amusing. >> >> 2. They will use 'u' in Python 3 only code, thereby making it >> incompatible with 3.2-, even if it otherwise would not be. >> >> These two actions will reinforce each other. > > Yes, this is what I mean. I can even make a prediction: in just 5 years, as > this feature would be banned in a decent society. The authors of the books > will be strongly advise not to use it, and in software companies 'u' will > be prohibited in coding style. But get rid of this will be difficult. Once Py2.7 is out of maintenance, we can deprecate that feature in one release and start warning about it in the next one. You're then free to use the corresponding 2to3 fixer to get it back out of your code with a single patch. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: when "normal" parallel computations in CPython will be implemented at last?
Dan Stromberg, 01.07.2012 21:28: > On Sun, Jul 1, 2012 at 11:58 AM, Thomas Jollans wrote: >> On 07/01/2012 08:44 PM, Dan Stromberg wrote: >>> IronPython, sadly, lacks a python standard library. >> >> Beg pardon? >> >> https://github.com/IronLanguages/main/tree/master/External.LCA_RESTRICTED/Languages/IronPython/27/Lib > > Perhaps things have changed. Yes, they have. The original restriction came from Microsoft enforcing a clean-room implementation for their code, which obviously excluded the standard library - which is well tested and licence cleared and all that, but nevertheless non-MS. When they dropped the IronPython project, it became free to integrate with other software. Clearly a cultural thing. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Best data structure for DFS on large graphs
Miheer Dewaskar, 03.07.2012 13:11: > I want to make a combinatorial game solver in python.The algorithm is to > perform Depth First Search (DFS) on the states of the game. > For DFS I,would need to keep a record of the visited states.What is the > best data structure for it,keeping in mind that the number of states could > be large? > > I was thinking about using Dictionary or a Binary Search Tree (BST) > ,assuming that the states can be made hashable and totally ordered > respectively. > I am not sure,but if there are large number of states Dictionaries wont > help much right? Dicts are fast for lookup, not for searching. > Anyway, does python have a built-in BST like data-structure ? It has lists and bisect: http://docs.python.org/library/bisect.html Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: 2 + 2 = 5
Paul Rubin, 04.07.2012 21:37: > I just came across this (https://gist.github.com/1208215): > > import sys > import ctypes > pyint_p = ctypes.POINTER(ctypes.c_byte*sys.getsizeof(5)) > five = ctypes.cast(id(5), pyint_p) > print(2 + 2 == 5) # False > five.contents[five.contents[:].index(5)] = 4 > print(2 + 2 == 5) # True (must be sufficiently large values of 2 there...) > > Heh. The author is apparently anonymous, I guess for good reason. That's not portable, though. ;) Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Interview Questions
Mark Lawrence, 10.07.2012 11:42: > I recall reading in a book in the local library > of a manager that wouldn't employ people unless they were wearing a new > pair of shoes. Guess they didn't take many people on. Managers tend to like wasting resources. Buying a new pair of shoes for each job interview sounds reasonable once you have a salary well beyond your own capabilities. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing a wrapper - any tips?
Martin P. Hellwig, 13.07.2012 09:39: > On Friday, 13 July 2012 05:03:23 UTC+1, Temia Eszteri wrote: >> I'm going to be looking into writing a wrapper for the Allegro 5 game >> development libraries, either with ctypes or Cython. They technically >> have a basic 1:1 ctypes wrapper currently, but I wanted to make >> something more pythonic, because it'd be next to impossible to deal >> with the memory management cleanly in the script itself. >> >> Anything I should keep in mind? Any tips to pass on to a first-time >> module writer, pitfalls to watch out for, etc.? > > I would split the wrapping in layers, the lowest layer is a one on one > exposure of the library with your wrapper, I would rather avoid ctypes > for performance reasons, however if performance is not a concern ctypes > is excellent and broadly available. > > The next layer is purely there to make the lower layer pythonic, i.e. > apply namespaces, automatic handling of memory, PEP8 naming convetions, > etc. etc. just what you would expect from a modern pure python module > > The next layer, if you want to, contains tools that are often used in > that concept, think in the line of design patterns. And the good thing about Cython in this context is that even if performance *is* a concern, you can move code around between all three layers freely in order to adjust it to your performance requirements and even drop the lowest layer entirely. In fact, my advice is to really skip that lowest layer in a Cython wrapper, because if it's really just a 1:1 mapping, it's going to end up with a lot of boring and redundant code and you won't gain anything from it. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Implicit conversion to boolean in if and while statements
Andrew Berg, 15.07.2012 10:34: > This has probably been discussed before, but why is there an implicit > conversion to a boolean in if and while statements? There isn't. This has nothing to do with "if" or "while". All objects have a truth value in Python, evaluating to True by default (object), unless they implement the test themselves. As Chris said, very convenient. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: jython lacks working xml processing modules?
[email protected], 17.07.2012 10:35: > hi,do you know the PyXML whether can be supported by Jython ? PyXML is a dead project, don't use it. You can use ElementTree in Jython, just as in Python. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: jython lacks working xml processing modules?
Matej Cepl, 17.07.2012 11:39: > On 17/07/12 10:35, [email protected] wrote: >>> > I'm trying to parse an xml file with jython (not through java >>> parsers >>> > like xerces). > > https://code.google.com/p/jython-elementtree/ ??? Note that this ships with Jython 2.5. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: dict: keys() and values() order guaranteed to be same?
Henrik Faber, 23.07.2012 13:23:
> I have a question of which I'm unsure if the specification guarantees
> it. With an arbitrary dictionaty d, are d.keys() and d.values()
> guaraneed to be in the same order? I.e. what I mean is:
>
> # For all dictionaries d:
> assert({ list(d.keys())[i]: list(d.values())[i] for i in range(len(d)) }
> == d)
>
> I'm curious if it's allowed because in a special case it would make for
> a nice shortcut and clean code. I think however that the implementation
> may chose not to have them in the same order necessarily -- then I'd
> obviously avoid relying on it.
You should.
Stefan
--
http://mail.python.org/mailman/listinfo/python-list
Re: dict: keys() and values() order guaranteed to be same?
Philipp Hagemeister, 23.07.2012 13:40: > On 07/23/2012 01:23 PM, Henrik Faber wrote: >> With an arbitrary dictionaty d, are d.keys() and d.values() >> guaraneed to be in the same order? > > Yes. From the documentation[1]: > > If items(), keys(), values(), iteritems(), iterkeys(), and itervalues() > are called with no intervening modifications to the dictionary, the > lists will directly correspond. > > [1] http://docs.python.org/library/stdtypes.html#dict.items Interesting. I wonder if other implementations like Jython and PyPy really adhere to this official guarantee. At least Jython has the same paragraph in its documentation and I would expect that PyPy follows it as well. http://www.jython.org/docs/library/stdtypes.html#mapping-types-dict Maybe this guarantee is just easy enough to build on the given implementation details of a platform that it's a common property. Iteration over data structures should tend to be deterministic, after all. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: catch UnicodeDecodeError
Jaroslav Dobrek, 26.07.2012 09:46:
> My problem is solved. What I need to do is explicitly decode text when
> reading it. Then I can catch exceptions. I might do this in future
> programs.
Yes, that's the standard procedure. Decode on the way in, encode on the way
out, use Unicode everywhere in between.
> I dislike about this solution that it complicates most programs
> unnecessarily. In programs that open, read and process many files I
> don't want to explicitly decode and encode characters all the time. I
> just want to write:
>
> for line in f:
And the cool thing is: you can! :)
In Python 2.6 and later, the new Py3 open() function is a bit more hidden,
but it's still available:
from io import open
filename = "somefile.txt"
try:
with open(filename, encoding="utf-8") as f:
for line in f:
process_line(line) # actually, I'd use "process_file(f)"
except IOError, e:
print("Reading file %s failed: %s" % (filename, e))
except UnicodeDecodeError, e:
print("Some error occurred decoding file %s: %s" % (filename, e))
Ok, maybe with a better way to handle the errors than "print" ...
For older Python versions, you'd use "codecs.open()" instead. That's a bit
messy, but only because it was finally cleaned up for Python 3.
> or something like that. Yet, writing this means to *implicitly* decode
> text. And, because the decoding is implicit, you cannot say
>
> try:
> for line in f: # here text is decoded implicitly
>do_something()
> except UnicodeDecodeError():
> do_something_different()
>
> This isn't possible for syntactic reasons.
Well, you'd normally want to leave out the parentheses after the exception
type, but otherwise, that's perfectly valid Python code. That's how these
things work.
> The problem is that vast majority of the thousands of files that I
> process are correctly encoded. But then, suddenly, there is a bad
> character in a new file. (This is so because most files today are
> generated by people who don't know that there is such a thing as
> encodings.) And then I need to rewrite my very complex program just
> because of one single character in one single file.
Why would that be the case? The places to change should be very local in
your code.
Stefan
--
http://mail.python.org/mailman/listinfo/python-list
Re: catch UnicodeDecodeError
Jaroslav Dobrek, 26.07.2012 12:51: >>> try: >>> for line in f: # here text is decoded implicitly >>>do_something() >>> except UnicodeDecodeError(): >>> do_something_different() > > the code above (without the brackets) is semantically bad: The > exception is not caught. Sure it is. Just to repeat myself: if the above doesn't catch the exception, then the exception did not originate from the place where you think it did. Again: look at the traceback. >>> The problem is that vast majority of the thousands of files that I >>> process are correctly encoded. But then, suddenly, there is a bad >>> character in a new file. (This is so because most files today are >>> generated by people who don't know that there is such a thing as >>> encodings.) And then I need to rewrite my very complex program just >>> because of one single character in one single file. >> >> Why would that be the case? The places to change should be very local in >> your code. > > This is the case in a program that has many different functions which > open and parse different > types of files. When I read and parse a directory with such different > types of files, a program that > uses > > for line in f: > > will not exit with any hint as to where the error occurred. I just > exits with a UnicodeDecodeError. ... that tells you the exact code line where the error occurred. No need to look around. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: catch UnicodeDecodeError
Philipp Hagemeister, 26.07.2012 14:17: > On 07/26/2012 01:15 PM, Stefan Behnel wrote: >>> exits with a UnicodeDecodeError. >> ... that tells you the exact code line where the error occurred. > > Which property of a UnicodeDecodeError does include that information? > > On cPython 2.7 and 3.2, I see only start and end, both of which refer to > the number of bytes read so far. Read again: "*code* line". The OP was apparently failing to see that the error did not originate in the source code lines that he had wrapped with a try-except statement but somewhere else, thus leading to the misguided impression that the exception was not properly caught by the except clause. Stefan -- http://mail.python.org/mailman/listinfo/python-list
counting source lines (was: Is Python a commercial proposition ?)
lipska the kat, 29.07.2012 18:01: > My most recent experience is with Java. The last project I was involved > with included 6775 java source files containing 1,145,785 lines of code. > How do I know this? because I managed to cobble together a python script > that walks the source tree and counts the lines of code. It ignores block > and line comments and whitespace lines so I'm fairly confident it's an > accurate total. It doesn't include web interface files (mainly ..jsp and > HTML) or configuration files (XML, properties files and what have you). In > fact it was remarkably easy to do this in python Just a comment on this part. An even easier way to count source lines is by using the tool "SLOCCount". It also works with various languages. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Python a commercial proposition ?
Tim Chase, 29.07.2012 20:28: > On 07/29/12 12:13, Michael Hrivnak wrote: >> - Operating system installer: http://fedoraproject.org/wiki/Anaconda >> - Software repository management: http://pulpproject.org/ >> - Software package installation: >> http://en.wikipedia.org/wiki/Ubuntu_Software_Center >> - Cloud computing: http://en.wikipedia.org/wiki/OpenStack > > I'll include both the Bazaar and Mercurial DVCS tools which are > mostly Python (I understand some inner loops drop to C, but both > have the option to fall back to a pure Python implementation). I find it perfectly reasonable to use C code (and other kinds of low-level code) in Python tools and applications. In fact, easy interfacing with low-level code is one of (C)Python's major selling points. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Python a commercial proposition ?
Rodrick Brown, 30.07.2012 02:12: > On Jul 29, 2012, at 12:07 PM, lipska the kat wrote: >> I'm trying to understand where Python fits into the set of commonly >> available, commercially used languages of the moment. > > Python is a glue language much like Perl was 10 years ago. Until the > GIL is fixed Enough people have commented on this piece of FUD already. > I doubt anyone will seriously look at Python as an option > for large enterprise standalone application development. I know enough examples to recognise this as nonsense. You mentioned working in "financials" and even there I know at least one not-so-small bank that's been developing their internal (EAI and business process) code in Python for almost a decade now. And their developers are quite happy with it, certainly happier than many of the Java developers I've met in other banks. Still, you may still get away with the above statement by providing a sufficiently narrow definition of "standalone". By my definition, there isn't much "standalone" code out there. Most code I know interfaces with a couple of external tools, libraries or backends, usually written in languages I don't have to care about because they provide a language independent interface. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: py2c - an open source Python to C/C++ is looking for developers
alex23, 31.07.2012 02:16: > On Jul 31, 2:42 am, MaxTheMouse wrote: >> What is the difference between this and Shedskin? Shedskin being a >> (restricted) python-to-C++ compiler. (http://code.google.com/p/ >> shedskin/) Is the goal to be able to handle any python code or a >> subset? > > There's also Nuitka, which is an unrestricted compiler, I believe: > http://nuitka.net/pages/overview.html Not to forget Cython, which is the only Python-to-C compiler that is in widespread use. > Is this a completely independent project, or are there plans to > leverage off of PyPy's toolchain, for example? >From a look at the source code, it seems hard to bring it together with anything. It looks very monolithic. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Python a commercial proposition ?
Paul Rubin, 31.07.2012 06:45: > A real compiler (PyPy) will help Python performance far more than > multi-core currently can. That's too general a statement to be meaningful. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: profiling and optimizing
Rita, 31.07.2012 14:13: > I recently inherented a large python process and everything is lovely. As a > learning experience I would like to optimize the code so I ran it thru the > profiler > > python -m cProfile myscript.py > > It seems majority of the time is taking in the deep copy but that seems to > come from a function (or functions) in the code. Is there a way to optimize > that? perhaps have a C implementation of the deep copy? Would that > be feasible? Those approaches would be my second (or maybe even later) step. The first one would be to check if the deep copy is really necessary, and if so, if it is really necessary to do it over the full depth of the entire object graph. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: py2c - an open source Python to C/C++ is looking for developers
Stefan Behnel, 31.07.2012 07:23: > From a look at the source code, it seems hard to bring it together with > anything. It looks very monolithic. Hmm, sorry, I mixed it up with "2c.py", which is yet another of those Python-to-C compilers with an all too similar name. https://code.google.com/p/2c-python/ There are a couple of others here: http://wiki.python.org/moin/PythonImplementations Seeing the number of those compilers, almost none of which is commonly used and/or still alive as a project, the question really is: why another one? I mean, it's totally fine as a hobby educational project, sure, and I really don't want to discourage anyone from going through this to have fun. But apart from "just for fun", what is the goal that would make this particular compiler different from the others? And also different enough to merit its own source base, instead of basing it on one of the existing projects? I don't consider "source is harder to read than to write" a good answer to this in general. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: docx/lxml
Cyrille Leroux, 31.07.2012 17:01: > I'm giving pip a try : > > > 1/ Linux (debian lenny) > - (as root) sh setuptools-0.6c11-py2.7.egg (ok) > - (as root) cd pip-1.1 ; python setup.py install (ok) > - pip : ImportError : No module named pkg_resources > - damn, I guess it's going to be a pain, again > - ... then I remember that anyway, I cannot access internet from my session > - end of story for linux > > > 2/ Back on Windows (7) > > - setuptools-0.6c11.win32-py2.7.exe (ok) > - (cmd) easy_install.exe pip : AttributeError : 'NoneType' object has no > attribute 'clone' > > Well, I think I'm starting to feel less and less enthusiastic about python... > I didn't suspect it would be so difficult to write a basic doc file :) "pip" goes with "distribute", not "setuptools". http://packages.python.org/distribute/ Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Python a commercial proposition ?
David, 01.08.2012 13:59: > On 01/08/2012, lipska the kat wrote: >> On 01/08/12 09:06, Mark Lawrence wrote: >>> >>> You complete ignoramus, if it gets poured in advance that's no good to >>> anybody as it'll go flat. Has to stay in the pump until you're ready to >>> drink it from the glass. Don't you know anything about the importance of >>> process and timing? :) >> >> Heh heh, obviously never got drunk ... er I mean served behind the bar >> at uni/college/pub %-} > > Nah, obviously *is* drunk ;p Would you mind taking this slightly off-topic discussion off the list? Thanks. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: C extension module doesn't throw exception after setting error indicator through PyErr_SetString()
rahul, 02.08.2012 11:50: > When I use same code base for Python 3.x, then behavior is different. You might want to take a look at Cython. It moves most of these "funny" little details off your shoulders. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Python a commercial proposition ?
Prasad, Ramit, 03.08.2012 08:51: >> I'm in stuck record mode here, but one of the things I really enjoy >> about reading here is the way things do go off topic. IMHO makes for a >> far more interesting experience. YMMV. > > +1 > > Ramit > This email is confidential and subject to important disclaimers and > conditions including on offers for the purchase or sale of > securities, accuracy and completeness of information, viruses, > confidentiality, legal privilege, and legal entity disclaimers, > available at http://www.jpmorgan.com/pages/disclosures/email. Huh? Who's still trying to sell viruses these days? I thought they came for free? Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Steven D'Aprano, 04.08.2012 08:15: > Most people are aware, if only vaguely, of the big Four Python > implementations: > > CPython, or just Python, the reference implementation written in C. > IronPython, written in .NET. > Jython, written in Java. > PyPy, the optimizing implementation written in Python (actually, it's > written in a subset of Python, RPython). > > But the Python ecosystem is a lot bigger than just those four. Here are > just a few other implementations that you might be interested in: > > > Stackless - the "forgetten Python", Stackless is, I believe, the oldest > implementation behind only CPython itself. It's a fork of CPython with > the calling stack removed and fast and lightweight microthreads, and is > used extensively in EVE Online. > > http://www.stackless.com/ > > > Nuitka - optimising Python compiler written in C++, supports Python 2.6 > and 2.7, claims to be up to twice as fast as CPython. > > http://nuitka.net/pages/overview.html > > > WPython - another optimizing version of Python with wordcodes instead of > bytecodes. > > http://code.google.com/p/wpython/ > > > CLPython, an implementation of Python written in Common Lisp. > > http://common-lisp.net/project/clpython/ > > > CapPython is an experimental restricted version of Python with > capabilities. > > http://plash.beasts.org/wiki/CapPython > http://en.wikipedia.org/wiki/Object-capability_model > > > Berp - a compiler which works by translating Python to Haskell and > compiling that. > > https://github.com/bjpop/berp/wiki And not to forget Cython, which is the only static Python compiler that is widely used. Compiles and optimises Python to C code that uses the CPython runtime and allows for easy manual optimisations to get C-like performance out of it. http://cython.org/ Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Steven D'Aprano, 04.08.2012 09:49: > On Sat, 04 Aug 2012 08:40:16 +0200, Stefan Behnel wrote: >> And not to forget Cython, which is the only static Python compiler that >> is widely used. Compiles and optimises Python to C code that uses the >> CPython runtime and allows for easy manual optimisations to get C-like >> performance out of it. >> >> http://cython.org/ > > Cython is great, but I question that it is a *Python* implementation. > That's not a criticism of Cython, but it is different from Python. Take > this example code from the tutorial: > > from libc.math cimport sin > > cdef double f(double x): > return sin(x*x) > > If that's Python code, then I'm Ethel the Aardvark. We never met in person, so I can't comment on the last part. However, the above is Cython code and, yes, that's a different language. Note that it uses a different file extension: ".pyx". Try putting the above code into a .py file and compiling that. Cython will reject it and tell you that "cimport" is not valid Python syntax. > Cython is very Python-like, but there is no doubt in my mind that it is a > superset of Python and therefore a different language. As long as you don't use any features of the Cython language, it's plain Python. That makes it a Python compiler in my eyes. The fact that you can easily use C features to optimise your code (also in Python syntax, BTW) doesn't impact that. You mentioned a couple of other implementations and I'm sure they have features (and bugs) that CPython doesn't have. Interaction with Lisp code in CLPython, for example. I don't think additional features or language implementation bugs make a Python implementation non-Python per se. Also note that most of the less widely known "alternative Python implementations" do not even publish their results of running the CPython test suite, so how do you actually know they can run Python code? Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Mark Lawrence, 04.08.2012 12:05: > I agree so it's off topic and can't be discussed here. Isn't that right, > Stefan? Hmm, in case you are referring to a recent friendly and diplomatic request of mine regarding a couple of people who were burdening a public high volume mailing list with a purely private back-and-forth chat about having beer and getting drunk - then, no, I don't think the discussion in this thread qualifies as yet another example for that so far. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Steven D'Aprano wrote: > Who would want to deal with C's idiosyncrasies, low-powered explicit type > system, difficult syntax, and core-dumps, when you could use something > better? In the free software world, apparently many people like C. C is also quite popular in the zero-fault software world: Several verification tools do exist and Leroy et al. are writing a certified compiler for C to plug the hole between the verified source code and the generated assembly. Stefan Krah -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Steven D'Aprano, 04.08.2012 12:54: > Berp is based on the Glasgow Haskell Compiler, which is a modern, > efficient, optimizing compiler capable of producing excellent quality > machine code on Windows, Mac, Linux and many Unixes. It gives you all the > advantages of a high-level language with high-level data structures, type > inference, and a compiler capable of generating optimized, fast, machine > code. Although all those optimisations don't mean that Python code would run fast on top of it. Just because you translate Python to another language and platform doesn't mean that there's any benefit from the underlying platform optimisations. Both PyPy and Cython run Python code faster than CPython, but not because they eventually translate it into machine code but because they optimise and specialise it along the way, based on its high-level code constructs. One big success of the Unladen Swallow project was to show that bare JIT compilation is mostly worthless for high level languages. > Who would want to deal with C's idiosyncrasies, low-powered explicit type > system, difficult syntax, and core-dumps, when you could use something > better? The core developers of both CPython and Cython aim for exactly that. They write C so you don't have to. But keep in mind that C is still *the* lingua franca of software development. A major reason why Python is (slowly) catching up these days is that the main implementation is written in C and makes it easy to interface with C code. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Thomas Rachel, 04.08.2012 14:51: > Am 04.08.2012 11:10 schrieb Stefan Behnel: >> As long as you don't use any features of the Cython language, it's plain >> Python. That makes it a Python compiler in my eyes. > > Tell that the C++ guys. C++ is mainly a superset of C. But nevertheless, C > and C++ are distinct languages and so are Python and Cython. So, if a C++ compiler takes a .c file and compiles it with C language semantics, it doesn't qualify as a C compiler? That implies a rather weird definition of a C compiler, I'd say. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Stefan Behnel, 04.08.2012 15:53: > Thomas Rachel, 04.08.2012 14:51: >> Am 04.08.2012 11:10 schrieb Stefan Behnel: >>> As long as you don't use any features of the Cython language, it's plain >>> Python. That makes it a Python compiler in my eyes. >> >> Tell that the C++ guys. C++ is mainly a superset of C. But nevertheless, C >> and C++ are distinct languages and so are Python and Cython. > > So, if a C++ compiler takes a .c file and compiles it with C language > semantics, it doesn't qualify as a C compiler? That implies a rather weird > definition of a C compiler, I'd say. Ah, sorry. Got it. You were again talking about Cython the language. Sure, Cython the language is different from Python the language. Cython the compiler can compile both. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Paul Rubin, 04.08.2012 17:59: > Stefan Krah writes: >> In the free software world, apparently many people like C. C is also >> quite popular in the zero-fault software world: Several verification >> tools do exist and Leroy et al. are writing a certified compiler for >> C to plug the hole between the verified source code and the generated >> assembly. > > C is pretty poor as a compiler target: how would you translate Python > generators into C, for example? Depends. If you have CPython available, that'd be a straight forward extension type. Otherwise, I guess you'd either have a class for them in C++ or a struct in C. Not exactly complex. For the yielding, you can use labels and goto. Given that you generate the code, that's pretty straight forward as well. > How would you handle garbage collection? CPython does it automatically for us at least. Lacking that, you'd use one of the available garbage collection implementations, or provide none at all. > Haskell doesn't sound all that great as a translation target for Python > either, unfortunately, because its execution semantics are so different. > GHC is a very powerful compiler but it was made to compile Haskell code > that people actually write, and may do less good of a job with compiler > output from an imperative language like Python. Compiling Python to > Scheme and then using a Scheme compiler might be a more natural fit. > But, compiling to Haskell was probably pretty convenient for that > particular project. You'd have some kind of emulation layer that does most of the translation at runtime. That's why I said that you shouldn't expect too much of a performance gain from what the platform gives you for the underlying implementation. It can optimise the emulator, but it won't see enough of the Python code to make anything efficient out of it. Jython is an example for that. > Finally, Python itself isn't all that well suited for compilation, given > its high dynamicity. You can get pretty far with static code analysis, optimistic optimisations and code specialisation. We've decided against whole program analysis in Cython not only for compiler complexity reasons but also because it would let the normal compilation time explode for gains that you can much easier get by manual optimisation. Obviously, optimising JIT compilers can do much more here (because they actually have to do less), although they won't always be able to figure out the right thing to do either. That's where manual optimisation wins again. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Paul Rubin, 04.08.2012 20:18: > Stefan Behnel writes: >>> C is pretty poor as a compiler target: how would you translate Python >>> generators into C, for example? >> Depends. If you have CPython available, that'd be a straight forward >> extension type. > > Calling CPython hardly counts as compiling Python into C. CPython is written in C, though. So anything that CPython does can be done in C. It's not like the CPython project used a completely unusual way of writing C code. Besides, I find your above statement questionable. You will always need some kind of runtime infrastructure when you "compile Python into C", so you can just as well use CPython for that instead of reimplementing it completely from scratch. Both Cython and Nuitka do exactly that, and one of the major advantages of that approach is that they can freely interact with arbitrary code (Python or not) that was written for CPython, regardless of its native dependencies. What good would it be to throw all of that away, just for the sake of having "pure C code generation"? >> For the yielding, you can use labels and goto. Given that you generate >> the code, that's pretty straight forward as well. > > You're going to compile the whole Python program into a single C > function so that you can do gotos inside of it? What happens if the > program imports a generator? No, you are going to compile only the generator function into a function that uses gotos, maybe with an additional in-out struct parameter that holds its state. Then, on entry, you read the label (or its ID) from the previous state, reset local variables and jump to the label. On exit, you store the state back end return. Cython does it that way. Totally straight forward, as I said. >>> How would you handle garbage collection? >> CPython does it automatically for us at least. > > You mean you're going to have all the same INCREF/DECREF stuff on every > operation in compiled data? Ugh. If you don't like that, you can experiment with anything from a dedicated GC to transactional memory. >> Lacking that, you'd use one of the available garbage collection >> implementations, > > What implementations would those be? There's the Boehm GC which is > useful for some purposes but not really suitable at large scale, from > what I can tell. Is there something else? No idea - I'll look it up when I need one. Last I heard, PyPy had a couple of GCs to choose from, but I don't know how closely the are tied into its infrastructure. >> or provide none at all. > > You're going to let the program just leak memory until it crashes?? Well, it's not like CPython leaks memory until it crashes, now does it? And it's written in C. So there must be ways to handle this also in C. Remember that CPython didn't even have a GC before something around 2.0, IIRC. That worked quite ok in most cases and simply left the tricky cases to the programmers. It really depends on what your requirements are. Small embedded systems, time critical code and real-time systems are often much better off without garbage collection. It's pure convenience, after all. >> you shouldn't expect too much of a performance gain from what the >> platform gives you for the underlying implementation. It can optimise >> the emulator, but it won't see enough of the Python code to make >> anything efficient out of it. Jython is an example for that. > > Compare that to the performance gain of LuaJIT and it starts to look > like something is wrong with that approach, or maybe some issue inherent > in Python itself. Huh? LuaJIT is a reimplementation of Lua that uses an optimising JIT compiler specifically for Lua code. How is that similar to the Jython runtime that runs *on top of* the JVM with its generic byte code based JIT compiler? Basically, LuaJIT's JIT compiler works at the same level as the one in PyPy, which is why both can theoretically provide the same level of performance gains. >> You can get pretty far with static code analysis, optimistic >> optimisations and code specialisation. > > It seems very hard to do reasonable optimizations in the presence of > standard Python techniques like dynamically poking class instance > attributes. I guess some optimizations are still possible, like storing > attributes named as literals in the program in fixed slots, saving some > dictionary lookups even though the slot contents would have to still be > mutable. Sure. Even when targeting the CPython runtime with the generated C code (like Cython or Nuitka), you can still do a lot. And sure, static code analysis will never be able to infer everything that a JIT compiler can see. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Paul Rubin, 04.08.2012 22:43: > Stefan Behnel writes: >>> Calling CPython hardly counts as compiling Python into C. >> CPython is written in C, though. So anything that CPython does can be >> done in C. It's not like the CPython project used a completely unusual >> way of writing C code. > > CPython is a relatively simple interpreter, and executing code > by invoking such an interpreter IMHO doesn't count as "compiling" it in > any meaningful way. Oh, CPython is substantially more than an interpreter. The eval loop is only *one* way to use the runtime environment. Remember that it has many builtin types and functions as well as a huge standard library. Much of the runtime environment is already written in C or can be compiled down to C. If you compile Python code into C code that avoids the eval loop and only uses the CPython runtime environment (which is what Cython does), I think that qualifies as compiling Python code to C. It's definitely the most practical and user friendly way to do it. >> You will always need some kind of runtime infrastructure when you >> "compile Python into C", so you can just as well use CPython for that >> instead of reimplementing it completely from scratch. > > Maybe there's parts of Cpython you can re-use, but having the CPython > interpreter be the execution engine for "compiled" Python generators > again fails the seriousness test of what it means to compile code. If > you mean something other than that, you might explain more clearly. See above. >> Both Cython and Nuitka do exactly that, > > I didn't know about Nuitka; it looks interesting but (at least after a > few minutes looking) I don't have much sense of how it works. It's mostly like Cython but without the type system, i.e. without all the stuff that makes it useful in real life. Just a bare Python-to-C++-in-CPython compiler, without much of a way to make it do what you want. >> Last I heard, PyPy had a couple of GCs to choose from, > > PyPy doesn't compile to C RPython (usually) does, though, and my guess is that the memory management part of the runtime is written in RPython. > but I guess compiling to C doesn't preclude > precise GC, as long as the generated C code carefully tracks what C > objects can contain GC-able pointers, and follows some constraints about > when the GC can run. Some other compilers do this so it's not as big a > deal as it sounded like at first. OK. Yep, C really becomes a lot nicer when you generate it. >> Huh? LuaJIT is a reimplementation of Lua that uses an optimising JIT >> compiler specifically for Lua code. How is that similar to the Jython >> runtime that runs *on top of* the JVM with its generic byte code based >> JIT compiler? > > I thought LuaJIT compiles the existing Lua VM code, but I haven't > looked at it closely or used it. Ok. It obviously reuses code, but the VM part of it is really different from standard Lua. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Paul Rubin, 05.08.2012 03:38: > Steven D'Aprano writes: >> Runtime optimizations that target the common case, but fall back to >> unoptimized code in the rare cases that the optimization doesn't apply, >> offer the opportunity of big speedups for most code at the cost of >> trivial slowdowns when you do something unusual. > > The problem is you can't always tell if the unusual case is being > exercised without an expensive dynamic check, which in some cases must > be repeated in every iteration of a critical inner loop, even though it > turns out that the program never actually uses the unusual case. Cython does a lot of optimistic optimisations. That's where a large part of that huge C file comes from that Cython generates from even simple Python code. For example, in CPython, C function calls are so ridiculously faster than Python function calls that it's worth some effort if it saves you from packing an argument tuple to call into a Python function. In fact, we've been thinking about ways to export C signatures from Python function objects, so that code implemented in C (or a C compatible language) can be called directly from other code implemented in C. That's very common in the CPython ecosystem. There are a lot of simple things that quickly add up into a much better performance on average. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Jürgen A. Erhard, 05.08.2012 01:25: > On Sat, Aug 04, 2012 at 08:40:16AM +0200, Stefan Behnel wrote: >> Steven D'Aprano, 04.08.2012 08:15: >>> Most people are aware, if only vaguely, of the big Four Python >>> implementations: >> >> And not to forget Cython, which is the only static Python compiler that is >> widely used. Compiles and optimises Python to C code that uses the CPython >> runtime and allows for easy manual optimisations to get C-like performance >> out of it. > > Cython is certainly *not* a Python *implementation*, since it always > uses the CPython runtime (and compiling Cython C files requires > Python.h). Yes, it avoids an unnecessary duplication of effort as well as a substantial loss of compatibility that all non-CPython based implementations suffer from. You'd be surprised to see how much of Python we implement, though, including some of the builtins. You might want to revise your opinion once you start digging into it. It's always easy to disagree at the surface. > None of the other implementations require Python for actually > compiling or running Python source. Nuitka was on the list as well. > Oh, yes, you can create a stand-alone... wait, a "stand-alone" app. > By embedding the Python runtime (dynamic linking with libpythonX.Y... > maybe static too? Sure, that works. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Stefan Behnel, 05.08.2012 07:46: > Jürgen A. Erhard, 05.08.2012 01:25: >> None of the other implementations require Python for actually >> compiling or running Python source. > > Nuitka was on the list as well. Oh, and Stackless was also on Steven's list, as well as WPython. That means that 50% of the "other implementations" that Steven presented are not "implementations" according to your apparent definition. BTW, what is you definition? Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
alex23, 06.08.2012 05:40: > On Aug 4, 4:15 pm, Steven D'Aprano [email protected]> wrote: >> But the Python ecosystem is a lot bigger than just those four. Here are >> just a few other implementations that you might be interested in: > > There's also HotPy: > > http://code.google.com/p/hotpy/ > http://www.hotpy.org/ And just in case anyone was wondering where the others are: http://wiki.python.org/moin/PythonImplementations Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Jürgen A. Erhard, 05.08.2012 14:28: > On Sun, Aug 05, 2012 at 07:46:59AM +0200, Stefan Behnel wrote: >> Jürgen A. Erhard, 05.08.2012 01:25: >>> None of the other implementations require Python for actually >>> compiling or running Python source. >> >> Nuitka was on the list as well. > > True, which I realized only after my missive. But doesn't change > much, only that the list is wrong. Agreed. > My definition, to also answer your following post, is "does not rely > on any executable part of the CPython source (which includes .c files > and executable code in header files if any, but of course can exclude > the stdlib)". Not sure that's precise enough, but... if it can't > run/work on a system that has no shred of CPython installed, it's not > an alternative *implementation*. I can live with that definition. Cython is (by design) not an independent reimplementation of Python. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Alternate Python extensions (was alternate Python implementations)
rusi, 07.08.2012 06:23: > On Aug 4, 11:15 am, Steven D'Aprano [email protected]> wrote: >> Most people are aware, if only vaguely, of the big Four Python >> implementations: > > I think the question about where Cython fits into this, raises the > need for a complementary list to Steven's. What are the different > ways in which python can be extended/embedded. eg > > 1. 'Classic' extending/embedding > 2. SCXX > 3. PyCXX > 4. Boost > 5. Cython > 6. Swig > 7. Sip > 8. ctypes > > Is such a list maintained somewhere? Hijacking this page would be a good place to start it IMHO: http://wiki.python.org/moin/Embedding%20and%20Extending Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: [c-api]Transmutation of an extension object into a read-only buffer adding an integer in-place.
Giacomo Alzetta, 10.08.2012 10:20: > I'm trying to implement a c-extension which defines a new class(ModPolynomial > on the python side, ModPoly on the C-side). > At the moment I'm writing the in-place addition, but I get a *really* strange > behaviour. You should take a look at Cython. It makes these things way easier and safer than with manually written C code. It will save you a lot of code, debugging and general hassle. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: [c-api]Transmutation of an extension object into a read-only buffer adding an integer in-place.
Giacomo Alzetta, 11.08.2012 08:21: > I'd prefer to stick to Python and C, without having to put cython > sources or cython-generated c modules (which I know are almost > completely unreadable from a human point of view. Or at least the ones I > saw). And the cool thing is: you don't have to read them. :) Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: [c-api]Transmutation of an extension object into a read-only buffer adding an integer in-place.
Giacomo Alzetta, 11.08.2012 10:55: > Il giorno sabato 11 agosto 2012 08:40:18 UTC+2, Stefan Behnel ha scritto: >> Giacomo Alzetta, 11.08.2012 08:21: >> >>> I'd prefer to stick to Python and C, without having to put cython >>> sources or cython-generated c modules (which I know are almost >>> completely unreadable from a human point of view. Or at least the ones I >>> saw). >> >> And the cool thing is: you don't have to read them. :) > > Yes, but since all this code will end-up in the hands of some examiner, he'll > have to read them. :) I'd just ask him what he prefers: beautiful Python code with a couple of static type declarations in it, or verbose C code with lots of C-isms and C-API-isms all over the place. If he's smart enough, he'll force you into writing the code in Cython. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Help needed installing easy_install and lxml2
Ian, 15.08.2012 21:39: > On a reasonably fresh (3 day old) install of 64 bit windows 7, I have > installed the 64 bit Python 2.7.3 from http://www.python.org/download/ > > Then I installed the 64 bit version of easy_install using > http://pypi.python.org/pypi/setuptools > > And then install lxml 2.3.5, for which I use the instructions at > http://lxml.de/installation.html#installation > > This ends with: > Processing lxml-2.3.5.tgz > Running lxml-2.3.5\setup.py -q bdist_egg --dist-dir > c:\users\ian\appdata\local\temp\easy_install-9__rq7\lxml-2.3.5\egg-dist-tmp-uj_v_2 > > Building lxml version 2.3.5. > Building without Cython. > ERROR: 'xslt-config' is not recognized as an internal or external command, > operable program or batch file. > > ** make sure the development packages of libxml2 and libxslt are installed ** > > Using build configuration of libxslt > error: Setup script exited with error: Unable to find vcvarsall.bat > > C:\Users\ian> > > Now what? I thought I was installing the binary package do I would not need > any development versions. > > What do these errors mean, and how can I overcome them? http://lxml.de/FAQ.html#where-are-the-binary-builds Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: type(None)()
Steven D'Aprano, 16.08.2012 15:58: >>> NoneType raises an error if you try to create a second instance. > In my opinion, this is a PITA for None and better behaviour would be to > return the pre-existing NoneType instance, but I didn't design the > language. The time machine strikes again. Python 3.3.0b1 (default:f7b59e890e30, Aug 11 2012, 05:30:10) [GCC 4.6.3] on linux Type "help", "copyright", "credits" or "license" for more information. >>> type(None)() >>> print(type(None)()) None Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: type(None)()
Ramchandra Apte, 16.08.2012 17:39: > On 16 August 2012 21:01, Stefan Behnel wrote: >> Steven D'Aprano, 16.08.2012 15:58: >>>>> NoneType raises an error if you try to create a second instance. >>> In my opinion, this is a PITA for None and better behaviour would be to >>> return the pre-existing NoneType instance, but I didn't design the >>> language. >> >> The time machine strikes again. >> >> Python 3.3.0b1 (default:f7b59e890e30, Aug 11 2012, 05:30:10) >> [GCC 4.6.3] on linux >> Type "help", "copyright", "credits" or "license" for more information. >>>>> type(None)() >>>>> print(type(None)()) >> None > > Are they the same object Obviously. None is a singleton (as was already mentioned a couple of times in this thread). Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: python+libxml2+scrapy AttributeError: 'module' object has no attribute 'HTML_PARSE_RECOVER'
Dmitry Arsentiev, 15.08.2012 14:49: > Has anybody already meet the problem like this? - > AttributeError: 'module' object has no attribute 'HTML_PARSE_RECOVER' > > When I run scrapy, I get > > File "/usr/local/lib/python2.7/site-packages/scrapy/selector/factories.py", > line 14, in > libxml2.HTML_PARSE_NOERROR + \ > AttributeError: 'module' object has no attribute 'HTML_PARSE_RECOVER' > > > When I run > python -c 'import libxml2; libxml2.HTML_PARSE_RECOVER' > > I get > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'module' object has no attribute 'HTML_PARSE_RECOVER' > > How can I cure it? > > Python 2.7 > libxml2-python 2.6.9 > 2.6.11-gentoo-r6 That version of libxml2 is way too old and doesn't support parsing real-world HTML. IIRC, that started with 2.6.21 and got improved a bit after that. Get a 2.8.0 installation, as someone pointed out already. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: XML parser: Element ordering?
Florian Lindner, 31.08.2012 14:03: > I plan to use the etree.ElementTree XML parser to parse a config file > in which the order of the elements matter, e.g.: > > > > > > is not equal to: > > > > > > I have found different answers to the question if order matters in XML > documents. So my question here: Does it matters (and is more or less > guarenteed to matter in the future) for the ElementTree parser of > python? It matters for XML documents, so, yes, any XML parser will definitely honour the document order (which is actually well defined for an XML document). What you might be referring to is the case of a specific XML document format, where the application (or sometimes specification) can explicitly state that the element order in a given subtree has no meaning for the semantics of the element and that therefore code must ignore the order in which elements occur. But that won't magically make the parser ignore it for you. That's a totally different level of abstraction and a deliberate decision of the code that *uses* the parser. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Cython 0.17 released
Hello everyone, on behalf of the Cython project team, I'm proud to announce the final release of Cython 0.17. This is a major step forward in the development of the Cython programming language that will make life easier for a lot of users, rounds up some rough edges of the compiler and adds (preliminary) support for CPython 3.3 and PyPy. With this release, the Cython compiler successfully passes over 14,000 regression tests of the CPython 3.3 test suite and almost 13,000 tests of the CPython 2.7 suite, with only some 300 and 200 failures respectively. This makes it a serious alternative to interpreted Python execution that integrates natively with the complete CPython ecosystem. It is also the first final release of an implementation of PEP 380 (generator delegation), before it will eventually appear in CPython 3.3. === Download and Documentation === Download: http://cython.org/release/Cython-0.17.tar.gz Release notes: http://wiki.cython.org/ReleaseNotes-0.17 Homepage: http://cython.org/ Documentation: http://docs.cython.org/ === Major features of this release === * vastly improved integration with the C++ STL containers http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#standard-library http://docs.cython.org/src/tutorial/strings.html#c-strings * "yield from" delegation between generators (PEP 380) http://www.python.org/dev/peps/pep-0380/ * alpha quality support for PyPy (via cpyext) http://docs.cython.org/src/userguide/pypy.html === What is Cython ? === Cython is a language with an optimising compiler that makes writing C extensions for the Python language as easy as Python itself. The Cython language is a superset of the Python language that additionally supports calling C functions and declaring C types on variables and class attributes. This allows the compiler to generate very efficient C code from Cython code. The C code is generated once and then compiles with all major C/C++ compilers in CPython 2.4 and later, including Python 3.x. PyPy support is work in progress (on both sides) and is considered mostly usable in Cython 0.17. All of this makes Cython the ideal language for wrapping external C libraries, embedding CPython into existing applications, and for fast C modules that speed up the execution of Python code. Have fun, Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: py2c - an open source Python to C/C++ is looking for developers
Ramchandra Apte, 02.09.2012 08:10: >> That's reason enough for it. Do you plan to port all the standard > python modules as well, though? > > Well, it should be quite easy because most of the _modules are either C > accelerator (which there is no need to port) or a C wrapper (which should be > trivial to port) Nope, not at all. They use the CPython C-API internally, so in order to port them, you'll have to reimplement that first. That's a huge amount of work, as proven by the incompleteness of all other Python implementations in that regard. If you think you can do better here then IronPython or PyPy, please go ahead. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: running Lua in Python
Arnaud Delobelle, 02.09.2012 11:04: > I'm looking for a way to run Lua scripts in Python, and also send and > receive data between the two. Something like lunatic-python [1] would > be ideal. However, so far I haven't been able to build it on the > machines it's supposed to run on (macs with OS X Lion) and it seems to > be dormant. > > My requirements are stock OS X Python (which is 2.7) and Lua 5.2. I'm > looking for either a way to make lunatic-python work or another tool > that would do the job. Try Lupa: http://pypi.python.org/pypi/lupa Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: running Lua in Python
Arnaud Delobelle, 02.09.2012 20:34: > On 2 September 2012 10:49, Arnaud Delobelle wrote: >> On 2 September 2012 10:39, Alec Taylor wrote: >>> http://pypi.python.org/pypi/lupa >> >> I'll check this out, thanks. > > Mmh it seems to be lua 5.1 and more importantly it seems to require a > custom build of Python, which I don't want. I have no idea why you say that, for neither of the two. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: running Lua in Python
Arnaud Delobelle, 02.09.2012 21:17: > On 2 September 2012 19:42, Stefan Behnel wrote: >> Arnaud Delobelle, 02.09.2012 20:34: >>> On 2 September 2012 10:49, Arnaud Delobelle wrote: >>>> On 2 September 2012 10:39, Alec Taylor wrote: >>>>> http://pypi.python.org/pypi/lupa >>>> >>>> I'll check this out, thanks. >>> >>> Mmh it seems to be lua 5.1 and more importantly it seems to require a >>> custom build of Python, which I don't want. >> >> I have no idea why you say that, for neither of the two. > > * For the custom build of Python on OS X, this blog post: > > > http://t-p-j.blogspot.co.uk/2010/11/lupa-on-os-x-with-macports-python-26.html > > which is actually linked to in lupa's own INSTALL.rst file Ah, ok, but that's a) a rather old blog post and it b) refers to LuaJIT, not plain Lua. Just because LuaJIT requires special build flags doesn't mean that Lupa or Lua do. > * For Lua 5.1, the official LuaJIT page: > > http://luajit.org/luajit.html > > where I quote: > > """ > LuaJIT implements the full set of language features defined by Lua 5.1. > """ > > Of course I'd be glad to be proved wrong on both counts. I was told that it works with Lua 5.1 and didn't try it with Lua 5.2 myself. But I think you should, if that's what you want to use. Stefan -- http://mail.python.org/mailman/listinfo/python-list
RPython, static type annotations, SafePython project (was: PhD Python & Smalltalk grant possibility (Lille/INRIA))
Mariano Reingart, 05.09.2012 18:53, on python-announce-list: > Context: Dynamically-typed languages cannot take advantage of static > type information. In this context we would like to study > the benefit of the introduction of static types *annotations* on > library design and general robustness. Works to a certain extent, but won't get you very far. You'll notice that static types don't have all that rich semantics, so you'll eventually end up wanting to extend the type declarations into contract specifications, and then you'll end up duplicating half of your code to describe the interface at some point. Has been done, IMHO not worth doing. > The benefits can be at the level of the robustness (bug > identification), but also tools (IDE) and support for assembly > generation/c in case of JIT. As the PyPy people keep reiterating, JITs (read: their JIT) don't need static type declarations. IDEs can benefit from them a bit (not much, just a bit - guessing usually works quite well here), but I'd hate to change my code only to make my IDE happy. Static compilers obviously benefit from them, so you should take a look at Cython if you are interested in a Python based type system with optional static type declarations and how people use them in real code. (There's definitely a lot more Cython code out there than RPython code...) > The PhD grant is financed in the context of the Safe Python project > therefore the Ph.D. will have to work in contact > do we with the project partners and help in the context of some deliverables. > > The following tasks have to be done: > > - RPython is not formally defined. One of the first task will > be to define the formal semantic of RPython. Just to be sure: you are talking about RPython, the language that PyPy is (mostly) written in, right? Why would you want to restrict yourself to that instead of looking at general Python code? Is that due to those who pay the project or because it's really of interest all by itself? > One approach is to write a RPython interpreter based on an > operational semantics and to compare the output > with the RPython interpreter. We will certainly use PLT redex > which is a domain-specific language to specify > and debug operational semantics. Sounds like a research-only project to me. From what I hear, RPython isn't really an interesting programming language. It's very restrictive and easy to get wrong. That might be related to the fact that the only definition of the language is the only existing (incomplete) implementation, but maybe not. (Sounds like some research could help to decide that at least). There's also ShedSkin, which, I believe, has its own definition of an "RPython", again based on what it understands and can statically analyse. > - We may use the abstract syntax tree of PyLint and use the > Frama-C infrastructure. We will also consider to > build a Python parser based on petitParser. My advice: skip the parser part, use the existing AST that Python provides. That's the only sane way to integrate with other tools that process Python code. > - Define of some default metrics may also be necessary for the > SafePython project. Their definition should > be trivial on ASTs. Not sure what you mean with "default metrics" here. > - Analysis of the benefits of static typing for RPython. One > idea is to study the existing python libraries > and analyze the "distance" to the RPython subset. That might be an interesting study in its own right: how static is real-world Python code? > - Exploring type checking in presence of inconsistent type > annotations. Yep, that's only one of the problems you'll run into. > - Since we are developing Pharo Which, if I searched correcly, is a Smalltalk IDE? > and that static type annotation are important to > support C or assembly generation, we would like to apply the > same technique to Pharo: > - define a syntax to support type annotation (reuse > the one developed by Pleaid team member) > - perform some analysis of existing library. I don't know Smalltalk well enough to comment on this. > About Lille and INRIA: > > Lille is located in the north of france at the border to Belgium > one hour from Paris, 1h20 from London, 35 min from Brussels, by train. And the most important thing (which you forgot to mention): a nice place to live. > French food, combined with belgian beer. Well, yes, but it's still a nice place to live. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Fastest web framework
Roy Smith, 23.09.2012 16:02: > Andriy Kornatskyy wrote: >> I have run recently a benchmark of a trivial 'hello world' application for >> various python web frameworks (bottle,�django, flask, pyramid, web.py, >> wheezy.web) hosted in uWSGI/cpython2.7 and gunicorn/pypy1.9... you might >> find >> it interesting: >> >> http://mindref.blogspot.com/2012/09/python-fastest-web-framework.html >> >> Comments or suggestions are welcome. > > That's a nice comparison, thanks for posting it. > > One thing that's worth pointing out, however, is that in a real world > application, as long as you're using something halfway decent, the speed > of the framework is probably not going to matter at all. It's much more > likely that database throughput will be the dominating factor. Yes, that makes the comparison (which may or may not be biased towards his own engine) a bit less interesting. Worth keeping this in mind: http://www.codeirony.com/?p=9 Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Fastest web framework
Andriy Kornatskyy, 23.09.2012 19:42: > If we take a look at web application we can split it into at least two > parts, one that renders things out and the other one that does data > extraction, e.g. from database (this is what you are pointing at). > > If you made a first call to database you get your list and can easily > cache it. The next call IS without impact that database call may > cause... but you still keep serving pages out... Well, if it was really that easy, you wouldn't be using a database in the first place but static pages, would you? Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pass FILE *
xDog Walker, 29.09.2012 10:45: > On Friday 2012 September 28 21:27, you wrote: >> A tiny bit of googling suggests the following approach: >> http://stackoverflow.com/questions/3794309/python-ctypes-python-file-object >> -c-file/3794401#3794401 > > Thanks for your response. > > My "tiny bit of Googling" brought no joy but I did try successfully the same > method suggested by Alex Martelli: use libc's fdopen to get a FILE * for > ctypes. I would have posted a reply to my own post but it didn't show up. Given your second post on stackoverflow, you may also consider switching from ctypes to Cython. It makes these things a bit more straight forward. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: [RELEASED] Python 3.3.0
Georg Brandl, 29.09.2012 14:18:
> On behalf of the Python development team, I'm delighted to announce the
> Python 3.3.0 final release.
> [...]
> * PEP 380, syntax for delegating to a subgenerator ("yield from")
Ah, you're so late! Cython has shipped its implementation almost a month
ago! ;)
Stefan
--
http://mail.python.org/mailman/listinfo/python-list
Re: [RELEASED] Python 3.3.0
Christian Heimes, 29.09.2012 16:06: > From now on you can't rely > on the order of an unordered type like dict or set. Tautologies tend to be true even without a temporal qualification. Stefan -- http://mail.python.org/mailman/listinfo/python-list
lxml 3.0 final released - efficient XML and HTML processing with Python
Hi everyone, it's been a while since the last stable release series appeared, so I'm proud to announce the final release of lxml 3.0. http://lxml.de/ http://pypi.python.org/pypi/lxml/3.0 Changelog: http://lxml.de/changes-3.0.html In short, lxml is the most feature-rich and easy-to-use library for processing XML and HTML in the Python language. It's also very fast, in case you're interested in that. It runs on Python 2.4 - 3.3. The 3.0 release features initial support for PyPy and a new tree API for DTDs (by Walter Dörwald), as well as some major cleanups in the code base. The lxml.cssselect package was moved into a separate PyPI project nicely extended and maintained by Simon Sapin. If you are interested in commercial support or customisations for the lxml package, please contact me directly. Have fun, Stefan -- http://mail.python.org/mailman/listinfo/python-list
