Re: regular expresions and dolar sign ($) in source string

[Peter Otten]

Jax wrote:

> I encountered problem with dolar sign in source string. It seems that $
> require special threatening. Below is copy of session with interactive
> Python's shell:
> 
> Python 2.5.2 (r252:60911, Jan  8 2009, 12:17:37)
> [GCC 4.3.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
 import re
 a = unicode(r"(instead of $399.99)", "utf8")
 print re.search(unicode(r"^\(instead of.*(\d+[.]\d+)\)$", "utf8"),
> a).group(1)
> 9.99
 print re.search(unicode(r"^\(.*(\d+[.]\d+)\)$", "utf8"), a).group(1)
> 9.99
 print re.search(unicode(r"^\(.*\$(\d+[.]\d+)\)$", "utf8"), a).group(1)
> 399.99
> 
> My question is: Why only third regular expression is correct?

They are all correct, they just don't give what you expect. This has nothing
to do with the $. The ".*" expression is "greedy", it tries to match as
many characters as possible. You can see that by adding another group:

>>> a = u"(instead of $399.99)"
>>> re.search(ur"^\(instead of(.*)(\d+[.]\d+)\)$", a).groups()
(u' $39', u'9.99')

Fortunately there is also a non-greedy variant ".*?" which matches as few
characters as possible:

>>> a = u"(instead of $399.99)"
>>> re.search(ur"^\(instead of.*?(\d+[.]\d+)\)$", a).group(1)
u'399.99'

Peter
--
http://mail.python.org/mailman/listinfo/python-list

Re: python command not working

[83nini]

On 23 Apr, 02:25, Dave Angel  wrote:
> 83nini wrote:
> > Christian,
> > at last i made the bat file (python25.bat) that contains the following
> > contents:
>
> > ---
> > @C:\Python25\python.exe %*
> > ---
>
> > could you tell me how to call python25 in the batch files? what batch
> > files do you mean? where do i find them?
>
> Your original question was how to get an interactive interpreter from a
> command prompt.  There are at least 3 ways:
>
> 1) just specify the full path to the executable on the command line
>      e:\mysource\>  c:\python25\python.exe
>
> 2) add c:\python25  to your PATH, either by using the SET statement, or
> control panel.
>      then you'll be able to just use:
>      e:\mysource\>  python
>
> 3) create a batch file in a directory on your PATH
>
> #3 is probably the best way.  You've created that batch file, now just
> use it.
>     e:\mysource\>  python25
>
> The remark about using "call python25" refers only to the case where you
> write additional batch files that use this one.  Since you're not likely
> to, don't worry about it yet.
>
> If typing python25 in a command box doesn't work, then you don't have
> that file on the PATH.  If that's the case, tell us exactly where the
> file is.

Thanks a million, it worked. :D
--
http://mail.python.org/mailman/listinfo/python-list

Re: Memory problems (garbage collection)

[Gerhard Häring]

Carbon Man wrote:
> Very new to Python, running 2.5 on windows.
> I am processing an XML file (7.2MB). Using the standard library I am 
> recursively processing each node and parsing it. The branches don't go 
> particularly deep. What is happening is that the program is running really 
> really slowly, so slow that even running it over night, it still doesn't 
> finish.
> Stepping through it I have noticed that memory usage has shot up from 190MB 
> to 624MB and continues to climb. 

That sounds indeed like a problem in the code. But even if the XML file
is only 7.2 MB the XML structures and what you create out of them have
some overhead.

> If I set a break point and then stop the 
> program the memory is not released. It is not until I shutdown PythonWin 
> that the memory gets released.

Then you're apparently looking at VSIZE or whatever it's called on
Windows. It's the maximum memory the process ever allocated. And this
usually *never* decreases, no matter what the application (Python or
otherwise).

> [GC experiments]

Unless you have circular references, in my experience automatic garbage
collection in Python works fine. I never had to mess with it myself in
10 years of Python usage.

> If I have the program at a break and do gc.collect() it doesn't fix it, so 
> whatever referencing is causing problems is still active.
> My program is parsing the XML and generating a Python program for 
> SQLalchemy, but the program never gets a chance to run the memory problem is 
> prior to that. It probably has something to do with the way I am string 
> building.

Yes, you're apparently concatenating strings. A lot. Don't do that. At
least not this way:

s = ""
s += "something"
s += "else"

instead do this:

from cStringIO import StringIO

s = StringIO()
s.write("something")
s.write("else")
...
s.seek(0)
print s.read()

or

lst = []
lst.append("something")
lst.append("else")
print "".join(lst)


> My apologies for the long post but without being able to see the code I 
> doubt anyone can give me a solid answer so here it goes (sorry for the lack 
> of comments): [...]

Code snipped.

Two tips: Use one of the above methods for concatenating strings. This
is a common problem in Python (and other languages, Java and C# also
have StringBuilder classes because of this).

If you want to speed up your XML processing, use the ElementTree module
in the standard library. It's a lot easier to use and also faster than
what you're using currently. A bonus is it can be swapped out for the
even faster lxml module (externally available, not in the standard
library) by changing a single import for another noticable performance
improvement.

HTH

-- Gerhard

--
http://mail.python.org/mailman/listinfo/python-list

Re: Memory problems (garbage collection)

[Gerhard Häring]

Here's a link for you:

http://wiki.python.org/moin/PythonSpeed/PerformanceTips

which also talks about string concatenation and othere do's and don'ts.

-- Gerhard

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Packages : A looming problem? packages might no longer work? (well not on your platform or python version anyway)

[Steven D'Aprano]

On Wed, 22 Apr 2009 23:47:08 -0700, Daniel Fetchinson wrote:

 Why? Why should every package on PyPI need to support all those
 Python versions? That should be the decision of the package
 maintainer. If they want to support every version of Python back to
 1.0, they can, and if they want to only support version 2.5 that's
 fine too.
>>>
>>> Why shouldn't packages support more than one python version?
>>
>> They can. That depends on the developer of the package.
> 
> And if the developer decides to support multiple versions currently
> there is no infrastructural help that would be available to him. On the
> other hand, there is infrastructural help available to every developer
> for distributing their apps: it's PyPI itself. You could ask, why do we
> need PyPI? Let the developers distribute their code in any way they
> like, python.org doesn't need to support this.

They don't *need* to support distribution. That they do is a nice-to-
have, not a must-have -- there are other solutions, like SourceForge. We 
can be grateful for PyPI without believing that it's a must-have.


> The OP is just thinking out loud that it would be great if developers
> could count on some help for testing various platforms and versions. And
> I agree, it would indeed be great.

To me, the OP doesn't seem to be asking "Wouldn't it be good if 
developers could get access to whatever systems they wanted?" and more 
like "Somebody needs to develop this amazing technology that will just 
magically make every package on PyPI work on all these different 
platforms, otherwise the Sky Will Fall". Just look at the subject line: 
the issue is described as "a looming problem", it's suggested that 
packages "might no longer work".

He's also worried about the combinational explosion of packages, 
platforms and versions, which to my mind is only a problem if you think 
that every package must work on every platform with every version of 
Python.

I suspect you're primed to read the OP's suggestion in the most positive 
light, because you're already aware of snakebite, and so you're seeing 
his posts in terms of what snakebite is offering. I had never even heard 
of snakebite before now, so I was seeing it in a very different light.


[...]
 What's the dire problem you are trying to solve?
>>>
>>> Backward and forward compatability of python package resources.
>>
>> That's not a problem, that's a feature.
>>
>> What's the *problem* that you are trying to solve? What bad thing will
>> happen if we don't build your proposed system?
> 
> I fail to understand what is so mysterious about this. The problem is
> the following: developer X wants to support python versions a, b, c, d
> on platforms U, V, W. Developer X has no access to platforms V and W and
> also has no access to python versions b, c and d. Developer X can not
> run tests on these platforms and python versions although he really
> wants to. This situation is what is commonly referred to as a problem.

That's *a* problem. It has many solutions, snakebite being just one. Is 
it the "looming" problem the OP is concerned about?


> This testing infrastructure should be available to developers who choose
> to opt in. Once they run their code and notice that there are errors,
> they will go back and fix these errors that otherwise would be
> undetected until a real person runs into them. Or they could clearly
> state what platform and python versions the code runs on for sure and on
> what platforms and python versions the code fails. This would be a great
> advantage to developers.

It certainly would. 

But the opt-in arrangement you are talking about doesn't sound much like 
the OP's suggestion that some cross-platform build-bot goes out and pulls 
in every single package on PyPI and tests them all on every combination 
of platforms and versions.


> Actually, the OP's suggestion is a really good one, the people behind
> snakebite.org realized this already some time ago. To me the idea is
> very clear and obviously would make the life of developers a whole lot
> easier and consequently the life of users as well.

Snakebite certainly sounds beneficial for development of Python. I'm not 
sure that it would scale to every developer with a package on PyPI, but 
maybe someday.



-- 
Steven
--
http://mail.python.org/mailman/listinfo/python-list

Supply a plugin interface

[Johannes Bauer]

Hello group,

I'm developing a GUI application in Python and having a blast so far :-)

What I'd like to add: I want the GUI users to supply plugin scripts,
i.e. offer some kind of API. That is, I want the user to write short
Python pieces which look something like

import guiapp

class myplugin():
def __init__(self):
guiapp.add_menu("foobar")

def exec(self, param):
print("foo")

the GUI application should now browse the plugin directory and read
those plugin python files and somehow incorporate (i.e. discover what
modules are there, instanciate, etc.)

How do I do that at runtime with Python?

Kind regards,
Johannes

-- 
"Meine Gegenklage gegen dich lautet dann auf bewusste Verlogenheit,
verlästerung von Gott, Bibel und mir und bewusster Blasphemie."
 -- Prophet und Visionär Hans Joss aka HJP in de.sci.physik
 <48d8bf1d$0$7510$54022...@news.sunrise.ch>
--
http://mail.python.org/mailman/listinfo/python-list

a py2exe feature for non-windows environments

[Tzury Bar Yochay]

Hi,

The nicest thing I like about py2exe is its library.zip which
encapsulate all the dependencies into one single file.
I wonder if there a script which can do the same for linux/mac osx so
one can ship a python solution as a single file
(instead of starting easy_install per used library on the target
machine).
--
http://mail.python.org/mailman/listinfo/python-list

Re: a py2exe feature for non-windows environments

[Chris Rebert]

On Thu, Apr 23, 2009 at 1:13 AM, Tzury Bar Yochay
 wrote:
> Hi,
>
> The nicest thing I like about py2exe is its library.zip which
> encapsulate all the dependencies into one single file.
> I wonder if there a script which can do the same for linux/mac osx so
> one can ship a python solution as a single file
> (instead of starting easy_install per used library on the target
> machine).

Mac OS X: http://svn.pythonmac.org/py2app/py2app/trunk/doc/index.html

Cheers,
Chris
-- 
I have a blog:
http://blog.rebertia.com
--
http://mail.python.org/mailman/listinfo/python-list

Re: Memory problems (garbage collection)

[Peter Otten]

Carbon Man wrote:

> Very new to Python, running 2.5 on windows.
> I am processing an XML file (7.2MB). Using the standard library I am
> recursively processing each node and parsing it. The branches don't go
> particularly deep. What is happening is that the program is running really
> really slowly, so slow that even running it over night, it still doesn't
> finish.
> Stepping through it I have noticed that memory usage has shot up from
> 190MB to 624MB and continues to climb. If I set a break point and then
> stop the program the memory is not released. It is not until I shutdown
> PythonWin that the memory gets released.
> I thought this might mean objects were not getting GCed, so through the
> interactive window I imported gc. gc.garbage is empty. gc.collect() seems
> to fix the problem (after much thinking) and reports 2524104. Running it
> again returns 0.
> I thought that garbage collection was automatic, if I use variables in a
> method do I have to del them?

No. Deleting a local variable only decreases the reference count. In your
code the next iteration of the for loop or returning from the method have
the same effect and occur directly after your del statements.

> I tried putting a "del node" in all my for node in  loops but that
> didn't help. collect() reports the same number. Tried putting gc.collect()
> at the end of the loops but that didn't help either.
> If I have the program at a break and do gc.collect() it doesn't fix it, so
> whatever referencing is causing problems is still active.
> My program is parsing the XML and generating a Python program for
> SQLalchemy, but the program never gets a chance to run the memory problem
> is prior to that. It probably has something to do with the way I am string
> building.
> 
> My apologies for the long post but without being able to see the code I
> doubt anyone can give me a solid answer so here it goes (sorry for the
> lack of comments):

First, use a small xml file to check if your program terminates and operates
correctly. Then try disabling cyclic garbage collection with gc.disable().
Remove the gc.collect() calls. 
 
This will not help with the memory footprint, but sometimes when you are
creating many new objects that you want to keep Python spends a lot of time
in vain looking for unreachable objects -- so there may be a speedup.

> from xml.dom import minidom
> import os
> import gc

gc.disable()

[snip more code]

Does this improve things?

Like Gerhard says, in the long run you are probably better off with
ElementTree.

Peter
--
http://mail.python.org/mailman/listinfo/python-list

Re: a py2exe feature for non-windows environments

[Tzury Bar Yochay]

> Mac OS X:http://svn.pythonmac.org/py2app/py2app/trunk/doc/index.html

Found these for linux:

http://www.pyinstaller.org/
http://wiki.python.org/moin/Freeze

thanks alot
--
http://mail.python.org/mailman/listinfo/python-list

Re: Ending data exchange through multiprocessing pipe

[Michal Chruszcz]

On Apr 22, 6:33 pm, MRAB  wrote:
> You could do this:
>
>      from multiprocessing.queues import Empty
>
>      queue = Queue()
>      Process(target=f, args=(queue,)).start()
>      while active_children():
>          try:
>              print queue.get(timeout=1)
>          except Empty:
>              pass

This one isn't generic. When I have tasks that all finish within 0.1
seconds the above gives 10x overhead. On the other hand, if I know the
results will be available after 10 hours there's no use in checking
every second.

Best regards,
Michal Chruszcz
--
http://mail.python.org/mailman/listinfo/python-list

Re: Supply a plugin interface

[Ulrich Eckhardt]

Johannes Bauer wrote:
> What I'd like to add: I want the GUI users to supply plugin scripts,
> i.e. offer some kind of API. That is, I want the user to write short
> Python pieces which look something like
> 
> import guiapp
> 
> class myplugin():
> def __init__(self):
> guiapp.add_menu("foobar")
> 
> def exec(self, param):
> print("foo")
> 
> the GUI application should now browse the plugin directory and read
> those plugin python files and somehow incorporate (i.e. discover what
> modules are there, instanciate, etc.)

You will find ways to scan a directory using os.path and related things. In
order to import a module, you add the base directory to sys.path. Since you
have just a variable with the module name (excluding the .py), you can't
use 'import' as it is. Instead, use the __import__ functio (I wonder if
there is anything more elegant) which returns the plugin.

Other than that, better steal ideas than reinventing them. I think e.g.
xchat already provides ways to script the UI, which you could take as an
inspiration.

Have fun!

Uli

-- 
Sator Laser GmbH
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932

--
http://mail.python.org/mailman/listinfo/python-list

Re: Supply a plugin interface

[Steven D'Aprano]

On Thu, 23 Apr 2009 09:47:56 +0200, Johannes Bauer wrote:

> Hello group,
> 
> I'm developing a GUI application in Python and having a blast so far :-)
> 
> What I'd like to add: I want the GUI users to supply plugin scripts,
> i.e. offer some kind of API. That is, I want the user to write short
> Python pieces which look something like
> 
> import guiapp
> 
> class myplugin():
>   def __init__(self):
>   guiapp.add_menu("foobar")
> 
>   def exec(self, param):
>   print("foo")

"exec" is a reserved word:

>>> def exec(self, param):
  File "", line 1
def exec(self, param):
   ^
SyntaxError: invalid syntax


> the GUI application should now browse the plugin directory and read
> those plugin python files and somehow incorporate (i.e. discover what
> modules are there, instanciate, etc.)
> 
> How do I do that at runtime with Python?

Untested:

import os
import sys
plugin_dir = os.path.expanduser('~/some/path/')
plugins = set()
for name in os.listdir(plugin_dir):
base, ext = os.path.splitext(name)
if ext in ('.py', '.pyc', '.pyo'):
plugins.add(base)
save_path = sys.path
plugin_modules = []
try:
sys.path = [plugin_dir]
for name in plugins:
plugin_modules.append(__import__(name))
finally:
sys.path = save_path




Hope this is useful.


-- 
Steven
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Packages : A looming problem?

[Lawrence D'Oliveiro]

In message , David 
Lyon wrote:

> There are now over 6,000 packages listed on PyPi - and this number can
> only get bigger.
> 
> Then, there are the three major operating systems:
> 
>  * Windows
>  * Mac
>  * Linix/Unix

Major Linux distros provide 20,000 or more packages each, without breaking a 
sweat. What's your big deal?

> To complicate the problem, there are now many versions of each operating
> system.

The Linux solution is to leave distro packaging to the distro maintainers. 
Release the source, and they will make up the necessary packages for their 
specific one-click installers. You don't have to worry about it.

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python, MS SQL, and batch inserts

[Lawrence D'Oliveiro]

In message , Dennis 
Lee Bieber wrote:

> You don't mean to say you are creating full SQL insert statements
> WITH THE DATA!

Nothing wrong with that 
.

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python, MS SQL, and batch inserts

[Lawrence D'Oliveiro]

In message , Philip 
Semanchuk wrote:

> Are you sure your logjam is in Python? Inserting 5500 rows can take a
> few seconds if you're COMMITting after each INSERT. Wrap the whole
> thing in an explicit transaction and see if that helps.

Does MS-SQL have the equivalent of "insert delayed" 
?

--
http://mail.python.org/mailman/listinfo/python-list

Reminder: Python Bug Day on Saturday

[Georg Brandl]

Hi,

I'd like to remind everyoone that there will be a Python Bug Day on
April 25.  As always, this is a perfect opportunity to get involved
in Python development, or bring your own issues to attention, discuss
them and (hopefully) resolve them together with the core developers.

We will coordinate over IRC, in #python-dev on irc.freenode.net,
and the Wiki page http://wiki.python.org/moin/PythonBugDay has all
important information and a short list of steps how to get set up.

Hope to see you there!

Georg


--
http://mail.python.org/mailman/listinfo/python-list

New fonts for python docs site

[Mark]

e.g. see http://docs.python.org/library/index.html

Please tell me this is a mistake? 3.X docs are the same.
--
http://mail.python.org/mailman/listinfo/python-list

Re: New fonts for python docs site

[Marco Mariani]

Mark wrote:


e.g. see http://docs.python.org/library/index.html

Please tell me this is a mistake? 3.X docs are the same.


Looks ok. What do you see?
--
http://mail.python.org/mailman/listinfo/python-list

Re: WINXP vs. LINUX in threading.Thread

[Kent]

Thanx you guys.

Now my program is working.

I used the Thread subclass. and at the end of the run method, i call
wx.CallAfter(mainFrame.somefunction, para) to show the dialog or
change some text. I tested in winxp&linux. both worked.


Kent

On Apr 23, 6:16 am, Carl Banks  wrote:
> On Apr 22, 5:34 pm, Lie Ryan  wrote:
>
>
>
> > Diez B. Roggisch wrote:
> > > Kent schrieb:
> > >> hello all,
>
> > >> i want to add a "new update notification" feature to my wxPython appl.
> > >> The codes below do the job. The logic is simple enough, I don't think
> > >> it needs to be explained.
>
> > >> since sometimes, under windows, proxy setting was a script. and was
> > >> set in IE. In this case, connecting to the HTML will take relative
> > >> long time. I therefore run the following codes in a new Thread
> > >> (subclass of threading.Thread), so that user don't have to wait during
> > >> the version checking period.
>
> > >> Under Linux, it worked smoothly. But under Windows XP, it didn't. If
> > >> there was new updates, the notification dialog can show, but no text,
> > >> icon,  on it. Then, the whole application didn't response any
> > >> longer. :( I have to force stop the application process.
>
> > >> where is the problem?
>
> > > GUI-toolkits and threads usually are not a good idea (Qt4 being an
> > > exception to that rule, at least they claim that). Google wxPython +
> > > threading for answers how to solve this - essentially, you need to
> > > create a timer or event-based solution that allows your
> > > background-thread to inject a status message to the main eventloop.
>
> > > Diez
>
> > You should use the GUI toolkit's own event loop for threading. An event
> > loop is some sort of cooperative multithreading mechanism,
>
> The function in question, gethostbyname, isn't cooperating, though.
> It blocks for a substantial amount of time from an event handler,
> freezing the application.
>
> > if you used
> > the threading mechanism from the threading library, there will be two


> > conflicting threading mechanism, resulting in many hard-to-predict
> > situation if you're not careful. The wxYield mechanism is much easier
> > option than multithreading.
>
> The problem is, during most of the delay wxYield can't be called
> becaust the function gethostbyname is blocking.
>
> I think Diez is correct.  To avoid the freeze, one should spawn a
> thread, and when it completes it should notify the GUI thread by
> pushing an event or scheduling an idle call.  Functions that do that
> are usually thread-safe.  (A permanent worker thread might be better
> but it would involve a lot more synchronization.)
>
> Carl Banks

--
http://mail.python.org/mailman/listinfo/python-list

Re: Supply a plugin interface

[Almar Klein]

Hi,

I did this a few times and put the code that loads the plugins in
the __init__.py of the plugin directory. You then do not have to
do the path stuff.

You can also make a rule that the class defined in each
plugin module should be a certain name, for example the same
name as the module (but starting with a capital to follow pep8).

Cheers,
  Almar


2009/4/23 Steven D'Aprano 

> On Thu, 23 Apr 2009 09:47:56 +0200, Johannes Bauer wrote:
>
> > Hello group,
> >
> > I'm developing a GUI application in Python and having a blast so far :-)
> >
> > What I'd like to add: I want the GUI users to supply plugin scripts,
> > i.e. offer some kind of API. That is, I want the user to write short
> > Python pieces which look something like
> >
> > import guiapp
> >
> > class myplugin():
> >   def __init__(self):
> >   guiapp.add_menu("foobar")
> >
> >   def exec(self, param):
> >   print("foo")
>
> "exec" is a reserved word:
>
> >>> def exec(self, param):
>  File "", line 1
>def exec(self, param):
>   ^
> SyntaxError: invalid syntax
>
>
> > the GUI application should now browse the plugin directory and read
> > those plugin python files and somehow incorporate (i.e. discover what
> > modules are there, instanciate, etc.)
> >
> > How do I do that at runtime with Python?
>
> Untested:
>
> import os
> import sys
> plugin_dir = os.path.expanduser('~/some/path/')
> plugins = set()
> for name in os.listdir(plugin_dir):
>base, ext = os.path.splitext(name)
>if ext in ('.py', '.pyc', '.pyo'):
>plugins.add(base)
> save_path = sys.path
> plugin_modules = []
> try:
>sys.path = [plugin_dir]
>for name in plugins:
>plugin_modules.append(__import__(name))
> finally:
>sys.path = save_path
>
>
>
>
> Hope this is useful.
>
>
> --
> Steven
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list

Re: New fonts for python docs site

[Mark]

On Thu, 23 Apr 2009 11:41:45 +0200, Marco Mariani wrote:
> Looks ok. What do you see?

Hmm, I was seeing a ugly jagged bit-mapped font, but only on the
docs.python.org site. So I cleared out my firefox cache and restarted and
it now looks fine.

Never seen that one before.

Sorry for the now silly post. I'd looked at the site yesterday and it 
looked fine and after seeing the ugly fonts coupled with the docs site
update of today, I jumped to a silly conclusion.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Ending data exchange through multiprocessing pipe

[Michal Chruszcz]

On Apr 22, 10:30 pm, Scott David Daniels 
wrote:
> Michal Chruszcz wrote:
> > ... First idea, which came to my mind, was using a queue. I've got many
> > producers (all of the workers) and one consumer. Seams quite simple,
> > but it isn't, at least for me. I presumed that each worker will put()
> > its results to the queue, and finally will close() it, while the
> > parent process will get() them as long as there is an active
> > subprocess
>
> Well, if the protocol for a worker is:
>      :
>           
>           queue.put(result)
>      queue.put()
>      queue.close()
>
> Then you can keep count of how many have finished in the consumer.

Yes, I could, but I don't like the idea of using a sentinel, if I
still need to close the queue. I mean, if I mark queue closed or close
a connection through a pipe why do I still have to "mark" it closed
using a sentinel? From my point of view it's a duplication. Thus I
dare to say multiprocessing module misses something quite important.

Probably it is possible to retain a pipe state using a multiprocessing
manager, thus omitting the state exchange duplication, but I haven't
tried it yet.

Best regards,
Michal Chruszcz
--
http://mail.python.org/mailman/listinfo/python-list

Re: and [True,True] --> [True, True]?????

[Leo]

Steven D'Aprano wrote:

> On Mon, 20 Apr 2009 15:13:15 -0500, Grant Edwards wrote:
> 
> 
>> I fail to see the difference between "length greater than 0" and "list
>> is not empty".  They are, by definition, the same thing, aren't they?
> 
> For built-in lists, but not necessarily for arbitrary list-like sequences.
> 
> There's also the performance issue. I might have a sequence-like
> structure where calling len() takes O(N**2) time, (say, a graph) but
> calling __nozero__ might be O(1). Why defeat the class designer's
> optimizations?
Is that true?
Calling len() actually traverses the whole list?
> 
> 


--
http://mail.python.org/mailman/listinfo/python-list

Large data arrays?

[Ole Streicher]

Hi,

for my application, I need to use quite large data arrays 
(100.000 x 4000 values) with floating point numbers where I need a fast
row-wise and column-wise access (main case: return a column with the sum 
over a number of selected rows, and vice versa).

I would use the numpy array for that, but they seem to be
memory-resistent. So, one of these arrays would use about 1.6 GB
memory which far too much. So I was thinking about a memory mapped
file for that. As far as I understand, there is one in numpy.

For this, I have two questions: 

1. Are the "numpy.memmap" array unlimited in size (resp. only limited
by the maximal file size)? And are they part of the system's memory
limit (~3GB for 32bit systems)?

2. Since I need row-wise as well as column-wise access, a simple usage
of a big array as memory mapped file will probably lead to a very poor
performance, since one of them would need to read values splattered
around the whole file. Are there any "plug and play" solutions for
that? If not: what would be the best way to solve this problem? 
Probably, one needs to use someting like the "Morton layout" for the
data. Would one then build a subclass of memmap (or ndarray?) that
implements this specific layout? How would one do that? (Sorry, I am
still a beginner with respect to python).

Best regards

Ole
--
http://mail.python.org/mailman/listinfo/python-list

Re: and [True,True] --> [True, True]?????

[Peter Otten]

Leo wrote:

> Steven D'Aprano wrote:
> 
>> On Mon, 20 Apr 2009 15:13:15 -0500, Grant Edwards wrote:
>> 
>> 
>>> I fail to see the difference between "length greater than 0" and "list
>>> is not empty".  They are, by definition, the same thing, aren't they?
>> 
>> For built-in lists, but not necessarily for arbitrary list-like
>> sequences.
>> 
>> There's also the performance issue. I might have a sequence-like
>> structure where calling len() takes O(N**2) time, (say, a graph) but
>> calling __nozero__ might be O(1). Why defeat the class designer's
>> optimizations?
> Is that true?
> Calling len() actually traverses the whole list?

No. Steven constructed the theoretical case where determining whether a
sequence (not Python's list) is empty is a cheap operation while getting
the actual length is costly. 

The only "popular" data structure I know that implements its length like
this are strings in C.

Peter
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Packages : A looming problem? packages might no longer work? (well not on your platform or python version anyway)

[David Stanek]

On Thu, Apr 23, 2009 at 2:47 AM, Daniel Fetchinson
 wrote:
>
> The OP is just thinking out loud that it would be great if developers
> could count on some help for testing various platforms and versions.
> And I agree, it would indeed be great.
>

I think you interpreted the OP differently. As I said before the idea
is not a bad one, but as a package developer I get to decide which
platforms and versions my code supports.

-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Packages : A looming problem? packages might no longer work? (well not on your platform or python version anyway)

[David Stanek]

On Thu, Apr 23, 2009 at 12:33 AM, David Lyon  wrote:
> Hi Steve,
>>> Why should the package developer dictacte which python version the
>>> package will run on ?
>>
>> Because they're the developer. Who else should decide what Python
>> versions to support?
> The developer shouldn't be making such decisions at all
> What hardware or operating systems we run his/her programs on is
> up to the real world to decide.
> If you think about it logically... why are we even asking our
> developers to even "build" their packages for specific python
> versions in the first place?
> They should just:
>

You should abandon this argument. Your original idea of a huge build
infrastructure had some merit. This is just off the wall.

If I use win32com how do you expect me to support Linux? What about
the many packages on PYPI containing C? What if I decide to write only
to Python 3? Who will support the other platforms if not the
developer?

--
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek
--
http://mail.python.org/mailman/listinfo/python-list

Re: Ending data exchange through multiprocessing pipe

[Jesse Noller]

On Thu, Apr 23, 2009 at 5:15 AM, Michal Chruszcz  wrote:
> On Apr 22, 10:30 pm, Scott David Daniels 
> wrote:
>> Michal Chruszcz wrote:
>> > ... First idea, which came to my mind, was using a queue. I've got many
>> > producers (all of the workers) and one consumer. Seams quite simple,
>> > but it isn't, at least for me. I presumed that each worker will put()
>> > its results to the queue, and finally will close() it, while the
>> > parent process will get() them as long as there is an active
>> > subprocess
>>
>> Well, if the protocol for a worker is:
>>      :
>>           
>>           queue.put(result)
>>      queue.put()
>>      queue.close()
>>
>> Then you can keep count of how many have finished in the consumer.
>
> Yes, I could, but I don't like the idea of using a sentinel, if I
> still need to close the queue. I mean, if I mark queue closed or close
> a connection through a pipe why do I still have to "mark" it closed
> using a sentinel? From my point of view it's a duplication. Thus I
> dare to say multiprocessing module misses something quite important.
>
> Probably it is possible to retain a pipe state using a multiprocessing
> manager, thus omitting the state exchange duplication, but I haven't
> tried it yet.
>
> Best regards,
> Michal Chruszcz
> --
> http://mail.python.org/mailman/listinfo/python-list
>

Using a sentinel, or looping on get/Empty pattern are both valid, and
correct suggestions.

If you think it's a bug, or you want a new feature, post it,
preferably with a patch, to bugs.python.org. Add me to the +noisy, or
if you can assign it to me.

Jesse
--
http://mail.python.org/mailman/listinfo/python-list

Re: pyflakes, pylint, pychecker - and other tools

[Nick Craig-Wood]

Colin J. Williams  wrote:
>  Esmail wrote:
> > What is the consensus of the Python community regarding these
> > code checkers?
> > 
> > In particular, are the stylistic recommendations that
> > pylint makes considered sensible/valid?
> 
>  pylint seems a bit heavy handled, a bit 
>  too much PEP 8, which was intended as a 
>  guide, rather than a prescription.

I've been very happy with pychecker.  I found pylint a bit too fussy
(rather like the original C lint!)

Note that if you run pychecker from emacs (M-x compile, then
"pychecker module_name") you can then click in its output window to go
to the correct line of code.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list

Re: Large data arrays?

[Nick Craig-Wood]

Ole Streicher  wrote:
>  for my application, I need to use quite large data arrays 
>  (100.000 x 4000 values) with floating point numbers where I need a fast
>  row-wise and column-wise access (main case: return a column with the sum 
>  over a number of selected rows, and vice versa).
> 
>  I would use the numpy array for that, but they seem to be
>  memory-resistent. So, one of these arrays would use about 1.6 GB
>  memory which far too much. So I was thinking about a memory mapped
>  file for that. As far as I understand, there is one in numpy.
> 
>  For this, I have two questions: 
> 
>  1. Are the "numpy.memmap" array unlimited in size (resp. only limited
>  by the maximal file size)? And are they part of the system's memory
>  limit (~3GB for 32bit systems)?

mmaps come out of your applications memory space, so out of that 3 GB
limit.  You don't need that much RAM of course but it does use up
address space.

>  2. Since I need row-wise as well as column-wise access, a simple usage
>  of a big array as memory mapped file will probably lead to a very poor
>  performance, since one of them would need to read values splattered
>  around the whole file. Are there any "plug and play" solutions for
>  that? If not: what would be the best way to solve this problem? 
>  Probably, one needs to use someting like the "Morton layout" for the
>  data. Would one then build a subclass of memmap (or ndarray?) that
>  implements this specific layout? How would one do that? (Sorry, I am
>  still a beginner with respect to python).

Sorry don't know very much about numpy, but it occurs to me that you
could have two copies of your mmapped array, one the transpose of the
other which would then speed up the two access patterns enormously.

You needn't mmap the two arrays (files) at the same time either.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list

Re: Memory footpring of python objects

[Nick Craig-Wood]

BlueBird  wrote:
>  I have a program that manages several thousands instances of one
>  object. To reduce memory
>  consumption, I want of course that specific object to have the
>  smallest memory footpring possible.
> 
>  I have a few ideas that I want to experiment with, like using
>  __slots__, using a tuple or using a dict. My
>  question is: how do I know the memory footprint of a given python
>  object ? I could not find any
>  builtin functions for this in the documentation.

I managed to 1/3 the memory requirement of our program by using
__slots__ on the most common class (several hundred thousand
instances!).

When doing these optimisations I ran a repeatable script and measured
the total memory usage using the OS tools (top in my case).

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list

Using optparse

[loial]

A shell script is passing parameters to my python script in the
following format

-PARAM1 12345 -PARAM2 67890

Can I parse these with optparse  ? If so how?

I can't seem to get it to work. It seems to expect --PARAM1 and --
PARAM2



--
http://mail.python.org/mailman/listinfo/python-list

Help with code! Gamepad?

[DC16]

I am using pygame and OpenGL.
How do I make a gamepad able to move the camera to a side of a cube on
screen.

Here is the code for keyboard use:



import pygame
from pygame.locals import *
import sys
from OpenGLLibrary import *

pygame.init()

Screen = (800,600)
Window = glLibWindow(Screen,caption="Camera Test")
View3D = glLibView3D((0,0,Screen[0],Screen[1]),45)
View3D.set_view()

Camera = glLibCamera([0,0.5,6],[0,0,0])

glLibColorMaterial(True)

drawing = 0
Objects = [glLibObjCube(),glLibObjTeapot(),glLibObjSphere
(64),glLibObjCylinder(0.5,1.0,64),glLibObjCone(0.5,1.8,64)]

while True:
key = pygame.key.get_pressed()
for event in pygame.event.get():
if event.type == QUIT:
pygame.quit()
sys.exit()
if event.type == KEYDOWN:
if event.key == K_ESCAPE:
pygame.quit()
sys.exit()
if event.key == K_RETURN:
drawing += 1
if drawing == 5:
drawing = 0
if event.key == K_1: glLibColor((255,255,255))
if event.key == K_2: glLibColor((255,0,0))
if event.key == K_3: glLibColor((255,128,0))
if event.key == K_4: glLibColor((255,255,0))
if event.key == K_5: glLibColor((0,255,0))
if event.key == K_6: glLibColor((0,0,255))
if event.key == K_7: glLibColor((128,0,255))
if   key[K_LEFT]: Camera.set_target_pos([-6,0.5,0])
elif key[K_RIGHT]: Camera.set_target_pos([6,0.5,0])
elif key[K_UP]: Camera.set_target_pos([0,6,2])
elif key[K_DOWN]: Camera.set_target_pos([0,-6,2])
else: Camera.set_target_pos([0,0.5,6])

Camera.update()

Window.clear()
Camera.set_camera()
Objects[drawing].draw()
Window.flip()

How do I change it so that a gampad can move the camera in the same
way.
(Ignore the colour changing code)

Thanks,
Dexter (DC16)

BTW the code was originally by someone else.
--
http://mail.python.org/mailman/listinfo/python-list

Re: need help with a multiloop error

[MRAB]

John Machin wrote:

On Apr 23, 10:54 am, delc...@gmail.com wrote:

Here is a little more of the code?  Does this help at all?


Not much. MRAB asked what is str2bool. Instead of answering the
question, you have supplied some code which is calling str2bool (and
doing interesting things if it fails).


I can only assume that 's' is meant to be a string representing a
Boolean value (eg "True"), but it's actually a float (or a float as a
string, ie "0.0"?), so a TypeError is being raised, but the code is
catching only ValueError.


You talk about a "multiloop" as though it's a well-known concept
("running a Python program with a multiloop") ... it might be in
industrial process automation but not in Python. *If* you believe that
whether your program has a "multiloop" or not is relevant to the
exception that you are getting, you had better explain what a
"multiloop" is.


I'd probably take it to mean a nested loop...


Traceback (most recent call last):
  File "C:\Python25\commodity\commoditytrading4.2.py  ", line 233, in 


That is a very interesting traceback output. How does the "http://";
get in there???


--
http://mail.python.org/mailman/listinfo/python-list

Re: Supply a plugin interface

[R. David Murray]

Ulrich Eckhardt  wrote:
> Johannes Bauer wrote:
> > What I'd like to add: I want the GUI users to supply plugin scripts,
> > i.e. offer some kind of API. That is, I want the user to write short
> > Python pieces which look something like
> > 
> > import guiapp
> > 
> > class myplugin():
> > def __init__(self):
> > guiapp.add_menu("foobar")
> > 
> > def exec(self, param):
> > print("foo")
> > 
> > the GUI application should now browse the plugin directory and read
> > those plugin python files and somehow incorporate (i.e. discover what
> > modules are there, instanciate, etc.)
> 
> You will find ways to scan a directory using os.path and related things. In
> order to import a module, you add the base directory to sys.path. Since you
> have just a variable with the module name (excluding the .py), you can't
> use 'import' as it is. Instead, use the __import__ functio (I wonder if
> there is anything more elegant) which returns the plugin.

In 2.7 and 3.1 there will be:

from importlib import import_module

myname = import_module('somename')

which has much more sensible semantics than __import__ does.

--
R. David Murray http://www.bitdance.com

--
http://mail.python.org/mailman/listinfo/python-list

Re: Ending data exchange through multiprocessing pipe

[MRAB]

Michal Chruszcz wrote:

On Apr 22, 6:33 pm, MRAB  wrote:

You could do this:

 from multiprocessing.queues import Empty

 queue = Queue()
 Process(target=f, args=(queue,)).start()
 while active_children():
 try:
 print queue.get(timeout=1)
 except Empty:
 pass


This one isn't generic. When I have tasks that all finish within 0.1
seconds the above gives 10x overhead. On the other hand, if I know the
results will be available after 10 hours there's no use in checking
every second.


If there is a result in the queue then it will return immediately; if
not, then it will wait for up to 1 second for a result to arrive, but if
none arrives in that time then it will raise an Empty exception. Raising
only 1 exception every second won't consume a negligible amount of
processing power (I wouldn't worry about it).
--
http://mail.python.org/mailman/listinfo/python-list

Unicode in writing to a file

[Carbon Man]

Py 2.5
Trying to write a string to a file.
self.dataUpdate.write(u"\nentry."+node.tagName+ u" = " + cValue)
cValue contains a unicode character. node.tagName is also a unicode string 
though it has no special characters in it.
Getting the error:
UnicodeEncodeError: 'ascii' codec can't encode character u'\x93' in position 
22: ordinal not in range(128)

Thanks. 


--
http://mail.python.org/mailman/listinfo/python-list

Re: and [True,True] --> [True, True]?????

[Hrvoje Niksic]

Peter Otten <__pete...@web.de> writes:

> The only "popular" data structure I know that implements its length
> like this are strings in C.

Linked lists and trees also tend to do the same, with the exception of
those that explicitly store their length to optimize length queries.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Memory problems (garbage collection)

[Carbon Man]

Thanks for the help.
I converted everything into the StringIO() format. Memory is still getting 
chewed up. I will look at ElementTree later but for now I believe the speed 
issue must be related to the amount of memory that is getting used. It is 
causing all of windows to slow to a crawl. gc.collect() still reports the 
same quantity as before.
Don't know what to try next. Updated program is below:

from xml.dom import minidom
import os
from cStringIO import StringIO

class xmlProcessing:
""" General class for XML processing"""

def process(self, filename="", xmlString=""):
if xmlString:
pass
elif filename:
xmldoc = minidom.parse(filename)
self.parse( xmldoc.documentElement )

def parseBranch(self, parentNode):
""" Process an XML branch """
for node in parentNode.childNodes:
try:
parseMethod = getattr(self, "parse_%s" % 
node.__class__.__name__)
except AttributeError:
continue
if parseMethod(node):
continue
self.parseBranch(node)
del node

def parse_Document(self, node):
pass

def parse_Text(self, node):
pass

def parse_Comment(self, node):
pass

def parse_Element(self, node):
try:
handlerMethod = getattr(self, "do_%s" % node.tagName)
except AttributeError:
return False
handlerMethod(node)
return True

class reptorParsing(xmlProcessing):
""" Specific class for generating a SQLalchemy program to create tables
and populate them with data"""

def __init__(self):
self.schemaPreface = StringIO()
self.schemaPreface.write("""from sqlalchemy import *
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine('sqlite:///tutorial.db', echo=False)
metadata = MetaData()
Base = declarative_base()""")
self.schemaTables = StringIO()
self.schemaFields = StringIO()
self.dataUpdate = StringIO()
self.tableDict = {}
self.tableName = StringIO()
self.tables = StringIO()

def parse(self, parentNode):
"""Main entry point to begin processing a XML document"""
self.parseBranch(parentNode)
# Properties such as schemaTables and .tables are populated by the 
various methods below
fupdate=open(os.path.join(os.getcwd(), "update.py"), 'w')
if self.schemaTables:
fupdate.write("import schema\n")
f=open(os.path.join(os.getcwd(), "schema.py"), 'w')
f.write(self.schemaPreface+"\n"+self.schemaTables+
'\n' + "metadata.create_all(engine)\n"+
"print 'hello 2'")
f.close()
if self.tables:
fupdate.write(self.tables)
fupdate.close()

def do_TABLES(self, tableNode):
"""Process schema for tables"""
for node in tableNode.childNodes:
self.tableName = node.tagName
# Define a declaritive mapping class
self.schemaTables.write("""\nclass %s(Base):
__tablename__ = '%s'
""" % (self.tableName, self.tableName))
self.schemaFields = StringIO()
# allow for userA = users("Billy","Bob") via a __init__()
self.schemaInitPreface = StringIO()
self.schemaInitPreface.write("def __init__(self")
self.schemaInitBody = StringIO()
self.parseBranch(node)
self.schemaInitPreface.write("):\n")
self.schemaTables.write(self.schemaFields.read() + "\n" + \
self.schemaInitPreface.read() + \
self.schemaInitBody.read() + "\n")

def do_FIELDS(self, fieldsNode):
"""Process schema for fields within tables"""
for node in fieldsNode.childNodes:
if self.schemaFields:
self.schemaFields.write("\n")
cType = ""
# The attribute type holds the type of field
crType = node.attributes["type"].value
if crType==u"C":
cType = "String(length=%s)" % node.attributes["len"].value
elif crType==u"N" and node.attributes["dec"].value==u'0':
cType = "Integer"
elif crType==u"N":
cType = "Numeric(precision=%s, scale=%s)" % 
(node.attributes["len"].value,node.attributes["dec"].value)
elif crType==u"L":
cType = "Boolean"
elif crType==u"T":
cType = "DateTime"
elif crType==u"D":
cType = "Date"
elif crType==u"M" or crType==u"G":
cType = "Text"

if node.attributes.getNamedItem("primary"):
cType += ", primary_key=True"
self.schemaFields.write("%s = Column(%s)" % (node.tagName, 
cType))
self.schemaInitPreface.write(", \\\n%s" % 
(node.tagName))
self.schemaInitBody.wri

Re: Python Packages : A looming problem? packages might no longer work? (well not on your platform or python version anyway)

[Marco Mariani]

David Lyon wrote:


What if I decide to write only to Python 3?


Fair enough. But don't forget it is open source.


So what?


Let me ask these two questions...

 - What about the use case where somebody likes the code and wants 
   to use it on Python 2.5?


A patch, a fork, whatever.

 - Should not that user be able to share back with other 
   Python 2.5 users?


Probably. But not necessarily on the same project or repository.
Especially if the port is not supported by the original/trunk developer.



If anything my suggestion promotes preserving the resources
of the original developer rather than letting them expire just
because their operating system does


I think your idea would be very costing in terms of man-years, and the 
net result would be... entropy, chaos, and a lot of CO2 :-)


--
http://mail.python.org/mailman/listinfo/python-list

Re: Using optparse

[Peter Otten]

loial wrote:

> A shell script is passing parameters to my python script in the
> following format
> 
> -PARAM1 12345 -PARAM2 67890
> 
> Can I parse these with optparse  ? If so how?
> 
> I can't seem to get it to work. It seems to expect --PARAM1 and --
> PARAM2

You are out of luck. Quoting

http://docs.python.org/library/optparse.html

"""
There are many different syntaxes for options; the traditional Unix syntax
is a hyphen (“-“) followed by a single letter, e.g. "-x" or "-F". Also,
traditional Unix syntax allows multiple options to be merged into a single
argument, e.g. "-x -F" is equivalent to "-xF". The GNU project
introduced "--" followed by a series of hyphen-separated words,
e.g. "--file" or "--dry-run". These are the only two option syntaxes
provided by optparse.
"""

Peter
--
http://mail.python.org/mailman/listinfo/python-list

Re: Unicode in writing to a file

[Marco Mariani]

Carbon Man wrote:


Py 2.5
Trying to write a string to a file.
self.dataUpdate.write(u"\nentry."+node.tagName+ u" = " + cValue)
cValue contains a unicode character. node.tagName is also a unicode string 
though it has no special characters in it.


So what's the encoding of your file?

If you didn't open dataUpdate with codecs.open, and you don't encode the 
string someway - i.e. text.encode('utf-8') python has no way to know it.


--
http://mail.python.org/mailman/listinfo/python-list

Re: Unicode in writing to a file

[Diez B. Roggisch]

Carbon Man wrote:

> Py 2.5
> Trying to write a string to a file.
> self.dataUpdate.write(u"\nentry."+node.tagName+ u" = " + cValue)
> cValue contains a unicode character. node.tagName is also a unicode string
> though it has no special characters in it.
> Getting the error:
> UnicodeEncodeError: 'ascii' codec can't encode character u'\x93' in
> position 22: ordinal not in range(128)

Please don't confuse utf-8 with unicode. The former is an encoding of the
latter. Which is a crucial difference, as something being encoded (either
in utf-8 or any other encoding) needs to be *decoded* before being dealt
with in python as unicode-object.

Which is what's causing your troubles here. cValue is a *byte*string
containing some non-ascii-characters. 

If you are sure cValue is utf-8-encoded, you can do

u" = " + cValue.decode("utf-8")

to remedy the problem.

Diez
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Packages : A looming problem? packages might no longer work? (well not on your platform or python version anyway)

[David Lyon]

On Thu, 23 Apr 2009 07:04:35 -0400, David Stanek 
wrote:

> If I use win32com how do you expect me to support Linux? 

Of course not...

> What about the many packages on PYPI containing C? 

Exactly. 

> What if I decide to write only to Python 3?

Fair enough. But don't forget it is open source.

Let me ask these two questions...

 - What about the use case where somebody likes the code and wants 
   to use it on Python 2.5?

 - Should not that user be able to share back with other 
   Python 2.5 users?

> Who will support the other platforms if not the developer?

It's Open Source don't forget

Fact is these days.. developers come and go

If anything my suggestion promotes preserving the resources
of the original developer rather than letting them expire just
because their operating system does

(I'm talking windows here)

David


--
http://mail.python.org/mailman/listinfo/python-list

Re: Unicode in writing to a file

[Ulrich Eckhardt]

Carbon Man wrote:
> self.dataUpdate.write(u"\nentry."+node.tagName+ u" = " + cValue)
> cValue contains a unicode character. node.tagName is also a unicode string
> though it has no special characters in it.
> Getting the error:
> UnicodeEncodeError: 'ascii' codec can't encode character u'\x93' in
> position 22: ordinal not in range(128)

There are two operations:
1. Concatenating the strings.
2. Invoking write().

First step I would make is to find out which of the two is raising the
exception. Anyway, it probably is the second one, i.e. the call to write().
What is happening there is that the file's codec is trying to convert the
Unicode string to bytes for the configured encoding ('ascii') but fails
because there is no representation for u'\x93' there (Note: ASCII only uses
the byte values from 0 to 126, the above is 147).

The remedy is to set the output encoding to e.g. UTF-8 (default for XML) or
ISO8859-1 (default for HTML) or whichever encoding you want. Otherwise,
just throw the error message at the search engine of your least distrust to
find a bazillion of other users that had similar problems. ;)

Uli

-- 
Sator Laser GmbH
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932

--
http://mail.python.org/mailman/listinfo/python-list

Re: Unicode in writing to a file

[Peter Otten]

Carbon Man wrote:

> Py 2.5
> Trying to write a string to a file.
> self.dataUpdate.write(u"\nentry."+node.tagName+ u" = " + cValue)
> cValue contains a unicode character. node.tagName is also a unicode string
> though it has no special characters in it.
> Getting the error:
> UnicodeEncodeError: 'ascii' codec can't encode character u'\x93' in
> position 22: ordinal not in range(128)

You have to decide in what encoding you want to store the data in your file.
UTF-8 is usually a good choice. Then open it with codecs.open() instead of
the built-in open():

import codecs

f = codecs.open(filename, "w", "UTF-8")
f.write(u"\nentry." + node.tagName + u" = " + cValue)

Peter

--
http://mail.python.org/mailman/listinfo/python-list

Re: Large data arrays?

[Ole Streicher]

Hi Nick,

Nick Craig-Wood  writes:
> mmaps come out of your applications memory space, so out of that 3 GB
> limit.  You don't need that much RAM of course but it does use up
> address space.

Hmm. So I have no chance to use >= 2 of these arrays simultaniously?

> Sorry don't know very much about numpy, but it occurs to me that you
> could have two copies of your mmapped array, one the transpose of the
> other which would then speed up the two access patterns enormously.

That would be a solution, but it takes twice the amount of address
space (which seems already to be the limiting factor). In my case (1.6
GB per array), I could even not use one array. 

Also, I would need to fill two large files at program start: one for
each orientation (row-wise or column-wise). Depending on the input
data (which are also either row-wise or column-wise), the filling of
the array with opposite direction would take a lot of time because of
the inefficiencies.

For that, using both directions probably would be not a good
solution. What I found is the "Morton layout" which uses a kind of
fractal interleaving and sound not that complicated. But I have no
idea on how to turn it into a "numpy" style: can I just extend from
numpy.ndarray (or numpy.memmap), and which functions/methods then need
to be overwritten? The best would be ofcourse that someone already did
this before that I could use without trapping in all these pitfalls
which occur when one implements a very generic algorithm.

Best regards

Ole
--
http://mail.python.org/mailman/listinfo/python-list

Re: sorting two corresponding lists?

[Piet van Oostrum]

> Saketh  (S) wrote:

>S> Why not use a dictionary instead of two lists? Then you can sort the
>S> dictionary by value -- e.g.

>S> d = dict(zip(items, values))
>S> sorted_items = sorted(d.iteritems(), key=lambda (k,v): (v,k))

>S> This produces a list of pairs, but demonstrates the general idea.

This will fail if there are duplicates in the items. And if we assume no
duplicates the key doesn't need the k part:

sorted_items = sorted(d.iteritems(), key=lambda (k,v): v)

or

from operator import itemgetter
sorted_items = sorted(d.iteritems(), key=itemgetter(1))

-- 
Piet van Oostrum 
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: p...@vanoostrum.org
--
http://mail.python.org/mailman/listinfo/python-list

Re: Ending data exchange through multiprocessing pipe

[Paul Boddie]

On 22 Apr, 17:43, Michal Chruszcz  wrote:
>
> I am adding support for parallel processing to an existing program
> which fetches some data and then performs some computation with
> results saved to a database. Everything went just fine until I wanted
> to gather all of the results from the subprocesses.

[Queue example]

I have to say that I'm not familiar with the multiprocessing API, but
for this kind of thing, there needs to be some reasonably complicated
stuff happening in the background to test for readable conditions on
the underlying pipes or sockets. In the pprocess module [1], I had to
implement a poll-based framework (probably quite similar to Twisted
and asyncore) to avoid deadlocks and other undesirable conditions.

[Pipe example]

Again, it's really awkward to monitor pipes between processes and to
have them "go away" when closed. Indeed, I found that you don't really
want them to disappear before everyone has finished reading from them,
but Linux (at least) tends to break pipes quite readily. I got round
this problem by having acknowledgements in pprocess, but it felt like
a hack.

> Most possibly I'm missing something in philosophy of multiprocessing,
> but I couldn't find anything covering such a situation. I'd appreciate
> any kind of hint on this topic, as it became a riddle I just have to
> solve. :-)

The multiprocessing module appears to offer map-based conveniences
(Pool objects) where you indicate that you want the same callable
executed multiple times and the results to be returned, so perhaps
this is really what you want. In pprocess, there's a certain amount of
flexibility exposed in the API, so that you can choose to use a map-
like function, or you can open everything up and use the
communications primitives directly (which would appear to be similar
to the queue-oriented programming mentioned in the multiprocessing
documentation).

One thing that pprocess exposes (and which must be there in some form
in the multiprocessing module) is the notion of an "exchange" which is
something that monitors a number of communications channels between
processes and is able to detect and act upon readable channels in an
efficient way. If it's not the Pool class in multiprocessing that
supports such things, I'd take a look for the component which does
support them, if I were you, because this seems to be the
functionality you need.

Paul

[1] http://www.boddie.org.uk/python/pprocess.html
--
http://mail.python.org/mailman/listinfo/python-list

Cleaning out the cobwebs in the PEP cupboard

[Larry Hastings]

There are some PEPs that seem to be stuck in a perpetual limbo, never to 
be decided upon.  Some of them seem hopelessly antiquated now--they were 
proposed long ago and the language has moved on.  With every passing day 
it becomes less likely they will ever be Accepted.


On the off chance that their fate is decided and we just hadn't noticed, 
I thought I'd take a couple of minutes, talk about some of those PEPs, 
and make some suggestions.  I fear I am jamming multiple feet into my 
mouth; honest, I mean no harm.  Just trying to get the wheels of fate 
turning once again.


We start with one I think is terribly obvious.

PEP 333: Python Web Server Gateway Interface v1.0
 http://www.python.org/dev/peps/pep-0333/

   The WSGI PEP.  Even though it's six years old, and mighty empires
   have been built upon its rock-solid foundations, it's still marked
   as "Draft".

   This is sillier than Gmail still being marked "beta".  In the name
   of Guido's shiny time machine: could PEP 333 please be formally
   Accepted?


The rest of these are marked "deferred", and date from very early in the 
decade, sometimes even before Python 2.0.


PEP 211: Adding A New Outer Product Operator
 http://www.python.org/dev/peps/pep-0211/

   This proposes @ as a "product" operator for iteration.  It would
   make "for i, j in S @ T:" equivalent to "for i in S: for j in T:".

   Since then we've added generator expressions and itertools.  In
   particular, itertools.product (added in 2.3) solves exactly the
   problem described, and as it's in C it wouldn't be any faster if
   this were an operator.  The only debate left is whether this is
   important enough to warrant the shorter spelling.  If brevity is
   that important, do "from itertools import product as p".  That would
   add a net of two extra characters per iteration.  I therefore gently
   suggest that this PEP be Rejected (or Withdrawn).


PEP 212: Loop counter iteration
 http://www.python.org/dev/peps/pep-0212/

   This PEP attempts to solve two problems: "for i in range(len(x)): e
   = x[i]" is inconvenient and wordy, and "for i in range(len(x)):" is
   wordy.  It proposes either a new keyword (indexing), two global
   functions (irange and indices respectively for the two problems),
   and/or two method calls on sequences mirroring the proposed global
   functions.

   The former problem was addressed with enumerate (added in 2.3, see
   PEP 279).  The latter problem is not severe or common enough to
   really need addressing.  Certainly it doesn't merit a keyword or a
   new method on every iterable; I don't think it even merits a global
   function.  The right place for this function, if we want it at all,
   would be in itertools.  I therefore gently suggest that this PEP
   also be Rejected (or Withdrawn).


PEP 213: Attribute Access Handlers
 http://www.python.org/dev/peps/pep-0213/

   This proposes an extension to classes to allow overriding access to
   members on a case-by-case basis.  Instead of writing one big
   __getattr__, you could write individual methods called __attr_XXX__
   to override behavior on self.XXX.

   In other words: properties.  This was years before Python 2.2 and
   the mighty PEP 252 which introduced properties.  Properties solves
   this exact problem.  This PEP actually discusses 252 at the end,
   staring its own fate in the eye.  While I admire its courage, there
   is no chance of this PEP being Accepted.  Once again I must
   recommend Rejection (or Withdrawl).


PEP 222: Web Library Enhancements
 http://www.python.org/dev/peps/pep-0222/

   This is an open-ended "we should make Python library support for CGI
   better" proposal.  It is not terribly specific; it seems to have
   been proposed in a deferred state.  Back before 2.1.

   Since then, CGI has fallen by the wayside; no serious server-side
   developer would willingly incur a new process per request.  We also
   have the aforementioned WSGI, which addresses the only specific area
   of the proposal (the Request and Response objects).

   In any case I suggest that this is not the proper way to approach
   this topic.  Instead, someone / some folks should /write/ such a
   module, release it on the 'net, get lots of happy users, and /then/
   propose submitting it to the standard library.  Therefore this PEP
   should be Rejected (or Withdrawn).


PEP 225: Elementwise/Objectwise Operators
 http://www.python.org/dev/peps/pep-0225/

   This proposes six new binary operators: ~+ ~- ~* ~/ ~% and ~**, and
   six new augmented assignment operators (add = to the end of the
   previous six).  These would be used for numeric expressions to
   differentiate between elementwise operations (operate on individual
   elements of the two operands) and objectwise operations (operate on
   each of the two operands as a whole).

   My math isn't good enough to say how necessary this is.  But it's
   been more than eight years since it was proposed; if it was that
   necessary su

Re: gethostbyname blocking

[Piet van Oostrum]

> marc wyburn  (MW) wrote:

>MW> Hi, I am writing an asynchronous ping app to check if 1000s of hosts
>MW> are alive very quickly.  Everything works extremely quickly unless the
>MW> host name doesn't have a DNS record.

>MW> when calling socket.gethostbyname if there is no record for the host
>MW> the result seems to block all other threads.  As an example I have 6
>MW> threads running and if I pass the class below a Queue with about 30
>MW> valid addresses with one invalid address in the middle the thread that
>MW> the exception occurs in seems to block the others.

What you could do is have two Queues, one with host names, and one with
results from gethostbyname. And an additional thread which gets from the
first queue, calls gethostbyname and puts the results in the second queue.
The ping threads then should get from the second queue. The lookup
thread could even do caching of the results if you often have to repeat
the pings for thew same host.
-- 
Piet van Oostrum 
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: p...@vanoostrum.org
--
http://mail.python.org/mailman/listinfo/python-list

OT: a metacomment on feedback comments

[Aaron Watters]

I just noticed this on my documentation
page.

http://aaron.oirt.rutgers.edu/myapp/docs/W.intro
"""
This is quite possible the most confusing
web framework ever.   It really deserves
some kind of award in a special category.
"""

This was posted just after someone on the same
computer (somewhere in Texas)
tried and failed to inject some javascript
into the page using a comment.

I don't see how this comment is helpful to
me or anyone reading the page.  How the
framework confusing?  Why should other readers dismiss
the project?  Can the documentation
or implementation be
improved, or is the whole thing beyond repair
and if so how or why?  Why do people bother
to write comments like this one?

Is there a link somewhere I can point to
to help people write useful or constructive
comments?

BTW: I just added support for several
Mootools javascript widgets and the
FCK wisiwig editor, checked in
to the repository and demoed in the
documentation, but not tarred up
in a release yet.

At least I'm having fun with it!

   -- Aaron Watters
  http://aaron.oirt.rutgers.edu/myapp/docs/W1500.whyIsWhiffCool

===
ban dhmo!
http://www.dhmo.org/
--
http://mail.python.org/mailman/listinfo/python-list

Re: WINXP vs. LINUX in threading.Thread

[Mike Driscoll]

On Apr 22, 2:57 pm, Kent  wrote:
> hello all,
>
> i want to add a "new update notification" feature to my wxPython appl.
> The codes below do the job. The logic is simple enough, I don't think
> it needs to be explained.
>
> since sometimes, under windows, proxy setting was a script. and was
> set in IE. In this case, connecting to the HTML will take relative
> long time. I therefore run the following codes in a new Thread
> (subclass of threading.Thread), so that user don't have to wait during
> the version checking period.
>
> Under Linux, it worked smoothly. But under Windows XP, it didn't. If
> there was new updates, the notification dialog can show, but no text,
> icon,  on it. Then, the whole application didn't response any
> longer. :( I have to force stop the application process.
>
> where is the problem?
>
> Thanks.
>
> code to get the version from a HTML:
>
> def getLatestVersion(versionUrl):
>     #proxy is only for testing
> #    proxy_support = urllib2.ProxyHandler
> ({"http":"10.48.187.80:8080"})
> #    opener = urllib2.build_opener(proxy_support)
> #    urllib2.install_opener(opener)
>     #proxy is only for testing
>
>     result = ''
>     try:
>         f = urllib2.urlopen(versionUrl)
>         s = f.read()
>         f.close()
>         result = s.split("#xxx#")[1]
>     except:
>         pass
>     return result
>
> Thread class:
>
> class UpdateChecker(Thread):
>     def __init__(self):
>         Thread.__init__(self)
>
>     def run(self):
>         lv = util.getLatestVersion(config.VERSION_URL)
>         if lv>config.VERSION:
>             showUpdateDialog(lv)
>
> codes to start the thread:
>
> updatechk = UpdateChecker()
> updatechk.start()

You are probably blocking wxPython's main loop. There are several
examples of using threads with wxPython on their wiki:
http://wiki.wxpython.org/LongRunningTasks

As you will find, there are various methods within wxPython that are
threadsafe (such as wx.CallAfter) and you can use those to communicate
with and update your GUI. From what I've read, this is normal across
GUI toolkits. They all have special ways to work with threads.

In the future, I highly recommend that you join and post to the
wxPython list as that's where you'll get the best targeted advice (see
their website). This list is great for general questions though.

- Mike
--
http://mail.python.org/mailman/listinfo/python-list

Re: Memory problems (garbage collection)

[Christian Heimes]

Peter Otten wrote:
> Like Gerhard says, in the long run you are probably better off with
> ElementTree.

In the long run it's even better to use lxml [1]. It's the fastest und
most powerful XML library for Python. It also supports element tree.

Christian

[1] http://codespeak.net/lxml/

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Packages : A looming problem?

[Paul Boddie]

On 23 Apr, 10:59, Lawrence D'Oliveiro  wrote:
>
> The Linux solution is to leave distro packaging to the distro maintainers.
> Release the source, and they will make up the necessary packages for their
> specific one-click installers. You don't have to worry about it.

And the crucial message, that may not have sunk in yet amongst the
many people messing around with "enhancing" distutils, is that in
order for the distro maintainers to have something decent to work
with, distutils and related tools should produce stuff which isn't too
clever for its own good: "dist" as in "distribute", not as "install,
but not uninstall, and then try and own everything from the
developer's toolchain to the user's installation environment".

Paul
--
http://mail.python.org/mailman/listinfo/python-list

Configuring pylint for local conventions (was: pyflakes, pylint, pychecker - and other tools)

[Ben Finney]

a...@pythoncraft.com (Aahz) writes:

> Second, you can configure pylint to respect your personal style

How? I haven't seen any decent documentation on doing so.

-- 
 \  “When we call others dogmatic, what we really object to is |
  `\   their holding dogmas that are different from our own.” —Charles |
_o__)   Issawi |
Ben Finney
--
http://mail.python.org/mailman/listinfo/python-list

Re: OT: a metacomment on feedback comments

[Paul Boddie]

On 23 Apr, 15:22, Aaron Watters  wrote:
>
> This was posted just after someone on the same
> computer (somewhere in Texas)
> tried and failed to inject some javascript
> into the page using a comment.

This just means that you may have reached the Reddit crowd. Don't
worry, though: ten seconds after writing their comment (having Googled
and copy-pasted the JavaScript "attack" moments before that), someone
dangled a shiny object outside their window and they ran outside to
take a look. They won't remember your framework now.

Paul
--
http://mail.python.org/mailman/listinfo/python-list

Re: gethostbyname blocking

[Jean-Paul Calderone]

On Thu, 23 Apr 2009 15:16:56 +0200, Piet van Oostrum  wrote:

marc wyburn  (MW) wrote:



MW> Hi, I am writing an asynchronous ping app to check if 1000s of hosts
MW> are alive very quickly.  Everything works extremely quickly unless the
MW> host name doesn't have a DNS record.



MW> when calling socket.gethostbyname if there is no record for the host
MW> the result seems to block all other threads.  As an example I have 6
MW> threads running and if I pass the class below a Queue with about 30
MW> valid addresses with one invalid address in the middle the thread that
MW> the exception occurs in seems to block the others.


What you could do is have two Queues, one with host names, and one with
results from gethostbyname. And an additional thread which gets from the
first queue, calls gethostbyname and puts the results in the second queue.
The ping threads then should get from the second queue. The lookup
thread could even do caching of the results if you often have to repeat
the pings for thew same host.


You still end up limited to a single host lookup at a time with this
approach.  Using a name lookup API which doesn't have this limitation
will probably produce significantly better results.

Jean-Paul
--
http://mail.python.org/mailman/listinfo/python-list

Re: Help with code! Gamepad?

[Mike Driscoll]

On Apr 23, 6:46 am, DC16  wrote:
> I am using pygame and OpenGL.
> How do I make a gamepad able to move the camera to a side of a cube on
> screen.
>
> Here is the code for keyboard use:
>
> import pygame
> from pygame.locals import *
> import sys
> from OpenGLLibrary import *
>
> pygame.init()
>
> Screen = (800,600)
> Window = glLibWindow(Screen,caption="Camera Test")
> View3D = glLibView3D((0,0,Screen[0],Screen[1]),45)
> View3D.set_view()
>
> Camera = glLibCamera([0,0.5,6],[0,0,0])
>
> glLibColorMaterial(True)
>
> drawing = 0
> Objects = [glLibObjCube(),glLibObjTeapot(),glLibObjSphere
> (64),glLibObjCylinder(0.5,1.0,64),glLibObjCone(0.5,1.8,64)]
>
> while True:
>     key = pygame.key.get_pressed()
>     for event in pygame.event.get():
>         if event.type == QUIT:
>             pygame.quit()
>             sys.exit()
>         if event.type == KEYDOWN:
>             if event.key == K_ESCAPE:
>                 pygame.quit()
>                 sys.exit()
>             if event.key == K_RETURN:
>                 drawing += 1
>                 if drawing == 5:
>                     drawing = 0
>             if event.key == K_1: glLibColor((255,255,255))
>             if event.key == K_2: glLibColor((255,0,0))
>             if event.key == K_3: glLibColor((255,128,0))
>             if event.key == K_4: glLibColor((255,255,0))
>             if event.key == K_5: glLibColor((0,255,0))
>             if event.key == K_6: glLibColor((0,0,255))
>             if event.key == K_7: glLibColor((128,0,255))
>     if   key[K_LEFT]: Camera.set_target_pos([-6,0.5,0])
>     elif key[K_RIGHT]: Camera.set_target_pos([6,0.5,0])
>     elif key[K_UP]: Camera.set_target_pos([0,6,2])
>     elif key[K_DOWN]: Camera.set_target_pos([0,-6,2])
>     else: Camera.set_target_pos([0,0.5,6])
>
>     Camera.update()
>
>     Window.clear()
>     Camera.set_camera()
>     Objects[drawing].draw()
>     Window.flip()
>
> How do I change it so that a gampad can move the camera in the same
> way.
> (Ignore the colour changing code)
>
> Thanks,
> Dexter (DC16)
>
> BTW the code was originally by someone else.

Try cross-posting to the pygame guys: http://www.pygame.org/wiki/info

I read that pyglet will have some sort of gamepad support soon too, so
you might want to check that project out as well.

- Mike
--
http://mail.python.org/mailman/listinfo/python-list

Cython + tuple unpacking

[Hugues Salamin]

Hello,

The following code will crash with a segfault when compiled using cython (v0.11)

def func():
for (a, b) ,c ,d in zip(zip(range(3), range(3)), range(3), range(3)):
print a, b
print c
print d # This line segfault

Compilation is done using distutils.

If the module is imported in python and func is called, I got a segmentation
fault at the line "print d".

I investigated the error with valgrind and gdb but I still have no clue what is
going on.

I'm on Ubuntu 8.04, Kernel 2.6.24-22-generic

Thanks
-- 
Hugues Salamin
--
http://mail.python.org/mailman/listinfo/python-list

Why can function definitions only use identifiers, and not attribute references or any other primaries?

[Jeremy Banks]

Hi. I'm sure there've been debates about this before, but I can't seem
to figure out what to search for to pull them up, so I'm asking here.

It seems to me that a lot of things could be made much easier if you
could use primaries other than basic identifiers for the target of
function definitions. For example, if attribute references were
allowed I'd be able to do:

def foo.bar():
return("I'm a method!")

If we wanted to get even more liberal (which I don't see as a bad
thing, but I could see being found more objectionable by some), we
could allow the use of anything that's a valid assignment target. For
example:

def foo["bar']():
return("I'm a function referenced in a mapping object!")


In this case I could see there being a problem in that there's nothing
to get the function's __name__ from, but that doesn't apply for the
first example.

Many uses of this may not be Pythonic, but I'm sure there are many
that are. It just feels like an arbitrary restriction, preventing
users from doing something that may be useful.

Any feedback or direction to previous discussion on the subject would
be appreciated. Thanks!
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why can function definitions only use identifiers, and not attribute references or any other primaries?

[Gary Herron]

Jeremy Banks wrote:

Hi. I'm sure there've been debates about this before, but I can't seem
to figure out what to search for to pull them up, so I'm asking here.

It seems to me that a lot of things could be made much easier if you
could use primaries other than basic identifiers for the target of
function definitions. For example, if attribute references were
allowed I'd be able to do:

def foo.bar():
return("I'm a method!")
  


There's no need for a specific addition to the syntax to do this.

Try this:

   def foo_bar():
   return(...)
   foo.bar = foo_bar


If we wanted to get even more liberal (which I don't see as a bad
thing, but I could see being found more objectionable by some), we
could allow the use of anything that's a valid assignment target. For
example:

def foo["bar']():
return("I'm a function referenced in a mapping object!")
  


and this:

   def foo_bar():
   return(...)
   foo["bar"] = foo_bar





In this case I could see there being a problem in that there's nothing
to get the function's __name__ from, but that doesn't apply for the
first example.
  


Not sure what you mean here.



Many uses of this may not be Pythonic, but I'm sure there are many
that are. It just feels like an arbitrary restriction, preventing
users from doing something that may be useful.

Any feedback or direction to previous discussion on the subject would
be appreciated. Thanks!
--
http://mail.python.org/mailman/listinfo/python-list
  


--
http://mail.python.org/mailman/listinfo/python-list

value error

[Francesco Pietra]

hi:
with script

data = open('134-176_rectified_edited.pdb', 'r')
outp = open('134-176_renumbered.pdb', 'w')

for L in data:
   if L[3] == 'M':
 L = L[:24] + "%4d" % (int(L[24-28])+133) + L[28:]
   outp.write(L)


i wanted to modify lines of the type:
ATOM  1 HH31 ACE 1   1.573   1.961   0.769  1.00  0.00   H

to add 133 to column 25, getting 134 there, and so on for next lines 2
-> 135, 3 -> 136, etc.


i must have heavily messed things because the file was not even read:

$ python renumber.py 134-176_rectified.pdb
Traceback (most recent call last):
  File "renumber.py", line 6, in 
L = L[:24] + "%4d" % (int(L[24-28])+133) + L[28:]
ValueError: invalid literal for int() with base 10: ''



thanks for having an expert look

chiendarret
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why can function definitions only use identifiers, and not attribute references or any other primaries?

[Jeremy Banks]

Thanks for your comments.

On Thu, Apr 23, 2009 at 11:52, Gary Herron  wrote:
> > [...]
>
> There's no need for a specific addition to the syntax to do this.
>
> Try this:
>
>   def foo_bar():
>       return(...)
>   foo.bar = foo_bar
>
>> [...]
>
> and this:
>
>   def foo_bar():
>       return(...)
>   foo["bar"] = foo_bar
>

I understand that this is possible now, but I just don't see why it's
necessary to do it at all.

>> In this case I could see there being a problem in that there's nothing
>> to get the function's __name__ from, but that doesn't apply for the
>> first example.
>>
>
> Not sure what you mean here.

>>> def foo(): pass
...
>>> bar = foo
>>> bar.__name__
'foo'
>>>

If I defined foo.bar it would know that the method name was "bar", but
if I defined foo["bar"] there's be no clear identifier to use for the
function's name. I don't see this as a great problem, since anonymous
functions already exist, but I thought it was worth acknowledging.

To be clear, I don't see this as a serious fault in the language, but
as an unnecessary restriction that makes code a little less direct
than it could be.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why can function definitions only use identifiers, and not attribute references or any other primaries?

[Alan Franzoni]

Jeremy Banks was kind enough to say:

> Hi. I'm sure there've been debates about this before, but I can't seem
> to figure out what to search for to pull them up, so I'm asking here.

I'm not exactly sure what you mean... if you want to add functions to
another fuction, just do it this way:

def foo():
return "i'm the main function."

foo.bar = lambda: "i'm the bar attr"

print foo()
print foo.bar()

> def foo.bar():
> return("I'm a method!")

So you'd just like some "syntactic sugar" to let function declarations to
be "shorter"?

I think this wouldn't make the language that clear. In order to accomplish
what you want, you could just use a class and the __call__ method for the
"main" function.

I think that what you really mean is that you would like an "improved
lambda" or "different def". E.g., that could you write anonymous functions
( or ruby-like code blocks ) in a different way. Something like this:

d["func"] = def pippo(a, b, c):
...
...
...

I suppose the reason behind this is that it's not really needed. There're
plenty of ways to do this  sort of things in python; if you just declare
methods in a class you can just use the plain def in order to achieve this,
and you can inherit from your container and specialize some methods in
order to achieve results which are similar to those you want.



-- 
Alan Franzoni 
-
Remove .xyzz from my email in order to contact me.
-
GPG Key Fingerprint:
5C77 9DC3 BD5B 3A28 E7BC 921A 0255 42AA FE06 8F3E
--
http://mail.python.org/mailman/listinfo/python-list

Re: Configuring pylint for local conventions (was: pyflakes, pylint, pychecker - and other tools)

[Jeremiah Dodds]

On Thu, Apr 23, 2009 at 2:58 PM, Ben Finney

> wrote:

> a...@pythoncraft.com (Aahz) writes:
>
> > Second, you can configure pylint to respect your personal style
>
> How? I haven't seen any decent documentation on doing so.
>
> --
>  \  “When we call others dogmatic, what we really object to is |
>  `\   their holding dogmas that are different from our own.” —Charles |
> _o__)   Issawi |
> Ben Finney
> --
> http://mail.python.org/mailman/listinfo/python-list


I'm fairly certain that ~/.pylint.rc is just the command line options
available to pylint made into a conf file.

See  http://flexget.com/browser/trunk/pylint.rc?rev=475 for an example.

So, for a section from pylint --help like so:

 Miscellaneous:
--notes=
List of note tags to take in consideration,
separated
by a comma. [current: FIXME,XXX,TODO]

You would make a configuration item like so:

[MISCELLANEOUS]
notes=FIXME,TODO

This is a fairly common thing for command-line app configurations.
--
http://mail.python.org/mailman/listinfo/python-list

Re: value error

[Gerhard Häring]

Francesco Pietra wrote:
> hi:
> with script
> 
> data = open('134-176_rectified_edited.pdb', 'r')
> outp = open('134-176_renumbered.pdb', 'w')
> 
> for L in data:
>if L[3] == 'M':
>  L = L[:24] + "%4d" % (int(L[24-28])+133) + L[28:]
>outp.write(L)
> 
> 
> i wanted to modify lines of the type:
> ATOM  1 HH31 ACE 1   1.573   1.961   0.769  1.00  0.00   H
> 
> to add 133 to column 25, getting 134 there, and so on for next lines 2
> -> 135, 3 -> 136, etc.
> 
> 
> i must have heavily messed things because the file was not even read:
> 
> $ python renumber.py 134-176_rectified.pdb
> Traceback (most recent call last):
>   File "renumber.py", line 6, in 
> L = L[:24] + "%4d" % (int(L[24-28])+133) + L[28:]
> ValueError: invalid literal for int() with base 10: ''

Instead of L[24-28] you want L[24:28]. Otherwise it's L[-4] ;-)

-- Gerhard

--
http://mail.python.org/mailman/listinfo/python-list

Re: value error

[Ben Finney]

Francesco Pietra  writes:

> $ python renumber.py 134-176_rectified.pdb
> Traceback (most recent call last):
>   File "renumber.py", line 6, in 
> L = L[:24] + "%4d" % (int(L[24-28])+133) + L[28:]
> ValueError: invalid literal for int() with base 10: ''

For this reason, it's best to break up big complex expressions like this
into a sequence of simpler statements, so when one of them fails it's
easier to see what went wrong.

In this case, it's because you have L[24-28] where that's almost
certainly not what you mean. It calculates 24-28, getting -4; then uses
that value as an index into L.

You probably wanted to say L[24:28], a slice instead of a single index.

But, really, why are all these magic numbers littering the source,
instead of using named values? You're writing code that will be a
nightmare to maintain.

-- 
 \   “A lot of people are afraid of heights. Not me, I'm afraid of |
  `\   widths.” —Steven Wright |
_o__)  |
Ben Finney
--
http://mail.python.org/mailman/listinfo/python-list

Re: value error

[Jeremiah Dodds]

On Thu, Apr 23, 2009 at 3:59 PM, Francesco Pietra wrote:

> hi:
> with script
>
> data = open('134-176_rectified_edited.pdb', 'r')
> outp = open('134-176_renumbered.pdb', 'w')
>
> for L in data:
>   if L[3] == 'M':
> L = L[:24] + "%4d" % (int(L[24-28])+133) + L[28:]
>   outp.write(L)
>
>
> i wanted to modify lines of the type:
> ATOM  1 HH31 ACE 1   1.573   1.961   0.769  1.00  0.00
>   H
>
> to add 133 to column 25, getting 134 there, and so on for next lines 2
> -> 135, 3 -> 136, etc.
>
>
> i must have heavily messed things because the file was not even read:
>
> $ python renumber.py 134-176_rectified.pdb
> Traceback (most recent call last):
>  File "renumber.py", line 6, in 
>L = L[:24] + "%4d" % (int(L[24-28])+133) + L[28:]
> ValueError: invalid literal for int() with base 10: ''
>
>
>
> thanks for having an expert look
>
> chiendarret
> --
> http://mail.python.org/mailman/listinfo/python-list
>


I wrote this function the other day for something similar I needed to do,
you may find it useful:

def chunk_line(line, steps):
"""Return a list of chunks from a string, with sizes as specified by the
list steps.

>>> line = '1121231234'
>>> steps = [1,2,3,4]
>>> chunk_line(line, steps)
['1', '12', '123', '1234']
"""
result = []
for step in steps:
result.append(line[:step])
line = line[step:]
return result
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why can function definitions only use identifiers, and not attribute references or any other primaries?

[Gary Herron]

Jeremy Banks wrote:

Thanks for your comments.

On Thu, Apr 23, 2009 at 11:52, Gary Herron  wrote:
  

[...]
  

There's no need for a specific addition to the syntax to do this.

Try this:

  def foo_bar():
  return(...)
  foo.bar = foo_bar



[...]
  

and this:

  def foo_bar():
  return(...)
  foo["bar"] = foo_bar




I understand that this is possible now, but I just don't see why it's
necessary to do it at all.

  

In this case I could see there being a problem in that there's nothing
to get the function's __name__ from, but that doesn't apply for the
first example.

  

Not sure what you mean here.



>>> def foo(): pass
...
>>> bar = foo
>>> bar.__name__
'foo'
>>>

If I defined foo.bar it would know that the method name was "bar", but
if I defined foo["bar"] there's be no clear identifier to use for the
function's name. I don't see this as a great problem, since anonymous
functions already exist, but I thought it was worth acknowledging.

To be clear, I don't see this as a serious fault in the language, but
as an unnecessary restriction that makes code a little less direct
than it could be.
  



Things like your suggestion are called "syntactic-sugar"  -- syntax that 
adds a convenience, but *no* new functionality.  Python has plenty of 
"syntactic-sugar"s, and more will be added in the future.  To make an 
argument for such an addition, one would have to describe some 
compelling (and general) use cases in a well-argued PEP.  You're welcome 
to try, but be forewarned, most PEP's (especially syntax changing PEPs) 
don't fly far.




--
http://mail.python.org/mailman/listinfo/python-list
  


--
http://mail.python.org/mailman/listinfo/python-list

Re: Why can function definitions only use identifiers, and not attribute references or any other primaries?

[Jeremy Banks]

> Things like your suggestion are called "syntactic-sugar"  -- syntax that
> adds a convenience, but *no* new functionality.  Python has plenty of
> "syntactic-sugar"s, and more will be added in the future.  To make an
> argument for such an addition, one would have to describe some compelling
> (and general) use cases in a well-argued PEP.  You're welcome to try, but be
> forewarned, most PEP's (especially syntax changing PEPs) don't fly far.

Thank you very much for the feedback. I might throw something at
Python-Ideas if I think I can come up with an adequate justification
and don't come accross a previous similar propsition (though if I do
miss it I'm sure it will be pointed out to me fairly quickly). I fully
appreciate the small chance of success, but it certainly couldn't hurt
to give it a try.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Regular expression to capture model numbers

[Piet van Oostrum]

> John Machin  (JM) wrote:

>JM> On Apr 23, 8:01 am, krishnaposti...@gmail.com wrote:

>>> Requirements:
>>>   The text must contain a combination of numbers, alphabets and hyphen
>>> with at least two of the three elements present.

>JM> Unfortunately(?), regular expressions can't express complicated
>JM> conditions like that.

Yes, they can but it is not pretty.

The pattern must start with a letter, a digit or a hyphen. 

If it starts with a letter, for example, there must be at least a hyphen
or a digit somewhere. So let us concentrate on the first one of these
that occurs in the string. Then the preceding things are only letters
and after it can be any combination of letters, digits and hyphens. So
the pattern for this is (when we write L for letters, and d for digits):

L+[-d][-Ld]*.

Similarly for strings starting with a digit and with a hyphen. Now
replacing L with [A-Za-z] and d with [0-9] or \d and factoring out the
[-Ld]* which is common to all 3 cases you get:

(?:[A-Za-z]+[-0-9]|[0-9]+[-A-Za-z]|-+[0-9A-Za-z])[-0-9A-Za-z]*

>>> obj = 
>>> re.compile(r'(?:[A-Za-z]+[-0-9]|[0-9]+[-A-Za-z]|-+[0-9A-Za-z])[-0-9A-Za-z]*')
>>> re.findall(obj, 'TestThis;1234;Test123AB-x')
['Test123AB-x']

Or you can use re.I and mention only one case of letters:

obj = re.compile(r'(?:[a-z]+[-0-9]|[0-9]+[-a-z]|-+[0-9a-z])[-0-9a-z]*', re.I)
-- 
Piet van Oostrum 
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: p...@vanoostrum.org
--
http://mail.python.org/mailman/listinfo/python-list

Numpy Performance

[timlash]

Still fairly new to Python.  I wrote a program that used a class
called RectangularArray as described here:

class RectangularArray:
   def __init__(self, rows, cols, value=0):
  self.arr = [None]*rows
  self.row = [value]*cols
   def __getitem__(self, (i, j)):
  return (self.arr[i] or self.row)[j]
   def __setitem__(self, (i, j), value):
  if self.arr[i]==None: self.arr[i] = self.row[:]
  self.arr[i][j] = value

This class was found in a 14 year old post:
http://www.python.org/search/hypermail/python-recent/0106.html

This worked great and let me process a few hundred thousand data
points with relative ease.  However, I soon wanted to start sorting
arbitrary portions of my arrays and to transpose others.  I turned to
Numpy rather than reinventing the wheel with custom methods within the
serviceable RectangularArray class.  However, once I refactored with
Numpy I was surprised to find that the execution time for my program
doubled!  I expected a purpose built array module to be more efficient
rather than less.

I'm not doing any linear algebra with my data.  I'm working with
rectangular datasets, evaluating individual rows, grouping, sorting
and summarizing various subsets of rows.

Is a Numpy implementation overkill for my data handling uses?  Should
I evaluate prior array modules such as Numeric or Numarray?  Are there
any other modules suited to handling tabular data?  Would I be best
off expanding the RectangularArray class for the few data
transformation methods I need?

Any guidance or suggestions would be greatly appreciated!

Cheers,

Tim
--
http://mail.python.org/mailman/listinfo/python-list

Re: Using optparse

[Piet van Oostrum]

> loial  (L) wrote:

>L> A shell script is passing parameters to my python script in the
>L> following format

>L> -PARAM1 12345 -PARAM2 67890

>L> Can I parse these with optparse  ? If so how?

>L> I can't seem to get it to work. It seems to expect --PARAM1 and --
>L> PARAM2

See the doc:

Some other option syntaxes that the world has seen include:
[...]
* a hyphen followed by a whole word, e.g. "-file" [...]

These option syntaxes are not supported by optparse, and they never will be.

-- 
Piet van Oostrum 
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: p...@vanoostrum.org
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Packages : A looming problem? packages might no longer work? (well not on your platform or python version anyway)

[Daniel Fetchinson]

> Why? Why should every package on PyPI need to support all those
> Python versions? That should be the decision of the package
> maintainer. If they want to support every version of Python back to
> 1.0, they can, and if they want to only support version 2.5 that's
> fine too.

 Why shouldn't packages support more than one python version?
>>>
>>> They can. That depends on the developer of the package.
>>
>> And if the developer decides to support multiple versions currently
>> there is no infrastructural help that would be available to him. On the
>> other hand, there is infrastructural help available to every developer
>> for distributing their apps: it's PyPI itself. You could ask, why do we
>> need PyPI? Let the developers distribute their code in any way they
>> like, python.org doesn't need to support this.
>
> They don't *need* to support distribution. That they do is a nice-to-
> have, not a must-have -- there are other solutions, like SourceForge. We
> can be grateful for PyPI without believing that it's a must-have.

Nobody believes PyPI is a must-have and for sure I didn't say it is a
must-have. PyPI is nice to have not only for developers but also for
users. A multiplatform/multiversion test farm would also be nice for
developers and users alike.

>> The OP is just thinking out loud that it would be great if developers
>> could count on some help for testing various platforms and versions. And
>> I agree, it would indeed be great.
>
> To me, the OP doesn't seem to be asking "Wouldn't it be good if
> developers could get access to whatever systems they wanted?" and more
> like "Somebody needs to develop this amazing technology that will just
> magically make every package on PyPI work on all these different
> platforms, otherwise the Sky Will Fall". Just look at the subject line:
> the issue is described as "a looming problem", it's suggested that
> packages "might no longer work".

You and I probably have a different approach to posts to c.l.p. I try
to interpret things in the best possible light and get the most out of
a suggestion. You are right, the OP was not formulating his proposal
clearly and didn't use a sophisticated enough language but what is
clear is that at the core of the proposal there is something valuable.
Having a test farm would be very useful and would benefit the python
community.

> He's also worried about the combinational explosion of packages,
> platforms and versions, which to my mind is only a problem if you think
> that every package must work on every platform with every version of
> Python.

I don't think every package should work on every platform and with
every version of python. But I do think that many developers want to
support more platforms and versions than what they have access to.
Having a test farm would be beneficial to these developers and their
users.

> I suspect you're primed to read the OP's suggestion in the most positive
> light, because you're already aware of snakebite, and so you're seeing
> his posts in terms of what snakebite is offering. I had never even heard
> of snakebite before now, so I was seeing it in a very different light.

I swear I didn't read this paragraph when I wrote above that I usually
try to interpret posts in the best possible light! This is actually
very funny, you used the exact same phrase :) But even if you have
never heard of snakebite it is simply beyond my understanding how a
developer can argue against a testing farm like this. You seriously
think it would not be a good idea?


> What's the dire problem you are trying to solve?

 Backward and forward compatability of python package resources.
>>>
>>> That's not a problem, that's a feature.
>>>
>>> What's the *problem* that you are trying to solve? What bad thing will
>>> happen if we don't build your proposed system?
>>
>> I fail to understand what is so mysterious about this. The problem is
>> the following: developer X wants to support python versions a, b, c, d
>> on platforms U, V, W. Developer X has no access to platforms V and W and
>> also has no access to python versions b, c and d. Developer X can not
>> run tests on these platforms and python versions although he really
>> wants to. This situation is what is commonly referred to as a problem.
>
> That's *a* problem.

Yes, a problem, that is why I wrote, a problem.

> It has many solutions, snakebite being just one.

Yes, something integrated into PyPI would be another one.

> Is  it the "looming" problem the OP is concerned about?

What are we worried about, the use of the English language or the
technical merit of a proposal?

>> This testing infrastructure should be available to developers who choose
>> to opt in. Once they run their code and notice that there are errors,
>> they will go back and fix these errors that otherwise would be
>> undetected until a real person runs into them. Or they could clearly
>> state what platform and python versions the code runs

Re: Python Packages : A looming problem? packages might no longer work? (well not on your platform or python version anyway)

[Daniel Fetchinson]

>> The OP is just thinking out loud that it would be great if developers
>> could count on some help for testing various platforms and versions.
>> And I agree, it would indeed be great.
>>
>
> I think you interpreted the OP differently. As I said before the idea
> is not a bad one, but as a package developer I get to decide which
> platforms and versions my code supports.

Yes, I also think the original idea of the OP is a good one. And I
agree with you that the developers need to be in charge what
platform/version their code supports. Trouble is that currently if
they decide to support multiple versions and platform they don't have
any help from python.org or PyPI. It would be great if after a
developer decided to support some platforms and some versions there
was some infrastructural help in place.

Cheers,
Daniel


-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown
--
http://mail.python.org/mailman/listinfo/python-list

Re: OT: a metacomment on feedback comments

[Johannes Bauer]

Aaron Watters schrieb:

> This was posted just after someone on the same
> computer (somewhere in Texas)
> tried and failed to inject some javascript
> into the page using a comment.

Script-kid moron. Delete and enjoy your life.

> I don't see how this comment is helpful to
> me or anyone reading the page.  How the
> framework confusing?  Why should other readers dismiss
> the project?  Can the documentation
> or implementation be
> improved, or is the whole thing beyond repair
> and if so how or why?  Why do people bother
> to write comments like this one?

Humm... I just checked your framework out and must say that I, too,
found it a little confusing. Especially the meaning of "use-url" seems
awkward since obviously this means more like "use-middleware or
"use-plugin" or "use-module" or even just plain "use" (which would be
IMHO clearer).

When you say "use-url" I first though "Ok, this part of the document
will be shown when the user surfs at the supplied URL". Which is not
what it does.

I also do not understand what the difference between a section and a
use-url ist. It seems "section" just creates a new paragraph and
"use-url" invokes some plugin behaviour - but I'm not sure.

I'm not sure why you chose to use the {{ }} syntax, but I'm guessing
that when you'd have used XML it would be a pain to include manual HTML
formatting - again I am unsure.

To sum it up, I think it's a neat idea, but it's not really intuitive.
After being quite startled at first it got better after I read the "why
it's cool" page.

Kind regards,
Johannes

-- 
"Meine Gegenklage gegen dich lautet dann auf bewusste Verlogenheit,
verlästerung von Gott, Bibel und mir und bewusster Blasphemie."
 -- Prophet und Visionär Hans Joss aka HJP in de.sci.physik
 <48d8bf1d$0$7510$54022...@news.sunrise.ch>
--
http://mail.python.org/mailman/listinfo/python-list

Re: Configuring pylint for local conventions (was: pyflakes, pylint, pychecker - and other tools)

[Aahz]

In article <874owf4gky.fsf...@benfinney.id.au>,
Ben Finney   wrote:
>a...@pythoncraft.com (Aahz) writes:
>>
>> Second, you can configure pylint to respect your personal style
>
>How? I haven't seen any decent documentation on doing so.

Actually, I don't know how, I'm just repeating what was claimed at a
presentation on pylint.  ;-)  I've traditionally used pychecker myself
and haven't seen any reason to switch.
-- 
Aahz (a...@pythoncraft.com)   <*> http://www.pythoncraft.com/

"If you think it's expensive to hire a professional to do the job, wait
until you hire an amateur."  --Red Adair
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why can function definitions only use identifiers, and not attribute references or any other primaries?

[Jeremy Banks]

On Thu, Apr 23, 2009 at 13:03, John Krukoff  wrote:
> You probably want to be searching for multi-line lambda to find the past
> decade or so of this argument, as that's where most people who argued
> for this came from. But, if you'd just like a bit of discouragement,
> here's GvR arguing that there's just no good way to mix statements and
> expressions in python:
> http://www.artima.com/weblogs/viewpost.jsp?thread=147358

I've read those discussion before, but somehow never made the
connection between those and this. I'll give that article a read, it
probably details exactly the perspective I'm looking for. Thank you!
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why can function definitions only use identifiers, and not attribute references or any other primaries?

[John Krukoff]

On Thu, 2009-04-23 at 12:26 -0300, Jeremy Banks wrote:
> > Things like your suggestion are called "syntactic-sugar"  -- syntax that
> > adds a convenience, but *no* new functionality.  Python has plenty of
> > "syntactic-sugar"s, and more will be added in the future.  To make an
> > argument for such an addition, one would have to describe some compelling
> > (and general) use cases in a well-argued PEP.  You're welcome to try, but be
> > forewarned, most PEP's (especially syntax changing PEPs) don't fly far.
> 
> Thank you very much for the feedback. I might throw something at
> Python-Ideas if I think I can come up with an adequate justification
> and don't come accross a previous similar propsition (though if I do
> miss it I'm sure it will be pointed out to me fairly quickly). I fully
> appreciate the small chance of success, but it certainly couldn't hurt
> to give it a try.
> --
> http://mail.python.org/mailman/listinfo/python-list

You probably want to be searching for multi-line lambda to find the past
decade or so of this argument, as that's where most people who argued
for this came from. But, if you'd just like a bit of discouragement,
here's GvR arguing that there's just no good way to mix statements and
expressions in python:
http://www.artima.com/weblogs/viewpost.jsp?thread=147358

-- 
John Krukoff 
Land Title Guarantee Company

--
http://mail.python.org/mailman/listinfo/python-list

Re: Numpy Performance

[Peter Otten]

timlash wrote:

> Still fairly new to Python.  I wrote a program that used a class
> called RectangularArray as described here:
> 
> class RectangularArray:
>def __init__(self, rows, cols, value=0):
>   self.arr = [None]*rows
>   self.row = [value]*cols
>def __getitem__(self, (i, j)):
>   return (self.arr[i] or self.row)[j]
>def __setitem__(self, (i, j), value):
>   if self.arr[i]==None: self.arr[i] = self.row[:]
>   self.arr[i][j] = value
> 
> This class was found in a 14 year old post:
> http://www.python.org/search/hypermail/python-recent/0106.html
> 
> This worked great and let me process a few hundred thousand data
> points with relative ease.  However, I soon wanted to start sorting
> arbitrary portions of my arrays and to transpose others.  I turned to
> Numpy rather than reinventing the wheel with custom methods within the
> serviceable RectangularArray class.  However, once I refactored with
> Numpy I was surprised to find that the execution time for my program
> doubled!  I expected a purpose built array module to be more efficient
> rather than less.
> 
> I'm not doing any linear algebra with my data.  I'm working with
> rectangular datasets, evaluating individual rows, grouping, sorting
> and summarizing various subsets of rows.
> 
> Is a Numpy implementation overkill for my data handling uses?  Should
> I evaluate prior array modules such as Numeric or Numarray?  Are there
> any other modules suited to handling tabular data?  Would I be best
> off expanding the RectangularArray class for the few data
> transformation methods I need?
> 
> Any guidance or suggestions would be greatly appreciated!

Do you have many rows with zeros? That might be the reason why your
self-made approach shows better performance.

Googling for "numpy sparse" finds:

http://www.scipy.org/SciPy_Tutorial

Maybe one of the sparse matrix implementations in scipy works for you.

Peter
--
http://mail.python.org/mailman/listinfo/python-list

Re: Supply a plugin interface

[Aahz]

In article <75akp8f17g29...@mid.dfncis.de>,
Johannes Bauer   wrote:
>
>the GUI application should now browse the plugin directory and read
>those plugin python files and somehow incorporate (i.e. discover what
>modules are there, instanciate, etc.)
>
>How do I do that at runtime with Python?

You might consider using execfile() but probably __import__ works better
for your purposes.  There's also importlib on PyPI, which is the backport
from 2.7.
-- 
Aahz (a...@pythoncraft.com)   <*> http://www.pythoncraft.com/

"If you think it's expensive to hire a professional to do the job, wait
until you hire an amateur."  --Red Adair
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why can function definitions only use identifiers, and not attribute references or any other primaries?

[Marco Mariani]

Jeremy Banks wrote:


I've read those discussion before, but somehow never made the
connection between those and this. I'll give that article a read, it
probably details exactly the perspective I'm looking for. Thank you!


You could also read this:

http://unlimitednovelty.com/2009/03/indentation-sensitivity-post-mortem.html

The author is writing a language for the Erlang VM inspired by Python 
and Ruby. He had some trouble (at the grammar level) in keeping both 
"indentation working like in python" (dear to Guido and many of us) and 
"anonymous blocks" (dear to functional languages).

So he got braces and was happy :-)

--
http://mail.python.org/mailman/listinfo/python-list

Re: pylab quick reference? (matplotlib)

[Esmail]

Arnaud Delobelle wrote:




 python -c 'import pylab;  help(pylab)' > pylab.txt


Do you know pydoc?


From the (shell) command line, try:


$ pydoc pylab # piped through less
$ pydoc pylab > pylab.txt # stored in a file
$ pydoc -w pylab  # Creates a 'pylab.html' file
wrote pylab.html


Ah .. and there I thought I'd impress everyone with my python
command line :-)

I had heard of pydoc before, but am not familiar with (it's on
my list of Python tools to explore), so many thanks for showing
me how to get a nice html formatted page.

Esmail
--
http://mail.python.org/mailman/listinfo/python-list

Re: Essential tools for Python development

[Esmail]

Jeremiah Dodds wrote:


pdb is good if you need to do step-through debugging.

What I normally do in emacs is the following (while working on python 
script, and with the python-mode that comes with emacs22):


C-x 3  #splits the screen into two vertical columns
C-c C-c   #fires up python and sends the current buffer to it
C-x o  #switches to the other column
C-x b Py #switches to the python interactive buffer

This is, I do believe, equivalent to running a script with python -i 
script.py . Drops you into an interactive session where you can interact 
with what you just wrote. Another very handy tool is etags, see the help 
in emacs (You can get to it through C-h i m emacs)


Thanks for the suggestions, I really like using emacs, so I am going to
see how I can integrate it with Python development.
(Unfortunately my Windows version of emacs dosen't seem to be able to
pop into python mode, and I haven't had time to find out why). At this
point I do most of my development under Linux/emacs anyway.

Best,
Esmail
--
http://mail.python.org/mailman/listinfo/python-list

Re: Essential tools for Python development

[Esmail]

Alex VanderWoude wrote:


I really like using Winpdb (http://winpdb.org/) which provides a GUI for 
pdb.  It allows you to single-step your code and inspect objects in 
memory.  What exactly is in that silly nextOne variable?  Hey, it's a 
list rather but it's supposed to be a number!  You get the picture.


thanks for the suggestion.

The name made me think that this would be only for the Windows
platform, but it turns out it's multi-platform.

I downloaded the Linux version, but haven't had time to explore it
yet, but plan to.

Esmail

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Packages : A looming problem? packages might no longer work? (well not on your platform or python version anyway)

[Paul Boddie]

On 23 Apr, 17:46, Daniel Fetchinson  wrote:
>
> You and I probably have a different approach to posts to c.l.p. I try
> to interpret things in the best possible light and get the most out of
> a suggestion.

There may be merit in the suggestion, but there also has to be
skepticism, even if it is regarded as "stop energy" in certain
circles.

[...]

> That is why the proposal is an important one. Let's say snakebite is
> not suitable for PyPI. What can be done on PyPI that helps developers
> with multiplatform/multiversion testing?

I think that PyPI plays an important role in publicising packages and
getting them out there in their "raw" form, but it's not the centre of
the universe. Maybe the answer doesn't involve PyPI at all or, more
likely, involves PyPI as a peripheral thing from which people acquire
those packages that have to work together on a specific platform. Such
a solution wouldn't have that much to do with PyPI, other than letting
people download large collections of packages very quickly,
potentially from mirror sites, which is something that various GNU/
Linux distribution services permit (and something which seems to have
been a struggle with PyPI).

Paul
--
http://mail.python.org/mailman/listinfo/python-list

Re: sorting two corresponding lists?

[Esmail]

My solution, I know the 'zip' version is more elegant, this is just for 
fun:)
 
 >>> items = ['apple', 'car', 'town', 'phone']

 >>> values = [5, 2, 7, 1]
 >>> new_values = sorted(values, reverse = True)
 >>> new_items = [items[x] for x in [i for i in map(values.index, 
new_values)]]

 >>> print(new_values)
[7, 5, 2, 1]
 >>> print(new_items)
['town', 'apple', 'car', 'phone']
 >>>
 


Cool .. always good to know alternative ways of accomplishing
a task.

Esmail
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Packages : A looming problem? packages might no longer work? (well not on your platform or python version anyway)

[Scott David Daniels]

Daniel Fetchinson wrote:

I don't think every package should work on every platform and with
every version of python. But I do think that many developers want to
support more platforms and versions than what they have access to.
Having a test farm would be beneficial to these developers and their
users.


This testing infrastructure should be available to developers who choose
to opt in. Once they run their code and notice that there are errors,
they will go back and fix these errors that otherwise would be
undetected until a real person runs into them. Or they could clearly
state what platform and python versions the code runs on for sure and on
what platforms and python versions the code fails. This would be a great
advantage to developers.

There is a cost here, however.  Core developers and large-userbase packages
like scipy or numpy are self-policing.  However, if you open machine
resources to every Python Package, you will have to police the resource
to make sure that you have provided a platform for malware distributors.
Now few of us think we want to provide that, but very few of us want to
spend the hours reading logs and ... to make sure that is not happening.


--Scott David Daniels
scott.dani...@acm.org
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Packages : A looming problem? packages might no longer work? (well not on your platform or python version anyway)

[norseman]

David Lyon wrote:


On Thu, 23 Apr 2009 07:04:35 -0400, David Stanek 
wrote:

If I use win32com how do you expect me to support Linux? 


Of course not...

What about the many packages on PYPI containing C? 


Exactly. 


What if I decide to write only to Python 3?


Fair enough. But don't forget it is open source.

Let me ask these two questions...

 - What about the use case where somebody likes the code and wants 
   to use it on Python 2.5?


 - Should not that user be able to share back with other 
   Python 2.5 users?



Who will support the other platforms if not the developer?


It's Open Source don't forget

Fact is these days.. developers come and go

If anything my suggestion promotes preserving the resources
of the original developer rather than letting them expire just
because their operating system does

(I'm talking windows here)


Go David!  I like your attitude!!



David


--
http://mail.python.org/mailman/listinfo/python-list


==

This topic has been going round and round and yet nobody has actually 
touched the "magic button".


Like whoever started this topic (and the rest of you) I have logged in 
somewhere and looked for Help, Snippets and more.  I find something I 
like and download and it fails miserably.  Thus the perceived "need" for 
some big brother certification thing.


It is far more simple to police ourselves.
The posting needs a DATE. The Help or Comment needs to explicitly 
specify the VERSION(s) it addresses.  The code needs to state OS and 
program and version used to write it.  And from there - user beware.


DATE is supplied for date of posting, but who says that was when the 
contents were created?  The content needs to specify its creation DATE.


VERSION(s) which HELP and COMMENTS were created under/towards/"at time 
of" need explicit noting. Even one liners in postings.


The author of the code needs to state OS (and version of) and the 
compiler/interpreter (and version of) and version the code was written 
to. (was it for a specific only or was the effort to be more general or 
was backward compatibility attempted, is the snippet even supposed to 
run?) As for programming language version compatibility - you the author 
need to download them and do that yourself if you need to know.


There is one other thing that really pisses me off when it's missing.
The idiot puts in the disclaimer but never tells anyone what the hell 
that damn thing is supposed to do.  AAARGGGghhh !!!


BB's, User Lists, all repositories can make these required before 
acceptance.



On the dream list:  It would be nice to have access to an OS (and or 
machine architecture) not in possession for testing compatibility in 
cases where it's an issue.

But then just asking usually gets it done. :)

Today is: 20090423
Steve
--
http://mail.python.org/mailman/listinfo/python-list

Re: Large data arrays?

[Scott David Daniels]

Ole Streicher wrote:

Nick Craig-Wood  writes:

mmaps come out of your applications memory space, so out of that 3 GB
limit.  You don't need that much RAM of course but it does use up
address space.


Hmm. So I have no chance to use >= 2 of these arrays simultaniously?
... That would be a solution, but it takes twice the amount of address
space (which seems already to be the limiting factor). In my case (1.6
GB per array), I could even not use one array. 


Looks like you are out of luck here.  I believe looking to do this
without managing the I/O yourself is likely to involve someone else's
trade-offs, which likely fail to match your use case.  You might check
gmane.comp.python.scientific.user
which I think is gmane's  mirror of comp.python.scipy.user, but I'm
not certain of the mailing list name.  Certainly the people over there
regularly deal with accessing large data.

--Scott David Daniels
scott.dani...@acm.org
--
http://mail.python.org/mailman/listinfo/python-list

Windows Shell Extensions - Out of Proc

[Ralf]

I'm trying to develop Windows Shell Extensions with the Python Win32
Extensions. Most of the samples are working. However, I have a
slightly different need: I want the Python COM server to run as a
separate process ("LocalServer" or "OutOfProc").

As I understand, both the InProc and LocalServer versions are created
automatically. So to disable the InProc version, I removed the
InProc32 key from the registry, for the context menu example
distributed with the Python Win32 Extensions. However, this removes
the context menu from Explorer, even though the LocalServer32 key
remains. I could access the COM objects in the example with a Python
client, so I have no idea why Explorer can't use the LocalServer
version.

Can anyone help with this? I've Google'd for hours, without much
success. I couldn't even find a good example for any other language,
only references saying it's possible for C#.
--
http://mail.python.org/mailman/listinfo/python-list

Re: JOB: Statistical Programmer | LOCATION: London, England, UK

[Scott David Daniels]

jt wrote:

JOB: Statistical Programmer  ...

Job postings are not in order here, they are off-topic.
If you go to
http://www.python.org/community/jobs/
You will see a jobs posting board that will happily post your job.

--Scott David Daniels
scott.dani...@acm.org
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why can function definitions only use identifiers, and not attribute references or any other primaries?

[Scott David Daniels]

Jeremy Banks wrote: (proposing def (): ...

On Thu, Apr 23, 2009 at 11:52, Gary Herron  wrote:

Try this:
  def foo_bar():
  return(...)
  foo.bar = foo_bar
and this:
  def foo_bar():
  return(...)
  foo["bar"] = foo_bar

I understand that this is possible now, but I just don't see why it's
necessary to do it at all.


I am afraid it will make it too easy to define functions in other
modules remotely, a tempting sharp stick to poke your eye out with.
Note also, that it will not be so easy to find the definition of a
function provided as a argument to a failing function.  Right now
you can get the function name and (with a bit more effort) its module.
Imagine debugging a pile of code that includes a module with:
import random
def random.random():
return .42

--Scott David Daniels
scott.dani...@acm.org
--
http://mail.python.org/mailman/listinfo/python-list

Presentation software for Python code

[Michael Hoffman]

I mean to give a presentation next week to my workgroup on good Python 
programming practice. Last time I did a Python presentation I used 
PowerPoint which was not totally well-suited for the task. In 
particular, formatting code snippets was a pain, and they weren't even 
prettyprinted.


Does anyone here have software they would suggest for making a 
presentation that includes Python code? Other than that it would 
probably be mainly bullet points. I'm willing to consider TeX- and 
HTML-based approaches.

--
Michael Hoffman
--
http://mail.python.org/mailman/listinfo/python-list

Re: Windows Shell Extensions - Out of Proc

[Thomas Heller]

Ralf schrieb:
> I'm trying to develop Windows Shell Extensions with the Python Win32
> Extensions. Most of the samples are working. However, I have a
> slightly different need: I want the Python COM server to run as a
> separate process ("LocalServer" or "OutOfProc").
> 
> As I understand, both the InProc and LocalServer versions are created
> automatically. So to disable the InProc version, I removed the
> InProc32 key from the registry, for the context menu example
> distributed with the Python Win32 Extensions. However, this removes
> the context menu from Explorer, even though the LocalServer32 key
> remains. I could access the COM objects in the example with a Python
> client, so I have no idea why Explorer can't use the LocalServer
> version.
> 
> Can anyone help with this? I've Google'd for hours, without much
> success. I couldn't even find a good example for any other language,
> only references saying it's possible for C#.

I think that for whatever reasons, explorer always tries to create
shell extensions as InProc.  CoCreateInstance, which is the usual
API to create COM instances, allows to specify which one you want.

Thomas
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Packages : A looming problem? packages might no longer work? (well not on your platform or python version anyway)

[David Stanek]

On Thu, Apr 23, 2009 at 1:12 PM, norseman  wrote:
>
> BB's, User Lists, all repositories can make these required before
> acceptance.
>
>

This is open source. I volunteer my time on the projects that I
maintain. If you don't like the quality or lack of documentations,
tests, etc. Contribute. Or just don't use the software.

What maybe another option is to have a karma rating on PYPI. Maybe
based off of home much you are included in other packages or some
other factors.

-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek
--
http://mail.python.org/mailman/listinfo/python-list

Re: Presentation software for Python code

[John Reid]

Michael Hoffman wrote:
Does anyone here have software they would suggest for making a 
presentation that includes Python code? Other than that it would 
probably be mainly bullet points. I'm willing to consider TeX- and 
HTML-based approaches.


I like pygments for formatting python code. It can generate TeX and you 
can use it inline it in your TeX source if you like.


--
http://mail.python.org/mailman/listinfo/python-list

Re: Presentation software for Python code

[Scott David Daniels]

Michael Hoffman wrote:
... Does anyone here have software they would suggest for making a 
presentation that includes Python code? Other than that it would 
probably be mainly bullet points. I'm willing to consider TeX- and 
HTML-based approaches.


You might take a look at Crunchy, and just do up your talk there.
Crunchy is a Python program that combines an otherwise static html
document with an interactive Python session within a browser
http://code.google.com/p/crunchy/

A demo from an earlier version of Crunchy  (when thedemo was made,
Firfox only). The basic features of that early version of Crunchy
are demonstrated in this screencast.

http://showmedo.com/videos/video?name=143&fromSeriesID=143

--Scott David Daniels
scott.dani...@acm.org
--
http://mail.python.org/mailman/listinfo/python-list