Re: Print message with Colors

[Laurent Pointal]

prk a écrit :
> Hi Folks,
> 
> Is there any procesure for print messages with colors.

Yes, see:

http://www.google.fr/search?q=python+print+color
http://groups.google.fr/groups?as_q=python+print+color&num=100&as_ugroup=comp.lang.python


See also ANSI escape sequences for the more general subject of color
printing on terminals.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Newbie - converting csv files to arrays in NumPy - Matlab vs. Numpy comparison

[John Machin]

sturlamolden wrote:
> oyekomova wrote:
> > Thanks for your help. I compared the following code in NumPy with the
> > csvread in Matlab for a very large csv file. Matlab read the file in
> > 577 seconds. On the other hand, this code below kept running for over 2
> > hours. Can this program be made more efficient? FYI - The csv file was
> > a simple 6 column file with a header row and more than a million
> > records.
> >
> >
> > import csv
> > from numpy import array
> > import time
> > t1=time.clock()
> > file_to_read = file('somename.csv','r')
> > read_from = csv.reader(file_to_read)
> > read_from.next()
>
> > datalist = [ map(float, row[:]) for row in read_from ]
>
> I'm willing to bet that this is your problem. Python lists are arrays
> under the hood!
>
> Try something like this instead:
>
>
> # read the whole file in one chunk
> lines = file_to_read.readlines()
> # count the number of columns
> n = 1
> for c in lines[1]:
>if c == ',': n += 1
> # count the number of rows
> m = len(lines[1:])

Please consider using
m = len(lines) - 1

> #allocate
> data = empty((m,n),dtype=float)
> # create csv reader, skip header
> reader = csv.reader(lines[1:])

lines[1:] again?
The OP set you an example:
read_from.next()
so you could use:
reader = csv.reader(lines)
_unused = reader.next()

> # read
> for i in arange(0,m):
>data[i,:] = map(float,reader.next())
>
> And if this is too slow, you may consider vectorizing the last loop:
>
> data = empty((m,n),dtype=float)
> newstr = ",".join(lines[1:])
> flatdata = data.reshape((n*m)) # flatdata is a view of data, not a copy
> reader = csv.reader([newstr])
> flatdata[:] = map(float,reader.next())
> 
> I hope this helps!

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Read from database, write to another database, simultaneously

[Laurent Pointal]

Bjoern Schliessmann a écrit :
> Sean Davis wrote:
> 
>> The author of one of the python database clients mentioned that
>> using one thread to retrieve the data from the oracle database and
>> another to insert the data into postgresql with something like a
>> pipe between the two threads might make sense, keeping both IO
>> streams busy.
> 
> IMHO he's wrong. Network interaction is quite slow compared with CPU
> performance, so there's no gain (maybe even overhead due to thread
> management and locking stuff). That's true even on multiprocessor
> machines, not only because there's almost nothing to compute but
> only IO traffic. CMIIW.


Not so sure, there is low CPU in the Python script, but there may be
CPU+disk activity on the database sides [with cache management and other
optimizations on disk access].
So, with a reader thread and a writer thread, he can have a select on a
database performed in parallel with an insert on the other database.
After, he must know if the two databases use same disks, same
controller, same host... or not.

But, if its only a do-once job, maybe the optimization is net really
necessary.
-- 
http://mail.python.org/mailman/listinfo/python-list

Antwort: Re: SubProcess _make_inheritable

[Roland Puntaier]

Thanks for pointing me to the tracker.
I've seen there is already an entry for this:
[ 1603907 ] subprocess: error redirecting i/o from non-console process 

Roland

Gabriel Genellina <[EMAIL PROTECTED]> schrieb am 11.01.2007 05:24:03:

> At Wednesday 10/1/2007 13:10, Roland Puntaier wrote:
> 
> >SubProcess.py needs to be patched - at least in 2.4 -  to work from
> >windows GUIs:
> >
> > def _make_inheritable(self, handle):
> > """Return a duplicate of handle, which is inheritable"""
> > if handle==None: handle=-1
> > return DuplicateHandle(GetCurrentProcess(), handle,
> >GetCurrentProcess(), 0, 1,
> >DUPLICATE_SAME_ACCESS)
> 
> You should submit it to the tracker, not post here and hope someone 
> would notice...
> http://sourceforge.net/tracker/?group_id=5470
> 
> Anyway, I don't see in which case would handle be None.
> 
> 
> -- 
> Gabriel Genellina
> Softlab SRL 
> 
> 
> 
> 
> 
> 
> __ 
> Preguntá. Respondé. Descubrí. 
> Todo lo que querías saber, y lo que ni imaginabas, 
> está en Yahoo! Respuestas (Beta). 
> ¡Probalo ya! 
> http://www.yahoo.com.ar/respuestas 
> 
> 
> 

-- 
http://mail.python.org/mailman/listinfo/python-list

RE: dynamic library loading, missing symbols

[Ames Andreas]

> -Original Message-
> From: [EMAIL PROTECTED] 
> [mailto:[EMAIL PROTECTED]
> g] On Behalf Of [EMAIL PROTECTED]
> Sent: Wednesday, January 10, 2007 8:52 PM
> Subject: Re: dynamic library loading, missing symbols
> 
> I suppose this means that any subsequent libraries dlopened will not
> see any of the symbols in my module?

Have you already checked the output of LD_DEBUG=all or sth. like that?


cheers,

aa

-- 
Andreas Ames | Programmer | Comergo GmbH |
Voice:  +49 711 13586 7789 | ames AT avaya DOT com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The Python Papers: Submit your Quotes

[cyberco]

"Like silence in music, whitespace is where Python Power shows"

"The odd thing is that Python results in what I call YoYo-code. After
writing some code I always discover a shorter, more elegant and more
readable way of doing the same thing in Python. The same happens after
adding more functionality. This goes on indefinitely, resulting in a
piece of code whose length YoYo's over time"

"Python's power is that it is fitted for both simple and complex tasks.
You're not bothered with the restrictions that are only relevant when
building complex systems if you just want to do something simple" -
Berco Beute

"Jack of all trades, master as well"

- Berco Beute

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Need startup suggestions for writing a MSA viewer GUI in python

[Joel Hedlund]

> This will probably be a major, but not humongous project. wxPython,
> pyGTk, and pyQt all have the architecture and basics you'll need, it
> will probably be about the same amount of work to create in all of
> them. Pick the one that best suites your licensing and platform needs.

Thanks for the suggestions. Now I'll start my reading up.

Cheers!
/Joel
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: maximum number of threads

[William Heymann]

On Wednesday 10 January 2007 7:11 am, Felipe Almeida Lessa wrote:
> ---
> $ python test.py
> 50
> 100
> 150
> 200
> 250
> 300
> 350
> Exception raised: can't start new thread
>
> Biggest number of threads: 382
> ---
>
> The test.py script is attached.

So you know I tried this on ubuntu edgy 64bit edition on a dual 2218 opteron 
system with 8G of ram and I got


Exception raised: can't start new thread
Biggest number of threads: 32274
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Using Excel With Python

[david brochu jr]

Try using ExcelApp.Close(). This should kill the entire application (it
might prompt for a Save).
-- 
http://mail.python.org/mailman/listinfo/python-list

Seattle Python Interest Group Thursday at 7:00 PM

[jamesthiele . usenet]

Seattle Python Interest Group Meeting Thursday, Jan 11th at 7:00 PM

Bar underneath the Third Place Books in Ravenna.
http://www.ravennathirdplace.com/
NE 65th St & 20th Ave NE

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a way to protect a piece of critical code?

[robert]

Hendrik van Rooyen wrote:
> "robert" <[EMAIL PROTECTED]> wrote:
> 
>> List .append() and .pop() will be atomic in any Python though its not
> mentioned explicitely - otherwise it would be time to leave Python.
>> There is also Queue.Queue - though it has unneccessary overhead for most
> purposes.
>>
> am aware of Queue module - the same app uses it for something else.
> I dont like too many try -- excepts in the code - I find they confuse
> me when I try to read it later - and in this case I cannot block on waiting 
> for
> the queue to fill.


pushing data objects through an inter-thread queue is a major source for 
trouble - as this thread shows again.
Everybody builds up a new protocol and worries about Empty/Full, 
Exception-handling/passing, None-Elements, ...
I've noticed that those troubles disappear when a functional queue is used - 
which is very easy with a functional language like Python.
For example with http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/491281


One would just use a  cq=CallQueue()

On the producer side one would just write the functional code one wants to 
execute in a target thread:

cq.call( what_i_want_do_func )


The consumer/receiver thread would just do (periodically) a non-blocking

cq.receive()


=> Without any object fumbling, protocol worries and very fast.

And note: This way - working with functional jobs - one can also "protect a 
piece of critical code" most naturally and specifically for certain threads 
without spreading locks throughout the code.
Even things which are commonly claimed "forbidden" (even when using lots of 
locks) can be magically done in perfect order and effectively this way. Think 
of worker threads doing things in the GUI or in master / database owner threads 
etc.

Similarly discrete background thread jobs can be used in a functional style 
this way:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/491280
( an alternative for the laborious OO-centric threading.Thread which mostly is 
a lazy copy from Java )
or for higher job frequencies by using "default consumer threads" as also shown 
in the 1st example of 
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/491281



Robert
-- 
http://mail.python.org/mailman/listinfo/python-list

os.popen() not executing command on windows xp

[nic]

On my system (WinXP) typing the following line into command
prompt(cmd.exe) successfully scans the file test1.txt:
"c:\Program Files\Grisoft\AVG Free\avgscan.exe" "c:\program
files\temp1\test1.txt"

Yet the python script:
import os
a = os.popen('"c:\Program Files\Grisoft\AVG Free\avgscan.exe"
"c:\program files\temp1\test1.txt"')
print a.read()

Returns a blank line, and I doesn't scan the file. (Note I've used
os.popen() successfully on my system in other situations like:
os.popen('taskkill /F /IM taskname.exe')).

I have a feeling my avgscan example shown above not working has
something to do with calling a 3rd party software program (avgscan.exe)
through popen, but I don't know why this won't work, when manually
typing it in the command line does?

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: os.popen() not executing command on windows xp

[Gabriel Genellina]

At Thursday 11/1/2007 06:42, nic wrote:


a = os.popen('"c:\Program Files\Grisoft\AVG Free\avgscan.exe"
"c:\program files\temp1\test1.txt"')


Your string contains backquotes, and they have to be escaped.
Either use a raw string: os.popen(r'"c:\Program...) or double all 
backquotes: os.popen('"c:\\Program...)



--
Gabriel Genellina
Softlab SRL 







__ 
Preguntá. Respondé. Descubrí. 
Todo lo que querías saber, y lo que ni imaginabas, 
está en Yahoo! Respuestas (Beta). 
¡Probalo ya! 
http://www.yahoo.com.ar/respuestas 

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: os.popen() not executing command on windows xp

[Justin Ezequiel]

> import os
> a = os.popen('"c:\Program Files\Grisoft\AVG Free\avgscan.exe"
> "c:\program files\temp1\test1.txt"')
> print a.read()
>

use raw strings

e.g., instead of '"c:...\avgscan...'
use r'"c:...\avgscan...'

http://www.ferg.org/projects/python_gotchas.html#contents_item_2

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python

[robert]

sturlamolden wrote:
> Nick Maclaren wrote:
> 
> I wonder if too much emphasis is put on thread programming these days.
> Threads may be nice for programming web servers and the like, but not
> for numerical computing. Reading books about thread programming, one
> can easily get the impression that it is 'the' way to parallelize
> numerical tasks on computers with multiple CPUs (or multiple CPU


Most threads on this planet are not used for number crunching jobs, but for 
"organization of execution".

Also if one wants to exploit the speed of upcoming multi-core CPUs for all 
kinds of fine grained programs, things need fast fine grained communication - 
and most important: huge data trees in memory have to be shared effectively.
CPU frequencies will not grow anymore in the future, but we will see 
multi-cores/SMP. How to exploit them in a manner as if we had really faster 
CPU's: threads and thread-like techniques.

Things like MPI, IPC are just for the area of "small message, big job" - 
typically sci number crunching, where you collect the results "at the end of 
day". Its more a slow network technique.

A most challenging example on this are probably games - not to discuss about 
gaming here, but as tech example to the point: Would you do MPI, RPC etc. while 
30fps 3D and real time physics simulation is going on?


Robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python

[robert]

Nick Maclaren wrote:
> In article <[EMAIL PROTECTED]>,
> Paul Rubin  writes:
> |>
> |> > Yes, I know that it is a bit Irish for the best way to use a shared
> |> > memory system to be to not share memory, but that's how it is.
> |> 
> |> But I thought serious MPI implementations use shared memory if they
> |> can.  That's the beauty of it, you can run your application on SMP
> |> processors getting the benefit of shared memory, or split it across
> |> multiple machines using ethernet or infiniband or whatever, without
> |> having to change the app code.
> 
> They use it for the communication, but don't expose it to the
> programmer.  It is therefore easy to put the processes on different
> CPUs, and get the memory consistency right.
> 

Thus communicated data is "serialized" - not directly used as with threads or 
with custom shared memory techniques like POSH object sharing.


Robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: maximum number of threads

[Cecil Westerhof]

Felipe Almeida Lessa wrote:

> $ python test.py
> 50
> 100
> 150
> 200
> 250
> 300
> 350
> Exception raised: can't start new thread

I tried your script on a PII 300 MHz and only 150 MB. I broke it of when it
reached more as 1,25 million. ;-}

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python

[Sergei Organov]

[EMAIL PROTECTED] (Nick Maclaren) writes:

> In article <[EMAIL PROTECTED]>,
> Sergei Organov <[EMAIL PROTECTED]> writes:
> |> 
> |> Do you mean that POSIX threads are inherently designed and implemented
> |> to stay idle most of the time?! If so, I'm afraid those guys that
> |> designed POSIX threads won't agree with you. In particular, as far as I
> |> remember, David R. Butenhof said a few times in comp.programming.threads
> |> that POSIX threads were primarily designed to meet parallel programming
> |> needs on SMP, or at least that was how I understood him.
>
> I do mean that, and I know that they don't agree.  However, the word
> "designed" doesn't really make a lot of sense for POSIX threads - the
> one I tend to use is "perpetrated".

OK, then I don't think the POSIX threads were "perpetrated" to be idle
most of time.

> The people who put the specification together were either unaware of
> most of the experience of the previous 30 years, or chose to ignore it.
> In particular, in this context, the importance of being able to control
> the scheduling was well-known, as was the fact that it is NOT possible
> to mix processes with different scheduling models on the same set of
> CPUs.  POSIX's facilities are completely hopeless for that purpose, and
> most of the systems I have used effectively ignore them.

I won't argue that. On the other hand, POSIX threads capabilities in the
field of I/O-bound and real-time threads are also limited, and that's
where "threads that are idle most of time" idiom comes from, I
think. What I argue, is that POSIX were "perpetrated" to support
I/O-bound or real-time apps any more than to support parallel
calculations apps. Besides, pthreads real-time extensions came later
than pthreads themselves.

What I do see, is that Microsoft designed their system so that it's
almost impossible to implement an interactive application without using
threads, and that fact leads to the current situation where threads are
considered to be beasts that are sleeping most of time.

> I could go on at great length, and the performance aspects are not even
> the worst aspect of POSIX threads.  The fact that there is no usable
> memory model, and the synchronisation depends on C to handle the
> low-level consistency, but there are no CONCEPTS in common between
> POSIX and C's memory consistency 'specifications' is perhaps the worst.

I won't argue that either. However, I don't see how does it make POSIX
threads to be "perpetrated" to be idle most of time.

> That is why many POSIX threads programs work until the genuinely
> shared memory accesses become frequent enough that you get some to the
> same location in a single machine cycle.

Sorry, I don't understand. Are you saying that it's inherently
impossible to write an application that uses POSIX threads and that
doesn't have bugs accessing shared state? I thought that pthreads
mutexes guarantee sequential access to shared data. Or do you mean
something entirely different? Lock-free algorithms maybe?

-- Sergei.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Maths error

[Nick Maclaren]

In article <[EMAIL PROTECTED]>,
Tim Peters <[EMAIL PROTECTED]> writes:
|> 
|> Sure.  Possibly even most.  Short of writing a long & gentle tutorial, 
|> can that be improved?  Alas, most people wouldn't read that either <0.5 
|> wink>.

Yes.  Improved wording would be only slightly longer, and it is never
appropriate to omit all negative aspects.  The truth, the whole truth
and nothing but the truth :-)

|> Worse, I expect most people have no real idea of that there's a possible 
|> difference between internal and external representations.  This is often 
|> given as a selling point for decimal arithmetic:  it's WYSIWYG in ways 
|> binary fp can't be (short of inventing power-of-2 fp representations for 
|> I/O, which few people would use).

Right.  Another case when none of the problems show up on dinky little
examples but do in real code :-(

|> > A lot of very well-respected numerical analysts said that larger bases
|> > led to a faster build-up of error (independent of the precision).  My
|> > limited investigations indicated that there was SOME truth in that,
|> > but it wasn't a major matter; I never say the matter settled
|> > definitively.
|> 
|> My point was that 28 decimal digits of precision is far greater than 
|> supplied even by 64-bit binary floats today (let alone the smaller sizes 
|> in most-common use back in the 60s and 70s).  "Pollution" of low-order 
|> bits is far less of a real concern when there are some number of low-
|> order bits you don't care about at all.

Yes, but that wasn't their point.  It was that in (say) iterative
algorithms, the error builds up by a factor of the base at every step.
If it wasn't for the fact that errors build up, almost all programs
could ignore numerical analysis and still get reliable answers!

Actually, my (limited) investigations indicated that such an error
build-up was extremely rare - I could achieve it only in VERY artificial
programs.  But I did find that the errors built up faster for higher
bases, so that a reasonable rule of thumb is that 28 digits with a decimal
base was comparable to (say) 80 bits with a binary base.

And, IN GENERAL, programs won't be using 128-bit IEEE representations.
Given Python's overheads, there is no reason not to, unless the hardware
is catastrophically slower (which is plausible).

|> If you know a < b, doing
|> 
|> c = a + (b-a)/2
|> 
|> instead of
|> 
|> c = (a+b)/2
|> 
|> at least guarantees (ignoring possible overflow) a <= c <= b.  As shown 
|> last time, it's not even always the case that (x+x)/2 == x in decimal fp 
|> (or in any fp base > 2, for that matter).

Yes.  Back in the days before binary floating-point started to dominate,
we taught that as a matter of routine, but it has not been taught to
all users of floating-point for a couple of decades.  Indeed, a lot of
modern programmers regard having to distort simple expressions in that
way as anathema.

It isn't a major issue, because our experience from then is that it is
both teachable and practical, but it IS a way in which any base above
2 is significantly worse than base 2.


Regards,
Nick Maclaren.
-- 
http://mail.python.org/mailman/listinfo/python-list

dot operations

[[EMAIL PROTECTED]]

Hi,
 Frequently I get to do like this:
a = (1, 2, 3, 4) # some dummy values
b = (4, 3, 2, 1)
import operator
c = map(operator.add, a, b)

I am finding the last line not very readable especially when I combine
couple of such operations into one line. Is it possible to overload
operators, so that, I can use .+ for element wise addition, as,
c = a .+ b
which is much more readable.

Similarly, I want to use .- , .*, ./   . Is it possible to do?

thanks.

-
Suresh

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python

[Nick Maclaren]

In article <[EMAIL PROTECTED]>,
Sergei Organov <[EMAIL PROTECTED]> writes:
|> 
|> OK, then I don't think the POSIX threads were "perpetrated" to be idle
|> most of time.

Perhaps I was being unclear.  I should have added "In the case where
there are more threads per system than CPUs per system".  The reasons
are extremely obscure and are to do with the scheduling, memory access
and communication.

I am in full agreement that the above effect was not INTENDED.

|> > That is why many POSIX threads programs work until the genuinely
|> > shared memory accesses become frequent enough that you get some to the
|> > same location in a single machine cycle.
|> 
|> Sorry, I don't understand. Are you saying that it's inherently
|> impossible to write an application that uses POSIX threads and that
|> doesn't have bugs accessing shared state? I thought that pthreads
|> mutexes guarantee sequential access to shared data. Or do you mean
|> something entirely different? Lock-free algorithms maybe?

I mean precisely the first.

The C99 standard uses a bizarre consistency model, which requires serial
execution, and its consistency is defined in terms of only volatile
objects and external I/O.  Any form of memory access, signalling or
whatever is outside that, and is undefined behaviour.

POSIX uses a different but equally bizarre one, based on some function
calls being "thread-safe" and others forcing "consistency" (which is
not actually defined, and there are many possible, incompatible,
interpretations).  It leaves all language aspects (including allowed
code movement) to C.

There are no concepts in common between C's and POSIX's consistency
specifications (even when they are precise enough to use), and so no
way of mapping the two standards together.


Regards,
Nick Maclaren.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dot operations

[robert]

[EMAIL PROTECTED] wrote:
> Hi,
>  Frequently I get to do like this:
> a = (1, 2, 3, 4) # some dummy values
> b = (4, 3, 2, 1)
> import operator
> c = map(operator.add, a, b)
> 
> I am finding the last line not very readable especially when I combine
> couple of such operations into one line. Is it possible to overload
> operators, so that, I can use .+ for element wise addition, as,
> c = a .+ b
> which is much more readable.
> 
> Similarly, I want to use .- , .*, ./   . Is it possible to do?

import numpy

You'll not even need dots
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python

[Nick Maclaren]

In article <[EMAIL PROTECTED]>,
robert <[EMAIL PROTECTED]> writes:
|> 
|> Most threads on this planet are not used for number crunching jobs,
|> but for "organization of execution".

That is true, and it is effectively what POSIX and Microsoft threads
are suitable for.  With reservations, even there.

|> Things like MPI, IPC are just for the area of "small message, big job"
|> - typically sci number crunching, where you collect the results "at
|> the end of day". Its more a slow network technique.

That is completely false.  Most dedicated HPC systems use MPI for high
levels of message passing over high-speed networks.

|> > They use it for the communication, but don't expose it to the
|> > programmer.  It is therefore easy to put the processes on different
|> > CPUs, and get the memory consistency right.
|> 
|> Thus communicated data is "serialized" - not directly used as with
|> threads or with custom shared memory techniques like POSH object
|> sharing.

It is not used as directly with threads as you might think.  Even
POSIX and Microsoft threads require synchronisation primitives, and
threading models like OpenMP and BSP have explicit control.

Also, MPI has asynchronous (non-blocking) communication.


Regards,
Nick Maclaren.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Joining threads but allowing signals to main thread?

[Lloyd Zusman]

Gabriel Genellina <[EMAIL PROTECTED]> writes:

> At Thursday 11/1/2007 03:43, Lloyd Zusman wrote:
>
>>   while threading.activeCount() > 1:
>>   time.sleep(0.001)
>>
>>   sys.exit(0)
>>
>>Is there any way to allow my program to respond to signals without
>>having to busy-wait in the main thread?
>
> Don't worry too much, this is *not* a busy wait, because you use
> sleep. You can use a longer sleep, it will be terminated by any
> signal. But not too long, or the check for activeCount() will be delayed
> in excess. (I use sleep(1) usually but your needs may be different)

Well, if this is the best that we can currently do in python, then so be
it.

My sincere thanks.


-- 
 Lloyd Zusman
 [EMAIL PROTECTED]
 God bless you.

-- 
http://mail.python.org/mailman/listinfo/python-list

sleep in asyncore

[billie]

Hi all.
I'm writing an authentication server by using asyncore / asynchat
modules.
I'd like to implement a basic brute-force protection by "freezing /
sleeping" the current client session for a period of time (e.g. 2
seconds) when the user sends a wrong password.
Does someone knows a "trick" to do that with asyncore?


Thanks in advance for your helping.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: creating simple Python scripting interfaces via C++

[Ben Sizer]

Ok, my first attempt at this creates proxy objects in Python, and
stores a pointer to the C++ instance in the Python object. I cast that
pointer to an int and pass it as a single parameter to the object's
__init__ function.

static PyObject* Actor_init(PyObject *self, PyObject *args)
{
PyObject* selfParam;
PyObject* ptrValue;
if (!PyArg_ParseTuple(args, "OO", &selfParam, &ptrValue))
return NULL;

PyObject_SetAttrString(selfParam, "_cpp_ptr", ptrValue);

Py_INCREF(Py_None);
return Py_None;
}

I have no idea why self is always NULL, when I'm calling the functions
as methods of an object. Any ideas why this is the case? For what it's
worth I attach each method via the PyMethodDef -> PyCFunction_New ->
PyMethod_New -> PyDict_SetItemString(classDict) route.

To get data back from the C++ object to Python, I extract that value
and cast it back to the appropriate pointer type.

static PyObject* Actor_showName(PyObject *self, PyObject *args)
{
PyObject* selfParam;
if (!PyArg_ParseTuple(args, "O", &selfParam))
return NULL;

PyObject* cppPtr = PyObject_GetAttrString(selfParam, "_cpp_ptr");
long cppPtrVal = PyInt_AsLong(cppPtr);
Actor* pActor = reinterpret_cast(cppPtrVal);

// Delegate to the C++ object
pActor->ShowName();

Py_INCREF(Py_None);
return Py_None;
}

I've omitted some error checking, but this is the way I'm going for
now. Are there any glaring errors I've made (apart from perhaps
assuming sizeof(pointer) <= sizeof(long), that is)? And is there
anywhere else more appropriate that I should be asking this question,
given the lack of responses to this and my other embedding topic so
far?

-- 
Ben Sizer

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Execute binary code

[Jorgen Grahn]

On 9 Jan 2007 07:04:11 -0800, sturlamolden <[EMAIL PROTECTED]> wrote:
>
> Jorgen Grahn wrote:
>
>> For what it's worth[1], under Unix it /is/ impossible. The only way to bring 
>> in
>> new code (short of dynamic libraries) is to call exec(2) or its variations,
>> and all need a file system object to load the code from.
>
> The x86 processor cannot tell the difference between code segments and
> data segments. If the executable code is stored in string, all you need
> is a pointer to the string holding the code. You can cast the string
> address to a function pointer (possibly through a void* if the compiler
> complains), then dereference (call) the function pointer.
>
> Trojans, viruses and JIT compilers do this all the time. Here is an
> (untested) example:
[...]

You probably need to flush the code cache somewhere there, too, don't you?
Or will that resolve itself because that memory area hasn't been executed
before?

I must admit I haven't contemplated this since the MC68000 was state of the
art, before caches became popular.

/Jorgen

-- 
  // Jorgen Grahn   R'lyeh wgah'nagl fhtagn!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Execute binary code

[Jorgen Grahn]

On Wed, 10 Jan 2007 10:31:50 -0600, Chris Mellon <[EMAIL PROTECTED]> wrote:
> On 10 Jan 2007 08:12:41 -0800, sturlamolden <[EMAIL PROTECTED]> wrote:
>>
>> Chris Mellon wrote:
>>
>> > This works fine if the binary data is "pure" asm, but the impresssion
>> > the OP gave is that it's a compiled binary, which you can't just "jump
>> > into" this way.
>>
>> You may have to offset the function pointer so the entry point becomes
>> correct.
>>
>
> That won't be enough. You basically would have to re-implement the OS
> loading process, handling relocations and loading any linked
> libraries. Possible, in theory, but very non-trivial.

Yeah, that was implicitly my thinking a bit up in the thread. If all you
have is an executable file (COFF/ELF/...) as a string, and you have no
os.exec(string) or similar, then you're in trouble.

At least if it has to work.

/Jorgen

-- 
  // Jorgen Grahn   R'lyeh wgah'nagl fhtagn!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python

[sturlamolden]

robert wrote:

> Thus communicated data is "serialized" - not directly used as with threads or 
> with custom shared memory techniques like POSH object sharing.

Correct, and that is precisely why MPI code is a lot easier to write
and debug than thread code. The OP used a similar technique in his
'parallel python' project.

This does not mean that MPI is inherently slower than threads however,
as there are overhead associated with thread synchronization as well.
With 'shared memory' between threads, a lot more fine grained
synchronization ans scheduling is needed, which impair performance and
often introduce obscure bugs.

-- 
http://mail.python.org/mailman/listinfo/python-list

Ref count oddness with embedded Python... memory leak?

[Ben Sizer]

Here's my test-case:

#include 
int main(int argc, char *argv[])
{
Py_Initialize(); Py_Finalize();
Py_Initialize(); Py_Finalize();
Py_Initialize(); Py_Finalize();
Py_Initialize(); Py_Finalize();
Py_Initialize(); Py_Finalize();
return 1;
}

Here's my output, with Python 2.5 built in debug mode on WinXP, no
modifications:

[7438 refs]
[7499 refs]
[7550 refs]
[7601 refs]
[7652 refs]

Is this normal? It doesn't look very promising to me.

-- 
Ben Sizer

-- 
http://mail.python.org/mailman/listinfo/python-list

Type casting a base class to a derived one?

[Frederic Rentsch]

Hi all,
   If I derive a class from another one because I need a few extra 
features, is there a way to promote the base class to the derived one 
without having to make copies of all attributes?

class Derived (Base):
   def __init__ (self, base_object):
  # ( copy all attributes )
  ...

This looks expensive. Moreover __init__ () may not be available if it 
needs to to something else.

Thanks for suggestions

Frederic


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Read from database, write to another database, simultaneously

[Sean Davis]

On Jan 10, 9:27 pm, johnf <[EMAIL PROTECTED]> wrote:
> Bjoern Schliessmann wrote:
> > Sean Davis wrote:
>
> >> The author of one of the python database clients mentioned that
> >> using one thread to retrieve the data from the oracle database and
> >> another to insert the data into postgresql with something like a
> >> pipe between the two threads might make sense, keeping both IO
> >> streams busy.
>
> > IMHO he's wrong. Network interaction is quite slow compared with CPU
> > performance, so there's no gain (maybe even overhead due to thread
> > management and locking stuff). That's true even on multiprocessor
> > machines, not only because there's almost nothing to compute but
> > only IO traffic. CMIIW.
>
> > Using multiplexing, you'll get good results with simple code without
> > the danger of deadlocks. Have a look at asyncore (standard library)
> > or the Twisted framework -- personally, I prefer the latter.
>
> > Regards,
>
> Sean you can't win - everyone has a different idea!  You need to explain
> that oracle has millions of records and it's possible to a pipe open to
> feed the Postgres side.
>
> One thing I didn't get - is this a one time transfer or something that is
> going to happen often.

Yes, some detail about the problem is definitely in order!

We have a collaborator that is going to maintain a large genome
database that is a component of a postgresql database that we currently
maintain.  There are going to be consumers of the oracle data using
both mysql and postgresql.  The oracle database is LARGE with around
100,000,000 rows spread over some 50-70 tables in multiple schemas.
The idea is that as publicly available data (from various datasources
on the web) become available, the oracle team will update the oracle
database, doing all the parsing and necessary data cleanup of the raw
data.  We then want to be able to update postgres with these oracle
data.  So the process may be done only once per month on some tables,
but as often as once a day on others.

As for the specifics, Oracle data is going to be coming in as a DB-API
2 cursor in manageable chunks (and at a relatively slow pace).  On the
postgres loading side, I wanted to use the pscycopg2 copy_from
function, which expects an open file-like object (that has read and
readline functionality) and is quite fast for loading data.  Note the
disconnect here--Oracle is coming in in discrete chunks, while
postgresql is looking for a file object.  I solved this problem by
creating a temporary file as an intermediary, but why wait for Oracle
to finish dumping data when I can potentially be loading into postgres
at the same time that the data is coming in?  So, I am actually looking
for a solution to this problem that doesn't require an intermediate
file and allows simultaneous reading and writing, with the caveat that
the data cannot all be read into memory simultaneously, so will need to
be buffered.

I hope that clarifies things.

Thanks,
Sean

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Type casting a base class to a derived one?

[bearophileHUGS]

Frederic Rentsch:
>If I derive a class from another one because I need a few extra
> features, is there a way to promote the base class to the derived one
> without having to make copies of all attributes?
>
> class Derived (Base):
>def __init__ (self, base_object):
>   # ( copy all attributes )
>   ...
>
> This looks expensive.

No need to copy attributes:

class Base(object):
def __init__ (self, x):
self.x = x

class Derived(Base):
def __init__ (self, x, y):
Base.__init__(self, x)
self.y = y

d = Derived(1, 2)
print d.x, d.y

Bye,
bearophile

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Type casting a base class to a derived one?

[Peter Otten]

Frederic Rentsch wrote:

>If I derive a class from another one because I need a few extra
> features, is there a way to promote the base class to the derived one
> without having to make copies of all attributes?
> 
> class Derived (Base):
>def __init__ (self, base_object):
>   # ( copy all attributes )
>   ...
> 
> This looks expensive. Moreover __init__ () may not be available if it
> needs to to something else.

base_instance = Base(...)
base_instance.__class__ = Derived

Peter
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Read from database, write to another database, simultaneously

[Sean Davis]

On Jan 11, 3:20 am, Laurent Pointal <[EMAIL PROTECTED]> wrote:
> Bjoern Schliessmann a écrit :
>
> > Sean Davis wrote:
>
> >> The author of one of the python database clients mentioned that
> >> using one thread to retrieve the data from the oracle database and
> >> another to insert the data into postgresql with something like a
> >> pipe between the two threads might make sense, keeping both IO
> >> streams busy.
>
> > IMHO he's wrong. Network interaction is quite slow compared with CPU
> > performance, so there's no gain (maybe even overhead due to thread
> > management and locking stuff). That's true even on multiprocessor
> > machines, not only because there's almost nothing to compute but
> > only IO traffic. CMIIW.Not so sure, there is low CPU in the Python script, 
> > but there may be
> CPU+disk activity on the database sides [with cache management and other
> optimizations on disk access].
> So, with a reader thread and a writer thread, he can have a select on a
> database performed in parallel with an insert on the other database.
> After, he must know if the two databases use same disks, same
> controller, same host... or not.

Some more detail:

The machine running the script is distinct from the Oracle machine
which is distinct from the Postgresql machine.  So, CPU usage is low
and because of the independent machines for the database end, it is
definitely possible to read from one database while writing to the
other.  That is the solution that I am looking for, and Dennis's post
seems pretty close to what I need.  I will have to use some kind of
buffer.  A Queue isn't quite right as it stands, as the data is coming
in as records, but for postgres loading, a file-like stream is what I
need, so there will need to be either a wrapper around the Queue on the
get() side.  Or is there a better way to go about this detail?  What
seems to make sense to me is to stringify the incoming oracle data into
some kind of buffer and then read on the postgresql side.

Thanks,
Sean

-- 
http://mail.python.org/mailman/listinfo/python-list

Reloading the already imported module

[anil . pundoor]

hi all,
  i have following code


if 
from SIPT.xml_param_mapping import
MESSAGE_PARAMETER_MAPPING
else:
from SIPT.msg_param_mapping import
MESSAGE_PARAMETER_MAPPING


parameter_list = MESSAGE_PARAMETER_MAPPING [
'ABC' ]
for parms in parameter_list:

parameter_list[parms][4] = 'EXPORT'

parameter_list[parms][5] = 'IMPORT'


   After this the xml_param_mapping gets altered. this process is
repeated over couple of time. But every time i need to get a fresh
xml_param_mapping. So how to relaod the already imported module??



regards
Anil

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python

[Sergei Organov]

[EMAIL PROTECTED] (Nick Maclaren) writes:
[...]
> I mean precisely the first.
>
> The C99 standard uses a bizarre consistency model, which requires serial
> execution, and its consistency is defined in terms of only volatile
> objects and external I/O.  Any form of memory access, signalling or
> whatever is outside that, and is undefined behaviour.
>
> POSIX uses a different but equally bizarre one, based on some function
> calls being "thread-safe" and others forcing "consistency" (which is
> not actually defined, and there are many possible, incompatible,
> interpretations).  It leaves all language aspects (including allowed
> code movement) to C.
>
> There are no concepts in common between C's and POSIX's consistency
> specifications (even when they are precise enough to use), and so no
> way of mapping the two standards together.

Ah, now I see what you mean. Even though I only partly agree with what
you've said above, I'll stop arguing as it gets too off-topic for this
group.

Thank you for explanations.

-- Sergei.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: what is the idiom for copy lots of params into self?

[Emin]

Dear Luis and everyone else who responded,

Thanks for your suggestions. One issue with using *args or **kw is that
I might no want to copy all the arguments to __init__ into self.

What made me ask the question in my original post was not so much that
I had to loop over the names I wanted to save, but whether it's okay to
mess with self.__dict__ or if there is another way I should be
assigning to self.

Thanks,
-Emin

On Jan 10, 9:05 pm, "Luis M. González" <[EMAIL PROTECTED]> wrote:
> Emin wrote:
> > Dear Experts,
>
> > When writing large classes, I sometimes find myself needing to copy a
> > lot of parameters from the argument of __init__ into self. Instead of
> > having twenty lines that all basically say something like self.x = x, I
> > often use __dict__ via something like:
>
> > class example:
> >  def __init__(self,a,b,c,d,e,f,g,h,i,j,k,l,m,n):
> >  for name in
> > ['a','b','c','d','e','f','g','h','i','j','k','l','m','n']:
> >  self.__dict__[name] = locals()[name]
>
> > This saves a lot of code and makes it easier to see what is going on,
> > but it seems like there should be a better idiom for this task. Any
> > suggestions?
>
> > Thanks,
> > -EminHow about using variable length argumens?
>
> class example:
> def __init__(self, *args):
> for i in args:
> self.__dict__[i] = i
> 
> x = example('uno','dos','tres')
> 
> Luis

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: creating simple Python scripting interfaces via C++

[David Boddie]

Ben Sizer wrote:  > And is there anywhere else more appropriate that I
should be asking > this question, given the lack of responses to this
and my other embedding > topic so far?  You could try asking on the C++
SIG mailing list at python.org:
http://mail.python.org/mailman/listinfo/c++-sig  David

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: sleep in asyncore

[Fredrik Lundh]

"billie" <[EMAIL PROTECTED]> wrote:

> I'm writing an authentication server by using asyncore / asynchat
> modules.
> I'd like to implement a basic brute-force protection by "freezing /
> sleeping" the current client session for a period of time (e.g. 2
> seconds) when the user sends a wrong password.
> Does someone knows a "trick" to do that with asyncore?

this could work:

add a timeout variable to your dispatcher instance, and have your "readable"
method returns False if timeout is set and the current time is before the given
value.  then set the timeout value to time.time() + 2 when you see a bad pass-
word.

 



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dot operations

[Paddy]

[EMAIL PROTECTED] wrote:
> Hi,
>  Frequently I get to do like this:
> a = (1, 2, 3, 4) # some dummy values
> b = (4, 3, 2, 1)
> import operator
> c = map(operator.add, a, b)
>
> I am finding the last line not very readable especially when I combine
> couple of such operations into one line. Is it possible to overload
> operators, so that, I can use .+ for element wise addition, as,
> c = a .+ b
> which is much more readable.
>
> Similarly, I want to use .- , .*, ./   . Is it possible to do?
>
> thanks.
>
> -
> Suresh

List comprehensions?

>>> a = (1, 2, 3, 4)
>>> b = (4, 3, 2, 1)
>>> import operator
>>> c = map(operator.add, a, b)
>>> c
[5, 5, 5, 5]
>>> c1 = [a1+b1 for a1,b1 in zip(a,b)]
>>> c1
[5, 5, 5, 5]
>>> 


- Paddy.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: what is the idiom for copy lots of params into self?

[Fredrik Lundh]

"Emin" <[EMAIL PROTECTED]> wrote:

> What made me ask the question in my original post was not so much that
> I had to loop over the names I wanted to save, but whether it's okay to
> mess with self.__dict__ or if there is another way I should be
> assigning to self.

http://effbot.org/pyref/setattr

 



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: globals accros modules

[alf]

Gabriel Genellina wrote:
> Change a=1 to amodule.a=1

I have multiple command line programs creating 'a' and amodule using it. 
   Plus some import sequence dependency. So it would not work. Currently 
the solution in amodule is:
import __main__
print __main__.a

> If you find yourself doing tricks with the module globals, think about 
> redesigning your application.

For what I do it is good enough. If I hit the same problem, will 
refactor it.

A.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Print message with Colors

[Neil Cerutti]

On 2007-01-11, Laurent Pointal <[EMAIL PROTECTED]> wrote:
> prk a écrit :
>> Hi Folks,
>> 
>> Is there any procesure for print messages with colors.
>
> Yes, see:
>
> http://www.google.fr/search?q=python+print+color
> http://groups.google.fr/groups?as_q=python+print+color&num=100&as_ugroup=comp.lang.python
>
> See also ANSI escape sequences for the more general subject of
> color printing on terminals.

If you're using Windows NT, 2000, or XP don't bother reading
about ANSI escape sequences. They will not work at all with
Python on those platforms, even if you use the crummy old
COMMAND.COM. I believe it's because Python on those platforms in
a console application, which on NT, 2000 and XP doesn't support
ANSI escape sequences. It makes IPython's appearance less cool. :-(

Try http://effbot.org/downloads/#console for color output that
works.

-- 
Neil Cerutti
We're going to be exciting.  Of course, it was exciting when the Titanic went
down. --Bob Weiss
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Internet Survey

[jmfbahciv]

In article <[EMAIL PROTECTED]>,
   krw <[EMAIL PROTECTED]> wrote:
>In article <[EMAIL PROTECTED]>, 
>[EMAIL PROTECTED] says...
>> In article <[EMAIL PROTECTED]>,
>>Lefty Bigfoot <[EMAIL PROTECTED]> wrote:
>> >On Wed, 10 Jan 2007 08:28:33 -0600, [EMAIL PROTECTED] wrote
>> >(in article <[EMAIL PROTECTED]>):
>> >
>> >> In article <[EMAIL PROTECTED]>,
>> >>"Elan Magavi" <[EMAIL PROTECTED]> wrote:
>> >>> Is that like.. OctaPussy?
>> >> 
>> >> I didn't read their stuff.  Are they really trying to put a 
>> >> round peg in a square hole?
>> >
>> >Sounds more like an octagonal pole in a round hole.
>> 
>> Nah, I figured they picked the word octal because it's never
>> been used before.
>
>and "HexaPussy" just wouldn't be right.
>
Right.  The PC types would put them in jail for being cruel
to animals.

/BAH
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Internet Survey

[jmfbahciv]

In article <[EMAIL PROTECTED]>, [EMAIL PROTECTED] wrote:
>
>
>On 10-Jan-2007, krw <[EMAIL PROTECTED]> wrote:
>
>> ...and "HexaPussy" just wouldn't be right.
>
>
>SexagesimalPussy (base 60) has kind of a nice ring to it.

That would cause kiddies to look up the word.  But 360
would have the correct ring.

/BAH
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Type casting a base class to a derived one?

[Neil Cerutti]

On 2007-01-11, Frederic Rentsch <[EMAIL PROTECTED]> wrote:
> If I derive a class from another one because I need a few extra
> features, is there a way to promote the base class to the
> derived one without having to make copies of all attributes?
>
> class Derived (Base):
>def __init__ (self, base_object):
>   # ( copy all attributes )
>   ...
>
> This looks expensive. Moreover __init__ () may not be available
> if it needs to to something else.
>
> Thanks for suggestions

How does it make sense to cast a base to a derived in your
application?

-- 
Neil Cerutti
I'm traveling to all 51 states to see who can stop 85. --Chad Johnson
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dot operations

[[EMAIL PROTECTED]]

Paddy wrote:
> [EMAIL PROTECTED] wrote:
> > Hi,
> >  Frequently I get to do like this:
> > a = (1, 2, 3, 4) # some dummy values
> > b = (4, 3, 2, 1)
> > import operator
> > c = map(operator.add, a, b)
> >
> > I am finding the last line not very readable especially when I combine
> > couple of such operations into one line. Is it possible to overload
> > operators, so that, I can use .+ for element wise addition, as,
> > c = a .+ b
> > which is much more readable.
> >
> > Similarly, I want to use .- , .*, ./   . Is it possible to do?
> >
> > thanks.
> >
> > -
> > Suresh
>
> List comprehensions?
>
> >>> a = (1, 2, 3, 4)
> >>> b = (4, 3, 2, 1)
> >>> import operator
> >>> c = map(operator.add, a, b)
> >>> c
> [5, 5, 5, 5]
> >>> c1 = [a1+b1 for a1,b1 in zip(a,b)]
> >>> c1
> [5, 5, 5, 5]
> >>>
>

I just found this from :
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/384122

class Infix(object):
def __init__(self, function):
self.function = function
def __ror__(self, other):
return Infix(lambda x, self=self, other=other:
self.function(other, x))
def __or__(self, other):
return self.function(other)
def __rlshift__(self, other):
return Infix(lambda x, self=self, other=other:
self.function(other, x))
def __rshift__(self, other):
return self.function(other)
def __call__(self, value1, value2):
return self.function(value1, value2)

import operator
dotplus = Infix(lambda x,y: map(operator.add, x, y))

a = range(4)
b = range(4)

c = a |dotplus| b

> 
> - Paddy.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Read from database, write to another database, simultaneously

[Istvan Albert]

Sean Davis wrote:

> at the same time that the data is coming in?  So, I am actually looking
> for a solution to this problem that doesn't require an intermediate
> file and allows simultaneous reading and writing, with the caveat that
> the data cannot all be read into memory simultaneously, so will need to
> be buffered.

IMO the problem that you need to solve is not well suited for the
python DBAPI as this API is meant to support programming and
interacting with the database not streaming large quantities from one
database into another.

I agree with another opinion in this thread.

All you need is a simple way to pipe the output from Oracle into
Postgresql. Just run the oracle client and start dumping to the
standard output. Pipe it through sed (or a python program) to reformat
the output for whatever minor fixes (you might not even need this step)
then continue piping it right into psql.

i.

-- 
http://mail.python.org/mailman/listinfo/python-list

Progress count in terminal (Mac OS X)

[Tommy Grav]

I have a program that does a lot of iterations and would like
to follow its progress by having it print out the current iteration
number as it progresses. How do I do this so that it appears
like a counter that increases in the same place in the terminal
window? I am using python2.5 on a Mac OSX and its terminal
tcsh window.

Cheers
Tommy
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Reloading the already imported module

[Paul McGuire]

<[EMAIL PROTECTED]> wrote in message 
news:[EMAIL PROTECTED]
> hi all,
>  i have following code
>
>
>  if 
>  from SIPT.xml_param_mapping import MESSAGE_PARAMETER_MAPPING
>  else:
>  from SIPT.msg_param_mapping import MESSAGE_PARAMETER_MAPPING
>
>  parameter_list = MESSAGE_PARAMETER_MAPPING ['ABC' ]
>  for parms in parameter_list:
>  parameter_list[parms][4] = 'EXPORT'
>  parameter_list[parms][5] = 'IMPORT'
>
>
>   After this the xml_param_mapping gets altered. this process is
> repeated over couple of time. But every time i need to get a fresh
> xml_param_mapping. So how to relaod the already imported module??
>

Why not work with a copy of MESSAGE_PARAMETER_MAPPING['ABC'], and keep the 
original intact?  Change:

parameter_list = MESSAGE_PARAMETER_MAPPING ['ABC' ]

to

parameter_list = MESSAGE_PARAMETER_MAPPING ['ABC' ][:]

Now no reimport is required.

-- Paul



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python

[robert]

sturlamolden wrote:
> robert wrote:
> 
>> Thus communicated data is "serialized" - not directly used as with threads 
>> or with custom shared memory techniques like POSH object sharing.
> 
> Correct, and that is precisely why MPI code is a lot easier to write
> and debug than thread code. The OP used a similar technique in his
> 'parallel python' project.

Thus there are different levels of parallelization:

1 file/database based; multiple batch jobs
2 Message Passing, IPC, RPC, ...
3 Object Sharing 
4 Sharing of global data space (Threads)
5 Local parallelism / Vector computing, MMX, 3DNow,...

There are good reasons for all of these levels.
Yet "parallel python" to me fakes to be on level 3 or 4 (or even 5 :-) ), while 
its just a level 2 system, where "passing", "remote", "inter-process" ... are 
the right vocables.

With all this fakes popping up - a GIL free CPython is a major feature request 
for Py3K - a name at least promising to run 3rd millenium CPU's ...


> This does not mean that MPI is inherently slower than threads however,
> as there are overhead associated with thread synchronization as well.

level 2 communication is slower. Just for selected apps it won't matter a lot.

> With 'shared memory' between threads, a lot more fine grained
> synchronization ans scheduling is needed, which impair performance and
> often introduce obscure bugs.

Its a question of chances and costs and nature of application.
Yet one can easily restrict inter-thread communcation to be as simple and 
modular or even simpler as IPC. Search e.g. "Python CallQueue" and 
"BackgroundCall" on Google.
Thread programming is less complicated as it seems. (Just Python's stdlib 
offers cumbersome 'non-functional' classes)


Robert
-- 
http://mail.python.org/mailman/listinfo/python-list

What about this?

[new]

www.magicoz.com
amazing

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Need startup suggestions for writing a MSA viewer GUI in python

[Istvan Albert]

Joel Hedlund wrote:

> ideas from you people to get me going in the right direction. Despite my
> GUI n00b-ness I need to get it good and usable with an intuitive look
> and feel.

UI design requires a different skillset than programming. It can be a
very frustrating and thankless task as well. It is incomparably easier
to see the flaws in existing interfaces than correcting them (or even
creating the said interface). Make sure to start with something simple,
and learn that way.

I would also recommend that you implement something novel, there are
many existing MSA viewers and it won't be easy to improve on them. Do
something that adds new value and people will be willing to expend
effort to use it.

i.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Progress count in terminal (Mac OS X)

[Fredrik Lundh]

Tommy Grav <[EMAIL PROTECTED]>:

>I have a program that does a lot of iterations and would like
> to follow its progress by having it print out the current iteration
> number as it progresses. How do I do this so that it appears
> like a counter that increases in the same place in the terminal
> window? I am using python2.5 on a Mac OSX and its terminal
> tcsh window.

print a carriage return before, and no line feed after, each line:

for i in range(100):
print "\r" + "count", i,
print # done

 



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Type casting a base class to a derived one?

[Paul McGuire]

"Peter Otten" <[EMAIL PROTECTED]> wrote in message 
news:[EMAIL PROTECTED]
> Frederic Rentsch wrote:
>
>>If I derive a class from another one because I need a few extra
>> features, is there a way to promote the base class to the derived one
>> without having to make copies of all attributes?
>>
>> class Derived (Base):
>>def __init__ (self, base_object):
>>   # ( copy all attributes )
>>   ...
>>
>> This looks expensive. Moreover __init__ () may not be available if it
>> needs to to something else.
>
> base_instance = Base(...)
> base_instance.__class__ = Derived
>
> Peter

This will change the class-level behavior of the object, but it wont 
automagically populate the instance with any attributes that are 
specifically added in Derived's __init__.

class A(object):
def __init__(self,x):
self.x = x
def method(self):
print "I am an A whose x value is", self.x

class B(A):
bb = 1000
def __init__(self,x,y):
super(B,self).__init__(x)
self.y = y
def method(self):
print "I am a B whose x value is", self.x

aobj = A(100)
aobj.method()
aobj.__class__ = B
aobj.method()
print aobj.bb
print aobj.y

prints:
I am an A whose x value is 100
I am a B whose x value is 100
1000
Traceback (most recent call last):
  File "dertest.py", line 20, in ?
print aobj.y
AttributeError: 'B' object has no attribute 'y'


But it wouldn't be hard to write a makeBaseIntoDerived method:

def makeAintoB(a,y=0):
a.y = y
a.__class__ = B

-- Paul 


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Type casting a base class to a derived one?

[Chris Mellon]

On 11 Jan 2007 15:01:48 +0100, Neil Cerutti <[EMAIL PROTECTED]> wrote:
> On 2007-01-11, Frederic Rentsch <[EMAIL PROTECTED]> wrote:
> > If I derive a class from another one because I need a few extra
> > features, is there a way to promote the base class to the
> > derived one without having to make copies of all attributes?
> >
> > class Derived (Base):
> >def __init__ (self, base_object):
> >   # ( copy all attributes )
> >   ...
> >
> > This looks expensive. Moreover __init__ () may not be available
> > if it needs to to something else.
> >
> > Thanks for suggestions
>
> How does it make sense to cast a base to a derived in your
> application?
>

I can't figure out any circumstance when you'd need to do this in
Python. Upcasting like this is something you do in statically typed
languages. I suspect that the OP doesn't really believe dynamic
casting works and doesn't want to pass a derived class for some
reason.
-- 
http://mail.python.org/mailman/listinfo/python-list

__init__ vs __new__

[Daniel Klein]

I've have a program that is using both of the methods below (in
different classes of course) for initializing the class. The example
below shows a class with the 2 methods with one commented out.

class JsubroutineParameters(list):
"Represents a list of arguments for external subroutine calls."
# We only need to override methods which add objects to the list.

def __init__(self, alist = []):
for objekt in alist: _validateParameter(objekt)
self.extend(alist)

#def __new__(cls, alist = []):
#for objekt in alist: _validateParameter(objekt)
#return list.__new__(cls, alist)

I don't really notice any behavioral difference. Is there in fact any
difference in using one over the other? Performance? Side effects? ???

I am using Python version 2.5.

Thanks,
Daniel Klein
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how to clean sys.path

[siggi]

"Tim Roberts"  wrote:
>>
>>when I do >>>sys.path in IDLE (winXP), i get a horrendously long list of
>>paths, paths I may have used during a lot of trials and errors. How can I
>>clean up sys.path? I mean, trim it of unnecessary paths?
>
> What do mean by "used during a lot of trials and errors"?  sys.path is
> recreated from scratch every time Python starts.  It doesn't accumulate
> over time, other than from new packages that you install.
> -- 
> Tim Roberts, [EMAIL PROTECTED]
> Providenza & Boekelheide, Inc.


Sorry Tim, my statement was not correct, due to my inexperience with Python.
And sorry, too, for my somewhat lengthy reply:
After having had inspected my current sys.path...

['C:\\Documents and Settings\\User\\My Documents\\My Python files',
'C:\\Documents and Settings\\User\\My Documents\\Python25\\Lib\\idlelib',
'C:\\Documents and Settings\\User\\My Documents\\Python25\\python25.zip',
'C:\\Documents and Settings\\User\\My Documents\\Python25\\DLLs',
'C:\\Documents and Settings\\User\\My Documents\\Python25\\lib',
'C:\\Documents and Settings\\User\\My Documents\\Python25\\lib\\plat-win',
'C:\\Documents and Settings\\User\\My Documents\\Python25\\lib\\lib-tk',
'C:\\Documents and Settings\\User\\My Documents\\Python25', 'C:\\Documents
and Settings\\User\\My Documents\\Python25\\lib\\site-packages',
'C:\\Documents and Settings\\User\\My
Documents\\Python25\\lib\\site-packages\\PIL', 'C:\\Documents and
Settings\\User\\My Documents\\Python25\\lib\\site-packages\\win32',
'C:\\Documents and Settings\\User\\My
Documents\\Python25\\lib\\site-packages\\win32\\lib', 'C:\\Documents and
Settings\\User\\My Documents\\Python25\\lib\\site-packages\\Pythonwin',
'C:\\Documents and Settings\\User\\My
Documents\\Python25\\lib\\site-packages\\wx-2.8-msw-ansi']

or in plain DOS:

C:\Documents and Settings\User\My Documents\My Python files
C:\Documents and Settings\User\My Documents\Python25\Lib\idlelib
C:\Documents and Settings\User\My Documents\Python25\python25.zip
C:\Documents and Settings\User\My Documents\Python25\DLLs
C:\Documents and Settings\User\My Documents\Python25\lib
C:\Documents and Settings\User\My Documents\Python25\lib\plat-win
C:\Documents and Settings\User\My Documents\Python25\lib\lib-tk
C:\Documents and Settings\User\My Documents\Python25
C:\Documents and Settings\User\My Documents\Python25\lib\site-packages
C:\Documents and Settings\User\My Documents\Python25\lib\site-packages\PIL
C:\Documents and Settings\User\My Documents\Python25\lib\site-packages\win32
C:\Documents and Settings\User\My
Documents\Python25\lib\site-packages\win32\lib
C:\Documents and Settings\User\My
Documents\Python25\lib\site-packages\Pythonwin
C:\Documents and Settings\User\My
Documents\Python25\lib\site-packages\wx-2.8-msw-ansi

...it just looked horrible to me at first sight!

If I interpret your explanation correctly, all these paths are necessary,
and not relics of previous installations and deinstallations.

What puzzles me, though, is, that e.g., when I run the wxPython application
"AnalogClock.py" with IDLE or in the command line , this works only in the
directory "...\My Python files\wxDemos\" . This directory contains all files
and folders from the original "\wx-2.8-msw-ansi\demos\").

When I copy AnalogClock.py to ...\My Python Files\  , nothing happens after
running it with IDLE or in the command line.
Appending 'C:\Documents and Settings\User\My Documents\My Python
files\wxDemos ' to the sys.path does not help either.

Thinking that I am clever, I  changed my sys.path with sclicing and
concatenation such that my sys.path starts with

'C:\Documents and Settings\User\My Documents\My Python files', 'C:\Documents
and Settings\User\My Documents\My Python files\wxDemos'. Now \wxDemos\ is
being searched very early.

... no way! After running AnalogClock.py again, this error message appears:

--
Traceback (most recent call last):
  File "C:\Documents and Settings\My Documents\My Python
files\wxAnalogClock.py", line 14, in 
import wx
ImportError: No module named wx.
--

Very strange! Because all this wx stuff IS IN the directory 'C:\Documents
and Settings\User\My Documents\My Python files\wxDemos'. And AnalogClock.py
does work when residing in that directory.

Can you help me again?

Thanks,

siggi

P.S. On another PC where the python program is in c:\programs\python25\, 
same as above!






-- 
http://mail.python.org/mailman/listinfo/python-list

Re: IDLE Python and Environment Variables

[Tristan]

Thanks Gabriel.

> What kind of environment variables? Those used by Python itself, like
> PYTHONPATH? Or your own variables, like FOO_LOCATION=C:\My\Projects\Lib\Foo

I need to add to PYTHONPATH and other enviroment variables asked, for
example, by DJANGO or other python products.

> It appears that your variables are some kind of configuration - in
> this case it has more sense to put such configuration in another
> place, like a config file, and forget about environment variables.
> You can use the usual .ini Windows format and read it with
> ConfigParser. You can pass your script the name of the ini file to
> read - this would be the equivalent of using different .bat files to
> call the same script.
> This way it doesn't matter whether you invoke your application using
> python command line, or inside IDLE, or inside another environment.

I considered to use it, thanks!!  Incidentally (beg your pardon if it
sounds to ignorance), can I define Environment variables in a config
file and then apply to my python program?

Thanks!!!

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Newbie - converting csv files to arrays in NumPy - Matlab vs. Numpy comparison

[Istvan Albert]

oyekomova wrote:

> csvread in Matlab for a very large csv file. Matlab read the file in
> 577 seconds. On the other hand, this code below kept running for over 2
> hours. Can this program be made more efficient? FYI

There must be something wrong with your setup/program. I work with
large csv files as well and  I never have performance problems of that
magnitude. Make sure you are not doing something else while parsing
your data.

Parsing 1 million lines with six columns with the program below takes
87 seconds on my laptop. Even your original version with extra slices
and all would still only be take about 50% more time.

import time, csv, random
from numpy import array

def make_data(rows=1E6, cols=6):
fp = open('data.txt', 'wt')
counter = range(cols)
for row in xrange( int(rows) ):
vals = map(str, [ random.random() for x in counter ] )
fp.write( '%s\n' % ','.join( vals ) )
fp.close()

def read_test():
start  = time.clock()
reader = csv.reader( file('data.txt') )
data   = [ map(float, row) for row in reader ]
data   = array(data, dtype = float)
print 'Data size', len(data)
print 'Elapsed', time.clock() - start

#make_data()
read_test()

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: globals accros modules

[stef]

>
> Change a=1 to amodule.a=1
> If you find yourself doing tricks with the module globals, think about 
> redesigning your application.
>
Of course I completely agree with you.

But ...
if you're moving from MatLab to Python,
and want to show your collegaes,
with how little effort they can reuse all their existing MatLab routines 
in Python,
then the global issue is a real pain !!

You can explain your collegaes, that
- the startindex of arrays changes from 1 to 0
- slices are upto, instead of including the final border
- indention is thé key
And tell them about all beautiful things in Python,
but tell them that they are going to loose all their globals ???

cheers,
Stef Mientki

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Universal Feed Parser - How do I keep attributes?

[Max Erickson]

Gabriel Genellina <[EMAIL PROTECTED]> wrote:

> At Wednesday 10/1/2007 14:38, [EMAIL PROTECTED] wrote:
> 
>> >>> d =
>> >>> feedparser.parse('http://weather.yahooapis.com/forecastrss?p=
> 94089')
>> >>> d.feed.yweather_location
>>u''
> 
> You have to feed it the *contents* of the page, not its URL.
> 
> 

The online documentation disagrees with you:

http://feedparser.org/docs/introduction.html


max

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Progress count in terminal (Mac OS X)

[Tommy Grav]

This certainly does work when running the interpreter interactively,
but when inserted into a script it seems to buffer the print statement
and not write it out to the terminal. How can I force the print  
statement
to not buffer the output?

Cheers
Tommy

On Jan 11, 2007, at 9:22 AM, Fredrik Lundh wrote:

> Tommy Grav <[EMAIL PROTECTED]>:
>
>> I have a program that does a lot of iterations and would like
>> to follow its progress by having it print out the current iteration
>> number as it progresses. How do I do this so that it appears
>> like a counter that increases in the same place in the terminal
>> window? I am using python2.5 on a Mac OSX and its terminal
>> tcsh window.
>
> print a carriage return before, and no line feed after, each line:
>
> for i in range(100):
> print "\r" + "count", i,
> print # done
>
> 
>
>
>
> -- 
> http://mail.python.org/mailman/listinfo/python-list

-- 
http://mail.python.org/mailman/listinfo/python-list

os.mkfifo

[Gigs_]

is os.mkfifo available on windows xp

thx
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Read from database, write to another database, simultaneously

[Paul Boddie]

Sean Davis wrote:
>
> As for the specifics, Oracle data is going to be coming in as a DB-API
> 2 cursor in manageable chunks (and at a relatively slow pace).  On the
> postgres loading side, I wanted to use the pscycopg2 copy_from
> function, which expects an open file-like object (that has read and
> readline functionality) and is quite fast for loading data.

And which seems to use the COPY FROM command in PostgreSQL...

> Note the disconnect here--Oracle is coming in in discrete chunks, while
> postgresql is looking for a file object.  I solved this problem by
> creating a temporary file as an intermediary, but why wait for Oracle
> to finish dumping data when I can potentially be loading into postgres
> at the same time that the data is coming in?

My experience along with the PostgreSQL documentation tells me that you
shouldn't worry about this problem too much: using COPY FROM is *far*
faster than performing many inserts or updates. The only thing you'd
need to worry about is data being copied into the database that
duplicates existing data, causing constraint violations, but since
you're already using this method I imagine that this is not a likely
problem.

Paul

-- 
http://mail.python.org/mailman/listinfo/python-list

Learning Python book, new edition?

[Demel, Jeff]

Does anyone know if there's a plan in the works for a new edition of
Learning Python?  The current edition (2nd) is a few years old and looks
like it only covers Python 2.3.

Anyone on the list have Lutz's ear?

-Jeff
This email is intended only for the individual or entity to which it is 
addressed.  This email may contain information that is privileged, confidential 
or otherwise protected from disclosure. Dissemination, distribution or copying 
of this e-mail or any attachments by anyone other than the intended recipient, 
or an employee or agent responsible for delivering the message to the intended 
recipient, is prohibited. If you are not the intended recipient of this message 
or the employee or agent responsible for delivery of this email to the intended 
recipient, please notify the sender by replying to this message and then delete 
it from your system.  Any use, dissemination, distribution, or reproduction of 
this message by unintended recipients is strictly prohibited and may be 
unlawful.
-- 
http://mail.python.org/mailman/listinfo/python-list

ValueError from dict - some detail would be helpful

[metaperl]

  File "/sw/lib/python2.5/csv.py", line 120, in _dict_to_list
raise ValueError, "dict contains fields not in fieldnames"


--- it would be nice if it said what field it was

I know that I can do a set difference on the two myself, but since it
know what wasn't there.. why not report it and save me some time?

-- 
http://mail.python.org/mailman/listinfo/python-list

Newbie question: SMTP -> SQL Server

[jrpfinch]

I have an externally-written piece of software that spits out emails
using SMTP (a few hundred per hour) and I would like to dump the
content of them in an MS SQL Server database.

I have barely used Python before but it looks as if it could do the
job.  I have no experience with mail agents.  My starting point is this
recipe:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440690

I know that there are various libraries that can connect to MS SQL
server and I am going to research these.

I have two questions:

i)  Does this sound like an efficient way of transferring data to the
database?
ii)  Looking through the Python documentation, I cannot see a way to
set the password on the SMTP server.  I would like to have a password -
does anyone know how to use the recipe above to set one?

Any other useful nudges in the right direction appreciated.

Many thanks

Jon

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: __init__ vs __new__

[Neil Cerutti]

On 2007-01-11, Daniel Klein <[EMAIL PROTECTED]> wrote:
> I've have a program that is using both of the methods below (in
> different classes of course) for initializing the class. The
> example below shows a class with the 2 methods with one
> commented out.
>
> class JsubroutineParameters(list):
> "Represents a list of arguments for external subroutine calls."
> # We only need to override methods which add objects to the list.
>
> def __init__(self, alist = []):
> for objekt in alist: _validateParameter(objekt)
> self.extend(alist)
>
> #def __new__(cls, alist = []):
> #for objekt in alist: _validateParameter(objekt)
> #return list.__new__(cls, alist)
>
> I don't really notice any behavioral difference. Is there in
> fact any difference in using one over the other? Performance?
> Side effects? ???
>
> I am using Python version 2.5.

The second version doesn't work the way you might be assuming.

Guido's paper says that mutable builtins like list have a dummy
__new__ static method. So 'return list.__new__(cls, alist)' does
nothing. It seems to work because the base class __init__
performs the initialization (assuming your __init__ above is
commented out). You can see this by providing a dummy __init__.

class Example(list):
  def __new__(cls, alist):
return list.__new__(cls, alist)
  def __init__(self, alist):
pass

>>> a = Example(range(5))
>>> a
[]

There is no need to use __new__ for mutable builtin types. Since
all you want from this construction is a side-effect, you can
nevertheless use it in this case.

Your __init__ should call the base-class __init__.

It's usually a bad idea to provide mutable types as default
arguments. Since you aren't modifying alist in __init__ you can
get away with it here, but getting in the habit of using the
below idiom might save you from a "gotcha" someday.

class JsubroutineParameters(list):
  def __init__(self, alist=None):
if alist is None:
  alist = []
for objekt in alist: _validateParameter(objekt)
list.__init__(self, alist)

You will no longer need to call append.

-- 
Neil Cerutti
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 2.5 install on Gentoo Linux: failed dmb and _tkinter

[Sorin Schwimmer]

> Did you add /usr/local/lib to /etc/ld.so.conf?

It's there

Sorin


 

Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail beta.
http://new.mail.yahoo.com
-- 
http://mail.python.org/mailman/listinfo/python-list

What happened to SPE?

[Paulo Pinto]

Hi,

does anyone know what happened to SPE?

It seems that the address http://pythonide.stani.be
is no longer valid. :(

Thanks in advance,
Paulo 


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Newbie question: SMTP -> SQL Server

[hg]

jrpfinch wrote:

> I have an externally-written piece of software that spits out emails
> using SMTP (a few hundred per hour) and I would like to dump the
> content of them in an MS SQL Server database.
> 
> I have barely used Python before but it looks as if it could do the
> job.  I have no experience with mail agents.  My starting point is this
> recipe:
> 
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440690
> 
> I know that there are various libraries that can connect to MS SQL
> server and I am going to research these.
http://sourceforge.net/projects/mysql-python
> 
> I have two questions:
> 
> i)  Does this sound like an efficient way of transferring data to the
> database?
Not certain I understand

> ii)  Looking through the Python documentation, I cannot see a way to
> set the password on the SMTP server.  I would like to have a password -
> does anyone know how to use the recipe above to set one?

s = smtplib.SMTP()

s.login(...,'password')
> 
> Any other useful nudges in the right direction appreciated.
> 
> Many thanks
> 
> Jon

hg


-- 
http://mail.python.org/mailman/listinfo/python-list

Problem with byte-compiled code

[eXt]

Hi, I have built some modules in C++to extend python. At first
everything worked but then I began splitting my python code into
several modules. The application begins in main.py and then imports my
other modules.

The first time I run the application it works, but when python imports
the byte compiled module all I get is a blank window. Moving back all
the code into the same source file works.

I have found that removing a call to a C++ function solves the problem.
However that function is needed and I found that if I print anything
inside that function it works. It isn't time related since I tried to
sleep instead but that didn't work either.

I'm not sure if this is the correct list but I hope someone can help me
a bit.

The C++ function:

TaskScheduler& createTaskScheduler(int threadCnt, const char* logfile){
//print("createTaskScheduler");
TaskScheduler::initialize(threadCnt,logfile);
return TaskScheduler::singleton();
}

Boost exporting:

def("createTaskScheduler", createTaskScheduler,
return_value_policy());

Python code:

ts = Core.createTaskScheduler(1,'')

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Need startup suggestions for writing a MSA viewer GUI in python

[Joel Hedlund]

> UI design requires a different skillset than programming. It can be a
> very frustrating and thankless task as well. It is incomparably easier
> to see the flaws in existing interfaces than correcting them (or even
> creating the said interface). Make sure to start with something simple,
> and learn that way.
> 
> I would also recommend that you implement something novel, there are
> many existing MSA viewers and it won't be easy to improve on them. Do
> something that adds new value and people will be willing to expend
> effort to use it.

That's true for any software. And for any aspect of life, come to think 
about it.

/Joel
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What happened to SPE?

[Neil Cerutti]

On 2007-01-11, Paulo Pinto <[EMAIL PROTECTED]> wrote:
> does anyone know what happened to SPE?
>
> It seems that the address http://pythonide.stani.be
> is no longer valid. :(

SPE lost its web host, and last I heard is looking for a new
home. For now you can get it here: 

  http://sourceforge.net/projects/spe/

-- 
Neil Cerutti
We don't necessarily discriminate.  We simply exclude certain types of people.
--Colonel Gerald Wellman
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Progress count in terminal (Mac OS X)

[reed]

> > for i in range(100):
> > sys.stdout.write( "\r" + "count   ", i,)
>>  sys.stdout.flush()
> > print # done

maybe

Tommy Grav wrote:
> This certainly does work when running the interpreter interactively,
> but when inserted into a script it seems to buffer the print statement
> and not write it out to the terminal. How can I force the print
> statement
> to not buffer the output?
>
> Cheers
> Tommy
>
> On Jan 11, 2007, at 9:22 AM, Fredrik Lundh wrote:
>
> > Tommy Grav <[EMAIL PROTECTED]>:
> >
> >> I have a program that does a lot of iterations and would like
> >> to follow its progress by having it print out the current iteration
> >> number as it progresses. How do I do this so that it appears
> >> like a counter that increases in the same place in the terminal
> >> window? I am using python2.5 on a Mac OSX and its terminal
> >> tcsh window.
> >
> > print a carriage return before, and no line feed after, each line:
> >
> > for i in range(100):
> > print "\r" + "count", i,
> > print # done
> >
> > 
> >
> >
> >
> > -- 
> > http://mail.python.org/mailman/listinfo/python-list

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Newbie question: SMTP -> SQL Server

[jrpfinch]

Thank you.  I have just realised I completely misunderstand how SMTP
servers work.  From what I can tell, when you run the cookbook script
it listens locally on port 8025.

You then have to configure a Linux (in my case) account with a username
and password so my external piece of software (on another Windows
machine) can log in and use the SMTP server.

Then I write the some code in the cookbook example to redirect any mail
content to the MS SQL server database.

Note that the external piece of software can only talk SMTP - this is
why I am having to develop this script.

Does this sound sensible?  Any tips on how to configure my Linux box (I
don't have much experience with Linux either)?

Many thanks

Jon

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dot operations

[Steven W. Orr]

On Thursday, Jan 11th 2007 at 11:41 +0100, quoth robert:

=>[EMAIL PROTECTED] wrote:
=>> Hi,
=>>  Frequently I get to do like this:
=>> a = (1, 2, 3, 4) # some dummy values
=>> b = (4, 3, 2, 1)
=>> import operator
=>> c = map(operator.add, a, b)
=>> 
=>> I am finding the last line not very readable especially when I combine
=>> couple of such operations into one line. Is it possible to overload
=>> operators, so that, I can use .+ for element wise addition, as,
=>> c = a .+ b
=>> which is much more readable.
=>> 
=>> Similarly, I want to use .- , .*, ./   . Is it possible to do?
=>
=>import numpy
=>
=>You'll not even need dots

I'm very new so my plea to be gentle is still on.

I just looked at the numpy package. Seems very cool, but for the life of 
me I didn't understand the method by which python allows for creation of 
infix operators. Can someone please explain or point me to a reference?

TIA

-- 
Time flies like the wind. Fruit flies like a banana. Stranger things have  .0.
happened but none stranger than this. Does your driver's license say Organ ..0
Donor?Black holes are where God divided by zero. Listen to me! We are all- 000
individuals! What if this weren't a hypothetical question?
steveo at syslang.net
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What happened to SPE?

[Dick Moores]

At 07:43 AM 1/11/2007, Paulo Pinto wrote:
>does anyone know what happened to SPE?
>
>It seems that the address http://pythonide.stani.be
>is no longer valid. :(

I'd suggest subscribing to the Python-spe-users list, 
, or 
reading the archive there.

Dick Moores



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: os.mkfifo

[Jerry Hill]

On 1/11/07, Gigs_ <[EMAIL PROTECTED]> wrote:
> is os.mkfifo available on windows xp

http://docs.python.org/lib/os-file-dir.html#l2h-2683

-- 
Jerry
-- 
http://mail.python.org/mailman/listinfo/python-list

Portable Python - free portable development environment !

[perica . zivkovic]

Hi there,

I would like to announce the *first* beta release of the Portable
Python 1.0 beta. From today Portable Python website is also online and
you can find it on the location www.PortablePython.com.

About:
Portable Python is a Python programming language preconfigured to run
directly from a portable device (USB stick, iPod etc.), enabling you to
have, at any time, portable programming environment. Just download and
extract to your portable device and in 10 minutes you are ready to
create your next Python application. No registry modifications,
installations, totally portable.

Where to find it:
http://www.portablepython.com

regards,

Perica Zivkovic

P.S. did I mentioned that it is free? :-)

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Newbie question: SMTP -> SQL Server

[Steve Holden]

jrpfinch wrote:
> Thank you.  I have just realised I completely misunderstand how SMTP
> servers work.  From what I can tell, when you run the cookbook script
> it listens locally on port 8025.
> 
> You then have to configure a Linux (in my case) account with a username
> and password so my external piece of software (on another Windows
> machine) can log in and use the SMTP server.
> 
> Then I write the some code in the cookbook example to redirect any mail
> content to the MS SQL server database.
> 
> Note that the external piece of software can only talk SMTP - this is
> why I am having to develop this script.
> 
> Does this sound sensible?  Any tips on how to configure my Linux box (I
> don't have much experience with Linux either)?
> 
> Many thanks
> 
> Jon
> 
Presumably you can configure the external piece of software at least to 
the extent of setting the email address it delivers to?

In that case it will be *much* simpler to set up an account on a mail 
server to receive these emails. That way the server will be available 
whenever the SMTP-based sender wants or needs to send mail, and you can 
use Python to collect the emails in batched using the POP protocol, as a 
client.

Once you receive the email messages then yes, it's perfectly practical 
to save the content in a SQL server database. mxODBC is one possible 
driver package (though it requires you to purchase a license for 
commercial use), and I seem to remember there's another one called 
adodbapi that is open source. There are almost certainly others, but 
it's a while since I used SQL Server with Python.

regards
  Steve
-- 
Steve Holden   +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd  http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Blog of Note:  http://holdenweb.blogspot.com

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Progress count in terminal (Mac OS X)

[James Thiele]

if you invoke python with the -u option the output of print is
unbuffered.

On Jan 11, 7:04 am, Tommy Grav <[EMAIL PROTECTED]> wrote:
> This certainly does work when running the interpreter interactively,
> but when inserted into a script it seems to buffer the print statement
> and not write it out to the terminal. How can I force the print
> statement
> to not buffer the output?
>
> Cheers
> Tommy
>
> On Jan 11, 2007, at 9:22 AM, Fredrik Lundh wrote:
>
> > Tommy Grav <[EMAIL PROTECTED]>:
>
> >> I have a program that does a lot of iterations and would like
> >> to follow its progress by having it print out the current iteration
> >> number as it progresses. How do I do this so that it appears
> >> like a counter that increases in the same place in the terminal
> >> window? I am using python2.5 on a Mac OSX and its terminal
> >> tcsh window.
>
> > print a carriage return before, and no line feed after, each line:
>
> > for i in range(100):
> > print "\r" + "count", i,
> > print # done
>
> > 
> 
> > --
> >http://mail.python.org/mailman/listinfo/python-list

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python

[Nick Maclaren]

In article <[EMAIL PROTECTED]>,
robert <[EMAIL PROTECTED]> writes:
|> 
|> Thus there are different levels of parallelization:
|> 
|> 1 file/database based; multiple batch jobs
|> 2 Message Passing, IPC, RPC, ...
|> 3 Object Sharing 
|> 4 Sharing of global data space (Threads)
|> 5 Local parallelism / Vector computing, MMX, 3DNow,...
|> 
|> There are good reasons for all of these levels.

Well, yes, but to call them "levels" is misleading, as they are closer
to communication methods of a comparable level.

|> > This does not mean that MPI is inherently slower than threads however,
|> > as there are overhead associated with thread synchronization as well.
|> 
|> level 2 communication is slower. Just for selected apps it won't matter a 
lot.

That is false.  It used to be true, but that was a long time ago.  The
reasons why what seems to be a more heavyweight mechanism (message
passing) can be faster than an apparently lightweight one (data sharing)
are both subtle and complicated.


Regards,
Nick Maclaren.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: globals accros modules

[Bruno Desthuilliers]

stef a écrit :
> 
>>
>> Change a=1 to amodule.a=1
>> If you find yourself doing tricks with the module globals, think about 
>> redesigning your application.
>>
> Of course I completely agree with you.
> 
> But ...
> if you're moving from MatLab to Python,
> and want to show your collegaes,
> with how little effort they can reuse all their existing MatLab routines 
> in Python,
> then the global issue is a real pain !!
> 
> You can explain your collegaes, that
> - the startindex of arrays changes from 1 to 0
> - slices are upto, instead of including the final border
> - indention is thé key
> And tell them about all beautiful things in Python,
> but tell them that they are going to loose all their globals ???

It's a feature. Globals are definitively a BadThing(tm).

> cheers,
> Stef Mientki
> 
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: os.popen() not executing command on windows xp

[nic]

Justin Ezequiel wrote:
> > import os
> > a = os.popen('"c:\Program Files\Grisoft\AVG Free\avgscan.exe"
> > "c:\program files\temp1\test1.txt"')
> > print a.read()
> >
>
> use raw strings
>
> e.g., instead of '"c:...\avgscan...'
> use r'"c:...\avgscan...'
>
> http://www.ferg.org/projects/python_gotchas.html#contents_item_2

Sorry I initally had retyped the code to replace some variables to make
it more clear, I already had them as raw strings:

import os
pstr = r'"c:\Program Files\Grisoft\AVG Free\avgscan.exe" "c:\program
files\temp1\test1.txt"'
a = os.popen(pstr)
print a.read()

I've confirmed the string I'm inputting to os.popen is EXACTLY the same
as the one I can successfully execute manually in command prompt, so
when I go:
print pstr, it yields:
"c:\Program Files\Grisoft\AVG Free\avgscan.exe" "c:\program
files\temp1\test1.txt"

The problem remains popen won't execute this line as it does when
inputted manually to command prompt.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Learning Python book, new edition?

[Bjoern Schliessmann]

Demel, Jeff wrote:

> Does anyone know if there's a plan in the works for a new edition
> of Learning Python?  The current edition (2nd) is a few years old
> and looks like it only covers Python 2.3.

IIRC, differences to 2.4 are in it, too.

> This email is intended only for the individual or entity to which
> it is addressed.  

Mails are _by_definition_ intended only for the addressee. :)

Regards,


Björn

-- 
BOFH excuse #57:

Groundskeepers stole the root password

-- 
http://mail.python.org/mailman/listinfo/python-list

COM compatibility

[Tmack]

I'm not a programmer!

I work for a software company. We have a SDK that customers can use to
customize the app. The requirement to use the SDK is:

XYZ App has been designed in such a way that all the business objects
used
in the application are automatically exposed through a thin COM wrapper
created
automatically during the application build process. This means that
using the
XYZ App SDK is very efficient, and all the same underlying objects and
properties that the application's programmers use to build the XYZ App
user
interface are also available to third-party programmers via the SDK.

Does it sound like Python would allow me use the apps SDK?

Thanks

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What about this?

[Bjoern Schliessmann]

new wrote:

> www.magicoz.com
> amazing

Yeah, it *is* really amazing that someone dares to spam for such an
unprofessional homepage. Even too stupid to include a doctype ...


Björn

-- 
BOFH excuse #61:

not approved by the FCC

-- 
http://mail.python.org/mailman/listinfo/python-list

Fixed keys() mapping

[George Sakkis]

I wrote an 'fkdict' dict-like class for mappings with a fixed set of
keys but I'm wondering if there's a simpler way to go about it.

First off, the main motivation for it is to save memory in case of many
dicts with the same keys, for example when reading from a
csv.DictReader or constructing dicts out of rows fetched from a
database. For example, test_mem(dict) takes up around 246 MB according
to the Windows task manager while test_mem(fkdict) takes around 49 MB:

def test_mem(maptype):
d = [(i,str(i)) for i in range(1000)]
ds = [maptype(d) for i in xrange(1)]
raw_input('finished')

An additional benefit is predictable ordering (e.g.
fkdict.fromkeys('abcd').keys() == list('abcd')), like several
ordered-dict recipes.

The implementation I came up with goes like this: each fkdict instance
stores only the values as a list in self._values. The keys and the
mapping of keys to indices are stored in a dynamically generated
subclass of fkdict, so that self._keys and self._key2index are also
accessible from the instance. The dynamically generated subclasses are
cached so that the second time an fkdict with the same keys is created,
the cached class is called.

Since the keys are determined in fkdict.__init__(), this scheme
requires changing self.__class__ to the dynamically generated subclass.
As much as I appreciate Python's dynamic nature, I am not particularly
comfortable with objects that change their class and the implications
this may have in the future (e.g. how well does this play with
inheritance). Is this a valid use case for type-changing behavior or is
there a better, more "mainstream" OO design pattern for this ? I can
post the relevant code if necessary.

George

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: what is the idiom for copy lots of params into self?

[Bjoern Schliessmann]

Emin wrote:

> Thanks for your suggestions. One issue with using *args or **kw is
> that I might no want to copy all the arguments to __init__ into
> self.

Try prepending something like

allowedParms = ("spam", "eggs", "yum")
args = dict([key,val for key,val in args.elements() if key in
allowedParms])

Regards,


Björn

-- 
BOFH excuse #64:

CPU needs recalibration

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Read from database, write to another database, simultaneously

[Bjoern Schliessmann]

Laurent Pointal wrote:

> Not so sure, there is low CPU in the Python script,

Yes.

> but there may be CPU+disk activity on the database sides [with
> cache management and other optimizations on disk access].

That's it. So data queues up on the database side and you won't get
much value from faked concurrency with CPU cycles.

> So, with a reader thread and a writer thread, he can have a select
> on a database performed in parallel with an insert on the other
> database.

Explain. Remember, threads aren't really working concurrently. Even
on a multiprocessor machine you have constraints for IO traffic.
(And the GIL exists too)

> But, if its only a do-once job, maybe the optimization is net
> really necessary.

I still don't understand how threads would help optimizing a task
that largely depends on IO and will probably be executed on one
CPU.

Regards,


Björn

-- 
BOFH excuse #14:

sounds like a Windows problem, try calling Microsoft support

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: COM compatibility

[Diez B. Roggisch]

Tmack wrote:

> I'm not a programmer!
> 
> I work for a software company. We have a SDK that customers can use to
> customize the app. The requirement to use the SDK is:
> 
> XYZ App has been designed in such a way that all the business objects
> used
> in the application are automatically exposed through a thin COM wrapper
> created
> automatically during the application build process. This means that
> using the
> XYZ App SDK is very efficient, and all the same underlying objects and
> properties that the application's programmers use to build the XYZ App
> user
> interface are also available to third-party programmers via the SDK.
> 
> Does it sound like Python would allow me use the apps SDK?

It sounds as if it is possible scripting it with python using Mark Hammond's
win32, yes. 

Diez
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Read from database, write to another database, simultaneously

[Bjoern Schliessmann]

Sean Davis wrote:

> I solved this problem by creating a temporary file as an
> intermediary, but why wait for Oracle to finish dumping data when
> I can potentially be loading into postgres at the same time that
> the data is coming in?  So, I am actually 
> looking for a solution to this problem that doesn't require an
> intermediate file and allows simultaneous reading and writing,
> with the caveat that the data cannot all be read into memory
> simultaneously, so will need to be buffered.

The functions you use don't seem very suited for such a
streaminglike task.

Regards,


Björn

-- 
BOFH excuse #282:

High altitude condensation from U.S.A.F prototype aircraft has
contaminated the primary subnet mask. Turn off your computer for 9
days to avoid damaging it.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python

[Konrad Hinsen]

On Jan 8, 2007, at 11:33, Duncan Booth wrote:

> The 'parallel python' site seems very sparse on the details of how  
> it is
> implemented but it looks like all it is doing is spawning some  
> subprocesses
> and using some simple ipc to pass details of the calls and results.  
> I can't
> tell from reading it what it is supposed to add over any of the other
> systems which do the same.
>
> Combined with the closed source 'no redistribution' license I can't  
> really
> see anyone using it.

I'd also like to see more details - even though I'd probably never  
use any Python module distributed in .pyc form only.

 From the bit of information there is on the Web site, the  
distribution strategy looks quite similar to my own master-slave  
distribution model (based on Pyro) which is part of ScientificPython.  
There is an example at

http://dirac.cnrs-orleans.fr/hg/ScientificPython/main/? 
f=08361040f00a;file=Examples/master_slave_demo.py

and the code itself can be consulted at

http://dirac.cnrs-orleans.fr/hg/ScientificPython/main/? 
f=bce321680116;file=Scientific/DistributedComputing/MasterSlave.py


The main difference seems to be that my implementation doesn't start  
compute jobs itself; it leaves it to the user to start any number he  
wants by any means that works for his setup, but it allows a lot of  
flexibility. In particular, it can work with a variable number of  
slave jobs and even handles disappearing slave jobs gracefully.

Konrad.
--
-
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: [EMAIL PROTECTED]
-


-- 
http://mail.python.org/mailman/listinfo/python-list

RE: Learning Python book, new edition?

[Demel, Jeff]

>Demel, Jeff wrote:
>> Does anyone know if there's a plan in the works for a new edition of 
>> Learning Python?  The current edition (2nd) is a few years old and 
>> looks like it only covers Python 2.3.

Björn replied:
>IIRC, differences to 2.4 are in it, too.

Interesting.  The description I read said 2.3.  That's good to know.

I'd still like to know if there will be a new edition soon, covering 2.5.  
Anyone?


>Jeff's mail server auto-appended:
>> This email is intended only for the individual or entity to which it 
>> is addressed.

Björn replied:
>Mails are _by_definition_ intended only for the addressee. :)

According to our legal department it must be specified, apparently.  They 
append that legal crap to every email, and it annoys me to no end.

-Jeff



This email is intended only for the individual or entity to which it is 
addressed.  This email may contain information that is privileged, confidential 
or otherwise protected from disclosure. Dissemination, distribution or copying 
of this e-mail or any attachments by anyone other than the intended recipient, 
or an employee or agent responsible for delivering the message to the intended 
recipient, is prohibited. If you are not the intended recipient of this message 
or the employee or agent responsible for delivery of this email to the intended 
recipient, please notify the sender by replying to this message and then delete 
it from your system.  Any use, dissemination, distribution, or reproduction of 
this message by unintended recipients is strictly prohibited and may be 
unlawful.
-- 
http://mail.python.org/mailman/listinfo/python-list

Frequency spectrum with fft of a real valued array...?

[Holger]

Dear all,

I need to do a FFT on an array of 20k real values. Origin of the sampled  
data is a sinus wave with light harmonics.
The goal is an frequency spectrum with the magnitudes of the first 50.  
harmonics.

I addressed python like:

test_arr = src_data_dict[ channel ][0:19599]
target_data_dict[ channel ] = FFT.fft(test_arr,n=50,axis=-1)

This results in an array of complex values but with unexpected big  
coefficients...
(much higher than the original signal magnitudes)
How do I get the absolute magnitudes of the harmonics, which are much  
lower than the
resulting absolute values of the coefficients?

The result should be like this:

1.harmonic (50 Hz)  1,0
2.harmonic (100 Hz) 0,01
3.harmonic (100 Hz) 0,08
4.harmonic (100 Hz) 0,0035
etc.


at the moment I get a resulting array like:

CH1
(1729.80103418+0j)
(366.689810532+19.5196963754j)
(370.688444025+32.162562652j)
(372.122246668+46.9545880507j)
(379.273599053+59.0724599622j)
(369.889589421+75.9247281559j)
(381.070551892+99.07345873j)
(378.800462354+106.761629308j)
(375.014128346+131.34177586j)
(389.110601354+149.320740829j)
(389.23247472+158.909042086j)
(398.875237165+197.86980788j)
(397.927158223+196.858459101j)
(402.455325066+234.651276425j)
(411.599088579+256.32156894j)
(414.469935576+254.512014918j)
(417.198515262+291.400509132j)
(426.745545674+320.769421334j)
(433.987466212+321.929780157j)
(446.124386798+350.810581686j)
(455.876025379+383.099789898j)
(458.083277747+405.592129477j)
(470.908512117+433.929598454j)
(482.083855098+468.256188814j)

What does it mean to me? How do I get to the wanted frequenca spectrum???
..

btw The maximum magnitudes of the original data are app. 70 peak

Thanks in advance for your help!!!

Regards Holger
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Frequency spectrum with fft of a real valued array...?

[Robert Kern]

Holger wrote:

> What does it mean to me? How do I get to the wanted frequenca spectrum???

It's packed in the conventional FFT format. Here is a function in numpy (the
successor to Numeric, which I assume that you are using) that generates the
corresponding frequencies in the same packed format:

In [324]: import numpy

In [325]: numpy.fft.fftfreq?
Type:   function
Base Class: 
Namespace:  Interactive
File:
/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy-1.0.2.dev3507-py2.5-macosx-10.4-i386.egg/numpy/fft/helper.py
Definition: numpy.fft.fftfreq(n, d=1.0)
Docstring:
fftfreq(n, d=1.0) -> f

DFT sample frequencies

The returned float array contains the frequency bins in
cycles/unit (with zero at the start) given a window length n and a
sample spacing d:

  f = [0,1,...,n/2-1,-n/2,...,-1]/(d*n) if n is even
  f = [0,1,...,(n-1)/2,-(n-1)/2,...,-1]/(d*n)   if n is odd


-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python - C# interoperability

[Larry Bates]

mc wrote:
> Is there an easy way to compile a Python class (or set of classes) into
> a .DLL that a C# program can call?  Or otherwise to use an existing
> library of Python classes from a C# program as seamlessly as possible?
> 

You can write COM objects that can be called from C# (or basically ANY
other language that can call COM objects) quite easily and compile them
with py2exe for distribution without python itself.

-Larry Bates
-- 
http://mail.python.org/mailman/listinfo/python-list