Re: slice iterator?

[ryles]

On May 8, 11:17 pm, Ross  wrote:
> I have a really long list that I would like segmented into smaller
> lists. Let's say I had a list a = [1,2,3,4,5,6,7,8,9,10,11,12] and I
> wanted to split it into groups of 2 or groups of 3 or 4, etc. Is there
> a way to do this without explicitly defining new lists? If the above
> problem were to be split into groups of 3, I've tried something like:
>
> start = 0
> stop = 3
> for i in range(len(a)):
>     segment = a[start:stop]
>     print segment
>     start += stop
>     stop += stop
>
> Unfortunately start and stop don't increment using this code. Ideally,
> my outcome would be
> [1,2,3]
> [4,5,6]
> [7,8,9]
> [10,11,12]

There is also an iterator version here:
http://code.activestate.com/recipes/303279
--
http://mail.python.org/mailman/listinfo/python-list

Re: unpickling a stream

[ryles]

On May 26, 3:25 pm, mso...@linuxmail.org wrote:
> Hello,
>
> I want to send a stream of pickled objects over a socket.  Is there a
> standard way of ensuring that only complete objects are unpickled on
> the receiving side.
>
> client pseudo code:
>   loop forever:
>     receive some bytes on the socket
>     if we have received a complete pickled object:  <== How is this
> done?
>       unpickle the object
>
> This isn't for a project, just trying to learn some more about the
> pickle module.
>
> Mike

If you'd like, you can turn a socket into a file-like object using its
makefile() method. An Unpickler will know when to stop reading from
the file (there's a STOP character in the pickle format). David
Beazley gives an example of this approach in Part 7 of "Generator
Tricks for Systems Programmers": http://www.dabeaz.com/generators/Generators.pdf
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: question about the GC implementation

[ryles]

On Oct 2, 11:20 am, Francis Moreau  wrote:
> Hello,
>
> I'm looking at gcmodule.c and in move_unreachable() function, the code
> assumes that if an object has its gc.gc_refs stuff to 0 then it *may*
> be unreachable.
>
> How can an object tagged as unreachable could suddenly become
> reachable later ?
>
> Thanks

It looks like you're not reading through all of the comments:

GC_TENTATIVELY_UNREACHABLE
move_unreachable() then moves objects not reachable (whether
directly or
indirectly) from outside the generation into an "unreachable" set.
Objects that are found to be reachable have gc_refs set to
GC_REACHABLE
again.  Objects that are found to be unreachable have gc_refs set
to
GC_TENTATIVELY_UNREACHABLE.  It's "tentatively" because the pass
doing
this can't be sure until it ends, and GC_TENTATIVELY_UNREACHABLE
may
transition back to GC_REACHABLE.

Only objects with GC_TENTATIVELY_UNREACHABLE still set are
candidates
for collection.  If it's decided not to collect such an object
(e.g.,
it has a __del__ method), its gc_refs is restored to GC_REACHABLE
again.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: weak reference to bound method

[ryles]

On Oct 2, 4:54 am, Ole Streicher  wrote:
> Hi group,
>
> I am trying to use a weak reference to a bound method:
>
> class MyClass(object):
>     def myfunc(self):
>         pass
>
> o = MyClass()
> print o.myfunc
>
>    >
>
> import weakref
> r = weakref.ref(o.myfunc)
> print r()
>
>    None
>
> This is what I do not understand. The object "o" is still alive, and
> therefore the bound method "o.myfunc" shall exists.
>
> Why does the weak reference claim that it is removed? And how can I hold
> the reference to the method until the object is removed?
>
> Is this a bug or a feature? (Python 2.6)
>
> Best regards
>
> Ole

Have a look at: http://mindtrove.info/articles/python-weak-references
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Need feedback on subprocess-using function

[ryles]

On Oct 4, 9:46 pm, Nobody  wrote:
> On Sat, 03 Oct 2009 13:21:00 +, gb345 wrote:
> > I'm relatively new to Python, and I'm trying to get the hang of
> > using Python's subprocess module.  As an exercise, I wrote the Tac
> > class below, which can prints output to a file "in reverse order",
> > by piping it through the Unix tac utility.  (The idea is to delegate
> > the problem of managing the memory for an arbitrarily large task
> > to tac.)
> >         self.pipe = subprocess.Popen(['tac'], stdout=out,
> >                                      stdin=subprocess.PIPE,
> >                                      stderr=subprocess.PIPE)
> > This works OK, as far as I can tell, but I'm not sure that I've
> > dotted all the i's and crossed all the t's...  E.g., I had to add
> > the line "p.stdin.close()" to the close method when I when I ran
> > into sporadic deadlock at the p.stderr.read() statement.  Are there
> > other similar problems lurking in this code?
>
> Yep. If the process writes more than a buffer-full of data to stderr, it
> will deadlock. tac will block trying to write to stderr, and won't be
> reading its stdin, so your program will block trying to write to tac.
>
> This is why the POSIX popen() call only lets you attach a pipe to stdin or
> stdout but not both.
>
> If you want a "double-ended" slave process, you need to use polling or
> non-blocking I/O or asynchronous I/O or multiple threads. I'm not aware of
> any single solution which works on all platforms.
>
> The easiest way around this problem is to redirect stderr to a temporary
> file and read in the file's contents in the close() method.

There is also Popen.communicate():

http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python shared lib

[ryles]

On Oct 4, 11:04 am, a...@pythoncraft.com (Aahz) wrote:
> I've got a dim memory that there's a reason for this -- you might try
> searching the python-dev archives and/or bugs.python.org.

Found mention of it in this discussion:

http://bugs.python.org/issue771998

"I disagree that --enable-shared should be the default. It is
difficult to maintain, as finding shared libraries is always
a PITA (in particular on Solaris). Its only real use is for
embedding, in even in the case of embedding, it is possible
to avoid using shared libraries."
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to find number of line that is currently executing?

[ryles]

On Oct 9, 11:46 pm, "Dr. Phillip M. Feldman" 
wrote:
> I would like to put a statement on line N of my program that prints the line
> number that is currently executing.

inspect.currentframe().f_lineno

http://docs.python.org/library/inspect.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Script to complete web form fields

[ryles]

On Oct 10, 9:39 pm, Feyo  wrote:
> How can I use Python to complete web form fields automatically? My
> work web-based email time-out is like 15 seconds. Every time I need to
> access my calendar, address book, or email, I have to type in my
> username and password. I'm just tired of it.
>
> I found the ClientForm module and have been working with it. Can do
> what I want in a unit testing sense to have responses sent and
> returned, but what I want to do is much simpler. Any ideas? Thanks.

See this recent thread for ideas:

http://groups.google.com/group/comp.lang.python/browse_thread/thread/58b7513596ab648c

However, why not take a more direct approach? Speak to management and
see if your company can correct this. It's obviously hindering
productivity, and that's something they may have overlooked. There
should be a more reasonable compromise between efficiency and
security. It's worth a shot.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: code in a module is executed twice (cyclic import problems) ?

[ryles]

On Oct 10, 7:36 pm, Stef Mientki  wrote:
> hello,
>
> I always thought code in a module was only executed once,
> but doesn't seem to be true.
>
> I'm using Python 2.5.
>
> And this is the example:
>
> == A.py ==
> My_List = []
>
> == B.py ==
> from A import *
> My_List.append ( 3 )
> print 'B', My_List
> import C
>
> == C.py ==
> from A import *
> from B import *
> print 'C', My_List
>
> Now when you start with B.py as the main program,
> this is the resulting output:
>
> B [3]
> B [3, 3]
> C [3, 3]
>
> Why is the B.py executed twice ?
>
> thanks,
> Stef

FYI, there was actually a related discussion about this just recently:

http://groups.google.com/group/comp.lang.python/browse_thread/thread/e24be42ecbee7cad
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: strange behaviour when inheriting from tuple

[ryles]

On Oct 11, 3:04 am, metal  wrote:
> Environment:
>
> PythonWin 2.5.4 (r254:67916, Apr 27 2009, 15:41:14) [MSC v.1310 32 bit
> (Intel)] on win32.
> Portions Copyright 1994-2008 Mark Hammond - see 'Help/About PythonWin'
> for further copyright information.
>
> Evil Code:
>
> class Foo:
>         def __init__(self, *args):
>                 print args
>
> Foo(1, 2, 3) # (1, 2, 3), good
>
> class Bar(tuple):
>         def __init__(self, *args):
>                 print args
>
> Bar(1, 2, 3) # TypeError: tuple() takes at most 1 argument (3 given)
>
> what the heck? I even didn't call tuple.__init__ yet

When subclassing immutable types you'll want to override __new__, and
should ensure that the base type's __new__ is called:

__ class MyTuple(tuple):
__ def __new__(cls, *args):
__ return tuple.__new__(cls, args)
__ print MyTuple(1, 2, 3)

(1, 2, 3)

See http://www.python.org/download/releases/2.2.3/descrintro/#__new__
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: restriction on sum: intentional bug?

[ryles]

On Oct 16, 11:39 am, Alan G Isaac  wrote:
> I expected this to be fixed in Python 3:
>
> >>> sum(['ab','cd'],'')
>
> Traceback (most recent call last):
>    File "", line 1, in 
> TypeError: sum() can't sum strings [use ''.join(seq) instead]

Then you probably haven't read through these discussions:

sum and strings: 
http://mail.python.org/pipermail/python-list/2006-August/subject.html
summing a bunch of numbers: 
http://mail.python.org/pipermail/python-dev/2003-April/subj
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How about adding slice notation to iterators/generators?

[ryles]

On Oct 16, 5:28 pm, Eloff  wrote:

> By giving iterators a default, overridable, __getitem__ that is just
> syntactic sugar for islice, they would share a slightly larger
> interface subset with the builtin container types. In a duck-typed
> language like python, that's almost always a good thing. You could use
> iterators in more situations which expect something more like a list.
>
> As long as it breaks no rationally existing code, I can think of no
> good reason why not to do this in a future python.

I think Python programmers have learned to expect certain things from
objects that support __getitem__. For example, indexing and slicing is
repeatable on the same object:

a[1] == a[1]

a[1:4] == a[1:4]

If you saw the above code would you want to think twice above whether
or not these expressions were true?

Iterators don't have a general concept of "get item" like types such
as strings, lists, etc. They have a concept of "get next item". So,
with your proposal, i[1] != i[1] and i[1:4] != i[1:4].

Not only that, it's also common for types with __getitem__ to have
__len__, which we obviously can't provide.

So, ultimately, although it could afford some small conveniences, I
think trying to mix iterators with __getitem__ would cause too much
confusion.

The use of islice() is both readable and explicit. It's very clear to
the reader that you're working with iterators and that items will be
consumed (something that's not reversible).
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Installation question 2.5.4

[ryles]

On Oct 17, 2:22 am, JimR  wrote:
> I completed the configure, the make and the make install.  However, the
> last instruction in README is to perform
> pkgmanager -a /usr/python
>
> As you may have guessed, Redhat does not have pkgmanager.  What can I do
> to work-around this problem and get this list running?

Don't worry, Jim, you don't need this.

It looks like you're reading from an irrelevant part of the README.

Here's the only part you typically need:

--

Congratulations on getting this far. :-)

To start building right away (on UNIX): type "./configure" in the
current directory and when it finishes, type "make".  This creates an
executable "./python"; to install in /usr/local, first do "su root"
and then "make install".

The section `Build instructions' below is still recommended reading.

--

I suspect you may have actually set your 'prefix' to /usr/python the
first time around. If that's the case, just remember to remove that
directory, since I doubt you'll want anything housed there. The
default (/usr/local) is usually fine. That's freely changeable with:

./configure --prefix=/my/prefix

Building from source can be a pain, because you may find that several
Python modules are not successfully built (it will tell you all about
this after 'make' completes). This is usually the result of various
'devel' packages (RPMs) not being installed. If this is the case, you
might have better luck not building python yourself, and just
installing a newer RPM for it.

Hope this helps. Good luck.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Installation question 2.5.4

[ryles]

On Sat, Oct 17, 2009 at 9:55 PM, JimR  wrote:
> Thanks.  As it turned out, I needed /usr/local/python instead of /usr/local
> as the prefix. After setting that, all worked as it should.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: restriction on sum: intentional bug?

[ryles]

On Oct 16, 6:53 pm, ryles  wrote:
> Then you probably haven't read through these discussions:
>
> sum and 
> strings:http://mail.python.org/pipermail/python-list/2006-August/subject.html
> summing a bunch of 
> numbers:http://mail.python.org/pipermail/python-dev/2003-April/subj

You meant:

sum and strings:
http://mail.python.org/pipermail/python-list/2006-August/subject.html

summing a bunch of numbers:
http://mail.python.org/pipermail/python-dev/2003-April/subject.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Anyone have python 3.1.1 installed on Solaris 10 ? (sparc or x86)

[ryles]

On Oct 19, 1:00 pm, Dave Crouse  wrote:
> Anyone have python 3.1.1 installed on Solaris 10 ? (sparc or x86)
>
> I've tried several times  on sparc, I keep getting:
>
> gcc -lintl -o python \
> Modules/python.o \
> libpython3.1.a -lsocket -lnsl -lintl -lrt -ldl -lm
> Undefined first referenced
> symbol in file
> libintl_bind_textdomain_codeset libpython3.1.a(_localemodule.o)
> libintl_gettext libpython3.1.a(_localemodule.o)
> libintl_textdomain libpython3.1.a(_localemodule.o)
> libintl_dcgettext libpython3.1.a(_localemodule.o)
> libintl_bindtextdomain libpython3.1.a(_localemodule.o)
> libintl_dgettext libpython3.1.a(_localemodule.o)
> ld: fatal: Symbol referencing errors. No output written to python
> collect2: ld returned 1 exit status
> *** Error code 1
> make: Fatal error: Command failed for target `python'
>
> Ideas ? (I'm not a c programmerso bear with me ;)  )

Have you tried building with cc instead of gcc? I've seen a few
botched gcc installations on Solaris (though maybe it's just my own
environment). Try adding --without-gcc to the ./configure line, and
before that export CC=cc (or however you set environment variables
with your own shell).
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Copying a ZipExtFile

[ryles]

On Oct 23, 1:15 pm, "Moore, Mathew L"  wrote:
> Hello all,
>
> A newbie here.  I was wondering why the following fails on Python 2.6.2 
> (r262:71605) on win32.  Am I doing something inappropriate?
>
> Interestingly, it works in 3.1, but would like to also get it working in 2.6.
>
> Thanks in advance,
> --Matt
>
> import io
> import shutil
> import tempfile
> import zipfile
>
> with tempfile.TemporaryFile() as f:
>     # (Real code retrieves archive via urllib2.urlopen().)
>     zip = zipfile.ZipFile(f, mode='w')
>     zip.writestr('unknowndir/src.txt', 'Hello, world!')
>     zip.close();
>
>     # (Pretend we just downloaded the zip file.)
>     f.seek(0)
>
>     # Result of urlopen() is not seekable, but ZipFile requires a
>     # seekable file.  Work around this by copying the file into a
>     # memory stream.
>     with io.BytesIO() as memio:
>         shutil.copyfileobj(f, memio)
>         zip = zipfile.ZipFile(file=memio)
>         # Can't use zip.extract(), because I want to ignore paths
>         # within archive.
>         src = zip.open('unknowndir/src.txt')
>         with open('dst.txt', mode='wb') as dst:
>             shutil.copyfileobj(src, dst)
>
> The last line throws an Error:
>
> Traceback (most recent call last):
>   File "test.py", line 25, in 
>     shutil.copyfileobj(src, dst)
>   File "C:\Python26\lib\shutil.py", line 27, in copyfileobj
>     buf = fsrc.read(length)
>   File "C:\Python26\lib\zipfile.py", line 594, in read
>     bytes = self.fileobj.read(bytesToRead)
> TypeError: integer argument expected, got 'long'

It should hopefully work if you use cStringIO/StringIO instead of
BytesIO.

I think the issue is essentially that StringIO.read() will accept a
long object while the backport of bytesio to to 2.6 does an explicit
check for int:

py> StringIO.StringIO("foo").read(long(1))
'f'

py> io.BytesIO("foo").read(long(1))
Traceback (most recent call last):
  File "", line 1, in 
TypeError: integer argument expected, got 'long'

Should this be amended? Perhaps someone on core can consider it.

As for why the bytesToRead calculation in ZipExtFile.read() results in
a long, I've not yet looked at it closely.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Building Python on Solaris 10?

[ryles]

On Oct 28, 3:46 pm, Judy Booth  wrote:
> Can anyone point me towards some instructions for building Python on
> Solaris 10?
> We need this for some of our test scripts and so far we cannot get this
> to build.
>
> We have tried both Python 2.6.4 and 3.1.1 and both fail with messages
> like this:
> Include/pyport.h:685:2: #error "LONG_BIT definition appears wrong for
> platform (bad gcc/glibc config?)."
> *** Error code 1
> make: Fatal error: Command failed for target `Modules/python.o'
>
> The configure command used is:
> ./configure --with-universal-archs=64-bit --enable-universalsdk
> LDFLAGS="-s
> -L/usr/local/lib -L/opt/ssl/lib -lgcc" --prefix=/opt/python
> --prefix=${PREFIX}
>
> and the compiler version is gcc 3.4.3.
>
> A search on Google showed this up as a problem several years ago and
> suggested that there might be a problem with the way configure is working
> on Solaris.
>
> If anyone can help with this that would be great.
>
> Thanks,
>
> Judy

We never heard back from the OP of this thread, but you might try
building with cc instead of gcc:

http://groups.google.com/group/comp.lang.python/browse_thread/thread/bf109ba2c86c3715
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Working threads progress

[ryles]

On Oct 28, 7:02 pm, mattia  wrote:
> Now, I would like to know the activity done (e.g. every two seconds) so I
> create another thread that checks the queue size (using .qsize()). Have
> you any suggestion to improve the code?

It's not uncommon to pass each thread a second queue for output, which
in your case might be tuples of (item, result). You can read from this
queue in a single thread, and since the results now accumulate
incrementally, you can easily collect more detailed status/progress
information (if that's what you're looking for).
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Copying a ZipExtFile

[ryles]

On Oct 28, 8:33 pm, ryles  wrote:
> As for why the bytesToRead calculation in ZipExtFile.read() results in
> a long, I've not yet looked at it closely.

Simple, actually:

In ZipExtFile.__init__():

self.bytes_read = 0L

In ZipExitFile.read():

bytesToRead = self.compress_size - self.bytes_read

13 - 0L == 13L
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Unhandled exception in thread

[ryles]

On Jun 11, 3:52 pm, Mads Michelsen  wrote:
> Here's the deal. The script in question is a screen scraping thing
> which downloads and parses the html in the background using a separate
> thread (the 'thread' module), while the main program serves up the
> data to the user, allowing some modicum of interaction. Sometimes, not
> always (though I cannot see a pattern), when I quit the script, the
> following error message is printed:
>
>     Unhandled exception in thread started by
>     Error in sys.excepthook:
>
>     Original exception was:

Are you using daemon threads? There are some issues with CPython when
exiting with daemon threads:

http://bugs.python.org/issue1722344

http://bugs.python.org/issue1856

http://bugs.python.org/issue4106

It's possible you are encountering this kind of problem. I have had to
deal with issue4106, the workaround being to explicitly join the
thread of multiprocessing.Queue before exiting.

I would follow Scott's advice and explicitly request and wait for your
threads to exit before terminating the process.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: generator expression works in shell, NameError in script

[ryles]

On Jun 18, 9:56 am, nn  wrote:
> On Jun 18, 8:38 am, guthrie  wrote:
>
>
>
> > On Jun 17, 6:38 pm, Steven Samuel Cole 
> > wrote:
>
> > > Still don't really understand why my initial code didn't work, though...
>
> > Your code certainly looks reasonable, and looks to me like it "should"
> > work. The comment of partial namespace is interesting, but
> > unconvincing (to me) - but I am not a Python expert! It would
> > certainly seem that within that code block it is in the local
> > namespace, not removed from that scope to be in another.
>
> > Seems that it should either be a bug, or else is a design error in the
> > language!
>
> > Just as in the above noted "WTF" - non-intuitive language constructs
> > that surprise users are poor design.
>
> This is certainly an odd one. This code works fine under 2.6 but fails
> in Python 3.1.
>
> >>> class x:
>
> ...     lst=[2]
> ...     gen=[lst.index(e) for e in lst]
> ...
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "", line 3, in x
>   File "", line 3, in 
> NameError: global name 'lst' is not defined
>
>
>
>

I believe it works in 2.x because unlike generator expressions, list
comprehensions do not create a new scope. However, in 3.0 list
comprehensions are actually treated as list().

http://docs.python.org/reference/expressions.html
http://www.python.org/dev/peps/pep-0289
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: generator expression works in shell, NameError in script

[ryles]

> Does the generator expression have its own little namespace or so ?

Yes, generators create a new scope:

http://docs.python.org/reference/expressions.html#grammar-token-generator_expression

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Making code run in both source tree and installation path

[ryles]

On Jun 29, 12:20 pm, Javier Collado  wrote:
> Hello,
>
> I would like to be able to run the main script in a python project
> from both the source tree and the path in which it's installed on
> Ubuntu. The script, among other things, imports a package which in
> turns makes use of some data files that contains some metadata that is
> needed in xml format.
>
> The source tree has an structure such as this one:
> setup.py
> debian/ (packaging files)
> src/ (source code)
> src/lib (package files)
> src/data (data files)
> src/bin (main script)
>
> However, when the project is installed using setup.py install, the
> directory structure is approximately this way:
> /usr/local/bin (main script)
> /usr/local/share/ (data files)
> /usr/local/lib/python2.x/dist-packages/ (library files)
>
> And when installing the code through a package, the structure is the
> same one, but removing "local".
>
> Hence, the data files aren't always in the same relative directories
> depending on we're executing code from the source tree or from the
> installation. To make it possible to run the code from both places,
> I've seen different approaches:
> - distutils trick in setup.py to modify the installed script (i.e.
> changing a global variable value) so that it has a reference to the
> data files location.
> - Heuristic in the package code to detect when it's being executed
> from the source tree and when it has been the installed
> - Just using an environment variable that the user must set according
> to his needs
>
> I guess that there are other options, for example, maybe using
> buildout. What would you say it's the best/more elegant option to
> solve this problem?
>
> Best regards,
>    Javier

It's kludgey, but one option may be to try and use __file__ to figure
out where the script is installed. Something like os.path.dirname
(os.path.abspath(__file__)) could tell you if it's in src/ or in the
bin/ directory, and then data files could be found in the appropriate
place.

I like the distutils/variable option better. Your script is more
likely to still behave correctly when copied to another directory.
Plus its code definitely remains cleaner.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Direct interaction with subprocess - the curse of blocking I/O

[ryles]

On Jun 29, 5:43 pm, Scott David Daniels  wrote:
> and I personally wouldn't have it any other way.  Simulating a shell
> with hooks on its I/O should be so complicated that a "script kiddie"
> has trouble writing a Trojan.

+1 QOTW
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Searching equivalent to C++ RAII or deterministic destructors

[ryles]

> You can go ahead and implement a __del__() method. It will often work in
> CPython, but you get no guarantees, especially when you have reference
> cycles and with other Python implementations that don't use refcounting.

And for resources whose lifetime is greater than your process (e.g. a
file), you can also use the atexit module to help ensure they are
cleaned/reclaimed at shutdown. One way to do this is to maintain a
weak dictionary (from the weakref module) to your objects and install
a handler which iterates this. This is useful for not only dealing
with reference cycles, but for objects which may exist at the time the
interpreter exits. These may not be deleted.

http://docs.python.org/reference/datamodel.html#object.__del__
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Multi thread reading a file

[ryles]

On Jul 2, 6:10 am, "Gabriel Genellina"  wrote:
> En Wed, 01 Jul 2009 12:49:31 -0300, Scott David Daniels  
>  escribió:
> > These loops work well with the two-argument version of iter,
> > which is easy to forget, but quite useful to have in your bag
> > of tricks:
>
> >      def convert(in_queue, out_queue):
> >          for row in iter(in_queue.get, None):
> >              # ... convert row
> >              out_queue.put(converted_line)
>
> Yep, I always forget about that variant of iter() -- very handy!

Yes, at first glance using iter() here seems quite elegant and clever.
You might even pat yourself on the back, or treat yourself to an ice
cream cone, as I once did. There is one subtle distinction, however.
Please allow me to demonstrate.

>>> import Queue
>>>
>>> queue = Queue.Queue()
>>>
>>> queue.put(1)
>>> queue.put("la la la")
>>> queue.put(None)
>>>
>>> list(iter(queue.get, None))
[1, 'la la la']
>>>
>>> # Cool, it really works! I'm going to change all my old code to use this... 
>>> new and *improved*
...
>>> # And then one day your user inevitably does something like this.
...
>>> class A(object):
... def __init__(self, value):
... self.value = value
...
... def __eq__(self, other):
... return self.value == other.value
...
>>> queue.put(A(1))
>>> queue.put(None)
>>>
>>> # And then this happens inside your 'generic' code (which probably even 
>>> passed your unit tests).
...
>>> list(iter(queue.get, None))
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 5, in __eq__
AttributeError: 'NoneType' object has no attribute 'value'
>>>
>>> # Oh... yeah. I really *did* want 'is None' and not '== None' which iter() 
>>> will do. Sorry guys!

Please don't let this happen to you too ;)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What are the limitations of cStringIO.StringIO() ?

[ryles]

This reminds me. Would anyone object to adding a sentence to
http://docs.python.org/library/stringio.html#module-cStringIO just to
mention that attributes cannot be added to a cStringIO, like such:

% import cStringIO, StringIO
% StringIO.StringIO().value = 1
% cStringIO.StringIO().value = 1
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'cStringIO.StringO' object has no attribute 'value'
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Adding an object to the global namespace through " f_globals" is that allowed ?

[ryles]

On Jul 2, 1:25 am, Terry Reedy  wrote:
> > The next statement works,
> > but I'm not sure if it will have any dramatical side effects,
> > other than overruling a possible object with the name A
>
> > def some_function ( ...) :
> >  A = object ( ...)
> >  sys._getframe(1).f_globals [ Name ] = A
>
> global name
> name = A
>
> or is name is a string var
> globals()[name] = A

It wasn't explicit, but I think Stef meant that the global should be
added to the caller's environment, which was the reason for
sys._getframe().

Is this environment only intended for interactive use? I wonder if you
might just set things up
in a PYTHONSTARTUP script instead.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Multi thread reading a file

[ryles]

On Jul 2, 10:20 pm, Paul Rubin <http://phr...@nospam.invalid> wrote:
> ryles  writes:
> > >>> # Oh... yeah. I really *did* want 'is None' and not '== None'
> > >>> which iter() will do. Sorry guys!
>
> > Please don't let this happen to you too ;)
>
> None is a perfectly good value to put onto a queue.  I prefer
> using a unique sentinel to mark the end of the stream:
>
>    sentinel = object()

I agree, this is cleaner than None. We're still in the same boat,
though, regarding iter(). Either it's 'item == None' or 'item == object
()', and depending on the type, __eq__ can introduce some avoidable
risk.

FWIW, even object() has its disadvantages. Namely, it doesn't work for
multiprocessing.Queue which pickles and unpickles, thus giving you a
new object. One way to deal with this is to define a "Stopper" class
and type check objects taken from the queue. This is not news to
anyone who's worked with multiprocessing.Queue, though.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing: pool with blocking queue

[ryles]

On Jul 2, 11:09 am, masher  wrote:
> My questions, then, is: Is there a more elegant/pythonic way of doing
> what I am trying to do with the current Pool class?

Another thing you might try is to subclass Pool and add an apply_async
() wrapper which would wait for _taskqueue.qsize() to reach the
desired size. You would probably do this wait in a loop with a small
sleep. This approach would avoid needing a second Queue, but you would
also add some delay to your producer due to the sleep (something
you're not currently worried about). The minimum sleep may be
something like 1 ms (it's really system dependent), but the time it
takes for a thread blocked on a mutex to wake up is often more on the
order of microseconds, which you have with your blocking queue.

I doubt this offers you much satisfaction, though.

> If the verdict is no, I'll be happy to file a bug report.

Yeah, I think it's a worth a try.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Multi thread reading a file

[ryles]

On Jul 2, 11:55 pm, Paul Rubin  wrote:
> Yeah, it should allow supplying a predicate instead of using == on
> a value.  How about (untested):
>
>    from itertools import *
>    ...
>    for row in takewhile(lambda x: x is sentinel,
>                          starmap(in_queue.get, repeat((:
>       ...

Yeah, it's a small recipe I'm sure a lot of others have written as
well. My old version:

def iterwhile(callable_, predicate):
""" Like iter() but with a predicate instead of a sentinel. """
return itertools.takewhile(predicate, repeatfunc(callable_))

where repeatfunc is as defined here:

http://docs.python.org/library/itertools.html#recipes

I wish all of these little recipes made their way into itertools or a
like module; itertools seems a bit tightly guarded.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how to set timeout while colling a soap method?

[ryles]

On Jul 13, 9:07 am, dzizes  wrote:
> Hello!
>
> I wrote some some python code for executing a soap method:
>
> import SOAPpy
> from SOAPpy import WSDL
>
> _server = WSDL.Proxy(some wsdl)
> r=_server.generuj(some parameters...)
> print r.encode('cp1250')
>
> It works fine. However, the execution time of this soap method might be
> long. Therefore, I would like to set a timeout like 1 minute, after which I
> would post a timeoute message.
>
> Any ideas?
> Greats,
>
> --
> View this message in 
> context:http://www.nabble.com/how-to-set-timeout-while-colling-a-soap-method-...
> Sent from the Python - python-list mailing list archive at Nabble.com.

I don't believe SOAPpy supports this directly. You can set the timeout
globally before you make your SOAP call:

import socket
socket.setdefaulttimeout(60)

http://docs.python.org/library/socket.html#socket.setdefaulttimeout
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Equivalent for dd & fold

[ryles]

On Jul 15, 1:14 pm, seldan24  wrote:
> On Jul 15, 12:47 pm, Michiel Overtoom  wrote:
>
>
>
> > seldan24 wrote:
> > > what can I use as the equivalent for the Unix 'fold' command?
>
> > def fold(s,len):
> >      while s:
> >          print s[:len]
> >          s=s[len:]
>
> > s="A very long string indeed. Really that long? Indeed."
> > fold(s,10)
>
> > Output:
>
> > A very lon
> > g string i
> > ndeed. Rea
> > lly that l
> > ong? Indee
> > d.
>
> > Greetings,
>
> > --
> > "The ability of the OSS process to collect and harness
> > the collective IQ of thousands of individuals across
> > the Internet is simply amazing." - Vinod 
> > Valloppillilhttp://www.catb.org/~esr/halloween/halloween4.html
>
> Wow, I feel like a dork.  I should have done more research prior to
> posting.  Anyway, thanks for the advice.  The trouble with Python is
> that things make 'too much' sense.  Loving this language.

You might also find the textwrap module useful:

http://docs.python.org/library/textwrap.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: difference in printing to screen Mac / Windows

[ryles]

On Jul 18, 7:03 am, Tim Chase  wrote:
> Lastly, you can force all standard-output in your program to be
> unbuffered without the "-u" parameter:

And if you're using -u a lot, the PYTHONUNBUFFERED environment
variable can also be set (but not empty), so that python adds the
option automatically.
-- 
http://mail.python.org/mailman/listinfo/python-list

Why doesn't `from pkg import mod' work after `del pkg.mod'?

[ryles]

According to http://www.python.org/doc/essays/packages.html:

"The import statement first tests whether the item is defined in the
package; if not, it assumes it is a module and attempts to load it."

However, I've noticed that once a module is imported using the
`from pkg import mod' syntax, if its name is deleted from the
package namespace then subsequent imports with `from' will fail.

Here is an example of this type of scenario:

$ ls -l pkg
total 8.0K
-rw-rw-r-- 1 ? general   0 Jul 24 20:21 _impl.py
-rw-rw-r-- 1 ? general 147 Jul 24 20:28 __init__.py
-rw-rw-r-- 1 ? general   0 Jul 24 20:33 public2.py
-rw-rw-r-- 1 ? general 208 Jul 24 20:32 public.py

$ cat pkg/__init__.py
from pkg import _impl

# Add functions which refer to objects in _impl
# ...

# Don't "currently" require this in the namespace anymore.
del _impl

$ cat pkg/public.py
# Implement something with the aid of _impl.

from pkg import public2  # OK, fine.
from pkg import _impl# The module exists, but this will fail.
# Could do import pkg._impl (or a relative import)

$ python
Python 2.6.2 (r262:71600, Jul 16 2009, 14:04:28)
[GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkg import public
Traceback (most recent call last):
  File "", line 1, in 
  File "pkg/public.py", line 4, in 
from pkg import _impl# The module exists, but this will fail.
ImportError: cannot import name _impl
>>> import sys
>>> sys.modules["pkg._impl"]

>>> from pkg import _impl
Traceback (most recent call last):
  File "", line 1, in 
ImportError: cannot import name _impl
>>> import pkg._impl  # Giving up!
>>>

I had previously noted that once a module is imported from a package
it is automatically added to the package namespace:

>>> import pkg
>>> dir(pkg)
['__builtins__', '__doc__', '__file__', '__name__', '__package__',
'__path__']
>>> from pkg import public2
>>> dir(pkg)
['__builtins__', '__doc__', '__file__', '__name__', '__package__',
'__path__', 'public2']
>>>

However, I didn't expect python to give up on importing those names
again
once removed.

Can anyone provide more details on what's occurring here?

Is this behavior intentional (a feature), or, is it in your opinion, a
defect?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Why doesn't `from pkg import mod' work after `del pkg.mod'?

[ryles]

On Jul 25, 8:57 am, Piet van Oostrum  wrote:
> >>>>> ryles  (r) wrote:
> >r> According tohttp://www.python.org/doc/essays/packages.html:
> >r> "The import statement first tests whether the item is defined in the
> >r> package; if not, it assumes it is a module and attempts to load it."
> >r> However, I've noticed that once a module is imported using the
> >r> `from pkg import mod' syntax, if its name is deleted from the
> >r> package namespace then subsequent imports with `from' will fail.
>
> This is incorrectly stated. Also on the initial import it will fail, not
> just on subsequent imports.
>
> piet$ python
> Python 2.6.2 (r262:71600, Apr 16 2009, 09:17:39)
> [GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.>>> 
> from pkg import _impl
>
> _impl # this is from a print statement in _impl to show that it did import
> Traceback (most recent call last):
>   File "", line 1, in 
> ImportError: cannot import name _impl
>

Well, since __init__.py is executed first, I would regard it as
causing the initial (successful) import, and yours being the
subsequent one, so that the "initial" import does not fail, the
subsequent one does. Wording aside, though, I think we are on the same
page here.

>
> According to the documentation
> <http://docs.python.org/library/functions.html#__import__> the statement
> from pkg import _impl
>
> _temp = __import__('pkg', globals(), locals(), ['_impl'], -1)
> _impl = _temp._impl
>
> This fails because _temp (the imported module) doesn't have a binding
> for _impl because you deleted it.

But supposing we hadn't deleted it, and __init__.py didn't import
_impl, there would still be no binding. Yet the importer would still
import and make the module available (presumably it "knows" to do this
since it was not already in sys.modules). The question really is that
since in our example pkg._impl is still in sys.modules, why won't the
importer add it to the pkg namespace again as it previous had? I would
imagine this is either by design, or is a case that was never
considered.

> for _impl because you deleted it. By the way, if you have a `normal'
> package that doesn't do anything with the name _impl, after the import
> the pkg module namespace will have got a binding for _impl although it is not
> in your code. It will have been put there by the import code.

Yes, this was noted further down in the original post with an example.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Wrapping prstat on Solaris

[ryles]

On Jul 27, 10:26 am, s...@pobox.com wrote:
> At work we currently use top to monitor ongoing system utilization on our
> Solaris systems.  As time has moved on though, use of top has become
> problematic.  Our admins want us to switch to prstat, Sun's top-like
> command.  It works fine however doesn't emit a timestamp at each display
> interval, so it's essentially impossible looking at a prstat output
> file to determine when the particular "screen" was emitted.
>
> If figured, "No problem.  I'll just write a little wrapper."  Alas, that is
> proving harder than I thought.  I can run prstat directing output to a file
> and it seems to blast out a block of lines every N seconds, but when run
> from inside a Python script I can't seem to make the damn thing not
> massively buffer its output.  Accordingly, my script doesn't really see the
> output as its emitted, so the timestamp line it prepends to a block of
> output is off.
>
> I'm currently using subprocess.Popen like so:
>
>     os.environ["TERM"] = "dumb"
>     cmd = "prstat -c %s" % " ".join(sys.argv[1:])
>     pipe = Popen(cmd, shell=True, bufsize=1, stdout=PIPE).stdout
>
> I've tried these other variants as well:
>
>   * prefacing the prstat command with unbuffer (the tcl/expect thingamabob)
>
>   * explicitly redirect the prstat output to /dev/stdout
>
>   * setting bufsize to 0
>
>   * used os.popen instead of Subprocess.Popen
>
> Nothing seems to help.  I always seem to see about 30 seconds of data at
> once (I'm currently using a 5-second interval and 10 lines of output).  I
> would have expected that prstat would simply flush stdout after each block
> of output.
>
> Any ideas about how to get prstat to cooperate better?
>
> Thanks,
>
> --
> Skip Montanaro - s...@pobox.com -http://www.smontanaro.net/
>     That's more than a dress. That's an Audrey Hepburn movie. -- Jerry Maguire

Hey Skip,

You haven't told us how you are actually reading from prstat's output
pipe, which may be the cause. For instance, if you are doing

for line in pipe:
  print line
  ...

then this could cause your buffering issue.

Instead try using readline():

while True:
  line = pipe.readline()
  ...
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The longest word

[ryles]

On Jul 28, 7:55 am, NiklasRTZ  wrote:

> Sincere thanks for strengthening python's superior flexibility. Same
> function also works around an exploding index problem returning
> results for longest word where otherwise a word with whitespace
> crashes the index:

Babelfish?

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is this a bug in multiprocessing or in my script?

[ryles]

On Aug 4, 10:37 pm, erikcw  wrote:
> Traceback (most recent call last):
>   File "scraper.py", line 144, in 
>     print pool.map(scrape, range(10))
>   File "/usr/lib/python2.6/multiprocessing/pool.py", line 148, in map
>     return self.map_async(func, iterable, chunksize).get()
>   File "/usr/lib/python2.6/multiprocessing/pool.py", line 422, in get
>     raise self._value
> TypeError: expected string or buffer

This is almost certainly due to your scrape call raising an exception.
In the parent process, multiprocessing will detect if one of its
workers have terminated with an exception and then re-raise it.
However, only the exception and not the original traceback is made
available, which is making debugging more difficult for you. Here's a
simple example which demonstrates this behavior:

*** from multiprocessing import Pool
*** def evil_on_8(x):
...   if x == 8: raise ValueError("I DONT LIKE THE NUMBER 8")
...   return x + 1
...
*** pool = Pool(processes=4)
>>> pool.map(evil_on_8, range(5))
[1, 2, 3, 4, 5]
*** pool.map(evil_on_8, range(10)) # 8 will cause evilness.
Traceback (most recent call last):
  File "", line 1, in 
  File "/bb/real/3ps/lib/python2.6/multiprocessing/pool.py", line 148,
in map
return self.map_async(func, iterable, chunksize).get()
  File "/bb/real/3ps/lib/python2.6/multiprocessing/pool.py", line 422,
in get
raise self._value
ValueError: I DONT LIKE THE NUMBER 8
***

My recommendation is that you wrap your scrape code inside a try/
except and log any exception. I usually do this with logging.exception
(), or if logging is not in use, the traceback module. After that you
can simply re-raise it.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Web page data and urllib2.urlopen

[ryles]

On Aug 5, 4:30 pm, Massi  wrote:
> Hi everyone, I'm using the urllib2 library to get the html source code
> of web pages. In general it works great, but I'm having to do with a
> financial web site which does not provide the souce code I expect. As
> a matter of fact if you try:
>
> import urllib2
> res = urllib2.urlopen("http://www.marketwatch.com/story/mondays-
> biggest-gaining-and-declining-stocks-2009-07-27")
> page = res.read()
> print page
>
> you will see that the printed code is very different from the one
> given, for example, by mozilla. Since I have really little knowledge
> in html I can't even understand if this is a python or html problem.
> Can anyone give me some help?
> Thanks in advance.

Check if setting your user agent to Mozilla results in a different
page:

http://diveintopython.org/http_web_services/user_agent.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: unique-ifying a list

[ryles]

On Aug 7, 4:53 pm, kj  wrote:
> Suppose that x is some list.  To produce a version of the list with
> duplicate elements removed one could, I suppose, do this:
>
>     x = list(set(x))
>
> but I expect that this will not preserve the original order of
> elements.
>

OrderedSet is most likely on the way, but for now Python 3.1 and 2.7
have OrderedDict. For 3.0 and 2.6 the recipe is here:

http://code.activestate.com/recipes/576669

With OrderedDict you can do:

OrderedDict.fromkeys(x).keys()  # Returns an iterator in 3.x, a list
in 2.x.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to do relpath implementation on 2.5

[ryles]

On Aug 8, 9:08 pm, Brian Allen Vanderburg II
 wrote:
> I've coded my own 'relpath' implementation for 2.5 (shown below) and I
> want to make sure it follows as closely as it should to 2.6 and later.  
> I've got a question regarding that.  When attempting to convert to a
> relative path and it is not possible for some reason (different drive or
> UNC share), should that be an error or should it return the absolute
> path of the target?  I'm using Debian so I don't have 2.6 available
> right now for testing.

Usually Python source code is easily browsed at:

http://svn.python.org/view/

Unfortunately the server is down right now.

The implementation in 2.6 raises a ValueError exception. Here is the
function from ntpath.py:

def relpath(path, start=curdir):
"""Return a relative version of a path"""

if not path:
raise ValueError("no path specified")
start_list = abspath(start).split(sep)
path_list = abspath(path).split(sep)
if start_list[0].lower() != path_list[0].lower():
unc_path, rest = splitunc(path)
unc_start, rest = splitunc(start)
if bool(unc_path) ^ bool(unc_start):
raise ValueError("Cannot mix UNC and non-UNC paths (%s and
%s)"
%
(path, start))
else:
raise ValueError("path is on drive %s, start on drive %s"
% (path_list[0],
start_list[0]))
# Work out how much of the filepath is shared by start and path.
for i in range(min(len(start_list), len(path_list))):
if start_list[i].lower() != path_list[i].lower():
break
else:
i += 1

rel_list = [pardir] * (len(start_list)-i) + path_list[i:]
if not rel_list:
return curdir
return join(*rel_list)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: run all scripts in sub-directory as subroutines?

[ryles]

On Aug 11, 5:12 pm, guthrie  wrote:
> This works fine, but in the sub-modules the sys.path appropriately
> returns the same as from the parent, I want them to know their own
> file names. How?? I can pass it to them, but wondered if there is a
> more self-sufficient way for a module to know from where it was
> invoked.

Instead of sys.path, you may be interested in the __file__ attribute
will give you a module's filename:

http://docs.python.org/reference/simple_stmts.html#index-1031

You could also be more explicit in matching *.py files in your
subdirectory by using glob:

http://docs.python.org/library/glob.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python "and" behavior

[ryles]

On Aug 13, 8:36 pm, goldtech  wrote:
> Could you explain or link me to an explanation of this?

http://docs.python.org/tutorial/datastructures.html#more-on-conditions

Give the whole tutorial a good read.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Splitting a string into substrings of equal size

[ryles]

On Aug 14, 8:22 pm, candide  wrote:
> Suppose you need to split a string into substrings of a given size (except
> possibly the last substring). I make the hypothesis the first slice is at the
> end of the string.
> A typical example is provided by formatting a decimal string with thousands
> separator.
>
> What is the pythonic way to do this ?
>
> For my part, i reach to this rather complicated code:
>
> # --
>
> def comaSep(z,k=3, sep=','):
>     z=z[::-1]
>     x=[z[k*i:k*(i+1)][::-1] for i in range(1+(len(z)-1)/k)][::-1]
>     return sep.join(x)
>
> # Test
> for z in ["75096042068045", "509", "12024", "7", "2009"]:
>     print z+" --> ", comaSep(z)
>
> # --
>
> outputting :
>
> 75096042068045 -->  75,096,042,068,045
> 509 -->  509
> 12024 -->  12,024
> 7 -->  7
> 2009 -->  2,009
>
> Thanks

py> s='1234567'
py> ','.join(_[::-1] for _ in re.findall('.{1,3}',s[::-1])[::-1])
'1,234,567'
py> # j/k ;)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Splitting a string into substrings of equal size

[ryles]

On Aug 15, 6:28 pm, MRAB  wrote:

> >      >>> for z in ["75096042068045", "509", "12024", "7", "2009"]:
> >            print re.sub(r"(?<=.)(?=(?:...)+$)", ",", z)
>
> >     75,096,042,068,045
> >     509
> >     12,024
> >     7
> >     2,009
>
> The call replaces a zero-width match with a comma, ie inserts a comma,
> if certain conditions are met:
>
> "(?<=.)"
>      Look behind for 1 character. There must be at least one previous
> character. This ensures that a comma is never inserted at the start of
> the string. I could also have used "(? whether the first character is a "-". That's left as an exercise for the
> reader. :-)
>
> "(?=(?:...)+$)"
>      Look ahead for a multiple of 3 characters, followed by the end of
> the string.

Wow, well done. An exceptional recipe from Python's unofficial regex
guru. And thanks for sharing the explanation.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 2.6 still not giving memory back to the OS...

[ryles]

On Aug 15, 7:55 am, Chris Withers  wrote:
> Hi All,
>
> I thought this was fixed back in Python 2.5, but I guess not?
>
> So, I'm playing in an interactive session:
>
>  >>> from xlrd import open_workbook
>  >>> b = open_workbook('some.xls',pickleable=0,formatting_info=1)
>
> At this point, top shows the process usage for python to be about 500Mb.
> That's okay, I'd expect that, b is big ;-)
>
>  >>> del b
>
> However, it still does now, maybe the garbage collector needs a kick?
>
>  >>> import gc
>  >>> gc.collect()
> 702614
>
> Nope, still 500Mb. What gives? How can I make Python give the memory its
> no longer using back to the OS?
>
> Okay, so maybe this is something to do with it being an interactive
> session? So I wrote this script:
>
> from xlrd import open_workbook
> import gc
> b = open_workbook('some.xls',pickleable=0,formatting_info=1)
> print 'opened'
> raw_input()
> del b
> print 'deleted'
> raw_input()
> gc.collect()
> print 'gc'
> raw_input()
>
> The raw inputs are there so I can check the memory usage in top.
> Even after the gc, Python still hasn't given the memory back to the OS :-(
>
> What am I doing wrong?
>
> Chris
>
> --
> Simplistix - Content Management, Batch Processing & Python Consulting
>             -http://www.simplistix.co.uk

Repeat the 'b = open_workbook ...' line in the interpreter several
times and track the memory usage after each execution. I noticed this
behavior on linux, but saw that after repeating it a few times memory
usage finally stop going up. Probably just the behavior of the
allocator.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Setuptools - help!

[ryles]

On Aug 7, 3:04 pm, Peter Chant  wrote:
> Robert Kern wrote:
> > You need to put main.py into the pphoto package.
>
> > $ mkdir pphoto/
> > $ mv main.py pphoto/
> > $ touch pphoto/__init__.py
>
> Thanks, it worked.  Any ideas how to run the resulting scripts without
> installing or running as root?
>
> Pete
>
> --http://www.petezilla.co.uk

If you are using Python 2.6+ you have your own site-packages
directory, e.g. ~/.local/lib/python2.6/site-packages. So, if this
package is only for your own use, you can use easy_install's --install-
dir option to have setuptools install it there, and can also set --
script-dir to ~/bin, where your console script will go. This, and
other options are discussed here:

http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Protecting against callbacks queuing up?

[ryles]

On Aug 23, 8:14 pm, Esben von Buchwald  wrote:
> I thought that this code would do the trick, but it obviously doesn't
> help at all, and i can't understand why...
>
>      def doCallback(self):
>          if self.process_busy==False:
>              self.process_busy=True
>              self.data_callback()
>              self.process_busy=False
>
> doCallback is defined as a callback function for the accelerometer
> instance, and data_callback does some calculations and show them on the
> display of the phone.
>
> What to do? Thanks...

As Dennis pointed out, it sounds like doCallback() is called serially.
That is, you can think of doCallback() as being called in a loop,
rather than entered multiple times simultaneously. Your 'busy' flag is
never actually checked until after it has already been reset to False.

If the data accessed by the callback as a unique copy and has a
timestamp then a simple approach which avoids multithreading would be
to have your callback ensure that any 'old' data is simply ignored.
This will allow it to quickly catch up to a 'new' event.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Object's nesting scope

[ryles]

On Aug 26, 8:51 am, zaur  wrote:
> Hi folk!
>
> What do you think about idea of "object's nesting scope" in python?
>
> Let's imaging this feature, for example, in this syntax:
>
> obj=:
>      
>
> or
>
> :
>    
>
> That's means that result object of  evaluation is used as
> nested scope for  evaluation.
>
> So is this idea useful?
>
> Zaur

FYI, there was recently a similar idea being discussed on the python-
ideas list:

http://groups.google.com/group/python-ideas/browse_thread/thread/1a13cd9a5189b01c?hl=en
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Q: multiprocessing.Queue size limitations or bug...

[ryles]

On Aug 26, 4:56 am, Michael Riedel 
wrote:
> Sorry for being not more specific but I'm not absolutely certain whether
> I encountered a bug or did anything wrong:
>
> The (stupid) code below results in a stall forever or not at 'p0.join()'
> depending on the value of TROUBLE_MAKER.
>
> Any help, thoughts, comments?
>
> Thank you for your time.
>
> Michael
>
> # --
>
> from multiprocessing import Process, Queue
>
> # bit vector size
> BVS=8
>
> #
> TROUBLE_MAKER=12  # for greater values p0.join() is never satisfied...
>
> def evaluate(q, id, start=0, stop=2**BVS):
>
>     cmin = {0: []}
>
>     for mask0 in range(start, stop):
>         for mask1 in range(0, 2**BVS):
>             for mask2 in range(mask1, TROUBLE_MAKER):
>                 cmin[0].append((mask0, mask1, mask2))
>
>     print 'process %d finished (dict layout: %d/%d)...' % (id,
> len(cmin), len(cmin[0]))
>     q.put(cmin.copy())
>     q.close()
>
> if __name__ == '__main__':
>
>     q0 = Queue()
>     q1 = Queue()
>     q2 = Queue()
>     q3 = Queue()
>
>     part = 2**BVS/4
>     p0 = Process(target=evaluate, args=(q0, 0, 0*part, 1*part),
> name='worker_0')
>     p1 = Process(target=evaluate, args=(q1, 1, 1*part, 2*part),
> name='worker_1')
>     p2 = Process(target=evaluate, args=(q2, 2, 2*part, 3*part),
> name='worker_2')
>     p3 = Process(target=evaluate, args=(q3, 3, 3*part, 4*part),
> name='worker_3')
>     p0.start()
>     print 'process 0 started...'
>     p1.start()
>     print 'process 1 started...'
>     p2.start()
>     print 'process 2 started...'
>     p3.start()
>     print 'process 3 started...'
>     # main process stalls at p0.join() for bigger TROUBLE_MAKER
>     p0.join()
>     p1.join()
>     p2.join()
>     p3.join()
>     res0 = q0.get()
>     res1 = q1.get()
>     res2 = q2.get()
>     res3 = q3.get()
>     print 'results fetched...'
>
> # --
>
> --

There is a warning related to this in the documentation:

http://docs.python.org/library/multiprocessing.html#pipes-and-queues

Basically, you should reverse the order of the get() and join() calls.

multiprocessing does a pretty nice job of abstracting away the low-
level details of IPC, but there are still some gotchas. As you've
noticed, your program will deadlock when there is a large enough
amount of data being put into the queue. This is related to a hidden
thread that exists inside each of your child processes. The thread is
responsible for taking your queue items from an internal buffer and
then writing them into a pipe that your parent process will read from
when get() is called. The pipe mechanism is what allows the two
processes to pass information, and is supported directly by the
Operating System. However, the pipe has a limited capacity, and when
it is full, the writer thread is stuck waiting for the reader to read
enough from the pipe so that it can finish its write. The problem is
that your parent process (reader) is not actually calling get() to
drain the pipe. Instead it's stuck in join() waiting for the writer to
complete.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: executable path finding

[ryles]

On Aug 31, 12:37 pm, koranthala  wrote:
> On Aug 31, 9:07 pm, "Diez B. Roggisch"  wrote:
>
>
>
> > koranthala wrote:
> > > Hi,
> > >     I am creating a python application using py2exe. I am facing a
> > > problem which I am not sure how to solve.
> > >     The application contains many other files associated with it -
> > > like icons, config files etc. The executable can be in any directory.
> > > If the user creates a shortcut to the executable to run in desktop,
> > > the program fails - saying that the icon image cannot be found. But I
> > > can run the application properly, if I run it from application
> > > directory - where all the other icons, config files etc are kept.
> > >     How is such issues usually solved? I do not want to hardcode the
> > > paths of icons and store the icons in those paths. I guess, we can
> > > change the directory to the application directory before running the
> > > application. (in __init__.py ???) But how do we find the current
> > > directory in that case? I am completely at sea in solving this.
> > >     This looks to be a very common issue. How is this usually solved?
>
> > You can get the location of a module via
>
> >  module.__file__
>
> > This can be used to find a file relative to e.g. the toplevel module/package
> > of your application.
>
> > The pkg_resources-module of setuptools encapsulates that even into a stream
> > and file-name api.
>
> > Diez
>
> Thank you Diez. It was what I was looking for.

Also see:

http://docs.python.org/library/pkgutil.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Multiple inheritance - How to call method_x in InheritedBaseB from method_x in InheritedBaseA?

[ryles]

On Sep 4, 6:01 am, The Music Guy  wrote:
> I have a peculiar problem that involves multiple inheritance and method 
> calling.
>
> I have a bunch of classes, one of which is called MyMixin and doesn't
> inherit from anything. MyMixin expects that it will be inherited along
> with one of several other classes that each define certain
> functionality. It defines method_x, which it assumes will also be
> defined in the other class that MyMixin ends up getting inherited
> with. For example,
>
> class MyMixin(object):
>     def method_x(self, a, b, c):
>         ...
>
> class BaseA(object):
>     def method_x(self, a, b, c):
>         ...
>
> class BaseB(object):
>     def method_x(self, a, b, c):
>         ...
>
> class BaseC(object):
>     def method_x(self, a, b, c):
>         ...
>
> class FooX(MyMixin, BaseA):
>     ...
>
> class FooY(MyMxin, BaseB):
>     ...
>
> class FooZ(MyMixin, BaseC):
>     ...
>
> This all appears fine at first, but there is a problem: Each Foo's
> method_x must call the method_x of MyMixin as well as the method_x of
> each respective Foo's second base class. One cannot simply call
> FooN.method_x, because that will only call MyMixin.method_x and not
> that of the other base.
>
> One might be tempted to amend MyMixin's method_x so that it calls the
> parent's method_x before doing anything else:
>
> class MyMixin(object):
>     def method_x(self, a, b, c):
>         super(MyMixin, self).method_x(a, b, c)
>         ...
>
> ...but of course, that will fail with an AttributeError because
> MyMixin's only superclass is object, which does not have a method_x.
>
> The only way I can think to solve the problem would be to implement a
> method_x for each Foo that calls the method_x for each of the bases:
>
> class FooX(MyMixin, BaseA):
>     def method_x(self, a, b, c):
>         MyMixin.method_x(self, a, b, c)
>         BaseA.method_x(self, a, b, c)
>
> class FooY(MyMxin, BaseB):
>     def method_x(self, a, b, c):
>         MyMixin.method_x(self, a, b, c)
>         BaseB.method_x(self, a, b, c)
>
> class FooZ(MyMixin, BaseC):
>     def method_x(self, a, b, c):
>         MyMixin.method_x(self, a, b, c)
>         BaseC.method_x(self, a, b, c)
>
> The problem with this solution is that method_x has to be explicitly
> created for each Foo, even though they all do just about the same
> thing, which kind of defeats the purpose of using multiple inheritance
> in this situation. Besides that, I just don't like it!
>
> So, does anyone have an idea about how to remedy this, or at least
> work around it?

Hopefully I've interpreted your requirement correctly.

In this particular case a workaround is to use super() in your base
classes and then have your concrete classes inherit from the base
*before* the mixin:

class MyMixin(object):
def method_x(self, a, b, c):
print "MyMixin"

class BaseA(object):
def method_x(self, a, b, c):
print "BaseA"
super(BaseA, self).method_x(a, b, c)

class FooX(BaseA, MyMixin):
pass

FooX().method_x(1, 2, 3)

BaseA
MyMixin


And if you want to define method_x() in FooX, then simply have that
use super() as well.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is "#!/usr/bin/env python" the better shebang line ?

[ryles]

On Sep 6, 10:01 am, Timothy Madden  wrote:
> Hello
>
> Sorry if this has been discussed before, my search did not find it.
> My questions is if I should use
>    #!/usr/bin/env python
> as the shebang line in a portable and open python script and if it does
> help with portability and usage.

Note that there is one limitation of /usr/bin/env on many systems: you
cannot supply any arguments to the interpreter.

e.g. #!/usr/bin/env python -u

The above will fail on many systems as env will literally try to find
a file named "python -u".

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/237208

Another, more obvious, case where you might avoid /usr/bin/env is when
you know (or might expect in the future) that there is another Python
version on the system which your script will not work with. Then it's
often convenient to hard code the interpreter path. This comes up
mostly in environments where you're not the administrator and have
another Python version installed in a custom place which you (and any
co-developers) have in their PATH. For example, if you wanted to share
your script with a non-team member, then having /usr/bin/env in your
script would force them to have to change their PATH first.

> Then, supposing /usr/bin/env is better than /usr/bin/python and I use
> it, is it not the case that many editors and code analysing programs,
> that want to know the script language for syntax highlighting, code
> completion, tags and source code browsing, etc, will loose their
> functionality because of this shebang line ? Is this not a good general
> reason to use the old one ?

I personally use vim, and it will recognize either shebang, or if none
is present (as in a typical importable module) it will notice the file
suffix, .py. It's not an issue.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: simple string question

[ryles]

> There's probably a more general method covering all the escape
> sequences, but for just \n:
>
> your_string = your_string.replace("\\n", "\n")

py> s = "hello\\r\\n"
py> s
'hello\\r\\n'
py> s.decode("string_escape")
'hello\r\n'
py>

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Multiple inheritance - How to call method_x in InheritedBaseB from method_x in InheritedBaseA?

[ryles]

On Sep 9, 1:47 am, The Music Guy  wrote:
> I should also mention--and I should have realized this much
> sooner--that each of the BaseN classes are themselves each going to
> have at least one common base which will define method_x, so each
> BaseN will be calling that if it defines its own method_x. Again,
> sorry I didn't mention that sooner. For some reason it didn't occur to
> me that it would be important. I feel dumb now... :P
>
> Here is the updated current example:
>
> class CommonBase(object):
>     def method_x(self, a, b c):
>         ... no call to superclass' method_x is needed here ...
>
> class MyMixin(object):
>    def method_x(self, a, b, c):
>        get_other_base(self).method_x(self, a, b, c)
>        ...
>
> class BaseA(CommonBase):
>    def method_x(self, a, b, c):
>        super(BaseA, self).method_x(a, b, c)
>        ...
>
> class BaseB(CommonBaset):
>    ...
>
> class BaseC(CommonBase):
>    def method_x(self, a, b, c):
>        super(BaseC, self).method_x(a, b, c)
>        ...
>
> class FooX(MyMixin, BaseA):
>    ...
>
> class FooY(MyMxin, BaseB):
>    ...
>
> class FooZ(MyMixin, BaseC):
>    ...

OK, how about this?

class CommonBase(object):
def method_x(self, a, b, c):
print "CommonBase"

class MyMixin(object):
   def method_x(self, a, b, c):
   print "MyMixin",
   super(MyMixin, self).method_x(a, b, c)

class BaseA(CommonBase):
   def method_x(self, a, b, c):
   print "BaseA",
   super(BaseA, self).method_x(a, b, c)

class BaseB(CommonBase):
   def method_x(self, a, b, c):
print "BaseB",
super(BaseB, self).method_x(a, b, c)

class BaseC(CommonBase):
   def method_x(self, a, b, c):
print "BaseC",
super(BaseC, self).method_x(a, b, c)

class FooX(MyMixin, BaseA):
def method_x(self, a, b, c):
print "FooX",
super(FooX, self).method_x(a, b, c)

class FooY(MyMixin, BaseB):
def method_x(self, a, b, c):
print "FooY",
super(FooY, self).method_x(a, b, c)

class FooZ(MyMixin, BaseC):
# Supposing this class doesn't require a specialized method_x.
pass


FooX().method_x(1, 2, 3)  # Prints 'FooX MyMixin BaseA CommonBase'
FooY().method_x(1, 2, 3)  # Prints 'FooY MyMixin BaseB CommonBase
FooZ().method_x(1, 2, 3)  # Prints 'MyMixin BaseC CommonBase'

---

Each method_x implementation uses super() to ensure that the next base
class method_x is called. The exception is CommonBase, which is
presumed to be the hierarchy root.

http://www.python.org/download/releases/2.2.3/descrintro/#cooperation
http://www.python.org/download/releases/2.3/mro
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Multiple inheritance - How to call method_x in InheritedBaseB from method_x in InheritedBaseA?

[ryles]

On Sep 9, 7:48 pm, The Music Guy  wrote:
> Btw, Carl, please forgive me if I frustrate you, because I'm trying my
> best not to. I'm trying to keep track of what I did and what you did
> and what Ryles and Scott did, while at the same time trying to keep a
> firm grasp of exactly what it is I'm trying to acheive. Besides that,
> I'm not a professional programmer--just a hobbyist. And of course,
> there's all the other IRL things I'm trying to deal with
> simultaneously with this...hopefully you can see how I would make
> mistakes.

Glad to see your project is afoot. Multiple inheritance is tricky
stuff. Try to keep class hierarchies simple and prefer composition
when feasible.

Have a look at the links I suggested in my previous post, particularly
the new-style classes paper. It's tough reading but if you get through
it you'll gain a lot.

And don't worry about Carl, he's a Calvin Klein model and, as you
know, sometimes they can get a bit moody ;)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Are min() and max() thread-safe?

[ryles]

On Sep 17, 7:02 am, Carl Banks  wrote:
> On Sep 16, 9:33 pm, Steven D'Aprano
>
>  wrote:
> > def minmax(seq):
> >     result = [None, None]
> >     t1 = MMThread(seq, min, result, 0)
> >     t2 = MMThread(seq, max, result, 1)
> >     t1.start()
> >     t2.start()
> >     # Block until all threads are done.
> >     while any([t1.isAlive(), t2.isAlive()]):
> >         time.sleep(0)
>
> Why not use "t1.join(); t2.join()" here?  Is there any benefit to do
> it this way instead?
>
> Carl Banks

The benefit is that SIGINT (Control-C) will be handled. Not sure why
sleep(0) is used, though, rather than something less busy like sleep
(0.5).
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Granularity of OSError

[ryles]

On Sep 19, 9:22 pm, MRAB  wrote:
> The point is that it's sometimes a good idea to do a cheap check first
> before attempting an operation that's 'expensive' even when it fails.

Strongly agree. Furthermore, with LBYL it's often easier to give a
user clearer error messages for common usage errors, rather than
waiting for an exception in a much lower-level place where it's less
clear to them what the cause is.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: on package import, have it conditionally import a subpackage

[ryles]

On Sep 19, 4:06 pm, Gabriel Rossetti 
wrote:
> Hello everyone,
>
> I'd like to ba able to import a package and have it's __init__
> conditionally import a subpackage. Suppose that you have this structure :
>
> mybase/
> mybase/__init__.py
> mybase/mypkg
> mybase/mypkg/__init__.py
> mybase/mypkg/module0.py
> mybase/mypkg/type1
> mybase/mypkg/type1/__init__.py
> mybase/mypkg/type1/module1.py
> mybase/mypkg/type1/module2.py
> mybase/mypkg/type2
> mybase/ mypkg/type2/__init__.py
> mybase/ mypkg/type2/module1.py
> mybase/ mypkg/type2/module2.py
>
> I'd like to do something like th following in my code
>
>  >>> import mybase
>  >>> mybase.setType("type1")
>  >>> from mybase.mypkg import module0, module1, module2
>  >>> module0.__file__
> 'mybase/mypkg/module0.pyc'
>  >>> module1.__file__
> 'mybase/mypkg/type1/module1.pyc'
>  >>> module2.__file__
> 'mybase/mypkg/type1/module2.pyc'
>
> but I can't figure out how to do this.
>
> Thank you,
> Gabriel

You might try something like the following:

$ export PYTHONPATH=/home/ryles


$ cat mybase_test.py

import mybase

mybase.type = "type2"

from mybase.mypkg import module0, module1, module2

print module0.__file__
print module1.__file__
print module2.__file__


$ python mybase_test.py
/home/ryles/mybase/mypkg/module0.pyc
/home/ryles/mybase/mypkg/type2/module1.pyc
/home/ryles/mybase/mypkg/type2/module2.pyc


$ ls -l mybase/
-rw-r--r--  1 ryles None  44 Sep 20 16:59 __init__.py
drwxr-xr-x+ 4 ryles None   0 Sep 20 17:06 mypkg


$ cat mybase/__init__.py

default_type = "type1"

type = default_type


$ ls -l mybase/mypkg/
-rw-r--r--  1 ryles None 307 Sep 20 17:06 __init__.py
-rw-r--r--  1 ryles None   0 Sep 20 16:51 module0.py
drwxr-xr-x+ 2 ryles None   0 Sep 20 17:02 type1
drwxr-xr-x+ 2 ryles None   0 Sep 20 17:02 type2


$ ls -l mybase/mypkg/type1/
-rw-r--r-- 1 ryles None   0 Sep 20 16:49 __init__.py
-rw-r--r-- 1 ryles None   0 Sep 20 16:49 module1.py
-rw-r--r-- 1 ryles None   0 Sep 20 16:51 module2.py


$ ls -l mybase/mypkg/type2/
-rw-r--r-- 1 ryles None   0 Sep 20 16:48 __init__.py
-rw-r--r-- 1 ryles None   0 Sep 20 16:51 module1.py
-rw-r--r-- 1 ryles None   0 Sep 20 16:49 module2.py


$ cat mybase/mypkg/__init__.py

# These modules are hard-coded.
from mybase.mypkg import module0

# These modules are resolved dynamically.
from mybase import type
for module_name in ["module1", "module2"]:
full_name = "mybase.mypkg." + type + '.' + module_name
globals()[module_name] = __import__(full_name, {}, {},
[full_name])

$
-- 
http://mail.python.org/mailman/listinfo/python-list