Re: [Tutor] Shelve del not reducing file size

[Barton David]

I mean no offense and lay no blame. It's simply that I feel like I've
been led up a nice gentle beach and suddenly I'm dodging boulders at the
bottom of a cliff.

I've learned to program with Python (and can hardly conceive of a better
language to be honest)- and I still think the core language is great:
elegant, easy to use and brilliantly documented.

But the more I explore the standard library and third party modules, the
more I run into trouble: a chaotic library structure that seems to
conceal capabilities rather than promote them, similar modules that
don't work in similar ways, a whole new level of opaque programming
lingo that makes me feel excluded, behaviours that I don't understand,
don't want, and that I can't find documentation to explain, and so on.

I guess it's not Python's fault: I'm guess I'm just too stupid. But I'm
just getting really disenchanted. Sorry.
 

-Original Message-
From: Kent Johnson [mailto:[EMAIL PROTECTED] 
Sent: 27 July 2007 13:07
To: Barton David
Cc: tutor@python.org
Subject: Re: [Tutor] Shelve del not reducing file size

Barton David wrote:
> *sigh* I'm really going off Python.

In what way is it Python's fault that the dbm database doesn't reclaim
disk space?

Kent

This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Shelve del not reducing file size

[Barton David]

Sadly I can't think of a plan B, hence the frustration! Python, as far
as I know, is as good as it gets. And I don't have the courage or the
capability to improve it myself.

So all I can really do is clasp my hands together and helplessly plead:
"Won't Somebody, Please, Think of the Children!" 

(Meaning struggling non-pro users like me, of course)

As far as I'm concerned, the core language has matured to be elegant and
terrificly newbie-friendly, but the extended functionality (the standard
library) absolutely has not. It disappoints me that Guido and many other
developers *seem* to be more concerned with strategies for revamping the
former than they are with strategies for improving the latter. If Kay
Schluehr, Paul Rubin and John Nagle are opposing this trend and being
dismissed as 'carpers' then I fear Python has lost sight of the
'friendliness' it once seemed to aspired to. 



-Original Message-
From: Kent Johnson [mailto:[EMAIL PROTECTED] 
Sent: 27 July 2007 14:11
To: Barton David
Cc: tutor@python.org
Subject: Re: [Tutor] Shelve del not reducing file size

If it's any solace, there is a small minority of Python users who agree
with you. There *are* rough edges in the library modules and the library
docs. The great majority of Python users seem to find them good enough
and are pleased and amazed at what you can do with the batteries
included. A minority find the warts, omissions and inconsistencies to be
very frustrating, and not because they (the users) are dumb. IIRC some
prominent carpers on comp.lang.python are Kay Schluehr, Paul Rubin and
John Nagle.

I'm curious, what is plan B? Do you have something better than Python to
try? I guess the above-named people are still with Python because the
benefits outweigh the warts.

Kent


This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Shelve del not reducing file size

[Barton David]

Eric Brunson wrote:

> It seems like new programmers today expect to be spoonfed their 
> information like they were in grammar school.  They don't know what it 
> is to hack a Makefile to get a package to compile or break out an RFC to 
> understand a protocol.  If you don't understand something and the 
> documentation is lacking, then strap on a pair and read the source, 
> write some test cases, dig a little.  In our environment most of the 
> code you'd have to read is just more Python, anyway.
> 
> Just me being a grouchy old programmer.  In my day we had to program in 
> 4 feet of snow, uphill... both ways!


heh. Well give me some credit. I taught myself to program, from scratch, 
without access to (or time for) any courses whatsoever, while doing a PhD in 
genetics. I've been using it for about 5 years now and I know the core language 
and certain standard modules pretty well. I doubt I would have got as far as I 
have if Python wasn't so newbie-friendly. My only complaint is that I'm 
starting to feel like I won't get much further than that without a computer 
science degree.

This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Shelve del not reducing file size

[Barton David]

*sigh* I'm really going off Python.

OK, thanks Andreas.
 

This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Shelve del not reducing file size

[Barton David]

Hi,
 
I've hit a snag with Python's shelve module. By way of example...
 
_
from shelve import DbfilenameShelf as StoreFile
import os
 
sf=StoreFile("mytest.db",writeback=False) # but same problem if
writeback=True
for i in range(1):
sf[str(i)]="TESTOBJECT"
sf.sync()
print len(sf)
sf.close()
predeletesize=os.path.getsize("mytest.db")
print predeletesize
 
sf=StoreFile("mytest.db",writeback=False) # but same problem if
writeback=True
for i in range(5000):
del sf[str(i)]
sf.sync()
print len(sf)
sf.close()
postdeletesize=os.path.getsize("mytest.db")
print postdeletesize
_
 
So why, when I run this, does predeletesize!=postdeletesize?
I gather that with most database types you have to invoke a special
command like VACUUM to perform cleanups after table deletions etc
so, since shelve uses database backends, is this related to that? Is
there a way to tell (e.g.) a DBfilenameShelf to do this? Or am I stuck
with having to delete the entire file and save it again?
 
 
Running PythonWin 2.5.1 on XP.
 
thanks
Dave

This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Shelve del not reducing file size

[Barton David]

Andreas Kostyrka wrote:
> Additionally, the language core is very very thought out, with glacial
> enhancements. "Fixing" the standard library OTOH would involve
renaming
> and removing names, which would make huge collections of programs
break.
> Not a good thing :(

Yes agreed. My comments there were primarily about the future of python
(i.e. Python 3000).



Kent Johnson wrote:
>  Ultimately it seems to come down to
> - Python is open source
> - Python developers are largely unpaid volunteers working on what
interests them
> - If you want something to change, you can
> -- do it yourself
> -- convince someone to volunteer to do it
> -- pay someone to do it
> 
> So far cleaning up the std lib has not attracted any of these three
options.

Yes of course. But not *all* Python developers are unpaid. And even if
they were- this ship isn't rudderless. It could be steered towards
standard library overhaul (in Py3000) if Guido so decreed (and if he
laid out a nice appealing framework, perhaps). But I guess if everybody
thinks it's ok or nobody thinks it can be improved then that isn't going
to happen.



Andreas Kostyrka wrote:
> Stop.

Somehow that one word just sums it all up!

Yeah I'll stop. It's not like I'm out to upset anybody. But please,
somebody up there, Think of the Children, eh?

This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Shelve del not reducing file size

[Barton David]

Eric Brunson wrote:
> You seem like a smart guy that's having a bad day, so I'm cutting you
> slack.
 
Thanks Eric. Yes I did indeed have a bad day (and it got much much worse),
and this is most definitely a case of a bad workman blaming his tools. I
apologise to all concerned for voicing my frustrations: it was clearly 
ill-advised.
Still.. call me idealistic but I feel like a good toolmaker should try to 
listen to her
clients.
 
I am not a dedicated programmer. I have other stuff on my plate. I probably
wouldn't be a programmer at all if Python wasn't (in the early stages) so
fabulously friendly.
 
Alan Gauld wrote:
> But Pythons library is not newbie friendly, sorry. How does 
> a newbie know when to use pickle v cpickle? or urllib v urllib2? And 
> which of the xml parsers? And as for thev mess that is glob/os/path/shutil?
> Its not clear to me even after 10 years of using Python which function
> sits where and why. And what about the confusion over system(),
> popen(),commands(),spawn(), subprocess() etc. or why is there time
> and datetime? Sure it makes sense once you've played with Python
> for a while it makes some sense and you learn the role of history.
 
This is very much how this particular 'newbie' has experienced things. I'm
not here to damn Python, but to praise it, for opening my eyes to a whole
bunch of stuff. But you know when I teach biology and genetics, and the
kids don't get it, I feel like the onus is on me to improve my teaching. And
if I code a tool for people in my lab, and they can't use it, then I feel like
I've got some work to do, either in teaching or in making the tool easier to
use.
 
That's just me, Tiger, and I'm sorry it makes you spit venom. Not my
intention at all. But it's Alan's hand that I want to shake, because as
far as I can tell, he's looking to the future, to the next generation, to
the ugly reality and the bright potential, and quite frankly, you're not.

This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] feeding data to subprocess exes and getting results without writing files

[Barton David]

Hi there,
I can easily use the subprocess module to run a separate exe from within
MyScript.py..
 
e.g.
import subprocess
process=subprocess.Popen("myprog.exe -i inputfile.txt -o
outputfile.txt")
 
..and that's just fine as far as it goes, if I don't mind creating
'inputfile.txt' first, and reading 'outputfile.txt' afterward.
 
But what if I do mind? I'm trying to do something like..
 
stdin="my input string"
process=subprocess.Popen("myprog.exe -i stdin -o stdout") # (infact
these are the default args for -i and -o, so it seems like myprog.exe
can be used this way)
myresults=stdout
 
I just can't wrap my head around stdin, stdout and the whole pipes
thing, but there's got to be a relatively simple way to do this, surely?
 
 
thanks for any help,
Dave (relative newbie using PythonWin & Python 2.4)

This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] feeding data to subprocess exes and getting results without writing files

[Barton David]

Thanks Chris,

I figured it out after a while..

import subprocess
inputtext="my input string"
process=subprocess.Popen("myprog.exe -i stdin -o
stdout",stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.S
TDOUT)
outputtext,errortext=process.communicate(inputtext)
 
..so it is fairly simple after all.

Thanks for the help,
Dave

(ps I inadvertantly replied directly to you Chris, rather than the
mailing list- sorry-, so this is just for posterity in case anybody else
wants to know the solution)


-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On
Behalf Of Christopher Arndt
Sent: 09 January 2007 14:26
To: Tutor@python.org
Subject: Re: [Tutor] feeding data to subprocess exes and getting results
without writing files

Barton David schrieb:
> I just can't wrap my head around stdin, stdout and the whole pipes 
> thing, but there's got to be a relatively simple way to do this,
surely?

You have to distinguish between three different concepts:

1) file names
2) file objects
3) file contents

1) Is just a string with some identifier (the file path). To use a file
with that identifier in Python, you have to create a file object from it
by using the builtin 'open' function.

2) File objects are builtin Python objects that are usually created by
the 'open' function or returned by some other function. There are a few
file objects that are already opened and accessible to your Python
program. These are sys.stdin, sys.stderr and sys.stdout. They are file
objects, *not* strings representing the (non-existant) file name or file
content!

3) File contents are just represented by binary strings in Python. You
read/write them with the appropriate methods of file objects.

==> The subprocess.Popen constructor expects *file objects* not strings
for its 'stdin' and 'stdout' arguments. Read the documentation of
subprocess.Popen very carefully again (ignore references to file
*descriptors*).

BTW: a pipe is just an object or function that reads from one file
object and writes to another.

Chris
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Generating all possible hits of a regex pattern

[Barton David]

I don't suppose there's a nifty trick (exploiting the internal workings
of the re module perhaps) to generate all possible hits of a given regex
pattern?
 
Something like:
 
>>> import re
>>> mypattern=re.compile( "[AB]C{1,2}D?" )
>>> print list( all_pattern_generator( mypattern ) )
["AC","BC","ACC","BCC","ACD","BCD","ACCD","BCCD"]
 
Or would I have to do this the hard way, parsing the expression myself
etc?
 
I suppose you might have to be a little careful what expression you put
into this: "A*" would produce an infinite number of results. But if it
was in a generator function that wouldn't necessarily be a problem,
and/or perhaps I could pre-check 'mypattern' and warn if it was.
 
cheers
Dave
 
 
 
 
 
 

This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] file.read() doesn't give full contents of compressed files

[Barton David]

Hi,
I'm really confused, and I hope somebody can explain this for me...
 
I've been playing with compression and archives, and have some .zip,
.tar, .gz and .tgz example files to test my code on.
I can read them using either zipfile, tarfile, gzip or zlib, and that's
fine. But just reading them in 'raw' doesn't give me the whole string of
(compressed) bytes.
 
i.e...
 
len( file("mytestfile","r").read() ) != os.path.getsize("mytestfile")
 
Not even close, in fact. It seems like file.read() just stops after
reading a small portion of each example file, but why would that happen?
And what could I do if I wanted to read in the entire (compressed)
contents as a string?
 
thanks for any insight,
 
Dave (using Python 2.4 and windows)

This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] file.read() doesn't give full contents of compressed files

[Barton David]

Oh... of course. Thanks and sorry for missing the bleeding obvious.

Mind you, when reading in 'txt mode' rather than binary, len() actually
gives a much *smaller* size than getsize. Does the conversion into txt
happen to introduce some sort of terminator character that stops
file.read() from going to the end?

Dave


-Original Message-
From: Kent Johnson [mailto:[EMAIL PROTECTED] 
Sent: 20 February 2007 12:53
To: Barton David
Cc: tutor@python.org
Subject: Re: [Tutor] file.read() doesn't give full contents of
compressed files

Barton David wrote:
> Hi,
> I'm really confused, and I hope somebody can explain this for me...
>  
> I've been playing with compression and archives, and have some .zip, 
> .tar, .gz and .tgz example files to test my code on.
> I can read them using either zipfile, tarfile, gzip or zlib, and 
> that's fine. But just reading them in 'raw' doesn't give me the whole 
> string of
> (compressed) bytes.
>  
> i.e...
>  
> len( file("mytestfile","r").read() ) != os.path.getsize("mytestfile")
>  
> Not even close, in fact. It seems like file.read() just stops after 
> reading a small portion of each example file, but why would that
happen?
> And what could I do if I wanted to read in the entire (compressed) 
> contents as a string?

Why do you think it stops reading? len() should be giving a bigger
number than getsize() because you are reading the file in text mode
which will convert \n to \r\n. Try file("mytestfile","rb").

Kent


This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] file.read() doesn't give full contents of compressed files

[Barton David]

I see. Thanks for that.

dave 

-Original Message-
From: Kent Johnson [mailto:[EMAIL PROTECTED] 
Sent: 20 February 2007 13:30
To: Barton David
Cc: tutor@python.org
Subject: Re: [Tutor] file.read() doesn't give full contents of
compressed files

Barton David wrote:
> Oh... of course. Thanks and sorry for missing the bleeding obvious.
> 
> Mind you, when reading in 'txt mode' rather than binary, len() 
> actually gives a much *smaller* size than getsize. Does the conversion

> into txt happen to introduce some sort of terminator character that 
> stops
> file.read() from going to the end?

Actually, yes. A ctrl-Z is treated as an end-of-file when reading a file
in text mode on Windows.
http://groups.google.com/group/comp.lang.python/msg/4604aac0222a0043?hl=
en&

Kent


This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] using tarfile on strings or filelike objects

[Barton David]

I like that I can access the contents of a zip archive that's stored in
memory (rather than on disk) by packing the archive contents into a
StringIO or cStringIO object and feeding that to ZipFile...
 
i.e. 
 
filelike=cStringIO.StringIO(archive_as_string)
zf=zipfile.ZipFile(filelike)
content=zf.read(archive_member_name)
zf.close()
filelike.close()
 
but I can't get the same thing to work with TarFile. Is there any way to
do this? (Other than first saving the archive data to disk and then
passing the path to TarFile.open?) The tarfile module documentation
talks about an optional fileobj flag but this doesn't seem to work.
 
cheers
Dave

This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] using tarfile on strings or filelike objects

[Barton David]

Thanks Kent,

But.. I've actually found that that doesn't work (raises a ReadError),
and neither does..
>>>tf=tarfile.open(mode="r|*",fileobj=filelike)
..which raises "AttributeError: _Stream instance has no attribute
'dbuf'"

However if I explicitly state the compression type.. e.g.
>>>tf=tarfile.open(mode="r|bz2",fileobj=filelike)
..then I can indeed go on to..
>>>print tf.getnames()
>>>assert archive_member_name in tf.getnames()
..and it works ok. Having to explicitly state the compression type isn't
exactly ideal, but I guess it'll do me for now.

Unfortunately, I'm still having trouble actually reading the contents of
'archive_member'.
i.e. ..
>>>tf_filelike=tf.extractfile(archive_member_name)
>>>print tf_filelike.read()
..raises..
File "C:\Python24\lib\tarfile.py", line 551, in _readnormal
self.fileobj.seek(self.offset + self.pos)
File "C:\Python24\lib\tarfile.py", line 420, in seek
raise StreamError, "seeking backwards is not allowed"

And I get the same error if I instead try..
>>>tf_infoobject=tf.getmember(archive_member_name)
>>>tf_filelike=tf.extractfile(tf_infoobject)
>>>print tf_filelike.read()

In fact I'm getting this even if I open the archive by passing the path
name (rather than using fileobj) so I guess this isn't the problem I
initially thought it was. I just don't get it.

Regards,
Dave

-----Original Message-
From: Kent Johnson [mailto:[EMAIL PROTECTED] 
Sent: 03 March 2007 22:26
To: Barton David
Cc: tutor@python.org
Subject: Re: [Tutor] using tarfile on strings or filelike objects

Barton David wrote:
> I like that I can access the contents of a zip archive that's stored 
> in memory (rather than on disk) by packing the archive contents into a

> StringIO or cStringIO object and feeding that to ZipFile...
>  
> i.e.
>  
> filelike=cStringIO.StringIO(archive_as_string)
> zf=zipfile.ZipFile(filelike)
> content=zf.read(archive_member_name)
> zf.close()
> filelike.close()
>  
> but I can't get the same thing to work with TarFile. Is there any way 
> to do this? (Other than first saving the archive data to disk and then

> passing the path to TarFile.open?) The tarfile module documentation 
> talks about an optional fileobj flag but this doesn't seem to work.

Just from reading the docs I would try
filelike = cStringIO...
tf = tarfile.open(mode='r|', fileobj=filelike)

This is based on the example in the docs of reading from stdin.

Kent


This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] using tarfile on strings or filelike objects

[Barton David]

Thanks Kent,

I think I've hacked my way around this but it's a little weird.
Simplest to demonstrate with code (assuming cStringIO and tarfile are
imported and my_tarfile_string & archive_member_name are specified):
__

#approach 1
filelike=cStringIO.StringIO(my_tarfile_string)
tar = tarfile.open(mode="r|bz2", fileobj=filelike)

for tarinfo in tar:
if tarinfo.name==archive_member_name:
tfl=tar.extractfile(tarinfo)
print tfl.read()
tfl.close()

tar.close()
filelike.close()

#This works. Hooray.
#However, if mode=="r" the following error occurs: "AttributeError:
'NoneType' object has no attribute 'rfind'"
#and if mode=="r|" the following error occurs: "ReadError: empty,
unreadable or compressed file"
#and if mode=="r|*" the following error occurs: "AttributeError: _Stream
instance has no attribute 'dbuf'"
#I only understand the second of those error messages, to be honest.

#approach 2
filelike=cStringIO.StringIO(my_tarfile_string)
tar = tarfile.open(mode="r|bz2", fileobj=filelike)

tfl=tar.extractfile(archive_member_name)
print tfl.read()
tfl.close()

tar.close()
filelike.close()

#approach 3
filelike=cStringIO.StringIO(my_tarfile_string)
tar = tarfile.open(mode="r|bz2", fileobj=filelike)

tarinfo=tar.getmember(archive_member_name)
tfl=tar.extractfile(tarinfo)
print tfl.read()
tfl.close()

tar.close()
filelike.close()

#These DON'T work, and produce the following error: "StreamError:
seeking backwards is not allowed"
#Maybe this is just some wacky bug?


__
At any rate, in the 'slightly related' thread you linked to, the
statement that..
>"bzip2 compressed files cannot be read from a "fake" (StringIO) file
object, only from real files"
..doesn't seem to be the case after all, so that's something. 

Regards
Dave

P.S. I'm very sorry but whenever I post replies to Tutor (I'm using
Outlook and hitting Reply All in response to a tutor's direct reply) it
doesn't seem to add my message to the bottom of the existing thread. If
somebody can tell me what I'm doing wrong, I'd appreciate it.



-Original Message-
From: Kent Johnson [mailto:[EMAIL PROTECTED] 
Sent: 05 March 2007 12:44
To: Barton David
Cc: tutor@python.org
Subject: Re: [Tutor] using tarfile on strings or filelike objects

Barton David wrote:
> Thanks Kent,
> 
> But.. I've actually found that that doesn't work (raises a ReadError),

> and neither does..
>>>> tf=tarfile.open(mode="r|*",fileobj=filelike)
> ..which raises "AttributeError: _Stream instance has no attribute 
> 'dbuf'"
> 
> However if I explicitly state the compression type.. e.g.
>>>> tf=tarfile.open(mode="r|bz2",fileobj=filelike)
> ..then I can indeed go on to..
>>>> print tf.getnames()
>>>> assert archive_member_name in tf.getnames()
> ..and it works ok. Having to explicitly state the compression type 
> isn't exactly ideal, but I guess it'll do me for now.
> 
> Unfortunately, I'm still having trouble actually reading the contents 
> of 'archive_member'.
> i.e. ..
>>>> tf_filelike=tf.extractfile(archive_member_name)
>>>> print tf_filelike.read()
> ..raises..
> File "C:\Python24\lib\tarfile.py", line 551, in _readnormal
> self.fileobj.seek(self.offset + self.pos) File 
> "C:\Python24\lib\tarfile.py", line 420, in seek
> raise StreamError, "seeking backwards is not allowed"

The docs for the | options say that seeking won't work with them. Maybe
try just 'r'? A hacky workaround might be to open the file once to get
the names out and a second time to read the data. Since the file is in
memory that should be pretty quick.
> 
> And I get the same error if I instead try..
>>>> tf_infoobject=tf.getmember(archive_member_name)
>>>> tf_filelike=tf.extractfile(tf_infoobject)
>>>> print tf_filelike.read()
> 
> In fact I'm getting this even if I open the archive by passing the 
> path name (rather than using fileobj) so I guess this isn't the 
> problem I initially thought it was. I just don't get it.

What if you try exactly the code shown in the tarfile examples for
reading from stdin, with your StringIO file substituted for stdin?

You might want to ask about this on comp.lang.python as no one here
seems to know the answer. This thread is slightly related:
http://tinyurl.com/2hrffs

Kent

> 
> Regards,
> Dave

This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor