reading binary data from a 32 bit machine on 64 bit machine
Hi I am very confused with the use of the struct module to read binary
data from a file.
( I have only worked with ascii files so far)
I have a file spec for a Data-logger (http://www.dataq.com/support/
techinfo/ff.htm)
I am collecting some voltage , time traces on one channel and they are
written to the binary file on a 32 bit windows machine
The file spec says that the number of header bytes in the data file
header is stored as a 16 bit eye "I" at bits 6-7
Now I want to get at that number. When I try format !h I get a
meaningful number
If f is my file handle opened with "rb" mode
>>> f.seek(5)
>>> (Integer,) = struct.unpack('!h',f.read(2))
>>> (Integer,)
(9348,)
I am assuming that means that there are 9348 header bytes . Can
someone look at the format spec and tell me if I am on the right
track.
Also if a binary guru can point me in the direction of a single line
format pythonic way to get at the header that will be great
Thanks a tonne
hari
--
http://mail.python.org/mailman/listinfo/python-list
NameError question - def(self,master) - master not in namespace within class?
Hi I am new to writing module and object oriented python code. I am
trying to understand namespaces and classes in python.
I have the following test case given in three files runner , master
and child. I am getting an error within child where in one line it
understands variable master.name and in the next line it gives a
NameError as given here
"print "Reset name now from %s to %s , oldname %s is saved in
mastertrash" % (master.trash, master.name , master.trash)
NameError: name 'master' is not defined"
Sorry for a long post because I dont know how to frame my question.
I am pasting the code contained in the three files and the error
message here
Thanks for your help
harijay
The detailed error I get is
hazel:tmp hari$ python runner.py
Traceback (most recent call last):
File "runner.py", line 3, in
import child
File "/Users/hari/rpc-ccp4/tmp/child.py", line 1, in
class child():
File "/Users/hari/rpc-ccp4/tmp/child.py", line 9, in child
print "Reset name now from %s to %s , oldname %s is saved in
mastertrash" % (master.trash, master.name , master.trash)
NameError: name 'master' is not defined
#File runner.py
#!/usr/bin/python
import master
import child
if __name__=="__main__":
print "RUNNING RUNNER"
m = master.master("hj","oldhj")
s = child.child(m)
print "Now I have the variable master name %s and master.trash %s" %
(m.name , m.trash)
#File master.py
class master():
name=""
trash=""
def __init__(self,name,trash):
self.name = name
self.trash = trash
#File child.py
class child():
def __init__(self,master):
print "Master name is %s" % master.name
print "Now seeting master name to setnameinchild in child.py "
tmp = master.trash
master.trash = master.name
master.name = "setnameinchild"
print "Reset name now from %s to %s , oldname %s is saved in
mastertrash" % (master.trash, master.name , master.trash)
--
http://mail.python.org/mailman/listinfo/python-list
Re: NameError question - def(self, master) - master not in namespace within class?
Thanks beiff for your prompt reply - But I shouldnt need to import
master in child.
Actually and very strangely. The problem fixed itself without any
change to my code.
I dont understand how. It may have been a problem with a bad *.pyc
lingering around . But now I cannot get the old NameError to repeat
itself..its very weird
Here is the new code and it runs just fine ..I am absolutely confused
as to why it failed several times in a row and then fixed itself.
Here is the new code just to convince myself that there is absolutely
no change;
#File runner.py
#!/usr/bin/python
import master
import child
if __name__=="__main__":
print "RUNNING RUNNER"
m = master.master("hj","oldhj")
s = child.child(m)
print "Now I have the variable master name %s and master.trash %s" %
(m.name , m.trash)
#File child.py
class child():
def __init__(self,master):
print "Master name is %s" % master.name
print "Now seeting master name to setnameinchild in child.py "
tmp = master.trash
master.trash = master.name
master.name = "setnameinchild"
print "Reset name now from %s to %s , oldname %s is saved in
mastertrash" % (master.trash, master.name , master.trash)
#File master.py
class master():
name=""
trash=""
def __init__(self,name,trash):
self.name = name
self.trash = trash
Produces the output:
hazel:tmp hari$ python runner.py
RUNNING RUNNER
Master name is hj
Now seeting master name to setnameinchild in child.py
Reset name now from hj to setnameinchild , oldname hj is saved in
mastertrash
Now I have the variable master name setnameinchild and master.trash hj
On Oct 9, 11:54 am, [EMAIL PROTECTED] wrote:
> On 9 Ott, 17:43, harijay <[EMAIL PROTECTED]> wrote:
>
>
>
> > Hi I am new to writing module and object oriented python code. I am
> > trying to understand namespaces and classes in python.
>
> > I have the following test case given in three files runner , master
> > and child. I am getting an error within child where in one line it
> > understands variable master.name and in the next line it gives a
> > NameError as given here
>
> > " print "Reset name now from %s to %s , oldname %s is saved in
> > mastertrash" % (master.trash, master.name , master.trash)
> > NameError: name 'master' is not defined"
>
> > Sorry for a long post because I dont know how to frame my question.
> > I am pasting the code contained in the three files and the error
> > message here
> > Thanks for your help
> > harijay
>
> > The detailed error I get is
> > hazel:tmp hari$ python runner.py
> > Traceback (most recent call last):
> > File "runner.py", line 3, in
> > import child
> > File "/Users/hari/rpc-ccp4/tmp/child.py", line 1, in
> > class child():
> > File "/Users/hari/rpc-ccp4/tmp/child.py", line 9, in child
> > print "Reset name now from %s to %s , oldname %s is saved in
> > mastertrash" % (master.trash, master.name , master.trash)
> > NameError: name 'master' is not defined
>
> > #File runner.py
> > #!/usr/bin/python
> > import master
> > import child
>
> > if __name__=="__main__":
> > print "RUNNING RUNNER"
> > m = master.master("hj","oldhj")
> > s = child.child(m)
> > print "Now I have the variable master name %s and master.trash %s" %
> > (m.name , m.trash)
>
> > #File master.py
> > class master():
> > name=""
> > trash=""
>
> > def __init__(self,name,trash):
> > self.name = name
> > self.trash = trash
>
> > #File child.py
> > class child():
> > def __init__(self,master):
> > print "Master name is %s" % master.name
> > print "Now seeting master name to setnameinchild in
> > child.py "
> > tmp = master.trash
> > master.trash = master.name
> > master.name = "setnameinchild"
> > print "Reset name now from %s to %s , oldname %s is saved in
> > mastertrash" % (master.trash, master.name , master.trash)
>
> You need to have an import master in child.py too.
>
> Ciao
> -
> FB
>
> Ciao
>
> FB
--
http://mail.python.org/mailman/listinfo/python-list
Re: NameError question - def(self, master) - master not in namespace within class?
Thanks Jerry and beiff , Jerry was right, it was an indent problem . Between using my text editor and running from commandline something went out of sync and I didnt catch it probably. I can now reproduce the error with a bad ident . These are my first posts to comp.lang.python..and I am very grateful to everyone for their time . harijay On Oct 9, 12:07 pm, "Jerry Hill" <[EMAIL PROTECTED]> wrote: > On Thu, Oct 9, 2008 at 11:43 AM, harijay <[EMAIL PROTECTED]> wrote: > > Hi I am new to writing module and object oriented python code. I am > > trying to understand namespaces and classes in python. > > > I have the following test case given in three files runner , master > > and child. I am getting an error within child where in one line it > > understands variable master.name and in the next line it gives a > > NameError as given here > > > > > #File child.py > > class child(): > > def __init__(self,master): > > print "Master name is %s" % master.name > > print "Now seeting master name to setnameinchild in > > child.py " > > tmp = master.trash > > master.trash = master.name > > master.name = "setnameinchild" > > print "Reset name now from %s to %s , oldname %s is saved in > > mastertrash" % (master.trash, master.name , master.trash) > > The last line (starting with print), is not indented to the correct > level. Assuming it's supposed to be part of the child class's > __init__ method, it needs to line up with the rest of that method. > > -- > Jerry -- http://mail.python.org/mailman/listinfo/python-list
Re: please solve
On Oct 12, 12:15 pm, Raymond Cote <[EMAIL PROTECTED]> wrote: > shweta mani wrote: > > hi folks, > > i have been assigned a project on Python. i need to execute a remote > > shell script file from a windows machine through SSH twisted or > > paramiko. if it is a normal file then directly with the command sh > > .sh it is getting executed. > > self.conn.sendRequest(self, 'exec', common.NS(sh test1.sh), wantReply > > = 1) > > You could take a look at the Fabric project for some ideas as to how to > do this: > <http://www.nongnu.org/fab/documentation.html> > > --R ALso please look at the pexpect module. This allows you to supply "human" input to automate scripts that require it. Look at http://www.noah.org/wiki/Pexpect#Description_of_Pexpect Hope this helps Harijay -- http://mail.python.org/mailman/listinfo/python-list
matching exactly a 4 digit number in python
Hi
I am a few months new into python. I have used regexps before in perl
and java but am a little confused with this problem.
I want to parse a number of strings and extract only those that
contain a 4 digit number anywhere inside a string
However the regexp
p = re.compile(r'\d{4}')
Matches even sentences that have longer than 4 numbers inside
strings ..for example it matches "I have 3324234 and more"
I am very confused. Shouldnt the \d{4,} match exactly four digit
numbers so a 5 digit number sentence should not be matched .
Here is my test program output and the test given below
Thanks for your help
Harijay
PyMate r8111 running Python 2.5.1 (/usr/bin/python)
>>> testdigit.py
Matched I have 2004 rupees
Matched I have 3324234 and more
Matched As 3233
Matched 2323423414 is good
Matched dc sav 2412441 asdf
SKIPPED random1341also and also
SKIPPED
SKIPPED 13
Matched a 1331 saves
SKIPPED and and as dad
SKIPPED A has 13123123
SKIPPED A 13123
SKIPPED 123 adn
Matched 1312 times I have told you
DONE
#!/usr/bin/python
import re
x = [" I have 2004 rupees "," I have 3324234 and more" , " As 3233 " ,
"2323423414 is good"," dc sav 2412441 asdf " , "random1341also and
also" ,"","13"," a 1331 saves" ," and and as dad"," A has 13123123","
A 13123","123 adn","1312 times I have told you"]
p = re.compile(r'\d{4} ')
for elem in x:
if re.search(p,elem):
print "Matched " + elem
else:
print "SKIPPED " + elem
print "DONE"
--
http://mail.python.org/mailman/listinfo/python-list
Re: matching exactly a 4 digit number in python
Thanks John Machin and Mark Tolonen ..
SO I guess the correct one is to use the word boundary meta character
"\b"
so r'\b\d{4}\b' is what I need since it reads
a 4 digit number in between word boundaries
Thanks a tonne, and this being my second post to comp.lang.python. I
am always amazed at how helpful everyone on this group is
Hari
On Nov 21, 5:12 pm, John Machin <[EMAIL PROTECTED]> wrote:
> On Nov 22, 8:46 am, harijay <[EMAIL PROTECTED]> wrote:
>
> > Hi
> > I am a few months new into python. I have used regexps before in perl
> > and java but am a little confused with this problem.
>
> > I want to parse a number of strings and extract only those that
> > contain a 4 digit number anywhere inside a string
>
> > However the regexp
> > p = re.compile(r'\d{4}')
>
> > Matches even sentences that have longer than 4 numbers inside
> > strings ..for example it matches "I have 3324234 and more"
>
> No it doesn't. When used with re.search on that string it matches
> 3324, it doesn't "match" the whole sentence.
>
>
>
> > I am very confused. Shouldnt the \d{4,} match exactly four digit
> > numbers so a 5 digit number sentence should not be matched .
>
> {4} does NOT mean the same as {4,}.
> {4} is the same as {4,4}
> {4,} means {4,INFINITY}
>
> Ignoring {4,}:
>
> You need to specify a regex that says "4 digits followed by (non-digit
> or end-of-string)". Have a try at that and come back here if you have
> any more problems.
>
> some test data:
> xxx1234
> xxx12345
> xxx1234xxx
> xxx12345xxx
> xxx1234xxx1235xxx
> xxx12345xxx1234xxx
--
http://mail.python.org/mailman/listinfo/python-list
Re: Reading a tab delimited text file.
You could also use the csv module
import csv
myfileobj = open("myfiletab.txt","read")
csv_read = csv.reader(myfileobj,dialect=csv.excel_tab)
myval1 = []
myval2 = []
myval3 = []
for line in csv_read:
# filter header and stuff using some criterion
if len(line) = 3:
myval1.append(line[0])
myval2.append(line[1])
myval3.append(line[2])
etc etc . The csv module defines a dialect excel_tab. Each line is
then parsed into an array . You can insert that array into another
array and have an array line mydata = [ [line1],[line2] ...]
Hopw this helps
harijay
On Feb 23, 5:45 pm, Tim Chase wrote:
> >> time_vec, ch1_vec, and_so_on = zip(*(
> >> map(float, line.split())
> >> for line in file('in.txt')))
>
> >> If this isn't homework, there are some less terse versions which
> >> are a bit easier on the eyes and less like some love-child
> >> between Perl and Python.
>
> > haha, no this isn't homework. I'm a mechanical engineering student
> > working on a research project and this program is for my personal use
> > to analyze the data.
>
> The "zip-star map-float" variant is a pretty unreadable way to go.
>
> The more readable versions look something like
>
> data = [map(float, line.split()) for line in file('in.txt')]
> time_vec = [bit[0] for bit in data]
> ch1_vec = [bit[1] for bit in data]
> and_so_on = [bit[2] for bit in data]
>
> or even
>
> time_vec = []
> ch1_vec = []
> and_so_on = []
> for line in file('in.txt'):
> a,b,c = map(float, line.split())
> time_vec.append(a)
> ch1_vec.append(b)
> and_so_on.append(c)
>
> which could also be written as
>
> for line in file('in.txt'):
> line = line.split()
> time_vec.append(float(line[0]))
> ch1_vec.append(float(line[1]))
> and_so_on.append(float(line[2]))
>
> -tkc
Another way would be to use the csv module .
import csv
f = file.open("mytabfile.txt"
--
http://mail.python.org/mailman/listinfo/python-list
getting at individual bits inside byte field: struct module : bitwise operator
In my last post I had asked about reading data from a binary file
using the struct module.
Thanks to some excellent help , I have managed to read in
successfully
most of the header of this binary format that I want to parse. These
are some time-voltage traces from a digital
to analog converter for my experiments. The output is according to a
format
mentioned here : ( http://www.dataq.com/support/techinfo/ff.htm)
I have a question about how to bitmask a bunch of bytes read in from
such a binary formatted file .
For eg the spec says the first two bytes have different parameters in
different bits .
Byte 1 Byte 0
SN16 SD9SD8 SD7 SD6 SD5 SD4 SD3
SD2 SD1 SD0 T4 T3 T2 T1 T0
I am reading in the two bytes using the following code
import struct
f.seek(0)
element1_format = struct.Struct("http://mail.python.org/mailman/listinfo/python-list
Re: getting at individual bits inside byte field: struct module : bitwise operator
Thanks Gabriel , MRAB and Mark
Instead of using the bitwise operator on the unpacked data I was
trying to do a bitwise operator directly after read. This was giving
the unsupported operand error
i.e
# version in my code - wrong
file.read(2) & 0x001f
Instead of
# version in my example and now in my working code
(element1,) = struct.unpack(" wrote:
> "harijay" wrote in message
>
> news:4c7d58a1-830f-4f02-ba07-aa4910f5f...@b16g2000yqb.googlegroups.com...
>
>
>
> > In my last post I had asked about reading data from a binary file
> > using the struct module.
> > Thanks to some excellent help , I have managed to read in
> > successfully
> > most of the header of this binary format that I want to parse. These
> > are some time-voltage traces from a digital
> > to analog converter for my experiments. The output is according to a
> > format
> > mentioned here : (http://www.dataq.com/support/techinfo/ff.htm)
>
> > I have a question about how to bitmask a bunch of bytes read in from
> > such a binary formatted file .
>
> > For eg the spec says the first two bytes have different parameters in
> > different bits .
> > Byte 1 Byte 0
> > SN16 SD9 SD8 SD7 SD6 SD5 SD4 SD3
> > SD2 SD1 SD0 T4 T3 T2 T1 T0
>
> > I am reading in the two bytes using the following code
>
> > import struct
> > f.seek(0)
> > element1_format = struct.Struct(" > (element1,) = element1_format.unpack(f.read(2))
>
> > Now element1 has type "str" . How do I apply a bitmask to this to get
> > at information in the component bits .
> > Since the entire file format has many such bitmasked fields and since
> > this is my first venture into binary formats and c-type structs , I
> > wanted to know how to read values inside a byte using python.
> > My few tries at using bitwise operators ( element1 & 0x001f) are
> > frustrated by messages that say " unsupported operand type(s) for &:
> > 'str' and 'int' " .
> > How can I keep my string objects as bits and apply bitmasks to them
> > Any help in this will be greatly appreciated.
> > Thanks
> > hari
>
> Please post a small example that fails as you describe. From your example,
> element1 should be an int (unpacked from two bytes), and there is no f
> defined. Without the actual code we can't figure out what you are doing
> wrong.
>
> This works as expected:
>
> >>> import struct
> >>> format = struct.Struct(" >>> (element1,) = format.unpack('aa') # a two-byte string to unpack
> >>> print hex(element1)
> 0x6161
> >>> print element1 & 0x1f
>
> 1
>
> -Mark
--
http://mail.python.org/mailman/listinfo/python-list
running shelscript inside python
Hi I want to run shell scripts of the following kind from inside python and for some reason either the os.system or the subprocess.call ways are not working for me . I am calling a fortran command (f2mtz ) with some keyworded input that is normally piped (<<) in. The section of code that deals with the shell script is as follows script="""HKLIN %s HKLOUT %s
Re: running shelscript inside python
Hello Albert ,
Thanks for your pointers
I did finally get it to work
Here is what worked:
script = """SYMM %s
CELL %s
skipline
LABOUT H K L FP FOM PHIS X
CTYPOUT H H H F W P R
FORMAT '(3f4.0,f11.2,f8.2,f8.1,f8.2)'
END
eof""" %(options.symm,cellparams)
import subprocess
f2mtzargs = ["f2mtz hklin %s hklout %s" %
(options.phs,options.mtzfile),"< wrote:
> On Thu, 2009-03-26 at 15:23 -0700, harijay wrote:
> > Hi
> > I want to run shell scripts of the following kind from inside python
> > and for some reason either the os.system or the subprocess.call ways
> > are not working for me .
>
> > I am calling a fortran command (f2mtz ) with some keyworded input that
> > is normally piped (<<) in.
> > The section of code that deals with the shell script is as follows
>
> > script="""HKLIN %s HKLOUT %s < > SYMM %s
> > CELL %s
> > format '(3f4.0,f11.2,f8.2,f7.3,4f11.3)'
> > skipline 0
> > LABOUT H K L IP FOMS PHIS HLA HLB HLC HLD
> > CTYPOUT H H H I W P A A A A
> > END
> > eof
> > """ %(options.phs,options.mtzfile,options.symm,cellparams)
> > import subprocess
> > subprocess.call(["f2mtz" , script])
>
> > Normally I would type the commands above into a shell script and then
> > execute it
> > for example a file (tmp.sh)
>
> > #!/usr/bin/sh
> > f2mtz HKLIN 32_4run1.phs HKLOUT 32_4run1.mtz < > SYMM p1
> > CELL 79.814 102.639 153.627 82.465 75.531 73.450
> > format '(3f4.0,f11.2,f8.2,f7.3,4f11.3)'
> > skipline 0
> > LABOUT H K L IP FOMS PHIS HLA HLB HLC HLD
> > CTYPOUT H H H I W P A A A A
> > END
> > eof
>
> > And then run that script
> > "sh tmp.sh"
>
> > When I try subprocess.call(["f2mtz" , script]) or os.system("f2mtz %s"
> > % script)
>
> Ask yourself what you are trying to do:
>
> * Run an executable
> * pass arguments to the executable
> * Use a string standard input to the call
>
> Then re-read over the subprocess module and it should become clearer to
> you.
--
http://mail.python.org/mailman/listinfo/python-list
win32com pythonwin extensions part of stdlib?
Hi I am using Python 2.6.5 on Windows. I wanted to start using the win32com extensions which I understand are "essentially part of the stdlib" ( quoted in http://tgolden.sc.sabren.com/python/win32_how_do_i.html) Since I didnt have the extensions as standard I went to sourceforge to get the module. However when I tried to do a distutils install of the python extensions for windows downloaded from sourceforge , I found out I need some proprietary components from Microsoft visual studio 7)."vcsvarsall.bat" needed. My question is : Can I use the win32com extensions on a licensed windows setup without having access to Visual Studio components. I did see some places where they suggested compiling with mingw32 , but the compilation didnt work. Any help in getting the module running would be greatly appreciated -- http://mail.python.org/mailman/listinfo/python-list
Re: win32com pythonwin extensions part of stdlib?
Using the "binary" offered on sourceforge and calling "python setup.py install" Has the install asking for "vcsvarsall.bat" . So this does not seem to be a binary build On Dec 7, 1:27 pm, Ian wrote: > On Dec 7, 11:02 am, harijay wrote: > > > > > > > > > > > Hi I am using Python 2.6.5 on Windows. > > > I wanted to start using the win32com extensions which I understand are > > "essentially part of the stdlib" ( quoted > > inhttp://tgolden.sc.sabren.com/python/win32_how_do_i.html) > > Since I didnt have the extensions as standard I went to sourceforge > > to get the module. > > > However when I tried to do a distutils install of the python > > extensions for windows downloaded from sourceforge , I found out I > > need some proprietary components from Microsoft visual studio > > 7)."vcsvarsall.bat" needed. > > > My question is : Can I use the win32com extensions on a licensed > > windows setup without having access to Visual Studio components. > > > I did see some places where they suggested compiling with mingw32 , > > but the compilation didnt work. > > > Any help in getting the module running would be greatly appreciated > > There are pre-built binary distributions available on sourceforge. > Why not just use one of those? -- http://mail.python.org/mailman/listinfo/python-list
Re: win32com pythonwin extensions part of stdlib?
Thanks Ian for your reply. I downloaded the zip file linked by the "Download Binaries/View All files link" . All that gave me was a zip file which unpacked to the source and documentation implying I had to use distutils. If instead I looked at the Build 214 link on sourceforge which corresponds to the latest build ( same time stamp as the link labeled "Download Binaries / View all Files") There was an exe file corresponding to my python version. Which came with an installer that installed the binaries Thanks for your help I looked in the right place. Hari On Dec 7, 3:14 pm, harijay wrote: > Using the "binary" offered on sourceforge and calling > "python setup.py install" > > Has the install asking for "vcsvarsall.bat" . So this does not seem to > be a binary build > > On Dec 7, 1:27 pm, Ian wrote: > > > > > > > > > On Dec 7, 11:02 am, harijay wrote: > > > > Hi I am using Python 2.6.5 on Windows. > > > > I wanted to start using the win32com extensions which I understand are > > > "essentially part of the stdlib" ( quoted > > > inhttp://tgolden.sc.sabren.com/python/win32_how_do_i.html) > > > Since I didnt have the extensions as standard I went to sourceforge > > > to get the module. > > > > However when I tried to do a distutils install of the python > > > extensions for windows downloaded from sourceforge , I found out I > > > need some proprietary components from Microsoft visual studio > > > 7)."vcsvarsall.bat" needed. > > > > My question is : Can I use the win32com extensions on a licensed > > > windows setup without having access to Visual Studio components. > > > > I did see some places where they suggested compiling with mingw32 , > > > but the compilation didnt work. > > > > Any help in getting the module running would be greatly appreciated > > > There are pre-built binary distributions available on sourceforge. > > Why not just use one of those? -- http://mail.python.org/mailman/listinfo/python-list
Re: win32com pythonwin extensions part of stdlib?
Thanks Tim for your reply. I did get that build . Also thanks for the examples. Looking forward to using the module Hari On Dec 7, 3:22 pm, Tim Golden wrote: > You want something from here: > > http://sourceforge.net/projects/pywin32/files/pywin32/Build%20214/ > > Which one you need will depend on your architecture > and the version of Python you're running > > TJG -- http://mail.python.org/mailman/listinfo/python-list
OSError: [Errno 26] Text file busy during subprocess.check_call() :seems os dependent
Hi,
I am writing some multithreaded code which aims to automate three
sequential data processing applications and distribute the processing
on my 16GB RAM, 64 bit Ubuntu box running Python 2.6.5
The basic class that orchestrates these jobs use Queue.Queue() to feed
the product of the first job into the Queue for the next job.
Each Thread receives a dynamically generated shell script from some
classes I wrote and then runs the script using
subprocess.call(["shell_script_file.sh"])
I tested the code on a mac laptop and also on ubuntu. Curiously on Mac
OSX 32 bit Core duo running snow leopard, the code always runs fine.
However on my ubuntu box I get sporadic errors detailed below.
I tried replacing the
subprocess.call() with
subprocess.Popen(["shellscriptfile.sh"],stdout=open("unique_process_id.log","w"),stderr=open("unique_error_log.log","w")
But I get the same "OSError: [Errno 26] Text file busy" error
Everytime I run the same job queue a different part of the job fails.
Unfortunately I dont see anybody else reporting this OSError. ANy help
in troubleshooting my "newbie" thread code will be greatly
appreciated.
Thanks
hari
The orchestrator class is at:
https://github.com/harijay/auriga/blob/master/process_multi.py
A sample thread subclass is at :
https://github.com/harijay/auriga/blob/master/scatomtzrunthread.py
Detailed error:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 532, in
__bootstrap_inner
self.run()
File "/home/hari/Dropbox/auriga/scatomtzrunthread.py", line 28, in
run
stat = subprocess.call([file])
File "/usr/lib/python2.6/subprocess.py", line 480, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1139, in
_execute_child
raise child_exception
OSError: [Errno 26] Text file busy
--
http://mail.python.org/mailman/listinfo/python-list
Re: OSError: [Errno 26] Text file busy during subprocess.check_call() :seems os dependent
I am writing to a unique script file . Each script file has prefixes
like script1.sh script2.sh and they reside in different directories .
The scripts will never trample each other since they are all
sequential and shell scripts and no directory will have more than one
shell script.
The only thing I am doing is dir1 and dir2 will each have a
script1.sh . But that still makes it mutually exclusive paths.
Hari
On Dec 30, 5:34 pm, "Thomas L. Shinnick" wrote:
> At 03:46 PM 12/30/2010, harijay wrote:
>
> >Hi,
> >I am writing some multithreaded code which aims to automate three
> >sequential data processing applications and distribute the processing
> >on my 16GB RAM, 64 bit Ubuntu box running Python 2.6.5
>
> >The basic class that orchestrates these jobs use Queue.Queue() to feed
> >the product of the first job into the Queue for the next job.
>
> >Each Thread receives a dynamically generated shell script from some
> >classes I wrote and then runs the script using
>
> >subprocess.call(["shell_script_file.sh"])
>
> You say dynamically generated. Any chance you are (re)using the same
> filename each time? Is it possible that two uses of that filename
> could occur at the same time? That is, is it possible that at the
> same time while one process is running from the script file, another
> process is trying to re-write the script file? And so maybe you need
> to have dynamically generated and unique filenames
>
> Most often I see references to binary executable files for the error
> message, but I've also seen references to script files, e.g.
> http://www.cyberciti.biz/faq/binbash-bad-interpreter-text-file-busy/
>
>
>
> >I tested the code on a mac laptop and also on ubuntu. Curiously on Mac
> >OSX 32 bit Core duo running snow leopard, the code always runs fine.
> >However on my ubuntu box I get sporadic errors detailed below.
>
> >I tried replacing the
> >subprocess.call() with
>
> >subprocess.Popen(["shellscriptfile.sh"],stdout=open("unique_process_id.log
> >","w"),stderr=open("unique_error_log.log","w")
>
> >But I get the same "OSError: [Errno 26] Text file busy" error
>
> >Everytime I run the same job queue a different part of the job fails.
>
> >Unfortunately I dont see anybody else reporting this OSError. ANy help
> >in troubleshooting my "newbie" thread code will be greatly
> >appreciated.
>
> >Thanks
> >hari
>
> >The orchestrator class is at:
> >https://github.com/harijay/auriga/blob/master/process_multi.py
>
> >A sample thread subclass is at :
> >https://github.com/harijay/auriga/blob/master/scatomtzrunthread.py
>
> >Detailed error:
>
> >Exception in thread Thread-1:
> >Traceback (most recent call last):
> > File "/usr/lib/python2.6/threading.py", line 532, in
> >__bootstrap_inner
> > self.run()
> > File "/home/hari/Dropbox/auriga/scatomtzrunthread.py", line 28, in
> >run
> > stat = subprocess.call([file])
> > File "/usr/lib/python2.6/subprocess.py", line 480, in call
> > return Popen(*popenargs, **kwargs).wait()
> > File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
> > errread, errwrite)
> > File "/usr/lib/python2.6/subprocess.py", line 1139, in
> >_execute_child
> > raise child_exception
> >OSError: [Errno 26] Text file busy
--
http://mail.python.org/mailman/listinfo/python-list
Using select.kqueue to monitor garageband transcode completion
I want to automate a series of functions in python that trigger when
the OSX application Garagband finishes writing to a file called
"todays_recording.mp3".
A Typical transcode process takes 20 minutes , and I fancy starting
the python program immediately after I start the transcode and then
walking away.
For the present moment I have a silly implementation that does
something like the code pasted below.
I was looking online for smarter I/O monitoring and I came across the
kqueue classes inside of the select module which could be used to
monitor the kernel events in BSD - so it should work on OSX. However
being a newbie , I cannot understand how to setup the select.kqueue
event to look for garageband closing , i.e finish writing the
transcoded mp3 file.
I did see some code on comp.lang.python about this in a post from
Ritesh Nadhani (pasted below as well). But I dont understand what the
events mean . Looking for help to monitor the file closing using
select.kqueue.
Thanks
Hari
My Pseudocode for clunky monitoring of file i/o completion:
while True:
try:
today_file = open("todays_recording.mp3","r")
my_custom_function_to_process_file(today_file)
except IOError:
print "File not ready yet..continuing to wait"
###
Some source code I came across on comp.lang.python ( courtesy Ritesh
Vadvani) related to this
##
import select26 as select
import os
kq = select.kqueue()
fd = os.open("/tmp/a.txt", os.O_RDONLY)
# I dont understand this line below
ev = [select.kevent(fd, filter=select.KQ_FILTER_VNODE,
flags=select.KQ_EV_ADD | select.KQ_EV_ENABLE | select.KQ_EV_ONESHOT,
fflags=select.KQ_NOTE_WRITE | select.KQ_NOTE_DELETE
| select.KQ_NOTE_EXTEND)]
kq.control(ev, 0, 0)
try:
while True:
evts = kq.control(ev, 1, None)
if len(evts):
print evts
except KeyboardInterrupt:
pass
else:
kq.close()
os.close(fd)
--
http://mail.python.org/mailman/listinfo/python-list
multicpu bzip2 using os.system or queue using python script
I want to quickly bzip2 compress several hundred gigabytes of data
using my 8 core , 16 GB ram workstation.
Currently I am using a simple python script to compress a whole
directory tree using bzip2 and a system call coupled to an os.walk
call.
I see that the bzip2 only uses a single cpu while the other cpus
remain relatively idle.
I am a newbie in queue and threaded processes . But I am wondering how
I can implement this such that I can have four bzip2 running threads
(actually I guess os.system threads ), each using probably their own
cpu , that deplete files from a queue as they bzip them.
Thanks for your suggestions in advance
hari
My single thread script is pasted here .
import os
import sys
for roots, dirlist , filelist in os.walk(os.curdir):
for file in [os.path.join(roots,filegot) for filegot in filelist]:
if "bz2" not in file:
print "Compressing %s" % (file)
os.system("bzip2 %s" % file)
print ":DONE"
--
http://mail.python.org/mailman/listinfo/python-list
Re: multicpu bzip2 using os.system or queue using python script
Thanks a tonne..That code works perfectly and also shows me how to
think of using queue and threads in my python programs
Hari
On Jul 27, 1:26 pm, MRAB wrote:
> harijay wrote:
> > I want to quickly bzip2 compress several hundred gigabytes of data
> > using my 8 core , 16 GB ram workstation.
> > Currently I am using a simple python script to compress a whole
> > directory tree using bzip2 and a system call coupled to an os.walk
> > call.
>
> > I see that the bzip2 only uses a single cpu while the other cpus
> > remain relatively idle.
>
> > I am a newbie in queue and threaded processes . But I am wondering how
> > I can implement this such that I can have four bzip2 running threads
> > (actually I guess os.system threads ), each using probably their own
> > cpu , that deplete files from a queue as they bzip them.
>
> > Thanks for your suggestions in advance
>
> [snip]
> Try this:
>
> import os
> import sys
> from threading import Thread, Lock
> from Queue import Queue
>
> def report(message):
> mutex.acquire()
> print message
> sys.stdout.flush()
> mutex.release()
>
> class Compressor(Thread):
> def __init__(self, in_queue, out_queue):
> Thread.__init__(self)
> self.in_queue = in_queue
> self.out_queue = out_queue
> def run(self):
> while True:
> path = self.in_queue.get()
> sys.stdout.flush()
> if path is None:
> break
> report("Compressing %s" % path)
> os.system("bzip2 %s" % path)
> report("Done %s" % path)
> self.out_queue.put(path)
>
> in_queue = Queue()
> out_queue = Queue()
> mutex = Lock()
>
> THREAD_COUNT = 4
>
> worker_list = []
> for i in range(THREAD_COUNT):
> worker = Compressor(in_queue, out_queue)
> worker.start()
> worker_list.append(worker)
>
> for roots, dirlist, filelist in os.walk(os.curdir):
> for file in [os.path.join(roots, filegot) for filegot in filelist]:
> if "bz2" not in file:
> in_queue.put(file)
>
> for i in range(THREAD_COUNT):
> in_queue.put(None)
>
> for worker in worker_list:
> worker.join()
--
http://mail.python.org/mailman/listinfo/python-list
