Catch script hangs
Due to an ftp server issue, my python script sometimes hangs whilst downloading, unable to receive any more data. Is there any way that I could have python check, maybe through a thread or something, whether it has hanged (or just, if it's still active after 10 seconds, stop it?). I have looked at threading but there does not seem to be a stop method on threading, which is a problem. Could the lower level thread module be a solution? I was thinking something like: thread: spawn threaded timer, if it gets to 20, close the thread attempt download of file cancel threaded timer. exit thread. would that work at all? -- http://mail.python.org/mailman/listinfo/python-list
Are *.pyd's universal?
Can I use a pyd compiled on linux in a Windows distribution? Or must I recompile it for windows users? -- http://mail.python.org/mailman/listinfo/python-list
FTP Offset larger than file.
I am writing a python script that performs an identical function to
the 'tail' unix utility, except that it connects to its files over
FTP, rather than the local hard disk.
I am currently using this python script to generate an increasing
'logfile' of garbage.
> import time
> for i in range(1, 2):
> time.sleep(0.2)
> print i
> f = open("data1.log","a")
> f.write('%s: This logfile is being automatically generated to help
> Bakes test his python ftptail. \n' % i)
> f.close()
and use this script to actually download it.
>import time
>import os.path
>from ftplib import FTP
>
>#Empty the file
>filename = 'data1.log'
>file = open(filename, 'w')
>file.write('')
>file.close()
>
>def handleDownload(block):
>file.write(block)
>print ".",
>
># Create an instance of the FTP object
># Optionally, you could specify username and password:
>ftp=FTP(host, user, pass)
>
>directory = '/temp'
>ftp.cwd(directory)
>
>file = open(filename, 'a')
>
>for i in range(1,2):
> size=os.path.getsize('data1.log')
> ftp.retrbinary('RETR ' + filename, handleDownload, rest=size)
>
>file.close()
>
>print ftp.close()
Now, my problem is that I get a very strange error. What should be
happening is the script gets the size of the local file before
downloading all of the external file after that offset.
The error I get is:
ftplib.error_temp: 451-Restart offset 24576 is too large for file size
22852.
451 Restart offset reset to 0
which tells me that the local file is larger than the external file,
by about a kilobyte. Certainly, the local file is indeed that size, so
my local script is doing the right things. I do wonder what is going
wrong, can anyone enlighten me?
--
http://mail.python.org/mailman/listinfo/python-list
Re: FTP Offset larger than file.
On 28 July, 15:01, Hrvoje Niksic wrote: > Bakes writes: > > The error I get is: > > ftplib.error_temp: 451-Restart offset 24576 is too large for file size > > 22852. > > 451 Restart offset reset to 0 > > which tells me that the local file is larger than the external file, > > by about a kilobyte. Certainly, the local file is indeed that size, so > > my local script is doing the right things. I do wonder what is going > > wrong, can anyone enlighten me? > > I'd say you failed to take buffering into account. You write into a > buffered file, yet you use os.path.getsize() to find out the current > file size. If the data is not yet flushed, you keep re-reading the same > stuff from the remote file, and writing it out. Once the buffer is > flushed, your file will contain more data than was retrieved from the > remote side, and eventually this will result in the error you see. > > As a quick fix, you can add a file.flush() line after the > file.write(...) line, and the problem should go away. Thank you very much, that worked perfectly. -- http://mail.python.org/mailman/listinfo/python-list
Re: FTP Offset larger than file.
On 28 July, 15:18, Bakes wrote: > On 28 July, 15:01, Hrvoje Niksic wrote: > > > > > Bakes writes: > > > The error I get is: > > > ftplib.error_temp: 451-Restart offset 24576 is too large for file size > > > 22852. > > > 451 Restart offset reset to 0 > > > which tells me that the local file is larger than the external file, > > > by about a kilobyte. Certainly, the local file is indeed that size, so > > > my local script is doing the right things. I do wonder what is going > > > wrong, can anyone enlighten me? > > > I'd say you failed to take buffering into account. You write into a > > buffered file, yet you use os.path.getsize() to find out the current > > file size. If the data is not yet flushed, you keep re-reading the same > > stuff from the remote file, and writing it out. Once the buffer is > > flushed, your file will contain more data than was retrieved from the > > remote side, and eventually this will result in the error you see. > > > As a quick fix, you can add a file.flush() line after the > > file.write(...) line, and the problem should go away. > > Thank you very much, that worked perfectly. Actually, no it didn't. That fix works seamlessly in Linux, but gave the same error in a Windows environment. Is that expected? -- http://mail.python.org/mailman/listinfo/python-list
Re: FTP Offset larger than file.
On Jul 28, 5:36 pm, Dave Angel wrote: > Bakes wrote: > > On 28 July, 15:18, Bakes wrote: > > >> On 28 July, 15:01, Hrvoje Niksic wrote: > > >>> Bakes writes: > > >>>> The error I get is: > >>>> ftplib.error_temp: 451-Restart offset 24576 is too large for file size > >>>> 22852. > >>>> 451 Restart offset reset to 0 > >>>> which tells me that the local file is larger than the external file, > >>>> by about a kilobyte. Certainly, the local file is indeed that size, so > >>>> my local script is doing the right things. I do wonder what is going > >>>> wrong, can anyone enlighten me? > > >>> I'd say you failed to take buffering into account. You write into a > >>> buffered file, yet you use os.path.getsize() to find out the current > >>> file size. If the data is not yet flushed, you keep re-reading the same > >>> stuff from the remote file, and writing it out. Once the buffer is > >>> flushed, your file will contain more data than was retrieved from the > >>> remote side, and eventually this will result in the error you see. > > >>> As a quick fix, you can add a file.flush() line after the > >>> file.write(...) line, and the problem should go away. > > >> Thank you very much, that worked perfectly. > > > Actually, no it didn't. That fix works seamlessly in Linux, but gave > > the same error in a Windows environment. Is that expected? > > This is a text file you're transferring. And you didn't specify "wb". > So the Windows size will be larger than the Unix size, since you're > expanding the newline characters. > > getsize() is looking at the size after newlines are expanded to 0d0a, > while The remote file, presumably a Unix system likely has just has 0a. > > I think you'd do best just keeping track of the bytes you've written. > > DaveA Thank you very much, that worked perfectly. -- http://mail.python.org/mailman/listinfo/python-list
regexp help
If I were using the code: (?P[0-9]+) to get an integer between 0 and 9, how would I allow it to register negative integers as well? -- http://mail.python.org/mailman/listinfo/python-list
Odd/Weird errors with FTPLib
I am using a simple python script to download my logfiles. This is on a while loop, the logfile grows rapidly, so it is necessary for python to start downloading the new script as soon as it has finished the old. It works fine (for about 20 minutes), then crashes. I have removed a couple of excepts, and have narrowed the error down to a 'error_perm: 550 logfile.log: The data is invalid.' error. Does anyone know what the problem might be regarding this, and what I might do to fix it? It's really irritating to have it break every so often, and at the moment, I have no chance in which to fix the error. -- http://mail.python.org/mailman/listinfo/python-list
Re: Odd/Weird errors with FTPLib
On 13 Sep, 22:41, Chris Rebert wrote:
> On Sun, Sep 13, 2009 at 2:34 PM, Bakes wrote:
> > I am using a simple python script to download my logfiles. This is on
> > a while loop, the logfile grows rapidly, so it is necessary for python
> > to start downloading the new script as soon as it has finished the
> > old.
>
> > It works fine (for about 20 minutes), then crashes. I have removed a
> > couple of excepts, and have narrowed the error down to a 'error_perm:
> > 550 logfile.log: The data is invalid.' error.
>
> > Does anyone know what the problem might be regarding this, and what I
> > might do to fix it?
>
> Including an actual code snippet and the full error traceback would help a
> lot.
>
> According tohttp://en.wikipedia.org/wiki/List_of_FTP_server_return_codes,
> error code 550 translates to:
> "Requested action not taken. File unavailable (e.g., file not found,
> no access)."
>
> Does the logfile get rotated or something, thus causing it to briefly not
> exist?
>
> It might also help if you explain how your logfile system works.
>
> Cheers,
> Chris
> --http://blog.rebertia.com
It's a cod4 gameserver logfile, being downloaded for a python bot to
parse.
The logfile is downloaded using this try/except while loop.
while True:
try:
if ftp == False:
self.debug('FTP connection not active, attempting to
(re)connect')
ftp = self.ftpconnect()
size=os.path.getsize('games_mp.log')
ftp.retrbinary('RETR ' + os.path.basename(self.ftpconfig
['path']), handleDownload, rest=size)
if self.console._paused:
self.console.unpause()
except:
print error
self.debug('Lost connection to server, pausing until
updated properly, Sleeping 10 seconds')
self.console.pause()
try:
ftp.close()
self.debug('FTP Connection Closed')
except:
self.debug('FTP does not appear to be open, so not
closed')
ftp = False
time.sleep(10)
I can only assume that occasionally, the logfile is being written to
by the gameserver at the same time that it's downloading.
If this was the case, do you think a try: download except: sleep
900msec then download loop would work?
--
http://mail.python.org/mailman/listinfo/python-list
Re: Odd/Weird errors with FTPLib
On Sep 13, 11:47 pm, MRAB wrote:
> Bakes wrote:
> > On 13 Sep, 22:41, Chris Rebert wrote:
> >> On Sun, Sep 13, 2009 at 2:34 PM, Bakes wrote:
> >>> I am using a simple python script to download my logfiles. This is on
> >>> a while loop, the logfile grows rapidly, so it is necessary for python
> >>> to start downloading the new script as soon as it has finished the
> >>> old.
> >>> It works fine (for about 20 minutes), then crashes. I have removed a
> >>> couple of excepts, and have narrowed the error down to a 'error_perm:
> >>> 550 logfile.log: The data is invalid.' error.
> >>> Does anyone know what the problem might be regarding this, and what I
> >>> might do to fix it?
> >> Including an actual code snippet and the full error traceback would help a
> >> lot.
>
> >> According tohttp://en.wikipedia.org/wiki/List_of_FTP_server_return_codes,
> >> error code 550 translates to:
> >> "Requested action not taken. File unavailable (e.g., file not found,
> >> no access)."
>
> >> Does the logfile get rotated or something, thus causing it to briefly not
> >> exist?
>
> >> It might also help if you explain how your logfile system works.
>
> >> Cheers,
> >> Chris
> >> --http://blog.rebertia.com
>
> > It's a cod4 gameserver logfile, being downloaded for a python bot to
> > parse.
>
> > The logfile is downloaded using this try/except while loop.
>
> > while True:
> > try:
> > if ftp == False:
> > self.debug('FTP connection not active, attempting to
> > (re)connect')
> > ftp = self.ftpconnect()
> > size=os.path.getsize('games_mp.log')
> > ftp.retrbinary('RETR ' + os.path.basename(self.ftpconfig
> > ['path']), handleDownload, rest=size)
> > if self.console._paused:
> > self.console.unpause()
> > except:
> > print error
> > self.debug('Lost connection to server, pausing until updated
> > properly, Sleeping 10 seconds')
> > self.console.pause()
> > try:
> > ftp.close()
> > self.debug('FTP Connection Closed')
> > except:
> > self.debug('FTP does not appear to be open, so not closed')
> > ftp = False
> > time.sleep(10)
>
> > I can only assume that occasionally, the logfile is being written to
> > by the gameserver at the same time that it's downloading.
> > If this was the case, do you think a try: download except: sleep
> > 900msec then download loop would work?
>
> Bare excepts are almost always a bad idea because they'll catch _all_
> exceptions, both those you expect could happen and those you don't.
> Catch only those you expect.
>
> For example, if the file 'games_mp.log' doesn't exist then
> os.path.getsize('games_mp.log') will raise an exception, and if you
> forgot to import the os module then that will raise a NameError
> exception.
>
> Anyway, I can't see how you leave the loop; I'd expect something like a
> 'break' statement.
>
> And as a matter of style, I'd prefer None to False to indicate when
> there's no FTP connection (and "if not ftp" instead of "if ftp ==
> False").
I removed the try/except and saw when it failed.
I'll change those things, games_mp.log is guaranteed to be there (file
made in another script), os is imported correctly.
So, what do you think the error could be?
--
http://mail.python.org/mailman/listinfo/python-list
Re: Odd/Weird errors with FTPLib
On Sep 15, 1:32 am, MRAB wrote:
> Bakes wrote:
> > On Sep 13, 11:47 pm, MRAB wrote:
> >> Bakes wrote:
> >>> On 13 Sep, 22:41, Chris Rebert wrote:
> >>>> On Sun, Sep 13, 2009 at 2:34 PM, Bakes wrote:
> >>>>> I am using a simple python script to download my logfiles. This is on
> >>>>> a while loop, the logfile grows rapidly, so it is necessary for python
> >>>>> to start downloading the new script as soon as it has finished the
> >>>>> old.
> >>>>> It works fine (for about 20 minutes), then crashes. I have removed a
> >>>>> couple of excepts, and have narrowed the error down to a 'error_perm:
> >>>>> 550 logfile.log: The data is invalid.' error.
> >>>>> Does anyone know what the problem might be regarding this, and what I
> >>>>> might do to fix it?
> >>>> Including an actual code snippet and the full error traceback would help
> >>>> a lot.
> >>>> According tohttp://en.wikipedia.org/wiki/List_of_FTP_server_return_codes,
> >>>> error code 550 translates to:
> >>>> "Requested action not taken. File unavailable (e.g., file not found,
> >>>> no access)."
> >>>> Does the logfile get rotated or something, thus causing it to briefly
> >>>> not exist?
> >>>> It might also help if you explain how your logfile system works.
> >>>> Cheers,
> >>>> Chris
> >>>> --http://blog.rebertia.com
> >>> It's a cod4 gameserver logfile, being downloaded for a python bot to
> >>> parse.
> >>> The logfile is downloaded using this try/except while loop.
> >>> while True:
> >>> try:
> >>> if ftp == False:
> >>> self.debug('FTP connection not active, attempting to
> >>> (re)connect')
> >>> ftp = self.ftpconnect()
> >>> size=os.path.getsize('games_mp.log')
> >>> ftp.retrbinary('RETR ' + os.path.basename(self.ftpconfig
> >>> ['path']), handleDownload, rest=size)
> >>> if self.console._paused:
> >>> self.console.unpause()
> >>> except:
> >>> print error
> >>> self.debug('Lost connection to server, pausing until updated
> >>> properly, Sleeping 10 seconds')
> >>> self.console.pause()
> >>> try:
> >>> ftp.close()
> >>> self.debug('FTP Connection Closed')
> >>> except:
> >>> self.debug('FTP does not appear to be open, so not
> >>> closed')
> >>> ftp = False
> >>> time.sleep(10)
> >>> I can only assume that occasionally, the logfile is being written to
> >>> by the gameserver at the same time that it's downloading.
> >>> If this was the case, do you think a try: download except: sleep
> >>> 900msec then download loop would work?
> >> Bare excepts are almost always a bad idea because they'll catch _all_
> >> exceptions, both those you expect could happen and those you don't.
> >> Catch only those you expect.
>
> >> For example, if the file 'games_mp.log' doesn't exist then
> >> os.path.getsize('games_mp.log') will raise an exception, and if you
> >> forgot to import the os module then that will raise a NameError
> >> exception.
>
> >> Anyway, I can't see how you leave the loop; I'd expect something like a
> >> 'break' statement.
>
> >> And as a matter of style, I'd prefer None to False to indicate when
> >> there's no FTP connection (and "if not ftp" instead of "if ftp ==
> >> False").
>
> > I removed the try/except and saw when it failed.
>
> > I'll change those things, games_mp.log is guaranteed to be there (file
> > made in another script), os is imported correctly.
>
> > So, what do you think the error could be?
>
> How does control leave the while loop?
>
> If you're running on Windows and you're opening the file 'games_mp.log'
> with mode 'w' then there'll be the issue of the line-endings. If the log
> file on the server uses '\n' for the line endings and you're using
> '\r\n' then os.path.getsize('games_mp.log') will return a larger size
> then you expect. In that case, could ftp.retrbinary() be complaining
> because it's given you the entire file and then you're trying to
> download from an offset that's beyond the end?
>
> For example, suppose the file on the server contains just "foo\n" (4
> bytes):
>
> 1. You open a file locally in text mode ('w').
>
> 2. You download the entire file, "foo\n".
>
> 3. You write out the data, but because you've opened in text mode it
> writes "foo\r\n" to the file. The local file size is now 5 bytes.
>
> 4. The loop means that you then try to download from offset 5, which is
> beyond the end of the file on the server.
>
> 5. Error?
It opens it in append mode, and if it errors with that message, the
ending is different. I have come across that problem before, and it
has been fixed (download in binary is the fix, mostly).
--
http://mail.python.org/mailman/listinfo/python-list
Get error message from FTPLib
I am using ftplib for a project, using a try/except loop. I would like to find out the exception, but I am a python newbie and do not know how. How, in a try/except loop would I find the ftplib exception? -- http://mail.python.org/mailman/listinfo/python-list
