Re: How to safely maintain a status file
> What are you keeping in this status file that needs to be saved > several times per second? Depending on what type of state you're > storing and how persistent it needs to be, there may be a better way > to store it. > > Michael This is for a threaded web crawler. I want to cache what URL's are currently in the queue so if terminated the crawler can continue next time from the same point. -- http://mail.python.org/mailman/listinfo/python-list
Re: How to safely maintain a status file
> > and then on startup read from tmp_file if status_file does not exist. > > But this seems awkward. > > It also violates your requirement -- since the "crash" could take > place with a partial "temp file". Can you explain why? My thinking was if crash took place when writing the temp file this would not matter because the status file would still exist and be read from. The temp file would only be renamed when fully written. -- http://mail.python.org/mailman/listinfo/python-list
Re: How to safely maintain a status file
> Windows doesn't suppport atomic renames if the right side exists. I > suggest that you implement two code paths: > > if os.name == "posix": > rename = os.rename > else: > def rename(a, b): > try: > os.rename(a, b) > except OSError, e: > if e.errno != 183: > raise > os.unlink(b) > os.rename(a, b) Problem is if the process is stopped between unlink and rename there would no status file. -- http://mail.python.org/mailman/listinfo/python-list
Re: generate Windows exe on Linux
thanks Jérôme. Closest I have found is pyinstaller added support for cross-compiling a year ago by mounting a Windows partition on Linux: https://groups.google.com/forum/?fromgroups#!topic/pyinstaller/KISZP5sHCWg But it was not stable so will be removed: https://groups.google.com/forum/?fromgroups#!searchin/PyInstaller/linux$20windows/pyinstaller/veq3BlA_Bns I have come across many vague suggestions to try using Wine with py2exe / pyinstaller / cx_Freeze, but few accounts from people who have actually succeeded. Richard -- http://mail.python.org/mailman/listinfo/python-list
asynchronous downloading
I want to download content asynchronously. This would be straightforward to do threaded or across processes, but difficult asynchronously so people seem to rely on external libraries (twisted / gevent / eventlet). (I would use gevent under different circumstances, but currently need to stick to standard libraries.) I looked around and found there is little interest in developing a proper HTTP client on asyncore. The best I found stopped development a decade ago: http://sourceforge.net/projects/asynchttp/ What do you recommend? And why is there poor support for asynchronous execution? Richard -- http://mail.python.org/mailman/listinfo/python-list
Re: asynchronous downloading
My current implementation works fine below a few hundred threads. But each thread takes up a lot of memory so does not scale well. I have been looking at Erlang for that reason, but found it is missing useful libraries in other areas. -- http://mail.python.org/mailman/listinfo/python-list
Re: asynchronous downloading
that example is excellent - best use of asynchat I have seen so far. I read through the python-dev archives and found the fundamental problem is no one maintains asnycore / asynchat. -- http://mail.python.org/mailman/listinfo/python-list
