Re: [Python-Dev] Ext4 data loss
> > > > Ok. In that use case, however, it is completely irrelevant whether the > > tempfile module calls fsync. After it has generated the non-conflicting > > filename, it's done. > > I agree, but my comment was that it would be nice if better fsync > support (if added) could be done in such a way that it helped not only > file objects, but also *file-like* objects (such as the wrappers in the > tempfile module) without making the file-like API any fatter. fsync() might not be "the answer". I found this blog post very entertaining to read: http://www.advogato.org/person/mjg59/diary.html?start=195 "So, on the one hand, we're trying to use things like relatime to batch data to reduce the amount of time a disk has to be spun up. And on the other hand, we're moving to filesystems that require us to generate *more* io in order to guarantee that our data hits disk, which is a guarantee we often don't want anyway! Users will be fine with losing their most recent changes to preferences if a machine crashes. They will not be fine with losing the entirity of their preferences. Arguing that applications need to use *fsync*() and are otherwise broken is ignoring the important difference between these use cases. It's no longer going to be possible to spin down a disk when any software is running at all, since otherwise it's probably going to write something and then have to *fsync* it out of sheer paranoia that something bad will happen. And then probably *fsync* the directory as well, because what if someone writes an even more pathological filesystem. And the disks sit there spinning gently and chitter away as they write tiny files[4] and never spin down and the polar bears all drown in the bitter tears of application developers who are forced to drink so much to forget that they all die of acute liver failure by the age of 35 and where are we then oh yes we're fucked." -M ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] VM imaging based launch optimizations for CPython?
Hi fellow snakemen and lizard ladies, We have been recently done lots of Python work on Nokia Series 60 phones and even managed to roll out some commercial Python based applications. In the future we plan to create some iPhone Python apps also. Python runs fine in phones - after it has been launched. Currently the biggest issue preventing the world dominance of Python based mobile applications is the start up time. We cope with the issue by using fancy splash screens and progress indicators, but it does't cure the fact that it takes a minute to show the main user interface of the application. Most of the time is spend in import executing opcodes and forming function and class structures in memory - something which cannot be easily boosted. Now, we have been thinking. Maemo has fork() based Python launcher ( http://blogs.gnome.org/johan/2007/01/18/introducing-python-launcher/) which greatly speed ups the start up time by holding Python in memory all the time. We cannot afford such luxury on Symbian and iPhone, since we do not control the operating system. So how about this 1. A Python application is launched normally 2. After VM has initialized module importing and reached a static launch state (meaning that the state is same on every launch) the VM state is written on to disk 3. Application continues execution and starts doing dynamic stuff 4. On the following launches, special init code is used which directly blits VM image from disk back to memory and we have reached the static state again without going whoops of executing import related opcodes 5. Also, I have heard a suggestion that VM image could be defragmented and analyzed offline Any opinions? Cheers, Mikko -- Mikko Ohtamaa Red Innovation Ltd. Oulu, Finland http://www.redinnovation.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] VM imaging based launch optimizations for CPython?
On Mon, Dec 22, 2008 at 12:09 PM, Erno Kuusela wrote: > > unexec probably work out of the box on symbian, but...: > > http://mail.python.org/pipermail/python-dev/2003-May/035727.html > > unexec() is pretty much what I was looking for. However, looks like its old hack from 80s and cannot be applied as is to the modern environment. Basically unexec() dumps the running application code (not specific to any interpreter) and data segments out as a.out binary. 1) Generating a binary file is not possible on Symbian and iPhone environments, because all binaries must be signed - however, we can probably use a generic stub exe which loads data segment only 2) a.out format is deprecated 3) Dynamic DLLs are not managed - basically a show stopper I hope I could find someone find enough OS fu to tell whether this is possible with DLLs at all and what data pointers must be patched on each unexec() call. -Mikko ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] VM imaging based launch optimizations for CPython?
> > > > Of course, you still have the actual interpretation of > the top-level module code - if it's not the marshalling > but this part that actually costs performance, this > efficient marshalling algorithm won't help. It would be > interesting to find out which modules have a particularly > high startup cost - perhaps they can be rewritten I am afraid this is the case. I hope we could marshal an arbitary application state (not even Python specific) into a fast loading dump file (hibernation/snapshot). We have tried to use lazy importing as much as possible to distribute the importing cost across the application UI states. Out of my head I know at least two particular module which could be refactored. I'd recommend as the best practice that everything should be imported lazily if it's possible. However, looks like currently Python community is moving to another direction, since doing explict imports in __init__ etc. makes APIs cleaner (think Django) and debugging more sane task - Python is mainly used on the server and limited environments haven't been particular interesting until lately. logging - defines lots of classes which are used only if they are specified by logging options. I once hacked this for my personal use to be little lighter. urllib - particular heavy, imports httplib, ftplib and stuff even if it is not used Nokia has just released Python 2.5 based PyS60. I think we'll come back this after a while with a nice generic profiler which will tell the import cost. Merry XMas, -Mikko ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
