BH Hello,
Hope you don't mind my less-waste-your-time more-get-to-the-point English. Begin ramble: Wanted to fix QFileSystemWatcher for Windows as per w00t's article: <http://blog.rburchell.com/2012/03/qt-51-aka-when-qfilesystemwatcher-might.html> (also see his fix: <https://codereview.qt-project.org/#change,19274> ) Realized that what I really needed was *recursive* monitoring, entire trees, which the WinAPI function "ReadDirectoryChangesW" <http://msdn.microsoft.com/en-us/library/windows/desktop/aa365465(v=vs.85).aspx> that was mentioned in w00t's article offers right out of the box. Others in #qt-labs also believe recursive file-system monitoring to be a useful feature to offer in Qt. Here is my research on the topic: #### 1. Status of file-system monitoring on all major platforms. Windows ------- API: ReadDirectoryChangesW Recursive: yes Granular info about what changed: yes Linux/X11 --------- API: inotify Recursive: no, but can be achieved manually* Granular info about what changed: yes *I like to refer to the Gnome Tracker project (and probably also the KDE Nepomuk project), they manually traverse & spawn another inotify monitor for each & every single dir in entire trees. They also claim that it doesn't hurt performance too much either. It's the best you can currently get, there's currently no other way of doing it. MacOSX ------ API: Kqueue Recursive: no, and cannot be achieved manually* Granular info about what changed: yes * Max 256 file descriptors anyway, so out of question for recursive purposes. ------ API: FSEvents Recursive: yes Granular info about what changed: no, but you are notified about exactly where the change happened (leaf-most directory, direct parent of where change happened) so at most you need to manually poll contents of just 1 directory. Also multiple changes are compounded into 1 event, configurable. MUST read: <http://arstechnica.com/apple/2007/10/mac-os-x-10-5/7/> #### 2. Problems with making QFileSystemWatcher recursive - Windows ReadDirectoryCHangesW is the winner here. Would be easiest to implement, we could even drop all the threading currently there now. - inotify gives us all the chills at the thought of manually generating watches recursively for entire directory trees, but it can be done satisfactorially enough behind the scenes inside QFileSystemWatcher. - Mac OS is the troublemaker here; Kqueue is out of question, as we established. With FSEvents you cannot emit signals such as fileChanged(), fileModified(), pathCreated(), or pathDeleted() if all you're notified about is that "something happened in this folder here", you MUST have some sort of snapshot of the folder to compare against, and QFileSystemWatcher generating an in-memory snapshot of the user's requested directories to be watched can be crazy time consuming and memory draining. Try doing this with an entire drive of some 40,000 folders and 300K files. #### 3. What are the use cases anyway? Reevaluate: what is file-system watching good for anyway? There are two general use-case categories I can think of: A) Your everyday lightweight necessities, like "has the config file changed". QFileSystemWarcher is just fine for this with inotify & kqueue (though Windows can still use some ReadDirectoryChangesW love instead of the current use of Find*ChangeNotification APIs). B) Heavy duty monitoring of entire hierachies, like monitoring dir trees in an image library, backup applications, file crawling / indexing, virus scanning etc. If you think about it, I suspect all B-style applications can be expected to have their own snapshot of the file system stored anyway. Any app that needs to monitor changes to THAT much stuff probably has it's own view of the file system that it just wants to keep updated. If this assumption is acceptable, we discover that granular info about what exactly changed is (useful but) not too urgent after all. So maybe FSEvents on mac isn't so bad. The application will only need to compare 1 folder contents with it's own snapshot to determine what changed. So now what; If we care to offer recursive FS monitoring in Qt, we can either try to stuff it into QFileSystemWatcher, or make a new class for it (perhaps platform specific API classes if necessary). #### 4. The first option: Enhanced QFileSystemWatcher If we do it this way, we'll one single awesome class that does it all. We just may have to make peace with varying usage/behavior on different platforms. If this is acceptable, we're in business. For example, here are some things we might need to mention in the docs; QFileSystemWatcher::fileModified() NOTE: This signal is not emitted on Mac OS X. Instead, directoryChanged() is emitted and the user is expected to investigate further about what exactly changed. (Believe me I tried developing a solution where even on Mac with FSEvents QFileSystemWatcher would do this "investigation" behind the scenes. I failed). QFileSystemWatcher::addPathRecursively() NOTE: this can take some time on Linux/X11 as inotify monitors are manually spawned recursively for each directory in the entire tree under the specified path. Also, very excessive amounts of inotify monitors may eventually cause system instability (slowing down). #### 5. Second option: QRecursiveFileSystemWatcher Let the current QFileSystemWatcher remain the simple everyday API for that "A" category of lightweight use cases. Let it even use Kqueue on Mac OS, it's better for that. And then, let's make a new Qt class for those "B" use cases where recursive is the way to go. This class (perhaps named QRecursiveFileSystemWatcher?) will have platform specific features/functionality from the very start, all of it's notes & warnings clearly documented. There can be common functions that work the same on all platforms, and then there can be those few functions or parameters to functions or return values from functions etc which are maybe ingonred on this or that platform, or they're never used on the other platform, or return an error on some platforms, etc. #### 6. Conclusion Please share your thoughts / opinions / comments, how should this best be done so that we can have the awesome feature of recursive file-system monitoring in Qt and still maintain a solid library API that works as best and as consistently as possible on each different platform. Thanks for your time -regedit _______________________________________________ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
