Darn, second attempt to fix my mess-up here in the mailing list only made things worse. Shoot me. -regedit
On Fri, Jul 20, 2012 at 1:21 AM, logic.cpp <logic....@gmail.com> wrote: > BH > > Hello, > > Hope you don't mind my less-waste-your-time more-get-to-the-point English. > Begin ramble: > > Wanted to fix QFileSystemWatcher for Windows as per w00t's article: > <http://blog.rburchell.com/2012/03/qt-51-aka-when-qfilesystemwatcher-might.html> > (also see his fix: <https://codereview.qt-project.org/#change,19274> ) > > Realized that what I really needed was *recursive* monitoring, entire trees, > which the WinAPI function "ReadDirectoryChangesW" > <http://msdn.microsoft.com/en-us/library/windows/desktop/aa365465(v=vs.85).aspx> > that was mentioned in w00t's article offers right out of the box. > > Others in #qt-labs also believe recursive file-system monitoring to be a > useful > feature to offer in Qt. > > Here is my research on the topic: > > > #### 1. Status of file-system monitoring on all major platforms. > > Windows > ------- > API: ReadDirectoryChangesW > Recursive: yes > Granular info about what changed: yes > > Linux/X11 > --------- > API: inotify > Recursive: no, but can be achieved manually* > Granular info about what changed: yes > *I like to refer to the Gnome Tracker project (and probably also the KDE > Nepomuk > project), they manually traverse & spawn another inotify monitor for each & > every single dir in entire trees. They also claim that it doesn't hurt > performance too much either. It's the best you can currently get, there's > currently no other way of doing it. > > MacOSX > ------ > API: Kqueue > Recursive: no, and cannot be achieved manually* > Granular info about what changed: yes > * Max 256 file descriptors anyway, so out of question for recursive purposes. > ------ > API: FSEvents > Recursive: yes > Granular info about what changed: no, but you are notified about exactly where > the change happened (leaf-most directory, direct parent of where change > happened) so at most you need to manually poll contents of just 1 directory. > Also multiple changes are compounded into 1 event, configurable. > > MUST read: <http://arstechnica.com/apple/2007/10/mac-os-x-10-5/7/> > > > #### 2. Problems with making QFileSystemWatcher recursive > > - Windows ReadDirectoryCHangesW is the winner here. Would be easiest to > implement, we could even drop all the threading currently there now. > - inotify gives us all the chills at the thought of manually generating > watches > recursively for entire directory trees, but it can be done satisfactorially > enough behind the scenes inside QFileSystemWatcher. > - Mac OS is the troublemaker here; > Kqueue is out of question, as we established. > With FSEvents you cannot emit signals such as fileChanged(), fileModified(), > pathCreated(), or pathDeleted() if all you're notified about is that > "something > happened in this folder here", you MUST have some sort of snapshot of the > folder > to compare against, and QFileSystemWatcher generating an in-memory snapshot of > the user's requested directories to be watched can be crazy time consuming and > memory draining. Try doing this with an entire drive of some 40,000 folders > and > 300K files. > > > #### 3. What are the use cases anyway? > > Reevaluate: what is file-system watching good for anyway? > > There are two general use-case categories I can think of: > A) Your everyday lightweight necessities, like "has the config file changed". > QFileSystemWarcher is just fine for this with inotify & kqueue (though Windows > can still use some ReadDirectoryChangesW love instead of the current use of > Find*ChangeNotification APIs). > B) Heavy duty monitoring of entire hierachies, like monitoring dir trees in an > image library, backup applications, file crawling / indexing, virus scanning > etc. > > If you think about it, I suspect all B-style applications can be expected to > have their own snapshot of the file system stored anyway. Any app that needs > to > monitor changes to THAT much stuff probably has it's own view of the file > system that it just wants to keep updated. If this assumption is acceptable, > we discover that granular info about what exactly changed is (useful but) not > too urgent after all. So maybe FSEvents on mac isn't so bad. The application > will only need to compare 1 folder contents with it's own snapshot to > determine > what changed. > > So now what; > If we care to offer recursive FS monitoring in Qt, we can either try to stuff > it > into QFileSystemWatcher, or make a new class for it (perhaps platform specific > API classes if necessary). > > > #### 4. The first option: Enhanced QFileSystemWatcher > > If we do it this way, we'll one single awesome class that does it all. We just > may have to make peace with varying usage/behavior on different platforms. If > this is acceptable, we're in business. > > For example, here are some things we might need to mention in the docs; > > QFileSystemWatcher::fileModified() > NOTE: This signal is not emitted on Mac OS X. Instead, directoryChanged() is > emitted and the user is expected to investigate further about what exactly > changed. > > (Believe me I tried developing a solution where even on Mac with FSEvents > QFileSystemWatcher would do this "investigation" behind the scenes. I failed). > > QFileSystemWatcher::addPathRecursively() > NOTE: this can take some time on Linux/X11 as inotify monitors are manually > spawned recursively for each directory in the entire tree under the specified > path. Also, very excessive amounts of inotify monitors may eventually cause > system instability (slowing down). > > > #### 5. Second option: QRecursiveFileSystemWatcher > > Let the current QFileSystemWatcher remain the simple everyday API for that "A" > category of lightweight use cases. Let it even use Kqueue on Mac OS, it's > better > for that. > > And then, let's make a new Qt class for those "B" use cases where recursive is > the way to go. This class (perhaps named QRecursiveFileSystemWatcher?) will > have > platform specific features/functionality from the very start, all of it's > notes > & warnings clearly documented. There can be common functions that work the > same > on all platforms, and then there can be those few functions or parameters to > functions or return values from functions etc which are maybe ingonred on this > or that platform, or they're never used on the other platform, or return an > error on some platforms, etc. > > #### 6. Conclusion > > Please share your thoughts / opinions / comments, how should this best be done > so that we can have the awesome feature of recursive file-system monitoring in > Qt and still maintain a solid library API that works as best and as > consistently > as possible on each different platform. > > Thanks for your time > -regedit _______________________________________________ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
