[Development] QFileSystemWatcher and Recursive Monitoring

[logic.cpp]

BH

Hello,

Hope you don't mind my less-waste-your-time more-get-to-the-point English.
Begin ramble:

Wanted to fix QFileSystemWatcher for Windows as per w00t's article:

(also see his fix:  )

Realized that what I really needed was *recursive* monitoring, entire trees,
which the WinAPI function "ReadDirectoryChangesW"

that was mentioned in w00t's article offers right out of the box.

Others in #qt-labs also believe recursive file-system monitoring to be a useful
feature to offer in Qt.

Here is my research on the topic:


 1. Status of file-system monitoring on all major platforms.

Windows
---
API: ReadDirectoryChangesW
Recursive: yes
Granular info about what changed: yes

Linux/X11
-
API: inotify
Recursive: no, but can be achieved manually*
Granular info about what changed: yes
*I like to refer to the Gnome Tracker project (and probably also the KDE Nepomuk
project), they manually traverse & spawn another inotify monitor for each &
every single dir in entire trees. They also claim that it doesn't hurt
performance too much either. It's the best you can currently get, there's
currently no other way of doing it.

MacOSX
--
API: Kqueue
Recursive: no, and cannot be achieved manually*
Granular info about what changed: yes
* Max 256 file descriptors anyway, so out of question for recursive purposes.
--
API: FSEvents
Recursive: yes
Granular info about what changed: no, but you are notified about exactly where
the change happened (leaf-most directory, direct parent of where change
happened) so at most you need to manually poll contents of just 1 directory.
Also multiple changes are compounded into 1 event, configurable.

MUST read: 


 2. Problems with making QFileSystemWatcher recursive

- Windows ReadDirectoryCHangesW is the winner here. Would be easiest to
implement, we could even drop all the threading currently there now.
- inotify gives us all the chills at the thought of manually generating watches
recursively for entire directory trees, but it can be done satisfactorially
enough behind the scenes inside QFileSystemWatcher.
- Mac OS is the troublemaker here;
Kqueue is out of question, as we established.
With FSEvents you cannot emit signals such as fileChanged(), fileModified(),
pathCreated(), or pathDeleted() if all you're notified about is that "something
happened in this folder here", you MUST have some sort of snapshot of the folder
to compare against, and QFileSystemWatcher generating an in-memory snapshot of
the user's requested directories to be watched can be crazy time consuming and
memory draining. Try doing this with an entire drive of some 40,000 folders and
300K files.


 3. What are the use cases anyway?

Reevaluate: what is file-system watching good for anyway?

There are two general use-case categories I can think of:
A) Your everyday lightweight necessities, like "has the config file changed".
QFileSystemWarcher is just fine for this with inotify & kqueue (though Windows
can still use some ReadDirectoryChangesW love instead of the current use of
Find*ChangeNotification APIs).
B) Heavy duty monitoring of entire hierachies, like monitoring dir trees in an
image library, backup applications, file crawling / indexing, virus scanning
etc.

If you think about it, I suspect all B-style applications can be expected to
have their own snapshot of the file system stored anyway. Any app that needs to
monitor changes to THAT much stuff probably has it's own view of the file
system that it just wants to keep updated. If this assumption is acceptable,
we discover that granular info about what exactly changed is (useful but) not
too urgent after all. So maybe FSEvents on mac isn't so bad. The application
will only need to compare 1 folder contents with it's own snapshot to determine
what changed.

So now what;
If we care to offer recursive FS monitoring in Qt, we can either try to stuff it
into QFileSystemWatcher, or make a new class for it (perhaps platform specific
API classes if necessary).


 4. The first option: Enhanced QFileSystemWatcher

If we do it this way, we'll one single awesome class that does it all. We just
may have to make peace with varying usage/behavior on different platforms. If
this is acceptable, we're in business.

For example, here are some things we might need to mention in the docs;

QFileSystemWatcher::fileModified()
NOTE: This signal is not emitted on Mac OS X. Instead, directoryChanged() is
emitted and the user is expected to investigate further about what exactly
changed.

(Believe me I tried developing a solution where even on Mac with FSEvents
QFileSystemWatcher would do this "investigation" behind the scenes. I failed).

QFileSystemWatcher::addPathRecursively()
NOTE: this can

Re: [Development] QFileSystemWatcher and Recursive Monitoring

[logic.cpp]

#1
Clarification of original post:

When I say "Making QFileSystemWatcher Recursive" I mean *adding* new functions
for recursive monitoring, such as "addPathRecursively()". I don't mean to
replace the current non-recursive functinality, which is very useful in its own
right (for "A" category use cases mentioned).
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QFileSystemWatcher and Recursive Monitoring (Arvid E. Picciani)

[logic.cpp]

BH

Hey'all, Thanks for the feedback.

I think we can still make everything work inside one single class, either
QFileSystemWatcher or perhaps a new class if we don't want to break QFSW too
much. We'll do this à la "Graceful Degradation".

Before I explain, consider also the fact that - as Arvid mentioned - QFSW itself
doesn't (yet) offer fully granular notifications as to what precisely changed.
All we have is fileChanged(), directoryChanged(), fileModified(), pathCreated()
and pathDeleted() (the last three are w00t's patch in review
 ). I propose that Qt should
offer more awesomely useful notifications about file-system changes. This can be
acheived either by creating even moar signals, or perhaps in a different way -
as will be explained.

Here's how it'll work;

Windows:
With ReadDirectoryChangesW we have single-item monitoring, recursive
monitoring, and we can get this cool notification "Renamed from" + 
"Renamed
to" which QFSW doesn't offer ATM.

Linux/X11:
With inotify we have single-item monitoring, recursive monitoring 
(gulp),
and we have even awesomer signals than windows, like "Moved from" + 
"Moved
to".

Mac OS X:
We'll use kqueue for single-item monitoring (addPath() for "A"-category 
use
cases) and we'll use FSEvents for recursive monitoring 
(addPathRecursive()
for "B" use cases).

>From what I gather, in Qt we don't like to have actual functions/signals that
behave differently (or are altogether ignored) on different platforms. So
instead of having many different signals in the file-system monitor class which
won't even be emitted on some platforms (Mac), how about we emit just ONE signal
that will pass to the user 1) a path string and 2) an enum flag about what excly
happened (akin to inotify events enum  ).
This ensures consistent behavior cross all platforms, there's always a path and
enum passed with the signal. The only difference will be in the granularity of
info that the path/enum-flag will have, offering the most info available on the
platform "gracefully degrading" as needed.

So for example, for a single file watch on "foo/bar/baz.txt":

Windows: "C:\foo\bar\baz.txt" QFSW_DELETED
Linux/X11: "/foo/bar/baz.txt" QFSW_DELETED
Mac (kqueue): "/foo/bar/baz.txt" QFSW_DELETED <- no degradation

And for a recursive watch on the "foo/" directory:

Windows: "C:\foo\bar\baz.txt" QFSW_DELETED
Linux/X11: "/foo/bar/baz.txt" QFSW_DELETED
Mac (FSEvents): "/foo/bar/" QFSW_DIR_CHANGED <- graceful degradation

Notice how on Mac we get notified on the exact leaf-most folder where the event
happened, and that the event is always just "something changed here".

This will all be very clearly documented in detail, so users will know that on
Mac they need to do further investigations when using recursive. They'll also
learn about the limits of single-item monitoring (256 kqueue file descriptors).
Alternatively we can expose a capabilities() function which can dynamically tell
the user about the API's platform limitations (granularity of notification info,
file-descriptor count limitations on Mac, etc).

Bottom line: Doing it like this will successfully exploit the capabilities of
all platforms' file-system monitoring facilities to their max, and will safely
cover most if not all use case scenarios.

A.k.a. what Qt is all about :)

-regedit


P.S. I was made aware by a smart fellow in #qt-labs that FSEvents began offering
file-level notifications since OS X Lion+. Seeing as Qt must support older
versions, the idea of a capabilities() function sounds more interesting here (so
Lion+ can already begin taking advantage of file-level events).
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QFileSystemWatcher and Recursive Monitoring (Arvid E. Picciani)

[logic.cpp]

Whoops I am so sorry, total noob here when it comes to mailing lists.
This was meant to be a reply to the existing thread
http://lists.qt-project.org/pipermail/development/2012-July/005279.html
The last post of which was by Arvid E. Picciani

Not sure if this means the discussion continues in this thread from
now on, or whether we resume the previous thread...

Someone halp please!
-regedit
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QFileSystemWatcher and Recursive Monitoring

[logic.cpp]

BH

Hey'all, Thanks for the feedback.

I think we can still make everything work inside one single class, either
QFileSystemWatcher or perhaps a new class if we don't want to break QFSW too
much. We'll do this à la "Graceful Degradation".

Before I explain, consider also the fact that - as Arvid mentioned - QFSW itself
doesn't (yet) offer fully granular notifications as to what precisely changed.
All we have is fileChanged(), directoryChanged(), fileModified(), pathCreated()
and pathDeleted() (the last three are w00t's patch in review
 ). I propose that Qt should
offer more awesomely useful notifications about file-system changes. This can be
acheived either by creating even moar signals, or perhaps in a different way -
as will be explained.

Here's how it'll work;

Windows:
With ReadDirectoryChangesW we have single-item monitoring, recursive
monitoring, and we can get this cool notification "Renamed
from" + "Renamed
to" which QFSW doesn't offer ATM.

Linux/X11:
With inotify we have single-item monitoring, recursive
monitoring (gulp),
and we have even awesomer signals than windows, like "Moved
from" + "Moved
to".

Mac OS X:
We'll use kqueue for single-item monitoring (addPath() for
"A"-category use
cases) and we'll use FSEvents for recursive monitoring
(addPathRecursive()
for "B" use cases).

>From what I gather, in Qt we don't like to have actual functions/signals that
behave differently (or are altogether ignored) on different platforms. So
instead of having many different signals in the file-system monitor class which
won't even be emitted on some platforms (Mac), how about we emit just ONE signal
that will pass to the user 1) a path string and 2) an enum flag about what excly
happened (akin to inotify events enum  ).
This ensures consistent behavior cross all platforms, there's always a path and
enum passed with the signal. The only difference will be in the granularity of
info that the path/enum-flag will have, offering the most info available on the
platform "gracefully degrading" as needed.

So for example, for a single file watch on "foo/bar/baz.txt":

Windows: "C:\foo\bar\baz.txt" QFSW_DELETED
Linux/X11: "/foo/bar/baz.txt" QFSW_DELETED
Mac (kqueue): "/foo/bar/baz.txt" QFSW_DELETED <- no degradation

And for a recursive watch on the "foo/" directory:

Windows: "C:\foo\bar\baz.txt" QFSW_DELETED
Linux/X11: "/foo/bar/baz.txt" QFSW_DELETED
Mac (FSEvents): "/foo/bar/" QFSW_DIR_CHANGED <- graceful degradation

Notice how on Mac we get notified on the exact leaf-most folder where the event
happened, and that the event is always just "something changed here".

This will all be very clearly documented in detail, so users will know that on
Mac they need to do further investigations when using recursive. They'll also
learn about the limits of single-item monitoring (256 kqueue file descriptors).
Alternatively we can expose a capabilities() function which can dynamically tell
the user about the API's platform limitations (granularity of notification info,
file-descriptor count limitations on Mac, etc).

Bottom line: Doing it like this will successfully exploit the capabilities of
all platforms' file-system monitoring facilities to their max, and will safely
cover most if not all use case scenarios.

A.k.a. what Qt is all about :)

-regedit


P.S. I was made aware by a smart fellow in #qt-labs that FSEvents began offering
file-level notifications since OS X Lion+. Seeing as Qt must support older
versions, the idea of a capabilities() function sounds more interesting here (so
Lion+ can already begin taking advantage of file-level events).
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QFileSystemWatcher and Recursive Monitoring

[logic.cpp]

Darn, second attempt to fix my mess-up here in the mailing list only
made things worse.
Shoot me.
-regedit

On Fri, Jul 20, 2012 at 1:21 AM, logic.cpp  wrote:
> BH
>
> Hello,
>
> Hope you don't mind my less-waste-your-time more-get-to-the-point English.
> Begin ramble:
>
> Wanted to fix QFileSystemWatcher for Windows as per w00t's article:
> <http://blog.rburchell.com/2012/03/qt-51-aka-when-qfilesystemwatcher-might.html>
> (also see his fix: <https://codereview.qt-project.org/#change,19274> )
>
> Realized that what I really needed was *recursive* monitoring, entire trees,
> which the WinAPI function "ReadDirectoryChangesW"
> <http://msdn.microsoft.com/en-us/library/windows/desktop/aa365465(v=vs.85).aspx>
> that was mentioned in w00t's article offers right out of the box.
>
> Others in #qt-labs also believe recursive file-system monitoring to be a 
> useful
> feature to offer in Qt.
>
> Here is my research on the topic:
>
>
>  1. Status of file-system monitoring on all major platforms.
>
> Windows
> ---
> API: ReadDirectoryChangesW
> Recursive: yes
> Granular info about what changed: yes
>
> Linux/X11
> -
> API: inotify
> Recursive: no, but can be achieved manually*
> Granular info about what changed: yes
> *I like to refer to the Gnome Tracker project (and probably also the KDE 
> Nepomuk
> project), they manually traverse & spawn another inotify monitor for each &
> every single dir in entire trees. They also claim that it doesn't hurt
> performance too much either. It's the best you can currently get, there's
> currently no other way of doing it.
>
> MacOSX
> --
> API: Kqueue
> Recursive: no, and cannot be achieved manually*
> Granular info about what changed: yes
> * Max 256 file descriptors anyway, so out of question for recursive purposes.
> --
> API: FSEvents
> Recursive: yes
> Granular info about what changed: no, but you are notified about exactly where
> the change happened (leaf-most directory, direct parent of where change
> happened) so at most you need to manually poll contents of just 1 directory.
> Also multiple changes are compounded into 1 event, configurable.
>
> MUST read: <http://arstechnica.com/apple/2007/10/mac-os-x-10-5/7/>
>
>
>  2. Problems with making QFileSystemWatcher recursive
>
> - Windows ReadDirectoryCHangesW is the winner here. Would be easiest to
> implement, we could even drop all the threading currently there now.
> - inotify gives us all the chills at the thought of manually generating 
> watches
> recursively for entire directory trees, but it can be done satisfactorially
> enough behind the scenes inside QFileSystemWatcher.
> - Mac OS is the troublemaker here;
> Kqueue is out of question, as we established.
> With FSEvents you cannot emit signals such as fileChanged(), fileModified(),
> pathCreated(), or pathDeleted() if all you're notified about is that 
> "something
> happened in this folder here", you MUST have some sort of snapshot of the 
> folder
> to compare against, and QFileSystemWatcher generating an in-memory snapshot of
> the user's requested directories to be watched can be crazy time consuming and
> memory draining. Try doing this with an entire drive of some 40,000 folders 
> and
> 300K files.
>
>
>  3. What are the use cases anyway?
>
> Reevaluate: what is file-system watching good for anyway?
>
> There are two general use-case categories I can think of:
> A) Your everyday lightweight necessities, like "has the config file changed".
> QFileSystemWarcher is just fine for this with inotify & kqueue (though Windows
> can still use some ReadDirectoryChangesW love instead of the current use of
> Find*ChangeNotification APIs).
> B) Heavy duty monitoring of entire hierachies, like monitoring dir trees in an
> image library, backup applications, file crawling / indexing, virus scanning
> etc.
>
> If you think about it, I suspect all B-style applications can be expected to
> have their own snapshot of the file system stored anyway. Any app that needs 
> to
> monitor changes to THAT much stuff probably has it's own view of the file
> system that it just wants to keep updated. If this assumption is acceptable,
> we discover that granular info about what exactly changed is (useful but) not
> too urgent after all. So maybe FSEvents on mac isn't so bad. The application
> will only need to compare 1 folder contents with it's own snapshot to 
> determine
> what changed.
>
> So now what;
> If we care to offer recursive FS monitoring in Qt, we can either try to stuff 
> it
> into QFileSystemWat

Re: [Development] QFileSystemWatcher and Recursive Monitoring

[logic.cpp]

> Are you sure about that?  The documentation specifies an
> kFSEventStreamEventFlagMustScanSubDirs flag for
> events which indicates that a problem happened either in the kernel or
> user space and events were coalesced.
>

Events being coalesced is another unrelated thing (a feature in fact - not an
issue). The 'kFSEventStreamEventFlagMustScanSubDirs' thing is an important event
that lets the user know that "Whoa- too much has happened and buffers have
overflowed, assume that some events are not recorded/missing so it's time for
you to to a bit of manual scanning." The same is available on Windows RDCW and
in inotify. Indeed this is yet another file-system event we should offer in Qt
that QFSW does not currently account for.
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QFileSystemWatcher and Recursive Monitoring

[logic.cpp]

> How will you handle renames?  It looks like ReadDirectoryChanges uses
> two separate events, leaving it up to the user to match them up.
>

As for renames (and moves on Linux/X11) we'd need to come up with some mechanism
to sebd 2 pieces of information in one signal. This can be either quick-n-dirty
passing a concatenated string (semicolon separated or whatever) as the path
value, or getting fancy with passing pointers to structures that may have
several pieces of info packed in depending on whether it's a regular event or a
rename/move event.

Or I dunno, any other ideas? It's reallys just a technicality that can
definitely be figured out easily.
-regedit
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QFileSystemWatcher and Recursive Monitoring

[logic.cpp]

> I don't like your single signal design. It's cross platform only in that it
> will compile/function on every platform. The application will behave
> differently in the slot connected to said signal on mac than on
> windows/linux. We should not expect every _user_ app to have separate
> codepaths for different platforms. A.k.a exactly what Qt isn't about.
>

What else do you suggest then, should we make 3 separate classes one for each
OS?

> Just because you attempted the perfect solution once and failed, doesn't
> mean it can't be done. At the end of the day we just need some #ifdef
> Q_MAC_OSX (or maybe this can/should be in QPA) with boring file property
> (filename,filesize,maybe-crc32?,etc) comparison stuff before determing the
> proper signal to emit.
>

By all means, please do try to generate this in-memory snapshot of a sizable
directory tree (say 20,000 folders 150,000 files) with all the necessary info
you listed (filename,filesize etc.) and tell me how it goes.
Last I tried it on my own hard drive full of stuff (40k folders 300k files) it
1) consumed too huge an amount of memory, and 2) took too long to complete.

***

What I'm trying to achieve here is to have some kind of common cross platform
interface for file-system monitoring (in contrast to, for example, making 3
separate classes QWinWatcher QNixWatcher QMacWatcher which will force users to
have platform specific code) that works as similarly as possible on all OSs and
gracefully degrades as necessary. Users will be made aware of this degradation
and will know to check for it and program their code to cover the different
eventualities (like with a capabilities() function or some such).

In fact - as was noted in a previous post - even on Mac OS itself there can be
different features available depending on which version (Snow Leopard(-) vs.
Lion(+)) and not just any trivial feature difference, it's a huge & important
one; getting actual file-level details on what exactly changed, eliminating the
need for an existing snapshot to compare against! Will Qt not take advantage of
these features where available? Maybe not, I don't know... but if yes, it would
only make sense to have some sort of capabilities() function or some-such which
can dynamically inform users about what features are available and what isn't.

Anyway, that's just one idea of mine. Dunno if it works into Qt's way of doing
things. Please let me know. The questions remain as follows;
- Should it all be in one class?
If yes, how to go about differences in features/functionality.
Should we emit many different kinds of signals, or just 1 general signal.
Should we do this all in QFileSystemWatcher, or another new class.
- Or should fs-watching exist in totally separate platform-specific classes?
- Other?

-regedit
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QFileSystemWatcher and Recursive Monitoring

[logic.cpp]

So - are you suggesting that Qt file-system monitoring facilities
should actually generate an in-memory snapshot of the necessary
directory tree(s) for platforms that need it (in this case Mac OSX <=
Snow Leopard) ?

Hey I'm not against, it would be really easy to implement. But you
should really try testing something like this yourself on your
computer first. It's /really/ memory consuming as well as /really/
time consuming for large trees of stuff, fairly common nowadays in
anyone's photo/music/video/doc etc. collection. Sure we can alleviate
the time-consuming part with a worker thread, but I'm telling you this
is quite a hunk of resources you're signing off Qt to consume for a
possibly large percentage of use cases (many many people out there
have <= Snow Leopard).

BTW d3fault are you on IRC? What's your handle & timezone, maybe we
can chat about this more.

-regedit

P.S. Just on a personal note, the reason I got caught up in all this
to begin with was some application I wanted to develop. My particular
project involves a snapshot of the file system anyway, so in my
particular case on MacOSX<=Snow Leopard I would definitely just obtain
directory-level information from FSEvents and proceed to compare the
directory's contents with that of my existing snapshot.
In fact if Qt is gonna do a whole in-memory snapshot as d3fault
suggests for this situation, I'm considering dropping this whole QFSW
campaign and just roll out my own solution :)
How about we offer a choice to the programmer whether they want Qt to
do the internal snapshot (perhaps the default) or not?
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QFileSystemWatcher and Recursive Monitoring

[logic.cpp]

So how about something like this (others have suggested something along these 
lines):

In order to work recursively, the file-system watcher class will require the 
user to implement a sort of snapshotting "backend" which will be 
interfaced/used for FS change comparisons. And only as a "bonus", the class 
will also offer a default such backend on applicable OSs (i.e. all except 
Mac<=Snow Leopard, which in fact don't even need any snapshotting at all).

I think this wins because first off, it makes things nice & easy for 
applications that DO involve FS snapshotting /anyway/ (which is possibly 70% of 
all use-cases that need recursive monitoring in the first place) you could 
simply interface your application's snapshotting functionalities straight into 
Qt's filesystem monitor! This is interesting for ALL OSs even those that 
already offer granular file change details.
And at the same time, it's very clear to users of this class that recursive 
monitoring must have snapshotting - unless a default is kindly provided by Qt 
BUT not on all OSs.

So in other words the question is:
Is it acceptable for Qt to say "Hey look; to use my QFoo facilities, you must 
implement  on your end and plug it in to QFoo. Y'know - most use-cases 
QFoo was designed for have implemented a  of their own anyway. But just 
because I'm so nice, I'll offer you a default  implementation wherever I 
can" ?
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QFileSystemWatcher and Recursive Monitoring

[logic.cpp]

BH

I think this is very much like what I was trying to describe.

It's really annoying that Mac<=Snow Leopard is forcing us into such corners, 
but if it works and it pleases all Qt evangelists :) then maybe this'll do.
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QFileSystemWatcher and Recursive Monitoring

[logic.cpp]

BH

Alright I'll begin trying to implement some of this, possibly inside
the QFileSystemWatcher class itself.

But don't expect much, this is opensauce after all so I might come
back in a few months and yawn "Oh right, QFileSystemWatcher... forgot
about that" :D

Though if some volunteers can help implement the Linux/X11 inotify and
MacOSX FSEvents/Kqueue parts, that would help motivate a little (and
get everything coded sooner) since I'm mainly familiar with the
Windows ReadDirectoryChangesW end.

Cheers

-regedit
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QFileSystemWatcher and Recursive Monitoring

[logic.cpp]

Hah 13 years later 😆
QFSW should probably just do what this codebase does:
https://github.com/emcrisostomo/fswatch
-- 
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development