[Development] QFileSystemWatcher and Recursive Monitoring

[logic.cpp]

BH

Hello,

Hope you don't mind my less-waste-your-time more-get-to-the-point English.
Begin ramble:

Wanted to fix QFileSystemWatcher for Windows as per w00t's article:
<http://blog.rburchell.com/2012/03/qt-51-aka-when-qfilesystemwatcher-might.html>
(also see his fix: <https://codereview.qt-project.org/#change,19274> )

Realized that what I really needed was *recursive* monitoring, entire trees,
which the WinAPI function "ReadDirectoryChangesW"
<http://msdn.microsoft.com/en-us/library/windows/desktop/aa365465(v=vs.85).aspx>
that was mentioned in w00t's article offers right out of the box.

Others in #qt-labs also believe recursive file-system monitoring to be a useful
feature to offer in Qt.

Here is my research on the topic:


#### 1. Status of file-system monitoring on all major platforms.

Windows
-------
API: ReadDirectoryChangesW
Recursive: yes
Granular info about what changed: yes

Linux/X11
---------
API: inotify
Recursive: no, but can be achieved manually*
Granular info about what changed: yes
*I like to refer to the Gnome Tracker project (and probably also the KDE Nepomuk
project), they manually traverse & spawn another inotify monitor for each &
every single dir in entire trees. They also claim that it doesn't hurt
performance too much either. It's the best you can currently get, there's
currently no other way of doing it.

MacOSX
------
API: Kqueue
Recursive: no, and cannot be achieved manually*
Granular info about what changed: yes
* Max 256 file descriptors anyway, so out of question for recursive purposes.
------
API: FSEvents
Recursive: yes
Granular info about what changed: no, but you are notified about exactly where
the change happened (leaf-most directory, direct parent of where change
happened) so at most you need to manually poll contents of just 1 directory.
Also multiple changes are compounded into 1 event, configurable.

MUST read: <http://arstechnica.com/apple/2007/10/mac-os-x-10-5/7/>


#### 2. Problems with making QFileSystemWatcher recursive

- Windows ReadDirectoryCHangesW is the winner here. Would be easiest to
implement, we could even drop all the threading currently there now.
- inotify gives us all the chills at the thought of manually generating watches
recursively for entire directory trees, but it can be done satisfactorially
enough behind the scenes inside QFileSystemWatcher.
- Mac OS is the troublemaker here;
Kqueue is out of question, as we established.
With FSEvents you cannot emit signals such as fileChanged(), fileModified(),
pathCreated(), or pathDeleted() if all you're notified about is that "something
happened in this folder here", you MUST have some sort of snapshot of the folder
to compare against, and QFileSystemWatcher generating an in-memory snapshot of
the user's requested directories to be watched can be crazy time consuming and
memory draining. Try doing this with an entire drive of some 40,000 folders and
300K files.


#### 3. What are the use cases anyway?

Reevaluate: what is file-system watching good for anyway?

There are two general use-case categories I can think of:
A) Your everyday lightweight necessities, like "has the config file changed".
QFileSystemWarcher is just fine for this with inotify & kqueue (though Windows
can still use some ReadDirectoryChangesW love instead of the current use of
Find*ChangeNotification APIs).
B) Heavy duty monitoring of entire hierachies, like monitoring dir trees in an
image library, backup applications, file crawling / indexing, virus scanning
etc.

If you think about it, I suspect all B-style applications can be expected to
have their own snapshot of the file system stored anyway. Any app that needs to
monitor changes to THAT much stuff probably has it's own view of the file
system that it just wants to keep updated. If this assumption is acceptable,
we discover that granular info about what exactly changed is (useful but) not
too urgent after all. So maybe FSEvents on mac isn't so bad. The application
will only need to compare 1 folder contents with it's own snapshot to determine
what changed.

So now what;
If we care to offer recursive FS monitoring in Qt, we can either try to stuff it
into QFileSystemWatcher, or make a new class for it (perhaps platform specific
API classes if necessary).


#### 4. The first option: Enhanced QFileSystemWatcher

If we do it this way, we'll one single awesome class that does it all. We just
may have to make peace with varying usage/behavior on different platforms. If
this is acceptable, we're in business.

For example, here are some things we might need to mention in the docs;

QFileSystemWatcher::fileModified()
NOTE: This signal is not emitted on Mac OS X. Instead, directoryChanged() is
emitted and the user is expected to investigate further about what exactly
changed.

(Believe me I tried developing a solution where even on Mac with FSEvents
QFileSystemWatcher would do this "investigation" behind the scenes. I failed).

QFileSystemWatcher::addPathRecursively()
NOTE: this can take some time on Linux/X11 as inotify monitors are manually
spawned recursively for each directory in the entire tree under the specified
path. Also, very excessive amounts of inotify monitors may eventually cause
system instability (slowing down).


#### 5. Second option: QRecursiveFileSystemWatcher

Let the current QFileSystemWatcher remain the simple everyday API for that "A"
category of lightweight use cases. Let it even use Kqueue on Mac OS, it's better
for that.

And then, let's make a new Qt class for those "B" use cases where recursive is
the way to go. This class (perhaps named QRecursiveFileSystemWatcher?) will have
platform specific features/functionality from the very start, all of it's notes
& warnings clearly documented. There can be common functions that work the same
on all platforms, and then there can be those few functions or parameters to
functions or return values from functions etc which are maybe ingonred on this
or that platform, or they're never used on the other platform, or return an
error on some platforms, etc.

#### 6. Conclusion

Please share your thoughts / opinions / comments, how should this best be done
so that we can have the awesome feature of recursive file-system monitoring in
Qt and still maintain a solid library API that works as best and as consistently
as possible on each different platform.

Thanks for your time
-regedit