[Development] QFileSystemWatcher and Recursive Monitoring

[logic.cpp]

Darn, second attempt to fix my mess-up here in the mailing list only
made things worse.
Shoot me.
-regedit

On Fri, Jul 20, 2012 at 1:21 AM, logic.cpp <logic.cpp at gmail.com> wrote:
> BH
>
> Hello,
>
> Hope you don't mind my less-waste-your-time more-get-to-the-point English.
> Begin ramble:
>
> Wanted to fix QFileSystemWatcher for Windows as per w00t's article:
> <http://blog.rburchell.com/2012/03/qt-51-aka-when-qfilesystemwatcher-might.html>
> (also see his fix: <https://codereview.qt-project.org/#change,19274> )
>
> Realized that what I really needed was *recursive* monitoring, entire trees,
> which the WinAPI function "ReadDirectoryChangesW"
> <http://msdn.microsoft.com/en-us/library/windows/desktop/aa365465(v=vs.85).aspx>
> that was mentioned in w00t's article offers right out of the box.
>
> Others in #qt-labs also believe recursive file-system monitoring to be a useful
> feature to offer in Qt.
>
> Here is my research on the topic:
>
>
> #### 1. Status of file-system monitoring on all major platforms.
>
> Windows
> -------
> API: ReadDirectoryChangesW
> Recursive: yes
> Granular info about what changed: yes
>
> Linux/X11
> ---------
> API: inotify
> Recursive: no, but can be achieved manually*
> Granular info about what changed: yes
> *I like to refer to the Gnome Tracker project (and probably also the KDE Nepomuk
> project), they manually traverse & spawn another inotify monitor for each &
> every single dir in entire trees. They also claim that it doesn't hurt
> performance too much either. It's the best you can currently get, there's
> currently no other way of doing it.
>
> MacOSX
> ------
> API: Kqueue
> Recursive: no, and cannot be achieved manually*
> Granular info about what changed: yes
> * Max 256 file descriptors anyway, so out of question for recursive purposes.
> ------
> API: FSEvents
> Recursive: yes
> Granular info about what changed: no, but you are notified about exactly where
> the change happened (leaf-most directory, direct parent of where change
> happened) so at most you need to manually poll contents of just 1 directory.
> Also multiple changes are compounded into 1 event, configurable.
>
> MUST read: <http://arstechnica.com/apple/2007/10/mac-os-x-10-5/7/>
>
>
> #### 2. Problems with making QFileSystemWatcher recursive
>
> - Windows ReadDirectoryCHangesW is the winner here. Would be easiest to
> implement, we could even drop all the threading currently there now.
> - inotify gives us all the chills at the thought of manually generating watches
> recursively for entire directory trees, but it can be done satisfactorially
> enough behind the scenes inside QFileSystemWatcher.
> - Mac OS is the troublemaker here;
> Kqueue is out of question, as we established.
> With FSEvents you cannot emit signals such as fileChanged(), fileModified(),
> pathCreated(), or pathDeleted() if all you're notified about is that "something
> happened in this folder here", you MUST have some sort of snapshot of the folder
> to compare against, and QFileSystemWatcher generating an in-memory snapshot of
> the user's requested directories to be watched can be crazy time consuming and
> memory draining. Try doing this with an entire drive of some 40,000 folders and
> 300K files.
>
>
> #### 3. What are the use cases anyway?
>
> Reevaluate: what is file-system watching good for anyway?
>
> There are two general use-case categories I can think of:
> A) Your everyday lightweight necessities, like "has the config file changed".
> QFileSystemWarcher is just fine for this with inotify & kqueue (though Windows
> can still use some ReadDirectoryChangesW love instead of the current use of
> Find*ChangeNotification APIs).
> B) Heavy duty monitoring of entire hierachies, like monitoring dir trees in an
> image library, backup applications, file crawling / indexing, virus scanning
> etc.
>
> If you think about it, I suspect all B-style applications can be expected to
> have their own snapshot of the file system stored anyway. Any app that needs to
> monitor changes to THAT much stuff probably has it's own view of the file
> system that it just wants to keep updated. If this assumption is acceptable,
> we discover that granular info about what exactly changed is (useful but) not
> too urgent after all. So maybe FSEvents on mac isn't so bad. The application
> will only need to compare 1 folder contents with it's own snapshot to determine
> what changed.
>
> So now what;
> If we care to offer recursive FS monitoring in Qt, we can either try to stuff it
> into QFileSystemWatcher, or make a new class for it (perhaps platform specific
> API classes if necessary).
>
>
> #### 4. The first option: Enhanced QFileSystemWatcher
>
> If we do it this way, we'll one single awesome class that does it all. We just
> may have to make peace with varying usage/behavior on different platforms. If
> this is acceptable, we're in business.
>
> For example, here are some things we might need to mention in the docs;
>
> QFileSystemWatcher::fileModified()
> NOTE: This signal is not emitted on Mac OS X. Instead, directoryChanged() is
> emitted and the user is expected to investigate further about what exactly
> changed.
>
> (Believe me I tried developing a solution where even on Mac with FSEvents
> QFileSystemWatcher would do this "investigation" behind the scenes. I failed).
>
> QFileSystemWatcher::addPathRecursively()
> NOTE: this can take some time on Linux/X11 as inotify monitors are manually
> spawned recursively for each directory in the entire tree under the specified
> path. Also, very excessive amounts of inotify monitors may eventually cause
> system instability (slowing down).
>
>
> #### 5. Second option: QRecursiveFileSystemWatcher
>
> Let the current QFileSystemWatcher remain the simple everyday API for that "A"
> category of lightweight use cases. Let it even use Kqueue on Mac OS, it's better
> for that.
>
> And then, let's make a new Qt class for those "B" use cases where recursive is
> the way to go. This class (perhaps named QRecursiveFileSystemWatcher?) will have
> platform specific features/functionality from the very start, all of it's notes
> & warnings clearly documented. There can be common functions that work the same
> on all platforms, and then there can be those few functions or parameters to
> functions or return values from functions etc which are maybe ingonred on this
> or that platform, or they're never used on the other platform, or return an
> error on some platforms, etc.
>
> #### 6. Conclusion
>
> Please share your thoughts / opinions / comments, how should this best be done
> so that we can have the awesome feature of recursive file-system monitoring in
> Qt and still maintain a solid library API that works as best and as consistently
> as possible on each different platform.
>
> Thanks for your time
> -regedit