[Development] QFileSystemWatcher and Recursive Monitoring

Robert Knight robertknight at gmail.com
Sun Jul 22 19:50:43 CEST 2012


> Notice how on Mac we get notified on the exact leaf-most folder where the event
> happened, and that the event is always just "something changed here".

Are you sure about that?  The documentation specifies an
kFSEventStreamEventFlagMustScanSubDirs flag for
events which indicates that a problem happened either in the kernel or
user space and events were coalesced.

> how about we emit just ONE signal
> that will pass to the user 1) a path string and 2) an enum flag about
> what exactly happened (akin to inotify events enum <http://en.wikipedia.org/wiki/Inotify> ).

Sounds sensible.  It looks as though this would map better to the
underlying APIs for FSEvents and ReadDirectoryChangesW.
How will you handle renames?  It looks like ReadDirectoryChanges uses
two separate events, leaving it up to the user to match them up.

> With inotify we have single-item monitoring, recursive monitoring (gulp),

But do document the issues with watch limits.  A related issue is that
scanning the file system in order to set up
the notifications on a sub-tree could take an arbitrary amount of time
with inotify.  You could potentially defer that work to a background
thread, or simply document the issue and leave it up to the app
developer - if the app is going to scan a whole dir tree
at some point, it will most likely not want to do that from the UI
thread anyway.

Regards,
Rob.

On 22 July 2012 11:15, logic.cpp <logic.cpp at gmail.com> wrote:
> Darn, second attempt to fix my mess-up here in the mailing list only
> made things worse.
> Shoot me.
> -regedit
>
> On Fri, Jul 20, 2012 at 1:21 AM, logic.cpp <logic.cpp at gmail.com> wrote:
>> BH
>>
>> Hello,
>>
>> Hope you don't mind my less-waste-your-time more-get-to-the-point English.
>> Begin ramble:
>>
>> Wanted to fix QFileSystemWatcher for Windows as per w00t's article:
>> <http://blog.rburchell.com/2012/03/qt-51-aka-when-qfilesystemwatcher-might.html>
>> (also see his fix: <https://codereview.qt-project.org/#change,19274> )
>>
>> Realized that what I really needed was *recursive* monitoring, entire trees,
>> which the WinAPI function "ReadDirectoryChangesW"
>> <http://msdn.microsoft.com/en-us/library/windows/desktop/aa365465(v=vs.85).aspx>
>> that was mentioned in w00t's article offers right out of the box.
>>
>> Others in #qt-labs also believe recursive file-system monitoring to be a useful
>> feature to offer in Qt.
>>
>> Here is my research on the topic:
>>
>>
>> #### 1. Status of file-system monitoring on all major platforms.
>>
>> Windows
>> -------
>> API: ReadDirectoryChangesW
>> Recursive: yes
>> Granular info about what changed: yes
>>
>> Linux/X11
>> ---------
>> API: inotify
>> Recursive: no, but can be achieved manually*
>> Granular info about what changed: yes
>> *I like to refer to the Gnome Tracker project (and probably also the KDE Nepomuk
>> project), they manually traverse & spawn another inotify monitor for each &
>> every single dir in entire trees. They also claim that it doesn't hurt
>> performance too much either. It's the best you can currently get, there's
>> currently no other way of doing it.
>>
>> MacOSX
>> ------
>> API: Kqueue
>> Recursive: no, and cannot be achieved manually*
>> Granular info about what changed: yes
>> * Max 256 file descriptors anyway, so out of question for recursive purposes.
>> ------
>> API: FSEvents
>> Recursive: yes
>> Granular info about what changed: no, but you are notified about exactly where
>> the change happened (leaf-most directory, direct parent of where change
>> happened) so at most you need to manually poll contents of just 1 directory.
>> Also multiple changes are compounded into 1 event, configurable.
>>
>> MUST read: <http://arstechnica.com/apple/2007/10/mac-os-x-10-5/7/>
>>
>>
>> #### 2. Problems with making QFileSystemWatcher recursive
>>
>> - Windows ReadDirectoryCHangesW is the winner here. Would be easiest to
>> implement, we could even drop all the threading currently there now.
>> - inotify gives us all the chills at the thought of manually generating watches
>> recursively for entire directory trees, but it can be done satisfactorially
>> enough behind the scenes inside QFileSystemWatcher.
>> - Mac OS is the troublemaker here;
>> Kqueue is out of question, as we established.
>> With FSEvents you cannot emit signals such as fileChanged(), fileModified(),
>> pathCreated(), or pathDeleted() if all you're notified about is that "something
>> happened in this folder here", you MUST have some sort of snapshot of the folder
>> to compare against, and QFileSystemWatcher generating an in-memory snapshot of
>> the user's requested directories to be watched can be crazy time consuming and
>> memory draining. Try doing this with an entire drive of some 40,000 folders and
>> 300K files.
>>
>>
>> #### 3. What are the use cases anyway?
>>
>> Reevaluate: what is file-system watching good for anyway?
>>
>> There are two general use-case categories I can think of:
>> A) Your everyday lightweight necessities, like "has the config file changed".
>> QFileSystemWarcher is just fine for this with inotify & kqueue (though Windows
>> can still use some ReadDirectoryChangesW love instead of the current use of
>> Find*ChangeNotification APIs).
>> B) Heavy duty monitoring of entire hierachies, like monitoring dir trees in an
>> image library, backup applications, file crawling / indexing, virus scanning
>> etc.
>>
>> If you think about it, I suspect all B-style applications can be expected to
>> have their own snapshot of the file system stored anyway. Any app that needs to
>> monitor changes to THAT much stuff probably has it's own view of the file
>> system that it just wants to keep updated. If this assumption is acceptable,
>> we discover that granular info about what exactly changed is (useful but) not
>> too urgent after all. So maybe FSEvents on mac isn't so bad. The application
>> will only need to compare 1 folder contents with it's own snapshot to determine
>> what changed.
>>
>> So now what;
>> If we care to offer recursive FS monitoring in Qt, we can either try to stuff it
>> into QFileSystemWatcher, or make a new class for it (perhaps platform specific
>> API classes if necessary).
>>
>>
>> #### 4. The first option: Enhanced QFileSystemWatcher
>>
>> If we do it this way, we'll one single awesome class that does it all. We just
>> may have to make peace with varying usage/behavior on different platforms. If
>> this is acceptable, we're in business.
>>
>> For example, here are some things we might need to mention in the docs;
>>
>> QFileSystemWatcher::fileModified()
>> NOTE: This signal is not emitted on Mac OS X. Instead, directoryChanged() is
>> emitted and the user is expected to investigate further about what exactly
>> changed.
>>
>> (Believe me I tried developing a solution where even on Mac with FSEvents
>> QFileSystemWatcher would do this "investigation" behind the scenes. I failed).
>>
>> QFileSystemWatcher::addPathRecursively()
>> NOTE: this can take some time on Linux/X11 as inotify monitors are manually
>> spawned recursively for each directory in the entire tree under the specified
>> path. Also, very excessive amounts of inotify monitors may eventually cause
>> system instability (slowing down).
>>
>>
>> #### 5. Second option: QRecursiveFileSystemWatcher
>>
>> Let the current QFileSystemWatcher remain the simple everyday API for that "A"
>> category of lightweight use cases. Let it even use Kqueue on Mac OS, it's better
>> for that.
>>
>> And then, let's make a new Qt class for those "B" use cases where recursive is
>> the way to go. This class (perhaps named QRecursiveFileSystemWatcher?) will have
>> platform specific features/functionality from the very start, all of it's notes
>> & warnings clearly documented. There can be common functions that work the same
>> on all platforms, and then there can be those few functions or parameters to
>> functions or return values from functions etc which are maybe ingonred on this
>> or that platform, or they're never used on the other platform, or return an
>> error on some platforms, etc.
>>
>> #### 6. Conclusion
>>
>> Please share your thoughts / opinions / comments, how should this best be done
>> so that we can have the awesome feature of recursive file-system monitoring in
>> Qt and still maintain a solid library API that works as best and as consistently
>> as possible on each different platform.
>>
>> Thanks for your time
>> -regedit
> _______________________________________________
> Development mailing list
> Development at qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development



More information about the Development mailing list