[Qt-interest] issues with QNetworkDiskCache and large number of cached files

[Bo Thorsen]

Den 05-03-2011 18:43, Crni Gorac skrev:
> We are using QNetworkDiskCache in an application with HTML/JavaScript
> user interface, displayed in QWebView based widgets.  The purpose of
> using QNetworkDiskCache is, of course, to speed-up application user
> interface; the application is of type so that large number of small
> images (like avatars etc.) appear to be used, so the cache directory
> on disk get quickly filled with thousands of small images.  We've
> noticed that this is affecting CPU usage, specifically on Windows -
> whenever some operation with the cache involved, disk activity seem to
> be shortly blocking main thread.  I remember facing alike problem,
> albeit with FAT file-system, on another project previously, and the
> approach that worked there was to split the given directory into
> number of subdirectories, and then to split these into another level
> of subdirectories, so that at the end only dozen or so files end up in
> each directory.  So I've looked into maybe sub-classing
> QNetworkDiskCache in order to implement such a scheme, but it appears
> this is not workable; I've also looked into just copying QNetworkCache
> implementation files into our project and then making needed changes,
> but this is not so easy either (pimpl idiom used, then
> QNetworkDiskCache private implementation class deriving from
> QAbstractNetworkCache private implementation class).  So I'm wondering
> - has anyone faced alike problem, or approached re-implementing
> QNetworkDiskCache?  Any suggestions here?  Or maybe it would be better
> to solve the problem some other way (I was thinking about somehow
> separating disk cache operations in dedicated thread, in order to keep
> main thread responsive, but it's not clear to me how that would be
> possible)?

I have had similar experiences with it and my solution have been to 
create a caching system that is detached from the loader - a completely 
seperate FileCache class.

Another solution is to subclass the QAbstractNetworkCache yourself 
instead of using QNetworkDiskCache. This way it can work in the 
background and you can do the split to subdirs.

In my opinion, QNetworkDiskCache shouldn't be used at the moment. It's 
the right idea, but the implementation sucks.

Bo Thorsen,
Fionia Software.

-- 

Expert Qt and C++ developer for hire
Contact me if you need expert Qt help
http://www.fioniasoftware.dk