[Qt-interest] How to design a "search and download" application in Qt with multiple threads in mind.

[Sven Grunewaldt]

Thank you both for your responses!

To Andre:
Yeah, I think you're right. I just thought about this whole thing way
too complicated. Your QtWebkit idea is actually quite nice and I will
keep it in mind, though I think it would be easier to maintain if I just
use some regex. I took some time and thought about the whole application
again... please see below.

To Martin:
I'm not sure why I wanted to have two download queues at all... I guess
my last experience with transferring hundreds of 1kb files over FTP was
still in my mind. What consumed the most time in this process was
connecting and so on, not the actual transfer. In my case I'll be
downloading images, so most of the time will be consumed by the transfer
and I will achieve nothing with two download processes at all. Thanks
for pointing that out ;) Normally I also try to keep it "nice" for the
servers.


Basically what I thought was: processing these HTML pages will take
quite some time so it is nice to have another thread which starts
downloading while the other one is processing the HTML pages.
And then my mind started going crazy and I wanted to put the actual
parsing of pages in an own thread. So I ended up with three threads at
minimum. One which downloads the HTML code, one which looks for images,
one which downloads these images. I guess I thought I could make use of
my quad-core when I have four parser threads...

I ditched completely. My current idea is this:
The main thread downloads the HTML pages and looks for images, these are
put into a download class (actually a thread safe QQueue).

This way I can start loading the already found images right away and
just block when the download queue is empty and unblock when something
new is put into the queue.

Regards,
Sven Grunewaldt

Am 13.09.2010 08:07, schrieb Martin Hofius:
> Am Sonntag, 12. September 2010 schrieb Andre Somers:
>>   Op 10-9-2010 17:24, Sven Grunewaldt schreef:
>>> Hi, I'm currently trying to plan an application that scans multiple web
>>> pages for images, puts them in a download queue and downloads these
>>> images.
>>>
>>> I'm struggling with the design of the classes and such. My initial idea
>>> was the following:
> ...
>> downloaded by a webpage wherever you like. Also, you should probably
>> ditch the idea of complicated queues and multiple threads. Using
>> multiple threads will not increase your network speed, nor will it
>> increase your disk IO speeds. *Those* will be your bottlenecks, not your
> you can archive multiple parallel transmissions by using non blocking calls - 
> without additional overhead for multiple threads. 
> I also don't think that disk IO is a real limitation for web downloads - 
> except you have a direkt connection to an internet backbone...
> 
> Even when many download accelerator programs use parallel transmissions to 
> increase download speed (to circumvent limitations of the server or to use 
> more than one server of a server farm) - please think about fair use of the 
> server and network capacities...
> 
> Martin
> _______________________________________________
> Qt-interest mailing list
> Qt-interest at trolltech.com
> http://lists.trolltech.com/mailman/listinfo/qt-interest