[Qt-interest] How to design a "search and download" application in Qt with multiple threads in mind.

[Andre Somers]

  Op 10-9-2010 17:24, Sven Grunewaldt schreef:
> Hi, I'm currently trying to plan an application that scans multiple web
> pages for images, puts them in a download queue and downloads these images.
>
> I'm struggling with the design of the classes and such. My initial idea
> was the following:
>
> The applications main method fetches the pages and puts the source code
> in a queue of some sort. This queue is processed by two threads which
> parse each source code for images and put these in another queue. THIS
> queue is processed by two threads which download the images.
>
> So basically there is the following scheme: One/two thread(s) find
> stuff, puts it into a queue, some other threads process this queue.
> Now the problem is - I don't know how to do that with the existing Qt
> classes. My hope was that I could just put a QQueue in the main class,
> enqueue stuff I find and let some other threads read the QQueue with
> parent()->queue->dequeue(), which I hoped to be blocking (which it is not).
>
> After I realized I would not accomplish much with my current knowledge I
> looked into my Qt book (the one by J. Blanchette and M. Summerfield) but
> couldn't find help there, so I have to bore you guys with my noobish
> question! :)
>
> Can anyone give me some pointers where I need to start? I'm really
> totally lost right now...
>
>
You need to think about what you want to accomplish exactly, and where 
in your process the bottlenecks will be. I think your design idea is way 
to complicated. Threads handling queues, parsing HTML yourself... what for?

Your first step is to simplify. Qt offers QtWebkit. Why not use it? If 
you use a subclassed QNetworkAccessManager, you can store the images 
downloaded by a webpage wherever you like. Also, you should probably 
ditch the idea of complicated queues and multiple threads. Using 
multiple threads will not increase your network speed, nor will it 
increase your disk IO speeds. *Those* will be your bottlenecks, not your 
CPU. You are not going to solve that using threads. Perhaps it would 
make sense to use webkit and its network IO from a separate thread, and 
the writing to disk as well so you can keep your UI responsive and 
easily make the disk IO asynchronic. However, I would first try without.

So... basic idea:
1) Subclass QNetworkAccessHandler to intercept the responses that 
contain images
2) Setup a QWebPage to use that new handler
3) Create a queue of URLs you want to get the images from
4) Feed the URLs one by one to your webpage object
5) Store the images that your network handler sees comming by

André


André