[Qt-interest] Multiple page rendering using webkit

[Arnold Krille]

Hi,

On Friday 15 April 2011 21:33:07 Tarandeep Singh wrote:
> I need to render a large number of pages and obviously need do this as
> efficiently as possible.
> Since GUI rendering needs to be done on the main thread, so I can't create
> multiple QWebPage threads.
> 
> So I am thinking of these 2 solutions-
> 
> 1) Create one process and have N QWebPage instances so that N pages can be
> rendered simultaneously. Here assumption is QT+Webkit behind the scenes
> create threads to do parallel IO etc.
> 
> 2) Create N processes where each process handles 1 page request at a given
> time.
> 
> Can someone please advice which solution should work better or is there is
> another way to achieve this.
> I have 16 cores and 32G of RAM.

I assume you have the to-be-rendered webpages on disk already as html and 
"only" want the rendered output in pdf/svg/png or whatever. (If not I think 
you can easily do with three to four webpages which in turns load the files 
from network and render and save to disk, net-io will be the slowest point.)

As all gui-stuff has to run in the thread that instantiated QApplication, you 
can really only use one thread for rendering. But who says that you can't run 
multiple processes in parallel?
One process, maybe even the master of your app, loads the html from disk into 
a queue which is in shared memory and then starts as many render-processes as 
you like. These render processes each take a pages html from the queue and 
render it and either put the rendered data back onto another shared queue or 
save directly to files on disk.
If you want to leave room for other processes to run (like the desktop), you 
should run like min(1, idealThreadCount()/2) render processes.

Hope that helps,

Arnold
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
Url : http://lists.qt-project.org/pipermail/qt-interest-old/attachments/20110415/4ba234d6/attachment.bin