[Qt-interest] load url using QWebPage in a thread

[Jason H]

Since Qt uses non-blocking I/O, there's not a huge amount of performance gain, but if you are using a lot of JS and performance *IS* a problem, then I am afraid you are stuck.  The only way to work around that in Qt is to spawn multiple local rendering processes and have a separate proxy process connect to one of the many processes. Each render can still service multiple requests, but you would want to start N renderers where N=number of CPUs. This way, you can distribute load across all cores, while having a central socket.

You didn't listen... you can't pass QWebPage to a thread and have it work [safely].
You need to emit the HTML to the main thread for rendering. To do that, define a signal, and connect a main GUI thread handler to the thread event that is emitted.







________________________________
From: Tarandeep Singh <tarandeep at gmail.com>
To: Jason H <scorp1us at yahoo.com>
Cc: qt-interest <qt-interest at trolltech.com>
Sent: Tue, April 20, 2010 8:43:40 PM
Subject: Re: [Qt-interest] load url using QWebPage in a thread


On Tue, Apr 20, 2010 at 1:31 PM, Jason H <scorp1us at yahoo.com> wrote:

>
>What you need to do, simply, is just use QWebFrame::load() in your regular (GUI) thread, just like in the examples. 
>page.mainFrame()->load(url);
>
>

Thanks Jason.  I made the changes & got it working, but occasionally  the application crashes (segmentation fault). 

The reason I want to use threads is- I want to have a muti-threaded server that will receive request (url), load the url into QWebPage and then get its html (after QWebPage runs javascripts, css etc) and send the html back to the client.

I cut my application to do the bare minimum and I am pasting the code snippet here, Please see if you can find some obvious flaw.

/////////////// main.cpp //////////////
#include <QApplication>
#include "server.h"

int main(int argc, char *argv[])
{
  QApplication app(argc, argv);
  Server server;
  server.listen( QHostAddress( ipaddress), port);
  return app.exec();
}

////////////////// server.cpp ///////////////
#include "server.h"
#include "thread.h"

Server::Server(QObject *parent) : QTcpServer(parent) { }

void Server::incomingConnection(int socketDescriptor)
{
  QUrl url ( "some url to load");
  QWebPage *page = new QWebPage( );
  page->mainFrame()->load( url);
  Thread *thread = new Thread(page, socketDescriptor, this);
  connect(thread, SIGNAL(finished()), thread, SLOT(deleteLater()));
  thread->start();
}


//////////////////////// thread.cpp ///////////////////
#include "thread.h"

Thread::Thread(QWebPage *p, int sd, QObject *parent) : QThread(parent)
{
  page = p;
  socketDescriptor = sd;
}

Thread::~Thread( )
{
  delete page;
}

void static waitForSignal(QObject* obj, const char* signal)
{
  QEventLoop loop;
  QObject::connect( obj, signal, &loop, SLOT( quit( )));
  loop.exec( );
}

void Thread::run()
{
  waitForSignal( page, SIGNAL( loadFinished(bool)));
  QString html = page->mainFrame( )->toHtml( );
  // send html back to client
}
/////////////////////////////////////////////////////////////////////////////////


Thanks,
Tarandeep



>
>I am not sure why you need to use threads, but lets indulge and say you do (maybe you're trying to spread SSL load across cores or something?)  In that case you should just use QNetworkAccessManager  in a thread to get the document, then use QWebFrane::setHtml(),  passing the HTML retrieved by QN.A.M. The thread would emit a signal in response to replyFinished, , for example, loadedHmtl(QString html) and you'd connect it to a function that would the load it in the page as:
>>
> page.mainFrame()->setHtml(html);
>
>Rememeber the implicitly shared types (like QString) can be used in signals/slots across thread
> boundaries. Widgets aren't implicitly shared, so you cannot, in a QThread subclass do page->mainFrame()->load().
>
>
>
>
>>
>
>
________________________________
From: Tarandeep Singh <tarandeep at gmail.com>
>To: Jason H <scorp1us at yahoo.com>
>Cc: qt-interest <qt-interest at trolltech.com>
>Sent: Tue, April 20, 2010 3:54:06 PM
>Subject: Re: [Qt-interest] load url using QWebPage in a thread
>
>
>
>On Tue, Apr 20, 2010 at 12:40 PM, Jason H <scorp1us at yahoo.com> wrote:
>
>>>
>>
>>
>>Non-GUI threads (there is only one) cannot use GUI things. 
>>page->mainFrame( )->load( url);  isn't right
>>You need to collect the page and return it to the GUI thread. You can use QNetworkAccessManager to get the page in a thread. Also, you can use QNetworkAccessManager for multiple requets in one (the GUI)  thread, since it is non-blocking. The performance differences should be minimal.
>>
>>
>
>Thanks Jason for the reply.
>
>I didn't understand this clearly- "collect the page and return it to the GUI thread". I am aware this is a method- moveToThread( ), so I should call page->moveToThread( guiThread). But which is the GUI thread here? the main program control flow? How do I get its thread?
>
>Also, I checked QNetworkAccessManager api, I don't see any method that will give me Page object.
>
>Can you please point me to some code snippet so that I can understand this.
>
>Appreciate your help,
>>
>
>Tarandeep
>
>
>>>
>>
>>
>>
>>
>>
>>
>>
________________________________
 >>
>>
>>From: Tarandeep Singh <tarandeep at gmail.com>
>>To: qt-interest at trolltech.com
>>Sent: Tue, April 20, 2010 1:15:49 PM
>>Subject: [Qt-interest] load url using QWebPage in a thread
>>
>>
>>>>Hi,
>>
>>I am trying to build a multi threaded server that loads a
>>url in QWebPage in a separate thread. But I am not able to get it
>>working. I am creating the QWebPage in the run method of the thread. Is
>>this right way? How do I connect QWebPage's loadFinished signal to find
>>out the page has been loaded (so I can do further processing, in my
>>case, get the HTML of the loaded page)
>>
>>Here is the code snippet. 
>>
>>/////////// MyServer extends QTcpServer /////////////////
>>MyServer::MyServer(QObject *parent) : QTctServer( parent) { }
>>    : QTcpServer(parent)
>>{
>>}
>>
>>void MyServer::incomingConnection(
>>>>
>>
>>
>>
>>
>>int socketDescriptor)
>>>>{
>>    QString url( "http://www.dummyurl.com/");
>>  MyThread *thread = new MyThread( socketDescriptor, url, this);
>>  connect(thread, SIGNAL(finished()), thread, SLOT(deleteLater()));
>>>>
>>
>>
>>
>>
>>
>>  thread->start();
>>}
>>
>>
>>///////////// MyThread ////////////////////
>>MyThread::MyThread( int socketDescriptor, const QString &urlstr, QObject *parent)
>>    : QThread(parent), socketDescriptor(socketDescriptor), urlstring(urlstr)
>>>>
>>
>>
>>
>>
>>
>>{
>>}
>>
>>void MyThread::run()
>>{
>>    QTcpSocket tcpSocket;
>>
>>    if (!tcpSocket.setSocketDescriptor(socketDescriptor)) {
>>        emit error(tcpSocket.error());
>>        return;
>>    }
>>
>>        QUrl url( urlstring);
>>>>
>>
>>
>>
>>
>>
>>        QWebPage *page = new QWebPage( );
>>        page->mainFrame( )->load( url); 
>>        
>>        //// HOW TO WAIT TILL LOADING FINISHED
>>        //// GET THE html of the page
>>        QString html = page->mainFrame( )->toHtml( );
>>>>
>>
>>
>>
>>
>>
>>        
>>        //// OTHER STUFF
>>        
>>        tcpSocket.write( returnValue);
>>        tcpSocket.disconnectFromHost();
>>        tcpSocket.waitForDisconnected();
>>}
>>
>>//////////// main.cpp //////////////////
>>>>
>>
>>
>>
>>
>>
>>int main(int argc, char *argv[])
>>{
>>  QApplication app(argc, argv, false);
>>  QApplication::setGraphicsSystem( "raster");
>>  app.setStyle( new QCleanlooksStyle( ));
>>  MyServer server;
>>  server.listen( QHostAddress( ip), portNo))
>>>>
>>
>>
>>
>>
>>
>>  return app.exec();
>>}
>>
>>
>>Thanks,
>>Tarandeep
>>
>
>



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.qt-project.org/pipermail/qt-interest-old/attachments/20100420/e8f857b7/attachment.html