[Qt-interest] QHttp upload won't stream?

Tue Oct 6 19:37:21 CEST 2009

Hi all,

I started this thread, and just now getting back to it.  Apologies.

Cross-reference (pretty much same content):
http://www.qtcentre.org/forum/f-qt-programming-2/t-qhttp-file-upload-doesnt-stream-24404.html

I also confirmed that using QNetworkAccessManager makes no difference.  Also that the problem seems to have to do with SSL:  when I use HTTP, things stream fine.

Haven't gotten any further than that.

Hawkeye Parker

> Date: Tue, 6 Oct 2009 17:49:00 +0100
> From: Info <info at misafe.com>
> Subject: Re: [Qt-interest] QHttp upload won't stream?
> To: Jason H <scorp1us at yahoo.com>
> Cc: qt-interest at trolltech.com
>
> I've tracked it down to the fact that when
> QNativeSocketEnginePrivate::nativeWrite() calls WSASend() it returns
> SOCKET_ERROR and the reason is WSAEWOULDBLOCK, now this appears to be
> because under normal HTTP, only chunks of 4096 are sent which are fine
> but using SSL, chunks of 30kB are sent which causes the socket to
> still be busy on the next attempt. This is find expect that somewhere
> it seems to report that the bytes were sent and read more from the
> input device.
> 
> That is as far as I've got with it.
> 
> 
> 
> On 6 Oct 2009, at 14:27, Jason H wrote:
> 
> > In my experience, the content-length header gets sent.
> >
> > What do you mean by "streaming" then? Streaming to me is you don't
> > know the final length and you'll be using chunked encoding.
> >
> > Https is interesting because its block encryption. You can't send 10
> > bytes and expect 10 bytes to come out without closing the transfer.
> > It'll be buffered until it gets a whole block to encrypt, or it is
> > closed.
> >
> >
> >
> > From: Info <info at misafe.com>
> > To: Jason H <scorp1us at yahoo.com>
> > Cc: qt-interest at trolltech.com
> > Sent: Tuesday, October 6, 2009 9:23:37 AM
> > Subject: Re: [Qt-interest] QHttp upload won't stream?
> >
> > But the same method works fine when not using HTTPS, surely that
> > would also require the same informations?
> >
> > The QIODevice (output) does implement bytesAvailable() and return
> > the correct number. Is that what you mean? I also explicitly set the
> > content-length header before the put/post request but it makes no
> > difference.
> >
> >
> > On 6 Oct 2009, at 14:19, Jason H wrote:
> >
> >> The problem (as I see it) is that Qt wants to know content-length
> >> for the header (by doing bytesAvailible() on the device). If you
> >> don't, you can only use chunked encoding, which is not supported
> >> from a send perspective. I rolled my own using QHttpHeader and a
> >> socket. I do have streamed transfers, however I used signals and
> >> slots.
> >>
> >> One signal is the start message (taking care of headers) then a
> >> signal for each data chunk, then ending with a done message which
> >> closes the socket.
> >>
> >>
> >>
> >> From: Info <info at misafe.com>
> >> To: qt-interest at trolltech.com
> >> Sent: Tuesday, October 6, 2009 9:00:52 AM
> >> Subject: Re: [Qt-interest] QHttp upload won't stream?
> >>
> >> Hi,
> >>
> >> Did you get anywhere with this? I'm having exactly the same problem
> >> with QNetworkAccessManager and have tracked it back to the same
> >> place you have QSslSocket::writeData, so it seems that you will
> >> still have the issue if you switch from QHttp to
> >> QNetworkAccessManager as they both use the same underlying socket.
> >>
> >>
> >> On 30 Sep 2009, at 18:02, qt-interest-request at trolltech.com wrote:
> >>
> >>> Hi All,
> >>>
> >>> (Note: also posted on Qt Centre.  Just trying to widen the net.)
> >>>
> >>> Using QT 4.5.2 on Windows XP.
> >>>
> >>> I have a file upload API which uses QHttp to send a PUT request to
> >>> a WebDav server.  It works fine except for one major problem:  it
> >>> loads the entire file into memory before it uploads the file.
> >>> This is a problem with large files, a show stopper if the client
> >>> runs out of memory.  See code example below (of course, you need a
> >>> dav server for it to work).
> >>>
> >>> int QHttp::request ( const QHttpRequestHeader & header, QIODevice
> >>> * data = 0, QIODevice * to = 0 )
> >>>
> >>> I'm using QHttp::request to start the request.  I can't seem to
> >>> come up with a device or combination of devices which will stream
> >>> or otherwise buffer the file from the disk.
> >>>
> >>> My working code (which doesn't stream) just passes in the QFile:
> >>>
> >>> _httpId = _qHttp->request(header, sourceFile);
> >>>
> >>> I tried passing a QDataStream (or TextStream), which is not a
> >>> QIODevice.  Tried to come up with a combination of QBuffer, QFile,
> >>> just to see if something would work.  No luck.
> >>>
> >>> I took a look at the QT source.  Interestingly, it _is_ buffering
> >>> the file read in qhttp.cpp, QHttpPrivate::_q_slotBytesWritten().
> >>> The memory load looks to me like it's happening in qsslsocket.cpp,
> >>> QSslSocket::writeData() when the method memcpy's the buffer.
> >>>
> >>> - Can I use QHttp to do what I want?  It doesn't seem like it, but
> >>> maybe I'm missing something.  Is this a bug (I don't think so, yet)?
> >>> - Is it worth trying QNetworkAccessManager, or am I going to run
> >>> into the same problem?  (I'm working on trying this now.  Looks to
> >>> me like QNetworkAccessManager is an entirely new impl.)
> >>> - If I need to write custom code to support this, could someone
> >>> outline a basic approach?  Will I need to create multiple requests
> >>> for this kind of "chunked" transfer?
> >>>
> >>> Code snippet (builds, haven't tried to run.  my actual code has a
> >>> bunch of classes abstracting/wrapping a lot of this.)
> >>>
> >>>
> >>>    QUrl
> destUrl(QString("http://www.someDAVserver.com/someFolder/someBigFile.da
> t
> >>> "));
> >>>    QFile* sourceFile = new QFile("D:\\someLocalDir\
> >>> \someBigFile.dat");
> >>>
> >>>    if (!sourceFile->open(QIODevice::ReadOnly)) {
> >>>        //throw
> >>>    }
> >>>
> >>>    QHttp::ConnectionMode mode = destUrl.scheme().toLower() ==
> >>> "https" ? QHttp::ConnectionModeHttps : QHttp::ConnectionModeHttp;
> >>>
> >>>    QHttp http;
> >>>    http.setHost(destUrl.host(), mode, destUrl.port() == -1 ? 0 :
> >>> destUrl.port());
> >>>
> >>>    QHttpRequestHeader header("PUT", destUrl.toEncoded
> >>> (QUrl::RemoveScheme|QUrl::RemoveAuthority));
> >>>    header.setValue("Host", destUrl.host());
> >>>    header.setValue("Accept-Language", QLocale::system().name());
> >>>    header.setValue("Content-Length", QString::number(sourceFile-
> >>> >size()));
> >>>
> >>>
> >>>    http.request(header, sourceFile);
> >>>
> >>>
> >>>
> >>> Thanks in advance,
> >>>
> >>> Hawkeye Parker
> >>
> >>
> >> __________________________________________________
> >> Do You Yahoo!?
> >> Tired of spam? Yahoo! Mail has the best spam protection around
> >> http://mail.yahoo.com
> >
> >
> >
> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://lists.trolltech.com/pipermail/qt-
> interest/attachments/20091006/0d873ed4/attachment.html
> 
> ------------------------------
> 
> _______________________________________________
> Qt-interest mailing list
> Qt-interest at trolltech.com
> http://lists.trolltech.com/mailman/listinfo/qt-interest
> 
> 
> End of Qt-interest Digest, Vol 11, Issue 46
> *******************************************