[Qt-interest] QHttp upload won't stream?
Hawkeye Parker
hawkeye.parker at autodesk.com
Fri Oct 16 21:04:07 CEST 2009
Hi Markus,
Thanks _very_ much for the patch. We'll be trying this shortly and let you and the list know how it goes.
Best regards,
Hawkeye Parker
-----Original Message-----
From: Markus Goetz [mailto:Markus.Goetz at nokia.com]
Sent: Wednesday, October 14, 2009 7:59 AM
To: Hawkeye Parker
Cc: qt-interest at trolltech.com
Subject: Re: Re: [Qt-interest] QHttp upload won't stream?
Hi,
I have a patch for Qt 4.5.x (should apply to 4.6. too) and QHttp:
http://pastebin.ca/1619204 (remove the tests/auto part if your Qt
sources do not have that)
Does it fix the issue for you?
Did anyone of the QNetworkAccessManager users try Qt 4.6.? It should be
fixed there too (not on Qt 4.5 though!)
thanks,
Markus
ext Hawkeye Parker wrote:
> Hi all,
>
> I started this thread, and just now getting back to it. Apologies.
>
> Cross-reference (pretty much same content):
> http://www.qtcentre.org/forum/f-qt-programming-2/t-qhttp-file-upload-doesnt-stream-24404.html
>
> I also confirmed that using QNetworkAccessManager makes no difference. Also that the problem seems to have to do with SSL: when I use HTTP, things stream fine.
>
> Haven't gotten any further than that.
>
> Hawkeye Parker
>
>
>> Date: Tue, 6 Oct 2009 17:49:00 +0100
>> From: Info <info at misafe.com>
>> Subject: Re: [Qt-interest] QHttp upload won't stream?
>> To: Jason H <scorp1us at yahoo.com>
>> Cc: qt-interest at trolltech.com
>>
>> I've tracked it down to the fact that when
>> QNativeSocketEnginePrivate::nativeWrite() calls WSASend() it returns
>> SOCKET_ERROR and the reason is WSAEWOULDBLOCK, now this appears to be
>> because under normal HTTP, only chunks of 4096 are sent which are fine
>> but using SSL, chunks of 30kB are sent which causes the socket to
>> still be busy on the next attempt. This is find expect that somewhere
>> it seems to report that the bytes were sent and read more from the
>> input device.
>>
>> That is as far as I've got with it.
>>
>>
>>
>> On 6 Oct 2009, at 14:27, Jason H wrote:
>>
>>
>>> In my experience, the content-length header gets sent.
>>>
>>> What do you mean by "streaming" then? Streaming to me is you don't
>>> know the final length and you'll be using chunked encoding.
>>>
>>> Https is interesting because its block encryption. You can't send 10
>>> bytes and expect 10 bytes to come out without closing the transfer.
>>> It'll be buffered until it gets a whole block to encrypt, or it is
>>> closed.
>>>
>>>
>>>
>>> From: Info <info at misafe.com>
>>> To: Jason H <scorp1us at yahoo.com>
>>> Cc: qt-interest at trolltech.com
>>> Sent: Tuesday, October 6, 2009 9:23:37 AM
>>> Subject: Re: [Qt-interest] QHttp upload won't stream?
>>>
>>> But the same method works fine when not using HTTPS, surely that
>>> would also require the same informations?
>>>
>>> The QIODevice (output) does implement bytesAvailable() and return
>>> the correct number. Is that what you mean? I also explicitly set the
>>> content-length header before the put/post request but it makes no
>>> difference.
>>>
>>>
>>> On 6 Oct 2009, at 14:19, Jason H wrote:
>>>
>>>
>>>> The problem (as I see it) is that Qt wants to know content-length
>>>> for the header (by doing bytesAvailible() on the device). If you
>>>> don't, you can only use chunked encoding, which is not supported
>>>> from a send perspective. I rolled my own using QHttpHeader and a
>>>> socket. I do have streamed transfers, however I used signals and
>>>> slots.
>>>>
>>>> One signal is the start message (taking care of headers) then a
>>>> signal for each data chunk, then ending with a done message which
>>>> closes the socket.
>>>>
>>>>
>>>>
>>>> From: Info <info at misafe.com>
>>>> To: qt-interest at trolltech.com
>>>> Sent: Tuesday, October 6, 2009 9:00:52 AM
>>>> Subject: Re: [Qt-interest] QHttp upload won't stream?
>>>>
>>>> Hi,
>>>>
>>>> Did you get anywhere with this? I'm having exactly the same problem
>>>> with QNetworkAccessManager and have tracked it back to the same
>>>> place you have QSslSocket::writeData, so it seems that you will
>>>> still have the issue if you switch from QHttp to
>>>> QNetworkAccessManager as they both use the same underlying socket.
>>>>
>>>>
>>>> On 30 Sep 2009, at 18:02, qt-interest-request at trolltech.com wrote:
>>>>
>>>>
>>>>> Hi All,
>>>>>
>>>>> (Note: also posted on Qt Centre. Just trying to widen the net.)
>>>>>
>>>>> Using QT 4.5.2 on Windows XP.
>>>>>
>>>>> I have a file upload API which uses QHttp to send a PUT request to
>>>>> a WebDav server. It works fine except for one major problem: it
>>>>> loads the entire file into memory before it uploads the file.
>>>>> This is a problem with large files, a show stopper if the client
>>>>> runs out of memory. See code example below (of course, you need a
>>>>> dav server for it to work).
>>>>>
>>>>> int QHttp::request ( const QHttpRequestHeader & header, QIODevice
>>>>> * data =, QIODevice * to = 0 )
>>>>>
>>>>> I'm using QHttp::request to start the request. I can't seem to
>>>>> come up with a device or combination of devices which will stream
>>>>> or otherwise buffer the file from the disk.
>>>>>
>>>>> My working code (which doesn't stream) just passes in the QFile:
>>>>>
>>>>> _httpId =qHttp->request(header, sourceFile);
>>>>>
>>>>> I tried passing a QDataStream (or TextStream), which is not a
>>>>> QIODevice. Tried to come up with a combination of QBuffer, QFile,
>>>>> just to see if something would work. No luck.
>>>>>
>>>>> I took a look at the QT source. Interestingly, it _is_ buffering
>>>>> the file read in qhttp.cpp, QHttpPrivate::_q_slotBytesWritten().
>>>>> The memory load looks to me like it's happening in qsslsocket.cpp,
>>>>> QSslSocket::writeData() when the method memcpy's the buffer.
>>>>>
>>>>> - Can I use QHttp to do what I want? It doesn't seem like it, but
>>>>> maybe I'm missing something. Is this a bug (I don't think so, yet)?
>>>>> - Is it worth trying QNetworkAccessManager, or am I going to run
>>>>> into the same problem? (I'm working on trying this now. Looks to
>>>>> me like QNetworkAccessManager is an entirely new impl.)
>>>>> - If I need to write custom code to support this, could someone
>>>>> outline a basic approach? Will I need to create multiple requests
>>>>> for this kind of "chunked" transfer?
>>>>>
>>>>> Code snippet (builds, haven't tried to run. my actual code has a
>>>>> bunch of classes abstracting/wrapping a lot of this.)
>>>>>
>>>>>
>>>>> QUrl
>>>>>
>> destUrl(QString("http://www.someDAVserver.com/someFolder/someBigFile.da
>> t
>>
>>>>> "));
>>>>> QFile* sourceFile =ew QFile("D:\\someLocalDir\
>>>>> \someBigFile.dat");
>>>>>
>>>>> if (!sourceFile->open(QIODevice::ReadOnly)) {
>>>>> //throw
>>>>> }
>>>>>
>>>>> QHttp::ConnectionMode mode =estUrl.scheme().toLower() ==
>>>>> "https" ? QHttp::ConnectionModeHttps : QHttp::ConnectionModeHttp;
>>>>>
>>>>> QHttp http;
>>>>> http.setHost(destUrl.host(), mode, destUrl.port() =-1 ? 0 :
>>>>> destUrl.port());
>>>>>
>>>>> QHttpRequestHeader header("PUT", destUrl.toEncoded
>>>>> (QUrl::RemoveScheme|QUrl::RemoveAuthority));
>>>>> header.setValue("Host", destUrl.host());
>>>>> header.setValue("Accept-Language", QLocale::system().name());
>>>>> header.setValue("Content-Length", QString::number(sourceFile-
>>>>>
>>>>>> size()));
>>>>>>
>>>>> http.request(header, sourceFile);
>>>>>
>>>>>
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Hawkeye Parker
>>>>>
>>>> __________________________________________________
>>>> Do You Yahoo!?
>>>> Tired of spam? Yahoo! Mail has the best spam protection around
>>>> http://mail.yahoo.com
>>>>
>>>
>>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: http://lists.trolltech.com/pipermail/qt-
>> interest/attachments/20091006/0d873ed4/attachment.html
>>
>> ------------------------------
>>
>> _______________________________________________
>> Qt-interest mailing list
>> Qt-interest at trolltech.com
>> http://lists.trolltech.com/mailman/listinfo/qt-interest
>>
>>
>> End of Qt-interest Digest, Vol 11, Issue 46
>> *******************************************
>>
>
> _______________________________________________
> Qt-interest mailing list
> Qt-interest at trolltech.com
> http://lists.trolltech.com/mailman/listinfo/qt-interest
>
More information about the Qt-interest-old
mailing list