[Qt-interest] QHttp upload won't stream?

Tue Oct 6 18:49:00 CEST 2009

I've tracked it down to the fact that when  
QNativeSocketEnginePrivate::nativeWrite() calls WSASend() it returns  
SOCKET_ERROR and the reason is WSAEWOULDBLOCK, now this appears to be  
because under normal HTTP, only chunks of 4096 are sent which are fine  
but using SSL, chunks of 30kB are sent which causes the socket to  
still be busy on the next attempt. This is find expect that somewhere  
it seems to report that the bytes were sent and read more from the  
input device.

That is as far as I've got with it.

On 6 Oct 2009, at 14:27, Jason H wrote:

> In my experience, the content-length header gets sent.
>
> What do you mean by "streaming" then? Streaming to me is you don't  
> know the final length and you'll be using chunked encoding.
>
> Https is interesting because its block encryption. You can't send 10  
> bytes and expect 10 bytes to come out without closing the transfer.  
> It'll be buffered until it gets a whole block to encrypt, or it is  
> closed.
>
>
>
> From: Info <info at misafe.com>
> To: Jason H <scorp1us at yahoo.com>
> Cc: qt-interest at trolltech.com
> Sent: Tuesday, October 6, 2009 9:23:37 AM
> Subject: Re: [Qt-interest] QHttp upload won't stream?
>
> But the same method works fine when not using HTTPS, surely that  
> would also require the same informations?
>
> The QIODevice (output) does implement bytesAvailable() and return  
> the correct number. Is that what you mean? I also explicitly set the  
> content-length header before the put/post request but it makes no  
> difference.
>
>
> On 6 Oct 2009, at 14:19, Jason H wrote:
>
>> The problem (as I see it) is that Qt wants to know content-length  
>> for the header (by doing bytesAvailible() on the device). If you  
>> don't, you can only use chunked encoding, which is not supported  
>> from a send perspective. I rolled my own using QHttpHeader and a  
>> socket. I do have streamed transfers, however I used signals and  
>> slots.
>>
>> One signal is the start message (taking care of headers) then a  
>> signal for each data chunk, then ending with a done message which  
>> closes the socket.
>>
>>
>>
>> From: Info <info at misafe.com>
>> To: qt-interest at trolltech.com
>> Sent: Tuesday, October 6, 2009 9:00:52 AM
>> Subject: Re: [Qt-interest] QHttp upload won't stream?
>>
>> Hi,
>>
>> Did you get anywhere with this? I'm having exactly the same problem  
>> with QNetworkAccessManager and have tracked it back to the same  
>> place you have QSslSocket::writeData, so it seems that you will  
>> still have the issue if you switch from QHttp to  
>> QNetworkAccessManager as they both use the same underlying socket.
>>
>>
>> On 30 Sep 2009, at 18:02, qt-interest-request at trolltech.com wrote:
>>
>>> Hi All,
>>>
>>> (Note: also posted on Qt Centre.  Just trying to widen the net.)
>>>
>>> Using QT 4.5.2 on Windows XP.
>>>
>>> I have a file upload API which uses QHttp to send a PUT request to  
>>> a WebDav server.  It works fine except for one major problem:  it  
>>> loads the entire file into memory before it uploads the file.   
>>> This is a problem with large files, a show stopper if the client  
>>> runs out of memory.  See code example below (of course, you need a  
>>> dav server for it to work).
>>>
>>> int QHttp::request ( const QHttpRequestHeader & header, QIODevice  
>>> * data = 0, QIODevice * to = 0 )
>>>
>>> I'm using QHttp::request to start the request.  I can't seem to  
>>> come up with a device or combination of devices which will stream  
>>> or otherwise buffer the file from the disk.
>>>
>>> My working code (which doesn't stream) just passes in the QFile:
>>>
>>> _httpId = _qHttp->request(header, sourceFile);
>>>
>>> I tried passing a QDataStream (or TextStream), which is not a  
>>> QIODevice.  Tried to come up with a combination of QBuffer, QFile,  
>>> just to see if something would work.  No luck.
>>>
>>> I took a look at the QT source.  Interestingly, it _is_ buffering  
>>> the file read in qhttp.cpp, QHttpPrivate::_q_slotBytesWritten().   
>>> The memory load looks to me like it's happening in qsslsocket.cpp,  
>>> QSslSocket::writeData() when the method memcpy's the buffer.
>>>
>>> - Can I use QHttp to do what I want?  It doesn't seem like it, but  
>>> maybe I'm missing something.  Is this a bug (I don't think so, yet)?
>>> - Is it worth trying QNetworkAccessManager, or am I going to run  
>>> into the same problem?  (I'm working on trying this now.  Looks to  
>>> me like QNetworkAccessManager is an entirely new impl.)
>>> - If I need to write custom code to support this, could someone  
>>> outline a basic approach?  Will I need to create multiple requests  
>>> for this kind of "chunked" transfer?
>>>
>>> Code snippet (builds, haven't tried to run.  my actual code has a  
>>> bunch of classes abstracting/wrapping a lot of this.)
>>>
>>>
>>>    QUrl destUrl(QString("http://www.someDAVserver.com/someFolder/someBigFile.dat 
>>> "));
>>>    QFile* sourceFile = new QFile("D:\\someLocalDir\ 
>>> \someBigFile.dat");
>>>
>>>    if (!sourceFile->open(QIODevice::ReadOnly)) {
>>>        //throw
>>>    }
>>>
>>>    QHttp::ConnectionMode mode = destUrl.scheme().toLower() ==  
>>> "https" ? QHttp::ConnectionModeHttps : QHttp::ConnectionModeHttp;
>>>
>>>    QHttp http;
>>>    http.setHost(destUrl.host(), mode, destUrl.port() == -1 ? 0 :  
>>> destUrl.port());
>>>
>>>    QHttpRequestHeader header("PUT", destUrl.toEncoded 
>>> (QUrl::RemoveScheme|QUrl::RemoveAuthority));
>>>    header.setValue("Host", destUrl.host());
>>>    header.setValue("Accept-Language", QLocale::system().name());
>>>    header.setValue("Content-Length", QString::number(sourceFile- 
>>> >size()));
>>>
>>>
>>>    http.request(header, sourceFile);
>>>
>>>
>>>
>>> Thanks in advance,
>>>
>>> Hawkeye Parker
>>
>>
>> __________________________________________________
>> Do You Yahoo!?
>> Tired of spam? Yahoo! Mail has the best spam protection around
>> http://mail.yahoo.com
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.qt-project.org/pipermail/qt-interest-old/attachments/20091006/0d873ed4/attachment.html 