[Qt-interest] QHttp upload won't stream?
Info
info at misafe.com
Tue Oct 6 21:21:25 CEST 2009
Are you using QHttp for your home grown solution? Or
QNetworkAccessManager?
On 6 Oct 2009, at 20:19, Jason H wrote:
> That would seem to explain why I am not seeing it with my home-grown
> solution.
>
>
> From: Info <info at misafe.com>
> To: Jason H <scorp1us at yahoo.com>
> Cc: qt-interest at trolltech.com
> Sent: Tuesday, October 6, 2009 2:34:28 PM
> Subject: Re: [Qt-interest] QHttp upload won't stream?
>
> My theory is now that somewhere in the Qt code, there is a line
> which is adding the expecting written bytes instead of the actual
> written bytes and firing the uploadProgress() signal, if this is the
> case then simply changing it to add the actual bytes instead should
> at worst result in the uploadProgress() being fired too many times
> but still indicate the correct progress. Guess I just need to find
> that line!.
>
> On another note, the method you are using, I assume that's using
> QHttp? Would it translate to QNetworkAccessManager/QNetworkReply? If
> it would then I would be very interested to hear more.
>
> Thanks.
>
>
> On 6 Oct 2009, at 18:39, Jason H wrote:
>
>> I kinda mentioned that under SSL, it waits until the cypher block
>> is done or the socket is closed.
>>
>> I see your point in that it is reporting bytes were sent when they
>> aren't. But what can you do? You've got a hard block device, not a
>> char device. It is similar to base64 encode/decode. You want to
>> fill the block if you can. And you can until you're done. There's
>> no easy answer.
>>
>> I do both SSL and plaintext http. But I use the the signals I
>> mentioned and it seems to work for me... nwo I am using an old Qt,
>> maybe things changed?
>>
>>
>>
>> From: Info <info at misafe.com>
>> To: Jason H <scorp1us at yahoo.com>
>> Cc: qt-interest at trolltech.com
>> Sent: Tuesday, October 6, 2009 12:49:00 PM
>> Subject: Re: [Qt-interest] QHttp upload won't stream?
>>
>> I've tracked it down to the fact that when
>> QNativeSocketEnginePrivate::nativeWrite() calls WSASend() it
>> returns SOCKET_ERROR and the reason is WSAEWOULDBLOCK, now this
>> appears to be because under normal HTTP, only chunks of 4096 are
>> sent which are fine but using SSL, chunks of 30kB are sent which
>> causes the socket to still be busy on the next attempt. This is
>> find expect that somewhere it seems to report that the bytes were
>> sent and read more from the input device.
>>
>> That is as far as I've got with it.
>>
>>
>>
>> On 6 Oct 2009, at 14:27, Jason H wrote:
>>
>>> In my experience, the content-length header gets sent.
>>>
>>> What do you mean by "streaming" then? Streaming to me is you don't
>>> know the final length and you'll be using chunked encoding.
>>>
>>> Https is interesting because its block encryption. You can't send
>>> 10 bytes and expect 10 bytes to come out without closing the
>>> transfer. It'll be buffered until it gets a whole block to
>>> encrypt, or it is closed.
>>>
>>>
>>>
>>> From: Info <info at misafe.com>
>>> To: Jason H <scorp1us at yahoo.com>
>>> Cc: qt-interest at trolltech.com
>>> Sent: Tuesday, October 6, 2009 9:23:37 AM
>>> Subject: Re: [Qt-interest] QHttp upload won't stream?
>>>
>>> But the same method works fine when not using HTTPS, surely that
>>> would also require the same informations?
>>>
>>> The QIODevice (output) does implement bytesAvailable() and return
>>> the correct number. Is that what you mean? I also explicitly set
>>> the content-length header before the put/post request but it makes
>>> no difference.
>>>
>>>
>>> On 6 Oct 2009, at 14:19, Jason H wrote:
>>>
>>>> The problem (as I see it) is that Qt wants to know content-length
>>>> for the header (by doing bytesAvailible() on the device). If you
>>>> don't, you can only use chunked encoding, which is not supported
>>>> from a send perspective. I rolled my own using QHttpHeader and a
>>>> socket. I do have streamed transfers, however I used signals and
>>>> slots.
>>>>
>>>> One signal is the start message (taking care of headers) then a
>>>> signal for each data chunk, then ending with a done message which
>>>> closes the socket.
>>>>
>>>>
>>>>
>>>> From: Info <info at misafe.com>
>>>> To: qt-interest at trolltech.com
>>>> Sent: Tuesday, October 6, 2009 9:00:52 AM
>>>> Subject: Re: [Qt-interest] QHttp upload won't stream?
>>>>
>>>> Hi,
>>>>
>>>> Did you get anywhere with this? I'm having exactly the same
>>>> problem with QNetworkAccessManager and have tracked it back to
>>>> the same place you have QSslSocket::writeData, so it seems that
>>>> you will still have the issue if you switch from QHttp to
>>>> QNetworkAccessManager as they both use the same underlying socket.
>>>>
>>>>
>>>> On 30 Sep 2009, at 18:02, qt-interest-request at trolltech.com wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> (Note: also posted on Qt Centre. Just trying to widen the net.)
>>>>>
>>>>> Using QT 4.5.2 on Windows XP.
>>>>>
>>>>> I have a file upload API which uses QHttp to send a PUT request
>>>>> to a WebDav server. It works fine except for one major
>>>>> problem: it loads the entire file into memory before it uploads
>>>>> the file. This is a problem with large files, a show stopper if
>>>>> the client runs out of memory. See code example below (of
>>>>> course, you need a dav server for it to work).
>>>>>
>>>>> int QHttp::request ( const QHttpRequestHeader & header,
>>>>> QIODevice * data = 0, QIODevice * to = 0 )
>>>>>
>>>>> I'm using QHttp::request to start the request. I can't seem to
>>>>> come up with a device or combination of devices which will
>>>>> stream or otherwise buffer the file from the disk.
>>>>>
>>>>> My working code (which doesn't stream) just passes in the QFile:
>>>>>
>>>>> _httpId = _qHttp->request(header, sourceFile);
>>>>>
>>>>> I tried passing a QDataStream (or TextStream), which is not a
>>>>> QIODevice. Tried to come up with a combination of QBuffer,
>>>>> QFile, just to see if something would work. No luck.
>>>>>
>>>>> I took a look at the QT source. Interestingly, it _is_
>>>>> buffering the file read in qhttp.cpp,
>>>>> QHttpPrivate::_q_slotBytesWritten(). The memory load looks to
>>>>> me like it's happening in qsslsocket.cpp, QSslSocket::writeData
>>>>> () when the method memcpy's the buffer.
>>>>>
>>>>> - Can I use QHttp to do what I want? It doesn't seem like it,
>>>>> but maybe I'm missing something. Is this a bug (I don't think
>>>>> so, yet)?
>>>>> - Is it worth trying QNetworkAccessManager, or am I going to run
>>>>> into the same problem? (I'm working on trying this now. Looks
>>>>> to me like QNetworkAccessManager is an entirely new impl.)
>>>>> - If I need to write custom code to support this, could someone
>>>>> outline a basic approach? Will I need to create multiple
>>>>> requests for this kind of "chunked" transfer?
>>>>>
>>>>> Code snippet (builds, haven't tried to run. my actual code has
>>>>> a bunch of classes abstracting/wrapping a lot of this.)
>>>>>
>>>>>
>>>>> QUrl destUrl(QString("http://www.someDAVserver.com/someFolder/someBigFile.dat
>>>>> "));
>>>>> QFile* sourceFile = new QFile("D:\\someLocalDir\\someBigFile.dat
>>>>> ");
>>>>>
>>>>> if (!sourceFile->open(QIODevice::ReadOnly)) {
>>>>> //throw
>>>>> }
>>>>>
>>>>> QHttp::ConnectionMode mode = destUrl.scheme().toLower() ==
>>>>> "https" ? QHttp::ConnectionModeHttps : QHttp::ConnectionModeHttp;
>>>>>
>>>>> QHttp http;
>>>>> http.setHost(destUrl.host(), mode, destUrl.port() == -1 ? 0 :
>>>>> destUrl.port());
>>>>>
>>>>> QHttpRequestHeader header("PUT", destUrl.toEncoded
>>>>> (QUrl::RemoveScheme|QUrl::RemoveAuthority));
>>>>> header.setValue("Host", destUrl.host());
>>>>> header.setValue("Accept-Language", QLocale::system().name());
>>>>> header.setValue("Content-Length", QString::number(sourceFile-
>>>>> >size()));
>>>>>
>>>>>
>>>>> http.request(header, sourceFile);
>>>>>
>>>>>
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Hawkeye Parker
>>>>
>>>>
>>>> __________________________________________________
>>>> Do You Yahoo!?
>>>> Tired of spam? Yahoo! Mail has the best spam protection around
>>>> http://mail.yahoo.com
>>>
>>>
>>>
>>
>>
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.qt-project.org/pipermail/qt-interest-old/attachments/20091006/843dc533/attachment.html
More information about the Qt-interest-old
mailing list