[Qt-interest] Problem using QNetworkAccessManager/QNetworkReply for large file download

[Thiago Macieira]

Em Quarta-feira 17 Fevereiro 2010, às 09:54:08, Mandeep Sandhu escreveu:
> Hi All,
> 
> I'm using Qt 4.5.3 on an embedded Linux platform (Linux 2.6.22.19).

Hi

Please upgrade to 4.6.2.

> I use the QNetworkAccessManager to setup the GET request and also
> setup the read buffer size
> (QNetworkReply::setReadBufferSize(BUFSIZE)). I use the "readyReady"
> signal to read data off the network reply object (read it in BUFSIZE
> chunks). I've also put a hack in my code which will stop reading data
> from the QIODevice(QNetworkReply) once I've downloaded some 16% of the
> file *(the 16% is chosen arbitrarily). See attached code. I've set my
> BUFSIZE to 128*1024 (128K).
> 
> Now I'm facing 2 issues:
> 
> 1) Whenever I read BUFSIZE in the readyRead signal, I'm seeing that I
> get LESS number of bytes than BUFSIZE! I get around 70K as size of the
> QByteArray. Is this expected?

Yes, at least for the first emissions of readyRead.

The signal is emitted whenever new data arrives, not when the buffer is full. 
So you'll probably receive an emission at 1480 bytes, then around 16k and 
upwards from that (from experience, an application reading from a socket). 

However, the signal should stop firing once the buffer has BUFSIZE bytes or more 
(it's not guaranteed to stop at exact BUFSIZE).

> 2) I've set the read buffer size to 128K and expect Qt to "throttle"
> the download whenever the read buffer gets full. I keep getting the
> readyRead signal till the buffer is full (QIODevice::bytesAvailable =
> BUFSIZE) and then after sometime the download completes (verified from
> webserver log which shows the full file is served). Does this mean
> that Qt actually downloads the _complete_ file and does not throttle
> at the network level? I'm asking this because the file download works
> for small files, but the program seg faults for a large file (eg the
> file i tested was 900MB). The webserver log shows whatever bytes that
> were served before the connection was close as a result of the program
> crashing.

Yes, this was a bug in Qt.

I think it was fixed for Qt 4.6.1. But, just in case, upgrade to 4.6.2.

> It's also fixed in Qt 4.6! :(

Yes, upgrade.
-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
  Senior Product Manager - Nokia, Qt Development Frameworks
      PGP/GPG: 0x6EF45358; fingerprint:
      E067 918B B660 DBD1 105C  966C 33F5 F005 6EF4 5358
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part.
Url : http://lists.qt-project.org/pipermail/qt-interest-old/attachments/20100217/c94951f3/attachment.bin