[Qt-interest] Problem using QNetworkAccessManager/QNetworkReply for large file download

[Mandeep Sandhu]

Hi All,

I'm using Qt 4.5.3 on an embedded Linux platform (Linux 2.6.22.19).

I've written a simple program which downloads a file via HTTP from a
webserver (nginx). The file is large in size (~900MB).

I use the QNetworkAccessManager to setup the GET request and also
setup the read buffer size
(QNetworkReply::setReadBufferSize(BUFSIZE)). I use the "readyReady"
signal to read data off the network reply object (read it in BUFSIZE
chunks). I've also put a hack in my code which will stop reading data
from the QIODevice(QNetworkReply) once I've downloaded some 16% of the
file *(the 16% is chosen arbitrarily). See attached code. I've set my
BUFSIZE to 128*1024 (128K).

Now I'm facing 2 issues:

1) Whenever I read BUFSIZE in the readyRead signal, I'm seeing that I
get LESS number of bytes than BUFSIZE! I get around 70K as size of the
QByteArray. Is this expected?

2) I've set the read buffer size to 128K and expect Qt to "throttle"
the download whenever the read buffer gets full. I keep getting the
readyRead signal till the buffer is full (QIODevice::bytesAvailable =
BUFSIZE) and then after sometime the download completes (verified from
webserver log which shows the full file is served). Does this mean
that Qt actually downloads the _complete_ file and does not throttle
at the network level? I'm asking this because the file download works
for small files, but the program seg faults for a large file (eg the
file i tested was 900MB). The webserver log shows whatever bytes that
were served before the connection was close as a result of the program
crashing.

Has anybody experienced similar issue?

I've found a bug that fixes something similar, though it mentions only
MAC and Windows as the affected platforms:

http://bugreports.qt.nokia.com/browse/QTBUG-6276?page=com.atlassian.jira.plugin.system.issuetabpanels%3Achangehistory-tabpanel

It's also fixed in Qt 4.6! :(

Thanks,
-mandeep
-------------- next part --------------
A non-text attachment was scrubbed...
Name: main.cpp
Type: text/x-c++src
Size: 3013 bytes
Desc: not available
Url : http://lists.qt-project.org/pipermail/qt-interest-old/attachments/20100217/c4f39d60/attachment.bin