[Interest] Flush after QProcess::write to prevent it from filling writeBuffer
philip at c0xc.net
Sat Feb 18 20:44:06 CET 2023
Hi Axel! Sure:
int main(int argc, char *argv)
QApplication app(argc, argv);
QString if_path = argv;
QString of_path = argv;
qInfo() << "test: reading from" << if_path << "writing to" << of_path;
if (!file.open(QIODevice::ReadOnly | QIODevice::Unbuffered))
qInfo() << "failed to open file";
dd_args << "bs=1M" << QString("of=") + of_path; //careful; just a test
qInfo() << "starting dd...";
qInfo() << "now starting read/write loop..." << proc.processId();
int bs = 1024 * 1024;
QByteArray block = file.read(bs);
qInfo() << "empty read";
qInfo() << "read:" << block.size();
qint64 written = proc.write(block);
if (written != block.size())
qInfo().noquote() << QString("write failed: %1 written (out of %2)").arg(written).arg(block.size());
qInfo() << "written:" << written;
//This should wait/block until all data are written/synced
//...but it doesn't...!?
qInfo() << "waiting for output process...";
bool ok = proc.waitForBytesWritten(-1); //this should block until all written bytes have been sent to the process
qInfo() << "output process confirmed bytes written" << ok;
qInfo() << "done!";
TARGET = qprocess_test
SOURCES = *.cpp
QT += widgets
$ qmake && make
$ dd bs=1M if=/dev/urandom of=/tmp/hugefile count=4096
$ ./qprocess_test /tmp/hugefile /mnt/tmp/targetfile
In my test case, /mnt/tmp/targetfile is on a slow NFS share but it could also be a USB 2.0 storage; don't use something fast like a tmpfs.
The program waits for five seconds so you have time to open: top -p $(pidof qprocess_test) (maybe hit e)
Then its memory usage grows to 4G which should never happen.
On 2/18/23 19:02, Axel Spoerl wrote:
> Hi Philip,
> Could you provide that minimal reproduction example?
>> On 18 Feb 2023, at 01:17, Philip Seeger<philip at c0xc.net> wrote:
>> Hi there!
>> I'm writing 1M chunks of data to a QProcess in a loop until all input data is consumed. How do I wait after calling QProcess::write() until all data has been sent to the process before calling write() again?
>> As documented, I have tried to call waitForBytesWritten(-1) after write() but it only helps in the first iteration; then it does not seem to block until all bytes are sent to the process. The result is that my program uses almost 4G of RES memory if a 4G input file is being read and the loop finishes in a second. Closing QProcess actually sends the data to the process (takes minutes) and I don't see any progress.
>> Internally, Qt seems to use a QByteArray as writeBuffer and I could not find where it checks if it exceeds a maximum size.
>> I believe QProcess storing 4 GB of data in an internal buffer is wrong under any circumstances. Have I overlooked something in the documentation? How do I turn this off?
>> I can provide a minimal example.
>> Interest mailing list
>> Interest at qt-project.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Interest