[Interest] Flush after QProcess::write to prevent it from filling writeBuffer

[Philip]

Hi Axel! Sure:

main.cpp:

#include <QDebug>
#include <QProcess>
#include <QFile>
#include <QThread>

int main(int argc, char *argv[])
{
     QApplication app(argc, argv);
     QString if_path = argv[1];
     QString of_path = argv[2];

     qInfo() << "test: reading from" << if_path << "writing to" << of_path;

     QFile file(if_path);
     if (!file.open(QIODevice::ReadOnly | QIODevice::Unbuffered))
     {
         qInfo() << "failed to open file";
         return 1;
     }
     QProcess proc;
     proc.setProgram("dd");
     QStringList dd_args;
     dd_args << "bs=1M" << QString("of=") + of_path; //careful; just a test
     proc.setArguments(dd_args);
     qInfo() << "starting dd...";
     proc.start();
     proc.waitForStarted();
     QThread::sleep(5);

     qInfo() << "now starting read/write loop..." << proc.processId();
     int bs = 1024 * 1024;
     while (true)
     {
         QByteArray block = file.read(bs);
         if (block.isEmpty())
         {
             qInfo() << "empty read";
             break;
         }
         qInfo() << "read:" << block.size();

         qint64 written = proc.write(block);
         if (written != block.size())
         {
             qInfo().noquote() << QString("write failed: %1 written (out of %2)").arg(written).arg(block.size());
             return 2;
         }
         qInfo() << "written:" << written;
         //This should wait/block until all data are written/synced
         //...but it doesn't...!?
         qInfo() << "waiting for output process...";
         bool ok = proc.waitForBytesWritten(-1); //this should block until all written bytes have been sent to the process
         qInfo() << "output process confirmed bytes written" << ok;
     }
     qInfo() << "done!";
     QThread::sleep(15);

     return app.exec();
}

qprocess_test.pro:

TARGET = qprocess_test
SOURCES = *.cpp
QT += widgets

$ qmake && make
$ dd bs=1M if=/dev/urandom of=/tmp/hugefile count=4096
$ ./qprocess_test /tmp/hugefile /mnt/tmp/targetfile

Note:
In my test case, /mnt/tmp/targetfile is on a slow NFS share but it could also be a USB 2.0 storage; don't use something fast like a tmpfs.
The program waits for five seconds so you have time to open: top -p $(pidof qprocess_test) (maybe hit e)
Then its memory usage grows to 4G which should never happen.


Philip


On 2/18/23 19:02, Axel Spoerl wrote:
> Hi Philip,
> Could you provide that minimal reproduction example?
> Thanks
> Axel
>
>> On 18 Feb 2023, at 01:17, Philip Seeger<philip at c0xc.net>  wrote:
>>
>> Hi there!
>>
>> I'm writing 1M chunks of data to a QProcess in a loop until all input data is consumed. How do I wait after calling QProcess::write() until all data has been sent to the process before calling write() again?
>>
>> As documented, I have tried to call waitForBytesWritten(-1) after write() but it only helps in the first iteration; then it does not seem to block until all bytes are sent to the process. The result is that my program uses almost 4G of RES memory if a 4G input file is being read and the loop finishes in a second. Closing QProcess actually sends the data to the process (takes minutes) and I don't see any progress.
>>
>> Internally, Qt seems to use a QByteArray as writeBuffer and I could not find where it checks if it exceeds a maximum size.
>>
>> I believe QProcess storing 4 GB of data in an internal buffer is wrong under any circumstances. Have I overlooked something in the documentation? How do I turn this off?
>>
>> I can provide a minimal example.
>>
>>
>> Philip
>>
>>
>>
>>
>> _______________________________________________
>> Interest mailing list
>> Interest at qt-project.org
>> https://lists.qt-project.org/listinfo/interest
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/interest/attachments/20230218/2714e3ab/attachment.htm>