[Interest] Help parsing output of "mksquashfs" command

[Elvis Stansvik]

2016-06-02 15:19 GMT+02:00 Elvis Stansvik <elvstone at gmail.com>:
> 2016-06-02 13:47 GMT+02:00 Juan Navarro <oneorjuan at gmail.com>:
>> Hello,
>>
>> I'm building a GUI tool under Kubuntu Linux 14.04, which among other things,
>> creates SquashFS (https://en.wikipedia.org/wiki/SquashFS) images with the
>> command-line command "mksquashfs". Due to the size of the images, this
>> process takes a good amount of time, so I'd like to show its progress with a
>> QProgressBar.
>>
>> Problem here is that the default output from this command is not really
>> intended to be parsed. It uses a dynamic progress bar + a progress
>> percentage which repaint themselves on the terminal, I guess that using the
>> technique of printing "\r" (carriage returns) without "\n" (line feeds), so
>> the progress bar is always drawn on the same row of the console output:
>>
>>     $ mksquashfs /home /tmp/test.sfs -noappend -processors 1
>>     Parallel mksquashfs: Using 1 processor
>>     Creating 4.0 filesystem on /tmp/test.sfs, block size 131072.
>>     [======================================/                ]  699/1250  75%
>>
>>
>> However I've found the way of printing the progress bar + percentage value
>> in a serial way, ie. each iteration gets printed on its own line, with the
>> following command line:
>>
>>     $ script --return --flush --quiet \
>>     --command "mksquashfs /home \
>>     /tmp/test.sfs -info -progress -noappend \
>>     -processors 1" > test.txt
>>
>> With this, the output shown in "file.txt" is somewhat similar to this:
>>
>>     file ..., uncompressed size 2755336 bytes
>>     [=========\                                             ]   481/12359
>> 3%
>>     file ..., uncompressed size 726904 bytes
>>     [==========\                                            ]   528/12359
>> 4%
>>     file ..., uncompressed size 577 bytes
>>     [==============\                                        ]   719/12359
>> 5%
>>
>> This should be really easy to parse, extract the "x%" from the end of the
>> line, and send that to the QProgressBar.
>>
>> So I've ported this command line to my Qt code. This is the relevant part of
>> the code:
>>
>>     class MyClass
>>     {
>>         QProcess p;
>>
>>         void start() {
>>             p.setProcessChannelMode(QProcess::SeparateChannels);
>>             p.setReadChannel(QProcess::StandardOutput);
>>             connect(&p, SIGNAL(readyReadStandardOutput()),
>>                 this, SLOT(onProcessReadyReadStdout()));
>>
>>             QString c = "script";
>>             QStringList a;
>>             a << "--return" << "--flush" << "--quiet"
>>             << "--command"
>>             << "mksquashfs /media/data/KSHOW_320/RO/home "
>>                "/tmp/test.sfs -info -progress "
>>                "-noappend -no-recovery -processors 1";
>>             p.start(c, a, QIODevice::ReadOnly | QIODevice::Unbuffered |
>> QIODevice::Text);
>>         }
>>
>>     private slots:
>>         void onProcessReadyReadStdout() {
>>             qDebug() << "New data:" << p.readAllStandardOutput();
>>         }
>>     }
>>
>>
>> The problem is that given this code, the output doesn't contain the progress
>> bar...
>>
>> With Qt 4.8.4:
>>     New data: "
>>     file ..., uncompressed size 2755336 bytes
>>     "
>>     New data: "
>>     file ..., uncompressed size 726904 bytes
>>     "
>>     New data: "
>>     file ..., uncompressed size 577 bytes
>>     "
>>
>> With Qt 5.6.0:
>>     New data: "\nfile ..., uncompressed size 2755336 bytes \n"
>>     New data: "\nfile ..., uncompressed size 726904 bytes \n"
>>     New data: "\nfile ..., uncompressed size 577 bytes \n"
>>
>> (Note how in Qt 5.6.0 the same command shows a different output, by
>> explicitly showing '\n' characters; also the flag "QIODevice::Text" doesn't
>> make a difference here).
>>
>> I'm quite lost here, because at this point I assume that QProcess is doing
>> the right thing, but I don't understand at what point in the command chain
>> the output is being generated differently between those two different ways
>> of running the same command.
>>
>> It's probably the way the carriage returns are managed in the QProcess vs.
>> shell redirection to a file, but I'd like to see if anyone here has some
>> comment.
>
> Maybe it needs a TTY to output the progress bar? That's out of scope
> for QProcess, but if you're fine with a dependency on kcoreaddons +
> kpty, it looks like the latter has a KPtyProcess you could use [1].

I was curious if that would work, so I tested it out with this:

#include <QCoreApplication>
#include <QRegularExpression>
#include <QObject>
#include <QDebug>

#include <KPtyProcess>
#include <KPtyDevice>

class MkSquashFs : public KPtyProcess
{
    Q_OBJECT

public:
    MkSquashFs(const QStringList &sources, const QString &destination,
QObject *parent = 0)
            : KPtyProcess(parent) {
        pty()->setWinSize(25, 80);
        setPtyChannels(KPtyProcess::AllChannels);
        setProgram("mksquashfs", QStringList() << sources << destination);
        connect(pty(), &KPtyDevice::readyRead, this, &MkSquashFs::readOutput);
    }

signals:
    void progressChanged(int progress);

private:
    void readOutput() {
        static QRegularExpression progressRe("(\\d+)%($|\\r)");
        auto match = progressRe.match(pty()->readAll());
        if (match.hasMatch()) {
            emit progressChanged(QString(match.captured(1)).toInt());
        }
    }


private:
    KPtyProcess process;
};

void printProgress(int progress) {
    qDebug() << "progress:" << progress;
}

int main(int argc, char *argv[]) {
    QCoreApplication app(argc, argv);

    MkSquashFs mkSquashFs({"/some/dir"}, "/some/file.sqsh");

    QObject::connect(&mkSquashFs, &MkSquashFs::progressChanged, printProgress);

    mkSquashFs.start();

    return app.exec();
}

#include "main.moc"

And it seems to work fine. One important thing was the
pty()->setWinSize(25, 80) to set the terminal window size.

Hope that helps,
Elvis

>
> Elvis
>
> [1] https://api.kde.org/frameworks/kpty/html/classKPtyProcess.html
>
>>
>> Regards,
>> Juan
>>
>> _______________________________________________
>> Interest mailing list
>> Interest at qt-project.org
>> http://lists.qt-project.org/mailman/listinfo/interest
>>