[Interest] Help parsing output of "mksquashfs" command
Elvis Stansvik
elvstone at gmail.com
Thu Jun 2 17:52:48 CEST 2016
2016-06-02 15:19 GMT+02:00 Elvis Stansvik <elvstone at gmail.com>:
> 2016-06-02 13:47 GMT+02:00 Juan Navarro <oneorjuan at gmail.com>:
>> Hello,
>>
>> I'm building a GUI tool under Kubuntu Linux 14.04, which among other things,
>> creates SquashFS (https://en.wikipedia.org/wiki/SquashFS) images with the
>> command-line command "mksquashfs". Due to the size of the images, this
>> process takes a good amount of time, so I'd like to show its progress with a
>> QProgressBar.
>>
>> Problem here is that the default output from this command is not really
>> intended to be parsed. It uses a dynamic progress bar + a progress
>> percentage which repaint themselves on the terminal, I guess that using the
>> technique of printing "\r" (carriage returns) without "\n" (line feeds), so
>> the progress bar is always drawn on the same row of the console output:
>>
>> $ mksquashfs /home /tmp/test.sfs -noappend -processors 1
>> Parallel mksquashfs: Using 1 processor
>> Creating 4.0 filesystem on /tmp/test.sfs, block size 131072.
>> [======================================/ ] 699/1250 75%
>>
>>
>> However I've found the way of printing the progress bar + percentage value
>> in a serial way, ie. each iteration gets printed on its own line, with the
>> following command line:
>>
>> $ script --return --flush --quiet \
>> --command "mksquashfs /home \
>> /tmp/test.sfs -info -progress -noappend \
>> -processors 1" > test.txt
>>
>> With this, the output shown in "file.txt" is somewhat similar to this:
>>
>> file ..., uncompressed size 2755336 bytes
>> [=========\ ] 481/12359
>> 3%
>> file ..., uncompressed size 726904 bytes
>> [==========\ ] 528/12359
>> 4%
>> file ..., uncompressed size 577 bytes
>> [==============\ ] 719/12359
>> 5%
>>
>> This should be really easy to parse, extract the "x%" from the end of the
>> line, and send that to the QProgressBar.
>>
>> So I've ported this command line to my Qt code. This is the relevant part of
>> the code:
>>
>> class MyClass
>> {
>> QProcess p;
>>
>> void start() {
>> p.setProcessChannelMode(QProcess::SeparateChannels);
>> p.setReadChannel(QProcess::StandardOutput);
>> connect(&p, SIGNAL(readyReadStandardOutput()),
>> this, SLOT(onProcessReadyReadStdout()));
>>
>> QString c = "script";
>> QStringList a;
>> a << "--return" << "--flush" << "--quiet"
>> << "--command"
>> << "mksquashfs /media/data/KSHOW_320/RO/home "
>> "/tmp/test.sfs -info -progress "
>> "-noappend -no-recovery -processors 1";
>> p.start(c, a, QIODevice::ReadOnly | QIODevice::Unbuffered |
>> QIODevice::Text);
>> }
>>
>> private slots:
>> void onProcessReadyReadStdout() {
>> qDebug() << "New data:" << p.readAllStandardOutput();
>> }
>> }
>>
>>
>> The problem is that given this code, the output doesn't contain the progress
>> bar...
>>
>> With Qt 4.8.4:
>> New data: "
>> file ..., uncompressed size 2755336 bytes
>> "
>> New data: "
>> file ..., uncompressed size 726904 bytes
>> "
>> New data: "
>> file ..., uncompressed size 577 bytes
>> "
>>
>> With Qt 5.6.0:
>> New data: "\nfile ..., uncompressed size 2755336 bytes \n"
>> New data: "\nfile ..., uncompressed size 726904 bytes \n"
>> New data: "\nfile ..., uncompressed size 577 bytes \n"
>>
>> (Note how in Qt 5.6.0 the same command shows a different output, by
>> explicitly showing '\n' characters; also the flag "QIODevice::Text" doesn't
>> make a difference here).
>>
>> I'm quite lost here, because at this point I assume that QProcess is doing
>> the right thing, but I don't understand at what point in the command chain
>> the output is being generated differently between those two different ways
>> of running the same command.
>>
>> It's probably the way the carriage returns are managed in the QProcess vs.
>> shell redirection to a file, but I'd like to see if anyone here has some
>> comment.
>
> Maybe it needs a TTY to output the progress bar? That's out of scope
> for QProcess, but if you're fine with a dependency on kcoreaddons +
> kpty, it looks like the latter has a KPtyProcess you could use [1].
I was curious if that would work, so I tested it out with this:
#include <QCoreApplication>
#include <QRegularExpression>
#include <QObject>
#include <QDebug>
#include <KPtyProcess>
#include <KPtyDevice>
class MkSquashFs : public KPtyProcess
{
Q_OBJECT
public:
MkSquashFs(const QStringList &sources, const QString &destination,
QObject *parent = 0)
: KPtyProcess(parent) {
pty()->setWinSize(25, 80);
setPtyChannels(KPtyProcess::AllChannels);
setProgram("mksquashfs", QStringList() << sources << destination);
connect(pty(), &KPtyDevice::readyRead, this, &MkSquashFs::readOutput);
}
signals:
void progressChanged(int progress);
private:
void readOutput() {
static QRegularExpression progressRe("(\\d+)%($|\\r)");
auto match = progressRe.match(pty()->readAll());
if (match.hasMatch()) {
emit progressChanged(QString(match.captured(1)).toInt());
}
}
private:
KPtyProcess process;
};
void printProgress(int progress) {
qDebug() << "progress:" << progress;
}
int main(int argc, char *argv[]) {
QCoreApplication app(argc, argv);
MkSquashFs mkSquashFs({"/some/dir"}, "/some/file.sqsh");
QObject::connect(&mkSquashFs, &MkSquashFs::progressChanged, printProgress);
mkSquashFs.start();
return app.exec();
}
#include "main.moc"
And it seems to work fine. One important thing was the
pty()->setWinSize(25, 80) to set the terminal window size.
Hope that helps,
Elvis
>
> Elvis
>
> [1] https://api.kde.org/frameworks/kpty/html/classKPtyProcess.html
>
>>
>> Regards,
>> Juan
>>
>> _______________________________________________
>> Interest mailing list
>> Interest at qt-project.org
>> http://lists.qt-project.org/mailman/listinfo/interest
>>
More information about the Interest
mailing list