[Development] A faster qUtf8Printable for static trace points?

[Milian Wolff]

On Dienstag, 6. September 2016 06:59:40 CEST Thiago Macieira wrote:
> Em segunda-feira, 5 de setembro de 2016, às 21:20:12 PDT, Milian Wolff
> 
> escreveu:
> > Hey Thiago, others.
> 
> Hey Milian
> 
> Is this what you were looking for me for on Sunday afternoon?

Yes :)

> > bool QLibraryPrivate::load()
> > {
> > 
> >     ...
> >     TraceScope<...> ts(qUtf8Printable(fileName))
> >     ...
> > 
> > }
> 
> Hmm... I only agreed to something that has no runtime impact when not in
> use. I thought the traces were totally out-of-line (recorded in a separate
> section). Annotating variables sounds like a good idea, but annotating the
> result of function calls... not so much.

True, I think this was not clear and shows the value of starting this 
discussion early.

> > Trace points have nearly zero runtime overhead when not used [1]. But the
> > above has a super high cost due to calling QString::toUtf8, which
> > constructs a temporary QByteArray and thus incurs a temporary runtime
> > allocation. An alternative would be to use qUtf16Printable, which would
> > be fast, but I fear it makes interoperability with existing trace point
> > consumers hard.
> > 
> > Can someone with experience on trace points confirm this assertion? If
> > that
> > is untrue, simply using qUtf16Printable above would make everything work,
> > and be fast.
> 
> qUtf16Printable may reallocate if the QString in question was created with
> fromRawData. But that's not a very common use-case. As for existing trace
> point consumers, sorry, you're the expert.

I wouldn't call myself an expert on that yet - I have used it, and have 
written custom analysation tools for LTTNG. But this does not necessarily mean 
I know all the ways in which this can be used :]

> I have no clue if there are any
> that dump UTF-16, but I wouldn't be surprised if there weren't. How hard
> would it be to add them? Remeber that C++11 has char16_t.

I will bring this question to the LTTNG/perf mailing lists and ask for 
guidance there. I imagine that xperf would support UTF16, considering that the 
Windows API uses it (right?). I will try to get input from an expert on that 
platform.

> > One way to handle the above case would be only doing the qUtf8Printable
> > call when the trace point is enabled, but afaik that checking also is not
> > done for free, and also makes the consumer code harder to write.
> 
> Right. Don't annotate with function calls. Annotate variables only. You can
> mostly do this by adding an unreachable rvalue-ref overload, like qAsConst.
> 
> > https://github.com/milianw/bench_qt/blob/master/bench_qstring/
> > bench_qstring.cpp#L118
> > 
> > Would something like this be acceptable for upstream? E.g. change
> > qUtf8Printable to not allocate memory for up to PATH_MAX chars or similar,
> > and otherwise fallback to the slow QByteArray (or simply truncate, which
> > would be an option for trace points actually).
> 
> That change is interesting and we could accept it, but I still wouldn't want
> it used in tracepoints. You avoid the allocation, but you're still
> converting to UTF-8. Your benchmark doesn't show it because your strings
> are probably very short and our UTF-8 algorithm is very optimised. It is
> dwarfed by the memory allocation.

OK, I will upstream this change independently of whether it will be used for 
trace points or not then.

> No, instead the policy should be that we annotate only direct variables or
> values that can be computed easily from the variable itself (like
> constData()).

I think this will severely limit the value of the trace points - even if we 
can find a way to print UTF-16 "cheaply" and thus can handle QString. I will 
try and see if there is maybe a way to opt out of calling the potentially 
costly functions in a nice to use API with minimal/no overhead and report my 
findings back to this list then. I have something like the category-based 
logging in mind ATM, where (afaik?) we short-circuit constructs like

    qCDebug(foo) << bar() << asdf();

when the category is not enabled

One example where "only" QString is not enough: the various cases where we 
operate with QUrl (such as QNetworkManager, or stuff in Qt Declarative)... In 
the customer project of ours, I wrote a wrapper around my "fast" UTF-8 
conversion and then use a static buffer with snprintf, such that you can use 
it with:

    TraceScope<...> ts("url: " QURL_TRACE_FORMAT, qUrlTraceFormat(myQUrl));

Ugly, but faster than constructing a QString for the QUrl first.

CTF (the common trace format) also seems to support structs, so we could do 
better there, i.e. get a cheap wrapper around the direct data getters of QUrl 
(protocol, host, port, user, path, query, fragment,....). But I'd have to 
check up what xperf has for that, and how to use it with sdt.h...

So, overly long email - sorry for that.

tl;dr; I will do more evaluation of the underlying trace point APIs to figure 
out how we can do this without cost

Bye, and thanks for the valuable input already

-- 
Milian Wolff | milian.wolff at kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5903 bytes
Desc: not available
URL: <http://lists.qt-project.org/pipermail/development/attachments/20160906/a6a93573/attachment.bin>