[Development] QtCS 2017 logging/tracing session notes

[Christian Gagneraud]

Hi,

Going through the wiki, the pdf and the codereview again, i see 3
different things:
- Qt: Logging framework
- ModMed: real-time dissection of organised/structured logs
- Ltt/ETW: event tracing/profiling

Different needs, different means, different reasons and different goals.

I like all 3! ;)

But i'm a bit worry of perf inpact on Ltt/ETW, if one wanted to
integrate them into the "structured" logging framework.
I've never heard of ETW before, so i won't comment. Regarding Ltt, I
would say that their primary design goal is stealthiness, the second
one being efficiency.
In the kernel mode case, you access the events from /proc, which is
backed by a kernel RCU list/buffer, i don't know how they have
implemented their userspace solution, but i'm expecting something
"pretty well done".

My point is that Ltt doesn't interfere (much) with the observed
system, and it's I think a primary quality that cannot be thrown away.
Letting Ltt traces go though the whole qDebug stuff makes me feel
nervous, traces shouldn't go though Qt locks.

Concerning the pending code review, what will happen next? Will this
be merged somewhere?
What if I want to share some experiments about Qt event tracing on top
of this review? (time permitting)
Now that the framework is there, it should be quite easy to pollute Qt
with new traces. I used pollute on purpose, b/c that's really the
drawback with Ltt, you have to pollute your code with macros at key
locations.

On the side: ModMed's ParTraP language looks very interesting, could
this be made accessible to Ltt/ETW?

Chris

On 10 October 2017 at 22:48, Thiago Macieira <thiago.macieira at intel.com> wrote:
> == Discussion ==
> * Structured logging:
> ** We want to extract information from logs and detect common messages. For
> example:
> {|
> ! From
> ! To
> |-
> |
>     Got message "Hello"
> |
>     Got message "%s"       Hello
> |-
> |
>     Got message "World"
> |
>     Got message "%s"       World
> |}
> ** We want to allow logging to store ancillary data, possibly in machine-
> readable format, likely not part of the message itself
> *** Like journald API can do
> *** How do we allow different formatting depending on backend? Needs to
> support formatting user types too.
> * Structured output
> ** Store to databases, for example
> ** Do we still obey QT_MESSAGE_PATTERN?
> ** Select backend with environment variable
>
> Need to review with [https://codereview.qt-project.org/185287 LTTNG and ETW
> tracing]
>
> ==== Requirements? ====
> performance similar to the current state (or provide choice)
>
> streaming generates a lot of code
>
> do not change qdebug but provide structured tracing as a new facility
>
> == Welcome text ==
>
> Qt-based devices such as Medical Cyber-Physical Systems could be developed,
> verified and validated more efficiently using structured logs. We suggest Qt
> Logging enhancements that would facilitate both human exploration and
> automated analysis with Python or other tools. The session will allow:
> * gathering your feedback
> * examining which enhancements would be useful to Qt
> and how to contribute them...
>
> ==== Enhancements? ====
> extended handler with format
>
> less literals in tracepoints
>
> TSV+JSON vs text-only
>
> Bind<TUserData>
>
> format-free handler
>
> --
> Thiago Macieira - thiago.macieira (AT) intel.com
>   Software Architect - Intel Open Source Technology Center
>
> _______________________________________________
> Development mailing list
> Development at qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development