[Development] [Qt-creator] Requesting repository for telemetry plugin in Qt Creator

[Tobias Hunger]

Hi Pasi,

On Feb 23, 2018 08:05, "Pasi Keränen" <pasi.keranen at qt.io> wrote:

Hi there,

+1 for having a generic telemetry plugin in Qt.


I planned to stay out of this, but

-1 since I am totally confused about the scope of this project at this
point.

So are we talking about a generic telemetry framework for Qt applications
or about watching Qt Creator users specifically?

Tino started by making the claim that this is a generic library and then
kept listing Qt Creator specific integrations. You focus entirely onto the
generic part, leaving out creator completely.

What exactly are we talking about here?

A Qt creator spyware plug-in (which does more than what we discuss here I
hope:-) would be a matter of a couple of weeks to do, putting a generic
framework into place with all the bits and pieces for that to be actually
useful in a wide list of possible contexts is a very different beast.

This is great initiative and very much the way today's app and application
industry works. UX studies performed by UX experts have been minimized and
targeted for specific (usually new/experimental) features or upcoming new
software (like we did with Qt 3D Studio back in last spring). And the mass
information on "how do our users use the SW? do they find the stuff we've
put in there? how often they hit a wall in doing sequence X? how many
crashes do they experience when doing Y?" is collected via automated
telemetry. It is great as it brings data from the actual user in their
actual work and you can then use that to concentrate on functionality that
really matters to your users. Making stuff they repeat hundred times a week
easier and faster, making them more productive.


I see value in this approach when you can do lots of small releases fast.
So you can do evaluate the effect of small changes by doing one change to
evaluate per release and measure how that effects usage.

We can not do more than two releases per year in Qt. Is this approach even
applicable to us?

I want to also point out that answering any of the questions you used as an
example require *way* more information than I am even remotely comfortable
to collect.

I see definite need for this in Qt 3D Studio so please don’t make this just
with Qt Creator in mind. Also, in my humble opinion, in order to be
relevant in today's UI development, Qt should also offer this kind of a
plug-in to our customers. A ready-to-go plug-in that automatically ensures
the data is collected in a way that fulfills data privacy acts and respects
the privacy of the user would be great. Especially for startups and smaller
companies, but also for bigger companies wanting to switch to the modern
way of doing UI development. It is not as easy to do as one might think at
glance.


I agree that having a general framework has value to some customers.

Such a framework would need to be integrated into big solutions like Google
analytics or something similar so the users can actually evaluate the
collected data and cross-reference that to other data they collect. Just
dumping data into some server somewhere does not help anybody.

Will the server-side code be part of this project by the way?

What about evaluation of the collected data? Is that in scope for this
project, too?

I would ask anyone who has not done work with usability and user experience
people in the past to give this way of working a chance. I've worked 7
years in application development while we grew usability knowledge in the
team over that time. The first time I got to observe a real world user
working with our software in actual real world situation was eye opening.
We had gotten so many important things wrong in our idealistic thinking and
forgotten to handle certain cases that occur on the field. Also you become
blind to your own creations faults as you just know how the software works.
It's just a fact of life.


Sure. We do way too little validation of what we do against the real world
usage.

Let us do some usability studies, let us talk to users, let us do surveys,
let's evaluate all the data users provide to us on the mailing list, the
bug tracker, the forums, stack overflow and in lots of other places. We do
have a very active and supportive community of users and customers, let us
use the feedback they provide already!

Collecting some semi-random data from a self-selected group of users and
dumping that onto a server somewhere is next to useless compared to all the
other options we have already. Even in an ideal situation (which we do not
have within Qt!), metrics are not comparable to a real user study where you
actually watch users doing their thing.

This is even more true when not leaving out any information that can be
used to identify individual users from the collected data, which we
obviously will do.

Best Regards,
Tobias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20180223/d00ba19e/attachment.html>