[Qt-creator] [Development] Requesting repository for telemetry plugin in Qt Creator

[Pasi Keränen]

Hi Tobias,

I planned to stay out of this, but
-1 since I am totally confused about the scope of this project at this point.
So are we talking about a generic telemetry framework for Qt applications or about watching Qt Creator users specifically?
Tino started by making the claim that this is a generic library and then kept listing Qt Creator specific integrations. You focus entirely onto the generic part, leaving out creator completely.
What exactly are we talking about here?

My understanding at this point is that this is meant to be a telemetry framework for Qt applications and first integration point that has been in discussions is Qt Creator. I intentionally left Qt Creator out as I don’t feel comfortable talking what they need/don’t need. So instead I concentrated on what I see Qt 3D Studio needs as I feel much more comfortable to talk about that based on our experiences, our engagements with customers and the usability and UI work we’ve done so far in Qt 3D Studio project.

I’m always in favor of iterative development as you very very rarely understand the scope and needs at the beginning. So I understand the need to have a “reference integration to try out” even if we talk about aiming towards generic telemetry plugin. And it seems quite obvious from these discussions that we don’t yet know ourselves a lot about how to collect data, what type of data is to be collected and how to process it. We seem to be a bit divided in to individuals thinking this is inherently bad and morally dubious. And individuals (like myself) who see this as great idea to be able to get more data on how people use our tools. Not just from big customers and people who come to our events. But also from people out there in universities, small companies, startups, the people we rarely meet in person.

Once we have understanding on what exactly are we after, then it would be good to move towards a generic Qt framework prototype. Maybe integrate against a known and trusted third party framework, add features to the API iteratively based on learnings and iterations. Then move on to publish this as tech preview and while we investigate integrating against another 3rd party framework so that we end up with a truly generic, Qt-like wrapper for specific functionality.

A Qt creator spyware plug-in (which does more than what we discuss here I hope:-) would be a matter of a couple of weeks to do, putting a generic framework into place with all the bits and pieces for that to be actually useful in a wide list of possible contexts is a very different beast.

I object to using “Spyware” term in this context. Spyware is SW that does things behind your back. Siphons contact information, user account info etc. without telling you. We’re NOT talking about such use cases here! I don’t think anyone in The Qt Company wants to do such things. From what I hear the intention is to track certain things in an open, transparent manner, respecting the communitys clear wish to keep things in the open and for the benefit of our end users.


This is great initiative and very much the way today's app and application industry works. UX studies performed by UX experts have been minimized and targeted for specific (usually new/experimental) features or upcoming new software (like we did with Qt 3D Studio back in last spring). And the mass information on "how do our users use the SW? do they find the stuff we've put in there? how often they hit a wall in doing sequence X? how many crashes do they experience when doing Y?" is collected via automated telemetry. It is great as it brings data from the actual user in their actual work and you can then use that to concentrate on functionality that really matters to your users. Making stuff they repeat hundred times a week easier and faster, making them more productive.

I see value in this approach when you can do lots of small releases fast. So you can do evaluate the effect of small changes by doing one change to evaluate per release and measure how that effects usage.
We can not do more than two releases per year in Qt. Is this approach even applicable to us?

Qt 3D Studio plans to do 4 releases per year, so for us this is definitely applicable. Qt Creator does also more than 2 releases a year so perhaps this could be useful for them as well, but as I said. I’m not comfortable talking on their behalf on this.

I want to also point out that answering any of the questions you used as an example require *way* more information than I am even remotely comfortable to collect.


This is where we definitely differ as individuals. I use software from Apple, Adobe etc. and I’m 100% fine giving them information on my usage statistics as it means the SW might get better over time for me. If e.g. Blender would some day ask me “are you ok for us to track your usage of the SW” I’d be 100% ok to do that as I see that it definitely could improve a lot from observing users. I see the poitive side of this type of usability and user data collection. So taking this in to account, perhaps our solution should somehow enforce possibility to opt-in/opt-out to accommodate for both you and me?

I agree that having a general framework has value to some customers.

Good to hear we’re in agreement on some level at least! :)

Such a framework would need to be integrated into big solutions like Google analytics or something similar so the users can actually evaluate the collected data and cross-reference that to other data they collect. Just dumping data into some server somewhere does not help anybody.
Will the server-side code be part of this project by the way?
What about evaluation of the collected data? Is that in scope for this project, too?

All these are good questions that I’d rather not try to answer in a meeting/email thread/chat thread, but rather prototype hands on iteratively. Just may way of approaching stuff and having experienced so many frustrating analysis paralysis moments and moments where theoretical discussions have proven something “absolutely impossible” only for competition then coming out with that exact feature a few moments later. Just try to do it and see how it goes, collect experience and modify your target setting based on that. Listen to customers, community and you shouldn’t go horribly wrong.

Let us do some usability studies, let us talk to users, let us do surveys, let's evaluate all the data users provide to us on the mailing list, the bug tracker, the forums, stack overflow and in lots of other places. We do have a very active and supportive community of users and customers, let us use the feedback they provide already!

Yup, we’ve done usability studies with Qt 3D Studio, we’ve done talking with SOME key customers and yes we’ve even fixed a lot of issues they’ve pointed out in Qt 3D Studio. I’m watching our bug tracker all the time, it’s part of the daily work. But let’s be honest.. we CAN’T talk to every user out there, only the biggest ones, only the ones that happen to participate our events. Qt 3D Studio is not talking to probably 95-97% of our users at the moment as we just can’t. Even this mailing list doesn’t cover all of our users. And crawling through all these email threads, no thanks, that is very very inefficient use of time in my opinion. This forum is great for discussing ideas, formulating, pointing out what needs to be considered (like in this case, it’s loud and clear that privacy and opt-in are of great importance to our community for atelemetry feature). Data collection would (when implemented correctly) automate the input collection and bring in data from all over the user base, that is why I see great value in it.

Collecting some semi-random data from a self-selected group of users and dumping that onto a server somewhere is next to useless compared to all the other options we have already. Even in an ideal situation (which we do not have within Qt!), metrics are not comparable to a real user study where you actually watch users doing their thing.

Perhaps first version will start with a set of data that might turn out to be ”semi random” and “not so useful for understanding users”. But I believe in learning, iterating and improvement. Next iteration we should improve and collect less random set of data that also then helps us in understanding our users better. And so on and so on.

Yes I agree, metrics are ADDITIONAL tool. Not replacement for anything. Visiting users and talking with them does sometime open a whole new point of view in to what you are doing. Those can provide revolutionary ideas for you. That isn’t replaced with metrics. But metrics are great for iterative development, as you point out above. Honing and fine tuning the set of functionality you already have in your application.

Regards,
Pasi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/qt-creator/attachments/20180223/456bef34/attachment.html>