[Development] Flakey tests heatmap analysis

[Tor Arne Vestbø]

Hello,

> On Oct 24, 2024, at 03:15, EXT Mitch Curtis <mitch.curtis at qt.io> wrote:
> 
> Hi,
> 
> Once I've gotten into a specific filter (i.e. https://testresults.qt.io/grafana/d/de08wetl3yq68f/Test%20failure%20heatmaps?var-integration=149569&var-table_name=blacklisted_tests&var-project=qt%2Fqtdeclarative&var-config=macos-13-arm64-developer-build-tests), is there a way to see a list of test functions (so that I can see which has the most occurrences), or is that not what this is for? I tried clicking on various things but nothing happened.

Yes, if you scroll down there’s four sections:

- Blacklisted test functions/data tags by configuration and module
- Blacklisted test suites by configuration and module
- Blacklisted test functions/data tags by test suite and module
- Blacklisted test functions/data tags in qt/qtdeclarative on macos-13-arm64-developer-build-tests

The last one is the one you’re looking for.

Cheers,
Tor Arne


> 
>> -----Original Message-----
>> From: Development <development-bounces at qt-project.org> On Behalf Of
>> Tor Arne Vestbø via Development
>> Sent: Friday, 11 October 2024 10:26 PM
>> To: development <development at qt-project.org>
>> Subject: [Development] Flakey tests heatmap analysis
>> 
>> Hi all,
>> 
>> We had a session during QtCS about flakey tests
>> (https://wiki.qt.io/QtCS2024_Flaky) where we discussed how we could gain
>> insights about which areas and platforms we are struggling the most.
>> 
>> Based on Jimis' excellent work on a Postgres database for our test results I
>> built a heatmap dashboard that might be helpful in analyzing things:
>> 
>> 
>> 
>> https://testresults.qt.io/grafana/d/de08wetl3yq68f/tor-arne-s-heatmap-
>> experiment?var-integration=149569&var-table_name=blacklisted_tests&var-
>> config=All&var-project=All&orgId=1
>> 
>> You can click around to drill down, and it also supports listing skipped tests
>> (but takes a while to load the data).
>> 
>> Let me know if there are suggestions on improvements.
>> 
>> Cheers,
>> Tor Arne