<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<div tabindex="-1" class="fEEQb BeMje TiApU allowTextSelection">
<div>
<div>
<div dir="ltr">
<div dir="ltr">
<div style="color:black;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif">
<div>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">Hello,</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><br>
</p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">I would like to present the results of the July fixing of flaky tests.</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><br>
</p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">short version:</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">19 - number of platform-specific flakiness to start with: (14 different tests) </span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><br>
</p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">as a result:</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">11 platform-specific flakiness </span><strong><span data-preserver-spaces="true">was fixed</span></strong><span data-preserver-spaces="true"> (caused by 8 different tests) </span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">4 platform specific flakiness was still </span><strong><span data-preserver-spaces="true">flaky</span></strong><span data-preserver-spaces="true"> (from 4 different tests)</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">4 cases of flakiness were</span><strong><span data-preserver-spaces="true"> blacklisted</span></strong><span data-preserver-spaces="true"> (2 different tests)</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><br>
</p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">The table under the link below shows more detailed information about fixed tests.</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><a href="https://wiki.qt.io/Fixed_flaky_tests_in_July_2022" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable" style="color:#4A6EE0" data-linkindex="0"><span data-preserver-spaces="true">https://wiki.qt.io/Fixed_flaky_tests_in_July_2022</span></a></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><br>
</p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">long version:</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><strong><span data-preserver-spaces="true">How was the problem approached?
<br>
</span></strong></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">We collected data about flakiness from June, in July we created a list of top "worst" cases that failed integrations and we contacted module maintainers. We gave some time
 for changes to be merged and run a sufficient number of times to gain confidence that the fix actually worked - and in late August we checked the results again.</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><br>
</p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">The complete lists of flaky tests from June that were being fixed in July can be found at this link:</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true"><span data-preserver-spaces="true"><a href="https://testresults.qt.io/grafana/d/000000007/flaky-summary-ci-test-info?orgId=1&from=1656626400000&to=1659304799000&viewPanel=65" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable" data-linkindex="1">https://testresults.qt.io/grafana/d/000000007/flaky-summary-ci-test-info?orgId=1&from=1656626400000&to=1659304799000&viewPanel=65</a></span><br>
</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true"><br>
</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><strong><span data-preserver-spaces="true">Which tests were taken into analysis?
<br>
</span></strong></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">The tests from dev branch that impacted negatively the integration system by causing at least 1 failure in any integrations and at least 1 flaky event.</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><br>
</p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><strong><span data-preserver-spaces="true">What is the difference between a failed and a flaky test?</span></strong></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">You can find a good explanation here:</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true"></span></p>
<p style="margin-top:0;margin-bottom:0"><span data-preserver-spaces="true"></span><a href="http://flaky_and_failed_test_definition" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable" style="color:#4A6EE0" data-linkindex="2"><span data-preserver-spaces="true">https://testresults.qt.io/grafana/d/000000007/flaky-summary-ci-test-info?orgId=1&viewPanel=55</span></a></p>
and here:</div>
<div><a href="https://testresults.qt.io/grafana/d/000000007/flaky-summary-ci-test-info?orgId=1&viewPanel=41" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable" data-linkindex="3">https://testresults.qt.io/grafana/d/000000007/flaky-summary-ci-test-info?orgId=1&viewPanel=41</a></div>
<div><br>
<p style="margin-top:0;margin-bottom:0"></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><strong><span data-preserver-spaces="true">What is understood by a "test"?</span></strong></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">A test is an umbrella term for a pair: test case and test function. A test case (usually a cpp file) contains several test functions that return results (pass, fail, or
 xfail). We collected and analyzed the results. Additionally, some tests contain data tags - test function arguments that also provide more detailed results, however, we do not store them, the granularity of the data ends at the test function level.</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><br>
</p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><strong><span data-preserver-spaces="true">What is understood by a "platform-specific flakiness"?</span></strong></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">A test runs on a specific platform - we describe it as "target operating system" and "target architecture". In most cases, flakiness is related to a particular test run
 on a specific platform. </span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">E.g., test case: "tst_qmutex" , test function "more stress" can return stable results most platforms but be flaky on: MacOS_11 X86_64 or on Windows_10_21H2 X86_64 . In
 such case, it will be counted as 2 "platform specific flakiness" (MacOS_11 and Windows_10_21H2_) caused by a single (unique, distinct) test.</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><br>
</p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">Since July fixing provided good results, in August we repeated the procedure: we gathered data about the most damaging (failing integrations) flaky tests and we compared
 it to July, to make sure only "new" tests are on the list. August's failing flakiness can be viewed under the link below.  Developers and maintainers are welcome to check if their tests are on the list.</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><a href="https://wiki.qt.io/Flaky_tests_that_caused_failures_in_August" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable" style="color:#4A6EE0" data-linkindex="4"><span data-preserver-spaces="true">https://wiki.qt.io/Flaky_tests_that_caused_failures_in_August</span></a></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><br>
</p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">Big thanks to everyone participating in fixing the tests!</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><br>
</p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">Anna Wojciechowska</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><br>
</p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><span data-preserver-spaces="true">The notebooks used to prepare this analysis can be found at:</span></p>
<p style="color:#0E101A;margin-top:0;margin-bottom:0"><a href="https://git.qt.io/qtqa/notebooks/-/tree/main/flakiness/august_2022" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable" style="color:#4A6EE0" data-linkindex="5"><span data-preserver-spaces="true">https://git.qt.io/qtqa/notebooks/-/tree/main/flakiness/august_2022</span></a></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</body>
</html>