[Development] Towards a Qt 5 beta

[Rohan McGovern]

jason.mcdonald at nokia.com said:
> > Random question of the day: do you happen to have stats about how
> > often those insignificant tests actually fail? That should help to
> > figure out which ones are actually working, and therefore should not
> > be marked as insignificant.
> 
> I'm glad you asked.  I spent some time yesterday and today doing that analysis for the 40-odd insignificant tests that had no Jira task associated.
> 
> I used the data from the last eleven CI runs for each module (times eight configs for which autotests are executed).  That's not hugely statistically significant, but it's not that easy to pull large numbers of build logs out of the CI system manually through the web interface.  (Any CI folks who can tell me how to extract/search all the stored logs?)

I'm using this pattern (which has some minor problems not relevant for
any test I've looked at so far):

  from=02238
  to=02538
  prj=QtBase_master_Integration
  cfg=win32-msvc2010_Windows_7
  test=tst_QLocale
  trailing_lines=3
  for i in $(seq -w $from $to); do
      echo "$i:"
      wget --quiet -O - http://testresults.qt-project.org/ci/$prj/build_$i/$cfg/log.txt.gz |
        zcat |
        egrep -A $trailing_lines "Testing $test" ||
        echo "did not run"
      echo ""
  done 2>&1 | tee results_${test}.txt

... and scanning at least one hundred logs.

Having a transparent caching proxy between self and
testresults.qt-project.org helps a lot.