[Development] Programs crashing left and right in the CI

[Jędrzej Nowacki]

On Monday 25 of July 2016 10:27:30 Thiago Macieira wrote:
> On segunda-feira, 25 de julho de 2016 08:51:36 PDT Thiago Macieira wrote:
> > On segunda-feira, 25 de julho de 2016 14:07:04 PDT Jędrzej Nowacki wrote:
> > > Ugh I haven't seen link.exe crashing, but in case of moc I have
> > > impression
> > > that it is a software problem. I remember that first time I have seen it
> > > I
> > > was  quite surprised, but now it is quite common and it happens mostly
> > > (only) on OSX (Clang miss-compilation?). If it would be memory then I
> > > would
> > > expect to see compiler crashing more often and I have experienced it
> > > maybe
> > > once...
> > > 
> > > Any other idea what can go wrong or what can be checked?
> > 
> > The fact that it is consitently crashing on moc indicates the problem is
> > actually moc or the compiler. You're also right that most of the crashes
> > are on macOS. I can't remember if I've seen it on other OSes. This
> > link.exe crash is a one-off and I haven't seen others.
> > 
> > Let's pay attention of what crashes and where. But we'll need to reproduce
> > this in order to debug it.
> 
> tst_qdir.exe crashed on Windows, on Qt 5.6:
> 
> A crash occurred in C:\Users\qt\work\qt\qtbase\tests\auto\corelib\io\qdir
> \release\tst_qdir.exe.
>  Function time: 2728ms Total time: 2731ms
> 
>  Exception address: 0x00007FF8DB7367B4
>  Exception code   : 0xc0000005
>  PASS   : tst_QDir::entryList(Sorting QDir::Type | QDir::DirsFirst)
>  PASS   : tst_QDir::entryList(Sorting QDir::Size)
>  PASS   : tst_QDir::entryList(Sorting QDir::Size | QDir::Reversed)
>  SKIP   : tst_QDir::entryListTimedSort() /bin/touch not found
>  tst_qdir.cpp(869) : failure location
>  PASS   : tst_QDir::entryListSimple(data2)
>  PASS   : tst_QDir::entryListSimple(simple dir)
>  PASS   : tst_QDir::entryListSimple(simple dir with slash)
>  Nearby symbol    : DllUnregisterServer
> 
>  Stack:
>  #  1: QTest::toString() - 0x00007FF8DEC08CD0
>  #  2: UnhandledExceptionFilter() - 0x00007FF8E4001AD0
>  #  3: TpDbgDumpHeapUsage() - 0x00007FF8E6D01BE0
>  #  4: TpDbgDumpHeapUsage() - 0x00007FF8E6D01BE0
>  #  5: memset() - 0x00007FF8E6C95140
>  #  6: _C_specific_handler() - 0x00007FF8E6C82800
>  #  7: wcstok_s() - 0x00007FF8E6C8D8C0
>  #  8: _chkstk() - 0x00007FF8E6C93E70
>  #  9: RtlRaiseException() - 0x00007FF8E6C53920
>  # 10: KiUserExceptionDispatcher() - 0x00007FF8E6C93060
>  # 11: DllUnregisterServer() - 0x00007FF8DB7097B0
>  # 12: DllUnregisterServer() - 0x00007FF8DB7097B0
>  # 13: DllUnregisterServer() - 0x00007FF8DB7097B0
>  # 14: DllGetActivationFactory() - 0x00007FF8DC9D1860
>  # 15: DllGetActivationFactory() - 0x00007FF8DC9D1860
>  # 16: DllGetActivationFactory() - 0x00007FF8DC9D1860
>  # 17: DllGetActivationFactory() - 0x00007FF8DC9D1860
>  # 18: DllGetActivationFactory() - 0x00007FF8DC9D1860
>  # 19: SwMemFree() - 0x00007FF8E3E29770
>  # 20: SwMemFree() - 0x00007FF8E3E29770
>  # 21: QueryProtectedPolicy() - 0x00007FF8E3F45C80
>  # 22: RtlGetActiveActivationContext() - 0x00007FF8E6C1C6B0
>  # 23: RtlFreeUnicodeString() - 0x00007FF8E6C37100
>  # 24: BaseThreadInitThunk() - 0x00007FF8E44B13B0
>  # 25: RtlUserThreadStart() - 0x00007FF8E6C15410
> 
> http://testresults.qt.io/coin/integration/qt/qtbase/tasks/
> 1469463068.thrift_bin

tst_qdir is known to be "ugly" it was backlisted for months and probably re-
added https://bugreports.qt.io/browse/QTBUG-50835 recently. The bug mentioned 
CI load, we see increased failure rate when the machinery is over a bigger 
load (~140 vms), but there are always more or less the same tests failing. So 
my interpretation is that tests/qt has a race condition that is exposed if IO 
gets slower. CPU and RAM are reserved separately without overallocation for 
every machine, so they do not overlap too much*.

So let's concentrate on code / apps that should be rock solid which are 
compilers, linkers and such.

Cheers,
 Jędrek

*CPU reservation is done in Mhz instead of cores, which may cause problems we 
try different tricks to enforce CPU affinity, but they are just "hints" to the 
VSphere scheduler.