[Development] Programs crashing left and right in the CI

Jędrzej Nowacki jedrzej.nowacki at qt.io
Tue Jul 26 11:22:04 CEST 2016


On Monday 25 of July 2016 10:27:30 Thiago Macieira wrote:
> On segunda-feira, 25 de julho de 2016 08:51:36 PDT Thiago Macieira wrote:
> > On segunda-feira, 25 de julho de 2016 14:07:04 PDT Jędrzej Nowacki wrote:
> > > Ugh I haven't seen link.exe crashing, but in case of moc I have
> > > impression
> > > that it is a software problem. I remember that first time I have seen it
> > > I
> > > was  quite surprised, but now it is quite common and it happens mostly
> > > (only) on OSX (Clang miss-compilation?). If it would be memory then I
> > > would
> > > expect to see compiler crashing more often and I have experienced it
> > > maybe
> > > once...
> > > 
> > > Any other idea what can go wrong or what can be checked?
> > 
> > The fact that it is consitently crashing on moc indicates the problem is
> > actually moc or the compiler. You're also right that most of the crashes
> > are on macOS. I can't remember if I've seen it on other OSes. This
> > link.exe crash is a one-off and I haven't seen others.
> > 
> > Let's pay attention of what crashes and where. But we'll need to reproduce
> > this in order to debug it.
> 
> tst_qdir.exe crashed on Windows, on Qt 5.6:
> 
> A crash occurred in C:\Users\qt\work\qt\qtbase\tests\auto\corelib\io\qdir
> \release\tst_qdir.exe.
>  Function time: 2728ms Total time: 2731ms
> 
>  Exception address: 0x00007FF8DB7367B4
>  Exception code   : 0xc0000005
>  PASS   : tst_QDir::entryList(Sorting QDir::Type | QDir::DirsFirst)
>  PASS   : tst_QDir::entryList(Sorting QDir::Size)
>  PASS   : tst_QDir::entryList(Sorting QDir::Size | QDir::Reversed)
>  SKIP   : tst_QDir::entryListTimedSort() /bin/touch not found
>  tst_qdir.cpp(869) : failure location
>  PASS   : tst_QDir::entryListSimple(data2)
>  PASS   : tst_QDir::entryListSimple(simple dir)
>  PASS   : tst_QDir::entryListSimple(simple dir with slash)
>  Nearby symbol    : DllUnregisterServer
> 
>  Stack:
>  #  1: QTest::toString() - 0x00007FF8DEC08CD0
>  #  2: UnhandledExceptionFilter() - 0x00007FF8E4001AD0
>  #  3: TpDbgDumpHeapUsage() - 0x00007FF8E6D01BE0
>  #  4: TpDbgDumpHeapUsage() - 0x00007FF8E6D01BE0
>  #  5: memset() - 0x00007FF8E6C95140
>  #  6: _C_specific_handler() - 0x00007FF8E6C82800
>  #  7: wcstok_s() - 0x00007FF8E6C8D8C0
>  #  8: _chkstk() - 0x00007FF8E6C93E70
>  #  9: RtlRaiseException() - 0x00007FF8E6C53920
>  # 10: KiUserExceptionDispatcher() - 0x00007FF8E6C93060
>  # 11: DllUnregisterServer() - 0x00007FF8DB7097B0
>  # 12: DllUnregisterServer() - 0x00007FF8DB7097B0
>  # 13: DllUnregisterServer() - 0x00007FF8DB7097B0
>  # 14: DllGetActivationFactory() - 0x00007FF8DC9D1860
>  # 15: DllGetActivationFactory() - 0x00007FF8DC9D1860
>  # 16: DllGetActivationFactory() - 0x00007FF8DC9D1860
>  # 17: DllGetActivationFactory() - 0x00007FF8DC9D1860
>  # 18: DllGetActivationFactory() - 0x00007FF8DC9D1860
>  # 19: SwMemFree() - 0x00007FF8E3E29770
>  # 20: SwMemFree() - 0x00007FF8E3E29770
>  # 21: QueryProtectedPolicy() - 0x00007FF8E3F45C80
>  # 22: RtlGetActiveActivationContext() - 0x00007FF8E6C1C6B0
>  # 23: RtlFreeUnicodeString() - 0x00007FF8E6C37100
>  # 24: BaseThreadInitThunk() - 0x00007FF8E44B13B0
>  # 25: RtlUserThreadStart() - 0x00007FF8E6C15410
> 
> http://testresults.qt.io/coin/integration/qt/qtbase/tasks/
> 1469463068.thrift_bin

tst_qdir is known to be "ugly" it was backlisted for months and probably re-
added https://bugreports.qt.io/browse/QTBUG-50835 recently. The bug mentioned 
CI load, we see increased failure rate when the machinery is over a bigger 
load (~140 vms), but there are always more or less the same tests failing. So 
my interpretation is that tests/qt has a race condition that is exposed if IO 
gets slower. CPU and RAM are reserved separately without overallocation for 
every machine, so they do not overlap too much*.

So let's concentrate on code / apps that should be rock solid which are 
compilers, linkers and such.

Cheers,
 Jędrek

*CPU reservation is done in Mhz instead of cores, which may cause problems we 
try different tricks to enforce CPU affinity, but they are just "hints" to the 
VSphere scheduler.





More information about the Development mailing list