[Development] Important recent changes in QList/QString/QByteArray

[Daniel Engelke]

Hey,


ASan for x86 Release targets is available in MSVC since https://devblogs.microsoft.com/cppblog/addresssanitizer-asan-for-windows-with-msvc/ , and recently also for x64 + Debug builds https://devblogs.microsoft.com/cppblog/asan-for-windows-x64-and-debug-build-support/
To add some details.


BR,
Dan


---

Daniel EngelkeSoftware Engineer
basysKom GmbH
Robert-Bosch-Str. 7 | 64293 Darmstadt | Germany
Tel: +49 6151 870 589 -0 | Fax: -199
daniel.engelke at basyskom.com | www.basyskom.com


Handelsregister: Darmstadt HRB 9352
Geschaeftsfuehrende Partner: Heike Ziegler, Alexander Sorg



 From:   Andrei Golubev <andrei.golubev at qt.io> 
 To:   "development at qt-project.org" <development at qt-project.org> 
 Sent:   8/31/2020 2:01 PM 
 Subject:   [Development] Important recent changes in QList/QString/QByteArray 


 
 
 
 Hello all, 
 
 
 
 
 Over the last week I pushed several patches to enable prepend optimization in some core containers, namely, QList, QString, QByteArray. 
 In the light of these changes, there may occur new subtle issues in your code. 
 
 
 
 
 New model
 
 ---
 
 
 
 In a nutshell, prepend optimization code does some magic with the memory layout and the stored elements location. 
 This, in turn, affects how you can and cannot use iterators/pointers/references to container's elements and their invalidation. 
 
 
 As of today, please consider that any non-const member function of the aforementioned container can invalidate "remembered" iterators, pointers or references to elements of that container. For example, an iterator is returned by QList::begin(), a pointer is  returned by QString::data() (and QString::constData()), a reference is returned by QList::at() - this is, of course, not an exhaustive list.
 
 The invalidation existed before for cases when a container could detach (due to copy-on-write) or reallocate (due to growing or squeezing). Now this is also true for non-detaching, non-reallocating, modifying operations. Examples of such operations: append,  prepend, insert, emplace, remove, etc.
 
 
 
 
 
 How this affects existing code? 
 --- 
 
 
 To the best of my knowledge, at least certain documentation pieces of QString and QByteArray already explained such a thing. So, hopefully, diligent users of these 2 containers should be safe.
 
 
 
 QList's behavior changed with regards to the iterators/pointers: as described above, using an iterator or a pointer to an element of a QList between non-const modifying operations is an undefined behavior (same true for references).
 
 
 
 A more subtle issue concerns the users of QVector. Since QVector got unified with QList and QList's implementation/behavior changed, old code that was valid before may now be a subject to UB. Consider a simple example of the difference: 
 
 
 QVector<int> ints { 0, 1, 2, 3, 4, 5, ... };  // now: effectively QList<int>
 
 auto thirdElemIt = ints.begin() + 2; 
 ints.remove(6);   // removing element at position 6. before: thirdElemIt remains valid 
                            //                                                       now: thirdElemIt may be invalidated
 
 print(*thirdElemIt);  // before: valid enough behavior; now: undefined
 
 
 
 
 
 
 Aiding the debugging process
 
 --- 
 
 
 Below are some basic steps one can do to simplify the debugging of issues:
 
 
 
 1) Enable assertions in the code. This should be possible even if you have a release build, there's a macro definition QT_FORCE_ASSERTS that should force-enable assertions (albeit not sure how exactly to specify it during build stage). 
 
 
 2) Build with address sanitizer. Even basic "sanitize=address" should suffice initially. It should be available both for GCC and Clang (guessing that any recent GCC/Clang that supports C++17 also supports ASan). MSVC developers some time ago also reported that  ASan is now supported. To enable ASan locally with CMake build toolchain, one can use "-DFEATURE_sanitize_address=ON" flag as part of the CMake input arguments (when building Qt sources). Don't know an alternative if you use configure script + qmake. 
 
 
 3) Valgrind's memory checker should cover roughly same issues as address sanitizer.
 
 
 
 Note that you do not need to rebuild Qt/your application to use valgrind which may be a preferred option sometimes. 
 
 
 
 
 
 
 Feel free to contact me in case you find an issue caused by new changes and need a help in resolving it. 
 Sorry for possible inconveniences!
 
 
 
 
 
 --
 
 Best Regards, 
 Andrei
 

_______________________________________________
Development mailing list
Development at qt-project.org
https://lists.qt-project.org/listinfo/development
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20200901/1bd8ca2b/attachment.html>