[Qt-creator] Unit testing private parts of shared libraries

[Elvis Stansvik]

2017-10-17 10:36 GMT+02:00 Nikolai Kosjar <nikolai.kosjar at qt.io>:
> On 10/16/2017 06:57 PM, Elvis Stansvik wrote:
>>
>> Hi all,
>>
>> (Posting this as a new thread instead of necroposting to my old thread
>> about the design of Qt Creator [1], which did end with some
>> discussions about testing.)
>>
>> I'm working on some tests for my own application, and started thinking
>> about unit testing of private parts of shared libraries (think the
>> _p.h/_p.cpp parts). Such parts are normally not exported AFAIK (in the
>> visibility attribute sense). This becomes a problem if you link the
>> unit tests dynamically against the code under test, as the tests won't
>> be able to see those symbols.
>>
>> So it seems to me there is only a few options:
>>
>>   1. Exporting the private parts after all,
>>   2. Adding the sources of the code under test to the unit tests,
>>   3. ?
>>
>> 1 is not really good, as you export more than you really have to. But
>> I guess it's not so bad after all, since you a) have to include _p.h
>> to get at them and b) they're normally called PrivateSomething or
>> InternalSomething, both of which should tell you you're doing
>> something bad. 2 is not really good either though, as you really want
>> to test the code as shipped, not as compiled as part of your test.
>
>
> Note that while "test the code as shipped" is very valuable, it does not
> scale very well (if used exclusively) - in the long run it can take quite
> some time to run those tests. If they take too long, you will less often run
> them as you probably should (or they can help you) and forget it at times.
> Eventually you might find they even stay in your way :>

Ah, sorry, I think you may have misunderstood me.

With "as shipped" vs "not as shipped" I was merely talking about
testing against the actual compiled library that will be shipped. It
was not about what code paths are exercised or testing methodologies
or anything like that. Just that there's a difference between having a
test link against your shared library (the exact one that will be
shipped) compared to compiling the sources again just for the sake of
the test, or #including the implementation. Testing against another
compilation of the code always brings some risk of behavioral
differences compared to the one that will be used in the wild (due to
compiler flags, different defines, ..). It's a small risk, but it's
there.

>
> That's why some proposed definitions of "unit testing" go to the extreme and
> use test doubles (fake, stub, mock...) where possible. "test the code as
> shipped"-tests are then considered as integration tests and are in quantity
> significantly less then those light-weight "mocked" unit tests. Downside
> with those unit tests are the needed extra interfaces/templates and what not
> which can make the real code look more complicated/abstract than actually
> necessary.

Yes, I do use mocking myself, and indeed it's a trade-off between the
value of being able to test code in isolation and "polluting" the code
with interfaces/templatization only for the purpose of testability.

Elvis

>
>> I had a quick look at Qt Creator because I know you folks are seasoned
>> developers, and it looks like you've gone with option 1. For example,
>> ExtensionSystem::Internal::PluginSpecPrivate is exported, and I guess
>> the reason is you want to be able to unit test it?
>>
>> I just wanted to ask in case you have any other ideas or better ways
>> to let unit tests access things that are normally not exported?
>>
>> Thanks in advance,
>> Elvis
>>
>> [1]
>> http://lists.qt-project.org/pipermail/qt-creator/2017-September/006712.html
>> _______________________________________________
>> Qt-creator mailing list
>> Qt-creator at qt-project.org
>> http://lists.qt-project.org/mailman/listinfo/qt-creator
>>
>