[Qbs] GCC: --start-group vs --whole-archive

[Christian Gagneraud]

On 24 May 2018 at 20:39, Christian Kandeler <christian.kandeler at qt.io> wrote:
> On Thu, 24 May 2018 17:12:26 +1200
> Christian Gagneraud <chgans at gmail.com> wrote:
>
>> I'm trying to link internal static libraries to an application, Qbs
>> made the choice to give the linker command a list of full path to the
>> library archives.
>> Because I cannot rely on proper dependency tracking b/w all the static
>> libraries, I get link errors due to how ld works: it picks up only
>> unseen symbols as it process the list of archives, which is different
>> from what msvc does: take all the static libs in a global pool and
>> make sure nothing is missing.
>>
>> My list of dependencies is "complete", but libs might have cyclic deps
>> (!) and i cannot sort out what is the proper link ordering.
>
> Why do they have cyclic dependencies? Did you design them this way?

They grew up like this, i guess because VisualStudio+msvc doesn't care
about this sort of things.
I know that this is bad, please don't start telling me that, you would
be preaching a converted.
This won't be addressed any time soon, first of all b/c this is not a
problem, as long as the set of required static libraries is given to
VisualStudio+msvc or qmake+gcc  so that an application can be linked.

Actaually nobody really knows if there are cyclic dependencies or not
(just heavy suspicion. Eg, bfd linker running out of memory on
machines that have 0.25TB of RAM) since they are considered as "a bag
of dependencies", without individual inter-depedency tracking.
Again, qmake can cope with that, and actually cmake can too (we use to
have a proof of concept, but is not maintained). So why not Qbs?

>> Both VisualStudio+msvc and QMake+gcc can deal with that.
>
> And qbs + msvc, I would assume?

I do not know, for now i'm just trying to get it working on x86-linux.
But i highly suspect that this statement is correct.
It all boil downs to GNU linker vs MSVC linker.
GNU picks up symbols as needed and discard the rest, for each library,
in given order, MSVC fills up a global symbol table from all
dependencies and then picks what it needs.

>> I have tried to use the linkWholeArchive this way:
>> Application {
>>  name: "Foo"
>>  Depends { name: "cpp" }
>>  cpp.linkerFlags: ["-fuse-ld=gold"] // bfd runs out of memory
>>  files: [ ... ]
>>  Depends { name: "Qt" ... }
>>  Depends { name: "Bar"; cpp.linkWholeArchive: true }
>>  Depends { name: "Baz"; cpp.linkWholeArchive: true }
>>  // And more ...
>> }
>>
>> My static library only declare dependencies on Qt, and they don't use
>> any "Export".
>>
>> Qbs generate a command-line using "-Wl,--whole-archive
>> /path/to/libBar.a /path/to/libBaz.a --no-whole-archive", but the
>> linker failed with weird messages about both missing symbols and
>> duplicated symbols.
>
> What kind of symbols are these?

I don't have the list right here, but i can tell you that it looked
really weird!
And again the list of static library is correct (msvc, qmake, cmake
can cope with that), and the order in which they appear in the qbs
file is the same as in the pro file or the vcxproj file.
I do agree that this is a degenerated use-case, but i'm not the first
one to ask for that.
The cpp.linkWholeArchive feature came from a user request for a very
similar scenario.

>> If I copy the generated command in the terminal and replace
>> --whole-archive with --start-group, and --no-whole-archive with
>> --end-group, then the link succeed (and is way faster). AFAIU, this
>> allows GNU ld to behave like msvc.
>>
>> Another way to get the linkage working is to replace all the
>> /path/to/libXYZ.a with -L /path/to -l XYZ. This is what qmake is
>> doing, and again it works well (same codebase). I know that doing that
>> can cause problems if the linker has the choice of shared and archived
>> libs, but in my case, "it just works" b/c we only build static
>> libraries.
>
> In which version of your command line did you do that change? The one with --whole-archive?

Yes, just copy/paste the command line generated by Qbs, replace
--whole-archive with --start-group and --no-whole-archive with
--end-group.
As all my in-project static dependencies all use
"cpp.linkWholeArchive: true" Qbs grouped them and generate a single
--whole-archive and a single --no-whole-archive, in the middle of them
are all my /path/to/libFoo|Bar|Baz.a

>
>> Is there a chance i can tweak qbs so that my scenario works?
>
> If you really have cyclic dependencies between your libraries, then obviously that defeats the whole dependency management approach. In such a case, you'll probably have to go with manually specifying the libraries using cpp.linkerFlags, including --start-group and --end-group.

Great! I do understand that Qbs try to be very smart when it comes to
depedencies, but maybe it's too restrictive.
I could build a program with GNU make with a broken dependency schema,
actually a lot of projects are broken in that respect. It usually
manifests when you build them on machine with "plenty" of cores.
QMake has the CONFIG+=ordered for that. It sub-efficient (each module
gets fully build one after another), but it allows you to solve
real-life problem.

Chris