[Qbs] Improving qbs resolve performance

[Christian Gagneraud]

On Sat, 13 Jul 2019 at 00:48, Christian Kandeler
<Christian.Kandeler at qt.io> wrote:
>
> On Fri, 12 Jul 2019 12:05:48 +0000
> Maximilian Hrabowski <hrabowski at clausmark.com> wrote:
>
> > >> I have a fairly large project with a root projects that pulls in several SubProjects, altogether with unit tests its about 313 qbs files that are pulled in.
> > >
> > > The number of project files is not all that important by itself. Qt Creator, for instance, has a comparable number, and resolves much faster than your project.
> >
> > Can you give any guidance what to avoid, i.e. what makes resolving slow if QtC is much faster resolved?
>
> Nothing beyond the obvious, such as calling external processes on the right-hand side of properties (but you should get warnings for that anyway). Related: If you have probes, be sure not to trigger them inadvertently. Excessive use of wildcards is also discouraged.
> This is what comes to mind spontaneously.

What about complex inter-dependencies? I tried once to convert a
legacy project. This was quite big (bigger than QtC) and complex (in
term of public/private inter-dependencies), and i do remember running
into "performance" issues.

I do agree that QtC is faster to resolve, but it is quite "clean", and
so facilitate the job of the build sytem in that respect.

Just imagine a chain of dependencies, let say an app has direct
dependency to say 20 static libs, in turn these libs have a "meander"
of inter-dependencies, with say the longest chain being 10 or more
item longs, and the final number of libs to (directly or indirectly)
link is something around 100. On top of that let's add "lots" of
system dependencies (eg, 10+ Qt modules, 10's of lower level libs,
...)

Could that cause serious slow down of the resolving?

The project i'm talking about is so big and complex, that we hit all
the walls. For example we manage to get a 32bits linker run out of
address space, we solved the issue by tweaking/massaging the linker
flags, but still it can take one to 2 minutes just to link the final
executable.
So if ld (bfd or gold) fight with linking such an executable, could it
be that Qbs fight with resolving it's build graph?

What about the cpp and the moc scanner? if you have a lot of
"moc'able" source files, with complex/crazy include hierarchy, could
this seriously impact the resolve work?

Pure speculation here, randomly picked:
I'm not familiar with Qbs internals, but just stumble on the "Opaq"
struct in cppscanner.cpp, which has a "QList<ScanResult>
includedFiles; " member. Given how lists are not CPU cache friendly,
could this sort of "minor details" greatly impact Qbs performance when
trying to resolve "poorly" organized giant projects? With modern CPU,
cache locality is paramount.


> (I certainly don't want to to dissuade anyone from trying, it's just that you need to be prepared for a disappointing outcome. For a much lower-hanging fruit in the area of performance improvement, take a look at https://bugreports.qt.io/browse/QBS-1448.)
(pure speculation here).

Is the build graph first built/resolve in-memory, or is it
built/visited on-disk?
If the former, then it is unlikely the bottle neck here, if the later,
that might indeed be a good lead.

Chris