[Qbs] Improving qbs resolve performance

[Christian Kandeler]

On Sat, 13 Jul 2019 17:37:04 +1200
Christian Gagneraud <chgans at gmail.com> wrote:

> On Sat, 13 Jul 2019 at 00:48, Christian Kandeler
> <Christian.Kandeler at qt.io> wrote:
> >
> > On Fri, 12 Jul 2019 12:05:48 +0000
> > Maximilian Hrabowski <hrabowski at clausmark.com> wrote:
> >  
> > > >> I have a fairly large project with a root projects that pulls in several SubProjects, altogether with unit tests its about 313 qbs files that are pulled in.  
> > > >
> > > > The number of project files is not all that important by itself. Qt Creator, for instance, has a comparable number, and resolves much faster than your project.  
> > >
> > > Can you give any guidance what to avoid, i.e. what makes resolving slow if QtC is much faster resolved?  
> >
> > Nothing beyond the obvious, such as calling external processes on the right-hand side of properties (but you should get warnings for that anyway). Related: If you have probes, be sure not to trigger them inadvertently. Excessive use of wildcards is also discouraged.
> > This is what comes to mind spontaneously.  
> 
> What about complex inter-dependencies? I tried once to convert a
> legacy project. This was quite big (bigger than QtC) and complex (in
> term of public/private inter-dependencies), and i do remember running
> into "performance" issues.

The number of modules pulled in per product is surely relevant (note that product dependencies are modules in this sense).

> What about the cpp and the moc scanner? if you have a lot of
> "moc'able" source files, with complex/crazy include hierarchy, could
> this seriously impact the resolve work?

No, that's done when building and has nothing to do with resolving.

> Pure speculation here, randomly picked:
> I'm not familiar with Qbs internals, but just stumble on the "Opaq"
> struct in cppscanner.cpp, which has a "QList<ScanResult>
> includedFiles; " member. Given how lists are not CPU cache friendly,
> could this sort of "minor details" greatly impact Qbs performance when
> trying to resolve "poorly" organized giant projects? With modern CPU,
> cache locality is paramount.

Again, that's a build time thing.

> > (I certainly don't want to to dissuade anyone from trying, it's just that you need to be prepared for a disappointing outcome. For a much lower-hanging fruit in the area of performance improvement, take a look at https://bugreports.qt.io/browse/QBS-1448.)  
> (pure speculation here).
> 
> Is the build graph first built/resolve in-memory, or is it
> built/visited on-disk?
> If the former, then it is unlikely the bottle neck here, if the later,
> that might indeed be a good lead.

That task is about the time is takes to load a stored build graph.


Christian