[Qbs] qbs-autoproject

[resurrection at centrum.cz]

We have discussed some configuration issues via separate e-mail but few things worth pointing out here:

- I should likely call close() explicitly after reading a file. One gets spoiled from QFile's destructor. :-) That might alleviate the too many open files issue. I reckon it ultimately hits the OS limit on open file handles otherwise.

- modules are for external dependencies like Qt or your custom made modules. The path is the one I use for Qt, you should put your path or remove it altogether if your project is not using Qt.


- As mentioned the other issues are to do with configuration of the patterns as the described project is fairly standard.

- I have tried it on legacy code base of 400+ products without optimizing or ignoring any patterns and while it took long the results seemed reasonable especially considering how messy it is.

I will look into the performance issue for sure. And the readme should really be turned into the wiki.

______________________________________________________________
> Od: Christian Gagneraud <chgans at gmail.com>
> Komu: resurrection at centrum.cz, qbs at qt-project.org
> Datum: 28.10.2017 02:33
> Předmět: Re: [Qbs] qbs-autoproject
>
On 28/10/17 00:59, resurrection at centrum.cz wrote:
> Thanks for trying it out Christian! I will try to answer some of your 
> questions:
> 
>  > qbs-autoproject tookssomething like 10 to 15 minutes (as advertised)
> 
> Take a look 
> at https://github.com/Resurr3ction/qbs-autoproject#performance-tips if 
> any of those could help you there. Particularly not using the 
> contentPattern and removing unused items should help a lot. If you just 
> want to see the structure (like the qbs-create-project does) you may 
> disable the dependency scanner by setting dependencyMode = 
> DependencyMode.Disabled.

I now have disabled the dependency, i will sort out that later.
I have as well adjusted ignorePattern so that qbs-autoproject only see 
the "easy" sub directories, mainly static libraries

> 
>> At this stage, i'm not sure if qbs-autoproject refused to write the qbs 
>> files, or if it allows to generate broken qbs files, that can then be 
>> fixed manually.
> 
> qbs-autoproject will write out only one file that will be named as your 
> root directory. Since you cannot edit it (because it would be 
> overwritten) I did not see any benefit in splitting it into multiple 
> files. It is also easier to debug when all projects/products are in 
> single file imho.

I'm fine with that.

However, qbs-autoproject fails to write the final qbs file:

$ qbs build
No build graph exists yet for this configuration.
Resolving project for configuration default
Nos @ Sat Oct 28 2017 12:38:16 GMT+1300 (NZDT)
--------------------------------------
Running steps...
[1/11] Parsing configuration...
     Project root: /home/krys/Projects/nos-build-ng
     Output path: /home/krys/Projects/nos-build-ng/.autoproject
     Install path: 
/home/krys/Projects/nos-build-ng/default/install-root/linux,unix-undefined-gcc
     Output format: Tree
     Dependency mode: Disabled
     Items:
         AutoprojectApp
         AutoprojectDynamicLib
         AutoprojectPlugin
         AutoprojectStaticLib
         AutoprojectInclude
         AutoprojectDoc
     Modules:
         Qt
[1/11] Done (0ms)
[2/11] Scanning modules...
     Qt
[2/11] Done (1ms)
[3/11] Creating projects...
<...> <- 1 project per subdir, recursively, starting with root
[3/11] Done (715ms)
[4/11] Creating products...
<...> <- 1 static lib per subdir, recursively
[4/11] Done (2390ms)
[5/11] Merging products...
<no log details here>
[5/11] Done (173ms)
[6/11] Consolidating products...
<...> <- Wrong association are made here
[6/11] Done (43859ms)
[7/11] Cleaning projects...
<...> <- Removes lot of empty project, including root :(
[7/11] Done (1949ms)
[8/11] Scanning dependencies...
Dependencies disabled --- skipping
[8/11] Done (0ms)
[9/11] Finding includes...
Dependencies disabled --- skipping
[9/11] Done (0ms)
[10/11] Setting dependencies...
Dependencies disabled --- skipping
[10/11] Done (0ms)
[11/11] Writing project...
ERROR: TypeError: Result of expression 'proj' [undefined] is not an object.

I'm suspecting a completely empty top project.


I think the problem comes from the consolidation phase:

Imagine you start with only 2 top level folders, that are 2 libraries:
- Feature
- Topic

If Topic contains a Feature sub-folder (which is not a library on it's 
own, it's just source code organisation), the the consolidation phase 
will merge Feature with Topic/Feature

A simple use case:
$ tree .
.
├── project.qbs
├── Core
│   ├── core.cpp
│   ├── core.qbs
│   └── Web
│       ├── web.cpp
│       └── web.h
├── Gui
│   ├── gui.cpp
│   ├── gui.qbs
│   └── Web
│       ├── web.cpp
│       └── web.h
└── Web
     ├── web.cpp
     ├── web.h
     └── web.qbs


In this case the consolidator will merge Core/Web, Gui/Web with Web/ 
instead of simplifying to Core, Gui and Web as top libraries.


>> After trying the "full" dependency-tracked custom Qbs (Export items), i 
>> decided that i don't want that. I like it, but I cannot afford so to 
>> speak. My DAG is so wild, that qbs fight to load it and i end up with 
>> Linux's limits.conf problems. Such as command line too long (true) and 
>> too many open files (Ubuntu's mistake).
> 
> Now this is interesting. Could you tell me more about these issues? 
> Export item vs include-directory should not be a significant difference 
> except for the cyclic dependency problem. Have you tried NoHeaderOnly 
> dependencyMode?

Let me clarify, this has nothing to do with qbs-autoproject, I have used 
my scripts as initial generator, and then manually polished the 
generated files. I have something like 150 products (apps and libs), the 
libraries have a complex inter-dependency. If I try to use the Export 
Item on each of these, i end up with an app build command that has lot 
of duplicated include search path (-I), a lot of duplicated library 
search path (-L) and a lot of duplicated link objects (-lfoo, -lbar, ...)
This is where i hit the Linux limits.conf, I am not sure if I do it 
wrong thought.

I as well hit the "too many open files" with qbs-autoproject, I have the 
feeling that it keeps all the source files open while scanning.
JS File objects not garbage collected?

>> It's not clear to me how i can inject my initial qbs support files into 
>> the qbs-autoproject ones, maybe i just need to edit '.qbs-autoproject/' 
>> files. I found it hard to define "where the project is", "what the 
>> project name is", and "where is my custom stuff".
> 
> All your custom stuff should be done via custom items and modules. So in 
> your case you would edit (or remove or add more) stuff in 
> .autoproject/imports/ and then you would link these to the 
> qbs-autproject via the `items` property - giving them the pattern etc.

Thanks, I get it now, this is pretty cool and very similar to how my 
python scripts work.

What about the 'modules' property, why there is a 'Qt' module there with 
an hard coded include path? Is it just an example?
I think i will want to use this feature, as i have a couple of special 
ThirdParty libraries.

Thanks,
Chris