[Qbs] Qbs JavaScript processing limits

[Christian Kandeler]

On Sat, 13 Oct 2018 10:46:47 +0200
<resurrection at centrum.cz> wrote:

> Hi, I have noticed that my probes that use heavy JavaScript crashes Qbs when processing very large files/data. The probes in question are here:
> 
> https://github.com/Resurr3ction/qbs-autoproject/blob/master/.autoproject/autoproject.qbs 
> 
> The most problematic Probe is the "dependencyscanner". It basically reads files, applies regexp to content to extract the includes and saves what it found. Some of the files have 30k+ lines.

Doesn't sound outlandishly big to me.

> Is there a limit in Qbs JS engine for this kind of processing? Or a memory limit what it can hold in? 

qbs.TextFile maps to QTextStream. I'd be surprised if this class had problems with files of the size you gave. You also seem to properly close your files after use.

> Would moving it into JS Command help? 

My assumption was that you need to do this at resolve time to fully read the project. Otherwise, why do it upfront if it can be done later?

> Or perhaps I need to scan the large files by chunks or something? 

It's generally a good idea to read text files line by line, unless you need more context, so yes, that would be recommended in any case. Why keep megabytes of data in memory if you don't have to? Who knows, maybe QtScript's garbage collector doesn't act fast enough and all the data adds up? Just a wild guess.

> I clearly need to optimize it somehow but I am not sure in what way exactly.

You might also want to take a look at the stack trace you get from your crash. That could give you some insight.


Christian