[Development] QProcess fork() failure and overcommit

[Christian Kandeler]

On 03/07/2017 02:04 PM, René J.V. Bertin wrote:
> I have a bit of an intriguing issue I hope someone here could help me understand. If not, sorry for the noise.
> 
> I'm seeing occasional QProcess failures where QProcess:waitForStarted() fails and gives rise to errors like
> 
> kdevplatform.vcs: "DVCSJob::start: git ls-files -- kclock.cpp failed to start: Resource error (fork failure): Cannot allocate memory"
> 
> Those come and go in episodes; restarting the affected application always helped too (for a while).
> 
> Last time this happened htop reported I had 6955Mb used of 7899Mb, and not even 10% swap used (1151Mb of 16Gb) but despite that the issue went away when I turned overcommit back on (I usually run with overcommit_memory=2 and overcommit_ratio=80).
> 
> This only happened to me with KDevelop5 until now, i.e. a Qt5/KF5 based application, running on a KDE4 (= Qt4-based) desktop.
> If memory contention were really the cause here I would expect to see traces of similar failures elsewhere too because KDevelop isn't exactly the only KDE application that makes generous use of QProcess.
> 
> Is there anything in QProces (Qt5) vs. the Qt4 version that could explain fork() failing, or else can it be the way QProcess is being used which causes this in KDevelop but not other applications?

This kind of stuff seems to happen when the parent process has allocated
a lot of memory. I haven't debugged into it, but one idea might be that
the page tables themselves get to a non-trivial size and thus copying
them causes allocation problems.
Note also that starting a QProcess becomes enormously slow in such
applications; for instance, on my machine here, I can only start about
20 processes per second from an application that has 1 GB of memory
allocated.


Christian