[Interest] memory fragmentation?

Mon Aug 20 18:08:40 CEST 2012

Jason:

> I've got a very long process that I want to protect against this.
> Switching to 64bit is not an option for at least another 10 years.

The classical answer from the realtime world is: Don't deallocate
  and reallocate.

  In our application, we instantiate at image startup time every
  Qt widget we're ever going to instantiate and after that, they
  remain in memory throughout the life of our application.
  This assures that if our application launches at all, it's not
  going to fail later trying to malloc Qt widgets. And in the
  spirit of the JPL "Power of Ten" rules, we try not to malloc
  at all (although we know that things like STL and some of
  our Linux libraries do it "behind our backs").

                                               Atlant

  From the JPL "Power of Ten" rules:

  http://spinroot.com/gerard/pdf/P10.pdf

  3. Rule: Do not use dynamic memory allocation after initialization.

  Rationale: This rule is common for safety critical software and appears in most
  coding guidelines. The reason is simple: memory allocators, such as malloc, and
  garbage collectors often have unpredictable behavior that can significantly impact
  performance. A notable class of coding errors also stems from mishandling of
  memory allocation and free routines: forgetting to free memory or continuing to use
  memory after it was freed, attempting to allocate more memory than physically
  available, overstepping boundaries on allocated memory, etc. Forcing all applications
  to live within a fixed, pre-allocated, area of memory can eliminate many of these
  problems and make it easier to verify memory use. Note that the only way to
  dynamically claim memory in the absence of memory allocation from the heap is
  to use stack memory. In the absence of recursion (Rule 1), an upper-bound on the
  use of stack memory can derived statically, thus making it possible to prove that
  an application will always live within its pre-allocated memory means.

From: interest-bounces+aschmidt=dekaresearch.com at qt-project.org [mailto:interest-bounces+aschmidt=dekaresearch.com at qt-project.org] On Behalf Of Jason H
Sent: Monday, August 20, 2012 11:51 AM
To: Thiago Macieira; interest at qt-project.org
Subject: Re: [Interest] memory fragmentation?

What can be done to combat this in C++/Qt?

Initially I thought there might be some kind of QObject d-ptr magic where the large allocations can be in the private class and re-allocate and copy the private class, without affecting the pointers that the application uses. Then have a function that runs on a QObject tree,  This does not fix the overall issue, but at least it could decrease the the chances of a failed allocation. But for me, this raises as many questions as it solves. Can we guarantee the new allocations are more compact than the old ones? That seems up to the heap manager. I don't know much about that. Also, when to run it? Periodically, or only when an allocation fails?

I've got a very long process that I want to protect against this. Switching to 64bit is not an option for at least another 10 years.

________________________________
From: Thiago Macieira <thiago.macieira at intel.com>
To: interest at qt-project.org
Sent: Monday, August 20, 2012 11:23 AM
Subject: Re: [Interest] memory fragmentation?

On segunda-feira, 20 de agosto de 2012 07.56.36, Jason H wrote:
> In comparing technologies (Qt vs .NET) one of the biggest architectural
> differences is the garbage collection model.
>
>
> Qt uses the standard C++ new/free, where .NET uses a garbage collector.
> Interestingly the GC not only manages overall usage, but can move objects
> around in memory. This is the prupose fo the ^ operator in their C++/CLI -
> it's a memory handle that is used to reference an object instead of a
> pointer, because the GC can change the address.
>
> In the entirety of my computing experience I have never seen a malloc or new
> fail because of memory fragmentation. Is this some low-level MS FUD?

No. It's a non-theoretical situation. It can happen.

Take, for example, a 32-bit application running on a 32-bit Linux. It has 3 GB
of addressing space. Since it's using some of it when it starts up, it has
less than 3 GB available for use.

Try this:
    malloc(1 GB);
    malloc(100 MB);
    free(the 1 GB block)
    malloc(2 GB);

There's a big chance it will fail, even though there's still roughly 2.8 GB of
addressing space available.

The above is, of course, an academic exercise. If your application is going to
use blocks in the order of gigabytes, you should be using 64-bit.

In a real-world case, memory fragmentation usually influences the total memory
usage of an application. Even if large chunks of memory are freed, they can't
be returned to the operating system because the application is still using
blocks further down the heap. Also, it might increase its consumption if it
does operations like:

    ptr1 = malloc(BIG);
    ptr2 = malloc(SMALL);
    free(ptr1);
    ptr3 = malloc(SMALL);
    ptr4 = malloc(BIG);
    free(ptr2);
    free(ptr3);
    return ptr4;

In a zero-overhead allocator, ptr1 = ptr2 + BIG and ptr3 = ptr1. That means
the room between ptr3 and ptr1 is only BIG - SMALL, which isn't big enough to
satisfy the fourth malloc. That code above will extend the heap by at least
2*BIG + SMALL, even though it only had BIG + 2*SMALL amount of memory
allocated at any given time.

E.g., for BIG = 65536 (64k) and SMALL = 16, the heap was extended by at least
128k + 16 bytes, even though the function actually allocated only 64k + 32
bytes.

If that operation is repeated in a loop, the heap might grow a lot. This is
known to happen under certain circumstances for Qt, for example: memory
allocations for QStrings, QByteArrays and QEvents, though small, happen quite
often.

--
Thiago Macieira - thiago.macieira (AT) intel.com<http://intel.com/>
  Software Architect - Intel Open Source Technology Center
    Intel Sweden AB - Registration Number: 556189-6027
    Knarrarnäsgatan 15, 164 40 Kista, Stockholm, Sweden

_______________________________________________
Interest mailing list
Interest at qt-project.org<mailto:Interest at qt-project.org>
http://lists.qt-project.org/mailman/listinfo/interest

Click here<https://www.mailcontrol.com/sr/CWaQ9NUrOajTndxI!oX7Uj80y4Ou3Kxp4k8WaAafW6d1U9xwWUBNZLZRnCe5RG0u39vSxKZjSn0ua8zCo+pDiw==> to report this email as spam.

________________________________
This e-mail and the information, including any attachments, it contains are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message.

Thank you.

Please consider the environment before printing this email.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/interest/attachments/20120820/0b517cd0/attachment.html>