[Interest] memory fragmentation?

[Jason H]

Given the network-centric world we live in, I usually think about networks, not disks, because as you point out, there are a variety of faster-than-rotational media - be RAM FS, flash FS or some other kind of mixed (SSD). Still the CPU is orders of magnitude faster than anything but a ramdisk. 

Still the whole-file-in memory issue I think is controlling because it limits your concurrency, else you'll be thrashing through virtual memory (still painful on a flash FS). Given that parallelism scales, it would seem benficial to work with just a few kB at at time, that way you can scale across more processors, disks or whole systems, while increasing fault tolerance. 





________________________________
 From: Atlant Schmidt <aschmidt at dekaresearch.com>
To: 'Jason H' <scorp1us at yahoo.com>; Tony Rietwyk <tony at rightsoft.com.au>; "interest at qt-project.org" <interest at qt-project.org> 
Sent: Tuesday, August 21, 2012 11:36 AM
Subject: RE: [Interest] memory fragmentation?
 

 
All:
 
> It can be very easily justified that many to Utf* calls are justified because
> of latency in the I/O system means it happens "for free" while reducing the
> overall latency.
 
In the “modern world”, be a little cautious when you decide what activities can
  overlap with other activities. For example, for many of us in the Linux world
  who are using Flash-based file systems (e.g., UBIfs), there’s really no such
  thing as overlapping disk I/O with other activities; all “disk” I/O to the Flash
  file system requires the CPU and will block ordinary interactive priority
  user-level activities that also need the CPU.
 
It’s not like in the old days when you queued an I/O request to a rotating
  magnetic disk and between seek latencies and DMA data transfers, you
  could slip-in a lot of computing. Nowadays, that “disk” I/O may still require
  the CPU.
 
                                              Atlant
 
From:interest-bounces+aschmidt=dekaresearch.com at qt-project.org [mailto:interest-bounces+aschmidt=dekaresearch.com at qt-project.org] On Behalf Of Jason H
Sent: Tuesday, August 21, 2012 10:17 AM
To: Tony Rietwyk; interest at qt-project.org
Subject: Re: [Interest] memory fragmentation?
 
Not true for really large XML. With the all-in-one approach, you have to have the entire document completed. This will create a long pause, and increase latency, and increase memory usage. It can be very easily justified that many to Utf* calls are justified because of latency in the I/O system means it happens "for free" while reducing the overall latency.


Would you prefer an image transcoding server that waited for an image to be completely uploaded and completed, or start receiving the conversion as soon as final blocks or scan lines were available?


 

________________________________
 
From:Tony Rietwyk <tony at rightsoft.com.au>
To: interest at qt-project.org 
Sent: Monday, August 20, 2012 9:52 PM
Subject: Re: [Interest] memory fragmentation?
 
...
In the end I chose Qt 4.8.1  QXmlStreamWriter(QString *).   The QByteArray version seemed the logical choice - but since the API only uses QString, each call resulted in temporaries being allocated and disposed.  It seems QString is crying out for an appendLatin1(const char *, int len) that re-allocates the unicode array if necessary, and then does the latin1 conversion without creating a temporary.   Doing a single toUtf8 at the end is also better than calling it a million times.  
 
 
 
Click here to report this email as spam.
________________________________
 This e-mail and the information, including any attachments, it contains are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message.

Thank you.

Please consider the environment before printing this email.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/interest/attachments/20120821/6a155417/attachment.html>