[Development] API review request: CBOR Stream reader and writer

[Thiago Macieira]

On segunda-feira, 22 de janeiro de 2018 16:56:56 PST Thiago Macieira wrote:
> After fixing the converter example to properly use mmap in all three cases,
> plus refactoring the CBOR parser to operate on a pre-loaded array and use
> larger buffer than single-digit byte counts, the numbers are:

A major difference between the three is the memory allocation strategy. The 
Binary JSON needs none, since it uses an in-memory format for the on-disk 
format. So the application runs with 53880 kB of the mmapped file in a total 
of 64384 kB RSS.

The parsed QJsonDocument uses the same allocation strategy, which one if its 
greatest advantages. It allocates a buffer equal to the size of the JSON 
input, so after unloading the 55872 kB of the source file, it keeps a huge 
block of 55792 kB allocated, in a total of 65928 kB RSS.

A major drawback of this is that the Binary JSON memory format is limited to 
128 MB in total. There's a bug report about this and changing the format to 
raise this limit, even for in-memory, will change the numbers above.

QCborValue does no such thing, but it's also not limited. Since it uses 
regular memory allocation, after unloading the 40244 kB file, it needs 110356 
kB of heap, for a total of 120428 kB RSS.

For comparison, loading the same data from a QDataStream and keeping in a 
QVariant requires 233944 kB of heap, for a total of 244088 kB RSS. The 
QDataStream file is 76156 kB and that includes all strings stored as UTF-16, 
so it's not the UTF-16 vs UTF-8 vs Latin1 that accounts for the majority of 
the overhead.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center