[Development] qmlbundle vs Qt Resource System

[Thiago Macieira]

On quarta-feira, 14 de agosto de 2013 11:10:28, Thomas Hartmann wrote:
> On most systems compression does save time, since computation 
> (decompression) is magnitudes faster then reading from any kind of 
> persistent memory. Of course real life benchmarks should make this 
> decision and compressing things twice does not make any sense and is 
> also a waste of memory.

That is true, but loading time is not the only component in the equation here. 
Decompressing puts on memory pressure, since the decompressed output must be 
kept in memory via generic heap allocations. For any large file, this might be 
a bad idea. Generic heap allocations are dirty memory and must be backed by 
the system swap.

mmap'ed contents are clean memory: the OS is free to discard the pages at will 
because they're backed by the filesystem.


> RCC could check if compression makes any sense like Thiago suggested.
> Also we might want to introduce "flags" on a per file basis to .qrc. So 
> people can flag files that get streamed or mmaped so they will not get 
> compressed. For example .wav files that get streamed by the application, 
> should not be compressed even if the compression ratio is around 50%, 
> because the decompression (dictionary etc.) takes extra memory.

Agreed.

> I am against using .zip.
> 
> I an ideal world our bundle format would have at least these features:
> 
> * zip and bzip2 compression (Huffman coding and Burrows–Wheeler 
> transform). Huffman coding needs less resources while BWT yields better 
> results. The user can use flags in .qrc files to control the choice. [1]

BWT is also a magnitude slower in decompressing, LZMA even worse. The higher 
compression ratio comes at a cost in decompression time.

I'm not sure what type of files should take those other compression algorithms.

In any case, BWT and LZMA are supported in .zip files too.

> * Files do not have to be compressed. Again the user can choose no 
> compression in .qrc files and .rcc can check if compression is feasible. [2]

Zip files can have that. "No compression" is just one of the multiple 
compression types.

https://en.wikipedia.org/wiki/Zip_file#Compression_methods

> * It makes sense to use one dictionary for several files to increase 
> compression ratio. Since this comes with additional (memory) costs it 
> should not be the default. [3]

You want the same file to be stored multiple times, but with different 
compression ratios / methods? That would leave the decision on which one to 
load to run-time, which is usually the worst possible time to make such a 
decision.

If you want to leave the decision to the user, then it's no different than 
storing multiple files.

Or did you mean a shared dictionary for compression, à la RAR and 7-Zip's 
"solid archive" and general .tar.gz compression?

Even if Zip doesn't support it, we can add our own extensions. If OS X can add 
__MACOSX dirs in zip files, why can't we do similarly?

> * For some use cases people might want to specify the order of files in 
> the .rcc file. There are even use cases where duplicating files makes 
> sense to speed up loading/seeking times. [4]

Can be done in Zip too, by creating the zip file incrementally, or with a 
specialised tool.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.qt-project.org/pipermail/development/attachments/20130814/cf6cb217/attachment.sig>