[Development] qmlbundle vs Qt Resource System

[Thomas Hartmann]

Hi,

Am 13/08/2013 18:32, schrieb Alan Alpert:
> On Tue, Aug 13, 2013 at 8:47 AM, Thiago Macieira
> <thiago.macieira at intel.com> wrote:
>> On terça-feira, 13 de agosto de 2013 12:40:20, Thomas Hartmann wrote:
>>> My conclusion would be to deprecate qmlbundle, since it is not really
>>> documented and supported (especially from the tooling side) and it does
>>> not seem to have any genuine advantage, while .qrc files and .rcc files
>>> have been part of Qt practically for ever and are supported by tooling.
>> [cut]
>>
>> Sounds good.
>
> You can't deprecate qmlbundle, as it's hasn't been initially supported
> yet ;) . It's just a prototype (and yes, there's a valid question of
> why it got into the release at this early point).
>
> Once we get past the prototype stage with a real implementation,
> dropping the qmlbundle code makes perfect sense. But I wouldn't advise
> hastily "deprecating" it until the alternative is ready, as we may
> need to resurrect it just to have something ready in time (should this
> become important to someone).

I agree that we have to find out if .rcc is good enough and maybe extend 
it slightly before deprecating/removing anything.
But I want to emphasise that Qt should only have one bundling mechanism 
(rcc). Rcc/qrc is already used also in the QML context and it can be 
extended if we find limitations. I see no reason to introduce 
another/different mechanism for bundle files.

>>> The only use case that came up, where .rcc should not be the solution,
>>> is a cache for the jit. I think any cache for the jit should go into a
>>> separate file and should not get mixed up with resource bundling.
>>
>> If the cache is platform-independent and future-proof, I don't see why it
>> couldn't be added to the bundle.
>
> It needs to be part of the bundle, platform-independent or not. Part
> of the qmlbundle plan was that you have a single file you can mmap in
> to get the result of the compilation phase with maximum efficiency.
> Part of the long-term QML plan is to JIT ahead of time, during the
> build phase, and to even extend that to the QML parts (not just JS
> snippets). If .rcc can't handle platform-dependent files then we'll
> need to go with qmlbundle (or we fix QRC until it does, has anyone
> investigated that?).

I have to disagree. Mixing a cache (read write access) with resources 
(read only) in a single file/virtual file system does not make much 
sense to me.

The cache might require a different kind of (flash) memory then the 
resources and generally mmaping the complete resource file is also not 
something you need to do.

I think the cache and the resources should not be mixed up. Another 
reason is that the .rcc/rsource file is platform independent, while the 
cache is not. Imagine people copying around .rcc files for testing/hot 
fixing/patching and suddenly there is also a cache inside.

Afaik .rcc are platform independent files, but I will test this properly 
until I make any real claims.

> >>> Another topic that came up was qmldir. Since Qt 5 now supports to .json
>>> to describe and annotate a plugin, shouldn't we use this instead of
>>> another arbitrary file format? Of course we will continue support for
>>> qmldir, but we should depreciate it and use .json instead. Like for any
>>> other plugin in Qt.
>>>
>>> The qmldir file could be replaced by qmldir.json in the pure .qml plugin
>>> case.
>
> Note that, outside Qt, the pure .qml (or mixed) case is more common I
> believe than the "just a C++ plugin" case.
>
> The other issue (possibly solvable) is that QML modules are always
> defined by a qmldir file, not a plugin, which would lead to a lot of
> issues (recognizing modules, finding the plugin in order to load the
> metadata file) if we tried to move the functionality into the new JSON
> metadata files.
>
>> I disagree, somewhat. Please don't ask people to manually write JSON files in
>> their text editors. But I can't say whether people often have to write those
>> files and if they do it by hand or whether there are tools to do it.
>
> Yes, and by hand. Every QML module needs at least one, and I don't
> think that QtCreator has any support to help.
>
>>  From what
>> I can tell, they're usually very simple files and are easy to generate using
>> shell tools. Therefore, I'd recommend they should stay as-is.
>
> They are indeed quite simple. The shell tools usecase is a boon, as
> often you want to generate one quickly based off the contents of a
> directory.
>
>>
>> This is also one of the requirements I've made for the new config API: the
>> actual config files should be plain text, preferably INI, but can be cached into
>> (binary) JSON.
>>
>>> If the plugin is converted to .rcc the .rcc file contains the
>>> qmldir.json file.
>>
>> That sounds more like my config proposal: it's a cache, not the actual user--
>> modified file.
>
> The point of having a pre-compiled bundle should be that you have the
> compiled output of the qmldir file instead. You don't need the actual
> file, it's far more like the cache which Thiago suggests.
>
> Given all those points, I'm against rewriting qmldir to be a JSON file.

This was just something that came up and I think it should be discussed.

>>>> - qrc resources can be stored in separate files (.rcc), or embedded into
>>>> the executable. Qmlbundle files are always separate files (.bundle).
>>>
>>> True. While most people (including me) did not know about .rcc files,
>>> they are supported.
>>
>> I didn't either.
>>
>> And since I haven't done any investigation... please make sure what happens to
>> the application once you load an arbitrary .rcc file. In particular, since
>> resources may simply add to the root of the resource system's path, it may
>> override assets that the host application needs, including those coming from
>> Qt itself.
>>
>> Should .rcc files be considered hostile, and an application should only load a
>> trusted one?
>
> Note that it's the same with QML files. They can execute native code,
> so should only be loaded when trusted.

Yes. Any QML plugin can introduce code in any Qt process. In conjunction 
with QML_IMPORT_PATH this is an issue, but it is not genuine related to 
bundles/.rcc files.

>>>> - rcc can optionally compress the content (on by default).
>>>
>>> If it does not already do this: rcc should not compress kwown image
>>> compressed formats by default (.jpg/.png).
>>
>> It should compress and verify whether the compression result is worth it. If
>> there's no gain, it should simply store uncompressed.
>>
>> One of the advantages of storing uncompressed is that QFile::map() returns the
>> actual pointer to the data.
>
> As mentioned earlier, we want the actual data ASAP. So the compression
> needs to both save space and time, or it should be disabled.

On most systems compression does save time, since computation 
(decompression) is magnitudes faster then reading from any kind of 
persistent memory. Of course real life benchmarks should make this 
decision and compressing things twice does not make any sense and is 
also a waste of memory.
RCC could check if compression makes any sense like Thiago suggested.
Also we might want to introduce "flags" on a per file basis to .qrc. So 
people can flag files that get streamed or mmaped so they will not get 
compressed. For example .wav files that get streamed by the application, 
should not be compressed even if the compression ratio is around 50%, 
because the decompression (dictionary etc.) takes extra memory.

>>>> - rcc always processes a .qrc file, while qmlbundle individually add /
>>>> remove /update files to a bundle by passing them on the command line.
>>> There was the idea to add similar functionality to rcc. Like any command
>>> line compression tool rcc then would take files and even (recursive)
>>> directories and convert them to .rcc files.
>>
>> You're now describing a Zip file. If we're going to go anywhere there, why not
>> use simply a Zip file?
>
> I think that ZIP was the original prototype of qmlbundle :P .
>

I am against using .zip.

I an ideal world our bundle format would have at least these features:

* zip and bzip2 compression (Huffman coding and Burrows–Wheeler 
transform). Huffman coding needs less resources while BWT yields better 
results. The user can use flags in .qrc files to control the choice. [1]

* Files do not have to be compressed. Again the user can choose no 
compression in .qrc files and .rcc can check if compression is feasible. [2]

* It makes sense to use one dictionary for several files to increase 
compression ratio. Since this comes with additional (memory) costs it 
should not be the default. [3]

* For some use cases people might want to specify the order of files in 
the .rcc file. There are even use cases where duplicating files makes 
sense to speed up loading/seeking times. [4]

* Features I do not anticipate.

While .zip might support all these use cases (I do not know about 3 and 
4) using it as a container is more work (speaking from experience), then 
just reinventing it for our specific use case. Of course using .zip has 
the advantage that 3thrd party file browsers do work. If anyone knows of 
a versatile free .zip library that supports choosing the compression 
(and not just defaults to Huffman coding) please tell me. ;)

Of course we use zlib/bzlib for the actual compression/decompression.

As for now I suggest we try how far we get with the current .rcc file 
format and if it is not good enough we introduce rcc2 which just has a 
different header and format, but otherwise is indistinguishable from the 
current .rcc.
A file format with the features I mentioned above is not THAT much  work 
(speaking from experience), but I prefer doing things if really needed 
not because we can. ;)

Kind Regards,
Thomas Hartmann


>>>> - qmlbundle has 'metalinks' that you can use e.g. store stripped /
>>>> processed versions of individual files.
>>> Is this used? Couldn't this be solved by file endings instead?
>>
>> Sounds like premature optimisation to me. A JIT cache sounds more interesting
>> to me.
>
> JIT cache is where we're going. It's because qmlbundle doesn't have
> any of these interesting and useful parts yet that it's neither
> supported nor documented. Nor used, it's in the repo right now either
> to facilitate future development (which was unexpectedly halted) or by
> mistake due to the turmoil of last year.
>
> --
> Alan Alpert
> _______________________________________________
> Development mailing list
> Development at qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development
>


-- 
Thomas Hartmann
Software Engineer
Nokia, Qt Development Frameworks

Digia Germany GmbH

Rudower Chausse 13, 12489 D-Berlin

Sitz der Gesellschaft: Berlin, Registergericht: Amtsgericht 
Charlottenburg, HRB 144331 B,
Geschäftsführer: Mika Pälsi, Juha Varelius, Anja Wasenius

Email: thomas.hartmann at digia.com


Digia Germany is a group company of Digia Plc,

Valimotie 21, FI-00380 Helsinki Finland

Visit us at: www.digia.com

------------------------------------------------------------------

PRIVACY AND CONFIDENTIALITY NOTICE

This message and any attachments are intended only for use by the named 
addressee and may contain privileged and/or confidential information. If 
you are not the named addressee you should not disseminate, copy or take 
any action in reliance on it. If you have received this message in 
error, please contact the sender immediately and delete the message and 
any attachments accompanying it. Digia Germany GmbH and Digia Plc do not 
accept liability for any corruption, interception, amendment, tampering 
or viruses occurring to this message.