[Development] File Selectors API, still for 5.1?

[Alan Alpert]

On Mon, Mar 25, 2013 at 4:23 PM, Aaron J. Seigo <aseigo at kde.org> wrote:
> tl;dr -> This is a step in a good direction. I don't think it is usable as-is
> for Plasma, but it could mature into such a thing. I'm unconvinced this is
> ready for 5.1, particularly not as public API, but could be polished into
> something shiny for 5.2

tl;dr -> Okay, let's make it private for 5.1 (also works for
BlackBerry) and aim to polish it for 5.2.

> and now I will ramble on below ;)

Oh good, I like a detailed discussion :) .

> (as I wrap this email up, it's nearly 00:30, and I have to get up in 6 hours
> .. I wish there were more hours in the day, or less going on at once. ;)
>
> On Friday, March 22, 2013 14:57:11 Alan Alpert wrote:
>> Secondly, this is time-sensitive. At least Plasma and BlackBerry
>> already have conflicting schemes to provide this functionality on top
>> of Qt 4. A disruptive change to their file structures is only going to
>> work on a major version change, like when they start using Qt 5 (both
>> aim to start with 5.1 I believe). Missing this window means that they
>> both continue to use their own conflicting directory structures in
>> their next major versions, which is a blow to cross-platform
>> application development.
>
> This is quite true. I'd also prefer it if we didn't try and get this Right(tm)
> by rushing it into 5.1 as public API. We can work with private API as long as
> we agree to a schedule as to when the API can change, e.g. only between x.y
> releases ( no API change in 5.1.1, 5.1.2, etc. 5.1->5.2 would be fine)

More time to iterate is always welcomed. This may lead to a
source-incompatible change in Plasma Packages though, ideally we'd
only do that once with the existing incompatible change of KDE
Frameworks 5. Would that be using the 5.1 or 5.2 APIs?

>> 3) Why directories? This change wasn't discussed on the ML yet, and is
>> a key difference from the consensus reached in
>> http://lists.qt-project.org/pipermail/development/2013-January/009359.html
>> .
>>
>> In one of the patch sets, the form changed from file at selector.png to
>> +selector/file.png . Feedback from designers in BlackBerry suggested
>> that they were more comfortable with directories, especially when
>> using it to swap out graphical assets for device variants (which can
>> lead to a lot of selected files). It's much easier for them to deal
>> with a group of selected files via directory, which I presume is
>> because from a GUI file manager you don't have the shell globbing that
>> I'm accustomed to using.
>
> This is also what i communicated to Alan by IRC some time ago when the
> consensus was for file name mangling. It is not only that many developers and
> designers find directory based separations easier to manage, but it ensures
> that accidentally naming your file something "special" does not result in
> unexpected behaviour.
>
> I'd suggest that it is quite unexpected to have a file called tiger_800x600.png
> get chosen over tiger.png just because someone thought it was a good idea to
> add "800x600" to the selections.
>
> In Plasma we handle this rather clearly: each package of QML has a contents/
> directory. In the root of the package, we keep various bits of metadata used
> by the runtime but which are not part of the actual set of resources that make
> up the component in that package. This guarantees no naming collisions between
> whatever special files the runtime may want now or in the future and what
> developers choose to call things.
>
> We then put platform components, or what Alan is called "selected files" (same
> difference :), in a *separate* root, resulting in a package structure that
> looks like this (from an actual example used in production):
>
> .
> ├── metadata.desktop
> ├── contents
> │   ├── code
> │   │   └── uiproperties.js
> │   └── ui
> │       ├── Icon.qml
> │       ├── menu
> │       │   ├── CommentForm.qml
> │       │   ├── Confirmation.qml
> │       │   ├── MenuArea.qml
> │       │   ├── MenuItem.qml
> │       │   ├── ServiceMenu.qml
> │       │   ├── SlcMenu.qml
> │       │   └── TargetChooser.qml
> │       └── sharelikeconnect.qml
> └── platformcontents
>     └── touch
>         └── code
>             └── uiproperties.js
>
> so platformcomponents/touch/code/uiproperties.js is selected for rather than
> contents/code/uiproperties.js whenever "touch" is one of the selectors and
> "code/uiproperties.js" is requested (well, usually it's in the form:
> file("code", "uiproperties.js"), but that's a matter of a different API, namely
> packages)
>
> This strategy ensures no chance of accidental file naming causing an unexpected
> selection or for files required by the runtime to collide with files provided by
> the component developer. It also becomes very clear what are default contents,
> what are selected files, etc. and it is very easy to separate out, remove or
> add new sets of selected files.
>
> Personally, I find the use of "+" in the directory names to be less desirable
> than the above approach, though that is because we do not nest selectors in
> Plasma.

The + allows for low chance of accidental collision, while still not
being dependent on package structure. You don't want to be dependent
on package structure, because an individual "unit" of interpreted code
(QML or JS) can be varying sizes and doesn't always come in a package.
There's the pre-package stage where you have so little code that you
want to test it without package overhead, and there's shared modules
that are used in multiple packages (which can be 'shared' in several
ways other than being made into a package!). BlackBerry had to make
some significant extensions to their platform content selection just
so that you could share components between applications (i.e. copy a
QML component and related files into your application package).

So even if we could unify all package formats as well (a much bigger
task than working together on emerging features), we'd still want to
be able to apply platform content selection in a way that doesn't rely
on packaging structure.

> Our typical selector usage has a device specialization, an input paradigm and
> a general form factor, e.g. "WhizzyGoProduct : Touch : Tablet" or "EyeAmaze :
> Desktop". This lends itself to encouraging developers to create common assets
> for generic input and form factor, and only as a finishing touch put product
> specific assets in.

Part of the problem I'm trying to solve is a unified API across
platforms - and platforms will certainly have different selectors
available. Even if they all agreed on the same general split, you'll
have different products, inputs and form factors. Someone uses this on
TV and wants selectors "UltraTV : Remote : HDTV". This means
supporting customized selectors, and I'd rather not try to categorize
them when that's impossible to enforce. Better to let people establish
their own de-facto categories in a free-form manner.

> Alan's patches on the other hand encourage an approach based on screen
> resolution and target OS. This is definitely another way of approaching it, but
> is generally an approach we try and discourage in Plasma. Why? Because if you
> are designing for screen resolution, you're doing it wrong. Much more
> interesting is the actual physical size of the device and the input methods,
> as those are the human concerns. I don't care if there are enough pixels on my
> 5" device to show your high density UI .. I want something that works well and
> looks proportionate on 5" of glass. In that sense, pixel density is equally
> important ... and then we're off to the races with an NxM matrix of pixel
> density + screen res of directories .. oh, and then add the target OS. :/

Platform/OS is a convenience for feature set. When would you add the
target OS for selecting a file? Only if you know that some feature is
supported on one platform but not another. I don't think the selectors
approach scales to full feature set exposure, it's better when the
list on a particular configuration is kept manageable, so this is
probably the best proxy for all feature selection that we have. if you
only use features supported on all your target platforms, you
hopefully won't need to use it.

Screen resolution is a good proxy for those human concerns, from an
easy implementation standpoint :P . Tablet/Phone/Desktop is both
harder to use (extremely generic terms nowadays) and harder to
implement (how can I get that string from the system?). Really, I
don't think you'll be able to make a perfect design without focusing
on exact model of hardware. Anything else is going to lead to
compromises. Tablet/Phone/Desktop is the approach that probably leads
to the fewest compromises with the devices from a few years ago, but
these days I'm looking forward to a phone with a keyboard and the
laptops have touchscreens...

For now I expect each platform to come up with its own guidelines for
UI adaptability. If we have a platform selector to start with, we can
start bringing the best selection approaches into the fold. Even if
that means encouraging platforms to add Phone/Tablet/Desktop manually
per device (only way I think it can be implemented). But the platform
selector allows you to have one directory structure that will work
across all platforms, even if they have conflicting selectors.

Screen resolution and DPI was just something concrete that can be
easily implemented. I'm happy to leave that out by default for now,
and let platforms add it if they specifically want it. I'd be happy to
add "input paradigm" and "general form factor" if there were a way to
reliably determine that at start up. But we can also leave the
selectors list relatively bare for now, and get most of the
functionality out of that runtime variable.

> Is this really what we want QML development to become? Hyper specialization of
> code bases into as many QML files as devices one hopes to run it on? That's the
> diametrical opposite of the goal of Qt in general.

Actually, Qt Quick did kindof move in a different direction. Because
Qt Widgets had already aced the "write once, deploy everywhere"
problem, Qt Quick focused on the missing piece of creating pixel
perfect custom UIs. It's not like Qt Quick had a completely different
goal to Qt (especially since it's just the UI layer, on top of a fully
cross-platform Qt data layer), but we did want it to be possible to
write a custom UI for every device with QtQuick.

So it's more like that's where we *currently* are with QML, that you
need to write a different UI for each device. And we want to improve
that situation to something more cross-platform so developers can
target full device spectrums with ease, but still leaving the
'per-device' route open for those who choose it.

> While it is evident that at least in the near-to-mid term we'll have to deal
> with different platforms having their own QML APIs to offer that are highly
> divergent, it is anything but evident that we ought to base file selection on
> screen resolution.
>
> ... and this does not even start to consider that this style of API is
> inherently biased to full screen applications. It is utterly irrelevant to a
> QML application running on a 1280x720 screen if it running in a window or only
> part of the screen.
>
> ... or that screen resolutions continue to be variable from vendor to vendor,
> year to year.

Okay okay, no screen resolutions ;) .

> So, in summary, all imho from a Plasma perspective:
>
> * selectors are a great concept, and directory based sorting is the way to go
> * I'm very much against the addition of screen resolution and DPI as static
> selectors. It essentially encodes the development paradigm in Qt itself in a
> way that we can not get rid of without patching .. and I really don't want to
> have to support developers who think that putting hardcoded screen sizes is a
> good idea (didn't we already live through that on Android?)
> * The operating system selectors make sense, but are currently limited and
> provide no way of being extended without patching Qt

Because there aren't fixed categories, you can extend easily by adding
one through the environment variable. Want to use
+plasma/usePlasmaAPIs ? Add plasma to the selectors env variable in
your script launcher.

> * I'm skeptical of the + prefix notation, though I can see how nesting can be
> useful depending on the development paradigms a platform wishes to encourage
> amongst its developers.

Yes, but it could happen even with your selector distinctions if you
had more entries in those categories. If you had keyboard-only,
keyboard-touch and touch-only then you might have different versions
of a custom UI control for desktop/keyboard-only, phone/touch-only,
phone/keyboard-touch . If desktop/keyboard-only was your base, then
you'd need +phone/+touch and +phone/+keyboard versions of the control.

> That's the design side ...
>
> On the implementation side:
>
> * in Plasma, we use a ':' separated list, not a ' ' separated list for the env
> var. This is more in line with traditional path lists on UNIX, and we already
> have this out in production systems.

Okay, that's easy enough to change.

>
> * I have concerns about the combination of nesting and strict list ordering as
> it can make it complex for the developer to predict the order things will be
> chosen. Consider the asset foo.png and the following directory layout:
>
> .
> ├── +A
> │   ├── foo.png
> │   ├── +B
> │   │   ├── foo.png
> │   │   ├── +C
> │   │   │   ├── foo.png
> ├── +B
> │   ├── +A
> │   │   ├── foo.png
>
> For the set of selectors { A, B } +A/+B/foo.png will be selected
> For the set of selectors { B, A } +B/+A/foo.png will be selected
> For the set of selectors { A, B, C } +A/+B/+C/foo.png will be selected
> For the set of selectors { B, A, C } +B/+A/foo.png will be selected
>
> So order of the selectors matters greatly. If one is wanting to match based on
> screen res and platform, then it becomes an absolute requirement that the
> order that those appear is consistent and logical and can *never* be changed.
> This ordering is, essentially, public API.

And it's documented as such. The problem is that without strict
ordering you get undefined ordering - even more confusing for the
developer is which file gets picked when A and B are set?
+B/+A/foo.png or +A/+B/foo.png? It would come down to implementation
details, and then we'd have to expose them... oh wait we pre-empted
that ;) .

> Currently the order is: env var (random stuff from the platform), screen res,
> locale name (language and country), target platform (none of which are Plasma,
> Sailfish or Ubuntu Phone , of course, though we see Android and Blackberry
> there).

That's because "platform" here is meaning OS. I don't think Qt can
pickup the runtime platform - all Qt sees for Plasma, Sailfish and
Ubunutu Phone right now is Linux/X11. If I actually had the "Desktop
Environment" level of platform available then I'd love to expose that.
If not, it goes in the env var.

>
> So how do we, as a not-important-enough-to-have-made-it-to-Q_OS_* levels (I'm
> being facetious there ;) ensure our platform is at the same place in the chain
> as others so that the ordering remains the same? Well, we can't. This makes
> this implementation very difficult for us to use and keep the developer story
> straight. I'd rather have the OS moved to right after the env var, or simply
> be required to be set by the runtime itself.
>
> I don't see any reason whatsoever for having the locale name in there.
> Localiation support belongs elsewhere, and most developers will get this wrong
> anyways.

Okay. Drop resolution, DPI, locale and platform... what's left? I'm
happy to scale back the API to just the ENV_VAR and the programmatic
interface, but then we have a tough time calling this a
"cross-platform Qt solution". With OS, resolution and DPI there's at
least something that all Qt developers can rely on. It's not an easy
problem to solve since a lot of the important factors, like "runtime
platform" even, are farther up the stack than Qt. A common interface
helps them to coordinate, but it's not as good as coordinating for
them and all Qt users - if that's possible.

If we leave it up to convention then there's also no real need for a
class in Qt. The only problem right now is that you can't reach into
the QML engine to handle it yourself,  we could just do something like
https://codereview.qt-project.org/#change,52228 . But that would mean
everyone gets to solve the same problem with an incompatible
structure, which is not as good as solving the problem right once and
then sharing it.

> Whatever the final set of selectors, before this goes into public API, the
> ordering needs to be well defined. Or, alternatively, the selectors must be
> marked up in such a way that the ordering can be done at runtime ... as this
> should only need to be done once the cost would be low to merge multiple sets
> of selectors consistently so that the application, runtime and Qt could all
> provide selectors and have them ordered based on what they are.
>
> This would resolve the issue of inserting a platform as the runtime could
> simple inject sth like "platform: Plasma". heck, it could be a snippet of json
> if we wanted to go all crazylike. this would at least guarantee app developers
> that whatever set of selectors existed, they would be in a consistent
> ordering.

If we want to go crazy, we probably would just make this a plugin
based interface (like QStyle was).

Currently there's no actual "category" conception in the code, so that
there's no issue with trying to add a new category or modify an
existing category. Just prepend your selector and it works (save
ordering). We could continue this approach by using alphabetical
ordering, which would be easy for developers to understand and which
would work universally. But I can see how that could prevent nested
selector trees from making sense.

This is the advantage of having Qt provide selectors, not only are
they available on all platforms but they're consistently ordered
across all platforms. If we can provide all the primary selectors,
even if that means loading stuff we don't know from other env vars
(like a QFILESELECTORS_PLATFORM instead of compile time), then we can
have the defined ordering for the pseudo-categories while still
allowing per-application or per-platform overrides to live in the main
environment variable (and there shouldn't be too many of those).

> * Due to the nesting, there are a lot of allocations on every check of the file
> system for a file. There is the concatenated QStringLists in
> QFileSelectors::allSelectors and the creation of new QStringLists in
> selectionHelper.

Yeah, I haven't optimized it yet. I was actually thinking about
caching results, but that got derailed on questions of how "dynamic"
the selectors need to be.

> * This is really one part of a Package definition; and I'm unsure (because I
> simply haven't sat down and tried to integrate this in any way with libplasma2
> code) if this would work seamlessly with our current concept of a package as
> described earlier.
>
> There is no results caching, but that's OK as that can be done in code that
> uses this, so not a major objection (we already do this in Plasma::Package,
> for instance, to prevent having to constantly hit disk and perform the various
> string and QFileInfo etc allocations in searching for matches)

There could be...

> .. and yes, there is the matter of dynamically changing the set. This is
> something we will have in Plasma in our initial Qt5 port, so it is important
> to us to get that "Right", though that can be done in another round of work
> later. Currently our plan is to have a DBus service that tracks the current
> set of selectors and signals the runtime if/when they change.
>
> This is important for supporting an on-the-fly switch at runtime between, say,
> a tablet form factor and a desktop form factor.

Would this apply to QML files? We were having this discussion in
BlackBerry and there's a conceptual problem of state transfer when
dynamically switching between QML files. Do you just load the new
component and lose all state related to the old item, or do you have
something more intelligent in mind?

But it's only "impossible" for code files. Reloading images and other
content is quite feasible, so the selectors should support that
eventually. I guess it's just moved up to 'first revision' material.

>
>> 4) One question (sort-of) that came up in the review (from Jędrzej):
>> "Without runtime detection of a selector value change, the
>> QFileSelectors is just a workaround for a deployment problem. It
>> solves issues that can be solved by an installer script or even by a
>> dynamically loaded QResource."
> ..
>> tied to compile-time artifacts. This allows for much faster
>> development iteration, as you can just copy graphical assets or data
>> to the install dir or device without needing a re-compile. This is a
>> valuable feature for most mobile development, and an essential feature
>> for interpreted languages like QML.
>
> This is indeed the most important point of all. It is about making developer's
> lives easier and more productive, not trying to solve a problem
> programatically just because we can.
>
> Basically, when developing, one gets to maintain *one* package that they can
> then test and deploy in-situ without first having to run installers that may
> well be platform dependent. Writing such installers to have all the correct
> permutations is error prone and on top of that .. what do you do about
> applications that sit on non-static platforms, where the form factor, screen
> res and other factors may change on the fly? Re-install at runtime in response
> to environment changes? Yeeah.
>
> I also would not worry about performance before we get to measuring it. In
> Plasma, this has not been a bottleneck issue at all and we have not yet even
> gone to such extremes as "put all files into a single file mmap'd read-only
> pseudo-fs" that would probably help with memory fragmentation and access on
> rotating media.

BlackBerry has run into performance issues reading from the
filesystem, but it's still not something to worry about until an API
is sorted out.

--
Alan Alpert