[Development] Extending touch semantic information and Qt backends

[Rutledge Shawn]

> On 11 Feb 2016, at 02:13, Ariel Molina <ariel at edis.mx> wrote:
> 
> Hi, 
> 
> Research is strong on natural user interfaces, and now market is stronger than ever and growing. I was working on the TUIO QPA and found a couple of bugs, which I already pushed patches [1] [2] which I hope make it to 5.6.x.
> 
> OTOH, to further enhance touch interfaces I want to bring the discussion about multiple touch layers, person-tagged touches, identified touches with area&rotation information and objects with unique marker identification.
> 
> - Multiple touch devs/layers and Person tagged touches. Can be easily added by extending QWindowSystemInterface::TouchPoint with a device identification, like device(). This will also work for TUIO. 
> - Area & rotation. There is TouchPoint::area() and ::pressure() but currently there is no support for ::angle(). Angle can be zero when not supported.
> - Unique identification. On research and advanced interactive surfaces the usage of "markers" [3] is quite common. This is just a code extracted either visually or by other mechanisms which is passed along the touch information. I conjunction with area, rotation and uuid this paves the way for very rich interaction.
> 
> I would like to know your thoughts about this topics, specially to know if it would be acceptable for me to submit patches enhancing those areas (and to whom add for review). I'd love to see marker uuid and angle() support at least.

I’m interested.  But I will warn you that my attempts to make MouseArea and Flickable handle touch events directly have run into issues with the complexity of handling two completely independent event types with similar-but-different code, backwards compatibility, getting reviews on the patches, getting tests to continue passing, etc.  They handle only mouse events, which are synthesized on a touchscreen; changing MouseArea to handle touch is too hard; and that makes it too hard to change Flickable, because so many applications have MouseAreas in delegates in ListViews.

So, in existing releases of QtQuick, only PinchArea and MultiPointTouchArea handle touch events directly.  Maybe you are thinking of enhancing MPTA and its TouchPoints?  That’s probably doable, but do you think that API is useful enough for your applications?  If not, what kind of API would you rather have?

Now the plan is to write a new set of event handlers which are lighter-weight: not Items themselves but declared inside any Item (inspired by the Keys attached property, but not quite the same).  There can potentially be lots of types of handlers, and even several handler instances inside one Item when necessary; some will be hardware-agnostic “gesture” recognizers (e.g. TapHandler and PinchHandler) while others will be hardware-specific.  So maybe we can come up with a good way to handle markers within that framework, although I don’t have access to a fancy touch table which supports them (but have a capacitive touchscreen).  And yes the multi-user scenarios are interesting too.  I figured the new QQuickPointerEvent can have some sort of ID for user or “seat", whichever makes most sense.  Do you think it should simply be a numeric ID?

Likewise I don’t have a touchscreen which supports rotation, but have a Wacom tablet which does.  QQuickPointerEvent has a rotation property, on the assumption that it will become more common (but I can only test it with the tablet for now).

What do you mean about layers?

The PointerEvent/PointerHandler patches have been mostly waiting for me to get back to them since last summer, but I plan to get back to them soon.  https://codereview.qt-project.org/#/c/123122/ is currently the ‘tip’, and you can follow dependencies backwards from there.  The plan is to put them into a branch soon so that it’s easier to follow progress.  Then we’ll see if it gets done in time for 5.8.  So far only the simplest Handlers work, others have issues remaining, and some aren’t written yet.