[Interest] [Qt3D] Virtual Reality (Vive/Oculus) Example

[Sean Harmer]

Hi Daniel,

awesome, thank you for this! We too have been starting work on something 
similar:

https://codereview.qt-project.org/#/c/195892/

The latency you mention can be avoided if we update the head tracking directly 
on the qt3d renderer backend. That way it can be used immediately on the 
current frame. The frontend doesn't necessarily need to know the eye matrices 
but we can make tracking these on the frontend opt-in if necessary (if 
something wants to bind to them - but even that use case may become obsolete 
soon with some backend bindings work we have ongoing).

I've pointed a couple of colleagues at your work and I'll try it out myself 
too once I get a few urgent things sorted locally.

You may not have seen it but, we now have some infrastructure for renderer 
plugins to allow deeper integration than having to go via the existing  public 
Qt 3D frontend types such as QTransform. I think this will be the way to 
remove the latency you mention.

We should certainly coordinate efforts and see how we can make this ready for 
production use. Feel free to ping us on development@ or on irc (#qt-3d) or 
even direct email if you want to discuss this.

Thank you once again for contributing!

Sean

On Tuesday 18 July 2017 10:29:25 Daniel Bulla wrote:
> Hi,
> 
> I started adding support for OpenVR/Vive and native OculusSDK to Qt3D.
> 
> You can find it on Github: https://github.com/dabulla/qt3d-vr-example
> 
> It can easily be setup, as described in the Readme.
> 
> 
> *What works?*
> 
>    - Stereoscopic 3D Rendering of Qt3D Qml scenes in the Headset.
>    - Headtracking
> 
> *What is still in progress?*
> 
>    - Headtracking has high latency
>    - Mirroring to Monitor
>    - Motion controllers + Input
>    - ... (lots of things imaginable)
> 
> *Implementation*
> 
> There is QHeadmountedDisplay which does the job of a Qt3DQuickWindow but
> uses Direct-Mode Rendering. It can be used just like a Window that is
> placed at the headset and will sync to it's refresh rate. OpenVR or
> OculusSDK (aka 'SDKs') also handle distortion correctly, so there is no
> need to do this manually.
> 
> The rendered Qt3D Qml scene must use a VrCamera to enable head-tracking and
> correct stereoscopic rendering. VrCamera configures projection-parameters
> for the headset by querying the SDK.
> 
> I only tested this on windows at the moment.
> 
> *Future*
> 
> I'd really like to proceed with the implementation and look forward to
> getting some feedback at this point. I read the qt styling guide and
> started a clean implementation on github and wanted to contribute it to
> qt3d at some point. I'm open for suggestions and could also need advice
> with the following problem:
> 
> *Tackling latency*
> 
> The pose read from the headset needs approx. 3 frames before it finally
> ends up as a shader-uniform for rendering. This takes 3 frames too long (it
> is extremely noticable and you get nausea very quickly)!
> 
>    - In the main-thread "getEyeMatrices" is called, this gets the
>    prediction of the head pose for the very next frame.
>    - The matrices are handed to Qt3D using the QTransform of the
>    Camera-Entity (VrCamera).
> 
> I suspect that it takes several frames before the transform ends up as a
> uniform in the shader and wonder how I can tweak Qt3D to keep this latency
> down. A tip of a Qt3D expert or architect would be really handy here.
> 
> My ideas thus far:
> 
>    - Skip the frontend and introduce a QVirtualRealityAspect (which has to
>    be done earlier or later anyway). I wonder if this could apply transforms
> faster. I guess it would not run in the same thread as QRenderAspect? -
> Introduce a way to set a callback function for a shader uniform. The shader
> could query a very recent head-pose just before rendering ("late
> latching").
>    - Ensure that UpdateWorldTransformJob is executed after setting the
>    transform, but before rendering (would this kill parallelism?).
>    - Don't use synchronous rendering. But this would require me to heavily
>    patch QRenderAspect I guess. At least two not-yet-customizable steps must
> be altered:
>       - Render to a texture owned by the SDK which we only have a
> GLuint texture-id
>       of (this is only true for Oculus).
>       - Call a custom swapBuffers method.
> 
> Hope you enjoy it and don't get sea sick :-)!
> 
> 
> --
> 
> Daniel Bulla

--
Dr Sean Harmer | sean.harmer at kdab.com | Managing Director UK
Klarälvdalens Datakonsult AB, a KDAB Group company
Tel. UK +44 (0)1625 809908, Sweden (HQ) +46-563-540090
KDAB - Qt Experts - Platform-independent software solutions