[Interest] [Qt3D] Virtual Reality (Vive/Oculus) Example

[Vlad Stelmahovsky]

Hi

great news. Hope you guys will combine efforts to make the solution
multiplatform. For example, compatible with GearVR
btw, did you checked this project: https://github.com/google/lullaby ?


On Tue, Jul 18, 2017 at 11:22 AM, Sean Harmer <sean.harmer at kdab.com> wrote:

> Hi Daniel,
>
> awesome, thank you for this! We too have been starting work on something
> similar:
>
> https://codereview.qt-project.org/#/c/195892/
>
> The latency you mention can be avoided if we update the head tracking
> directly
> on the qt3d renderer backend. That way it can be used immediately on the
> current frame. The frontend doesn't necessarily need to know the eye
> matrices
> but we can make tracking these on the frontend opt-in if necessary (if
> something wants to bind to them - but even that use case may become
> obsolete
> soon with some backend bindings work we have ongoing).
>
> I've pointed a couple of colleagues at your work and I'll try it out myself
> too once I get a few urgent things sorted locally.
>
> You may not have seen it but, we now have some infrastructure for renderer
> plugins to allow deeper integration than having to go via the existing
> public
> Qt 3D frontend types such as QTransform. I think this will be the way to
> remove the latency you mention.
>
> We should certainly coordinate efforts and see how we can make this ready
> for
> production use. Feel free to ping us on development@ or on irc (#qt-3d) or
> even direct email if you want to discuss this.
>
> Thank you once again for contributing!
>
> Sean
>
> On Tuesday 18 July 2017 10:29:25 Daniel Bulla wrote:
> > Hi,
> >
> > I started adding support for OpenVR/Vive and native OculusSDK to Qt3D.
> >
> > You can find it on Github: https://github.com/dabulla/qt3d-vr-example
> >
> > It can easily be setup, as described in the Readme.
> >
> >
> > *What works?*
> >
> >    - Stereoscopic 3D Rendering of Qt3D Qml scenes in the Headset.
> >    - Headtracking
> >
> > *What is still in progress?*
> >
> >    - Headtracking has high latency
> >    - Mirroring to Monitor
> >    - Motion controllers + Input
> >    - ... (lots of things imaginable)
> >
> > *Implementation*
> >
> > There is QHeadmountedDisplay which does the job of a Qt3DQuickWindow but
> > uses Direct-Mode Rendering. It can be used just like a Window that is
> > placed at the headset and will sync to it's refresh rate. OpenVR or
> > OculusSDK (aka 'SDKs') also handle distortion correctly, so there is no
> > need to do this manually.
> >
> > The rendered Qt3D Qml scene must use a VrCamera to enable head-tracking
> and
> > correct stereoscopic rendering. VrCamera configures projection-parameters
> > for the headset by querying the SDK.
> >
> > I only tested this on windows at the moment.
> >
> > *Future*
> >
> > I'd really like to proceed with the implementation and look forward to
> > getting some feedback at this point. I read the qt styling guide and
> > started a clean implementation on github and wanted to contribute it to
> > qt3d at some point. I'm open for suggestions and could also need advice
> > with the following problem:
> >
> > *Tackling latency*
> >
> > The pose read from the headset needs approx. 3 frames before it finally
> > ends up as a shader-uniform for rendering. This takes 3 frames too long
> (it
> > is extremely noticable and you get nausea very quickly)!
> >
> >    - In the main-thread "getEyeMatrices" is called, this gets the
> >    prediction of the head pose for the very next frame.
> >    - The matrices are handed to Qt3D using the QTransform of the
> >    Camera-Entity (VrCamera).
> >
> > I suspect that it takes several frames before the transform ends up as a
> > uniform in the shader and wonder how I can tweak Qt3D to keep this
> latency
> > down. A tip of a Qt3D expert or architect would be really handy here.
> >
> > My ideas thus far:
> >
> >    - Skip the frontend and introduce a QVirtualRealityAspect (which has
> to
> >    be done earlier or later anyway). I wonder if this could apply
> transforms
> > faster. I guess it would not run in the same thread as QRenderAspect? -
> > Introduce a way to set a callback function for a shader uniform. The
> shader
> > could query a very recent head-pose just before rendering ("late
> > latching").
> >    - Ensure that UpdateWorldTransformJob is executed after setting the
> >    transform, but before rendering (would this kill parallelism?).
> >    - Don't use synchronous rendering. But this would require me to
> heavily
> >    patch QRenderAspect I guess. At least two not-yet-customizable steps
> must
> > be altered:
> >       - Render to a texture owned by the SDK which we only have a
> > GLuint texture-id
> >       of (this is only true for Oculus).
> >       - Call a custom swapBuffers method.
> >
> > Hope you enjoy it and don't get sea sick :-)!
> >
> >
> > --
> >
> > Daniel Bulla
>
> --
> Dr Sean Harmer | sean.harmer at kdab.com | Managing Director UK
> Klarälvdalens Datakonsult AB, a KDAB Group company
> Tel. UK +44 (0)1625 809908, Sweden (HQ) +46-563-540090
> KDAB - Qt Experts - Platform-independent software solutions
> _______________________________________________
> Interest mailing list
> Interest at qt-project.org
> http://lists.qt-project.org/mailman/listinfo/interest
>



-- 
Best regards,
Vlad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/interest/attachments/20170718/f1edd09c/attachment.html>