[Development] [QtMultimedia] How to make to use a HW codecs by QML media player on Linux?

Denis Shienkov denis.shienkov at gmail.com
Thu May 26 11:58:10 CEST 2016 

I have created a BUG: https://bugreports.qt.io/browse/QTBUG-53646

2016-05-26 12:01 GMT+03:00 Denis Shienkov <denis.shienkov at gmail.com>:

> UPD: Seems the problem is ***NOT IN QtMM***..
>
> For example, when I have compiled an usual cpp-based QMediaPlayer example,
> and to set there:
>
> {quote}
> root at apalis-t30:~/Downloads# export
> QT_GSTREAMER_WIDGET_VIDEOSINK=nv_omx_hdmi_videosink
> root at apalis-t30:~/Downloads# export
> QT_GSTREAMER_WINDOW_VIDEOSINK=nv_omx_hdmi_videosink
> {quote}
>
> then CPU loas ~50%, I is good!
>
> So, seems, the problem is in QML && OpenGL!!! It is very-very sad....:(
>
> BR,
> Denis
>
>
> 2016-05-25 17:12 GMT+03:00 Denis Shienkov <denis.shienkov at gmail.com>:
>
>> Now, I have added this output:
>>
>> {code}
>> qDebug() << "*** TEST Best sink name" << elementName << ", m_videoSink:"
>> << m_videoSink;
>> {code}
>>
>> to the ctor of QGStreamerVideoOverlay in file
>> /src/gsttools/qgstreamervideooverlay.cpp,
>> and now I see this output:
>>
>> {quote}
>> root at apalis-t30:~/Downloads# ./video-player welcome.avi
>> QIconvCodec::convertToUnicode: using Latin-1 for conversion, iconv_open
>> failed
>> QIconvCodec::convertFromUnicode: using Latin-1 for conversion, iconv_open
>> failed
>> Unable to query physical screen size, defaulting to 100 dpi.
>> To override, set QT_QPA_EGLFS_PHYSICAL_WIDTH and
>> QT_QPA_EGLFS_PHYSICAL_HEIGHT (in millimeters).
>> Setting videosink to  "nv_omx_hdmi_videosink"
>> NvxLiteH264DecoderInit : Opening TVMR H264 block
>> NvxLiteH264DecoderInit : Opening TVMR H264 block
>> *** TEST Best sink name "nv_omx_hdmi_videosink" , m_videoSink: 0x15e1e8
>> *** TEST Best sink name "nv_omx_hdmi_videosink" , m_videoSink: 0x15e5c0
>> NvMMLiteOpen : Block : BlockType = 260
>> ++++++ NvAvpOpen +++++++
>> NvMMLiteBlockCreate : Block : BlockType = 260
>> ++++++++++++ TVMRFrameDelivery +++++++++++++++
>> BeginSequence 1920x720
>> pnvsi->nDecodeBuffers = 3
>> Display Resolution : (1920x720)
>> Display Aspect Ratio : (1920x720)
>> cbBeginSequence at 433: SurfaceLayout = 2
>> pStreamInfo->NumOfSurfaces = 7, MaxDPB = 24, InteraceStream = 0,
>> InterlaceEnabled = 0
>> Allocating new output: 1920x720 (x 9)
>> Warning: "A lot of buffers are being dropped."
>> Warning: "A lot of buffers are being dropped."
>> Warning: "A lot of buffers are being dropped."
>> Warning: "A lot of buffers are being dropped."
>> ^Croot at apalis-t30:~/Downloads#
>> {quote}
>>
>> where are:
>>
>> {quote}
>> *** TEST Best sink name "nv_omx_hdmi_videosink" , m_videoSink: 0x15e1e8
>> *** TEST Best sink name "nv_omx_hdmi_videosink" , m_videoSink: 0x15e5c0
>> {quote}
>>
>> BUT: Nothing changes, the CPU loading ~300% as before... So, seems, the
>> problem is *NOT IN VIDEO-SINK* selection...
>>
>> So, now, I am confused.. WTF? WTF? WTF? :(
>>
>> BR,
>> Denis
>>
>>
>>
>> 2016-05-25 16:53 GMT+03:00 Denis Shienkov <denis.shienkov at gmail.com>:
>>
>>> The solution from:
>>>
>>> https://devtalk.nvidia.com/default/topic/894891/jetson-tk1/gstreamer-1-0-and-qt5-video-playback/
>>>
>>> with adding the:
>>>
>>> {code}
>>>         qDebug() << "Setting videosink to " << videoSinkString;
>>>         qputenv("QT_GSTREAMER_WINDOW_VIDEOSINK",
>>> videoSinkString.toUtf8());
>>> {code}
>>>
>>> to the main.cpp, where videoSinkString is nv_omx_hdmi_videosink, does
>>> not work too...
>>>
>>> WTF?
>>>
>>>
>>>
>>> 2016-05-25 15:07 GMT+03:00 Denis Shienkov <denis.shienkov at gmail.com>:
>>>
>>>> Now, I have recompiled the QtMM plugin (with default gstreamer 0.10),
>>>> but with adding of debug macro DEBUG_PLAYBIN.
>>>> Then I got following log: https://paste.kde.org/p6ztesnsq
>>>>
>>>> Where I do not see any mentions about the selected gst sinks.. I see
>>>> only this:
>>>>
>>>> {quote}
>>>> Set video output: QGstreamerVideoRenderer(0x153fe0)
>>>> Current sink: fakesink0 0x13e590 pending:  0x0 new sink: fakesink0
>>>> 0x13e590
>>>> Video sink has not changed, skip video output reconfiguration
>>>> Video sink has chaged, reload video output
>>>> void QGstreamerPlayerSession::setVideoRenderer(QObject*)
>>>> Set video output: QGstreamerVideoRenderer(0x153fe0)
>>>> Current sink: fakesink0 0x13e590 pending:  0x0 new sink:
>>>> qvideosurfacegstsink0 0x1bc470
>>>> Reconfigure video output
>>>> The pipeline has not started yet, pending state:
>>>> QMediaPlayer::StoppedState
>>>> Video sink has chaged, reload video output
>>>> void QGstreamerPlayerSession::setVideoRenderer(QObject*)
>>>> Set video output: QGstreamerVideoRenderer(0x153fe0)
>>>> Current sink: qvideosurfacegstsink0 0x1bc470 pending:  0x0 new sink:
>>>> qvideosurfacegstsink0 0x1bc470
>>>> Video sink has not changed, skip video output reconfiguration
>>>> {quote}
>>>>
>>>> what is it 'fakesink', 'qvideosurfacegstsink0' ? where is any of
>>>> desired:
>>>>
>>>> {quote}
>>>> root at apalis-t30:~/Downloads# gst-inspect | grep sink | grep nv
>>>> omx:  nv_omx_audiosink: OpenMAX IL audiosink element
>>>> omx:  nv_gstbin_videosink: OpenMAX IL videosink element
>>>> omx:  nv_omx_videosink: OpenMAX IL videosink element
>>>> omx:  nv_omx_hdmi_videosink: OpenMAX IL hdmi videosink element
>>>> omx:  nv_omx_overlaysink: OpenMAX IL overlaysink element
>>>> omx:  nv_gl_eglimagesink: OpenMAX IL videosink element
>>>> nvxvimagesink:  nvxvimagesink: Video sink
>>>> root at apalis-t30:~/Downloads#
>>>> {quote}
>>>>
>>>> sinks? I do not understand nothing.. o_O How I can see the selected
>>>> sink name?
>>>>
>>>>
>>>>
>>>>
>>>> 2016-05-25 14:35 GMT+03:00 Denis Shienkov <denis.shienkov at gmail.com>:
>>>>
>>>>> > BTW: As I remember, I saw some environment variable which allows to
>>>>> setup custom video sink... I need to see sources of QtMM... if I'm not
>>>>> mistaked..
>>>>> maybe it will help...
>>>>>
>>>>> I have found this env variables:
>>>>>
>>>>> * QT_GSTREAMER_WIDGET_VIDEOSINK
>>>>> * QT_GSTREAMER_WINDOW_VIDEOSINK
>>>>>
>>>>> and try to use it:
>>>>>
>>>>> $export QT_GSTREAMER_WIDGET_VIDEOSINK=nv_omx_hdmi_videosink
>>>>> $export QT_GSTREAMER_WINDOW_VIDEOSINK=nv_omx_hdmi_videosink
>>>>> $./my-qml-video-player
>>>>>
>>>>> but, this does not help...
>>>>>
>>>>> BR,
>>>>> Denis
>>>>>
>>>>>
>>>>>
>>>>> 2016-05-25 8:30 GMT+03:00 Denis Shienkov <denis.shienkov at gmail.com>:
>>>>>
>>>>>> > Gstreamer 1.0+ goes some way to addressing these issues....
>>>>>>
>>>>>> The Apalis T30 board has GPU from the NVidia. So, when I use
>>>>>> 'gst-inspect' I see some of 'nv' sinks.
>>>>>> So, as I understand, the easy ways to do it are:
>>>>>>
>>>>>> 1) I should try to use GStreamer 1.0 (instead of 0.10) "as is" with
>>>>>> QtMM (rebuild QtMM with this GStreamer), at first stage.
>>>>>>
>>>>>> 2) I need try to change QtMM sources and try to set the specific
>>>>>> vendor-video sinks there, at second stage... (in case the first stage does
>>>>>> not work).
>>>>>>
>>>>>> Is it?
>>>>>>
>>>>>> BTW: As I remember, I saw some environment variable which allows to
>>>>>> setup custom video sink... I need to see sources of QtMM... if I'm not
>>>>>> mistaked..
>>>>>> maybe it will help...
>>>>>>
>>>>>> > but even within the standard packages there still seems to be
>>>>>> multiple APIs for codecs which produce EGL surfaces and none of them have
>>>>>> been utilized in QtMultimedia yet
>>>>>>
>>>>>> Could you please provide an examples of this API which is not adopted
>>>>>> in QtMM?
>>>>>>
>>>>>>
>>>>>>
>>>>>> BR,
>>>>>> Denis
>>>>>>
>>>>>>
>>>>>> 25.05.2016 4:44, Andrew den Exter пишет:
>>>>>>
>>>>>> The short answer is some amount of platform adaptation will be
>>>>>> required to enable hardware acceleration on your board.  Looking at the
>>>>>> page you linked to; that adaptation might be achieved by creating an
>>>>>> implementation of QGstBufferPoolPlugin (gstreamer 0.10 only) which exposes
>>>>>> EGL surfaces associated with the x-nv-yuv buffers produced by the
>>>>>> accelerated codec.  Or you might also need to create your own video node by
>>>>>> implementing QSGVideoNodeFactoryPlugin or something more invasive
>>>>>> may be required, it really depends on how the vendor implemented their
>>>>>> gstreamer plugins
>>>>>>
>>>>>> The reason hardware acceleration doesn't just work is that hardware
>>>>>> vendors have traditionally had to create extensions to gstreamer in order
>>>>>> to implement their hardware accelerated plugins, and these extensions tend
>>>>>> to require a vendor specific video sink to work.  If all you care about is
>>>>>> rendering to the full dimensions of a native window then autovideosink will
>>>>>> generally resolve to that vendor specific video sink and that's why the
>>>>>> standard example command line examples tend to work.  But when it comes to
>>>>>> something more complicated like compositing into an opengl scene or even as
>>>>>> simple as rendering to a sub rect of a window then you're into the realm of
>>>>>> vendor extensions and without a port to your specific hardware it probably
>>>>>> won't work.  Gstreamer 1.0+ goes some way to addressing these issues, but
>>>>>> even within the standard packages there still seems to be multiple APIs for
>>>>>> codecs which produce EGL surfaces and none of them have been utilized in
>>>>>> QtMultimedia yet.
>>>>>>
>>>>>>
>>>>>> Andrew
>>>>>>
>>>>>>
>>>>>> On Wed, May 25, 2016 at 12:40 AM, Denis Shienkov <
>>>>>> denis.shienkov at gmail.com> wrote:
>>>>>>
>>>>>>> Hi all.
>>>>>>>
>>>>>>> I have the Toradex (Apalis T30) embedded board:
>>>>>>>
>>>>>>> https://www.toradex.com/computer-on-modules/apalis-arm-family/nvidia-tegra-3
>>>>>>>
>>>>>>> which has:
>>>>>>>
>>>>>>> http://developer.toradex.com/files/toradex-dev/uploads/media/Colibri/Linux/Images/Apalis_T30_LinuxImageV2.6Beta1_20160331.tar.bz2
>>>>>>>
>>>>>>> Linux Image, with the GStreamer 0.10.
>>>>>>>
>>>>>>> Also I have compiled Qt 5.6 and installed it to the board.
>>>>>>>
>>>>>>> When I try to play any *.MP4 video file from the QML Video item:
>>>>>>> http://doc.qt.io/qt-5/qml-qtmultimedia-video.html
>>>>>>>
>>>>>>> I see, that my video has very-very lags, and I see (with htop
>>>>>>> utility) that CPU loading is ~300% (
>>>>>>> three cores running on ~100%).
>>>>>>>
>>>>>>> But, when I try to play the video, using the 'nvgstplayer' utility,
>>>>>>> I see that CPU loading is ~46% and all fine. Seems, the QML player loads
>>>>>>> the SW codecs instead of HW codecs...
>>>>>>>
>>>>>>> Is there are any way to say to QML player to use a HW codecs? Or,
>>>>>>> maybe, I need to fix a sources of QtMM (where I need to change it?) Or,
>>>>>>> maybe, is there are another way to do this?
>>>>>>>
>>>>>>> Otherwise, is just a **HELL** and there is no sense in QML
>>>>>>> QtMultimedia at all (But I need QML for anumations). And I do not know, how
>>>>>>> to fix it (I'm not expert in GStreamer).
>>>>>>>
>>>>>>> PS: I have tried the 'gst-launch' pipelines instead of
>>>>>>> 'nvgstplayer', as described here:
>>>>>>>
>>>>>>> http://developer.toradex.com/knowledge-base/video-playback-%28linux%29
>>>>>>> but it crashes with sigsegv....
>>>>>>>
>>>>>>> BR,
>>>>>>> Denis
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Development mailing list
>>>>>>> Development at qt-project.org
>>>>>>> http://lists.qt-project.org/mailman/listinfo/development
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20160526/f7c593be/attachment.html>

More information about the Development mailing list