[Interest] (code) Cannot map videoforames for QAbstractVideoSurface::present
Gil H
qt at tastytech.ca
Tue Jun 19 17:17:13 CEST 2018
I’m not sure about why you’re getting only a single frame, but the video frames coming back on Mac, I believe, are in YUV format. You need to do extra work to convert them to RGB for display or whatever else. Unfortunately, it’s next to impossible to work with video on Mac without knowing some basic OpenGL.
If the frame is totally black, it’s a different problem you’ll need to debug, but just a heads up that once you do get frames back, they won’t be what you expect…
From: Jason H <jhihn at gmx.com>
Date: Monday, June 18, 2018 at 12:34 PM
To: Gil H <qt at tastytech.ca>
Cc: Qt Interest <interest at qt-project.org>
Subject: Re: [Interest] (code) Cannot map videoforames for QAbstractVideoSurface::present
After adding your code, I get something... not sure what, but a black QImage... It could be a black initial frame but present() is only ever getting called once. I have no idea why. I should be getting 30fps (29.97). I am returning true.
Sent: Sunday, June 17, 2018 at 7:29 PM
From: "Gil H" <qt at tastytech.ca>
To: "Qt Interest" <interest at qt-project.org>
Subject: Re: [Interest] (code) Cannot map videoforames for QAbstractVideoSurface::present
Hi Jason, you need to check the handle type on the QVideoFrame. If input->handleType() == QAbstractVideoBuffer::NoHandle then you can map the memory directly, as you've tried. But on Mac, the handle type is GLTextureHandle and you need to write some OpenGL code to get the texture handle, read it, and then get at the pixels that way.
Here's a bit of code I used to do this:
// declare as class members QByteArray _frameBuffer; int _frameBytes; GLuint _fbo;
// in processing function: GLuint textureId = input->handle().toUInt(); if (_fbo == 0)
glGenFramebuffers(1, &_fbo);
_frameBytes = input->width()*input->height() * 4;
GLuint prevFbo; glGetIntegerv(GL_FRAMEBUFFER_BINDING, (GLint *) &prevFbo); glBindFramebuffer(GL_FRAMEBUFFER, _fbo); glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, textureId, 0);
#ifdef Q_OS_IOS
glReadPixels(0, 0, input->width(), input->height(), GL_RGBA, GL_UNSIGNED_BYTE, _frameBuffer.data());
#else
glReadPixels(0, 0, input->width(), input->height(), GL_BGRA_EXT, GL_UNSIGNED_BYTE, _frameBuffer.data());
#endif
glBindFramebuffer(GL_FRAMEBUFFER, prevFbo);
// at this point _frameBuffer bytearray contains your pixels
Hope this saves you some of the headaches I went through as well!
Gil
On 6/14/18, 11:26 AM, “Interest on behalf of Jason H” <interest-bounces+qt=tastytech.ca at qt-project.org on behalf of jhihn at gmx.com> wrote:
Yes, I essentially have that (your message just made it to my email after I sent my code)
I don't know why map() would fail, and why bits() would not be valid.
I'm on OSX, reading from an MP4...
> Sent: Thursday, June 14, 2018 at 7:12 AM
> From: "Igor Mironchik" <igor.mironchik at gmail.com>
> To: interest at qt-project.org
> Subject: Re: [Interest] (code) Cannot map videoforames for QAbstractVideoSurface::present
>
> Hi,
>
> Start from here:
>
> bool
> Frames::present( const QVideoFrame & frame )
> {
> if( !isActive() )
> return false;
>
> QVideoFrame f = frame;
> f.map( QAbstractVideoBuffer::ReadOnly );
>
> QImage image( f.bits(), f.width(), f.height(), f.bytesPerLine(),
> QVideoFrame::imageFormatFromPixelFormat( f.pixelFormat() ) );
>
> f.unmap();
>
> emit newFrame( image.copy( image.rect() ) );
>
> return true;
> }
>
> This works for me...
>
>
> 14-Jun-18 17:52, Jason H пишет:
> > I have no idea what I'm doing wrong. Can someone this and let me know?
> > The frames I'm getting are BGR32, so I have special code for that.
> > But I'm getting the message "QVideoFrame::unmap() was called more times then QVideoFrame::map()"
> > Which is obviously not the case.
> >
> > To restate the problems:
> > 1. I only get to present() once
> > 2. I's failing to map frames
> > 3. "then" in the message should be "than"
> >
> > Here's the first 3 lines of the .pro and complete code:
> > extractframes.pro-----
> > QT += multimedia
> > CONFIG += c++11 console
> > CONFIG -= app_bundle
> >
> > main.cpp--------------
> > #include <QAbstractVideoSurface>
> > #include <QGuiApplication>
> > #include <QMediaPlayer>
> > #include <QVideoSurfaceFormat>
> >
> > #include <QtDebug>
> >
> > class VideoSurface: public QAbstractVideoSurface {
> > int _frameNumber;
> > QVideoSurfaceFormat _format;
> > QString _prefix;
> > int _frameInterval;
> > public:
> > VideoSurface(const QString &prefix, int frameInterval): _prefix(prefix), _frameInterval(frameInterval) { }
> >
> > bool present(const QVideoFrame &frame) override {
> > qDebug() << Q_FUNC_INFO << _frameNumber << frame.isValid();
> >
> > if (_frameNumber % _frameInterval == 0) {
> > QVideoFrame frame2(frame);
> > frame2.map(QAbstractVideoBuffer::ReadOnly);
> > QImage image;
> >
> > if (frame2.pixelFormat() == QVideoFrame::Format_BGR32) {
> > image = QImage(frame2.bits(), frame2.width(), frame2.height(), QImage::Format_RGB32);
> > image = image.rgbSwapped();
> > } else {
> > image = QImage(frame2.bits(), frame2.width(), frame2.height(), QVideoFrame::imageFormatFromPixelFormat(frame2.pixelFormat()));
> > }
> >
> > frame2.unmap();
> > QString filename = QString(_prefix).arg(_frameNumber);
> > bool result = image.save(filename);
> > qDebug() << Q_FUNC_INFO << filename << result;
> > }
> > _frameNumber ++;
> > return true;
> > }
> >
> > bool start(const QVideoSurfaceFormat &format) override {
> > qDebug() << Q_FUNC_INFO;
> > _format = format;
> > _frameNumber = 0;
> > QAbstractVideoSurface::start(format);
> > return true;
> > }
> > void stop() override {
> > qDebug() << Q_FUNC_INFO;
> > QAbstractVideoSurface::stop();
> > }
> > QList<QVideoFrame::PixelFormat> supportedPixelFormats(QAbstractVideoBuffer::HandleType handleType) const override
> > {
> > Q_UNUSED(handleType);
> > return QList<QVideoFrame::PixelFormat>()
> > << QVideoFrame::Format_ARGB32
> > << QVideoFrame::Format_ARGB32_Premultiplied
> > << QVideoFrame::Format_RGB32
> > << QVideoFrame::Format_RGB24
> > << QVideoFrame::Format_RGB565
> > << QVideoFrame::Format_RGB555
> > << QVideoFrame::Format_ARGB8565_Premultiplied
> > << QVideoFrame::Format_BGRA32
> > << QVideoFrame::Format_BGRA32_Premultiplied
> > << QVideoFrame::Format_BGR32
> > << QVideoFrame::Format_BGR24
> > << QVideoFrame::Format_BGR565
> > << QVideoFrame::Format_BGR555
> > << QVideoFrame::Format_BGRA5658_Premultiplied
> > << QVideoFrame::Format_AYUV444
> > << QVideoFrame::Format_AYUV444_Premultiplied
> > << QVideoFrame::Format_YUV444
> > << QVideoFrame::Format_YUV420P
> > << QVideoFrame::Format_YV12
> > << QVideoFrame::Format_UYVY
> > << QVideoFrame::Format_YUYV
> > << QVideoFrame::Format_NV12
> > << QVideoFrame::Format_NV21
> > << QVideoFrame::Format_IMC1
> > << QVideoFrame::Format_IMC2
> > << QVideoFrame::Format_IMC3
> > << QVideoFrame::Format_IMC4
> > << QVideoFrame::Format_Y8
> > << QVideoFrame::Format_Y16
> > << QVideoFrame::Format_Jpeg
> > << QVideoFrame::Format_CameraRaw
> > << QVideoFrame::Format_AdobeDng;
> > }
> >
> >
> > };
> >
> > // Command line: extractframes format interval
> > // format:QString, with %1 = frame number
> > // Example:
> > // extractframes filename.mp4 image_%1.png 30
> >
> > int main(int argc, char *argv[])
> > {
> > QGuiApplication a(argc, argv);
> > QMediaPlayer player;
> > VideoSurface surface(argv[2], QString(argv[3]).toInt());
> > QObject::connect(&player, &QMediaPlayer::mediaStatusChanged, [=]( QMediaPlayer::MediaStatus status ) {
> > qDebug() << Q_FUNC_INFO << status;
> > });
> > QObject::connect(&player, QOverload<QMediaPlayer::Error>::of(&QMediaPlayer::error), [=]( QMediaPlayer::Error error ) {
> > qDebug() << Q_FUNC_INFO << error;
> > });
> > QObject::connect(&player, &QMediaPlayer::stateChanged, [=]( const QMediaPlayer::State &state ) {
> > qDebug() << Q_FUNC_INFO << state;
> > if (QMediaPlayer::StoppedState == state) qApp->quit();
> > });
> >
> > player.setMedia(QUrl::fromLocalFile(argv[1]));
> > player.setVideoOutput(&surface);
> > player.setMuted(true);
> >
> > player.play();
> >
> > return a.exec();
> > }
> > _______________________________________________
> > Interest mailing list
> > Interest at qt-project.org
> > http://lists.qt-project.org/mailman/listinfo/interest
>
> _______________________________________________
> Interest mailing list
> Interest at qt-project.org
> http://lists.qt-project.org/mailman/listinfo/interest
>
_______________________________________________
Interest mailing list
Interest at qt-project.org
http://lists.qt-project.org/mailman/listinfo/interest
_______________________________________________ Interest mailing list Interest at qt-project.org http://lists.qt-project.org/mailman/listinfo/interest
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/interest/attachments/20180619/d74c5282/attachment.html>
More information about the Interest
mailing list