• State Resolved
  • Locked Locked
  • Replies 28 replies
  • Answers 3 answers
  • Subscribers 105 subscribers
  • Views 19705 views
  • Users 0 members are here
Support feedback
Options
Options
Similar topics

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM5728 using VPE and Ducati to encode 1080p30 video from USB camera

Lucas Kinion
Prodigy 130 points
Other Parts Discussed in Thread: AM5728

Hello,

I have a couple of questions regarding the use of the VPE and Ducatih264enc elements together in Gstreamer on the AM5728 processor using the TI Processor SDK Linux distribution. I'm trying to generate a pipeline that pulls in raw video from a USB 3.0 camera, uses the VPE element to convert it from YUYV to NV12, and encodes it using the Ducati H264 hardware encoder. I'm having issues interfacing with the VPE element, specifically relating to DMA buffer allocation.

The kernel module appears to be loaded correctly, so my issue seems to be with my userspace implementation. My ultimate goal is to use the Appsrc with a camera that has a custom Api, but if I can get a pipeline working with V4l2src or Videotestsrc, that would help as well. Is there some documentation or sample code on allocating and sharing buffers between my source, VPE, and encoder properly?

My pipeline looks roughly like this
Source (v4l2/app/videotest) ! capsfilter YUYV 1920x1080@30p ! vpe ! capsfilter NV12 1920x1080@30p ! ducatih264enc ! h264parse ! rtph264pay ! udpsink

And the errors that this returns are generally related to the DMA buffer handling, so I believe that's where my problem lies. Depending on what options I pass to the VPE and the source, I get errors such as "basesrc internal data flow error,", "vpebufferpool alloc function failed", or "gstvpe this plugin does not support buffers not allocated by self"


I have not found any sample pipelines or code that use the VPE in conjunction with a raw video source and the Ducati encoder, so any info on that would be appreciated. Please let me know if you have any info that could help me solve my problem.

Thank you
Lucas 

  • 0
    TI__Guru**** 393215 points
    Hi Lucas,

    I will forward this to our video experts.
  • 0 in reply to Biser Gatchev-XID
    TI__Mastermind 22916 points

    Hi Lucas,

    Which version of PSDK are you trying? Here's one example gst pipeline of simultaneous encode decode. I tried it at my end using PSDK 2.2 and do not see any warnings or errors thrown. You will need weston running to run this example pipeline as this is using waylandsink.

    gst-launch-1.0 -v -e videotestsrc num-buffers=500 ! queue ! 'video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080, framerate=(fraction)22/1' ! queue ! ducatih264enc ! queue ! h264parse ! queue ! ducatih264dec ! queue ! waylandsink

     

  • 0 in reply to Manisha Agrawal
    Prodigy 130 points

    I am using a distribution by Compulab, who makes the board I'm using, which is based on PSDK version 2.00.01. I'm not having issues with the encode or decode on its own. Rather, I'm having problems with the connection to VPE. I can run your example just fine (using a different sink since I'm using X without Weston), but as soon as I try to modify it to something like this is when I have issues:

    gst-launch-1.0 -v -e videotestsrc num-buffers=500 ! queue ! 'video/x-raw,format=(string)YUYV,width=(int)1920,height=(int)1080,framerate=(int)22/1' ! vpe ! 'video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080, framerate=(fraction)22/1' ! queue ! ducatih264enc ! queue ! h264parse ! queue ! ducatih264dec ! queue ! waylandsink

    So my question is more specifically related to putting VPE in the pipeline. Sorry if I was unclear

    Thanks

    Lucas

  • 0 in reply to Lucas Kinion
    TI__Mastermind 22916 points
    vpe plugin doesn't connect to every other gstreamer plugin due to the way it accepts buffer. It's only able to connect to ti provided hardware accelerated gstreamer plugin. The v4l2src warpper for vip was modified to be able to connect to vpe plugin. We plan to remove this restriction when we migrate to gstreamer version 1.6. Expect the changes to be incorporated in 3Q PSDK release.

    This example pipleline works well using capture from VIP on GP EVM-
    gst-launch-1.0 –v v4l2src device=/dev/video1 num-buffers=1000 io-mode=4 ! 'video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=30/1' ! queue ! vpe num-input-buffers=8 ! 'video/x-raw, format=(string)NV12, width=(int)1280, height=(int)720' ! ducatih264enc bitrate=4000 ! queue ! h264parse ! queue ! ducatih264dec ! queue ! vpe ! 'video/x-raw, format=(string)NV12, width=(int)800, height=(int)480' ! queue ! waylandsink
  • 0 in reply to Manisha Agrawal
    Prodigy 130 points

    Thanks for the help Manisha, it has been very valuable so far.

    So if I'm able to use the VIP (which I am not yet able to with the version of board I am using), then the VPE should be compatible. I assume that doesn't apply to USB 3.0 V4L2 sources as well, correct? Is there no way to use VPE with different sources using manual buffer allocation in a C++ Gstreamer application?

    If neither of these is possible, do you have any other tips to help me encode 1080p@30fps using a different method? I'm running into processor limitations using videoconvert instead of vpe to convert YUYV into NV12, and I haven't found any ways to optimize it further. I tried to attached my code for reference, if it helps.

    Thank you 

    Lucas

    pixelink_test.cpp

  • 0 in reply to Lucas Kinion
    TI__Mastermind 22916 points
    Hi Lucas,

    Yes, once you use vip for video capture, you should be able to connect the VIP with VPE. Infact, VIP can do the YUYV capture format to NV12 conversion for you. USB3 v4l2src also will have problem. For USB3 v4l2src discussion, check the discussions here - e2e.ti.com/.../1908760

    Let me see if I can get the changes required at src plugin side to get it connected with vpe.

    Regards,
    Manisha
  • 0 in reply to Manisha Agrawal
    Prodigy 130 points
    Hi Manisha

    Are there any updates on Gstreamer 1.6 or the plugin changes? I noticed PSDK 3.0 came out; does it contain the update, or will that still come at a later date?

    Thanks
    Lucas
  • 0 in reply to Lucas Kinion
    TI__Mastermind 22916 points

    Hi Lucas,

    Migration to gstreamer 1.6 is planned for PSDK 3.1 release scheduled end of 3Q, 2016. 

    Regards,

    Manisha

  • 0 in reply to Manisha Agrawal
    Prodigy 10 points

    Hi Manisha,


    I'm working with Lucas to get the VPE plugin working with gstreamer  and was wondering if PSDK 3.1 is out yet, or if possible an early version of if with a new plugin available?

    Dave

  • 0 in reply to David Retherford
    TI__Mastermind 22916 points

    PSDK 3.1 is expected to be released in next couple of days. In this release vpe plugin can work with other plugins that supports dmabuf.

  • 0 in reply to Manisha Agrawal
    Prodigy 130 points

    Manisha,

    Thanks for the work on the PSDK update. I have been experimenting with building and using it over the past couple of weeks, and I'm still having issues. To show where I'm having trouble, I'm following the "Capture and Display Fullscreen" pipeline on the Multimedia Training page:

    Capture and Display Fullscreen
  target #  gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=1000 io-mode=4 ! 'video/x-raw, \
format=(string)YUY2, width=(int)1280, height=(int)720' ! vpe num-input-buffers=8 ! queue ! kmssink

    But I get the following errors. I'll copy and paste the output with debug level 2, and attempt to attach the console output with debug level 4.

    Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
0:00:00.644739255   809   0x170660 WARN          v4l2bufferpool gstv4l2bufferpool.c:1197:gst_v4l2_buffer_pool_dqbuf:<v4l2src0:pool:src> Driver should never set v4l2_buffer.field to ANY
0:00:00.644852308   809   0x170660 WARN          v4l2bufferpool gstv4l2bufferpool.c:1949:gst_v4l2_buffer_pool_process:<v4l2src0:pool:src> Dropping corrupted buffer without payload
0:00:00.644932665   809   0x170660 WARN          v4l2bufferpool gstv4l2bufferpool.c:1949:gst_v4l2_buffer_pool_process:<v4l2src0:pool:src> Dropping corrupted buffer without payload
0:00:01.344712138   809   0x170660 WARN                     vpe gstvpebufferpool.c:607:gst_vpe_buffer_pool_import: Allocating a new input buffer index: 0/128, 0
0:00:01.380326540   809   0x170660 ERROR                    vpe gstvpebufferpool.c:519:gst_vpe_buffer_pool_queue: vpebufferpool: QBUF failed: Bad address, index = 31
0:00:01.380660820   809   0x170600 WARN              bufferpool gstbufferpool.c:300:do_alloc_buffer:<vpebufferpool1> alloc function failed
0:00:01.380838127   809   0x170660 WARN                 basesrc gstbasesrc.c:2948:gst_base_src_loop:<v4l2src0> error: Internal data flow error.
0:00:01.380882372   809   0x170660 WARN                 basesrc gstbasesrc.c:2948:gst_base_src_loop:<v4l2src0> error: streaming task paused, reason error (-5)
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data flow error.
Additional debug info:
gstbasesrc.c(2948): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming task paused, reason error (-5)
Execution ended after 0:00:01.175735469
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

    vpepipeline_detailed.txt
    Fullscreen
    1
    2
    3
    4
    0:00:00.000228710 ·[333m 818·[00m 0x2fc00 ·[36mINFO ·[00m ·[00;01;31m GST_INIT gst.c:511:init_pre:·[00m Initializing GStreamer Core Library version 1.8.3
    0:00:00.000426024 ·[333m 818·[00m 0x2fc00 ·[36mINFO ·[00m ·[00;01;31m GST_INIT gst.c:512:init_pre:·[00m Using library installed in /usr/lib/arm-linux-gnueabihf
    0:00:00.000474336 ·[333m 818·[00m 0x2fc00 ·[36mINFO ·[00m ·[00;01;31m GST_INIT gst.c:523:init_pre:·[00m Linux cl-som-am57x-sid 4.4.19-cl-som-am57x-ti-03.01.00.06-gdb0b54cdad #13 SMP PREEMPT Thu Oct 20 11:12:59 CDT 2016 armv7l
    0:00:00.001448222 ·[333m 818·[00m 0x2fc00 ·[36mINFO ·[00m ·[00;01;31m GST_INIT gstmessage.c:119:_priv_gst_message_initialize:·[00m init messages
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    It appears that with a USB V4L2 camera, it is still unable to allocate a dmabuf to share with VPE, and I don't see an option for VPE to allocate the buffer and pass it to the v4l2src. Am I missing something obvious, or is there something else I need to be doing?

    Thanks

    Lucas

  • 0 in reply to Lucas Kinion
    Prodigy 130 points
    Is it going to take a modification to gst-v4l2src, or even to videobuf-core? It appears that gstv4l2bufferpool.c is setting the CORRUPTED flag which throws an error, and I suspect it's related to the fact that videobuf-core.c says that it doesn't handle DMABUF.

    Or perhaps do I need to figure out how to let vpe or ducatih264enc allocate the buffer and then share it back to v4l2src?
  • 0 in reply to Lucas Kinion
    TI__Mastermind 22916 points
    From some of the log info, it seems QBUF is failing. Looks like dmabuf operations are failing. Is the USB driver capable of exporting and importing dmabufs?

    Can you please share log with just v4l2src debug (--gst-debug=*v4l2*:5). Add –v option too for negotiation info. Also, please share video4linux/video1 debug log.
  • 0 in reply to Manisha Agrawal
    Prodigy 130 points

    I've attached the outputs from gst_debug and from dmesg. I'm not familiar with the USB driver, but I'm using whatever came with the Processor SDK 3.01. I assumed it was capable of it, but I can start researching on it if you think that's the missing link.

    v4l2_debug.txt

    v4l_dmesg.txt
    Fullscreen
    1
    2
    3
    4
    5
    [18870.681864] video1: VIDIOC_QUERYCAP
    [18870.695982] video1: VIDIOC_QUERYCAP
    [18870.696096] video1: VIDIOC_ENUMINPUT
    [18870.696173] video1: VIDIOC_ENUMINPUT: error -22
    [18870.696227] video1: VIDIOC_ENUMSTD: error -25
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  • 0 in reply to Lucas Kinion
    Prodigy 130 points

    Also, is v4l2video0convert something I should try troubleshooting as well? According to the description, it sounds like it points to vpe but it has some more parameters that can be set.

  • 0 in reply to Lucas Kinion
    TI__Mastermind 22916 points
    Hi Lucas,

    Sorry for the late reply. I expect to get usb camera at my end, replicate the issue and look into it by end of this week. I shall update you on the suggestion to resolve the issue as soon I have some answer for you.

    Regards,
    Manisha
  • 0 in reply to Manisha Agrawal
    Prodigy 165 points
    HI Manisha,

    I am also looking for same solution to get work . (i.e USB 3.0 camera pipeline working with VPE and Ducati) , Can you share when PSDK 3.1 will be released for AM5728 .

    Regards
    Pallab Sarkar

    e2e.ti.com/.../553891
  • 0 in reply to Pallab Sarkar
    TI__Mastermind 22916 points
    Pallab,

    PSDK 3.1 is already released and available for download on ti.com. USB camera is still having issue connecting to vpe in gst pipeline for this version. We are looking into this.

    Regards,
    Manisha
  • 0 in reply to Manisha Agrawal
    TI__Mastermind 22916 points

    Root caused the issue to be with usb/uvc driver. The usb/uvc driver was using vmalloc to allocate the video buffers and hence large allocated buffers were not contiguous.  VPE driver need contiguous buffers to work on and hence the handshake was failing. I modified the usb/uvc driver to work in dma-contiguous mode and with that change, below gstreamer pipeline works great. Attached patch of the changes done to usb/uvc driver. 

    gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=1000 io-mode=4 ! 'video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720' ! queue !  vpe ! queue ! kmssink

    usb_capture.txt
    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    diff --git a/drivers/media/usb/uvc/Kconfig b/drivers/media/usb/uvc/Kconfig
    index 6ed85efa..d7eb610 100644
    --- a/drivers/media/usb/uvc/Kconfig
    +++ b/drivers/media/usb/uvc/Kconfig
    @@ -1,7 +1,8 @@
    config USB_VIDEO_CLASS
    tristate "USB Video Class (UVC)"
    depends on VIDEO_V4L2
    - select VIDEOBUF2_VMALLOC
    +#select VIDEOBUF2_VMALLOC
    + select VIDEOBUF2_DMA_CONTIG
    ---help---
    Support for the USB Video Class (UVC). Currently only video
    input devices, such as webcams, are supported.
    diff --git a/drivers/media/usb/uvc/uvc_driver.c b/drivers/media/usb/uvc/uvc_driver.c
    index d11fd6a..ed04bcc 100644
    --- a/drivers/media/usb/uvc/uvc_driver.c
    +++ b/drivers/media/usb/uvc/uvc_driver.c
    @@ -23,6 +23,7 @@
    #include <linux/version.h>
    #include <asm/unaligned.h>
    +#include <media/videobuf2-dma-contig.h>
    #include <media/v4l2-common.h>
    #include "uvcvideo.h"
    @@ -1924,6 +1925,8 @@ static int uvc_probe(struct usb_interface *intf,
    if (v4l2_device_register(&intf->dev, &dev->vdev) < 0)
    goto error;
    + dev->alloc_ctx = vb2_dma_contig_init_ctx(&intf->dev);
    + intf->dev.coherent_dma_mask = DMA_BIT_MASK(32);
    /* Initialize controls. */
    if (uvc_ctrl_init_device(dev) < 0)
    goto error;
    diff --git a/drivers/media/usb/uvc/uvc_queue.c b/drivers/media/usb/uvc/uvc_queue.c
    index cfb868a..b3278c0 100644
    --- a/drivers/media/usb/uvc/uvc_queue.c
    +++ b/drivers/media/usb/uvc/uvc_queue.c
    @@ -18,10 +18,11 @@
    #include <linux/module.h>
    #include <linux/usb.h>
    #include <linux/videodev2.h>
    -#include <linux/vmalloc.h>
    +//#include <linux/vmalloc.h>
    #include <linux/wait.h>
    #include <media/videobuf2-v4l2.h>
    -#include <media/videobuf2-vmalloc.h>
    +//#include <media/videobuf2-vmalloc.h>
    +#include <media/videobuf2-dma-contig.h>
    #include "uvcvideo.h"
    @@ -80,7 +81,7 @@ static int uvc_queue_setup(struct vb2_queue *vq, const void *parg,
    /* Make sure the image size is large enough. */
    if (fmt && fmt->fmt.pix.sizeimage < stream->ctrl.dwMaxVideoFrameSize)
    return -EINVAL;
    -
    + alloc_ctxs[0] = stream->dev->alloc_ctx;
    *nplanes = 1;
    sizes[0] = fmt ? fmt->fmt.pix.sizeimage
    @@ -202,7 +203,7 @@ int uvc_queue_init(struct uvc_video_queue *queue, enum v4l2_buf_type type,
    queue->queue.drv_priv = queue;
    queue->queue.buf_struct_size = sizeof(struct uvc_buffer);
    queue->queue.ops = &uvc_queue_qops;
    - queue->queue.mem_ops = &vb2_vmalloc_memops;
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  • 0 in reply to Manisha Agrawal
    Prodigy 130 points

    Thank you so much for the help Manisha. That appears to have fixed the problem. The VPE returns data now. It will take some tweaking to make sure the memory operations are configured correctly so it can run at full speed, and interface it with non-V4L2 cameras, but this is a great start.


    Thanks

    Lucas

  • 0 in reply to Lucas Kinion
    TI__Mastermind 22916 points
    Thank you for the update Lucas! I am glad to know that the patch helped to resolve the issue at your end.
  • 0 in reply to Manisha Agrawal
    Intellectual 650 points
    Hello manisha,

    I have the same problem and i want to know how to use the "usb_capture.txt" file to fix this issue.
    Thank you,

    BR :)
    Faiez
  • 0 in reply to Faiez
    TI__Mastermind 22916 points
    Faiez,

    Copy the file usb_capture under top linux directory (eg - on PLSDK 3.3 - board-support/linux-4.4.41+gitAUTOINC+f9f6f0db2d-gf9f6f0db2d directory), then on linux terminal window, make your current directory the directory where the file is residing
    #cd ti-processor-sdk-linux-am57xx-evm-03.03.00.04/board-support/linux-4.4.41+gitAUTOINC+f9f6f0db2d-gf9f6f0db2d

    After that use following command to apply the patch.
    #patch -p1 < usb_capture.txt

    If their is no incompatibility between the source code on top of which patch was created and the source code on which you are applying the patch, patch application will be successful, else you may see some error. In that case you should apply the patch manually, by looking at the modifications in patch file and reading the new source file on which you want to apply the patch.
  • 0 in reply to Manisha Agrawal
    Intellectual 650 points
    Manisha,

    i don't have this directory ("ti-processor-sdk-linux-am57xx-evm-03.03.00.04/board-support/linux-4.4.41+gitAUTOINC+f9f6f0db2d-gf9f6f0db2d") in the latest processor SDK and i have this:

    root@am57xx-evm:/# ls
    + gitAUTOINC mnt ti-processor-sdk-linux-am57xx-evm-03.03.00.04
    bin home opt tmp
    board-support include proc usb_capture.txt
    boot lib run usr
    dev linux-4.4.41 sbin var
    etc lost+found srv www
    f9f6f0db2d-gf9f6f0db2d media sys
    root@am57xx-evm:/#

    Best regards,
    Faiez :)
  • 0 in reply to Faiez
    TI__Mastermind 22916 points

    Faiez,

    Looks like you are not that familiar with linux. Please do google on basic linux commands and patching the files. 

    Regards,

    Manisha

  • 0 in reply to Faiez
    TI__Guru* 80190 points

    Hello,

    Faiez said:
    ti-processor-sdk-linux-am57xx-evm-03.03.00.04/board-support/linux-4.4.41+gitAUTOINC+f9f6f0db2d-gf9f6f0db2d

    This folder is allocated on the PC where your PSDK is installed not on the board.

    If you do not have PSDK installed on the PC please install it. 

    BR
    Margarita

  • 0 in reply to Manisha Agrawal
    Intellectual 840 points

    What version of the SDK is the patch for?I used this patch in sdk3.0,but it did not work.So,I want to know what version of the SDK is appropriate with the patch which you provided?Thank you。

  • 0 in reply to hsg
    TI__Mastermind 22916 points
    The patch was tested on PLSDK version 3.1.