Tool/software:
Hi,
We are using edgeai-gst-plugins in our GStreamer pipeline with OpenVX and v4l2h265enc. We noticed image corruption when v4l2h265enc uses dma-buf import. Our "simplified" pipeline:
gst-launch-1.0 videotestsrc pattern=smpte num-buffers=30 ! video/x-raw, format=NV12, width=1920, height=1080 , framerate=30/1 ! tiovxmultiscaler ! video/x-raw, format=NV12, width=1280, height=720, framerate=30/1 ! OUR_CUSTOM_ELEMENT ! tiovxmemalloc ! video/x-raw,stride-y-align=32 ! v4l2h265enc output-io-mode=5 ! h265parse ! mp4mux ! filesink location=/tmp/out.mp4 sync=false -v
With this pipeline, we see image corruption like shown below:
The corruption pattern is not consistent and changes from frame to frame. It looks like a cache coherency issue. However, the pipeline that doesn't use dma-buf import (output-io-mode=0) works without problems. But this causes video frame copying, which we need to avoid for performance reasons.
gst-launch-1.0 videotestsrc pattern=smpte num-buffers=30 ! video/x-raw, format=NV12, width=1920, height=1080 , framerate=30/1 ! tiovxmultiscaler ! video/x-raw, format=NV12, width=1280, height=720, framerate=30/1 ! OUR_CUSTOM_ELEMENT ! tiovxmemalloc ! video/x-raw,stride-y-align=32 ! v4l2h265enc output-io-mode=0 ! h265parse ! mp4mux ! filesink location=/tmp/out.mp4 sync=false -v
OUR_CUSTOM_ELEMENT: Our custom GStreamer element is very simple in this case. It just gets the vx_image from the buffer, maps it, sets the content, and unmaps it. Below is example code that shows the corruption. In both working and non-working cases, our custom element is the same. The corruption only happens with v4l2h265enc output-io-mode=5.
static GstFlowReturn gst_tiovx_custom_transform_ip(GstBaseTransform *trans, GstBuffer *buf)
{
auto self{trans};
vx_uint32 constexpr OVX_ARRAY_PRIMARY_IMAGE_INDEX{0};
auto meta{reinterpret_cast<GstTIOVXImageMeta *>(gst_buffer_get_meta(buf, GST_TYPE_TIOVX_IMAGE_META_API))};
if (not meta)
{
GST_ERROR("gst_buffer_get_meta failed: GstTIOVXImageMeta missing!");
return GST_FLOW_ERROR;
}
vx_reference fullImageReference{vxGetObjectArrayItem(meta->array, OVX_ARRAY_PRIMARY_IMAGE_INDEX)};
if (not fullImageReference) {
GST_ELEMENT_ERROR(self, STREAM, FAILED, ("vxGetObjectArrayItem failed"), (NULL));
return GST_FLOW_ERROR;
}
const int patchX = 320;
const int patchY = 180;
vx_image fullImage{reinterpret_cast<vx_image>(fullImageReference)};
vx_rectangle_t const rect = {patchX, patchY, patchX*3, patchY*3};
vx_map_id map_id{};
void *ptr{nullptr};
vx_imagepatch_addressing_t addr{};
for (vx_uint32 plane{0}; plane < NV12_PLANE_COUNT; ++plane)
{
if (vxMapImagePatch(fullImage, &rect, plane, &map_id, &addr, &ptr, VX_WRITE_ONLY, VX_MEMORY_TYPE_HOST, VX_NOGAP_X) != VX_SUCCESS)
{
GST_ELEMENT_ERROR(self, STREAM, FAILED, ("vxMapImagePatch failed"), (NULL));
return GST_FLOW_ERROR;
}
auto const yRows{addr.dim_y * addr.scale_y / VX_SCALE_UNITY};
for (vx_uint32 y = 0; y < yRows; y++)
{
vx_uint8 *rowPtr = (vx_uint8 *)ptr + y * addr.stride_y;
auto const xBytes{addr.dim_x * addr.stride_x * addr.scale_x / VX_SCALE_UNITY};
memset(rowPtr, 0,xBytes);
}
if (vxUnmapImagePatch(fullImage, map_id) != VX_SUCCESS)
{
GST_ELEMENT_ERROR(self, STREAM, FAILED, ("vxUnmapImagePatch failed"), (NULL));
return GST_FLOW_ERROR;
}
}
vxReleaseReference (&fullImageReference);
return GST_FLOW_OK;
}Could you please look into this and help fix the issue? Are there any interoperability issues? Is there anything we missed here to keep tiovx and dma buf consistent
