OpenMAX VFPC scaler reconfiguration

See attached for a patch to ilclient.c in the decode_display example showing the suggested method not working.  This hacks it to call OMX_SetConfig OMX_TI_IndexConfigVidChResolution with a modified resolution after 100 frames of scaler input.  In my testing, it works fine up to that point but the scaler always hangs after the call - it accepts more buffers, but never returns any.

- Mark

diff -ur decode_display/src/ilclient.c decode_display.modified/src/ilclient.c
--- decode_display/src/ilclient.c	2012-04-19 12:22:39.000000000 +0100
+++ decode_display.modified/src/ilclient.c	2012-08-10 15:14:33.000000000 +0100
@@ -80,6 +80,60 @@
 #include <omx_vfdc.h>
 #include <omx_ctrl.h>
 
+
+#include <OMX_TI_Index.h>
+#include <OMX_TI_Common.h>
+
+// Hackery to dynamically modify scaler resolution.
+// After being called 100 times, it modifies the scaler input resolution to upscale
+// part of the image rather than the whole image.  Numbers are correct for 720p
+// decoder output, for other resolutions this should just change it to produce
+// something random (but still succeed).
+
+static IL_CLIENT_COMP_PRIVATE *hack_scaler_component;
+static int hack_scaler_input_frames, hack_scaler_output_frames;
+
+static void hack_scaler_input_hook(IL_CLIENT_COMP_PRIVATE *comp)
+{
+  ++hack_scaler_input_frames;
+  printf("scaler input frame %d\n", hack_scaler_input_frames);
+
+  if(hack_scaler_input_frames == 100) {
+    OMX_ERRORTYPE err;
+    OMX_CONFIG_VIDCHANNEL_RESOLUTION res;
+
+    OMX_INIT_PARAM(&res);
+    res.eDir          = OMX_DirInput;
+    res.nChId         = 0;
+    res.Frm0Width     = 640;
+    res.Frm0Height    = 360;
+    res.Frm0Pitch     = 1408;
+    res.Frm1Width     = 0;
+    res.Frm1Height    = 0;
+    res.Frm1Pitch     = 0;
+    res.FrmStartX     = 0;
+    res.FrmStartY     = 0;
+    res.FrmCropWidth  = 0;
+    res.FrmCropHeight = 0;
+
+    err = OMX_SetConfig(comp->handle,
+                        (OMX_INDEXTYPE) OMX_TI_IndexConfigVidChResolution,
+                        &res);
+    if(err != OMX_ErrorNone) {
+      ERROR("failed to modify scaler input channel resolution\n");
+    } else {
+      printf("successfully modified scaler input channel resolution\n");
+    }
+  }
+}
+static void hack_scaler_output_hook(IL_CLIENT_COMP_PRIVATE *comp)
+{
+  ++hack_scaler_output_frames;
+  printf("scaler output frame %d\n", hack_scaler_output_frames);
+}
+
+
+
 OMX_U8 PADX;
 OMX_U8 PADY;
 extern void IL_ClientFbDevAppTask (void *threadsArg);
@@ -273,6 +327,9 @@
   OMX_ERRORTYPE eError = OMX_ErrorNone;
   int retVal = 0;
 
+  if(thisComp == hack_scaler_component)
+    hack_scaler_output_hook(thisComp);
+
   /* get the pipe corrsponding to this port, portIndex is part of bufferheader
      structure */
   outPortParamsPtr =
@@ -499,6 +556,9 @@
   pBufferIn->pMarkData = pipeMsg->bufHeader.pMarkData;
   pBufferIn->nTickCount = 0;
 
+  if(thisComp == hack_scaler_component)
+    hack_scaler_input_hook(thisComp);
+
   /* call etb to the component */
   err = OMX_EmptyThisBuffer (thisComp->handle, pBufferIn);
   return (err);
@@ -817,6 +877,8 @@
   }
   pAppData->scILComp->handle = pAppData->pScHandle;
 
+  hack_scaler_component = pAppData->scILComp;
+
   printf (" scalar compoenent is created \n");
 
   /* omx calls are made in this function for setting the parameters for scalar

Hi Mark,

It is not clear that, you want to change input stream dynamically or want to change output scaling ? for scaling o/p port properties needs to be changed. If you want to take  a portion of picture, you can use crop width/height for doing so.

Regards

Vimal

Hi Vimal; thank you for replying!

Vimal Jain said:

It is not clear that, you want to change input stream dynamically or want to change output scaling ? for scaling o/p port properties needs to be changed. If you want to take  a portion of picture, you can use crop width/height for doing so.

Ideally I would like to reconfigure both input and output, but in this case input resolution is the interesting one.  Cropping the image is not the intent at all, though I admit it ends up doing that in the example I gave (the example was intended as a simple demonstration that trying to configure it like this just hangs the scaler, rather than that it's exactly what I want to do).

Consider a scaler which is scaling decoded video up to be sent to an output.  The H.264 stream being fed to the decoder may change resolution at any moment, so the scaler needs to be able to change its configured input resolution when that happens in order for the output to be correct.

For example, the scaler input and decoder might be configured with a maximum resolution of 1280x720, and the scaler output at 1920x1080.  Initially the H.264 stream going in to the decoder is 1280x720 so the scaler input is configured to 1280x720 with pitch 1408.  Suppose now the stream changes to be 512x288, then when the first  frame of the new resolution comes out of the decoder the scaler needs to be reconfigured to have input 512x288 with pitch 640 but otherwise continue exactly as before (same buffers, same output).

How might this be achieved?

Thanks,

- Mark

Mark,

in OMX height/width/pitch is port format, and unfortunately can not be changed in execute state. Scalar was provided these as runtime parameter, to take from buffer header. However  standard OMX header does not alllow these fields in buffer header ! so Scalar ends up taking those values from port properties. Typically if buffer sizes are same for whole stream, decoder should be able to decode smaller resolution with buffer-pitch, and then crop width/height alongwith startX, startY can be used to cut actual video from the buffer and then scale it up. 

Regards

Vimal

Vimal Jain said:

Typically if buffer sizes are same for whole stream, decoder should be able to decode smaller resolution with buffer-pitch, and then crop width/height alongwith startX, startY can be used to cut actual video from the buffer and then scale it up.

Ok, that sounds like a good approach.  See attached for another attempt (as above, it's a patch to ilclient.c in the decode_display example; 720p only still).  Now I'm setting the StartX, StartY, CropWidth and CropHeight while keeping the Width, Height and Pitch fields the same as they were at startup.  This does not hang the scaler, but it also doesn't appear to actually do anything - the output video stays exactly the same.  Am I missing some other parameters I need to set somewhere?

Thanks,

- Mark

diff -ur decode_display/src/ilclient.c decode_display.modified/src/ilclient.c
--- decode_display/src/ilclient.c	2012-08-13 11:06:35.000000000 +0100
+++ decode_display.modified/src/ilclient.c	2012-08-13 11:05:23.000000000 +0100
@@ -80,6 +80,60 @@
 #include <omx_vfdc.h>
 #include <omx_ctrl.h>
 
+
+#include <OMX_TI_Index.h>
+#include <OMX_TI_Common.h>
+
+// Hackery to dynamically modify scaler resolution.
+// After being called 100 times, it modifies the scaler input cropping parameters
+// to upscale the middle quarter of the image rather than the whole image.  Numbers
+// are correct for 720p decoder output, for other resolutions this will likely just
+// hang because it modifies the whole image resolution dynamically.
+
+static IL_CLIENT_COMP_PRIVATE *hack_scaler_component;
+static int hack_scaler_input_frames, hack_scaler_output_frames;
+
+static void hack_scaler_input_hook(IL_CLIENT_COMP_PRIVATE *comp)
+{
+  ++hack_scaler_input_frames;
+  printf("scaler input frame %d\n", hack_scaler_input_frames);
+
+  if(hack_scaler_input_frames == 100) {
+    OMX_ERRORTYPE err;
+    OMX_CONFIG_VIDCHANNEL_RESOLUTION res;
+
+    OMX_INIT_PARAM(&res);
+    res.eDir          = OMX_DirInput;
+    res.nChId         = 0;
+    res.Frm0Width     = 1280;
+    res.Frm0Height    = 720;
+    res.Frm0Pitch     = 1408;
+    res.Frm1Width     = 0;
+    res.Frm1Height    = 0;
+    res.Frm1Pitch     = 0;
+    res.FrmStartX     = 320;
+    res.FrmStartY     = 180;
+    res.FrmCropWidth  = 640;
+    res.FrmCropHeight = 360;
+
+    err = OMX_SetConfig(comp->handle,
+                        (OMX_INDEXTYPE) OMX_TI_IndexConfigVidChResolution,
+                        &res);
+    if(err != OMX_ErrorNone) {
+      ERROR("failed to modify scaler input channel resolution\n");
+    } else {
+      printf("successfully modified scaler input channel resolution\n");
+    }
+  }
+}
+static void hack_scaler_output_hook(IL_CLIENT_COMP_PRIVATE *comp)
+{
+  ++hack_scaler_output_frames;
+  printf("scaler output frame %d\n", hack_scaler_output_frames);
+}
+
+
+
 OMX_U8 PADX;
 OMX_U8 PADY;
 extern void IL_ClientFbDevAppTask (void *threadsArg);
@@ -273,6 +327,9 @@
   OMX_ERRORTYPE eError = OMX_ErrorNone;
   int retVal = 0;
 
+  if(thisComp == hack_scaler_component)
+    hack_scaler_output_hook(thisComp);
+
   /* get the pipe corrsponding to this port, portIndex is part of bufferheader
      structure */
   outPortParamsPtr =
@@ -499,6 +556,9 @@
   pBufferIn->pMarkData = pipeMsg->bufHeader.pMarkData;
   pBufferIn->nTickCount = 0;
 
+  if(thisComp == hack_scaler_component)
+    hack_scaler_input_hook(thisComp);
+
   /* call etb to the component */
   err = OMX_EmptyThisBuffer (thisComp->handle, pBufferIn);
   return (err);
@@ -817,6 +877,8 @@
   }
   pAppData->scILComp->handle = pAppData->pScHandle;
 
+  hack_scaler_component = pAppData->scILComp;
+
   printf (" scalar compoenent is created \n");
 
   /* omx calls are made in this function for setting the parameters for scalar

Hi Mark,

sorry, cropping/scaling is broken in last release, and it takes only port properties in effect. Issue is identified, and we will be able to provide fix in next release.

Regards

Vimal

Hi Vimal,

Thank you for that answer.  When will the next release, with this feature working, be available?

- Mark

As a strongly related question, will this dynamic reconfiguration also work on the noise filter component?  I am unable to test anything about the noise filter  because it doesn't work at all in 5.04.

This would be useful for the capture -> scale -> noise filter -> encoder use case, to allow different encode resolutions from the same input source (using the noise filter in bypass mode for its 422 -> 420 conversion, since the scaler can only output 422 and the encoder can only input 420).

Thanks,

- Mark

Mark,

NF does not have run time change parameters. You could use DEI path for capture and scaling, It can provide 420 output for encoder. 

Regards

Vimal

Is DEI dynamically reconfigurable as above?

Also, what are the resource use implications of using it?  In an application already using using three instances of the scaler, will it continue to work?  (Since it generates two outputs (one of which will just be discarded in this case), I am guessing that it must use up two scaler instances.)

Thanks,

- Mark

I have now tried scaling with DEIH and it mostly works.

Some observations:

- DEIH seems to coexist perfectly well with existing INDTXSCWB scalers.

- Dynamic reconfiguration doesn't appear to work here either (I assume this is the same problem as with the vanilla scalers).

- When downscaling to an output with pitch not equal to width, the two output (luma and chroma) planes overlap, which messes up large sections of the output image.  It looks like it chooses the start address of the chroma plane based on the width and height inputs initially (ignoring the pitch), so this would probably be solved by working reconfiguration (by setting an initial maximum size with pitch equal to width and then switching to the actual intended size immediately after).

- The fact that it is pointlessly generating an additional 422 output which is discarded doesn't appears to harm anything, though to avoid wasting memory bandwidth it would be nice to switch this off.

So, comment on dynamic reconfiguration of DEIH too?

Thanks,

- Mark

Hi,

DEI has dynamic reconfiguration provision, but probably is not verified as you observed. Please note De-interlace algorithm has to be in bypass mode for dynamic reconfiguration. (bAlgbypass = 1).

for downscaling, do you want to retain original pitch and scale the buffer ? This is with SC or DEI ?

DEI component is using dual-out driver, however you can turn off one port to disable the dual  o/p.

Regards

Vimal

Hi Vimal,

Thank you for your reply.

Vimal Jain said:

for downscaling, do you want to retain original pitch and scale the buffer ? This is with SC or DEI ?

This is with DEI.  The aim is to end up with whatever pitch combination the encoder accepts.  I have tried setting videnc2DynamicParams.captureWidth on the encoder, but it seems to always use the pitch from the original port definition anyway (though it does respect the inputWidth and inputHeight parameters).  Therefore, I am trying to make the DEI output use the encoder input pitch.  Setting Frm1Pitch on the output of the DEI component has the right result for luma (in that the encoded video has correct luma), but the chroma is not right - it seems like the two planes are overlapping somehow.

Note that this problem may instead be on the encoder end, if it is finding the location of the chroma plane incorrectly when inputWidth and inputHeight are changed.  (In all cases the luma output is correct, so it is definitely mostly working.)

I will have a more careful look at the buffers coming out of the DEI component to try to work out where the problem is, but if you are aware of any issues around this it would be very helpful.

Vimal Jain said:

DEI component is using dual-out driver, however you can turn off one port to disable the dual  o/p.

Thank you for that information, I will make this change to avoid writing the 422 output at all.

I have a further question regarding scaler use.  It seems that the VPSS firmware only supports four scalers of either kind (DEI or INDTXSCWB), and crashes when a fifth is allocated.  However, I believe there should be more than this available (the OpenMAX manual suggests that there are five, labelling them SC1 to SC5).  Are more scaling resources available?  If so, how can they be accessed?

Thanks,

- Mark

I have resolved the incorrectly placed chroma plane issue.

When DEI is configured (via VidChResolution) to have a output width and height which is different from the original port width and height, it still calculates the position of the output chroma plane based on the original values rather than the configured values.

For example, if set up with 1280x720 (1280 pitch) ports and then configured to 768x432 (1280 pitch), the luma plane starts at offset 0 as expected but the chroma plane starts at 1280*720 = 921600 rather than 1280*432 = 552960.  This is a made worse by the fact that on the output buffer the nFilledLen is totally wrong, as it shows 768*432*3/2 = 497664 in this case.  So, when fed into the encoder, it calculates that the chroma plane should be at 2/3*497664 = 331776, and ends starting halfway through the luma plane.  To work around the problem, the filled length of the DEI output buffer should be ignored and instead the expected value from the original port setup should be used (that is, 1280*720*3/2 = 1382400).  With this change, the encoder will then look for the chroma plane at 1382400*2/3 = 921600, which is correct.

Following further testing around this, I think that my original suggestion that DEI dynamic reconfiguration does not work was in fact wrong (somewhat confused by the above issue).  This does now work for at least simple cases - I will investigate more complex cases and repeated reconfiguration in the next few days.

- Mark

Mark,

Thanks..

that points out to issue in filledLength calculation in DEI component. We would look into that issue.

Regards

Vimal

Hi Vimal, 

    The decode_mosaicdisplay example in ezsdk5.05 now could do four input videos mixed and display. I want to link the output of the VSWMOSAIC component to the DUCATI.VIDENC component, but the colorformat does not match. So I want to use the VFPC.NF component change the 422 format to 420SP format. Could you give some suggestion or sample codes on how to do it?

Thanks so much ! Appreciate for your reply!

Hello Mark/Vimal,

I have similar requirement of dynamically configuring DEI (used as scaler). I am using capture_encode example which does

capture --> (Dual) DEI --> (one DEI out) to Display  --> (second DEI out) to Encode

I want to scale down captured input and want encode it accordingly. I know regarding encoder's dynamic parameters but don't know about DEI (as scaler)'s dynamic paramer.

Can you please let me know how can i achieve this or share me steps or piece of code by which you fixed you issue ?

Regards,

Hitesh

Hello Mark,

We are in the middle of some development and your help will move ahead us, can you please guide me to resolve the issue which you resolved ?

Regards,

Hitesh

Hitesh Viradiya said:

I have similar requirement of dynamically configuring DEI (used as scaler). I am using capture_encode example which does

capture --> (Dual) DEI --> (one DEI out) to Display  --> (second DEI out) to Encode

I want to scale down captured input and want encode it accordingly. I know regarding encoder's dynamic parameters but don't know about DEI (as scaler)'s dynamic paramer.

I found that DEI output reconfiguration worked in EZSDK 5.05.01.04 using the OMX_SetConfig call on OMX_CONFIG_VIDCHANNEL_RESOLUTION as in the capture_encode demo (the demo only calls it when starting, but actually that OMX_SetConfig can be called on the DEI component at any time to do the reconfiguration you want).  I have not used it in dual output mode - only to reconfigure an encode prescaler taking 4:2:0 to 4:2:0 (with the 4:2:2 output channel unused) - but I don't think there should be any meaningful difference.

Also, IIRC the input pitch change on the encoder (via captureWidth in the DynamicParams) doesn't work, so you will want to keep the 4:2:0 output pitch (that is, Frm1Pitch in OMX_CONFIG_VIDCHANNEL_RESOLUTION) constant  and ensure that it is the same as the pitch on the encoder input port - make it the width of the largest possible frame you might process.

- Mark

Hello Mark,

Many thanks for a quick response. To give you more idea about the problem, let me elaborate what so ever i did to reconfigure the DEI as scaler.

I had verified scaler at starting and it worked for me. I tried reconfiguring the scaler run-time (i.e. in Executing state of DEI). I have a requirement of Keeping capture resolution constant (i mean to say, same which is set during start of application to the end of application), i want to scaled it down (to minimum possible resolution support in scaler) and scaled it up (up to capture resolution) run-time, so as my encoder dynamic parameter reconfiguration required run-time. I know encoder and tested that it supports resolution change run-time. My code is as below, from result commented along with the code, leads me to issues in buffers which were created during application start time, and there is some issues in offset of chroma/luma, or some issue in setting pitch.

      OMX_INIT_PARAM (&chResolution);

      /* first output to display */
      chResolution.Frm0Width = scaled_nWidth;
      chResolution.Frm0Height = scaled_nHeight;
      chResolution.Frm0Pitch = scaled_nWidth * 2;         ------>  With this i got chroma and luma overlapped to each other on display.
      chResolution.Frm0Pitch = pAppData->nWidth * 2;  ------>  With this i Improved result but in display i got last frame in background and scaled video on it.

      /* second output to encode */
      chResolution.Frm1Width = scaled_nWidth;
      chResolution.Frm1Height = scaled_nHeight;
      chResolution.Frm1Pitch  = scaled_nWidth;            --->  With this i got chroma and luma overlapped to each other in encoded data.
      chResolution.Frm1Pitch  = pAppData->nWidth;     --->  With this i Improved result then above case, i got luma proper but still chroma overlaps in encoded data.
      chResolution.FrmStartX  = 0;
      chResolution.FrmStartY  = 0;
      chResolution.FrmCropWidth = 0;
      chResolution.FrmCropHeight = 0;
      chResolution.FrmCropWidth = 0;
      chResolution.FrmCropHeight = 0;
      chResolution.eDir = OMX_DirOutput;
      chResolution.nChId = 0;

      eError = OMX_SetConfig (pAppData->pDeiHandle,
                              (OMX_INDEXTYPE) OMX_TI_IndexConfigVidChResolution,
                              &chResolution);
      if (eError != OMX_ErrorNone)
      {
        ERROR ("failed to set output channel resolution\n");
      }
      else
      {
        printf ("set output resolution to %dx%d\n", scaled_nWidth, scaled_nHeight);
      }

      OMX_INIT_PARAM (&tDynParams);

      tDynParams.nPortIndex = OMX_VIDENC_OUTPUT_PORT;

      eError = OMX_GetConfig (pAppData->pEncHandle, OMX_TI_IndexParamVideoDynamicParams,
                              &tDynParams);

      /* setting resolution (width x height)*/
      tDynParams.videoDynamicParams.h264EncDynamicParams.videnc2DynamicParams.inputWidth = scaled_nWidth;
      tDynParams.videoDynamicParams.h264EncDynamicParams.videnc2DynamicParams.inputHeight = scaled_nHeight;
      tDynParams.videoDynamicParams.h264EncDynamicParams.videnc2DynamicParams.captureWidth = scaled_nWidth;

      eError = OMX_SetConfig (pAppData->pEncHandle, OMX_TI_IndexParamVideoDynamicParams,
                              &tDynParams);

My encoder configuration works perfectly, can you help me in identifying the root cause and to fix the above mentioned issue ?

Regards,

Hitesh

I think your problem with encoder input is that nFilledLen is not set correctly on output from the DEI component.  Inside OMX_BUFFERHEADERTYPE, nFilledLen is somewhat misleadingly named for what it is actually used for.  When a buffer is passed into OMX here, the only use made of nFilledLen is to calculate the position of the chroma plane relative to the luma plane: the pointer to the chroma plane is made as pBuffer + ((nFilledLen + nOffset) / 3 << 1).

In the case of the DEI component, the chroma output offset doesn't change after reconfiguration but nFilledLen does change (to be wrong).  So, you will need to adjust nFilledLen before passing the OMX_BUFFERHEADERTYPE to the next component.  In the case of the encoder input you probably want it to be (pAppData->nWidth * pAppData->nHeight / 2) * 3.  (nOffset will be zero for DEI output, so the chroma buffer is at nFilledLen + nWidth * nHeight.)

For the display input, do you actually want the video to be scaled for display?  The DEI output works perfectly well with different sizes on the two output ports, so I would suggest not scaling the 4:2:2 data (set Frm0Width = pAppData->nWidth, etc.) and just sending it directly, thus avoiding the reconfiguration problem on that link.

- Mark

Hello Marks,

Many thanks for a quick response. You are right i don't want to scale down the DEI data that goes to display, but i want to scale down DEI ouput that goes as input to encoder.

Can you help me, how can i achieve this using patch work with nFilledLen settings ?

Regards,

Hitesh

Your OMX_CONFIG_VIDCHANNEL_RESOLUTION on reconfiguring should probably look something like:

.Frm0Width  = pAppData->nWidth;
.Frm0Height = pAppData->nHeight;
.Frm0Pitch  = pAppData->nWidth * 2;
.Frm1Width  = scaled_nWidth;
.Frm1Height = scaled_nHeight;
.Frm1Pitch  = pAppData->nWidth;

This passes through 4:2:2 at the same resolution to go straight to the display without needing any additional configuration, and it downscales 4:2:0 to feed to the encoder while keeping the pitch constant at the original value.

To fix up nFilledLen, you need to edit the buffer headers as they either come out of the DEI component or when they get passed in to the encoder component.  In the ilclient demo setup it would probably be easiest to change IL_ClientProcessPipeCmdETB to use a different value of nFilledLen when you identify that the target component is the encoder.

- Mark

Hello Mark,

Many thanks. I got the solution. It works perfectly as per my need.

Thanks again.

Regards,

Hitesh