NetworkRx to Isp (Image Processing Issue)

Aneesh Peram

Expert 1255 points

I am sending the bayer_raw image using network_tx.exe tool to a usecase for processing.

THe usecase structure is :

NetworkRx -> IssM2mIsp -> Display

IssM2mIsp -> Alg_aewb

I am able to send the image and i am seeing output over display but the image on the display is not expected one.

I have an Bayer_raw with grbg format (8 bit and 12 bit ) which will result into an green image when i converted it into rgb888, but when i send the same image into the usecase it is producing a different colour.

I am using tda2px and vsdk3.0.3.

Please help me in processing the image correctly and also please mention the configuration which i have to setup to do this image processing.

Thanks and Regards,

Aneesh

over 5 years ago

0 Mayank Mangla over 5 years ago

TI__Mastermind 24525 points

can you please attach the input (12b bayer) and ISP output (8b NV12) images?

0 Aneesh Peram over 5 years ago in reply to Mayank Mangla

Expert 1255 points

grbg_12b.raw.txt

Hi Mayank,

The Input Image is attached above .

.raw extension is not allowed. So,please remove the .txt extension of the file.

I don't have the output image,the output is over hdmi.

Regards,

Aneesh

0 Mayank Mangla over 5 years ago in reply to Aneesh Peram

TI__Mastermind 24525 points

Thanks.. RAW image looks OK.

In the standard usecase (image sensor driven), Process function M2MISP Link is called every time a frame is received from the sensor. Sensor driver has linkages to DCC, which is a tuning library. All of this is needed to correctly process the RAW images and get correct ISP output.

I am not sure what changes you have done to the SW, but if you do it correctly you should be able to save YUV images after ISP. Please refer to 1-ch ISP capture display usecase.

0 Aneesh Peram over 5 years ago in reply to Mayank Mangla

Expert 1255 points

Hi Mayank,

I did not found the difference when compared with the 1-ch ISP capture display usecase and i did not understand your first statement clearly (specially "DCC"). Can you please elaborate that part.

Regards,

Aneesh

0 Aneesh Peram over 5 years ago in reply to Aneesh Peram

Expert 1255 points

Hi Mayank,

Can I contact you directly regarding this issue.

Regards,

Aneesh

0 Mayank Mangla over 5 years ago in reply to Aneesh Peram

TI__Mastermind 24525 points

Aneesh,

It is better to communicate through E2E. This allows for all the experts to watch this thread and contribute.

DCC is Dynamic Camera Configuration. This is a database of all ISP parameters required to process a RAW image correctly. This needs to be developed fro every sensor. This would need to be done for Network source also.

Regards,

Mayank

0 Aneesh Peram over 5 years ago in reply to Mayank Mangla

Expert 1255 points

Fullscreen chains_network_isp_display.c Download

#include "chains_network_isp_display_priv.h"
#include <src/include/chains_common.h>
#include <src/rtos/iss/include/app_util_iss.h>
#include <src/rtos/iss/include/iss_sensors.h>
#include <src/rtos/iss/include/iss_utils.h>
#include "../hm_raw_option/hm_raw_option_settings.h"

#define ALG_AEWB_MEM_SIZE (896 * 1024)

#define NETWORK_RX_FRAME_WIDTH      (1820)
#define NETWORK_RX_FRAME_HEIGHT     (940)

Void ChainsCommon_SetNetworkRxPrms_ND(
                        NetworkRxLink_CreateParams *pPrm,
                        UInt32 maxWidth,
                        UInt32 maxHeight,
                        UInt32 numCh,
                        UInt32 fps /* in msecs */
                        );

typedef struct
{
	chains_network_isp_displayObj ucObj;
	UInt32 captureOutWidth;
	UInt32 captureOutHeight;
	UInt32 displayWidth;
	UInt32 displayHeight;
	Chains_Ctrl *chainsCfg;
	AppUtils_Obj appSensInfo;
	UInt32  netProcId;
	char enable_network_raw_frames[8];
}chains_network_isp_displayAppObj;

char gchains_network_isp_display_runTimeMenu[]={
			"\r\n "
		    "\r\n ===================="
		    "\r\n chains Run-time Menu"
		    "\r\n ===================="
		    "\r\n "
		    "\r\n 0: Stop Chain"
		    "\r\n p: Print Performance Statistics "
		    "\r\n "
		    "\r\n Enter Choice: "
		    "\r\n "};

void chains_network_isp_display_FreeMemory(chains_network_isp_displayAppObj *pObj)
{
	Int32 status;

	    IssM2mIspLink_CreateParams *pIssM2mIspPrm;
	    AlgorithmLink_IssAewbCreateParams *pAlg_IssAewbPrm;


	    pIssM2mIspPrm = &pObj->ucObj.IssM2mIspPrm;
	    pAlg_IssAewbPrm = &pObj->ucObj.Alg_IssAewbPrm;

	    if (pAlg_IssAewbPrm->memAllocInfo.memAddr)
	    {
	        status = Utils_memFree(UTILS_HEAPID_DDR_CACHED_SR,
	                               (Ptr) pAlg_IssAewbPrm->memAllocInfo.memAddr,
	                               pAlg_IssAewbPrm->memAllocInfo.memSize);
	        UTILS_assert(status == 0);
	    }

	    if (pIssM2mIspPrm->memAllocInfo.memAddr)
	    {
	        status = Utils_memFree(UTILS_HEAPID_DDR_CACHED_SR,
	                               (Ptr) pIssM2mIspPrm->memAllocInfo.memAddr,
	                               pIssM2mIspPrm->memAllocInfo.memSize);
	        UTILS_assert(status == 0);
	    }
}
Void chains_network_isp_display_SetDisplayPrms(DisplayLink_CreateParams *pPrm_Video,
                                       Chains_DisplayType displayType,
                                       UInt32 displayWidth,
                                       UInt32 displayHeight,
                                       UInt32 captureWidth,
                                       UInt32 captureHeight)
{
    UInt32 dispWidth, dispHeight;

    if (pPrm_Video)
    {
        if ((displayType == CHAINS_DISPLAY_TYPE_SDTV_NTSC)
                || (displayType == CHAINS_DISPLAY_TYPE_SDTV_PAL))
        {
            pPrm_Video->displayScanFormat = SYSTEM_SF_INTERLACED;
        }

        /* To maintain the aspect ratio, change the display tarWidth */
        dispWidth = captureWidth * displayHeight / captureHeight;
        dispHeight = displayHeight;

        if (dispWidth > displayWidth)
        {
            dispWidth = displayWidth;
            dispHeight = displayWidth * captureHeight / captureWidth;
        }
//        dispWidth=1820;
//        dispHeight=940;
//		displayWidth=1920;
//		displayHeight=1080;

        pPrm_Video->rtParams.tarWidth = dispWidth;
        pPrm_Video->rtParams.tarHeight = dispHeight;
        pPrm_Video->rtParams.posX = (displayWidth - dispWidth) / 2U;
        pPrm_Video->rtParams.posY = (displayHeight - dispHeight) / 2U;

        pPrm_Video->displayId = DISPLAY_LINK_INST_DSS_VID1;
    }
}

Void chains_network_isp_display_SetIspConfig(chains_network_isp_displayAppObj *pObj)
{
    appSetDefaultIspParams(&pObj->appSensInfo);
    appSetIspParamsFromDCC(&pObj->appSensInfo);
}

Void chains_network_isp_display_setMuxes(
		chains_network_isp_displayAppObj *pObj)
{
    /* Setup PinMux */
    ChainsPlatform_SetPinMux(pObj->chainsCfg->sensorName);

    /* Init Board Mux */
    ChainsBoard_SetBoardMux(pObj->chainsCfg->sensorName);
}


Void chains_network_isp_display_load_network_raw_config(chains_network_isp_displayAppObj *pObj)
{
	LoadNetworkRawFramesOption(pObj->enable_network_raw_frames);
		if(0 == strcmp("ON", pObj->enable_network_raw_frames))
		{
			pObj->ucObj.NetworkRxPrm.enable_network_raw_frames=1;
		}
		else if(0 == strcmp("OFF", pObj->enable_network_raw_frames))
		{
			pObj->ucObj.NetworkRxPrm.enable_network_raw_frames=0;
		}
		else
		{
			pObj->ucObj.NetworkRxPrm.enable_network_raw_frames=0;
		}


}


Void chains_network_isp_display_SetAppPrms(chains_network_isp_displayObj *pUcObj,
                                   Void *appObj)
{

	UInt32 chId;
	System_LinkChInfo *pChInfo;
	chains_network_isp_displayAppObj *pObj = (chains_network_isp_displayAppObj *)appObj;
	chains_network_isp_display_load_network_raw_config(pObj);

	pUcObj->IssM2mIspPrm.channelParams[0].inBpp=SYSTEM_BPP_BITS16;
	pUcObj->IssM2mIspPrm.channelParams[1].inBpp=SYSTEM_BPP_BITS16;
	pUcObj->IssM2mIspPrm.channelParams[2].inBpp=SYSTEM_BPP_BITS16;
	pUcObj->IssM2mIspPrm.channelParams[3].inBpp=SYSTEM_BPP_BITS16;

	Vps_printf("the value is %d",pUcObj->NetworkRxPrm.enable_network_raw_frames);
	for (chId = 0; chId < pUcObj->NetworkRxPrm.outQueInfo.numCh; chId++)
	{
	  pChInfo = &pUcObj->NetworkRxPrm.outQueInfo.chInfo[chId];
	  if(pUcObj->NetworkRxPrm.enable_network_raw_frames==1)
	  {
		  pChInfo->flags = System_Link_Ch_Info_Set_Flag_Data_Format(pChInfo->flags,SYSTEM_DF_BAYER_RGGB);
	  }
	}


    ChainsCommon_GetDisplayWidthHeight(pObj->chainsCfg->displayType,
                                       &pObj->displayWidth, &pObj->displayHeight);

    pUcObj->IPCOut_A15_0_IPU1_0_0LinkID
            = SYSTEM_MAKE_LINK_ID(
                pObj->netProcId,
                pUcObj->IPCOut_A15_0_IPU1_0_0LinkID);

        pUcObj->NetworkRxLinkID
            = SYSTEM_MAKE_LINK_ID(
                pObj->netProcId,
                pUcObj->NetworkRxLinkID);

        pObj->captureOutWidth  = NETWORK_RX_FRAME_WIDTH;
        pObj->captureOutHeight = NETWORK_RX_FRAME_HEIGHT;


    /* Create the sensor now */
    strncpy(pObj->appSensInfo.sensorName, pObj->chainsCfg->sensorName,
            ISS_SENSORS_MAX_NAME);
    pObj->appSensInfo.numCh = 1;
    pObj->appSensInfo.ispOpMode = pObj->chainsCfg->ispOpMode;
    //pObj->appSensInfo.numCSI2Inst = 1;
    appCreateISSSensor(&pObj->appSensInfo);


    ChainsCommon_SetNetworkRxPrms_ND(
               &pUcObj->NetworkRxPrm,
               pObj->captureOutWidth,
               pObj->captureOutHeight,
               1,
               30
           );

    appInitIssIspParams(&pObj->appSensInfo, pUcObj->IssM2mIspLinkID,
                        &pUcObj->IssM2mIspPrm);
    appInitIssAewbParams(&pObj->appSensInfo, pUcObj->Alg_IssAewbLinkID,
                         &pUcObj->Alg_IssAewbPrm);

    pUcObj->IssM2mIspPrm.allocBufferForDump = TRUE;

    ChainsCommon_SetGrpxSrcPrms(&pUcObj->GrpxSrcPrm,
                                                   pObj->displayWidth,
                                                   pObj->displayHeight
                                                  );

    ChainsCommon_SetDisplayPrms(&pUcObj->Display_VideoPrm,
                                    &pUcObj->Display_GrpxPrm,
                                    pObj->chainsCfg->displayType,
                                    pObj->displayWidth,
                                    pObj->displayHeight
                                   );

    chains_network_isp_display_SetDisplayPrms(
            &pUcObj->Display_VideoPrm,
            pObj->chainsCfg->displayType,
            pObj->displayWidth, pObj->displayHeight,
            pUcObj->IssM2mIspPrm.channelParams[0U].outParams.widthRszA,
            pUcObj->IssM2mIspPrm.channelParams[0U].outParams.heightRszA);

    ChainsCommon_StartDisplayCtrl(pObj->chainsCfg->displayType,
                                  pObj->displayWidth, pObj->displayHeight);
}
Void ChainsCommon_SetNetworkRxPrms_ND(
                        NetworkRxLink_CreateParams *pPrm,
                        UInt32 maxWidth,
                        UInt32 maxHeight,
                        UInt32 numCh,
                        UInt32 fps /* in msecs */
                        )
{
    UInt32 chId;
    System_LinkChInfo *pChInfo;

    pPrm->outQueInfo.numCh = numCh;

    pPrm->timerPeriodMilliSecs = 1000/fps;

    for (chId = 0; chId < pPrm->outQueInfo.numCh; chId++)
    {
        pPrm->channelParams[chId].numBuffers = 6;

        pChInfo = &pPrm->outQueInfo.chInfo[chId];



        pChInfo->flags = System_Link_Ch_Info_Set_Flag_Scan_Format(pChInfo->flags,
                                                    SYSTEM_SF_PROGRESSIVE);
        pChInfo->width = maxWidth;
        pChInfo->height = maxHeight;
        pChInfo->startX = 0;
        pChInfo->startY = 0;
        pChInfo->pitch[0] = SystemUtils_align(pChInfo->width, 32);
        pChInfo->pitch[1] = SystemUtils_align(pChInfo->width, 32);

    }

    pPrm->networkServerPort = NETWORK_RX_SERVER_PORT;
    /* Select network mode for null source link */
    ChainsCommon_networkRxSetMode(pPrm);


}
Void chains_network_isp_display_StartApp(chains_network_isp_displayAppObj *pObj)
{
    Chains_memPrintHeapStatus();
    chains_network_isp_display_SetIspConfig(pObj);
    chains_network_isp_display_setMuxes(pObj);
    appStartIssSensorSerDes(&pObj->appSensInfo);
    ChainsCommon_StartDisplayDevice(pObj->chainsCfg->displayType);
    chains_network_isp_display_Start(&pObj->ucObj);
    Chains_prfLoadCalcEnable(TRUE, FALSE, FALSE);
}

Void chains_network_isp_display_StopAndDeleteApp(chains_network_isp_displayAppObj *pObj)
{
	chains_network_isp_display_Stop(&pObj->ucObj);
	chains_network_isp_display_Delete(&pObj->ucObj);

	chains_network_isp_display_FreeMemory(pObj);

    ChainsCommon_StopDisplayCtrl();

    appStopISSSensor(&pObj->appSensInfo);

    appDeleteISSSensor(&pObj->appSensInfo);

    ChainsCommon_StopDisplayDevice(pObj->chainsCfg->displayType);

    /* Print the HWI, SWI and all tasks load */
    /* Reset the accumulated timer ticks */
    Chains_prfLoadCalcEnable(FALSE, TRUE, TRUE);
}

Void chains_network_isp_display(Chains_Ctrl *chainsCfg)
{
    char ch;
    UInt32 done = FALSE;
    chains_network_isp_displayAppObj* pChainsObj = (chains_network_isp_displayAppObj*)Utils_memAlloc(
    UTILS_HEAPID_DDR_CACHED_SR, sizeof(chains_network_isp_displayAppObj), 16);
    UTILS_assert(pChainsObj != NULL);
    memset(pChainsObj, 0, sizeof(chains_network_isp_displayAppObj));

    pChainsObj->netProcId = Utils_netGetProcId();
    UTILS_assert(chainsCfg != NULL);
    strncpy(chainsCfg->sensorName, SENSOR_AR0220, ISS_SENSORS_MAX_NAME);

    if (TRUE == IssM2mIspLink_IsWdrMode(chainsCfg->ispOpMode))
    {
        Vps_printf(" chains: Please make sure BSP is build with WDR and LDC "
                   "enabled !!!\n");
    }

    pChainsObj->chainsCfg = chainsCfg;

    /* Initialize Video Sensor, so that Algorithm can use Params
   from Vid Sensor layer */
    chains_network_isp_display_Create(&pChainsObj->ucObj, pChainsObj);
    chains_network_isp_display_StartApp(pChainsObj);

    while (!done)
    {
        Vps_printf(gchains_network_isp_display_runTimeMenu);

        ch = Chains_readChar();

        switch (ch)
        {
        case '0':
            done = TRUE;
            break;
        case 'p':
        case 'P':
            ChainsCommon_PrintStatistics();
            chains_network_isp_display_printStatistics(&pChainsObj->ucObj);
            break;
        default:
            Vps_printf("\nUnsupported option '%c'. Please try again\n", ch);
            break;
        }
    }

    chains_network_isp_display_StopAndDeleteApp(pChainsObj);
}

Hi Mayank,

Did i have to set the DCC in usecase ? I have attched my usecase configuration file for your refernce above.

Can you please check the file and help in setting the configuration correctly.

Regards,

Aneesh

0 Mayank Mangla over 5 years ago in reply to Aneesh Peram

TI__Mastermind 24525 points

Aneesh,

DCC APIs are called by M2MISP link. The usecase does not need any changes.

Sensor driver must assign a DCC id and have DCC database included as a header file. Please refer to one of the existing sensor drvers.

For e.g. AR0233 -> iss_sensor_ar0233.c

#include "ar0233_dcc.h"

prms->dccId = 233U;

You may start by reusing the DCC header file and ID from an existing sensor. Please pick a sensor which is closes to the RAW files you are providing.

Regards,

Mayank

0 Aneesh Peram over 5 years ago in reply to Mayank Mangla

Expert 1255 points

Hi Mayank,

The details you have given are already there in my code, but the sensor is ar0220.

Still, Its not working perfectly.

Regards,

Aneesh

0 Mayank Mangla over 5 years ago in reply to Aneesh Peram

TI__Mastermind 24525 points

Aneesh,

Couple of suggestions:

1. Do you have AEWB link in the usecase? If yes, you will need to remove it since that link tries to communicate with an image sensor.

2. Please check sewbMode in the Sensor Parameters. To be safe, please set it to ALGORITHMS_ISS_AEWB_MODE_NONE

Regards,

Mayank

0 Aneesh Peram over 5 years ago in reply to Mayank Mangla

Expert 1255 points

7002.grbg_12b.raw.txt

Hi Mayank,

I tried removing the aewb link and tried but there is no improvement.

I have attached my input file. Please check it and it is giving out a grey colour approxmately(R=G=B=128) for your reference.

If possible give me a input file and tell the expected output of that input, because i am not able to understand where exactly the problem is there wheather the input or configuration.

Can you please tell he 1-ch iss capture display usecase is isscapture_isp_simcop display. or another one.If it is another one can you please give the usecase.

Is it possile to save the raw data before sending to isp?

Regards,

Aneesh

0 Mayank Mangla over 5 years ago in reply to Aneesh Peram

TI__Mastermind 24525 points

Please share the image dimensions for visualization.

I will suggest you use a real world image or a test pattern captured from the image sensor of your choice. While working with synthetic images it is hard to know the ground truth.

0 Aneesh Peram over 5 years ago in reply to Mayank Mangla

Expert 1255 points

Hi Mayank,

The image dimensions are 1820x940.

Is it possible to store raw data from camera (Iss Capture)?

If yes, can you please help me in saving raw bayer data from iss capture .

I found a link in which it is discussed but i cannot find the usecase(single channel ISS). The forum link is given below.

https://e2e.ti.com/support/processors/f/791/t/667341?TDA2EG-iss-capture-link-dump-frame-command

Regards,

Aneesh

0 Mayank Mangla over 5 years ago in reply to Aneesh Peram

TI__Mastermind 24525 points

img_D65.zip

0 Mayank Mangla over 5 years ago in reply to Mayank Mangla

TI__Mastermind 24525 points

I don't have any images from AR0220. Attached is an image from AR0233. The resolution is 1920x1280 and Bayer phase is GRGB

0 Aneesh Peram over 5 years ago in reply to Mayank Mangla

Expert 1255 points

orginal.zip

board output.zip

Hi Mayank,

I used the bayer image you have given and i did not got the correct output.

I have attached the output for your reference and also the original image.

The output on hdmi is with Aewb off.

Regards,

Aneesh

0 Mayank Mangla over 5 years ago in reply to Aneesh Peram

TI__Mastermind 24525 points

Can you please share the ISP output buffer in YUV420SemiPlanar format?

0 Aneesh Peram over 5 years ago in reply to Mayank Mangla

Expert 1255 points

isp output.zip

Hi Mayank,

The bayer raw image you have sent me is GRBG or GRGB. If it is GRGB then i dont have that format in my code to proceess.

If it is GRBG then the ISPoutput buffer is attached above.

Regards,

Aneesh

0 Mayank Mangla over 5 years ago in reply to Aneesh Peram

TI__Mastermind 24525 points

The raw image is 1920x1280. Are you processing it with the same settings?

0 Aneesh Peram over 5 years ago in reply to Mayank Mangla

Expert 1255 points

Hi Mayank,

I cropped the image to 1920*1080 and used it.

Regards,

Aneesh

0 Mayank Mangla over 5 years ago in reply to Aneesh Peram

TI__Mastermind 24525 points

Can you please share the cropped Bayer image?

Please also elaborate on how you are configuring the ISP for 1920x1080.

0 Aneesh Peram over 5 years ago in reply to Mayank Mangla

Expert 1255 points

grbg_16b_1080.zip

Hi Mayank,

I have attched the cropped bayer image.

About configuration, I have changed the resolutions to 1920x1080 in sensor parameters.

I have given the sensor parameters for your reference.

IssSensorIf_Params *prms = &SensorParams_Ar0220;
IssSensor_Info *info = &SensorParams_Ar0220.info;

IssSensorIf_Params_Init(prms);
strncpy(prms->name, SENSOR_AR0220, ISS_SENSORS_MAX_NAME);

prms->dccId = 233U;
info->width = 1920;
info->height = 1280;
info->dataFormat = SYSTEM_DF_BAYER_GRBG;
info->bpp = SYSTEM_BPP_BITS12;
info->features = ISS_SENSORS_AR0220_FEATURES;
info->aewbMode = ALGORITHMS_ISS_AEWB_MODE_AEWB;
info->maxExp = 1;
info->ramOffset = 24*1024*1024;

prms->start = AR0220_Start;
prms->stop = AR0220_Stop;
prms->setAeParams = AR0220_SetAeParams;
prms->getExpParams = AR0220_GetExpParams;
prms->getDccParams = AR0220_GetDccParams;
prms->initAewbConfig = AR0220_InitAewbConfig;
prms->getIspConfig = AR0220_GetIspConfig;
prms->readWriteReg = AR0220_ReadWriteReg;

and also i am changing the height and width values of the network_tx tool and network rx.

Thanks and Regards

Aneesh

0 Mayank Mangla over 5 years ago in reply to Aneesh Peram

TI__Mastermind 24525 points

nfo->width = 1920;
info->height = 1280;

Why is the height 1280?

0 Aneesh Peram over 5 years ago in reply to Aneesh Peram

Expert 1255 points

Hi Mayank,

Sorry for that i just changed that before for testing i sent that to you. The isp output i have sent is with 1920x1080 .

Regards,

Aneesh

0 Mayank Mangla over 5 years ago in reply to Aneesh Peram

TI__Mastermind 24525 points

Aneesh,

ISP output looks garbled up. It is not clear where it is getting corrupted. A few suggestions:

1. Double check that the networking layer is sending the image correctly.

2. If you have LDC in your usecase, please disable it.

3. Dump the ISP input and output images from the function IssM2mIspLink_drvProcessFrame. You will need CCS and a JTAG.

0 Mayank Mangla over 5 years ago in reply to Mayank Mangla

TI__Mastermind 24525 points

As discussed on Dec-12-2019, the wrong output was caused by incorrect pitch in the network utility used to send the RAW image. After correcting the pitch, Harman is able to get the correct output from ISP.

It was agreed that this thread can be closed.

Processors

Processors forum

NetworkRx to Isp (Image Processing Issue)