Camera Cannot Capture

Hi Ricky,

What I see in the log you shared are heap warnings like this:

"[CORE1]: [     22.251] [t=0x595cc15b] xdc.runtime.Memory: ERROR: line 52: out of memory: heap=0x85a2bfb0, size=532608
[CORE1]: [     22.251] xdc.runtime.Memory: line 52: out of memory: heap=0x85a2bfb0, size=532608
[CORE1]: [     22.251] [ERR=1137] src/msp_video.c:[2363]:Warning: Pipe processing is slower than the requested framerate!!! "

This kind of warning traces are output in Preview camera mode and raise a warning that some of the algo-s in the image pipe are performing more slowly than expected and this slows down the whole pipe. (The exact heap error message is because the sensor frame buffer queue becomes full - sensor outputs frames regularly at 30 fps but the image processing pipe cannot process them at this rate.) The system is designed to recover gracefully in this case - this is not a fatal error in any way.

So the problem you observe in the capture phase is not related to the above heap error messages. I checked that the sensor you use OV5647 is also 5Mpix like the OV5640 and OV5650 supported in the baseline releases. So most probably it is not necessary to make changes in the heap configuration.

Anyway please give the RAW image resolution you use with OV5647.

I see a lot of DCC errors in the log. Did you update the DCC XML-s for OV5647 sensor?

Do you use OV5647 like SoC (Smart) camera or in RAW mode?

Also you can see where exactly the crash on Ducati side happens - in the log there is a register dump.

You can use the addr2line tool with the PC register value.

Also you can check what were the last few functions executed in the call stack trace.

Best regards,

Hristo

Hi Hristo,

We are using OV5647 in RAW mode, and the resolution is 1920x1080 both in preview and capture mode, frame rate is 30fps. This resolution and frame rate is supported by BLAZE/BLAZE TABLET2, so I just can't understand how this error occured on our custom board. By the way, we still didn't update the DCC XMLs, shouldn't it be the last thing to concern about?

BR,

Ricky

Hi Ricky,

"the resolution is 1920x1080 both in preview and capture mode"

You mean image sensor output resolution is 1920x1080, i.e. you use some binning from the sensor?

Unfortunately the log you shared doesn't contain some pointer what goes wrong. Could you please check (like I suggested earlier) where the PC is pointing at the time of the crash. That could give us some idea of the problem.

About the DCC files - it is recommended that you supply valid DCC binaries for the respective sensor. You might reuse the OV5650 DCC files at the beginning. Please note that DCC bin files contain sensor ID inside (so you need either to use the same sensor ID for OV5647 of to re-generate the DCC bin-s from DCC xml-s).

BTW, I see a line in the log:

[CORE1]: [      6.070] [ERR=926] src/config/omx_config/params/omx_params_custom.c:[331]:Mode is not allowed!!!

What mode configuration was tried and not accepted? (I assume you are doing capture in HQ Capture mode.)

Regards,

Hristo

Hi Hristo,

Yes, the sensor output resolution is 1920x1080. We also tried 2592x1944 & 1280x720, and got the same results.

About the capture mode, the log was base on HQ capture mode, but it doesn't make any difference if we set the capture mode to HS, I've tried that.

Do you mean that we can use the DCC binaries for OV5650 by changing the DCC name in new_camera.c and using the same ID with OV5650 in OV5647_desc.c?

By the way, we are still tring to figure out where the PC is pointing at.

BR,

Ricky

Ricky,

Is should be sufficient if you define the sensor ID for OV5647 with the same value as for OV5650: please check "CAMERA_OV5650" that is defined in new_sensor_MSP.h.

It is not needed to make equal the DCC names in new_camera.c (in case the DCC-s from Ducati image are used and not loaded from Android FS) and ID-s from OV5647_desc.c.

Both OV5650 and OV5647 are 5Mpix ones. If the OV5647 Bayer pattern is the same like OV5650 (I don't have the datasheet) -> you can use the OV5650 DCC files and get relatively good image for quick start. Later you can make separate OV5647 tuning in separate DCC set of files.

Regards,

Hristo

Hi Hristo,

We got the same error although we used the sensor ID for OV5650. So I thought this might not be the solution to this problem.

I've also got some informations from base_image_app_m3.xem3.map, PC was pointing at 003d2746 when crash happens:

003d26fe 0000000a WTSD_DucatiMMSW.drivers.csl.iss.isp.aem3 : rsz.oem3 (.text:rsz_validate_params)
003d2708 0000000a WTSD_DucatiMMSW.drivers.drv_tCtrl_iss.aem3 : tCtrl_interface.oem3 (.text:tCtrl_close)
003d2712 0000000a ipc.lib : Ipc.obj (.text:ti_sdo_ipc_GateMP_getSharedAddr__F)
003d271c 0000000a sysbios.lib : BIOS.obj (.text:ti_sysbios_gates_GateAll_Instance_init__F)
003d2726 0000000a : BIOS.obj (.text:ti_sysbios_hal_Hwi_Instance_finalize__F)
003d2730 0000000a : BIOS.obj (.text:ti_sysbios_heaps_HeapMem_getExtendedStats__E)
003d273a 0000000a base_image_app_m3_pem3.oem3 (.text:xdc_runtime_IHeap_alloc)
003d2744 0000000a base_image_app_m3_pem3.oem3 (.text:xdc_runtime_IHeap_free)
003d274e 0000000a base_image_app_m3_pem3.oem3 (.text:xdc_runtime_IHeap_getStats)
003d2758 0000000a ti.targets.arm.rtsarm.aem3 : LoggerSys.oem3 (.text:xdc_runtime_LoggerSys_disable__F)
003d2762 0000000a : LoggerSys.oem3 (.text:xdc_runtime_LoggerSys_enable__F)
003d276c 0000000a base_image_app_m3_pem3.oem3 (.text:xdc_runtime_knl_ISemaphore_pend)
003d2776 0000000a base_image_app_m3_pem3.oem3 (.text:xdc_runtime_knl_ISemaphore_post)
003d2780 0000000a ti.targets.arm.rtsarm.aem3 : Thread.oem3 (.text:xdc_runtime_knl_Thread_Instance_finalize__

And LR was pointing at 003c1501:

003c13f0 00000038 WTSD_DucatiMMSW.platform.osal.aem3 : STMHelper.oem3 (.text:STMLog_EnableCIOMode)
003c1428 00000038 WTSD_DucatiMMSW.ext_rel.frameq_mod.aem3 : shmem.oem3 (.text:Shmem_getFreeBufCnt)
003c1460 00000038 WTSD_DucatiMMSW.platform.osal.aem3 : timm_osal_pipes.oem3 (.text:TIMM_OSAL_DeletePipe)
003c1498 00000038 : timm_osal_events.oem3 (.text:TIMM_OSAL_EventSet)
003c14d0 00000038 : timm_osal_memory.oem3 (.text:TIMM_OSAL_Free)
003c1508 00000038 WTSD_DucatiMMSW.drivers.csl.iss.simcop.nsf.aem3 : _csl_nsf.oem3 (.text:_CSL_nsfAddrConfig)
003c1540 00000038 ti.resmgr.aem3 : IpcResource.oem3 (.text:_IpcResource_translateError)
003c1578 00000038 fw3a.aem3 (.text:af_get_inter_err_acc:af_tools.oem3)
003c15b0 00000038 fw3as.aem3 (.text:af_get_inter_err_acc:af_tools.oem3)
003c15e8 00000038 fw3as.aem3 (.text:affw_h3a_time_send:af_fw_misc.oem3)
003c1620 00000038 fw3a.aem3 (.text:caf_get_time_4frms:caf_comm.oem3)
003c1658 00000038 WTSD_DucatiMMSW.drivers.csl.iss.simcop.common.aem3 : simcop_irq_mgr.oem3 (.text:create_swi)

Is it useful? Does this have anything to do with the heap memory error when preview? Because I can see the memory leak detected by ducati after doing capture.

BR,

Ricky

Hi Hristo,

Another question, what's the correct ducati version if we use 4AI.1.7? We are now using ducatimm-2.1.11.

BR,

Ricky

Hi Ricky,

From the MAP excerpt above it seems the crash happens during memory Free operation - this can be a result of wrong pointer, corrupt memory, etc.

To get more information who and what memory exactly was tried to be freed - it would be useful if you provide also the function names from the call stack.

It seems there is a change in the code between your latest experiment and the first log you shared in the thread; could you please provide updated log, I'd like to see also that the DCC errors are gone after OV5650 DCC-s are used.

Unfortunately I don't have access to Ducati sources at the present - that limits my support abilities.

Best regards,

Hristo

"Another question, what's the correct ducati version if we use 4AI.1.7? We are now using ducatimm-2.1.11."

This is an internal information that I don't have access to. Hopefully some TI-er would clarify.

Regards,

Hristo

Hi Hristo,

It seems that the tool addr2line is only for android, because the toolchain for ducati is different from which for android FS, so how can I find the exact function name in ducati side?

Update the log: 3731.ov5647.log

BR, 

Ricky

Hi Ricky,

The addr2line tool should work with Ducati image (ducati-m3-core0.xem3).

I assume that you compiled Ducati image with debug info enabled. The Ducati output image is in ELF format and thus addr2line can parse it.

(I was thinking in case you still experience difficulties - whether it's possible to send me the Ducati XEM3 image and the MAP file. The Ducati image is bulky with debug info enabled, so it might  be difficult.)

I still see DCC errors in the log you shared. I'm not sure why this still happens. I suppose you defined "CAMERA_OV5647" to be equal to 42 as it is with "CAMERA_OV5650".

You might do some other trials:

- You may connect OV2722 on first Camera port (as primary camera) and try doing capture with it -> if there is a different behaviour it'd point us that the problem is sensor specific (OV2722 is also 1080p resolution).

- You may try to disable different post-processing algo-s (like LDCNSF, GLBCE, etc - not sure what is currently enabled for preview and capture), that would make the image pipe simpler. Usually the "out of memory: heap=0x .... Warning: Pipe processing is slower than the requested framerate!!! " kind of warnings in preview are because LDCNSF/VNF is performing more slowly for bigger resolutions (that might be caused of not optimal configuration because of incorrect DCC).

Best regards,

Hristo

Hi Hristo,

I set the "CAMERA_OV5647" to 42, here is the log: 0844.ov5647_42.log

I'm so sorry to tell you that the result of capturing on OV2722 is the same with OV5647, so this is not a problem of sensor specific.

As for the Ducati images, I will ask my leader, and I can post them to you via email if it is allowed. So please kindly tell me your email address. Thanks!

And how should I disable these post-processing algo-s?

BR,

Ricky

Ricky,

The latest log contains confirmation that the hang comes from invalid mem free:

"[CORE1]: [     30.726] [t=0xd7226f83] ti.sysbios.heaps.HeapMem: ERROR: line 331: assertion failure: A_invalidFree: Invalid free"

The crash happens in the OS mem routines; it is necessary to look in the call stack to find the latest function calls and decide which was the incorrect buffer (in worse case it could be a mem corruption also).

I've sent you my mail address separately.

The post-processing algo-s can be disabled from Android Camera setting (in the GUI). Though some of them are hardcoded (not available in the GUI) so the other way is to disable them in Camera HAL on A9 side.

BTW, what is the "GLBCE_WTABLE_SIZE" value in build.h defined for your HW environment?

Also I see in the log some trace that bothers me:

[CORE1]: [      5.413] [ERR=921] src/config/omx_config/params/omx_params_custom.c:[331]:Mode is not allowed!!!

I don't remember such a print in the original Ducati code. Is this a new modification on your side; did you check what exactly fails there?

Regards,

Hristo

Hristo,

Sorry, but I didn't get any conversation from you.

[CORE1]: [      5.413] [ERR=921] src/config/omx_config/params/omx_params_custom.c:[331]:Mode is not allowed!!!

Maybe this is decided by the capture mode (camera setting), I will check it out tomorrow.

BR,

Ricky

Hi Ricky,

My mail address is hhristov at mm-sol dot com (I sent it in the invitation request to you yesterday).

You may send me the MAP file that corresponds to "3731.ov5647.log" - this log contains complete stack dump which might be important for the debug.

Regards,

Hristo

Hi Hristo,

The unallowed mode is 'OMX_CaptureImageProfileExtended1' in omx_params_custom.c, I'm focusing on it.

BR,

Ricky

Hi Hristo,

I found the unsupported mode is OMX_CaptureStereoImageCapture in omx_params_custom.c, this can be fixed by disable the macro "BLD_STEREO_CAP_DOUBLE_STEREO_RES" in build.h. It has nothing to do with the crash issue.

BR,

Ricky

Ricky,

Yes, I agree that this is something separate and doesn't have relation to the crash.

Best regards,

Hristo

Hi Ricky,

I debug the ov5647 and changed the ov5650 driver  " #define     OV5650_MODEL_ID_REG_VALUE      (0x5647) ",

Now , the i2c is time out, read no id .

But ，the same hardware platform if change the ov5640 sensor , the preview and capture function is ok. 

the log

Sensor Power Sequence for :

static SensorPowerSequence tPwrUpSequence[] = {

{SEN_PWR_CMD_CLOCK, 1},
{SEN_PWR_CMD_DELAY, 5},

{SEN_PWR_CMD_POWER_2, 1},
{SEN_PWR_CMD_DELAY, 1},

{SEN_PWR_CMD_POWER_1, 1},
{SEN_PWR_CMD_DELAY, 1},

{SEN_PWR_CMD_POWER_3, 1},
{SEN_PWR_CMD_DELAY, 1},

{SEN_PWR_CMD_POWER_4, 1},
{SEN_PWR_CMD_DELAY, 5},

{SEN_PWR_CMD_RESET, 0},
{SEN_PWR_CMD_DELAY, 20},

{SEN_PWR_CMD_XSHTDWN, 1},
{SEN_PWR_CMD_DELAY, 20},

{SEN_PWR_CMD_XSHTDWN, 0},
{SEN_PWR_CMD_DELAY, 20},

{SEN_PWR_CMD_RESET, 1},
{SEN_PWR_CMD_DELAY, 5},


{SEN_PWR_CMD_TERMINATE, 0},

};

Please help analyze and provide some suggestions.

Thanks.

Hi Pei,

Attach my power on sequence for your referance:

{SEN_PWR_CMD_POWER_1, 1},
{SEN_PWR_CMD_DELAY, 5},

{SEN_PWR_CMD_XSHTDWN, 1},
{SEN_PWR_CMD_DELAY, 5},

{SEN_PWR_CMD_XSHTDWN, 0},
{SEN_PWR_CMD_DELAY, 5},

{SEN_PWR_CMD_RESET, 1},
{SEN_PWR_CMD_DELAY, 5},

{SEN_PWR_CMD_CLOCK, 1},
{SEN_PWR_CMD_DELAY, 10},

{SEN_PWR_CMD_TERMINATE, 0},

I'm not sure if this is workable on your platform, but I think you should try it.

BR,

Ricky

Hi, Ricky

I tried, the same results. 

Hi Pei,

Can you please send me the log file? I wanna know what's happening.

BR,

Ricky

Sorry, I forgot to upload log files.

8686.07051506.ov5647&ov2655.log

Sensor Power On Sequence for :

{SEN_PWR_CMD_POWER_2, 1},
{SEN_PWR_CMD_DELAY, 1},

{SEN_PWR_CMD_POWER_1, 1},
{SEN_PWR_CMD_DELAY, 1},

{SEN_PWR_CMD_POWER_3, 1},
{SEN_PWR_CMD_DELAY, 1},

{SEN_PWR_CMD_POWER_4, 1},
{SEN_PWR_CMD_DELAY, 5},

{SEN_PWR_CMD_XSHTDWN, 1},
{SEN_PWR_CMD_DELAY, 5},

{SEN_PWR_CMD_XSHTDWN, 0},
{SEN_PWR_CMD_DELAY, 5},

{SEN_PWR_CMD_RESET, 1},
{SEN_PWR_CMD_DELAY, 5},

{SEN_PWR_CMD_CLOCK, 1},
{SEN_PWR_CMD_DELAY, 10},

{SEN_PWR_CMD_TERMINATE, 0}

Thanks.

Hi Pei,

Have you ever checked the GPIO level when reading sensor ID? Are they correct?

My code version is 4AI.1.7, different from yours, it's difficult to find the reason for your error from the log.

BR,

Ricky