This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VM: camera no image

Part Number: TDA4VM

Hello,

Consult camera image problem
platform:TDA4  j721e 
PSDK:rtos:ti-processor-sdk-rtos-j721e-evm-08_01_00_13
             linux: ti-processor-sdk-linux-j7-evm-08_01_00_07
We use your company's platform and SDK, and there is a small probability that the camera application will not receive images. The phenomenon is described as follows:
After the platform is powered on for the first time, the camera application does not receive the frame rate sent by the driver.
Check the vsdk log, the driver log is normal. However, the vsdk log does not receive logs after 6.79 seconds.
Using the IPC_TEST program to test, it is found that the communication between MCU2_0 and MUP1_0 fails.
The log of the camera application shows that the capture_note takes a long time.
See the screenshot of the log for details.
Regards,
Gary
  • Hi Gary,

    From the remote core logs (i.e. [MCUx_0], [C7x_y ] ), Do you see all the logs being printed when you run the vision_apps_init.sh?

    i.e. Are all the cores reaching "App: Run ... Done" ?

    Regards,

    Nikhil

  • hi TI

    The vsdk log is as follows:

     

    It should be noted that the probability of this phenomenon occurring is very, very low. I think there are two phenomena that are unusual.
    1. IPC communication is abnormal. Who caused the communication abnormality between IPU1_0 and MCU2_0?
    2. capture_note seriously times out and the received frame rate is incorrect.

    This problem is very difficult now. Please help me analyze and solve it. Thank you very much.
  • Hi,

    IPC communication is abnormal. Who caused the communication abnormality between IPU1_0 and MCU2_0?

    IPC communication is expected to fail when the core MCU2_0 hangs.

    From the shared logs, I see all the cores are initialized and reached "App run".

    1. Since you mentioned that this phenomenon is not occurring every time, Could you confirm in the Cores reach "App_run" when the IPC communication fails?

    We use your company's platform and SDK, and there is a small probability that the camera application will not receive images. The phenomenon is described as follows:
    After the platform is powered on for the first time, the camera application does not receive the frame rate sent by the driver.
    Check the vsdk log, the driver log is normal. However, the vsdk log does not receive logs after 6.79 seconds.

    2. Also, do you see the IPC error after running the camera application? If this is the case, I think it could be because MCU2_0 is hanged here.

    I would like to understand your multi-cam application. From the graph i see it is a YUV sensor right? What is thecsirx configuration done here?

    Regards,

    Nikhil

  • hi TI

    1. From the vsdk log, IPC did not fail, so the program went to App_run. 2. No IPC related errors occur when running the camera application. 3. The application accepts four-channel YUV cameras. What configurations of CSIRX do you want to know? 4. Capture_node receives a frame rate exception and executes a timeout. What does this have to do with?
  • Hi,

    From the vsdk log, IPC did not fail, so the program went to App_run.

    Sorry for the misunderstanding here. I didn't mean the IPC init() that happens before App_run. I was referring to the below IPC demo example failing scenario. In this below scenario, does the MCU2_0 core reaches App_run()?

    The application accepts four-channel YUV cameras. What configurations of CSIRX do you want to know?

    As part of your sensor params, you would be mentioning the resoution of sensor, the landband speed, the metadatas etc. which will be used to configure the capture node in the application. I would like you to share this information.

    From the above application logs, I see modifications done to the multi-cam graph. Could you help me out in identifying these modifications?

    Regards,

    Nikhil

  • hi TI

    The log screenshot above is after the application failed to receive the frame rate. After 71462 seconds, ipc_test was run to test the IPC communication. It was found that the communication failed at this time. There is no way to know at this moment whether MCU2_0 is running.



    As part of your sensor params, you would be mentioning the resoution of sensor, the landband speed, the metadatas etc. which will be used to configure the capture node in the application. I would like you to share this information.

    From the above application logs, I see modifications done to the multi-cam graph. Could you help me out in identifying these modifications?

    image_resolution:1920*1080
    numDataLanes:4
    laneBandSpeed:CSIRX_LANE_BAND_SPEED_1350_TO_1500_MBPS
    fps:30
     

  • Hi,

    The log screenshot above is after the application failed to receive the frame rate. After 71462 seconds, ipc_test was run to test the IPC communication. It was found that the communication failed at this time. There is no way to know at this moment whether MCU2_0 is running.

    Yes, this is expected as the MCU2_0 core would have been hanged or crashed. You could confirm the same by connecting xds110 or JTAG debugger or use CCS (Code composer studio) and check the status of the core.

    image_resolution:1920*1080
    numDataLanes:4
    laneBandSpeed:CSIRX_LANE_BAND_SPEED_1350_TO_1500_MBPS
    fps:30

    4 YUV422 inputs with this resolution should be fine at this datarate. 

    Does the issue occur when you leave the application to run for a long time or is it when you restart the application multiple times?

    Also, you only have 3 nodes in your application (i.e. Capture, Conv and Scaler node). Did you change the buffer depth of the nodes in the application or is it still set to 4 ?

    Also do you see this issue with 1 or 2 cameras?

    Regards,

    Nikhil

  • hi ti 

    1. We ran the camera driver and Ethernet driver on MCU2_0. This phenomenon occurs when the machine is powered on for the first time. The probability is very high. After the phenomenon occurs, the Ethernet can be used normally, so I understand that MCU2_0 is in a running state.

    2. I don’t use the debugging tools you mentioned locally.

    3. the issue occur when i leave the application to run for a long time 

    4. I don't change the buffer depth.

  • so I understand that MCU2_0 is in a running state.

    Ok, so that is good. In this case, could you get the capture stats printed once the issue occurs?

    In the multicam graph, when you press "p" (while running in interactive mode), it should print out the stats from the csi-rx. Could you share the same when the issue occurs?

    Regards,

    Nikhil

  • hi TI

    1. We do not support the way of communication you mentioned.
    2. This problem has a very small probability and is difficult to reproduce. It has not reoccurred so far.
    3. This problem occurs when the machine is powered on for the first time. My question now is: Why does the IPC have problems?
  • Hi,

    This problem occurs when the machine is powered on for the first time. My question now is: Why does the IPC have problems?

    I think the hypothesis here would be that the IPC is blocked by the framework (i.e. some core IPC communication in this endpoint). 

    Hence, I think the other IPCs are not going through.


    In this case, could you get the capture stats printed once the issue occurs?

    Only the capture stats in this case would help us identify what is the issue that is occuring here.

    Regards,

    Nikhil

  • hi ti

    What do capture stats refer to and what logs are they?

  • HI TI

    When a problem occurs, the IPC code will display the following error. May I ask what is causing this

  • Hi,

    Could you enable APP_IPC_DEBUG macro in the file app_ipc_rtos.c and app_ipc_linux_priv.h present in vision_apps/utils/ipc/src/ folder.

    With this you could get more logs from the IPC transaction. 

    If you have already run ./vision_apps_init.sh , then you would get the ipc logs from remote cores as well after enabling this macro.

    What do capture stats refer to and what logs are they?

    Capture stats logs would give you an idea on why the hang on the video stream occured (i.e. there is crccount which increases if the sender is faulty, then there is overflow flags, and frame drops could be seen if any etc)

    Regards,

    Nikhil

  • hi TI 

    This is the IPC related log we obtained
    APP: Init ... !!!
    MEM: Init ... !!!
    MEM: Initialized DMA HEAP (fd=5) !!!
    MEM: Init ... Done !!!
    IPC: Init ... !!!
    IPC: Init ... Done !!!
    REMOTE_SERVICE: Init ... !!!
    REMOTE_SERVICE: Init ... Done !!!
     71362.089994 s: GTC Frequency = 0 MHz
    APP: Init ... Done !!!
     71362.090057 s:  VX_ZONE_INIT:Enabled
     71362.090059 s:  VX_ZONE_ERROR:Enabled
     71362.090061 s:  VX_ZONE_WARNING:Enabled
     71362.091705 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
     71362.091981 s:  VX_ZONE_INIT:[tivxHostInitLocal:86] Initialization Done for HOST !!!
     71377.254269 s: IPC: TX: FAILED: mpu1_0 -> mcu2_0 (port 13) msg = 0xdead0000
    APP IPC: ERROR: Send msg 1 to CPU [mcu2_0] failed !!!
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000f
     71392.358021 s: IPC: TX: FAILED: mpu1_0 -> mcu2_0 (port 13) msg = 0xdead0000
    APP IPC: ERROR: Send msg 1 to CPU [mcu2_0] failed !!!
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000f
     71407.462610 s: IPC: TX: FAILED: mpu1_0 -> mcu2_0 (port 13) msg = 0xdead0000
    APP IPC: ERROR: Send msg 1 to CPU [mcu2_0] failed !!!
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000f
     71422.567013 s: IPC: TX: FAILED: mpu1_0 -> mcu2_0 (port 13) msg = 0xdead0000
    APP IPC: ERROR: Send msg 1 to CPU [mcu2_0] failed !!!
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: mcu2_1 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: mcu3_0 -> mpu1_0 (port 13) msg = 0xdead000f
    APP IPC: Waiting for all messages to get echoed from remote core...
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0000
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0001
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0002
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0003
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c6x_1 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0004
    IPC: RX: mcu3_1 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0005
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0006
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0007
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0008
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead0009
    IPC: RX: c6x_2 -> mpu1_0 (port 13) msg = 0xdead000f
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000a
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000b
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000c
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000d
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000e
    IPC: RX: c7x_1 -> mpu1_0 (port 13) msg = 0xdead000f
    APP IPC: Waiting for all messages to get echoed ... Done.
    APP IPC: Running remote service test ... 
     71432.568057 s: REMOTE_SERVICE_TEST: Running test for CPU mcu2_0 !!!
    REMOTE_SERVICE: TX: FAILED: mpu1_0 -> mcu2_0 (port 21) cmd = 0x00001234, prm_size = 4 bytes
     71447.658136 s: REMOTE_SERVICE_TEST: Test failed @ iteration 0 !!!
     71447.658257 s: REMOTE_SERVICE_TEST: Running test @ 0xc99a0000 of 1024 bytes size for CPU mcu2_0 !!!
    REMOTE_SERVICE: TX: FAILED: mpu1_0 -> mcu2_0 (port 21) cmd = 0x00005678, prm_size = 4 bytes
     71462.758531 s: REMOTE_SERVICE_TEST: Test failed @ iteration 0 !!!
     71462.758542 s: REMOTE_SERVICE_TEST: Running timer test of 10000 msecs for CPU mcu2_0 !!!
    REMOTE_SERVICE: TX: FAILED: mpu1_0 -> mcu2_0 (port 21) cmd = 0x00000002, prm_size = 4 bytes
     71477.862692 s: REMOTE_SERVICE_TEST: ERROR: Timer test !!!
     71477.862696 s: REMOTE_SERVICE_TEST: Running timer test of 10000 msecs for CPU mcu2_0 ... DONE !!!
     71477.862700 s: REMOTE_SERVICE_TEST: Running test for CPU mcu2_1 !!!
     71477.864500 s: REMOTE_SERVICE_TEST: Test passed !!!
     71477.864579 s: REMOTE_SERVICE_TEST: Running test @ 0xc99a0000 of 1024 bytes size for CPU mcu2_1 !!!
     71477.865871 s: REMOTE_SERVICE_TEST: Test passed !!!
     71477.865876 s: REMOTE_SERVICE_TEST: Running timer test of 10000 msecs for CPU mcu2_1 !!!
     71487.865547 s: REMOTE_SERVICE_TEST: Running timer test of 10000 msecs for CPU mcu2_1 ... DONE !!!
     71487.865561 s: REMOTE_SERVICE_TEST: Running test for CPU mcu3_0 !!!
     71487.866584 s: REMOTE_SERVICE_TEST: Test passed !!!
     71487.866672 s: REMOTE_SERVICE_TEST: Running test @ 0xc99a0000 of 1024 bytes size for CPU mcu3_0 !!!
     71487.867733 s: REMOTE_SERVICE_TEST: Test passed !!!
     71487.867736 s: REMOTE_SERVICE_TEST: Running timer test of 10000 msecs for CPU mcu3_0 !!!
     71497.867742 s: REMOTE_SERVICE_TEST: Running timer test of 10000 msecs for CPU mcu3_0 ... DONE !!!
     71497.867756 s: REMOTE_SERVICE_TEST: Running test for CPU mcu3_1 !!!
     71497.868569 s: REMOTE_SERVICE_TEST: Test passed !!!
     71497.868655 s: REMOTE_SERVICE_TEST: Running test @ 0xc99a0000 of 1024 bytes size for CPU mcu3_1 !!!
     71497.869561 s: REMOTE_SERVICE_TEST: Test passed !!!
     71497.869564 s: REMOTE_SERVICE_TEST: Running timer test of 10000 msecs for CPU mcu3_1 !!!
     72014.835259 s: REMOTE_SERVICE_TEST: Running timer test of 10000 msecs for CPU mcu3_1 ... DONE !!!
     72014.835273 s: REMOTE_SERVICE_TEST: Running test for CPU c6x_1 !!!
     72014.837388 s: REMOTE_SERVICE_TEST: Test passed !!!
     72014.837487 s: REMOTE_SERVICE_TEST: Running test @ 0xc99a0000 of 1024 bytes size for CPU c6x_1 !!!
     72014.837890 s: REMOTE_SERVICE_TEST: Test passed !!!
     72014.837893 s: REMOTE_SERVICE_TEST: Running timer test of 10000 msecs for CPU c6x_1 !!!
     72024.837098 s: REMOTE_SERVICE_TEST: Running timer test of 10000 msecs for CPU c6x_1 ... DONE !!!
     72024.837114 s: REMOTE_SERVICE_TEST: Running test for CPU c6x_2 !!!
     72024.837434 s: REMOTE_SERVICE_TEST: Test passed !!!
     72024.837523 s: REMOTE_SERVICE_TEST: Running test @ 0xc99a0000 of 1024 bytes size for CPU c6x_2 !!!
     72024.837920 s: REMOTE_SERVICE_TEST: Test passed !!!
     72024.837922 s: REMOTE_SERVICE_TEST: Running timer test of 10000 msecs for CPU c6x_2 !!!
     72034.837983 s: REMOTE_SERVICE_TEST: Running timer test of 10000 msecs for CPU c6x_2 ... DONE !!!
     72034.838098 s: REMOTE_SERVICE_TEST: Running test for CPU c7x_1 !!!
     72034.838689 s: REMOTE_SERVICE_TEST: Test passed !!!
     72034.838779 s: REMOTE_SERVICE_TEST: Running test @ 0xc99a0000 of 1024 bytes size for CPU c7x_1 !!!
     72034.839118 s: REMOTE_SERVICE_TEST: Test passed !!!
     72034.839120 s: REMOTE_SERVICE_TEST: Running timer test of 10000 msecs for CPU c7x_1 !!!
     72044.838407 s: REMOTE_SERVICE_TEST: Running timer test of 10000 msecs for CPU c7x_1 ... DONE !!!
    APP IPC: Running remote service test ... Done.
     72044.838434 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:100] De-Initialization Done for HOST !!!
     72044.842813 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
    APP: Deinit ... !!!
    REMOTE_SERVICE: Deinit ... !!!
    REMOTE_SERVICE: Deinit ... Done !!!
    IPC: Deinit ... !!!
    IPC: DeInit ... Done !!!
    MEM: Deinit ... !!!
    MEM: Alloc's: 7 alloc's of 7168 bytes 
    MEM: Free's : 7 free's  of 7168 bytes 
    MEM: Open's : 0 allocs  of 0 bytes 
    MEM: Deinit ... Done !!!
    APP: Deinit ... Done !!!
    APP IPC: Done !!!
    



    What is the reason for the error in the log?
  • Hi,

    If you have already run ./vision_apps_init.sh , then you would get the ipc logs from remote cores as well after enabling this macro.

    As mentioned above, please run this scirpt before and then take the ipc application logs, to get error logs from [MCU2_0].

    Could you enable APP_IPC_DEBUG macro in the file app_ipc_rtos.c

    Since you have enabled the macro in rtos side too, you should get some logs from [MCU2_0] when running the IPC application. Please let me know if you already tried this?

    Regards,

    Nikhil

  • hi

    The probability of this problem occurring is very small and difficult to reproduce.
  • Hi,

    I totally understand that.

    But we would require this information to understand the issue.

    Capture stats logs would give you an idea on why the hang on the video stream occured (i.e. there is crccount which increases if the sender is faulty, then there is overflow flags, and frame drops could be seen if any etc)

    Please share this data too.

    Also when there is a hang in the multicam application, is it possible to check where exactly is the hang? It would be great if you could setup a debugger here.

    Regards,

    Nikhil

  • hi

    May I ask how to set up the debugger