TDA2P-ACD: Another issue with corrupted behavior - EVE rendering issue

Hi,

    Both init and process function of remap application returns error for unsupported configuration. Can  you confirm if you are checking errors from these API's?

Regards,
Anshu

Hi Anshu, 

We have a check on the return of "Not ok" return. 

In such a case we got a not ok return there is a validate point that will cause the SOC to reset, but we are not seeing a reset. 

We need to understand what are the possible failure cases that can cause such a failure in EVE core, Can you please share analysis of those failure modes ? 

Also, please provide list of APIs that can help in root cause analysis of the issue. 

• What are the possible errors 
• How to simulate them
• Diagnostic information

output resolution 1120 x 664

input camera resolution 1280 x 800

input tile size : 32 x 32 

output tile size 32 x 35

Hi Mahmoud,

     Can  you share the following information to us to analyze :

• EVE software version being used?
• Dumps of the following structures: 
  • REMAP_MERGE_TI_CreateParams
  • RemapParms::maps:srcMap
  • RemapParms::maps structure along with content of all the pointers in this data structure
• Also do you have any update on some of the experiments we discussed yesterday?

Regards,

Anshu

eve version- eve_sw_01_06_01_00_valeo_003

Update from experiments so far :

- by commenting out the remap function we are seeing black screen in the single camera view, we are still seeing the 360 portion normally.

- by stopping the trigger of EVE : we are seeing complete black screen without the 360 portion

- by injecting a while(1) loop in EVE we are seeing complete black screen without the 360 portion

For the memory dump from the requested structures : we are working to extract that from debugger and share here.

memory dump from the requested structures is below : 

LUT_Extract.zip

Hello Mahmoud and team,

• Does you system enable parity checks?
• Can you check for parity erors in the following registers:

• • EVE_PMEM_ED_STAT
  • EVE_DMEM_ED_STAT
  • EVE_WBUF_ED_STAT
  • EVE_IBUF_ED_STAT

Thanks,

Kyle

Trial by giving repeated pattern in srcMap (from DSP side) resulted in below pattern : 

It seems a "similar" pattern to what was seen at FORD site , can you please help to identify what are the causes that can lead to a repeated pattern ? I checked internally and the generation of the LUT was based on a TI example so I believe we need to review that in a call. Let us know when you are available. Also, we need to consider the probability if the LUT data is passed correct and EVE is using the data from one tile only repeatedly to construct output video image.

Hi Kyle, 

We don't see parity check configuration in application side, how to check if they are enabled? 

We are going to get an extract from the registers mentioned in a normal working case. Note : issue reported is not reproducible so far, so we can't get the register reading while the issue is active.

Hi Mahmoud,

Parity is enabled in the following registers:

• EVE_PMEM_ED_CTL
• EVE_DMEM_ED_CTL
• EVE_WBUF_ED_CTL
• EVE_IBUF_ED_CTL

Regards,

Kyle

Hello,

As we discussed, another register to check is EVE_MSW_ERR:

Regards,

Kyle

Hello Kyle , 

When we read the registers from EVE core we are seeing a exception  being raised   

 and we tried  reading through the memory  window as well , looks like its not accessible register as all zeros are shown  for the above listed EVE reg

From DSP we tried reading them , but the values are all zeros 

Can you please suggest how to read these registers ?

Thanks 

Satish

Hello ,  

current MMU  configuration for the EVE 1 core  , please suggest the MMU  changes for the reading the  EVE register 

void configureAndProgramMMU_EVE_1(void)
{

/* Enable access to entire DDR */
SetupEveMmuEntry( 5U, 0x4a000000U, 0x4a000000U);
SetupEveMmuEntry( 6U, 0xC3000000U, 0xC3000000U);
SetupEveMmuEntry( 7U, 0x81000000U, 0x81000000U);
SetupEveMmuEntry( 8U, 0x83000000U, 0x83000000U);

/* Non Cache Area Access for IPC */
SetupEveMmuEntry( 9U, 0xA0000000U, 0xA0000000U);
SetupEveMmuEntry( 10U,0xA1000000U, 0xA1000000U);

SetupEveMmuEntry( 11U, 0x84000000U, 0x84000000U);
SetupEveMmuEntry( 12U, 0xC2000000U, 0xC2000000U);
SetupEveMmuEntry( 13U, 0xC1000000U, 0xC1000000U);
SetupEveMmuEntry( 14U, 0xC0000000U, 0xC0000000U);

/* Video Buffers Access */
SetupEveMmuEntry( 15U, 0x88000000U, 0x88000000U);
SetupEveMmuEntry( 16U, 0x89000000U, 0x89000000U);
SetupEveMmuEntry( 17U, 0x8A000000U, 0x8A000000U);
SetupEveMmuEntry( 18U, 0x8B000000U, 0x8B000000U);
SetupEveMmuEntry( 19U, 0x8C000000U, 0x8C000000U);
SetupEveMmuEntry( 20U, 0x8D000000U, 0x8D000000U);
SetupEveMmuEntry( 21U, 0x8E000000U, 0x8E000000U);
SetupEveMmuEntry( 22U, 0x8F000000U, 0x8F000000U);
SetupEveMmuEntry( 23U, 0x90000000U, 0x90000000U);
SetupEveMmuEntry( 24U, 0x91000000U, 0x91000000U);
SetupEveMmuEntry( 25U, 0x92000000U, 0x92000000U);
SetupEveMmuEntry( 26U, 0x93000000U, 0x93000000U);
SetupEveMmuEntry( 27U, 0x94000000U, 0x94000000U);
SetupEveMmuEntry( 28U, 0x95000000U, 0x95000000U);

SetupEveMmuEntry( 29U, 0xC4000000U, 0xC4000000U);
SetupEveMmuEntry( 30U, 0xC5000000U, 0xC5000000U);
SetupEveMmuEntry( 31U, 0xC6000000U, 0xC6000000U);

//Program the Translation Table Base address in DDR
WR_REG(EVE_1_MMU_TTB, TRANS_BUFFER_IN_DDR_EVE_0);
//Program the Enabling of the MMU through the MMU_CNTL
WR_REG(EVE_1_MMU_CNTL, 0xEU); //Enabled MMUENABLE, TWLENABLE, EMUTLBUPDATE


return;
}

Hello Mahmoud and team,

Extending on the previous recommendations, can you collect all of these registers (For all memories, <MEM> = PMEM, DMEM, IBUF, WBUF)

- EVE_<MEM>_ED_CTL

- EVE_<MEM>_ED_STAT

- EVE_<MEM>_EDADDR

- EVE_<MEM>_EDADDR_BO

And MSW related:

- EVE_MSW_CTL

- EVE_MSW_ERR

- EVE_MSW_ERRADDR

Note, that these registers are accessed within the EVE subsystem and do not use the MMU, as described here:

You need to use base address of 0x4008_xxxx to access this bank of registers:

I will post separately on ARP32 and EDMA related registers.

Regards,

Kyle

Hello,

Please also collect these regs related to ARP32:

- ARP32_IRQSTATUS_RAW

Related to EDMA:

- EDMA_TPCC_EMR

- EDMA_TPCC_EMRH

- EDMA_TPCC_QEMR

- EDMA_TPTCn_ERRSTAT (n=0 and 1 for two TCs)

- EDMA_TPTCn_ERRDET (n=0 and 1 for two TCs)

Thanks,

Kyle

Hi Kyle, 

We are still not getting correct values for register reading. Please clarify the below points : 

1- from your quote "You need to use base address of 0x4008_xxxx to access this bank of registers" , the register address offset for registers like MSW register is not matching the region size as below : 

While for example the EVE_MSW_ERR register address offset is 0x0008 0028 

So , Can you please clarify what address exactly shall be used to read MSW register for example ? 

Note : The team is going to reply soon with the result of reading of the registers so far.

Hello  Kyle , 

when we read for base register 0x4008_xxxx   , we are getting 0  read as values ,  

Can you please provide the method to read the EVE  status registers . 

Hello,

The specific registers at 0x80, 90, 0xA0, 0xB0 are the <MEM>_ED_CTL R/W registers that default to 0x0, so if you didn't enable them it would make sense to see 0x0 data.

As an experiment, I sat with one of our engineers, and we wrote to the following registers with 0xFFFFFFFFF

EVE_PMEM_ED_CTL @ 0x40080080 => Result is 0x3 (only bits 0,1 are writeable)

EVE_MSW_CTL @ 0x40080024 => Result is 0x00011111 (bits 0, 4, 8, 12, 16 are writeable)

This experiment was done independently on EVE1 and EVE2 instances.  This should proves that the "local address" for EVE resources is fixed within an EVE instance.

As a "just in case" you may also set the MMU as follows:

SetupEveMmuEntry( ?? , 0x40000000U, 0x40000000U);

This was originally in the GEL file we were using, but we removed it and got the same result.  The "just in case" is in case we somehow misinterpreted the quick experiment.

Regards,
Kyle

Hi Kyle,

Initialized the eve control regs 80, 90, A0, B0, 24 with 0xFFFFFFFF.

Still the register value showing as zero except for 80 and 90 registers.

- with A0 and B0 - setting to 0xFFFF FFFF - their is no streaming

- tried mapping the 0x40000000 in mmu - with the ecu is not booting up.

-Can you please let us know how to enable all the error registers which you have mentioned to get in previous reply.

Hello,

Just to be explicit, we coded up the sequence and log into a GEL file script.

Output log is here:

Output log:

ARP32_EVE_1: GEL Output: Reading Registers

ARP32_EVE_1: GEL Output: 0x40080080: 0x00000000

ARP32_EVE_1: GEL Output: 0x40080090: 0x00000000

ARP32_EVE_1: GEL Output: 0x400800A0: 0x00000000

ARP32_EVE_1: GEL Output: 0x400800B0: 0x00000000

ARP32_EVE_1: GEL Output: 0x40080024: 0x00000000

ARP32_EVE_1: GEL Output:

Writing 0xFFFFFFFF to each register...

ARP32_EVE_1: GEL Output: Reading Registers

ARP32_EVE_1: GEL Output: 0x40080080: 0x00000003

ARP32_EVE_1: GEL Output: 0x40080090: 0x00000003

ARP32_EVE_1: GEL Output: 0x400800A0: 0x00000003

ARP32_EVE_1: GEL Output: 0x400800B0: 0x00000003

ARP32_EVE_1: GEL Output: 0x40080024: 0x00011111

GEL script is here:

menuitem "EVE Register Check"

#define WR_MEM_32(addr, data) *(unsigned int*)(addr) =(unsigned int)(data)
#define RD_MEM_32(addr) *(unsigned int*)(addr)
#define uint32_t unsigned int

hotmenu register_check()
{
GEL_TextOut("Reading Registers \n");
GEL_TextOut("0x40080080: %x\n",,,,, RD_MEM_32(0x40080080));
GEL_TextOut("0x40080090: %x\n",,,,, RD_MEM_32(0x40080090));
GEL_TextOut("0x400800A0: %x\n",,,,, RD_MEM_32(0x400800A0));
GEL_TextOut("0x400800B0: %x\n",,,,, RD_MEM_32(0x400800B0));
GEL_TextOut("0x40080024: %x\n",,,,, RD_MEM_32(0x40080024));

GEL_TextOut("\nWriting 0xFFFFFFFF to each register...\n");
WR_MEM_32(0x40080080, 0xFFFFFFFF);
WR_MEM_32(0x40080090, 0xFFFFFFFF);
WR_MEM_32(0x400800A0, 0xFFFFFFFF);
WR_MEM_32(0x400800B0, 0xFFFFFFFF);
WR_MEM_32(0x40080024, 0xFFFFFFFF);


GEL_TextOut("Reading Registers \n");
GEL_TextOut("0x40080080: %x\n",,,,, RD_MEM_32(0x40080080));
GEL_TextOut("0x40080090: %x\n",,,,, RD_MEM_32(0x40080090));
GEL_TextOut("0x400800A0: %x\n",,,,, RD_MEM_32(0x400800A0));
GEL_TextOut("0x400800B0: %x\n",,,,, RD_MEM_32(0x400800B0));
GEL_TextOut("0x40080024: %x\n",,,,, RD_MEM_32(0x40080024));

}

Regards,

Kyle

Hello Kyle,

I have a question related to the testing that we discussed previously. You proposed to increase/decrease EVE voltage by 30 and 50 mvolts. 
what is the max and min voltage ranges that we can test EVE on. 
Ford team was proposing testing with 100mvolts step. Do you recommend that?

Thanks,

Sara

Hi Kyle, 

Can you please also provide if there are similar registers that we can check at the DSP side ? 

Hello Kyle, 

We have observed that ARP32ERR error is set for EVE_PMEM_ED_STAT even after clearing it. it becomes set again. We check the view and camera was streaming normally. we have captured a video for this behavior. 

Please check and let us know if we are doing something incorrect and how does this error impact the system? 

Thanks,

Sara

Sara,

-1-

I would recommend 50mV to start.  Reducing by 100mV results may not be indicative of the real failure.

-2- 

Re: PMEM parity errors ...If the cache is already active when parity is enabled then the parity bit is likely uninitialized leading to the parity error.  Immediately after PMEM parity is enabled can you initiate a Global Invalidate:

Regards,

Kyle

As per our discussion today. Please find the fishbone analysis stating all actions done.

JIRA 60912 CX430 _Tessellated_ RVC Image.pdf

Thanks,

Sara

Hello,  As we discussed, can you extract the DIE_ID (and neighboring) registers highlighted here:

Regards,

Kyle

Hello Kyle ,

As we discussed, We are facing issues in running DDR memtester for the following cores

- EVE2 DDR
- A15 DDR
- IPU2 DDR
- EVE1 DDR

Please find attached the logs for the failed testes and the test package we used.

memtester_dat2p2.zip

Failed.zip

Please check and let me know your feedback.

Thanks,

Sara

Hello Sara,

Do the same tests which failed here fail on a reference unit?   The failures in the logs look to be likely from incompatibilities in running from your base image.  Can you also attach the passing logs? 

I would tend to guess the point you launched the tests (to include gels) scripts in some cases were hindered and some cases helped by system initialization.   The tests normally assume a 'boot by gel' where it initialized and parks cores in a safe state.   I recall your team successfully ran many of these against different base software versions but I don't understand what all was there.

Unlocking this thread.

Sara, can you please respond to Richards questions please

regards

Dave C

Dave and Richard,

We managed to run the memtester on a reference ECU (Golden ECU) and as well as on the suspect ECU  and both were passing.