Hi,

Is there any update on this?

Thank you.

Regards,
Kim

I have a few questions to get better context on the problem:

-Is it the servicing of the uPP interrupt that is taking too long (some other activity is delaying it), or other write accesses are taking too long when the uPP interrupt is being serviced? 

-What type of activity is involved in the uPP ISR?  Is the ISR moving data to/from the peripheral to/from the DDR?

-Where is the ISR handled for the uPP interrupt?  Is it in the DSP?  

-are there potentially any DMAs going on when the uPP interrupt is being serviced?

-what type of peripheral is connected to uPP?

-is it possible that the latency is coming from possibly a slow peripheral interface while DDR writes are being performed from DSP?

Regards,

James

Hi James,

Thank you for the reply.

-Is it the servicing of the uPP interrupt that is taking too long (some other activity is delaying it), or other write accesses are taking too long when the uPP interrupt is being serviced? 

Yes, the issue is that it sometimes takes too long to service the uPP interrupt, because something must be blocking it.

We are measuring the uPP interrupt delay by reading a base counter from an FPGA, which follows a 100MHz clock. This specific clock frequency is due to the EMIFA interface is running at 100MHz. The base counter works by incrementing every 10ns and reversing the counting direction when it reaches either 0 or 16660.

When the FPGA transmits an uPP frame, the counter starts incrementing and the interrupt is triggered on the DSP. When the uPP ISR starts to actually execute, we initially read the FPGA counter value to determine the interrupt latency, since the FPGA transmitted the uPP frame.

An example:

On e.g. The lower graph, the highest spike indicates that the counter value is at 4123. This means that it took about 41.23us to service the uPP interrupt.

We have observed that if we either remove the code that writes to DDR memory or execute it from the uPP ISR itself, a large portion of the interrupt delay seems to disappear. This is the reason we suspect that write operations to DDR memory is delaying the uPP interrupt from being serviced somehow. Originally, the code that writes to DDR memory happens in a seperate thread on the DSP that runs periodically at a 1KHz update rate.

-What type of activity is involved in the uPP ISR?  Is the ISR moving data to/from the peripheral to/from the DDR?

The uPP ISR reads and verifies measurement data from a Xilinx Spartan 6 FPGA through uPP. This is done by using a 128 bytes data buffer associated with uPP DMA channel UPID0. Subsequently, the ISR will trigger DMA transfers that moves data from 64 bytes IRAM buffers to EMIFA async data (CS2) memory area, so it can be fetched by the FPGA via the EMIFA interface.

But all of this takes place after the uPP interrupt has been successfully serviced, so the uPP interrupt is already delayed before it begins to read measurements from the FPGA.

-Where is the ISR handled for the uPP interrupt?  Is it in the DSP?  

Yes, the ISR for uPP is handled on the DSP.

-are there potentially any DMAs going on when the uPP interrupt is being serviced?

We are triggering DMA transfers from the uPP ISR, but this happens after the uPP interrupt is already delayed. Aside from the uPP ISR, we are not triggering any DMA transfers on the DSP. However, Linux on the ARM might be doing some DMA activity in kernel drivers etc. To which we are not aware of. It is not something we have been tinkering with.

-what type of peripheral is connected to uPP?

If you mean the uPP port, then it is connected to the FPGA.

As mentioned, we are using L138-DI-225-RI SOM from Critical Link. They have provided the following block diagram, which indicates outside I/O to and from OMAP-L138.

-is it possible that the latency is coming from possibly a slow peripheral interface while DDR writes are being performed from DSP?

Could you name some examples of slow peripheral interfaces?

We have speculated that cache might be causing this problem or at least have an impact on the interrupt latency. We have configured MAR registers to use cache for specific DDR memory ranges. We are caching everything from C000 0000h – C6FF FFFFh. However, the DDR memory we are writing to from the ISR and the 1KHz update thread is allocated in C700 0000h – C7FF FFFFh, which is not cached.

Please let me know if you require any further information (drawings, code, graphs etc.)

Regards,
Kim

After tomorrow, I'll be on vacation. I will be back January 3rd, 2024.

Regards,
Kim

Kim, seems you are on the right track, in that some write is consuming the bus for longer than normal time.  A couple of ideas:

-you might be able to use the performance counter to see what the offending master is.  See section 15.4.8 in the TRM

-check  section 15.2.6.2 Command Starvation, to help prevent possible starvation of certain commands.

-can you enable cache in the region the ISR is accessing?  Maybe not as a final solution, but just a debug step to identify if the uncached memory is a source of the issue 

Regards,

James

unlocking thread

Hi James,

Sorry for the late reply.

- can you enable cache in the region the ISR is accessing?  Maybe not as a final solution, but just a debug step to identify if the uncached memory is a source of the issue 

I tried to enable cache in the C700 0000 to C7FF FFFFh area, but it did not seem to have any affect at all, unfortunately.

- you might be able to use the performance counter to see what the offending master is.  See section 15.4.8 in the TRM

As you suggested, I tried to read the performance counter 1 (PC1) on the DSP and tried to configure the master filter to only count DDR access by UPP (0x66).

In particular, I have configured the PCC and PCMRS registers like the following:

  // Setup PCC & PCMRS performance counters for DDR controller
  ddrRegs->PCC =
      (1    << 31) | // 31      = CNTR2_MSTID_EN
      (0    << 30) | // 30      = CNTR2_REGION_EN
      (0    << 20) | // 29-20   = reserved
      (0    << 16) | // 19-16   = CNTR2_CFG
      (1    << 15) | // 15      = CNTR1_MSTID_EN - Use counter 1 for upp
      (0    << 14) | // 14      = CNTR1_REGION_EN
      (0    << 4)  | // 13-4    = reserved
      (0);           // 3-0     = CNTR1_CFG

  ddrRegs->PCMRS =
      (0   << 24) | // 31-24  =  MST_ID2
      (0   << 20) | // 23-20  =  reserved
      (0   << 16) | // 19-16  =  REGION_SEL2
      (0x66 << 8) | // 15-8   =  MST_ID1
      (0   << 4)  | // 7-4    =  reserved
      (0);          // 3-0    =  REGION_SEL1

I have written some small code that runs on the DSP. The code periodically reads PC1, but the result seems a little strange to me. As I mentioned in the third post, we use IRAM (L2 RAM) to allocate memory for a pair of 128 bytes buffers, which are used for UPP.

The counter displays a very large value like e.g. 179678811, but doesn’t seem to update at all. It just stays at that large value. My periodic readings of the counter from the DSP starts a few minutes after the OMAP has booted.

To test this further, I have also attempted to move the UPP buffers from IRAM and to DDR. Specifically, I moved the buffers to a non-cached area in DDR (C7FF FC00h to C800 0000h). However, the PC1 counter is still not updating, but simply stuck on some large number.

Move UPP buffers away from IRAM (L2 RAM) completely

We are currently experimenting with moving the UPP buffers away from L2 RAM because theoretically, we think that it might be possible that the UPP interrupt could be stalled by something inside the C674x Megamodule (DSP). As an alternative we have moved the UPP buffers to Shared RAM (0x8000 0000). We are currently experiencing a few issues with our software because the location of the buffers has changed. I am currently looking into that.

In other words, our theory is that since UPP is a separate master to the DSP, something internal in the DSP might be delaying the UPP interrupt.

Regards,

Kim