This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CCS/AM6548: R5F interrupt jitter with no OS

Part Number: AM6548

Tool/software: Code Composer Studio

Hello,

Environment:

  • Hardware: AM65x IDK
  • Software: ti-processor-sdk-rtos-am65xx-evm-06_01_00_08

We have conducted the same experiment in this post without TI-RTOS. This time we took your advice and put everything inside TCMA and TCMB.

The experiment is like:

  1. we utilize the example code <PDK>\packages\ti\csl\test\dmTimerUt\
  2. create a timer which generates interrupts every 125us
  3. get CPU tick returned by CSL_armR5PmuReadCntr() in the beginning of the Timer ISR
  4. calculate the difference time between every two interrupts

The theoretical value of the difference time should be 125us; however, after running the program for one hour we get the min value is 124.035 us and max value is 126.045 us which shows the jitter is around 2.01 (126.045 - 124.035) us.

Are the numbers reasonable? Or is there anything I understand or configure wrong?

The source code:

0878.dmTimerUt.zip

The linker file:

7776.linker_r5.zip

  • Hi Hungwei,

    Thanks for sharing the bare-metal code. I need to review the code and measurements.

    Regards,
    Frank

  • Hello Frank,

    We have run the test for for 13 hours and the result is the 2.090 us. FYI.

    --------

    The result for running the test for one hour (which is shown in my last post):

    maxdelta: 126.045
    mindelta: 124.035

    jitter = (126.045 - 124.035) = 2.010 (ns)

    --------

    The result for running the test for 13 hours

    maxdelta: 126.090
    mindelta: 124.000

    jitter = (126.090 - 124.000) = 2.090 (ns)

  • Hi Hungwei,

    Thanks for the additional data. I'm currently a bit delayed in looking at your example code. I'll get back with you soon.

    Regards,
    Frank

  • Hi Hungwei,

    I'm looking at the bare-metal example first. Thanks for your patience on this issue.

    Regards,
    Frank

  • Hi Hungwei,

    I compiled your example code and executed it on AM65xx R5F_0 for one hour using "Free Run" in CCS. I observe similar results to those you've mentioned above. 

    I see you're using MCU_TIMER2 with a 25 MHz functional clock.

    I see the positive increment register, negative increment register, and counter value registers are correctly zeroed.

    I reviewed your memory allocation, and it looks good. I see all code/data are allocated to TCMA/B. I see you're using the default CSL MPU settings from pdk_am65xx_1_0_6\packages\ti\csl\arch\r5\src\startup\startup.c, which are proper for TCMA/B.

    I'll continue my investigation and keep you posted.

    Regards,
    Frank

  • Hi Hungwei,

    I instrumented the DM Timer ISR with GPIO toggle and captured the resulting signal using a logic analyzer. I set the logic analyzer sampling frequency to 500 Msps (-/+ 2 ns resolution) and set the capture time for 60 min. The results of the capture are shown below. I've added measurements showing the min/max positive and negative pulse duration. As can be seen, the GPIO toggle capture also shows approximately -/+ 1 usec. of jitter.

    Here I've zoomed in on a long negative pulse. It appears the edge located at the A1/B1 timing markers is delayed by ~1.03 usec, so the next positive pulse is shortened by about the same amount of time.

    I'm continuing my investigation, and I'll keep you posted.

    Regards,
    Frank

  • Hi Hungwei,

    I routed MCU_TIMER2 output externally to MCU_TIMER_IO0, and set the timer output to toggle on overflow. I captured the results on a logic analyzer. As before, I set the logic analyzer sampling frequency to 500 Msps and set the capture time for 60 min. The results are shown below.

    As can be seen, there isn't any jitter in the MCU_TIMER2 output.

    My next step is to look at the VIM/VIC software configuration.

    Regards,
    Frank

  • Hi Hungwei,

    I added GPIO toggle to the CSL masterIsr() (this ISR calls the registered user ISR function for a particular interrupt, see <PDK>\packages\ti\csl\arch\r5\src\interrupt.c) and captured three signals using a logic analyzer:

    1. GPIO toggle @ entry to masterIsr()
    2. GPIO toggle @ entry to registered ISR for test case, csldmTimerTest_Stub()
    3. MCU_TIMER2 PWM output from MCU_TIMER_IO0

    For captures with these signals, I observe unexpected GPIO toggles (i.e. interrupts) on #1 which occur with ~1 sec. period. For example, this screen capture shows a GPIO toggle ~3/4 through the 125 usec. period:

    I latched the interrupt number in masterIsr(), and found these interrupts are for interrupt number 38 which is MCU_TIMER0. Further, it appears the MCU_TIMER0 interrupts cause the jitter in the MCU_TIMER2 interrupts. This occurs when the MCU_TIMER0 interrupt occurs slightly before the MCU_TIMER2 interrupt. The is shown in this screen capture:

    Thus it appears the MCU_TIMER0 interrupts are delaying the MCU_TIMER2 interrupts.

    If I disable MCU_TIMER0 by writing TIMER_TCLR.ST==0, then the MCU_TIMER2 interrupt jitter is eliminated.

    Regards,
    Frank

  • Hi Hungwei,

    I found MCU_TIMER0 is used in the example because of this call in main():

    cslApp_print("\n DmTimer Test Application\r\n");

    Apparently this timer is used to implement a delay function in the bare-metal UART code. Below is call graph showing where the timer is configured and started:

    main()
        cslApp_print()
            UART_printf()
                UART_dataWrite()
                    UART_putc()
                        uart_console_putc()
                            UART_writePolling()
                                UART_writePolling_v1()
                                    UART_charPut_v1()
                                        Osal_delay()
                                            TimerP_create()     // FL: create timer
                                                TimerP_dmTimerInstanceInit()
                                                    TimerP_dmTimerDeviceCfg()
                                                        TimerP_dmTimerInitDevice()
                                                            TIMERDisable()
                                                                HW_WR_FIELD32()
    
                                        Osal_delay()
                                            osal_core_delay()
                                                osalArch_Timestampinit()
                                                    TimerP_start() // FL: start timer

    MCU_TIMER2 jitter is eliminated if the call to cslApp_print() is removed from main().

    Regards,
    Frank

  • Hi Hungwei,

    I inspected VIM_VEC_INT_38 & VIM_VEC_INT_40 for MCU_TIMER0 and MCU_TIMER2. I found both registers contain the address of the CSL masterIsr().

    • For MCU_TIMER2, the registered user ISR called from masterIsr() is located in fxnArray[40] (see <PDK>\packages\ti\csl\arch\r5\src\interrupt.c) and is csldmTimerTest_Stub().
    • For MCU_TIMER0, the registered user ISR called from masterIsr() is located in fxnArray[38] and is TimerP_dmTimerStub() (see <PDK>\packages\ti\osal\src\nonos\timer\v1\TimerP_nonos.c:153:static void TimerP_dmTimerStub(uintptr_t arg))

    I also inspected VIM_PRI_INT_38 & VIM_PRI_INT_40 for the timers, and found both registers contain 0x0 (highest priority). According to AM65xx TRM Section 6.3.3.6.5 VIM Interrupt Prioritization, the VIM is able to prioritize interrupts and interrupt an existing ISR if a higher priority interrupt occurs. Therefore I decided to experiment with changing VIM_PRI_INT_38 to 0x1 (lower priority) to see if MCU_TIMER2 interrupt jitter would be eliminated. Unfortunately the jitter is still present.

    I'll continue investigating this and keep you posted.

    Regards,
    Frank

  • Hello Frank,

    Thanks for the update! I will check if disabling MCU_TIMER0 eliminates the MCU_TIMER2 jitter on my side. 

    >the VIM is able to prioritize interrupts and interrupt an existing ISR if a higher priority interrupt occurs. Therefore I decided to experiment with changing VIM_PRI_INT_38 to 0x1 (lower priority) to see if MCU_TIMER2 interrupt jitter would be eliminated. Unfortunately the jitter is still present.

    It seems the VIM does not prioritize correctly these interrupts? Looking forward for your further response.

  • Hi Hungwei,

    >> It seems the VIM does not prioritize correctly these interrupts?

    This may be the case, I'm investigating this now.

    As part of this effort, I decided to add debug GPIO outputs as follows:

    1. GPIO #1: masterIsr(), toggle @ entry.
    2. GPIO #2: csldmTimerTest_Stub(), drive high @ entry, drive low @ exit.
    3. GPIO #3: TimerP_dmTimerStub(), drive high @ entry, drive low @ exit.

    The capture below shows the MCU_TIMER2 jitter for VIM_PRI_INT_38==0x0 & VIM_PRI_INT_40==0x0:

    • A1-A2 is the time difference between subsequent entries to masterIsr() for the MCU_TIMER2 interrupt
    • B1-B2 is the time difference between subsequent entries to csldmTimerTest_Stub()
    • The portions of the capture circled in red show entry into masterIsr() for the MCU_TIMER0 interrupt, and the TimerP_dmTimerStub() entry/exit.

    Regards,
    Frank

  • Hi Hungwei,

    I added the following code to masterIsr() to re-enable IRQ interrupts in the ISR and allow nested IRQ interrupts:

        CSL_armR5IntrEnableIrq(1); // enable nested interrupts

    I then manually set VIM_PRI_INT_38=0xF via JTAG to reduce the MCU_TIMER0 priority.

    In this case, I observe what might be incorrect hardware behavior. The marker at A2 shows the MCU_TIMER2 event is taken (masterIsr() executes), and the MCU_TIMER2 ISR executes (pulse on GPIO1). However (see red circles), there is a spurious event generated for MCU_TIMER0 ISR and the MCU_TIMER0 ISR runs again from the start.


    One possible solution to this is to disable the VIM/VIC hardware handshake mode.

    It's possible I need to add more code to the DM Timer example to properly enable nested interrupt processing.

    I'll check into this issue further and keep you posted.

    Regards,
    Frank

  • Hi Hungwei,

    I consulted internally on this issue, and discovered there is a hardware bug for AM65xx R5F VIM/VIC hardware handshake mode. Because of this bug, prioritized / nested interrupt processing doesn't work correctly.

    The AM65xx R5F CSL doesn't currently support nested hardware interrupts, and doesn't implement a workaround for the VIM/VIC hardware bug.

    Regards,
    Frank

  • Hello Frank,

    Thanks for the update. Does this mean any other interrupts might delay my MCU_TIMER2's ISR which makes MCU_TIMER2 jitter be higher than 2 us? Like 3 or 5 us when we create more than two interrupts.

    What would you recommend us to do then? We really need a hardware interrupt with jitter of 1us or less.

  • Hello Frank,

    Would TI propose a workaround on AM65x for the VIM/VIC hardware bug?

  • Hello Frank,

    The MCU_TIMER jitter is decreased from 1.98us to 0.107us after removing cslApp_print("..."); on my side.

    Before removing cslApp_print("...");

    After removing cslApp_print("...");

  • Hello Hungwei,
    Hello Frank,

    this is the thread where I reported the issue about VIM interrupt nesting: https://e2e.ti.com/support/processors/f/791/t/848213

    The proposed workaround is not to use the hardware handshake mechanism (VE bit, "Vector Enable"), and the thread I linked contains a fix for SYS/BIOS. That fix has been working fine for us since then. The latest SYS/BIOS version which is available for download separately from the processor SDK should already contain that fix. Of course this only fixes the issue for SYS/BIOS, not for bare-metal / CSL. On SYS/BIOS you don't loose anything if you disable the hardware handshake mechanism, since in SYS/BIOS every interrupt handler goes through the same multiplexer function that looks up the actual handler from a table anway.

    You should be able to prioritize interrupts on SYS/BIOS with the hardware handshake disabled. If you put all of the "common" (used by all ISRs) interrupt handling code and your critical code in TCM I'd expect you to achieve an interrupt latency for your high-priority interrupt that is below 1us. Please note that this is something we never tested, since we've moved from the R5f to the A53 since then.

    Regards,

    Dominic

  • Hi Hungwei,

    >> Does this mean any other interrupts might delay my MCU_TIMER2's ISR which makes MCU_TIMER2 jitter be higher than 2 us?

    Yes, the amount of delay depends on the duration of the ISR for the other interrupt.

    >> What would you recommend us to do then? We really need a hardware interrupt with jitter of 1us or less.

    For AM65xx and bare-metal code, I think a CSL update is required. I'll ask about this internally and get back with you.

    Regards,
    Frank

  • Hi Hungwei,

    CSL doesn't currently support VIM nested interrupts. If this feature were added, then I think the workaround for the VIM/VIC bug which has been implemented in SYS/BIOS (mentioned by Dominic) should work. I'm not clear yet on the performance impact (interrupt latency) for this workaround. CSL also goes through a master interrupt handler and determines the interrupt number by reading a VIM register, so maybe there is little or no impact.

    Regards,
    Frank

  • Dominic,

    >> The proposed workaround is not to use the hardware handshake mechanism (VE bit, "Vector Enable")

    It's unfortunate the hardware handshake mechanism isn't working, since I believe this mode has the potential to provide the lowest possible interrupt latency.

    Regards,
    Frank

  • Hello Frank and Dominic,

    Thanks for the info! I will study the posts in https://e2e.ti.com/support/processors/f/791/t/848213 and get back with you.

  • Hello Frank,

    I am trying MCU_TIME_IO0 but it does not work. Could you see this thread (https://e2e.ti.com/support/processors/f/791/t/886587) and help me check if there is anything I missed?

  • Hungwei,

    I see your thread concerning MCU_TIMER_IO0 has been assigned to a colleague, so I'll let him respond.

    Regards,
    Frank