Task Execution Rate Changing Over Time

Hi Jonathan,

I think the first thing to check is whether or not the SYS/BIOS system tick rate is slowing down over time. An easy way to do this is in ROV in CCS go to the Clock section and open the Module tab. Halt your program, make a note of the number of ticks, run for 30 seconds, halt your program, and check the number of ticks again making sure that roughly the correct number has passed--so if your tick period is 1000 microseconds, you should see that after 30 seconds the number of ticks has increased by about 30,000.

You can do this when you first start running the application, and then do it again after a few hours when things start running at a slower rate. Seeing whether or not the number of ticks that pass changes, should help us narrow things down a bit.

Whitney

Hi Whitney,

Here is some more information about this problem: I can recompile the code after changing as little as a single #define statement from '1' to '0'  in my code, and the task execution rate will speed up instead of slow down. It seems arbitrary whether the execution rate speeds up or slows down, although I have not yet seen the exact same compiled code become faster and then slower, or vice versa.

In this particular case, I was able to verify that the SYS/BIOS system tick rate was normal at first (~ 1,000 ticks per second), and then after about 10 minutes, I noticed things were running faster than usual (via a blinking LED), and I verified that the SYS/BIOS system tick rate was indeed faster (~ 1,280 ticks per second). I verified this faster SYS/BIOS system tick rate over several measurement intervals, and it is consistently ~ 1,280 ticks per second. So the SYS/BIOS system tick rate is not changing all the time. It seems to change once and then stay that way for a long time.

I thought I would post a little more evidence: the execution diagram of the application running as usual, and the execution diagram of the application running slower than normal (in this case, 4x slower).

Normal Execution:

Slow Execution (4x Slower):

Hi Jonathan,

I apologize for the delayed response. I've talked about this with some colleagues and it seems to us like it isn't a BIOS problem. I believe someone who knows the hardware better than I do should be getting in touch with you soon if they have not already.

I can at least suggest that it seems like over time the application may be doing something to accidentally modify timer peripheral, system clock divider, or PLL settings. Try taking a snapshot of the timer, system clock divider, and PLL setting registers initially, run until the timing starts to change, and then compare. Knowing what is changing should help track down what's causing it.

Whitney

Hi Jonathan,

As discussed yesterday, please keep us updated on the results of your debug steps. In addition to those, I have a few more suggestions to narrow down the cause of these two issues.

• Break upon register change. I know we mentioned this, and yes, there is a way to set this up. Check out section 3 of this CCS application note. It is based on CCSv4, so it may not be 1:1 with v5; However, within the break-point properties window, you can setup a breakpoint on the change of a watch expression. Creating these for the critical clocking registers will help pinpoint the section of code altering the sys tick timings.
• Look for DVFS APIs. Look around for any usage of the DVFS. This is the Dynamic Voltage Frequency Scaling APIs used within SYS BIOS to alter the core freq/voltage.
• Measure the AM335x Core Voltage. If your system frequency is changing, and continuing to run safely, this likely means its being re-configured in software. I say this because a freq change actually requires a call to the PMIC to alter the voltage settings. Is the am335x core voltage changing after this task execution rate change.  

Hi Michael,

I will keep you updated with the results as I continue to debug.

I measured the AM335x Core Voltage when the SYS/BIOS system tick was running too fast and when it was normal, and there is essentially no difference. It looks correct for OPP120 (600 MHZ). Here are the measurements when it was running too fast:

VDD_MPU:  1.211 V (within expected range of 1.2 V +/- 4%)

VDD_CORE: 1.107 V (within expected range of 1.1 V +/- 4%)

I have captured the relevant AM335x registers before and after the SYS/BIOS system tick rate issue occurs, and I cannot see any difference in configuration of the timers, PLL, etc.

My SYS/BIOS cfg file configures DMTimer2 (Clock.timerID = 0) as the system clock.

In this case, the SYS/BIOS system tick rate became faster. Here are some before and after screenshots of the registers:

Clock Module (CM) registers (0x44E0_0000) - Normal Operation

Clock Module (CM) registers (0x44E0_0000) - SYS/BIOS Issue

DMTimer2 registers (0x4804_0000) - Normal Operation

DMTimer2 registers (0x4804_0000) - SYS/BIOS Issue

DMTimer3 registers (0x4804_2000) - Normal Operation

DMTimer3 registers (0x4804_2000) - SYS/BIOS Issue

DMTimer0 (not used by my application) registers (0x44E0_5000) - Normal Operation

DMTimer0 (not used by my application) registers (0x44E0_5000) - SYS/BIOS Issue

I am guessing that I can't see the registers of DMTimer1_1ms because it might be disabled. I don't think it is used by my application so it shouldn't matter.

DMTimer1_1ms (not used by my application) registers (0x44E3_1000) - Normal Operation

DMTimer1_1ms (not used by my application) registers (0x44E3_1000) - SYS/BIOS Issue

Power Reset Module (PRM) registers (0x44E0_0B00) - Normal Operation

Power Reset Module (PRM) registers (0x44E0_0B00) - SYS/BIOS Issue

These were all of the registers that I thought would be relevant to the issue. Let me know if there are any other registers that I should investigate.

Thanks,

Jon

I am still trying to solve this issue. At the moment, I am trying to change the timer that SYS/BIOS uses for the system clock to see if it has any effect.

For the AM335x, it looks like DMTimer0 and DMTimer1_1ms should be used for SYS/BIOS, and DMTimers 2-7 should be used for the application.

In my program, SYS/BIOS is configured to use DMTimer2 (Timer ID 0) for the SYS/BIOS system clock, since that was the default setting in the example application "ecat_appl" in the AM335x SYS/BIOS Industrial SDK, which my application is based on.

I would like to configure SYS/BIOS to use DMTimer1_1ms, since apparently this timer is designed to be used for accurate 1ms OS ticks. I can't figure out how to tell SYS/BIOS that it should use DMTimer1. It looks like I can only configure SYS/BIOS to use the application timers (DMTimers 2-7). There does not appear to be a "Timer ID" for DMTimer1 on the AM335x:

http://software-dl.ti.com/dsps/dsps_public_sw/sdo_sb/targetcontent/sysbios/6_35_00_20/exports/bios_6_35_00_20/docs/cdoc/ti/sysbios/timers/dmtimer/doc-files/TimerTables.html#AM3359

ti.catalog.arm.cortexa8

AM3359

Is it possible to configure SYS/BIOS to use DMTimer1 on the AM335x? If so, how?

Thanks,

Jon

Only DMTimers2-7 are managed by SYS/BIOS, so you won't be able to use DMTimer1 to generate the OS ticks.

Have you tried any of the other timer peripherals yet? Make sure no other code is using whichever timer peripheral you choose.

Whitney

Jon,

I've looked over your schematics, and for the most part things conform well to the ICE board. I do have a few recommendations/questions:

• Any reason why SW3 & VFB3 of U9 are NC's?
• Why the additional 33ohm R's on the GPMC_ADx and GPIOx bank lines?
• There is no pull-up on B9. This is inconsistent with the ICE platform.
• CAP_VDD_SRAM_CORE and others hvae a parallel 0.1 and 0.01uF capacitance. The recommended cap is 1uF to stabilize these.
• The recommended capacitance on VDD_OSC is 10nF at the output to suppress high-freq switching noise and stabilize this power supply. R11 has 110nF (prob not an issue).
• Spotted a few extra caps on VDDSHVx_x.
• OSC32KIN of your PMIC isn't grounded, but terminated. I'm unsure if this is a problem, but the ICE board grounds this pin.

I set SYSBOOT[5] to 1 to output the system clock on CLKOUT1.

Using an oscilloscope, I looked at the CLKOUT1 signal before and after the issue occured. The clock looks the same before and after: 24MHz, 3.3V.

So the issue should theoretically be further downstream, in the ARM processor. If this is a hardware issue, could it be possible that the PLL circuitry  in the ARM is damaged, causing the CLK_32KHZ clock to run at a different harmonic? Is there some way for me to check the integrity of the PLL circuitry or CLK_32KHZ?

Thanks,

Jon

Michael Risley said:

Jon,

I've looked over your schematics, and for the most part things conform well to the ICE board. I do have a few recommendations/questions:

• Any reason why SW3 & VFB3 of U9 are NC's?
• Why the additional 33ohm R's on the GPMC_ADx and GPIOx bank lines?
• There is no pull-up on B9. This is inconsistent with the ICE platform.
• CAP_VDD_SRAM_CORE and others hvae a parallel 0.1 and 0.01uF capacitance. The recommended cap is 1uF to stabilize these.
• The recommended capacitance on VDD_OSC is 10nF at the output to suppress high-freq switching noise and stabilize this power supply. R11 has 110nF (prob not an issue).
• Spotted a few extra caps on VDDSHVx_x.
• OSC32KIN of your PMIC isn't grounded, but terminated. I'm unsure if this is a problem, but the ICE board grounds this pin.

Hi Michael,

Answers to questions below:

1. The DC-DC switcher portion of the PMIC (SW3, VFB#) is used to boost the output voltage to +5V, which we do not need on our design – we already have a +5V to power the PMC. Threfore we just do not connect that portion of the PMIC
2. 33ohm R's on the GPMC_ADx and GPIOx bank lines are required per TI ICE board design. They are shown on the ICE schematic sheet with the ARM chip.
3. In our design the net B9 (which is the VREFP for the ARM ADC) is driven by the PMIC 1.8V output – at that time we could not locate a good 1.8V reference and therefore use a PMIC 1.8V output as VRef. Found out later that TI does have 1.8V Ref (LM4132 for example)
4. We misread the cap values on the pins CAP_VDD_SRAM_CORE (pin D9), CAP_VDD_SRAM_MPU(pin D11) and CAP_VBB_MPU(pin C10) on the ICE reference design and have since made changes on our schematics to add the 1uF caps
5. We increase capacitance on VDD_OSC to insure stable supply to critical internal  ARM oscillator
6. We increase capacitance on some supplies again to minimize supplies ripples
7. The PMIC pin 20 OSCK32KN IS grounded. The ARM pin A6- OSC1_IN is indeed grounded trough a 100K resistor. We verify that there is no oscillation on the OSC1_OUT pin A4, indicating that the 100K is OK- Will change in next design value to 10K.

These recommendations/questions do not appear to be the root cause of the problem. Any other ideas?

Regards,

Jon

Jonathan Marrs said:

I set SYSBOOT[5] to 1 to output the system clock on CLKOUT1.

Using an oscilloscope, I looked at the CLKOUT1 signal before and after the issue occured. The clock looks the same before and after: 24MHz, 3.3V.

So the issue should theoretically be further downstream, in the ARM processor. If this is a hardware issue, could it be possible that the PLL circuitry  in the ARM is damaged, causing the CLK_32KHZ clock to run at a different harmonic? Is there some way for me to check the integrity of the PLL circuitry or CLK_32KHZ?

Thanks,

Jon

Hey Everyone,

It looks like we have found the cause of the SYS/BIOS System Tick Rate Issue on the AM335x ARM Processor.

Apparently the issue is occurring at the Peripheral PLL (ADPLLLJ).

We have an external crystal oscillator that generates a 24 MHz clock, which is sent to the ARM chip. This 24 MHz clock then goes to several different places, one of which is the Peripheral PLL.

Using the 24 MHz clock, the Peripheral PLL (ADPLLLJ) generates a 192 MHz clock called PER_CLKOUTM2. This clock is then divided to several different clock frequencies to be used by all the different ARM peripherals.

One of these frequencies is CLK_32KHZ, which is a 32 KHz clock that is used for the timer peripherals.

One of the timers (DMTIMER2) is used for the SYS/BIOS System Tick Rate, which is the 1 ms system tick that controls the execution rate of all software tasks, such as our Heartbeat task.

Using an oscilloscope, I looked at the 24MHz clock via CLKOUT1 before and after the issue occurred, and I could see the same 24 MHz frequency, so I decided to look further downstream at the clock after it exits the Peripheral PLL.

I was able to detect this Peripheral PLL issue by looking at PER_CLKOUTM2 (192 MHz) by configuring it to be divided by 6 and sending this 32 MHz signal out on the CLKOUT2 pin of the ARM processor.

Before the SYS/BIOS System Tick Rate issue occurred, I saw the expected clock frequency of 32 MHz. After the issue occurred, I saw 41.3 MHz.  Our software tasks were running ~ 30% faster due to this ~ 30% increase in the PER_CLKOUTM2 (192 MHz) clock frequency.

This leads me to believe that something is going wrong in the Peripheral PLL (ADPLLLJ).

 When I first started troubleshooting this issue, I checked to see if any of the Clock Module (CM) registers or PRCM registers were changing. At that time, I could not see any register configuration changes before and after the issue. I will double check to make sure no registers are changing that could affect the Peripheral PLL.

There may be a problem with the AM335x ARM Processor itself. Perhaps the Peripheral PLL (ADPLLLJ) is damaged or vulnerable in some way. We could have a bad batch of ARM processors, or perhaps the processors were damaged during PCB fabrication or during operation. Hopefully it is not a design flaw of the AM335x ARM Processor. We will continue to investigate this issue and see if we can correlate it with ARM processor batches, PCB assembly batches, etc.

Perhaps TI could look into potential issues with the AM335x Peripheral PLL (ADPLLLJ), since we are not well equipped to troubleshoot this issue on our own, especially if it turns out to be an AM335x design flaw or a bad batch of processors.

Note that this issue can take several days to appear, so it is possible that it may not have been caught in Quality Control tests.

Regards,

Jon

Jon,

This is good troubleshooting; You're definitely narrowing down on the issue! I've looped in some members of the Sitara team, who will have some feedback on your latest post shortly.

It seems the big question of HW or SW, has yet to be answered. Our team suggested running the TI EtherCAT slave example from the IA-SDK on your custom board. Doing this would definitively point the issue to HW or SW:

• Install a new task that runs at Bios rate (usually 1ms) and toggle a GPIO. Measure the GPIO usign a scope. Keep it running for hours/days. Note: TI has been running our examples on ICE boards based on standard Sys/Bios for days and weeks without observing a slow down... If you see the isuse yet again, this would point to the customized HW.

Of course the reverse case could be made as well: Try running your application on an ICE board, and do the same monitoring of the cycle time. 

In parallel, we will go through the clocking portions of your schematics to double-check for any missed items.

Hi Jon,

After reviewing your schematics, I see that M5 (VPP) is connected to the VDD2_SMPS net. This voltage is intended to allow programming of the e-fuses. Holding this pin high during steady state could lead to some unpredictable behavior. The BeagleBone reference platform routes this pin as a NC.

Can you make the VPP pin of the processor a NC, and try to re-produce this issue?

Hi Jon,

As mentioned offline, if physically trying to pull up the VPP line is difficult, you try a work-around in the firmware.

There is an undocumented register that may be able to reduce the effects of having this voltage tied to VPP.

• Set bit 11 of 0x44e10100

The default value of this register will be 0x3281. Set the 11th, such that it is 0x3A81. Doing this may disable any potential efuse programming.

Keep us updated.

Michael Risley said:

Hi Jon,

As mentioned offline, if physically trying to pull up the VPP line is difficult, you try a work-around in the firmware.

There is an undocumented register that may be able to reduce the effects of having this voltage tied to VPP.

• Set bit 11 of 0x44e10100

The default value of this register will be 0x3281. Set the 11th, such that it is 0x3A81. Doing this may disable any potential efuse programming.

Keep us updated.

Hi Michael,

I just tested some boards with some firmware that set bit 11 of register 0x44E10100, so that the value changed from 0x3281 to 0x3A81. However, the Peripheral PLL frequency still changed spontaneously.

I saw a post on the TI E2E forum in which someone had a similar issue with the Peripheral PLL frequency changing, and their board had VPP connected to a power rail instead of floating / NC, just like our board:

http://e2e.ti.com/support/arm/sitara_arm/f/791/t/268225.aspx

We are going to try drilling a hole to cut the trace that connects VPP to VDD2_SMPS / VDD_CORE (1.1V). Hopefully that will solve the problem, even though changing the value of register 0x44E10100 did not work.

Thanks,

Jon

Hi Jon,

We had the same issue (see the link) but drilling a hole wasn’t possible, so we will have to wait until the next batch to cut the connection.

Meanwhile we implemented 3 different software patches:

- disable the effuse (set bit 11 of 0x44e10100) like TI proposed.

- reinitiate the peripheral PLL when we detect the freq. haze changed (= if we lose serial communication, baud rate is wrong).

- If all doesn’t' help we issue software reset to reboot.

I haven't had any issue any more in the lab.

regards,

Steve

Michael Risley said:

Jon,

This is good troubleshooting; You're definitely narrowing down on the issue! I've looped in some members of the Sitara team, who will have some feedback on your latest post shortly.

It seems the big question of HW or SW, has yet to be answered. Our team suggested running the TI EtherCAT slave example from the IA-SDK on your custom board. Doing this would definitively point the issue to HW or SW:

• Install a new task that runs at Bios rate (usually 1ms) and toggle a GPIO. Measure the GPIO usign a scope. Keep it running for hours/days. Note: TI has been running our examples on ICE boards based on standard Sys/Bios for days and weeks without observing a slow down... If you see the isuse yet again, this would point to the customized HW.

Of course the reverse case could be made as well: Try running your application on an ICE board, and do the same monitoring of the cycle time. 

In parallel, we will go through the clocking portions of your schematics to double-check for any missed items.

Hi Michael,

Using several of our boards, I ran two of the sample applications from the AM335x SYS/BIOS Industrial SDK (v1.0.0.7):  "i2c_led" and "ecat_appl". Our application is based on "ecat_appl" so it is the most relevant sample application.

The only modification I made in each sample application was to blink an LED at 1 Hz to monitor whether the Peripheral PLL frequency issue was happening. Neither of the two sample applications failed after 5+ days of testing.

Before we conclude that the firmware is definitely the problem, you should know some facts about the boards when running our own customized application:

• We see different but repeatable rates of failure in our boards running our own application: Some boards will routinely fail within 2 days, others will fail after about 4 days, while other boards have never failed.
• These boards that fail at different rates all have the same hardware design, and are all running the exact same firmware.
• Our own customized application is more complex than "ecat_appl" since we make use of additional AM335x peripherals, several external ICs (DACs, ADCs, DO drivers, etc.) and have additional tasks running. We therefore exercise more of the hardware on our board and within the AM335x itself while running our own application.

Based on these facts, I think the issue is not caused by the firmware alone, but a combination of the firmware and the hardware.

If hardware was not a factor, we would not see such different but repeatable failure rates in our boards running the exact same firmware. The only potential differences in hardware that I can think of would be slight variations in manufacturing quality or slight variations/damage to electronic components. It seems that some portion of our design is marginally stable, and small variations in hardware integrity is causing the Peripheral PLL issue to appear to a greater extent on some boards and not others. Since the issue can take several days to appear, it further suggests that something in the hardware is marginally stable.

The fact that we see the issue while running our own application and not while running the sample applications may be due to the hardware being exercised/stressed more while running our own application, perhaps introducing more noise/glitches/EMI into the system.

It may turn out that having VPP connected to VDD2_SMPS / VDD_CORE (1.1V) makes our boards more susceptible to whatever slight variations exist in our hardware, but that remains to be seen.

I will continue to keep you updated with our debugging progress.

Thanks,

Jon 

Jonathan Marrs said:

Hi Michael,

I just tested some boards with some firmware that set bit 11 of register 0x44E10100, so that the value changed from 0x3281 to 0x3A81. However, the Peripheral PLL frequency still changed spontaneously.

I saw a post on the TI E2E forum in which someone had a similar issue with the Peripheral PLL frequency changing, and their board had VPP connected to a power rail instead of floating / NC, just like our board:

http://e2e.ti.com/support/arm/sitara_arm/f/791/t/268225.aspx

We are going to try drilling a hole to cut the trace that connects VPP to VDD2_SMPS / VDD_CORE (1.1V). Hopefully that will solve the problem, even though changing the value of register 0x44E10100 did not work.

Thanks,

Jon

Hi Michael,

We drilled a hole to cut the trace for the VPP pin on a few of our boards, but we still experience the Peripheral PLL frequency issue. So I think we can assume that the VPP pin for efuse programming is not the root cause of the problem.

After reading the thread referenced above (http://e2e.ti.com/support/arm/sitara_arm/f/791/t/268225.aspx), it seems likely that VDDA1P8V_USB0 of the AM335x could be the issue. VDDA1P8V_USB0 (1.8V) supplies the Peripheral PLL of the AM335x, and comes directly from VAUX1 of TPS65910A (PMIC). Since we copied the ICE schematic, our board has nothing else connected to this power rail... VAUX1 goes straight to VDDA1P8V_USB0, and nowhere else. We have a 0.01uF capacitor and a 2.2uF capacitor on this line, but no inductors or ferrite beads, just like the ICE.

Following the thread referenced above, we tried shorting VDDA1P8V_USB0 to ground and were able to create the Peripheral PLL frequency issue. Shorting this line to ground does not always produce the issue, but after several iterations of perturbing the line, we are able to cause the Peripheral PLL frequency to change, and it stays at this new frequency after VDDA1P8V_USB0 returns to the normal 1.8V level. The frequency that the Peripheral PLL changes to seems to be random, but we have noticed that most often it speeds up by ~25%. Using this method, we were able to produce the Peripheral PLL frequency issue on our own board as well as on the ICE board.

Due to these experimental results, we suspect that glitches may be occuring sporadically on  VDDA1P8V_USB0. We are now monitoring  VDDA1P8V_USB0 with an oscilloscope to detect such glicthes. If we detect glitches or spikes, we will have to find out why they are happening.

We are also considering adding an inductor or ferrite bead to VDDA1P8V_USB0 to suppress glitches/spikes. In the thread referenced above, James said that TI uses a ferrite bead on sensitive PLL rails to shield them from noise. What ferrite bead or inductor does TI recommend for VDDA1P8V_USB0 of the AM335x, in addition to the 0.01uF and 2.2uF capacitors?

Also, it would be useful to know if TI has any insight about what could cause a glitch/spike on VDDA1P8V_USB0. I think it could be caused by either the TPS65910A (PMIC) or the AM335x itself, since those are the only two ICs connected to VDDA1P8V_USB0. Any troubleshooting tips would be appreciated.

Thanks,

Jon 

Jonathan,

I understand that this issue is getting resolved via direct communication, so we are marking this thread as Answered/Closed.

Any progress?

 We also tested this path and inserted small spikes on this power line with a generator but the PLL stayed stable.

Only when we dropped the voltage to less than 1.2V and then inserted the spikes we could stop the PLL, and only stop the PLL.

This is not the same as the +25% we got in the field so we abounded this source.