SS1BTPM gains thecomplete CPU resources after a while

Hi,
Your query has been assigned to our stack expert.
BR,
Eyal

Just to add some more detail, we are using Standard SPP profil to connect to our sensor unit.
The sensor unit has an integrated BT121 module from BlueGIGA for BT communication.

We still phase thr problem that the BT datatransfer with 50Hz data rate and 60Bytes of data needs nearly the CPU power for the SS1BTPM Stack.
We removed all our code around the Stack as far as possilbe.
Do you have any suggestions or ideas what could cause that effect.

The amount of data and the data rate is to low, to cause such a problem.

Regards Bernd

Hi Bernd,

How did you ascertain that SS1BTPM daemon is chewing all the CPU? Did, you check with top or htop?
For isolating the issue, you may consider the following -
a) Check with, vmstat is the CPU is spent most in Kernel or user space, when you see the problem.
b) Try running target GDB server and do a <Ctrl-C> to pause the process in GDB and check the stack (with 'where'), where the process is hogging when you see the problem..

If, there is a sample application, we can use to reproduce the issue at our end please send it across.

Thanks

Hello Hari,

• we checked the load with htop
• the ratio user space/kernel space is about 50/50. Both values rise concurrently

In case of overload, there seems to be an internal buffer on the BT module filling up with data, which cannot be sent fast enough to the application.
We see this behaviour:

• Program starts: load is normal (buffer fills)
  If we stop the sender, our app immediately stops receiving data. This is OK.
  
  
• Buffer filled to a certain level: load is very high, both of SS1BTPM and our Application (linked to Bluetopia).
  User space and Kernel space loads are high.
  
  
• If we stop the BT sender, our App still receives data, for seconds to minutes.
  We think the internal buffer of the BT Module is emptied during this phase.
  SS1BTPM has low load then, our App still high load.
  
  
• If the buffer is emptied, all loads are normal.
  If we start the BT-sender, the cycle begins again.

Environment:

• Custom Board with AM335x
• BT-Module:  wl18xx, connected via Uart. HW: 183x or 180x, PG 2.2 (ROM 0x11).
• Firmware BT-Module: TIInit_11.8.32.bts, Ver. TI_P2.136
• Bluetopia Stack Version: 4.0.3.0.1.0
• Linux SDK: ti-sdk-am335x-evm-07.00.00.00

Configuration of the Serial Interface to the BT-Module in Devicetree

        tibt {
                compatible = "tibt";
                nshutdown_gpio = <63>; /* Bank1, pin31 */
                dev_name = "/dev/ttyO2";
                flow_cntrl = <1>;
                baud_rate = <3000000>; 
        };

Hello Hari,

here ist the callstack of SS1BTPM in case of high load:

(gdb) thread apply all bt
[New Thread 2117]
[New Thread 2118]
[New Thread 2119]
[New Thread 2189]
[New Thread 2196]
[New Thread 2197]

Thread 7 (Thread 2197):
#0  0x4862b404 in write () from /opt/mecha1-psp-V7.1.0.7-SNAPSHOT/custom-filesystem/lib/libpthread.so.0
#1  0x001fcd68 in ?? ()
#2  0x001fcd68 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 6 (Thread 2196):
#0  0x4862d494 in __libc_do_syscall () from /opt/mecha1-psp-V7.1.0.7-SNAPSHOT/custom-filesystem/lib/libpthread.so.0
#1  0x4862a7ea in do_futex_wait () from /opt/mecha1-psp-V7.1.0.7-SNAPSHOT/custom-filesystem/lib/libpthread.so.0
#2  0x4862a866 in sem_wait@@GLIBC_2.4 () from /opt/mecha1-psp-V7.1.0.7-SNAPSHOT/custom-filesystem/lib/libpthread.so.0
#3  0x002001d4 in ?? ()
#4  0x002001d4 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 5 (Thread 2189):
#0  0x4862d494 in __libc_do_syscall () from /opt/mecha1-psp-V7.1.0.7-SNAPSHOT/custom-filesystem/lib/libpthread.so.0
#1  0x4862a7ea in do_futex_wait () from /opt/mecha1-psp-V7.1.0.7-SNAPSHOT/custom-filesystem/lib/libpthread.so.0
#2  0x4862a866 in sem_wait@@GLIBC_2.4 () from /opt/mecha1-psp-V7.1.0.7-SNAPSHOT/custom-filesystem/lib/libpthread.so.0
#3  0x002001d4 in ?? ()
Cannot access memory at address 0x32

It appears the thread is spending more time in system call.. Not sure why this would lead more CPU load.. Let me try to reproduce the issue on my setup..

Thanks
Hari

To, reproduce the issue, i used the below environment.

System SW: TI-SDK - 02.00.02, released in April 2016, and Blutopia: 4.0.3.0.1.0, released Jan 2016.

Test setup:

- LinuxSSPM, running on AM335 EVM in loopback mode.. 
- Teraterm serial console on PC using BT serial port connected to SPPM port
- Text file (480KB) transfer from tera term, repeated several times

And, we observed the CPU load with htop.. When the transfer is taking place, SS1BTPM load shoots to 20 to 25% and LinuxSPPM between 10 to 15 %. And the CPU load falls down, once the transfer is done.

Can, you also try with TI-SDK 02.00.02 and see, if the problem goes away?

Thanks

Hi Hari,

to switch our custom board to TI-SDK 02.00.02 does not work we already tried some time ago and failed to port all the batch to the new SDK.
We already use Bluetopia 4.0.3.0.1.0.

But proceed with our testing and found the reason for our problem.
It looks like the problem is caused by having the following setting to enable i2c-0 in the device tree.

i2c0: i2c@44e0b000 {
status = "okay";
};

If we remove this our problem with BT is gone.
But then shutdown is not working as the shutdown seems to need i2c0.

To you have any idea what is the relation between isC0 and the BT WMOD1835 connected to UART0?
And what is wrong with enabling the i2C0 zero in the device tree like we did it?

Thanks for your support.

Hi Bernd,

Looking at the AM335x EVM schematics, it seems I2C0 is wired to PMIC. So, it is _not_ a good idea to disable I2C0.
I wonder, if your problems are due to device power management. Do, you have any power policy/governor enabled? I will also check on my setup on monday..

Thanks

Hi Hari,

we now confirmed that the issue is related to the configuration of I2C0.
If we do not configure i2C0 device than the BT issue is gone.

As soon as we enable i2C0 device in the micro we see the BT issue after some time of operation.

But if we do not configure i2C0 we have a crash when starting the linux.
As there is no device defined for I2C0 in the device tree.
Looks like the kernel requires a definition here.

[    1.717946] ------------[ cut here ]------------

[    1.722857] WARNING: CPU: 0 PID: 1 at arch/arm/mach-omap2/omap_device.c:888 omap_device_get_by_hwmod_name+0x94/0xb4()

[    1.734002] omap_device_get_by_hwmod_name: no omap_device for i2c1

[    1.740490] Modules linked in:

[    1.743705] CPU: 0 PID: 1 Comm: swapper Not tainted 3.12.10-ti2013.12.01 #5

[    1.751020] Backtrace:

[    1.753598] [<c0017880>] (dump_backtrace+0x0/0x10c) from [<c0017a1c>] (show_stack+0x18/0x1c)

[    1.762456]  r6:c002f564 r5:00000009 r4:ed083e28 r3:00000000

[    1.768421] [<c0017a04>] (show_stack+0x0/0x1c) from [<c05d0a08>] (dump_stack+0x20/0x28)

[    1.776840] [<c05d09e8>] (dump_stack+0x0/0x28) from [<c0046d08>] (warn_slowpath_common+0x70/0x90)

[    1.786164] [<c0046c98>] (warn_slowpath_common+0x0/0x90) from [<c0046dcc>] (warn_slowpath_fmt+0x38/0x40)

[    1.796134]  r8:c07db3fc r7:c07d8410 r6:c11c2744 r5:c097f508 r4:c0717d64

[    1.803198] [<c0046d94>] (warn_slowpath_fmt+0x0/0x40) from [<c002f564>] (omap_device_get_by_hwmod_name+0x94/0xb4)

[    1.813972]  r3:c05dbe84 r2:c071434c

[    1.817748] [<c002f4d0>] (omap_device_get_by_hwmod_name+0x0/0xb4) from [<c07e8280>] (am33xx_pm_init+0x204/0x45c)

[    1.828432]  r4:00000000

[    1.831105] [<c07e807c>] (am33xx_pm_init+0x0/0x45c) from [<c07e0c00>] (am33xx_init_late+0x18/0x20)

[    1.840507]  r8:c07db3fc r7:c07d8410 r6:c097e400 r5:00000007 r4:c08ee14c

[    1.847573] [<c07e0be8>] (am33xx_init_late+0x0/0x20) from [<c07db420>] (init_machine_late+0x24/0x30)

[    1.857173] [<c07db3fc>] (init_machine_late+0x0/0x30) from [<c00089ec>] (do_one_initcall+0xf4/0x154)

[    1.866766] [<c00088f8>] (do_one_initcall+0x0/0x154) from [<c07d8b94>] (kernel_init_freeable+0xf0/0x1bc)

[    1.876726] [<c07d8aa4>] (kernel_init_freeable+0x0/0x1bc) from [<c05cb654>] (kernel_init+0x10/0xec)

[    1.886237] [<c05cb644>] (kernel_init+0x0/0xec) from [<c00146f8>] (ret_from_fork+0x14/0x3c)

[    1.895005]  r4:00000000 r3:00000000

[    1.898795] ---[ end trace 746e089ac4a8ece1 ]---[ 

In additon Standby mode is not working without having i2C0 enabled.

Our board has the PMIC connected to I2C1 instead of I2C0, might be that ther is any issue as linux is assuming the OMIC in I2C0.
See the post here.

Hi Bernd,

I spoke with couple of AM335x folks here. The recommendation is to use I2C0 for PMIC control. This is because I2C0 is in wakeup domain and so can be active in low power state as well for throttling PMIC.

However, we are not sure how I2C0 is effecting the HCI performance. My guess feeling is that device is trying to enter low power states in CPUIdle (between HCI transfers) and the wakeup latencies are high resulting in poor throughput.. 

Can, you build kernel with CPI Idle disabled (disable CONFIG_ARM_CPUIDLE in the menuconfig) and give it a try? Also, ensure that the governer is set to 'performance', before you start the test.

Thanks

Additional Information -

You, can also disable CPU Idle states from console as below :

echo 1 > /sys/devices/system/cpu/cpu0/cpuidle/state0/disable

echo 1 > /sys/devices/system/cpu/cpu0/cpuidle/state1/disable

Thanks