Linux/TMDX654GPEVM: Kernel panic - CryptoDev/OpenSSL

Prateek Khatri

Part Number: TMDX654GPEVM

Tool/software: Linux

Hello,

I am facing an issue where the board is resetting after a few minutes and I am also seeing a prompt for Kernel Panic.

root@am65xx-evm:~# journalctl -f
-- Logs begin at Thu 2019-06-13 19:58:51 UTC. --
Jun 18 17:44:56 am65xx-evm systemd-logind[403]: New session c1 of user root.
Jun 18 17:44:56 am65xx-evm systemd[1]: Started Session c1 of user root.
Jun 18 17:44:56 am65xx-evm systemd[872]: Reached target Timers.
Jun 18 17:44:56 am65xx-evm systemd[872]: Reached target Sockets.
Jun 18 17:44:56 am65xx-evm systemd[872]: Reached target Paths.
Jun 18 17:44:56 am65xx-evm systemd[872]: Reached target Basic System.
Jun 18 17:44:56 am65xx-evm systemd[872]: Reached target Default.
Jun 18 17:44:56 am65xx-evm systemd[872]: Startup finished in 95ms.
Jun 18 17:44:56 am65xx-evm systemd[1]: Started User Manager for UID 0.
Jun 18 17:44:56 am65xx-evm login[877]: ROOT LOGIN  on '/dev/ttyS2'
Jun 18 17:45:06 am65xx-evm systemd-networkd[232]: eth6: Configured
Jun 18 17:45:06 am65xx-evm systemd-timesyncd[185]: Network configuration changed, trying to establish .
Jun 18 17:50:01 am65xx-evm systemd-timesyncd[185]: Synchronized to time server 216.239.35.8:123 (time3.
[   97.370971] Unable to handle kernel paging request at virtual address 1473cee744
[   97.378390] Mem abort info:
[   97.381185]   Exception class = DABT (current EL), IL = 32 bits
[   97.387097]   SET = 0, FnV = 0
[   97.390148]   EA = 0, S1PTW = 0
[   97.393285] Data abort info:
[   97.396162]   ISV = 0, ISS = 0x00000046
[   97.399994]   CM = 0, WnR = 1
[   97.402962] user pgtable: 64k pages, 48-bit VAs, pgd = ffff800841519a00
[   97.409588] [0000001473cee744] *pgd=00000008c5070003, *pud=00000008c5070003, *pmd=0000000000000000
[   97.418544] Internal error: Oops: 96000046 [#1] PREEMPT SMP
[   97.424104] Modules linked in: xhci_plat_hcd rpmsg_proto xhci_hcd sha512_arm64 usbcore virtio_rpmsg6
[   97.472438] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O    4.14.79-ge669d52447 #1
[   97.480682] Hardware name: Texas Instruments AM654 Base Board (DT)
[   97.486847] task: ffff000008b3b280 task.stack: ffff000008b00000
[   97.492780] PC is at emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
[   97.498875] LR is at emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
[   97.504954] pc : [<ffff000001012b00>] lr : [<ffff000001012b00>] pstate: 60000145
[   97.512330] sp : ffff00000800fdb0
[   97.515634] x29: ffff00000800fdb0 x28: ffff0000087e0110 
[   97.520938] x27: ffff80084171b9c0 x26: 0000001473cee734 
[   97.526239] x25: ffff800841001810 x24: ffff80084171baf0 
[   97.531541] x23: 0000000000000040 x22: 0000000040000255 
[   97.536842] x21: ffff80084171b900 x20: ffff80084171b000 
[   97.542143] x19: 0000000000000020 x18: 0000000000000000 
[   97.547444] x17: 0000000000000000 x16: 0000000000000000 
[   97.552747] x15: 000000000a640b63 x14: 0000000000000000 
[   97.558047] x13: 0000000000000000 x12: ffff800843a7f580 
[   97.563349] x11: 0000000000000001 x10: 00000000ffffffe9 
[   97.568649] x9 : 0000000000000000 x8 : 0000000000000000 
[   97.573950] x7 : 0000010000000000 x6 : ffff80087ff30000 
[   97.579251] x5 : 0000000000000000 x4 : ffff800843a7f5b8 
[   97.584554] x3 : ffff800843a7f590 x2 : 0000000000000000 
[   97.589855] x1 : 0000000000000040 x0 : 0000000000000000 
[   97.595158] Process swapper/0 (pid: 0, stack limit = 0xffff000008b00000)
[   97.601843] Call trace:
[   97.604282] Exception stack(0xffff00000800fc70 to 0xffff00000800fdb0)
[   97.610709] fc60:                                   0000000000000000 0000000000000040
[   97.618522] fc80: 0000000000000000 ffff800843a7f590 ffff800843a7f5b8 0000000000000000
[   97.626335] fca0: ffff80087ff30000 0000010000000000 0000000000000000 0000000000000000
[   97.634149] fcc0: 00000000ffffffe9 0000000000000001 ffff800843a7f580 0000000000000000
[   97.641962] fce0: 0000000000000000 000000000a640b63 0000000000000000 0000000000000000
[   97.649775] fd00: 0000000000000000 0000000000000020 ffff80084171b000 ffff80084171b900
[   97.657589] fd20: 0000000040000255 0000000000000040 ffff80084171baf0 ffff800841001810
[   97.665402] fd40: 0000001473cee734 ffff80084171b9c0 ffff0000087e0110 ffff00000800fdb0
[   97.673216] fd60: ffff000001012b00 ffff00000800fdb0 ffff000001012b00 0000000060000145
[   97.681029] fd80: ffff80084171b900 0000000040000255 0000ffffffffffff ffff000008449a60
[   97.688842] fda0: ffff00000800fdb0 ffff000001012b00
[   97.693724] [<ffff000001012b00>] emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
[   97.700854] [<ffff00000869df54>] net_rx_action+0xf4/0x2b0
[   97.706246] [<ffff0000080811ec>] __do_softirq+0x12c/0x228
[   97.711634] [<ffff0000080acd10>] irq_exit+0xc8/0xf8
[   97.716504] [<ffff0000080fd8c0>] __handle_domain_irq+0x60/0xb0
[   97.722323] [<ffff000008080fc4>] gic_handle_irq+0x7c/0x178
[   97.727795] Exception stack(0xffff000008b0fdf0 to 0xffff000008b0ff30)
[   97.734221] fde0:                                   0000000000000000 00008008773c0000
[   97.742034] fe00: 0000000000000000 ffff000008af3930 ffff000008af3908 ffff000008b0ff20
[   97.749847] fe20: 00008008773c0000 0000000000000000 0000000000000002 ffff000008b0feb0
[   97.757660] fe40: 0000000000000920 0000000000000000 0000000000000001 0000000000000000
[   97.765474] fe60: 000000000000002b 0000000000000000 ffff0000087e18d8 0000ffffa4f5d2d0
[   97.773287] fe80: 0000000000000014 ffff000008b33ac4 ffff000008b33a30 ffff000008af0018
[   97.781101] fea0: ffff000008b3b280 ffff80087ffff480 ffff000008ae0028 0000000000000000
[   97.788914] fec0: 00000000ffef3ce0 0000000000000400 0000000080a90018 ffff000008b0ff30
[   97.796728] fee0: ffff000008084e2c ffff000008b0ff30 ffff000008084e30 0000000060000145
[   97.804542] ff00: 0000000000000000 00000000ffef3ce0 ffffffffffffffff ffff000008123f84
[   97.812354] ff20: ffff000008b0ff30 ffff000008084e30
[   97.817220] [<ffff000008082a30>] el1_irq+0xb0/0x124
[   97.822088] [<ffff000008084e30>] arch_cpu_idle+0x10/0x18
[   97.827389] [<ffff0000080e5ee0>] do_idle+0xd8/0x118
[   97.832255] [<ffff0000080e60b4>] cpu_startup_entry+0x24/0x28
[   97.837905] [<ffff00000879621c>] rest_init+0xcc/0xd8
[   97.842863] [<ffff000008a90b50>] start_kernel+0x378/0x38c
[   97.848253] Code: f94037a5 f9400300 aa0503e1 95d0e719 (f9000b54) 
[   97.854339] ---[ end trace a0caa21cf2697537 ]---
[   97.858946] Kernel panic - not syncing: Fatal exception in interrupt
[   97.865286] SMP: stopping secondary CPUs
[   97.869201] Kernel Offset: disabled
[   97.872680] CPU features: 0x080200c
[   97.876157] Memory Limit: none
[   97.879205] ---[ end Kernel panic - not syncing: Fatal exception in interrupt

How do I go about figuring out what is causing this Kernel Panic? What one of our vendor is mentioning is that it is the Cryptodev module provided with the SDK.

Please advise.

over 6 years ago

0 Prateek Khatri over 6 years ago

Intellectual 400 points

Seems like using the Crytpo dev causes the Kernel Panic. If I don't invoke Cryptodev by using any of the test commands given here:

http://software-dl.ti.com/processor-sdk-linux/esd/docs/latest/linux/Foundational_Components_Kernel_Drivers.html#crypto

Then the kernel doesn't crash.

0 Prateek Khatri over 6 years ago in reply to Prateek Khatri

Intellectual 400 points

I created a fresh SD card with one of the provided images with processor SDK. The ti-rootfs image without the docker (option 2 in create-sd-card.sh script).

I am still seeing the Kernel Panic message after some time regardless of executing cryptodev related commands.

How do I debug this?

0 Prateek Khatri over 6 years ago in reply to Prateek Khatri

Intellectual 400 points

I re-flashed the provided rootfs image and left the board overnight. It did not go into Kernel Panic or rebooted.

However, I ran the following commands on the provided rootfs image:

# modprobe cryptodev
# openssl req -nodes -newkey rsa:2048 -keyout example.key -out example.crt -x509 -batch
# openssl pkcs12 -export -out certificate.pfx -inkey example.key -in example.crt -password "pass:"

After 5 minutes or so the board rebooted.

Seems to be definitely related to Cryptodev drivers.

0 Prateek Khatri over 6 years ago in reply to Prateek Khatri

Intellectual 400 points

There seems to be a bug in UDMA driver sa2ul. I removed that kernel module and could not reproduce the Kernel Panic anymore.

This would need to be fixed in the processor SDK.

Could anyone please confirm this?

0 Bin Liu over 6 years ago in reply to Prateek Khatri

TI__Guru**** 168411 points

Hi Prateek,

Which SDK version do you use?

I tested SDK v5.1.0.11 on the AM65x EVM and ran the two openssl commands you provided above with cryptodev module loaded (it is loaded automatically during linux boots), but I don't see such kernel panic.

And I am not convinced the kernel panic is related to cryptodev.

- the two openssl commands run very quick, they finish within a second, how could it cause kernel panic in a few minutes later;

- the call trace dump in kernel panic you provided above shows there is kernel memory paging failure happened in emac driver, which shouldn't be related to cryptodev either.

0 Prateek Khatri over 6 years ago in reply to Bin Liu

Intellectual 400 points

I am using the config to initialize the SDK at:

processor-sdk-05.03.00.07-config.txt

When I was building the SDK that was the highest version available.

Also, on the website the version availble is the same:

How do I update to that version of SDK?

If it's happening with EMACS using the default SD card generated by the SDK, how can I try to solve this?

0 Bin Liu over 6 years ago in reply to Prateek Khatri

TI__Guru**** 168411 points

Here is the link to the sdk v5.1.0.11:

0 Prateek Khatri over 6 years ago in reply to Bin Liu

Intellectual 400 points

Thanks,

Will update the SDK and report back.

0 Prateek Khatri over 6 years ago in reply to Prateek Khatri

Intellectual 400 points

I downloaded the new SDK and started a build however, all throughout the weekend it was stuck at this state:

I don't see the pid shown here utilizing any CPU. How do I fix this now :/ ?

(I restarted the computer and ran bitbake again, that's why PIDs are different).

Thanks

0 Prateek Khatri over 6 years ago in reply to Prateek Khatri

Intellectual 400 points

Found the answer to the build being stuck.

0 Prateek Khatri over 6 years ago in reply to Prateek Khatri

Intellectual 400 points

Hi Bin,

You are absolutely right, it is the EMAC driver. As soon as I plug in the Ethernet to eth4 I see Kernel panic.

Will try on the new version of SDK soon.

Thanks,
Prateek

0 Prateek Khatri over 6 years ago in reply to Prateek Khatri

Intellectual 400 points

I just build the new SDK with arago-core-tisdk-image and I can't see any eth* listed when I do ifconfig.

How do I build the Rootfs with the Eth drivers included?

Thanks,

Prateek

0 Bin Liu over 6 years ago in reply to Prateek Khatri

TI__Guru**** 168411 points

Hi Prateek,

Thanks for confirming the issue is not related to crypto, that matches to the kernel dump showing the crash point is at emac_napi_rx_poll() called in the prueth driver. Ethernet is out of my support scope, but I just heard that the pruss ethernet driver is not stable in the current Processor SDK releases which could cause kernel panic. The issue is fixed in the coming SDK release. At the moment please only use eth0 if you don't need additional ethernet ports or pruss ethernet. I will let our ethernet expert to comment if he has additional information.

0 Nick Saulnier over 6 years ago in reply to Bin Liu

TI__Guru** 101330 points

Hello Prateek,

I have observed PRU Ethernet to cause kernel panics on Linux Processor SDK 5.3 in the following situations:
* bidirectional iperf tests
* multiple PRU Ethernet ports plugged in, where at least one PRU Ethernet port is on the main board (i.e., I did not observe kernel panics with multiple PRU Ethernet ports plugged in on the IDK daughter card).
* occasionally during an scp file transfer

There may be other kernel panics that I have not personally observed. We expect the PRU Ethernet kernel panics to be fixed in the July SDK release. Until then, we advise sticking to CPSW port eth0 if you only need one Ethernet port.

Regards,

Nick

0 Prateek Khatri over 6 years ago in reply to Nick Saulnier

Intellectual 400 points

Hi Bin,

I did not update the boot partition of the SD card for the new SDK causing the ETh initialization to fail. After replacing the boot files it works fine.

We will stick to eth0 for now, what is the release cycle for the processor SDK?

0 Bin Liu over 6 years ago in reply to Prateek Khatri

TI__Guru**** 168411 points

Prateek,

The Processor SDKs are released quarterly. The next release will be available in July.

Processors

Processors forum

Linux/TMDX654GPEVM: Kernel panic - CryptoDev/OpenSSL