AM3352: crash when using HW crypto accelerator API

Colombo Carlo

Mastermind 21530 points

Part Number: AM3352

Hi everybody ,

in latest SDK there are crypto modules using HW acceleration

[79446.411865] omap-aes 53500000.aes: OMAP AES hw accel rev: 3.2

[79446.438446] omap-aes 53500000.aes: will run requests pump with realtime priority

[79462.248420] omap-sham 53100000.sham: hw accel on OMAP rev 4.3

now after creating a IPsec tunnerl , transferring data kernel goes in Panic using these crypto API .

if we do not load these modules everything is fine .

if we use HW acceleration with Openssl everything is fine .

please what is wrong ? what shoudl we do ?

Carlo

over 3 years ago

0 Hong Guan64 over 3 years ago

TI__Guru 54345 points

Hi Carlo,

Colombo Carlo said:
in latest SDK there are crypto modules using HW acceleration

1. What is Linux SDK version?

Colombo Carlo said:
now after creating a IPsec tunnerl , transferring data kernel goes in Panic using these crypto API .

2. Can we upload the full log file showing error/panic message in attachment?

Colombo Carlo said:
if we do not load these modules everything is fine .

3. Can we upload the full working log file for correlation analysis in attachment?

Best,

-Hong

0 Filippo Carlini over 3 years ago in reply to Hong Guan64

Prodigy 230 points

Hello Hong,

at the moment we are using Kernel 4.14.221 and the logs have been taken with that kernel version.

I attached the kernel panic log file to this post.

The kernel modules that cause the crash are:

crypto_engine.ko
omap-crypto.ko
omap-aes-driver.ko
omap-des.ko
omap-sham.ko

If I don't' load these modules everything works.

Tee issue seems to be exactly the same as reported years ago, without a solution, into this post:

https://e2e.ti.com/support/processors/f/791/t/349219?AM335x-Hardware-Crypto-Engine

So the IPSEC tunnel performance decreases instead of improving when using hw acceleration and immediately after some data transfers kernel crashes .

Thank you and best regards

Filippo

Log kernel panic.txt

Wed Feb 24 13:50:01 2021 daemon.info : 08[IKE] scheduling reauthentication in 28234s
Wed Feb 24 13:50:01 2021 daemon.info : 08[IKE] maximum IKE_SA lifetime 28774s
Wed Feb 24 13:50:01 2021 daemon.info : 08[ENC] generating ID_PROT response 0 [ ID HASH ]
Wed Feb 24 13:50:01 2021 daemon.info : 08[NET] sending packet: from 192.168.1.1[4500] to 192.168.1.156[55282] (92 bytes)
Wed Feb 24 13:50:01 2021 daemon.info : 10[NET] received packet: from 192.168.1.156[55282] to 192.168.1.1[4500] (188 bytes)
Wed Feb 24 13:50:01 2021 daemon.info : 10[ENC] parsed QUICK_MODE request 1402759692 [ HASH SA No ID ID ]
Wed Feb 24 13:50:01 2021 daemon.info : 10[CFG] selected proposal: ESP:AES_CBC_128/HMAC_SHA2_256_128/NO_EXT_SEQ
Wed Feb 24 13:50:01 2021 daemon.info : 10[ENC] generating QUICK_MODE response 1402759692 [ HASH SA No ID ID ]
Wed Feb 24 13:50:01 2021 daemon.info : 10[NET] sending packet: from 192.168.1.1[4500] to 192.168.1.156[55282] (188 bytes)
Wed Feb 24 13:50:01 2021 daemon.info : 11[NET] received packet: from 192.168.1.156[55282] to 192.168.1.1[4500] (76 bytes)
Wed Feb 24 13:50:01 2021 daemon.info : 11[ENC] parsed QUICK_MODE request 1402759692 [ HASH ]


[79528.542796] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[79528.553859] pgd = cd7ac000
[79528.557349] [00000000] *pgd=8d06b831, *pte=00000000, *ppte=00000000
[79528.563943] Internal error: Oops: 17 [#1] SMP ARM
[79528.568708] Modules linked in: omap_sham omap_des omap_aes_driver omap_crypto crypto_engine pppoe ppp_async option l2tp_ppp usb_wwan pppox ppp_generic nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_state xt_policy xt_nat xt_multiport xt_mark xt_mac xt_limit xt_esp xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD usbserial slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ipt_ah ip_tables cdc_acm cryptodev nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 l2tp_netlink l2tp_core udp_tunnel ip6_udp_tunnel xfrm6_mode_tunnel
[79528.641034]  xfrm6_mode_transport xfrm6_mode_beet ipcomp6 xfrm6_tunnel esp6 ah6 xfrm4_tunnel xfrm4_mode_tunnel xfrm4_mode_transport xfrm4_mode_beet ipcomp esp4 ah4 tunnel6 tunnel4 tun xfrm_user xfrm_ipcomp algif_skcipher algif_hash af_alg echainiv authenc usb_f_acm u_serial g_serial libcomposite
[79528.667762] CPU: 0 PID: 2259 Comm: apkrouter_owrt_ Not tainted 4.14.221 #0
[79528.674697] Hardware name: Generic AM33XX (Flattened Device Tree)
[79528.680850] task: cda2cc80 task.stack: cd06c000
[79528.685458] PC is at page_address+0xc/0xd4
[79528.689631] LR is at omap_crypto_cleanup+0x40/0xc64 [omap_crypto]
[79528.695785] pc : [<c01fa624>]    lr : [<bf3373dc>]    psr: 00000113
[79528.702111] sp : cd06dc58  ip : cd06dc80  fp : cd06dc7c
[79528.707389] r10: ffffe000  r9 : c0c502fc  r8 : 00000000
[79528.712670] r7 : 000005a0  r6 : cd10f4c0  r5 : cd6f80ec  r4 : 00000002
[79528.719258] r3 : 000005a0  r2 : 00000000  r1 : cd10f4c0  r0 : 00000000
[79528.725851] Flags: nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[79528.733055] Control: 10c5387d  Table: 8d7ac019  DAC: 00000051
[79528.738863] Process apkrouter_owrt_ (pid: 2259, stack limit = 0xcd06c218)
[79528.745715] Stack: (0xcd06dc58 to 0xcd06e000)
[79528.750132] dc40:                                                       00000002 cd6f80ec
[79528.758395] dc60: cd10f4c0 000005a0 00000000 c0c502fc cd06dcb4 cd06dc80 bf3373dc c01fa624
[79528.766659] dc80: 00000000 c0425610 00000002 cd6f8040 c0110768 00000000 c0d5f580 ffffe000
[79528.774923] dca0: c0c502fc ffffe000 cd06dcd4 cd06dcb8 bf33ee74 bf3373a8 0000000a 00000923
[79528.783186] dcc0: cd6f8060 cd6f8064 cd06dcfc cd06dcd8 c0136054 bf33ecfc 00000006 c0d02098
[79528.791448] dce0: c0d02080 40000006 00000100 00000000 cd06dd5c cd06dd00 c0101628 c0135fc4
[79528.799713] dd00: cd06dd2c cf56c068 cf56c068 00400040 c0d04900 0078e364 c0a53780 c0826434
[79528.807975] dd20: 00000009 c0d5f580 c0c50360 c0d02080 cf56c000 c0c56a24 00000000 00000000
[79528.816238] dd40: 00000001 cf404400 cd06c000 c0d60c08 cd06dd6c cd06dd60 c0136540 c0101534
[79528.824501] dd60: cd06dd94 cd06dd70 c017c2b8 c0136470 c1280c30 60000013 ffffffff cd06dde4
[79528.832764] dd80: fa8c4e71 cd06c000 cd06ddac cd06dd98 c010148c c017c220 c0823970 60000013
[79528.841028] dda0: cd06de14 cd06ddb0 c010c50c c0101408 cfbb8f40 cda2cc80 00000000 00006b1e
[79528.849289] ddc0: cfbb8f40 cd0e9800 cd870000 00000001 fa8c4e71 cd0e9800 c0d60c08 cd06de14
[79528.857551] dde0: cd06de00 cd06de00 c082396c c0823970 60000013 ffffffff 00000051 bf000000
[79528.865815] de00: cfbb8f40 cd0e9800 cd06de4c cd06de18 c01549f8 c0823950 00000001 00000000
[79528.874078] de20: c01549a4 cd0e9800 cd06de4c cfbb8f40 cda2cc80 cfbb8f50 cf5f8cc0 fa8c4e71
[79528.882341] de40: cd06de9c cd06de50 c081d7d8 c0154910 ade5e8b7 00004854 cd06df20 cfbb8f50
[79528.890605] de60: ade7733f 00004854 c01904d8 0000a145 c0c55540 ffffe000 ffffe000 0000c350
[79528.898869] de80: 00000000 00989680 00000000 00000001 cd06deb4 cd06dea0 c081d8d8 c081d13c
[79528.907133] dea0: cd06df20 ffffe000 cd06df14 cd06deb8 c08231f4 c081d860 0000c350 00000000
[79528.915396] dec0: 00000001 c0173710 00000080 c0192988 c0192988 00000000 00000000 c0d2b340
[79528.923659] dee0: 00000001 cd06df80 a0000013 00000001 cd06df80 0000c350 00000000 009959d0
[79528.931921] df00: 00000000 000000a2 cd06df7c cd06df18 c0190fec c0823180 00989680 00000000
[79528.940182] df20: cd06df20 00000000 00000000 00000000 edd9b726 00004853 edd8f3d6 00004853
[79528.948445] df40: c0190078 cfbb75c0 00000000 00000000 00000000 00000000 b6f45ce8 00000000
[79528.956708] df60: 00000000 000000a2 c0107e64 cd06c000 cd06dfa4 cd06df80 c01910d0 c0190ee8
[79528.964971] df80: 00000000 00000000 00989680 c014d924 b6f45ce8 00000000 00000000 cd06dfa8
[79528.973232] dfa0: c0107c60 c019105c 00000000 00000000 b6f45ce8 b6f45ce8 00000000 00000000
[79528.981494] dfc0: 00000000 00000000 00000000 000000a2 be9a7da8 b6f23000 b6fdc5d8 b6f45d4c
[79528.989757] dfe0: b6f45cb0 b6f45ca0 b6fb07cc b6fafc8c 60000010 b6f45ce8 00000000 00000000
[79528.997995] Backtrace: 
[79529.000529] [<c01fa618>] (page_address) from [<bf3373dc>] (omap_crypto_cleanup+0x40/0xc64 [omap_crypto])
[79529.010105]  r9:c0c502fc r8:00000000 r7:000005a0 r6:cd10f4c0 r5:cd6f80ec r4:00000002
[79529.017974] [<bf33739c>] (omap_crypto_cleanup [omap_crypto]) from [<bf33ee74>] (omap_aes_done_task+0x184/0x1f4 [omap_aes_driver])
[79529.029727]  r10:ffffe000 r9:c0c502fc r8:ffffe000 r7:c0d5f580 r6:00000000 r5:c0110768
[79529.037622]  r4:cd6f8040
[79529.040258] [<bf33ecf0>] (omap_aes_done_task [omap_aes_driver]) from [<c0136054>] (tasklet_action+0x9c/0x114)
[79529.050253]  r5:cd6f8064 r4:cd6f8060
[79529.053895] [<c0135fb8>] (tasklet_action) from [<c0101628>] (__do_softirq+0x100/0x25c)
[79529.061897]  r9:00000000 r8:00000100 r7:40000006 r6:c0d02080 r5:c0d02098 r4:00000006
[79529.069721] [<c0101528>] (__do_softirq) from [<c0136540>] (irq_exit+0xdc/0x140)
[79529.077111]  r10:c0d60c08 r9:cd06c000 r8:cf404400 r7:00000001 r6:00000000 r5:00000000
[79529.085005]  r4:c0c56a24
[79529.087613] [<c0136464>] (irq_exit) from [<c017c2b8>] (__handle_domain_irq+0xa4/0xb4)
[79529.095525] [<c017c214>] (__handle_domain_irq) from [<c010148c>] (omap_intc_handle_irq+0x90/0x94)
[79529.104487]  r9:cd06c000 r8:fa8c4e71 r7:cd06dde4 r6:ffffffff r5:60000013 r4:c1280c30
[79529.112313] [<c01013fc>] (omap_intc_handle_irq) from [<c010c50c>] (__irq_svc+0x6c/0x90)
[79529.120385] Exception stack(0xcd06ddb0 to 0xcd06ddf8)
[79529.125500] dda0:                                     cfbb8f40 cda2cc80 00000000 00006b1e
[79529.133763] ddc0: cfbb8f40 cd0e9800 cd870000 00000001 fa8c4e71 cd0e9800 c0d60c08 cd06de14
[79529.142022] dde0: cd06de00 cd06de00 c082396c c0823970 60000013 ffffffff
[79529.148701]  r5:60000013 r4:c0823970
[79529.152371] [<c0823944>] (_raw_spin_unlock_irq) from [<c01549f8>] (finish_task_switch+0xf4/0x1a4)
[79529.161318]  r5:cd0e9800 r4:cfbb8f40
[79529.164975] [<c0154904>] (finish_task_switch) from [<c081d7d8>] (__schedule+0x6a8/0x724)
[79529.173147]  r8:fa8c4e71 r7:cf5f8cc0 r6:cfbb8f50 r5:cda2cc80 r4:cfbb8f40
[79529.179925] [<c081d130>] (__schedule) from [<c081d8d8>] (schedule+0x84/0x9c)
[79529.187053]  r10:00000001 r9:00000000 r8:00989680 r7:00000000 r6:0000c350 r5:ffffe000
[79529.194948]  r4:ffffe000
[79529.197551] [<c081d854>] (schedule) from [<c08231f4>] (do_nanosleep+0x80/0x154)
[79529.204929]  r5:ffffe000 r4:cd06df20
[79529.208584] [<c0823174>] (do_nanosleep) from [<c0190fec>] (hrtimer_nanosleep+0x110/0x174)
[79529.216846]  r10:000000a2 r9:00000000 r8:009959d0 r7:00000000 r6:0000c350 r5:cd06df80
[79529.224741]  r4:00000001
[79529.227332] [<c0190edc>] (hrtimer_nanosleep) from [<c01910d0>] (SyS_nanosleep+0x80/0x98)
[79529.235507]  r9:cd06c000 r8:c0107e64 r7:000000a2 r6:00000000 r5:00000000 r4:b6f45ce8
[79529.243348] [<c0191050>] (SyS_nanosleep) from [<c0107c60>] (ret_fast_syscall+0x0/0x54)
[79529.251331]  r4:00000000
[79529.253924] Code: e89da800 e1a0c00d e92ddbf0 e24cb004 (e5903000) 
[79529.260308] ---[ end trace 482e6115e6308d53 ]---
[79529.265065] Kernel panic - not syncing: Fatal exception in interrupt
[79529.271530] Rebooting in 3 seconds..

0 Bin Liu over 3 years ago in reply to Filippo Carlini

TI__Guru*** 125066 points

Hi Filippo,

The last TI Processor SDK release which uses kernel v4.14 is Processor SDK Linux v5.3.0.7

https://software-dl.ti.com/processor-sdk-linux/esd/AM335X/05_03_00_07/index_FDS.html

It is based on kernel v4.14.79. IPSEC is validated in this release. I guess the kernel v4.14.221 has community update and am wondering if it breaks the crypto hw acceleration on AM3352.

Please you please tested with the kernel in TI Processor SDK v5.3.0.7 release to see if the kernel crash still happens?

0 Filippo Carlini over 3 years ago in reply to Bin Liu

Prodigy 230 points

Hello,

I already made tests with the last TI SDK : ("PROCESSOR-SDK-LINUX-AM335X 06_03_00_106") which include the kernel "Linux am335x-evm 4.19.94" and the behaviour is exactly the same.

Please find enclosed to this post the kernel panic log taken with kernel 4.19.94 from the TI SDK.

Regards

Log kernel panic Arago TI SDK.txt

[ 1630.140501] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[ 1630.148828] pgd = 1bdec059
[ 1630.151566] [00000000] *pgd=88105831, *pte=00000000, *ppte=00000000
[ 1630.157975] Internal error: Oops: 17 [#1] ARM
[ 1630.162367] Modules linked in: cbc aes_arm_bs crypto_simd cryptd sha256_generic sha256_arm hmac drbg authenc echainiv xfrm6_mode_tunnel xfrm4_mode_tunnel ppp_deflate bsd_comp ppp_async crc_ccitt ppp_generic slhc musb_dsps phy_am335x musb_hdrc phy_generic phy_am335x_control af_alg xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo pm33xx wkup_m3_ipc wkup_m3_rproc remoteproc omap_aes_driver crypto_engine omap_sham omap_crypto ti_emif_sram at24 musb_am335x rtc_omap omap_wdt sch_fq_codel g_serial option usb_wwan usbserial usbcore usb_f_acm u_serial libcomposite udc_core usb_common
[ 1630.216151] CPU: 0 PID: 1218 Comm: iperf3 Not tainted 4.19.94-gbe5389fd85 #10
[ 1630.223330] Hardware name: Generic AM33XX (Flattened Device Tree)
[ 1630.229492] PC is at page_address+0xc/0xe8
[ 1630.233650] LR is at omap_crypto_cleanup+0x48/0xbc [omap_crypto]
[ 1630.239698] pc : [<c020148c>]    lr : [<bf0f2048>]    psr: 00070113
[ 1630.246006] sp : caf73e5c  ip : caf73e70  fp : caf73e6c
[ 1630.251266] r10: 000005a0  r9 : 00000000  r8 : caff24c0
[ 1630.256527] r7 : c0d0ca28  r6 : caa2d2cc  r5 : c0113e94  r4 : 00000002
[ 1630.263096] r3 : 000005a0  r2 : 00000000  r1 : caff24c0  r0 : 00000000
[ 1630.269672] Flags: nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[ 1630.276854] Control: 10c5387d  Table: 8aad8019  DAC: 00000051
[ 1630.282641] Process iperf3 (pid: 1218, stack limit = 0x187adba2)
[ 1630.288690] Stack: (0xcaf73e5c to 0xcaf74000)
[ 1630.293084] 3e40:                                                                00000002
[ 1630.301328] 3e60: caf73ea4 caf73e70 bf0f2048 c020148c 00000000 c0402368 00000002 caa2d240
[ 1630.309573] 3e80: c0113e94 00000040 c0d0ca28 c0d47840 00000006 00000100 caf73ec4 caf73ea8
[ 1630.317815] 3ea0: bf10df80 bf0f200c 0000000a 00000923 00000000 00000000 caf73eec caf73ec8
[ 1630.326058] 3ec0: c012c398 bf10de2c 00000000 c0d49ca8 c0d49c90 caf72000 40000006 00000006
[ 1630.334301] 3ee0: caf73efc caf73ef0 c012c400 c012c334 caf73f5c caf73f00 c0102250 c012c3f0
[ 1630.342547] 3f00: caf73f2c caf73f10 c016c120 c0902b4c 00404000 c0abf504 c0d141a0 0002078f
[ 1630.350790] 3f20: 00000009 c0d47840 c0d0f3d8 c0d49c90 cf007700 c0d46218 00000000 00000001
[ 1630.359034] 3f40: 00000000 cf008000 00000000 bedfabb0 caf73f6c caf73f60 c012c754 c010214c
[ 1630.367277] 3f60: caf73f94 caf73f70 c0166574 c012c658 c0d7a264 a0070010 ffffffff 10c5387d
[ 1630.375520] 3f80: 10c53c7d 00000000 caf73fac caf73f98 c042b0c8 c0166520 b6f472d8 a0070010
[ 1630.383763] 3fa0: 00000000 caf73fb0 c0101dd4 c042b098 b6f63000 00000768 b6f13a00 b6f35f68
[ 1630.392006] 3fc0: 00019fac b6aed053 00000005 00020000 00022208 00000000 bedfabb0 0000000a
[ 1630.400249] 3fe0: b6f63000 bedfaa68 b6f14168 b6f472d8 a0070010 ffffffff 00000000 00000000
[ 1630.408471] Backtrace:
[ 1630.410970] [<c0201480>] (page_address) from [<bf0f2048>] (omap_crypto_cleanup+0x48/0xbc [omap_crypto])
[ 1630.420421]  r4:00000002
[ 1630.423037] [<bf0f2000>] (omap_crypto_cleanup [omap_crypto]) from [<bf10df80>] (omap_aes_done_task+0x160/0x1ec [omap_aes_driver])
[ 1630.434772]  r10:00000100 r9:00000006 r8:c0d47840 r7:c0d0ca28 r6:00000040 r5:c0113e94
[ 1630.442648]  r4:caa2d240
[ 1630.445262] [<bf10de20>] (omap_aes_done_task [omap_aes_driver]) from [<c012c398>] (tasklet_action_common.constprop.4+0x70/0xbc)
[ 1630.456808]  r5:00000000 r4:00000000
[ 1630.460431] [<c012c328>] (tasklet_action_common.constprop.4) from [<c012c400>] (tasklet_action+0x1c/0x20)
[ 1630.470068]  r9:00000006 r8:40000006 r7:caf72000 r6:c0d49c90 r5:c0d49ca8 r4:00000000
[ 1630.477882] [<c012c3e4>] (tasklet_action) from [<c0102250>] (__do_softirq+0x110/0x284)
[ 1630.485864] [<c0102140>] (__do_softirq) from [<c012c754>] (irq_exit+0x108/0x10c)
[ 1630.493319]  r10:bedfabb0 r9:00000000 r8:cf008000 r7:00000000 r6:00000001 r5:00000000
[ 1630.501195]  r4:c0d46218
[ 1630.503774] [<c012c64c>] (irq_exit) from [<c0166574>] (__handle_domain_irq+0x60/0xb0)
[ 1630.511685] [<c0166514>] (__handle_domain_irq) from [<c042b0c8>] (omap_intc_handle_irq+0x3c/0x94)
[ 1630.520626]  r9:00000000 r8:10c53c7d r7:10c5387d r6:ffffffff r5:a0070010 r4:c0d7a264
[ 1630.528432] [<c042b08c>] (omap_intc_handle_irq) from [<c0101dd4>] (__irq_usr+0x54/0x80)
[ 1630.536485] Exception stack(0xcaf73fb0 to 0xcaf73ff8)
[ 1630.541580] 3fa0:                                     b6f63000 00000768 b6f13a00 b6f35f68
[ 1630.549823] 3fc0: 00019fac b6aed053 00000005 00020000 00022208 00000000 bedfabb0 0000000a
[ 1630.558063] 3fe0: b6f63000 bedfaa68 b6f14168 b6f472d8 a0070010 ffffffff
[ 1630.564721]  r5:a0070010 r4:b6f472d8
[ 1630.568340] Code: e89da800 e1a0c00d e92dd810 e24cb004 (e5903000)
[ 1630.574582] ---[ end trace 6cee8b5e0db4f788 ]---
[ 1630.579272] Kernel panic - not syncing: Fatal exception in interrupt
[ 1630.585683] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

Filippo

0 Bin Liu over 3 years ago in reply to Filippo Carlini

TI__Guru*** 125066 points

Hi Filippo,

Can you please provide some details of the setup how to trigger the crash? I want to try to reproduce the issue so we can debug it.

0 Filippo Carlini over 3 years ago in reply to Bin Liu

Prodigy 230 points

Hello,

you can reproduce the issue with a BeagleBone black, installing kernel 4.19.94 from the last SDK and the Arago distribution.

Of course you have to enable Omap hw acceleration when building kernel.
Then you have to install Strongswan package (I built it using the yocto recipe inside the SDK) and configure it using the /etc/ipsec.conf and /etc/ipsec.secrets enclosed to this message.

As a server I installed Strongswan on an Ubuntu virtual machine on my PC configured with the attached server ipsec.conf and ipsec.secrets.
(I tried with other IPSEC installations anyway and the behaviour doesn't change)

The Beaglebone is connected to the PC via Ethernet.
When the two peers are connected trough the IPSEC tunnel you can make a performance test with iperf or a data transfer through the tunnel and you can reproduce the kernel panic.

Thank You and Regards

Filippo

IPSEC_configs.zip

0 Bin Liu over 3 years ago in reply to Filippo Carlini

TI__Guru*** 125066 points

Hi Filippo,

Thanks for the details. It sounds like the setup has nothing special, it is about the same as our setup running iperf test with IPSec, but we don't observe such kernel crash.

The AM335x Processor SDK release v6.03 on ti.com has prebuilt binaries for kernel and root filesystem which has Strongswan included, you don't have to rebuild anything for Beaglebone Black to test IPSec. Have you tried to use the prebuilt image from SDK v6.03 on Beaglebone Black to reproduce the issue?

0 Filippo Carlini over 3 years ago in reply to Bin Liu

Prodigy 230 points

No, I didn't try the prebuilt binaries because I wanted to be able to configure my kernel and my distribution.
I will try. If it works it means that there should be something that drives to a crash with different configurations.

I tried with two kernel versions and two different distributions (Arago from the SDK and Openwrt).

Thank you and Regards

Filippo

0 Bin Liu over 3 years ago in reply to Filippo Carlini

TI__Guru*** 125066 points

Hi Filippo,

Yes, trying the prebuilt will be a good data point for comparison, if it doesn't show the issue.

Looking forward to your test result with the prebuilt.

0 Filippo Carlini over 3 years ago in reply to Bin Liu

Prodigy 230 points

Ok, so I did....

I wrote an SD card using the SDK script:

ti-processor-sdk-linux-am335x-evm-06.03.00.106/bin$ sudo ./create-sdcard.sh

I put the SD card into my Beaglebone Black and I booted it up.
I assigned do the eth0 the address 192.168.123.1.
I put the ipsec.conf file that I sent you (client side).
I restarted ipsec and the two peers connected each other activating the tunnel.

On the server I executed:

iperf3 -s -f K

On the Beaglebone I executed:

iperf3 -c 192.168.6.234 -f M

and......

kernel panic !!!

Please find enclosed the log.

Regards

Filippo

Log kernel panic Arago TI SDK Beaglebone.txt

root@am335x-evm:~# iperf3 -c 192.168.6.234 -f M
Connecting to host 192.168.6.234, port 5201
[  5] local 192.168.123.1 port 50480 connected to 192.168.6.234 port 5201
[  126.598683] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[  126.606912] pgd = 79fb0153
[  126.609631] [00000000] *pgd=95bb3831, *pte=00000000, *ppte=00000000
[  126.615936] Internal error: Oops: 17 [#1] PREEMPT ARM
[  126.621007] Modules linked in: cbc aes_arm_bs crypto_simd cryptd sha256_generic sha256_arm hmac drbg authenc echainiv xfrm4_mode_tunnel xfrm_user xfrm4_tunnel ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo usb_f_acm u_serial usb_f_ecm g_multi usb_f_mass_storage usb_f_rndis u_ether libcomposite pru_rproc irq_pruss_intc pruss musb_dsps musb_hdrc udc_core phy_am335x phy_generic phy_am335x_control snd_soc_simple_card snd_soc_simple_card_utils pm33xx wkup_m3_rproc wkup_m3_ipc remoteproc pvrsrvkm(O) omap_aes_driver crypto_engine omap_crypto omap_sham pruss_soc_bus ti_emif_sram at24 musb_am335x omap_wdt rtc_omap sch_fq_codel uio_module_drv(O) uio ftdi_sio usbserial usbcore usb_common cryptodev(O)
[  126.682273] CPU: 0 PID: 1008 Comm: iperf3 Tainted: G           O      4.19.94-gbe5389fd85 #1
[  126.690744] Hardware name: Generic AM33XX (Flattened Device Tree)
[  126.696876] PC is at page_address+0x10/0x134
[  126.701169] LR is at omap_crypto_cleanup+0x48/0xbc [omap_crypto]
[  126.707199] pc : [<c0201058>]    lr : [<bf0e5048>]    psr: 00070113
[  126.713490] sp : d5ab9b98  ip : d5ab9bb8  fp : d5ab9bb4
[  126.718733] r10: 000005a0  r9 : 00000000  r8 : d5be78c0
[  126.723977] r7 : c0e0cae8  r6 : db7afacc  r5 : c0114654  r4 : 00000002
[  126.730529] r3 : 000005a0  r2 : 00000000  r1 : d5be78c0  r0 : 00000000
[  126.737084] Flags: nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[  126.744246] Control: 10c5387d  Table: 95c14019  DAC: 00000051
[  126.750014] Process iperf3 (pid: 1008, stack limit = 0xc60e2e97)
[  126.756043] Stack: (0xd5ab9b98 to 0xd5aba000)
[  126.760418] 9b80:                                                       d5ab9bb4 d5ab9ba8
[  126.768632] 9ba0: c01da318 00000002 d5ab9bec d5ab9bb8 bf0e5048 c0201054 00000000 c04297e4
[  126.776845] 9bc0: 00000002 db7afa40 c0114654 00000040 c0e0cae8 c0e4b540 00000006 00000100
[  126.785058] 9be0: d5ab9c0c d5ab9bf0 bf0f5004 bf0e500c 0000000a 00000923 00000000 00000000
[  126.793272] 9c00: d5ab9c34 d5ab9c10 c012e168 bf0f4eb0 00000000 c0e4d9e8 c0e4d9d0 d5ab8000
[  126.801485] 9c20: 40000006 00000006 d5ab9c44 d5ab9c38 c012e1d0 c012e104 d5ab9ca4 d5ab9c48
[  126.809699] 9c40: c0102290 c012e1c0 d5ab9c74 d5ab9c58 c01752e0 c0a02b34 00404000 c0bc3e0c
[  126.817912] 9c60: c0e14a60 ffffbc43 00000009 c0e4b540 c0e0f4c0 c0e4d9d0 dc007700 c0e49f20
[  126.826126] 9c80: 00000000 00000001 00000000 dc008000 d5ab8000 33e7794d d5ab9cb4 d5ab9ca8
[  126.834339] 9ca0: c012e540 c010218c d5ab9cdc d5ab9cb8 c016eea8 c012e444 c0e7e09c 60070013
[  126.842552] 9cc0: ffffffff d5ab9d2c 0000055a d5ab8000 d5ab9cf4 d5ab9ce0 c0452e40 c016ee54
[  126.850765] 9ce0: c0830b08 60070013 d5ab9df4 d5ab9cf8 c0101a0c c0452e10 d5c60000 0001faa6
[  126.858979] 9d00: 0000055a 00000001 d5c60000 d5c600ec 00000001 d5ab9e4c 0000055a 0000055a
[  126.867193] 9d20: 33e7794d d5ab9df4 d5ab9d30 d5ab9d48 c082c6e0 c0830b08 60070013 ffffffff
[  126.875406] 9d40: 00000051 bf000000 d5ab9d6c d5ab9d58 00000000 00000000 db61ce40 d5ab9db8
[  126.883620] 9d60: d5ab9e1c ffffe000 c0e03048 c0e03868 00202000 00000000 c022eca4 00000000
[  126.891833] 9d80: 00000000 00000001 00000000 0000055a 00000040 db6e6000 c0e03048 0000055a
[  126.900047] 9da0: 00000000 d5ab9e54 00000000 00000000 00000000 00000000 00000000 00000000
[  126.908261] 9dc0: bef379c0 ce0ea58d d5c60000 d5c60000 d5ab9e4c 00020000 00000000 d5ab9ed8
[  126.916475] 9de0: d5bb5a80 d724c600 d5ab9e14 d5ab9df8 c0831670 c0830964 d5c60000 c0e03048
[  126.924688] 9e00: 00000000 00000000 d5ab9e34 d5ab9e18 c085d9ec c083164c c085d370 c082d76c
[  126.932901] 9e20: 00000000 d5ab9ec0 d5ab9e44 d5ab9e38 c07af340 c085d9bc d5ab9eac d5ab9e48
[  126.941115] 9e40: c07af3ec c07af330 c07ad75c 00000000 00000000 00000001 0000055a 0001faa6
[  126.949329] 9e60: d5ab9eb8 00000001 00000000 00000000 00000000 00000040 d5ab9ed8 ce0ea58d
[  126.957542] 9e80: d5ab9ed8 c0e03048 d5bb5a80 d5ab9f60 00000000 ce0ea58d 00000000 00000004
[  126.965755] 9ea0: d5ab9f24 d5ab9eb0 c0236d94 c07af35c 00020000 00000000 b6aa7000 00020000
[  126.973969] 9ec0: 00000001 00000000 00020000 d5ab9eb8 00000001 00000000 d5bb5a80 00000000
[  126.982182] 9ee0: 00000000 00000000 00000000 00000000 00000000 00000000 3b990cb0 ce0ea58d
[  126.990396] 9f00: 00020000 d5bb5a80 b6aa7000 d5ab9f60 00000000 00000000 d5ab9f54 d5ab9f28
[  126.998610] 9f20: c0236f78 c0236c84 d5bb5a80 00000802 d5ab9f54 d5bb5a80 c0e03048 d5bb5a80
[  127.006823] 9f40: b6aa7000 00000000 d5ab9f94 d5ab9f58 c02371fc c0236ed8 00000000 00000000
[  127.015036] 9f60: 00000000 00000000 bef37938 ce0ea58d 00020000 b6aa7000 00000005 00000004
[  127.023250] 9f80: c0101204 d5ab8000 d5ab9fa4 d5ab9f98 c0237280 c023719c 00000000 d5ab9fa8
[  127.031463] 9fa0: c0101000 c023727c 00020000 b6aa7000 00000005 b6aa7000 00020000 00000000
[  127.039676] 9fc0: 00020000 b6aa7000 00000005 00000004 00022208 00000000 bef379c0 0000000a
[  127.047889] 9fe0: 0000006c bef378a8 b6ee0337 b6e4c22c 60070010 00000005 00000000 00000000
[  127.056097] Backtrace:
[  127.058560] [<c0201048>] (page_address) from [<bf0e5048>] (omap_crypto_cleanup+0x48/0xbc [omap_crypto])
[  127.067990]  r4:00000002
[  127.070553] [<bf0e5000>] (omap_crypto_cleanup [omap_crypto]) from [<bf0f5004>] (omap_aes_done_task+0x160/0x1ec [omap_aes_driver])
[  127.082255]  r10:00000100 r9:00000006 r8:c0e4b540 r7:c0e0cae8 r6:00000040 r5:c0114654
[  127.090114]  r4:db7afa40
[  127.092677] [<bf0f4ea4>] (omap_aes_done_task [omap_aes_driver]) from [<c012e168>] (tasklet_action_common.constprop.3+0x70/0xbc)
[  127.104201]  r5:00000000 r4:00000000
[  127.107794] [<c012e0f8>] (tasklet_action_common.constprop.3) from [<c012e1d0>] (tasklet_action+0x1c/0x20)
[  127.117402]  r9:00000006 r8:40000006 r7:d5ab8000 r6:c0e4d9d0 r5:c0e4d9e8 r4:00000000
[  127.125183] [<c012e1b4>] (tasklet_action) from [<c0102290>] (__do_softirq+0x110/0x284)
[  127.133136] [<c0102180>] (__do_softirq) from [<c012e540>] (irq_exit+0x108/0x114)
[  127.140564]  r10:33e7794d r9:d5ab8000 r8:dc008000 r7:00000000 r6:00000001 r5:00000000
[  127.148422]  r4:c0e49f20
[  127.150972] [<c012e438>] (irq_exit) from [<c016eea8>] (__handle_domain_irq+0x60/0xb0)
[  127.158838] [<c016ee48>] (__handle_domain_irq) from [<c0452e40>] (omap_intc_handle_irq+0x3c/0x94)
[  127.167748]  r9:d5ab8000 r8:0000055a r7:d5ab9d2c r6:ffffffff r5:60070013 r4:c0e7e09c
[  127.175525] [<c0452e04>] (omap_intc_handle_irq) from [<c0101a0c>] (__irq_svc+0x6c/0xa8)
[  127.183560] Exception stack(0xd5ab9cf8 to 0xd5ab9d40)
[  127.188630] 9ce0:                                                       d5c60000 0001faa6
[  127.196843] 9d00: 0000055a 00000001 d5c60000 d5c600ec 00000001 d5ab9e4c 0000055a 0000055a
[  127.205057] 9d20: 33e7794d d5ab9df4 d5ab9d30 d5ab9d48 c082c6e0 c0830b08 60070013 ffffffff
[  127.213266]  r5:60070013 r4:c0830b08
[  127.216863] [<c0830958>] (tcp_sendmsg_locked) from [<c0831670>] (tcp_sendmsg+0x30/0x44)
[  127.224901]  r10:d724c600 r9:d5bb5a80 r8:d5ab9ed8 r7:00000000 r6:00020000 r5:d5ab9e4c
[  127.232760]  r4:d5c60000
[  127.235307] [<c0831640>] (tcp_sendmsg) from [<c085d9ec>] (inet_sendmsg+0x3c/0x70)
[  127.242821]  r7:00000000 r6:00000000 r5:c0e03048 r4:d5c60000
[  127.248512] [<c085d9b0>] (inet_sendmsg) from [<c07af340>] (sock_sendmsg+0x1c/0x2c)
[  127.256111]  r4:d5ab9ec0
[  127.258656] [<c07af324>] (sock_sendmsg) from [<c07af3ec>] (sock_write_iter+0x9c/0xf0)
[  127.266525] [<c07af350>] (sock_write_iter) from [<c0236d94>] (__vfs_write+0x11c/0x168)
[  127.274476]  r10:00000004 r9:00000000 r8:ce0ea58d r7:00000000 r6:d5ab9f60 r5:d5bb5a80
[  127.282336]  r4:c0e03048
[  127.284879] [<c0236c78>] (__vfs_write) from [<c0236f78>] (vfs_write+0xac/0x184)
[  127.292220]  r9:00000000 r8:00000000 r7:d5ab9f60 r6:b6aa7000 r5:d5bb5a80 r4:00020000
[  127.299998] [<c0236ecc>] (vfs_write) from [<c02371fc>] (ksys_write+0x6c/0xe0)
[  127.307164]  r8:00000000 r7:b6aa7000 r6:d5bb5a80 r5:c0e03048 r4:d5bb5a80
[  127.313893] [<c0237190>] (ksys_write) from [<c0237280>] (sys_write+0x10/0x14)
[  127.321059]  r9:d5ab8000 r8:c0101204 r7:00000004 r6:00000005 r5:b6aa7000 r4:00020000
[  127.328835] [<c0237270>] (sys_write) from [<c0101000>] (ret_fast_syscall+0x0/0x54)
[  127.336435] Exception stack(0xd5ab9fa8 to 0xd5ab9ff0)
[  127.341508] 9fa0:                   00020000 b6aa7000 00000005 b6aa7000 00020000 00000000
[  127.349721] 9fc0: 00020000 b6aa7000 00000005 00000004 00022208 00000000 bef379c0 0000000a
[  127.357932] 9fe0: 0000006c bef378a8 b6ee0337 b6e4c22c
[  127.363007] Code: e1a0c00d e92dd810 e24cb004 e24dd00c (e5903000)
[  127.369219] ---[ end trace a4b1829e57dfe7e8 ]---
[  127.373859] Kernel panic - not syncing: Fatal exception in interrupt
[  127.380243] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

0 Bin Liu over 3 years ago in reply to Filippo Carlini

TI__Guru*** 125066 points

Hi Filippo,

Thanks for the quick update. The crash log seems to be the same as that on your platform.

I am interested to know how the test is different from our SystemTest did... I will look into the issue and might reproduce it on my setup so that I can debug it. I should have an update around the mid of next week.

0 Bin Liu over 3 years ago in reply to Filippo Carlini

TI__Guru*** 125066 points

Hi Filippo,

My apologies for the delay. I am having a hard time to create an IPSec setup. I support kernel crypto, but am not an expert on Network/IPSec. The IPSec config files/certificates I have was used about 5 years ago while debugging crypto drivers, and I only found out now that the certificates are already expired in 2018.

I had also tried to use your IPSec config, but it seems the setup uses a router in the middle:

192.168.6.234 <--> 192.168.123.1 <--> 192.168.123.113

while my setup is to use PC <--> EVM directly.

I will continue to work on creating an IPSec setup, and keep you posted.

0 Bin Liu over 2 years ago in reply to Filippo Carlini

TI__Guru*** 125066 points

Hi Filippo,

As mentioned previously above, the IPSec tunnel didn't establish using your config files on my EVM setup.

However I am able to run iperf on an IPSec UDP tunnel using the IPSec config files attached below with the prebuilt kernel and rootfs in Processor SDK Linux v6.3, but I didn't see any kernel crash or panic. These config files are used in the automation test in Processor SDK releases.

ipsec_files.tgz

The EVM or Beaglebone Black is directly connected to a LInux PC via ethenet. On PC, run

$ iperf -s -u

On EVM, run:

# date -s '2021-11-20 15:40'
# ifconfig eth0 down && ifconfig eth0 192.168.0.214 up
# ipsec stop && sudo ipsec start
# ipsec up Beta-Conn1
# iperf -c 192.168.0.1 -u -i2 -t600 -b100M

0 Colombo Carlo over 2 years ago in reply to Bin Liu

TI__Mastermind 21530 points

Hi Bin ,

we need to use IPSEC config as files attached in this thread on feb 26 --> that is the configuration we must work with .

could you kindly test with that server setting ?

thank you

Carlo

0 Bin Liu over 2 years ago in reply to Colombo Carlo

TI__Guru*** 125066 points

Hi Carlo,

In my test setup I detailed yesterday, the Beaglebone Black ethernet port is directly connected to the PC, the PC ip address is 192.168.0.1, while the BBB ip address is 192.168.0.214.

Please explain the customer's setup in detail how to run the test using the customer's config files. I am not sure how to use it since its leftside and rightside ip addresses are not in the same subnet.

If I keep my version of ipsec.conf but only replace the customer's version of ipsec.secrets, I got the following failure message on BBB when starting the IPSec connection.

root@am335x-evm:~# ipsec start
Starting strongSwan 5.7.1 IPsec [starter]...
root@am335x-evm:~# ipsec up Beta-Conn1
initiating IKE_SA Beta-Conn1[1] to 192.168.0.1
generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
sending packet: from 192.168.0.214[500] to 192.168.0.1[500] (464 bytes)
received packet: from 192.168.0.1[500] to 192.168.0.214[500] (487 bytes)
parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(FRAG_SUP) N(HASH_ALG) N(MULT_AUTH) ]
selected proposal: IKE:AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_2048
received cert request for "C=US, O=TI, CN=Strongswan CA"
sending cert request for "C=US, O=TI, CN=Strongswan CA"
no private key found for 'C=US, O=Test, CN=beta.test.org'
establishing connection 'Beta-Conn1' failed

0 Filippo Carlini over 2 years ago in reply to Bin Liu

Prodigy 230 points

Hi Bin,

in our configuration, the beaglebone eth0 is connected to a 4G router to have internet access and reach the IPSEC server on an AWS virtual machine.
The 4G router assigns to the beaglebone an IP address trough its dhcp server.

I can provide you our "/etc/network/interfaces", "/etc/ipsec.conf", /etc/ipsec.secrets" configuration files so you will be able to connect your beaglebone to our IPSEC server. You'll have to find a router to which connect the beaglebone eth0 in order to have internet access.

We should schedule the test when we both will be available. I will turn on the IPSEC server and iperf3 server during the test.

Do you agree ?

Best Regards

Filippo

0 Bin Liu over 2 years ago in reply to Filippo Carlini

TI__Guru*** 125066 points

Hi Filippo,

It would be difficult for me to create the setup with a router to have internet access.

Is it possible for your to deploy your IPSec server on a local Linux PC next to BBB? It is pretty straight forward to install Strongswan on Ubuntu, just a matter of apt install then copy over the IPSec config files to /etc/ folder. To have a standalone setup also should help debugging the issue once I am able to reproduce it, without much of your involvement on your AWS server side.

0 Filippo Carlini over 2 years ago in reply to Bin Liu

Prodigy 230 points

Hi Bin,

I already sent you our IPSEC configuration files in this thread on February 26th .
That configuration (server side) was on an Ubuntu 16.04 machine.

Our AWS machine runs Centos 8 that uses a slightly different configuration because has a more recent version of strongswan that uses swanclt.conf instead of ipsec.conf with a different syntax.

Regards

Filippo

0 Bin Liu over 2 years ago in reply to Filippo Carlini

TI__Guru*** 125066 points

Hi Filippo,

But how should I use your config files you provided on 2/26. As I asked before, the leftside and rightside IP in ipsec.conf are on different subnet, how should I configure the IP address of my BBB and PC to use your config files? Do you use a local router between BBB and Ubuntu 16.04? I need to understand your setup (network topology) to use your config files.

Also the config files only include ipsec.conf and ipsec.secrets, no keys or certificates are provided. Does your setup only use ipsec.conf and ipsec.secrets, no certificates under /etc/ipsec.d/ are needed?

Please review the config files I provided a couple days ago to understand what config files are used in my IPSec test.

0 Filippo Carlini over 2 years ago in reply to Bin Liu

Prodigy 230 points

Hi Bin,

my Ubuntu IPSEC server is on a vmware virtual machine so, in the middle, there is vmware virtual network that acts like a router.

That configuration doesn't need certificates, only a preshared key:

leftauth=psk
rightauth=psk

Regards

Filippo

0 Bin Liu over 2 years ago in reply to Filippo Carlini

TI__Guru*** 125066 points

Hi Filippo,

I think I modified your version of ipsec.conf back in February or March to use the same subnet on both leftside and rightside, but failed to establish the IPSec tunnel. Let me give it another try next week, and I will get back to you.

0 Bin Liu over 2 years ago in reply to Filippo Carlini

TI__Guru*** 125066 points

Hi Filippo,

I modified your ipsec.conf to put both EVM and PC on the same subnet, but I got the following error when trying to connect the IPSec tunnel.

root@am335x-evm:~# ipsec up prodgateway-to-devgateway
initiating Main Mode IKE_SA prodgateway-to-devgateway[1] to 192.168.0.1
generating ID_PROT request 0 [ SA V V V V V ]
sending packet: from 192.168.0.214[500] to 192.168.0.1[500] (204 bytes)
received packet: from 192.168.0.1[500] to 192.168.0.214[500] (56 bytes)
parsed INFORMATIONAL_V1 request 1691271572 [ N(NO_PROP) ]
received NO_PROPOSAL_CHOSEN error notify
establishing connection 'prodgateway-to-devgateway' failed

Following is the full logs on the PC and the EVM.

ipsec-failure-server.log

dev@uda:~$ cat /etc/ipsec.conf
# ipsec.conf - strongSwan IPsec configuration file

config setup
        charondebug="all"
        uniqueids=yes

conn devgateway-to-prodgateway
        type=tunnel
        auto=start
        keyexchange=ikev1
        authby=secret
        left=192.168.0.1
        leftsubnet=192.168.0.0/24
        right=192.168.0.214
        rightsubnet=192.168.0.0/24
	leftid=192.168.0.1
	leftauth=psk
        rightauth=psk
        rightid=192.168.0.214
        ike=aes128-sha1-modp1024!
#        ike=aes128-sha1-modp1024,3des-sha1-modp1024!
#        esp=aes128-sha1!
#        aggressive=yes
        keyingtries=%forever
        ikelifetime=28800s
        lifetime=3600s
        dpddelay=30s
        dpdtimeout=120s
        dpdaction=restart
dev@uda~$ 
dev@uda~$ $ sudo cat /etc/ipsec.secrets 
# : RSA /etc/ipsec.d/private/alphaKey.der
 : PSK "pippopluto"

dev@uda~$ 
dev@uda~$ sudo ipsec version
Linux strongSwan U5.6.2/K4.15.0-159-generic
Institute for Internet Technologies and Applications
University of Applied Sciences Rapperswil, Switzerland
See 'ipsec --copyright' for copyright information.
dev@uda~$ 
dev@uda~$ sudo ipsec stop && sudo ipsec start
[sudo] password for lb: 
Stopping strongSwan IPsec...
Starting strongSwan 5.6.2 IPsec [starter]...
dev@uda~$

ipsec-failure-gpevm.log

root@am335x-evm:~# uname -a 
Linux am335x-evm 4.19.94-gbe5389fd85 #1 PREEMPT Sun Apr 19 03:43:09 UTC 2020 armv7l GNU/Linux
root@am335x-evm:~# 
root@am335x-evm:~# date -s '2021-10-25 10:49'
Mon Oct 25 10:49:00 UTC 2021
root@am335x-evm:~# 
root@am335x-evm:~# cat /etc/ipsec.conf
# ipsec.conf - strongSwan IPsec configuration file

# basic configuration

config setup
        # strictcrlpolicy=yes
        # uniqueids = no
        charondebug="all"
        uniqueids=yes

# Add connections here.

conn prodgateway-to-devgateway
        type=tunnel
        auto=start
        keyexchange=ikev1
        authby=secret
        left=192.168.0.214
        right=192.168.0.1
        rightsubnet=192.168.0.0/24
        leftid=192.168.0.214
        leftauth=psk
        rightauth=psk
        auto=add
        rekey=yes
        rightid=192.168.0.1
#        ike=aes128-sha1-modp1024!
#        esp=aes128-sha1!
#        aggressive=yes
        keyingtries=%forever
        ikelifetime=28800s
        lifetime=3600s
        dpddelay=30s
        dpdtimeout=120s
        dpdaction=restart
root@am335x-evm:~# 
root@am335x-evm:~# cat /etc/ipsec.secrets 
# ipsec.secrets - strongSwan IPsec secrets file
#
# : RSA /etc/ipsec.d/private/betaKey.der
 : PSK "pippopluto"
root@am335x-evm:~# 
root@am335x-evm:~# ipsec version
Linux strongSwan U5.7.1/K4.19.94-gbe5389fd85
University of Applied Sciences Rapperswil, Switzerland
See 'ipsec --copyright' for copyright information.
root@am335x-evm:~# 
root@am335x-evm:~# ipsec stop && ipsec start
Stopping strongSwan IPsec...
Starting strongSwan 5.7.1 IPsec [starter]...
root@am335x-evm:~# 
root@am335x-evm:~# ifconfig eth1 down
root@am335x-evm:~# 
root@am335x-evm:~# ifconfig eth1 192.168.0.1 up
[  177.668695] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
root@am335x-evm:~# [  177.771614] ax88179_178a 2-1:1.0 eth1: ax88179 - Link status is: 1
[  181.227585] ax88179_178a 2-1:1.0 eth1: ax88179 - Link status is: 1
[  181.255025] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

root@am335x-evm:~# 
root@am335x-evm:~# ping -c 3 192.168.0.1
PING 192.168.0.1 (192.168.0.1): 56 data bytes
64 bytes from 192.168.0.1: seq=0 ttl=64 time=1.618 ms
64 bytes from 192.168.0.1: seq=1 ttl=64 time=0.955 ms
64 bytes from 192.168.0.1: seq=2 ttl=64 time=1.022 ms

--- 192.168.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.955/1.198/1.618 ms
root@am335x-evm:~# 
root@am335x-evm:~# ipsec up prodgateway-to-devgateway
initiating Main Mode IKE_SA prodgateway-to-devgateway[1] to 192.168.0.1
generating ID_PROT request 0 [ SA V V V V V ]
sending packet: from 192.168.0.214[500] to 192.168.0.1[500] (204 bytes)
received packet: from 192.168.0.1[500] to 192.168.0.214[500] (56 bytes)
parsed INFORMATIONAL_V1 request 1691271572 [ N(NO_PROP) ]
received NO_PROPOSAL_CHOSEN error notify
establishing connection 'prodgateway-to-devgateway' failed
root@am335x-evm:~#

0 Filippo Carlini over 2 years ago in reply to Bin Liu

Prodigy 230 points

Hi Bin,
I rearranged my test bed on xubuntu 20.04.

Please find enclosed the configuration files and the log of the test until the kernel panic.

The xubuntu machine has IP address 192.168.1.131 and the beaglebone has IP address 192.168.1.1.

Directly connected each other without any router in the middle.beaglebone_configuration.zip xubuntu_configuration.zip

kernel panic log.txt

root@am335x-evm:~# ipsec start
Starting strongSwan 5.7.1 IPsec [starter]...
[   94.169295] NET: Registered protocol family 15
[   94.313579] Initializing XFRM netlink socket
root@am335x-evm:~# [   94.859764] cryptd: max_cpu_qlen set to 1000

root@am335x-evm:~# ipsec statusall
Status of IKE charon daemon (strongSwan 5.7.1, Linux 4.19.94-gbe5389fd85, armv7l):
  uptime: 9 seconds, since Oct 22 11:21:38 2021
  malloc: sbrk 675840, mmap 0, used 348352, free 327488
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 3
  loaded plugins: charon aes des rc2 sha2 sha1 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl curve25519 xcbc cmac hmac curl sqlite attr kernel-netlink resolve socket-default stroke vici updown xauth-generic
Listening IP addresses:
  192.168.1.1
  192.168.7.2
Connections:
        vpn1:  %any...192.168.1.131  IKEv1, dpddelay=30s
        vpn1:   local:  [apr410] uses pre-shared key authentication
        vpn1:   remote: [xubuntu] uses pre-shared key authentication
        vpn1:   child:  192.168.1.0/24 === 192.168.1.0/24 TUNNEL, dpdaction=restart
Security Associations (1 up, 0 connecting):
        vpn1[1]: ESTABLISHED 9 seconds ago, 192.168.1.1[apr410]...192.168.1.131[xubuntu]
        vpn1[1]: IKEv1 SPIs: b57ccba96c4ab973_i* 87f92cf7b29e5ff3_r, pre-shared key reauthentication in 42 minutes
        vpn1[1]: IKE proposal: AES_CBC_128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256
        vpn1{1}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: cd3ffc8b_i c96dd37a_o
        vpn1{1}:  AES_CBC_128/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 7 hours
        vpn1{1}:   192.168.1.0/24 === 192.168.1.0/24


root@am335x-evm:~# lsmod | grep crypt
crypto_simd            16384  1 aes_arm_bs
cryptd                 24576  1 crypto_simd
crypto_engine          16384  1 omap_aes_driver
omap_crypto            16384  1 omap_aes_driver
cryptodev              49152  0


root@am335x-evm:~# iperf3 -c 192.168.1.131 -f M
Connecting to host 192.168.1.131, port 5201
[  5] local 192.168.1.1 port 34778 connected to 192.168.1.131 port 5201
[  262.688449] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[  262.696672] pgd = d29b93e3
[  262.699403] [00000000] *pgd=95ad6831, *pte=00000000, *ppte=00000000
[  262.705718] Internal error: Oops: 17 [#1] PREEMPT ARM
[  262.710789] Modules linked in: cbc aes_arm_bs crypto_simd cryptd sha256_generic sha256_arm hmac drbg authenc echainiv xfrm4_mode_tunnel xfrm_user xfrm4_tunnel ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo usb_f_acm u_serial usb_f_ecm g_multi usb_f_mass_storage usb_f_rndis u_ether libcomposite pru_rproc irq_pruss_intc pruss musb_dsps musb_hdrc phy_am335x udc_core phy_generic phy_am335x_control snd_soc_simple_card snd_soc_simple_card_utils pm33xx wkup_m3_rproc wkup_m3_ipc remoteproc pvrsrvkm(O) omap_aes_driver crypto_engine omap_crypto omap_sham ti_emif_sram pruss_soc_bus at24 musb_am335x rtc_omap omap_wdt sch_fq_codel uio_module_drv(O) uio ftdi_sio usbserial usbcore usb_common cryptodev(O)
[  262.772054] CPU: 0 PID: 1002 Comm: iperf3 Tainted: G           O      4.19.94-gbe5389fd85 #1
[  262.780524] Hardware name: Generic AM33XX (Flattened Device Tree)
[  262.786656] PC is at page_address+0x10/0x134
[  262.790952] LR is at omap_crypto_cleanup+0x48/0xbc [omap_crypto]
[  262.796980] pc : [<c0201058>]    lr : [<bf0e5048>]    psr: 00070113
[  262.803271] sp : d5c35b68  ip : d5c35b88  fp : d5c35b84
[  262.808513] r10: 000005b0  r9 : 00000000  r8 : d5c1b8c0
[  262.813757] r7 : c0e0cae8  r6 : db7cc8cc  r5 : c0114654  r4 : 00000002
[  262.820310] r3 : 000005b0  r2 : 00000000  r1 : d5c1b8c0  r0 : 00000000
[  262.826865] Flags: nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[  262.834028] Control: 10c5387d  Table: 95cb4019  DAC: 00000051
[  262.839796] Process iperf3 (pid: 1002, stack limit = 0x5d173391)
[  262.845826] Stack: (0xd5c35b68 to 0xd5c36000)
[  262.850203] 5b60:                   d5c35b84 d5c35b78 c01da318 00000002 d5c35bbc d5c35b88
[  262.858417] 5b80: bf0e5048 c0201054 00000000 c04297e4 00000002 db7cc840 c0114654 00000040
[  262.866630] 5ba0: c0e0cae8 c0e4b540 00000006 00000100 d5c35bdc d5c35bc0 bf0f5004 bf0e500c
[  262.874843] 5bc0: 0000000a 00000923 00000000 00000000 d5c35c04 d5c35be0 c012e168 bf0f4eb0
[  262.883057] 5be0: 00000000 c0e4d9e8 c0e4d9d0 d5c34000 40000006 00000006 d5c35c14 d5c35c08
[  262.891271] 5c00: c012e1d0 c012e104 d5c35c74 d5c35c18 c0102290 c012e1c0 d5c35c44 d5c35c28
[  262.899484] 5c20: c01752e0 c0a02b34 00404000 c0bc3e0c c0e14a60 fffff16c 00000009 c0e4b540
[  262.907698] 5c40: c0e0f4c0 c0e4d9d0 dc007700 c0e49f20 00000000 00000001 00000000 dc008000
[  262.915911] 5c60: d5c34000 00000000 d5c35c84 d5c35c78 c012e540 c010218c d5c35cac d5c35c88
[  262.924125] 5c80: c016eea8 c012e444 c0e7e09c 60070013 ffffffff d5c35cfc 0000056a d5c34000
[  262.932339] 5ca0: d5c35cc4 d5c35cb0 c0452e40 c016ee54 c07b2f90 60070013 d5c35d2c d5c35cc8
[  262.940552] 5cc0: c0101a0c c0452e10 00000020 d5bc910c 00000001 dced88e0 d5bc910c 006000c0
[  262.948765] 5ce0: d5bc910c d5c35e4c 0000056a 0000056a 00000000 d5c35d2c d5c35d30 d5c35d18
[  262.956978] 5d00: c07b2fe4 c07b2f90 60070013 ffffffff 00000051 bf000000 d5c60000 d5c600ec
[  262.965191] 5d20: d5c35d44 d5c35d30 c07b2fe4 c07b2ed8 d5c60000 d5c600ec d5c35df4 d5c35d48
[  262.973405] 5d40: c0830de8 c07b2fd4 d5c35d6c d5c35d58 00000000 00000000 db5beb40 d5c35db8
[  262.981618] 5d60: d5c35e1c ffffe000 c0e03048 c0e03868 00202000 00000000 c022eca4 00000000
[  262.989832] 5d80: 00000000 00000001 00000000 0000056a 00000040 d5c19000 c0e03048 0000056a
[  262.998044] 5da0: 00000000 d5c35e54 00000000 00000000 00000000 00000000 00000000 00000000
[  263.006257] 5dc0: bef1a9c0 075aa3b1 d5c60000 d5c60000 d5c35e4c 00020000 00000000 d5c35ed8
[  263.014471] 5de0: db337a80 d6a2d480 d5c35e14 d5c35df8 c0831670 c0830964 d5c60000 c0e03048
[  263.022685] 5e00: 00000000 00000000 d5c35e34 d5c35e18 c085d9ec c083164c c085d370 c082d76c
[  263.030898] 5e20: 00000000 d5c35ec0 d5c35e44 d5c35e38 c07af340 c085d9bc d5c35eac d5c35e48
[  263.039112] 5e40: c07af3ec c07af330 c07ad75c 00000000 00000000 00000001 0000056a 0001fa96
[  263.047325] 5e60: d5c35eb8 00000001 00000000 00000000 00000000 00000040 d5c35ed8 075aa3b1
[  263.055538] 5e80: d5c35ed8 c0e03048 db337a80 d5c35f60 00000000 075aa3b1 00000000 00000004
[  263.063752] 5ea0: d5c35f24 d5c35eb0 c0236d94 c07af35c 00020000 00000000 b6ac2000 00020000
[  263.071965] 5ec0: 00000001 00000000 00020000 d5c35eb8 00000001 00000000 db337a80 00000000
[  263.080178] 5ee0: 00000000 00000000 00000000 00000000 00000000 00000000 3b991e44 075aa3b1
[  263.088392] 5f00: 00020000 db337a80 b6ac2000 d5c35f60 00000000 00000000 d5c35f54 d5c35f28
[  263.096605] 5f20: c0236f78 c0236c84 db337a80 00000802 d5c35f54 db337a80 c0e03048 db337a80
[  263.104818] 5f40: b6ac2000 00000000 d5c35f94 d5c35f58 c02371fc c0236ed8 00000000 00000000
[  263.113031] 5f60: 00000000 00000000 bef1a938 075aa3b1 00020000 b6ac2000 00000005 00000004
[  263.121246] 5f80: c0101204 d5c34000 d5c35fa4 d5c35f98 c0237280 c023719c 00000000 d5c35fa8
[  263.129459] 5fa0: c0101000 c023727c 00020000 b6ac2000 00000005 b6ac2000 00020000 00000000
[  263.137672] 5fc0: 00020000 b6ac2000 00000005 00000004 00022208 00000000 bef1a9c0 0000000a
[  263.145885] 5fe0: 0000006c bef1a8a8 b6efb337 b6e6722c 60070010 00000005 00000000 00000000
[  263.154093] Backtrace:
[  263.156556] [<c0201048>] (page_address) from [<bf0e5048>] (omap_crypto_cleanup+0x48/0xbc [omap_crypto])
[  263.165987]  r4:00000002
[  263.168550] [<bf0e5000>] (omap_crypto_cleanup [omap_crypto]) from [<bf0f5004>] (omap_aes_done_task+0x160/0x1ec [omap_aes_driver])
[  263.180252]  r10:00000100 r9:00000006 r8:c0e4b540 r7:c0e0cae8 r6:00000040 r5:c0114654
[  263.188111]  r4:db7cc840
[  263.190675] [<bf0f4ea4>] (omap_aes_done_task [omap_aes_driver]) from [<c012e168>] (tasklet_action_common.constprop.3+0x70/0xbc)
[  263.202200]  r5:00000000 r4:00000000
[  263.205793] [<c012e0f8>] (tasklet_action_common.constprop.3) from [<c012e1d0>] (tasklet_action+0x1c/0x20)
[  263.215401]  r9:00000006 r8:40000006 r7:d5c34000 r6:c0e4d9d0 r5:c0e4d9e8 r4:00000000
[  263.223182] [<c012e1b4>] (tasklet_action) from [<c0102290>] (__do_softirq+0x110/0x284)
[  263.231136] [<c0102180>] (__do_softirq) from [<c012e540>] (irq_exit+0x108/0x114)
[  263.238564]  r10:00000000 r9:d5c34000 r8:dc008000 r7:00000000 r6:00000001 r5:00000000
[  263.246423]  r4:c0e49f20
[  263.248972] [<c012e438>] (irq_exit) from [<c016eea8>] (__handle_domain_irq+0x60/0xb0)
[  263.256838] [<c016ee48>] (__handle_domain_irq) from [<c0452e40>] (omap_intc_handle_irq+0x3c/0x94)
[  263.265749]  r9:d5c34000 r8:0000056a r7:d5c35cfc r6:ffffffff r5:60070013 r4:c0e7e09c
[  263.273526] [<c0452e04>] (omap_intc_handle_irq) from [<c0101a0c>] (__irq_svc+0x6c/0xa8)
[  263.281560] Exception stack(0xd5c35cc8 to 0xd5c35d10)
[  263.286634] 5cc0:                   00000020 d5bc910c 00000001 dced88e0 d5bc910c 006000c0
[  263.294847] 5ce0: d5bc910c d5c35e4c 0000056a 0000056a 00000000 d5c35d2c d5c35d30 d5c35d18
[  263.303058] 5d00: c07b2fe4 c07b2f90 60070013 ffffffff
[  263.308127]  r5:60070013 r4:c07b2f90
[  263.311721] [<c07b2ecc>] (skb_page_frag_refill) from [<c07b2fe4>] (sk_page_frag_refill+0x1c/0x74)
[  263.320629]  r5:d5c600ec r4:d5c60000
[  263.324225] [<c07b2fc8>] (sk_page_frag_refill) from [<c0830de8>] (tcp_sendmsg_locked+0x490/0xce8)
[  263.333132]  r5:d5c600ec r4:d5c60000
[  263.336723] [<c0830958>] (tcp_sendmsg_locked) from [<c0831670>] (tcp_sendmsg+0x30/0x44)
[  263.344760]  r10:d6a2d480 r9:db337a80 r8:d5c35ed8 r7:00000000 r6:00020000 r5:d5c35e4c
[  263.352620]  r4:d5c60000
[  263.355168] [<c0831640>] (tcp_sendmsg) from [<c085d9ec>] (inet_sendmsg+0x3c/0x70)
[  263.362681]  r7:00000000 r6:00000000 r5:c0e03048 r4:d5c60000
[  263.368373] [<c085d9b0>] (inet_sendmsg) from [<c07af340>] (sock_sendmsg+0x1c/0x2c)
[  263.375972]  r4:d5c35ec0
[  263.378517] [<c07af324>] (sock_sendmsg) from [<c07af3ec>] (sock_write_iter+0x9c/0xf0)
[  263.386386] [<c07af350>] (sock_write_iter) from [<c0236d94>] (__vfs_write+0x11c/0x168)
[  263.394337]  r10:00000004 r9:00000000 r8:075aa3b1 r7:00000000 r6:d5c35f60 r5:db337a80
[  263.402195]  r4:c0e03048
[  263.404739] [<c0236c78>] (__vfs_write) from [<c0236f78>] (vfs_write+0xac/0x184)
[  263.412080]  r9:00000000 r8:00000000 r7:d5c35f60 r6:b6ac2000 r5:db337a80 r4:00020000
[  263.419856] [<c0236ecc>] (vfs_write) from [<c02371fc>] (ksys_write+0x6c/0xe0)
[  263.427021]  r8:00000000 r7:b6ac2000 r6:db337a80 r5:c0e03048 r4:db337a80
[  263.433751] [<c0237190>] (ksys_write) from [<c0237280>] (sys_write+0x10/0x14)
[  263.440917]  r9:d5c34000 r8:c0101204 r7:00000004 r6:00000005 r5:b6ac2000 r4:00020000
[  263.448695] [<c0237270>] (sys_write) from [<c0101000>] (ret_fast_syscall+0x0/0x54)
[  263.456293] Exception stack(0xd5c35fa8 to 0xd5c35ff0)
[  263.461365] 5fa0:                   00020000 b6ac2000 00000005 b6ac2000 00020000 00000000
[  263.469578] 5fc0: 00020000 b6ac2000 00000005 00000004 00022208 00000000 bef1a9c0 0000000a
[  263.477789] 5fe0: 0000006c bef1a8a8 b6efb337 b6e6722c
[  263.482863] Code: e1a0c00d e92dd810 e24cb004 e24dd00c (e5903000)
[  263.489066] ---[ end trace 2342b225a6045421 ]---
[  263.493706] Kernel panic - not syncing: Fatal exception in interrupt
[  263.500089] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

I hope that now you will be able to reproduce the issue.

The log contains also a list of the modules concerned by the problem.

Best Regards
Filippo

0 Bin Liu over 2 years ago in reply to Filippo Carlini

TI__Guru*** 125066 points

Hi Filippo,

Thanks for the new config files, I am able to reproduce the kernel crash with them using kernel v4.19 in Processor SDK v6.3.

The good news is the issue seems to be fixed in the new Processor SDK v7.3 which was released a few months ago. I can successfully run iperf test with the kernel v5.4 in SDK v7.3.

Can you please test with SDK v7.3 on your setup?

www.ti.com/.../07.03.00.005

0 Filippo Carlini over 2 years ago in reply to Bin Liu

Prodigy 230 points

Hi Bin,

I downloaded and installed the SDK v7.3 on an SD card and booted up the Beaglebone Black.

I confirm you that the kernel panic doesn't happen anymore but there is something wrong anyway now.

The performance is better if I unload the crypto modules (managing the acceleration hw) than with the modules loaded !

Please find enclosed a test summary of the 4 configurations:

Module unloaded trough IPSEC tunnel and with direct connection.

Model loaded trough IPSEC tunnel and with direct connection.

Best Regards
Filippo

IPERF_IPSEC_SDK7_Test.txt

--------------------------------
BBB crypto modules UNLOADED, 600 Mhz, via IPSEC

root@am335x-evm:~# iperf3 -c 192.168.99.1 -f M
Connecting to host 192.168.99.1, port 5201
[  5] local 192.168.7.2 port 56386 connected to 192.168.99.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  3.09 MBytes  3.09 MBytes/sec    0    329 KBytes
[  5]   1.00-2.00   sec  3.69 MBytes  3.69 MBytes/sec   40    286 KBytes
[  5]   2.00-3.00   sec  2.10 MBytes  2.10 MBytes/sec    0    329 KBytes
[  5]   3.00-4.05   sec  1.68 MBytes  1.61 MBytes/sec  127    245 KBytes
[  5]   4.05-5.00   sec  4.36 MBytes  4.56 MBytes/sec    0    268 KBytes
[  5]   5.00-6.00   sec  3.37 MBytes  3.36 MBytes/sec   75    200 KBytes
[  5]   6.00-7.00   sec  2.52 MBytes  2.52 MBytes/sec    0    218 KBytes
[  5]   7.00-8.00   sec  2.55 MBytes  2.55 MBytes/sec   14    164 KBytes
[  5]   8.00-9.00   sec  2.10 MBytes  2.10 MBytes/sec    0    180 KBytes
[  5]   9.00-10.00  sec  2.94 MBytes  2.94 MBytes/sec    0    186 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  28.4 MBytes  2.84 MBytes/sec  256             sender
[  5]   0.00-10.07  sec  27.3 MBytes  2.71 MBytes/sec                  receiver

iperf Done.


--------------------------------
BBB crypto modules UNLOADED, 600 Mhz, direct

root@am335x-evm:~# iperf3 -c 195.231.67.140 -f M
Connecting to host 195.231.67.140, port 5201
[  5] local 192.168.137.61 port 45638 connected to 195.231.67.140 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  6.60 MBytes  6.60 MBytes/sec    0    443 KBytes
[  5]   1.00-2.00   sec  6.73 MBytes  6.73 MBytes/sec    0    731 KBytes
[  5]   2.00-3.00   sec  3.75 MBytes  3.75 MBytes/sec  144    567 KBytes
[  5]   3.00-4.00   sec  6.25 MBytes  6.25 MBytes/sec    0    646 KBytes
[  5]   4.00-5.00   sec  6.25 MBytes  6.25 MBytes/sec    0    700 KBytes
[  5]   5.00-6.00   sec  5.00 MBytes  5.00 MBytes/sec    0    735 KBytes
[  5]   6.00-7.00   sec  6.25 MBytes  6.25 MBytes/sec    0    757 KBytes
[  5]   7.00-8.00   sec  2.50 MBytes  2.50 MBytes/sec    0    769 KBytes
[  5]   8.00-9.00   sec  3.75 MBytes  3.75 MBytes/sec    0    775 KBytes
[  5]   9.00-10.00  sec  5.00 MBytes  5.00 MBytes/sec    0    775 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  52.1 MBytes  5.21 MBytes/sec  144             sender
[  5]   0.00-10.12  sec  49.9 MBytes  4.93 MBytes/sec                  receiver

iperf Done.


--------------------------------
BBB crypto modules LOADED, 600 Mhz, via IPSEC

root@am335x-evm:~# iperf3 -c 192.168.99.1 -f M
Connecting to host 192.168.99.1, port 5201
[  5] local 192.168.7.2 port 49194 connected to 192.168.99.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   836 KBytes  0.81 MBytes/sec    0   67.4 KBytes
[  5]   1.00-2.00   sec  1.03 MBytes  1.03 MBytes/sec    0    114 KBytes
[  5]   2.00-3.00   sec  1.25 MBytes  1.25 MBytes/sec    0    171 KBytes
[  5]   3.00-4.04   sec  1.65 MBytes  1.59 MBytes/sec    0    226 KBytes
[  5]   4.04-5.52   sec  1.83 MBytes  1.23 MBytes/sec    0    284 KBytes
[  5]   5.52-6.06   sec  1.18 MBytes  2.18 MBytes/sec    0    333 KBytes
[  5]   6.06-7.00   sec  1.31 MBytes  1.39 MBytes/sec    0    333 KBytes
[  5]   7.00-8.01   sec  1.15 MBytes  1.14 MBytes/sec    0    333 KBytes
[  5]   8.01-9.06   sec  1.98 MBytes  1.89 MBytes/sec    0    385 KBytes
[  5]   9.06-10.34  sec  1.58 MBytes  1.23 MBytes/sec    0    385 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.34  sec  13.8 MBytes  1.33 MBytes/sec    0             sender
[  5]   0.00-10.39  sec  12.5 MBytes  1.21 MBytes/sec                  receiver

iperf Done.


--------------------------------
BBB crypto modules LOADED, 600 Mhz, direct

root@am335x-evm:~# iperf3 -c 195.231.67.140 -f M
Connecting to host 195.231.67.140, port 5201
[  5] local 192.168.137.193 port 46044 connected to 195.231.67.140 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.54 MBytes  4.54 MBytes/sec    0    369 KBytes
[  5]   1.00-2.00   sec  5.97 MBytes  5.96 MBytes/sec    0    594 KBytes
[  5]   2.00-3.00   sec  6.24 MBytes  6.25 MBytes/sec    0    889 KBytes
[  5]   3.00-4.00   sec  3.75 MBytes  3.75 MBytes/sec  360    465 KBytes
[  5]   4.00-5.00   sec  5.00 MBytes  5.00 MBytes/sec   86    349 KBytes
[  5]   5.00-6.00   sec  6.25 MBytes  6.25 MBytes/sec    0    386 KBytes
[  5]   6.00-7.00   sec  3.75 MBytes  3.75 MBytes/sec    0    407 KBytes
[  5]   7.00-8.00   sec  6.25 MBytes  6.25 MBytes/sec    0    420 KBytes
[  5]   8.00-9.00   sec  5.00 MBytes  5.00 MBytes/sec    0    424 KBytes
[  5]   9.00-10.00  sec  6.25 MBytes  6.25 MBytes/sec    0    424 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  53.0 MBytes  5.30 MBytes/sec  446             sender
[  5]   0.00-10.07  sec  50.2 MBytes  4.99 MBytes/sec                  receiver

iperf Done.

0 Bin Liu over 2 years ago in reply to Filippo Carlini

TI__Guru*** 125066 points

Hi Filippo,

I didn't pay attention on the numbers when I tested it, but I remembered I got around 3MB/sec w/ hw crypto in IPSec.

Anyway, have you compared CPU load in both cases w/ and w/o hw crypto?

0 Filippo Carlini over 2 years ago in reply to Bin Liu

Prodigy 230 points

Hi Bin,

the CPU load sounds strange, always in the wrong direction compared to what I was thinking, it seems a little lower when the modules are unloaded.

With crypto modules I am always above 95% of CPU load during the iperf test and without the modules I am around 75% or lower.

Regards

Fillippo

0 Bin Liu over 2 years ago in reply to Filippo Carlini

TI__Guru*** 125066 points

Hi Filippo,

For whatever reason, my PC has some network issue today to communicate to my board (I use network boot). Let me do more test next week on the performance.

The last time I looked at IPSec was a few years ago, I remembered the iperf (not iperf3) UDP numbers w/ hw crypto was slightly less than w/ sw crypto, but the CPU load was much less. So basically the hw crypto doesn't improve the throughput but it off-load the CPU.

Processors

Processors forum

AM3352: crash when using HW crypto accelerator API