PROCESSOR-SDK-AM62X: Cryptodev hardware accelerator does not take effect

Hi Gary,

Could you please share the verbose output of these tests with the help of `time -v`

# with sa2ul
time -v openssl speed -evp aes-128-cbc -engine devcrypto

# without sa2ul
time -v openssl speed -evp aes-128-cbc

For reference, following are the logs from the tests done on my TI AM62X HSFS board. The logs shows that the CPU usage was almost 50% less when the openssl was using sa2ul engine.

root@am62xx-evm:~# time -v openssl speed -evp aes-128-cbc -engine devcrypto
Engine "devcrypto" set.
Doing AES-128-CBC for 3s on 16 size blocks: 63085 AES-128-CBC's in 0.10s
Doing AES-128-CBC for 3s on 64 size blocks: 66531 AES-128-CBC's in 0.08s
Doing AES-128-CBC for 3s on 256 size blocks: 59120 AES-128-CBC's in 0.07s
Doing AES-128-CBC for 3s on 1024 size blocks: 49130 AES-128-CBC's in 0.02s
Doing AES-128-CBC for 3s on 8192 size blocks: 22171 AES-128-CBC's in 0.04s
Doing AES-128-CBC for 3s on 16384 size blocks: 12943 AES-128-CBC's in 0.03s
version: 3.0.12
built on: Tue Oct 24 13:48:41 2023 UTC
options: bn(64,64)
compiler: aarch64-oe-linux-gcc  -mbranch-protection=standard --sysroot=recipe-sysroot -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=  -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG
CPUINFO: OPENSSL_armcap=0xbd
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
AES-128-CBC      10093.60k    53224.80k   216210.29k  2515456.00k  4540620.80k  7068603.73k
        Command being timed: "openssl speed -evp aes-128-cbc -engine devcrypto"
        User time (seconds): 0.37
        System time (seconds): 7.49
        Percent of CPU this job got: 43%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 18.16s
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 23744
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 314
        Voluntary context switches: 273070
        Involuntary context switches: 29
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
root@am62xx-evm:~# time -v openssl speed -evp aes-128-cbc
Doing AES-128-CBC for 3s on 16 size blocks: 22747996 AES-128-CBC's in 2.99s
Doing AES-128-CBC for 3s on 64 size blocks: 16963282 AES-128-CBC's in 2.99s
Doing AES-128-CBC for 3s on 256 size blocks: 8155871 AES-128-CBC's in 3.00s
Doing AES-128-CBC for 3s on 1024 size blocks: 2753497 AES-128-CBC's in 2.99s
Doing AES-128-CBC for 3s on 8192 size blocks: 382388 AES-128-CBC's in 3.00s
Doing AES-128-CBC for 3s on 16384 size blocks: 192330 AES-128-CBC's in 2.99s
version: 3.0.12
built on: Tue Oct 24 13:48:41 2023 UTC
options: bn(64,64)
compiler: aarch64-oe-linux-gcc  -mbranch-protection=standard --sysroot=recipe-sysroot -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=  -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG
CPUINFO: OPENSSL_armcap=0xbd
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
AES-128-CBC     121728.41k   363093.66k   695967.66k   943003.65k  1044174.17k  1053891.21k
        Command being timed: "openssl speed -evp aes-128-cbc"
        User time (seconds): 17.97
        System time (seconds): 0.00
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 18.02s
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 23536
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 307
        Voluntary context switches: 1
        Involuntary context switches: 243
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
root@am62xx-evm:~#

Regards,

Prashant

Hello Gary,

Thank you for the logs. Indeed, it looks like the SA2UL hardware acceleration is not working.

Let us move away from openssl for now to a simpler cipher test available in the cryptodev driver (board-support/extra-drivers/cryptodev-module-*/tests). Simply copy the following two files to the target filesystem: 1) cipher.c & 2) testhelper.h. Then, perform the following procedure:

root@am62xx-evm:~# ls
cipher.c  testhelper.h
root@am62xx-evm:~# gcc -o cipher cipher.c
root@am62xx-evm:~# modprobe -r sa2ul
root@am62xx-evm:~# ./cipher -v
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-ce
AES Test passed
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-ce
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-ce
Test passed
root@am62xx-evm:~# modprobe sa2ul
root@am62xx-evm:~# ./cipher -v
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-sa2ul
AES Test passed
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-sa2ul
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-sa2ul
Test passed
root@am62xx-evm:~#

The above logs shows that on disabling the SA2UL module the test was performed using the software algorithm while on loading the SA2UL module the test was performed using the SA2UL hardware accelerator.

Regards,

Prashant

Hello Gary,

Thank you for doing the test. This mostly looks like an issue with the SA2UL driver itself.

Could you please let me know if you have modified the SA2UL driver (board-support/ti-linux-kernel/drivers/crypto/sa2ul.c) or using it as it is? In case it is modified, please make sure the supported algorithms field is filled properly

static struct sa_match_data am64_match_data = {
	.priv = 0,
	.priv_id = 0,
	.supported_algos = BIT(SA_ALG_CBC_AES) |
			   BIT(SA_ALG_EBC_AES) |
			   BIT(SA_ALG_SHA256) |
			   BIT(SA_ALG_SHA512) |
			   BIT(SA_ALG_AUTHENC_SHA256_AES),
};

static const struct of_device_id of_match[] = {
	{ .compatible = "ti,j721e-sa2ul", .data = &am654_match_data, },
	{ .compatible = "ti,am654-sa2ul", .data = &am654_match_data, },
	{ .compatible = "ti,am64-sa2ul", .data = &am64_match_data, },
	{ .compatible = "ti,am62-sa3ul", .data = &am64_match_data, },
	{},
};

Regards,

Prashant

Hello Gary,

I assume that the customer is working on top of PSDK v8.6. All this time, I was thinking we are talking about PSDK v9 & so was doing the testing with the same. Now that I tested the PSDK v8.6 prebuilt WIC image, I do see the SA2UL is not working meaning the issue could be reproduced.

And with that, I started working through the differences between the PSDK v8.6 & PSDK v9 and stumbled upon the following difference in the crypto node

At first, I simply copy pasted the crypto node as it is from PSDK v9 to PSDK v8.6. This resulted in the SA2UL driver working meaning the cipher test was using the cbc-aes-sa2ul driver for the test. Then the minimal change in the PSDK v8.6 DTS that also worked is

diff --git a/board-support/linux-5.10.168+gitAUTOINC+2c23e6c538-g2c23e6c538/arch/arm64/boot/dts/ti/k3-am62-main.dtsi b/board-support/linux-5.10.168+gitAUTOINC+2c23e6c538-g2c23e6c538/arch/arm64/boot/dts/ti/k3-am62-main.dtsi
index 510775e9a..1689b8278 100644
--- a/board-support/linux-5.10.168+gitAUTOINC+2c23e6c538-g2c23e6c538/arch/arm64/boot/dts/ti/k3-am62-main.dtsi
+++ b/board-support/linux-5.10.168+gitAUTOINC+2c23e6c538-g2c23e6c538/arch/arm64/boot/dts/ti/k3-am62-main.dtsi
@@ -179,7 +179,6 @@
 	crypto: crypto@40900000 {
 		compatible = "ti,am62-sa3ul";
 		reg = <0x00 0x40900000 0x00 0x1200>;
-		power-domains = <&k3_pds 70 TI_SCI_PD_SHARED>;
 		#address-cells = <2>;
 		#size-cells = <2>;
 		ranges = <0x00 0x40900000 0x00 0x40900000 0x00 0x30000>;

Could you please ask the customer to try out the above changes in the crypto node & see if this gets the SA2UL working.

Regards,

Prashant

Hello Gary,

Attaching the logs just in case if they help.

Default PSDK v8.6 WIC Image

root@am62xx-evm:~# uname -r
5.10.168-g2c23e6c538
root@am62xx-evm:~# modprobe -r sa2ul
root@am62xx-evm:~# ./cipher -v
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-ce
AES Test passed
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-ce
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-ce
Test passed
root@am62xx-evm:~# modprobe sa2ul
root@am62xx-evm:~# ./cipher -v
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-ce
AES Test passed
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-ce
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-ce
Test passed
root@am62xx-evm:~#

After DTS changes

root@am62xx-evm:~# uname -r
5.10.168-g2c23e6c538
root@am62xx-evm:~# modprobe -r sa2ul
root@am62xx-evm:~# ./cipher -v
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-ce
AES Test passed
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-ce
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-ce
Test passed
root@am62xx-evm:~# modprobe sa2ul
root@am62xx-evm:~# ./cipher -v
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-sa2ul
AES Test passed
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-sa2ul
requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver cbc-aes-sa2ul
Test passed
root@am62xx-evm:~#

Regards,

Prashant