Linux/AM3352: AES driver random crashes

Yes same thing. I also tried without pre-emption  (No Forced Preemption). But I still get the crash.

Hello again,

I haven't been able to create a generic method to recreate the error. But I have compiled the kernel with KDB so I can get the log out. It looks like we're out of 64kB buffers, but the memory seems to be very fragmented. Maybe you have a suggestion on how to resolve this. See log below:

<4>[  319.763538] enter, keylen: 16
<4>[  319.763602] nbytes: 48, enc: 1, cbc: 1
<4>[  319.768733] enter, keylen: 16
<4>[  319.768855] nbytes: 32816, enc: 0, cbc: 1
<4>[  319.768942] 53500000.aes-en: page allocation failure: order:4, mode:0x2080020(GFP_ATOMIC)
<4>[  319.768977] CPU: 0 PID: 821 Comm: 53500000.aes-en Tainted: G           O    4.9.59-rt23+ #84
<4>[  319.768981] Hardware name: Generic AM33XX (Flattened Device Tree)
<4>[  319.768985] Backtrace:
<4>[  319.769020] [<c00197ec>] (dump_backtrace) from [<c0019aa8>] (show_stack+0x18/0x1c)
<4>[  319.769030]  r7:c0a92954 r6:c0aa35c0 r5:00000030 r4:00000000
<4>[  319.769046] [<c0019a90>] (show_stack) from [<c02d420c>] (dump_stack+0x24/0x28)
<4>[  319.769064] [<c02d41e8>] (dump_stack) from [<c00c1cb4>] (warn_alloc+0x100/0x114)
<4>[  319.769073] [<c00c1bb8>] (warn_alloc) from [<c00c2038>] (__alloc_pages_nodemask+0x2cc/0xb28)
<4>[  319.769079]  r3:00000000 r2:00000004 r1:c06f129c
<4>[  319.769083]  r4:02080020
<4>[  319.769093] [<c00c1d6c>] (__alloc_pages_nodemask) from [<c00c28ac>] (__get_free_pages+0x18/0x34)
<4>[  319.769102]  r10:00000000 r9:cdc30000 r8:00000004 r7:00008030 r6:cc61da00 r5:cdddd280
<4>[  319.769106]  r4:cca42610
<4>[  319.769123] [<c00c2894>] (__get_free_pages) from [<c04e432c>] (omap_aes_prepare_req+0xc0/0x270)
<4>[  319.769143] [<c04e426c>] (omap_aes_prepare_req) from [<c0298b90>] (crypto_pump_work+0x1d0/0x298)
<4>[  319.769152]  r10:00000000 r9:00000000 r8:cc91db7c r7:cc91db50 r6:a0030013 r5:cdddd280
<4>[  319.769156]  r4:cc91dbd4
<4>[  319.769179] [<c02989c0>] (crypto_pump_work) from [<c0053ec0>] (kthread_worker_fn+0xf4/0x198)
<4>[  319.769188]  r9:00000000 r8:ffffe000 r7:00000001 r6:c0ae5978 r5:cca16000 r4:cc91dbb4
<4>[  319.769200] [<c0053dcc>] (kthread_worker_fn) from [<c0053d80>] (kthread+0xf8/0x110)
<4>[  319.769210]  r9:00000000 r8:c0053dcc r7:cc91dbb4 r6:cca16000 r5:cca45c80 r4:00000000
<4>[  319.769232] [<c0053c88>] (kthread) from [<c0015720>] (ret_from_fork+0x14/0x34)
<4>[  319.769240]  r8:00000000 r7:00000000 r6:00000000 r5:c0053c88 r4:cca45c80
<4>[  319.769243] Mem-Info:
<4>[  319.769265] active_anon:18689 inactive_anon:1451 isolated_anon:0
<4>[  319.769265]  active_file:7223 inactive_file:7187 isolated_file:0
<4>[  319.769308] lowmem_reserve[]: 0 0 0
<4>[  319.769317] Normal: 104*4kB (UMEHC) 156*8kB (UMEH) 296*16kB (UMEH) 1128*32kB (UMEH) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 2*8192kB (C) = 58880kB
<4>[  319.769358] 15965 total pagecache pages
<4>[  319.769369] 0 pages in swap cache
<4>[  319.769374] Swap cache stats: add 0, delete 0, find 0/0
<4>[  319.769376] Free swap  = 0kB
<4>[  319.769378] Total swap = 0kB
<4>[  319.769382] 65536 pages RAM
<4>[  319.769384] 0 pages HighMem/MovableOnly
<4>[  319.769387] 2758 pages reserved
<4>[  319.769389] 6144 pages cma reserved
<3>[  319.769398]    omap_aes_copy_sgs: Couldn't allocated pages for unaligned cases.
<3>[  319.777172] omap_aes_prepare_req: Failed to copy SGs for unaligned cases
<4>[  319.784208] total: 32816
<4>[  319.785243] enter done_task

Magnus,

I think if you could explain how to use encrypted filesystem, I could try to reproduce it on my setup. I haven't used encrypted filesystem before.

The second log you provided seems telling out of memory. How much DDR does your board have?

Since your system works in v4.4 but failed in v4.9, looking at the crypto driver changes in kernel source code from git://gitorious.ti.com/ti-linux-kernel/ti-linux-kernel.git:

$ git log --oneline ti2016.01..ti2017.05 drivers/crypto/omap-aes*

02a424f69f73 Revert "crypto: omap-aes: prevent probe time race condition with multiple cores"
3f3c3de21b5c crypto: omap-aes: Verify page zone of scatterlists before starting DMA
4f15d74a4338 crypto: omap-aes: Use bool instead of int for omap_aes_check_aligned()
18b7177f5e6e crypto: omap-aes: add spinlock init for GCM mode lock
49f6bf75eced crypto: omap-aes: prevent probe time race condition with multiple cores
ff9ca8c009a0 crypto: omap-aes: make queue length configurable
fb73ee38836c crypto: omap-aes: make fallback size configurable
ca1e39992036 crypto: omap-aes: Add support for GCM mode
ecbcbd3f4306 crypto: omap-aes: export some AES driver functionality locally
460e7be69189 crypto: omap-aes: fix context handling for multiple cores
d7b43357ab94 crypto: omap-aes: move definitions over to a separate header file
99f2c84ba255 crypto: omap-aes: drop unused flags attribute from omap_aes_ctx
c98ef8dbca18 crypto: omap-aes - fix error return code in omap_aes_probe()
0d0cda93de95 crypto: omap-aes - fix crypto engine initialization order
9fcb191a84eb crypto: omap-aes - Add fallback support
164f3ef301bf crypto: omap-aes - Add support for multiple cores
f303b455bb95 crypto: omap-aes - use runtime_pm autosuspend for clock handling
4cba7cf025f3 crypto: engine - permit to enqueue ashash_request
2589ad84047f crypto: engine - move crypto engine to its own header
7c001a8650e6 crypto: omap - Stop using crypto scatterwalk_bytes_sglen
b973eaab68db crypto: omap - do not call dmaengine_terminate_all
3741bbb207f7 crypto: omap-aes - Fix registration of algorithms
da8b29a6c189 crypto: omap-aes - Use dma_request_chan() for requesting DMA channel
0529900a01cb crypto: omap-aes - Support crypto engine framework
16f080aaadcb crypto: omap - Removed unused variable "err"

The two highlighted above are more interesting to the issue you have. Is it possible for you to manually revert the two changes one at a time to see if they are related?

Bin,

To use encrypted files systems you should use the tool cryptsetup:

# Create a key
head -c 32 /dev/urandom  > /tmp/key

# Map a partition to the decrypted device /dev/mapper/encpart
cryptsetup --cipher=aes-cbc-essiv:sha256 --key-file=/tmp/key open --type=plain /dev/<partition> encpart

# Create file system
mkfs.ext4 /dev/mapper/encpart

# Mount the decrypted partition
mkdir /tmp/encpart
mount /dev/mapper/encpart /tmp/encpart

I have 256MB RAM which should be more than enough. I think that the main problem is that the memory gets very fragmented.

I tried to revert the commits you suggested. I managed to revert "Stop using crypto scatterwalk_bytes_sglen", but failed with the other one. However this didn't solve the problem. There has been a major refactoring of the crypto drivers between the 4.4 and 4.9 release, which makes it hard to revert individual commits. Linus Torvalds made a large commit on Oct 10th 2016. I tried to go back to the version before this commit, but then md5 calculations seems to be incorrect. Actually I'm not sure how to proceed now.

Magnus,

Can you please try the mainline v4.14 kernel to see if the issue still exist?
Can you please get memory usage statistics with omap-sham to see if there is mem leak?
How soon after kernel booted to see the failure?

Bin,

The patches did not resolve my problem. But I have finally been able to make a small test program to trigger the error. It seems like it is the md5 calculation of a certain size that is problematic.

The test program looks like this:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/ioctl.h>
#include <crypto/cryptodev.h>

#define BUF_LEN 32784

char buf[BUF_LEN];
char digest[16];

int main(void)
{
    int count = 0;

    while (1)
    {
        int cfd = 0;
        struct session_op sess = {0};
        struct crypt_op cryp = {0};

        cfd = open("/dev/crypto", O_RDWR, 0);
        if (cfd < 0) {
            perror("open(/dev/crypto)");
            exit(1);
        }

        sess.mac = CRYPTO_MD5;
        if (ioctl(cfd, CIOCGSESSION, &sess)) {
            perror("ioctl(CIOCGSESSION)");
            exit(1);
        }

        cryp.ses = sess.ses;
        cryp.flags = COP_FLAG_UPDATE;
        cryp.len = BUF_LEN;
        cryp.src = buf;
        cryp.mac = digest;
        if (ioctl(cfd, CIOCCRYPT, &cryp)) {
            perror("ioctl(CIOCCRYPT)");
            exit(1);
        }

        cryp.flags = COP_FLAG_FINAL;
        cryp.len = 0;
        cryp.src = NULL;
        if (ioctl(cfd, CIOCCRYPT, &cryp)) {
            perror("ioctl(CIOCCRYPT)");
            exit(1);
        }

        if (ioctl(cfd, CIOCFSESSION, &sess.ses)) {
            perror("ioctl(CIOCFSESSION)");
            exit(1);
        }
        close(cfd);

        printf("%d\n", count++);
    }
}

After a number of runs the kernel will run out of memory when I run on kernel version 4.14.

Before I run the test program /proc/pagetypeinfo shows this:

Page block order: 11
Pages per block:  2048

Free pages count per migrate type at order       0      1      2      3      4      5      6      7      8      9     10     11 
Node    0, zone   Normal, type    Unmovable      1      1      1      0      1      1      1      1      2      0      1      0 
Node    0, zone   Normal, type      Movable      1      1      1      2      0      1      0      0      0      1      1      4 
Node    0, zone   Normal, type  Reclaimable      0      0      0      1      0      1      1      1      1      1      0      0 
Node    0, zone   Normal, type   HighAtomic      0      0      0      0      0      0      0      0      0      0      0      0 
Node    0, zone   Normal, type          CMA      1      0      1      1      1      1      0      1      1      1      1      2 
Node    0, zone   Normal, type      Isolate      0      0      0      0      0      0      0      0      0      0      0      0 

Number of blocks type     Unmovable      Movable  Reclaimable   HighAtomic          CMA      Isolate 
Node 0, zone   Normal            4           24            1            0            3            0 

After I run the test program it looks like this:

Page block order: 11
Pages per block:  2048

Free pages count per migrate type at order       0      1      2      3      4      5      6      7      8      9     10     11
Node    0, zone   Normal, type    Unmovable    100     45     14    781      1      0      0      0      0      0      0      0
Node    0, zone   Normal, type      Movable      6      0      0      0      0      0      0      0      0      0      0      0
Node    0, zone   Normal, type  Reclaimable      3      0      0      0      0      0      0      0      0      0      0      0
Node    0, zone   Normal, type   HighAtomic      0      0      0      0      0      0      0      0      0      0      0      0
Node    0, zone   Normal, type          CMA      5      2      1      1      1      1      1      0      1      1      1      2
Node    0, zone   Normal, type      Isolate      0      0      0      0      0      0      0      0      0      0      0      0

Number of blocks type     Unmovable      Movable  Reclaimable   HighAtomic          CMA      Isolate
Node 0, zone   Normal           10           19            0            0            3            0 

It is clear that the memory has been very fragmented after the test. If I run this on the 4.4 kernel I don't get this fragmentation.