fork13 -i 1000000 failed on Keystone2 board with arago kernel

Hi, Guoqing,

Our system test logs from for MCSDK3.0.0.11 automation test show that fork13 was passed (see below). I'll see if I can reproduce the failure on keystone-2 EVM.

<<<test_start>>>
tag=fork13 stime=1366720536
cmdline="fork13 -i 1000000"
contacts=""
analysis=exit
<<<test_output>>>
fork13      1  TPASS  :  1000000 pids forked, all passed
<<<execution_status>>>
initiation_status="ok"
duration=675 termination_type=exited termination_id=0 corefile=no
cutime=784 cstime=25786
<<<test_end>>>

Rex

Hi, Guoqing,

I ran either the test script or individual fork13 command, and none of them failed. You mentioned you get the kernel from arago git server. Did you run the test on the image you built? Could you install the mcsdk_3_00_00_11_setuplinux.bin from http://software-dl.ti.com/sdoemb/sdoemb_public_sw/mcsdk/latest/index_FDS.html and run the test agaist that kernel image? Also, did you get your ltp from open source? Could I have your fork13? Attached is my fork13 (please rename it before running it) and result which I have run a few times without any issues.

5621.fork13.txt

root@keystone-evm:/opt/ltp/testcases/bin# ./fork13 -i 1000000
fork13      1  TPASS  :  1000000 pids forked, all passed
root@keystone-evm:

Rex

Hi Rex,

Thanks for you reply.

Where is the source tree of the kernel image under ti/mcsdk_linux_3_00_00_11/images, I thought all the kernel are built from arago git tree. I tried the mcsdk_linux_3_00_00_11/images/uImage-keystone-evm.bin and mcsdk_linux_3_00_00_11/images/uImage-k2hk-evm.dtb, but it doesn't work well too.

root@localhost:/opt/ltp/testcases/bin# uname -a
Linux localhost 3.8.4 #1 SMP Fri May 31 00:00:49 EDT 2013 armv7l GNU/Linux
root@localhost:/opt/ltp/testcases/bin# ./fork13 -i 1000000
fork13      1  TFAIL  :  Unexpected exit status 0, iteration 21302.
root@localhost:/opt/ltp/testcases/bin# ./fork13 -i 1000000
[  103.330959] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[  103.330959]
[  103.340102] [<c0015260>] (unwind_backtrace+0x0/0xf8) from [<c04722d4>] (panic+0x90/0x1e4)
[  103.348268] [<c04722d4>] (panic+0x90/0x1e4) from [<c0025c88>] (do_exit+0x818/0x8ac)
[  103.355910] [<c0025c88>] (do_exit+0x818/0x8ac) from [<c0025e80>] (do_group_exit+0x3c/0xd0)
[  103.364157] [<c0025e80>] (do_group_exit+0x3c/0xd0) from [<c0033884>] (get_signal_to_deliver+0x194/0x60c)
[  103.373615] [<c0033884>] (get_signal_to_deliver+0x194/0x60c) from [<c0010ec0>] (do_signal+0x7c/0x480)
[  103.382814] [<c0010ec0>] (do_signal+0x7c/0x480) from [<c001175c>] (do_work_pending+0x68/0xa8)
[  103.391320] [<c001175c>] (do_work_pending+0x68/0xa8) from [<c000db60>] (work_pending+0xc/0x20)
[  103.399908] CPU1: stopping
[  103.402626] [<c0015260>] (unwind_backtrace+0x0/0xf8) from [<c0013834>] (handle_IPI+0x11c/0x160)
[  103.411304] [<c0013834>] (handle_IPI+0x11c/0x160) from [<c000854c>] (gic_handle_irq+0x54/0x5c)
[  103.419897] [<c000854c>] (gic_handle_irq+0x54/0x5c) from [<c0477f00>] (__irq_svc+0x40/0x50)
[  103.428219] Exception stack(0xdc0b9cd0 to 0xdc0b9d18)
[  103.433255] 9cc0:                                     00000000 00000000 0001a168 00000000
[  103.441409] 9ce0: c06e8900 dd9d3a40 00000000 00000028 de4f4f50 00000008 de4f501c ddaacf00
[  103.449559] 9d00: 00000000 dc0b9d18 c0098a60 c0098a9c 60000113 ffffffff
[  103.456161] [<c0477f00>] (__irq_svc+0x40/0x50) from [<c0098a9c>] (filemap_fault+0xf4/0x410)
[  103.464492] [<c0098a9c>] (filemap_fault+0xf4/0x410) from [<c00b610c>] (__do_fault+0x68/0x538)
[  103.472995] [<c00b610c>] (__do_fault+0x68/0x538) from [<c00b86ec>] (handle_pte_fault+0x84/0x1024)
[  103.481843] [<c00b86ec>] (handle_pte_fault+0x84/0x1024) from [<c00ba698>] (handle_mm_fault+0xec/0x154)
[  103.491126] [<c00ba698>] (handle_mm_fault+0xec/0x154) from [<c0479648>] (do_page_fault+0x284/0x3b8)
[  103.500148] [<c0479648>] (do_page_fault+0x284/0x3b8) from [<c00083fc>] (do_DataAbort+0x2c/0x94)
[  103.508821] [<c00083fc>] (do_DataAbort+0x2c/0x94) from [<c0478034>] (__dabt_usr+0x34/0x40)
[  103.517055] Exception stack(0xdc0b9fb0 to 0xdc0b9ff8)
[  103.522089] 9fa0:                                     00008794 00000000 710e0d60 00008798
[  103.530240] 9fc0: 7c967e3f b6fa7000 00000000 be959b80 00000000 00000000 00008794 be959c2c
[  103.538390] 9fe0: 00000000 be959b60 4c1fb267 4c1f7b08 60000030 ffffffff
[  103.544982] CPU3: stopping
[  103.547697] [<c0015260>] (unwind_backtrace+0x0/0xf8) from [<c0013834>] (handle_IPI+0x11c/0x160)
[  103.556373] [<c0013834>] (handle_IPI+0x11c/0x160) from [<c000854c>] (gic_handle_irq+0x54/0x5c)
[  103.564965] [<c000854c>] (gic_handle_irq+0x54/0x5c) from [<c0477f00>] (__irq_svc+0x40/0x50)
[  103.573287] Exception stack(0xde8e9f90 to 0xde8e9fd8)
[  103.578322] 9f80:                                     c0b0cb20 00000000 00006ad2 00000000
[  103.586475] 9fa0: de8e8000 de8e8000 c06b0e08 de8e8000 c0682450 412fc0f4 00000000 00000000
[  103.594625] 9fc0: 00000000 de8e9fd8 c000eafc c000eb00 60000013 ffffffff
[  103.601227] [<c0477f00>] (__irq_svc+0x40/0x50) from [<c000eb00>] (default_idle+0x28/0x30)
[  103.609384] [<c000eb00>] (default_idle+0x28/0x30) from [<c000ec44>] (cpu_idle+0x64/0xa0)
[  103.617454] [<c000ec44>] (cpu_idle+0x64/0xa0) from [<80657a28>] (0x80657a28)
[  103.624479] CPU0: stopping
[  103.627195] [<c0015260>] (unwind_backtrace+0x0/0xf8) from [<c0013834>] (handle_IPI+0x11c/0x160)
[  103.635871] [<c0013834>] (handle_IPI+0x11c/0x160) from [<c000854c>] (gic_handle_irq+0x54/0x5c)
[  103.644462] [<c000854c>] (gic_handle_irq+0x54/0x5c) from [<c0477f00>] (__irq_svc+0x40/0x50)
[  103.652785] Exception stack(0xc0675f60 to 0xc0675fa8)
[  103.657823] 5f60: c0af4b20 00000000 00054d5a 00000000 c0674000 c0674000 c06b0e08 c0674000
[  103.665975] 5f80: c0682450 412fc0f4 00000000 00000000 00000000 c0675fa8 c000eafc c000eb00
[  103.674124] 5fa0: 60000013 ffffffff
[  103.677609] [<c0477f00>] (__irq_svc+0x40/0x50) from [<c000eb00>] (default_idle+0x28/0x30)
[  103.685766] [<c000eb00>] (default_idle+0x28/0x30) from [<c000ec44>] (cpu_idle+0x64/0xa0)
[  103.693841] [<c000ec44>] (cpu_idle+0x64/0xa0) from [<c06337f8>] (start_kernel+0x2ec/0x340)

And I also run mcsdk image (same as above) with your fork13 under mcsdk filesystem, it also has problem as follows.

root@keystone-evm:~# uname -a
Linux keystone-evm 3.8.4 #1 SMP Fri May 31 00:00:49 EDT 2013 armv7l GNU/Linux
root@keystone-evm:~# ./fork13 -i 1000000
fork13      1  TBROK  :  unexpected signal 11 received (pid = 1577).
fork13      2  TBROK  :  Remaining cases broken
root@keystone-evm:~# ./fork13 -i 1000000
[  333.400437] Alignment trap: fork13 (15244) PC=0xb6ec1278 Instr=0xf8452200 Address=0xffffffff FSR 0xa21
[  333.409717] Alignment trap: not handling ldm with s-bit set
[  333.415289] Alignment trap: not handling instruction f8452200 at [<b6ec1278>]
[  333.422415] Unhandled fault: alignment exception (0xa21) at 0xffffffff
fork13      1  T[  333.422635] Alignment trap: fork13 (15244) PC=0xb6eb0076 Instr=0xe8501f00 Address=0x0000000b FSR 0x221
FAIL  :  Unexpec[  333.422640] Alignment trap: not handling instruction e8501f00 at [<b6eb0076>]
ted exit status [  333.422645] Unhandled fault: alignment exception (0x221) at 0x0000000b
0, iteration 79799.
2013 May 31 10:01:22 keystone-evm [  333.422415] Unhandled fault: alignment exception (0xa21) at 0xffffffff
2013 May 31 10:01:22 keystone-evm [  333.422645] Unhandled fault: alignment exception (0x221) at 0x0000000b

I attached my fork13 here  0003.fork13.txt

Do you run smp kernel or up kernel on the board? I know up kernel can pass this test.

PS: I used the following cmd to boot kernel, pls let me know if something wrong here, thanks again.

set fdt_high 0xffffffff;set serverip 128.224.163.23;set ipaddr 128.224.162.22;setenv bootargs root=/dev/nfs console=ttyS0,115200n8 rootwait=1 rw init=/linuxrc nfsroot=128.224.163.23:${nfsroot} earlyprintk ip=dhcp;tftpboot 0x88000000 uImage;tftpboot 0x87000000 k2hk-evm.dtb;tftpboot 0xc5f0000 skern-keystone-evm.bin;mon_install 0x0c5f0000;bootm 0x88000000 - 0x87000000

Regards,

Guoqing

Hi, Guoqing,

Does my fork13 run on your EVM and what is the result?

Your fork13 does not run on my EVM which is using the 3.00.00.11 (GA version) file system. I changed to beta version file system and it runs successfully. That says you are using 2012.03 Linaro compiler instead of 2013.03. Could you confirm that you are using the older version of compiler and also older version of file system?

You should be able to run the pre-build image which is what I do and it is SMP kernel. Both my fork13 and your fork13 (running on old file system) run successfully.

root@tci6614-evm:/opt/ltp/testcases/bin# ls 0003.fork13
0003.fork13
root@tci6614-evm:/opt/ltp/testcases/bin# ./0003.fork13 -i 1000000
fork13      1  TPASS  :  1000000 pids forked, all passed
root@tci6614-evm:/opt/ltp/testcases/bin# uname -a
Linux tci6614-evm 3.8.4 #1 SMP Fri May 31 00:00:49 EDT 2013 armv7l unknown
root@tci6614-evm:/opt/ltp/testcases/bin#

I suggest you to follow Keystone 2 User's Guide for the u-boot env setting to get a baseline first. "env default -f -a" command resets the env variables, and you can change boot using nfs "setenv boot net" and corresponding nfs_root, serverip, etc.

Rex

Hi Rex,

Sorry, your fork13 can't run with my rootfs, it can only works with mcsdk rootfs.
I re-run your fork13 with mcsdk fs + mcsdk pre-build images, also have encountered
failed result as before, and the case can't always pass even it maybe pass at some
time.

root@keystone-evm:~# ./fork13 -i 1000000
fork13      1  TPASS  :  1000000 pids forked, all passed
root@keystone-evm:~# ./fork13 -i 1000000
[ 1123.662820] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000004
[ 1123.662820]
[ 1123.671967] [<c0015260>] (unwind_backtrace+0x0/0xf8) from [<c04722d4>] (panic+0x90/0x1e4)
[ 1123.680140] [<c04722d4>] (panic+0x90/0x1e4) from [<c0025c88>] (do_exit+0x818/0x8ac)
[ 1123.687792] [<c0025c88>] (do_exit+0x818/0x8ac) from [<c0025e80>] (do_group_exit+0x3c/0xd0)
[ 1123.696042] [<c0025e80>] (do_group_exit+0x3c/0xd0) from [<c0033884>] (get_signal_to_deliver+0x194/0x60c)
[ 1123.705505] [<c0033884>] (get_signal_to_deliver+0x194/0x60c) from [<c0010ec0>] (do_signal+0x7c/0x480)
[ 1123.714708] [<c0010ec0>] (do_signal+0x7c/0x480) from [<c001175c>] (do_work_pending+0x68/0xa8)
[ 1123.723224] [<c001175c>] (do_work_pending+0x68/0xa8) from [<c000db60>] (work_pending+0xc/0x20)
[ 1123.731815] CPU0: stopping
[ 1123.734531] [<c0015260>] (unwind_backtrace+0x0/0xf8) from [<c0013834>] (handle_IPI+0x11c/0x160)
[ 1123.743208] [<c0013834>] (handle_IPI+0x11c/0x160) from [<c000854c>] (gic_handle_irq+0x54/0x5c)
[ 1123.751800] [<c000854c>] (gic_handle_irq+0x54/0x5c) from [<c0477f00>] (__irq_svc+0x40/0x50)
[ 1123.760124] Exception stack(0xdd9b5ee8 to 0xdd9b5f30)
[ 1123.765161] 5ee0:                   dd017b80 00000000 00000000 c0676040 dd9b4000 01200011
[ 1123.773313] 5f00: de298e00 dd017c90 c06b1480 dd017b80 de9f5800 00000000 00000000 dd9b5f30
[ 1123.781462] 5f20: c001f6b4 c001f71c 60000013 ffffffff
[ 1123.786506] [<c0477f00>] (__irq_svc+0x40/0x50) from [<c001f71c>] (copy_process.part.62+0xb44/0xe08)
[ 1123.795528] [<c001f71c>] (copy_process.part.62+0xb44/0xe08) from [<c001fa94>] (do_fork+0x90/0x2bc)
[ 1123.804464] [<c001fa94>] (do_fork+0x90/0x2bc) from [<c000db20>] (ret_fast_syscall+0x0/0x30)
[ 1123.812788] CPU3: stopping
[ 1123.815502] [<c0015260>] (unwind_backtrace+0x0/0xf8) from [<c0013834>] (handle_IPI+0x11c/0x160)
[ 1123.824180] [<c0013834>] (handle_IPI+0x11c/0x160) from [<c000854c>] (gic_handle_irq+0x54/0x5c)
[ 1123.832773] [<c000854c>] (gic_handle_irq+0x54/0x5c) from [<c0477f00>] (__irq_svc+0x40/0x50)

What is the revision of your keystone2 evm board? The label on my board is "XTCIEVMK2X Rev 1.0",
pls see the attached photo of the board.

From /proc/cpuinfo I got the following
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x2
CPU part        : 0xc0f
CPU revision    : 4

Hardware        : KeyStone2
Revision        : 0000
Serial          : 0000000000000000

BTW: Could you tell me what is the difference between pre-build kernel image and kernel buit from git tree?
Where can I get the source code of the pre-build kernel image?

Regards,
Guoqing

Hi, Guoqing,

What's the difference between your file system and that in mcsdk release? Before I post my last reply, I had fork13 ran 3 times and they were all successful. My EVM is Rev 1.0. When MCSDK is release, the package does not include source code and they need to get from arago git server. Instructions are in the User's Guide. There isn't any difference between pre-build and built from git tree, but we exclude any possible change from customer's site. I can give it a try to run fork13 multiple times to see if the issue happens.

Rex

Hi Rex,

Not know the exactly differences, but for ltp I think there are no big gap.
After created ld-linux-armhf.so.3 link under /lib, I can run your fork13 within my fs, but it still failed as follows:

root@localhost:~# uname -a
Linux localhost 3.8.4 #1 SMP Fri May 31 00:00:49 EDT 2013 armv7l GNU/Linux

root@localhost:~# ls -l
total 40
-rw-r--r-- 1 root root 18508 Jul 17 10:52 5621.fork13.txt
-rwxr-xr-x 1 root root 18508 Jul 18 13:50 fork13

root@localhost:~# ./fork13 -i 1000000
fork13      1  T[  124.983621] Alignment trap: fork13 (10852) PC=0x4c2767ca Instr=0xe8540f00 Address=0xffffffff FSR 0x221
BROK  :  unexpec[  124.994221] Alignment trap: not handling instruction e8540f00 at [<4c2767ca>]
ted signal 11 re[  125.002697] Unhandled fault: alignment exception (0x221) at 0xffffffff
ceived (pid = 10[  125.010682] Alignment trap: fork13 (10852) PC=0x4c2767ca Instr=0xe8540f00 Address=0xffffffff FSR 0x221
852).
fork13   [  125.021244] Alignment trap: not handling instruction e8540f00 at [<4c2767ca>]
   2  TBROK  :  [  125.029725] Unhandled fault: alignment exception (0x221) at 0xffffffff
Remaining cases broken
fork13      3  TBROK  :  unexpected signal 7 received (pid = 10852).
fork13      1  TFAIL  :  Unexpected exit status 0, iteration 140154.

or the below.

root@localhost:~# ./fork13 -i 1000000
fork13      1  TFAIL  :  Unexpected exit status 0, iteration 401743.

If possible, could you share me the source code of fork13? thanks.

Regards,
Guoqing

Hi, Guoqing,

Sorry, I don't have source code, but we'll take a look at this further. I'll let you know what we get.

Rex

Hi, Guoqing,

Did you change the frequencies of the EVM and what frequencies is your EVM running at? Could you send me the outputs from the commands below in u-boot prompt?

getclk 0

getclk 2

Also, how many successful runs do you get before it starts failing?

Thanks!

Rex

Hi Rex,

Pls see the following output:

TCI6638 EVM # getclk 0
clock index [0] - frequency 798720000
TCI6638 EVM # getclk 2
clock index [2] - frequency 625000000

And the test result is not stable so it is hard to tell the exact successfully run numbers.
For example, I got kernel hung when I first run the case, I also got below result after two
successfully runs.

root@localhost:~# ./fork13 -i 1000000
fork13      1  TFAIL  :  Unexpected exit status 0, iteration 366627.

BTW: 3.4 kernel runs this case well so far with my own fork13, I'll run more tests with it.

Thanks,
Guoqing

Hi, Guoqing,

We are looking at this and trying to exclude non-related variables. We are able to reproduce this issue with multiple runs but don't have any conclusion yet. Your info are very valuable to us. If I am correct, MCSDK pre-GA is using kernel 3.6, and 3.8 in GA. What MCSDK version is kernel 3.4 in, or is this on non-Keystone-2 silicon?

Rex

Hi Rex,

Thanks for confirming this issue, do you have plan to fix it?

The 3.4 kernel is our own version which just finished the port
for keystone2, I suggest enable 720789 errata and disable 798181
errata in your kernel config, It works with 3.4 kernel but I'm
still confused since 720789 errata is apply to Cortex-A9 (prior
to r2p0) and 798181 errata is apply to Cortex-A15 (r0p0..r3p2),
that is to say, 798181 should work for keystone2 since the version
is a15 r2p4, but the fact is that I only need 720789 not 798181.

Regards,
Guoqing

Hi, Guoqing,

We have tried to disable the SMP in the bootargs, and was able to run the test successfully for multiple times with released kernel image. Yes, we plan to fix this. We'll look into the errata and thanks for the suggestion.

Rex

Hi Rex,

Right, UP is ok as I had said before, but for SMP the circumstance is more complicated and there must have incoherency problems among cores.

Pls update me if you have some new discovery.

Regards,
Guoqing