This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Linux/AM3358: MPU hangs

Part Number: AM3358
Other Parts Discussed in Thread: AM3352, TPS65910

Tool/software: Linux

Hello, I am newbie here.

We developed the custom board based on AM335x board. Everything run well. But after a random amount of time, CPU freezed without any log. 

Platform: AM335x

SDK: TI processor SDK 04.02.00.09

OSC0 frequency: 24M Hz.

DDR frequency: 400M Hz.

W followed this article: processors.wiki.ti.com/.../Sitara_Linux_Training:_Tuning_the_DDR3_Timings_on_BeagleBoneBlack

We used kingston's D2516EC4BXGGB 4G bits DDR3L SDRAM which is used by Beagle Bone Black.

We attched DDR3L schematics.

DS_CBN_CORE_R0.1_17.08.25-1.pdf

P.S The odd things is that the custom board is working without freezing if OSC0 frequency is 24M Hz but SYSBOOT[15:14] is 26MHz.

I would appreciate your advice.

 

  • Hi,

    Please check if Errata Advisory 1.0.30 workaround is implemented on your board. By the way it's absolutely incorrect to configure SYSBOOT[15:14] for 26MHz if the crystal you use is 24MHz.
  • Thank you for your advice. We applied Errata Advisory 1.0.30 workaround.

    We attached the am3358 core schematics.3480.DS_CBN_CORE_R0.1_17.08.25-1.pdf


    Can you give me any other advice?

  • On this sheet there is nothing else that looks wrong. Check that you have external 4.7kOHM pullups to VDDSHV6 on EMU0 and EMU1.
  • We already had external 4.7kOHM pullups to VDDSHV6 on EMU0 and EMU1.

    We used kingston's D2516EC4BXGGB 4G bits DDR3L SDRAM in our custom board instead of MT41K256M16HA-125. The value of memory data symbol of MT41K256M16HA-125 seems to be same as kingston's D2516EC4BXGGB 4G's. And so we filled up "AM35X DDR Timing Configuration Tool" sheet according to MT41K256M16HA-125 data sheet because we can't find all memory data symbols in kingston's D2516EC4BXGGB data sheet. Is there any problem? I attached "AM35X DDR Timing Configuration Tool" sheet.(please, refer to AM335x-DDR3_400Mhz sheet)AM335x_DDR_register_calc_tool_20180319.xls

  • The fact that you can boot to kernel means that this is not very likely to be a DDR leveling/timing issue. Did you perform the EDMA memory test as described here (step 5): processors.wiki.ti.com/.../Sitara_Linux_Training:_Tuning_the_DDR3_Timings_on_BeagleBoneBlack
  • By the way, one thing I noticed is that you have 33Ohm termination resistors on the DDR. This is not correct. They should be either 47Ohm or 51Ohm, and precise 50Ohm resistors on the clock lines.
  • We were dubugging deeper. We found that CPU hang under certain conditions.

    If i run ssh-keygen in arago filesystem, CPU hang.

    We need the specific version of openssh in arago filesystem . And so we cross compile openssh in host computer(ubuntu 16.04). I described the procedures below.

    - cross compile environment setup
    $ source ti-processor-sdk-linux-am335x-evm-04.02.00.09/linux-devkit/environment-setup

    - zlib-1.2.3
    $ LAGS="-Os" ./configure --shared
    $ make

    - openssl-1.0.2g
    $ ./Configure linux-generic32 shared -DL_ENDIAN -DHAVE_CRYPTODEV -DUSE_CRYPTDEV_DIGESTS
    $ make

    - openssh-4.6p1
    $ ./configure --host=arm-linux --with-libs --with-zlib=/home/reti/kkch/zlib-1.2.3 --with-ssl-dir=/home/reti/kkch/openssl-1.0.2g --disable-etc-default-login CC=arm-linux-gnueabihf-gcc AR=arm-linux-gnueabihf-ar
    $ make

    -And then I copied the compiled openssh to target board.
    -If run ssh-keygen in target board, CPU hang


    Can this be said to be caused by memory configuration? Or is it another matter?
    I would appreciate your advice.

  • Thanks. I have notified the software team. They will continue from here.
  • Ethan,

    Please attach the full kernel boot log, and the output of "lsmod" command.
  • Thanks for your interest.

    We attached kernel boot log and insmod output. 

    1.-------------------------------------- kernel boot log --------------------------------------

    U-Boot SPL 2016.05-00118-gabc6b2a-dirty (Mar 20 2018 - 17:59:04)
    Trying to boot from MMC1
    reading args
    spl_load_image_fat_os: error reading image args, err - -1
    reading u-boot.img
    reading u-boot.img
    reading u-boot.img
    reading u-boot.img


    U-Boot 2016.05-00118-gabc6b2a-dirty (Mar 20 2018 - 17:59:04 +0900)

    Watchdog enabled
    DRAM: 512 MiB
    NAND: 0 MiB
    MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1
    reading uboot.env

    ** Unable to read "uboot.env" from mmc0:1 **
    Using default environment

    <ethaddr> not set. Validating first E-fuse MAC
    Net: eth0: ethernet@4a100000
    Warning: usb_ether MAC addresses don't match:
    Address in SROM is de:ad:be:ef:00:01
    Address in environment is a8:1b:6a:33:52:e8
    , eth1: usb_ether
    switch to partitions #0, OK
    mmc0 is current device
    SD/MMC found on device 0
    reading boot.scr
    ** Unable to read file boot.scr **
    reading uEnv.txt
    ** Unable to read file uEnv.txt **
    switch to partitions #0, OK
    mmc0 is current device
    Scanning mmc 0:1...
    switch to partitions #0, OK
    mmc0 is current device
    SD/MMC found on device 0
    switch to partitions #0, OK
    mmc0 is current device
    SD/MMC found on device 0
    reading boot.scr
    ** Unable to read file boot.scr **
    reading uEnv.txt
    ** Unable to read file uEnv.txt **
    3621296 bytes read in 272 ms (12.7 MiB/s)
    41541 bytes read in 38 ms (1 MiB/s)
    Kernel image @ 0x82000000 [ 0x000000 - 0x3741b0 ]
    ## Flattened Device Tree blob at 88000000
    Booting using the fdt blob at 0x88000000
    Loading Device Tree to 8fff2000, end 8ffff244 ... OK

    Starting kernel ...

    [ 0.000000] Booting Linux on physical CPU 0x0
    [ 0.000000] Linux version 4.9.59-ga75d8e9305 (gtbldadm@ubuntu-16) (gcc version 6.2.1 20161016 (Linaro GCC 6.2-2016.11) ) #1 PREEMPT Sat Dec 23 09:46:08 EST 2017
    [ 0.000000] CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c5387d
    [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
    [ 0.000000] OF: fdt:Machine model: TI AM335x EVM-SK
    [ 0.000000] efi: Getting EFI parameters from FDT:
    [ 0.000000] efi: UEFI not found.
    [ 0.000000] cma: Reserved 48 MiB at 0x9d000000
    [ 0.000000] Memory policy: Data cache writeback
    [ 0.000000] CPU: All CPU(s) started in SVC mode.
    [ 0.000000] AM335X ES2.1 (neon)
    [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 129920
    [ 0.000000] Kernel command line: console=ttyO0,115200n8 root=PARTUUID=560100e7-02 rw rootfstype=ext4 rootwait
    [ 0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
    [ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
    [ 0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
    [ 0.000000] Memory: 457960K/524288K available (7168K kernel code, 281K rwdata, 2420K rodata, 1024K init, 280K bss, 17176K reserved, 49152K cma-reserved, 0K highmem)
    [ 0.000000] Virtual kernel memory layout:
    [ 0.000000] vector : 0xffff0000 - 0xffff1000 ( 4 kB)
    [ 0.000000] fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
    [ 0.000000] vmalloc : 0xe0800000 - 0xff800000 ( 496 MB)
    [ 0.000000] lowmem : 0xc0000000 - 0xe0000000 ( 512 MB)
    [ 0.000000] pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
    [ 0.000000] modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
    [ 0.000000] .text : 0xc0008000 - 0xc0800000 (8160 kB)
    [ 0.000000] .init : 0xc0b00000 - 0xc0c00000 (1024 kB)
    [ 0.000000] .data : 0xc0c00000 - 0xc0c465d0 ( 282 kB)
    [ 0.000000] .bss : 0xc0c465d0 - 0xc0c8c834 ( 281 kB)
    [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
    [ 0.000000] Preemptible hierarchical RCU implementation.
    [ 0.000000] Build-time adjustment of leaf fanout to 32.
    [ 0.000000] NR_IRQS:16 nr_irqs:16 16
    [ 0.000000] IRQ: Found an INTC at 0xfa200000 (revision 5.0) with 128 interrupts
    [ 0.000000] OMAP clockevent source: timer2 at 24000000 Hz
    [ 0.000014] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns
    [ 0.000033] clocksource: timer1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
    [ 0.000042] OMAP clocksource: timer1 at 24000000 Hz
    [ 0.000210] clocksource_probe: no matching clocksources found
    [ 0.000381] Console: colour dummy device 80x30
    [ 0.000407] WARNING: Your 'console=ttyO0' has been replaced by 'ttyS0'
    [ 0.000413] This ensures that you still see kernel messages. Please
    [ 0.000417] update your kernel commandline.
    [ 0.000440] Calibrating delay loop... 996.14 BogoMIPS (lpj=4980736)
    [ 0.089189] pid_max: default: 32768 minimum: 301
    [ 0.089315] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
    [ 0.089326] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
    [ 0.090109] CPU: Testing write buffer coherency: ok
    [ 0.090500] Setting up static identity map for 0x80100000 - 0x80100060
    [ 0.091296] EFI services will not be available.
    [ 0.092480] devtmpfs: initialized
    [ 0.103961] VFP support v0.3: implementor 41 architecture 3 part 30 variant c rev 3
    [ 0.104299] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
    [ 0.104325] futex hash table entries: 256 (order: -1, 3072 bytes)
    [ 0.107965] pinctrl core: initialized pinctrl subsystem
    [ 0.109405] NET: Registered protocol family 16
    [ 0.111212] DMA: preallocated 256 KiB pool for atomic coherent allocations
    [ 0.124327] omap_hwmod: debugss: _wait_target_disable failed
    [ 0.199180] cpuidle: using governor ladder
    [ 0.229168] cpuidle: using governor menu
    [ 0.234875] OMAP GPIO hardware version 0.1
    [ 0.249965] No ATAGs?
    [ 0.249991] hw-breakpoint: debug architecture 0x4 unsupported.
    [ 0.286914] edma 49000000.edma: TI EDMA DMA engine driver
    [ 0.287820] reg-fixed-voltage fixedregulator2: could not find pctldev for node /ocp/l4_wkup@44c00000/scm@210000/pinmux@800/pinmux_wl12xx_gpio, deferring probe
    [ 0.290992] omap_i2c 44e0b000.i2c: could not find pctldev for node /ocp/l4_wkup@44c00000/scm@210000/pinmux@800/pinmux_i2c0_pins, deferring probe
    [ 0.291152] media: Linux media interface: v0.10
    [ 0.291213] Linux video capture interface: v2.00
    [ 0.291252] pps_core: LinuxPPS API ver. 1 registered
    [ 0.291259] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
    [ 0.291282] PTP clock support registered
    [ 0.291317] EDAC MC: Ver: 3.0.0
    [ 0.292396] omap-mailbox 480c8000.mailbox: omap mailbox rev 0x400
    [ 0.292732] Advanced Linux Sound Architecture Driver Initialized.
    [ 0.293914] clocksource: Switched to clocksource timer1
    [ 0.303435] NET: Registered protocol family 2
    [ 0.304264] TCP established hash table entries: 4096 (order: 2, 16384 bytes)
    [ 0.304312] TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
    [ 0.304351] TCP: Hash tables configured (established 4096 bind 4096)
    [ 0.304429] UDP hash table entries: 256 (order: 0, 4096 bytes)
    [ 0.304445] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
    [ 0.304569] NET: Registered protocol family 1
    [ 0.304990] RPC: Registered named UNIX socket transport module.
    [ 0.305003] RPC: Registered udp transport module.
    [ 0.305009] RPC: Registered tcp transport module.
    [ 0.305015] RPC: Registered tcp NFSv4.1 backchannel transport module.
    [ 0.305948] hw perfevents: enabled with armv7_cortex_a8 PMU driver, 5 counters available
    [ 0.308210] workingset: timestamp_bits=14 max_order=17 bucket_order=3
    [ 0.315387] squashfs: version 4.0 (2009/01/31) Phillip Lougher
    [ 0.316294] NFS: Registering the id_resolver key type
    [ 0.316340] Key type id_resolver registered
    [ 0.316347] Key type id_legacy registered
    [ 0.316390] ntfs: driver 2.1.32 [Flags: R/O].
    [ 0.318193] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
    [ 0.318213] io scheduler noop registered
    [ 0.318219] io scheduler deadline registered
    [ 0.318351] io scheduler cfq registered (default)
    [ 0.319699] pinctrl-single 44e10800.pinmux: 142 pins at pa f9e10800 size 568
    [ 0.322808] backlight supply power not found, using dummy regulator
    [ 0.378065] Serial: 8250/16550 driver, 10 ports, IRQ sharing disabled
    [ 0.381720] 44e09000.serial: ttyS0 at MMIO 0x44e09000 (irq = 158, base_baud = 3000000) is a 8250
    [ 0.982882] console [ttyS0] enabled
    [ 0.988192] omap_rng 48310000.rng: OMAP Random Number Generator ver. 20
    [ 0.995057] [drm] Initialized
    [ 0.999068] panel panel: found backlight
    [ 1.004051] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
    [ 1.010706] [drm] No driver support for vblank timestamp query.
    [ 1.042386] Console: switching to colour frame buffer device 60x34
    [ 1.050316] tilcdc 4830e000.lcdc: fb0: frame buffer device
    [ 1.097469] brd: module loaded
    [ 1.106847] loop: module loaded
    [ 1.112394] libphy: Fixed MDIO Bus: probed
    [ 1.183953] davinci_mdio 4a101000.mdio: davinci mdio revision 1.6
    [ 1.190093] davinci_mdio 4a101000.mdio: detected phy mask fffffffc
    [ 1.197792] libphy: 4a101000.mdio: probed
    [ 1.201901] davinci_mdio 4a101000.mdio: phy[0]: device 4a101000.mdio:00, driver SMSC LAN8710/LAN8720
    [ 1.211158] davinci_mdio 4a101000.mdio: phy[1]: device 4a101000.mdio:01, driver SMSC LAN8710/LAN8720
    [ 1.221159] cpsw 4a100000.ethernet: Detected MACID = a8:1b:6a:33:52:e6
    [ 1.228064] cpsw 4a100000.ethernet: device node lookup for pps timer failed
    [ 1.235162] cpsw 4a100000.ethernet: cpts: overflow check period 500 (jiffies)
    [ 1.243388] cpsw 4a100000.ethernet: cpsw: Detected MACID = a8:1b:6a:33:52:e8
    [ 1.252094] mousedev: PS/2 mouse device common for all mice
    [ 1.258192] i2c /dev entries driver
    [ 1.263500] cpuidle: enable-method property 'ti,am3352' found operations
    [ 1.271409] omap_hsmmc 48060000.mmc: Got CD GPIO
    [ 1.278758] ledtrig-cpu: registered to indicate activity on CPUs
    [ 1.287872] NET: Registered protocol family 10
    [ 1.293804] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
    [ 1.300710] NET: Registered protocol family 17
    [ 1.305724] Key type dns_resolver registered
    [ 1.310347] omap_voltage_late_init: Voltage driver support not added
    [ 1.354139] tps65910 0-002d: No interrupt support, no core IRQ
    [ 1.362017] vrtc: supplied by vbat
    [ 1.369292] vio: supplied by vbat
    [ 1.374164] vdd1: supplied by vbat
    [ 1.379492] vdd2: supplied by vbat
    [ 1.385529] random: fast init done
    [ 1.389625] vdig1: supplied by vbat
    [ 1.394536] vdig2: supplied by vbat
    [ 1.399490] vpll: supplied by vbat
    [ 1.404359] vdac: supplied by vbat
    [ 1.409187] vaux1: supplied by vbat
    [ 1.414143] vaux2: supplied by vbat
    [ 1.419060] vaux33: supplied by vbat
    [ 1.424081] vmmc: supplied by vbat
    [ 1.428895] vbb: supplied by vbat
    [ 1.433798] omap_i2c 44e0b000.i2c: bus 0 rev0.11 at 400 kHz
    [ 1.441303] omap_hsmmc 48060000.mmc: Got CD GPIO
    [ 1.551338] mmc0: host does not support reading read-only switch, assuming write-enable
    [ 1.561462] mmc0: new high speed SDHC card at address 59b4
    [ 1.567914] mmcblk0: mmc0:59b4 SD 14.7 GiB
    [ 1.575199] mmcblk0: p1 p2
    [ 1.655809] input: gpio_buttons0 as /devices/platform/gpio_buttons0/input/input0
    [ 1.667202] hctosys: unable to open rtc device (rtc0)
    [ 1.673180] lis3_reg: disabling
    [ 1.676714] ALSA device list:
    [ 1.679816] No soundcards found.
    [ 3.044134] EXT4-fs (mmcblk0p2): recovery complete
    [ 3.064999] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
    [ 3.073289] VFS: Mounted root (ext4 filesystem) on device 179:2.
    [ 3.084109] devtmpfs: mounted
    [ 3.091436] Freeing unused kernel memory: 1024K
    [ 3.381417] systemd[1]: System time before build time, advancing clock.
    [ 3.428690] systemd[1]: systemd 230 running in system mode. (+PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP -GCRYPT -GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN)
    [ 3.447841] systemd[1]: Detected architecture arm.

    Welcome to Arago 2017.10!

    [ 3.486068] systemd[1]: Set hostname to <am335x-evm>.
    [ 3.798141] systemd[1]: [/lib/systemd/system/gadget-init.service:15] Unknown lvalue 'ExecStopPre' in section 'Service'
    [ 4.088459] systemd[1]: Created slice User and Session Slice.
    [ OK ] Created slice User and Session Slice.
    [ 4.124562] systemd[1]: Listening on Journal Socket (/dev/log).
    [ OK ] Listening on Journal Socket (/dev/log).
    [ 4.154564] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
    [ OK ] Started Dispatch Password Requests to Console Directory Watch.
    [ 4.194518] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.
    [ OK ] Listening on /dev/initctl Compatibility Named Pipe.
    [ 4.234234] systemd[1]: Reached target Remote File Systems.
    [ OK ] Reached target Remote File Systems.
    [ 4.264156] systemd[1]: Reached target Swap.
    [ OK ] Reached target Swap.
    [ OK ] Started Forward Password Requests to Wall Directory Watch.
    [ OK ] Reached target Paths.
    [ OK ] Listening on udev Control Socket.
    [ OK ] Listening on Syslog Socket.
    [ OK ] Listening on udev Kernel Socket.
    [ OK ] Listening on Network Service Netlink Socket.
    [ OK ] Created slice System Slice.
    [ OK ] Reached target Slices.
    [ OK ] Created slice system-serial\x2dgetty.slice.
    [ OK ] Created slice system-getty.slice.
    [ OK ] Listening on Process Core Dump Socket.
    [ OK ] Listening on Journal Socket.
    Mounting Debug File System...
    Mounting POSIX Message Queue File System...
    Starting Create Static Device Nodes in /dev...
    Starting alignment.service...
    Starting Setup Virtual Console...
    Starting Load Kernel Modules...
    Starting Remount Root and Kernel File Systems...
    Mounting Temporary Directory...
    [ 5.043445] cryptodev: loading out-of-tree module taints kernel.
    [ 5.067137] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
    Starting Journal Service...
    [ 5.126994] cryptodev: driver 1.8 loaded.
    [ OK ] Mounted Debug File System.
    [ OK ] Mounted POSIX Message Queue File System.
    [ OK ] Mounted Temporary Directory.
    [ OK ] Started Create Static Device Nodes in /dev.
    [ OK ] Started alignment.service.
    [ OK ] Started Setup Virtual Console.
    [ OK ] Started Load Kernel Modules.
    [ OK ] Started Remount Root and Kernel File Systems.
    [ OK ] Started Journal Service.
    Starting udev Coldplug all Devices...
    Starting Flush Journal to Persistent Storage...
    Mounting Configuration File System...
    Starting Apply Kernel Variables...
    Starting udev Kernel Device Manager...
    [ OK ] Reached target Local File Systems (Pre).
    Mounting /media/ram...
    Mounting /var/volatile...
    [ OK ] Mounted Configuration File System.
    [ OK ] Mounted /var/volatile.
    [ OK ] Mounted /media/ram.
    [ OK ] Started udev Kernel Device Manager.
    [ OK ] Started Apply Kernel Variables.
    [ 6.024390] systemd-journald[102]: Received request to flush runtime journal from PID 1
    [ OK ] Started Flush Journal to Persistent Storage.
    [ OK ] Reached target Local File Systems.
    Starting Create Volatile Files and Directories...
    Starting Load/Save Random Seed...
    [ OK ] Started Load/Save Random Seed.
    [ OK ] Started Create Volatile Files and Directories.
    Starting Network Time Synchronization...
    Starting Update UTMP about System Boot/Shutdown...
    [ OK ] Started Update UTMP about System Boot/Shutdown.
    [ OK ] Started Network Time Synchronization.
    [ OK ] Reached target System Time Synchronized.
    [ OK ] Created slice system-systemd\x2dbacklight.slice.
    Starting Load/Save Screen Backlight...htness of backlight:backlight...
    [ 7.524521] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
    [ OK ] Started Load/Save Screen Backlight Brightness of backlight:backlight.
    [ 7.596812] omap_rtc 44e3e000.rtc: already running
    [ OK ] Found device /dev/ttyS0.
    [ OK ] Started udev Coldplug all Devices.
    [ 7.656073] omap_rtc 44e3e000.rtc: rtc core: registered 44e3e000.rtc as rtc0
    [ OK ] Reached target System Initialization.
    [ OK ] Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
    [ OK ] Started Daily Cleanup of Temporary Directories.
    [ OK ] Reached target Timers.
    [ OK ] Listening on D-Bus System Message Bus Socket.
    [ OK ] Listening on RPCbind Server Activation Socket.
    [ OK ] Listening on dropbear.socket.
    [ OK ] Reached target Sockets.
    [ OK ] Reached target Basic System.
    Starting Avahi mDNS/DNS-SD Stack...
    [ OK ] Started Kernel Logging Service.
    [ OK ] Started System Logging Service.
    Starting uim-sysfs.service...
    Starting Login Service...
    [ OK ] Started D-Bus System Message Bus.
    [ 8.900901] omap_hwmod: pruss: _wait_target_ready failed: -16
    [ 9.020825] pruss-soc-bus 4a326000.pruss-soc-bus: use pm_runtime_put_sync_suspend() in driver?
    [ 9.116442] pruss-soc-bus 4a326000.pruss-soc-bus: couldn't enable module
    [ 9.179155] pruss-soc-bus: probe of 4a326000.pruss-soc-bus failed with error -16
    [ 9.191062] remoteproc remoteproc0: wkup_m3 is available
    [ 9.368639] omap-sham 53100000.sham: hw accel on OMAP rev 4.3
    [ 9.494388] omap-aes 53500000.aes: OMAP AES hw accel rev: 3.2
    [ OK ] Started Avahi mDNS/DNS-SD Stack.
    [ 9.567594] omap-aes 53500000.aes: will run requests pump with realtime priority
    [ 9.640805] remoteproc remoteproc0: powering up wkup_m3
    [ 9.653441] PM: Cannot get wkup_m3_ipc handle
    [ 9.656878] remoteproc remoteproc0: Booting fw image am335x-pm-firmware.elf, size 224268
    [ 9.657147] remoteproc remoteproc0: remote processor wkup_m3 is now up
    [ 9.657175] wkup_m3_ipc 44e11324.wkup_m3_ipc: CM3 Firmware Version = 0x192
    [ 9.949938] asoc-simple-card sound: tlv320aic3x-hifi <-> 4803c000.mcasp mapping ok
    [ 9.999665] PM: bootloader does not support rtc-only!
    Starting Network Service...
    Starting Telephony service...
    [ 10.395336] omap_hwmod: gfx: failed to hardreset
    [ 10.408914] net eth1: initializing cpsw version 1.12 (0)
    Starting Print notice about GPLv3 packages...
    [ OK ] Started Network Service.
    [ 10.491443] cpsw 4a100000.ethernet: initialized cpsw ale version 1.4
    [ 10.628609] pvrsrvkm 56000000.sgx: Unable to reset SGX!
    [ 10.657506] omap_hwmod: gfx: _wait_target_ready failed: -16
    [ 10.657530] pvrsrvkm 56000000.sgx: use pm_runtime_put_sync_suspend() in driver?
    [ 10.657546] PVR_K:(Error): EnableSGXClocks: pm_runtime_get_sync failed (16)
    [ 10.657557] PVR_K:(Error): SysInitialise: Failed to Enable system clocks (175)
    [ 11.042066] Bluetooth: Core ver 2.22
    [ 11.042171] NET: Registered protocol family 31
    [ 11.042175] Bluetooth: HCI device and connection manager initialized
    [ 11.042201] Bluetooth: HCI socket layer initialized
    [ 11.042212] Bluetooth: L2CAP socket layer initialized
    [ 11.042254] Bluetooth: SCO socket layer initialized
    [ 11.477503] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
    [ 12.014203] cpsw 4a100000.ethernet: ALE Table size 1024
    [ 12.174984] SMSC LAN8710/LAN8720 4a101000.mdio:01: attached PHY driver [SMSC LAN8710/LAN8720] (mii_bus:phy_addr=4a101000.mdio:01, irq=-1)
    [ 12.455803] cpts ptp bc clkid 0
    [ OK ] Found device /dev/ttyS3.
    [ 12.523185] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    [ 12.727908] net eth0: initializing cpsw version 1.12 (0)
    [ 12.904706] SMSC LAN8710/LAN8720 4a101000.mdio:00: attached PHY driver [SMSC LAN8710/LAN8720] (mii_bus:phy_addr=4a101000.mdio:00, irq=-1)
    [ 13.083363] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
    [ OK ] Found device /dev/mmcblk0p1.
    [ OK ] Started Telephony service.
    [ OK ] Started Login Service.
    Starting Start USB gadget...
    [ OK ] Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
    [ OK ] Reached target Network.
    [ OK ] Started strongSwan IPsec IKEv1/IKEv2 daemon using ipsec.conf.
    Starting Enable and configure wl18xx bluetooth stack...
    Starting Simple Network Management Protocol (SNMP) Daemon....
    Starting Permit User Sessions...
    Starting Lightning Fast Webserver With Light System Requirements...
    Starting Network Name Resolution...
    [ OK ] Started Permit User Sessions.
    [ 14.360399] udc-core: couldn't find an available UDC - added [g_multi] to list of pending drivers
    [ OK ] Started Serial Getty on ttyS0.
    [ OK ] Started Getty on tty1.
    [ OK ] Started Serial Getty on ttyS3.
    [ OK ] Started Network Name Resolution.
    [ OK ] Started Lightning Fast Webserver With Light System Requirements.
    [ 15.124726] cpsw 4a100000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
    [ 15.132838] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    [ 15.289940] NET: Registered protocol family 15
    [FAILED] Failed to start Start USB gadget.
    See 'systemctl status gadget-init.service' for details.
    [ OK ] Started Enable and configure wl18xx bluetooth stack.
    [ 16.448406] Initializing XFRM netlink socket
    [ OK ] Started Simple Network Management Protocol (SNMP) Daemon..
    [ 17.750994] random: crng init done
    [ 18.716504] TI-am335x-tsc TI-am335x-tsc: ti,charge-delay not specified
    [ 18.801362] input: ti-tsc as /devices/platform/ocp/44e0d000.tscadc/TI-am335x-tsc/input/input1
    [ 18.817015] 47401300.usb-phy supply vcc not found, using dummy regulator
    [ 18.817863] 47401b00.usb-phy supply vcc not found, using dummy regulator
    [ 18.845908] usbcore: registered new interface driver usbfs
    [ 18.845979] usbcore: registered new interface driver hub
    [ 18.846102] usbcore: registered new device driver usb
    [ 19.049426] musb-hdrc musb-hdrc.0: MUSB HDRC host driver
    [ 19.049470] musb-hdrc musb-hdrc.0: new USB bus registered, assigned bus number 1
    [ 19.067279] hub 1-0:1.0: USB hub found
    [ 19.067365] hub 1-0:1.0: 1 port detected
    [ 19.187525] using random self ethernet address
    [ 19.187532] using random host ethernet address
    [ 19.187570] using host ethernet address: A8:1B:6A:33:52:E8
    [ 19.187571] using random self ethernet address
    [ 19.187574] using random host ethernet address
    [ 19.190456] using host ethernet address: A8:1B:6A:33:52:E8
    [ 19.190458] usb0: HOST MAC a8:1b:6a:33:52:e8
    [ 19.190744] usb0: MAC 6e:04:ae:10:79:16
    [ 19.329032] Mass Storage Function, version: 2009/09/11
    [ 19.329042] LUN: removable file: (no medium)
    [ 19.329262] LUN: removable read only file: /dev/mmcblk0p1
    [ 19.329267] Number of LUNs=1
    [ 19.335154] g_multi gadget: Multifunction Composite Gadget
    [ 19.335163] g_multi gadget: g_multi ready
    [ 19.367668] musb-hdrc musb-hdrc.1: MUSB HDRC host driver
    [ 19.367709] musb-hdrc musb-hdrc.1: new USB bus registered, assigned bus number 2
    [ 19.411264] hub 2-0:1.0: USB hub found
    [ 19.411326] hub 2-0:1.0: 1 port detected
    [ 19.579595] g_multi gadget: high-speed config #1: Multifunction with RNDIS

    _____ _____ _ _
    | _ |___ ___ ___ ___ | _ |___ ___ |_|___ ___| |_
    | | _| .'| . | . | | __| _| . | | | -_| _| _|
    |__|__|_| |__,|_ |___| |__| |_| |___|_| |___|___|_|
    |___| |___|

    Arago Project http://arago-project.org am335x-evm ttyS0

    Arago 2017.10 am335x-evm ttyS0

    am335x-evm login: ***************************************************************
    ***************************************************************
    NOTICE: This file system contains the following GPLv3 packages:
    autoconf
    binutils
    cifs-utils
    cpp-symlinks
    cpp
    dosfstools
    g++-symlinks
    g++
    gawk-dev
    gawk
    gcc-symlinks
    gcc
    gdb
    gdbserver
    glmark2
    gstreamer1.0-libav
    hidapi
    libcairo-perf-utils
    libgmp10
    libidn11
    libmpc3
    libmpfr4
    libreadline-dev
    libreadline6
    m4-dev
    m4
    make
    nettle
    swig-dev
    swig

    If you do not wish to distribute GPLv3 components please remove
    the above packages prior to distribution. This can be done using
    the opkg remove command. i.e.:
    opkg remove <package>
    Where <package> is the name printed in the list above

    NOTE: If the package is a dependency of another package you
    will be notified of the dependent packages. You should
    use the --force-removal-of-dependent-packages option to
    also remove the dependent packages as well
    ***************************************************************
    ***************************************************************
    [ 27.975486] hrtimer: interrupt took 194667 ns

    _____ _____ _ _
    | _ |___ ___ ___ ___ | _ |___ ___ |_|___ ___| |_
    | | _| .'| . | . | | __| _| . | | | -_| _| _|
    |__|__|_| |__,|_ |___| |__| |_| |___|_| |___|___|_|
    |___| |___|

    Arago Project http://arago-project.org am335x-evm ttyS0

    Arago 2017.10 am335x-evm ttyS0

    am335x-evm login:

    2.-------------------------------------- lsmod command output --------------------------------------
    root@am335x-evm:~# lsmod
    Module Size Used by
    usb_f_acm 5276 1
    u_serial 11171 3 usb_f_acm
    sha512_generic 9967 0
    usb_f_ecm 6492 1
    sha512_arm 12252 0
    sha256_generic 10178 0
    sha1_generic 2864 0
    sha1_arm_neon 6261 0
    sha1_arm 3862 1 sha1_arm_neon
    md5 2219 0
    musb_dsps 9734 0
    musb_hdrc 87874 1 musb_dsps
    usbcore 201869 1 musb_hdrc
    phy_am335x 2372 2
    phy_am335x_control 2937 1 phy_am335x
    phy_generic 5372 1 phy_am335x
    des_generic 17608 0
    cbc 2388 0
    xfrm_user 22370 2
    xfrm4_tunnel 2040 0
    ipcomp 2257 0
    xfrm_ipcomp 4311 1 ipcomp
    ti_am335x_tsc 6029 0
    esp4 6402 0
    ah4 5627 0
    af_key 27032 0
    xfrm_algo 6989 5 xfrm_user,esp4,ah4,af_key,xfrm_ipcomp
    g_multi 6145 0
    usb_f_mass_storage 37218 2 g_multi
    usb_f_rndis 15482 2 g_multi
    u_ether 13001 3 usb_f_ecm,g_multi,usb_f_rndis
    libcomposite 44446 5 usb_f_ecm,usb_f_acm,usb_f_mass_storage,g_multi,usb_f_rndis
    udc_core 19450 8 usb_f_ecm,usb_f_acm,usb_f_mass_storage,u_serial,musb_hdrc,u_ether,usb_f_rndis,libcomposite
    usb_common 4737 6 phy_am335x,udc_core,musb_hdrc,libcomposite,musb_dsps,usbcore
    bluetooth 337541 2
    snd_soc_simple_card 5844 0
    snd_soc_simple_card_utils 5095 1 snd_soc_simple_card
    pm33xx 6433 0
    wkup_m3_ipc 8240 1 pm33xx
    wkup_m3_rproc 3670 1
    remoteproc 28645 2 wkup_m3_rproc,wkup_m3_ipc
    pvrsrvkm 403500 0
    omap_aes_driver 19334 0
    crypto_engine 7098 1 omap_aes_driver
    omap_sham 21950 0
    ti_emif_sram 5663 1 pm33xx
    pruss_soc_bus 4105 0
    snd_soc_tlv320aic3x 48423 1
    musb_am335x 1426 0
    rtc_omap 9662 1
    omap_wdt 4634 0
    ti_am335x_tscadc 6290 1 ti_am335x_tsc
    sch_fq_codel 9115 4
    uio_module_drv 7251 0
    uio 9121 1 uio_module_drv
    cryptodev 38083 1

    I would appreciate your advice.

  • Ethan,

    Can you please run the following commands right after kernel booted to remove the crypto hw drivers then run your ssh-keygen test to see if MPU hangs? This would tell if it is cypto hw related.

    # modprobe -r omap-aes-driver
    # modprobe -r omap-sham
    # modprobe -r omap-des
  • Thank you for reply.

    What does it mean to remove the crypto hw drivers?

    I deleted /lib/modules/4.9.59-ga75d8e9305/extra/cryptodev.ko file. Is this what you mean?

    And then run following command.

    # modprobe -r omap-aes-driver

    # modprobe -r omap-sham

    # modprobe -r omap-des

    I attached command output and lnsmod output.

    But, when I run ssh-keygen, cpu stil hang.

    Can you give me any other advice?

    output.rtf

  • Ethan,

    ethan jo said:
    What does it mean to remove the crypto hw drivers?

    With all those mentioned omap-* drivers loaded, crypto related operations (such as aes/sham) would be processed by the AM3358 Crypto HW Accelerator. But if we removed those drivers, those crypto operations are done by CPU.

    ethan jo said:
    I deleted /lib/modules/4.9.59-ga75d8e9305/extra/cryptodev.ko file. Is this what you mean?

    Not exactly what I asked, but it does what I needed.

    ethan jo said:

    But, when I run ssh-keygen, cpu stil hang.

    This tells the issue is not related to Crypto Accelerator. There would be two other possibilities I can think of.

    1. Board related, for example, clock, power, DDR config. But based on Biser's comments, this is unlikely to be the factor;

    2. Software related, could be a bug in the version of openssh you use. can you test the openssh which is in the Processor SDK root filesystem to see if the issue also happens?

  • Thank you for your advice.

    But, I can't find the built-in ssh-keygen which is in the Processor SDK root filesystem.

    Where is it located in the filesystem?

    I made the filesystem using create-sdcard.sh script provided by Processor SDK(ti-processor-sdk-linux-am335x-evm-04.02.00.09).

  • Ethan,

    Sorry, I meant to use the openssl version (not openssh) provided in the SDK, which is 1.0.2j.
    The Processor SDK doesn't have openssh, but dropbear. so you still need to compile openssh, but try to use openssl which comes in the SDK root filesystem.

    If the lockup still happens. There would be two possibilities.
    1. hw issue: the CPU hangs, the program counter goes to somewhere and never changed.
    2. sw issue: userspace sw program dead loop, the program counter would change be stay in a loop. Please note that this takes some effort to verify, because Linux kernel still runs, so userspace and kernel space context switching still happens.
  • Ethan,

    I haven’t heard back from you, I’m assuming you were able to resolve your issue. If not, just post a reply below (or create a new thread if the thread has locked due to time-out). thanks.