Linux/AM3358: SD card EXT4 error

Filip Kotoucek

Part Number: AM3358

Tool/software: Linux

Hello,

I would like to ask you for advice about system configuration running from SD card. I'm using kernel from TI 4.9.59. And due to need of our own filesystem I chose to use Buildroot. I know the TI mainly supports Yocto, but I believe someone could help me anyway.

There is the thing. The compilation is OK, I create SD card as recomended by buildroot by: dd if=${BASEDIR}/../output/images/sdcard.img of=/dev/mmcblk0. It creates two partitions on the SD. The Linux boots OKish but after while wierd Ext-4 errors appear. Here is some example I got.

[ 17.243011] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:5434: Out y
[ 17.256024] EXT4-fs error (device mmcblk0p2) in ext4_orphan_add:2845: Out of memory
[ 17.268580] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:5434: Out of memory
[ 17.280781] EXT4-fs error (device mmcblk0p2) in ext4_setattr:5261: Out of memory
[ 19.682658] EXT4-fs error (device mmcblk0p2): ext4_map_blocks:567: inode #8: block 114770: comm jbd2)
[ 19.700038] jbd2_journal_bmap: journal block not found at offset 81 on mmcblk0p2-8
[ 19.709030] Aborting journal on device mmcblk0p2-8.
[ 37.046792] EXT4-fs error (device mmcblk0p2): ext4_journal_check_start:56: Detected aborted journal
[ 37.056102] EXT4-fs (mmcblk0p2): Remounting filesystem read-only
[ 37.067945] EXT4-fs (mmcblk0p2): ext4_writepages: jbd2_start: 3072 pages, ino 4342; err -30
[ 60.772476] random: crng init done
[ 305.122423] EXT4-fs (mmcblk0p2): error count since last fsck: 39
[ 305.128516] EXT4-fs (mmcblk0p2): initial error at time 65536:
�:0: inode 8: block 524288
[ 305.136965] EXT4-fs (mmcblk0p2): last error at time 1513289431: ext4_journal_check_start:56: inode 80

Eventually it leads to total kernel panic. I tried multiple SD cards, same behavior. I switched the journald to use volatile memory, but I'm definitely missing something.

Below is attaged my Buildroot defconfig.

Any help would be appreciated.

Fullscreen prusa_pro_defconfig.txt Download

BR2_arm=y
BR2_cortex_a8=y
BR2_CCACHE=y
BR2_GLOBAL_PATCH_DIR="board/prusa/einsy_pro/patches"
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TARGET_GENERIC_HOSTNAME="prusa_pro"
BR2_TARGET_GENERIC_ISSUE="Welcome to Prusa"
BR2_INIT_SYSTEMD=y
BR2_TARGET_GENERIC_ROOT_PASSWD="toor"
BR2_SYSTEM_BIN_SH_BASH=y
BR2_TARGET_GENERIC_GETTY_PORT="ttyO0"
BR2_ROOTFS_OVERLAY="board/prusa/einsy_pro/rootfs_overlay"
BR2_ROOTFS_POST_IMAGE_SCRIPT="board/prusa/einsy_pro/post-image.sh"
BR2_LINUX_KERNEL=y
BR2_LINUX_KERNEL_CUSTOM_GIT=y
BR2_LINUX_KERNEL_CUSTOM_REPO_URL="https://github.com/filipkotoucek/tisdk_linux.git"
BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION="development"
BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/prusa/einsy_pro/linux_defconfig"
BR2_LINUX_KERNEL_DTS_SUPPORT=y
BR2_LINUX_KERNEL_INTREE_DTS_NAME="am335x-bone am335x-boneblack am335x-prusa-pro"
BR2_PACKAGE_BUSYBOX_CONFIG="board/prusa/einsy_pro/busybox.config"
BR2_PACKAGE_SPIDEV_TEST=y
BR2_PACKAGE_PYTHON=y
BR2_PACKAGE_PYTHON_SPIDEV=y
BR2_PACKAGE_DTC=y
BR2_PACKAGE_DTC_PROGRAMS=y
BR2_PACKAGE_DROPBEAR=y
BR2_PACKAGE_VIM=y
BR2_TARGET_ROOTFS_EXT2=y
BR2_TARGET_ROOTFS_EXT2_4=y
BR2_TARGET_ROOTFS_EXT2_SIZE="256M"
BR2_TARGET_UBOOT=y
BR2_TARGET_UBOOT_BUILD_SYSTEM_KCONFIG=y
BR2_TARGET_UBOOT_CUSTOM_VERSION=y
BR2_TARGET_UBOOT_CUSTOM_VERSION_VALUE="2017.01"
BR2_TARGET_UBOOT_BOARD_DEFCONFIG="am335x_boneblack"
BR2_TARGET_UBOOT_NEEDS_DTC=y
# BR2_TARGET_UBOOT_FORMAT_BIN is not set
BR2_TARGET_UBOOT_FORMAT_IMG=y
BR2_TARGET_UBOOT_FORMAT_CUSTOM=y
BR2_TARGET_UBOOT_FORMAT_CUSTOM_NAME="spl/u-boot-spl.bin"
BR2_TARGET_UBOOT_SPL=y
BR2_TARGET_UBOOT_SPL_NAME="MLO"
BR2_PACKAGE_HOST_DOSFSTOOLS=y
BR2_PACKAGE_HOST_GENIMAGE=y
BR2_PACKAGE_HOST_MTOOLS=y

over 7 years ago

0 Biser Gatchev-XID over 7 years ago

TI__Guru**** 393215 points

Hi,

Current Linux Processor SDK release is based on kernel v4.9.41. Other releases or Linux versions are not supported on this forum.

0 Filip Kotoucek over 7 years ago in reply to Biser Gatchev-XID

Prodigy 130 points

The TI SDK 04.02.00.09 includes kernel 4.9.59 which I am using.

0 Biser Gatchev-XID over 7 years ago in reply to Filip Kotoucek

TI__Guru**** 393215 points

Sorry, my mistake there. I have notified the software team. They will respond here.

0 Kemal R. Shakir over 7 years ago in reply to Biser Gatchev-XID

TI__Guru 55345 points

Hello Filip,

There are remains from your previous formatting which confuses the ext4 filesystem. Please overwrite the complete data on your SD card with zeroes before flashing your new image.

dd if=/dev/zero of=/dev/sdX

Best regards,
Kemal

0 Filip Kotoucek over 7 years ago in reply to Kemal R. Shakir

Prodigy 130 points

I already did that but nothing changed.

When I tried to DD card again and write the image into it, I got these errors after boot:

[ 18.422901] EXT4-fs error (device mmcblk0p2): mb_free_blocks:1470: group 1, block 33283:freeing already freed block (bit 515); block bitmap corrupt.
[ 18.436628] EXT4-fs error (device mmcblk0p2): ext4_mb_generate_buddy:758: group 1, block bitmap and bg descriptor inconsistent: 30010 vs 30011 free clusters

After next reboot I got the error messages when the device is booting up:

[ 1.731485] ALSA device list:
[ 1.734557] No soundcards found.
[ 1.750369] EXT4-fs (mmcblk0p2): warning: mounting fs with errors, running e2fsck is recommended
[ 1.763438] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 1.771630] VFS: Mounted root (ext4 filesystem) on device 179:2.
[ 1.779902] devtmpfs: mounted
[ 1.784394] Freeing unused kernel memory: 1024K
...
[ OK ] Reached target Local File Systems.
Starting Rebuild Journal Catalog...
[ 3.953640] systemd-journald[89]: Received request to flush runtime journal from PID 1
[ 4.020297] EXT4-fs error (device mmcblk0p2): ext4_mb_generate_buddy:758: group 1, block bitmap and bg descriptor inconsistent: 30517 vs 30518 free clusters
[ OK ] Started Flush Journal to Persistent Storage.
Starting Create Volatile Files and Directories...
[ 4.094186] JBD2: Spotted dirty metadata buffer (dev = mmcblk0p2, blocknr = 0). There's a risk of filesystem corruption in case of system crash.
[ OK ] Started Rebuild Journal Catalog.

Could it be caused by some bad allocation? When I write the image to 8 GB SD card it creates two partitions:

/dev/mmcblk0p1 16MB first sector 1, last sector 32768, flasgs boot lba
/dev/mmcblk0p2 512MB first sector 32769, last sector 1081344
unalocated first sector 1081345, last sector 15759359

Or does the linux writes something many times somewhere into SD card so it causes the damaged sector?
I have the second partition mounted via USB gadget driver as mass storage. So could it be caused by improper device handling?

0 Kemal R. Shakir over 7 years ago in reply to Filip Kotoucek

TI__Guru 55345 points

We need to narrow down the issue to understand why this happens. Can you boot with the kernel from the latest Processor SDK since this issue can be seen only with bildroot's kernel? Can you attach that SD card to your host machine or tftpboot and run e2fsck on that mmcblk0p2 partition? If there is no success, can you copy the rootfs files with cp -rp to another location then mkfs.ext4 that partition and copy the files back?

0 Filip Kotoucek over 7 years ago in reply to Kemal R. Shakir

Prodigy 130 points

It seems the problem is in mounting mmblck0p2 over g_multi driver. When I switch to using just g_ether without sharing the filesystem from SD card it seems it is working. I send the configuration and shell scripts handling g_multi.

Fullscreen usb-ethernet.sh.txt Download

#!/bin/sh

# Based off:
# https://github.com/RobertCNelson/boot-scripts/blob/master/boot/am335x_evm.sh

# Assume some defaults
SERIAL_NUMBER="1234BBBK5678"
PRODUCT="am335x_evm"
manufacturer="ti.com"

# Read serial numberfrom the eerpom
eeprom="/sys/bus/i2c/devices/0-0050/eeprom"
if [ -f ${eeprom} ] ; then
	SERIAL_NUMBER=$(hexdump -e '8/1 "%c"' ${eeprom} -n 28 | cut -b 17-28)
	PRODUCT="BeagleBoneBlack"
	manufacturer="BeagleBoard.org"
fi

echo "SERIAL_NUMBER = ${SERIAL_NUMBER}"
echo "PRODUCT = ${PRODUCT}"

#save location of MAC address, otherwise exit
mac_address="/proc/device-tree/ocp/ethernet@4a100000/slave@4a100300/mac-address"
[ -r "$mac_address" ] || exit 0

#save MAC address as address of this device
dev_addr=$(/usr/bin/hexdump -v -e '5/1 "%02X:" 1/1 "%02X"' "$mac_address")
echo "dev_addr = ${dev_addr}"

#define static MAC for host PC. Swap last digit to 0
host_addr="${dev_addr:0:-1}0"
echo "host_addr = ${host_addr}"

# Set the g_multi parameters (read-write, removable)
g_defaults="cdrom=0 ro=0 stall=0 removable=1 nofua=1"
g_product="iSerialNumber=${SERIAL_NUMBER} iManufacturer=${manufacturer} iProduct=${PRODUCT}"
g_network="dev_addr=${dev_addr} host_addr=${host_addr}"
g_storage="file=/dev/mmcblk0p2"

#start g_multi module. It starts usb0 interface and mounts rootfs as removable device
modprobe g_multi ${g_storage} ${g_defaults} ${g_product} ${g_network}
#modprobe g_ether ${g_network}

# Bring up the USB network interface
ifdown usb0
ifup usb0

Fullscreen usb-ethernet.service.txt Download

[Unit]
Description=Start USB gadget
Requires=dev-mmcblk0p2.device
#Wants=serial-getty@ttyS0.service
Wants=udhcpd.service

After=dev-mmcblk0p2.device
#Before=serial-getty@ttyS0.service
Before=udhcpd.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/usb-ethernet.sh
#ExecStopPre=/bin/systemctl stop serial-getty@ttyGS0.service
ExecStop=/sbin/rmmod g_multi

[Install]
WantedBy=basic.target

0 Kemal R. Shakir over 7 years ago in reply to Filip Kotoucek

TI__Guru 55345 points

Yes, it is possible to be caused by g_multi driver. I am sending you the scripts from the latest Processor SDK.

Fullscreen gadget-init.sh.txt Download

#!/bin/sh

# Based off:
# https://github.com/RobertCNelson/boot-scripts/blob/master/boot/am335x_evm.sh

# Assume some defaults
SERIAL_NUMBER="1234BBBK5678"
PRODUCT="am335x_evm"
manufacturer="ti.com"

# Read serial numberfrom the eerpom
eeprom="/sys/bus/i2c/devices/0-0050/eeprom"
if [ -f ${eeprom} ] ; then
    SERIAL_NUMBER=$(hexdump -e '8/1 "%c"' ${eeprom} -n 28 | cut -b 17-28)
    PRODUCT="BeagleBoneBlack"
    manufacturer="BeagleBoard.org"
fi

echo "SERIAL_NUMBER = ${SERIAL_NUMBER}"
echo "PRODUCT = ${PRODUCT}"


mac_address="/proc/device-tree/ocp/ethernet@4a100000/slave@4a100300/mac-address"
[ -r "$mac_address" ] || exit 0

host_addr=$(/usr/bin/hexdump -v -e '5/1 "%02X:" 1/1 "%02X"' "$mac_address")
echo "host_addr = ${host_addr}"

# Set the g_multi parameters
g_defaults="cdrom=0 ro=1 stall=0 removable=1 nofua=1"

g_product="iSerialNumber=${SERIAL_NUMBER} iManufacturer=${manufacturer} iProduct=${PRODUCT}"

g_network="host_addr=${host_addr}"

g_storage="file=/dev/mmcblk0p1"

modprobe g_multi ${g_storage} ${g_defaults} ${g_product} ${g_network}

# Bring up the USB network interface
ifup usb0

Fullscreen gadget-init.service.txt Download

[Unit]
Description=Start USB gadget
Requires=dev-mmcblk0p1.device
Wants=serial-getty@ttyGS0.service
Wants=busybox-udhcp.service

After=dev-mmcblk0p1.device
Before=serial-getty@ttyGS0.service
Before=busybox-udhcp.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/gadget-init.sh
ExecStopPre=/bin/systemctl stop serial-getty@ttyGS0.service
ExecStop=/sbin/rmmod g_multi

[Install]
WantedBy=basic.target

Processors

Processors forum

Linux/AM3358: SD card EXT4 error