J722SXH01EVM: OSPI NOR FLASH random read failures.

Hi,

Are you using EVM for these tests? Or is it based on your custom board?

If its custom board based, what is the flash part being used?

Regards,
Tanmay

Hi Tanmay,

I am using the EVM board for this test.

Regards,

Vishnupriya J

Hi,

Let me try this out on my end and respond back to you.

Regards,
Tanmay

Hi Tanmay,

Thanks for the update and we look forward to your response.

Regards,

Vishnupriya J

Hi Tanmay,

Is there any updates?
We are waiting for your response.

Regards,

Vishnupriya J

HI Tanmay,

Is there any progress or updates?!
Kindly respond us.

Regards,

Vishnupriya J

Vishnupriya,

Apologies for the long silence on this. Is this seen even on the latest 10.1 SDK version. Have you tried 10.1 SDK?

https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-J722S/10.01.00.04

If you see the same issue, please share the logs.

Thanks for your patience,
Keerthy

Hi Keerthy,

Thanks for your response. We will check and update you.

Hi Keerthy,

Even with the latest J722S 10.01 SDK I am facing the issue, I used the pre-built binaries.

root@j722s-evm:~# flash_erase /dev/mtd3 0 0 
Erasing 256 Kibyte @ 0 -- 100 % complete 
root@j722s-evm:~# dd if=256K_file of=/dev/mtd3 bs=256K count=1
0+1 records in
0+1 records out
131072 bytes (131 kB, 128 KiB) copied, 0.246157 s, 532 kB/s
root@j722s-evm:~# dd if=/dev/mtd3 of=256K_file_read bs=256K count=1
1+0 records in
1+0 records out
262144 bytes (262 kB, 256 KiB) copied, 0.00419299 s, 62.5 MB/s
root@j722s-evm:~# diff 256K_file 256K_file_read
Files 256K_file and 256K_file_read differ
root@j722s-evm:~# 

Can you please help us to figure out the root cause of this issue?

Regards,

Vishnupriya J

Hi Vishnupriya,

The number of bytes written to flash (131072) and read (262144) differ. They are exactly double. Can you please check this.

Also have you tried to write and readback from phy in u-boot? Does that maintain the data integrity?

Regards,
Tanmay

Hi Vishnupriya,

Also this is not seen on the EVM on my end. See the attached logs

root@j722s-evm:~# dd if=/dev/urandom of=text.txt  count=16  bs=16K                                                                                                           
16+0 records in
16+0 records out
262144 bytes (262 kB, 256 KiB) copied, 0.00387583 s, 67.6 MB/s
root@j722s-evm:~# ls -lh text.txt 
-rw-r--r-- 1 root root 256K Oct 10 09:42 text.txt
root@j722s-evm:~# dd if=text.txt of=/dev/mtd3 bs=256K count=1                                                                                                                
1+0 records in
1+0 records out
262144 bytes (262 kB, 256 KiB) copied, 0.612383 s, 428 kB/s
root@j722s-evm:~# dd if=/dev/mtd3 of=256K_file_read bs=256K count=1
1+0 records in
1+0 records out
262144 bytes (262 kB, 256 KiB) copied, 0.00314416 s, 83.4 MB/s
root@j722s-evm:~# diff text.txt 256K_file_read 
root@j722s-evm:~# diff text.txt 256K_file_read 
root@j722s-evm:~# diff text.txt 256K_file_read 
root@j722s-evm:~# diff text.txt 256K_file_read 
root@j722s-evm:~# 

Is it possible for you to try with a different hardware?

Regards,
Tanmay

Hi Tanmay,

Tanmay Patil said:

The number of bytes written to flash (131072) and read (262144) differ. They are exactly double. Can you please check this.

I apologize for this, it is a mistake from our side. Despite using proper file's on testing, I got the error with subsequent read and writes in kernel. I have attached the log for your reference.

root@j722s-evm:~# flash_erase /dev/mtd1 0 0
256K count=1
Erasing 2048 Kibyte @ 0 -- 100 % complete 
root@j722s-evm:~# dd if=256K_file  of=/dev/mtd1 bs=256K count=1
1+0 records in
1+0 records out
262144 bytes (262 kB, 256 KiB) copied, 0.670315 s, 391 kB/s
root@j722s-evm:~# dd if=/dev/mtd1 of=256K_file_read bs=256K count=1
1+0 records in
1+0 records out
262144 bytes (262 kB, 256 KiB) copied, 0.00269191 s, 97.4 MB/s
root@j722s-evm:~# cmp 256K_file 256K_file_read
root@j722s-evm:~# 
root@j722s-evm:~# 
root@j722s-evm:~# 
root@j722s-evm:~# flash_erase /dev/mtd1 0 0
Erasing 2048 Kibyte @ 0 --  0 % complete 
Erasing 2048 Kibyte @ 0 -- 100 % complete 
root@j722s-evm:~# dd if=512K_file of=/dev/mtd1 bs=512K count=1
1+0 records in
1+0 records out
524288 bytes (524 kB, 512 KiB) copied, 1.3534 s, 387 kB/s
root@j722s-evm:~# dd if=/dev/mtd1 of=512K_file_read bs=512K count=1
1+0 records in
1+0 records out
524288 bytes (524 kB, 512 KiB) copied, 0.0049774 s, 105 MB/s
root@j722s-evm:~# cmp 512K_file 512K_file_read
root@j722s-evm:~# 
root@j722s-evm:~# 
root@j722s-evm:~# 
root@j722s-evm:~# dd if=/dev/urandom of=2M_file   bs=2M   count=1
1+0 records in
1+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0306677 s, 68.4 MB/s
root@j722s-evm:~# flash_erase /dev/mtd1 0 0
Erasing 2048 Kibyte @ 0 --  0 % complete 
Erasing 2048 Kibyte @ 0 -- 100 % complete 
root@j722s-evm:~# dd if=2M_file of=/dev/mtd1 bs=2M count=1
1+0 records in
1+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 5.40781 s, 388 kB/s
root@j722s-evm:~# dd if=/dev/mtd1 of=2M_file_read bs=2M count=1
1+0 records in
1+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.017506 s, 120 MB/s
root@j722s-evm:~# cmp 2M_file 2M_file_read
2M_file 2M_file_read differ: char 524289, line 1998
root@j722s-evm:~# 
root@j722s-evm:~# 
root@j722s-evm:~# 
root@j722s-evm:~# dd if=/dev/urandom of=128K_file bs=128K count=1
1+0 records in
1+0 records out
131072 bytes (131 kB, 128 KiB) copied, 0.00089186 s, 147 MB/s
root@j722s-evm:~# flash_erase /dev/mtd1 0 0
Erasing 2048 Kibyte @ 0 -- 100 % complete 
root@j722s-evm:~# dd if=128K_file of=/dev/mtd1 bs=128K count=1
1+0 records in
1+0 records out
131072 bytes (131 kB, 128 KiB) copied, 0.228921 s, 573 kB/s
root@j722s-evm:~# dd if=/dev/mtd1 of=128K_file_read bs=128K count=1
1+0 records in
1+0 records out
131072 bytes (131 kB, 128 KiB) copied, 0.00168421 s, 77.8 MB/s
root@j722s-evm:~# cmp 128K_file 128K_file_read
128K_file 128K_file_read differ: char 12289, line 1
root@j722s-evm:~# 

Tanmay Patil said:

Also have you tried to write and readback from phy in u-boot? Does that maintain the data integrity?

Yes, I have tried read and write from u-boot and I am facing intermittent failures on the u-boot also.

Attached the inconsistent read failure on u-boot for your reference.

=> sf probe
SF: Detected s28hs512t with page size 256 Bytes, erase size 256 KiB, total 64 MiB
=> sf erase 0x0 0x80000                                                                                               
SF: 524288 bytes @ 0x0 Erased: OK
=> fatload mmc 1:1 ${loadaddr} 128K_file
131072 bytes read in 78 ms (1.6 MiB/s)
=> sf write ${loadaddr} 0x0 ${filesize}
device 0 offset 0x0, size 0x20000
SF: 131072 bytes @ 0x0 Written: OK
=> sf read ${fdtaddr} 0x0 ${filesize}
device 0 offset 0x0, size 0x20000
SF: 131072 bytes @ 0x0 Read: OK
=> cmp ${loadaddr} ${fdtaddr} ${filesize}
Total of 131072 word(s) were the same
=> sf read ${fdtaddr} 0x0 ${filesize}    
device 0 offset 0x0, size 0x20000
SF: 131072 bytes @ 0x0 Read: OK
=> cmp ${loadaddr} ${fdtaddr} ${filesize}
Total of 131072 word(s) were the same
=> sf read ${fdtaddr} 0x0 ${filesize}    
device 0 offset 0x0, size 0x20000
SF: 131072 bytes @ 0x0 Read: OK
=> cmp ${loadaddr} ${fdtaddr} ${filesize}
Total of 131072 word(s) were the same
=> sf read ${fdtaddr} 0x0 ${filesize}    
device 0 offset 0x0, size 0x20000
SF: 131072 bytes @ 0x0 Read: OK
=> cmp ${loadaddr} ${fdtaddr} ${filesize}
Total of 131072 word(s) were the same
=>
=>
=>
=> sf erase 0x0 0x80000
SF: 524288 bytes @ 0x0 Erased: OK
=> fatload mmc 1:1 ${loadaddr} 256K_file
262144 bytes read in 84 ms (3 MiB/s)
=> sf write ${loadaddr} 0x0 ${filesize}
device 0 offset 0x0, size 0x40000
SF: 262144 bytes @ 0x0 Written: OK
=> sf read ${fdtaddr} 0x0 ${filesize}
device 0 offset 0x0, size 0x40000
SF: 262144 bytes @ 0x0 Read: OK
=> cmp ${loadaddr} ${fdtaddr} ${filesize}
word at 0x82080000 (0x00) != word at 0x88080000 (0x800000)
Total of 131072 word(s) were the same
=>

Thanks & Regards,

Vishnupriya J

Hi Tanmay,

Tanmay Patil said:

Also this is not seen on the EVM on my end. See the attached logs

As I have mentioned already I'm getting intermittent failures on subsequent reads and also I used to test with all the partition for nearly 10 times. Have you tested with subsequent reads in all the partition of the OSPI NOR flash device?

Tanmay Patil said:

Is it possible for you to try with a different hardware?

I have tested with another hardware even in that also I'm facing the failures.

Thanks & Regards,

Vishnupriya J

Hi Tanmay,

Is there any updates?

Kindly respond us.

Regards,

Vishnupriya J

Hi,

VISHNUPRIYA J said:

Have you tested with subsequent reads in all the partition of the OSPI NOR flash device?

I have tried with all the partitions for about 400 iterations and didn't see the issue. I will run the test overnight and come back.

What is the flash part being used.

Regards,
Tanmay

Hi Tanmay,

Tanmay Patil said:

What is the flash part being used.

When testing the read & write operations in OSPI NOR flash of the J722S EVM board, error occurs on the random partitions.

Can you share us the files and commands used for the testing.

Regards,

Vishnupriya J

Hi,

I used the following script for testing for about 400 iterations :

do_test(){
flash_erase /dev/mtd$1 0 0
dd if=text.txt of=/dev/mtd$1 bs=256K count=1
dd if=/dev/mtd$1 of=256K_file_read bs=256K count=1
}


c=0
i=0
p=0
while [ $c -eq 0 ]; do
        do_test $p
        ((p++))
        if [ $p -eq 6 ]; then
                p=0
        fi

        diff -q text.txt 256K_file_read
        c=$(diff -q text.txt 256K_file_read | grep differ | wc -c)
        ((i++))
        echo ""
        echo "Iteration $i partition $p"
        echo ""
        sleep 1
done

Regards,
Tanmay

Hi Tanmay,

I have observed that the shared script utilizes a uniform file size of 256KB across all the partitions. Additionally, the read-back file is not being deleted after comparison.

For testing purpose, I have used varying file sizes in my script, ranging from 128KB to the full size of each respective partition. However, while doing so I observed the issue on subsequent reads with random partitions.

Herewith I have attached the commands for creating the file and the script file that I have used for the testing for your reference.

dd if=/dev/urandom of=<file_name> bs=<file_size> count=1

#!/bin/bash

test_flash() {
    local partition=$1
    local file=$2
    local size=$3

    echo ""
    echo "Testing partition $partition with file $file ($size)..."
    echo ""

    # Erase the partition
    flash_erase "$partition" 0 0
    if [ $? -ne 0 ]; then
	echo ""
        echo "Failed to erase $partition"
	echo ""
        exit 1
    fi

    # Write to partition
    dd if="$file" of="$partition" bs="$size" count=1
    if [ $? -ne 0 ]; then
	echo ""
        echo "Failed to write $file to $partition"
	echo ""
        exit 1
    fi

    # Read from partition
    local read_file="${file}_read"
    dd if="$partition" of="$read_file" bs="$size" count=1
    if [ $? -ne 0 ]; then
	echo ""
        echo "Failed to read from $partition to $read_file"
	echo ""
        exit 1
    fi

    # Compare
    diff -q $file  $read_file
    if [ $? -ne 0 ]; then
	echo ""
	echo "**********************Comparison failed between $file and $read_file**********************"
	echo ""
	exit 1
    fi

    # Delete the read file
    rm "$read_file"
    echo ""

    flash_erase "$partition" 0 0
    if [ $? -ne 0 ]; then       
        echo ""                 
        echo "Failed to erase $partition"
        echo ""                          
        exit 1                           
    fi  

}

declare -a tests=(
    "/dev/mtd0 128K_file 128K"
    "/dev/mtd0 256K_file 256K"
    "/dev/mtd0 512K_file 512K"
    "/dev/mtd1 128K_file 128K"
    "/dev/mtd1 256K_file 256K"
    "/dev/mtd1 512K_file 512K"
    "/dev/mtd1 2M_file 2M"
    "/dev/mtd2 128K_file 128K"
    "/dev/mtd2 256K_file 256K"
    "/dev/mtd2 512K_file 512K"
    "/dev/mtd2 2M_file 2M"
    "/dev/mtd2 4M_file 4M"
    "/dev/mtd3 128K_file 128K"
    "/dev/mtd3 256K_file 256K"
    "/dev/mtd4 128K_file 128K"
    "/dev/mtd4 256K_file 256K"
    "/dev/mtd5 128K_file 128K"
    "/dev/mtd5 256K_file 256K"
    "/dev/mtd5 512K_file 512K"
    "/dev/mtd5 2M_file 2M"
    "/dev/mtd5 4M_file 4M"
    "/dev/mtd5 10M_file 10M"
)

iterations=100

for ((i=1; i<=iterations; i++)); do
    echo ""
    echo "---------------- Iteration $i ----------------"
    echo ""

    for test in "${tests[@]}"; do
        set -- $test
        partition=$1
        file=$2
        size=$3

	flash_erase "$partition" 0 0                  
	if [ $? -ne 0 ]; then                         
       		 echo ""                                   
        	echo "Failed to erase $partition"
        	echo ""                                                                                          
        	exit 1                                                                                           
   	fi  

        test_flash "$partition" "$file" "$size"
        if [ $? -ne 0 ]; then
		echo ""
		echo "Test failed on iteration $i for partition $partition with file $file"
		echo ""
		exit 1
	else
	 	echo ""
		echo "Test success on iteration $i for partition $partition with file $file"
		echo ""
	fi
    done
	echo ""
	echo "________________________________________________________________________________________________"
	echo "" 
done

echo ""
echo ""
echo ""
echo "*************************************All tests completed successfully*************************************"
echo ""
echo ""
echo ""

Regards,

Vishnupriya J

Hi Tanmay,

Any updates on this?

Regards,

Vishnupriya J

Hi,

I ran tests with your scripts, but I couldn't reproduce the error. But the test was a bit short (~ few hours). I will rerun this script for overnight testing today. I will respond tomorrow.

Regards,
Tanmay

Hi Tanmay,

Thanks for the update and we look forward to your response.

Regards,

Vishnupriya J