AM3352: AM3352: UART module issue reproduced

Pawel

Part Number: AM3352

Hi Bin

The issue has been reproduced for the SERIAL_OMAP driver. Please find requested data below.

1. the output of command "serialstats", the link below has the information of this tool.

https://software-dl.ti.com/processor-sdk-linux/esd/docs/06_03_00_106/AM335X/linux/Foundational_Components/Kernel/Kernel_Drivers/UART.html#basic-external-loopback-testing

The serialstats and serialcheck tools have been cross-compiled and uploaded to target device, but it seems that serialcheck tool doesn't work with OMAP driver:

./serialcheck -h -d /dev/ttyO1 -f binary -m t -l 10 -b 19200
tiocsserial failed: Invalid argument

serialstats is working properly:

./serialstats -d /dev/ttyO5 -i 1
cts: 0 dsr: 0 rng: 0 dcd: 0 rx: 0 tx: 0 frame error 0 overuns 0 parity: 0 break: 0 buffer overrun: 0
cts: 0 dsr: 0 rng: 0 dcd: 0 rx: 0 tx: 0 frame error 0 overuns 0 parity: 0 break: 0 buffer overrun: 0

2. get the register dump of the locked port. you can use devmem2 command to read the registers.

It will be in the file UART5registers.txt which will be sent to the local support team

3. get all the tasks in the system by command "echo t > /proc/sysrq-trigger". You can send this log offline if you are not comfortable to post it here.

It will be in the file sysrq.txt which will be sent to the local support team

4. what is the state of RTS line (high or low) after the port is locked up?

It is high:

Once above states and logs are collected, please check if the UART IP is in sane state:

- write a char directly to the TX FIFO to see if there is any activity on the TX line, for example on UART5, "devmem2 0x481aa000 b 0x55", where 0xa81aa000 is the base address of UART5.

When writing to the failed port, there is no effect on the TX line

devmem2 0x481aa000 b 0x55
/dev/mem opened.
Memory mapped at address 0xb6f01000.
Read at address 0x481AA000 (0xb6f01000): 0x00
Write at address 0x481AA000 (0xb6f01000): 0x55, readback 0x55

When writing to a working port, data can be observed on the TX line:

- send some data from the other side and see if AM3352 UART FIFO occupancy increases in offset 0x64 (UART_RXFIFO_LVL register).

For the failed port RXFIFO doesn't increase:

/dev/mem opened.
Memory mapped at address 0xb6f73000.
Read at address  0x481AA064 (0xb6f73064): 0x00

For the working port RXFIFO increases:

/dev/mem opened.
Memory mapped at address 0xb6f49000.
Read at address  0x481A8064 (0xb6f49064): 0x01

Also do you understand when exactly the lockup happens? Is it just after switching baud rate or in the middle of transfer or at the start or random? When the AM3352 UART lockup happens, it is a rs485 master, not slave, right?

It is after switching when trying to transfer data. Device is working as rs485 master.

over 5 years ago

0 Bin Liu over 5 years ago

TI__Guru**** 173621 points

Hi Pawel,

Thanks for the update. I will discuss the information with our dev team and get you back.

Pawel said:

The serialstats and serialcheck tools have been cross-compiled and uploaded to target device, but it seems that serialcheck tool doesn't work with OMAP driver:

?

1

2

./serialcheck -h -d /dev/ttyO1 -f binary -m t -l 10 -b 19200

tiocsserial failed: Invalid argument

Not sure what cause the issue. I ran the same on my Beaglebone Black and it works fine.

root@am335x-evm:~# serialcheck -h -d /dev/ttyS0 -f ascii.txt -m t -l 10 -b 19200
Needed 0 reads 1 writes loops 10 / 10
cts: 0 dsr: 0 rng: 0 dcd: 0 rx: 0 tx: 640 frame 0 ovr 0 par: 0 brk: 0 buf_ovrr: 0
root@am335x-evm:~# uname -a
Linux am335x-evm 4.19.94 #45 PREEMPT 2020-09-21 armv7l GNU/Linux

0 Bin Liu over 5 years ago

TI__Guru**** 173621 points

Hi Pawel,

Sorry for the delay. I am still working on this with our dev team. I will keep you posted.

0 Bin Liu over 5 years ago

TI__Guru**** 173621 points

Hi Pawel,

We didn't find anything unusual from the log and information you collected. Can you please collect and provide more information as detailed below?

Can you please send the schematics of RS485 connection to the SoC UART?
We want to understand if reset the UART module will unlock the issue. Please do the following on the locked UART port:
- close your application so that the locked UART port is closed;
- use devmem2 to reset the UART port - the function omap8250_soft_reset() in drivers/tty/serial/8250/8250_omap.cshows the sequence of the register access, basically
  - write 0x1 to SCR register (offset 0x40)
  - write 0x3 to SCR register
  - read the SYSC register (offset 0x54), set its bit1 [SOFTRESET], and write the value back to SYSC register
  - read the SYSS register (offset 0x58) and check its bit0 to ensure the reset is done
- re-run your application to see if the RS-485 communicate is recovered.
When trying to reproduce the issue, did you have serialstats program running simultaneously at the background until lockup? We want to see if the program reports any UART errors, particularly Line error or FIFO overrun error when the lockup happens.

0 Pawel over 5 years ago in reply to Bin Liu

Prodigy 250 points

Hi Bin

I will send schematic to the local support team. There are 5 UART ports. Baudrate is changing on UART1 and UART5. Issue is reproducing on both ports.

I've restarted UART port according to your instruction. So first I've closed all user space applications and then:

root@test:~# devmem2 0x481a6040 b 0x1
/dev/mem opened.
Memory mapped at address 0xb6fcf000.
Read at address  0x481A6040 (0xb6fcf040): 0x80
Write at address 0x481A6040 (0xb6fcf040): 0x01, readback 0x01
root@test:~# devmem2 0x481a6040 b 0x3
/dev/mem opened.
Memory mapped at address 0xb6fa7000.
Read at address  0x481A6040 (0xb6fa7040): 0x01
Write at address 0x481A6040 (0xb6fa7040): 0x03, readback 0x03
root@test:~# devmem2 0x481a6054 b
/dev/mem opened.
Memory mapped at address 0xb6f30000.
Read at address  0x481A6054 (0xb6f30054): 0x0D
root@test:~# devmem2 0x481a6054 b 0xF
/dev/mem opened.
Memory mapped at address 0xb6fd7000.
Read at address  0x481A6054 (0xb6fd7054): 0x0D
Write at address 0x481A6054 (0xb6fd7054): 0x0F, readback 0x0F
root@test:~# devmem2 0x481a6058 b
/dev/mem opened.
Memory mapped at address 0xb6f03000.
Read at address  0x481A6058 (0xb6f03058): 0x01

The result is that port is still not working. The TX line is not changing after writing to register:

root@test:~# devmem2 0x481aa000 b 0x55
/dev/mem opened.
Memory mapped at address 0xb6f2b000.
Read at address  0x481AA000 (0xb6f2b000): 0x00
Write at address 0x481AA000 (0xb6f2b000): 0x55, readback 0x55

Our application is also not working with this port. What I observed is that after reset the following Python script is not hanging now. Means properly exits when finished.

#!/usr/bin/env python3

import time
import serial
import serial.rs485

port = serial.Serial('/dev/ttyO5')
port.rs485_mode = serial.rs485.RS485Settings()
port.stopbits = serial.STOPBITS_ONE
port.bytesize = serial.EIGHTBITS
port.parity = serial.PARITY_EVEN
port.timeout = 5
port.baudrate = 19200
time.sleep(1)
port.write('x'.encode('utf-8'))
port.flush()

After running it, the TX counter in /proc/tty/driver/OMAP-SERIAL increases by 1. But there is no effect on TX line. Only RTS line reacts (the same as before port reset):

Then SYSC register is 0x0 after reset:

root@test:~# devmem2 0x481a6054 b
/dev/mem opened.
Memory mapped at address 0xb6ff0000.
Read at address  0x481A6054 (0xb6ff0054): 0x00

The serialstats program was not running during issue reproduction. It can be run in the background, but it will be needed to restart the device and reproduce the issue again which can take some weeks. Please let me know whether you'd like to read something more from the device with locked port or restart it with serialstats in background.

0 Bin Liu over 5 years ago in reply to Pawel

TI__Guru**** 173621 points

Hi Pawel,

Pawel said:
root@test:~# devmem2 0x481a6040 b 0x1

You mentioned the locked ports are UART1 and UART5, but the base address 0x481a6000 in the command above is for UART3. Why do you modify UART3 register but not UART1 or UART5?

Pawel said:
The serialstats program was not running during issue reproduction.

Understood. Please run it when reproduce the issue again.

0 Pawel over 5 years ago in reply to Bin Liu

Prodigy 250 points

Hi Bin

You're right, I've reset wrong port. I used address from shell history and didn't notice that it is for UART3. Sorry for that.

The good news is that resetting the UART5 unlocks it:

root@test:~# root@test:~# devmem2 0x481aa040 b 0x1
-sh: root@test:~#: command not found
root@test:~# devmem2 0x481aa040 b 0x1
/dev/mem opened.
Memory mapped at address 0xb6f26000.
Read at address  0x481AA040 (0xb6f26040): 0x88
Write at address 0x481AA040 (0xb6f26040): 0x01, readback 0x01
root@test:~# devmem2 0x481aa040 b 0x3
/dev/mem opened.
Memory mapped at address 0xb6fac000.
Read at address  0x481AA040 (0xb6fac040): 0x01
Write at address 0x481AA040 (0xb6fac040): 0x03, readback 0x03
root@test:~# devmem2 0x481aa054 b
/dev/mem opened.
Memory mapped at address 0xb6f84000.
Read at address  0x481AA054 (0xb6f84054): 0x0D
root@test:~# devmem2 0x481aa054 b 0xF
/dev/mem opened.
Memory mapped at address 0xb6f13000.
Read at address  0x481AA054 (0xb6f13054): 0x0D
Write at address 0x481AA054 (0xb6f13054): 0x0F, readback 0x0F
root@test:~# devmem2 0x481aa058 b
/dev/mem opened.
Memory mapped at address 0xb6f1e000.
Read at address  0x481AA058 (0xb6f1e058): 0x01

Based on that we can try to implement in our application a workaround solution. However the best option would be to have it on the driver level.

Regarding serialstats I will run it with 1s period as it is the smallest possible.

0 Bin Liu over 5 years ago in reply to Pawel

TI__Guru**** 173621 points

Hi Pawel,

Thanks for the update. I will discuss the new observation with our dev team.

Pawel said:
Regarding serialstats I will run it with 1s period as it is the smallest possible.

Yes, 1s period is fine. We want to check if the serialstats log shows any error when the lockup happens.

I received the link for the schmatics, and I can download the first 2 small files, but not the last file (which is about 34MB). Can you please check if the download link is valid?

0 Bin Liu over 5 years ago in reply to Pawel

TI__Guru**** 173621 points

It seem your download site has max file size limit as 25MB, but this PDF is 34MB, so it download allow download.

0 Bin Liu over 5 years ago in reply to Pawel

TI__Guru**** 173621 points

It is strange that the local support can download all the 3 PDF files, and have forwarded them to me. We will review them. Thanks.

0 Bin Liu over 5 years ago in reply to Pawel

TI__Guru**** 173621 points

Hi Pawel,

Which UART driver do you use in the last test, serial-omap.c or 8250-omap.c?

0 Pawel over 5 years ago in reply to Bin Liu

Prodigy 250 points

Hi Bin

The serial-omap.c driver was used during last test.

0 Bin Liu over 5 years ago in reply to Pawel

TI__Guru**** 173621 points

Hi Pawel,

The following kernel patch does UART module reset when user space application setting termios(). Can you please run your test with this patch applied to see if the lockup still happens? Please run serialstat tool in background in your test, so that we will know if any line error happens.

Fullscreen 0001-serial-omap-do-softreset-in-set_termios.patch.txt Download

From 1bd5819a99d3d7019bb9a4cd9c870616158e317e Mon Sep 17 00:00:00 2001
From: Bin Liu <b-liu@ti.com>
Date: Thu, 15 Oct 2020 11:00:46 -0500
Subject: [PATCH] serial: omap: do softreset in set_termios()

Signed-off-by: Bin Liu <b-liu@ti.com>
---
 drivers/tty/serial/omap-serial.c | 40 ++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/drivers/tty/serial/omap-serial.c b/drivers/tty/serial/omap-serial.c
index 6420ae581a80..a26ed1502ddd 100644
--- a/drivers/tty/serial/omap-serial.c
+++ b/drivers/tty/serial/omap-serial.c
@@ -67,6 +67,12 @@
 #define OMAP_UART_SCR_TX_TRIG_GRANU1_MASK		(1 << 6)
 #define OMAP_UART_SCR_TX_EMPTY			(1 << 3)
 
+/* SYSC register bitmasks */
+#define OMAP_UART_SYSC_SOFTRESET	(1 << 1)
+
+/* SYSS register bitmasks */
+#define OMAP_UART_SYSS_RESETDONE	(1 << 0)
+
 /* FCR register bitmasks */
 #define OMAP_UART_FCR_RX_FIFO_TRIG_MASK			(0x3 << 6)
 #define OMAP_UART_FCR_TX_FIFO_TRIG_MASK			(0x3 << 4)
@@ -835,6 +841,32 @@ static void serial_omap_uart_qos_work(struct work_struct *work)
 	pm_qos_update_request(&up->pm_qos_request, up->latency);
 }
 
+static int serial_omap_soft_reset(struct uart_omap_port *up)
+{
+	int timeout = 100;
+	int sysc;
+	int syss;
+
+	sysc = serial_in(up, UART_OMAP_SYSC);
+
+	/* softreset the UART */
+	sysc |= OMAP_UART_SYSC_SOFTRESET;
+	serial_out(up, UART_OMAP_SYSC, sysc);
+
+	/* By experiments, 1us enough for reset complete on AM335x */
+	do {
+		udelay(1);
+		syss = serial_in(up, UART_OMAP_SYSS);
+	} while (--timeout && !(syss & OMAP_UART_SYSS_RESETDONE));
+
+	if (!timeout) {
+		dev_err(up->port.dev, "timed out waiting for reset done\n");
+		return -ETIMEDOUT;
+	}
+
+	return 0;
+}
+
 static void
 serial_omap_set_termios(struct uart_port *port, struct ktermios *termios,
 			struct ktermios *old)
@@ -895,6 +927,14 @@ serial_omap_set_termios(struct uart_port *port, struct ktermios *termios,
 	pm_runtime_get_sync(up->dev);
 	spin_lock_irqsave(&up->port.lock, flags);
 
+	if (!uart_console(port)) {
+		if (serial_omap_soft_reset(up))
+			return;
+
+		/* Restore wakeup enable register */
+		serial_out(up, UART_OMAP_WER, up->wer);
+	}
+
 	/*
 	 * Update the per-port timeout.
 	 */
-- 
2.17.1

0 Pawel over 5 years ago in reply to Bin Liu

Prodigy 250 points

Hi Bin

Great! I've compiled the Linux kernel with the patch and ran a test with serialstats in a background. I will inform you if the issue will be reproduced or any error will be reported.

0 Bin Liu over 5 years ago in reply to Pawel

TI__Guru**** 173621 points

Thanks for the update. Looking forward to the result.

0 Pawel over 4 years ago

Prodigy 250 points

0 Bin Liu over 4 years ago

TI__Guru**** 173621 points

Hi Pawel,

It seems you created this thread then edited it, now I don't see any of your message

Is there still standing issue or the original issue is solved?

0 Pawel over 4 years ago in reply to Bin Liu

Prodigy 250 points

Hi Bin

Yes, I've created this thread with the message but found that text is invisible after posting so I've tried to edit it. What is interesting the entry was visible in the edit mode, but again not visible after posting. I don't what could be the reason.

The message was that end of the previous year we finalized tests of the Linux kernel patch. The permanent lock of the UART port didn't happen during these tests as well as later during development activities. So we can treat this issue as solved.

Thank you for your support!

Best Regards

Paweł

0 Bin Liu over 4 years ago in reply to Pawel

TI__Guru**** 173621 points

Hi Pawel,

Glad the issue is solved. Thanks for the update.

I am going to merge this thread with the original one, so that all the information is in one place.

Processors

Processors forum

AM3352: AM3352: UART module issue reproduced