AM3357: UART Driver Issue - AM335X

 Bin Liu and Remi Leveque Andreas Dannenberg1  we have below config changes,

&uart2 {
pinctrl-names = "default", "sleep";
pinctrl-0 = <&rs485_pins_default>;
pinctrl-1 = <&rs485_pins_sleep>;
linux,rs485-enabled-at-boot-time;
rs485-rts-delay = <2 2>;
rs485-rts-active-high;
rts-gpio = <&gpio0 22 GPIO_ACTIVE_HIGH>;
status = "okay";
};

rs485_pins_default: pinmux_rs485_pins_default {
pinctrl-single,pins = <
AM33XX_PADCONF(AM335X_PIN_MII1_TX_CLK, PIN_INPUT, MUX_MODE1)
AM33XX_PADCONF(AM335X_PIN_MII1_RX_CLK, PIN_OUTPUT, MUX_MODE1) 
>;
};

when we see in logic analyzer we are receiving data correctly when we probe TX and RX as below which comes from transviver (SN65HVD11DR),

we are reading NULL bytes in our application and also we are receiving broken data as we are reading 10 bytes of data not all the time we are suspecting UART driver is adding it as its having switched between DE/REdue to fast transition in differential signals. please help us on this 

Hi Manu,

I will review the information you provided tomorrow and get back to you.

Hi Manu,

Can you please try to add "rs485-rx-during-tx" property in the &uart2 node to see if this resolves the issue?

ok will check and get back.. for now i will change padmux to pullup as per our design

We got to our most significant "root cause" fairly quickly. We found that the RxD output from the RS-485 transceiver has no pull-up on the board to keep it from floating when DE/RE-not was raised high. (Note, as is very typical in RS-485 applications, DE and RE-not are tied together in our design.) Manu was able to program the pad mux to provide the pull-up and the original anomaly was eliminated. During qualification testing we discovered more though. We're seeing two different cases of uncommanded signals on the DE/RE-not line that are (a) sometimes causing communications errors and (b) are unexplained even when they don't.

1. We occasionally see a short positive pulse on DE/RE-not. When it occurs in the middle of a receive packet, it garbles the message, forcing an unnecessary retry. When it occurs when the RS-485 bus is idle, it does no harm, but of course shouldn't be happening.
2. We also occasionally see DE/RE-not remain high for a significant time after the transmitted packet is completed. In some case this lingering drive enable almost reaches to the reply coming in on the bus. Again, this particular anomaly is not currently causing us problems, but if our downstream module replied just a tad quicker, it would.

It appears that these anomalies are occurring at the driver level. We've looked at our application layer code very carefully and nothing is touching the DE/RE-not signal. We'd appreciate your advice and help in tracking this down.

We also had a question about the value of the selectable pull-up that the pad mux provides.

Thanks in advance for the help.

John R. Graham
Siemens

Hi John,

Bin is out of office until April 29th. Let me see if someone on the hardware team can take a look. Please expect a delayed response.

Hello John,

Please refer below FAQ and E2Es regarding the leakage current with pullup or pulldown enabled.

(+) [FAQ] AM3351 , AM3352 , AM3354 , AM3356 , AM3357 , AM3358 , AM3359 Custom board design – E2E, FAQ references and design guidelines for reference during custom board design - Processors forum - Processors - TI E2E support forums

E2Es related to pullup/pulldown

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/532832/am572x-i-o-pullup-pulldown-resistor-value

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1089444/am3354-gpmc_a9-internal-pull-down-resistor/5733402

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1282287/am3354-am335x-internal-pull-up-pull-down-resistors-resistance/4862944

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/253870/resistance-of-am335x-internal-pull-ups-pull-downs

5.7 DC Electrical Characteristics

Regards,

Sreenivasa 

Hello John,

The weak internal pulls may not be able to hod the attached device input in a known state.

Checking if you have had a chance to add an external pull and perform testing.

Regards,

Sreenivasa 

Hi Sreenivasa,

Very first thing, we installed a proper pull-up resistor. The floating Rx issue was immediately resolve. By experimentation we discovered that we got identical behavior with the pin-mux-commanded internal pull-up. The anomalies I've reported most recently here are occurring in both configurations.

Regards,
John

Hello John, 

Thank you.

Trying to understand the frequency of occurrence. Are you updating the DE/RE every time data is sent or at the start of transmission etc.

Can you please explain the sequence.

Regards,

Sreenivasa

Part Number: AM3357

Continued from https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1633431/am3357-uart-driver-issue---am335x which was closed due to non-response on my part.

Kallikuppa Sreenivasa had written:
Hello John, 

Thank you.

Trying to understand the frequency of occurrence. Are you updating the DE/RE every time data is sent or at the start of transmission etc.

Can you please explain the sequence.

Regards,
Sreenivasa 

--------------------
Hi Sreenivasa,

The frequency of occurrence is relatively modest, perhaps every vew dozen messages we send out on the RS-485 link. In our current patch release, we retry the command to the device from which we received the corrupted message and this almost always succeeds. We're just looking to find the root cause.

Regarding how we drive the DE/RE-not line (single signal tied together on the board), we don't. None of our application layer code touches that signal. This is all handled at the driver level, so we're looking for some assistance / insight at that level.

We see three variants of the anomaly, all related to DE being high when it shouldn't be:

1. DE glitches high and then back low outside of any incoming receive packet. Harmless but still cause for concern.
2. DE glitches high and then back low inside the bounds of an incoming packet. Causes parity and/or framing errors. Now mitigated by retries.
3. DE remains high for too long after the end of an outgoing command packet, causing the incoming packet to be damaged. Also now mitigated by retries. 

Regards,
John

Hi John,

Manu kumar said:

rts-gpio = <&gpio0 22 GPIO_ACTIVE_HIGH>;

Do you have internal pull enabled in the gpio0_22 pinmux setting?

If yes, I am wondering if the DE glitch issue is caused by the kernel uart driver framework. Is it possible that you can modify your board to use UART2 native RTS pin for the DE control instead of using the gpio pin?

Hi Bin Liu,

Sometime during my own investigations this last month I realized that our design was not using the most correct / natural implementation (native RTS). Alas, we have shipped significant volume so switching is not logistically feasible on this product. We need to address the issues using the existing pin.

I will ask Manu to comment on the pinmux question, but it shouldn't matter, right? From our MSO captures, is seems clear that DE is being actively driven.

Regards,
John

Bin Liu  Yes we are using gpio0_22 which is configured as DE_RE_SEL_1 and its enabled with the PADCONF as  AM33XX_PADCONF(AM335X_PIN_GPMC_AD8, PIN_OUTPUT_PULLDOWN, MUX_MODE7)

Thank you Manu.

Hi John,

I remembered several years ago, I debugged a couple RS485 GPIO based DE pin control issues in older kernels, turned out the problems were due to kernel UART framework bugs in GPIO based RTS control. I don't remember the kernel versions which had the problems, but the solution was to move to native RTS pin for DE.

For this issue on your board, I am thinking of two ways to debug it, either testing with the latest kernel or modifying the board to use native RTS to see if the issue still happens.

1. modify one board to use native RTS to see if the DE pin glitch still happens. If not, very likely the GPIO DE pin glitch is caused by software;

2. port the latest kernel to your board to see if the GPIO DE pin glitch still happens. If not, it is a software issue which has been fixed in new kernel. If it still happens, we will figure out how to debug from here.

Of cause, you can skip #1 and directly test #2 if porting the latest kernel has less effort than modifying the board.

Unlocking the thread for any update.

Hi Bin Liu,

Manu identified a commit ID that appeared to be about the subject matter discussed. He said

As per John's suggestions, I analyzed the UART driver and found that SEM3T is using Omap-Serial driver for UART development. I looked at the changes to this driver for the current kernel and compared it with SEM3T version. Found a few of them, out of which I felt one commit was towards RS485 settings.

The commit ID for of this RS-485 is 51f93776b84d. This change ensures DT configured RS485 flags aren't accidentally over ridden. All other changes are not related to RS485 handshake. I am planning to cherry pick this commit into my current file and try compiling the kernel with this updated driver file.

We cherry-picked this commit and integrated it into our kernel source but it did not affect the problem. We're still seeing exactly the same symptoms. So we may have mis-identified the  commit. Our request is that you help us identify the exact commit IDs that you recall fixed this problem.

Hi John,

I review the commit 51f93776b84d, but don't feel it directly links to the issue you have. The commit is about RS485 config during init but the issue you have is at UART runtime.

As discussed in the meeting, here is the debug plan I proposed in order:

1. In your current kernel device tree, remove "rs485-rts-active-high;" but add "rs485-rx-during-tx". Setting SER_RS485_ENABLED flag in your application rs485_settings to enable RS485 mode. The relevant e2e thread is

 AM3351: RS485 spurious receive data problem and test case 

2. Test the same use case with kernel v6.12 in AM335x SDK11.02, to see if the issue still happens. If not, try to identify which kernel patch(es) resolve the issue and back port.

3. If the issue still happens in v6.12, debug the UART runtime using kernel FTRACE to get the correlation of RS485 packet transfers with GPIO DE pin control to see if it is the GPIO DE pin control issue in the UART driver.