Linux/AM3352: Simultaneous RS-485 and RS-422 communication

Yes, it's a custom board.
If you mean rtsn pin, it's wired to CPU only in UART3, I am not sure if it's pulled up/down in other UARTs (I am not designer of this board) but it's uncontrollable from cpu - that's for sure.
About release tag, I'm not sure which tag you have in mind? (sorry if it's obvious - I'm not a pro, just self-learner)
Maybe it's possible to use both drivers, but to distinguish them somehow between those devices?

>> Yes the RTS pin is what I am meaning. After looking at the DTS pin mux definitions, as a quick check of the mux modes for the RTS signal on UART3, the mux mode is 7 for the RTS signal which is gpio mode. Have you tried mux mode 6 for this pin? 

I have just checked it on both muxmodes (6 and 7) and it doesn't change anything. UART3 is set in device tree with "rts-gpio = <&gpio2 17 0>;" so I think it should be in gpio mode (muxmode 7)

>> You also will want to use the 8250 omap serial driver, the other is now considered legacy code.

Ok, so I'm using the first configuration with 8250.

I have tried to probe with external device if there is any 3.3V signal on RTS pin and there is nothing. For comparison I have once more used the legacy driver and there is 3.3V (blinking diode on my tool) when transferring data (either direction).

I am wondering if in device tree node property compatible = "ti,am3352-uart", "ti,omap3-uart"; is set correctly... in Documentation/devicetree/bindings/serial/rs485.txt there is example for atmel at91. Loooks similar but there is no list of compatible values.

Sorry for such long delay in checking this, but I finally had possibility to check this out.

Now my device tree for this node looks like this:

uart3_pins: pinmux_uart3_pins {
    pinctrl-single,pins = <
    0x160 (PIN_INPUT_PULLUP | MUX_MODE1)            /* (C15) spi0_cs1.uart3_rxd */
    0x164 (PIN_OUTPUT_PULLDOWN | MUX_MODE1)         /* (C18) ecap0_in_pwm0_out.uart3_txd */
    0xcc ( PIN_OUTPUT_PULLDOWN | MUX_MODE6 )        /* (U4)  lcd_data11.uart3_rtsn */


uart3: serial@481a6000 {                // RS485
    pinctrl-names = "default";
    pinctrl-0 = <&uart3_pins>;
    status = "okay";
    rs485-rts-delay = <200 200>;
    linux,rs485-enabled-at-boot-time;
    /* rts-gpio = <&gpio2 17 0>; */
    rs485-rts-active-high;
};

But it behaves exactly the same as earlier.

here is decompiled device tree fragment (to include things that came from am33xx.dtsi)

serial@481a6000 {
    compatible = "ti,am3352-uart", "ti,omap3-uart";
    ti,hwmods = "uart4";
    clock-frequency = <0x2dc6c00>;
    reg = <0x481a6000 0x2000>;
    interrupts = <0x2c>;
    status = "okay";
    pinctrl-names = "default";
    pinctrl-0 = <0x2d>;
    rs485-rts-delay = <0xc8 0xc8>;
    linux,rs485-enabled-at-boot-time;
    rs485-rts-active-high;
};

Everything seems to be fine, but still rts is not working.

here is the schematic:

I have connected Logic analyzer to pins Tx, Rx, rts and cts.

only result I got is here:

CTS and RTS didn't move even a bit. For comparison (and testing if i connected it properly) I have recompiled kernel with legacy driver (SERIAL_OMAP), I have high signal on RTS during send but still nothing on CTS.

Here is my device tree for this node with CTS pin:

uart3: serial@481a6000 {                //user RS485
               pinctrl-names = "default";
               pinctrl-0 = <&uart3_pins>;
               status = "okay";
               rs485-rts-delay = <200 200>;
               /* linux,rs485-enabled-at-boot-time; */
               rts-gpio = <&gpio2 17 0>;
               rs485-rts-active-high;
       };

uart3_pins: pinmux_uart3_pins {
                    pinctrl-single,pins = <
                            0x160 (PIN_INPUT_PULLUP | MUX_MODE1)            /* (C15) spi0_cs1.uart3_rxd */
                            0x164 (PIN_OUTPUT_PULLDOWN | MUX_MODE1)         /* (C18) ecap0_in_pwm0_out.uart3_txd */
                            0xcc ( PIN_OUTPUT_PULLDOWN | MUX_MODE6 )        /* (U4)  lcd_data11.uart3_rtsn */
                            0xc8 ( PIN_OUTPUT_PULLDOWN | MUX_MODE6 )        /* (U3)  lcd_data10.uart3_ctsn */
                            >;
            };

Marek,

I helped Schuyler recreate this using a BBB. I used UART1 as the pins are available on the P9 header (P9-19 and P9-24 to be exact). 

Here are the DTS changes I made:

diff --git a/arch/arm/boot/dts/am335x-boneblack.dts b/arch/arm/boot/dts/am335x-boneblack.dts
index 1230fd0..27df4e7 100644
--- a/arch/arm/boot/dts/am335x-boneblack.dts
+++ b/arch/arm/boot/dts/am335x-boneblack.dts
@@ -86,6 +86,24 @@
 			AM33XX_IOPAD(0x86c, PIN_OUTPUT_PULLDOWN | MUX_MODE7) /* gpmc_a11.GPIO1_27 */
 		>;
 	};
+
+	uart1_pins_default: uart1_pins_default {
+		pinctrl-single,pins = <
+			0x180 ( PIN_INPUT | MUX_MODE0 ) /* (D16) uart1_rxd.uart1_rxd */
+			0x184 ( PIN_OUTPUT | MUX_MODE0 ) /* (D15) uart1_txd.uart1_txd */
+			0x178 ( PIN_INPUT | MUX_MODE0 ) /* (D18) uart1_ctsn.uart1_ctsn */
+			0x17c ( PIN_OUTPUT | MUX_MODE0 ) /* (D17) uart1_rtsn.uart1_rtsn */
+		>;
+	};
+};
+
+&uart1 {                // RS485
+    pinctrl-names = "default";
+    pinctrl-0 = <&uart1_pins_default>;
+    status = "okay";
+    rs485-rts-delay = <200 200>;
+    linux,rs485-enabled-at-boot-time;
+    rs485-rts-active-high;
 };
 
 &lcdc {

Here's what we captured:

As you can see, we see RTS strobing as we would expect.

It seems that something must be different in your set up. If you have a Beaglebone Black, you should be able to recreate the test above easily enough.

We hope this is helpful.

I have checked your configuration on my BBB and it seems to be working fine on logic analyser. But...
Lately, for most of time, to observe pins I was sending some strings to appropriate ttySx using echo - I used two grouped terminals, to send in both directions in about 100ms interval (first Tx than Rx).
When it seemed (by the logic analyser plot) to be working fine, I have moved to connecting to terminal myself using screen. This worked perfectly on old SERIAL_OMAP module, but now when I start screen, I noticed, the RTS pin is pulled down permanently. RTS will be back in high state only after reboot. Earlier (when I posted this topic) I was mainly using screen for this, but the RTS must have been reverted - that was why I was able to read but I was unable to send anything, now after some changes it's oposite - after I launch screen, RTS will be permanently down so only sending will be possible. 

I think my main question of this topic will now change a bit: Is this normal what happens? is it screen's fault? Is there a way to 'unlock' RTS without reboot?

I really appreciate your help,
Kind regards,
Marek

PS: We have made some minor changes in HW, but as it's working when echoing to each other I won't get into details of changes.

Yes, it's still working as I described it first... But currently I am using the legacy driver for this one RS485 serial and the 8250 for all others. 
I don't want to waste any more of your time as it could be something on my side (considering it's working fine on your end).

Only one last question: If you take two BBBs, connect them with RS and open tty using screen: is the RTSn pin behaving properly?