This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Linux/AM4378: Spurious EDMA trigger from UART

Part Number: AM4378


Tool/software: Linux

I am using a custom board based on the TI EVM for AM437x.

I have tested using the linux kernel from ti-processor-sdk-linux-am437x-evm-04.00.00.04, and newer versions (4.12.y, 4.13.y, 4.14.y) from the linux mainline repo.  All exhibit the following behaviour.

I need to receive "high speed" (1.5Mbs - 3Mbs) data through the UART (ttyS2).  I get RX overruns using the default interrupt driven receiver, so I have enabled DMA for that UART in the device tree.  The serial ports have been configured to pass raw data, and all flow control has been turned off.  I have captured the serial data on a logic analyzer and the transmitted data is correct.

With DMA enabled, the transfers contain extra NULL bytes.  That is, when I send a payload of 8192 bytes, I sometimes receive more that 8192 bytes.  The extra bytes are always 0x00's, and the remain data is correct.

To test, I generated an 8k file that contains the following string "0123456789ABCDEF" repeated 512 times.  Here is a hexdump of the source file:

hexdump -C /root/8k
00000000  30 31 32 33 34 35 36 37  38 39 41 42 43 44 45 46  |0123456789ABCDEF|
*
00002000

I sent and received the data using the following command:

cat /dev/ttyS2 > /tmp/dump & sleep 1 ; cat /root/8k > /dev/ttyS4 ; sleep 10 ; kill -HUP %1 ; sleep 2 ; wc /tmp/dump ; hexdump -C /tmp/dump

The sleeps are just to make sure the processes are "ready".  Here is an example output of the above command when the extra 0x00's are captured:

   0    1 8248 /tmp/dump
00000000  30 31 32 33 34 35 36 37  38 39 41 42 43 44 45 46  |0123456789ABCDEF|
*
000000c0  00 30 31 32 33 34 35 36  37 38 39 41 42 43 44 45  |.0123456789ABCDE|
000000d0  46 30 31 32 33 34 35 36  37 38 39 41 42 43 44 45  |F0123456789ABCDE|
*
00000140  46 00 00 00 00 30 31 32  33 34 35 36 37 38 39 41  |F....0123456789A|
00000150  42 43 44 45 46 30 31 32  33 34 35 36 37 38 39 41  |BCDEF0123456789A|
*
00000720  42 43 44 45 46 30 31 00  00 00 00 00 32 33 34 35  |BCDEF01.....2345|
00000730  36 37 38 39 41 42 43 44  45 46 30 31 32 33 34 35  |6789ABCDEF012345|
*
00001380  36 37 38 39 41 42 43 44  45 46 00 00 00 00 00 00  |6789ABCDEF......|
00001390  00 30 31 32 33 34 35 36  37 38 39 41 42 43 44 45  |.0123456789ABCDE|
000013a0  46 30 31 32 33 34 35 36  37 38 39 41 42 43 44 45  |F0123456789ABCDE|
*
000015d0  46 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |F...............|
000015e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000015f0  00 30 31 32 33 34 35 36  37 38 39 41 42 43 44 45  |.0123456789ABCDE|
00001600  46 30 31 32 33 34 35 36  37 38 39 41 42 43 44 45  |F0123456789ABCDE|
*
00001970  46 00 00 00 00 00 00 00  30 31 32 33 34 35 36 37  |F.......01234567|
00001980  38 39 41 42 43 44 45 46  30 31 32 33 34 35 36 37  |89ABCDEF01234567|
*
00002038

The extra 56 bytes are 0x00's.  The other 8192 bytes match the source file.  As seen above, the extra 0x00's are inserted into the stream at multiple locations.

With the standard kernel module, the Rx DMA size is set to 48 bytes with a 48 byte FIFO threshold and 48 byte DMA burst size.  At the end of one DMA transfer, the driver directly reads any remaining bytes from the UART FIFO, then enables another DMA transfer.

With a customized kernel, I have determined that the extra 0x00's occur during the DMA of data from the UART FIFO to memory, not during the direct reads.  Also, the extra 0x00's have not appeared in the first DMA transfer (after opening the serial port).  I think the 0x00s are caused by the DMA reading from an empty FIFO.

I have also tested with a larger DMA buffer size of 1k with various FIFO thresholds and matching DMA burst sizes.  In these tests, the first 1k of data is always good, and the directly read bytes from the FIFO between the end of one DMA transfer and the start of the next is always good.  The extra 0x00's may occur anywhere during the DMA transfer (e.g starting 252 bytes into a 1k buffer with threshold = 16 burst = 16).

It appears that the DMA is triggered before the FIFO has enough data.  Perhaps the DMA is incorrectly triggered at the start of the subsequent transfers, and it take a few transfers to "catch" up and overtake the FIFO.

I haven't found any errata directly related to the above problem, and the TRM is somewhat vague about the Rx Trigger from the UART.

Is this a spurious trigger?  Are there work-arounds?  Anyone else notice this problem?

  • We're looking into this. Feedback will be posted here.

    Best Regards,
    Yordan
  • Hi Patrick,

    Thank you for your patience.

    What are you termios settings? It is possible NULL characters are getting injected by the driver to indicate a break condition, or parity or framing errors [1].

    In an attempt to better determine if you're hitting one of these conditions, my understanding is you could set your termios c_iflag the following:

    config.c_iflag &= ~(IGNBRK | BRKINT | IGNPAR | ISTRIP );
    config.c_iflag |= (PARMRK | INPCK);

    Instead of a single NULL character, you'd see extra characters to indicate a comm error, and then could take appropriate action in code.

    [1] man7.org/.../termios.3.html
  • Hi Michael,

    My termios iflag setting is: (IGNPAR | IGNBRK).  I set it this way to get "raw" 8-bit input and to avoid insertion of "markers". but I will try the same test with the settings you suggest.

    As far as I can tell, no comm errors occur on the line because I receive all of the data with sets of extra 0x00s inserted in the stream.

    I have also tested with a modified driver that reports the stats (rx, overflow, parity, framing, etc) whenever the port is closed, and these show that no comm errors occurred during the tests.

    Here is a hexdump of a capture with the settings you suggested above:

    00000000  36 37 38 39 41 42 43 44  45 46 30 31 32 33 34 35  |6789ABCDEF012345|
    *
    00000110  36 37 38 39 41 42 00 43  00 00 00 00 00 00 00 00  |6789AB.C........|
    00000120  44 45 46 30 31 32 33 34  35 36 37 38 39 41 42 43  |DEF0123456789ABC|
    *
    00000800
    

    As you can see, there are no 0xFF markers indicating comm errors, and all of the data is present, and extra 0x00's are inserted in the stream.

    I am going to try a version of the driver with cyclic DMA transfers (based on a patch from John Ogness: http://lkml.iu.edu/hypermail/linux/kernel/1601.2/04139.html

    Patrick

  • I tried to use the cyclic DMA transfer mentioned above, but this driver suffered the same extra 0x00's.  They appear to occur when the line goes idle just long enough to trigger an RX_TIMEOUT interrupt before more data arrives.

    To work around the problem, I used the fsl_lpuart code as a guide and did the following:

    • reduced the DMA burst size and FIFO trigger level to 1 (to avoid RX_TIMEOUT due to "stale" data in FIFO)
    • set the DMA length to approximately 10ms worth of data at configured baud rate.
    • DMA completion interrupt flips data to tty level
    • added a linux timer interrupt to flip data to tty level after 20ms (granularity is 10ms).
    • restart the timer when a DMA interrupt occurs.

    With the above, I am able to stream data from one port to another at 3Mbaud without loss or additional 0x00s at a throughput of about 284kB/s.

    I still haven't found a way to properly use DMA with the RX_TIMEOUT interrupt.  Any suggestions?

    Patrick

  • Patrick,
    We recently added advisory 27
    www.ti.com/.../sprz408c.pdf

    I wonder if you hitting this errata.

    Regards
    Mukul
  • Hi Mukul,

    I have considered that errata early in my investigation.  I verified that the stock drivers were already checking and acknowledging those spurious interrupts.  The problem appears to occur when an actual RX_TIMEOUT interrupt occurs.

    The only way I have been able to avoid the problem is to use a FIFO threshold of 1 so that no RX_TIMEOUT interrupts occur.

    Patrick

  • Patrick,

    Discussing with other engineers internally, this issue has come up recently, and actively debugging.

    UART instances 1-3 are physically implemented as a USART, while UART instances 0, 4 and 5 are standard UART. At this time it appears this issue is isolated to the USART instances. Are you able to test with one of the UART instances?

    Regards,
    Mike
  • Hi Mike,

    I modified my device tree file to enable DMA on UART 4 and ran my tests  (at 3Mbs) again from UART2 to UART4.  I do not see extra 0x00s in this direction, so the result supports your the USART vs. UART hypothesis.

    I do have overrun errors while receiving, but this is expected since this driver has to issue a new DMA transfer every 48 bytes with very little tolerance to latency at this data rate.

    These tests use the stock drivers built from the TI git repo (linux version 4.9.59-05832-g153e6c41224f).

    Patrick

  • Patrick,

    Thank you sharing your test results!

    Regards,
    Mike