This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TMS320F28388D: Faulty LWIP F2838X port (tested both UDP and TCP/IP via MQTT)

Part Number: TMS320F28388D
Other Parts Discussed in Thread: C2000WARE

Hello all.

As you well know from these threads

https://e2e.ti.com/support/microcontrollers/c2000/f/c2000-microcontrollers-forum/932820/cm-ethernet-issue-lwip

https://e2e.ti.com/support/microcontrollers/c2000/f/c2000-microcontrollers-forum/915952/tms320f28388d-tcp-client-running-on-f2838xd-cm-cannot-connect-to-a-server

https://e2e.ti.com/support/microcontrollers/c2000/f/c2000-microcontrollers-forum/880995/tms320f28388d-tcp-communication-work-slow/3266744

There is a serious issue about Lwip implementation on F2838X family, acknowledge by Texas Instruments over UDP. However, I have been testing MQTT example provided on the same library, which uses TCP and the issue is also present there.

The messages are lost in the stack, and lwip loses control of the connection because of the port provided by Texas Instruments.

To provide an example, if I run the program against a Mosquitto broker on a PC connected to the F28388D device, the connection fails after some time. We have seen that MQTT messages arrive to the broker but suddenly, a message (TCP) get lost in the stack and a retransmission starts by the lwip MQTT app....however, the stack at this point is corrupted, so all messages are lost and the CM4 core freezes waiting for some event to happen.

Luckily, some times the lost connection is detected (or the disconnection message from the broker is received), other times, the disconnection message is not received at the lwip app level, and the CM4 core waits for ever.

Depending on the circumstances, we have seen the system stuck at tcp_output function because useg->next is pointing to itself!!!

The problems are now demostrated to be on TCP and UDP.

I hope there is a strong effort in correcting and providing a realiable software, since many users are reporting these issues. The reason we decided to use F2838X family (and many others) is the Ethernet connectivity to talk to the world and connect easily.

This issue is taking too long, almost a year and no solution yet.

Thank you

  • Depending on the circumstances, we have seen the system stuck at tcp_output function because useg->next is pointing to itself!!!

    The same issue was reported in TM4C129 Launchpad lwIP Buffer issue with high TCP/IP network traffic

    The referenced thread suggests the issue could be caused by calling Lwip from difference contexts because Lwip is not re-entrant.

    I don't have a F2838X device at the moment, but does the failure occur on an unmodified example from TI (in which cause could look at the code)?

  • Thank you Chester.

    Yes I am very aware that lwip is not reentrant. We are running MQTT over TCP that comes with Lwip library included in C2000ware 3.04.00....so you can test it your self. On light and modest (and of course on heavy) traffic or when the cable is disconnected, the program get stuck and many packets are lost in the buffer.

    I am also investigating aboutsome worng values of ERR_OK and ERR_CONN that these MQTT and TCP functions are passing up as return. TCP says is always disconnected....while it is connected even at application level!!

    We are calling our MQTT functions from SysTickIntHandler which calls lwIPTimer, which comes from the main loop. On the other hand, and also stated in e2e forums is that SysTickIntHandler  counter value provided in the library is wrong (and still not corrected), and so the lwIPTimer.

    Regards

  • Pak,

    an owner has been assigned. They are out of the office today, and tomorrow is a holiday. Expect a reply 4/14.


    Regards,
    Cody 

  • The messages are lost in the stack, and lwip loses control of the connection because of the port provided by Texas Instruments.

    When the device receives high traffic there is a buffer overflow condition(generic ISR) and it tries to recover for the scenario in which the fifo ring put and get indices might be going out of sync.

    I am attaching an updated lwip Project with following changes. Can you try this once.

    1205.enet_lwip.zip

    1) Complete RX DMA is being initialized in the GenericISR.

    2) Updated the pcbGetBuffer to not use custom.

    3) Pushing the same received packet to queue that was already used in f2838xif_receive() to maintain consistency in HW descriptor chain.

    4) Added necessary low level drivers to enet_lwip.c

    Regards,

    Yashwant

  • Hello Yashwant.

    Could you just point to the specific files that changes. We do not want to overwrite anything not needed, since our project is updated with our own MQTT or UDP application.

    Thank you

  • Hello Yashwant. I changed the code and now I see a lot of improvement.

    I have tested the pings and UDP unicorn up to the limit and the CM4 core recovers and continous pinging. Congratulations on a good work.

    On the other hand, I would like to know if you have tested it in long term and how. I would need to check this before going into deployment with Ethernet.

    Best regards

  • PAk,

    We performed ping tests along with TCP http, ICMP packet attack tests. Still overnight tests remain.

    You can expect this update for next release.

    Regards,

    Yashwant

  • Bad news.

    It breaks after a while because of Memory leak(?)

    Error

    This  solution without proper testing is not valid for production or deployment.

    Regards

  • We are looking into this and will get back to you.

    Regards,

    Yashwant

  • How often is this issue occurring. Can you provide us the steps to recreate this. We have put the device to overnight packet transmission and receiving and have not see the issue. 

    Could it be possible that your updated application has some mem_alloc without mem_free?

    Regards,

    Yashwant

  • Hello.

    Over our UDP application, which sends ~8kB every 5ms, we created a MQTT app which just sends data to a topic after subscription. The packets are around 180bytes (also every 5ms).

    When we enable MQTT using the libray included in lwip (of course with the latest changes you provided on the Eth driver), we see it crashing maybe after 1 minute or similar.

    What I see is that lwip MQTT uses altcp_write, alttcp_out, alt_tcp_abort and similar. I have not found any other mem_alloc and mem_free than in the f2838xif.c file but these:

    Which are only called once at the program.

    From my point of view, maybe the issue is coming from TCP signaling between broker and MQTT client in the CM4 core

  • Can you replace __asm("   bkpt #0"); with return(ERR_MEM); in f2838xif_transmit() and see if the lwip stack does some recovery. Seems like the mqtt is not properly integrated else without that as well we should be seeming memory error.

  • Hello Yashwant.

    I did what you suggested, however the program crashes on a FaultISR but I cannot find why.

    I just copied you the Registers state.

    521177 14
    R PC 0x0000000B 0x002133E2
    R SP 0x0000000B 0x1FFFD648
    R LR 0x0000000B 0xFFFFFFF9
    R xPSR 0x0000000B 0x21000003
    R R0 0x0000000B 0x04F1ADB5
    R R1 0x0000000B 0x0000000F
    R R2 0x0000000B 0x0000000F
    R R3 0x0000000B 0x00000003
    R R4 0x0000000B 0x00000000
    R R5 0x0000000B 0x00000000
    R R6 0x0000000B 0x00000000
    R R7 0x0000000B 0x1FFFD708
    R R8 0x0000000B 0x00000000
    R R9 0x0000000B 0x00000000
    R R10 0x0000000B 0x00000000
    R R11 0x0000000B 0x00000000
    R R12 0x0000000B 0x200021F8
    R R13 0x0000000B 0x1FFFD648
    R R14 0x0000000B 0xFFFFFFF9
    R MSP 0x0000000B 0x1FFFD648
    R PSP 0x0000000B 0x00000000
    R DSP 0x0000000B 0x00000000
    R CTRL_FAULT_BASE_PRI 0x0000000B 0x00000000
    R AES_AES_KEY2_6 0x0000000B 0x00000000
    R AES_AES_KEY2_7 0x0000000B 0x00000000
    R AES_AES_KEY2_4 0x0000000B 0x00000000
    R AES_AES_KEY2_5 0x0000000B 0x00000000
    R AES_AES_KEY2_2 0x0000000B 0x00000000
    R AES_AES_KEY2_3 0x0000000B 0x00000000
    R AES_AES_KEY2_0 0x0000000B 0x00000000
    R AES_AES_KEY2_1 0x0000000B 0x00000000
    R AES_AES_KEY1_6 0x0000000B 0x00000000
    R AES_AES_KEY1_7 0x0000000B 0x00000000
    R AES_AES_KEY1_4 0x0000000B 0x00000000
    R AES_AES_KEY1_5 0x0000000B 0x00000000
    R AES_AES_KEY1_2 0x0000000B 0x00000000
    R AES_AES_KEY1_3 0x0000000B 0x00000000
    R AES_AES_KEY1_0 0x0000000B 0x00000000
    R AES_AES_KEY1_1 0x0000000B 0x00000000
    R AES_AES_IV_IN_OUT_0 0x0000000B 0x00000000
    R AES_AES_IV_IN_OUT_1 0x0000000B 0x00000000
    R AES_AES_IV_IN_OUT_2 0x0000000B 0x00000000
    R AES_AES_IV_IN_OUT_3 0x0000000B 0x00000000
    R AES_AES_CTRL 0x0000000B 0x00000000
    R AES_AES_C_LENGTH_0 0x0000000B 0x00000000
    R AES_AES_C_LENGTH_1 0x0000000B 0x00000000
    R AES_AES_AUTH_LENGTH 0x0000000B 0x00000000
    R AES_AES_DATA_IN_OUT_0 0x0000000B 0x00000000
    R AES_AES_DATA_IN_OUT_1 0x0000000B 0x00000000
    R AES_AES_DATA_IN_OUT_2 0x0000000B 0x00000000
    R AES_AES_DATA_IN_OUT_3 0x0000000B 0x00000000
    R AES_AES_TAG_OUT_0 0x0000000B 0x00000000
    R AES_AES_TAG_OUT_1 0x0000000B 0x00000000
    R AES_AES_TAG_OUT_2 0x0000000B 0x00000000
    R AES_AES_TAG_OUT_3 0x0000000B 0x00000000
    R AES_AES_REV 0x0000000B 0x00000000
    R AES_AES_SYSCONFIG 0x0000000B 0x00000000
    R AES_AES_SYSSTATUS 0x0000000B 0x00000000
    R AES_AES_IRQSTATUS 0x0000000B 0x00000000
    R AES_AES_IRQENABLE 0x0000000B 0x00000000
    R AES_AES_DIRTY_BITS 0x0000000B 0x00000000
    R AES_SS_AESDMAINTEN 0x0000000B 0x00000000
    R AES_SS_AESDMASTATUS 0x0000000B 0x00000000
    R AES_SS_AESDMASTATUSCLR 0x0000000B 0x00000000
    R CANA_CAN_CTL 0x0000000B 0x00000000
    R CANA_CAN_ES 0x0000000B 0x00000000
    R CANA_CAN_ERRC 0x0000000B 0x00000000
    R CANA_CAN_BTR 0x0000000B 0x00000000
    R CANA_CAN_INT 0x0000000B 0x00000000
    R CANA_CAN_TEST 0x0000000B 0x00000000
    R CANA_CAN_PERR 0x0000000B 0x00000000
    R CANA_CAN_RAM_INIT 0x0000000B 0x00000000
    R CANA_CAN_GLB_INT_EN 0x0000000B 0x00000000
    R CANA_CAN_GLB_INT_FLG 0x0000000B 0x00000000
    R CANA_CAN_GLB_INT_CLR 0x0000000B 0x00000000
    R CANA_CAN_ABOTR 0x0000000B 0x00000000
    R CANA_CAN_TXRQ_X 0x0000000B 0x00000000
    R CANA_CAN_TXRQ_21 0x0000000B 0x00000000
    R CANA_CAN_NDAT_X 0x0000000B 0x00000000
    R CANA_CAN_NDAT_21 0x0000000B 0x00000000
    R CANA_CAN_IPEN_X 0x0000000B 0x00000000
    R CANA_CAN_IPEN_21 0x0000000B 0x00000000
    R CANA_CAN_MVAL_X 0x0000000B 0x00000000
    R CANA_CAN_MVAL_21 0x0000000B 0x00000000
    R CANA_CAN_IP_MUX21 0x0000000B 0x00000000
    R CANA_CAN_IF1CMD 0x0000000B 0x00000000
    R CANA_CAN_IF1MSK 0x0000000B 0x00000000
    R CANA_CAN_IF1ARB 0x0000000B 0x00000000
    R CANA_CAN_IF1MCTL 0x0000000B 0x00000000
    R CANA_CAN_IF1DATA 0x0000000B 0x00000000
    R CANA_CAN_IF1DATB 0x0000000B 0x00000000
    R CANA_CAN_IF2CMD 0x0000000B 0x00000000
    R CANA_CAN_IF2MSK 0x0000000B 0x00000000
    R CANA_CAN_IF2ARB 0x0000000B 0x00000000
    R CANA_CAN_IF2MCTL 0x0000000B 0x00000000
    R CANA_CAN_IF2DATA 0x0000000B 0x00000000
    R CANA_CAN_IF2DATB 0x0000000B 0x00000000
    R CANA_CAN_IF3OBS 0x0000000B 0x00000000
    R CANA_CAN_IF3MSK 0x0000000B 0x00000000
    R CANA_CAN_IF3ARB 0x0000000B 0x00000000
    R CANA_CAN_IF3MCTL 0x0000000B 0x00000000
    R CANA_CAN_IF3DATA 0x0000000B 0x00000000
    R CANA_CAN_IF3DATB 0x0000000B 0x00000000
    R CANA_CAN_IF3UPD 0x0000000B 0x00000000
    R CANB_CAN_CTL 0x0000000B 0x00000000
    R CANB_CAN_ES 0x0000000B 0x00000000
    R CANB_CAN_ERRC 0x0000000B 0x00000000
    R CANB_CAN_BTR 0x0000000B 0x00000000
    R CANB_CAN_INT 0x0000000B 0x00000000
    R CANB_CAN_TEST 0x0000000B 0x00000000
    R CANB_CAN_PERR 0x0000000B 0x00000000
    R CANB_CAN_RAM_INIT 0x0000000B 0x00000000
    R CANB_CAN_GLB_INT_EN 0x0000000B 0x00000000
    R CANB_CAN_GLB_INT_FLG 0x0000000B 0x00000000
    R CANB_CAN_GLB_INT_CLR 0x0000000B 0x00000000
    R CANB_CAN_ABOTR 0x0000000B 0x00000000
    R CANB_CAN_TXRQ_X 0x0000000B 0x00000000
    R CANB_CAN_TXRQ_21 0x0000000B 0x00000000
    R CANB_CAN_NDAT_X 0x0000000B 0x00000000
    R CANB_CAN_NDAT_21 0x0000000B 0x00000000
    R CANB_CAN_IPEN_X 0x0000000B 0x00000000
    R CANB_CAN_IPEN_21 0x0000000B 0x00000000
    R CANB_CAN_MVAL_X 0x0000000B 0x00000000
    R CANB_CAN_MVAL_21 0x0000000B 0x00000000
    R CANB_CAN_IP_MUX21 0x0000000B 0x00000000
    R CANB_CAN_IF1CMD 0x0000000B 0x00000000
    R CANB_CAN_IF1MSK 0x0000000B 0x00000000
    R CANB_CAN_IF1ARB 0x0000000B 0x00000000
    R CANB_CAN_IF1MCTL 0x0000000B 0x00000000
    R CANB_CAN_IF1DATA 0x0000000B 0x00000000
    R CANB_CAN_IF1DATB 0x0000000B 0x00000000
    R CANB_CAN_IF2CMD 0x0000000B 0x00000000
    R CANB_CAN_IF2MSK 0x0000000B 0x00000000
    R CANB_CAN_IF2ARB 0x0000000B 0x00000000
    R CANB_CAN_IF2MCTL 0x0000000B 0x00000000
    R CANB_CAN_IF2DATA 0x0000000B 0x00000000
    R CANB_CAN_IF2DATB 0x0000000B 0x00000000
    R CANB_CAN_IF3OBS 0x0000000B 0x00000000
    R CANB_CAN_IF3MSK 0x0000000B 0x00000000
    R CANB_CAN_IF3ARB 0x0000000B 0x00000000
    R CANB_CAN_IF3MCTL 0x0000000B 0x00000000
    R CANB_CAN_IF3DATA 0x0000000B 0x00000000
    R CANB_CAN_IF3DATB 0x0000000B 0x00000000
    R CANB_CAN_IF3UPD 0x0000000B 0x00000000
    R ERRORLOG_UCERRFLG 0x0000000B 0x00000000
    R ERRORLOG_UCERRSET 0x0000000B 0x00000000
    R ERRORLOG_UCERRCLR 0x0000000B 0x00000000
    R ERRORLOG_UCM4EADDR 0x0000000B 0x00000000
    R ERRORLOG_UCEMACEADDR 0x0000000B 0x00000000
    R ERRORLOG_UCuDMAEADDR 0x0000000B 0x00000000
    R ERRORLOG_UCEtherCATMEMREADDR 0x0000000B 0x00000000
    R ERRORLOG_UCEMACMEMREADDR 0x0000000B 0x00000000
    R ERRORLOG_BUSFAULTFLG 0x0000000B 0x00000001
    R ERRORLOG_BUSFAULTCLR 0x0000000B 0x00000000
    R ERRORLOG_M4BUSFAULTADDR 0x0000000B 0x04F1ADB5
    R ERRORLOG_uDMABUSFAULTADDR 0x0000000B 0x00000000
    R ERRORLOG_EMACBUSFAULTADDR 0x0000000B 0x00000000
    R ERRORLOG_CERRFLG 0x0000000B 0x00000000
    R ERRORLOG_CERRSET 0x0000000B 0x00000000
    R ERRORLOG_CERRCLR 0x0000000B 0x00000000
    R ERRORLOG_CM4EADDR 0x0000000B 0x00000000
    R ERRORLOG_CEMACEADDR 0x0000000B 0x00000000
    R ERRORLOG_CuDMAEADDR 0x0000000B 0x00000000
    R ERRORLOG_CERRCNT 0x0000000B 0x00000000
    R ERRORLOG_CERRTHRES 0x0000000B 0x00000000
    R ERRORLOG_CEINTFLG 0x0000000B 0x00000000
    R ERRORLOG_CEINTSET 0x0000000B 0x00000000
    R ERRORLOG_CEINTCLR 0x0000000B 0x00000000
    R ERRORLOG_CEINTEN 0x0000000B 0x00000000
    R DcsmCommonRegs_FLSEM 0x0000000B 0x00000000
    R DcsmCommonRegs_SECTSTAT1 0x0000000B 0x0FFFFFFF
    R DcsmCommonRegs_SECTSTAT2 0x0000000B 0x0FFFFFFF
    R DcsmCommonRegs_SECTSTAT3 0x0000000B 0x0FFFFFFF
    R DcsmCommonRegs_RAMSTAT1 0x0000000B 0x000FFFFF
    R DcsmCommonRegs_RAMSTAT2 0x0000000B 0xFFFFFF0F
    R DcsmCommonRegs_RAMSTAT3 0x0000000B 0x000FFFFF
    R DcsmCommonRegs_SECERRSTAT 0x0000000B 0x00000000
    R DcsmCommonRegs_SECERRCLR 0x0000000B 0x00000000
    R DcsmCommonRegs_SECERRFRC 0x0000000B 0x00000000
    R DcsmZ1Regs_Z1_LINKPOINTER 0x0000000B 0xFFFFFFFF
    R DcsmZ1Regs_Z1_OTPSECLOCK 0x0000000B 0x00000FF0
    R DcsmZ1Regs_Z1_JLM_ENABLE 0x0000000B 0x0000000F
    R DcsmZ1Regs_Z1_LINKPOINTERERR 0x0000000B 0x00000000
    R DcsmZ1Regs_Z1_GPREG1 0x0000000B 0xFFFFFFFF
    R DcsmZ1Regs_Z1_GPREG2 0x0000000B 0xFFFFFFFF
    R DcsmZ1Regs_Z1_GPREG3 0x0000000B 0xFFFFFFFF
    R DcsmZ1Regs_Z1_GPREG4 0x0000000B 0xFFFFFFFF
    R DcsmZ1Regs_Z1_CSMKEY0 0x0000000B 0x00000000
    R DcsmZ1Regs_Z1_CSMKEY1 0x0000000B 0x00000000
    R DcsmZ1Regs_Z1_CSMKEY2 0x0000000B 0x00000000
    R DcsmZ1Regs_Z1_CSMKEY3 0x0000000B 0x00000000
    R DcsmZ1Regs_Z1_CR 0x0000000B 0x0060000F
    R DcsmZ1Regs_Z1_GRABSECT1R 0x0000000B 0x0FFFFFFF
    R DcsmZ1Regs_Z1_GRABSECT2R 0x0000000B 0x0FFFFFFF
    R DcsmZ1Regs_Z1_GRABSECT3R 0x0000000B 0x0FFFFFFF
    R DcsmZ1Regs_Z1_GRABRAM1R 0x0000000B 0x000FFFFF
    R DcsmZ1Regs_Z1_GRABRAM2R 0x0000000B 0xFFFFFF0F
    R DcsmZ1Regs_Z1_GRABRAM3R 0x0000000B 0x000FFFFF
    R DcsmZ1Regs_Z1_EXEONLYSECT1R 0x0000000B 0x3FFF3FFF
    R DcsmZ1Regs_Z1_EXEONLYSECT2R 0x0000000B 0x00003FFF
    R DcsmZ1Regs_Z1_EXEONLYRAM1R 0x0000000B 0xFFC303FF
    R DcsmZ1Regs_Z1_JTAGKEY0 0x0000000B 0xFFFFFFFF
    R DcsmZ1Regs_Z1_JTAGKEY1 0x0000000B 0x2BFFFFFF
    R DcsmZ1Regs_Z1_JTAGKEY2 0x0000000B 0x4BFFFFFF
    R DcsmZ1Regs_Z1_JTAGKEY3 0x0000000B 0x3FFFFFFF
    R DcsmZ1Regs_Z1_CMACKEY0 0x0000000B 0x00000000
    R DcsmZ1Regs_Z1_CMACKEY1 0x0000000B 0x00000000
    R DcsmZ1Regs_Z1_CMACKEY2 0x0000000B 0x00000000
    R DcsmZ1Regs_Z1_CMACKEY3 0x0000000B 0x00000000
    R DcsmZ2Regs_Z2_LINKPOINTER 0x0000000B 0xFFFFFFFF
    R DcsmZ2Regs_Z2_OTPSECLOCK 0x0000000B 0x00000FF0
    R DcsmZ2Regs_Z2_LINKPOINTERERR 0x0000000B 0x00000000
    R DcsmZ2Regs_Z2_GPREG1 0x0000000B 0xFFFFFFFF
    R DcsmZ2Regs_Z2_GPREG2 0x0000000B 0xFFFFFFFF
    R DcsmZ2Regs_Z2_GPREG3 0x0000000B 0xFFFFFFFF
    R DcsmZ2Regs_Z2_GPREG4 0x0000000B 0xFFFFFFFF
    R DcsmZ2Regs_Z2_CSMKEY0 0x0000000B 0x00000000
    R DcsmZ2Regs_Z2_CSMKEY1 0x0000000B 0x00000000
    R DcsmZ2Regs_Z2_CSMKEY2 0x0000000B 0x00000000
    R DcsmZ2Regs_Z2_CSMKEY3 0x0000000B 0x00000000
    R DcsmZ2Regs_Z2_CR 0x0000000B 0x0060000F
    R DcsmZ2Regs_Z2_GRABSECT1R 0x0000000B 0x0FFFFFFF
    R DcsmZ2Regs_Z2_GRABSECT2R 0x0000000B 0x0FFFFFFF
    R DcsmZ2Regs_Z2_GRABSECT3R 0x0000000B 0x0FFFFFFF
    R DcsmZ2Regs_Z2_GRABRAM1R 0x0000000B 0x000FFFFF
    R DcsmZ2Regs_Z2_GRABRAM2R 0x0000000B 0xFFFFFF0F
    R DcsmZ2Regs_Z2_GRABRAM3R 0x0000000B 0x000FFFFF
    R DcsmZ2Regs_Z2_EXEONLYSECT1R 0x0000000B 0x3FFF3FFF
    R DcsmZ2Regs_Z2_EXEONLYSECT2R 0x0000000B 0x00003FFF
    R DcsmZ2Regs_Z2_EXEONLYRAM1R 0x0000000B 0xFFC303FF
    R DIAGERRORLOG_DIAGERRFLG 0x0000000B 0x00000000
    R DIAGERRORLOG_DIAGERRCLR 0x0000000B 0x00000000
    R DIAGERRORLOG_DIAGERRADDR 0x0000000B 0x00000000
    R CSFR_MMSR 0x0000000D 0x00
    R CSFR_BFSR 0x0000000D 0x82
    R CSFR_UFSR 0x0000000F 0x0000
    R ESCSS_ESCSS_IPREVNUM 0x0000000B 0x00000000
    R ESCSS_ESCSS_INTR_RIS 0x0000000B 0x00000000
    R ESCSS_ESCSS_INTR_MASK 0x0000000B 0x00000000
    R ESCSS_ESCSS_INTR_MIS 0x0000000B 0x00000000
    R ESCSS_ESCSS_INTR_CLR 0x0000000B 0x00000000
    R ESCSS_ESCSS_INTR_SET 0x0000000B 0x00000000
    R ESCSS_ESCSS_LATCH_SEL 0x0000000B 0x00000000
    R ESCSS_ESCSS_ACCESS_CTRL 0x0000000B 0x00000000
    R ESCSS_ESCSS_GPIN_DAT 0x0000000B 0x00000000
    R ESCSS_ESCSS_GPIN_PIPE 0x0000000B 0x00000000
    R ESCSS_ESCSS_GPIN_GRP_CAP_SEL 0x0000000B 0x00000000
    R ESCSS_ESCSS_GPOUT_DAT 0x0000000B 0x00000000
    R ESCSS_ESCSS_GPOUT_PIPE 0x0000000B 0x00000000
    R ESCSS_ESCSS_GPOUT_GRP_CAP_SEL 0x0000000B 0x00000000
    R ESCSS_ESCSS_MEM_TEST 0x0000000B 0x00000000
    R ESCSS_ESCSS_RESET_DEST_CONFIG 0x0000000B 0x00000000
    R ESCSS_ESCSS_SYNC0_CONFIG 0x0000000B 0x00000000
    R ESCSS_ESCSS_SYNC1_CONFIG 0x0000000B 0x00000000
    R ESCSSCONFIG_ESCSS_CONFIG_LOCK 0x0000000B 0x00000000
    R ESCSSCONFIG_ESCSS_MISC_IO_CONFIG 0x0000000B 0x00000000
    R ESCSSCONFIG_ESCSS_PHY_IO_CONFIG 0x0000000B 0x00000000
    R ESCSSCONFIG_ESCSS_SYNC_IO_CONFIG 0x0000000B 0x00000000
    R ESCSSCONFIG_ESCSS_LATCH_IO_CONFIG 0x0000000B 0x00000000
    R ESCSSCONFIG_ESCSS_GPIN_SEL 0x0000000B 0x00000000
    R ESCSSCONFIG_ESCSS_GPIN_IOPAD_SEL 0x0000000B 0x00000000
    R ESCSSCONFIG_ESCSS_GPOUT_SEL 0x0000000B 0x00000000
    R ESCSSCONFIG_ESCSS_GPOUT_IOPAD_SEL 0x0000000B 0x00000000
    R ESCSSCONFIG_ESCSS_LED_CONFIG 0x0000000B 0x00000000
    R ESCSSCONFIG_ESCSS_MISC_CONFIG 0x0000000B 0x00000000
    R ETHERNET_MAC_Configuration 0x0000000B 0x0000A003
    R ETHERNET_MAC_Ext_Configuration 0x0000000B 0x00000000
    R ETHERNET_MAC_Packet_Filter 0x0000000B 0x00000000
    R ETHERNET_MAC_Watchdog_Timeout 0x0000000B 0x00000000
    R ETHERNET_MAC_Hash_Table_Reg0 0x0000000B 0x00000000
    R ETHERNET_MAC_Hash_Table_Reg1 0x0000000B 0x00000000
    R ETHERNET_MAC_VLAN_Tag_Ctrl 0x0000000B 0x00000000
    R ETHERNET_MAC_VLAN_Tag_Data 0x0000000B 0x00000000
    R ETHERNET_MAC_VLAN_Hash_Table 0x0000000B 0x00000000
    R ETHERNET_MAC_VLAN_Incl 0x0000000B 0x00000000
    R ETHERNET_MAC_Inner_VLAN_Incl 0x0000000B 0x00000000
    R ETHERNET_MAC_Q0_Tx_Flow_Ctrl 0x0000000B 0x00000000
    R ETHERNET_MAC_Rx_Flow_Ctrl 0x0000000B 0x00000000
    R ETHERNET_MAC_RxQ_Ctrl4 0x0000000B 0x00000000
    R ETHERNET_MAC_RxQ_Ctrl0 0x0000000B 0x0000000A
    R ETHERNET_MAC_RxQ_Ctrl1 0x0000000B 0x00000000
    R ETHERNET_MAC_RxQ_Ctrl2 0x0000000B 0x00000000
    R ETHERNET_MAC_Interrupt_Status 0x0000000B 0x00040000
    R ETHERNET_MAC_Interrupt_Enable 0x0000000B 0x00000000
    R ETHERNET_MAC_Rx_Tx_Status 0x0000000B 0x00000000
    R ETHERNET_MAC_PMT_Control_Status 0x0000000B 0x00000000
    R ETHERNET_MAC_RWK_Packet_Filter 0x0000000B 0x00000000
    R ETHERNET_MAC_LPI_Control_Status 0x0000000B 0x00000000
    R ETHERNET_MAC_LPI_Timers_Control 0x0000000B 0x03E80000
    R ETHERNET_MAC_LPI_Entry_Timer 0x0000000B 0x00000000
    R ETHERNET_MAC_1US_Tic_Counter 0x0000000B 0x00000063
    R ETHERNET_MAC_Version 0x0000000B 0x00000050
    R ETHERNET_MAC_Debug 0x0000000B 0x00000000
    R ETHERNET_MAC_HW_Feature0 0x0000000B 0x0E1D73F5
    R ETHERNET_MAC_HW_Feature1 0x0000000B 0x218E3965
    R ETHERNET_MAC_HW_Feature2 0x0000000B 0x22041041
    R ETHERNET_MAC_HW_Feature3 0x0000000B 0x00320031
    R ETHERNET_MAC_MDIO_Address 0x0000000B 0x0001080C
    R ETHERNET_MAC_MDIO_Data 0x0000000B 0x0000FFFF
    R ETHERNET_MAC_ARP_Address 0x0000000B 0x00000000
    R ETHERNET_MAC_CSR_SW_Ctrl 0x0000000B 0x00000000
    R ETHERNET_MAC_Ext_Cfg1 0x0000000B 0x00000002
    R ETHERNET_MAC_Address0_High 0x0000000B 0x80008000
    R ETHERNET_MAC_Address0_Low 0x0000000B 0x00F263A8
    R ETHERNET_MAC_Address1_High 0x0000000B 0x0000FFFF
    R ETHERNET_MAC_Address1_Low 0x0000000B 0xFFFFFFFF
    R ETHERNET_MAC_Address2_High 0x0000000B 0x0000FFFF
    R ETHERNET_MAC_Address2_Low 0x0000000B 0xFFFFFFFF
    R ETHERNET_MAC_Address3_High 0x0000000B 0x0000FFFF
    R ETHERNET_MAC_Address3_Low 0x0000000B 0xFFFFFFFF
    R ETHERNET_MAC_Address4_High 0x0000000B 0x0000FFFF
    R ETHERNET_MAC_Address4_Low 0x0000000B 0xFFFFFFFF
    R ETHERNET_MAC_Address5_High 0x0000000B 0x0000FFFF
    R ETHERNET_MAC_Address5_Low 0x0000000B 0xFFFFFFFF
    R ETHERNET_MAC_Address6_High 0x0000000B 0x0000FFFF
    R ETHERNET_MAC_Address6_Low 0x0000000B 0xFFFFFFFF
    R ETHERNET_MAC_Address7_High 0x0000000B 0x0000FFFF
    R ETHERNET_MAC_Address7_Low 0x0000000B 0xFFFFFFFF
    R ETHERNET_MMC_Control 0x0000000B 0x00000000
    R ETHERNET_MMC_Rx_Interrupt 0x0000000B 0x00000000
    R ETHERNET_MMC_Tx_Interrupt 0x0000000B 0x00000000
    R ETHERNET_MMC_Rx_Interrupt_Mask 0x0000000B 0x0FFFFFFF
    R ETHERNET_MMC_Tx_Interrupt_Mask 0x0000000B 0x00000000
    R ETHERNET_Tx_Octet_Count_Good_Bad 0x0000000B 0x033DC51C
    R ETHERNET_Tx_Packet_Count_Good_Bad 0x0000000B 0x0000E243
    R ETHERNET_Tx_Broadcast_Packets_Good 0x0000000B 0x00000006
    R ETHERNET_Tx_Multicast_Packets_Good 0x0000000B 0x00000000
    R ETHERNET_Tx_64Octets_Packets_Good_Bad 0x0000000B 0x00000F2F
    R ETHERNET_Tx_65To127Octets_Packets_Good_Bad 0x0000000B 0x0000000C
    R ETHERNET_Tx_128To255Octets_Packets_Good_Bad 0x0000000B 0x00000001
    R ETHERNET_Tx_256To511Octets_Packets_Good_Bad 0x0000000B 0x00002179
    R ETHERNET_Tx_512To1023Octets_Packets_Good_Bad 0x0000000B 0x0000005A
    R ETHERNET_Tx_1024ToMaxOctets_Packets_Good_Bad 0x0000000B 0x0000B134
    R ETHERNET_Tx_Unicast_Packets_Good_Bad 0x0000000B 0x0000E23D
    R ETHERNET_Tx_Multicast_Packets_Good_Bad 0x0000000B 0x00000000
    R ETHERNET_Tx_Broadcast_Packets_Good_Bad 0x0000000B 0x00000006
    R ETHERNET_Tx_Underflow_Error_Packets 0x0000000B 0x00000000
    R ETHERNET_Tx_Single_Collision_Good_Packets 0x0000000B 0x00000000
    R ETHERNET_Tx_Multiple_Collision_Good_Packets 0x0000000B 0x00000000
    R ETHERNET_Tx_Deferred_Packets 0x0000000B 0x00000000
    R ETHERNET_Tx_Late_Collision_Packets 0x0000000B 0x00000000
    R ETHERNET_Tx_Excessive_Collision_Packets 0x0000000B 0x00000000
    R ETHERNET_Tx_Carrier_Error_Packets 0x0000000B 0x00000000
    R ETHERNET_Tx_Octet_Count_Good 0x0000000B 0x033DC51C
    R ETHERNET_Tx_Packet_Count_Good 0x0000000B 0x0000E243
    R ETHERNET_Tx_Excessive_Deferral_Error 0x0000000B 0x00000000
    R ETHERNET_Tx_Pause_Packets 0x0000000B 0x00000000
    R ETHERNET_Tx_VLAN_Packets_Good 0x0000000B 0x00000000
    R ETHERNET_Tx_OSize_Packets_Good 0x0000000B 0x00000000
    R ETHERNET_Rx_Packets_Count_Good_Bad 0x0000000B 0x000032C0
    R ETHERNET_Rx_Octet_Count_Good_Bad 0x0000000B 0x000CB6CF
    R ETHERNET_Rx_Octet_Count_Good 0x0000000B 0x000CB6CF
    R ETHERNET_Rx_Broadcast_Packets_Good 0x0000000B 0x00000036
    R ETHERNET_Rx_Multicast_Packets_Good 0x0000000B 0x00000000
    R ETHERNET_Rx_CRC_Error_Packets 0x0000000B 0x00000000
    R ETHERNET_Rx_Alignment_Error_Packets 0x0000000B 0x00000000
    R ETHERNET_Rx_Runt_Error_Packets 0x0000000B 0x00000000
    R ETHERNET_Rx_Jabber_Error_Packets 0x0000000B 0x00000000
    R ETHERNET_Rx_Undersize_Packets_Good 0x0000000B 0x00000000
    R ETHERNET_Rx_Oversize_Packets_Good 0x0000000B 0x00000000
    R ETHERNET_Rx_64Octets_Packets_Good_Bad 0x0000000B 0x00003295
    R ETHERNET_Rx_65To127Octets_Packets_Good_Bad 0x0000000B 0x0000002A
    R ETHERNET_Rx_128To255Octets_Packets_Good_Bad 0x0000000B 0x00000001
    R ETHERNET_Rx_256To511Octets_Packets_Good_Bad 0x0000000B 0x00000000
    R ETHERNET_Rx_512To1023Octets_Packets_Good_Bad 0x0000000B 0x00000000
    R ETHERNET_Rx_1024ToMaxOctets_Packets_Good_Bad 0x0000000B 0x00000000
    R ETHERNET_Rx_Unicast_Packets_Good 0x0000000B 0x0000328A
    R ETHERNET_Rx_Length_Error_Packets 0x0000000B 0x00000000
    R ETHERNET_Rx_Out_Of_Range_Type_Packets 0x0000000B 0x00000000
    R ETHERNET_Rx_Pause_Packets 0x0000000B 0x00000000
    R ETHERNET_Rx_FIFO_Overflow_Packets 0x0000000B 0x00000000
    R ETHERNET_Rx_VLAN_Packets_Good_Bad 0x0000000B 0x00000000
    R ETHERNET_Rx_Watchdog_Error_Packets 0x0000000B 0x00000000
    R ETHERNET_Rx_Receive_Error_Packets 0x0000000B 0x00000000
    R ETHERNET_Rx_Control_Packets_Good 0x0000000B 0x00000000
    R ETHERNET_Tx_LPI_USEC_Cntr 0x0000000B 0x00000000
    R ETHERNET_Tx_LPI_Tran_Cntr 0x0000000B 0x00000000
    R ETHERNET_Rx_LPI_USEC_Cntr 0x0000000B 0x00000000
    R ETHERNET_Rx_LPI_Tran_Cntr 0x0000000B 0x00000000
    R ETHERNET_MMC_IPC_Rx_Interrupt_Mask 0x0000000B 0x3FFF3FFF
    R ETHERNET_MMC_IPC_Rx_Interrupt 0x0000000B 0x00000000
    R ETHERNET_RxIPv4_Good_Packets 0x0000000B 0x000032AE
    R ETHERNET_RxIPv4_Header_Error_Packets 0x0000000B 0x00000000
    R ETHERNET_RxIPv4_No_Payload_Packets 0x0000000B 0x00000000
    R ETHERNET_RxIPv4_Fragmented_Packets 0x0000000B 0x00000000
    R ETHERNET_RxIPv4_UDP_Checksum_Disabled_Packets 0x0000000B 0x00000000
    R ETHERNET_RxIPv6_Good_Packets 0x0000000B 0x00000000
    R ETHERNET_RxIPv6_Header_Error_Packets 0x0000000B 0x00000000
    R ETHERNET_RxIPv6_No_Payload_Packets 0x0000000B 0x00000000
    R ETHERNET_RxUDP_Good_Packets 0x0000000B 0x000001A6
    R ETHERNET_RxUDP_Error_Packets 0x0000000B 0x00000000
    R ETHERNET_RxTCP_Good_Packets 0x0000000B 0x00003108
    R ETHERNET_RxTCP_Error_Packets 0x0000000B 0x00000000
    R ETHERNET_RxICMP_Good_Packets 0x0000000B 0x00000000
    R ETHERNET_RxICMP_Error_Packets 0x0000000B 0x00000000
    R ETHERNET_RxIPv4_Good_Octets 0x0000000B 0x0007E5B7
    R ETHERNET_RxIPv4_Header_Error_Octets 0x0000000B 0x00000000
    R ETHERNET_RxIPv4_No_Payload_Octets 0x0000000B 0x00000000
    R ETHERNET_RxIPv4_Fragmented_Octets 0x0000000B 0x00000000
    R ETHERNET_RxIPv4_UDP_Checksum_Disable_Octets 0x0000000B 0x00000000
    R ETHERNET_RxIPv6_Good_Octets 0x0000000B 0x00000000
    R ETHERNET_RxIPv6_Header_Error_Octets 0x0000000B 0x00000000
    R ETHERNET_RxIPv6_No_Payload_Octets 0x0000000B 0x00000000
    R ETHERNET_RxUDP_Good_Octets 0x0000000B 0x00001B4F
    R ETHERNET_RxUDP_Error_Octets 0x0000000B 0x00000000
    R ETHERNET_RxTCP_Good_Octets 0x0000000B 0x0003D4D0
    R ETHERNET_RxTCP_Error_Octets 0x0000000B 0x00000000
    R ETHERNET_RxICMP_Good_Octets 0x0000000B 0x00000000
    R ETHERNET_RxICMP_Error_Octets 0x0000000B 0x00000000
    R ETHERNET_MAC_L3_L4_Control0 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer4_Address0 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr0_Reg0 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr1_Reg0 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr2_Reg0 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr3_Reg0 0x0000000B 0x00000000
    R ETHERNET_MAC_L3_L4_Control1 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer4_Address1 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr0_Reg1 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr1_Reg1 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr2_Reg1 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr3_Reg1 0x0000000B 0x00000000
    R ETHERNET_MAC_L3_L4_Control2 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer4_Address2 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr0_Reg2 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr1_Reg2 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr2_Reg2 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr3_Reg2 0x0000000B 0x00000000
    R ETHERNET_MAC_L3_L4_Control3 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer4_Address3 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr0_Reg3 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr1_Reg3 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr2_Reg3 0x0000000B 0x00000000
    R ETHERNET_MAC_Layer3_Addr3_Reg3 0x0000000B 0x00000000
    R ETHERNET_MAC_Timestamp_Control 0x0000000B 0x00002000
    R ETHERNET_MAC_Sub_Second_Increment 0x0000000B 0x00000000
    R ETHERNET_MAC_System_Time_Seconds 0x0000000B 0x00000000
    R ETHERNET_MAC_System_Time_Nanoseconds 0x0000000B 0x00000000
    R ETHERNET_MAC_System_Time_Seconds_Update 0x0000000B 0x00000000
    R ETHERNET_MAC_System_Time_Nanoseconds_Update 0x0000000B 0x00000000
    R ETHERNET_MAC_Timestamp_Addend 0x0000000B 0x00000000
    R ETHERNET_MAC_System_Time_Higher_Word_Seconds 0x0000000B 0x00000000
    R ETHERNET_MAC_Timestamp_Status 0x0000000B 0x00000000
    R ETHERNET_MAC_Tx_Timestamp_Status_Nanoseconds 0x0000000B 0x00000000
    R ETHERNET_MAC_Tx_Timestamp_Status_Seconds 0x0000000B 0x00000000
    R ETHERNET_MAC_Auxiliary_Control 0x0000000B 0x00000000
    R ETHERNET_MAC_Auxiliary_Timestamp_Nanoseconds 0x0000000B 0x00000000
    R ETHERNET_MAC_Auxiliary_Timestamp_Seconds 0x0000000B 0x00000000
    R ETHERNET_MAC_Timestamp_Ingress_Asym_Corr 0x0000000B 0x00000000
    R ETHERNET_MAC_Timestamp_Egress_Asym_Corr 0x0000000B 0x00000000
    R ETHERNET_MAC_Timestamp_Ingress_Corr_Nanosecond 0x0000000B 0x00000000
    R ETHERNET_MAC_Timestamp_Egress_Corr_Nanosecond 0x0000000B 0x00000000
    R ETHERNET_MAC_Timestamp_Ingress_Corr_Subnanosec 0x0000000B 0x00000000
    R ETHERNET_MAC_Timestamp_Egress_Corr_Subnanosec 0x0000000B 0x00000000
    R ETHERNET_MAC_PPS_Control 0x0000000B 0x00000000
    R ETHERNET_MAC_PPS0_Target_Time_Seconds 0x0000000B 0x00000000
    R ETHERNET_MAC_PPS0_Target_Time_Nanoseconds 0x0000000B 0x00000000
    R ETHERNET_MAC_PPS0_Interval 0x0000000B 0x00000000
    R ETHERNET_MAC_PPS0_Width 0x0000000B 0x00000000
    R ETHERNET_MAC_PPS1_Target_Time_Seconds 0x0000000B 0x00000000
    R ETHERNET_MAC_PPS1_Target_Time_Nanoseconds 0x0000000B 0x00000000
    R ETHERNET_MAC_PPS1_Interval 0x0000000B 0x00000000
    R ETHERNET_MAC_PPS1_Width 0x0000000B 0x00000000
    R ETHERNET_MAC_PTO_Control 0x0000000B 0x00000000
    R ETHERNET_MAC_Source_Port_Identity0 0x0000000B 0x00000000
    R ETHERNET_MAC_Source_Port_Identity1 0x0000000B 0x00000000
    R ETHERNET_MAC_Source_Port_Identity2 0x0000000B 0x00000000
    R ETHERNET_MAC_Log_Message_Interval 0x0000000B 0x00000000
    R ETHERNETSS_ETHERNETSS_IPREVNUM 0x0000000B 0x00000000
    R ETHERNETSS_ETHERNETSS_CTRLSTS 0x0000000B 0x00000380
    R ETHERNETSS_ETHERNETSS_PTPTSTRIGSEL0 0x0000000B 0x00000000
    R ETHERNETSS_ETHERNETSS_PTPTSTRIGSEL1 0x0000000B 0x00000000
    R ETHERNETSS_ETHERNETSS_PTPTSSWTRIG0 0x0000000B 0x00000000
    R ETHERNETSS_ETHERNETSS_PTPTSSWTRIG1 0x0000000B 0x00000000
    R ETHERNETSS_ETHERNETSS_PTPPPSR0 0x0000000B 0x00000000
    R ETHERNETSS_ETHERNETSS_PTPPPSR1 0x0000000B 0x00000000
    R ETHERNETSS_ETHERNETSS_PTP_TSRL 0x0000000B 0x00000000
    R ETHERNETSS_ETHERNETSS_PTP_TSRH 0x0000000B 0x00000000
    R ETHERNETSS_ETHERNETSS_PTP_TSWL 0x0000000B 0x00000000
    R ETHERNETSS_ETHERNETSS_PTP_TSWH 0x0000000B 0x00000000
    R ETHERNETSS_ETHERNETSS_REVMII_CTRL 0x0000000B 0x00000000
    R FLASHCTRL_FRDCNTL 0x0000000B 0x00000200
    R FLASHCTRL_FBAC 0x0000000B 0x0000000F
    R FLASHCTRL_FBFALLBACK 0x0000000B 0x00000003
    R FLASHCTRL_FBPRDY 0x0000000B 0x00008001
    R FLASHCTRL_FPAC1 0x0000000B 0x04E20001
    R FLASHCTRL_FMSTAT 0x0000000B 0x00000000
    R FLASHCTRL_FRD_INTF_CTRL_LOCK 0x0000000B 0xA5A5A5A5
    R FLASHCTRL_FRD_INTF_CTRL 0x0000000B 0x00000003
    R FLASHECC_ECC_ENABLE 0x0000000B 0x0000000A
    R FLASHECC_SINGLE_ERR_ADDR_LOW 0x0000000B 0x00000000
    R FLASHECC_SINGLE_ERR_ADDR_HIGH 0x0000000B 0x00000000
    R FLASHECC_UNC_ERR_ADDR_LOW 0x0000000B 0x00000000
    R FLASHECC_UNC_ERR_ADDR_HIGH 0x0000000B 0x00000000
    R FLASHECC_ERR_STATUS 0x0000000B 0x00000000
    R FLASHECC_ERR_POS 0x0000000B 0x00000000
    R FLASHECC_ERR_STATUS_CLR 0x0000000B 0x00000000
    R FLASHECC_ERR_CNT 0x0000000B 0x00000000
    R FLASHECC_ERR_THRESHOLD 0x0000000B 0x00000000
    R FLASHECC_ERR_INTFLG 0x0000000B 0x00000000
    R FLASHECC_ERR_INTCLR 0x0000000B 0x00000000
    R FLASHECC_FDATAH_TEST 0x0000000B 0x00000000
    R FLASHECC_FDATAL_TEST 0x0000000B 0x00000000
    R FLASHECC_FADDR_TEST 0x0000000B 0x00000000
    R FLASHECC_FECC_TEST 0x0000000B 0x00000000
    R FLASHECC_FECC_CTRL 0x0000000B 0x00000000
    R FLASHECC_FOUTH_TEST 0x0000000B 0x00000000
    R FLASHECC_FOUTL_TEST 0x0000000B 0x00000000
    R FLASHECC_FECC_STATUS 0x0000000B 0x00000000
    R FLASHECC_FLASH_ECC_REGS_LOCK 0x0000000B 0xA5A5A5A5
    R GCRC_CRCCTRL 0x0000000B 0x00000000
    R GCRC_CRCPOLY 0x0000000B 0x00000000
    R GCRC_CRCDATAMASK 0x0000000B 0x00000000
    R GCRC_CRCDATAIN 0x0000000B 0x00000000
    R GCRC_CRCDATAOUT 0x0000000B 0x00000000
    R GCRC_CRCDATATRANS 0x0000000B 0x00000000
    R GPIODATA_GPADAT 0x0000000B 0xFFBFDFA3
    R GPIODATA_GPASET 0x0000000B 0x00000000
    R GPIODATA_GPACLEAR 0x0000000B 0x00000000
    R GPIODATA_GPATOGGLE 0x0000000B 0x00000000
    R GPIODATA_GPBDAT 0x0000000B 0xEBFFEFFF
    R GPIODATA_GPBSET 0x0000000B 0x00000000
    R GPIODATA_GPBCLEAR 0x0000000B 0x00000000
    R GPIODATA_GPBTOGGLE 0x0000000B 0x00000000
    R GPIODATA_GPCDAT 0x0000000B 0xFFFFF7FF
    R GPIODATA_GPCSET 0x0000000B 0x00000000
    R GPIODATA_GPCCLEAR 0x0000000B 0x00000000
    R GPIODATA_GPCTOGGLE 0x0000000B 0x00000000
    R GPIODATA_GPDDAT 0x0000000B 0xE38C9DFF
    R GPIODATA_GPDSET 0x0000000B 0x00000000
    R GPIODATA_GPDCLEAR 0x0000000B 0x00000000
    R GPIODATA_GPDTOGGLE 0x0000000B 0x00000000
    R GPIODATA_GPEDAT 0x0000000B 0xFFFFFFFF
    R GPIODATA_GPESET 0x0000000B 0x00000000
    R GPIODATA_GPECLEAR 0x0000000B 0x00000000
    R GPIODATA_GPETOGGLE 0x0000000B 0x00000000
    R GPIODATA_GPFDAT 0x0000000B 0x000001FF
    R GPIODATA_GPFSET 0x0000000B 0x00000000
    R GPIODATA_GPFCLEAR 0x0000000B 0x00000000
    R GPIODATA_GPFTOGGLE 0x0000000B 0x00000000
    R GPIODATAREAD_GPADAT_R 0x0000000B 0x80000000
    R GPIODATAREAD_GPBDAT_R 0x0000000B 0x00000024
    R GPIODATAREAD_GPCDAT_R 0x0000000B 0x00000040
    R GPIODATAREAD_GPDDAT_R 0x0000000B 0x10801000
    R GPIODATAREAD_GPEDAT_R 0x0000000B 0x00000000
    R GPIODATAREAD_GPFDAT_R 0x0000000B 0x00000000
    R GPIODATAREAD_GPGDAT_R 0x0000000B 0x00000000
    R GPIODATAREAD_GPHDAT_R 0x0000000B 0x00000000
    R I2C0_I2CMSA 0x0000000B 0x00000000
    R I2C0_I2CMCS 0x0000000B 0x00000020
    R I2C0_I2CMDR 0x0000000B 0x00000000
    R I2C0_I2CMTPR 0x0000000B 0x00010001
    R I2C0_I2CMIMR 0x0000000B 0x00000000
    R I2C0_I2CMRIS 0x0000000B 0x00000000
    R I2C0_I2CMMIS 0x0000000B 0x00000000
    R I2C0_I2CMICR 0x0000000B 0x00000000
    R I2C0_I2CMCR 0x0000000B 0x00000000
    R I2C0_I2CMCLKOCNT 0x0000000B 0x00000000
    R I2C0_I2CMBMON 0x0000000B 0x00000003
    R I2C0_I2CMBLEN 0x0000000B 0x00000000
    R I2C0_I2CMBCNT 0x0000000B 0x00000000
    R I2C0_I2CSOAR 0x0000000B 0x00000000
    R I2C0_I2CSCSR 0x0000000B 0x00000000
    R I2C0_I2CSDR 0x0000000B 0x00000000
    R I2C0_I2CSIMR 0x0000000B 0x00000000
    R I2C0_I2CSRIS 0x0000000B 0x00000000
    R I2C0_I2CSMIS 0x0000000B 0x00000000
    R I2C0_I2CSICR 0x0000000B 0x00000000
    R I2C0_I2CSOAR2 0x0000000B 0x00000000
    R I2C0_I2CSACKCTL 0x0000000B 0x00000000
    R I2C0_I2CFIFODATARX 0x0000000B 0x0000004B
    R I2C0_I2CFIFOCTL 0x0000000B 0x00040004
    R I2C0_I2CFIFOSTATUS 0x0000000B 0x00010005
    R I2C0_I2CPP 0x0000000B 0x00000001
    R I2C0_I2CPC 0x0000000B 0x00000001
    R IPCCMTOCPU1_CMTOCPU1IPCACK 0x0000000B 0x00000000
    R IPCCMTOCPU1_CPU1TOCMIPCSTS 0x0000000B 0x00000001
    R IPCCMTOCPU1_CMTOCPU1IPCSET 0x0000000B 0x00000000
    R IPCCMTOCPU1_CMTOCPU1IPCCLR 0x0000000B 0x00000000
    R IPCCMTOCPU1_CMTOCPU1IPCFLG 0x0000000B 0x00000000
    R IPCCMTOCPU1_IPCCOUNTERL 0x0000000B 0x581F018C
    R IPCCMTOCPU1_IPCCOUNTERH 0x0000000B 0x00000019
    R IPCCMTOCPU1_CPU1TOCMIPCRECVCOM 0x0000000B 0x00001001
    R IPCCMTOCPU1_CPU1TOCMIPCRECVADDR 0x0000000B 0x00000000
    R IPCCMTOCPU1_CPU1TOCMIPCRECVDATA 0x0000000B 0x00000440
    R IPCCMTOCPU1_CMTOCPU1IPCREPLY 0x0000000B 0x00005555
    R IPCCMTOCPU1_CMTOCPU1IPCSENDCOM 0x0000000B 0x00001001
    R IPCCMTOCPU1_CMTOCPU1IPCSENDADDR 0x0000000B 0x00000000
    R IPCCMTOCPU1_CMTOCPU1IPCSENDDATA 0x0000000B 0x00000002
    R IPCCMTOCPU1_CPU1TOCMIPCREPLY 0x0000000B 0x00005555
    R IPCCMTOCPU1_CMTOCPU1IPCBOOTSTS 0x0000000B 0x80001002
    R IPCCMTOCPU1_CPU1TOCMIPCBOOTMODE 0x0000000B 0x5A007D03
    R IPCCMTOCPU1_PUMPREQUEST 0x0000000B 0x00000000
    R IPCCMTOCPU2_CMTOCPU2IPCACK 0x0000000B 0x00000000
    R IPCCMTOCPU2_CPU2TOCMIPCSTS 0x0000000B 0x00000000
    R IPCCMTOCPU2_CMTOCPU2IPCSET 0x0000000B 0x00000000
    R IPCCMTOCPU2_CMTOCPU2IPCCLR 0x0000000B 0x00000000
    R IPCCMTOCPU2_CMTOCPU2IPCFLG 0x0000000B 0x00000000
    R IPCCMTOCPU2_IPCCOUNTERL 0x0000000B 0x35E756A5
    R IPCCMTOCPU2_IPCCOUNTERH 0x0000000B 0x0000001B
    R IPCCMTOCPU2_CPU2TOCMIPCRECVCOM 0x0000000B 0x00000000
    R IPCCMTOCPU2_CPU2TOCMIPCRECVADDR 0x0000000B 0x00000000
    R IPCCMTOCPU2_CPU2TOCMIPCRECVDATA 0x0000000B 0x00000000
    R IPCCMTOCPU2_CMTOCPU2IPCREPLY 0x0000000B 0x00000000
    R IPCCMTOCPU2_CMTOCPU2IPCSENDCOM 0x0000000B 0x00000000
    R IPCCMTOCPU2_CMTOCPU2IPCSENDADDR 0x0000000B 0x00000000
    R IPCCMTOCPU2_CMTOCPU2IPCSENDDATA 0x0000000B 0x00000000
    R IPCCMTOCPU2_CPU2TOCMIPCREPLY 0x0000000B 0x00000000
    R MCANSS_MCANSS_PID 0x0000000B 0x00000000
    R MCANSS_MCANSS_CTRL 0x0000000B 0x00000000
    R MCANSS_MCANSS_STAT 0x0000000B 0x00000000
    R MCANSS_MCANSS_ICS 0x0000000B 0x00000000
    R MCANSS_MCANSS_IRS 0x0000000B 0x00000000
    R MCANSS_MCANSS_IECS 0x0000000B 0x00000000
    R MCANSS_MCANSS_IE 0x0000000B 0x00000000
    R MCANSS_MCANSS_IES 0x0000000B 0x00000000
    R MCANSS_MCANSS_EOI 0x0000000B 0x00000000
    R MCANSS_MCANSS_EXT_TS_PRESCALER 0x0000000B 0x00000000
    R MCANSS_MCANSS_EXT_TS_UNSERVICED_INTR_CNTR 0x0000000B 0x00000000
    R MCAN_MCAN_CREL 0x0000000B 0x00000000
    R MCAN_MCAN_ENDN 0x0000000B 0x00000000
    R MCAN_MCAN_DBTP 0x0000000B 0x00000000
    R MCAN_MCAN_TEST 0x0000000B 0x00000000
    R MCAN_MCAN_RWD 0x0000000B 0x00000000
    R MCAN_MCAN_CCCR 0x0000000B 0x00000000
    R MCAN_MCAN_NBTP 0x0000000B 0x00000000
    R MCAN_MCAN_TSCC 0x0000000B 0x00000000
    R MCAN_MCAN_TSCV 0x0000000B 0x00000000
    R MCAN_MCAN_TOCC 0x0000000B 0x00000000
    R MCAN_MCAN_TOCV 0x0000000B 0x00000000
    R MCAN_MCAN_ECR 0x0000000B 0x00000000
    R MCAN_MCAN_PSR 0x0000000B 0x00000000
    R MCAN_MCAN_TDCR 0x0000000B 0x00000000
    R MCAN_MCAN_IR 0x0000000B 0x00000000
    R MCAN_MCAN_IE 0x0000000B 0x00000000
    R MCAN_MCAN_ILS 0x0000000B 0x00000000
    R MCAN_MCAN_ILE 0x0000000B 0x00000000
    R MCAN_MCAN_GFC 0x0000000B 0x00000000
    R MCAN_MCAN_SIDFC 0x0000000B 0x00000000
    R MCAN_MCAN_XIDFC 0x0000000B 0x00000000
    R MCAN_MCAN_XIDAM 0x0000000B 0x00000000
    R MCAN_MCAN_HPMS 0x0000000B 0x00000000
    R MCAN_MCAN_NDAT1 0x0000000B 0x00000000
    R MCAN_MCAN_NDAT2 0x0000000B 0x00000000
    R MCAN_MCAN_RXF0C 0x0000000B 0x00000000
    R MCAN_MCAN_RXF0S 0x0000000B 0x00000000
    R MCAN_MCAN_RXF0A 0x0000000B 0x00000000
    R MCAN_MCAN_RXBC 0x0000000B 0x00000000
    R MCAN_MCAN_RXF1C 0x0000000B 0x00000000
    R MCAN_MCAN_RXF1S 0x0000000B 0x00000000
    R MCAN_MCAN_RXF1A 0x0000000B 0x00000000
    R MCAN_MCAN_RXESC 0x0000000B 0x00000000
    R MCAN_MCAN_TXBC 0x0000000B 0x00000000
    R MCAN_MCAN_TXFQS 0x0000000B 0x00000000
    R MCAN_MCAN_TXESC 0x0000000B 0x00000000
    R MCAN_MCAN_TXBRP 0x0000000B 0x00000000
    R MCAN_MCAN_TXBAR 0x0000000B 0x00000000
    R MCAN_MCAN_TXBCR 0x0000000B 0x00000000
    R MCAN_MCAN_TXBTO 0x0000000B 0x00000000
    R MCAN_MCAN_TXBCF 0x0000000B 0x00000000
    R MCAN_MCAN_TXBTIE 0x0000000B 0x00000000
    R MCAN_MCAN_TXBCIE 0x0000000B 0x00000000
    R MCAN_MCAN_TXEFC 0x0000000B 0x00000000
    R MCAN_MCAN_TXEFS 0x0000000B 0x00000000
    R MCAN_MCAN_TXEFA 0x0000000B 0x00000000
    R MCANERR_MCANERR_REV 0x0000000B 0x00000000
    R MCANERR_MCANERR_VECTOR 0x0000000B 0x00000000
    R MCANERR_MCANERR_STAT 0x0000000B 0x00000000
    R MCANERR_MCANERR_WRAP_REV 0x0000000B 0x00000000
    R MCANERR_MCANERR_CTRL 0x0000000B 0x00000000
    R MCANERR_MCANERR_ERR_CTRL1 0x0000000B 0x00000000
    R MCANERR_MCANERR_ERR_CTRL2 0x0000000B 0x00000000
    R MCANERR_MCANERR_ERR_STAT1 0x0000000B 0x00000000
    R MCANERR_MCANERR_ERR_STAT2 0x0000000B 0x00000000
    R MCANERR_MCANERR_ERR_STAT3 0x0000000B 0x00000000
    R MCANERR_MCANERR_SEC_EOI 0x0000000B 0x00000000
    R MCANERR_MCANERR_SEC_STATUS 0x0000000B 0x00000000
    R MCANERR_MCANERR_SEC_ENABLE_SET 0x0000000B 0x00000000
    R MCANERR_MCANERR_SEC_ENABLE_CLR 0x0000000B 0x00000000
    R MCANERR_MCANERR_DED_EOI 0x0000000B 0x00000000
    R MCANERR_MCANERR_DED_STATUS 0x0000000B 0x00000000
    R MCANERR_MCANERR_DED_ENABLE_SET 0x0000000B 0x00000000
    R MCANERR_MCANERR_DED_ENABLE_CLR 0x0000000B 0x00000000
    R MCANERR_MCANERR_AGGR_ENABLE_SET 0x0000000B 0x00000000
    R MCANERR_MCANERR_AGGR_ENABLE_CLR 0x0000000B 0x00000000
    R MCANERR_MCANERR_AGGR_STATUS_SET 0x0000000B 0x00000000
    R MCANERR_MCANERR_AGGR_STATUS_CLR 0x0000000B 0x00000000
    R MEMINITANDTEST_CxLOCK 0x0000000B 0x00000000
    R MEMINITANDTEST_CxTEST 0x0000000B 0x00000000
    R MEMINITANDTEST_CxINIT 0x0000000B 0x00000000
    R MEMINITANDTEST_CxINITDONE 0x0000000B 0x00000000
    R MEMINITANDTEST_CMMSGxLOCK 0x0000000B 0x00000000
    R MEMINITANDTEST_CMMSGxTEST 0x0000000B 0x00000000
    R MEMINITANDTEST_CMMSGxINIT 0x0000000B 0x00000000
    R MEMINITANDTEST_CMMSGxINITDONE 0x0000000B 0x00000000
    R MEMINITANDTEST_SxGROUP1_LOCK 0x0000000B 0x00000000
    R MEMINITANDTEST_SxGROUP1_TEST 0x0000000B 0x00000000
    R MEMINITANDTEST_SxGROUP1_INIT 0x0000000B 0x00000000
    R MEMINITANDTEST_SxGROUP1_INITDONE 0x0000000B 0x00000000
    R MEMINITANDTEST_ROM_LOCK 0x0000000B 0x00000000
    R MEMINITANDTEST_ROM_TEST 0x0000000B 0x00000000
    R MEMINITANDTEST_ROM_FORCE_ERROR 0x0000000B 0x00000000
    R MEMINITANDTEST_PERI_MEM_TEST_LOCK 0x0000000B 0x00000000
    R MEMINITANDTEST_PERI_MEM_TEST_CONTROL 0x0000000B 0x00000000
    R MPU_MPU_TYPE 0x0000000B 0x00000800
    R MPU_MPU_CTRL 0x0000000B 0x00000000
    R MPU_MPU_RNR 0x0000000B 0x00000000
    R MPU_MPU_RBAR 0x0000000B 0x00000000
    R MPU_MPU_RASR 0x0000000B 0x00000000
    R MPU_MPU_RBAR_A1 0x0000000B 0x00000000
    R MPU_MPU_RASR_A1 0x0000000B 0x00000000
    R MPU_MPU_RBAR_A2 0x0000000B 0x00000000
    R MPU_MPU_RASR_A2 0x0000000B 0x00000000
    R MPU_MPU_RBAR_A3 0x0000000B 0x00000000
    R MPU_MPU_RASR_A3 0x0000000B 0x00000000
    R NMI_CMNMICFG 0x0000000B 0x00000001
    R NMI_CMNMIFLG 0x0000000B 0x00000000
    R NMI_CMNMIFLGCLR 0x0000000B 0x00000000
    R NMI_CMNMIFLGFRC 0x0000000B 0x00000000
    R NMI_CMNMIWDCNT 0x0000000B 0x00000000
    R NMI_CMNMIWDPRD 0x0000000B 0x0000FFFF
    R NMI_CMNMISHDWFLG 0x0000000B 0x00000000
    R NVIC_NVIC_ISER0 0x0000000B 0x20012000
    R NVIC_NVIC_ISER1 0x0000000B 0x00000000
    R NVIC_NVIC_ICER0 0x0000000B 0x20012000
    R NVIC_NVIC_ICER1 0x0000000B 0x00000000
    R NVIC_NVIC_ISPR0 0x0000000B 0x20016000
    R NVIC_NVIC_ISPR1 0x0000000B 0x00000000
    R NVIC_NVIC_ISPR2 0x0000000B 0x00000000
    R NVIC_NVIC_ICPR0 0x0000000B 0x20016000
    R NVIC_NVIC_ICPR1 0x0000000B 0x00000000
    R NVIC_NVIC_IABR0 0x0000000B 0x00000000
    R NVIC_NVIC_IABR1 0x0000000B 0x00000000
    R NVIC_NVIC_IPR0 0x0000000B 0x00000000
    R NVIC_NVIC_IPR1 0x0000000B 0x00000000
    R NVIC_NVIC_IPR2 0x0000000B 0x00000000
    R NVIC_NVIC_IPR3 0x0000000B 0x00000000
    R NVIC_NVIC_IPR4 0x0000000B 0x00000000
    R NVIC_NVIC_IPR5 0x0000000B 0x00000000
    R NVIC_NVIC_IPR6 0x0000000B 0x00000000
    R NVIC_NVIC_IPR7 0x0000000B 0x00000000
    R NVIC_NVIC_IPR8 0x0000000B 0x00000000
    R NVIC_NVIC_IPR9 0x0000000B 0x00000000
    R NVIC_NVIC_IPR10 0x0000000B 0x00000000
    R NVIC_NVIC_IPR11 0x0000000B 0x00000000
    R NVIC_NVIC_IPR12 0x0000000B 0x00000000
    R NVIC_NVIC_IPR13 0x0000000B 0x00000000
    R NVIC_NVIC_IPR14 0x0000000B 0x00000000
    R NVIC_NVIC_IPR15 0x0000000B 0x00000000
    R NVIC_STIR 0x0000000B 0x00000000
    R SSI0_SSICR0 0x0000000B 0x00000000
    R SSI0_SSICR1 0x0000000B 0x00000000
    R SSI0_SSIDR 0x0000000B 0x00000000
    R SSI0_SSISR 0x0000000B 0x00000003
    R SSI0_SSICPSR 0x0000000B 0x00000000
    R SSI0_SSIIM 0x0000000B 0x00000000
    R SSI0_SSIRIS 0x0000000B 0x00000008
    R SSI0_SSIMIS 0x0000000B 0x00000000
    R SSI0_SSIICR 0x0000000B 0x00000000
    R SSI0_SSIDMACTL 0x0000000B 0x00000000
    R SSI0_SSIPV 0x0000000B 0x00000000
    R SSI0_SSIPP 0x0000000B 0x00000009
    R SSI0_SSIPC 0x0000000B 0x00000000
    R SSI0_SSIPeriphID4 0x0000000B 0x00000000
    R SSI0_SSIPeriphID5 0x0000000B 0x00000000
    R SSI0_SSIPeriphID6 0x0000000B 0x00000000
    R SSI0_SSIPeriphID7 0x0000000B 0x00000000
    R SSI0_SSIPeriphID0 0x0000000B 0x00000022
    R SSI0_SSIPeriphID1 0x0000000B 0x00000000
    R SSI0_SSIPeriphID2 0x0000000B 0x00000018
    R SSI0_SSIPeriphID3 0x0000000B 0x00000001
    R SSI0_SSIPCellID0 0x0000000B 0x0000000D
    R SSI0_SSIPCellID1 0x0000000B 0x000000F0
    R SSI0_SSIPCellID2 0x0000000B 0x00000005
    R SSI0_SSIPCellID3 0x0000000B 0x000000B1
    R CMSYSCTL_CMPCLKCR0 0x0000000B 0x00000000
    R CMSYSCTL_CMPCLKCR1 0x0000000B 0x00000001
    R CMSYSCTL_CMPCLKCR2 0x0000000B 0x00000000
    R CMSYSCTL_CMSOFTPRESET0 0x0000000B 0x00000000
    R CMSYSCTL_CMSOFTPRESET1 0x0000000B 0x00000004
    R CMSYSCTL_CMSOFTPRESET2 0x0000000B 0x00000000
    R CMSYSCTL_CMCLKSTOPREQ0 0x0000000B 0x00000000
    R CMSYSCTL_CMCLKSTOPREQ1 0x0000000B 0x00000000
    R CMSYSCTL_CMCLKSTOPREQ2 0x0000000B 0x00000000
    R CMSYSCTL_CMCLKSTOPACK0 0x0000000B 0x00000000
    R CMSYSCTL_CMCLKSTOPACK1 0x0000000B 0x00000000
    R CMSYSCTL_CMCLKSTOPACK2 0x0000000B 0x00000000
    R CMSYSCTL_MCANWAKESTATUS 0x0000000B 0x00000000
    R CMSYSCTL_MCANWAKESTATUSCLR 0x0000000B 0x00000000
    R CMSYSCTL_CMECATCTL 0x0000000B 0x00000000
    R CMSYSCTL_PALLOCATESTS 0x0000000B 0x00000010
    R CMSYSCTL_CMRESCCLR 0x0000000B 0x00000000
    R CMSYSCTL_CMRESC 0x0000000B 0x00010112
    R CMSYSCTL_CMSYSCTLLOCK 0x0000000B 0x00000000
    R SCB_ACTLR 0x0000000B 0x00000000
    R SCB_CPUID 0x0000000B 0x410FC241
    R SCB_ICSR 0x0000000B 0x0440F803
    R SCB_VTOR 0x0000000B 0x20013000
    R SCB_AIRCR 0x0000000B 0xFA050000
    R SCB_SCR 0x0000000B 0x00000000
    R SCB_CCR 0x0000000B 0x00000200
    R SCB_SHPR1 0x0000000B 0x00000000
    R SCB_SHPR2 0x0000000B 0x00000000
    R SCB_SHPR3 0x0000000B 0x00000000
    R SCB_SHCSRS 0x0000000B 0x00000000
    R SCB_CFSR 0x0000000B 0x00008200
    R SCB_HFSR 0x0000000B 0x40000000
    R SCB_MMFAR 0x0000000B 0x04F1ADB5
    R SCB_BFAR 0x0000000B 0x04F1ADB5
    R SCB_AFSR 0x0000000B 0x00000000
    R SYSTICK_SYST_CSR 0x0000000B 0x00010007
    R SYSTICK_SYST_RVR 0x0000000B 0x0001E847
    R SYSTICK_SYST_CVR 0x0000000B 0x00001071
    R SYSTICK_SYST_CALIB 0x0000000B 0x80000000
    R TIMER0_TIM 0x0000000B 0x00000000
    R TIMER0_PRD 0x0000000B 0x00000000
    R TIMER0_TCR 0x0000000B 0x00000000
    R TIMER0_TPR 0x0000000B 0x00000000
    R TIMER1_TIM 0x0000000B 0x00000000
    R TIMER1_PRD 0x0000000B 0x00000000
    R TIMER1_TCR 0x0000000B 0x00000000
    R TIMER1_TPR 0x0000000B 0x00000000
    R TIMER2_TIM 0x0000000B 0x00000000
    R TIMER2_PRD 0x0000000B 0x00000000
    R TIMER2_TCR 0x0000000B 0x00000000
    R TIMER2_TPR 0x0000000B 0x00000000
    R UART0_UARTECR 0x0000000B 0x00000000
    R UDMA_DMASTAT 0x0000000B 0x001F0000
    R UDMA_DMACFG 0x0000000B 0x00000000
    R UDMA_DMACTLBASE 0x0000000B 0x00000000
    R UDMA_DMAALTBASE 0x0000000B 0x00000200
    R UDMA_DMASWREQ 0x0000000B 0x00000000
    R UDMA_DMAUSEBURSTSET 0x0000000B 0x00000000
    R UDMA_DMAUSEBURSTCLR 0x0000000B 0x00000000
    R UDMA_DMAREQMASKSET 0x0000000B 0x00000000
    R UDMA_DMAREQMASKCLR 0x0000000B 0x00000000
    R UDMA_DMAENASET 0x0000000B 0x00000000
    R UDMA_DMAENACLR 0x0000000B 0x00000000
    R UDMA_DMAALTSET 0x0000000B 0x00000000
    R UDMA_DMAALTCLR 0x0000000B 0x00000000
    R UDMA_DMAPRIOSET 0x0000000B 0x00000000
    R UDMA_DMAPRIOCLR 0x0000000B 0x00000000
    R UDMA_DMAERRCLR 0x0000000B 0x00000000
    R UDMA_DMACHMAP0 0x0000000B 0x00000000
    R UDMA_DMACHMAP1 0x0000000B 0x00000000
    R UDMA_DMACHMAP2 0x0000000B 0x00000000
    R UDMA_DMACHMAP3 0x0000000B 0x00000000
    R UDMA_DMAPeriphID4 0x0000000B 0x00000004
    R UDMA_DMAPeriphID0 0x0000000B 0x00000030
    R UDMA_DMAPeriphID1 0x0000000B 0x000000B2
    R UDMA_DMAPeriphID2 0x0000000B 0x0000000B
    R UDMA_DMAPeriphID3 0x0000000B 0x00000000
    R UDMA_DMAPCellID0 0x0000000B 0x0000000D
    R UDMA_DMAPCellID1 0x0000000B 0x000000F0
    R UDMA_DMAPCellID2 0x0000000B 0x00000005
    R UDMA_DMAPCellID3 0x0000000B 0x000000B1
    R WDT_SCSR 0x0000000B 0x00000003
    R WDT_WDCNTR 0x0000000B 0x00000000
    R WDT_WDKEY 0x0000000B 0x00000040
    R WDT_WDCR 0x0000000B 0x00000040
    R WDT_WDWCR 0x0000000B 0x00000000
    

  • Have you tried increasing increasing the #define MEM_SIZE  in lwipopts.h. Can we put any limit to number of mqtt subscriptions?

    It is difficult for us to debug the cause without the whole system. We request you to debug it yourself and we will get back to you if we come into similar issue at our end.

    Regards,

    Yashwant

  • My mem_size is:

    #define MEM_SIZE                        (28 * 1024)  // default is 1600, was 16K

    I am finding that the program before failing, when it loses connection, spends a lot of time on the mem_malloc function of mem.c, which is called on f2838xif_transmit

    At the end, it seems a memory leak on the UDP and TCP /IP stack, I see the problem in both protocols.

    I think you need to review the driver about this. Sometimes get longer and sometimes shorter, but at the end it will fail, so I find it is not ready for a production environment.

  • Please, take a deep look at Tivaware LWIP port, in specific at file tiva-tm4c129.c in:

    C:\ti\tivaware_c_series_2_1_4_178\third_party\lwip-1.4.1\ports\tiva-tm4c129\netif

    I think it will save you a lot of work and effort.

    We did not have any problem with Tivaware lwip implementation during years, and I am seeing it with f28388D one.

    BR

  • We have tested today the system in a real-life environment and when any UDP or TCP protocol is enabled (via MQTT or just TCP alone) the memory degrades in time, making the device unusable in minutes (sometimes 5 min other times 20min, but it will fail).

    Please, check the driver and test it thoroughly. This device is not ready for production regarding Ethernet communications.

  • I have increased the MEM_SIZE up to 41 kB with no result. It takes a little longer but it dies at the end with both protocols: UDP and TCP.

  • Another issue i noticed with the lwip implementation.

    When a packet arrives, the application calls lwIPEthernetIntHandler() to handle the packet. If this function does not return fast enough, the data in this packet is overridden with newer packets that arrives, while you are handling the data. This happens because the DMA runs in background and it overrides the packet that is currently being handled with new data that it transfers from the PHY into the MCU.

    I noticed this issue while debugging and placing breakpoints where my application handles the incoming packet (UDP receive function). I solved it by halting the DMA interrupt while handling the packet (Ethernet_disableRxDMAReception). It might happen also during normal work.

    p.s. there is very low traffic on my net. i don't think its because i run out of packet buffer but im not sure. Anyway i think that new packets should be discarded rather then overrides packet that are currently being handled.

  • Ideally the Rx HW descriptor is owned by application while it is processing and the ownership is passed to DMA after processing in receiveISR only then the DMA can access the memory. But if the same memory region is shared across the packets then maybe there could be overwriting. We will look into this and will add Ethernet_disableRxDMAReception if necessary. Thanks for the suggestion.

  • Thank you Timor for the feedback. I do see the same behavior of lwIPEthernetIntHandler. Many times RX is not fast enough.

    I think this issue is overriding the pbufs, and at the end the MCU runs out of memory, because these overrided packets are never freed.

    Watch out about disabling DMA, because the process could be still running on the background and if not handled correctly, even launch a FaultISR interrupt.

    I hope Yashwant these leads could help you to find out the issue here. From my experience, tthe problems are happening with every protocol if you just give enough time to the program to run.

    I will wait for your feedback, but I would like to expect a long test from your side.

    Best Regards

  • Hello Timor. I have tried your proposal with no luck.

    Where are you placing your Ethernet_disableRxDMAReception calls? I am dong so in Ethernet_receivePacketCallbackCustom

    My code:

    Ethernet_Pkt_Desc* Ethernet_receivePacketCallbackCustom(
            Ethernet_Handle handleApplication,
            Ethernet_Pkt_Desc *pPacket)
    {
    
    	Ethernet_Pkt_Desc* temp_eth_pkt;
        //
        // Book-keeping to maintain number of callbacks received.
        //
    #ifdef ETHERNET_DEBUG
        Ethernet_numRxCallbackCustom++;
    #endif
    
    
    
        //////////////////////////////////////////////////////////////////////////
          //Pak test ERROR
          int i=0;
    
          for(i = 0U;i < Ethernet_device_struct.initConfig.numChannels;i++)
           {
          	Ethernet_disableRxDMAReception(
                     Ethernet_device_struct.baseAddresses.enet_base,
                     i);
           }
          ///////////////////////////////////////////////////////////////////////////
    
    
    
        //
        // This is a placeholder for Application specific handling
        // We are replenishing the buffer received with another buffer
        //
      //  return lwIPEthernetIntHandler(pPacket);
    
          temp_eth_pkt=lwIPEthernetIntHandler(pPacket);
    
    
        //////////////////////////////////////////////////////////////////////////
          //Pak test ERROR
    
    
          for(i = 0U;i < Ethernet_device_struct.initConfig.numChannels;i++)
           {
               Ethernet_enableRxDMAReception(
                     Ethernet_device_struct.baseAddresses.enet_base,
                     i);
           }
          ///////////////////////////////////////////////////////////////////////////
    
          return temp_eth_pkt;
    }

  • PAk, i have placed the calls at a higher level, in my UDP function. This is the function that is being called from udp_input. Its the receive function that you specify when you initialize the pcb using udp_new() and udp_recv(). But, for my understanding, placing the calls in Ethernet_receivePacketCallbackCustom() is the right place and i think it should work just as you did.

    I used a simpler calls:

    Ethernet_disableRxDMAReception(EMAC_BASE, 0);
    Ethernet_enableRxDMAReception(EMAC_BASE, 0);

    p.s. are you using any OS (RTOS)?

    I will try to move my calls to Ethernet_receivePacketCallbackCustom() and let you know.

  • Thank you Timor..

    No I am not using any OS.

    I have two setups, one using UDP as a server and another using UDP+ MQTT over TCP. In both of them the MCU is transmitting 1kB of data every 5ms through the UDP socket. Besides, a small MQTT message is sent if MQTT is enabled.

    At the end, it fails in both modes.

    Are you able to generate a scenario like that on your side to check that this is happening.

    I will wait for your test. Regards

  • Good Morning Yashwant.

    I would like to know if there is any advance on your side.

    Have you checked in medium time tests  (10 to 20 min) to debug this issue?

    I see the system crashes no matter the protocol used.

    Thank you

  • Hi,

    Yes we were able to recreate your issue at memsize of 500 bytes. Still need to debug further on why the mem_free for those pkt descriptors is not being called in time by process_transmit(). Will try to add a recovery if mem_alloc fails transmit() function.

    Will get back to you in a day or so.

    Regards,

    Yashwant

  • PAk,

    Can you paste the lwip_stats value after transmitting 5-10 MQTT packets (lwip_stats.mem.used)? We want to see if this memory used value keeps on increasing or it gets reduced after successful transmission. 

    In our case this value gets back to 0 after successful transmission of all packets be it tcp, icmp pkt.

    -Yashwant

  • I am trying to create an indepent project with both UDP and MQTT/TCP for you.

    Would you be able to provide me a example project (with other name different to enet_lwip) ready to import and modify on top of it?

    It could be useful if the files are local in src folder (not linked to C:\ti\c2000\C2000Ware_3_04_00_00\libraries\communications\Ethernet\third_party\lwip\examples\enet_lwip...., folder).

    Best regards

  • I have attached the latest enet_lwip project and linked files copied. You can use different workspace if name matches. As asked earlier, have you faced the memory leak issue with just the provided source code without any updates?

    If yes how did you recreate that issue? In my earlier reply 500 bytes of mem size is too low for the number of new packet descriptor being created in f2838xif_transmit() but at more than 5KB i see the (lwip_stats.mem.used) value going back to 0 after all packets are transmitted. So no memory leak is found. 

    Can you also reply to this.

    Can you paste the lwip_stats value after transmitting 5-10 MQTT packets (lwip_stats.mem.used)? We want to see if this memory used value keeps on increasing or it gets reduced after successful transmission. 

    enet_lwip_new.zip

    -Yashwant

  • Hello.

    With your code I am getting:

     undefined              first referenced
      symbol                    in file     
     ---------              ----------------
     Ethernet_initRxChannel ./enet_lwip.obj

    Because on the file ethernet.c  the function is defined as:

    static uint32_t Ethernet_initRxChannel(Ethernet_ChInfo *chInfo)

    and not defined in ethernet.h, hence we need to add:

    //#include "driverlib_cm/ethernet.h"
    #include "driverlib_cm/ethernet.c"

  • Attached are latest ethernet driver files.

    ethernet.cethernet.h

    -Yashwant

  • Hello Yashwant. I am seeing the same error. Is it possitble that Driverlib_cm.lib has to be updated?

    I am seeing that ethernet.h is not detected but it is inclided and linked!! Quite strange. You can see my CCS screen.

  • Yes. You would have to import the f2838x/driverlib_cm project and rebuild it which will generate new driverlib_cm.lib after updating the ethernet files.

    -Yashwant

  • With respect to my earlier discussion with PAk,

    I moved the DMA enable/disable to the Ethernet_receivePacketCallbackCustom() and it works fine

  • Following my last post and after watching the system for some time i see that, with the new position of the DMA disable, the stack is much more stable. Originally I was getting genericISR every now and again with RBU and after moving the DMA disable to Ethernet_receivePacketCallbackCustom there are much less genericISR interrupts (actually almost none).

    Regarding the genericISR interrupt (Ethernet_genericISRCustom), in a RBU (receive buffer unavailable) condition this function drains the RX queue and that causes the device to lose connection with clients and it takes a few seconds until it reconnects. Why draining all the RX queue is necessary at this point? Actually I wonder if draining is needed at all. Why not dropping only the new packet that arrived? If I understand correctly, the RBU condition occurs because the application is holding the descriptor buffer while handling incoming packet and it will release it soon. If so, than the DMA can wait a moment for the descriptor to be released or drop the current packet but there is no need to drop all the RX queue.

    I appreciate your thoughts.

  • Thank you Timor for that insightful explanation. I am seeing that same behavior in both Receiving and Trasmitting buffers.

    Hello Yashwant. This can answer your question about lwip_stats:

    I have created an independent LWIP example supporting UDP and MQTT for testing. I will send it you via IM

  • Hello again Yashwant, please conform you have received my code

  • Yes, Received the code will try to test it. 

    Can you paste the lwip_stats value after transmitting 5-10 MQTT packets (lwip_stats.mem.used)? We want to see if this memory used value keeps on increasing or it gets reduced after successful transmission. 

    Can you reply to the above query.

  • I am getting this error 

    undefined first referenced
    symbol in file
    --------- ----------------
    mqtt_cyclic_timer ./lwip_mqtt.obj

    while building.

    Can you share the definition of mqtt_cyclic_timer function.

  • Hello Yashwant.

    You need to link mqtt.c and mqtt.h. They are located in:

    C:\ti\c2000\C2000Ware_3_04_00_00\libraries\communications\Ethernet\third_party\lwip\lwip-2.1.2\src\include\lwip

  • I have tested the MQTT, UDP application you have provided. 

    When MQTT disabled and continuous UDP data reception every 2ms. The lwip_stats.mem value stays at 0. So there is no memory leak in the driver or the udp application. And also i did not see any crash.

    When MQTT enabled, the lwip_stats.mem value increases by some amount in the beginning and stays at the value during UDP and continuous ping. Even in this scenario i did not see any crash in the system.

    Even when the UDP socket is disconnected the lwip.tcp.xmit value keeps increasing. You may want to look into why this is happening.

    As the error is caused because of the updated application we request you to debug it yourself.

    Regards,

    Yashwant

  • Could you share a wireshark log?

    Are you using it from flash or JTAG?

    You need to connect UDP client with CCC, and you will start to receive many UDP packets, and also have to see a MQTT packet every 2ms.

    I have seen the  error also with only UDP, and Timor is constantly reporting issues with lwip. So please, don't try to avoid this issue.

    Timor said:

    "Another issue i noticed with the lwip implementation.

    When a packet arrives, the application calls lwIPEthernetIntHandler() to handle the packet. If this function does not return fast enough, the data in this packet is overridden with newer packets that arrives, while you are handling the data. This happens because the DMA runs in background and it overrides the packet that is currently being handled with new data that it transfers from the PHY into the MCU.

    I noticed this issue while debugging and placing breakpoints where my application handles the incoming packet (UDP receive function). I solved it by halting the DMA interrupt while handling the packet (Ethernet_disableRxDMAReception). It might happen also during normal work.

    p.s. there is very low traffic on my net. i don't think its because i run out of packet buffer but im not sure. Anyway i think that new packets should be discarded rather then overrides packet that are currently being handled."

  • It is clear that the error is hitted on the f2838xif_transmit function and Timor reported at lwIPEthernetIntHandler, so it is a driver problem.

    I have run again the test with the same results, so the problem is in  TI's driver implementation. MQTT application is part of lwip, and the application has not been updated. Please, try to correct the F2838xif driver, or state that F2838X family is not compatible with LWIP as announced, since for the moment is a false statement.

  • Yashwant, do you have any insights regarding the GenericISR? Is it necessary to drain the all RX queue in a case of RBU?

    And also, i noticed another issue with the lwip. We had a bug on our client side (PC) which caused it to send multiple UDP request almost simultaneously to our server (F2838X) with less than 100usec between requests. The first request was missed and didn't get any respond and i found that the second packet is overriding the first packet. The interesting thing in this case is that when i captured this event in the lwip stack i saw that the packet length (p-len) was of the first packet but the actual data was of the second packet. I don't know if this is the DMA overriding data in this case or is it down in the PHY?

    Anyway, after we found that this is because of zero delay between UDP packets we added a delay on the client side and that solved the problem. In real world multiple packets might arrive on the same time and that must be handled correctly. A wrong packet length might explain the memory leakage that PAk is seeing.

    p.s. i also worked with lwip on TM4C for long time without any problem. I think that all the problems here are with attaching the lwip to the specific hardware. i.e. the problems are related to the external PHY, the DMA, the buffers and the interrupts handling.

    Thank you for doing your best to get the lwip to work correctly.

  • I think that all the problems here are with attaching the lwip to the specific hardware. i.e. the problems are related to the external PHY, the DMA, the buffers and the interrupts handling.

    I don't have any TMS320F2838x hardware to investigate the problem. Does anyone know if the EMAC IP in the TMS320F2838x CM is specific to that device family, or if the same EMAC IP is used in a different device family?

    If the same EMAC IP exists in a different device family, that could allow other drivers to be compared against f2838xif.c

  • CM4 is an ARM M4 core, so probaly is similar to Tiva TM4C129X, but some of them come with PHY+MAC and others only with EMAC IP.

    Lwip code is present at Tivaware SW package in TI website.

    In fact, I have used Tivaware LWIP implementation with no issues for years, so I am surprised with all these issues ocurring for more than a year.

    PS: Thank you Timor for sharing your experience with Lwip with both Tiva and F2838X devices. I totally agree with you, this is related to the Lwip port.

    PS2: Thank you Yashwant for your work, I know this is not a simple issue, but as you will understand, the port is not finished yet. The problems are not only MQTT related, but also TCP and UDP.

  • The EMAC IP of F28388x is different from TM4C. You can refer to TRM Ethernet chapter for IP description. Any help from external community is much appreciated.

  • Is it necessary to drain the all RX queue in a case of RBU?

    Noticed that there is a C2000Ware 3.04.00.00 Libraries -> Communication -> Ethernet -> Examples -> tcpEchoF2838X example which uses the SYS/BIOS NDK. There is a ndk_f2838x_3_61_01_01/source/ti/ndk/drivers/f2838x/EMACF2838X.c driver, which has the EMAC_deletePackets() function with the comment:

    /*
     *  ======== EMAC_deletePackets ========
     *  Called to free any packets held in the RX queue that the NDK has not yet
     *  been able to consume.
     *  This happens when the target is halted and results in these packets never
     *  being freed due to:
     *
     *  1. The RBU error occuring and in result halting DMA operation.
     *  2. The NDK thread being continuously starved due to the generic ISR running
     *     over and over
     *
     *  Freeing these packets allows the next, new batch of incoming packets to be
     *  allocated and
     *  properly handled (dequeued and processed/free by the upper layers of the
     *  stack)
     */

    Haven't yet attempted to compare the NDK driver against the LWIP driver to see if differences which explain the failures when the LWIP driver is used.

  • Thank you Chester.

    This is a very clever finding and excellent point. FMO, totally related with the issues we are seeing with Lwip in both RX and TX.

  • Hello again Yashwant.

    I have run a new test disabling MQTT and TCP, and enabling only UDP and the error happens in the same place. Here is the capture of the sequence with Lwip_stats.

    As you can see, this time instead of tcp_output, the mem overflow happens with a udp_sento call.

    All of the time, in lwip_stats is at 0 value (or just 1 packet), however, suddenly, the error is triggered and lwip_stats is full. I think the issue here is the driver stops working and memory get full quickly.

    This proves that the error comes from the driver and not from the application, since it is protocol independent.

    Best Regards