EK-TM4C129EXL: Updating firmware using I2C with the ROM Bootloader - Hanging after successful COMMAND_GET_STATUS

Hi Terence,

Terence Darwen said:

Then I start with my COMMAND_SEND_DATA commands:

1. Send a COMMAND_SEND_DATA command appropriately packetized.
2. Check for the ACK to the COMMAND_SEND_DATA command 
3. Send the COMMAND_GET_STATUS command appropriately packetized.
4. Check for the ACK to the COMMAND_GET_STATUS command 
5. Make sure the return code from COMMAND_GET_STATUS is COMMAND_RET_SUCCESS.

When doing the first COMMAND_SEND_DATA command it hangs indefinitely at step 2.  The bootloader never sends back an ACK no matter how long I wait.  Here's my logic analyzer output:

In general, if the slave bootloader does not send an ACK, then it may mean the start address to be programmed is not a valid 32-word address or the size of the image may have exceeded the available flash size. It is also possible that the checksum does not match and hence NAK is replied. Let's analyze your waveform.

Terence Darwen said:

As far as I can see in the above waveform, you are sending:

- byte length: 8

- Checksum: 0x8C, Can you confirm if 0x8C is correct?

- Command: 0x24 (SEND DATA)

- Data:1 0x20000200

- Data2: 0x0000073F

After SEND_DATA is transmitted, you then try to read the response. You begin with "R 0x42". However, I do not see any more data being transmitted. Comparing to the working waveform, I see "R 0x42" -> "0x00" -> "0xCC", the non working waveform does not follow with the 0x00. I'm a bit confused with myself with this 0x00 the second byte. I believe this is supposed to come from the slave because master issues a I2C read command. I don't know why the slave is not responding. Can you issue a PING command to see if the slave is in bootloader mode or not?

[EDIT] I believe the master must still generate the SCL clocks in order for the slave to return the data in I2C read mode. You might want to check your master side why it is not generating the clocks after "R 0x42".

Can you refer to this serial bootloader app note. Not sure if you have reference this app note when you develop your serial programmer on the master I2C side.   

https://www.ti.com/lit/an/spma074a/spma074a.pdf

Terence Darwen said:

Now, here's the funny thing: If I omit the COMMAND_GET_STATUS when doing both the COMMAND_DOWNLOAD and COMMAND_SEND_DATA commands, everything works.  The entire 0x9A8 bytes of the new firmware will successfully get sent from the EK-TM4C129EXL to the EK-TM4C1294XL and written to flash on the EK-TM4C1294XL.  When rebooting, it successfully runs the new firmware.

So, I'm confused on two points and hoping someone has some suggestions:

• Why is the I2C communication hanging after that first COMMAND_SEND_DATA command.
• Why does everything work when removing the COMMAND_GET_STATUS commands.

  That is a interesting observation that I don't have an answer yet. Here I'm a bit confused here. Earlier you said the I2C slave hangs with the ACK response check. Here you seem to suggest the hang is due to GET_STATUS. There are two types of response: Type-1 (ACK or NAK response) and Type-2 (GET_STATUS). Which one is hanging?

Hi Charles, thanks for the detailed response.  Addressing your questions below:

Charles Tsai said:

Checksum: 0x8C, Can you confirm if 0x8C is correct?

I believe the checksum is correct.  The checksum is calculated using the following data: 0x24 + 0x00 + 0x02 + 0x20 + 0x3F + 0x07 + 0x00 + 0x00.  This comes to 0x8C.

Charles Tsai said:

Comparing to the working waveform, I see "R 0x42" -> "0x00" -> "0xCC", the non working waveform does not follow with the 0x00. I'm a bit confused with myself with this 0x00 the second byte. I believe this is supposed to come from the slave because master issues a I2C read command.

Yes, I agree the second (as well as third) byte is supposed to come from the slave.  This is where things fail and I get no more response from the slave.  To explain a bit more: My understanding of how the default bootloader works is when the master sends a valid command the bootloader will respond with an ACK.  This ACK consists of two bytes (0x00 followed by 0xCC).  If there is some issue the bootloader will respond with a NACK (0x33 instead of 0xCC).  Of course to obtain the ACK or NACK the master must do a read.  This is what is happening when you see the master send "R 0x42" - it's initiating the read of the ACK/NACK.  And you're correct, the first sign of a problem is when the master sends the "R 0x42" and gets nothing back...  And this is when everything fails and I no longer get any responses from the slave (i.e. the target).

Charles Tsai said:

Can you issue a PING command to see if the slave is in bootloader mode or not?

Yes, I've added code so the very first thing the master does is issue a ping to make sure it can talk to the target's bootloader.  The following images from my logic analyzer shows the ping command, the ACK, followed by a "get status" command, the ACK for it and then the successful status - all looks good.  I had to break up the images a bit because it would be too long to all show vertically:

I can actually continue on with this to show the issue I'm experiencing.  So, after the ping, I send the "download" command.  Here it is:

Then I go to read the ACK for the "download" command and, just like in my example from my original post, I get no response from the bootloader.  The SCL line is held low indefinitely:

Looking at this closer, I originally thought the slave (the bootloader) was holding the clock line low but I'm not sure that's the case.  I wonder if all the master needs to do is send more clock signals to get a response from the bootloader.  I will investigate this.

Charles Tsai said:

Can you refer to this serial bootloader app note. Not sure if you have reference this app note when you develop your serial programmer on the master I2C side.   

https://www.ti.com/lit/an/spma074a/spma074a.pdf

Thank you, I was not aware of this app note.  I was using SPMU301E (TivaWare Bootloader User's Guide) and the TivaWare bootloader code to create my code.  I will review SPMA074A as well.

Charles Tsai said:

Here I'm a bit confused here. Earlier you said the I2C slave hangs with the ACK response check. Here you seem to suggest the hang is due to GET_STATUS. There are two types of response: Type-1 (ACK or NAK response) and Type-2 (GET_STATUS). Which one is hanging?

Apologies for the confusion.  To be clear, when I remove my GET_STATUS calls and leave in my ACK reads, everything works.  The ping command works, the download command works, the many send data commands work as well as the reset command after finished with my send data commands and my target board is successfully updated with new firmware.

Now, as shown above with my ping example, as soon as I add a GET_STATUS command the *ACK* on the *following* command will hang. 

Does that description make sense?  ...I know, it's weird and I can't currently make sense of it.

Just a follow up on this:

Terence Darwen said:

Looking at this closer, I originally thought the slave (the bootloader) was holding the clock line low but I'm not sure that's the case.  I wonder if all the master needs to do is send more clock signals to get a response from the bootloader.  I will investigate this.

It's hanging due to the master bus being busy after starting the read as shown below:

The bus remains busy indefinitely:

This is the same function I use to do all other I2C reads and have not experienced any problems....  I'm not sure why the bus busy bit remains set indefinitely...  

Also, reviewing SPMA074A, it appears to state that the ACK/NACK should only be a single byte returned by the bootloader and not the two bytes I'm normally receiving as I showed in my ping example in the previous post.

Any idea why I'm receiving two bytes instead of one?  I originally thought two bytes for an ACK was to be expected since before writing my own updater firmware I used a Total Phase Aardvark to send a ping command and then read back data for the ACK and it received two bytes (0x00 then 0xCC) as well.

Also, if it would be helpful at all, I could attach the code I'm using on both eval boards.  These are basic CCS projects.  The code is well commented and should be fairly easy to read.

Apologies for excessive posting, but reviewing the bootloader code I see that the ACK is indeed a 0x00 followed with 0xCC.  From bl_packet.c in TivaWare 2.2.0.295 boot loader:

//*****************************************************************************
//
// The packet that is sent to acknowledge a received packet.
//
//*****************************************************************************
static const uint8_t g_pui8ACK[2] = { 0, COMMAND_ACK };

...then further down:

//*****************************************************************************
//
//! Sends an Acknowledge packet.
//!
//! This function is called to acknowledge that a packet has been received by
//! the microcontroller.
//!
//! \return None.
//
//*****************************************************************************
void
AckPacket(void)
{
    //
    // ACK/NAK packets are the only ones with no size.
    //
    SendData(g_pui8ACK, 2);
}

I'm still very much open to the idea that something is wrong in my I2CReadData function.  Here it is:

#define I2C_DELAY                 2000     // Wait this many ticks between I2C actions

void I2CReadData(uint8_t* outputBuffer, const uint32_t bytesToRead)
{
    // Make sure the bus isn't busy before trying to start the transaction
    MAP_SysCtlDelay(I2C_DELAY);
    while(MAP_I2CMasterBusy(I2C0_BASE));

    // Set the slave address for receiving
    MAP_I2CMasterSlaveAddrSet(I2C0_BASE, SLAVE_ADDRESS, true);
    MAP_SysCtlDelay(I2C_DELAY);
    while(MAP_I2CMasterBusy(I2C0_BASE));

    // Start receiving data in burst mode
    MAP_I2CMasterControl(I2C0_BASE, I2C_MASTER_CMD_BURST_RECEIVE_START);
    MAP_SysCtlDelay(I2C_DELAY);
    while(I2CMasterBusy(I2C0_BASE));

    // Get the first byte
    outputBuffer[0] = MAP_I2CMasterDataGet(I2C0_BASE);
    MAP_SysCtlDelay(I2C_DELAY);
    while(MAP_I2CMasterBusy(I2C0_BASE));

    // Read the remaining data
    uint32_t i = 1;
    for(; i < bytesToRead; ++i)
    {
        // If we're not yet at the last byte, send a continue, otherwise, send a finish
        if(i < (bytesToRead - 1))
        {
            MAP_I2CMasterControl(I2C0_BASE, I2C_MASTER_CMD_BURST_RECEIVE_CONT);
        }
        else
        {
            MAP_I2CMasterControl(I2C0_BASE, I2C_MASTER_CMD_BURST_RECEIVE_FINISH);
        }

        MAP_SysCtlDelay(I2C_DELAY);
        while(MAP_I2CMasterBusy(I2C0_BASE));

        // Get the current byte of data
        outputBuffer[i] = (uint8_t)MAP_I2CMasterDataGet(I2C0_BASE);
        MAP_SysCtlDelay(I2C_DELAY);
        while(MAP_I2CMasterBusy(I2C0_BASE));
    }
}

I'll rework this.  Possibly doing two single reads instead of burst mode will work better.

Terence Darwen said:

To explain a bit more: My understanding of how the default bootloader works is when the master sends a valid command the bootloader will respond with an ACK.  This ACK consists of two bytes (0x00 followed by 0xCC).  If there is some issue the bootloader will respond with a NACK (0x33 instead of 0xCC). 

That is correct. 

Terence Darwen said:

Of course to obtain the ACK or NACK the master must do a read.  This is what is happening when you see the master send "R 0x42" - it's initiating the read of the ACK/NACK.  And you're correct, the first sign of a problem is when the master sends the "R 0x42" and gets nothing back...  And this is when everything fails and I no longer get any responses from the slave (i.e. the target).

Terence Darwen said:

I revised my original reply while you sent your response. In my revised reply, I mentioned that the master must still generate clocks for the slave to return the data. However, looking at your above waveform closely, it seems to me that the SCL is low. This can mean two things but I don't know which is which. The first possibility is that the master is not putting out the clocks or in the second case the slave is stretching the clock as means to hold the master in wait state per I2C protocol. 

Terence Darwen said:

but reviewing the bootloader code I see that the ACK is indeed a 0x00 followed with 0xCC.  From bl_packet.c in TivaWare 2.2.0.295 boot loader:

Fullscreen

1

2

3

4

5

6

//*****************************************************************************

//

// The packet that is sent to acknowledge a received packet.

//

//*****************************************************************************

static const uint8_t g_pui8ACK[2] = { 0, COMMAND_ACK };

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

//*****************************************************************************
//
// The packet that is sent to acknowledge a received packet.
//
//*****************************************************************************
static const uint8_t g_pui8ACK[2] = { 0, COMMAND_ACK };

...then further down:

Fullscreen

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

//*****************************************************************************

//

//! Sends an Acknowledge packet.

//!

//! This function is called to acknowledge that a packet has been received by

//! the microcontroller.

//!

//! \return None.

//

//*****************************************************************************

void

AckPacket(void)

{

//

// ACK/NAK packets are the only ones with no size.

//

SendData(g_pui8ACK, 2);

}

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

//*****************************************************************************
//
//! Sends an Acknowledge packet.
//!
//! This function is called to acknowledge that a packet has been received by
//! the microcontroller.
//!
//! \return None.
//
//*****************************************************************************
void
AckPacket(void)
{
    //
    // ACK/NAK packets are the only ones with no size.
    //
    SendData(g_pui8ACK, 2);
}

I

Yes, the response is two bytes with 0x00, 0xCC for ACK. 

Terence Darwen said:

I'll rework this.  Possibly doing two single reads instead of burst mode will work better.

Let me know if that makes a difference. But also check if the slave is stretching the clock. 

Okay, thanks, Charles.  I replaced the burst read with two I2C_MASTER_CMD_SINGLE_RECEIVE and am still getting the same results.  I also extended the I2C_DELAY to 6000 ticks to give the receiver more time, but also get the same results.

Here's my new I2CRead function:

#define I2C_DELAY                 6000     // Wait this many ticks between I2C actions

void I2CReadDataACK(uint8_t* outputBuffer)
{
    // Make sure the bus isn't busy before trying to start the transaction
    MAP_SysCtlDelay(I2C_DELAY);
    while(MAP_I2CMasterBusy(I2C0_BASE));

    // Set the slave address for receiving
    MAP_I2CMasterSlaveAddrSet(I2C0_BASE, SLAVE_ADDRESS, true);
    MAP_SysCtlDelay(I2C_DELAY);
    while(MAP_I2CMasterBusy(I2C0_BASE));

    // Request to receive the single byte
    MAP_I2CMasterControl(I2C0_BASE, I2C_MASTER_CMD_SINGLE_RECEIVE);
    MAP_SysCtlDelay(I2C_DELAY);
    while(MAP_I2CMasterBusy(I2C0_BASE));

    // Read the first byte of data
    outputBuffer[0] = I2CMasterDataGet(I2C0_BASE);
    MAP_SysCtlDelay(I2C_DELAY);
    while(MAP_I2CMasterBusy(I2C0_BASE));

    // Request to receive the single byte
    MAP_I2CMasterControl(I2C0_BASE, I2C_MASTER_CMD_SINGLE_RECEIVE);
    MAP_SysCtlDelay(I2C_DELAY);
    while(MAP_I2CMasterBusy(I2C0_BASE));

    // Read the second byte of data
    outputBuffer[1] = I2CMasterDataGet(I2C0_BASE);
    MAP_SysCtlDelay(I2C_DELAY);
    while(MAP_I2CMasterBusy(I2C0_BASE));
}

Similar to my old I2CRead function, it gets stuck indefinitely here:

The bus is busy indefinitely with SCL held low after the read attempt:

Zooming out showing it hanging indefinitely:

With clock stretching, it should be up to the I2C slave to release the SCL at some point, right?

Also, I believe it is the target device (the I2C slave) holding SCL low since if I reset only that board without touching the other board (the master) the SCL line goes back high.

I have a hard time believing this is a bug in the ROM bootloader since I would imagine it is in very wide use but I'm not sure what else is could be.

Any thoughts?

Terence Darwen said:

Also, I believe it is the target device (the I2C slave) holding SCL low since if I reset only that board without touching the other board (the master) the SCL line goes back high.

Hi Terence,

  Can you double check on the master side by setting up a Clock Low Timeout? If you see the timeout flag or interrupt then it confirms that the slave was indeed holding the bus. As far as why the slave side ROM bootloader was holding the bus, I really don't know. As a matter of fact, I cannot recall ROM-based I2C bootloader questions on e2e. You might be the first one or one of the very few. Most people use UART, Ethernet and USB with some people use SPI and CAN. Since you are using the ROM bootloader, the visibility and debug into what it is doing is somewhat limited unlike the flash-based bootloader. When a slave stretches the clock, it means it is not ready to take the data. What if you send just 4 bytes or less at a time in your SENT_DATA command instead of 8 bytes? Will that make a difference? I will start with sending just one byte at at time to see how the bootloader responds and gradually increase the byte count. 

18.3.1.6 Clock Low Timeout (CLTO)
The I2C slave can extend the transaction by pulling the clock low periodically to create a slow bit
transfer rate. The I2C module has a 12-bit programmable counter that is used to track how long the
clock has been held low. The upper 8 bits of the count value are software programmable through
the I2C Master Clock Low Timeout Count (I2CMCLKOCNT) register. The lower four bits are not
user visible and are 0x0. The CNTL value programmed in the I2CMCLKOCNT register has to be
greater than 0x01. The application can program the eight most significant bits of the counter to
reflect the acceptable cumulative low period in transaction. The count is loaded at the START
condition and counts down on each falling edge of the internal bus clock of the Master. Note that
the internal bus clock generated for this counter keeps running at the programmed I2C speed even
if SCL is held low on the bus. Upon reaching terminal count, the master state machine forces ABORT
on the bus by issuing a STOP condition at the instance of SCL and SDA release.
As an example, if an I2C module was operating at 100 kHz speed, programming the I2CMCLKOCNT
register to 0xDA would translate to the value 0xDA0 since the lower four bits are set to 0x0. This
would translate to a decimal value of 3488 clocks or a cumulative clock low period of 34.88 ms at
100 kHz.

Terence Darwen said:

Rerunning the exact same test as before shows the Clock Timeout Error now when it did not exist before:

Seems to be additional indication the ROM bootloader is causing the issue.

Hi Terence,

  Thanks for the confirmation. The slave indeed is holding the bus for whatever reason that I don't have an explanation. 

Terence Darwen said:

Good idea!  BUT then that made me think of trying something else: Just send pings (only a single data byte) endlessly once every 50 milliseconds and see if it can handle that.  Unfortunately, same issue:

And then the ACK read on the second ping fails with SCL stuck low as usual:

Ok, I see. Perhaps there is some unknown issue with the ROM-based I2C bootloader. Since the bootloader is stored in the ROM, there is no way of patching it without a silicon respin which is not possible. 

Terence Darwen said:

Despite this, I don't think it's a showstopper.  Like I mentioned before, if I don't send the "get status" commands the firmware update is successful every time.  I've tested this many times without fail.  Yes, it would be nice to have the status *but* I do at least have the ACK's for each command I issue.

Glad that you have tested the workaround (by not sending the GET_STATUS command) without failing. I will definitely take note of this and bookmark this post so anyone in the community can benefit from your workaround if I'm asked. Thanks so much!