uNPI handshake with an external MCU

Hi Gregory,

As you have observed, this is not the expected behavior. On startup, it is expected that the communication flow go like this (assuming SNP up first)

1. SNP chirps
2. SAP chirps in rsp
3. SNP sends RST (0xFA)
4. SAP sends RST in rsp *Reset handshake is complete here*
5. SNP chirps
6. SAP chirps
7. SNP sends Power up indication


To help us get started with the debug process, can you please post a screenshot or picture of your logic trace?
I would also be interested to see the steps you took in porting the SAPlib with software handshaking to the SAP.

The trace I've seen was using an oscilloscope & UART RX/TX on the SAP unfortunately as I don't have a logic analyzer yet.
I'm probably dealing with timing issues as increasing the chirp timeout at the SNP allows me to proceed with the protocol. Currently the SAP has

#define NPI_SWHS_CHIRP_TIMEOUT_PERIOD       50000

and that fixes the first handshake. The SAP receives the power up indication, but doesn't continue.
This is the trace from the SAP (which wakes up first) and it's stuck at the last receive:

[  124457]o
[  124471]c
[  124476]R w: l: 1;
[  124476]T w: 55; l: 1;
[  124477]T r: l: 1;
[  124554]R r: fa; l: 1;
[  124561]R w: l: 1;
[  124561]T w: fa; l: 1;
[  124563]T r: l: 1;
[  124642]R r: 55; l: 1;
[  124649]T w: 55; l: 1;
[  124649]R w: l: 1;
[  124651]T r: l: 1;
[  124728]R r: fe; l: 1;
[  124737]R w: l: 4;
[  124740]R r: 0; l: 4;
[  124748]R w: l: 1;
[  124751]R r: 54; l: 1;
[  124759]x
[  124778]o
[  124793]c
[  124798]R w: l: 1;
[  124798]T w: 55; l: 1;
[  124800]T r: l: 1;
[  128418]R r: 55; l: 1;
[  128433]R w: l: 1;
[  128433]T w: fe; l: 9;
[  128434]T r: l: 9;
[  128535]x
[  128648]o
[  128664]c
[  128668]T w: 55; l: 1;
[  128668]R w: l: 1;
[  128670]T r: l: 1;
[  128747]R r: fa; l: 1;
[  128754]R w: l: 1;
[  128754]T w: fa; l: 1;
[  128755]T r: l: 1;
[  128832]R r: 55; l: 1;
[  128840]T w: 55; l: 1;
[  128840]R w: l: 1;
[  128842]T r: l: 1;
[  128918]R r: fe; l: 1;
[  128926]R w: l: 4;
[  128929]R r: 0; l: 4;
[  128938]R w: l: 1;
[  128941]R r: 54; l: 1;
[  128948]x
[  128950]o
[  128964]c
[  128970]T w: 55; l: 1;
[  128970]R w: l: 1;
[  128972]T r: l: 1;
[  132680]R r: 55; l: 1;

where

• the timestamp is in 10s microseconds (tick every 10 microseconds)
• R w is RX wait
• R r is RX read
• T w is TX
• T r is TX complete
• where applicable, adding the first byte of the RX/TX packet
• o is UART channel openning
• c is clearing
• x is UART channel closure

I guess that the uNPI doesn't deal well with slow responses from other processors.

Regarding the porting, I'm using FreeRTOS on STM32F401.
I've only changed the RTOS API calls to be compliant with FreeRTOS (semaphore, tasks creations) with the same initialization values (i.e. semaphore count 0/1) and modified the driver calls to be compliant with STM32F4 HAL drivers.

Hi Gregory,

This will be pretty hard to debug without a logic analyzer, but I will help out as best I can.

Here is my hypothesis:

The SAP receives the first chirp from the SNP, and starts to react, before it is able to chirp, the SNP hits its chirp timeout (assuming the SAP missed the first one) and chirps again. Now the SAP has rx'd two chirps and is in a bad state. It completes the reset handshake (0xFA) successfully but then the regular handshake (0x55) but then seems to close its TL. This is what worries me.

Can you define NPI_SW_HANDSHAKING_DEBUG, which will toggle a GPIO in the NPITL_transmissionCallBack (this is where the UART is closed).

This is indeed the behavior of the transactions - the chirp timeouts and then the SNP sends another chirp, causing the protocol to hang.
The transactions were slow because of the tracing itself. I was using SWV and that caused delays.
Other than that. there were several other porting issues.

I've managed to proceed with the implementation, and there's now successful bring up of the SNP by the SAP, even some reading from characteristics.
The protocol itself is not prone to signal errors and it can stop at some points.
Other than the early chirp timeout, I'm receiving unwanted interrupts on the RX channel of the SAP (as if the SNP sent a handshake chirp), but the SNP didn't send any chirp, so it's not replying with a RST, but with a chirp. That's causing the protocol to go into an unknown state and can't leave it.

I'll keep checking what's causing it.

Best regards,

Gregory

Follow up:

I got a logic analyzer and now I'm seeing this behavior on the bus:

Above seen is the reset handshake, power up indication and then the HCI reset sent from the SAP to the SNP.
After the first NPI packet , the SNP is responding to a CHIRP after a long delay of about 30ms. Is this normal? I didn't change the SNP code.

Hi Gregory,

While it is un-tested as to how the protocol will behave while running SWV, it shouldn't be closing the TL prematurely like that. Since you now have a logic analyzer, can you try enabling NPI_SW_HANDSHAKING_DEBUG and attaching to the trace pin to verify when the TL_CB is being triggered?

Regarding your update, it is expected to see a delay when an HCI reset is issued since this trickles down to a call that sets a bit in a HW register and waits for the proc to reset. This is the only call that should have a delay this large.

In summary:
*startup*
1. PU-IND from SNP
2. HCI reset from SAP
3. HCI reset confirm from SNP
* SNP resets, you will see its UART TX line drop low for this *
**large expected delay here while SNP resets**

The traces I used using the SWV were on the SAP on the STM32 MCU. As I've disabled it, it's not an issue and the MCUs are responsive.

The capture from the logic analyzer is showing something else. The power up in was sent and then it's time for the SNP reset, like you've mentioned. What you see is the chirp from the SAP, a 30 ms delay, chirp from the SNP, and the reset command. I see the delay caused by the CC2650 reset later and that is about 100ms delay.
The 30ms delay is puzzling.

Hi Gregory,

"The traces I used using the SWV were on the SAP on the STM32 MCU. As I've disabled it, it's not an issue and the MCUs are responsive. " I agree here.

Can you attach the entire logic file? I can't really make out the individual bytes being sent from the screenshot above. From your description, the 30ms delay sounds like it is unexpected.

I've attached the trace.

Best regards,

Gregory

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/538/1-MHz_2C00_-2-M-Samples-_5B00_8_5D00_.7z

This delay is indeed coming from the CC2650. Changing optimization of the SNP project modifies the delay. When the optimization is low, the delay is about 30ms, as shown in the earlier logic trace. When I change the optimization to high (size/speed) then it comes down to ~22-24ms. I've attached a trace of a run with high optimization.

Is there any logic trace that you can provide me with to use as a reference?

Thanks.

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/538/3-MHz_2C00_-6-M-Samples-_5B00_45_5D00_.7z

Hi Gregory,

If possible it is best to leave optimizations on for the SNP.

I have attached a logic trace for your reference.

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/538/two_5F00_wire_5F00_unpi.logicdata

I was playing with the optimizations just to understand the protocol behavior. I'll leave the optimization on high from now on.

It seems that this behavior is the same in your trace - that it takes ~24ms for the cc2650 to respond to a chirp.
This is affecting severely on the NPI throughput and BLE throughput.

Any idea on where is this coming from and how to lower this delay?

Thanks.

So the delay is coming from the NPITLUART_openTransport function's loop:

void NPITLUART_openTransport(uint8_t portID, UART_Params *portParams,
                             hsTransactionRole role)
{
  hsRxByte[0] = NPI_UART_HS_JUNK;
  hsTxByte[0] = NPI_UART_HS_JUNK;
  // Open / power on the UART.
  uartHandle = UART_open(portID, portParams);
  //Remove any mystery bytes from the FIFO...i.e. 0xF8
  volatile uint16_t count = 0;
  while( count < 32000)
  {
    UARTCharGetNonBlocking(((UARTCC26XX_HWAttrs const *)(uartHandle->hwAttrs))->baseAddr);
    count++;
  }
  NPITLUART_doHandshake(uartHandle,role);
}

lowering the count limit (32000) reduces the delay.

However, I'm not sure of the behavior of lowering the count as I'm not that familiar with TI-RTOS. Is there a way to replace the loop block with something else - something more responsive?

Thanks.

Hi Gregory,

I apologize for the long response time. I was out of the office. 

The short answer is that since you are not implementing this library on a 26xx device, (your NPI Master is the STM4) it is safe to remove this loop entirely.

If you are interested, the reason why the loop is required is because occasionally when opening/closing the UART frequently on the CC26xx a situation can occur where a read callback is triggered before any data is sent on the line. Since this interferes with the flow of the uNPI handshaking protocol, I have placed the loop there to ensure the FIFO is empty. 

I will add better comments/documentation to this issue. However, again, since you are implementing on a different platform it is completely safe to remove the while loop entirely.

Thanks for the explanation.

The loop itself is still on the SNP, the cc2650, so that's why I'm asking.

How low can I lower that loop count to make the system still work? The loop is affecting the throughput.

Thanks.

Gregory,

Right, sorry. I was thinking from the master side.

There are a couple options we can try. Let me know which sounds optimal to you and I will of course support the changes:

1. The latest TI-RTOS version supports a UART control command to flush the RX FIFO. This is ideally what we need to do here. This control command was not available in v2.13.00.06 which is the reason for the loop. However, while I have no reason to believe there will be any issues, I haven't tested this code (two wire uNPI) with the latest RTOS yet. This will also require the rest of the SNP project to be ported which will take some time, but I believe is the best long term solution

2. We can work to characterize the delay on your system and see if we can find an acceptable number lower than 32000.

I recommend #1 as the solution and am fully willing to support the effort if you think it is the best course of action

I agree with your suggestion, but it sounds like porting the project to 2.15 will take some time so the other options are more appealing.

1. Is there a way to simulate the UART flush with the available drivers?
2. If I go with the second option, do you know what I can expect to encounter when the loop counter is too low?

Lastly, about the updates to the SNP project, are there plans to support updates to the BLE stack? and BT 4.2 support?

Thanks.

Yes, porting to the latest RTOS will take some time. There have been changes at the driver level.

I can confirm that our next SDK release will use an updated version of the RTOS. I also plan to port this example as well once the SDK is released so if you wait, then you can simply reference that.

1. Unfortunately, no. This is why I had resorted to the above while loop code. A much better design would have been to flush the FIFO.

2. Yes, you will get a UART readCallBack immediately after calling doHandshake even when no bytes have been Rx'd on the line. This is the mystery byte that the comment alludes to. For me, this was always 0xF8. You could modify the readCallback in the npi_tl_uart_swhs.c file to scan for this case, ignore the byte, and trigger another read.

Alternatively, since the RTOS UART drivers are provided in source format, you could add the command to the UART driver implementation.

Below is the UARTCC26XX_control() call that implements the flush.

int UARTCC26XX_control(UART_Handle handle, unsigned int cmd, void *arg)
{
    /* Locals */
    UARTCC26XX_Object            *object;
    UARTCC26XX_HWAttrsV1 const   *hwAttrs;
    unsigned int                 key;

    /* Get the pointer to the object and hwAttrs */
    object = handle->object;

    /* Initialize return value*/
    int ret = UART_STATUS_UNDEFINEDCMD;
    /* Do command*/
    switch(cmd)
    {
        case UARTCC26XX_CMD_RETURN_PARTIAL_ENABLE:
            /* Enable RETURN_PARTIAL */
            object->readRetPartial = true;
            ret = UART_STATUS_SUCCESS;
        break;

        case UARTCC26XX_CMD_RETURN_PARTIAL_DISABLE:
            /* Disable RETURN_PARTIAL */
            object->readRetPartial = false;
            ret = UART_STATUS_SUCCESS;
        break;

        case UARTCC26XX_CMD_RX_FIFO_FLUSH:
            /* Flush RX FIFO */
            hwAttrs = handle->hwAttrs;

            /* Disable interrupts to avoid reading data while changing state. */
            key = Hwi_disable();

            /* Read RX FIFO until empty */
            while (((int32_t)UARTCharGetNonBlocking(hwAttrs->baseAddr)) != -1);

            /* Set size = 0 to prevent reading and restore interrupts. */
            object->readSize = 0;
            Hwi_restore(key);

            ret = UART_STATUS_SUCCESS;
        break;

        default:
            /* This command is not defined */
            ret = UART_STATUS_UNDEFINEDCMD;
        break;
    }

    /* Return */
    return (ret);
}

Awesome. I'll check this out.
What the schedule for the SDK release?

Best regards

The SDK is planned for public release in Q2 this year.
Contact your TI rep for more info if you are interested in the beta program.

Looking forward for the updated SDK.

I've added the UART flush and it works great. Now the time it takes for the CC2650 to chrip back is 150us, which is much better.
Along with higher UART baudrate, I think it's possible for the NPI protocol not to be the bottleneck in the BLE channel.

I'll keep on working on it.
Many thanks for the help.

Best regards,
Gregory

Next issue I'm encountering with the SW handshake:

1. SNP (cc2650) sends a chirp
2. SAP ACKs the chirp within 90-150us
3. read callback is not registered on the SNP
4. protocol continues with chirp timeouts from the SNP forever

This behavior does not happen in the same place at the same time, but it's occurring seldom.
Not sure about it, but this fault looks like it's happening when the SAP starts sending a NPI packet after successful packet transmission from the SAP.

I'm attaching a logic trace of a sample failure.
I've added to the SAP protocol ability to acknowledge timeout chirp, even if the SAP already chirped back already.
Also, for debug purpose, I've removed the SNP reset from the initialization protocol. It does not affect the bug occurrence.

I'll appreciate the help.

Best regards,

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/538/6-MHz_2C00_-1-B-Samples-_5B00_81_5D00_.7z

I have further questions not directly talking about the handshake procedure. I can open a new question, if you suggest so.

1. What happens when there's a collision on the NPI bus? When both SAP and SNP want to transmit, like when there's a BLE read/write request from the BLE central and an indication/notification from the SAP and both want to use the NPI channel?
2. What's the point in the notification confirmation packet (SNP_SEND_NOTIF_IND_CNF)? Is there any practical use for it? Notifications are not acknowledged by the BLE protocol (that's what indications are for) so I don't understand the necessity. If it can be removed, I'll be happy to do so as the acknowledge takes about 70% of the notification procedure time.

Thanks.

Hi Gregory,

I have review your logic trace and agree with your analysis. Can you try enabling NPI_SW_HANDSHAKING_DEBUG which will toggle a GPIO during the critical section where a chirp may be missed?

This can provide some insight as to why the SNP may be missing the chirp. In any case it should be handling the repeated chirps.

I've added the symbol SWHS_DEBUG. It changed in commit d5421b5c.

Attached is the logic trace. This time it happened after connecting to the BLE central and after sending a few dozen notifications.

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/538/6-MHz_2C00_-1-B-Samples-_5B00_7_5D00_.7z

Hi Sean,

I'm still unable to resolve this issue. It looks like a bug in the CC26xx UART driver.

I'm trying to update the cc26xx UART driver to the one in 2.16.00.08, hoping I don't have to adapt the entire TI-RTOS. So far I've done the following:

1. Taken extracts from UARTCC26XX.c/.h and basically added the swi in the read callback. The files are attached.
2. Changed Board.c to this
   

   /* UART hardware parameter structure, also used to assign UART pins */
   const UARTCC26XX_HWAttrs uartCC26XXHWAttrs[CC2650_UARTCOUNT] = {
       {    /* CC2650_UART0 */
           .baseAddr = UART0_BASE,
           .intNum = INT_UART0,
           .powerMngrId = PERIPH_UART0,
           .intPriority = ~0,
           .swiPriority = 0,
           .txPin = Board_UART_TX,
           .rxPin = Board_UART_RX,
           .ctsPin = PIN_UNASSIGNED,
           .rtsPin = PIN_UNASSIGNED
       },
   };

3. Change in appBLE.cfg (because of not enough RAM):
   Swi.numPriorities = 3;

There's somewhat better behavior (faster response), but the SNP is stuck after the first characteristic declaration in the SAP bringup part (attached logic trace).
I'm really stuck on this. Is there anything else that I can do?

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/538/UARTCC26XX.7z

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/538/cc2640uart-new.7z

I've ported the SNP project to TIRTOS 2.15.00.17 and all I can say is that the UART driver is behaving badly. The addition of the SW interrupt in the cc26xx UART RX callback is doing something wrong in the NPI flow.
On arrival of packets larger than the RX FIFO threshold (16 bytes by default), the NPI protocol stops because NPITLUART_readCallBack receives a size 0 payload when expecting payload of size 9, for example.

Just in case, I've also modified enter/exit CS:

_npiCSKey_t NPIUtil_EnterCS(void)
{
    _npiCSKey_t key;
    key.taskkey = (uint_least16_t) Task_disable();
    key.swikey = (uint_least16_t) Swi_disable();
    key.hwikey = (uint_least16_t) Hwi_disable();
    return key;
}

// -----------------------------------------------------------------------------
//! \brief      Critical section exit. Enables Tasks and HWI
//!
//! \param    key   key obtained with corresponding call to EnterCS()
//!
//! \return   void
// -----------------------------------------------------------------------------
void NPIUtil_ExitCS(_npiCSKey_t key)
{
    Hwi_restore((UInt) key.hwikey);
    Swi_restore((UInt) key.swikey);
    Task_restore((UInt) key.taskkey);
}

Best regards.

Hi Gregory,

I apologize again for the long wait time. Would it be possible for you to share you code with me? (only NPI layer and UART driver). 

We need to work together to root cause this bug so that it may be improved for subsequent releases.

Hi Sean,
I've sent you an email (hopefully it's really you) with an invite to our bitbucket repository.
Please see if you get it and email me back at gkovelman at motionize dash inc dot com

Best regards