CC2564C: mSBC over HCI issue.

Here is the link for the issue we raised on this topic.

cc2564c-msbc-over-hci-packets-getting-corrupted

Thanks!

why are you using the msbc over hci and not the assisted WBS?

Hi Kobi,

It is not an option. Because we need BLE to co-exist with the HFP & A2DP profiles.

Regards!

Hi Kobi,

You can refer to the following link for the Background of our issue and project.

cc2564c-msbc-over-hci-packets-getting-corrupted

We have also worked closely with TI on this specific product. I will share the details as DM. 

Key Points to note: Assisted mode is not an option for us at this point, as we need BLE to Co-exist.

Points we like to get Answered: Why is the issue observed on some specific phones? Is there any dependency on SCO parameters that affects the cc2564C to enter into this state?

Regards!

ok. i understand now.

looking at the other thread - does the wrong mSBC data always starts with "6c, 00" as in you example?  Did you try to skip those? (i.e. update the pointer before calling SBC_Decode_Data)

looking at another related thread (https://e2e.ti.com/support/wireless-connectivity/bluetooth-group/bluetooth/f/bluetooth-forum/845317/cc256xc-bt-sp-msbc-audio-data-problem-for-sco-over-hci) it seems that there are couple of 2 bytes patterns that may come before the sync word ("6c 00", "76 DB", "6D B7" )  that you may skip (look for the "0xAD" sync word and provide this to the SBC_Decode_Data()). i'll try to check the spec to see if they make any sense but it seems that the Decode function doesn't expect them. 

looking on it a little more (i'm not an expert of the HFP) it seems that the buffer can be fragmented (so one mSBC frame can start at one HCI frame and continue at the next one). I think you should copy it to some temporary buffer and call when the full frame is available starting from the 0xAD sync word. 

I'll try to read about it and provide more details later this week. 

Hi Kobi,

Yes, as a patch we are doing the process of updating the pointer.

But the issue is that,  if the bit padding or offset is more than 4 or 6 bytes then the audio Quality goes bad with graininess etc. As the mSBC data comes as 60 bytes, and we are offsetting the pointers, so for sure the last bytes mSBC decodes will be Junk(from the 60 if we offset 4 bytes the data available is only 56 bytes the rest of the 4 bytes will be out of that packet. 

on the Second Approach, which is storing into a buffer and doing the offset also doesn't seem to be appealing. As we analyzed the data over time, it doesn't seem like the data is split over HCI_Packet.

Regards,

Vishnu

Hi Kobi,

One more data we like to get.

In the following e2e thread cc2564c-corrupted-msbc-data-seen-with-voice-over-hci ,

Vihang Parmar has mentioned something communicated over Email in the last comment.

Can you get that detail please?  If TI has already some findings on this  it will be better to take a look on those.

Thanks!

Thanks for the reference. I will try to get the info from Vihang.

Hi Kobi,

Any update on this Issue?

Thanks!

no update. i'm still waiting for Vihang's response.

He remembered couple of issues with voice over HCI but needed to check to the details.

I'll ping him and let you know the findings.

Ok. Please do.

The product is already in market.

We need to get this addressed as fast as possible. 

We are getting a lot of complaints regarding this issue. 

Thanks!

We will try to find an answer as soon as possible.

Hi kobi,

Any update?

I know, you are still waiting for Vihnags response. 

Meantime, Is there any parameter we can try playing with, so the issue won't happen or the issue rate we can reduce?

Regards,

we are still looking into this but it is taking longer than we thought.

I'll update as soon as I'll have anything.

Hi Kobi,

Thanks for the efforts you are taking. 

Waiting for the response.

(FYI If you are trying to reproduce the issue. For  that I can help you.)

Regards

Hi Kobi,

Any update?

can you share at least the minimum details you know on this?

Which will help us move forward. 

At least let me know whether the issue has some fix or not. 

Whether it is hardware or in the stack?

We are very locked with this issue. Please help.

Regards,

Any update ?

Thanks for the information.

Is this limitation  due to hardware or Stack related ? 

Regards,

Is it related to any sco parameters? Why we are observing the issue only with some specific phones?

In some phones the rate of issue is too low.

Some phones like Samsung , iPhone 7  shows the issue very high.

Some iPhones doesn't show the issue, some redmi also we are not observing the issue. Is there any justification for these? 

Thanks!

Hi Kobi,

Thanks for the  genuine responses. 

I understand the issue is part of both hardware and stack. 

It will be better if it is mentioned in the datasheet clearly. 

Also I understand TI is not giving further support for this chipset.

Considering all those points, 

Is it possible for TI to provide  the stack as source code so that we can try to work and tune to get it into some better state. As I already mentioned we will be in a very bad situation with this issue. The devices already reached the customer hands. HFP calls are one of our Major feature. 

 Is it possible to reset the internal SCO buffers periodically, without dropping the sco connection? 

Thanks! 

Hi,

We can't provide the stack sources due to legal issues.

I'll try to check about the periodic reset for the buffer.

Br,

Kobi

Hi Kobi,

Yes, please check and let me know about the buffer reset option. That will be helpful.

On the other hand, we are planning to do some tests as you suggested on our test setup.

Test setup we thought of

STM32F4 MCU as I2S master.

CC564C as I2S slave. 

When a narrow Band HFP connection comes we will switch to the PCM mode.

In other cases  ( A2DP / mSBC ) we will continue with the  Audio over HCI itself.

Do you see any Potential Issues with this approach?

Thanks & Regards

Vishnu.

Hi Kobi, 

Any update on this ?

is there any possibility to reset the Buffers?

thanks!

It looks like there is no simple way to reset the buffers without resetting the entire connection.

Thanks for the Information.

Hi kobi,

Thanks for the helps.

I like to know something related to this.

When the SCO is active and if some ACL traffic is coming in parallel, we observed that the system is entering Hangstate and we found that  the hang is happening consistently on the execution of a particular line in  RXThread( ) (this function is inside hcitrans.c).   

The line is as follows,

 Any info regarding this behavior?

Just before this issue is getting triggered, We observed that the Sco data packets go in a pattern of  GBGBGBGBGB... sometimes GBGBGBBGBGBG...,(where G represents Goodpacket, and B represents BadPacket in the SCO over HCI).

Any information on this behavior?

Thanks!

So you are still trying to work with SCO over HCI, right?

As i told you before, this interface had couple of issues before - although i'm not familiar with the specific one you mentioned.

So you are still trying to work with SCO over HCI, right?

As the Product is already with 1000+ customers, We have to do something right? 

In the coming months, it will be reaching more than 4000+ hands.

Previously When I raised the Issue, It was not clearly conveyed by TI that the issue is with Hardware and it doesn't have any fix.

Just FYI, we have even sent our boards to TI for these particular issues. 

If TI had confirmed the issue and clearly mention it in the datasheet, we wouldn't have ended up in this situation.

Yes, on the next hardware version, we are considering removing the voice-over HCI part.

But that doesn't solve the issue for existing users.

It will be very bad to tell users there are issues with hardware right?

We are trying to mask the behavior by doing the detection using FW and testing the SCO, so that user no need to suffer from these issues.

I understand  the helplessness on your side, 

But I still have to explore any possible way we can make the user experience better.

Thanks!

what is the nature of this "Hang state"? Do you see any abort or exception? 

It may be related to stack size of the thread - can you try to increase the the RxThread stack (RX_THREAD_STACK_SIZE)?

Also try to increase heap size of the project to eliminate other memory issues.

what is the nature of this "Hang state"? Do you see any abort or exception?

We didn't observe any abort or exception when this happens.

When this hang triggers, (FYI we are using FreeRTOS version of Bluetopia Stack and we are on the latest service pack.)

*The ostimers stop working.

*The UART communication with the CC2564C is broken.

*Debug UART print works fine.

*One other Audio Thread in our system is still running.

* We observed that the osMessageput() is not able to queue the messages. (even though it is not related to Bluetopia stack, this scenario is also observed).

* We also observed that the Stack size of Bluetopia is getting decreased rapidly. But when the issue happens stack still has Memory free of around 7 KB.  If we increase the Bluetopia stack size, then we are seeing that the 7KB point changes to 12 KB (assume we increased  stack by 5 KB). The figures are not exact values but around those. This rapid stack reduction happens in 3-4  s from when it triggers to the point when it hangs, by this time stack reduces by an amount of  3-6 KB. (this reduction I think, it is because of the UART RX data is feeding in with BTPS_Malloc()). For the memory usage data, we used  "BTPS_QueryMemoryUsage()" function.

One other major observation we have here is that when the device has 2 Masters (scenario when CC2564C is connected to 2 other devices).

ie. CC2564C acts as a Slave to both devices, in this scenario this issue happens within 10-20s.

If the connection on CC2564C is like 1 Master and 1 Slave, then it takes about 3-7 minutes to get this issue.

if CC2564C is the Master to both Connections, The issue is not observed. 

I am not sure how can we relate this to the Stack entering the BadState, but this is our observation.

It may be related to stack size of the thread - can you try to increase the the RxThread stack (RX_THREAD_STACK_SIZE)?

 I am not able to find "RX_THREAD_STACK_SIZE" in the HCITRANS Configuration Files.

But we have tried Increasing the RXThread StackSize on the following line 

/* Create a thread that will process the received data. */
UartContext.ReceiveThreadHandle = BTPS_CreateThread(RxThread, 1600, NULL); 

In this 1600 is what we normally use.  I tried Increasing that to 3200 but still got the issue, didn't observe any change.

Also try to increase heap size of the project to eliminate other memory issues

Yes, we tried changing those. but didn't observe the change. Also, we tried changing thread priority to RealTime, but Still no luck.

we have  80KB allocated for FreeRTOS, 30KB for Bluetopia. and We are running on STM32F4 processor with 125DMIPS(100MHz).

Thanks!

Hi Kobi,

I am attaching the BT logger Logs when the device hangs.

This is the portion when the Issue happens. (It is not exactly this portion, around Here).

You can look around line no -  922254.

Device_Hangedin RXTHread_Reduced.zip

The behavior of issue is similar to ConfigASSERT getting triggered.  ( #define configASSERT( x ) if ((x) == 0) { for( ;; );} )   // This I reproduced while trying to start a timer whose Timer ID was NULL.

Behavior is ostimer Misbehaves, Message queues not working etc.

I tried replacing the ConfigASSERT function with 

#define configASSERT( x ) if ((x) == 0) {taskDISABLE_INTERRUPTS(); rvUserAssertInformFunction(__FILE__, __LINE__); for( ;; );}

But when this issue happens configASSERT is not getting called.

is it possible that there exists some infinite for loop or while loop inside the stack which is getting triggered?

Also, on the other point you made

"what is the nature of this "Hang state"? Do you see any abort or exception? "

Do I have to Register for Specific Callbacks or exception Catchers to get these exceptions?

Regards,

what is osMessageput? 

How do you knw the Audio thread is still working if you don't get any input?

How do you know the debug UART is working? what messages are being sent?

If the app doesn't crash (as it seems to be), then it is probably not related to memory corruption but to a thread synchronization.

If you fill up a message queue, try to increase the queue size.

what is osMessageput? 

This is a FreeRTOS function to Manage the Queue. This we are using for Statemachine Handling.

How do you knw the Audio thread is still working if you don't get any input?

The Audio is a free Running thread triggered by I2S events. So it will keep on Running. Doesn't matter whether data came or not. 

How do you know the debug UART is working? what messages are being sent?

We are able to see the Display Statements we added in the Audio Thread. The debug Prints say loop run count etc.

Reagrds,

I will need to consult with our Bluetooth FW experts about log.

I'll respond as soon as we find anything.

Thanks, Kobi.

Waiting for the Response from your Team.

Some more inputs regarding what we found and the mechanism we use now to recover from the hang condition is attached here.

The logic seems to work as per our testing, But we are unclear about why this is happening and how it is getting fixed.

The logic we use is as Follows

Just before this issue is getting triggered, We observed that the Sco data packets go in a pattern of  GBGBGBGBGB... sometimes GBGBGBBGBGBG...,(where G represents Goodpacket, and B represents BadPacket in the SCO over HCI).

But this pattern seems to trigger very frequently sometimes, even when the issue is not happening.

So we found one more checkpoint to detect the condition.

We observed that the Stack is reducing Rapidly, this also we are considering doing the Reset.

PseudoCode for Issue Detection is added Below

void PseudoCode_IssueDetection(){
	
	BTPS_MemoryStatistics_t MemoryStatistics;
	BTPS_QueryMemoryUsage(&MemoryStatistics, TRUE);
	
	if(GOODPACKET){
		ContinuousBadPacket =0;
		if(PrevWasGoodPacket == TRUE){
			DetectHangStatePattern = 0;
		}
		PrevWasGoodPacket = TRUE;
		ContinuousGoodPacket++;
		if(DetectBadStateMemoryUsedinitially > (int32_t)MemoryStatistics.CurrentHeapUsed ){
			DetectBadStateMemoryUsedinitially = (int32_t)MemoryStatistics.CurrentHeapUsed;
		}		
	}else{ // BADPACKET
		ContinuousBadPacket++;
		DetectHangStatePattern++;
		if((DetectHangStatePattern - ContinuousBadPacket) > 4){
			if(((int32_t)MemoryStatistics.CurrentHeapUsed - DetectBadStateMemoryUsedinitially ) > 2000){
				 Display(("ISSUE_DETECTED\n"));
				 AudioQualityReconnectionManage();
			}
			DetectHangStatePattern =0;
		}
	} 
}

***DetectBadStateMemoryUsedinitially is the Stack used initially after the HFP is started and the mSBC encoder is Initialized.

As specific to our application, When SCO is active we won't be allocating any huge Resources from Bluetopia Stack, so we are taking advantage of that here.

In AudioQualityReconnectionManage() function we  use HCI_Disconnect(BluetoothStackID,SCOConnectionHandle, HCI_ERROR_CODE_OTHER_END_TERMINATED_CONNECTION_USER_ENDED,0);

Then we reinitiate the SCO-connection  with HFRE_Setup_Audio_Connection();

This gives a bad User experience as the audio will go for around 3 Seconds, but still better than Hang.

One extra observation is that if the device is Master to both devices the rate of reproducibility of issue is very very low.

Thanks,

It is great that you found some kind of a patch.

While trying to help here please note that not many customers (if any) used the SCO over HCI and we have minimal experience with this so we are mostly relying on your tests currently.

Thanks Kobi!