SRIO ChipToChip Failed: Invalid queueID

I have same problem,

I'm using our custom board.

Our hardware is....

C6678 <--x4(5Gbps)-->  SRIO Switch <--x4 (5Gbps)--> C6678

The  "SRIO_TputBenchMarkingTest"  example works fine..

But when I run "srioChipToChipProducer/Consumer" example.

I have same problem.

Return status always -5, the val of &remoteQId always 0

How to fix or  debug it ? 

Hi Yanuar,

Is there a particular reason you are using the IPC example? You might want to use the throughput benchmarking example provided. The CCS project can be found in the following location:

<install_dir>\pdk_C6670_1_1_2_6\packages\ti\drv\exampleProjects\SRIO_TputBenchmarkingTestProject

This example is thoroughly documented and provides easy customization for  different configurations. The guide for this example can be found at the following location:

<install_dir>\pdk_C6670_1_1_2_6\packages\ti\drv\srio\test\tput_benchmarking\docs

Let us know if you have any additional questions or comments.

Thanks,

Clinton

DaeHwan,

Glad you were able to get the SRIO_TputBenchmarkingTest working. How do you use SRIO in your application? Are you intending to use SRIO for IPC? Is there something missing in the TputBenchmarking example that you need for your configuration?

Thanks,

Clinton

We are also experiencing these problem with the srioChipToChip example on our hardware (invalid QueueID as well as nameserver issues), along with some other difficulties reported here: http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/t/298908.aspx

After digging a bit in the source of the SRIO IPC transport, we found the issue is caused by an invalid MessageQ-ID (65536) received in the SRIO isr (TransportSrio_isr). The ISR then calls MessageQ_put to queue the received message in a local queue, which asserts at:

    if (dstProcId != MultiProc_self()) {
......
} else
{
        Assert_isTrue((UInt16)queueId < MessageQ_module->numQueues,
                      ti_sdo_ipc_MessageQ_A_invalidQueueId);
....
}

Unfourtunately we weren't able to debug it further, as the transport seems to be compiled with optimizations, and BIOS.libType = LibType.Custom rebuilds Sys/BIOS+IPC but not the SRIO transport. Any hints how to manually re-compile the SRIO/IPC transport are very welcome.

Thanks, Clemens

Because it was asked: We would like to use IPC over SRIO as it provides a consistent interface with the shared memory implementation and furthermore allows us to avoid to get too much involved into SRIO internals.

Clinton,

I think. I did not missing anything. I checked several times.

Could you explain more detail. What kind of reasons to make this problem? 

Many persons talking about this issue(ChipToChip Srio IPC).

Thanks,

DaeHwan Park.

Still no suitable answer for this InvalidQueueID problem I guess.

Anyone maybe have same workflow like me. So, I want to connect:

|DSP| ---SRIO--> |FPGA w/ RF| --> wave --> |RF w/ FPGA| ---SRIO--> |DSP|

But, because at the start and the end this flow, we have DSPs with SRIO, I want to try DSP-to-DSP connection using SRIO.

Maybe for an alternative, for now, I am using standard loopback SRIO example with PLL changing (need to have same baudrate for DSP and the FPGA, I am using 1.25 Gbaud/s), only port-0 enable and successfully write the packets to FPGA (Terasic DE4). Now, working on the Rx. 

But still, it will be a very great help if this InvalidQueueID problem resolved.

Thanks,

Nugraha

Hi there! 

Good news. Somehow, I managed to get this problem done.

I've found an incorrect value of PLL on the example (PDK 1_1_2_6). CMIIW.

All of these ideas about PLL, Sample Rate and Line Rate can be seen on sprugw1a.pdf (SRIO User Guide). But, in short

//the idea 

Line Rate = PLL clock vs MPY vs Rate

//Line Rate (GBaud/s) = {1.25, 2.5, 3.125, 5 etc)
//PLL clock (MHz) = {156.25, 250, 312.5}
//MPY (x) = from 4x to 25x, obtained from SERDES PLL value
//Rate (samples/clock) = {4, 2, 1, 0.5}

In the consumer_device_srio.c and producer_device_srio.c, the PLL value is 0x233 (1000110011, italicized binaries are the MPY), that means the MPY is not determined (at least i couldn't find MPY value which correspond to the value of 0b00011001 in table 2-4). 

But, from the SERDES Tx and Rx BootCFG value, the value of Rate is 0b01, which means there are two samples per clock.

So, I just changed the MPY value to 10x, to get 250MHz x 10 x 2 samples/clock to obtain 5 GBaud/s of Line Rate. From the table 2-4, the PLL value now 0x251 (1001010001). Run the test again in both of EVM and the console reported:

===============================================================

-----------------------Initializing---------------------------
Core 0 : L1D cache size 4. L2 cache size 0.
Core 0 : Memory region 0 inserted
[C66xx_1] Local Core ("CORE1") starting
Local Core ID: 1
Global Core ID: 3
Core 3: Waiting for SRIO to be initialized.
[C66xx_0] Port 0 did not initialize
Core 0: SRIO Driver has been initialized
[C66xx_1] Core 1: SRIO can now be used.
localQueueName=CORE1
remoteQueueName=CORE3
Core 1: tsk0 starting
[C66xx_0] localQueueName=CORE0
remoteQueueName=CORE2
Core 0: tsk0 starting
Global Core 0: Sending packets to an off-chip core.
Global Core 0 attempting to open remote board Queue CORE2
[C66xx_1] Global Core 1: Sending packets to an off-chip core.
Global Core 1 attempting to open remote board Queue CORE3
[C66xx_0] Local Core ("CORE0") starting
Local Core ID: 0
Global Core ID: 2

-----------------------Initializing---------------------------
Core 2 : L1D cache size 4. L2 cache size 0.
Core 2 : Memory region 0 inserted
Core 2: SRIO Driver has been initialized
[C66xx_1] Core 3: SRIO can now be used.
localQueueName=CORE3
Core 3: tsk0 starting
[C66xx_0] localQueueName=CORE2
Core 2: tsk0 starting
Global Core 2: Receiving packets from an off-chip core.
[C66xx_1] Global Core 3: Receiving packets from an off-chip core.
[C66xx_0] Global Core 0 opened messageQ with name CORE2 and ID 20001
Core 0: Sent 4 messages.
Core 2: received 4 messages from core 0
[C66xx_1] Global Core 1 opened messageQ with name CORE3 and ID 30001
[C66xx_0] Chip to chip message transfer passed!
[C66xx_1] Core 1: Sent 4 messages.
Core 3: received 4 messages from core 1
Chip to chip message transfer passed!

===============================================================

Now, I am trying to use lower Line Rate (2500 MBaud/s and 1250 MBaud/s)

Regards,

Nugraha

| update |

I've tested on 1.25 and 2.5 GBaud/s of Line Rate and failed. Even going back to 5 GBaud/s setting also failed. Checked on SW3 to SW6 and both DSPs are already in no-boot mode.

Any helps appreciated.

Regards,

Aditya

> I've tested on 1.25 and 2.5 GBaud/s of Line Rate and failed.
> Even going back to 5 GBaud/s setting also failed.

Probably it was a lucky try. On my system the example has a sucess-rate of ~30%, so it is quite hard to check wether a change fixed it or wether it was just pure luck.

In the end it seems like the IPC SRIO transport has some serious bugs which hopefully wil be fixed by TI soon.
However the lack of qualified feedback and/or instructions is quite unsettling for me.

Regards, Clemens

Hi Nugraha,

Thanks for sharing your progress. What changes did you make to the example to move to the lower line rates?

I have been testing this example and have run into similar issues. I'm currently debugging my setup and doing further research. I will provide updates as they are available.

Any other feedback or progress updates from users having issues with the IPC example is appreciated.

Thanks,

Clinton

Hi,

I made some further investigations on the above issue, as clemens reported.

This assertion in MessageQ_put is caused by an invalid SRIO message.

[C66xx_0] ti.sdo.ipc.MessageQ: line 494: assertion failure: A_invalidQueueId: Invalid queueId is used

xdc.runtime.Error.raise: terminating execution

Actually SRIO transport receives messages with msgId = 0xFFFF (and dstID=0xFFF). This is an inavlid message, according MessageQ.h

#define MessageQ_INVALIDMSGID           (0xffff)
#define MessageQ_INVALIDMESSAGEQ        (0xffff)

The reception of those invalid message seems to be arbitary. During a test, this kind of messages can be received on any core, independent of how dspProc had been set. Even messages might be received on the the same core, from which they had been sent before. I found no indications that message with bad IDs have been transmittet by a sender.

So the first important question arrises.

What are the potential sources (or causes) for the reception of invalid messages ?

This is a serious issue and  I need urgent solution !

Second:

Messages from srio transport, identified with msgID= MessageQ_INVALIDMESSAGEQ are not handled properly by MessageQ and trasnport_SRIO.

See TransportSrio.c: TransportSrio_isr

Void TransportSrio_isr(UArg arg)
{

    /* Reset heapId to rxMsgQHeapId.  This heap is overwritten when the message is received over SRIO */
    rxMsg->heapId = TransportSrio_module->rxMsgQHeapId;
    
    queueId = MessageQ_getDstQueue(rxMsg);
    
    /* Put messageQ buffer on queue */
    MessageQ_put(queueId, rxMsg);
    

When queueID of an invalid Message is derived, the dstProc Number field ist set to 0.

(see MessageQ.c, #define MessageQ_getDstQueue )

Which means, any invalid message is routed to processor0.
Furthermode MessageQ_put does not check for invalid msqID, it will transport an already invalid message via SRIO to some other place, from the context of an interrupt.

Question 2: What ist the intented action when MessageQ receives an invalid message from srio ?

best regards

Alois

-----------------------------------------------------------------------------------------------------------

appendix 21-Jan-2014

Here some details about the configuration of my testcase

I use a custom board with two TMS320C6678 (DSPA and DSPB)

SRIO lines of DSPA and DSPB  connected back-to-back, running at 3.125 GBaud, one x4 lanes

TNS320C6678 Silicon Revision 1.0

Code of SRIO testcase is attached, srio_test_ti.c

#include <xdc/std.h>
#include <xdc/runtime/System.h>
#include <xdc/runtime/IHeap.h>
#include <xdc/cfg/global.h>

#include <ti/ipc/HeapBufMP.h>
#include <ti/ipc/MessageQ.h>

#include "vision_box.h"

/* These are the garbage queues used in the test Application */
const UInt32 GARBAGE_LEN_QUEUE = 905;
const UInt32 GARBAGE_TOUT_QUEUE = SrioGarbageQ; /* Timeout errors must be cleaned up by transport */
const UInt32 GARBAGE_RETRY_QUEUE = 907;
const UInt32 GARBAGE_TRANS_ERR_QUEUE = SrioGarbageQ; /* Transmission errors must be cleaned up by transport */
const UInt32 GARBAGE_PROG_QUEUE = 909;
const UInt32 GARBAGE_SSIZE_QUEUE = 910;

#define HEAP_NAME	"MsqHeapBuf"
#define HEAP_ID		0

#define NUM_HOST_DESC 256

extern char RemoteQueueName[16][20];
extern MessageQ_Handle messageQ;
MessageQ_Handle messageQ = NULL;
char LocalQueueName[20];
char RemoteQueueName[16][20];

struct TestMsg
{
	MessageQ_MsgHeader header;
	unsigned int data[20];
};

void srio_test()
{
	Int32 status;
	HeapBufMP_Handle heapHandle;
	HeapBufMP_Params heapBufParams;
	struct TestMsg *pTestMsg;
	int i, loop = 0;
	MessageQ_QueueId remoteQId;

	// Create the local queue name (VIB_HWInfo.CPUID identifies DSPA/DSPB, 0=DSPA, 1=DSPB)
	sprintf(LocalQueueName, "CORE%u", VIB_HWInfo.CPUID*8+DNUM);

	// Create the remote queue names "CORE0...15"
	int core;
	for (core=0; core<16; core++)
		sprintf(RemoteQueueName[core], "CORE%u", core);

	// Create the local message queue
	messageQ = MessageQ_create(LocalQueueName, NULL);
	if (messageQ == NULL)
		System_abort("ERROR: MessageQ_create() failed\n");

	if (DNUM == 0)
	{
		/* Create the heap that will be used to allocate messages. */
		HeapBufMP_Params_init(&heapBufParams);
		heapBufParams.regionId       = 0;	// SharedRegion ID
		heapBufParams.name           = HEAP_NAME;
		heapBufParams.numBlocks      = NUM_HOST_DESC;
		heapBufParams.blockSize      = SRIO_MTU_SIZE;
		heapHandle = HeapBufMP_create(&heapBufParams);
		if (heapHandle == NULL)
			System_abort("HeapBufMP_create failed\n" );
	}
	else
	{
		/* Open the heap created by the other processor. Loop until opened. */
		do
		{
			status = HeapBufMP_open(HEAP_NAME, &heapHandle);
		} while (status < 0);
	}

	/* Each core must register this heap with MessageQ */
	MessageQ_registerHeap((IHeap_Handle)heapHandle, HEAP_ID);

	if (DNUM == 0)
	{
		System_printf("Waiting for messages...\n");
		while (1)
		{
			status = MessageQ_get(messageQ, (MessageQ_Msg *)&pTestMsg, MessageQ_FOREVER);
			if (status < 0)
				System_abort("MessageQ_get failed\n");

			// reply message
			if (MessageQ_put(MessageQ_getReplyQueue((MessageQ_Msg)pTestMsg), (MessageQ_Msg)pTestMsg) < 0)
				System_abort("MessageQ_put failed\n");

      if ((loop % 100000) == 0) {
        System_printf("looped (%u) messages\n", loop);
      }
      
			loop++;
		}
	}
	else
	{
		// Send to core0 of other DSP always, VIB_HWInfo.CPUID identifies DSPA/DSPB, 0=DSPA, 1=DSPB)
		unsigned int remoteCoreId = (VIB_HWInfo.CPUID == 0 ? 8 : 0);

		System_printf("Opening message queue '%s'\n", RemoteQueueName[remoteCoreId]);
		do
		{
			status = MessageQ_open(RemoteQueueName[remoteCoreId], &remoteQId);
		}
		while (status < 0);

		while (1)
		{
			//System_printf("Sending massage (%u)\n", loop);
			pTestMsg = (struct TestMsg *)MessageQ_alloc(HEAP_ID, sizeof(struct TestMsg));
			if (pTestMsg == NULL)
				System_abort("MessageQ_alloc failed\n");

			MessageQ_setReplyQueue(messageQ, (MessageQ_Msg)pTestMsg);

			for (i=0; i<20; i++)
			{
				pTestMsg->data[i] = i + DNUM + 8*VIB_HWInfo.CPUID;
			}

			if (MessageQ_put(remoteQId, (MessageQ_Msg)pTestMsg) < 0)
				System_abort("MessageQ_put failed\n");

			status = MessageQ_get(messageQ, (MessageQ_Msg *)&pTestMsg, MessageQ_FOREVER);
			if (status < 0)
				System_abort("MessageQ_get failed\n");

			for (i=0; i<20; i++)
			{
				if (pTestMsg->data[i] != i + DNUM + 8*VIB_HWInfo.CPUID)
					System_printf("Data error: pTestMsg->data[%u] = %u\n", i, pTestMsg->data[i]);
			}

			MessageQ_free((MessageQ_Msg)pTestMsg);

			loop++;
		}
	}
}

Alois, All,

Thanks for posting your issues and providing feedback. I wanted to give an update on this issue. It has been escalated and additional resources are investigating this issue. I will post progress as we work on a solution. 

Please continue to post any other feedback or developments that you have with this example.

Thanks,

Clinton

All,

The IPC Transports can be rebuilt for experimentation and single-step debug by following the instructions located here:

http://processors.wiki.ti.com/index.php/BIOS_MCSDK_2.0_User_Guide#Rebuilding_the_IPC_Transports

Don't forget to clean and rebuild the CCS example projects after rebuilding the IPC transport so that the newly generated library gets picked up the example executable.

Justin

Additionally, you can set the XDCOPTIONS for verbose on the command line to verify the compiler is using the flags that have been set in the config.bld files

> set XDCOPTIONS=-v

Hi,

I use the BIOS.libType setting in the .cfg file for rebuilding IPC and SYS/BIOS (according this post)

http://e2e.ti.com/support/embedded/bios/f/355/t/298692.aspx

I found it very important and helpful, to remove the --optimize_with_debug compiler option. Otherwise call stacks are not displayed.

BIOS.libType = BIOS.LibType_Custom;
BIOS.customCCOpts = BIOS.customCCOpts.replace("-o3", "-o0");
BIOS.customCCOpts = BIOS.customCCOpts.replace("--optimize_with_debug", "");
print(BIOS.customCCOpts);

regards

Alois

Hi Clinton,

do you have any news about the IPC SRIO transport issue ?

regards

ClintonRodgers said:

Alois, All,

Thanks for posting your issues and providing feedback. I wanted to give an update on this issue. It has been escalated and additional resources are investigating this issue. I will post progress as we work on a solution. 

Please continue to post any other feedback or developments that you have with this example.

Thanks,

Clinton

Clinton,

Any update from TI?  I wonder how many other people have spent time on this.

http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/p/314856/1099690.aspx#1099690

Hi Alois,

Just to let you know that the problem is being analyzed and some update will be posted asap.

Anthony.

HI Anthony,

thats good news !

So you could reproduce the issue.

We did some futher invetigations here as well. We found a cache coherency issue in the  TransportSrio.c:TransportSrio_isr(). It seems that it  fixes the problem.

I dont know that state of your analyzes, just in the case it would be helpfull, I could provide you more information.

Alois

Besides the cache issues in the SRIO transport, there seems to be a bug in the NameServerMessageQ module:

In NameServerMessageQ_get(), the null characters for the names of the NameServer instance (msg->instanceName) and the NameServer entry (msg->name) are not copied into the message. This leads to random comparison results at the receiver in NameServerMessageQ_swiFxn().

Ralf

Hi Garrett,

I agree with your fix, but there are also some cache issues which occur with real-world applications:

The SRIO LLD expects cache coherent memory for the QMSS accumulator list. But the SRIO Transport allocates memory for the accumulator list from the default heap. In ohter words, it only works if the default heap gets placed in L2, which is a huge restriction in my opinion.
See also: http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/p/283507/997367.aspx

In addition, the SRIO transport doesn't handle cache properly. In TransportSrio_initRxDescBufs() and TransportSrio_isr(), the cache is not written back for the allocated message before it gets phushed into the QMSS queue.

I also think the cache operation should work on L2 instead of L1D. This would allow messages to be allocated in external memory.

Thanks,
Ralf

I'm wondering if someone is still working on the SRIO transport issues.

Ralf

Ralf,

Once it is analyzed updates will be posted regarding the issue you reported with cache.

A.

Hi all,

     I rebuild the IPC and then link the new build.

     But, 'Chip to Chip example' occur same problem (MessageQ_get() ---- return status -5).

     Did i miss something??.. or is there additional amendments??

regards, soon hwan jeong