This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

QMSS gets stuck and does not return descriptor to Tx completion queue

Hi,

Our DSP application makes intensive use of QMSS/PA to send packets to Ethernet and move data among DSP cores. At any given time, all 8 DSP cores can be sending Ethernet packets through PA or SW using PA queue, and moving data among cores through packet DMA using infrastructure queue. 

We noticed that sometimes after a buffer descriptor is pushed onto a transmission queue, the QMSS seems to get stuck, and never puts the buffer descriptor back into the Tx completion queue.

If we put a 10000 cycles delay after each pushing, then the problem goes away. And if we do a queue peek using Qmss_getQueueEntryCount(hnd) before each pushing, the problem also disappears. These observations all point to the possibility that there is a hardware bug in QMSS.

Another observation is that the Qmss_getQueueEntryCount(hnd) function call sometimes takes 8000 cycles to complete, even though most of the time it takes about 200 cycles. Our DSP runs at 1GHz. The Qmss_getQueueEntryCount(hnd) function call essentially is just reading Register A from the Queue Peek Region of the queue. 

Any idea what could be the cause of the problem, namely the QMSS may get stuck when there is intensive use of the queues?

Do you have a known issue that is related to the problem that we have,  If so, how do you suggest we get around this problem? 

Daniel

  • Daniel,

    Please have a look at the Mulitcore Navigator User's Guide.
    http://focus.ti.com/lit/ug/sprugr9c/sprugr9c.pdf
    This API gives, reading the queue’s status register until it returns a non-zero count.
    Provide more details to check this issue, How you are validating this is a hardware bug?
    This is not an known issue. You can see the release notes: C:\ti\pdk_C6678_1_1_2_6\docs

  • Hi Pubesh,

    I checked the Mulitcore Navigator User's Guide, http://focus.ti.com/lit/ug/sprugr9f/sprugr9f.pdf, but I could not find any reference to your statement: "This API gives, reading the queue’s status register until it returns a non-zero count."  Do you mean the API actually waits until it returns a non-zero count?  

    As I said in my previous post:

    1)I checked the code of this Qmss_getQueueEntryCount(hnd)  function, and it is essentially just reading a register.

    2) The problem went away after I applied some delay or did a queue entry count query, while nothing else in the software was changed. This makes me think it is a hardware issue. 

    Daniel

     

  • Just verified that with Qmss_getQueueEntryCount(hnd) before each sending, the QMSS sometimes still gets stuck, meaning the descriptor pushed onto the transmission queue is not returned to Tx  completion queue.

    The problem goes away with the delay of 10000 cycles though. So delay is the key. 

     

     

     

     

  • Daniel,

    Do you have an example project that you can share that exhibits this behavior? Preferably on an EVM. 

    I have seen similar symptoms before that were caused by the return queue number not being written all the way out to the descriptor because of cache. However, if your issue is solved with a delay then this seems like it is not the issue.

    Thanks,

    Jason Reeder 

  • Hi Jason,

    It may take me quite some time before I can come up with an example project. This problem is observed in the application running on our custom board, it happens when the DSP is highly loaded. 

    Each 10ms on each core, there are about 100 packets sent to Ethernet through PA/SW queue (648), and 125 packets sent to other DSP cores (18 packets per destination core) on the same chip through infrastructure queue and packet DMA.

    The sending is done in bursts, meaning every 10ms all the pushes(100 ethernet + 125 on-chip) happen almost one immediately after another, followed by a period of nothing.

    If we reduce the DSP load by half, then the problem is not observed. It may still happen, but we just did not observe it during the tests.

    I don't believe it is cache related. At one point of time, I actually put the descriptors in a region that is not cacheable, but it did not solve the problem.

    Daniel

  • Daniel,

    For your RX flow configuration what do you have the RX_ERROR_HANDLING bit set to? If you have this bit set to 1 then any starvation errors at your RX free descriptor queue will cause the packet transfer to be retried after a TIMEOUT period has elapsed. This can cause the descriptor to be popped from your TX queue by the infrastructure PKTDMA but not returned to the TX free descriptor queue. The packet will also not arrive at the destined RX queue. 

    See the image below for the series of steps that take place during an infrastructure transfer. If there are no packets available in the RX Free Descriptor Queue at step 4 then starvation will occur and the packet popped during step 3 will either be recycled back to the TX Free Descriptor Queue (RX_ERROR_HANDLING = 0) or the packet will be held during the TIMEOUT period until the RX Free Descriptor Queue is checked once again for a descriptor (RX_ERROR_HANDLING = 1). When you get stuck in your error state can you check the number of descriptors on each of the 4 queues below (this can be done through register reads in the memory browser view in CCS)? This should give us an idea of what is going wrong. It could be the case that there is some sort of condition where the Consumer Core is not returning descriptors back to the RX Free Desc Queue fast enough and is causing a starvation to occur.

    1) The producer core has data to transfer to another core so it pops a free descriptor from the TX Free Descriptor Queue and attached the buffer

    2) The producer core then pushes that descriptor onto Infra PKTDMA TX Queue to begin the transfer

    3) The Infrastructure PKTDMA sees a descriptor on its TX queue so the TX Core pops the descriptor to start moving the data

    4) The TX Core notifies the RX Core that data is coming so the RX Core pops a descriptor from the RX Free Descriptor Queue specified by the RX Flow being used

    5) The Infrastructure PKTDMA transfers the data from the TX buffer to the RX buffer

    6) After the data transfer is completed the TX Core returns the descriptor that it popped back to queue specified by the return queue field in the descriptor. Also after the transfer is complete the RX Core pushes the descriptor with the newly filled buffer to the RX Queue that was specified by the RX Flow

    7) The Consumer Core pops a descriptor from the RX Queue and consumes the data

    8) Once the Consumer Core is finished with the data then the descriptor is returned to the RX Free Descriptor Queue 

  • Hi Jason,

    Thanks for the explanation.

    I checked our code, and the RX_ERROR_HANDLING bit is set to 0. So it is probably not the cause of the problem.

    And we have configured a lot of buffers and buffer-descriptors in the Rx free queue (3071) of each core, and in this particular test it is impossible for them to be exhausted.

    Daniel

  • Daniel,

    Can you share your code where you initialize the descriptors and their buffers? As well as your infrastructure PKTDMA channel and RX Flow configurations?

    Jason Reeder

  • Hi Jason,

    Here is the code.


    /** ============================================================================
     *   @n@b Init_Qmss
     *
     *   @b Description
     *   @n This API initializes the QMSS LLD on core 0 only.
     *
     *   @param[in]  
     *   @n None
     * 
     *   @return    Int32
     *              -1      -   Error
     *              0       -   Success
     * =============================================================================
     */
    Int32 Init_Qmss (Void)
    {
        Int32                       result;
        Qmss_MemRegInfo             memCfg;
        Qmss_InitCfg                qmssInitConfig;
        Cppi_DescCfg                cppiDescCfg;
        UInt32                      numAllocated;
    
        // Initialize QMSS
        memset (&qmssInitConfig, 0, sizeof (Qmss_InitCfg));
    
        // Set up QMSS configuration
    
        // Use internal linking RAM
        qmssInitConfig.linkingRAM0Base  =   0;   
        qmssInitConfig.linkingRAM0Size  =   0;
        qmssInitConfig.linkingRAM1Base  =   (uint32_t)gExtLinkRam;
        qmssInitConfig.maxDescNum       =   NUM_HOST_DESC;
        
        qmssInitConfig.pdspFirmware[0].pdspId = Qmss_PdspId_PDSP1;
    #ifdef _LITTLE_ENDIAN    
        qmssInitConfig.pdspFirmware[0].firmware = (void *) &acc48_le;
        qmssInitConfig.pdspFirmware[0].size = sizeof (acc48_le);
    #else
        qmssInitConfig.pdspFirmware[0].firmware = (void *) &acc48_be;
        qmssInitConfig.pdspFirmware[0].size = sizeof (acc48_be);
    #endif    
    
        // Initialize the Queue Manager
        result = Qmss_init (&qmssInitConfig, &qmssGblCfgParams);
        if (result != QMSS_SOK)
        {
            #ifdef EVM
            System_printf ("Error initializing Queue Manager SubSystem, Error code : %d\n", result);
    		#endif
            return -1;
        }
    
        // Start Queue manager on this core 
        Qmss_start ();
    
        // Setup the descriptor memory regions. 
        //
        // The Descriptor base addresses MUST be global addresses and
        // all memory regions MUST be setup in ascending order of the
        // descriptor base addresses.
        //
    
        // Initialize and setup CPSW Host Descriptors required for example
        memset (gHostDesc, 0, SIZE_HOST_DESC * NUM_HOST_DESC);
        memCfg.descBase             =   (UInt32 *) gHostDesc;
        memCfg.descSize             =   SIZE_HOST_DESC;
        memCfg.descNum              =   NUM_HOST_DESC;
        memCfg.manageDescFlag       =   Qmss_ManageDesc_MANAGE_DESCRIPTOR;
        memCfg.memRegion            =   Qmss_MemRegion_MEMORY_REGION0;
        memCfg.startIndex           =   0;
    
        // Insert Host Descriptor memory region 
        result = Qmss_insertMemoryRegion(&memCfg);
        if (result == QMSS_MEMREGION_ALREADY_INITIALIZED)
        {
            #ifdef EVM
            System_printf ("Memory Region %d already Initialized \n", memCfg.memRegion);
    		#endif
        }
        else if (result < QMSS_SOK)
        {
            #ifdef EVM
            System_printf ("Error: Inserting memory region %d, Error code : %d\n", memCfg.memRegion, result);
    		#endif
            return -1;
        }    
    
        // Initialize all the descriptors we just allocated on the
        // memory region above. Setup the descriptors with some well
        // known values before we use them for data transfers.
        //
        memset (&cppiDescCfg, 0, sizeof (cppiDescCfg));
        cppiDescCfg.memRegion       =   Qmss_MemRegion_MEMORY_REGION0;
        cppiDescCfg.descNum         =   NUM_HOST_DESC;
        cppiDescCfg.destQueueNum    =   QMSS_PARAM_NOT_SPECIFIED;     
        cppiDescCfg.queueType       =   Qmss_QueueType_GENERAL_PURPOSE_QUEUE;
        cppiDescCfg.initDesc        =   Cppi_InitDesc_INIT_DESCRIPTOR;
        cppiDescCfg.descType        =   Cppi_DescType_HOST;
        
        // By default:
        //      (1) Return descriptors to tail of queue 
        //      (2) Always return entire packet to this free queue
        //      (3) Set that PS Data is always present in start of SOP buffer
        //      (4) Configure free q num < 4K, hence qMgr = 0
        //      (5) Recycle back to the same Free queue by default.
        //
        cppiDescCfg.returnPushPolicy            =   Qmss_Location_TAIL;    
        cppiDescCfg.cfg.host.returnPolicy       =   Cppi_ReturnPolicy_RETURN_ENTIRE_PACKET;    
        cppiDescCfg.cfg.host.psLocation         =   Cppi_PSLoc_PS_IN_DESC;         
        cppiDescCfg.returnQueue.qMgr            =   0;    
        cppiDescCfg.returnQueue.qNum            =   QMSS_PARAM_NOT_SPECIFIED; 
        cppiDescCfg.epibPresent                 =   Cppi_EPIB_EPIB_PRESENT;
        
        // Initialize the descriptors, create a free queue and push descriptors to a global free queue 
        if ((gGlobalFreeQHnd = Cppi_initDescriptor (&cppiDescCfg, &numAllocated)) <= 0)
        {
            #ifdef EVM
            System_printf ("Error Initializing Free Descriptors, Error: %d \n", gGlobalFreeQHnd);
    		#endif
            return -1;
        }
        else
        {
            #ifdef EVM
            System_printf ("Initializing Free Descriptors. %x \n",&gHostDesc[0]);
    		#endif
        }        
    
        // Need to invalidate and writeback from the cache all descriptors that will be used by cores other than zero
        // hence this is NUM_HOST_DESC_PER_CORE*(NUM_CORES-1) buffers of size SIZE_HOST_DESC
        // starting at location NUM_HOST_DESC_PER_CORE*SIZE_HOST_DESC for core 1 from start of gHostDesc
        //SYS_CACHE_INV ((void *)&gHostDesc[SIZE_HOST_DESC*NUM_HOST_DESC_PER_CORE], NUM_HOST_DESC_PER_CORE*(NUM_CORES-1)*SIZE_HOST_DESC, CACHE_FENCE_WAIT);
        SYS_CACHE_WBINV ((void *)gHostDesc, sizeof(gHostDesc), CACHE_FENCE_WAIT);
       
        // Queue Manager Initialization Done 
        return 0;
    }
    
    
    /** ============================================================================
     *   @n@b Init_Qmss_Local
     *
     *   @b Description
     *   @n This API initializes the QMSS LLD in cores other than core 0.
     *
     *   @param[in]  
     *   @n None
     * 
     *   @return    Int32
     *              -1      -   Error
     *              0       -   Success
     * =============================================================================
     */
    Int32 Init_Qmss_Local (Void)
    {
      Int32            result;
    
      while(1)
      {
          // Block until Qmss_init() has completed by core 0 
          result = Qmss_start();
          if(result == QMSS_NOT_INITIALIZED)
          {
    		  #ifdef EVM
              System_printf ("QMSS Not yet Initialized\n");
    		  #endif
              continue;
          }
          else if (result != QMSS_SOK)  {
            #ifdef EVM
            System_printf ("Qmss_start failed with error code %d\n", result);
    		#endif
            return (-1);
          }
    
          if (result == QMSS_SOK) 
          {
              break;
          }
      }
    
      return 0;
    }
    
    
    /** ============================================================================
     *   @n@b Init_Cppi
     *
     *   @b Description
     *   @n This API initializes the CPPI LLD, opens the PASS CPDMA and opens up
     *      the Tx, Rx channels required for data transfers.
     *
     *   @param[in]  
     *   @n None
     * 
     *   @return    Int32
     *              -1      -   Error
     *              0       -   Success
     * =============================================================================
     */
    Int32 Init_Cppi (Void)
    {
        Int32                       result, i;        
        Cppi_CpDmaInitCfg           cpdmaCfg;
        UInt8                       isAllocated;        
        Cppi_TxChInitCfg            txChCfg;
        Cppi_RxChInitCfg            rxChInitCfg;
    
        // Initialize CPPI LLD
        result = Cppi_init (&cppiGblCfgParams);
        if (result != CPPI_SOK)
        {
            #ifdef EVM
            System_printf ("Error initializing CPPI LLD, Error code : %d\n", result);
    		#endif
            return -1;
        }
    
        // Initialize PASS CPDMA
        memset (&cpdmaCfg, 0, sizeof (Cppi_CpDmaInitCfg));
        cpdmaCfg.dmaNum     = Cppi_CpDma_PASS_CPDMA;
        if ((gCpdmaHnd = Cppi_open (&cpdmaCfg)) == NULL)
        {
            #ifdef EVM
            System_printf ("Error initializing CPPI for PASS CPDMA %d \n", cpdmaCfg.dmaNum);
    		#endif
            return -1;
        }    
    
        // Open all CPPI Tx Channels. These will be used to send data to PASS/CPSW             
        for (i = 0; i < NUM_PA_TX_QUEUES; i ++)
        {
            txChCfg.channelNum      =   i;       // CPPI channels are mapped one-one to the PA Tx queues
            txChCfg.txEnable        =   Cppi_ChState_CHANNEL_DISABLE;  // Disable the channel for now.
            txChCfg.filterEPIB      =   0;
            txChCfg.filterPS        =   0;
            txChCfg.aifMonoMode     =   0;
            txChCfg.priority        =   2;
            if ((gCpdmaTxChanHnd[i] = Cppi_txChannelOpen (gCpdmaHnd, &txChCfg, &isAllocated)) == NULL)
            {
    	        #ifdef EVM
                System_printf ("Error opening Tx channel %d\n", txChCfg.channelNum);
    			#endif
                return -1;
            }
    
            Cppi_channelEnable (gCpdmaTxChanHnd[i]);
        }
    
        // Open all CPPI Rx channels. These will be used by PA to stream data out.
        for (i = 0; i < NUM_PA_RX_CHANNELS; i++)
        {
            // Open a CPPI Rx channel that will be used by PA to stream data out.
            rxChInitCfg.channelNum  =   i; 
            rxChInitCfg.rxEnable    =   Cppi_ChState_CHANNEL_DISABLE; 
            if ((gCpdmaRxChanHnd[i] = Cppi_rxChannelOpen (gCpdmaHnd, &rxChInitCfg, &isAllocated)) == NULL)
            {
    	        #ifdef EVM
                System_printf ("Error opening Rx channel: %d \n", rxChInitCfg.channelNum);
    			#endif
                return -1;
            }
    
            // Also enable Rx Channel
            Cppi_channelEnable (gCpdmaRxChanHnd[i]);    
        }
        
        // Clear CPPI Loobpack bit in PASS CDMA Global Emulation Control Register
        Cppi_setCpdmaLoopback(gCpdmaHnd, 0);   
    
        // Initialize QMSS CPDMA
        memset (&cpdmaCfg, 0, sizeof (Cppi_CpDmaInitCfg));
        cpdmaCfg.dmaNum     = Cppi_CpDma_QMSS_CPDMA;
        if ((gQmssCpdmaHnd = Cppi_open (&cpdmaCfg)) == NULL)
        {
            #ifdef EVM
            System_printf ("Error initializing CPPI for QMSS CPDMA %d \n", cpdmaCfg.dmaNum);
    		#endif
            return -1;
        }    
    
        // Open all CPPI Tx Channels. These will be used to send data to QMSS             
        for (i = 0; i < NUM_CORES; i ++)
        {
            txChCfg.channelNum      =   i;       // CPPI channels are mapped one-one to the PA Tx queues
            txChCfg.txEnable        =   Cppi_ChState_CHANNEL_DISABLE;  // Disable the channel for now.
            txChCfg.filterEPIB      =   0;
            txChCfg.filterPS        =   0;
            txChCfg.aifMonoMode     =   0;
            txChCfg.priority        =   2;
            if ((gQmssCpdmaTxChanHnd[i] = Cppi_txChannelOpen (gQmssCpdmaHnd, &txChCfg, &isAllocated)) == NULL)
            {
    	        #ifdef EVM
                System_printf ("Error opening Tx channel %d\n", txChCfg.channelNum);
    			#endif
                return -1;
            }
    
            Cppi_channelEnable (gQmssCpdmaTxChanHnd[i]);
        }
    
        // Open all CPPI Rx channels. These will be used by QMSS to stream data out.
        for (i = 0; i < NUM_QMSS_RX_CHANNELS; i++)
        {
            // Open a CPPI Rx channel that will be used by QMSS to stream data out.
            rxChInitCfg.channelNum  =   i; 
            rxChInitCfg.rxEnable    =   Cppi_ChState_CHANNEL_DISABLE; 
            if ((gQmssCpdmaRxChanHnd[i] = Cppi_rxChannelOpen (gQmssCpdmaHnd, &rxChInitCfg, &isAllocated)) == NULL)
            {
    	        #ifdef EVM
                System_printf ("Error opening Rx channel: %d \n", rxChInitCfg.channelNum);
    			#endif
                return -1;
            }
    
            // Also enable Rx Channel
            Cppi_channelEnable (gQmssCpdmaRxChanHnd[i]);    
        }
        
        // Clear CPPI Loobpack bit in PASS CDMA Global Emulation Control Register
        Cppi_setCpdmaLoopback(gQmssCpdmaHnd, 1);   
    
        // CPPI Init Done. Return success
        return 0;
    }    
    
    /** ============================================================================
     *   @n@b Setup_Tx
     *
     *   @b Description
     *   @n This API sets up all relevant data structures and configuration required
     *      for sending data to PASS/Ethernet. It sets up a Tx free descriptor queue,
     *      PASS Tx queues required for send.
     *
     *   @param[in]  
     *   @n None
     * 
     *   @return    Int32
     *              -1      -   Error
     *              0       -   Success
     * =============================================================================
     */
    Int32 Setup_Tx (Void)
    {
        UInt8                       isAllocated, i;        
        Qmss_Queue                  qInfo;
        Ptr                   		pCppiDesc;
    
        if (g_uiCoreNum == 0)
        {
    	    // Open all Transmit (Tx) queues. 
    	    //
    	    // These queues are used to send data to PA PDSP/CPSW.
    	    //
    	    for (i = 0; i < NUM_PA_TX_QUEUES; i ++)
    	    {
                
    	        if ((gPaTxQHnd[i] = Qmss_queueOpen (Qmss_QueueType_PASS_QUEUE, QMSS_PARAM_NOT_SPECIFIED, &isAllocated)) < 0)
    	        {
    		        #ifdef EVM
    	            System_printf ("Error opening PA Tx queue \n");
    				#endif
    	            return -1;
    	        }            
    	    }
        
    	    SYS_CACHE_WB ((void *)gPaTxQHnd, 128, CACHE_WAIT);
        
    	    // Open all infrastructure core to core Transmit (Tx) queues. 
    	    //
    	    // These queues are used to send data core-to-core.
    	    //
    	    for (i = 0; i < NUM_CORES; i ++)
    	    {
                
    	        if ((gQmssTxQHnd[i] = Qmss_queueOpen (Qmss_QueueType_INFRASTRUCTURE_QUEUE, QMSS_INFRASTRUCTURE_QUEUE_BASE + i, &isAllocated)) < 0)
    	        {
    				#ifdef EVM
    	            System_printf ("Error opening QMSS Tx queue %d\n",i);
    				#endif
    	            return -1;
    	        }            
    	    }
        
    	    SYS_CACHE_WB ((void *)gQmssTxQHnd, sizeof(gQmssTxQHnd), CACHE_WAIT);
    	}
    
        // Open a Tx Free Descriptor Queue (Tx FDQ). 
        //
        // This queue will be used to hold Tx free decriptors that can be filled
        // later with data buffers for transmission onto wire.
        //
        if ((gTxFreeQHnd[g_uiCoreNum] = Qmss_queueOpen (Qmss_QueueType_STARVATION_COUNTER_QUEUE, TX_FREE_QUEUE_NUM_INIT+g_uiCoreNum, &isAllocated)) < 0)
        {
            #ifdef EVM
            System_printf ("Error opening Tx Free descriptor queue \n");
    		#endif
            return -1;
        }    
        
        SYS_CACHE_WB ((void *)&gTxFreeQHnd[g_uiCoreNum], sizeof(Qmss_QueueHnd), CACHE_WAIT);
                
    
        qInfo = Qmss_getQueueNumber (gTxFreeQHnd[g_uiCoreNum]);
    	#ifdef EVM
        System_printf ("Tx Free descriptor queue = %d\n",gTxFreeQHnd[g_uiCoreNum]);
    	#endif
    
        // Attach some free descriptors to the Tx free queue we just opened.
        for (i = 0; i < NUM_TX_DESC; i++)
        {
            // Get a free descriptor from the global free queue we setup 
            // during initialization.
            //
            if ((pCppiDesc = Qmss_queuePop (gGlobalFreeQHnd)) == NULL)
            {
                break;                
            }
    
            // The descriptor address returned from the hardware has the 
            // descriptor size appended to the address in the last 4 bits.
            //
            // To get the true descriptor size, always mask off the last 
            // 4 bits of the address.
            //
            pCppiDesc = (Ptr) ((UInt32) pCppiDesc & 0xFFFFFFF0);
    
            // Populate the Tx free descriptor with the buffer.
            Cppi_setData (Cppi_DescType_HOST, pCppiDesc, (Uint8 *)(&cppiMemTX[g_uiCoreNum][i]), TX_BUF_SIZE);
    
            // Save original buffer information
            Cppi_setOriginalBufInfo (Cppi_DescType_HOST, pCppiDesc, (Uint8 *)(&cppiMemTX[g_uiCoreNum][i]), TX_BUF_SIZE);
    
            // Setup the Completion queue:
            //
            // Setup the return policy for this desc to return to the free q we just
            // setup instead of the global free queue.
            //
            Cppi_setReturnQueue ((Cppi_DescType) Cppi_DescType_HOST, pCppiDesc, qInfo);
    
            Cppi_setPacketLen    (Cppi_DescType_HOST, pCppiDesc, TX_BUF_SIZE);
            
            SYS_CACHE_WB (pCppiDesc, SIZE_HOST_DESC, CACHE_FENCE_WAIT);
    
            // Push descriptor to Tx free queue
            Qmss_queuePushDescSize (gTxFreeQHnd[g_uiCoreNum], pCppiDesc, SIZE_HOST_DESC);           
        }
        if (i != NUM_TX_DESC)
        {
            #ifdef EVM
            System_printf ("Error allocating Tx free descriptors \n");
    		#endif
            return -1;
        }
    
        // All done with Tx configuration. Return success.
        return 0;
    }
    
    /** ============================================================================
     *   @n@b Setup_Rx
     *
     *   @b Description
     *   @n This API sets up all relevant data structures and configuration required
     *      for receiving data from PASS/Ethernet. It sets up a Rx free descriptor queue
     *      with some empty pre-allocated buffers to receive data, and an Rx queue
     *      to which the Rxed data is streamed for the example application. 
     *
     *   @param[in]  
     *   @n None
     * 
     *   @return    Int32
     *              -1      -   Error
     *              0       -   Success
     * =============================================================================
     */
    Int32 Setup_Rx (Void)
    {
        UInt8                       isAllocated;        
        UInt32						i;
        Qmss_Queue                  rxFreeQInfo, rxQInfo;
        Ptr                   		pCppiDesc;
        Cppi_RxFlowCfg              rxFlowCfg;
        Ptr                         pDataBuffer;
        Uint32                      mySWInfo[] = {0x11112222, 0x33334444};
    
        // Open a Receive (Rx) queue. 
        //
        // This queue will be used to hold all the packets received by PASS/CPSW
        //
        //
        if ((gRxQHnd[g_uiCoreNum] = Qmss_queueOpen (Qmss_QueueType_GENERAL_PURPOSE_QUEUE, RX_QUEUE_NUM_INIT+g_uiCoreNum, &isAllocated)) < 0)
        {
            #ifdef EVM
            System_printf ("Error opening gRxQHnd queue \n");
    		#endif
            return -1;
        }            
        rxQInfo = Qmss_getQueueNumber (gRxQHnd[g_uiCoreNum]);
    
    	// Disable Interrupts 
    	//key = Hwi_disable();
    
    	// Cleanup the prefetch buffer also. 
    	//CSL_XMC_invalidatePrefetchBuffer();
        SYS_CACHE_WB ((void *)gRxQHnd, 128, CACHE_WAIT);
    
    	// Reenable Interrupts. 
    	//Hwi_restore(key);
    
        // The following RX queues are not shared between cores
    
        // Open a Rx Free Descriptor Queue for command port traffic per core (Rx FDQ). 
        //
        // This queue will hold all the Rx free decriptors for command port traffic. 
        // These descriptors will be used by the PASS CPDMA to hold data received via CPSW.
        //
        if ((gRxCommandFreeQHnd[g_uiCoreNum] = Qmss_queueOpen (Qmss_QueueType_STARVATION_COUNTER_QUEUE, RX_FREE_COMMAND_QUEUE_NUM_INIT+g_uiCoreNum, &isAllocated)) < 0)
        {
            #ifdef EVM
            System_printf ("Error opening Rx Free descriptor queue \n");
    		#endif
            return -1;
        }            
            
        SYS_CACHE_WB ((void *)gRxCommandFreeQHnd, 128, CACHE_WAIT);
            
        rxFreeQInfo = Qmss_getQueueNumber (gRxCommandFreeQHnd[g_uiCoreNum]);
    	#ifdef EVM
    	System_printf ("Rx Free command descriptor queue = %d\n",gRxCommandFreeQHnd[g_uiCoreNum]);
    	#endif
    
        // Attach some free descriptors to the Rx free queue we just opened.
        for (i = 0; i < NUM_RX_COMMAND_DESC; i++)
        {
            // Get a free descriptor from the global free queue we setup 
            // during initialization.
            //
            if ((pCppiDesc = Qmss_queuePop (gGlobalFreeQHnd)) == NULL)
            {
    	        #ifdef EVM
                System_printf ("Error popping descriptor from Global Free Q = %d.\n", gGlobalFreeQHnd);
    			#endif
                break;                
            }
    
            // The descriptor address returned from the hardware has the 
            // descriptor size appended to the address in the last 4 bits.
            //
            // To get the true descriptor size, always mask off the last 
            // 4 bits of the address.
            //
            pCppiDesc = (Ptr) ((UInt32) pCppiDesc & 0xFFFFFFF0);
                
            pDataBuffer = (Uint8 *)(&cppiMemRXCommand[g_uiCoreNum][i]);
            // Populate the Rx free descriptor with the buffer we just allocated.
            Cppi_setData (Cppi_DescType_HOST, pCppiDesc, (UInt8 *)pDataBuffer, RX_BUF_SIZE*2);
    
            // Save original buffer information 
            Cppi_setOriginalBufInfo (Cppi_DescType_HOST, pCppiDesc, (UInt8 *)pDataBuffer, RX_BUF_SIZE*2);
                
            // Setup the Completion queue:
            //
            // Setup the return policy for this desc to return to the free q we just
            // setup instead of the global free queue.
            //
            Cppi_setReturnQueue (Cppi_DescType_HOST, pCppiDesc, rxFreeQInfo);
    
            Cppi_setSoftwareInfo (Cppi_DescType_HOST, pCppiDesc, (UInt8 *) mySWInfo);
    
            Cppi_setPacketLen    (Cppi_DescType_HOST, pCppiDesc, RX_BUF_SIZE*2);
                
            SYS_CACHE_WB (pCppiDesc, SIZE_HOST_DESC, CACHE_FENCE_WAIT);
                
            // Push descriptor to Rx free queue 
            Qmss_queuePushDescSize (gRxCommandFreeQHnd[g_uiCoreNum], pCppiDesc, SIZE_HOST_DESC);           
        }        
        if (i != NUM_RX_COMMAND_DESC)
        {
            #ifdef EVM
            System_printf ("Error allocating Rx Command free descriptors \n");
    		#endif
            return -1;
        }
    
        // Open a Rx Free Descriptor Queue for ethernet port traffic per core (Rx FDQ). 
        //
        // This queue will hold all the Rx free decriptors for command port traffic. 
        // These descriptors will be used by the PASS CPDMA to hold data received via CPSW.
        //
        if ((gRxEthernetFreeQHnd[g_uiCoreNum] = Qmss_queueOpen (Qmss_QueueType_STARVATION_COUNTER_QUEUE, RX_FREE_ETHERNET_QUEUE_NUM_INIT+g_uiCoreNum, &isAllocated)) < 0)
        {
            #ifdef EVM
            System_printf ("Error opening Rx Free descriptor queue \n");
    		#endif
            return -1;
        }            
            
        SYS_CACHE_WB ((void *)gRxEthernetFreeQHnd, 128, CACHE_WAIT);
            
        rxFreeQInfo = Qmss_getQueueNumber (gRxEthernetFreeQHnd[g_uiCoreNum]);
    	#ifdef EVM
        System_printf ("Rx Free ethernet descriptor queue = %d\n",gRxEthernetFreeQHnd[g_uiCoreNum]);
    	#endif
    
        // Attach some free descriptors to the Rx free queue we just opened. 
        for (i = 0; i < NUM_RX_ETHERNET_DESC; i++)
        {
            // Get a free descriptor from the global free queue we setup 
            // during initialization.
            //
            if ((pCppiDesc = Qmss_queuePop (gGlobalFreeQHnd)) == NULL)
            {
    	        #ifdef EVM
                System_printf ("Error popping descriptor from global Free Q.\n");
    			#endif
                break;                
            }
    
            // The descriptor address returned from the hardware has the 
            // descriptor size appended to the address in the last 4 bits.
            //
            // To get the true descriptor size, always mask off the last 
            // 4 bits of the address.
            //
            pCppiDesc = (Ptr) ((UInt32) pCppiDesc & 0xFFFFFFF0);
                
            pDataBuffer = ((Uint8 *)(&cppiMemRXEthernet[g_uiCoreNum][i]) + 12);
            // Populate the Rx free descriptor with the buffer we just allocated. 
            // This updates the packet descriptor word 3 with length and word 4 with data buffer ptr
            Cppi_setData (Cppi_DescType_HOST, pCppiDesc, (UInt8 *)pDataBuffer, RX_BUF_SIZE-12);
    
            // Save original buffer information 
            // This updates the packet descriptor word 6 with length and word 7 with data buffer ptr
            Cppi_setOriginalBufInfo (Cppi_DescType_HOST, pCppiDesc, (UInt8 *)pDataBuffer, RX_BUF_SIZE-12);
                
            // Setup the Completion queue:
            //
            // Setup the return policy for this desc to return to the free q we just
            // setup instead of the global free queue.
            //
            Cppi_setReturnQueue (Cppi_DescType_HOST, pCppiDesc, rxFreeQInfo);
    
            Cppi_setSoftwareInfo (Cppi_DescType_HOST, pCppiDesc, (UInt8 *) mySWInfo);
    
            // Set the packet length in the packet descriptor word 0
            Cppi_setPacketLen    (Cppi_DescType_HOST, pCppiDesc, RX_BUF_SIZE);
                
            SYS_CACHE_WB (pCppiDesc, SIZE_HOST_DESC, CACHE_FENCE_WAIT);
                
            // Push descriptor to Rx free queue 
            Qmss_queuePushDescSize (gRxEthernetFreeQHnd[g_uiCoreNum], pCppiDesc, SIZE_HOST_DESC);           
        }        
        if (i != NUM_RX_ETHERNET_DESC)
        {
            #ifdef EVM
            System_printf ("Error allocating Rx Ethernet free descriptors \n");
    		#endif
            return -1;
        }
        
        // Setup command Rx Flow on each core. The only difference among the cores is the rxQInfo.
        //
        // A Rx flow encapsulates all relevant data properties that CPDMA would
        // have to know in order to successfully receive data.
        //
        // Initialize the flow configuration 
        memset (&rxFlowCfg, 0, sizeof(Cppi_RxFlowCfg));
        rxFreeQInfo = Qmss_getQueueNumber (gRxCommandFreeQHnd[g_uiCoreNum]);
    	#ifdef EVM
        System_printf ("Rx Free command descriptor queue = %d\n",gRxCommandFreeQHnd[g_uiCoreNum]);
    	#endif
    
        // Let CPPI pick the next available flow 
        rxFlowCfg.flowIdNum             =   CPPI_PARAM_NOT_SPECIFIED;    
    
        rxFlowCfg.rx_dest_qmgr          =   rxQInfo.qMgr;    
        rxFlowCfg.rx_dest_qnum          =   rxQInfo.qNum;  
        rxFlowCfg.rx_desc_type          =   Cppi_DescType_HOST; 
    
        rxFlowCfg.rx_sop_offset         =   0;
        rxFlowCfg.rx_ps_location        =   Cppi_PSLoc_PS_IN_DESC;  
        rxFlowCfg.rx_psinfo_present     =   0;    // Disable PS info
        
        rxFlowCfg.rx_error_handling     =   0;    // Drop the packet, do not retry on starvation by default        
        rxFlowCfg.rx_einfo_present      =   1;    // EPIB info present        
        
        rxFlowCfg.rx_dest_tag_lo_sel    =   0;    // Disable tagging 
        rxFlowCfg.rx_dest_tag_hi_sel    =   0;    
        rxFlowCfg.rx_src_tag_lo_sel     =   0;    
        rxFlowCfg.rx_src_tag_hi_sel     =   0;    
    
        rxFlowCfg.rx_size_thresh0_en    =   0;    // By default, we disable Rx Thresholds 
        rxFlowCfg.rx_size_thresh1_en    =   0;    // By default, we disable Rx Thresholds 
        rxFlowCfg.rx_size_thresh2_en    =   0;    // By default, we disable Rx Thresholds 
        rxFlowCfg.rx_size_thresh0       =   0x0;
        rxFlowCfg.rx_size_thresh1       =   0x0;
        rxFlowCfg.rx_size_thresh2       =   0x0;
    
        rxFlowCfg.rx_fdq0_sz0_qmgr      =   rxFreeQInfo.qMgr; // Setup the Receive free queue for the flow 
        rxFlowCfg.rx_fdq0_sz0_qnum      =   rxFreeQInfo.qNum;    
        rxFlowCfg.rx_fdq0_sz1_qnum      =   0x0; 
        rxFlowCfg.rx_fdq0_sz1_qmgr      =   0x0;
        rxFlowCfg.rx_fdq0_sz2_qnum      =   0x0;
        rxFlowCfg.rx_fdq0_sz2_qmgr      =   0x0;
        rxFlowCfg.rx_fdq0_sz3_qnum      =   0x0;
        rxFlowCfg.rx_fdq0_sz3_qmgr      =   0x0;
    
        rxFlowCfg.rx_fdq1_qnum          =   rxFreeQInfo.qNum;  // Use the Rx Queue to pick descriptors 
        rxFlowCfg.rx_fdq1_qmgr          =   rxFreeQInfo.qMgr;
        rxFlowCfg.rx_fdq2_qnum          =   rxFreeQInfo.qNum;  // Use the Rx Queue to pick descriptors 
        rxFlowCfg.rx_fdq2_qmgr          =   rxFreeQInfo.qMgr;
        rxFlowCfg.rx_fdq3_qnum          =   rxFreeQInfo.qNum;  // Use the Rx Queue to pick descriptors 
        rxFlowCfg.rx_fdq3_qmgr          =   rxFreeQInfo.qMgr;
    
        // Configure the Rx Command flow 
        if ((gRxCommandFlowHnd[g_uiCoreNum] = Cppi_configureRxFlow (gCpdmaHnd, &rxFlowCfg, &isAllocated)) == NULL)
        {
            #ifdef EVM
            System_printf ("Error configuring command Rx flow \n");
    		#endif
            return -1;
        }
    
        SYS_CACHE_WB ((void *)gRxCommandFlowHnd, 128, CACHE_FENCE_WAIT);
    
        // Configure the Rx Command flow for QMSS
        if ((gRxQmssCommandFlowHnd = Cppi_configureRxFlow (gQmssCpdmaHnd, &rxFlowCfg, &isAllocated)) == NULL)
        {
            #ifdef EVM
            System_printf ("Error configuring command Qmss Rx flow \n");
    		#endif
            return -1;
        }
    
        // Setup Ethernet Rx Flow on each core. The only difference among the cores is the rxQInfo.
        //
        // A Rx flow encapsulates all relevant data properties that CPDMA would
        // have to know in order to succefully receive data.
        //
        // Initialize the flow configuration 
        memset (&rxFlowCfg, 0, sizeof(Cppi_RxFlowCfg));
        rxFreeQInfo = Qmss_getQueueNumber (gRxEthernetFreeQHnd[g_uiCoreNum]);
    	#ifdef EVM
        System_printf ("Rx Free ethernet descriptor queue = %d\n",gRxEthernetFreeQHnd[g_uiCoreNum]);
    	#endif
    
        // Let CPPI pick the next available flow 
        rxFlowCfg.flowIdNum             =   CPPI_PARAM_NOT_SPECIFIED;    
    
        rxFlowCfg.rx_dest_qmgr          =   rxQInfo.qMgr;    
        rxFlowCfg.rx_dest_qnum          =   rxQInfo.qNum;  
        rxFlowCfg.rx_desc_type          =   Cppi_DescType_HOST; 
    
        rxFlowCfg.rx_sop_offset         =   0;
        rxFlowCfg.rx_ps_location        =   Cppi_PSLoc_PS_IN_DESC;  
        rxFlowCfg.rx_psinfo_present     =   0;    // Disable PS info
        
        rxFlowCfg.rx_error_handling     =   0;    // Drop the packet, do not retry on starvation by default        
        rxFlowCfg.rx_einfo_present      =   1;    // EPIB info present        
        
        rxFlowCfg.rx_dest_tag_lo_sel    =   0;    // Disable tagging 
        rxFlowCfg.rx_dest_tag_hi_sel    =   0;    
        rxFlowCfg.rx_src_tag_lo_sel     =   0;    
        rxFlowCfg.rx_src_tag_hi_sel     =   0;    
    
        rxFlowCfg.rx_size_thresh0_en    =   0;    // By default, we disable Rx Thresholds 
        rxFlowCfg.rx_size_thresh1_en    =   0;    // By default, we disable Rx Thresholds 
        rxFlowCfg.rx_size_thresh2_en    =   0;    // By default, we disable Rx Thresholds 
        rxFlowCfg.rx_size_thresh0       =   0x0;
        rxFlowCfg.rx_size_thresh1       =   0x0;
        rxFlowCfg.rx_size_thresh2       =   0x0;
    
        rxFlowCfg.rx_fdq0_sz0_qmgr      =   rxFreeQInfo.qMgr; // Setup the Receive free queue for the flow 
        rxFlowCfg.rx_fdq0_sz0_qnum      =   rxFreeQInfo.qNum;    
        rxFlowCfg.rx_fdq0_sz1_qnum      =   0x0; 
        rxFlowCfg.rx_fdq0_sz1_qmgr      =   0x0;
        rxFlowCfg.rx_fdq0_sz2_qnum      =   0x0;
        rxFlowCfg.rx_fdq0_sz2_qmgr      =   0x0;
        rxFlowCfg.rx_fdq0_sz3_qnum      =   0x0;
        rxFlowCfg.rx_fdq0_sz3_qmgr      =   0x0;
    
        rxFlowCfg.rx_fdq1_qnum          =   rxFreeQInfo.qNum;  // Use the Rx Queue to pick descriptors 
        rxFlowCfg.rx_fdq1_qmgr          =   rxFreeQInfo.qMgr;
        rxFlowCfg.rx_fdq2_qnum          =   rxFreeQInfo.qNum;  // Use the Rx Queue to pick descriptors 
        rxFlowCfg.rx_fdq2_qmgr          =   rxFreeQInfo.qMgr;
        rxFlowCfg.rx_fdq3_qnum          =   rxFreeQInfo.qNum;  // Use the Rx Queue to pick descriptors 
        rxFlowCfg.rx_fdq3_qmgr          =   rxFreeQInfo.qMgr;
    
        // Configure the Rx Ethernet flow 
        if ((gRxEthernetFlowHnd[g_uiCoreNum] = Cppi_configureRxFlow (gCpdmaHnd, &rxFlowCfg, &isAllocated)) == NULL)
        {
            #ifdef EVM
            System_printf ("Error configuring Ethernet Rx flow \n");
    		#endif
            return -1;
        }
    
        SYS_CACHE_WB ((void *)gRxEthernetFlowHnd, 128, CACHE_FENCE_WAIT);
    
        // All done with Rx configuration. Return success. 
        return 0;
    }
    
    
    
    

    Daniel

  • Daniel,

    Nothing jumps out at me in your initialization. If there's any way that you could create a project that replicates the error on an EVM it would be much easier for me to track down this issue.

    Jason Reeder