PCIe and NDK Integration

Saurabh Kumar Singh

Hi.

We are trying to send data, which is received by the board over PCIe, using UDP to a windows application. The problem is that at a time only one of the two communication links work, i.e the board is able to communicate over PCIe but data being sent to the windows application using UDP sub-module in NDK fails. Although the return value from the sendto() is correct, the data is not received by the windows appl nor is it visible in Wireshark tool. When these links, (PCIe and UDP Socket), were tested independently, there was no problem in communication in either of these two links. The problem seems to be occur only when both of these links are used in the same project.

The PCIe initialization and configuration is done as shown in the attached file Intc_Config.cpp. UDP initialization is done as per the example client_evm6678 given in NDK example folder.

The question we are having at this point is, are there any known issue/bugs when PCIe and NDK are used together in EVM6678?

5123.Intc_Config.cpp

over 12 years ago

0 Varada Bellary over 12 years ago

TI__Expert 5750 points

There is not apparent conflict in the operation of PCIe and NDK. I am going to ask the PCIe expert also to take a look. Couple of things :

Which PCIe interrupt are you using? I am assuming it is the PCI MSIe, and not PCI legacy.

EMAC uses Queue Manager and PktDMA. Network RX interrupt event is #48 . So group 1. This group 1 is mapped to VECT_ID =8, in the ECM, of client.cfg file.

PCIe MSI interrupt event is #17. So group 0. Are you enabling group 0 ? And where is it mapped to ?

I understand that you are integrating both PCIe and NDK in your program. In this integrated codebase, does UDP transmit alone work? Then does PCIe alone work ? This is in part to determine if there is some initialization/configuration issue.

When you mention, UDP Xmit does not work, though sendto() function returns okay, can you check for network statistics to see if network TX statistics are increasing.

Lastly – what is your boot mode of the EVM. Is it ‘no-boot’ ?

Hope this helps.

0 Saurabh Kumar Singh over 12 years ago in reply to Varada Bellary

Prodigy 235 points

Hello Varada,

Thanks for the response.

1. The PCIe interrupt we are using is PCI Legacy interrupt and not PCI MSIe. And though we are enabling group 0 in the .cfg file, the PCI legacy interrupt is event #63 and hence its mapped to group 1.

2. Yes, we are integrating both PCIe and NDK in our program. If the PCIe Initial configuration statements are not called (these functions are in the file Intc_Config.cpp attached in the first post) then UDP (Tx and Rx) works. And if the configuration statements are called PCIe works and UDP does not.

3. We will check the network statistics.

4. EVM is in "I2C boot mode"

0 Varada Bellary over 12 years ago in reply to Saurabh Kumar Singh

TI__Expert 5750 points

Hi Saurabh,

It seems to intc mapping conflict. In the Intc_Config.cpp, can you try mapping the legacy interrupt to “CSL_INTC_VECTID_8”, just so that all the group1 events are mapped to same vector.

Hope this resolves the issue. Below wiki page will clarify interrupt configuration.

http://processors.wiki.ti.com/index.php/Configuring_Interrupts_on_Keystone_Devices

Hope this resolves. Keep us posted.

0 Saurabh Kumar Singh over 12 years ago in reply to Varada Bellary

Prodigy 235 points

Varada Bellary said:
It seems to intc mapping conflict. In the Intc_Config.cpp, can you try mapping the legacy interrupt to “CSL_INTC_VECTID_8”, just so that all the group1 events are mapped to same vector.

We have made the change so that all group1 events are mapped to the same vector as suggested by you. But it did not solve the problem that we are facing. On more careful inspection, we notice that doing this actually makes no change in the behavior.

We tried to send the UDP message even before the PCIe initialization. The sending of message and reception of acknowledgement is successful. Messages sent after PCIe initialization are also successful, but the receiving of the messages still fails. Though the messages are visible in Wireshark, the network statistics does not reflect the same messages.

0 Varada Bellary over 12 years ago in reply to Saurabh Kumar Singh

TI__Expert 5750 points

In case of EMAC, there is no TX interrupt. There are only RX events that is due to RX data arriving in the high priority accumulation Qs. The fact that only RX not working after PCIe initialization, hints to maybe RX interrupt is not working.

We can check if the status register to see if interrupts are getting generated.

I have attached a gel file, that you can use to print interrupt details and status.

After you see that RX not happening, can you halt the DSP program. Then you could load the gel file and run the function “corepac_intc_qm_status”.

0131.c6678_Interrupts_v0.2.gel

0 Saurabh Kumar Singh over 12 years ago in reply to Varada Bellary

Prodigy 235 points

Hi Varada,

We tried using the GEL file supplied by you and running the function "qm_corepac_intc_status"

What are we supposed to look for in the output generated by the gel file? Could you kindly explain?

Please find below the output generated using the GEL file:

"C66xx_0: GEL Output: ***********************************************************************************************
C66xx_0: GEL Output: ********************************** QM COREPAC INTC INTERRUPTS MAPPING ***********************
C66xx_0: GEL Output: ***********************************************************************************************

C66xx_0: GEL Output: QM Interrupt for 0~31 Queues (QM_INT_LOW_0) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for 32~63 Queues (QM_INT_LOW_1) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for 64~95 Queues (QM_INT_LOW_2) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for 96~127 Queues (QM_INT_LOW_3) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for 128~159 Queues (QM_INT_LOW_4) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for 160~191 Queues (QM_INT_LOW_5) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for 192~223 Queues (QM_INT_LOW_6) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for 224~255 Queues (QM_INT_LOW_7) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for 256~287 Queues (QM_INT_LOW_8) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for 288~319 Queues (QM_INT_LOW_9) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for 320~351 Queues (QM_INT_LOW_10) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for 352~383 Queues (QM_INT_LOW_11) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for 384~415 Queues (QM_INT_LOW_12) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for 416~447 Queues (QM_INT_LOW_13) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for 448~479 Queues (QM_INT_LOW_14) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for 480~511 Queues (QM_INT_LOW_15) (Broadcast to all cores) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for Queue708 (QM_INT_HIGH_0) ---> EVT[1] ---> CORE0 CPU INTERRUPT VECTOR7
C66xx_0: GEL Output: QM Interrupt for Queue716 (QM_INT_HIGH_8) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for Queue724 (QM_INT_HIGH_16) ---> **MASKED**
C66xx_0: GEL Output: QM Interrupt for Queue732 (QM_INT_HIGH_24) ---> **MASKED** "

Also, could you tell us where we may find more information about the rx event that you mention? Is there a way we can check on the occurrence of these events and interrupts?

Thank you for all your help.

0 Varada Bellary over 12 years ago in reply to Saurabh Kumar Singh

TI__Expert 5750 points

I am trying to get advice from another expert.

The best way to come up to speed with EMAC and NetCP is through training material.

http://focus.ti.com/docs/training/catalog/events/event.jhtml?sku=OLT110027

0 Varada Bellary over 12 years ago in reply to Varada Bellary

TI__Expert 5750 points

Below is the advice from device expert .

I have seen some issues that if we re-initialize the INTC for multiple interrupt events, the later setup will prevent the previous setup working.

I think the “client.cfg” configures the INTC already for EMAC, but in “intc_config()” the PCIe interrupt setup re-initializes the INTC again.

Could you let the customer try to comment out the following sections in “intc_config()” (Intc_Config.cpp) to see if both PCIe and EMAC interrupts could be working please?

#if 0

/* INTC module initialization */

intcContext.eventhandlerRecord = EventHandler;

intcContext.numEvtEntries = 10;

if (CSL_intcInit(&intcContext) != CSL_SOK)

{

Dbg_printf("Error: GEM-INTC initialization failed\n");

return FALSE;

}

#endif

0 Saurabh Kumar Singh over 12 years ago in reply to Varada Bellary

Prodigy 235 points

Hi Varada,

Thank you for your suggestion but it did not resolve our issue. PCIe does not work if the specified line is commented.

We found that the specified function consists of the following lines of code:

"    Uint16       i;

    asm(" dint");

    if (pContext != NULL) {
        _CSL_intcEventOffsetMap = pContext->offsetResv;
        _CSL_intcAllocMask = pContext->eventAllocMask;
        _CSL_intcNumEvents = pContext->numEvtEntries;

        for (i = 0; i < ((CSL_INTC_EVENTID_CNT + 31) / 32); i++)
            _CSL_intcAllocMask[i] = 0;

        for (i = 0; i < CSL_INTC_EVENTID_CNT ; i++)
            _CSL_intcEventOffsetMap[i] = CSL_INTC_MAPPED_NONE;
        if (pContext->eventhandlerRecord != NULL) {
            _CSL_intcEventHandlerRecord = pContext->eventhandlerRecord;
            for (i = 0; i < _CSL_intcNumEvents; i++)
                _CSL_intcEventHandlerRecord[i].handler = CSL_INTC_EVTHANDLER_NONE;
        }
    }
    _CSL_intcIvpSet();
    _CSL_intcCpuIntrTable.nmiIsr = _CSL_intcNmiDummy;
    _CSL_intcCpuIntrTable.isr4 = _CSL_intcDispatcher;
    _CSL_intcCpuIntrTable.isr5 = _CSL_intcDispatcher;
    _CSL_intcCpuIntrTable.isr6 = _CSL_intcDispatcher;
    _CSL_intcCpuIntrTable.isr7 = _CSL_intcDispatcher;
    _CSL_intcCpuIntrTable.isr8 = _CSL_intcDispatcher;
    _CSL_intcCpuIntrTable.isr9 = _CSL_intcDispatcher;
    _CSL_intcCpuIntrTable.isr10 = _CSL_intcDispatcher;
    _CSL_intcCpuIntrTable.isr11 = _CSL_intcDispatcher;
    _CSL_intcCpuIntrTable.isr12 = _CSL_intcDispatcher;
    _CSL_intcCpuIntrTable.isr13 = _CSL_intcDispatcher;
    _CSL_intcCpuIntrTable.isr14 = _CSL_intcDispatcher;
    _CSL_intcCpuIntrTable.isr15 = _CSL_intcDispatcher;

    asm(" rint");"

We tried commenting out various parts of the above code. None of the changes we tried worked.

Do you have any more suggestions?

0 Steven Ji over 12 years ago in reply to Saurabh Kumar Singh

TI__Genius 12065 points

Saurabh,

Please take a look at the wiki page about configuring interrupts on KeyStone devices as follows:

http://ap-fpdsp-swapps.dal.design.ti.com/index.php/Configuring_Interrupts_on_Keystone_Devices

In "Software Implementation" section, it mentions that in the application, if both SYS/BIOS and CSL APIs are being used, there will be conflicts since both CSL and SYS/BIOS will assume that they own ISTP (Interrupt Service Table Pointer).

So it will be better to use only one method for the interrupt setup in the application.

Could you please try to configure the PCIe interrupt using BIOS or configure the EMAC interrupt using CSL please?

There is one SYS/BIOS example at the end of the above page. You probably can just change the system/host interrupt number to match it for PCIe interrupt and it is in BIOS format. Hope it could solve your issues by avoiding using both BIOS and CSL together.

0 Saurabh Kumar Singh over 12 years ago in reply to Steven Ji

Prodigy 235 points

Hi Steven,

Yes configuring PCIe interrupts using BIOS solved the issue.

Thank you

Processors

Processors forum

PCIe and NDK Integration