Linux/AM5K2E02: Enable more then two 1Gbe interfaces on linux kernel 4.4 for k2e

Hey Rex,

on the k2e Evalkit and on our customer board I get following error message if I bring up all network interfaceses:

[   35.817510] keystone-navigator-qmss soc:qmss@2a40000: out of descs in region(12) for pool(tx-pool-0-eth0)
[   35.827107] net eth0: Couldn't create tx-pool-0-eth0 tx pool 0
[   35.865522] net eth0: Failed to setup navigator resources
ifconfig: SIOCSIFFLAGS: Cannot allocate memory

I have following interfaces eth0 to eth3 (two GBE and two XGBE)

Can you help me, what is wrong?

The Evalkit use the following devicetree.

6052.keystone-k2e-evm.txt

Hi, Sebastian,

The tx descriptors are supposed to be recycled and released back to the queue. It seems that one of the packets got stucked in the driver and the subsequent packets got queued up and finally out of descriptors. Do you see some error messages earlier before this out of descs message?

Rex

Hey Rex,

I see no error messages earlier.

On the Evalboard, when the interface eth1 is up and then I want to bring up eth0 I get this error messages.

But when I bring eth1 down I can bring up eth0 up without error messages.

Best regards,

Sebastian

Hi, Sebastian,

If issue only happens when you bring up one more interface, it sounds to me that the descriptors in region-12 are all used up. Could you try to increase the descriptors?

Rex 

Hey Rex,

I found several issues to bring up six 1G and two 10G ethernet interfaces on the device-tree.

I will upload a device-tree to see what has to be changed.

Issues:

1. A interrupt number is assigned twice (number 59).

2. The descriptor size under qmss has to be increased.

3. The tx and rx queue number has to be adapted between ethernet and crypto

4. The qrange under qmss hast to be increased.

8228.k2e-device-tree-for-ethernet.txt

Hi, Sebastian,

could you show me which 2 nodes has interrupt 59 assigned twiec? I see queue conflicts with crypto. We may need to move crypto to other queue if we still have queue available. I think 2 and 4 are specific to your environment. Your configuration seems to need more descriptors.

Rex

Hey Rex,

look to file keystone-k2e-netcp.dts.

You see following configuration:

            qpend-1 {
                qrange = <528 16>;
                interrupts = <0 48 0xf04 0 49 0xf04 0 50 0xf04
                          0 51 0xf04 0 52 0x104 0 53 0x204
                          0 54 0xf04 0 55 0xf04 0 56 0xf04
                          0 57 0xf04 0 58 0xf04 0 59 0xf04
                          0 60 0xf04 0 61 0xf04 0 62 0xf04
                          0 63 0xf04>;
                qalloc-by-id;
            };
            qpend-2 {
                qrange = <544 16>;
                interrupts = <0 64 0xf04 0 65 0xf04 0 66 0xf04
                          0 59 0xf04 0 68 0xf04 0 69 0xf04
                          0 70 0xf04 0 71 0xf04 0 72 0xf04
                          0 73 0xf04 0 74 0xf04 0 75 0xf04
                          0 76 0xf04 0 77 0xf04 0 78 0xf04
                          0 79 0xf04>;
            };

I make the twice configuration fat.

Hi, Sebastian,

I see. Yes, that should be avoid.

I ran through TI's cofiguration when all 8 GbE ports are enabled on K2H, and I will modify the NetCP User's Guide to address the reuse of the queues. In TI's example configuration, qpend-1 has 16 queues starting at 528 and qpend-2 also has 16 queues starting at 544. You can see in the document that XGE ports gave up their queues in qpend-1 and move to qpend-2 to accomodate 8 ethernet ports, . The bottom line is that all rx-queues and tx-completion-queues for GbE ports and for XGE ports need to be in consecutive order.

The rx-queue of 8 GbE ports take 528, 529, .., to 535, and tx-completion queue take 536-543.

The rx-queue of 2 XGE ports take 544 and 545, the tx-completion-queue take 546 and 547.

If you need more queues, you need to increase the queue pools, and the number of queues in the qpend, and follow the rx-queue and tx-completion-queue assignment rules.

Rex

Hey Rex,

what you say is right.

I write you this information so that other people know how to configure the ethernet part.

I need two days for driver debugging to get the right configuration of the device tree.

I'm disappointed about the quality of the documentation. It is nice when you update the documentation.

Regards,

Sebastian