TMS320C6657: Non-broadcast Ethernet Packets Not Sent

Any ideas on how I can make progress on this issue?

Sorry for the delay here Christopher. We will need little more time to look at this issue in a little more detail since I am tied up with some other tasks. I have also looped in out networking expert to comment on your setup. 

Could you indicate why you need to update the ethBoot setting in IBL source to get TFTP to work. My understanding from the MCSDK/Processor SDK RTOS test log is that you should be able to follow the steps in here to generate the IBL config to boot using TFTP:

http://software-dl.ti.com/processor-sdk-rtos/esd/docs/latest/rtos/index_Foundational_Components.html#tftp-boot

Can you confirm that you tried these steps.

Regards,

Rahul

Let me walk through the steps in the document you cited and comment on each.

First issue I have is the title of the section:  4.6.3.4.6.3. TFTP Boot.  Based on my examination of the source code for this in the function c6657_ibl_config() located in the file "C:\ti_new\mcsdk_2_01_01_04\tools\boot_loader\ibl\src\util\iblConfig\src\device.c"

ibl.bootModes[2].u.ethBoot.doBootp          = TRUE;

The default method is Bootp not TFTP.  This was confirmed by my sniffing of the interface with Wireshark, both with a completely stock EVM and after I had rebuilt the EEPROM code but had not yet changed any parameters.  The reason I initially did not think to change any parameters was all the associated documentation, including the description of the EVM DIP Switch settings, would lead you to believe TFTP should be the mode used.  That was not true.

The linked document also goes on to say "EMAC boot is a multi-stage process which is designed to boot an application from TFTP server after reset."  That is also not true, by default as loaded on a stock EVM, the EMAC boot mode is Bootp.

Moving on to 4.6.3.4.6.3.1. Updating the IBL Ethernet Configurations.  Specifically the section Using CCS.  It is mentioned to run the GEL script for the device.  In my case the gel would be setConfig_c6657_main().  At the time I did that, I was not looking at the GEL itself, I was just running it as directed and it had no effect, the boot mode was still Bootp.  Now that I know more, examining the GEL file itself, you will see that for the C6657 Bootp is the mode selected.

ibl.bootModes[2].u.ethBoot.doBootp          = TRUE;

So, now it makes total sense why using the GEL as directed made no difference.  Of course, now in hindsight, the comment following the directions about modifying the GEL in a text editor has new meaning, but at the time, I just moved on to recompiling the IBL itself.

The section labeled Using iblConfig Utility Program is somewhat confusing to me even after working through the entire process.  It ultimately seemed to me that iblConfig was part of the overall make of the ibl itself.  So, I ended up making the entire ibl rather than making just the iblConfig.out.  Once I was able to get the ibl to build, I modified the source code for c6657_ibl_config() located in the file "C:\ti_new\mcsdk_2_01_01_04\tools\boot_loader\ibl\src\util\iblConfig\src\device.c"

ibl.bootModes[2].u.ethBoot.doBootp          = FALSE;

I wrote the newly compiled file i2crom_0x51_c6657_le.bin to the EEPROM using the eepromwriter utility.  Now when I booted the device I no longer saw Bootp messages, but I saw ARP Requests and Replies.  But, I saw no TFTP Requests.  This is when I really started digging into the code and was able to see the raw packet data being sent to the EMAC, saw the correct MAC Address which was not coming out on the wire, and saw that if I manually changed the MAC address in the buffer to FF's then the TFTP request was sent as described in my original message. 

For others reading this post, it is worth mentioning that if you look at the device.c file, which defines the parameters for the various boot modes, for all defined soc's, the default is Bootp, not TFTP.

    /* Ethernet configuration for Boot mode 0 */
    ibl.bootModes[0].bootMode = ibl_BOOT_MODE_TFTP;
    ibl.bootModes[0].priority = ibl_HIGHEST_PRIORITY;
    ibl.bootModes[0].port     = 0;

    /* Bootp is disabled. The server and file name are provided here */
    ibl.bootModes[0].u.ethBoot.doBootp          = TRUE;
    ibl.bootModes[0].u.ethBoot.useBootpServerIp = TRUE;
    ibl.bootModes[0].u.ethBoot.useBootpFileName = TRUE;
    ibl.bootModes[0].u.ethBoot.bootFormat       = ibl_BOOT_FORMAT_BBLOB;

Even though it says  ibl_BOOT_MODE_TFTP, having doBootp = TRUE forces the code to use Bootp instead of TFTP.  The comment in code even says "Bootp is disabled." but that is not true.

If you follow the code and see what the effect of bootp = TRUE is …

In ethboot.c

    if (ibl.bootModes[eIdx].u.ethBoot.doBootp == TRUE)
        nDevice.ip_address = 0;
    else
        nDevice.ip_address = htonl(nl);

So, ip_address is set to 0.  nDevice is then passed to net_open via this call:

    /* Open the network device */
    if ((*net_boot_module.open) ((void *)&nDevice, ibl_rec_params) != 0)
        return;

In the file net.c the function net_open() you have the code:

    /* Determine if we need to execute BOOTP or not? */
    if (netmcb.net_device.ip_address != 0)
    {
        /* IP Address was supplied by the device team. There is no need
         * to execute the BOOTP protocol; manually create the network route also. */
        ip_add_route (FLG_RT_NETWORK, netmcb.net_device.ip_address, netmcb.net_device.net_mask, 0);

        /* Optional call back with boot server info */
        if (asyncComplete != NULL)
            (*asyncComplete)((void *)&netmcb.net_device);

        /* Lets download the file from the TFTP Server. */
        tftp_get_file (netmcb.net_device.server_ip, netmcb.net_device.file_name);
    }
    else
    {
        /* No IP Address was supplied we need to execute the BOOTP protocol. */
        bootp_init (asyncComplete);
    }

So, as you can see, if the ip_address is 0, as it would be by default, the function bootp_init() is called not tftp_get_file.

Hi,

"This is when I really started digging into the code and was able to see the raw packet data being sent to the EMAC, saw the correct MAC Address which was not coming out on the wire, and saw that if I manually changed the MAC address in the buffer to FF's then the TFTP request was sent as described in my original message."

This doesn't make sense to me.  
    /* Fill out the transmit buffer descriptor */

    pDesc->pNext     = 0;
    pDesc->pBuffer   = (Uint8 *)deviceLocalAddrToGlobal((Uint32)buffer);
    pDesc->BufOffLen = num_bytes;
    pDesc->PktFlgLen = EMAC_DSC_FLAG_SOP | EMAC_DSC_FLAG_EOP | num_bytes | EMAC_DSC_FLAG_OWNER;

    /* Send the packet out. */
    ptr_EMACRegs->TX0HDP = (Uint32)pDesc;

The EMAC has no knowledge what inside the buffer, whether it is a broadcast packet with destination MAC of all FF, or a unicast packet to your PC's MAC. Between those two packets, what is the difference between the pDesc? Are they the same buffer location or same alignment? What is the BufOffLen? At the first time, it is supposed to send out a broadcast packet, if you replace the destination MAC with your PC's MAC, will you see it on wireshark? In the second time, it is supposed to send out a unicast packet (which you can't see on the wire), if you replace it with MAC with FF, will you see it?

Regards, Eric  

Just to dot the i's and cross the t's, I got a new C6657EVM which has not been reprogrammed or modified in any way.  I configured the DIP switches for TFTP boot:

SW3 (1-8) off, off, on, off, on, on, off, on

SW5 (1-8) on, on, on, off, on, on, on, on

As expected on wireshark I saw broadcast bootp requests.

I then followed the directions in the linked document under

4.6.3.4.6.3.1. Updating the IBL Ethernet Configurations - Using CCS

I used the files from pdk_c665x_2_0_14

I modified the gel as shown below for the C6657.

 ibl.bootModes[2].u.ethBoot.doBootp          = FALSE;
 ibl.bootModes[2].u.ethBoot.useBootpServerIp = FALSE;
 ibl.bootModes[2].u.ethBoot.useBootpFileName = FALSE;
 ibl.bootModes[2].u.ethBoot.bootFormat       = ibl_BOOT_FORMAT_BBLOB;

I loaded the program i2cparam_0x51_c6657_le_0x500.out.  Ran the program, ran the gel setConfig_c6657_main, then hit enter.

I power cycled the EVM and I saw no packets of any kind from the EVM.  No bootp, no Arp, no TFTP.  Nothing.

Thinking that pdk_c665x_2_0_14 might not be compatible with these older EVM's, I then repeated the procedure above using files from mcsdk_2_01_01_04.  Still no packets. 

I thought that maybe my initial attempt to modify the parameters using pdk_c665x_2_0_14 corrupted something, I then reloaded the entire I2C EEPROM from mcsdk_2_01_01_04, loading i2crom_0x51_c6657_le.bin into memory at 0x0c000000 and running the program eepromwriter_evm6657l.out.  I power cycled the EVM and once again I saw bootp messages.  I then repeated the "Using CCS" procedure using files from mcsdk_2_01_01_04 but the result was the same, no packets. 

The only success I've had is a complete rebuild of the IBL, modifying the device.c file, reprogramming the EEPROM using my rebuilt i2crom_0x51_c6657_le.bin.  Then I see ARP packets as expected, but no TFTP request.

Hello again Eric,

I completely agree, "This doesn't make sense to me."  LOL

I repeated my tests using my rebuilt ibl that does send ARP requests and at least attempts to send TFTP Requests.  One issue I had getting it to even send the ARP requests was memory caching.  It is worth noting that the file cpmacdrv.c which contains the code referenced by you above has the following:

#ifdef EMAC_CACHE_SUPPORT
    /* Clean the cache for external addesses */
    Cache_wbL2((void *)buffer, num_bytes);
#endif

However, I was not able to figure out how to get the ibl to compile with EMAC_CACHE_SUPPORT defined.  Cache libraries or objects are not part of the ibl.  So, I did the next best thing, I moved the buffer to non-cached memory.

So, when the arp request is sent, which is sent successfully, I see this for pDesc:

00000000
0C000666
00000040
E0000040

I believe that is exactly as expected.  The buffer is located at 0x0c000666.  Dumping that address you get the following:

FF FF FF FF FF FF 00 18 30 0A 0E 35 08 06 00 01
08 00 06 04 00 01 00 18 30 0A 0E 35 C0 A8 01 03
00 00 00 00 00 00 C0 A8 01 02 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

The first 6 bytes (FF FF FF FF FF FF) are the destination MAC address indicating broadcast packet.  The next 6 bytes (00 18 30 0A 0E 35) is the source MAC Address, which is the EVM as expected.  The next two bytes (08 06) is the packet type of ARP. 

Wireshark correctly captures this packets as soon as I step the line of code that moves the pDesc pointer to the TX0HDP, 

STW.D2T1 A6, *B6[0]

where A6 = 0x02C0A010 and B6 = 0x02C08600.

Since the host replies with an ARP reply, the next time this code runs, it is trying to send the TFTP Request packet.

Again, pDesc is exactly the same, same buffer, same flags, same length.

00000000
0C000666
00000040
E0000040

The buffer is this

90 E2 BA 6E 5E 1D 00 18 30 0A 0E 35 08 00 45 00
00 31 01 00 00 00 40 11 F6 66 C0 A8 01 03 C0 A8
01 02 04 D2 00 45 00 1D D9 1F 00 01 63 36 36 35
37 2D 6C 65 2E 62 69 6E 00 6F 63 74 65 74 00 00

Now you can see the destination mac address is 90 E2 BA 6E 5E 1D.  This matches the mac address of the host machine.  The source mac address is the same as the EVM, the packet type has changed to 0800 which is IPv4.  However, when I step code as before, I see nothing on wireshark. 

If I then break the code again, modify the destination mac address to all FF's.

Here is the packet as modified in memory at 0x0c000666:

FF FF FF FF FF FF 00 18 30 0A 0E 35 08 00 45 00
00 31 02 00 00 00 40 11 F5 66 C0 A8 01 03 C0 A8
01 02 04 D2 00 45 00 1D D9 1F 00 01 63 36 36 35
37 2D 6C 65 2E 62 69 6E 00 6F 63 74 65 74 00 00

 Wireshark does decode the TFTP Read request, it does show a Destination mac address of FF:FF:FF:FF:FF:FF.  The packet looks correct otherwise, no decoding errors.

You also asked what happens when I take the ARP request and modify the destination mac address to be the host mac address instead of all FF's.

90 E2 BA 6E 5E 1D 00 18 30 0A 0E 35 08 06 00 01  <- Arp
08 00 06 04 00 01 00 18 30 0A 0E 35 C0 A8 01 03
00 00 00 00 00 00 C0 A8 01 02 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Interestingly enough that worked.  So, that was a good test, unexpected by myself.  Thanks.  This means it is not just a mac address issue.

So, I have to admit I am more confused than ever now.  I did another test where I changed the FF's of the Arp Request to 11:22:33:44:55:66 and that also worked.  However, changing the TFTP request to 11:22:33:44:55:66 did not work. 

So, it seems like the combination of the IPv4 packet and a destination mac address other than FF's does not get transmitted. 

Here is the wireshark decode of the TFTP Request when I change the destination mac address to FF's.

Frame 183: 64 bytes on wire (512 bits), 64 bytes captured (512 bits) on interface 0
Ethernet II, Src: TexasIns_0a:0e:35 (00:18:30:0a:0e:35), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
    Source: TexasIns_0a:0e:35 (00:18:30:0a:0e:35)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 192.168.1.3, Dst: 192.168.1.2
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 49
    Identification: 0x0100 (256)
    Flags: 0x0000
    ...0 0000 0000 0000 = Fragment offset: 0
    Time to live: 64
    Protocol: UDP (17)
    Header checksum: 0xf666 [validation disabled]
    [Header checksum status: Unverified]
    Source: 192.168.1.3
    Destination: 192.168.1.2
User Datagram Protocol, Src Port: 1234, Dst Port: 69
    Source Port: 1234
    Destination Port: 69
    Length: 29
    Checksum: 0xd91f [unverified]
    [Checksum Status: Unverified]
    [Stream index: 21]
    [Timestamps]
Trivial File Transfer Protocol
    Opcode: Read Request (1)
    Source File: c6657-le.bin
    Type: octet
VSS-Monitoring ethernet trailer, Source Port: 0
    Src Port: 0

I should mention, the length is the same because there is a minimum length set in the code of 64.  So that makes sense.

Hi,

Thanks for the testing, it was interesting results! It looks that for combination of the IPv4 packet and a destination mac address other than FF's does not get transmitted. This would means that EMAC can't send out regular IPV4 unicast packets. I don't know how this can be.

I used another EMAC example (the EMAC_evmc6657_C66Loopback_testProject in Non-loopback mode, connected to network) to test the idea:

• I put the data buffer hard-coded at 0x0C00_0666
• I created a data dump exactly match your data buffer and loaded into this address

90 E2 BA 6E 5E 1D 00 18 30 0A 0E 35 08 00 45 00
00 31 01 00 00 00 40 11 F6 66 C0 A8 01 03 C0 A8
01 02 04 D2 00 45 00 1D D9 1F 00 01 63 36 36 35
37 2D 6C 65 2E 62 69 6E 00 6F 63 74 65 74 00 00

• I was able to capture such packet on wire:

In your failure case, are you able to check if the EMAC statistics for Tx packet increase? 8.15 Good Transmit Frames Register (TXGOODFRAMES) offset 0x34?

And if you saw any error in MACSTATUS (0x02c08164)?

Regards, Eric

Hi,

My second screenshot was wrong, please see this:

Sorry for the confusion!

Regards, Eric

I stepped through the code as before, monitoring the status registers you mentioned.  For each iteration, the sequence is this,

Send Broadcast ARP

Send Unicast Tftp Request (attempt 1)

Send Unicast Tftp Request (attempt 2)

Send Unicast Tftp Request (attempt 3)

Send Unicast Tftp Request (attempt 4)

Send Unicast Tftp Request (attempt 5)

After each Send I see TXGOODFRAMES increment by 1 until the value 6 is reached, then the process starts over with a EMAC Reset I believe and the count restarts at 0.

Looking at MACSTATUS, 0x02c08164, I initially saw 0x80000018, which is IDLE, and EXTGIG and EXTFULLDUPLEX.  No errors.

So, that gave me an idea, maybe it is related to running the interface at 1G, so I dropped back to 100M, MACSTATUS changed to 0x80000000, but the problem remained and the other registers behaved the same.

Hi,

As the TXGOODFRAMES has increment and no error was found, the EMAC is assumed to send out the packets at the switch level. The EMAC has no knowledge what data is inside the buffer and will not treat different packet types differently.

Do you know the Wireshark or PC environment have no issue to capture those packets? Do you have a direct connection or there is switch or router in between and possibly dropped the packet? 

Regards, Eric

Hi,

Also, if you don't change the IBL code, so it is bootp case. It worked or not? If it did, you would see a few bootp packets came out of the EVM, then did you see any unicast packet after that?

Regards, Eric

I agree this should just work.  I'm not ruling out the possibility of it being somewhere outside the EVM, I was hoping someone at TI might have more experience with BootROM/IBL TFTP and EVMs and could comment.

I've tried it both ways, connected to a switch and direct from EVM to PC.  Behavior is the same, ARP works but TFTP Request is not seen.  I initially thought the C6657 had a Ethernet Switch built into it.  I guess other Keystone devices do, but this one does not.  It appears the EMAC is connected directly to the PHY.  I just don't understand how the PC could filter out these packets when wireshark can normally see packets not addressed to the PC itself, but on the wire.  I'll keep poking at it, maybe I'll stumble across something.

For the second post above, we have had the system working successfully using the ROM Ethernet Boot mode.  This is not the IBL mode, just the straight up ROM mode.  That uses Bootp.  Bootp are broadcast.  After the server sees the Bootp it send a stream of UDP packets to the DSP which does not reply, UDP is unacknowledged.  We use this to load a bootloader to internal SRAM, which configures DDR RAM and then I re-used the ROM code as my second level bootloader, so it does the Bootp again, UDP's the actual program to DDR.  Once that is complete, the NDK takes over and we have full two Ethernet comms between the DSP and host.  However, in this case the host is a Linux board.  Once the DSP is booted, we have the DSP send a TFTP request to the host to download an FPGA image.  That also works.  That should be unicast. I'll make sure I can see all that in wireshark from the PC using the switch to monitor the EVM/Host connection.

Hi,

As you mentioned the C6657 EMAC is not a switch, it doesn't have ALE (Address lookup engineer). Other Keystone devices (like C6678 or K2H or K2E) may have a 3/5//9-port switch and direct a packet to another port.

And thanks very much for dig into the EMAC boot so deep between ROM and IBL ways and hope you have good results next week. 

Regards, Eric

Hi,

Are you still working on this? Any news?

Regards, Eric